[01:02] <jsalisbury> hallyn, ack.  thanks for the heads up
[07:42]  * apw yawns
[07:59] <tjaalton> I guess we don't have debugsym builds for the lts backport kernels?
[07:59] <tjaalton> *dbgsym
[08:03] <infinity> tjaalton: We should do.
[08:04] <infinity> tjaalton: http://ddebs.ubuntu.com/pool/main/l/linux-lts-raring/
[08:04] <infinity> tjaalton: http://ddebs.ubuntu.com/pool/main/l/linux-lts-quantal/
[08:04] <tjaalton> infinity: oh of course..
[08:05] <tjaalton> different source package
[08:09] <tjaalton> just to clarify, should this work when a dbgsym package is installed? http://stackoverflow.com/questions/6151538/addr2line-on-kernel-module
[08:09] <tjaalton> what I'm trying is to get the line of code that matches an oops trace
[08:10] <tjaalton> the gdb approach just says 'no debugging symbols'
[08:12] <tjaalton> nevermind, used the wrong argument.. needed the one in /usr/lib/debug/lib/modules/..
[12:14]  * henrix -> lunch
[14:25] <tjaalton> guess there are no dbgsym packages for mainline kernels?
[15:03] <tjaalton> would be awesome to have those
[16:01] <hallyn> hey - is there a signalfd+epoll wizard here?
[16:01] <hallyn> If I run http://people.canonical.com/~serge/signalfd.c 10 times very quickly, one of the runs will end up hanging because main never gets an epoll event for the sigchild from the first cloned child
[16:02] <hallyn> the clone is done after the signalfd is set up, so I would expect epoll to fire for that child's sigchld
[16:02] <hallyn> (when it hangs, the cloned child and its own child both are <defunct>, waiting for parent to reap them...)
[16:06] <apw> hallyn, you know you cannot do IO in a signal handler right ?
[16:06] <apw> hallyn, or indeed very little else
[16:06] <hallyn> apw: but this isn't a real signal handler...
[16:06] <hallyn> despite the name :)
[16:07] <hallyn> it runs from main() in the epoll loop
[16:07] <apw> oh heh
[16:08] <hallyn> but i'm hoping someone can point out what i'm doing wrong there, before i go to lkml to present it as a possible bug...  somewher ein that stack...
[16:08] <apw> hallyn, so you clone the children the make the epoll thingy, why is that not racy
[16:09] <apw> oh i see you make the signalfd first
[16:09] <hallyn> oh yeah, lemme change that order,
[16:09] <apw> though i would expect that to behave right i think :)
[16:09] <hallyn> yeah, but maybe i need to make the epollfd and attach the signalfd first.  Mind you, that *is* being done right int he code I'm trying to reproduce :)
[16:12] <hallyn> I also apparently inserted the clone block twice
[16:13] <apw> heh
[16:13] <apw> post up the fixed version when you have it :)
[16:13] <hallyn> well, the double clone seems to be the trigger
[16:14] <hallyn> updated http://people.canonical.com/~serge/signalfd.c
[16:18] <apw> hallyn, what the heck is the code trying to even do at the bottom
[16:18] <apw> it seems to be using the number of live fds reported to loop over the signal handler
[16:18] <hallyn> apw: it's waiting on an epoll event for the signalfd fd, and calling 'signal_handler' on the fd when it arrives.
[16:19] <apw> yeah but looping over 0 -> nfds and reading for each doesn't make sense
[16:19] <apw> as only one of them is a signafs
[16:19] <hallyn> apw: admittedly since i only have one fd the loop seems silly - the more generalized case is in src/lxc/mainloop.c
[16:19] <apw> signalfd anyhow, and the number returned is not related to the number of signals ... is it ?
[16:20] <hallyn> right, the number returned is # available fds,
[16:20] <hallyn> and yes that can only ever be 1
[16:20] <apw> so for you yes, but for them?
[16:20] <apw> if this code you have is cut down from theirs i mean
[16:20] <hallyn> right, there they also add console fds and such
[16:20] <apw> it seems to be reading the signalfd once per the number of fds they have awake
[16:20] <apw> that doesn't make sense really does it?
[16:21] <hallyn> no they call a different handler for each fd type
[16:21] <hallyn> (one sec, ...)
[16:22] <hallyn> apw: see mainloop.c in https://github.com/lxc/lxc/blob/staging/src/lxc/mainloop.c
[16:23] <apw> ahh that makes more sense
[16:24] <hallyn> apw: if you lxc-start a container with a init which just does exit(0), the mainloop handler often hangs, while init sits around defunct.  i'm trying to reproduce that... 
[16:25] <hallyn> i'll pull out the inner loop since it only obfuscates the problem
[16:26] <apw> hallyn, well the original code does have a case (i think) where it can hide an event coming in
[16:27] <apw> if the handler return non-zero it will silently go back to sleep, it might be worth instrumenting when it does things like that
[16:30] <hallyn> apw: right, the handler returns 1 if the container init's sigchild was received, 0 otherwise
[16:30] <hallyn> apw: by 'to sleep' you mean wiating on an epoll event right?
[16:30] <apw> hallyn, yes to sleep == called epoll_wait and blocks again
[16:30] <hallyn> apw: I was wondering last night whether the epoll event came in for two siganls, and i was only reading one - but trying another read doesn't seem to help,
[16:31] <hallyn> and near as I can tell epoll_wait should return again if i haven't consumed all the data
[16:31] <apw> well in theory if there was two, and you read one and poll again, it should wait again ... yes that
[16:31] <hallyn> right - unless i was doing edge triggered, but i'm not (EPOLL_ET or whatever)
[16:34] <apw> hallyn, so the differnce in your code is you should really be doing
[16:34] <apw> you should be running 'all' the ones it returns and doing
[16:34] <apw> if (event[N].data == signal_handler)
[16:35] <apw>   signal_handler()
[16:35] <apw> just in case it is returning something else
[16:35] <apw> if there was a bug in which it returned the wrong data, you wouldn't see it with your code as is
[16:36] <hallyn> wella ctually epoll-wait(0 seems to be always returning 0, even when it has data.  (huh?)
[16:38] <apw> well if it returned 0 ther should be no data
[16:38] <apw> you might want to seed the events with poison to confirm
[16:41] <hallyn> easy enough, when it hangs, just hit ctrl-c :)
[16:44] <apw> for me this code does behave odd
[16:44] <apw> oh this is the old old code
[16:44] <hallyn> hm, i see, 0 means don't wait, that's why i'm getting 0 often.
[16:45]  * hallyn tries with timeout -1
[16:45] <apw> yeah thats non-blocking
[16:47] <hallyn> apw: verifying data ... actually may fix this!  haven't reproduced so far
[16:47] <hallyn> updated http://people.canonical.com/~serge/signalfd.c
[16:47] <hallyn> nope, here we go
[16:48] <apw> how often do you run it to get a hang
[16:49] <hallyn> with the updated code, 100 times
[16:50] <apw> for me i got one where it say 'got an event' a lot over and over
[16:51] <apw> got an event (nfds 1)
[16:51] <apw> bad epoll event!
[16:52] <hallyn> right
[16:53] <hallyn> so why is there a bad epoll event.  weird
[16:53] <hallyn> am i not initializing something before setting up?
[16:54]  * apw pokes, at least this would explain the hang
[16:54] <apw> i think
[16:54] <apw> anyhow, poking away, as i can repro it as well
[16:55] <hallyn> doh
[16:56] <hallyn> apw: nm, that's a bogus one.  i is never set :)
[16:56] <hallyn> before we check events[i].data.ptr
[16:56] <apw> hallyn, heh indeed, doh
[16:56] <apw> and i just got it to hang witht hat fixed
[16:56] <hallyn> double-doh, though - fixing it (new http://people.canonical.com/~serge/signalfd.c )
[16:57] <hallyn> yeah
[16:57] <hallyn> hangs immediately
[16:57] <hallyn> what's an ascii emoticon for tortured agonizing troll?
[17:21] <hallyn> apw: well, to my surprise, even replacing clone with fork eventually reproduces it.  that's good i guess.
[17:22] <apw> hallyn, what are you testing on
[17:22] <apw> which release
[17:22] <apw> hallyn, as it seems reproducible on 3.8 (whatever this is)
[17:23] <hallyn> apw: yeah, raring.  oh, snap, i thought my other vm was saucy, but no i've only tested on raring
[17:23] <hallyn> wait a sec, i'm marking nonblock wrongly, perhaps
[17:24] <apw> hallyn, same here
[17:26] <apw> are you marking anything noblock at all?
[17:27] <hallyn> apw: yeah (refresh http://people.canonical.com/~serge/signalfd.c  :)
[17:27] <hallyn> apw: i was doing it with fcntl for awhile, but then saw that signalfd takes a SFD_NONBLOCK flag
[17:27] <hallyn> anyway, doesn't fix it
[17:27] <hallyn> so the signal really really does seem to just get lost
[17:28] <hallyn> ok lemme try on saucy
[17:28] <hallyn> or, uh, precise
[17:29] <hallyn> happens same way on precise
[17:29] <apw> hallyn, so it always hangs when it gets the other exit first right?
[17:30] <hallyn> yeah
[17:30] <hallyn> apw: no
[17:31] <hallyn> sometimes it gets the other exit first and still succeeds
[17:31] <hallyn> but every time it fails it does first get the other exit
[17:31]  * apw isn't seeing that, but it is hanging very quick for me
[17:31] <hallyn> apw: http://paste.ubuntu.com/5840986/  here is an example
[17:32] <apw> it does feel like it is behaving like it is ET
[17:33] <apw> yeah
[17:34] <hallyn> apw: but then i'd expect my looping of the read to fix it
[17:37] <apw> yep that we would
[17:39] <hallyn> apw: d'oh.  
[17:39] <apw> hallyn, ?
[17:39] <hallyn> signalfd() does not do flags.
[17:39] <hallyn> wonder if signalfd() library call uses signalfd4(0 syscall
[17:40] <apw> oh
[17:40] <hallyn> yeah strace says it uses signalfd4
[17:40] <apw> and mine is seemingly non-b
[17:40] <apw> blocking
[17:40] <apw> so doesn't that mean it does work
[17:41] <apw> waiting ...
[17:41] <apw> got an event (nfds 1)
[17:41] <apw> got sig for 20171, init was 20173
[17:41] <apw> failed to read signal info errno=11
[17:41] <apw> failed to read signal info errno=11
[17:41] <apw> waiting ...
[17:41] <apw> for me it does that, on a hang, so it saw the wrong pid, read 3 times (two failed cause nothing there) and then we went back
[17:42] <hallyn> but why 2 in a row failures?
[17:42] <hallyn> it should only fail once, then return to the epoll-wait loop
[17:43] <hallyn> apw: that printf looks differnet from mine
[17:43] <hallyn>                 perror("read");
[17:43] <hallyn>                 printf("errno is %d\n", errno);
[17:43] <hallyn>                 if (errno == EAGAIN || errno == EWOULDBLOCK)
[17:43] <hallyn>                         return 0;
[17:43] <hallyn>                 printf("failed to read signal info");
[17:43] <apw> yeah it is mine indeed
[17:43] <hallyn> ok
[17:43] <hallyn> i'm tempted to ping mkerrisk
[17:45] <hallyn> mind you the corresponding lxc bug has been somewhat low priority for me, but i thought we were doing something more wrong than this...  if it's a kernel or libc bug i think it's a bigger deal
[17:49] <apw> hallyn, i'll keep poking for a bit too
[17:51] <apw> hallyn, of course it might be a bug in signalfd not epoll, hmm
[17:56] <hallyn> apw: yup
[17:56] <hallyn> apw: that's why i was happy that fork could reproduce it - at least it doesn't involve CLONE_NEWPID as well :)
[17:57] <apw> hallyn, and it might be an alignment issue, as it is random and we have address psace randomisation on
[17:57] <apw> as SIGCLD is one of the large ones in that structure
[17:58] <apw> so if the structure was too small, and aligned bad or something
[17:58] <apw> or too big or something
[17:58] <apw>         struct signalfd_siginfo __user *siginfo;
[17:58] <apw> if that isn't the same size as the one in the kernel ... we might fail
[17:59] <apw> as if we dequeue it and then fail to return it ... we just lose it
[18:06] <apw> hallyn, what makes you think (i assume you have docs somewhere) that just because you are using this interface that you can guarentee to see all signals ?
[18:07] <apw> hallyn, as you say when it hangs you can see that you have pending children events
[18:07] <apw> hallyn, are you sure you can expect to see both signals as separate events with separate children ?
[18:07] <apw> hallyn, would i not expect you to actuall ignore the siginfo and use wait to get the childre
[18:07] <apw> hallyn, and there i htink i would expect you to get both as they are there waiting to be waited for
[18:08] <apw> hallyn, ie reading signalfd is _not_ waiting for them nor reporting them
[18:10] <apw> hallyn, and you need to wait to clean them up regardless
[18:13] <hallyn> interesting
[18:13] <hallyn> so i have to waitpid() on the first to clear it up before i can get a sigfd for the second sigchld?
[18:14] <hallyn> lemme try
[18:14] <hallyn> it makes sense
[18:14] <hallyn> (and it briefly came to mind last night, but i figured "nah!" :)
[18:15] <apw> hallyn, no i am not saying you have to waitpid to clear it
[18:15] <apw> hallyn, i am saying that signalfd is reporting pending signals
[18:16] <apw> and pending signals with their 'latest' siginfo
[18:16] <apw> but if you get two signals of the same type only the last one will be reported
[18:16] <apw> that is normal for signals and siginfo
[18:16] <apw> i am syaing when you get a signal, you should be looking on wait (with NOWAIT) and using
[18:16] <hallyn> i see
[18:16] <apw> the data it returns
[18:16] <apw> as the pids not anything int he siginfo.
[18:17] <hallyn> so i (and the original author) am (is) completely misunderstanding sigchld delivery
[18:17] <apw> as we see both as zombies i bet money that you will find two waits
[18:17] <hallyn> i did indeed assume i'd get N sigchlds
[18:17] <apw> that is not normal with signals at the process level for sure
[18:17] <apw> now signalfd could have either semantic and it does not say which
[18:18] <apw> but if we conjecture it has the smae semantics as signals
[18:18] <apw> and the way POLLIN is calculated i suspect it is
[18:18] <apw> then you may not indeed, but a looping wait should always be sage
[18:18] <apw> safe, and i think you need to be waiting anyhow to clean up the carcasses
[18:18] <apw> so if i am right, a wait loop will repair things, if it is broken
[18:19] <apw> then it will only ever have one and make no differende
[18:23] <hallyn> apw: thanks - final signalfd.c testcase uploaded - this one doesn't seem to reproduce.  seems you were right!
[18:23] <hallyn> now i can fix lxc :)
[18:23] <hallyn> \o/
[18:23] <hallyn> thanks apw
[18:23] <apw> hallyn, heh... there is point in having a kernel team after all :)
[18:23] <apw> hallyn, and i am glad we got to the bottom of it
[18:24]  * hallyn goes to take a walk outside before writing the actual patches
[18:26] <apw> hallyn, i know the feeling, i think i need a beer after thinking that hard
[18:27] <hallyn> that sounds.  good.
[18:29] <apw> hallyn, ok .. you need to break that loop for == 0 i think
[18:29] <apw> as it spins in there till they exit otherwise
[18:30] <apw> hallyn, and waitid() may be a more natural fit for your code base maybe
[18:30] <apw> hallyn, switching your loop from >= 0 to > 0 still seems to work
[18:30] <apw> and removes the repeated looping returning 0
[18:31] <apw> got sig for 2791, init was 2793
[18:31] <apw> waitpid ret=2791
[18:31] <apw> waitpid ret=2793
[18:31] <apw> got init's sigchld: done
[18:31] <apw> hallyn, and we see the pattern there which i conjectured .. coool
[18:31] <apw> beer it is
[19:08] <hallyn> apw: (drat, no beer for me) do you have a copy of your final one you can post?
[19:08] <hallyn> curious which loop you ended up going with
[19:09] <hallyn> apw: oh, right, i misread the == 0 return value case!  lol
[19:09] <hallyn> "... in any state" well that does me no good :)
[19:10] <hallyn> waitid is too foreign to me :)  i'll muck it up somehow
[19:15] <apw> hallyn, that version of yours looked ok once it was >= 0 :)
[19:15] <apw> > 0 even
[19:17] <hallyn> getting ready to test the lxc ase
[19:17] <hallyn> lxc case
[19:25] <hallyn> hm, not a slam dunk fix.
[19:27] <hallyn> odd 
[19:36] <apw> hallyn, ?
[19:39] <hallyn> apw: added the waitpid loop into signal_handler in lxc, but lxc-start still ends up with defunct init 
[19:39] <hallyn> so i'm missing something else that's going wrong there
[19:41] <apw> sounds like a tommorrow fun
[19:42] <hallyn> tomorrow's a holiday here - but ttyl
[19:43] <apw> hallyn, drop me a line if you have any other examples to play with and i can look at it tommorrow
[19:44] <hallyn> thx
[19:46] <seb128> hey
[19:46] <seb128> is there any issue with the saucy 3.10 kernel and screen light?
[19:47] <seb128> changing it using the multimedia keys hangs my latitude laptop for a second
[19:47] <seb128> where it works fine with 3.9
[19:50] <apw> seb128, noting i have seen reported
[19:50] <seb128> apw, hum, ok
[19:51] <seb128> apw, is there any special info that would be useful in a bug report?
[19:52] <apw> seb128, h/w info which ubuntu-bug will collect
[19:52] <seb128> ok
[19:52] <seb128> that's a latitude e6410
[20:19] <hallyn> apw: well huh - particularly interesting in the lxc case is that after i call the parent task, the [init] task remains defunct, while parented by 1
[20:19] <hallyn> *that* probably shouldn't happen
[20:20] <apw> hallyn, if you reap a parent before its child then its children go to init don't they ?
[20:20] <apw> you could test that
[20:20] <apw> with a basic fork test
[20:23] <hallyn> apw: (not sure i was clear)  i'm saying:  root      1878     1  0 21:32 pts/2    00:00:00 [init] <defunct>                                   
[20:23] <hallyn> '[init]'is the container init, which has exited;  it's now parented by the host init, but is still rockin' the zombie life
[20:23] <apw> right so noone has waited for it
[20:23] <apw> have you waited for its parent
[20:24] <hallyn> bc it's not sending a new sigchld?
[20:24] <apw> that is not what i mean
[20:24] <hallyn> i figured init would reap zombies regardless of whether it got a sigchld
[20:24] <apw> oh i see
[20:24] <hallyn> i killed its parent, and the parent is gone.
[20:24] <apw> right so yes it should jump to real init, and that should be reaping it 
[20:25] <apw> that is its job
[20:26] <hallyn> all right i'd better step away a few hours.  sorry if i pulled you away from that beer :)
[20:27] <apw> hallyn, that seems wrong, and in the first place i would be blaming init itself, which we cannot trace easily hmmm, i wonder if --verbose would give us any info
[20:29] <hallyn> apw: but as i say if the exiting task has already sent its sigchld, then gets reparented to init, then init won't get a sigchld - 
[20:29] <hallyn> i'd actually expect the kernel at that point, instead of reparenting, to just reap ...  
[20:29] <hallyn> (i guess that's nonsense-talk)
[20:30] <hallyn> so upstart would need to walk its pid list and check for zombies... which is not efficient.  still, yeah, cleaning up after bad users is its job...
[20:30] <hallyn> this should be trivial to write a test case for :)  will do so later