jsalisburyhallyn, ack.  thanks for the heads up01:02
=== fmasi_afk is now known as fmasi
=== fmasi is now known as Guest32901
=== elmo__ is now known as elmo
* apw yawns07:42
tjaaltonI guess we don't have debugsym builds for the lts backport kernels?07:59
infinitytjaalton: We should do.08:03
infinitytjaalton: http://ddebs.ubuntu.com/pool/main/l/linux-lts-raring/08:04
infinitytjaalton: http://ddebs.ubuntu.com/pool/main/l/linux-lts-quantal/08:04
tjaaltoninfinity: oh of course..08:04
tjaaltondifferent source package08:05
tjaaltonjust to clarify, should this work when a dbgsym package is installed? http://stackoverflow.com/questions/6151538/addr2line-on-kernel-module08:09
tjaaltonwhat I'm trying is to get the line of code that matches an oops trace08:09
tjaaltonthe gdb approach just says 'no debugging symbols'08:10
tjaaltonnevermind, used the wrong argument.. needed the one in /usr/lib/debug/lib/modules/..08:12
=== yofel_ is now known as yofel
* henrix -> lunch12:14
=== Guest32901 is now known as fmasi
=== ayan_ is now known as ayan
tjaaltonguess there are no dbgsym packages for mainline kernels?14:25
tjaaltonwould be awesome to have those15:03
hallynhey - is there a signalfd+epoll wizard here?16:01
hallynIf I run http://people.canonical.com/~serge/signalfd.c 10 times very quickly, one of the runs will end up hanging because main never gets an epoll event for the sigchild from the first cloned child16:01
hallynthe clone is done after the signalfd is set up, so I would expect epoll to fire for that child's sigchld16:02
hallyn(when it hangs, the cloned child and its own child both are <defunct>, waiting for parent to reap them...)16:02
apwhallyn, you know you cannot do IO in a signal handler right ?16:06
apwhallyn, or indeed very little else16:06
hallynapw: but this isn't a real signal handler...16:06
hallyndespite the name :)16:06
hallynit runs from main() in the epoll loop16:07
apwoh heh16:07
hallynbut i'm hoping someone can point out what i'm doing wrong there, before i go to lkml to present it as a possible bug...  somewher ein that stack...16:08
apwhallyn, so you clone the children the make the epoll thingy, why is that not racy16:08
apwoh i see you make the signalfd first16:09
hallynoh yeah, lemme change that order,16:09
apwthough i would expect that to behave right i think :)16:09
hallynyeah, but maybe i need to make the epollfd and attach the signalfd first.  Mind you, that *is* being done right int he code I'm trying to reproduce :)16:09
hallynI also apparently inserted the clone block twice16:12
apwpost up the fixed version when you have it :)16:13
hallynwell, the double clone seems to be the trigger16:13
hallynupdated http://people.canonical.com/~serge/signalfd.c16:14
apwhallyn, what the heck is the code trying to even do at the bottom16:18
apwit seems to be using the number of live fds reported to loop over the signal handler16:18
hallynapw: it's waiting on an epoll event for the signalfd fd, and calling 'signal_handler' on the fd when it arrives.16:18
apwyeah but looping over 0 -> nfds and reading for each doesn't make sense16:19
apwas only one of them is a signafs16:19
hallynapw: admittedly since i only have one fd the loop seems silly - the more generalized case is in src/lxc/mainloop.c16:19
apwsignalfd anyhow, and the number returned is not related to the number of signals ... is it ?16:19
hallynright, the number returned is # available fds,16:20
hallynand yes that can only ever be 116:20
apwso for you yes, but for them?16:20
apwif this code you have is cut down from theirs i mean16:20
hallynright, there they also add console fds and such16:20
apwit seems to be reading the signalfd once per the number of fds they have awake16:20
apwthat doesn't make sense really does it?16:20
hallynno they call a different handler for each fd type16:21
hallyn(one sec, ...)16:21
hallynapw: see mainloop.c in https://github.com/lxc/lxc/blob/staging/src/lxc/mainloop.c16:22
apwahh that makes more sense16:23
hallynapw: if you lxc-start a container with a init which just does exit(0), the mainloop handler often hangs, while init sits around defunct.  i'm trying to reproduce that... 16:24
hallyni'll pull out the inner loop since it only obfuscates the problem16:25
apwhallyn, well the original code does have a case (i think) where it can hide an event coming in16:26
apwif the handler return non-zero it will silently go back to sleep, it might be worth instrumenting when it does things like that16:27
hallynapw: right, the handler returns 1 if the container init's sigchild was received, 0 otherwise16:30
hallynapw: by 'to sleep' you mean wiating on an epoll event right?16:30
apwhallyn, yes to sleep == called epoll_wait and blocks again16:30
hallynapw: I was wondering last night whether the epoll event came in for two siganls, and i was only reading one - but trying another read doesn't seem to help,16:30
hallynand near as I can tell epoll_wait should return again if i haven't consumed all the data16:31
apwwell in theory if there was two, and you read one and poll again, it should wait again ... yes that16:31
hallynright - unless i was doing edge triggered, but i'm not (EPOLL_ET or whatever)16:31
apwhallyn, so the differnce in your code is you should really be doing16:34
apwyou should be running 'all' the ones it returns and doing16:34
apwif (event[N].data == signal_handler)16:34
apw  signal_handler()16:35
apwjust in case it is returning something else16:35
apwif there was a bug in which it returned the wrong data, you wouldn't see it with your code as is16:35
hallynwella ctually epoll-wait(0 seems to be always returning 0, even when it has data.  (huh?)16:36
apwwell if it returned 0 ther should be no data16:38
apwyou might want to seed the events with poison to confirm16:38
hallyneasy enough, when it hangs, just hit ctrl-c :)16:41
apwfor me this code does behave odd16:44
apwoh this is the old old code16:44
hallynhm, i see, 0 means don't wait, that's why i'm getting 0 often.16:44
* hallyn tries with timeout -116:45
apwyeah thats non-blocking16:45
hallynapw: verifying data ... actually may fix this!  haven't reproduced so far16:47
hallynupdated http://people.canonical.com/~serge/signalfd.c16:47
hallynnope, here we go16:47
apwhow often do you run it to get a hang16:48
hallynwith the updated code, 100 times16:49
apwfor me i got one where it say 'got an event' a lot over and over16:50
apwgot an event (nfds 1)16:51
apwbad epoll event!16:51
hallynso why is there a bad epoll event.  weird16:53
hallynam i not initializing something before setting up?16:53
* apw pokes, at least this would explain the hang16:54
apwi think16:54
apwanyhow, poking away, as i can repro it as well16:54
hallynapw: nm, that's a bogus one.  i is never set :)16:56
hallynbefore we check events[i].data.ptr16:56
apwhallyn, heh indeed, doh16:56
apwand i just got it to hang witht hat fixed16:56
hallyndouble-doh, though - fixing it (new http://people.canonical.com/~serge/signalfd.c )16:56
hallynhangs immediately16:57
hallynwhat's an ascii emoticon for tortured agonizing troll?16:57
hallynapw: well, to my surprise, even replacing clone with fork eventually reproduces it.  that's good i guess.17:21
apwhallyn, what are you testing on17:22
apwwhich release17:22
apwhallyn, as it seems reproducible on 3.8 (whatever this is)17:22
hallynapw: yeah, raring.  oh, snap, i thought my other vm was saucy, but no i've only tested on raring17:23
hallynwait a sec, i'm marking nonblock wrongly, perhaps17:23
apwhallyn, same here17:24
apware you marking anything noblock at all?17:26
hallynapw: yeah (refresh http://people.canonical.com/~serge/signalfd.c  :)17:27
hallynapw: i was doing it with fcntl for awhile, but then saw that signalfd takes a SFD_NONBLOCK flag17:27
hallynanyway, doesn't fix it17:27
hallynso the signal really really does seem to just get lost17:27
hallynok lemme try on saucy17:28
hallynor, uh, precise17:28
hallynhappens same way on precise17:29
apwhallyn, so it always hangs when it gets the other exit first right?17:29
hallynapw: no17:30
hallynsometimes it gets the other exit first and still succeeds17:31
hallynbut every time it fails it does first get the other exit17:31
* apw isn't seeing that, but it is hanging very quick for me17:31
hallynapw: http://paste.ubuntu.com/5840986/  here is an example17:31
apwit does feel like it is behaving like it is ET17:32
hallynapw: but then i'd expect my looping of the read to fix it17:34
apwyep that we would17:37
hallynapw: d'oh.  17:39
apwhallyn, ?17:39
hallynsignalfd() does not do flags.17:39
hallynwonder if signalfd() library call uses signalfd4(0 syscall17:39
hallynyeah strace says it uses signalfd417:40
apwand mine is seemingly non-b17:40
apwso doesn't that mean it does work17:40
apwwaiting ...17:41
apwgot an event (nfds 1)17:41
apwgot sig for 20171, init was 2017317:41
apwfailed to read signal info errno=1117:41
apwfailed to read signal info errno=1117:41
apwwaiting ...17:41
apwfor me it does that, on a hang, so it saw the wrong pid, read 3 times (two failed cause nothing there) and then we went back17:41
hallynbut why 2 in a row failures?17:42
hallynit should only fail once, then return to the epoll-wait loop17:42
hallynapw: that printf looks differnet from mine17:43
hallyn                perror("read");17:43
hallyn                printf("errno is %d\n", errno);17:43
hallyn                if (errno == EAGAIN || errno == EWOULDBLOCK)17:43
hallyn                        return 0;17:43
hallyn                printf("failed to read signal info");17:43
apwyeah it is mine indeed17:43
hallyni'm tempted to ping mkerrisk17:43
hallynmind you the corresponding lxc bug has been somewhat low priority for me, but i thought we were doing something more wrong than this...  if it's a kernel or libc bug i think it's a bigger deal17:45
apwhallyn, i'll keep poking for a bit too17:49
apwhallyn, of course it might be a bug in signalfd not epoll, hmm17:51
hallynapw: yup17:56
hallynapw: that's why i was happy that fork could reproduce it - at least it doesn't involve CLONE_NEWPID as well :)17:56
apwhallyn, and it might be an alignment issue, as it is random and we have address psace randomisation on17:57
apwas SIGCLD is one of the large ones in that structure17:57
apwso if the structure was too small, and aligned bad or something17:58
apwor too big or something17:58
apw        struct signalfd_siginfo __user *siginfo;17:58
apwif that isn't the same size as the one in the kernel ... we might fail17:58
apwas if we dequeue it and then fail to return it ... we just lose it17:59
=== fmasi is now known as fmasi_afk
apwhallyn, what makes you think (i assume you have docs somewhere) that just because you are using this interface that you can guarentee to see all signals ?18:06
apwhallyn, as you say when it hangs you can see that you have pending children events18:07
apwhallyn, are you sure you can expect to see both signals as separate events with separate children ?18:07
apwhallyn, would i not expect you to actuall ignore the siginfo and use wait to get the childre18:07
apwhallyn, and there i htink i would expect you to get both as they are there waiting to be waited for18:07
apwhallyn, ie reading signalfd is _not_ waiting for them nor reporting them18:08
apwhallyn, and you need to wait to clean them up regardless18:10
hallynso i have to waitpid() on the first to clear it up before i can get a sigfd for the second sigchld?18:13
hallynlemme try18:14
hallynit makes sense18:14
hallyn(and it briefly came to mind last night, but i figured "nah!" :)18:14
apwhallyn, no i am not saying you have to waitpid to clear it18:15
apwhallyn, i am saying that signalfd is reporting pending signals18:15
apwand pending signals with their 'latest' siginfo18:16
apwbut if you get two signals of the same type only the last one will be reported18:16
apwthat is normal for signals and siginfo18:16
apwi am syaing when you get a signal, you should be looking on wait (with NOWAIT) and using18:16
hallyni see18:16
apwthe data it returns18:16
apwas the pids not anything int he siginfo.18:16
hallynso i (and the original author) am (is) completely misunderstanding sigchld delivery18:17
apwas we see both as zombies i bet money that you will find two waits18:17
hallyni did indeed assume i'd get N sigchlds18:17
apwthat is not normal with signals at the process level for sure18:17
apwnow signalfd could have either semantic and it does not say which18:17
apwbut if we conjecture it has the smae semantics as signals18:18
apwand the way POLLIN is calculated i suspect it is18:18
apwthen you may not indeed, but a looping wait should always be sage18:18
apwsafe, and i think you need to be waiting anyhow to clean up the carcasses18:18
apwso if i am right, a wait loop will repair things, if it is broken18:18
apwthen it will only ever have one and make no differende18:19
hallynapw: thanks - final signalfd.c testcase uploaded - this one doesn't seem to reproduce.  seems you were right!18:23
hallynnow i can fix lxc :)18:23
hallynthanks apw18:23
apwhallyn, heh... there is point in having a kernel team after all :)18:23
apwhallyn, and i am glad we got to the bottom of it18:23
* hallyn goes to take a walk outside before writing the actual patches18:24
apwhallyn, i know the feeling, i think i need a beer after thinking that hard18:26
hallynthat sounds.  good.18:27
apwhallyn, ok .. you need to break that loop for == 0 i think18:29
apwas it spins in there till they exit otherwise18:29
apwhallyn, and waitid() may be a more natural fit for your code base maybe18:30
apwhallyn, switching your loop from >= 0 to > 0 still seems to work18:30
apwand removes the repeated looping returning 018:30
apwgot sig for 2791, init was 279318:31
apwwaitpid ret=279118:31
apwwaitpid ret=279318:31
apwgot init's sigchld: done18:31
apwhallyn, and we see the pattern there which i conjectured .. coool18:31
apwbeer it is18:31
=== chiluk` is now known as chiluk
hallynapw: (drat, no beer for me) do you have a copy of your final one you can post?19:08
hallyncurious which loop you ended up going with19:08
hallynapw: oh, right, i misread the == 0 return value case!  lol19:09
hallyn"... in any state" well that does me no good :)19:09
hallynwaitid is too foreign to me :)  i'll muck it up somehow19:10
apwhallyn, that version of yours looked ok once it was >= 0 :)19:15
apw> 0 even19:15
hallyngetting ready to test the lxc ase19:17
hallynlxc case19:17
hallynhm, not a slam dunk fix.19:25
hallynodd 19:27
apwhallyn, ?19:36
hallynapw: added the waitpid loop into signal_handler in lxc, but lxc-start still ends up with defunct init 19:39
hallynso i'm missing something else that's going wrong there19:39
apwsounds like a tommorrow fun19:41
hallyntomorrow's a holiday here - but ttyl19:42
apwhallyn, drop me a line if you have any other examples to play with and i can look at it tommorrow19:43
seb128is there any issue with the saucy 3.10 kernel and screen light?19:46
seb128changing it using the multimedia keys hangs my latitude laptop for a second19:47
seb128where it works fine with 3.919:47
apwseb128, noting i have seen reported19:50
seb128apw, hum, ok19:50
seb128apw, is there any special info that would be useful in a bug report?19:51
apwseb128, h/w info which ubuntu-bug will collect19:52
seb128that's a latitude e641019:52
hallynapw: well huh - particularly interesting in the lxc case is that after i call the parent task, the [init] task remains defunct, while parented by 120:19
hallyn*that* probably shouldn't happen20:19
apwhallyn, if you reap a parent before its child then its children go to init don't they ?20:20
apwyou could test that20:20
apwwith a basic fork test20:20
hallynapw: (not sure i was clear)  i'm saying:  root      1878     1  0 21:32 pts/2    00:00:00 [init] <defunct>                                   20:23
hallyn'[init]'is the container init, which has exited;  it's now parented by the host init, but is still rockin' the zombie life20:23
apwright so noone has waited for it20:23
apwhave you waited for its parent20:23
hallynbc it's not sending a new sigchld?20:24
apwthat is not what i mean20:24
hallyni figured init would reap zombies regardless of whether it got a sigchld20:24
apwoh i see20:24
hallyni killed its parent, and the parent is gone.20:24
apwright so yes it should jump to real init, and that should be reaping it 20:24
apwthat is its job20:25
hallynall right i'd better step away a few hours.  sorry if i pulled you away from that beer :)20:26
apwhallyn, that seems wrong, and in the first place i would be blaming init itself, which we cannot trace easily hmmm, i wonder if --verbose would give us any info20:27
hallynapw: but as i say if the exiting task has already sent its sigchld, then gets reparented to init, then init won't get a sigchld - 20:29
hallyni'd actually expect the kernel at that point, instead of reparenting, to just reap ...  20:29
hallyn(i guess that's nonsense-talk)20:29
hallynso upstart would need to walk its pid list and check for zombies... which is not efficient.  still, yeah, cleaning up after bad users is its job...20:30
hallynthis should be trivial to write a test case for :)  will do so later20:30
=== fmasi_afk is now known as fmasi
=== fmasi is now known as fmasi_afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!