[20:26] <ion> O hai
[20:27] <ion> keybuk: A couple of small changes: https://code.edge.launchpad.net/~ion/ubuntu/lucid/upstart/lucid
[20:28] <sadmac2> Keybuk: you're alive!
[20:30] <ion> You guys might get a kick out of a hack i did: ‘evil’ implements ‘reference’ without branching. http://gist.github.com/290066
[20:31] <ion> An excellent introduction to Erlang, btw: http://www.viddler.com/explore/rentzsch/videos/8/
[20:32] <Keybuk> hi
[20:32] <Keybuk> I am
[20:32] <Keybuk> very much alive - and very much refreshed
[20:32] <Keybuk> holidays FTW
[20:32] <ion> Good to hear
[20:33] <sadmac2> Keybuk: I just got my first upstart 0.6/SELinux clash bug today. We're leaking pipes.
[20:34] <sadmac2> why O_CLOEXEC wasn't the default behavior in POSIX... well, actually there's weirder shit in POSIX to wonder about.
[20:36] <ion> Someone should form a committee to design a better POSIX!
[20:37] <sadmac2> ion: one day when I gain superpowers I will submit a giant patch to linux introducing a second process tree with better semantics and interfaces.
[20:37] <sadmac2> then we can phase POSIX out altogether
[20:40] <soren> I'm curious.. Has any thought been put into the impact on memory requirements caused by running a whole bunch of things in parallel rather than sequentially?
[20:41] <sadmac2> soren: the majority of those things are long running processes.
[20:41] <Keybuk> sadmac2: which pipe is being leaked?
[20:42] <soren> sadmac2: I'm not sure what significance that has?
[20:42] <sadmac2> Keybuk: dunno yet. It could quite possibly be something inside of libdbus
[20:42] <sadmac2> soren: eventually they'll all be running in parallel anyway.
[20:42] <soren> sadmac2: True.
[20:43] <sadmac2> soren: I doubt there's anything interesting to explore in that avenue, but if you'd care to measure something it might be interesting to look at.
[20:43] <soren> sadmac2: We're just seeing a problem somewhere that might be due to everything running in parallel rather than sequentially. I'm not entirely sure yet.
[20:43] <Keybuk> sadmac2: at some point, upstart will use the newer syscalls that I can specify the cloexec flag
[20:43] <Keybuk> when I have some time
[20:43] <Keybuk> *cry*
[20:44] <sadmac2> Keybuk: yeah.
[20:45]  * Keybuk plans to beg for upstart time next cycle
[20:45] <sadmac2> Keybuk: I've been having some nasty interruptions to my coding time lately, the last one involving a new car and some bruised ribs that I really need to get looked at.
[20:45] <soren> sadmac2: Short story: We have some virtual machines with 256 MB of RAM and no swap.. During boot, the OOM killer gets invoked for some reason.
[20:45] <sadmac2> but my airbags were in working order, which was nice to find out.
[20:46] <Keybuk> soren: fsck probably
[20:46] <Keybuk> fsck consumes terabytes of memory
[20:46] <Keybuk> its why things try so hard to activate swap first
[20:46] <soren> sadmac2: ...and one of the working hypotheses is that it's caused by things being run in parallel that used to be run sequentially. I don't really buy that hypothesis, but was just curious about the general problem.
[20:46] <soren> Keybuk: Interesting.
[20:47] <sadmac2> soren: you'd have to instrument and see. certainly upstart has tools to serialize out stuff if you need them. certainly those tools could get better if there's many people like you.
[20:47] <sadmac2> but yeah, counting up how many of those things you're running are tasks which will actually release memory in the short term would help test your theory.
[20:48] <Keybuk> perf ftw
[20:48] <soren> Keybuk: These should be clean images, though. fsck shouldn't be doing anything other than telling me that everything is fine and that it won't bother with a proper check.
[20:49] <soren> Keybuk: "perf"?
[20:49] <sadmac2> "performance"
[20:49] <sadmac2> the genre of debuggering you're entering into
[20:50] <soren> Ah, I thought it was a reference to another tool in Keybuk's shiny magic box of tools.
[20:51] <sadmac2> there might be a perf tool called perf. I think there is.
[20:51] <soren> Perhaps. It's not in ubuntu.
[20:51]  * sadmac2 doesn't have it
[20:51] <sadmac2> soren: there's oprofile, which is awful, but also the only thing that does what it does. Thankfully your problem isn't suited to it.
[20:52] <sadmac2> soren: and there's systemtap, which may come into play here
[20:52] <sadmac2> <3 systemtap
[20:52] <Keybuk> it comes in the kernel source
[20:52] <sadmac2> It keeps me employed, and as things people pay for go, its not so bad to work with
[20:54] <soren> sadmac2: How can systemtap be useful here? This it the first time I've heard of it.
[20:55] <sadmac2> soren: hard to say. Its problem scope is gigantic.
[20:55] <soren> I see.
[20:55] <sadmac2> soren: it lets you write custom trace programs
[20:56] <soren> sadmac2: I'm curious about your "things people pay for" remark. We have it in Ubuntu, so it must be free software?
[20:56] <sadmac2> soren: for your issue though, getting top running early enough would do it.
[20:56] <sadmac2> soren: people pay me to /use/ it
[20:56] <sadmac2> soren: I'm a support engineer. I solve fiddly performance issues in a hurry while the customer's hair is on fire.
[20:57] <sadmac2> fiddly...
[20:57]  * sadmac2 notices his diction getting Britisher now that Keybuk is back
[20:58] <soren> sadmac2: Ah, I see what you mean.
[20:59] <soren> sadmac2: As for getting top to run that early, I think that will be hard. The OOM killer gets upset quite early.
[20:59] <soren> smoser: can you reproduce this problem in a regular VM?
[21:00] <smoser> yes. under "normal" kvm
[21:01] <sadmac2> soren: actually, just turn on kdump and disable the OOM killer, and let it core. the crash dump tool should be able to get you what you need.
[21:01] <smoser> lucid: http://pastebin.com/f16fba185
[21:02] <smoser> karmic: http://pastebin.com/f5d42cbf8
[21:02] <soren> Maybe a low-tech approach would do: Run a script very early on that dumps the output of "ps -eFl" somewhere every 0.1 seconds and go and look at it afterwards?
[21:02] <smoser> ^^^ output of ps -eo pid,vsz,args
[21:02] <Keybuk> sadmac2: perf is the new systemtap
[21:03] <sadmac2> Keybuk: really?
[21:04] <sadmac2> its hard to google because google expands "perf" into "performance"
[21:04] <sadmac2> "prioritize results with the least amount of interpolation" doesn't seem to have made it into google's secret sauce.
[21:04] <ion> Google for +perf
[21:05] <sadmac2> as of 2005 the readme declares it dead.
[21:05] <sadmac2> OR there's 2 things called perf
[21:05] <sadmac2> Keybuk: ah, this is looking familiar
[21:06] <sadmac2> Keybuk: I think you have it wrong though. This looks like the new oprofile, not the new systemtap
[22:06] <smoser> soren, http://pastebin.com/f37a27768 is updated with i386 
[22:07] <Keybuk> it's the new systemtap too
[22:07] <Keybuk> there's nothing you can do with systemtap that you can't do with perf_event/trace_event
[22:35] <sadmac> Keybuk: I'll be interested to test that.
[22:36]  * sadmac remembers to check the RHEL 6 kernel packages for it