[20:26] O hai [20:27] keybuk: A couple of small changes: https://code.edge.launchpad.net/~ion/ubuntu/lucid/upstart/lucid [20:28] Keybuk: you're alive! [20:30] You guys might get a kick out of a hack i did: ‘evil’ implements ‘reference’ without branching. http://gist.github.com/290066 [20:31] An excellent introduction to Erlang, btw: http://www.viddler.com/explore/rentzsch/videos/8/ [20:32] hi [20:32] I am [20:32] very much alive - and very much refreshed [20:32] holidays FTW [20:32] Good to hear [20:33] Keybuk: I just got my first upstart 0.6/SELinux clash bug today. We're leaking pipes. [20:34] why O_CLOEXEC wasn't the default behavior in POSIX... well, actually there's weirder shit in POSIX to wonder about. [20:36] Someone should form a committee to design a better POSIX! [20:37] ion: one day when I gain superpowers I will submit a giant patch to linux introducing a second process tree with better semantics and interfaces. [20:37] then we can phase POSIX out altogether [20:40] I'm curious.. Has any thought been put into the impact on memory requirements caused by running a whole bunch of things in parallel rather than sequentially? [20:41] soren: the majority of those things are long running processes. [20:41] sadmac2: which pipe is being leaked? [20:42] sadmac2: I'm not sure what significance that has? [20:42] Keybuk: dunno yet. It could quite possibly be something inside of libdbus [20:42] soren: eventually they'll all be running in parallel anyway. [20:42] sadmac2: True. [20:43] soren: I doubt there's anything interesting to explore in that avenue, but if you'd care to measure something it might be interesting to look at. [20:43] sadmac2: We're just seeing a problem somewhere that might be due to everything running in parallel rather than sequentially. I'm not entirely sure yet. [20:43] sadmac2: at some point, upstart will use the newer syscalls that I can specify the cloexec flag [20:43] when I have some time [20:43] *cry* [20:44] Keybuk: yeah. [20:45] * Keybuk plans to beg for upstart time next cycle [20:45] Keybuk: I've been having some nasty interruptions to my coding time lately, the last one involving a new car and some bruised ribs that I really need to get looked at. [20:45] sadmac2: Short story: We have some virtual machines with 256 MB of RAM and no swap.. During boot, the OOM killer gets invoked for some reason. [20:45] but my airbags were in working order, which was nice to find out. [20:46] soren: fsck probably [20:46] fsck consumes terabytes of memory [20:46] its why things try so hard to activate swap first [20:46] sadmac2: ...and one of the working hypotheses is that it's caused by things being run in parallel that used to be run sequentially. I don't really buy that hypothesis, but was just curious about the general problem. [20:46] Keybuk: Interesting. [20:47] soren: you'd have to instrument and see. certainly upstart has tools to serialize out stuff if you need them. certainly those tools could get better if there's many people like you. [20:47] but yeah, counting up how many of those things you're running are tasks which will actually release memory in the short term would help test your theory. [20:48] perf ftw [20:48] Keybuk: These should be clean images, though. fsck shouldn't be doing anything other than telling me that everything is fine and that it won't bother with a proper check. [20:49] Keybuk: "perf"? [20:49] "performance" [20:49] the genre of debuggering you're entering into [20:50] Ah, I thought it was a reference to another tool in Keybuk's shiny magic box of tools. [20:51] there might be a perf tool called perf. I think there is. [20:51] Perhaps. It's not in ubuntu. [20:51] * sadmac2 doesn't have it [20:51] soren: there's oprofile, which is awful, but also the only thing that does what it does. Thankfully your problem isn't suited to it. [20:52] soren: and there's systemtap, which may come into play here [20:52] <3 systemtap [20:52] it comes in the kernel source [20:52] It keeps me employed, and as things people pay for go, its not so bad to work with [20:54] sadmac2: How can systemtap be useful here? This it the first time I've heard of it. [20:55] soren: hard to say. Its problem scope is gigantic. [20:55] I see. [20:55] soren: it lets you write custom trace programs [20:56] sadmac2: I'm curious about your "things people pay for" remark. We have it in Ubuntu, so it must be free software? [20:56] soren: for your issue though, getting top running early enough would do it. [20:56] soren: people pay me to /use/ it [20:56] soren: I'm a support engineer. I solve fiddly performance issues in a hurry while the customer's hair is on fire. [20:57] fiddly... [20:57] * sadmac2 notices his diction getting Britisher now that Keybuk is back [20:58] sadmac2: Ah, I see what you mean. [20:59] sadmac2: As for getting top to run that early, I think that will be hard. The OOM killer gets upset quite early. [20:59] smoser: can you reproduce this problem in a regular VM? [21:00] yes. under "normal" kvm [21:01] soren: actually, just turn on kdump and disable the OOM killer, and let it core. the crash dump tool should be able to get you what you need. [21:01] lucid: http://pastebin.com/f16fba185 [21:02] karmic: http://pastebin.com/f5d42cbf8 [21:02] Maybe a low-tech approach would do: Run a script very early on that dumps the output of "ps -eFl" somewhere every 0.1 seconds and go and look at it afterwards? [21:02] ^^^ output of ps -eo pid,vsz,args [21:02] sadmac2: perf is the new systemtap [21:03] Keybuk: really? [21:04] its hard to google because google expands "perf" into "performance" [21:04] "prioritize results with the least amount of interpolation" doesn't seem to have made it into google's secret sauce. [21:04] Google for +perf [21:05] as of 2005 the readme declares it dead. [21:05] OR there's 2 things called perf [21:05] Keybuk: ah, this is looking familiar [21:06] Keybuk: I think you have it wrong though. This looks like the new oprofile, not the new systemtap [22:06] soren, http://pastebin.com/f37a27768 is updated with i386 [22:07] it's the new systemtap too [22:07] there's nothing you can do with systemtap that you can't do with perf_event/trace_event [22:35] Keybuk: I'll be interested to test that. [22:36] * sadmac remembers to check the RHEL 6 kernel packages for it