| ion | O hai | 20:26 |
|---|---|---|
| ion | keybuk: A couple of small changes: https://code.edge.launchpad.net/~ion/ubuntu/lucid/upstart/lucid | 20:27 |
| sadmac2 | Keybuk: you're alive! | 20:28 |
| ion | You guys might get a kick out of a hack i did: ‘evil’ implements ‘reference’ without branching. http://gist.github.com/290066 | 20:30 |
| ion | An excellent introduction to Erlang, btw: http://www.viddler.com/explore/rentzsch/videos/8/ | 20:31 |
| Keybuk | hi | 20:32 |
| Keybuk | I am | 20:32 |
| Keybuk | very much alive - and very much refreshed | 20:32 |
| Keybuk | holidays FTW | 20:32 |
| ion | Good to hear | 20:32 |
| sadmac2 | Keybuk: I just got my first upstart 0.6/SELinux clash bug today. We're leaking pipes. | 20:33 |
| sadmac2 | why O_CLOEXEC wasn't the default behavior in POSIX... well, actually there's weirder shit in POSIX to wonder about. | 20:34 |
| ion | Someone should form a committee to design a better POSIX! | 20:36 |
| sadmac2 | ion: one day when I gain superpowers I will submit a giant patch to linux introducing a second process tree with better semantics and interfaces. | 20:37 |
| sadmac2 | then we can phase POSIX out altogether | 20:37 |
| soren | I'm curious.. Has any thought been put into the impact on memory requirements caused by running a whole bunch of things in parallel rather than sequentially? | 20:40 |
| sadmac2 | soren: the majority of those things are long running processes. | 20:41 |
| Keybuk | sadmac2: which pipe is being leaked? | 20:41 |
| soren | sadmac2: I'm not sure what significance that has? | 20:42 |
| sadmac2 | Keybuk: dunno yet. It could quite possibly be something inside of libdbus | 20:42 |
| sadmac2 | soren: eventually they'll all be running in parallel anyway. | 20:42 |
| soren | sadmac2: True. | 20:42 |
| sadmac2 | soren: I doubt there's anything interesting to explore in that avenue, but if you'd care to measure something it might be interesting to look at. | 20:43 |
| soren | sadmac2: We're just seeing a problem somewhere that might be due to everything running in parallel rather than sequentially. I'm not entirely sure yet. | 20:43 |
| Keybuk | sadmac2: at some point, upstart will use the newer syscalls that I can specify the cloexec flag | 20:43 |
| Keybuk | when I have some time | 20:43 |
| Keybuk | *cry* | 20:43 |
| sadmac2 | Keybuk: yeah. | 20:44 |
| * Keybuk plans to beg for upstart time next cycle | 20:45 | |
| sadmac2 | Keybuk: I've been having some nasty interruptions to my coding time lately, the last one involving a new car and some bruised ribs that I really need to get looked at. | 20:45 |
| soren | sadmac2: Short story: We have some virtual machines with 256 MB of RAM and no swap.. During boot, the OOM killer gets invoked for some reason. | 20:45 |
| sadmac2 | but my airbags were in working order, which was nice to find out. | 20:45 |
| Keybuk | soren: fsck probably | 20:46 |
| Keybuk | fsck consumes terabytes of memory | 20:46 |
| Keybuk | its why things try so hard to activate swap first | 20:46 |
| soren | sadmac2: ...and one of the working hypotheses is that it's caused by things being run in parallel that used to be run sequentially. I don't really buy that hypothesis, but was just curious about the general problem. | 20:46 |
| soren | Keybuk: Interesting. | 20:46 |
| sadmac2 | soren: you'd have to instrument and see. certainly upstart has tools to serialize out stuff if you need them. certainly those tools could get better if there's many people like you. | 20:47 |
| sadmac2 | but yeah, counting up how many of those things you're running are tasks which will actually release memory in the short term would help test your theory. | 20:47 |
| Keybuk | perf ftw | 20:48 |
| soren | Keybuk: These should be clean images, though. fsck shouldn't be doing anything other than telling me that everything is fine and that it won't bother with a proper check. | 20:48 |
| soren | Keybuk: "perf"? | 20:49 |
| sadmac2 | "performance" | 20:49 |
| sadmac2 | the genre of debuggering you're entering into | 20:49 |
| soren | Ah, I thought it was a reference to another tool in Keybuk's shiny magic box of tools. | 20:50 |
| sadmac2 | there might be a perf tool called perf. I think there is. | 20:51 |
| soren | Perhaps. It's not in ubuntu. | 20:51 |
| * sadmac2 doesn't have it | 20:51 | |
| sadmac2 | soren: there's oprofile, which is awful, but also the only thing that does what it does. Thankfully your problem isn't suited to it. | 20:51 |
| sadmac2 | soren: and there's systemtap, which may come into play here | 20:52 |
| sadmac2 | <3 systemtap | 20:52 |
| Keybuk | it comes in the kernel source | 20:52 |
| sadmac2 | It keeps me employed, and as things people pay for go, its not so bad to work with | 20:52 |
| soren | sadmac2: How can systemtap be useful here? This it the first time I've heard of it. | 20:54 |
| sadmac2 | soren: hard to say. Its problem scope is gigantic. | 20:55 |
| soren | I see. | 20:55 |
| sadmac2 | soren: it lets you write custom trace programs | 20:55 |
| soren | sadmac2: I'm curious about your "things people pay for" remark. We have it in Ubuntu, so it must be free software? | 20:56 |
| sadmac2 | soren: for your issue though, getting top running early enough would do it. | 20:56 |
| sadmac2 | soren: people pay me to /use/ it | 20:56 |
| sadmac2 | soren: I'm a support engineer. I solve fiddly performance issues in a hurry while the customer's hair is on fire. | 20:56 |
| sadmac2 | fiddly... | 20:57 |
| * sadmac2 notices his diction getting Britisher now that Keybuk is back | 20:57 | |
| soren | sadmac2: Ah, I see what you mean. | 20:58 |
| soren | sadmac2: As for getting top to run that early, I think that will be hard. The OOM killer gets upset quite early. | 20:59 |
| soren | smoser: can you reproduce this problem in a regular VM? | 20:59 |
| smoser | yes. under "normal" kvm | 21:00 |
| sadmac2 | soren: actually, just turn on kdump and disable the OOM killer, and let it core. the crash dump tool should be able to get you what you need. | 21:01 |
| smoser | lucid: http://pastebin.com/f16fba185 | 21:01 |
| smoser | karmic: http://pastebin.com/f5d42cbf8 | 21:02 |
| soren | Maybe a low-tech approach would do: Run a script very early on that dumps the output of "ps -eFl" somewhere every 0.1 seconds and go and look at it afterwards? | 21:02 |
| smoser | ^^^ output of ps -eo pid,vsz,args | 21:02 |
| Keybuk | sadmac2: perf is the new systemtap | 21:02 |
| sadmac2 | Keybuk: really? | 21:03 |
| sadmac2 | its hard to google because google expands "perf" into "performance" | 21:04 |
| sadmac2 | "prioritize results with the least amount of interpolation" doesn't seem to have made it into google's secret sauce. | 21:04 |
| ion | Google for +perf | 21:04 |
| sadmac2 | as of 2005 the readme declares it dead. | 21:05 |
| sadmac2 | OR there's 2 things called perf | 21:05 |
| sadmac2 | Keybuk: ah, this is looking familiar | 21:05 |
| sadmac2 | Keybuk: I think you have it wrong though. This looks like the new oprofile, not the new systemtap | 21:06 |
| smoser | soren, http://pastebin.com/f37a27768 is updated with i386 | 22:06 |
| Keybuk | it's the new systemtap too | 22:07 |
| Keybuk | there's nothing you can do with systemtap that you can't do with perf_event/trace_event | 22:07 |
| sadmac | Keybuk: I'll be interested to test that. | 22:35 |
| * sadmac remembers to check the RHEL 6 kernel packages for it | 22:36 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!