[11:16] <tom__> I'm trying to debug/understand high init cpu/memory use when creating lots of network interfaces
[11:16] <tom__> these are virtual interfaces, and I'm creating of the order of 10-15 of them per second
[11:17] <tom__> on my ubuntu 12.04 server install I see init using up to 90% CPU at times, with memory use increasing from around 0.8% when I boot to around 8% when I get to around 30k virtual interfaces
[11:18] <tom__> If I disable the upstart-udev-bridge process (service upstart-udev-bridge stop) before I create the virtual interfaces, the high cpu/memory usage goes away -- so it looks like upstart is somehow behaving badly in the face of lots of virtual interfaces
[11:19] <tom__> Does anyone have any ideas on this?  Am I being foolish in stressing upstart in this way?
[12:07] <SpamapS> tom__: are you creating them, and then removing them?
[12:08] <SpamapS> tom__: I think we identified a problem with the network-interface-security jobs where instances are not stopped recently
[12:41]  * JanC wonders why anybody would want to create 30k virtual network interfaces  ☺
[12:48] <tom__> SpamapS: no, I'm just creating them.  They may then be bought up using the ip command, but they stay in existence 
[12:48] <tom__> JanC: good question ;-) it's for an L2TP configuration, bridging lots of ethernet sessions at the server end
[12:54] <JanC> ah, a VPN server, that makes sense indeed   ☺
[14:07] <SpamapS> tom__: I doubt anybody has tested 30,000 job instances, so perhaps there is a scalability problem with upstart
[14:07] <SpamapS> tom__: I'd be interested to attach gdb and measure where its spending its time
[14:19] <tom__> SpamapS: is there a good way to measure runtime behaviour with gdb?  I've used it as a "poor man's profiler" before (interrupting periodically, grabbing a backtrace, analysing the resulting traces), but that's a bit tedious
[14:48] <tom__> Hmm, it looks like one may reproduce at least some of the same behaviour using veth interfaces
[14:48] <tom__> For example: for i in {1..30000}; do sudo ip link add name vbl${i}_0 type veth peer name vbl${i}_1; done
[14:49] <tom__> Running this up in a VM shows init CPU usage at around 10-15%, memory usage currently at 5.7% and counting
[14:50] <tom__> My VM is a somewhat different environment to the box I referred to earlier -- the VM has an i7 core to play with, while the hardware machine I typically use for testing is a venerable Athlon 64
[15:53] <SpamapS> tom__: http://poormansprofiler.org/
[15:53] <SpamapS> tom__: actually there's one huge problem with that which is that init is doing ptraces, so it can't be ptraced properly IIRC
[16:15] <tom__> SpamapS: cool link -- I'd not come across that before.  I had my own version of the same thing!
[16:16] <tom__> SpamapS: FWIW, I tried it out on my VM, which is now rather bogged down with some 29k veth interfaces, and has upstart eating around 80% CPU
[16:16] <tom__> SpamapS: here's 10 iterations of the Poor Man's Profiler:
[16:17] <tom__>      10 ??,job_process_handler,nih_child_poll,nih_main_loop,main