[11:16] I'm trying to debug/understand high init cpu/memory use when creating lots of network interfaces [11:16] these are virtual interfaces, and I'm creating of the order of 10-15 of them per second [11:17] on my ubuntu 12.04 server install I see init using up to 90% CPU at times, with memory use increasing from around 0.8% when I boot to around 8% when I get to around 30k virtual interfaces [11:18] If I disable the upstart-udev-bridge process (service upstart-udev-bridge stop) before I create the virtual interfaces, the high cpu/memory usage goes away -- so it looks like upstart is somehow behaving badly in the face of lots of virtual interfaces [11:19] Does anyone have any ideas on this? Am I being foolish in stressing upstart in this way? [12:07] tom__: are you creating them, and then removing them? [12:08] tom__: I think we identified a problem with the network-interface-security jobs where instances are not stopped recently [12:41] * JanC wonders why anybody would want to create 30k virtual network interfaces ☺ [12:48] SpamapS: no, I'm just creating them. They may then be bought up using the ip command, but they stay in existence [12:48] JanC: good question ;-) it's for an L2TP configuration, bridging lots of ethernet sessions at the server end [12:54] ah, a VPN server, that makes sense indeed ☺ [14:07] tom__: I doubt anybody has tested 30,000 job instances, so perhaps there is a scalability problem with upstart [14:07] tom__: I'd be interested to attach gdb and measure where its spending its time [14:19] SpamapS: is there a good way to measure runtime behaviour with gdb? I've used it as a "poor man's profiler" before (interrupting periodically, grabbing a backtrace, analysing the resulting traces), but that's a bit tedious [14:48] Hmm, it looks like one may reproduce at least some of the same behaviour using veth interfaces [14:48] For example: for i in {1..30000}; do sudo ip link add name vbl${i}_0 type veth peer name vbl${i}_1; done [14:49] Running this up in a VM shows init CPU usage at around 10-15%, memory usage currently at 5.7% and counting [14:50] My VM is a somewhat different environment to the box I referred to earlier -- the VM has an i7 core to play with, while the hardware machine I typically use for testing is a venerable Athlon 64 [15:53] tom__: http://poormansprofiler.org/ [15:53] tom__: actually there's one huge problem with that which is that init is doing ptraces, so it can't be ptraced properly IIRC [16:15] SpamapS: cool link -- I'd not come across that before. I had my own version of the same thing! [16:16] SpamapS: FWIW, I tried it out on my VM, which is now rather bogged down with some 29k veth interfaces, and has upstart eating around 80% CPU [16:16] SpamapS: here's 10 iterations of the Poor Man's Profiler: [16:17] 10 ??,job_process_handler,nih_child_poll,nih_main_loop,main