| tom__ | I'm trying to debug/understand high init cpu/memory use when creating lots of network interfaces | 11:16 |
|---|---|---|
| tom__ | these are virtual interfaces, and I'm creating of the order of 10-15 of them per second | 11:16 |
| tom__ | on my ubuntu 12.04 server install I see init using up to 90% CPU at times, with memory use increasing from around 0.8% when I boot to around 8% when I get to around 30k virtual interfaces | 11:17 |
| tom__ | If I disable the upstart-udev-bridge process (service upstart-udev-bridge stop) before I create the virtual interfaces, the high cpu/memory usage goes away -- so it looks like upstart is somehow behaving badly in the face of lots of virtual interfaces | 11:18 |
| tom__ | Does anyone have any ideas on this? Am I being foolish in stressing upstart in this way? | 11:19 |
| SpamapS | tom__: are you creating them, and then removing them? | 12:07 |
| SpamapS | tom__: I think we identified a problem with the network-interface-security jobs where instances are not stopped recently | 12:08 |
| * JanC wonders why anybody would want to create 30k virtual network interfaces ☺ | 12:41 | |
| tom__ | SpamapS: no, I'm just creating them. They may then be bought up using the ip command, but they stay in existence | 12:48 |
| tom__ | JanC: good question ;-) it's for an L2TP configuration, bridging lots of ethernet sessions at the server end | 12:48 |
| JanC | ah, a VPN server, that makes sense indeed ☺ | 12:54 |
| SpamapS | tom__: I doubt anybody has tested 30,000 job instances, so perhaps there is a scalability problem with upstart | 14:07 |
| SpamapS | tom__: I'd be interested to attach gdb and measure where its spending its time | 14:07 |
| tom__ | SpamapS: is there a good way to measure runtime behaviour with gdb? I've used it as a "poor man's profiler" before (interrupting periodically, grabbing a backtrace, analysing the resulting traces), but that's a bit tedious | 14:19 |
| tom__ | Hmm, it looks like one may reproduce at least some of the same behaviour using veth interfaces | 14:48 |
| tom__ | For example: for i in {1..30000}; do sudo ip link add name vbl${i}_0 type veth peer name vbl${i}_1; done | 14:48 |
| tom__ | Running this up in a VM shows init CPU usage at around 10-15%, memory usage currently at 5.7% and counting | 14:49 |
| tom__ | My VM is a somewhat different environment to the box I referred to earlier -- the VM has an i7 core to play with, while the hardware machine I typically use for testing is a venerable Athlon 64 | 14:50 |
| SpamapS | tom__: http://poormansprofiler.org/ | 15:53 |
| SpamapS | tom__: actually there's one huge problem with that which is that init is doing ptraces, so it can't be ptraced properly IIRC | 15:53 |
| tom__ | SpamapS: cool link -- I'd not come across that before. I had my own version of the same thing! | 16:15 |
| tom__ | SpamapS: FWIW, I tried it out on my VM, which is now rather bogged down with some 29k veth interfaces, and has upstart eating around 80% CPU | 16:16 |
| tom__ | SpamapS: here's 10 iterations of the Poor Man's Profiler: | 16:16 |
| tom__ | 10 ??,job_process_handler,nih_child_poll,nih_main_loop,main | 16:17 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!