[18:48] Hope someone can help: I'm having trouble with gluster where the daemon 'glusterd' is starting before the network is properly up, despite using 'start on static-network-up'. My network contains a bridge (for KVM use), and it appears to appear up to upstart (ouch) before it's actually up and passing packets. Is there any way I can make glusterd wait on actual network connectivity through the bridge? [18:50] geek65535: how are you configuring the bridge? [21:20] SpamapS: so sorry! Got distracted by the real world. (I hate when that happens). This is my current config: [21:20] iface bridge0 inet static [21:20] address 10.8.1.21 [21:20] netmask 255.255.255.128 [21:20] network 10.8.1.0 [21:20] broadcast 10.8.1.127 [21:20] gateway 10.8.1.1 [21:20] # dns-* options are implemented by the resolvconf package, if installed [21:20] dns-nameservers 10.8.1.33 10.8.1.44 [21:20] dns-search houston8.topgolf.com [21:20] bridge_ports eth0 [21:20] bridge_stp off [21:20] bridge_maxwait 6 [21:46] I seem to have a slight problem with upstart when trying to run a python script as a service; when I try to manually invoke the process, it will just lock up the screen until I press ctrl-c. any ideas how to fix this? [21:51] what is the recommended approach for starting python scripts with upstart? [22:05] and now the server has become unusable... [22:08] TimothyA: python scripts aren't special over any other thing [22:09] TimothyA: can you pastebin your upstart script? [22:09] geek65535: do you have 'auto bridge0' in /etc/network/interfaces ? [22:12] SpamapS: I can't. the server has become unreachable. [22:13] SpamapS: yes. I just missed it on the copy-and-paste [22:21] geek65535: ok, static-network-up *should* wait for that.. you can see the exact time static-network-up was emitted by looking at the mtime of the dir /run/network/static-network-up-emitted ... [22:21] geek65535: perhaps ifup returns before the bridge is actually working? [22:23] SpamapS: removed the file and rebooted server [22:23] but it's nothing more than a basic respawn and exec python main.py lines [22:24] SpamapS: let me take a look... [22:24] TimothyA: yeah, that should work. If you don't have any post-start or expect lines, it should return immediately on 'start your-job-name' [22:27] run/network/static-network-up-emitted is empty... [22:27] er, sorry, missed the 'mtime' part... [22:27] geek65535: right, its the dir itself that is used as a flag [22:28] geek65535: you should also see the ifup.bridge0 there, with an earlier mtime/ctime [22:28] root@vmhost1:/run/network# ls -l --full-time [22:28] total 4 [22:28] -rw-r--r-- 1 root root 22 2012-10-17 17:26:27.217981181 -0500 ifstate [22:28] -rw-r--r-- 1 root root 0 2012-10-17 17:26:27.209981181 -0500 ifup.bridge0 [22:28] -rw-r--r-- 1 root root 0 2012-10-17 17:26:24.089981092 -0500 ifup.lo [22:28] drwxr-xr-x 2 root root 40 2012-10-17 17:26:27.209981181 -0500 static-network-up-emitted [22:29] geek65535: right, so ifup is the problem then [22:29] geek65535: it must return before the bridge is actually working [22:31] That's what I had been thinking. I just need to find something that emits a signal after the bridge is active. (Or live with the 'sleep 10' stuck in my /etc/init/glusterd.conf) [22:31] geek65535: thats a bug.. perhaps we should find it and fix it :) [22:33] What can I do to help? (I'm an experienced admin, but I don't code outside of scripting languages.) [22:35] geek65535: I'm looking now.. it may be an issue with the way ifupdown delegates this duty to the bridge plugin [22:36] SpamapS: it did not [22:36] and it locks up my screen when I try to do it manually [22:36] TimothyA: "locks up my screen" meaning the terminal does not return? [22:37] yep [22:37] geek65535: ok, this is handled by /etc/network/if-pre-up.d/bridge ... [22:38] geek65535: of particular interest to you may be the MAXWAIT= area [22:39] geek65535: bridge_maxwait 10 would solve your problem ;) [22:40] geek65535: (from man bridge-utils-interfaces) [22:40] geek65535: actualy, bridge_fd seems more appropriate [22:41] geek65535: I suspect yours is failing because brctl showstp bridge0 is not reporting forward delay / bridge forward delay properly [22:42] geek65535: its usually 15.00 which should be 15 seconds.. so maybe you need more? [22:42] SpamapS: I have maxwait at 6 and bridge_fd at 0, allong with bridge_hello at 0 (although that shouldn't matter with stp off). [22:43] TimothyA: ok, well that still should work fine. When you can, pastebin the upstart job file and we can try to help [22:43] SpamapS: the python script already implements a signal handler. [22:43] TimothyA: thats not realy important in 'start' [22:43] TimothyA: signals are only sent on 'reload' and 'stop' (HUP, and TERM, then KILL) [22:43] exec python main.py [22:43] that's about all [22:46] SpamapS: upped maxwait to 20, rebooting to see if it make a difference. [22:48] SpamapS: no joy. [22:48] geek65535: how strange [22:49] geek65535: another possibility is that brctl is showing it as ready, when its not [22:49] SpamapS: I'd buy that. [22:50] geek65535: because maxwait is just how long it waits for brctl showstp to show 'listening, learning, forwarding, blocking' [22:51] oh oh oh .. I think that maxwait is 10th's of seconds [22:52] SpamapS: 10ths? That would make a difference! [22:53] yeah [22:53] well it sleeps 0.1, then inc's count +1 [22:53] so yeah, 10ths [22:53] ok, I have to run [22:53] SpamapS: me, too, actually. dinnertime... [22:54] geek65535: I think you have enough to take a look. If you find that the script is doing something wrong, or brctl is, I urge you to do an 'ubuntu-bug bridge-utils' or 'ubuntu-bug linux' ;) good luck [22:55] SpamapS: thanks!