[18:48] <geek65535> Hope someone can help: I'm having trouble with gluster where the daemon 'glusterd' is starting before the network is properly up, despite using 'start on static-network-up'. My network contains a bridge (for KVM use), and it appears to appear up to upstart (ouch) before it's actually up and passing packets. Is there any way I can make glusterd wait on actual network connectivity through the bridge?
[18:50] <SpamapS> geek65535: how are you configuring the bridge?
[21:20] <geek65535> SpamapS: so sorry! Got distracted by the real world. (I hate when that happens). This is my current config:
[21:20] <geek65535> iface bridge0 inet static
[21:20] <geek65535>         address 10.8.1.21
[21:20] <geek65535>         netmask 255.255.255.128
[21:20] <geek65535>         network 10.8.1.0
[21:20] <geek65535>         broadcast 10.8.1.127
[21:20] <geek65535>         gateway 10.8.1.1
[21:20] <geek65535>         # dns-* options are implemented by the resolvconf package, if installed
[21:20] <geek65535>         dns-nameservers 10.8.1.33 10.8.1.44
[21:20] <geek65535>         dns-search houston8.topgolf.com
[21:20] <geek65535>         bridge_ports eth0
[21:20] <geek65535>         bridge_stp off
[21:20] <geek65535>         bridge_maxwait 6
[21:46] <TimothyA> I seem to have a slight problem with upstart when trying to run a python script as a service; when I try to manually invoke the process, it will just lock up the screen until I press ctrl-c. any ideas how to fix this?
[21:51] <TimothyA> what is the recommended approach for starting python scripts with upstart?
[22:05] <TimothyA> and now the server has become unusable...
[22:08] <SpamapS> TimothyA: python scripts aren't special over any other thing
[22:09] <SpamapS> TimothyA: can you pastebin your upstart script?
[22:09] <SpamapS> geek65535: do you have 'auto bridge0' in /etc/network/interfaces ?
[22:12] <TimothyA> SpamapS: I can't. the server has become unreachable.
[22:13] <geek65535> SpamapS: yes. I just missed it on the copy-and-paste
[22:21] <SpamapS> geek65535: ok, static-network-up *should* wait for that.. you can see the exact time static-network-up was emitted by looking at the mtime of the dir /run/network/static-network-up-emitted ...
[22:21] <SpamapS> geek65535: perhaps ifup returns before the bridge is actually working?
[22:23] <TimothyA> SpamapS: removed the file and rebooted server
[22:23] <TimothyA> but it's nothing more than a basic respawn and exec python main.py lines
[22:24] <geek65535> SpamapS: let me take a look...
[22:24] <SpamapS> TimothyA: yeah, that should work. If you don't have any post-start or expect lines, it should return immediately on 'start your-job-name'
[22:27] <geek65535> run/network/static-network-up-emitted is empty...
[22:27] <geek65535> er, sorry, missed the 'mtime' part...
[22:27] <SpamapS> geek65535: right, its the dir itself that is used as a flag
[22:28] <SpamapS> geek65535: you should also see the ifup.bridge0 there, with an earlier mtime/ctime
[22:28] <geek65535> root@vmhost1:/run/network# ls -l --full-time
[22:28] <geek65535> total 4
[22:28] <geek65535> -rw-r--r-- 1 root root 22 2012-10-17 17:26:27.217981181 -0500 ifstate
[22:28] <geek65535> -rw-r--r-- 1 root root  0 2012-10-17 17:26:27.209981181 -0500 ifup.bridge0
[22:28] <geek65535> -rw-r--r-- 1 root root  0 2012-10-17 17:26:24.089981092 -0500 ifup.lo
[22:28] <geek65535> drwxr-xr-x 2 root root 40 2012-10-17 17:26:27.209981181 -0500 static-network-up-emitted
[22:29] <SpamapS> geek65535: right, so ifup is the problem then
[22:29] <SpamapS> geek65535: it must return before the bridge is actually working
[22:31] <geek65535> That's what I had been thinking. I just need to find something that emits a signal after the bridge is active. (Or live with the 'sleep 10' stuck in my /etc/init/glusterd.conf)
[22:31] <SpamapS> geek65535: thats a bug.. perhaps we should find it and fix it :)
[22:33] <geek65535> What can I do to help? (I'm an experienced admin, but I don't code outside of scripting languages.)
[22:35] <SpamapS> geek65535: I'm looking now.. it may be an issue with the way ifupdown delegates this duty to the bridge plugin
[22:36] <TimothyA> SpamapS: it did not
[22:36] <TimothyA> and it locks up my screen when I try to do it manually
[22:36] <SpamapS> TimothyA: "locks up my screen" meaning the terminal does not return?
[22:37] <TimothyA> yep
[22:37] <SpamapS> geek65535: ok, this is handled by /etc/network/if-pre-up.d/bridge ...
[22:38] <SpamapS> geek65535: of particular interest to you may be the MAXWAIT= area
[22:39] <SpamapS> geek65535: bridge_maxwait 10 would solve your problem ;)
[22:40] <SpamapS> geek65535: (from man bridge-utils-interfaces)
[22:40] <SpamapS> geek65535: actualy, bridge_fd seems more appropriate
[22:41] <SpamapS> geek65535: I suspect yours is failing because brctl showstp bridge0 is not reporting forward delay / bridge forward delay properly
[22:42] <SpamapS> geek65535: its usually 15.00 which should be 15 seconds.. so maybe you need more?
[22:42] <geek65535>  SpamapS: I have maxwait at 6  and bridge_fd at 0, allong with bridge_hello at 0 (although that shouldn't matter with stp off).
[22:43] <SpamapS> TimothyA: ok, well that still should work fine. When you can, pastebin the upstart job file and we can try to help
[22:43] <TimothyA> SpamapS: the python script already implements a signal handler.
[22:43] <SpamapS> TimothyA: thats not realy important in 'start'
[22:43] <SpamapS> TimothyA: signals are only sent on 'reload' and 'stop' (HUP, and TERM, then KILL)
[22:43] <TimothyA> exec python main.py
[22:43] <TimothyA> that's about all
[22:46] <geek65535>  SpamapS: upped maxwait to 20, rebooting to see if it make a difference.
[22:48] <geek65535>  SpamapS: no joy.
[22:48] <SpamapS> geek65535: how strange
[22:49] <SpamapS> geek65535: another possibility is that brctl is showing it as ready, when its not
[22:49] <geek65535>  SpamapS: I'd buy that.
[22:50] <SpamapS> geek65535: because maxwait is just how long it waits for brctl showstp to show 'listening, learning, forwarding, blocking'
[22:51] <SpamapS> oh oh oh .. I think that maxwait is 10th's of seconds
[22:52] <geek65535>  SpamapS: 10ths? That would make a difference!
[22:53] <SpamapS> yeah
[22:53] <SpamapS> well it sleeps 0.1, then inc's count +1
[22:53] <SpamapS> so yeah, 10ths
[22:53] <SpamapS> ok, I have to run
[22:53] <geek65535>  SpamapS: me, too, actually. dinnertime...
[22:54] <SpamapS> geek65535: I think you have enough to take a look. If you find that the script is doing something wrong, or brctl is, I urge you to do an 'ubuntu-bug bridge-utils' or 'ubuntu-bug linux' ;) good luck
[22:55] <geek65535>  SpamapS: thanks!