| geek65535 | Hope someone can help: I'm having trouble with gluster where the daemon 'glusterd' is starting before the network is properly up, despite using 'start on static-network-up'. My network contains a bridge (for KVM use), and it appears to appear up to upstart (ouch) before it's actually up and passing packets. Is there any way I can make glusterd wait on actual network connectivity through the bridge? | 18:48 |
|---|---|---|
| SpamapS | geek65535: how are you configuring the bridge? | 18:50 |
| geek65535 | SpamapS: so sorry! Got distracted by the real world. (I hate when that happens). This is my current config: | 21:20 |
| geek65535 | iface bridge0 inet static | 21:20 |
| geek65535 | address 10.8.1.21 | 21:20 |
| geek65535 | netmask 255.255.255.128 | 21:20 |
| geek65535 | network 10.8.1.0 | 21:20 |
| geek65535 | broadcast 10.8.1.127 | 21:20 |
| geek65535 | gateway 10.8.1.1 | 21:20 |
| geek65535 | # dns-* options are implemented by the resolvconf package, if installed | 21:20 |
| geek65535 | dns-nameservers 10.8.1.33 10.8.1.44 | 21:20 |
| geek65535 | dns-search houston8.topgolf.com | 21:20 |
| geek65535 | bridge_ports eth0 | 21:20 |
| geek65535 | bridge_stp off | 21:20 |
| geek65535 | bridge_maxwait 6 | 21:20 |
| TimothyA | I seem to have a slight problem with upstart when trying to run a python script as a service; when I try to manually invoke the process, it will just lock up the screen until I press ctrl-c. any ideas how to fix this? | 21:46 |
| TimothyA | what is the recommended approach for starting python scripts with upstart? | 21:51 |
| TimothyA | and now the server has become unusable... | 22:05 |
| SpamapS | TimothyA: python scripts aren't special over any other thing | 22:08 |
| SpamapS | TimothyA: can you pastebin your upstart script? | 22:09 |
| SpamapS | geek65535: do you have 'auto bridge0' in /etc/network/interfaces ? | 22:09 |
| TimothyA | SpamapS: I can't. the server has become unreachable. | 22:12 |
| geek65535 | SpamapS: yes. I just missed it on the copy-and-paste | 22:13 |
| SpamapS | geek65535: ok, static-network-up *should* wait for that.. you can see the exact time static-network-up was emitted by looking at the mtime of the dir /run/network/static-network-up-emitted ... | 22:21 |
| SpamapS | geek65535: perhaps ifup returns before the bridge is actually working? | 22:21 |
| TimothyA | SpamapS: removed the file and rebooted server | 22:23 |
| TimothyA | but it's nothing more than a basic respawn and exec python main.py lines | 22:23 |
| geek65535 | SpamapS: let me take a look... | 22:24 |
| SpamapS | TimothyA: yeah, that should work. If you don't have any post-start or expect lines, it should return immediately on 'start your-job-name' | 22:24 |
| geek65535 | run/network/static-network-up-emitted is empty... | 22:27 |
| geek65535 | er, sorry, missed the 'mtime' part... | 22:27 |
| SpamapS | geek65535: right, its the dir itself that is used as a flag | 22:27 |
| SpamapS | geek65535: you should also see the ifup.bridge0 there, with an earlier mtime/ctime | 22:28 |
| geek65535 | root@vmhost1:/run/network# ls -l --full-time | 22:28 |
| geek65535 | total 4 | 22:28 |
| geek65535 | -rw-r--r-- 1 root root 22 2012-10-17 17:26:27.217981181 -0500 ifstate | 22:28 |
| geek65535 | -rw-r--r-- 1 root root 0 2012-10-17 17:26:27.209981181 -0500 ifup.bridge0 | 22:28 |
| geek65535 | -rw-r--r-- 1 root root 0 2012-10-17 17:26:24.089981092 -0500 ifup.lo | 22:28 |
| geek65535 | drwxr-xr-x 2 root root 40 2012-10-17 17:26:27.209981181 -0500 static-network-up-emitted | 22:28 |
| SpamapS | geek65535: right, so ifup is the problem then | 22:29 |
| SpamapS | geek65535: it must return before the bridge is actually working | 22:29 |
| geek65535 | That's what I had been thinking. I just need to find something that emits a signal after the bridge is active. (Or live with the 'sleep 10' stuck in my /etc/init/glusterd.conf) | 22:31 |
| SpamapS | geek65535: thats a bug.. perhaps we should find it and fix it :) | 22:31 |
| geek65535 | What can I do to help? (I'm an experienced admin, but I don't code outside of scripting languages.) | 22:33 |
| SpamapS | geek65535: I'm looking now.. it may be an issue with the way ifupdown delegates this duty to the bridge plugin | 22:35 |
| TimothyA | SpamapS: it did not | 22:36 |
| TimothyA | and it locks up my screen when I try to do it manually | 22:36 |
| SpamapS | TimothyA: "locks up my screen" meaning the terminal does not return? | 22:36 |
| TimothyA | yep | 22:37 |
| SpamapS | geek65535: ok, this is handled by /etc/network/if-pre-up.d/bridge ... | 22:37 |
| SpamapS | geek65535: of particular interest to you may be the MAXWAIT= area | 22:38 |
| SpamapS | geek65535: bridge_maxwait 10 would solve your problem ;) | 22:39 |
| SpamapS | geek65535: (from man bridge-utils-interfaces) | 22:40 |
| SpamapS | geek65535: actualy, bridge_fd seems more appropriate | 22:40 |
| SpamapS | geek65535: I suspect yours is failing because brctl showstp bridge0 is not reporting forward delay / bridge forward delay properly | 22:41 |
| SpamapS | geek65535: its usually 15.00 which should be 15 seconds.. so maybe you need more? | 22:42 |
| geek65535 | SpamapS: I have maxwait at 6 and bridge_fd at 0, allong with bridge_hello at 0 (although that shouldn't matter with stp off). | 22:42 |
| SpamapS | TimothyA: ok, well that still should work fine. When you can, pastebin the upstart job file and we can try to help | 22:43 |
| TimothyA | SpamapS: the python script already implements a signal handler. | 22:43 |
| SpamapS | TimothyA: thats not realy important in 'start' | 22:43 |
| SpamapS | TimothyA: signals are only sent on 'reload' and 'stop' (HUP, and TERM, then KILL) | 22:43 |
| TimothyA | exec python main.py | 22:43 |
| TimothyA | that's about all | 22:43 |
| geek65535 | SpamapS: upped maxwait to 20, rebooting to see if it make a difference. | 22:46 |
| geek65535 | SpamapS: no joy. | 22:48 |
| SpamapS | geek65535: how strange | 22:48 |
| SpamapS | geek65535: another possibility is that brctl is showing it as ready, when its not | 22:49 |
| geek65535 | SpamapS: I'd buy that. | 22:49 |
| SpamapS | geek65535: because maxwait is just how long it waits for brctl showstp to show 'listening, learning, forwarding, blocking' | 22:50 |
| SpamapS | oh oh oh .. I think that maxwait is 10th's of seconds | 22:51 |
| geek65535 | SpamapS: 10ths? That would make a difference! | 22:52 |
| SpamapS | yeah | 22:53 |
| SpamapS | well it sleeps 0.1, then inc's count +1 | 22:53 |
| SpamapS | so yeah, 10ths | 22:53 |
| SpamapS | ok, I have to run | 22:53 |
| geek65535 | SpamapS: me, too, actually. dinnertime... | 22:53 |
| SpamapS | geek65535: I think you have enough to take a look. If you find that the script is doing something wrong, or brctl is, I urge you to do an 'ubuntu-bug bridge-utils' or 'ubuntu-bug linux' ;) good luck | 22:54 |
| geek65535 | SpamapS: thanks! | 22:55 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!