| [diablo] | MaddaGaska, you or me take the lead? | 11:11 |
|---|---|---|
| MaddaGaska | diablo: Happy for you to. | 11:12 |
| [diablo] | OK | 11:12 |
| [diablo] | guys, we have a problem with libvirt starting prior to multipath in Ubuntu 10.04 | 11:12 |
| [diablo] | so our LVM pool is not being started | 11:13 |
| [diablo] | how can we please modify the start on runlevel [2345] so that it checks that multipath is running (but this is not an upstart job) | 11:13 |
| JanC | somehow you need an upstart event that the libvirt upstart job can wait for | 12:09 |
| soren | [diablo]: So, for instance, extend the multipath init script to call: "initctl emit multipath-up" | 12:46 |
| soren | [diablo]: ...and add that to your libvirt job's "start" line. | 12:46 |
| soren | "that" being "and multipath-up". | 12:47 |
| [diablo] | hi soren , just reading | 12:47 |
| [diablo] | MaddaGaska, is the one actually with the box | 12:47 |
| [diablo] | MaddaGaska, can you test that please | 12:48 |
| MaddaGaska | Sounds worth a shot. Have just tried something recommended in a libvirt reported bug with an iSCSI device not coming up, server is still rebooting. | 12:48 |
| MaddaGaska | And when I say 'still rebooting', I may mean that I've bricked it. Oh well, I wanted to do a reinstall today, and there's one elft. | 12:48 |
| MaddaGaska | *left | 12:48 |
| MaddaGaska | Trying it on server 2 now. | 12:49 |
| [diablo] | k | 12:50 |
| MaddaGaska | libvirt job should then say: start on started multipath-up ? | 12:52 |
| JanC | you probably want more than only that | 12:55 |
| MaddaGaska | Possibly... though it doesn't state any dependencies in its default- it simply states runlevels 2345, and multipath-tools theoretically starts at the same run levels according to its init.d... | 12:58 |
| MaddaGaska | So based on that I don't think we'll run into dependency problems as it's not guaranteed to start before or after any specific services at runlevel 2 normally? | 12:58 |
| MaddaGaska | Ah well... libvirt started but we're still running into the same problem so apparently that's not where the issue lies. | 12:59 |
| MaddaGaska | Thanks soren/JanC | 13:00 |
| MaddaGaska | (Rearrange in desired order of precedence) | 13:00 |
| JanC | MaddaGaska: oh, and you want "start on multipath-up" | 13:04 |
| JanC | not "start on started multipath-up" | 13:05 |
| MaddaGaska | JanC: Yeah, that's what I put after rereading one of the upstart help pages. | 13:05 |
| MaddaGaska | (Upstart Intro, cook book, and best practises) | 13:05 |
| JanC | also, maybe you need to wait for the multipath device to be ready | 13:06 |
| MaddaGaska | I'm assuming I got that bit right because otherwise libvirt wouldn't have started at all... hoping not to have to resort to an ugly hack here like just restarting libvirt when it finishes booting. | 13:06 |
| JanC | (I missed part of the discussion, so not sure when the event is emitted) | 13:06 |
| MaddaGaska | This may be true... | 13:07 |
| MaddaGaska | I've put the initctl command just before the exit 0 at the end of /etc/init.d/multipath-tools | 13:07 |
| MaddaGaska | So yeah, it could be taking its time actually bringing up the specific multipath device | 13:08 |
| JanC | I've never used multipath, but isn't there some way to "hook" into it, e.g. when a device comes up or goes down? | 13:09 |
| MaddaGaska | Good question... I'll dive into the man page | 13:10 |
| JanC | something similar to /etc/network/if-up.d/upstart etc. | 13:12 |
| MaddaGaska | The only thing related to it I can find is /etc/multipath.conf, and I can't find anything in there to fire triggers when it brings up a device | 13:14 |
| MaddaGaska | I've raised this in OFTC:/#virt as well, but it seems to be asleep. | 13:15 |
| MaddaGaska | Hmm... is there an upstart event when the system is ready for user logon? | 14:00 |
| MaddaGaska | Not for when a user logs on, because this server will be unattended. | 14:00 |
| MaddaGaska | Okay. solved with a dirty hack in libvirt's upstart script to make it sleep for 30 seconds before starting. | 14:32 |
| JanC | isn't there an udev event when the device becomes available? | 14:49 |
| MaddaGaska | Possibly... I'll see if I can get it to show me one... | 14:52 |
| MaddaGaska | Is there a replacement for the command 'initctl events'? | 14:57 |
| MaddaGaska | To show events in real time? | 14:57 |
| MaddaGaska | According to the upstart-udev-bridge, it's not to be relied on. Is there any best-practise reason not to ensure it is up by means of, for example, calling vgdisplay -s on the expected volume group, then sleeping for a second if it's not up? | 15:11 |
| MaddaGaska | e.g. while [ ! vgdisplay -s ] ; do ; sleep 1 ; done | 15:11 |
| MaddaGaska | *with the vg name after -s | 15:12 |
| JanC | MaddaGaska: I think the "not to be relied on" phrase is about the fact that somewhere in the future 'upstart-udev-bridge' will be replaced by built-in upstart functionality (without guarantees that things will keep working exactly the same) | 15:18 |
| * MaddaGaska nods | 15:18 | |
| MaddaGaska | That was my reading of it too, hence my reluctance to use it- because if it changes then suddenly we have a pair of kvm hosts that don't actually bring up their guests automatically. | 15:19 |
| JanC | I assume that would only happen after a release upgrade though | 15:20 |
| MaddaGaska | Hmm... fair point, we're on an LTS release so it wouldn't be a problem for a couple of years... | 15:22 |
| JanC | MaddaGaska: now that I think about it, you need to wait until *LVM* is ready, which will be somewhere after multipath is started, so indeed that's where you need to hook into or what you need to monitor or whatever | 15:22 |
| MaddaGaska | Yup, that's what I've done in a slightly hackish way | 15:22 |
| MaddaGaska | while ! vgdisplay -s vms | 15:22 |
| MaddaGaska | do | 15:22 |
| MaddaGaska | sleep 0.5 | 15:22 |
| MaddaGaska | done | 15:22 |
| MaddaGaska | In the pre-start script | 15:22 |
| MaddaGaska | So it shouldn't actually bring up libvirt until it sees the storage group it wants. | 15:22 |
| MaddaGaska | Although I'll check the /etc/init- I'm not sure whether LVM is run as an upstart service? | 15:23 |
| MaddaGaska | Bingo | 15:25 |
| JanC | mdadm is still a sysvinit script it seems... | 15:26 |
| MaddaGaska | That's lvm's? | 15:26 |
| MaddaGaska | So, in absence of libvirt actually allowing time for a logical volume group to come up, does this look like a reasonable workaround to you? | 15:28 |
| JanC | it looks reasonable, although I would expect there is some way to handle this with udev-originating events too | 15:31 |
| MaddaGaska | That would be a better way to do it, thinking about it... but I think I'll work with this as an acceptable workaround as I've spent most of today trying to get it working. | 15:48 |
| MaddaGaska | Thanks for your help. | 15:51 |
| SpamapS | shaatar: you may want to try piping the command to logger to see if it is printing out errors | 16:42 |
| SpamapS | oh hahahaha | 16:42 |
| SpamapS | I love when that happens | 16:42 |
| * SpamapS was scrolled back 24 hours | 16:42 | |
| * SpamapS vows to implement a big red "you are scrolled back" reminder indicator for irssi.. | 16:43 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!