/srv/irclogs.ubuntu.com/2011/04/01/#upstart.txt

[diablo]MaddaGaska, you or me take the lead?11:11
MaddaGaskadiablo: Happy for you to.11:12
[diablo]OK11:12
[diablo]guys, we have a problem with libvirt starting prior to multipath in Ubuntu 10.04 11:12
[diablo]so our LVM pool is not being started 11:13
[diablo]how can we please modify the start on runlevel [2345] so that it checks that multipath is running (but this is not an upstart job)11:13
JanCsomehow you need an upstart event that the libvirt upstart job can wait for12:09
soren[diablo]: So, for instance, extend the multipath init script to call: "initctl emit multipath-up"12:46
soren[diablo]: ...and add that to your libvirt job's "start" line.12:46
soren"that" being "and multipath-up".12:47
[diablo]hi soren , just reading12:47
[diablo]MaddaGaska, is the one actually with the box12:47
[diablo]MaddaGaska, can you test that please12:48
MaddaGaskaSounds worth a shot. Have just tried something recommended in a libvirt reported bug with an iSCSI device not coming up, server is still rebooting.12:48
MaddaGaskaAnd when I say 'still rebooting', I may mean that I've bricked it. Oh well, I wanted to do a reinstall today, and there's one elft.12:48
MaddaGaska*left12:48
MaddaGaskaTrying it on server 2 now.12:49
[diablo]k12:50
MaddaGaskalibvirt job should then say: start on started multipath-up ?12:52
JanCyou probably want more than only that12:55
MaddaGaskaPossibly... though it doesn't state any dependencies in its default- it simply states runlevels 2345, and multipath-tools theoretically starts at the same run levels according to its init.d...12:58
MaddaGaskaSo based on that I don't think we'll run into dependency problems as it's not guaranteed to start before or after any specific services at runlevel 2 normally?12:58
MaddaGaskaAh well... libvirt started but we're still running into the same problem so apparently that's not where the issue lies.12:59
MaddaGaskaThanks soren/JanC13:00
MaddaGaska(Rearrange in desired order of precedence)13:00
JanCMaddaGaska: oh, and you want "start on multipath-up"13:04
JanCnot "start on started multipath-up"13:05
MaddaGaskaJanC: Yeah, that's what I put after rereading one of the upstart help pages.13:05
MaddaGaska(Upstart Intro, cook book, and best practises)13:05
JanCalso, maybe you need to wait for the multipath device to be ready13:06
MaddaGaskaI'm assuming I got that bit right because otherwise libvirt wouldn't have started at all... hoping not to have to resort to an ugly hack here like just restarting libvirt when it finishes booting.13:06
JanC(I missed part of the discussion, so not sure when the event is emitted)13:06
MaddaGaskaThis may be true...13:07
MaddaGaskaI've put the initctl command just before the exit 0 at the end of /etc/init.d/multipath-tools13:07
MaddaGaskaSo yeah, it could be taking its time actually bringing up the specific multipath device13:08
JanCI've never used multipath, but isn't there some way to "hook" into it, e.g. when a device comes up or goes down?13:09
MaddaGaskaGood question... I'll dive into the man page13:10
JanCsomething similar to /etc/network/if-up.d/upstart etc.13:12
MaddaGaskaThe only thing related to it I can find is /etc/multipath.conf, and I can't find anything in there to fire triggers when it brings up a device13:14
MaddaGaskaI've raised this in OFTC:/#virt as well, but it seems to be asleep.13:15
MaddaGaskaHmm... is there an upstart event when the system is ready for user logon?14:00
MaddaGaskaNot for when a user logs on, because this server will be unattended.14:00
MaddaGaskaOkay. solved with a dirty hack in libvirt's upstart script to make it sleep for 30 seconds before starting.14:32
JanCisn't there an udev event when the device becomes available?14:49
MaddaGaskaPossibly... I'll see if I can get it to show me one...14:52
MaddaGaskaIs there a replacement for the command 'initctl events'?14:57
MaddaGaskaTo show events in real time?14:57
MaddaGaskaAccording to the upstart-udev-bridge, it's not to be relied on. Is there any best-practise reason not to ensure it is up by means of, for example, calling vgdisplay -s on the expected volume group, then sleeping for a second if it's not up?15:11
MaddaGaskae.g. while [ ! vgdisplay -s ] ; do ; sleep 1 ; done15:11
MaddaGaska*with the vg name after -s15:12
JanCMaddaGaska: I think the "not to be relied on" phrase is about the fact that somewhere in the future 'upstart-udev-bridge' will be replaced by built-in upstart functionality (without guarantees that things will keep working exactly the same)15:18
* MaddaGaska nods15:18
MaddaGaskaThat was my reading of it too, hence my reluctance to use it- because if it changes then suddenly we have a pair of kvm hosts that don't actually bring up their guests automatically.15:19
JanCI assume that would only happen after a release upgrade though15:20
MaddaGaskaHmm... fair point, we're on an LTS release so it wouldn't be a problem for a couple of years...15:22
JanCMaddaGaska: now that I think about it, you need to wait until *LVM* is ready, which will be somewhere after multipath is started, so indeed that's where you need to hook into or what you need to monitor or whatever15:22
MaddaGaskaYup, that's what I've done in a slightly hackish way15:22
MaddaGaskawhile ! vgdisplay -s vms15:22
MaddaGaskado15:22
MaddaGaska sleep 0.515:22
MaddaGaskadone15:22
MaddaGaskaIn the pre-start script15:22
MaddaGaskaSo it shouldn't actually bring up libvirt until it sees the storage group it wants.15:22
MaddaGaskaAlthough I'll check the /etc/init- I'm not sure whether LVM is run as an upstart service?15:23
MaddaGaskaBingo15:25
JanCmdadm is still a sysvinit script it seems...15:26
MaddaGaskaThat's lvm's?15:26
MaddaGaskaSo, in absence of libvirt actually allowing time for a logical volume group to come up, does this look like a reasonable workaround to you? 15:28
JanCit looks reasonable, although I would expect there is some way to handle this with udev-originating events too15:31
MaddaGaskaThat would be a better way to do it, thinking about it... but I think I'll work with this as an acceptable workaround as I've spent most of today trying to get it working.15:48
MaddaGaskaThanks for your help.15:51
SpamapSshaatar: you may want to try piping the command to logger to see if it is printing out errors16:42
SpamapSoh hahahaha16:42
SpamapSI love when that happens16:42
* SpamapS was scrolled back 24 hours16:42
* SpamapS vows to implement a big red "you are scrolled back" reminder indicator for irssi..16:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!