[10:27] <ttx> cjwatson: I CCed you on the server boot testing thread... there is a worrying tendency on /some/ servers to not display anything at all on console (no login prompt). I'll have fader do another round making sure splash is disabled, but do you have any idea where it could come from ?
[10:31] <slangasek> ttx: either plymouth hanging and failing to give back the console; or something preventing the 'filesystem' event from being emitted, blocking any of the gettys from starting
[10:31] <ttx> slangasek: fader can ssh into them alright, fwiw
[10:32] <slangasek> that leaves option 1)
[10:32] <ttx> slangasek: looks like a display issue. Those servers pass all hw enablement tests, but that doesn't include a visual test
[10:33] <slangasek> do VT switches work?
[10:34] <ttx> slangasek: I'd say no, based on fader's limited description. I suggest we coordinate some time with him to clearly debug that
[10:34] <ttx> I've not reproduced on my hardware
[10:35] <ttx> slangasek: would anything be sensitive to the type of graphics card on the server ?
[10:35] <slangasek> possible, but unlikely
[10:35] <slangasek> it might be sensitive to the disk layout, or whether fsck was called during boot
[10:36] <ttx> I'm pretty sure he runs the same test on the server that work and those that fail.
[10:36] <slangasek> there's mounting evidence that fsck-on-boot breaks plymouth+mountall interaction quite badly; I still haven't had time to reproduce this, but need to in the next day or so
[10:36] <slangasek> right
[10:36] <slangasek> there might be a plain old race condition in there somewhere, too
[10:37] <ttx> Will gather more data with fader and potentially ping one of you guys when we get stuck
[10:37] <slangasek> ttx: in your local tests, are you booting w/o splash?
[10:38] <ttx> in my tests, yes. Fader's tests all failed to get anything on VT7 yesterday, but he pxebooted the installer and got "splash" on the kernel line at the end
[10:39] <ttx> I asked him to run without splash, but I don't think that will affect the lack on login prompt at the end, we had evidence of this issue elsewhere from someone trying out Beta2
[10:40] <ttx> https://lists.ubuntu.com/archives/ubuntu-server/2010-April/004022.html
[10:40] <ttx> That's why I asked to do some testing on the hw enablement servers to get confirmation that it fails or confidence that it works
[10:41] <ttx> Let me sync with fader and get a proper bug report set up.
[10:42] <ttx> At that point we are both blind :)
[10:46] <ttx> slangasek: you might want to keep an eye on bug 557909, in case that's not already on your radar. Seem to be some race with LVM/mountall, I'm hitting it quite steadily on my workstation. Hitting M and mounting manually works.
[10:46] <ubottu> Launchpad bug 557909 in devmapper "lucid hangs on boot because of device ownership" [Undecided,Confirmed] https://launchpad.net/bugs/557909
[10:50] <ttx> bug 561390 seems to be the same LVM/mountall issue
[10:50] <ubottu> Launchpad bug 561390 in mountall "LVM - /var failed to mount during boot" [Undecided,New] https://launchpad.net/bugs/561390
[10:52] <slangasek> "I do an fsck on my root filesystem everytime, when booting (set by 'tune2fs -c 1')."
[10:52] <slangasek> well, no wonder I could never reproduce that bug :P
[10:52] <slangasek> (not the LVM/mountall one, a different one)
[10:52] <ttx> heh
[10:52] <ttx> the LVM thing is definitely a race, I win about half the time
[10:55] <slangasek> the comments in 557909 make no sense
[10:55] <slangasek> if you can reproduce this, can you grab a log of mountall --debug?
[10:56] <ttx> slangasek: how do you do that ?
[10:56] <slangasek> ttx: edit /etc/init/mountall.conf to add --debug and redirect output to a logfile under /dev
[10:56] <ttx> ok
[10:58] <ttx> I'll brb
[10:58] <slangasek> huh, I guess technically we don't need the redirect anymore, you can just pick it up from /var/log/boot.log instead now \o/
[11:08] <ttx> slangasek: log at chinstrap:~ttx/mountall.log
[11:09] <ttx> slangasek: want it attached to that bug ?
[11:09] <slangasek> probably to bug 561390
[11:09] <ubottu> Launchpad bug 561390 in mountall "LVM - /var failed to mount during boot" [Undecided,New] https://launchpad.net/bugs/561390
[11:09] <slangasek> the other one is mostly just noise
[11:09] <ttx> yes
[11:15]  * ttx can't seem to find back the error messages that were present in the log when he manually mounted /home
[11:15]  * ttx investigates again
[11:16]  * ttx grumbles and restarts
[11:40] <ttx> slangasek: done, that was a lot more painful than expected, sorry
[11:47] <slangasek> ttx: and /dev/cassini/cassini-home exists?
[11:48] <slangasek> ttx: I think we need a udev log to go with this, but I'm not sure the right way to grab the log of the udev bits we need
[11:52] <slangasek> oh, 554737 is awesome, plymouth and mountall are deadlocked because they're both in a blocking write
[12:05] <slangasek> also: memory leaks
[12:37] <ttx> slangasek: lrwxrwxrwx 1 root root 33 2010-04-13 12:37 /dev/cassini/cassini-home -> /dev/mapper/cassini-cassini--home
=== rgreening_ is now known as rgreening
[15:00] <NCommander> to any RM around: we're going to push a fix for OOO on ARM to a PPA (we're not 100% sure its going to work due the extreme build times involved). Is there an issue if we copy OOo from the devirtualized PPA to the main archive?
[15:07] <pitti> NCommander: that's done with security all the time, should work
[15:07] <NCommander> pitti: right, just wanted to clear it by RM since my PPA is going to be eating a buildd
[15:08] <NCommander> pitti: alternatively, it might just be worth uploading to the archive properly since the OOo build for ARM is fairly stale and not supportable, so I can't see us being much worse off if we break it more (at least it builds in that case ;-))
[15:19] <jdstrand> NCommander: if this is supposed to go to -security, be sure to build without -proposed and -updates. other than that, you mentioned it is not a virtual ppa, so it should be ok
[15:19] <jdstrand> NCommander: also, if it goes to -security, you may want to communicate with a member of the security team ;)
[15:19] <NCommander> jdstrand: thanks, just wanted to check that this was OK
[15:19] <NCommander> jdstrand: its not for security
[15:20]  * jdstrand nods
[15:20] <jdstrand> NCommander: ubuntu-security-proposed and ubuntu-mozilla-security are two PPAs that we do this sort of thing with all the time
[15:21] <jdstrand> NCommander: the only tricky part is when you do the pocket copy, everything goes to main, so you need to do a change-orverride on anything that shouldn't be in main
[15:21] <jdstrand> NCommander: this is a manual process, so providing that list to the AA that does the pocket copy is a very nice thing to do :)
[16:13] <lamont> so those ppa buildds will be right back.  just fyi
[16:14] <ttx> slangasek, cjwatson: the "no login prompt" server bug is bug 546743
[16:14] <ubottu> Launchpad bug 546743 in linux "Blank screen at first boot with ATI ES1000 and 10.04 server" [High,Confirmed] https://launchpad.net/bugs/546743
[16:15] <ttx> all spottings of that issue so far involve ATi ES1000
[16:17] <ttx> cjwatson: I was wondering if setting nomodeset for server on RC would make sense if we can't workaround this by any other way.
[16:17] <ttx> that would defuse the "you broke my server boot for fancy graphics I don't care about" argument
=== barry is now known as flufl
=== flufl is now known as barry
=== ttx_ is now known as ttx
[21:05] <slangasek> NCommander: what's the point of using a PPA instead of doing the build in the archive directly, though?  Doesn't seem to save any build time, and costs archive admin time later to copy it
[21:08] <NCommander> slangasek: the fix isn't fully tested due to three days of OOo build time. I'm personally not diffed either way, but there was concern that by doing an OOo upload, we can remove the working but stale binaries.
[21:12] <slangasek> ah
[22:08] <lamont> oh hai.  mucking about with ppc for a few minutes
[22:31] <lamont> and ppc is back to the normal fun