[00:11] <tmpRAOF> doko: How can I usefully review this request? The diff between the two packages is huge (there's 42K changed lines in one of the debian patches alone), gccgo doesn't have a MRE, and enabling new features is generally against the purpose of SRUs.
[00:12] <tmpRAOF> doko: So, I guess one question is why are we enabling cgo on aarch64 in an SRU?
[00:26] <tmpRAOF> Ok. I'm going to leave the trusty queue at the same length as when I started processing it.
[00:26] <tmpRAOF> BAh.
[00:33] <doko> tmpRAOF, let me discuss this with slangasek, it's not urgent
[00:33] <tmpRAOF> doko: As is evident from the way that gccgo update has sat in the queue since November 2014 :)
[00:34] <tmpRAOF> Ok. Thanks.
=== tmpRAOF is now known as RAOF
=== doko_ is now known as doko
=== txspud|ORS is now known as txspud
[13:05] <smoser> hey, can I get a manual override of failing systemd tests ?
[13:06] <smoser> and let systemd through from proposed
[13:06] <smoser> https://jenkins.qa.ubuntu.com/job/vivid-adt-systemd/ARCH=amd64,label=adt/121/consoleFull is the failure, versus https://jenkins.qa.ubuntu.com/job/vivid-adt-systemd/ARCH=amd64,label=adt/119/consoleFull is the pass.
[13:06] <smoser> but something else changed. the failure fails on installation of upstart-bin
[13:06] <smoser> when the pass never did the installation ("upstart-bin is already the newest version.")
[13:07] <cjwatson> No, the failure was not on installation of upstart-bin.
[13:08] <cjwatson> It clearly gets past that.
[13:08] <smoser> ?
[13:08] <cjwatson> It says "Processing triggers" etc. before it fails.
[13:08] <smoser> something caused debconf prompt during installation of upstart-bin
[13:08] <cjwatson> That isn't consistent with the log.
[13:08] <smoser> the pass it never installed upstart-bin
[13:09] <cjwatson> It gets past upstart-bin's postinst and into the libc-bin trigger.
[13:09] <cjwatson> If it's a debconf prompt, it's in the libc-bin trigger, not in upstart-bin.
[13:11] <cjwatson> But libc-bin.postinst doesn't use debconf.
[13:11] <cjwatson> So I'm curious where this debconf theory comes from.
[13:12] <smoser> cjwatson, http://paste.ubuntu.com/10620817/ ?
[13:13] <smoser> maybe thats noise
[13:13] <cjwatson> That's irrelevant.
[13:13] <infinity> It is.  Minor chroot misconfiguration, but not the failure.
[13:13] <cjwatson> Although it might be a good idea for the systemd test to use DEBIAN_FRONTEND=noninteractive.
[13:14] <cjwatson> (In debian/tests/cmdline-upstart-boot)
[13:16] <cjwatson> Given that this test is doing a reboot, and that pitti added that code, it seems likely to be a non-trivial interaction between the reboot logic, autopkgtest, and the testbed
[13:16] <infinity> Didn't there used to be a big retry button on this jenkins UI somewhere?
[13:17] <cjwatson> It's on d-jenkins.ubuntu-ci:8080, not on the public jenkins.
[13:17] <apw> infinity, and only if you are logged in
[13:17] <infinity> cjwatson: Yes, that's where I am.
[13:17] <apw> top right and tiny iirc
[13:17] <cjwatson> It's already been retried, though.
[13:18] <infinity> Oh, wait.  http://d-jenkins.ubuntu-ci:8080/view/Vivid/view/AutoPkgTest/job/vivid-adt-systemd/ <-- Build Now in the top left?
[13:18] <cjwatson> Anyway, the retry facility is vivid-adt-systemd -> Build Now
[13:18] <cjwatson> yeah
[13:18] <cjwatson> I wonder what version of autopkgtest is being used here?
[13:19] <cjwatson> jibel: ^-
[13:19] <infinity> Jenkins makes me wonder if life as a serial killer would really be all that bad.
[13:20]  * smoser was just thinking "wow, this jenkins is fast!". infinity you just need to use a very slow jenkins for a while, then you'll think its great.
[13:20] <smoser> ok.. so i cannot rcreate failure on 'apt-get install upstart-bin' from -proposed. on a cloud image.
[13:20] <cjwatson> smoser: But that's not where it's failing!  It's failing on the *reboot*.
[13:21] <cjwatson> <VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0"
[13:21] <jibel> cjwatson, it uses git rev. 73ad9a5 which is from 14 days ago
[13:21] <cjwatson> The test process requests a reboot after installing upstart-bin, it reboots, waits five minutes, never gets a login prompt.
[13:23] <cjwatson> So my worry here about overriding this is that it is in fact indicating that the one-time boot with upstart isn't working, which would be bad.
[13:23] <smoser> yeah. sorry for being dense. just tested that, and 'reboot' worked . system came back up, console had login prompt.
[13:23] <cjwatson> But I'd suggest trying to reproduce this locally with adt-run.
[13:23] <cjwatson> http://packaging.ubuntu.com/html/auto-pkg-test.html#executing-the-test
[13:25] <smoser> yeah.
[13:25] <cjwatson> Not that it's my decision any more, but overriding failing tests of our init daemon is a pretty scary thing to ask for.  What if some later failure is masked by this?
[13:26] <jibel> cjwatson, it's a different revision than on other nodes.
[13:26] <infinity> cjwatson: You're still a core-dev, you get to have opinions.
[13:26] <cjwatson> Oh, sure, I just mean that I no longer get to commit overrides :)
[13:27] <infinity> cjwatson: Oh, true, but that inability aligns with the correct course of action (doing nothing), so it all works out.
[13:31] <cjwatson> Heh.
[13:33] <cjwatson> jibel: Mm, I wonder if the changes since then would help.  There are some things related to rebooting, but I don't know if we're using /sbin/autopkgtest-reboot
[13:33] <jibel> cjwatson, yes, I'm double-checking the configuration of the other nodes then will update it
[13:34] <cjwatson> jibel: It's failed previously on aldebaran too.
[13:34] <cjwatson> http://d-jenkins.ubuntu-ci:8080/view/Vivid/view/AutoPkgTest/job/vivid-adt-systemd/120/ARCH=amd64,label=adt/
[14:02] <didrocks> smoser: cjwatson: seems to be a real regression/issue (maybe only in the testbed) anyway
[14:02] <didrocks> I can reproduce it under adt-run here
[14:02] <didrocks> tried latest autopkgtest, the one beforehand as well (last version where systemd passed with)
[14:02] <didrocks> and they are all stuck when rebooting with upstart
[14:02] <didrocks> other reboots are fine
[14:02] <didrocks> I tried to not get the latest grub-legacy-ec2 update, still the same
[14:03] <didrocks> so I get my cloud/adt image is after the regression went in though
[14:03] <didrocks> (just tried the same test, commenting the grub changes to boot to upstart, and the reboot pass)
[14:04] <smoser> didrocks, so you're saying that you can recreate pass of '219-4ubuntu5'  and failure of ' 219-4ubuntu6' ?
[14:04] <didrocks> smoser: sorry, no, I meant this was already failing with 219-4ubuntu5 as well
[14:04] <smoser> oh. no. you're saying with latest autopkgtest they both hang.
[14:04] <smoser> right.
[14:04] <didrocks> just it's not systemd, not autopkgtest, not grub-legacy-ec2
[14:05] <didrocks> and only when rebooting with upstart
[14:06] <didrocks> (I wonder why pitti is sedding in /boot/grub/menu.lst, or maybe some other tests don't pass under some testbed config as I just call update-grub on some other tests)
[14:08] <didrocks> ok, only calling update-grub and rebooting doesn't fail, so what triggers the issue is line 75
[14:18] <didrocks> the sed and updated default init cmdline are sane
[14:52] <smoser> didrocks, line 75 where ?
[14:54] <didrocks> smoser: debian/tests/cmdline-upstart-boot
[14:56] <smoser>  /boot/grub/menu.lst isn't really read anywhere to my knowledge except for 'pvgrub' boot (on ec2 instances)
[15:09] <didrocks> smoser: anyway, on my local adt, it's clearly the sed setting upstart by default which causes this issue at reboot
[15:09] <didrocks> I just checked, /sbin/upstart is installed
[15:10] <didrocks> and I didn't see anything especially related since last systemd upload to vivid being the define root cause of that issue
[15:11] <didrocks> I guess another approach would be to take a cloud image a week old, and then, doing partial upgrades until finding the culpurit
[15:11] <smoser> well, one thing that changed, although i'm not quite sure how it affected this
[15:11] <didrocks> smoser: did you try to reboot on a cloud image using upstart?
[15:11] <smoser> is that cloud images *had* init=/lib/systemd/init
[15:11] <smoser> and now they do not.
[15:13] <didrocks> the grub patches handles those situations and only add the overriden init= (and the config file looks good)
[15:15] <smoser> right. i thought so too
[15:21] <didrocks> smoser: do you have handy some week-old images? I think it worthes giving it a simple try
[15:22] <smoser> didrocks, i'll look.
[15:23] <smoser> so it does fail to boot.
[15:23] <smoser> i can verify that. that taking current instance, adding 'init=/sbin/upstart' and rebooting fails
[15:23] <didrocks> right
[15:24] <smoser> http://paste.ubuntu.com/10621472/
[15:25] <didrocks> smoser: do we have ttyS0 accessible?
[15:26] <didrocks> as it seems it's what adt is using
[15:26] <smoser> well that is from ttyS0
[15:26] <didrocks> VirtSubproc.expect(term, b' login: ', 300, 'login prompt on ttyS0', …)
[15:27] <didrocks> so, we don't have the login prompt?
[15:34] <didrocks> smoser: just used a recent cloud image, and after switching to upstart, can't boot as well (qemu stays on "Booting from hard disk…")
[15:35] <smoser> didrocks, well, you need to look at console.
[15:35] <smoser> serial
[15:35] <smoser> probalby. i'lll look
[15:36] <cjwatson> I wonder if the Upstart job for ttyS0 isn't in place
[15:36] <cjwatson> That was traditionally set up by the installer if booting using Upstart
[15:37] <cjwatson> systemd's autopkgtest might need to put it in place
[15:38] <cjwatson> See http://bazaar.launchpad.net/~ubuntu-core-dev/finish-install/ubuntu/view/head:/finish-install.d/90console, presumably something similar somewhere for cloud images
[15:42] <didrocks> the job is present in cloud image though
[15:43] <smoser> cjwatson, its nto that.
[15:43] <smoser> booted current systemd cloud instance. install upstart-bin, chnage grub to init=/sbin/upstart
[15:43] <smoser> reboot
[15:43] <smoser> then... cloud-init-nonet is blocking.
[15:45] <smoser> http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/cloud-init/vivid/view/head:/upstart/cloud-init-nonet.conf
[15:46] <smoser> that normally gets hooked all through the /etc/network/if-up.d/upstart .
[15:46] <smoser> so it seems that that isnt working
[15:47] <didrocks> hum, latest ifupdown was uploaded quite a while ago…
[15:47] <smoser> well, udev fires events that call upstart net sutff... that calls /etc/network/if-up.d/upstart ..
[15:49] <smoser> i suspect it is 'init_is_upstart;' in /etc/network/if-up.d/upstart
[15:50] <cjwatson> Well, it doesn't have any non-Upstart logic past that
[15:50] <cjwatson> And with systemd you surely aren't waiting for the "static-network-up" Upstart event
[15:50] <cjwatson> Or "net-device-up"
[15:51] <cjwatson> Oh, wait, I can't read
[15:51] <cjwatson>    if [ -x /sbin/initctl ] && /sbin/initctl version 2>/dev/null | /bin/grep -q upstart; then
[15:51] <cjwatson>        return 0
[15:51] <cjwatson>    fi
[15:51] <cjwatson> That should work if init=/sbin/upstart?
[15:52] <smoser> thats what i think is failing
[15:52] <smoser> but i'll test that.
[15:52] <cjwatson> It would be perplexing for that to fail if pid 1 is upstart
[15:52] <didrocks> it's not
[15:52] <didrocks> I was looking at that
[15:53] <didrocks> even if pid1 is systemd, this works if upstart-bin is installed
[15:54] <cjwatson> Hm, yeah, that's kind of the wrong test when what we care about is pid 1, but it should be incorrect in a direction that's harmless here
[15:54] <didrocks> yeah, it shouldn't matter for that case, it's just that it will have a wrong claim under systemd :)
[15:54] <didrocks> so yeah, as you told, this shouldn't fail anyway…
[15:55] <smoser> so i commented out that 'if ! init_is_upstart'
[15:55] <smoser> and still fails
[15:56] <didrocks> yeah, at least, it makes sense
[15:56] <smoser> right. but for some reason, it sure appears that static-network-up is not getting run
[15:57] <smoser> indeed, it even looks like networking is just not coming up at all
[15:57] <smoser> because even after those timeouts, the node isnt there.
[15:57] <smoser> network wise.
[16:13] <smoser> ok., this is silly.
[16:13] <smoser> didrocks, http://paste.ubuntu.com/10621727/
[16:13] <smoser> 'apt-get install upstart-bin' does not get 'upstart'
[16:13] <smoser> which means none of that stuff is there.
[16:13] <smoser> which just can't be expected to boot.
[16:13] <smoser> jodh, ^
[16:15] <smoser> and previously, when these things were passing.. upstart was installed
[16:16] <smoser> ok. so essentially, the test used to work because 'upstart' was installed.
[16:16] <didrocks> smoser: ahah, good catch! I was thinking the upstart package was way more empty (only the basic tools)
[16:17] <didrocks> so, I guess we should move some jobs to -bin or a -common package
[16:17] <smoser> test is bogus. or woudl require a lot more work to make it work.
[16:17] <smoser> yeah.
[16:18] <smoser> i'm guessing its /etc/init/upstart-udev-bridge.conf
[16:18] <didrocks> well, the test doesn't want "upstart" itself to be installed, really, we want to try having systemd as default and having upstart that we can optionnally boot
[16:18] <smoser> that was the "no network" issue
[16:18] <didrocks> so yeah, moving the jobs would fix it
[16:18] <infinity> Moving the jobs is "hard".
[16:18] <smoser> right.
[16:18] <infinity> We discussed redoing the split instead to match systemd.
[16:19] <infinity> So, "upstart" would be jobs and binaries, and "upstart-sysv" would be the overlapping bits that make it boot by default.
[16:19] <infinity> I might do that this afternoon.
[16:19] <didrocks> infinity: I guess you are telling this because they all are conffiles, right?
[16:19] <infinity> didrocks: Right.
[16:19] <smoser> so... can we let that -proposed through ?
[16:19] <didrocks> yeah, the upstart-sysv sounds better as matching what systemd has
[16:19] <infinity> smoser: Yeah, now that it's been root-caused, we can let it through.
[16:20] <didrocks> smoser: can you run all the other tests locally?
[16:20] <didrocks> as this one was in the middle of the stack
[16:20] <didrocks> I can deal with it if you prefer (but probably tomorrow for me)
[16:20] <infinity> ubuntu-drivers-common being completely broken is fun too.
[16:22] <smoser> ok. i'll run tests
[16:22] <didrocks> thanks :)
[16:24] <smoser> i can reasonably expect to run them on trusty ?
[16:24] <smoser> didrocks, ^ as host.
[16:24] <didrocks> smoser: oh sure, as long as you have a vivid adt cloud image
[16:24] <smoser> right
[16:24] <didrocks> and just comment the upstart test in debian/test/control
[16:24] <didrocks> upstart*
[16:39] <smoser> bah. didrocks doesnt seem to want to run on trusty
[16:40] <smoser> http://paste.ubuntu.com/10621864/ and eventual hang
[16:41] <smoser> http://paste.ubuntu.com/10621878/
[16:43] <didrocks> smoser: argh, ok… do you mind if I run them tomorrow morning? I'm finishing up another systemd issue and will wrap up then
[17:01] <smoser> didrocks, yeah. i have a fix i want to get in to systemd also .
[17:01] <smoser> for http://pad.lv/1432829
[17:01] <ubot93> Launchpad bug 1432829 in resolvconf (Ubuntu) "resolvconf not updated correctly for interfaces configured in initramfs" [Undecided,New]
[17:03] <didrocks> smoser: ok, I only gives debdiff against experimental (we are using a git here, having an ubuntu branch), but my patches are not as urgent as yours I guess (they just need to be in for vivid)
[18:04] <bdmurray> cjwatson: How is regression detection done in http://people.canonical.com/~ubuntu-archive/proposed-migration/trusty/update_excuses.yaml?
[18:05] <bdmurray> cjwatson: I ask as apport version 2.14.1-0ubuntu3.8 is failing but so did 2.14.1-0ubuntu3.7
[18:06] <cjwatson> bdmurray: It's up to autopkgtest.py, but I think it's "has this test ever passed at all in this series"
=== Streamstormer_ is now known as Streamstormer