| tmpRAOF | doko: How can I usefully review this request? The diff between the two packages is huge (there's 42K changed lines in one of the debian patches alone), gccgo doesn't have a MRE, and enabling new features is generally against the purpose of SRUs. | 00:11 |
|---|---|---|
| tmpRAOF | doko: So, I guess one question is why are we enabling cgo on aarch64 in an SRU? | 00:12 |
| tmpRAOF | Ok. I'm going to leave the trusty queue at the same length as when I started processing it. | 00:26 |
| tmpRAOF | BAh. | 00:26 |
| doko | tmpRAOF, let me discuss this with slangasek, it's not urgent | 00:33 |
| tmpRAOF | doko: As is evident from the way that gccgo update has sat in the queue since November 2014 :) | 00:33 |
| tmpRAOF | Ok. Thanks. | 00:34 |
| === tmpRAOF is now known as RAOF | ||
| === doko_ is now known as doko | ||
| === txspud|ORS is now known as txspud | ||
| smoser | hey, can I get a manual override of failing systemd tests ? | 13:05 |
| smoser | and let systemd through from proposed | 13:06 |
| smoser | https://jenkins.qa.ubuntu.com/job/vivid-adt-systemd/ARCH=amd64,label=adt/121/consoleFull is the failure, versus https://jenkins.qa.ubuntu.com/job/vivid-adt-systemd/ARCH=amd64,label=adt/119/consoleFull is the pass. | 13:06 |
| smoser | but something else changed. the failure fails on installation of upstart-bin | 13:06 |
| smoser | when the pass never did the installation ("upstart-bin is already the newest version.") | 13:06 |
| cjwatson | No, the failure was not on installation of upstart-bin. | 13:07 |
| cjwatson | It clearly gets past that. | 13:08 |
| smoser | ? | 13:08 |
| cjwatson | It says "Processing triggers" etc. before it fails. | 13:08 |
| smoser | something caused debconf prompt during installation of upstart-bin | 13:08 |
| cjwatson | That isn't consistent with the log. | 13:08 |
| smoser | the pass it never installed upstart-bin | 13:08 |
| cjwatson | It gets past upstart-bin's postinst and into the libc-bin trigger. | 13:09 |
| cjwatson | If it's a debconf prompt, it's in the libc-bin trigger, not in upstart-bin. | 13:09 |
| cjwatson | But libc-bin.postinst doesn't use debconf. | 13:11 |
| cjwatson | So I'm curious where this debconf theory comes from. | 13:11 |
| smoser | cjwatson, http://paste.ubuntu.com/10620817/ ? | 13:12 |
| smoser | maybe thats noise | 13:13 |
| cjwatson | That's irrelevant. | 13:13 |
| infinity | It is. Minor chroot misconfiguration, but not the failure. | 13:13 |
| cjwatson | Although it might be a good idea for the systemd test to use DEBIAN_FRONTEND=noninteractive. | 13:13 |
| cjwatson | (In debian/tests/cmdline-upstart-boot) | 13:14 |
| cjwatson | Given that this test is doing a reboot, and that pitti added that code, it seems likely to be a non-trivial interaction between the reboot logic, autopkgtest, and the testbed | 13:16 |
| infinity | Didn't there used to be a big retry button on this jenkins UI somewhere? | 13:16 |
| cjwatson | It's on d-jenkins.ubuntu-ci:8080, not on the public jenkins. | 13:17 |
| apw | infinity, and only if you are logged in | 13:17 |
| infinity | cjwatson: Yes, that's where I am. | 13:17 |
| apw | top right and tiny iirc | 13:17 |
| cjwatson | It's already been retried, though. | 13:17 |
| infinity | Oh, wait. http://d-jenkins.ubuntu-ci:8080/view/Vivid/view/AutoPkgTest/job/vivid-adt-systemd/ <-- Build Now in the top left? | 13:18 |
| cjwatson | Anyway, the retry facility is vivid-adt-systemd -> Build Now | 13:18 |
| cjwatson | yeah | 13:18 |
| cjwatson | I wonder what version of autopkgtest is being used here? | 13:18 |
| cjwatson | jibel: ^- | 13:19 |
| infinity | Jenkins makes me wonder if life as a serial killer would really be all that bad. | 13:19 |
| * smoser was just thinking "wow, this jenkins is fast!". infinity you just need to use a very slow jenkins for a while, then you'll think its great. | 13:20 | |
| smoser | ok.. so i cannot rcreate failure on 'apt-get install upstart-bin' from -proposed. on a cloud image. | 13:20 |
| cjwatson | smoser: But that's not where it's failing! It's failing on the *reboot*. | 13:20 |
| cjwatson | <VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0" | 13:21 |
| jibel | cjwatson, it uses git rev. 73ad9a5 which is from 14 days ago | 13:21 |
| cjwatson | The test process requests a reboot after installing upstart-bin, it reboots, waits five minutes, never gets a login prompt. | 13:21 |
| cjwatson | So my worry here about overriding this is that it is in fact indicating that the one-time boot with upstart isn't working, which would be bad. | 13:23 |
| smoser | yeah. sorry for being dense. just tested that, and 'reboot' worked . system came back up, console had login prompt. | 13:23 |
| cjwatson | But I'd suggest trying to reproduce this locally with adt-run. | 13:23 |
| cjwatson | http://packaging.ubuntu.com/html/auto-pkg-test.html#executing-the-test | 13:23 |
| smoser | yeah. | 13:25 |
| cjwatson | Not that it's my decision any more, but overriding failing tests of our init daemon is a pretty scary thing to ask for. What if some later failure is masked by this? | 13:25 |
| jibel | cjwatson, it's a different revision than on other nodes. | 13:26 |
| infinity | cjwatson: You're still a core-dev, you get to have opinions. | 13:26 |
| cjwatson | Oh, sure, I just mean that I no longer get to commit overrides :) | 13:26 |
| infinity | cjwatson: Oh, true, but that inability aligns with the correct course of action (doing nothing), so it all works out. | 13:27 |
| cjwatson | Heh. | 13:31 |
| cjwatson | jibel: Mm, I wonder if the changes since then would help. There are some things related to rebooting, but I don't know if we're using /sbin/autopkgtest-reboot | 13:33 |
| jibel | cjwatson, yes, I'm double-checking the configuration of the other nodes then will update it | 13:33 |
| cjwatson | jibel: It's failed previously on aldebaran too. | 13:34 |
| cjwatson | http://d-jenkins.ubuntu-ci:8080/view/Vivid/view/AutoPkgTest/job/vivid-adt-systemd/120/ARCH=amd64,label=adt/ | 13:34 |
| didrocks | smoser: cjwatson: seems to be a real regression/issue (maybe only in the testbed) anyway | 14:02 |
| didrocks | I can reproduce it under adt-run here | 14:02 |
| didrocks | tried latest autopkgtest, the one beforehand as well (last version where systemd passed with) | 14:02 |
| didrocks | and they are all stuck when rebooting with upstart | 14:02 |
| didrocks | other reboots are fine | 14:02 |
| didrocks | I tried to not get the latest grub-legacy-ec2 update, still the same | 14:02 |
| didrocks | so I get my cloud/adt image is after the regression went in though | 14:03 |
| didrocks | (just tried the same test, commenting the grub changes to boot to upstart, and the reboot pass) | 14:03 |
| smoser | didrocks, so you're saying that you can recreate pass of '219-4ubuntu5' and failure of ' 219-4ubuntu6' ? | 14:04 |
| didrocks | smoser: sorry, no, I meant this was already failing with 219-4ubuntu5 as well | 14:04 |
| smoser | oh. no. you're saying with latest autopkgtest they both hang. | 14:04 |
| smoser | right. | 14:04 |
| didrocks | just it's not systemd, not autopkgtest, not grub-legacy-ec2 | 14:04 |
| didrocks | and only when rebooting with upstart | 14:05 |
| didrocks | (I wonder why pitti is sedding in /boot/grub/menu.lst, or maybe some other tests don't pass under some testbed config as I just call update-grub on some other tests) | 14:06 |
| didrocks | ok, only calling update-grub and rebooting doesn't fail, so what triggers the issue is line 75 | 14:08 |
| didrocks | the sed and updated default init cmdline are sane | 14:18 |
| smoser | didrocks, line 75 where ? | 14:52 |
| didrocks | smoser: debian/tests/cmdline-upstart-boot | 14:54 |
| smoser | /boot/grub/menu.lst isn't really read anywhere to my knowledge except for 'pvgrub' boot (on ec2 instances) | 14:56 |
| didrocks | smoser: anyway, on my local adt, it's clearly the sed setting upstart by default which causes this issue at reboot | 15:09 |
| didrocks | I just checked, /sbin/upstart is installed | 15:09 |
| didrocks | and I didn't see anything especially related since last systemd upload to vivid being the define root cause of that issue | 15:10 |
| didrocks | I guess another approach would be to take a cloud image a week old, and then, doing partial upgrades until finding the culpurit | 15:11 |
| smoser | well, one thing that changed, although i'm not quite sure how it affected this | 15:11 |
| didrocks | smoser: did you try to reboot on a cloud image using upstart? | 15:11 |
| smoser | is that cloud images *had* init=/lib/systemd/init | 15:11 |
| smoser | and now they do not. | 15:11 |
| didrocks | the grub patches handles those situations and only add the overriden init= (and the config file looks good) | 15:13 |
| smoser | right. i thought so too | 15:15 |
| didrocks | smoser: do you have handy some week-old images? I think it worthes giving it a simple try | 15:21 |
| smoser | didrocks, i'll look. | 15:22 |
| smoser | so it does fail to boot. | 15:23 |
| smoser | i can verify that. that taking current instance, adding 'init=/sbin/upstart' and rebooting fails | 15:23 |
| didrocks | right | 15:23 |
| smoser | http://paste.ubuntu.com/10621472/ | 15:24 |
| didrocks | smoser: do we have ttyS0 accessible? | 15:25 |
| didrocks | as it seems it's what adt is using | 15:26 |
| smoser | well that is from ttyS0 | 15:26 |
| didrocks | VirtSubproc.expect(term, b' login: ', 300, 'login prompt on ttyS0', …) | 15:26 |
| didrocks | so, we don't have the login prompt? | 15:27 |
| didrocks | smoser: just used a recent cloud image, and after switching to upstart, can't boot as well (qemu stays on "Booting from hard disk…") | 15:34 |
| smoser | didrocks, well, you need to look at console. | 15:35 |
| smoser | serial | 15:35 |
| smoser | probalby. i'lll look | 15:35 |
| cjwatson | I wonder if the Upstart job for ttyS0 isn't in place | 15:36 |
| cjwatson | That was traditionally set up by the installer if booting using Upstart | 15:36 |
| cjwatson | systemd's autopkgtest might need to put it in place | 15:37 |
| cjwatson | See http://bazaar.launchpad.net/~ubuntu-core-dev/finish-install/ubuntu/view/head:/finish-install.d/90console, presumably something similar somewhere for cloud images | 15:38 |
| didrocks | the job is present in cloud image though | 15:42 |
| smoser | cjwatson, its nto that. | 15:43 |
| smoser | booted current systemd cloud instance. install upstart-bin, chnage grub to init=/sbin/upstart | 15:43 |
| smoser | reboot | 15:43 |
| smoser | then... cloud-init-nonet is blocking. | 15:43 |
| smoser | http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/cloud-init/vivid/view/head:/upstart/cloud-init-nonet.conf | 15:45 |
| smoser | that normally gets hooked all through the /etc/network/if-up.d/upstart . | 15:46 |
| smoser | so it seems that that isnt working | 15:46 |
| didrocks | hum, latest ifupdown was uploaded quite a while ago… | 15:47 |
| smoser | well, udev fires events that call upstart net sutff... that calls /etc/network/if-up.d/upstart .. | 15:47 |
| smoser | i suspect it is 'init_is_upstart;' in /etc/network/if-up.d/upstart | 15:49 |
| cjwatson | Well, it doesn't have any non-Upstart logic past that | 15:50 |
| cjwatson | And with systemd you surely aren't waiting for the "static-network-up" Upstart event | 15:50 |
| cjwatson | Or "net-device-up" | 15:50 |
| cjwatson | Oh, wait, I can't read | 15:51 |
| cjwatson | if [ -x /sbin/initctl ] && /sbin/initctl version 2>/dev/null | /bin/grep -q upstart; then | 15:51 |
| cjwatson | return 0 | 15:51 |
| cjwatson | fi | 15:51 |
| cjwatson | That should work if init=/sbin/upstart? | 15:51 |
| smoser | thats what i think is failing | 15:52 |
| smoser | but i'll test that. | 15:52 |
| cjwatson | It would be perplexing for that to fail if pid 1 is upstart | 15:52 |
| didrocks | it's not | 15:52 |
| didrocks | I was looking at that | 15:52 |
| didrocks | even if pid1 is systemd, this works if upstart-bin is installed | 15:53 |
| cjwatson | Hm, yeah, that's kind of the wrong test when what we care about is pid 1, but it should be incorrect in a direction that's harmless here | 15:54 |
| didrocks | yeah, it shouldn't matter for that case, it's just that it will have a wrong claim under systemd :) | 15:54 |
| didrocks | so yeah, as you told, this shouldn't fail anyway… | 15:54 |
| smoser | so i commented out that 'if ! init_is_upstart' | 15:55 |
| smoser | and still fails | 15:55 |
| didrocks | yeah, at least, it makes sense | 15:56 |
| smoser | right. but for some reason, it sure appears that static-network-up is not getting run | 15:56 |
| smoser | indeed, it even looks like networking is just not coming up at all | 15:57 |
| smoser | because even after those timeouts, the node isnt there. | 15:57 |
| smoser | network wise. | 15:57 |
| smoser | ok., this is silly. | 16:13 |
| smoser | didrocks, http://paste.ubuntu.com/10621727/ | 16:13 |
| smoser | 'apt-get install upstart-bin' does not get 'upstart' | 16:13 |
| smoser | which means none of that stuff is there. | 16:13 |
| smoser | which just can't be expected to boot. | 16:13 |
| smoser | jodh, ^ | 16:13 |
| smoser | and previously, when these things were passing.. upstart was installed | 16:15 |
| smoser | ok. so essentially, the test used to work because 'upstart' was installed. | 16:16 |
| didrocks | smoser: ahah, good catch! I was thinking the upstart package was way more empty (only the basic tools) | 16:16 |
| didrocks | so, I guess we should move some jobs to -bin or a -common package | 16:17 |
| smoser | test is bogus. or woudl require a lot more work to make it work. | 16:17 |
| smoser | yeah. | 16:17 |
| smoser | i'm guessing its /etc/init/upstart-udev-bridge.conf | 16:18 |
| didrocks | well, the test doesn't want "upstart" itself to be installed, really, we want to try having systemd as default and having upstart that we can optionnally boot | 16:18 |
| smoser | that was the "no network" issue | 16:18 |
| didrocks | so yeah, moving the jobs would fix it | 16:18 |
| infinity | Moving the jobs is "hard". | 16:18 |
| smoser | right. | 16:18 |
| infinity | We discussed redoing the split instead to match systemd. | 16:18 |
| infinity | So, "upstart" would be jobs and binaries, and "upstart-sysv" would be the overlapping bits that make it boot by default. | 16:19 |
| infinity | I might do that this afternoon. | 16:19 |
| didrocks | infinity: I guess you are telling this because they all are conffiles, right? | 16:19 |
| infinity | didrocks: Right. | 16:19 |
| smoser | so... can we let that -proposed through ? | 16:19 |
| didrocks | yeah, the upstart-sysv sounds better as matching what systemd has | 16:19 |
| infinity | smoser: Yeah, now that it's been root-caused, we can let it through. | 16:19 |
| didrocks | smoser: can you run all the other tests locally? | 16:20 |
| didrocks | as this one was in the middle of the stack | 16:20 |
| didrocks | I can deal with it if you prefer (but probably tomorrow for me) | 16:20 |
| infinity | ubuntu-drivers-common being completely broken is fun too. | 16:20 |
| smoser | ok. i'll run tests | 16:22 |
| didrocks | thanks :) | 16:22 |
| smoser | i can reasonably expect to run them on trusty ? | 16:24 |
| smoser | didrocks, ^ as host. | 16:24 |
| didrocks | smoser: oh sure, as long as you have a vivid adt cloud image | 16:24 |
| smoser | right | 16:24 |
| didrocks | and just comment the upstart test in debian/test/control | 16:24 |
| didrocks | upstart* | 16:24 |
| smoser | bah. didrocks doesnt seem to want to run on trusty | 16:39 |
| smoser | http://paste.ubuntu.com/10621864/ and eventual hang | 16:40 |
| smoser | http://paste.ubuntu.com/10621878/ | 16:41 |
| didrocks | smoser: argh, ok… do you mind if I run them tomorrow morning? I'm finishing up another systemd issue and will wrap up then | 16:43 |
| smoser | didrocks, yeah. i have a fix i want to get in to systemd also . | 17:01 |
| smoser | for http://pad.lv/1432829 | 17:01 |
| ubot93 | Launchpad bug 1432829 in resolvconf (Ubuntu) "resolvconf not updated correctly for interfaces configured in initramfs" [Undecided,New] | 17:01 |
| didrocks | smoser: ok, I only gives debdiff against experimental (we are using a git here, having an ubuntu branch), but my patches are not as urgent as yours I guess (they just need to be in for vivid) | 17:03 |
| bdmurray | cjwatson: How is regression detection done in http://people.canonical.com/~ubuntu-archive/proposed-migration/trusty/update_excuses.yaml? | 18:04 |
| bdmurray | cjwatson: I ask as apport version 2.14.1-0ubuntu3.8 is failing but so did 2.14.1-0ubuntu3.7 | 18:05 |
| cjwatson | bdmurray: It's up to autopkgtest.py, but I think it's "has this test ever passed at all in this series" | 18:06 |
| === Streamstormer_ is now known as Streamstormer | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!