=== lifeless_ is now known as lifeless [10:35] lxd snap builds appear to be painfully slow [10:35] https://code.launchpad.net/~xnox/+snap/xnox-subiquity/+build/75817 [10:35] it's been 9 minutes doing apt update =( [10:36] and previous builds at https://code.launchpad.net/~xnox/+snap/xnox-subiquity would take just 1 minute to get a lot further (and fail, but whatever) [10:37] It may have hung [10:37] it is progressing, but slowly [10:37] Bluff [10:37] That Ign sounds a bit suspicious [10:37] i made coffee refreshed and it had an extra 'Ign:5 http://ppa.launchpad.net/snappy-dev/tools/ubuntu xenial Release' [10:37] before refresh it was at Ign:3 [10:37] There's no way the chroot should have the snappy-dev PPA up to date already. [10:37] Yeah, it's almost certainly timed out [10:38] We'll hopefully get more information once it finishes timing out [10:38] https://launchpadlibrarian.net/335415335/buildlog_snap_ubuntu_xenial_amd64_xnox-subiquity_BUILDING.txt.gz -> took 1 minute to do everything, and it has a lot of Get rather than Ign. [10:38] Yeah yeah I know [10:39] It's not "slow", it's broken :) [10:39] ok, thanks. [10:39] But I won't have much idea of why until it gets further [10:39] (worked fine on dogfood) [10:40] Huh, there it managed to get something from ppa.launchpad.net [10:43] let's see what dogfood lcy01 makes of it [11:01] trying https://code.dogfood.paddev.net/~cjwatson/+snap/subiquity/+build/65491 [11:04] Looks like it's doing the same thing [11:05] wgrant: Do you have your "get shell on dogfood builder" package handy somewhere? [11:06] cjwatson: labbu:~wgrant/buildd-rooter.tar.xz [11:06] thanks [11:06] You need to adjust the key in wgrant-rooter, build it, publish it, then install wgrant-rootit [11:07] cjwatson: Should we roll prod back for now? [11:07] s/install wgrant-rootit/build wgrant-rootit and wait for it to hang/ [11:08] wgrant: Let's. Could you organise that? If you could leave buildd-staging alone that'd be good. [11:08] cjwatson: kk [11:09] I wonder if breaking out of the buildd chroot will work when it's using schroot, though. [11:10] I don't see why not. [11:10] Unless schroot has started using mount namespaces. [11:10] Pretty sure I put in way too many ..s for that reason. [11:13] Seems to leave me in /home [11:14] At least when I try locally in schroot -c unstable-amd64 -u root [11:17] I guess overlayfs might matter there [11:17] Yeah, I see the same here, but that is aufs so is going to be pretty weird. [11:17] Slightly surprised it wouldn't work, but I don't see how schroot could possibly break it with the dir backend and no mount namespaces. [11:18] Well, let's try I guess. [11:19] cjwatson: Wait no I typoed, it works fine [11:19] >>> import os [11:19] >>> os.getcwd() [11:19] '/root' [11:19] >>> os.chdir('..') [11:19] >>> os.chroot('/root') [11:19] >>> os.getcwd() [11:19] '(unreachable)/run/schroot/mount/artful-amd64-a0c7ae2d-d598-4f05-8b8f-2e60df09df2a' [11:19] >>> os.chdir('../../../../../../../../') [11:19] >>> os.getcwd() [11:20] '(unreachable)/' [11:45] cjwatson: All active vbuilders are 147, though lgw01 is being sluggish at coming back, as always [12:48] wgrant: https://code.launchpad.net/~cjwatson/launchpad-buildd/lxd-clamp-mss/+merge/330078 [12:48] I don't feel too bad about not spotting that in advance ... [12:49] Indeed [12:50] cjwatson: lgtm [12:50] thanks [12:50] will get it onto dogfood today but I'm not going to attempt another prod rollout until next week [12:51] cjwatson: The non-vbuilders are probably 1500 anyway, so I suppose we need not downgrade them. [12:52] z13-* are 1500 at least [20:58] wgrant: Would you be OK with reversioning our currently-forked dependencies using PEP 440-compliant versions (e.g. Twisted 13.0.0-p2 -> 13.0.0.post3 or similar)? I had another poke at my pip branch today and discovered that this would be helpful for that project; notably, unlike old pkg_resources, modern pkg_resources (and hence pip) considers zope.pagetemplate 3.5.0-p1 < 3.5.0, so much ... [20:58] ... confusion arises. [20:59] wgrant: I think we just need to make sure that whatever versions we use are also handled correctly by the old pkg_resources we're currently using, but that's possible. [21:00] (I made some progress; I have a bin/py now, though it's currently a pkg_resources entry point which means that running it causes pkg_resources to try to resolve stuff and hence I run into the above. [21:00] )