[10:35] <xnox> lxd snap builds appear to be painfully slow
[10:35] <xnox> https://code.launchpad.net/~xnox/+snap/xnox-subiquity/+build/75817
[10:35] <xnox> it's been 9 minutes doing apt update =(
[10:36] <xnox> and previous builds at https://code.launchpad.net/~xnox/+snap/xnox-subiquity would take just 1 minute to get a lot further (and fail, but whatever)
[10:37] <cjwatson> It may have hung
[10:37] <xnox> it is progressing, but slowly
[10:37] <cjwatson> Bluff
[10:37] <wgrant> That Ign sounds a bit suspicious
[10:37] <xnox> i made coffee refreshed and it had an extra 'Ign:5 http://ppa.launchpad.net/snappy-dev/tools/ubuntu xenial Release'
[10:37] <xnox> before refresh it was at Ign:3
[10:37] <wgrant> There's no way the chroot should have the snappy-dev PPA up to date already.
[10:37] <cjwatson> Yeah, it's almost certainly timed out
[10:38] <cjwatson> We'll hopefully get more information once it finishes timing out
[10:38] <xnox> https://launchpadlibrarian.net/335415335/buildlog_snap_ubuntu_xenial_amd64_xnox-subiquity_BUILDING.txt.gz -> took 1 minute to do everything, and it has a lot of Get rather than Ign.
[10:38] <cjwatson> Yeah yeah I know
[10:39] <cjwatson> It's not "slow", it's broken :)
[10:39] <xnox> ok, thanks.
[10:39] <cjwatson> But I won't have much idea of why until it gets further
[10:39] <cjwatson> (worked fine on dogfood)
[10:40] <cjwatson> Huh, there it managed to get something from ppa.launchpad.net
[10:43] <cjwatson> let's see what dogfood lcy01 makes of it
[11:01] <cjwatson> trying https://code.dogfood.paddev.net/~cjwatson/+snap/subiquity/+build/65491
[11:04] <cjwatson> Looks like it's doing the same thing
[11:05] <cjwatson> wgrant: Do you have your "get shell on dogfood builder" package handy somewhere?
[11:06] <wgrant> cjwatson: labbu:~wgrant/buildd-rooter.tar.xz
[11:06] <cjwatson> thanks
[11:06] <wgrant> You need to adjust the key in wgrant-rooter, build it, publish it, then install wgrant-rootit
[11:07] <wgrant> cjwatson: Should we roll prod back for now?
[11:07] <wgrant> s/install wgrant-rootit/build wgrant-rootit and wait for it to hang/
[11:08] <cjwatson> wgrant: Let's.  Could you organise that?  If you could leave buildd-staging alone that'd be good.
[11:08] <wgrant> cjwatson: kk
[11:09] <cjwatson> I wonder if breaking out of the buildd chroot will work when it's using schroot, though.
[11:10] <wgrant> I don't see why not.
[11:10] <wgrant> Unless schroot has started using mount namespaces.
[11:10] <wgrant> Pretty sure I put in way too many ..s for that reason.
[11:13] <cjwatson> Seems to leave me in /home
[11:14] <cjwatson> At least when I try locally in schroot -c unstable-amd64 -u root
[11:17] <cjwatson> I guess overlayfs might matter there
[11:17] <wgrant> Yeah, I see the same here, but that is aufs so is going to be pretty weird.
[11:17] <wgrant> Slightly surprised it wouldn't work, but I don't see how schroot could possibly break it with the dir backend and no mount namespaces.
[11:18] <cjwatson> Well, let's try I guess.
[11:19] <wgrant> cjwatson: Wait no I typoed, it works fine
[11:19] <wgrant> >>> import os
[11:19] <wgrant> >>> os.getcwd()
[11:19] <wgrant> '/root'
[11:19] <wgrant> >>> os.chdir('..')
[11:19] <wgrant> >>> os.chroot('/root')
[11:19] <wgrant> >>> os.getcwd()
[11:19] <wgrant> '(unreachable)/run/schroot/mount/artful-amd64-a0c7ae2d-d598-4f05-8b8f-2e60df09df2a'
[11:19] <wgrant> >>> os.chdir('../../../../../../../../')
[11:19] <wgrant> >>> os.getcwd()
[11:20] <wgrant> '(unreachable)/'
[11:45] <wgrant> cjwatson: All active vbuilders are 147, though lgw01 is being sluggish at coming back, as always
[12:48] <cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad-buildd/lxd-clamp-mss/+merge/330078
[12:48] <cjwatson> I don't feel too bad about not spotting that in advance ...
[12:49] <wgrant> Indeed
[12:50] <wgrant> cjwatson: lgtm
[12:50] <cjwatson> thanks
[12:50] <cjwatson> will get it onto dogfood today but I'm not going to attempt another prod rollout until next week
[12:51] <wgrant> cjwatson: The non-vbuilders are probably 1500 anyway, so I suppose we need not downgrade them.
[12:52] <cjwatson> z13-* are 1500 at least
[20:58] <cjwatson> wgrant: Would you be OK with reversioning our currently-forked dependencies using PEP 440-compliant versions (e.g. Twisted 13.0.0-p2 -> 13.0.0.post3 or similar)?  I had another poke at my pip branch today and discovered that this would be helpful for that project; notably, unlike old pkg_resources, modern pkg_resources (and hence pip) considers zope.pagetemplate 3.5.0-p1 < 3.5.0, so much ...
[20:58] <cjwatson> ... confusion arises.
[20:59] <cjwatson> wgrant: I think we just need to make sure that whatever versions we use are also handled correctly by the old pkg_resources we're currently using, but that's possible.
[21:00] <cjwatson> (I made some progress; I have a bin/py now, though it's currently a pkg_resources entry point which means that running it causes pkg_resources to try to resolve stuff and hence I run into the above.
[21:00] <cjwatson> )