[00:23] -queuebot:#ubuntu-release- Unapproved: python-pip (focal-proposed/universe) [20.0.2-5ubuntu1.3 => 20.0.2-5ubuntu1.4] (no packageset) [07:14] -queuebot:#ubuntu-release- Unapproved: ubuntu-settings (focal-proposed/main) [20.04.5 => 20.04.6] (ubuntu-desktop) [07:24] -queuebot:#ubuntu-release- Unapproved: rejected evolution-data-server [source] (focal-proposed) [3.36.5-0ubuntu1] [07:25] -queuebot:#ubuntu-release- Unapproved: evolution-data-server (focal-proposed/main) [3.36.4-0ubuntu1 => 3.36.5-0ubuntu1] (i386-whitelist, ubuntu-desktop) [08:40] Laney: it seems a lot of xenial arm64 tests fail with grub-install: error: relocation 0x113 is not implemented yet; wondering if we should work around that somehow, e.g. by uninstalling grub [08:41] Laney: hang on, they worked a day earlier [08:42] is that a fail fail or a temp fail? [08:43] those seem to fail fail as /unknown [08:45] hmm [08:47] bdmurray, hey, could you check what's going on there? lp 1923267 the gjs update was accepted as a SRU in hirsute-proposed but seems it vanished from there? [08:47] Launchpad bug 1923267 in gjs (Ubuntu Hirsute) "Gjs promises on Gio.File operations don't work anymore after upgrading libglib2.0-0 2.67.5-2 to 2.68.0-1 (and indicator-multiload app broke)" [High, Fix Committed] https://launchpad.net/bugs/1923267 [08:47] it's in impish only now?! [08:48] hum [08:48] Deleted on 2021-04-23 by Iain Lane [08:48] moved to impish-proposed [08:48] Laney: I see in journal also that "Flavor m1.large could not be found." on bos01 for arm64, causing a bunch of errors [08:48] Laney, ^ could you check what happened to that SRU? [08:49] seb128: Copy it back. We don't have a good way to tell which pre release uploads should be kept, so it's guess work and sometimes we guess wrong [08:49] juliank: umm yeah, can you check that? if true, get IS to create it [08:49] on it [08:49] Laney, so pocket copy from impish? [08:49] looks like bos* just went down though [08:49] seb128: yeah [08:49] openstack flavor list is either slow or timing out [08:50] yeah, wait for the cloud to come back [08:50] :/ [08:50] Laney, you don't have a command line in your backlog handy to share so I don't have to figure out the parameters to use again? ;) [08:50] seb128: I shared it to Rik_Mills the other day so it's probably in here, one second [08:50] sweet waiting time [08:51] #ubuntu-release.log-20210427.gz:26/04 10:24:01 copy-package --from=ubuntu --to=ubuntu --include-binaries --from-suite=impish-proposed --to-suite=hirsute-proposed gwenview [08:51] Laney, thanks! [08:51] juliank: follow along in the outage channel! [08:51] I wonder also if there is some sort of query I could do to get a list of potential other SRUs we lost [08:52] you could probably find the last proposed-migration run before I did that in https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses/ [08:52] Laney, thanks [08:53] Laney, last question on that, any idea of tweaks we could do the process or archive opening to avoid it happens again in futur cycles? [08:54] I don't know, we have a trello board 'devel unapproved', maybe that could be used somehow, but not super sure atm [08:54] In that particular case, was there a bug reference with a verification-{needed,done}-hirsute tag? Would filtering on that have helped, if we had appropriate scripting to do it? [08:54] https://paste.ubuntu.com/p/kpZGJVVjFp/ [08:54] that's the list by the way [08:54] rbasak, yes we have a bug with proper tags [08:55] I think maybe the difficulty could be reduced to things still in unapproved then maybe? [08:55] bug #1923267 [08:55] Bug 1923267 in gjs (Ubuntu Hirsute) "Gjs promises on Gio.File operations don't work anymore after upgrading libglib2.0-0 2.67.5-2 to 2.68.0-1 (and indicator-multiload app broke)" [High, Fix Committed] https://launchpad.net/bugs/1923267 [08:55] I'm sure scripting could help, but I was starting from dists/ [08:55] -queuebot:#ubuntu-release- Unapproved: gjs (hirsute-proposed/main) [1.67.2-2ubuntu1 => 1.67.2-2ubuntu2] (desktop-core, desktop-extra, i386-whitelist, mozilla) (sync) [08:55] so LP bugs aren't really super easily available there [08:56] What do you mean by "starting from dists/"? [08:57] That's what I used to list -proposed (grep-dctrl). That would need to be input to another script or something which can figure out whatever it is there is to figure out. [08:57] Or instead the script could list proposed itself. Just saying, I wasn't using the LP API at all to do this. [08:57] I see - thanks. [08:59] juliank: it was funny, I was literally in the middle of typing shell commands on the lxd-armhf machines when they went down [08:59] don't think I've caught an outage that live before :D [09:00] Laney: heh [09:02] is there a way to send ~. to an inner ssh btw? [09:02] ~~.? [09:02] IOW, ~~ is the escape for ~ [09:02] * Laney tries [09:02] yes! thanks [09:06] juliank: back [09:06] * juliank lists flavors [09:07] keystone still seems down [09:07] nah [09:07] it's probably that bug where if you source one file first and then another you can't use the cloud [09:07] quit and go back in [09:09] Laney: I only ever sourced that one file :D [09:10] Laney: now it works [09:10] m1.large is there [09:10] maybe it was the cloud going down that caused the error [09:10] anything is possible in the wonderful world of openstack [09:10] got to say it would have been surprising if the flavor didn't exist [09:14] Laney: I wonder if I should try reinstating the port cleanup in cleanup-instances [09:14] We only kept copy-security-group, and it should be enough - at least if the worker restarts [09:15] maybe the worker should purge all ports in its security group before starting a job too [09:18] I think workers are down, autopkgtest@bos01-arm64-6.service: Job autopkgtest@bos01-arm64-6.service/start failed with result 'dependency'. [09:18] yeah I'm running the maintenance thingy [09:18] I dunno if the port thing we're seeing atm is stray ports, did you confirm that? [09:18] I think it's something quotaish on our new users [09:19] Laney: I haven't checked, need to write a script and run it I suppose [09:40] there's still something wrong in Boston [09:40] I'm trying to get IS to help out [10:00] (without much success so far, it must be said) [10:02] I'm digging into bootloader stuff :/ [10:19] the bootloader stuff https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1926748 [10:19] Launchpad bug 1926748 in grub2 (Ubuntu Xenial) "regression in xenial updates - grub2 cannot handle new arm64 relocations" [Undecided, New] [10:24] huh [11:01] hoping that one patch will do it; cause the other relocations need 13 patches =( [11:06] regression-updates can we please downgrade grub2 grub2-unsigned grub-signed from xenial-updates to xenial-proposed please? as otherwise upgrades are broken on aws ec2 arm64. [11:06] !regression-alert [11:06] !regression [11:06] * xnox is not sure what the bot command is [11:07] Laney: ^^^^ [11:09] Ok, on it [11:10] sil2100: tah. [11:10] (I don't think we have highlights on this btw. ;) ) [11:10] * sil2100 feels like he is reverting stuff all the time this week [11:10] sil2100: and previous grub2 & grub2-signed need to put back into xenial-updates, as you obviously know. [11:13] -queuebot:#ubuntu-release- Unapproved: ceph (groovy-proposed/main) [15.2.11-0ubuntu0.20.10.1 => 15.2.11-0ubuntu0.20.10.2] (ubuntu-desktop, ubuntu-server) [11:15] xnox: That first one used to be a factoid, but someone deleted it. [11:15] Ukikie: cool. thanks. [11:16] hi ubuntu-sru - I've just uploaded new ceph versions to focal and groovy proposed - one of the cherry picks included in the last uploads for SRU did not actually fix a UX issue [11:16] -queuebot:#ubuntu-release- Unapproved: ceph (focal-proposed/main) [15.2.11-0ubuntu0.20.04.1 => 15.2.11-0ubuntu0.20.04.2] (desktop-core, ubuntu-server) [11:16] sil2100: will go get lunch/coffee and then will test proposed fixed, hopefully we might be able to have a fixed up grub2 sru for xenial-proposed for arm64 at least. [11:16] so I've elected to drop that patch from the other SRU's in flight so we can move forwards [11:18] ACK o/ [11:22] -queuebot:#ubuntu-release- Unapproved: grub2 (xenial-updates/main) [2.02~beta2-36ubuntu3.31 => 2.02~beta2-36ubuntu3.29] (core) (sync) [11:23] -queuebot:#ubuntu-release- Unapproved: accepted grub2 [sync] (xenial-updates) [2.02~beta2-36ubuntu3.29] [11:23] xnox: ok, old versions deleted from -updates, still present in -proposed, old versions copied over to -updates (hopefully the correct ones) [11:24] I hope it's all good regarding signing bits [11:24] I'll go grab a bite now as well and then take a look if all is good [11:26] also confused how come we have released xenial ahead of bionic. [11:26] and will double check that bionic is not affected otherwise. [11:49] xnox: the reason was that xenial goes ESM today [11:50] I mean, Steve probably should have released both bionic and xenial at this point, but now I'm happy he didn't [11:50] Since we have the liberty of just checking this without rush [12:22] i feel like i don't even know how computers work => none of the commands on xenial work for me, and i have to look up old syntax for all the things. Even like debuild. [12:34] -queuebot:#ubuntu-release- Unapproved: accepted google-cloud-sdk [sync] (xenial-release) [335.0.0-0ubuntu1~16.04.1] [12:38] -queuebot:#ubuntu-release- Unapproved: grub2 (xenial-proposed/main) [2.02~beta2-36ubuntu3.31 => 2.02~beta2-36ubuntu3.32] (core) [12:52] sil2100: juliank: please review http://launchpadlibrarian.net/536308504/grub2_2.02~beta2-36ubuntu3.31_2.02~beta2-36ubuntu3.32.diff.gz on amazon ec2 arm64 xenial instance that fixes things for me. [13:48] xnox: looking [13:59] xnox: ok, I don't know much about these parts of grub2 but upload seems sane - from the SRU POV I guess I'd like if it was built with -v to include the previous version and hm, possible since this is to go to -security, maybe actually building it in a security pocket first? Or something [14:00] sil2100: -v & security is a must [14:00] sil2100: please reject. [14:01] sil2100: re what this does "it reimplements binutils in grub basically, because it is like lets take these elf objects which are grub .mod modules and lets assemble a core.efi pe/coff binary out of it" it is black magic in binutils, and more so, in grub2. So this is an upstream cherrypick. [14:08] sil2100: there is also shim signing to review in impish unapproved https://launchpad.net/ubuntu/impish/+queue?queue_state=1&queue_text=shim [14:09] sil2100: the shim-signed with MS signature is already in impish-proposed (ftbfs, as it is waiting on shim signing to be approved) [14:15] sil2100: rebuilding grub in bileto [14:18] xnox: do you have a ppa? I spent some time building myself to realize that I don't have a local reproducer for the linking issue [14:18] xnox: but I can rerun failed autopkgtest in the cloud [14:18] when did the autopkgtests catch it? like when it needed dist-upgrading in the base system? [14:18] [NEEDSBUILD] Needs building Cancel build [14:18] not in an actual triggered test, right? [14:18] Start in 2 hours [14:19] Laney: all the tests triggered by python-apt yesterday-today/last night [14:19] Laney: They all upgrade first, get grub 2.04 which was released yesterday, and then fail [14:19] yeah [14:20] so I guess the question is how can we catch this in a real test triggered by the actual upload [14:20] so it doesn't get released [14:21] juliank: it's building [14:22] Laney: grub must get an autopkgtest, consisting just of Depends [14:22] juliank: i built in amazon ec2 xenial arm64 instance; and tested there. [14:22] Laney: and true as test case [14:22] normal apt upgrade failed to configure grub; but manual call to ./grub-install from the built package worked [14:22] Laney: Ok, optimally you do want to run grub-install :D [14:23] But in the cloud as we have it now, it will fail either way [14:23] but i guess it will not build on arm64 https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4545/+build/21482739 because all builders are borked with that grub2? [14:23] juliank: do you want me to build the package in amazon cloud and give you the debs? [14:24] xnox: do the builders build on bare (virtual) metal, and not in a chroot? [14:24] juliank: well we reverted grub from xenial-updates so all the vms should be fine now? [14:24] If they build in a chroot, they should be fine [14:24] juliank: i think it's chroot, on a VM [14:24] that should be fine either way [14:24] but like some arm64 builders look dead https://launchpad.net/builders [14:26] bos02 is looking really bad from where I'm sitting, independent of the grub2 stuff [14:28] cool. [14:29] (and bos01, but that's not relevant for launchpad builds) [14:29] fun outages [14:32] like https://ubuntu-release.kpi.ubuntu.com/d/76Oe_0-Gz/autopkgtest?orgId=1 check the middle two, doesn't look great] [14:36] separately there's a lot of instance reboot failures in this latest set of linux tests [14:36] in impish [14:37] xnox: looking! [14:37] -queuebot:#ubuntu-release- Unapproved: rejected grub2 [source] (xenial-proposed) [2.02~beta2-36ubuntu3.32] [14:42] everything is baddddddd [14:47] Everything is terrible [14:48] let's all moan together [14:51] sigh madness [14:56] * xnox has "Everything is awesome" in my head now, as sung by cjwatson [14:57] checking if something in impish-proposed is bad [14:57] there's lots of suspicious things in there :D [14:58] today I am thankful for lxd VMs [15:08] https://paste.ubuntu.com/p/5VZVg7F8z4/ [15:13] sil2100: so it looks like my grub2 xenial sru will not build today, so we shall come back to it like on tuesday. (bank holiday here in the uk) [15:14] sil2100: vorlon: or if https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4545/+packages finishes building feel free to binary copy that into xenial-proposed. [15:14] * xnox checks if i uploaded it with the right -v, i think i did not. [15:14] https://launchpadlibrarian.net/536318662/grub2_2.02~beta2-36ubuntu3.32_source.changes => i did not [15:14] * xnox facepalm [15:16] xnox: why are you building this in bileto instead of just uploading to -proposed? [15:17] binary copies from bileto are annoying [15:18] xnox: and you say "if it finishes building" but it's ftbfs already on arm64 and ppc64el? [15:19] vorlon: I think your initramfs-tools is bad, it seems to break booting somehow https://paste.ubuntu.com/p/5VZVg7F8z4/ [15:19] Laney: caught by autopkgtests? [15:21] vorlon: sort of, yes [15:21] k [15:21] I'll look into it, thanks [15:21] I noticed that all-proposed runs are failing to reboot [15:21] ah [15:21] and then bisected until I found that [15:21] so I think we should drop it in the meantime [15:21] ack [15:21] you want to do that, or me? [15:22] I will [15:22] cheers [15:22] also, separately, I've seen that the boston clouds are kinda unhappy, lots of keystone errors and timeouts in lxd [15:23] I've not managed to get IS to bite properly yet (think it's affecting buildds too) [15:23] so if you want to track that it would be helpful; I'll be EODing in 90 minuts [15:25] vorlon: ftbfs without any build-logs..... cause bos02 is sad [15:29] Laney: ugh it's the damn +x bit being dropped from init again by MoM; I remembered this issue and tried to check for missing +x bits but apparently I mistakenly looked only in the debian/ subdir [15:30] urgh :( [15:30] reuploaded [15:30] cheers for the quick fix [15:30] xnox: and now https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4545/+packages has disappeared? [15:32] vorlon: because wrong -v, so uploaded to 4546 [15:32] instead of just uploading to the archive [15:32] https://launchpadlibrarian.net/536325290/grub2_2.02~beta2-36ubuntu3.32_source.changes => looks better [15:32] vorlon: must be built against -security pocket [15:33] hmm ok [15:33] build scores adjusted [15:36] hmm why are the -proposed links all broken on https://people.canonical.com/~ubuntu-archive/pending-sru.html [15:57] seb128 merged some changes that I think could have caused this [16:50] sil2100: yeah I don't see anything in seb128's diff that obviously explains; I guess I'll dig [17:02] I'm out - hopefully the clouds get fixed so we can make queue progress over the weekend ... [17:43] vorlon, I can check if it's my changes [17:44] vorlon, ah, I see a typo in there, let me try with that fixed [17:45] sru_item['url'] = rpkg['proposed'] = pkgurl + rpkg['proposed'] [17:50] vorlon, sil2100, sorry for the inconvenience, I pushed a fix [18:19] -queuebot:#ubuntu-release- Unapproved: software-properties (focal-proposed/main) [0.98.9.4 => 0.98.9.5] (core) [18:41] -queuebot:#ubuntu-release- New binary: resource-agents [amd64] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [18:41] -queuebot:#ubuntu-release- New binary: resource-agents [s390x] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [18:43] -queuebot:#ubuntu-release- New binary: resource-agents [arm64] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [18:43] -queuebot:#ubuntu-release- New binary: resource-agents [ppc64el] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [18:45] -queuebot:#ubuntu-release- New binary: resource-agents [armhf] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [18:57] seb128: aha, cheers! [18:58] -queuebot:#ubuntu-release- Unapproved: python-pip (bionic-proposed/universe) [9.0.1-2.3~ubuntu1.18.04.4 => 9.0.1-2.3~ubuntu1.18.04.5] (no packageset) [19:18] -queuebot:#ubuntu-release- New binary: resource-agents [riscv64] (impish-proposed/main) [1:4.7.0-1ubuntu2] (ubuntu-server) [19:23] -queuebot:#ubuntu-release- Unapproved: grub2 (xenial-proposed/main) [2.02~beta2-36ubuntu3.31 => 2.02~beta2-36ubuntu3.32] (core) (sync) [19:33] I'd appreciate if any AA could process resource-agents in impish NEW queue :) [19:36] -queuebot:#ubuntu-release- Unapproved: accepted grub2 [sync] (xenial-proposed) [2.02~beta2-36ubuntu3.32] [19:36] xnox: ^^ accepted [20:11] xnox: how did shim migrate to impish release without corresponding shim-signed? [20:13] hmm I guess we no longer have a dep on unsigned shim package, right [20:20] xnox: why did https://launchpad.net/ubuntu/+source/shim-signed/1.47/+build/21482148 download from impish-proposed but https://launchpad.net/ubuntu/+source/shim-signed/1.47/+build/21482149 download from impish?! [21:49] vorlon: because it checks apt where the candidate for shim is at, and downloads from there. now that shim is not in -proposed anymore it downloads from release [21:50] vorlon: but what is more confusing is that https://launchpad.net/ubuntu/+source/shim-signed/1.47/+build/21482149 downloads current and gets ubuntu1; wheres when i download the same url from ftpmaster.internal from people.canonical.com i get ubuntu2 signed tarball. [21:50] vorlon: it's as if arm64 bos02 builders see some different ftpmaster.internal?! [21:50] i don't know if I should ping #launchpad people but not over labor weekened :/ [21:50] let me try #is [21:52] yes, I don't know what's going on there >_< [22:01] -queuebot:#ubuntu-release- Unapproved: nvidia-graphics-drivers-340 (focal-proposed/restricted) [340.108-0ubuntu5.20.04.1 => 340.108-0ubuntu5.20.04.2] (kernel-dkms, ubuntu-desktop) (sync) [22:01] -queuebot:#ubuntu-release- Unapproved: oss4 (focal-proposed/universe) [4.2-build2010-5ubuntu6~20.04.1 => 4.2-build2010-5ubuntu6~20.04.2] (kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: r8168 (focal-proposed/multiverse) [8.048.00-1ubuntu0.20.04.1 => 8.048.00-1ubuntu0.20.04.2] (kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: sysdig (focal-proposed/universe) [0.26.4-1ubuntu0.2 => 0.26.4-1ubuntu0.3] (kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: virtualbox-hwe (focal-proposed/multiverse) [6.1.16-dfsg-6ubuntu1.20.04.1 => 6.1.16-dfsg-6ubuntu1.20.04.2] (kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: zfs-linux (focal-proposed/main) [0.8.3-1ubuntu12.8 => 0.8.3-1ubuntu12.9] (core, kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: rtl8812au (focal-proposed/universe) [4.3.8.12175.20140902+dfsg-0ubuntu13~20.04.1 => 4.3.8.12175.20140902+dfsg-0ubuntu13~20.04.2] (kernel-dkms) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: virtualbox (focal-proposed/multiverse) [6.1.16-dfsg-6~ubuntu1.20.04.1 => 6.1.16-dfsg-6~ubuntu1.20.04.2] (kernel-dkms, ubuntu-cloud) (sync) [22:02] -queuebot:#ubuntu-release- Unapproved: v4l2loopback (focal-proposed/universe) [0.12.3-1ubuntu0.3 => 0.12.3-1ubuntu0.4] (kernel-dkms) (sync) [22:13] -queuebot:#ubuntu-release- Unapproved: grub2 (xenial-proposed/main) [2.02~beta2-36ubuntu3.32 => 2.02~beta2-36ubuntu3.32] (core) (sync) [22:13] all of the above is from me =) [22:22] kanashiro_: so your resource-agents package split mentions resource-agents-common as a base for both supported and unsupported agents, but I only see an addition of resource-agents-supported, no corresponding -unsupported package? [22:22] thus I'm confused at the split [22:24] also resource-agents-supported breaks/replaces old resource-agents, but new resource-agents doesn't depend on resource-agents-supported, so this agent (there seems to be only one) disappears on upgrade? [22:34] vorlon: not Lucas, but I was involved in the review of this change. there's no corresponding -unsupported package; it is implicit that the resource-agents will contain everything that is not in the -supported package, thus being the unsupported version [22:35] weird but ok? [22:35] yeah :) [22:35] sergiodj: in that case, why does the unsupported package not depend on the supported one, for upgrades? [22:36] vorlon: I don't know offhand, sorry [22:37] I mean, they're two separate packages, and the user can install just one or the other [22:39] but maybe I misunderstood your question