[00:02] <vorlon> bumping various build scores; skipping over k* because it sounds like wgrant is already handling (hopefully on both s390x and riscv64?)
[00:05] <vorlon> also skipping plasma-*
[00:06] <wgrant> vorlon: Yep, I'm handling everything k* or building a libkf5*
[00:06] <vorlon> wgrant: but not plasma?
[00:06] <wgrant> Some of those have been caught up in it, but will check.
[00:09] <cpete> kanashiro, vorlon, dbungert: can someone requeue kcmutils build on riscv64? The build dep should be available now
[00:09] <vorlon> doing
[00:09] <cpete> thanks
[00:19] <cpete> So I've been scanning through the no_hide_arches list looking for build failures or dependency waits. I've stopped at knotifyconfig and there's nothing actionable above that currently. afk for dinner and when I come back I'll see if there's something automated I can setup to consume Dan's lists
[00:24] <vorlon> openimageio bootstrapped, will copy back to the archive once the armhf binaries are available
[00:27] <vpa1977> anyone fixing faketime?
[00:28] <vpa1977> (next actionable on list is sssd and its failing tests partially due to faketime)
[00:31] <vpa1977> openjdk-8 did not build in ppa, will try to solve it
[01:00] <vorlon> vpa1977: lolno
[01:00] <vorlon> vpa1977: we should XFAIL the tests
[01:01] <kanashiro> vpa1977 I think ahasenack was working on sssd,  we can check the status with him tomorrow
[01:01] <vorlon> dbungert: so a list of packages that are blocked by tests and marked permanent failure would be helpful, those are packages not blocked by queued tests but by failed tests which need analyzed
[01:01] <vpa1977> vorlon: that's a build time failure, we can run the build with nocheck on armhf
[01:02] <vorlon> vpa1977: I don't think it's a good idea to nocheck the whole testsuite as opposed to XFAILing the specific tests that are failing due to faketime - and I think the other test failures are not yet fully analyzed
[01:26] <dbungert> vorlon: makes sense, I'll do that next
[02:30] <kanashiro> I am checking libgpod and it is blocked in -proposed because libgpod-dev is depending on libimobiledevice-dev which is in universe in -proposed and -updates, but in main in the release pocket (and all previous ubuntu series). Not sure what I can do to fix this
[03:00] <vorlon> mwhudson, kanashiro: so the buildd tarball also seems to have noble-updates enabled by default, but that appears to get overwritten for a livefs build, which explains the much longer list of uninstallables in our build log
[03:07] <mwhudson> vorlon: right yes, all builds of all kinds get some sources.list mangling i think?
[03:07] <vorlon> yeah I expect so, though it's not part of the build log afaics
[03:07] <vorlon> I'm not sure why noble-updates gets disabled here when it doesn't for stable releases, must be some magicking
[03:17] <tsimonq2> > the buildd tarball also seems to have noble-updates enabled by default waaaaaat?
[03:23] <wgrant> vorlon: You can see it in the override-sources-list invocation around line 13 of each build log.
[03:29] <tsimonq2> For reference, this is Jammy's, which is probably what it should be for Noble:
[03:29] <tsimonq2> RUN: /usr/share/launchpad-buildd/bin/in-target override-sources-list --backend=lxd --series=jammy --arch=amd64 LIVEFSBUILD-600220 'deb http://ftpmaster.internal/ubuntu jammy main universe' 'deb http://ftpmaster.internal/ubuntu jammy-security main universe' 'deb http://ftpmaster.internal/ubuntu jammy-updates main universe' 'deb http://ftpmaster.internal/ubuntu jammy-proposed main universe'
[03:35] <dbungert> vorlon: something like this?  https://paste.ubuntu.com/p/Qw3Btdds5J/
[03:42] <tsimonq2> vorlon: My best guess for this magic are lines 192-193 of lib/cdimage/livefs.py in ubuntu-cdimage, those make their way to the actual livefs.requestBuild() API call.
[03:42] <tsimonq2> I can't find any such magic elsewhere, such as debian-cd or livecd-rootfs itself.
[03:43] <tsimonq2> My *theory* is that commenting out those two lines as a cowboy would DTRT.
[04:00] <mwhudson> tsimonq2: have you looked at launchpad-buildd?
[04:00] <tsimonq2> mwhudson: I have, and I didn't find anything useful in there.
[04:00] <mwhudson> ok
[04:01] <tsimonq2> :)
[05:08] -queuebot:#ubuntu-release- New binary: gnome-shell-pomodoro [s390x] (noble-proposed/universe) [0.25.1-0ubuntu1] (no packageset)
[06:17] -queuebot:#ubuntu-release- New: accepted gnome-shell-pomodoro [s390x] (noble-proposed) [0.25.1-0ubuntu1]
[08:05] <wgrant> s390x caught up, doing a mass give-back
[08:06] <vorlon> packages whose migration is currently on the critical path for livecd-rootfs: cdrkit openldap glib2.0 systemd parted qemu libpng1.6 snapd
[08:06] <vorlon> there may be some others hidden under python deps which we'll be able to see more clearly after the next publication run
[08:11] -queuebot:#ubuntu-release- New binary: gnome-shell-pomodoro [amd64] (noble-proposed/universe) [0.25.1-0ubuntu1] (no packageset)
[08:16] <vorlon> I've scheduled retries of all autopkgtest failures for the above packages but there's going to need to be some manual review and overriding of failures here
[08:17] <vorlon> I'm also going to schedule the blocking 'in progress' tests over to the regular queue ahead of the huge queue
[08:29] <wgrant> I suppose there's no way for mortals to jump the autopkgtest queue?
[08:30] <vorlon> no but we've put the mass of tests in the 'huge' queue which gets loadbalanced vs the regular queue
[08:31] <vorlon> so anything that needs retried and is directly blocking someone working on this gets priority queuing
[08:32] <wgrant> Ah cool
[08:47] <sil2100> o/
[08:58] <ricotz> hello SRU team, please take a look at https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/2058687
[08:58] -ubottu:#ubuntu-release- Launchpad bug 2058687 in libreoffice (Ubuntu Mantic) "[SRU] libreoffice 7.6.6 for mantic" [Medium, In Progress]
[08:59] <vorlon> one additions to that list now that python3 has migrated: python-apt
[09:02] <vorlon> uh maybe apt itself should be in that list
[09:52] <upils> vorlon, I am looking at parted
[10:01] <slyon> I'm tackling systemd vs ayatana-indicator-session
[10:02] <doko> vorlon: do you have a list of packages temporarily removed from -proposed?
[10:07] <doko> component-mismatches-proposed.txt has some strange binary-only promotions to main, mostly for old library binaries. but looking at recent builds, there are none. e.g. libprotobuf-lite32 for mysql-8.0
[10:11] -queuebot:#ubuntu-release- New: accepted gnome-shell-pomodoro [amd64] (noble-proposed) [0.25.1-0ubuntu1]
[10:22] <juliank> Note that due to the removal of the previous release pocket versions of amd64 packages, apt/unattended-upgrades will now go and install -proposed versions for you, as your installed version is no longer present in a higher-pinned repository
[10:22] <juliank> Which is normally by design but a bit annoying right now
[10:23] <juliank> Note that it only upgrades packages you have currently installed that were removed to proposed
[10:23] <juliank> New installs would pull pre-t64 versions from -updates
[10:24] <juliank> And Architecture: all packages are still in the release pocket, so e.g. systemd-dev will have a candidate of 255.4-1ubuntu5 whereas it will try to upgrade systemd to proposed 255.4-1ubuntu6 (or -updates 255.2-3ubuntu2 if you did not have the t64 version installed)
[10:29] <slyon> systemd/255.4-1ubuntu6 is skiptested, so I think we should ignore those regressions and focus on glib2.0 instead (which is blocking systemd migration)
[10:35] <upils> About parted regressions: All were retried or are tests are already requested by vorlon except 2
[10:36] <upils> - livecd-rootfs/24.04.55. Should I request a retest with parted/3.6-4build1 or use every currently problematic pkgs as triggers?
[10:37] <upils> (on amd64)
[10:37] <bdrung> I am looking into cdrkit
[10:37] <upils> - nova/3:29.0.0~rc1-0ubuntu2: a retest for amd64 is already queued but without parted/3.6-4build1. Should I request one with parted/3.6-4build1 as trigger? or wait for the other one to succeed?
[10:39] <slyon> upils: just parted/3.6-4build1 if you think the other pkgs could cause unrelated regressions. But all of them if you're confident that it will pass (e.g. from looking at the test history or running local tests).
[10:41] <slyon> upils: wrt nova: Yes, queue a test for nova vs parted/3.6-4build1
[10:42] <upils> thanks!
[10:42] <schopin> taking qemu
[10:43] <doko> what are the critereas, investigate each test failure, including i386?
[10:44] <schopin> sil2100: ^
[10:48] <bdrung> retriggered most tests for cdrkit.
[10:48] <bdrung> looking at snapd now.
[10:52] <schopin> Can we consider that a test is OK if all-proposed run is good, like last week?
[10:53] <bdrung> I triggered livecd-rootfs migration-reference/0 to let them fail. then snapd should migrate.
[10:57] <schopin> bdrung: did you do it on all archs?
[11:00] <bdrung> schopin, yes
[11:00] <schopin> thanks :)
[11:10] <pushkarnk> I'll starting working through openldap
[11:20] <bdrung> I'll start looking at glib2.0 from the bottom up
[11:22] <pushkarnk> openldap has a long list, I have started from the top (balsa)
[11:24] <ravikant_> triggered sbuild with migration-reference/0 for qemu migration
[11:24] <schopin> for qemu, cinder/i386 and isa-support/i386 are badpkg, I've retriggered cryptsetup/arm64 with the last succesful trigger list, and the rest (up to livecd-rootfs) are either in progress or already solved.
[11:25] <schopin> The rest of the list is with ravikant_ :)
[11:25] <schopin> I'll circle back to qemu after I get some grub.
[11:26] <juliank> Please restore amd64 builds of shim-signed 1.57+15.8-0ubuntu1 in noble release pocket and remove shim-signed from updates
[11:27] <juliank> In all hilariousness this was actually built in mantic
[11:27] <ravikant_> for qemu, nova is bad-pkg, open-iscsi is a pass, osk-sdl is flaky.
[11:30] <ginggs> qemu is not blocked by autopkgtests
[11:31] <ginggs> "Should wait for tests relating to qemu 1:8.2.1+ds-1ubuntu9, but forced by ubuntu-release "
[11:31] <ginggs> it is blocked by
[11:31] <ginggs> Depends: qemu libpng1.6 (not considered)
[11:31] <ginggs> Depends: qemu glib2.0 (not considered)
[11:40] <juliank> can we clean out the -updates packages that have newer versions in release again now?
[11:47] <juliank> Did we just break Depends in amd64 release pocket?
[11:48] <juliank> libkpim5kmanagesieve5 4:23.08.5-0ubuntu3 has Depends: libqt5core5t64 and migrated, but libqt5core5t64 is only in proposed at least here
[11:48] <juliank> (launchpad is too slow to say anything)
[11:49] <juliank> Is it again a problem of non-atomic migration
[11:58] <juliank> I'm debugging EDSP files with upgrade failures that people on reddit send me
[11:59] <juliank> s/upgrade failures/packages being removed/
[12:15] <schopin> ginggs: damned, I missed that line!
[12:16] <jbicha> ginggs: could you update the force-skiptest hint for the new gnome-remote-desktop version? (armhf was removed last week but is back in -updates with the older version)
[12:18] <ginggs> jbicha: looking
[12:20]  * schopin wonders why we have inkscape on i386?????
[12:22] <juliank> heh
[12:26] <jbicha> inkscape is a build dependency but it might be possible to rearrange packaging to not need it on i386
[12:28] <schopin> Probably just a matter of adding a :native somewhere.
[12:29] <ginggs> jbicha: i added a force hint for gnome-remote-desktop/46.0-2
[12:32] <rbasak> Would someone like to give me a task to work on?
[12:36] <slyon> rbasak: trying to get glib2.0 and/or libpng1.6 migrated, e.g. picking a random spot in their autopkgtest regressions and working down from there.
[12:36] <ogra_> rbasak, my dishes need doing ... want to come over ?
[12:36] <ogra_> 😛
[12:37] <rbasak> ogra_: would that be a Noble cause :-P
[12:37] <slyon> those two libs are currently on the critical path
[12:37] <rbasak> slyon: looking
[12:37] <ogra_> rbasak, yeah, that would be extremely noble of you 😉
[12:37] <slyon> thanks!
[12:38] <rbasak> OK I'll start looking from glib2.0's gphotofs entry and work my way down from there
[12:38] <slyon> I'm currently at glib2.0 vs libgepub
[12:44] <bdrung> and i am working my way up on glib2.0 and currently at webp-pixbuf-loader
[12:48] <schopin> I keep seeing installability issues for linux-libc-dev on i386, but I can't figure out why. Does anyone has a clue?
[12:49] <bdrung> schopin, IIRC i have seen that on amd64 in a chroot as well. can you point to an example?
[12:49] <schopin> https://autopkgtest.ubuntu.com/packages/z/zvbi/noble/i386
[12:50] <schopin> bdrung: ^
[12:59] <bdrung> schopin, sorry, i fail to explain that
[13:09] <ginggs> the last britney run (10:34) failed due to the brief launchpad outage
[13:09] <ravikant_> looking at libpng1.6, moving up from zvbi
[13:19] <ravikant_> for libpng, zvbi, wxwidgets3.2, vlc, visp, sox, poppler, openjdk-8, opencv, odin, are FAIL badpkg. openjdk-23 is flaky.
[13:22] <rbasak> A lot of these (under glib2.0) seem to already have autopkgtests queued?
[13:23] <upils> For nova/3:29.0.0~rc1-0ubuntu2 on amd64, blocking livecd-rootfs I have unmet dependencies (see https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/n/nova/20240404_114824_7e32f@/log.gz) on libvirt-daemon-system I do not understand why. Binaries for libvirt-daemon-* version 10.0.0-1ubuntu1 are still published in main (updates)
[13:24] <slyon> ravikant_: are you triggering corresponding tests to resolve the badpkg situation?
[13:24] <slyon> rbasak: yes. I'd suggest skipping over those that have (releveant) tests queued already
[13:24] <rbasak> OK
[13:24] <ravikant_> slyon: I did not. What should I use as Trigger?
[13:25] <ravikant_> libpng1.6 in this case?
[13:25] <ravikant_> I meant libpng1.6/1.6.43-5build1
[13:26] <schopin> IMHO if it's only on i386 we shouldn't waste too much time on it.
[13:27] <slyon> ravikant_: It always depends a little bit on the apt output from the respective logfile. Sometimes it can be the package to be migrated itself (if there's a newer version in -proposed), or there can be anohter hint at the end of the logfile, which might be worth checking for a newer version in -proposed. A last resort would be &all-proposed=1 (that's what I did on i386, as I didn't want to waste too much time on int)
[13:28] <ravikant_> ok, I'll try that.
[13:29] <rbasak> So https://autopkgtest.ubuntu.com/packages/g/graphviz/noble/i386 for example seems to have a bunch of attempts and hasn't resolved.
[13:29] <rbasak> Should I investigate deeper? What does not wasting time on it mean exactly?
[13:40] <schopin> rbasak: our short term goal being to be able to build images, if it's a badpkg issue and not an actual test failure I think hint through the failure is acceptable.
[13:40] -queuebot:#ubuntu-release- New sync: python-influxdb-client (noble-proposed/primary) [1.40.0-1]
[13:40] -queuebot:#ubuntu-release- New sync: python-reactivex (noble-proposed/primary) [4.0.4-1]
[13:47] <sil2100> I don't know what was vorlon's recommendation for this, but I'd like for someone at least to take a look at the badpkg issue? Since I'd be curious what's up
[13:48] <schopin> I'm trying to figure out the linux-libc-dev thing but I'm actually having issues just having a suitable i386 environment to begin with :/
[13:49] <schopin> https://autopkgtest.ubuntu.com/packages/g/graphviz/noble/i386 -> that one has all-proposed=1 badpkg :/
[13:50] <athos> Are we working in any specific list ATM?
[13:50] <jamespage> those two syncs above are to support a sync for cloudkitty to 20.0.0 which is currently blocking autopkgtests in noble (as its the old release which is not compatible with the oslo's in noble).
[13:54] <sil2100> New britney run just finished \o/ waiting for the new update_excuses
[13:55] <slyon> athos: cdrkit openldap glib2.0 systemd parted qemu libpng1.6 snapd apt python-apt
[13:56] <slyon> but beware: some are hinted/skiptest already. So ignore those
[14:00] <sil2100> huh, weird, snapd still doesn't want to migrate even the reference tests failed, let me just hint it to save time
[14:00] <vorlon> doko: abseil; mpg123; .
[14:04] <athos> apt is interesting since it is hindering mksbuild usage (for noble schroots) is it hinted. it seems taht steve is getting those fixed now.
[14:12] <schopin> If anyone hasn't done so already, remember to refresh your excuses page :)
[14:14] <sil2100> It's getting shorter every moment
[14:14] <upils> I am bit lost about the test failure for nova (as explained above). The same error is also happening on i386. Anyone available to quickly check and tell me if this is something obvious? (like something we should hint for now)
[14:15] <schopin> upils: I don't see nova on the new excuses.
[14:16] <upils> I do :D (but maybe I am looking in the wrong place -> https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_excuses.html#parted)
[14:16] <schopin> Ah, you mentioned it blocking livecd-rootfs earlier.
[14:18] <schopin> upils: I'd start with an all-proposed=1 run
[14:18] <upils> > Ah, you mentioned it blocking livecd-rootfs earlier.
[14:18] <upils> I did, yeah, my bad
[14:19] <upils> ok, will do, thanks, but I do not understand why it would fix the issue
[14:21] <jamespage> upils: looking that that nova failure
[14:22] <schopin> upils: it wouldn't fix the root issue, but it could potentially allow the tests to run.
[14:23] <jamespage> upils: I think the package install is confused by the mismatch of 'all' and 'amd64' package versions between noble-updates and nobel{-proposed}
[14:24] <jamespage> I've also seen a few of these but I've stopped hitting recheck until rebuilds are all through
[14:24] <upils> jamespage, thanks. You think that because you see `522s Get:143 http://ftpmaster.internal/ubuntu noble-updates/main amd64 libvirt0 amd64 10.0.0-1ubuntu1 [1822 kB]` ?
[14:25] <upils> schopin, oh ok. So I guess it could at least unblock the situation
[14:26] <jamespage> upils: I think so - only the amd64 binaries got removed from the release pocket so we're in this weird mismatched version position right now
[14:26] <jamespage> this exploded my laptop this morning :)
[14:26] <vorlon> doing a retrigger now of all failed tests for the key packages for livecd-rootfs just in case
[14:27] <vorlon> juliank: cleaning out -updates requires freezing and unfreezing the archive, so no
[14:27] <vorlon> jbicha, schopin: :native is not relevant to how we build packages, so no
[14:28] <vorlon> schopin: auotpkgtest runners are cross-arch, launchpad builds are not
[14:28] <schopin> oh? TIL.
[14:29] <jamespage> upils: https://dpaste.com/5JFPVHJZY - fairly easily reproduced locally
[14:30] <vorlon> rbasak: glib2.0 tests queued> tests will be queued but no guarantee they'll pass on a retry! so analysis needed
[14:30] <vorlon> rbasak: we will ignore i386 test failures liberally
[14:32] <jbicha> I'm uploading gtk+2.0 now, because gtk+2.0 already migrated for xz-utils. This will allow overlay-scrollbar to build on armhf
[14:32] <vorlon> sil2100: badpkg issue> which one?
[14:32] <juliank> vorlon: hmm, isn't it just for package in packagesd; if version is newer in release pocket; then remove-package -s noble-updates package; fi; done
[14:33] <vorlon> juliank: we *cannot remove packages from -updates*
[14:34] <vorlon> without freezing the archive
[14:34] <juliank> sounds peculiar
[14:34] <schopin> vorlon: a lot of i386 tests are failing due to installability problems
[14:35] <vorlon> schopin: if those are the only tests blocking, ask the release team to skiptest hint the package
[14:35] <vorlon> but if there are tests pending, we wouldn't hint yet
[14:35] <vorlon> also now queuing --no-proposed retests for all failing tests
[14:39] <RikMills> I think KDE stacks are finallt geting there. thank you for being patient and letting our retry script do its job
[14:39] <RikMills> *finally
[14:40] <vorlon> yes, the archive mails for KDE stack landing in the release pocket have been promising
[14:40] <vorlon> now if only we could get glib2.0 in ;)
[14:40] <juliank> Like I said this morning EU time
[14:40] <juliank> > libkpim5kmanagesieve5 4:23.08.5-0ubuntu3 has Depends: libqt5core5t64 and migrated, but libqt5core5t64 is only in proposed at least here
[14:40] <schopin> qa-help: the last two attempts on https://autopkgtest.ubuntu.com/packages/a/adios2/noble/arm64 seem to be an infra failure?
[14:40] <vorlon> juliank: sure, so someone (on Kubuntu) could run the qtbase-opensource-src blocking tests to ground
[14:41] <juliank> i.e. KDE PIM stuff migrated *before* the Qt it depends on
[14:41] <juliank> Well this unsat dependency is making apt just remove plasma-desktop
[14:42] <juliank> why did we migrate it in teh wrong order?
[14:42] <juliank> I'd have expected us to not break amd64 dependencies harder
[14:42] <vorlon> only if it didn't increase net uninstallability on amd64
[14:42] <schopin> Maybe the LP outage interrupted the publisher?
[14:43] <juliank> Nah I don't think Qt is ready to migrate yet
[14:43] <vorlon> juliank: it could have been either noble-updates nonsense because britney looks at it; or because we removed amd64 binaries from release pocket but not arch: all, which britney counts uninstallibilities of specifically on amd64
[14:44] <juliank> Ah so everything is horrible basically
[14:44] <jbicha> yay, webkit2gtk finished building
[14:44] <vorlon> orig: 20349+0: a-3297:a-3167:a-4876:i-266:p-2940:r-2912:s-2891
[14:45] <upils> looking at libvirt to try and possibly unblock parted
[14:45] <vorlon> that's a lot of packages britney says are uninstallable on baseline :P
[14:46] <vorlon> the only meaningful blocking test for cdrkit is nova, which is blocked by installability of test deps
[14:47] <vorlon> livecd-rootfs ignorable, ubiquity ignorable (on arm64 heh), i386 ignorable, all other tests pass on amd64
[14:47] <upils> vorlon, that is what I am trying to solve
[14:47] <vorlon> so hinting
[14:47] <upils> (the installability of test deps for nova)
[14:47] <Skia> schopin: sorry, was in a call
[14:47] <Skia> looking
[14:48] <vorlon> upils: fixing it is not the requirement, vs analyzing it and making sure it doesn't pose a real risk of regression
[14:48] <vorlon> upils: (the installability should *also* be fixed but async)
[14:48] <schopin> Skia: it's not just that package either, I've seen this on a couple of others too.
[14:49] <schopin> looking into libpng1.6 vs gtk4 (looks like a real regression)
[14:49] <andersson1234> yes, those look like infra failures
[14:49] <andersson1234> we can re-enable retries if you'd like, though that may slow down our throughput
[14:50] <Skia> schopin: yeah, that definitely looks like an infra issue, and I've had other issues with the infra today. I currently have an open RT about those, even though not on bos02. IS also told us that the Launchpad fire earlier got them quite busy, so maybe related.
[14:50] <Skia> andersson1234: are the three usual retries disabled?
[14:51] <andersson1234> yes, let me double check
[14:51] <Skia> andersson1234: okay, I missed that. That may be worth mentionning on our status page.
[14:52] <andersson1234> yes, since 28th march https://git.launchpad.net/autopkgtest-cloud/commit/?id=17dc61a0867ebf23d482fd7d6e0d695849ec343b
[14:53] -ubottu:#ubuntu-release- Commit 17dc61a in autopkgtest-cloud "Merge remote-tracking branch 'andersson123/reduce-retries'"
[14:53] <andersson1234> (and then your amendment after) - I will add it to our status page
[14:55] <sil2100> vorlon: quick question about apt in noble-proposed - I see that it's actually a no-change rebuild - I'm probably missing context, but any reason not to skiptest it?
[14:55] <vorlon> sil2100: I don't have any reason not to - it just was missed by my scripts because it was a double-rebuild (oops)
[14:56] <sil2100> Let me do that then
[15:19] <vorlon> hmm the curl autopkgtest failure on openldap looks bad, it's not caused by openldap but random rebuild ordering means something still depends on libelf1 instead of libelf1t64 on amd64 and makes a sad
[15:20] <vorlon> oh but that's in the release pocket too
[15:20] <vorlon> so maybe fixed by migrations alone
[15:22] <pushkarnk> vorlon: I guess its debugedit
[15:22] <vorlon> looks like it may be, yes
[15:28] <vorlon> pushkarnk: interestingly, debugedit is blocked in -proposed by a similar installation failure on debhelper
[15:28] <vorlon> I'll just hint it through
[15:29] <pushkarnk> ok
[15:29] <vorlon> wait, ehm
[15:32] <vorlon> ok weird debugedit was uploaded on March 1 but wasn't built on amd64 until March 30 so missed the xz-utils window?!
[15:32] <vorlon> and the new debugedit still depends on libelf1 because of the random rebuild ordering heh
[15:33] <bdmurray> vorlon: Could there be other double-rebuilds that are missing?
[15:35] <vorlon> bdmurray: yes but I don't have a good way to systematically detect them without changelog parsing etc because in some cases the first rebuild will be for time_t so not automatically ignorable by the policy we have so far agreed
[15:35] -queuebot:#ubuntu-release- New binary: libqofono [amd64] (noble-proposed/universe) [0.122-2] (no packageset)
[15:35] -queuebot:#ubuntu-release- New binary: libqofono [ppc64el] (noble-proposed/universe) [0.122-2] (no packageset)
[15:35] <vorlon> so anyway, debugedit reuploaded
[15:35] <vorlon> and curl vs openldap ignorable
[15:36] <andersson1234> I've been looking at openldap vs freedombox btw, as well as a few others
[15:40] -queuebot:#ubuntu-release- New binary: libqofono [armhf] (noble-proposed/universe) [0.122-2] (no packageset)
[15:40] <andersson1234> pushkarnk: where are you up to on openldap?
[15:41] -queuebot:#ubuntu-release- New binary: libqofono [s390x] (noble-proposed/universe) [0.122-2] (no packageset)
[15:43] <pushkarnk> andersson1234: curl, to be honest. I did run through some other packages test failures, but didn't really spend quality time on them
[15:43] <bdrung> i am looking at mtd-utils now
[15:44] <vorlon> andersson1234, pushkarnk: so anyone tried to reproduce the lighttpd/amd64 autopkgtest failure? (a priority because it's amd64)
[15:45] <schopin> vorlon: I think we should badtest gtk4/4.14.1+ds-0ubuntu2 (and ubuntu1), see LP: #2059158
[15:45] -ubottu:#ubuntu-release- Launchpad bug 2059158 in gtk4 (Ubuntu) "proposed-migration for gtk4 4.14.1+ds-0ubuntu1" [Critical, New] https://launchpad.net/bugs/2059158
[15:46] <vorlon> schopin: badtest, or baseline retest?
[15:47] <dbungert> logs: missingbuild log with arch filtering https://paste.ubuntu.com/p/F4C3G5RMhb/
[15:47] <dbungert> missingbuild, no arch filtering https://paste.ubuntu.com/p/KNWWhPHWB8/
[15:47] <schopin> vorlon: Well, on amd64 baseline would succeed since the tests are properly executable in the -updates version.
[15:47] <dbungert> autopkgtests in a PERMAFAIL state (regressions) https://paste.ubuntu.com/p/fb3t7VQ7pF/
[15:47] <dbungert> blocked chains https://paste.ubuntu.com/p/SjyHTxmjhY/
[15:49] <vorlon> schopin: I don't think the baseline pulls from -updates?
[15:49] <jbicha> schopin: yes, nteodosio also recognized the executable problem for autopkgtests but I don't think we were intending to upload until gtk4 migrated first
[15:49] <jbicha> https://salsa.debian.org/gnome-team/gtk4/-/merge_requests/22
[15:49] -ubottu:#ubuntu-release- Merge 22 in gnome-team/gtk4 "Make installed-tests executable." [Closed]
[15:49] <vorlon> schopin: correct me if I'm wrong
[15:50] <schopin> jbicha: I figured, and yes I think it'd be bad to upload just to fix that right now :)
[15:51] <schopin> vorlon: experience tells me that when we disagree I'm usually the one being wrong.
[15:51] <vorlon> schopin: well let's look at the logs :)
[15:52] <vorlon> schopin: https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/l/lighttpd/20240404_002824_ea614@/log.gz the one I'm looking at shows noble-updates so you're right
[15:52] <schopin> but where would the gtk package be pulled from in baseline if not from updates?
[15:52] <vorlon> schopin: badtesting now
[15:52] <vorlon> schopin: well if it failed to be pulled in at all then the tests would also definitely fail!
[15:52] <schopin> heh
[15:53] -queuebot:#ubuntu-release- New binary: libqofono [arm64] (noble-proposed/universe) [0.122-2] (no packageset)
[15:54] <vorlon> schopin: so did you want this badtested on amd64 or on all archs?
[15:55] <vorlon> schopin: it *fails* on all archs but the baseline should fail the same way on !amd64
[15:55] <schopin> well there you go, amd64 only.
[15:55] <vorlon> ok
[16:07] <vorlon> 2725s Duplicate config variable in conditional 0 global: server.name
[16:07] <vorlon> ok lighttpd failure is Not Our Fault
[16:09] <ahasenack_> do we have a wiki or discourse page that summarizes the time_t effort, for public consumption?
[16:09] <vorlon> no
[16:12] <vorlon> ahasenack_: is https://autopkgtest.ubuntu.com/packages/n/nfs-utils/noble/s390x interesting to the Server Team
[16:12] <ahasenack_> 938s mount.nfs: mount system call failed for /mnt
[16:12] <ahasenack_> yes
[16:14] <ahasenack_> even a simple mount seems to be failing, not just kerberos
[16:15] <vorlon> andersson1234, pushkarnk: I'm hinting openldap now
[16:15] <vorlon> freedombox vs ufw Breaks is not an openldap regression
[16:15] <ahasenack_> oh, s390x only
[16:15] <ahasenack_> well, that's easier than armhf to get a VM on
[16:15] <vorlon> and it looks like it passed when andersson1234 retried the test
[16:15] <vorlon> ahasenack_: :)
[16:16] <sergiodj> ahasenack_: I have a machine available if you need
[16:16] <ahasenack_> sergiodj: armhf?
[16:16] <vorlon> looking at glib2.0 now
[16:16] <sergiodj> ahasenack_: s390x
[16:17] <vorlon> glib2.0 a bit trickier because I see a lot of armhf test failures and this isn't a no-change rebuild
[16:17] <sergiodj> vorlon: I'm a bit out of the loop (meeting right now), but is there any openldap problem that needs to be investigated?
[16:17] <vorlon> note to all: the regular autopkgtest queue is empty, don't be shy about triggering retries of failing tests if you need to
[16:18] <vorlon> sergiodj: no, openldap hinted, all failing revdep tests are analyzed or cleared via retry
[16:18] <sergiodj> vorlon: ack, thanks
[16:20] <cpaelzer> vorlon: I saw on https://bugs.launchpad.net/ubuntu/+source/bpftrace/+bug/2052809 you only promoted -release, but the versions not -updates and -proposed. Would those not also be needed to not set back the component when it migrates?
[16:20] -ubottu:#ubuntu-release- Launchpad bug 2052809 in bpftrace (Ubuntu) "[MIR] bpftrace" [Undecided, Fix Released]
[16:20] <vorlon> oh, I see ubuntu-server images are building successfully on non-amd64, confirming that remaining bootstrap issues are amd64-only, excellent
[16:20] <vorlon> cpaelzer: -proposed yes, -updates will be deleted so doesn't matter. Can you take care of it?
[16:20] <cpaelzer> I will
[16:21] <cpaelzer> also I'll do the same for src:bpfcc so that this mismatch isn't a migration blocker anymore
[16:21] <sergiodj> vorlon: is there a list of issues you guys are using today?
[16:21] <vorlon> cpaelzer: sorry, no real way from the component-mismatches reports to see that packages need promoting in both pockets
[16:21] <schopin> sergiodj: I think https://paste.ubuntu.com/p/SjyHTxmjhY/
[16:22] <vorlon> sergiodj: the remaining packages for stage 1 that need to get migrated asap are: glib2.0 systemd parted qemu libpng1.6 snapd
[16:22] <sergiodj> schopin: thanks
[16:22] <vorlon> sergiodj: see scrollback for information about any blocking test failures that people are already investigating
[16:22] <sergiodj> vorlon: thanks, will do
[16:23] <vorlon> once those are all cleared, the list schopin points to is good
[16:23] <bdrung> Has anyone seen such kind of failure? https://bugs.launchpad.net/ubuntu/+source/mtd-utils/+bug/2060214
[16:23] -ubottu:#ubuntu-release- Launchpad bug 2060214 in mtd-utils (Ubuntu) "mtd-utils 1:2.1.6-1build1 FTBFS" [Undecided, New]
[16:25] <vorlon> someone could go deep on https://autopkgtest.ubuntu.com/packages/a/ayatana-indicator-session/noble/armhf to see what's still pulling in libelf1 on armhf instead of libelf1t64, and get that resolved in the release pocket
[16:27] <bdrung> let me check ayatana-indicator-session and leave mtd-utils to someone else.
[16:27] <juliank> vorlon: bdrung: https://magenta.jak-linux.org/ubuntu-archive/distcheck/noble.armhf/release.txt says debugedit:armhf depends on libelf1, cuasing lots of unsat build-depends
[16:27] <vorlon> analyzed armhf test failures against glib2.0 of ayatana-indicator-session, booth, cjs, cpdb-backend-file, dbus-test-runner, all ignorable for promotion
[16:28] <vorlon> juliank: well I just uploaded debugedit
[16:28] <vorlon> juliank: because it was depending on libelf1 on amd64 after the latest rebuild and breaking things; when that migrates to the release pocket it should be cleaner?
[16:29] <juliank> I think retrying with new debugedit should clear it out?
[16:29] <juliank> if it's published yet
[16:29] <juliank> but wild guess
[16:29] <vorlon> juliank: the existing debugedit on armhf should already have been built against libelf1t64
[16:30] <juliank> That said I can actually run dose against debugedit on the release pocket
[16:30] <vorlon> juliank: https://launchpad.net/ubuntu/noble/armhf/debugedit/1:5.0-5build1 confirms, the rebuild from the beginning of March already has the libelf1t64 dep
[16:31] <juliank> But my report from release pocket also says:
[16:31] <juliank>   * ayatana-indicator-session                build-depends on missing cmake-data:armhf (= 3.28.3-1build5) via cmake:armhf, but not a regression
[16:31] <juliank> did you kill cmake entirely in the release pocket?
[16:32] <vorlon> I probably had to kill cmake-data:all because of cmake self-build-depends, so that it could pull the older version from -updates
[16:32] <juliank> zeah so ugh
[16:33] <cpaelzer> Also the build log says debugedit -> libelf1t64 in https://launchpadlibrarian.net/723142362/buildlog_ubuntu-noble-armhf.debugedit_1%3A5.0-5build2_BUILDING.txt.gz
[16:33] <juliank> nothing that has cmake in build/depends and tests having buildddeps in it is going to pass
[16:33] <cpaelzer> vorlon: FYI the bpfcc and bpftrace component mismatches are fixed now
[16:33] <vorlon> cpaelzer: cheers
[16:33] <juliank>  debugedit  | 1:5.0-5                 | noble           | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
[16:33] <juliank>  debugedit  | 1:5.0-5build1           | noble-proposed  | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
[16:33] <juliank> debugedit with the rebuild against t64 never left proposed
[16:34] <juliank> So yes, broken cmake and debugedit still broken
[16:34] <vorlon> oh, fwiw i386 autopkgtests can fail because of multi-arch: same libraries not being installable because i386 and amd64 have different versions available: https://autopkgtest.ubuntu.com/packages/d/dbus/noble/i386
[16:34] <vorlon> which just reinforces that we should be ignoring i386 right now
[16:34] <juliank> Yes
[16:35] <juliank> I figured as much when I read lots of i386 failures earlier
[16:35] <vorlon> juliank: yeah debugedit curiously was ftbfs (dep-wait?) on amd64 for a month, so missed xz-utils entirely and didn't have to be removed, *but* when it finally built it built against libelf1 instead of libelf1t64 :P  so now that's fixed and it should be migratable today
[16:35] <juliank> So realistically you need to drop *all* cmake binaries from the release pocket
[16:35] <vorlon> I haven't looked at cmake it's not critical path
[16:35] <vorlon> juliank: no because old cmake in -updates on armhf is not t64
[16:35] <juliank> i.e. they are not installable due to = cmake-data depends
[16:36] <vorlon> juliank: where are you seeing problems from this?
[16:36] <juliank> vorlon: that's the other half of why the autopkgtest is failing
[16:36] <andersson1234> vorlon: good to hear the test passed, I'm going to try and reproduce the lighttpd/amd64 test now
[16:36] <vorlon> juliank: "the" autopkgtest?
[16:36] <juliank> And if you were to try apt in the release pocket it would fail too
[16:37] <juliank> vorlon: the one you mentioned ayatana-indicator-session
[16:37] <vorlon> andersson1234: I reproduced lighttpd/amd64 and it's a garbage test harness in lighttpd, generates a bogus config file for lighttpd itself
[16:37] <juliank> Like I said, the same applies to e.g. apt migration-reference/0 tests - they would fail now
[16:37] <vorlon> juliank: ok cool - so the only action here is to make sure cmake gets promoted
[16:37] <juliank> very urgently
[16:38] <vorlon> not as urgently as unblocking image builds
[16:38] <andersson1234> vorlon: okay i'll hold off on that and continue looking at regressions
[16:38] <juliank> before britney goes retries migration-reference/0 tests of packages depending on it and just migrates all
[16:38] <vorlon> but if migrating cmake makes a lot of other tests pass so we don't have to manually deal with them that's a good thing
[16:39] <vorlon> juliank: tests for cmake are currently not queued at all since it's not seeded in any images.  Feel free to queue up those tests now to move cmake forward
[16:40] <vorlon> bdmurray, andersson1234: do we still have horrible shell code in autopkgtest-cloud/autopkgtest to compute the list of binary packages to set a pin for, and if so should we replace that with Pin: src:$sourcepackage which I think didn't exist at the time this was first implemented
[16:41] <juliank> Atriggered
[16:43] <bdmurray> paride: ^^
[16:43] <schopin> ww/quit
[16:44] <schopin> Hem. Disregard this :)
[16:45] <vorlon> someone please take care of util-linux
[16:49] <vorlon> well gst-omx/armhf autopkgtest failures are ignorable, that package was removed from release this week as a Debian removal
[16:50] <vorlon> juliank: fwiw this is an argument for prioritizing --all-proposed retests right now instead of --no-proposed
[16:53] <juliank> I wish I could help more but the time_t stuff and DST kind of killed much of me
[16:54] <juliank> I stayed up later and later during time_t and it's all broken now :(
[16:54] <andersson1234> tackling openldap vs krb5
[16:54] <vorlon> seb128, jbicha: libgdata autopkgtest failures on armhf and ppc64el are concerning, mention of sysprof vs libsoup here
[16:57] <vorlon> seb128, jbicha: similar failure for libsoup2.4 itself https://autopkgtest.ubuntu.com/packages/libs/libsoup2.4/noble/armhf looks to me like libsoup2.4 has an undeclared dependency on sysprof, that was masked by glib2.0 depending on it, which has now been backed out
[16:58] <ahasenack_> what about krb5? I might be able to help, between sssd debugging sessions
[17:00] <vorlon> I haven't looked but if it hasn't migrated yes that's important
[17:00] <vorlon> can't have an amd64 server build without it :)
[17:00] <vorlon> (and probably not desktop)
[17:01] <ahasenack_> looking at https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_excuses.html#krb5, seeing some reds
[17:01] <ahasenack_> some double listed packages in different versions, usually a sign that a new run could clarify things
[17:02] <andersson1234> openldap vs nfs-utils (s390x) looks like a legitimate test failure, nothing I can amend with additional triggers or all-proposed
[17:02] <ahasenack_> I filed a bug about the nfs-utils failure on s390x
[17:03] <vorlon> ahasenack_: https://paste.ubuntu.com/p/F4C3G5RMhb/ is the list of seeded packages with build failures that need resolved, but https://paste.ubuntu.com/p/fb3t7VQ7pF/ is the list of all seeded packages that need tests resolved; I don't see krb5 listed there so maybe dbungert needs to refresh something
[17:03] <vorlon> andersson1234: I already hinted openldap, see above
[17:03] <dbungert> I'm setting up hosting of the logs at https://people.canonical.com/~dbungert/ausrede/
[17:06] <andersson1234> ack
[17:06] <ahasenack_> excuses is a bit old
[17:06] <vorlon> ahasenack_: current runtime is ~2h
[17:06] <ahasenack_> ~2h
[17:06] <ahasenack_> ok
[17:06] <vorlon> so you should see an update soon
[17:09] <vorlon> almost done reviewing glib2.0 blocking tests
[17:11] <vorlon> I thought I saw/heard mention somewhere of issues with tracker?
[17:11] <vorlon> tracker/armhf has a SIGABRT in its autopkgtest.  Won't block on this
[17:13] <andersson1234> looking at python-apt vs breezy-debian
[17:13] <cpete> I've started working on a script to parse the ausrede output and query the launchpad api for missing builds. my first attempt looks something like this https://paste.ubuntu.com/p/CdM7XzrRkm/
[17:14] <cpete> everything else is either built or building
[17:14] <cpete> would anyone else find this useful?
[17:16] <vorlon> doko: I don't understand the dbusada/s390x failure
[17:17] <vorlon> doko: (autopkgtest)
[17:17] <vorlon> bdmurray: ^^ maybe you can remind me what 'erroneous package: rules extract failed' points to?
[17:18] <vorlon> stopping there for a bit. next on my list would be systemd
[17:19] <vorlon> remaining packages for installability of livecd-rootfs: systemd parted qemu libpng1.6 snapd
[17:20] <jbicha> vorlon: libsoup2.4 used an auto feature for sysprof integration, so glib's change meant that the build configuration was different. I can upload libsoup2.4 with sysprof explicitly disabled now
[17:21] <jbicha> I did get a change into debhelper so that compat 14 will make meson features enabled instead of auto; that didn't make it into noble's debhelper & compat 14 is still "beta" anyway
[17:24] <jbicha> disabling the sysprof feature avoids component-mismatch churn now since we don't have time to deal with it
[17:26] <bdmurray> vorlon: looks like the `dpkg-source` call in runner/autopkgtest
[17:43] <vorlon> jbicha: I suggest saving the upload of libsoup2.4 until after the current version migrates
[17:45] <vorlon> bdmurray, sil2100, ginggs: I'm going to prepare a list of --all-proposed autopkgtest retries for all the failed tests blocking seeded packages.  This a) has the highest chance of success on retry, b) is fairly safe because we can be confident that ~everything currently in -proposed that's seeded will migrate before release so any runtime regressions this might mask would be transient, c) has least
[17:45] <vorlon> load on autopkgtest runners and human eyeballs of any of the available strategies for clearing thees
[17:47] <sil2100> I think that's fair. Basically we already had all autopkgtests running all-proposed for a long time when we were hacking down on the time_t transition, so it's not like this is that much worse
[17:47] <vorlon> I'm afk now for a bit and it takes a while to generate the list so I won't be dispatching for ~1h
[17:47] <ginggs> vorlon: ack
[17:53] <bdmurray> Its weird that dbusada only hits the rules extract error one some arches
[17:54] <doko> vorlon: please ignore, the dbusada binaries for armhf and s390x were removed in the release pocket, but apparently are back again. I'll go over these again and remove
[18:00] <dbungert> log generation now automated and hosted at https://people.canonical.com/~dbungert/ausrede/
[18:03] <bdmurray> doko: when you say please ignore are you suggesting to a hint for dbusada on armhf and s390x?
[18:03] <bdmurray> s/to a/to add a/
[18:05] <ginggs> doko: i'll hints for dbusada
[18:05] <ginggs> add, even
[18:08] <doko> so these were all back again in the release pocket ... removed again. I even had all the commands in my shell history ...
[18:11] <bdmurray> you might see what else is in your shell history to find lazarus binaries
[18:14] <doko> ginggs: please also for adasockets libgnatcoll libtexttools libxmlada libfloristlibgmpada
[18:15] <doko> bdmurray: I didn't touch lazarus
[18:17] <doko> ginggs: also gprbuild
[18:18] <bdmurray> doko: lazarus was a joke about binaries coming back from the dead
[18:24] <ginggs> doko: some of those already have force hints, i added badtest hints for dbusada and adasockets
[18:25] <ginggs> i'll see what else is needed now that glib2.0 has been hinted
[18:29] -queuebot:#ubuntu-release- New binary: libqofono [riscv64] (noble-proposed/universe) [0.122-2] (no packageset)
[18:32] <enr0n> uploaded conntrack-tools as a part of unblocking systemd
[18:35] <jbicha> vorlon: oops, libsoup2.4 is already uploaded
[18:37] <ginggs> doko: i think gnat will still be blocked by
[18:37] <ginggs> Depends: plplot numpy (not considered)
[18:40] <doko> ginggs: yes, known
[19:06] <enr0n> network-manager autopkgtest fails because python3-gi gets removed (https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/n/network-manager/20240404_183130_7a1cc@/log.gz)
[19:06] <enr0n> does it make sense to do an upload with Depends: python3-gi in the d/t/control, even though it would normally be overkill?
[19:07] <enr0n> (at least I think it would be overkill normally, since python3-gi is Task: minimal)
[19:16] <bdmurray> enr0n: well the advantage would be network-manager tests would run when python3-gi was uploaded so I think it does make sense
[19:16] <vorlon> jbicha: ok well deleting it now
[19:17] <enr0n> bdmurray: okay, thanks. I will do the upload then
[19:20] <vorlon> doko: ah ok, again this is not in the release pocket but in -updates
[19:21] <vorlon> doko: oh, or rather it was in both pockets?
[19:21] <vorlon> doko, ginggs: anyway, already ignored it for glib2.0 due to low care factor, so this is fine
[19:25] <vorlon> doko: ok I can explain what happened with dbusada.  same version in release and updates pockets; because there was a newer, xz-contaminated version in -proposed we had copied to -updates and removed the amd64 binaries from the release pocket; upon realizing the mistake it was self-copied back to the release pocket to restore the amd64 binaries, and the other archs came along for the ride
[19:25] <vorlon> bdmurray: I prefer "zombie"
[19:29] <vorlon> confusing timestamps on proposed-migration.  Previous run started 16:45, update_excuses generated 17:28, update_output generated 17:30, next run didn't start until 18:36
[19:30] <vorlon> also archive mail quite delayed, I'm still getting migration notifications from 7am UTC?
[19:50] <vorlon> jbicha: the removal was unnecessary because the current britney run caught libsoup2.4 before your upload so it's migrating now; restoring your update
[19:50] <vorlon> also: good news this means glib2.0 is migrating now
[19:54] <vorlon> a lot of pending tests for systemd, and britney is almost done running, so not digging into systemd until the next update_excuses publication
[19:54] <vorlon> hinting parted now (only failing test is the known freedombox/armhf)
[19:56] <vorlon> schopin: mpg123 is back (no-change rebuild has migrated)
[20:01] <vorlon> libpng1.6 comes down to poppler/i386 and zvbi/i386; will take a peek at those but also skiptesting now ahead of the next britney run
[20:03] <vorlon> 3311s autopkgtest [18:28:33]: ERROR: |  libc6-dev:i386 : Depends: linux-libc-dev:i386 but it is not installable
[20:03] <vorlon> recurrent theme
[20:03] <vorlon> why is libc-linux-dev:i386 uninstallable?
[20:06] <vorlon> answer: because of linux binary removals on amd64 meaning multiarch mismatch
[20:11] <vorlon> update_excuses has refreshed
[20:13] <vorlon> hinting systemd (no-change rebuild only for CVE-2024-3094; all amd64 tests finished successfully except for those that are 'Test in progress (will not be considered a regression)')
[20:13] -ubottu:#ubuntu-release- Malicious code was discovered in the upstream tarballs of xz, starting with version 5.6.0. Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can b... <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-3094>
[20:13] <vorlon> .... uh thanks for letting us know, ubottu?
[20:14] <vorlon> (since when does that flag CVEs)
[20:17] <bdmurray> Who manages ubottu?
[20:28] <ahasenack_> made good progress on the sssd ftbfs on armhf, fix worked locally, testing in a ppa (which is faster to build than my pi4)
[20:29] <dbungert> liushuyu: my updated logs are now posted at https://people.canonical.com/~dbungert/ausrede/
[20:32] <mwhudson> dbungert: stupid feature request, html reports and clickable links
[20:32] <dbungert> mwhudson: absolutely sensible I think.  I was considering it anyhow.
[20:32] <dbungert> also now that I'm hosting it more sensibly than pastebin this is reasonble.
[20:33] <mwhudson> right
[20:36] <liushuyu> libcdio fix for armhf t64: https://code.launchpad.net/~liushuyu-011/ubuntu/+source/libcdio/+git/libcdio/+merge/463629
[20:36] <jbicha> vorlon: could you remove rust-zstd from proposed? easier not to do that transition
[20:37] <liushuyu> can we mass re-queue the tests w/ all-proposed=1 if they have "FAIL badpkg" in the log?
[20:42] <vorlon> liushuyu: I already requeued with all-proposed for all failed tests as of a few hours ago. what test are you looking at?
[20:45] <liushuyu> vorlon: some rev-dep tests of glib2.0, but if you have re-queued all the failed tests then it should be good
[20:45] <vorlon> liushuyu: but which one specifically do you see this on
[20:46] <liushuyu> also how should we coordinate the work with https://people.canonical.com/~dbungert/ausrede/blockedtree.log this list?
[20:48] <vorlon> liushuyu: by calling it out here
[20:48] <liushuyu> vorlon: for instance https://autopkgtest.ubuntu.com/packages/g/gphotofs/noble/amd64
[20:48] <liushuyu> I can see some of the tests have already finished with new results
[20:49] <vorlon> liushuyu: ah so that one shows it's already passed with all-proposed yes
[20:49] <vorlon> liushuyu: it's generally useful to check those pages before digging further
[20:50] <bdmurray> I'll dequeue  60faeb26-be86-4221-a061-cc867033b030 from that page then
[20:50] <ahasenack_> something I just noticed: a noble container I just deployed earlier today, on armhf, came with ifconfig installed instead of ip
[20:51] <ahasenack_> didn't investigate yet, but thought to mention it before I forget
[20:53] <vorlon> ahasenack_: you should talk to the CPC team about that I think
[20:56] <vorlon> woo python-apt migrated before I got to it
[20:56] <vorlon> ahasenack_: note that ifupdown isn't even in main so should never be pulled into a container image
[20:57] <ahasenack_> vorlon: a new image seems fine
[20:57] <mwhudson> didn't the cinnamon binary get removed on armhf as part of the t64 wash up?
[20:57] <liushuyu> taking glib2.0 vs rust-gio-sys
[20:57] <ahasenack_> I guess maybe something was removed while I was trying to install the build-deps for sssd mixing proposed, updates, and release
[20:57] <vorlon> liushuyu: glib2.0 is already done
[20:57] <vorlon> liushuyu: sorry, you said glib2.0 but I didn't key in on it - see scrollback
[20:58] <mwhudson> vorlon: for things like cinnamon should we (you?) remove the armhf binaries from noble-updates to placate britney?
[20:58] <vorlon> mwhudson: can't remove from noble-updates!
[20:58] <vorlon> mwhudson: needs a force hint
[20:58] <mwhudson> vorlon: hnngh what why
[20:58] <mwhudson> but ok
[20:58] <vorlon> mwhudson: because launchpad
[20:58] <mwhudson> vorlon: bugs or misfeatures?
[20:59] <liushuyu> vorlon: Ah I see you handled this
[20:59] <mwhudson> vorlon: can we forcibly empty the noble-updates pocket before release at least?
[21:00] <bdmurray> mwhudson: when it is frozen yes
[21:00] <bdmurray> so next week
[21:00] <vorlon> mwhudson: misfeature
[21:00] <mwhudson> ah
[21:00] <mwhudson> ok. or maybe "ok"
[21:01] <mwhudson> vorlon: so um, hint cinnamon?
[21:01] <liushuyu> taking libpng1.6 vs jskeus
[21:02] <vorlon> mwhudson: done thanks
[21:02] <vorlon> liushuyu: libpng1.6 also done!
[21:02] <vorlon> and I'm working on snapd now
[21:02] <liushuyu> Am I looking at the incorrect version of https://people.canonical.com/~dbungert/ausrede/blockedtree.log?
[21:03] <vpa1977> Please call out which package to start from =)
[21:03] <vorlon> liushuyu: no but you need to check scrollback whether a package is claimed
[21:05] <vorlon> snapd also migrated before I got to it, hurrah
[21:05] <mwhudson> what does "Package 'sysprof-capture-4', required by 'libsoup-2.4', not found" actually mean?
[21:06] <mwhudson> from here https://launchpad.net/ubuntu/+source/libmateweather/1.26.3-3.1build2/+build/27975754 but also another build i looked at
[21:06] <vorlon> mwhudson: glib2.0 pulled in sysprof stuff from Debian, which has now been backed out; libsoup2.4 built against glib2.0 while it was in this state, picking this up as a dependency; fixed in libsoup2.4 that j_bicha uploaded earler today
[21:07] <mwhudson> ah so these builds should be retryable now
[21:07] <liushuyu> libcdio is now done
[21:07] <vorlon> mwhudson: if they are build failures then yes
[21:07] <mwhudson> they are
[21:09] <mwhudson> retrying libmateweather goodvibes osm-gps-map for this
[21:09] <vorlon> snapd migrated but needs openssh
[21:09] <vorlon> looking at this now
[21:10] <vorlon> openssh migrated in the current round
[21:10] <liushuyu> which order are we looking at the list? The list starts with libpng1.6 at-spi2-core qtbase-opensource-src libarchive...
[21:11] <liushuyu> ... I searched the message history and it seems like at least at-spi2-core and libarchive are not mentioned anywhere
[21:12] <mwhudson> ahasenack_: you are still looking at sssd?
[21:12] <liushuyu> Also SRU team, please take a look at https://code.launchpad.net/~liushuyu-011/ubuntu/+source/gdb/+git/gdb/+merge/463546
[21:12] <ahasenack_> mwhudson: yes, seconds away fro confirming 1 fix
[21:12] <mwhudson> ahasenack_: hooray!
[21:13] <vorlon> liushuyu: that particular list is broader than the list I was working from.  I mentioned at various points that I was driving the list to unblock installability of livecd-rootfs; which is now done, it just means they've been worked out of order wrt that list
[21:13] <mwhudson> i think everything from https://people.canonical.com/~dbungert/ausrede/missingbuild_allarches.log is in progress or handled or hinted then
[21:13] <vorlon> dbungert: I wonder if you want to try to filter your reports against 'accepted: $src' in https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_output.txt
[21:14] <vorlon> dbungert: nevermind, the things still on that page seem to be the ones that *didn't* migrate in the last run but are hinted for the next one - so whatever filtering you're doing is already fine
[21:15] <dbungert> ack, thanks
[21:16] <vpa1977> would it be ok to start looking at qtbase-opensource-src/cmake (amd64 test failure) ?
[21:16] <liushuyu> taking at-spi2-core vs castle-game-engine
[21:17] <liushuyu> this one's queued as well
[21:18] <vpa1977> i think they are all queued and current failure is due to badpkg
[21:19] <vorlon> liushuyu, vpa1977: https://irclogs.ubuntu.com/2024/04/04/%23ubuntu-release.html#t17:45
[21:20] <vorlon> debootstrap currently blocked by coreutils; looking
[21:22] <vorlon> coreutils is a no-change rebuild, hinting
[21:23] <mwhudson> vorlon: do we have a way to check the pseudo-essential set?
[21:24] <vorlon> mwhudson: well they're all seeded (ubuntu-minimal)
[21:24] <mwhudson> vorlon: true
[21:24] <vorlon> but also Priority: required
[21:24] <vorlon> which is what I was just about to audit
[21:26] <mwhudson> ok fair enough
[21:26] <mwhudson> the latency on everything is a bit much
[21:32] -queuebot:#ubuntu-release- New binary: rust-zbus-1 [amd64] (noble-proposed/universe) [1.9.3-5] (no packageset)
[21:35] <vorlon> Priority: required binary packages currently missing from the release pocket on amd64 compared with noble-updates (false-positives possible due to t64 renames): https://paste.ubuntu.com/p/yZSKz2wPRv/
[21:36] <vorlon> I love that we've gone this far and glibc is missing from the release pocket
[21:37] <dbungert> the permafail list against glibc is impressive
[21:38] <vpa1977> looking at audit
[21:38] <vorlon> dbungert: but a good number of tests that passed on retry
[21:38] <dbungert> that is good news
[21:38] <vorlon> don't know why ableton-link/arm64 has neither a pending nor a finished all-proposed test
[21:45] <vpa1977> audit/dbus-broker armhf - needs all-proposed retry, audit/dbus - i386 failure (ignore), audit/network-manager (amd64) - bug in network manager
[21:48] <vpa1977> raised https://bugs.launchpad.net/ubuntu/+source/audit/+bug/2060241
[21:48] -ubottu:#ubuntu-release- Launchpad bug 2060241 in network-manager (Ubuntu) "prosposed migration audit 1:3.1.2-2.1build1 vs network-manager 1.46.0-1ubuntu1" [Undecided, New]
[21:49] <vpa1977> looking at util-linux
[21:49] <vorlon> vpa1977: so are you saying the release team should skiptest hint audit?
[21:51] <vpa1977> vorlon: I guess so? audit/dbus-broker passed before, audit is a no change rebuild, dbus-broker is in release.
[21:51] <vorlon> vpa1977: ok because you didn't *say* this :) please be explicit and assertive
[21:51] <vpa1977> vorlon: ack
[21:52] <vorlon> vpa1977: also fwiw audit was already hinted: Should wait for tests relating to audit 1:3.1.2-2.1build1, but forced by ubuntu-release
[21:52] <vorlon> (because it was a no-change rebuild)
[21:52] <vorlon> vpa1977: it's easy to miss this line, no angry fruit salad
[22:02] <cpete> can't tell where we're at in the list, but I will start looking at numpy I guess
[22:02] <vpa1977> util-linux - bugs in autopkgtests for gdcm/3.0.22-2.1build2,  openvswitch/3.3.0-1build1, also needs upload for sssd. Will raise bugs. Skip for now?
[22:17] <vorlon> vpa1977: why does it need an upload for sssd? that's known broken on armhf in the release pocket
[22:17] <vpa1977> vorlon: I mean for the sssd test to pass.
[22:18] <vorlon> ok so when you say "skip" are you saying you're skipping it and moving on or you're asking for a skiptest hint?
[22:18] <vpa1977> skipping and moving on
[22:18] <vorlon> why
[22:18] <kanashiro> I'm seeing some installability issues with gcc-13 (when trying to build some packages to sponsor the upload), is anyone working on that?
[22:18] <mwhudson> dbungert: how often is https://people.canonical.com/~dbungert/ausrede/ supposed to update?
[22:18] <vorlon> "buggy revdeps" is not a reason to skip, it's a reason to confirm this is ok to migrate and get it in?
[22:18] <vpa1977> A number of tests are still running for the package. The failures that I've seen seems to be unrelated though
[22:19] <vorlon> so that it's possible to debootstrap
[22:19] <vorlon> vpa1977: ok - which archs are the tests still running on?
[22:19] <dbungert> mwhudson: should update within 10 minutes of update_excuses.yaml updating
[22:20] <mwhudson> ah i am being impatient i see
[22:21] <dbungert> the timestamp I log is the same as the timestamp in the first line of update_excuses.yaml, but that seems a curiously long time
[22:21] <dbungert> before that timestamp and the mtime on https://ubuntu-archive-team.ubuntu.com/proposed-migration/ were closeish
[22:24] <vpa1977> vorlon: oh, need to click through "test in progress" - a few failed. neavark , openswitch amd64 still running, sbd , zonefs-tools, zuluscript queued,
[22:28] <vorlon> enough outstanding on glibc at the moment that even I'm not willing to hint it just yet
[22:28] <vorlon> will come back to it in a britney cycle or two
[22:29] <ahasenack_> got a good build of sssd on armhf, time to polish the changes and upload https://launchpad.net/~ahasenack/+archive/ubuntu/sssd-tests/+packages
[22:29] <kanashiro> I re-triggered some tests blocking gcc-13, the test results were error (not fail), so I think they may pass in the next run
[22:30] <vorlon> ahasenack_: a) huzzah; but also b) sssd was only blocked because of the revivified binaries in noble-updates so maybe you want me to hint the current one through first before you upload
[22:31] <ahasenack_> vorlon: I don't know. It's a real time_t 64 bug that the tests caught
[22:31] <ahasenack_> vorlon: do you have a reference to "faketime is broken on armhf"? Because that was indeed one of the "fixes" (don't install it)
[22:31] <vorlon> ahasenack_: yes but the armhf binaries were *removed* from the release pocket already for the time_t migration
[22:32] <vorlon> ahasenack_: britney is *only* blocking sssd migration because of noble-updates, all of which goes away before release; so this is an artifact not a real policy violation
[22:32] <vorlon> ahasenack_: https://bugs.launchpad.net/ubuntu/+source/faketime/+bug/2059078
[22:32] <ahasenack_> vorlon: ok then
[22:32] -ubottu:#ubuntu-release- Launchpad bug 2059078 in faketime (Ubuntu) "proposed-migration for faketime 0.9.10-2.1ubuntu1" [Undecided, New]
[22:33] <ahasenack_> I'll need a few minutes anyway to polish the changes (not too much: but the minimal necessary)
[22:35] <vorlon> mwhudson: when was the last time https://ubuntu-archive-team.ubuntu.com/proposed-migration/noble/rcbuggy-problem-packages.html was used
[22:35] <mwhudson> vorlon: i certainly don't know!
[22:35] <vorlon> I forgot it existed
[22:36] <vorlon> and it's been running for 20 minutes now and is the thing stopping the next proposed-migration run from starting
[22:36] <vorlon> lalala <stab>
[22:36] <mwhudson> uh 20 minutes?
[22:36] <vorlon> so far
[22:37] <mwhudson> sounds stuck. the slowest part iirc is parsing the yaml which might take 20 *seconds* on a bad day
[22:37] <vorlon> and I noted earlier the discrepancies between update_excuses/output generation times and the start time of the next p-m run
[22:38] <vorlon> maybe we should deprecate this in favor of cpaelzer's script added to ubuntu-archive-tools (if he ever finishes it :)
[22:38] <dbungert> mwhudson: html logs now for missingbuild* and permafail
[22:39] <mwhudson> vorlon: yeah if it's not being looked at there's no real point i guess
[22:39] <mwhudson> dbungert: nice
[22:41] <vorlon> so, next proposed-migration run has started which means it *should* be safe for ahasenack_ to upload sssd without it preventing migration of the current version
[22:44] <vorlon> looking at procps
[22:48] <mwhudson> vorlon: does the output from run-proposed-migration itself go anywhere?
[22:50] <vorlon> mwhudson: no, it runs as part of archive-reports which spawns processes in parallel and also runs a whole bunch of stuff so deliberately suppresses output
[22:54] <vpa1977> added https://bugs.launchpad.net/ubuntu/+source/xen-tools/+bug/2060247, https://bugs.launchpad.net/ubuntu/+source/gdcm/+bug/2060243, https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/2060245.
[22:54] -ubottu:#ubuntu-release- Launchpad bug 2060247 in xen-tools (Ubuntu) "xen-tools autopkgtest failure" [Undecided, New]
[22:54] -ubottu:#ubuntu-release- Launchpad bug 2060243 in util-linux (Ubuntu) "proposed migration  util-linux 2.39.3-9ubuntu4 vs  gdcm/3.0.22-2.1build2" [Undecided, New]
[22:54] -ubottu:#ubuntu-release- Launchpad bug 2060245 in openvswitch (Ubuntu) "[s390x] proposed migration util-linux  2.39.3-9ubuntu4 vs openvswitch/3.3.0-1build1" [Undecided, New]
[22:58] <vorlon> hinting procps
[23:00] <vorlon> also hinting debugedit (gets rid of libelf1 and cleans stuff up)
[23:06] <vorlon> hinting libarchive
[23:07] <vorlon> liushuyu: are you working on any packages right now? do you want to look at cmake?
[23:09] <vorlon> afk for a bit; will tackle glibc again after the next britney run
[23:10] <mwhudson> afk for a couple hours
[23:13] <vorlon> was going to schedule glibc tests to the regular queue to speed things up but this is pointless because that's basically the full contents of the huge queues right now, heh
[23:17] -queuebot:#ubuntu-release- New binary: rust-zbus-1 [ppc64el] (noble-proposed/universe) [1.9.3-5] (no packageset)
[23:17] <vorlon> kanashiro: hi, I see you mentioned gcc-13; this is not really a priority right now as it doesn't build any binary packages that are seeded on images afaik? all the runtime libs have been taken over by gcc-14, and are present in noble release pocket already
[23:18] <kanashiro> vorlon ack. So ideally, those package depending on gcc-13 should be using gcc-14?
[23:18] <vorlon> kanashiro: gcc-13 is the default compiler but gcc-14 provides the shared libs
[23:18] <vorlon> and I don't think gcc is seeded in the livefses for any images
[23:19] <vorlon> hmmmm well ok gcc is seeded in ubuntustudio ubuntu-mate ubuntu-unity edubuntu ubuntucinnamon
[23:19] <vorlon> which sounds like a bug to me
[23:19] <kanashiro> yeah, I do not think it is, I was trying to sponsor some upload for time_t transition but I cannot build those packages because of this gcc-13 issue
[23:20] <vorlon> well you mentioned tests blocking gcc-13, that shouldn't affect installability
[23:20] <vorlon> gcc-13 should be installable in -proposed which is what launchpad uses
[23:20] <kanashiro> I am building with -proposed but I get this:
[23:20] <kanashiro> https://chat.debianbsb.org/file/3/YLYcMabgq74lDDR7
[23:22] -queuebot:#ubuntu-release- New binary: rust-zbus-1 [armhf] (noble-proposed/universe) [1.9.3-5] (no packageset)
[23:22] <vorlon> kanashiro: neither of those is the current version in -proposed
[23:22] <vorlon>  libobjc-13-dev                    | 13.2.0-23ubuntu3 | noble-proposed/universe | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
[23:23] <kanashiro> yeah, sorry, this specific build I was trying with -updates
[23:23] <vorlon> but also uh what are you uploading that is a) implemented in ObjC and b) is in time_t
[23:23] -queuebot:#ubuntu-release- New binary: rust-zbus-1 [arm64] (noble-proposed/universe) [1.9.3-5] (no packageset)
[23:24] <kanashiro> I was trying to sponsor this: https://bugs.launchpad.net/ubuntu/+source/gvmd/+bug/2058958
[23:24] -ubottu:#ubuntu-release- Launchpad bug 2058958 in gvmd (Ubuntu) "armhf autopkgtest failure (due to time_t format args)" [Undecided, New]
[23:24] <vorlon> I see, ok
[23:25] <vorlon> (that shouldn't really be blocking anything for image builds however as gvmd is in universe)
[23:25] <vorlon> anyway, libobjc-13-dev should certainly be installable in -proposed
[23:27] <kanashiro> I see, I will try again to make sure I included -proposed
[23:27] <vorlon> ooh qtbase-opensource-src just got accepted
[23:28] <vorlon> I: [2024-04-04T23:27:15+0000] - trying: qtbase-opensource-src
[23:28] <vorlon> I: [2024-04-04T23:27:30+0000] - accepted: qtbase-opensource-src
[23:28] <vorlon> I: [2024-04-04T23:27:30+0000] -    ori: 5971+0: a-2729:a-436:a-1960:i-108:p-259:r-242:s-237
[23:28] <vorlon> I: [2024-04-04T23:27:30+0000] -    pre: 3796+0: a-1645:a-359:a-1224:i-9:p-190:r-189:s-180
[23:28] <vorlon> I: [2024-04-04T23:27:30+0000] -    now: 3701+0: a-1550:a-359:a-1224:i-9:p-190:r-189:s-180
[23:28] <kanashiro> vorlon if there is something requiring attention right now let me know, I can try to help
[23:28] <vorlon> starting to look something like a coherent archive?
[23:28] <vorlon> kanashiro: well liushuyu hasn't claimed cmake yet, maybe you want to
[23:38] <kanashiro> I see many tests are in progress, I'll check the ones that failed already
[23:38] <vpa1977> cmake failure is due to gobject introspection putting arch specific names into /usr/bin. I will add patch to cmake and forward it upstream
[23:40] <vorlon> vpa1977: juliank wants cmake migrated sooner rather than later because of the concern that cmake inconsistency in the release pocket will lead to lots of failing tests that will hide potential regressions
[23:40] <vpa1977> vorlon: then the failure can be ignored, this is a flaky assertion /usr/bin/x86_64-linux-gnu-g-ir-scanner on disk vs asserted /usr/bin/g-ir-scanner
[23:41] <sil2100> o/
[23:41] <kanashiro> vpa1977 since you are doing it already I'll leave it to you :)
[23:42] <vorlon> vpa1977: it's perfectly fine to stage the change (locally or as a bug report) but if you upload now, cmake will migrate "later", not "sooner"
[23:42] <sil2100> What's the status on the xz rebuild? Did we get livecd-rootfs installable?
[23:43] <vpa1977> vorlon: will raise a bug and link MR, but would not be uploading anything until clear
[23:43] <vorlon> sil2100: yes, with the next publication run
[23:43] <sil2100> \o/
[23:43] <vorlon> sil2100: known blockers for debootstrap now are util-linux and glibc
[23:43] <sil2100> Anything I can help out with right now?
[23:43] <vorlon> (coreutils also already hinted)
[23:43] <sil2100> oh no
[23:43] <sil2100> Usual excuses work, right?
[23:44] <vorlon> sil2100: see link in topic for lists that prioritized seeded packages
[23:47] <sil2100> ACK o/
[23:47] <sil2100> People looking already at util-linux and glibc, or would me looking at it additionally help?
[23:48] <vorlon> I'm waiting for the next britney run to complete before I dive back in to glibc and util-linux
[23:48] <vorlon> because of the number of pending tests
[23:48] <vorlon> and britney is finishing up right now
[23:48] <sil2100> Ah, of course util-linux is a no-change rebuild on top of a security update!
[23:53] <vorlon> I've tried to write up some guidance here wrt how I'm triaging autopkgtest failures on packages https://docs.google.com/document/d/14ed3ANBzpKPU_ZPSu5c9oVTDSVIneZ6Y5E6NJ2wAuTw/edit
[23:55] <vorlon> STATS:
[23:55] <vorlon> Finished at: Thu, 04 Apr 2024 23:54:49 +0000
[23:55] <sil2100> \o/
[23:57] <dbungert> ausrede logs updated
[23:58] <sil2100> Still some glibc tests in progress tho
[23:58] <vorlon> yes
[23:59] <vorlon> they are most of the outstanding autopkgtests in the queue
[23:59] <vorlon> I'm going to look at util-linux now which does look manageable
[23:59] <vorlon> and then probably let glibc go another round through britney