[00:12] <bdmurray> I see gnudatalanguage and eccodes being a blocker now
[00:13] <bdmurray> Retrying amd64 tests
[00:13] <bdmurray> actually arm64 and ppc64el too
[01:20] <bdmurray> Right, I can't read and 4-1build is greater than 1-4build so retrying didn't help.
=== chris14_ is now known as chris14
[07:36] <tjaalton> release team, check the logs at https://code.launchpad.net/~p-pisati/britney/+git/hints-ubuntu/+merge/461043 and hopefully accept it :)
[07:37] <tjaalton> blocking kernel 6.8 migration
[07:40] <LocutusOfBorg> hello ubuntu-archive please kick out liquidsoap if possible, ftbfs and blocking ocaml migration (debian rc bug is open 1064128)
=== guiverc2 is now known as guiverc
[08:44] <tjaalton> ginggs: hey, mind having a look at the the merge request above?
[09:02] <ginggs> tjaalton: i can take a look in a bit
[09:07] <tjaalton> thanks!
[10:06] <ginggs> tjaalton: so dracut/amd64 and initramfs-tools/s390x both passed, so no need for a hint for them
[10:07] <ginggs> however, all the linux-meta* packages seem to be failing with glibc that just migrated
[10:08] <ginggs> is this just a matter of retrying with the right triggers?
[10:11] <ginggs> i'll leave a comment in the MP
[10:12] <tjaalton> oh..
[10:13] <tjaalton> I've retriggered for amd64, see how it goes
[11:39] <juliank> I guess britney is extra super slow now
[12:06] <ahasenack> ugh, where did https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_excuses.html#gnudatalanguage come from
[12:14] <ahasenack> needs a rebuild, since eccodes 2.33 was removed
[12:19] <ahasenack> uploaded
[12:24] <tjaalton> ginggs: meh, still failing
[12:25] <tjaalton> trying to find ndbm.h
[12:33] <tjaalton> hmm not that
[12:50] <tjaalton> ginggs: arighi ran the glibc tests on a vm and it passed fine
[16:45] <bdmurray> ahasenack: What's the status of gnudatalanguage now?
[16:46] <ahasenack> bdmurray: built some hours ago, waiting for britney
[16:46] <ahasenack> one rdep test still running (in excuses)
[16:46] <ahasenack> there is a missing build on riscv64, but it's not new, so shouldn't impact migration
[16:47] <ahasenack> ah, actually two rdep tests running
[16:47] <ahasenack> coyote and mpfit
[16:47] <bdmurray> riscv64 did build
[16:48] <ahasenack> correct, n/m then
[16:48] <ahasenack> coyote/ppc64el is running still
[16:48] <bdmurray> looks like mpfit just passed
[16:50] <ahasenack> coyote/ppc64el running for 2h already
[16:50] <ahasenack> but the last line of the log says 425s
[16:50] <ahasenack> I guess the "running for" also includes time waiting for a vm?
[16:51] <bdmurray> running for is cumulative IIRC
[16:51] <bdmurray> I did see a lot of ppc64el failures in the KPI so will dig
[16:51] <ahasenack> the log hasn't moved since we started talking
[16:51] <bdmurray> ahasenack: could you retry poetry w/ all-proposed for noble/ppc64el as I also saw that as a blocker
[16:52] <ahasenack> poetry? let me check, I didn't see that
[16:52] <ahasenack> oh, unter pytest
[16:53] <bdmurray> I see it under python3-defaults too
[16:53] <ahasenack> bdmurray: "server error"
[16:53] <ahasenack> n/m
[16:53] <ahasenack> forgot one parameter
[16:56] <ahasenack> poetry started on ppc64el
[16:56] <ahasenack> I see live logs
[16:56] <ahasenack> coyote still stalled at 425s
[16:57] <bdmurray> Yeah, I'm connected to the instance and its stuck waiting for something
[16:59] <bdmurray> I'm gonna kill it
[17:02] <ahasenack> ok, it's running again, and alive
[17:04] <bdmurray> Why is ginggs pass for coyote wildly different than the failures?
[17:05] <ahasenack> one of the fails is a timeout
[17:05] <ahasenack> https://autopkgtest.ubuntu.com/request.cgi?release=noble&arch=ppc64el&package=poetry&trigger=python3-defaults/3.12.1-0ubuntu1
[17:05] <ahasenack> er
[17:05] <ahasenack> 263s % Compiled module: CGERRORMSG.
[17:05] <ahasenack> 10262s autopkgtest
[17:05] <ahasenack> see the time delta
[17:06] <ahasenack> looks like it was stuck, maybe just like what was about to happen to that job you killed
[17:06] <bdmurray> Yes, but look at https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/ppc64el/c/coyote/20240129_094213_cfe37@/log.gz
[17:06] <bdmurray> CGERRORMSG doesn't show at all
[17:06] <ahasenack> that's from a month ago
[17:07] <ahasenack> poetry failed :/
[17:08] <ahasenack> that looks like a network error
[17:09] <bdmurray> Are they in the same dc? bos01 vs bos02?
[17:10] <bdmurray> No, it looks like the poetry failures switch between DCs
[17:10] <ahasenack> 561s DEBUG    urllib3.connectionpool:connectionpool.py:1053 Starting new HTTPS connection (1): upload.pypi.org:443
[17:11]  * ahasenack checks other arches where it pssed, if they try that too
[17:11] <bdmurray> The firewall rules should be the same for s390x and ppc64el. (Mostly for arm64 too.)
[17:11] <ahasenack> even on ppc64el it passed before
[17:11] <ahasenack> looking at test_publish_returns_non_zero_code_for_upload_errors explicitly
[17:12] <ahasenack> but the ppc64el history isn't good, just one green?
[17:12] <ahasenack> time for migration-reference/0?
[17:12] <bdmurray> +1
[17:12] <ahasenack> done
[17:13] <ahasenack> oh, poetry was failing everywhere
[17:13] <bdmurray> Oh wow
[17:13] <ahasenack> looks like it was just that one lucky green that reset the status on ppc64el
[17:14] <ahasenack> maybe while the firewall was too open? :)
[17:14] <ahasenack> firewall/proxy
[17:16] <athos> Hi ubuntu-release! Could anyone please take a look at LP: #2054843 and add a few hints so we can complete the PHP transition as we did last time?
[17:16] -ubottu:#ubuntu-release- Launchpad bug 2054843 in php-defaults (Ubuntu) "Please force php-defaults 93ubuntu2 in nn" [Undecided, New] https://launchpad.net/bugs/2054843
[17:16] <ahasenack> bdmurray: look at this list of test exceptions: https://git.launchpad.net/ubuntu/+source/poetry/tree/debian/rules#n7
[17:18] <ahasenack> bdmurray: coyote/ppc64el passed
[17:18] <bdmurray> ahasenack: coyote - really? I think I'm still looking at the log file
[17:19] <bdmurray> Oh I see jbicha queued it too
[17:20] <ahasenack> https://autopkgtest.ubuntu.com/packages/c/coyote/noble/ppc64el shows a green run at the top with gnudatalanguage build2
[17:21] <bdmurray> Yeah, I found that too
[17:25] <ahasenack> poetry's migration-reference/0 failed too
[17:25] <bdmurray> \o/ for a failure?
[17:25] <ahasenack> yep
[17:25] <bdmurray> I wonder if we should hint coyote since jeremy's request is hung
[17:26] <ahasenack> where was that? ppc64el too?
[17:26] <ahasenack> 27min running, 303s in logs
[17:26] <bdmurray> yeah
[17:28] <ahasenack> it's not all runs that hit that, just a few
[17:28] <ahasenack> I count 4 in all the history of https://autopkgtest.ubuntu.com/packages/c/coyote/noble/ppc64el
[17:28] <ahasenack> you can see the large duration
[17:28] <ahasenack> all around 2h51min
[17:28] <ahasenack> should closely match the autopkgtest timeout
[17:28] <ahasenack> (my guess)
[17:29] <juliank> AFAICT python3-defaults should migrate with the next britney run due to the pass on coyote
[17:29] <ahasenack> I see bos01 and bos02 in the failures
[17:29] <bdmurray> I don't want britney to interpret the jeremy's run as a failure though
[17:30] <ahasenack> can you kill it?
[17:30] <bdmurray> It'll get requeued
[17:30] <ahasenack> it might pass then :)
[17:30] <bdmurray> Killing and removing it from the queue requires great luck
[17:30] <ahasenack> (or not)
[17:30] <ahasenack> 32min 303s, that one is definitely going to timeout at 2h51min
[17:30] <juliank> britney is fine, it already has a pass and it ignores later fails
[17:31] <juliank> It just needs one pass for the version
[17:31] <bdmurray> ack, thanks juliank!
[17:31] <juliank> I reran so many tests today it's fun
[17:32] <juliank> So I expect base-files, glibc and the entire python3-defaults chain to migrate today
[17:32] <juliank> Then things will be so much nicer :)
[17:32] <bdmurray> until the next thundering herd
[17:35] <juliank> python3-defaults may need some massaging with `hint` hints with all the implicit dependencies in it in case britney goes down the wrong path
[17:35] <ahasenack> 1h since the last excuses report, it's close
[17:36] <ahasenack> but I don't think it will migrate now, when the next report is published, but the one after that
[17:36] <juliank> I'm not sure it will be this one
[17:36] <ahasenack> yeah, it's running on data 1h old
[17:36] <juliank> We might get more britney runs though
[17:36] <juliank> It runs up to twice an hour afaict
[17:36] <juliank> It's just too slow right now
[17:38] <juliank> the current run started at :51, about 47 minutes ago
[17:38] <juliank> so exciting
[17:39] <juliank> results are in
[17:39] <juliank> sigh
[17:39] <juliank> Implicit dependency: python3-defaults pyvkfft (not considered)
[17:40] <juliank> Depends: pyvkfft nvidia-graphics-drivers-535 (not considered)
[17:41] <juliank> Why would a no-change rebuild of a python library depend on a new nvidia graphic driver
[17:42] <juliank> ah versioned depends
[17:42] <juliank> OK um it has no reverse depends, let's demote pyvkfft to proposed?
[17:43] <juliank> ubuntu-archive: let's demote pyvkfft to proposed, it depends on new nvidia driver, has no reverse {build-,}depends and block python3-defaults
[17:44] <juliank> i.e. remove the version in release
[17:48] <vorlon> juliank: looking
[18:07] <ahasenack> why did pvkfft show up just now, it was uploaded a month ago
[18:20] <juliank> britney doing britney things
[18:21] <juliank> ahasenack: So what happens I believe it may have given up earlier trying to determine a transition set, until we hinted it, and then it failed first on the other one, and now this one
[18:26] <juliank> glibc migrated so update_excuses will be bit nicer now :)
[18:28] <vorlon> juliank: pyvkfft removed
[18:30] <juliank> vorlon: thank you!
[20:00] <ahasenack> something started happening...
[20:00] <ahasenack> samba migrated
[20:00] <ahasenack> and I think it was waiting on python3-defaults
[20:04] <tjaalton> vorlon: hi, do you think the linux-meta glibc fail could be hinted to let it migrate? arighi tested it locally on a vm and the tests passed fine. the MR is at https://code.launchpad.net/~p-pisati/britney/+git/hints-ubuntu/+merge/461043
[20:04] <tsimonq2> Looking forward to https://ubuntu-archive-team.ubuntu.com/proposed-migration/log/noble/2024-02-23/19:28:36.log updating which will show exactly what packages ;)
[20:05] <tsimonq2> yup Python migrated \o/
[20:06] <tsimonq2> And so did Qt \o/
[20:06] <ahasenack> @5pm on Friday
[20:06] <tsimonq2> Sounds like it'll be a productive weekend ;D
[20:06] <tsimonq2> (For me, at least)
=== mfo is now known as Guest4913
=== mfo_ is now known as mfo
[20:23] <athos> \o/
[20:45] <athos> Hi ubuntu-release! Now that could anyone take a look at LP: #2054843 and add a few hints so we can complete the PHP transition as we did last time?
[20:45] -ubottu:#ubuntu-release- Launchpad bug 2054843 in php-defaults (Ubuntu) "Please force php-defaults 93ubuntu2 in nn" [Undecided, New] https://launchpad.net/bugs/2054843
[20:48] <bdmurray> I'm gonna re-enable autosync if python migrated
[20:48] <bdmurray> vorlon: ^^
[20:51] <athos> s/Not that/Now that python migrated/
[21:04] <juliank> bdmurray: Oh autosync got disabled again?
[21:05] <juliank> I thought it was on
[21:06] <juliank> I'm so happy we managed to migrate python :D
[21:06] <juliank> PHP too sounds awesome 😎 👍
[21:10] <vorlon> bdmurray: yes please
[21:13] <vorlon> athos: hints added
[21:16] <athos> Thanks!
[21:18] <bdmurray> juliank: I disabled it 21 or so hours ago per a discussion with d_oko
[21:18] <juliank> Ah
[21:19] <vorlon> tjaalton: hi, so I don't understand why a new linux-meta is causing the glibc autopkgtest to fail; it doesn't look like a flaky test, it seems to consistently fail with new linux-meta and pass with new versions of other userspace packages.  So I'm reluctant to hint this
[21:24] <vorlon> tjaalton: something changed debug/tst-fortify-syslog result from 'unsupported' to 'fail'; same for elf/tst-dlopen-self-container, elf/tst-dlopen-tlsmodid-container, elf/tst-glibc-hwcaps-2-cache, elf/tst-glibc-hwcaps-cache, elf/tst-glibc-hwcaps-prepend-cache...
[21:25] <vorlon> schopin: ^^ glibc autopkgtest regressing with new linux
[21:25] <vorlon> 3052s error: test-container.c:1136: could not create a private mount namespace
[21:26] <sarnold> vorlon: wild-guess time! https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2046844
[21:26] -ubottu:#ubuntu-release- Launchpad bug 2046844 in angelfish (Ubuntu) "AppArmor user namespace creation restrictions cause many applications to crash with SIGTRAP" [Critical, In Progress]
[21:29] <vorlon> sarnold: how would that regress as a result of a new linux, vs a new apparmor package?
[21:30] <sarnold> vorlon: because the "you don't get capabilities inside a usernamespace without an apparmor policy" is enforced by the kernel
[21:30] <vorlon> ok but I thought this had all landed already in the release pocket, to great consternation?
[21:31] <sarnold> good point :(
[22:15] <LocutusOfBorg> hello ubuntu-archive please kick out liquidsoap if possible, ftbfs and blocking ocaml migration (debian rc bug is open 1064128)
[22:15] <LocutusOfBorg> vorlon, ^^
[22:15] <LocutusOfBorg> pleeeease?
[22:19] <LocutusOfBorg> also flask can migrate if we kick out fava and searx from noble... both rc buggy
[22:19] <tjaalton> vorlon: hmm ok.. I don't see apparmor being installed on the test instance though?
[22:21] <tjaalton> unless it's already on the base image
=== mfo_ is now known as mfo
[22:45] <vorlon> tjaalton: I would definitely expect apparmor to be present on the base image
[22:47] <vorlon> LocutusOfBorg: liquidsoap removed
[22:52] <jbicha> are you able to restore ceph 18.2.0-0ubuntu7 and its binaries to noble-proposed to save LP build time?
[22:53] <bdmurray> I believe the riscv build was cancelled
[22:53] <jbicha> yes, but can you restore the other architecture builds?
[22:55] <vorlon> technically anyone with upload rights can do a self-binary-copy back to -proposed
[22:56] <vorlon> anyway yes we should restore it
[22:57] <vorlon> riscv64 build wasn't cancelled, but failed because it was "for superseded source" when it finished
[22:57] <vorlon> I've done the copy-back and also retried the riscv64 build
[23:07] <bdmurray> vorlon: what would the command for a "self-binary-copy" look like?
[23:07] <vorlon> $ copy-package -b --force-same-destination --auto-approve -y -s noble-proposed --to-suite noble-proposed ceph -e 18.2.0-0ubuntu7