[00:20] <arraybolt3> whew, fixed one
[00:29] -queuebot:#ubuntu-release- Unapproved: unity-lens-applications (noble-proposed/universe) [7.1.0+16.10.20160927-0ubuntu6 => 7.1.0+16.10.20160927-0ubuntu7] (no packageset)
[00:45] -queuebot:#ubuntu-release- Unapproved: unity-lens-files (noble-proposed/universe) [7.1.0+17.10.20170605-0ubuntu3 => 7.1.0+17.10.20170605-0ubuntu4] (no packageset)
[00:58] -queuebot:#ubuntu-release- Unapproved: unity-lens-music (noble-proposed/universe) [6.9.1+16.04-0ubuntu4 => 6.9.1+16.04-0ubuntu5] (no packageset)
[01:15] <cjwatson> vorlon: I'm curious what the stack trace looks like, even though it's not my problem :)
[01:19] -queuebot:#ubuntu-release- Unapproved: unity-scope-home (noble-proposed/universe) [6.8.2+19.04.20190412-0ubuntu3 => 6.8.2+19.04.20190412-0ubuntu4] (no packageset)
[01:19] <arraybolt3> to whom it may concern (vorlon possibly?) All problematic Unity packages in but 2051343 now have fixes in the queue.
[01:19] <arraybolt3> *bug 2051343
[01:19] -ubottu:#ubuntu-release- Bug 2051343 in unity-scope-home (Ubuntu) "unity: FTBFS in Noble" [Undecided, In Progress] https://launchpad.net/bugs/2051343
[01:57] -queuebot:#ubuntu-release- Unapproved: arctica-greeter (noble-proposed/universe) [0.99.5.0-2build3 => 0.99.5.0-3ubuntu1] (ubuntu-mate)
[01:58] <Wimpy> Please accept arctica-greeter/0.99.5.0-3ubuntu1
=== chris14_ is now known as chris14
[03:00] -queuebot:#ubuntu-release- New binary: nvidia-imex-550 [arm64] (noble-proposed/none) [550.54.15-0ubuntu0] (no packageset)
[03:01] -queuebot:#ubuntu-release- New binary: nvidia-imex-550 [amd64] (noble-proposed/none) [550.54.15-0ubuntu0] (no packageset)
[03:03] -queuebot:#ubuntu-release- New binary: nvidia-graphics-drivers-550-server [amd64] (noble-proposed/multiverse) [550.54.15-0ubuntu0] (i386-whitelist)
[03:04] -queuebot:#ubuntu-release- New binary: nvidia-graphics-drivers-550-server [arm64] (noble-proposed/multiverse) [550.54.15-0ubuntu0] (i386-whitelist)
[03:09] <vorlon> cjwatson: turns out the PS5 migration lost track of some config to stop the stuck-job-killer from thinking the publisher was a stuck job :/
[03:10] <vorlon> cjwatson: (which also explains why the stacktrace itself made no sense)
[03:10] <vorlon> anyway we've got that handled and the publisher is running again hopefully without rude interruption
[03:11] <vorlon> arraybolt3: cheers, I will have a look through the queue a bit later this evening
[03:11] <arraybolt3> thank you!
[03:11] <vorlon> cjwatson: (storm.exceptions.DisconnectionError: terminating connection due to administrator command
[03:12] <vorlon> )
[04:05] <wgrant> cjwatson: I ended up having to apply ImmutablePgJSON to BPPH/SPPH._channel, as your workaround to manually flush in dominator was too slow.
[04:15] -queuebot:#ubuntu-release- Unapproved: accepted liferea [source] (noble-proposed) [1.15.4-1ubuntu1]
[04:16] -queuebot:#ubuntu-release- Unapproved: accepted software-properties [source] (noble-proposed) [0.99.46]
[04:16] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-release-upgrader [source] (noble-proposed) [1:24.04.10]
[04:18] -queuebot:#ubuntu-release- Unapproved: accepted ubuntu-mate-settings [source] (noble-proposed) [24.04.3]
[04:24] -queuebot:#ubuntu-release- Unapproved: accepted interimap [source] (noble-proposed) [0.5.7-2ubuntu2]
[04:25] -queuebot:#ubuntu-release- Unapproved: accepted c2esp [source] (noble-proposed) [27-11ubuntu4]
[04:25] -queuebot:#ubuntu-release- Unapproved: accepted unity-lens-applications [source] (noble-proposed) [7.1.0+16.10.20160927-0ubuntu7]
[04:26] -queuebot:#ubuntu-release- Unapproved: accepted unity-lens-files [source] (noble-proposed) [7.1.0+17.10.20170605-0ubuntu4]
[04:26] -queuebot:#ubuntu-release- Unapproved: accepted unity-scope-home [source] (noble-proposed) [6.8.2+19.04.20190412-0ubuntu4]
[04:26] -queuebot:#ubuntu-release- Unapproved: accepted unity-lens-music [source] (noble-proposed) [6.9.1+16.04-0ubuntu5]
[04:31] -queuebot:#ubuntu-release- Unapproved: accepted arctica-greeter [source] (noble-proposed) [0.99.5.0-3ubuntu1]
[04:37] <tsimonq2> Please accept calamares-settings-ubuntu, it's tested but it depends on the latest version of snapd... will be depwait until Monday, or whenever they upload it.
[04:37] -queuebot:#ubuntu-release- Unapproved: calamares-settings-ubuntu (noble-proposed/universe) [1:24.04.20 => 1:24.04.21] (lubuntu, ubuntustudio)
[04:37] <tsimonq2> that ^
[04:46] -queuebot:#ubuntu-release- Unapproved: accepted gopacket [sync] (noble-proposed) [1.1.19-6]
[05:15] <vorlon> tsimonq2: the only acceptance criterion right now is "doesn't clobber something in -proposed that we're actively trying to migrate in the current publisher run", this isn't beta freeze and no further explanation needed
[05:15] <vorlon> and happily we seem to be getting close to being able to unfreeze
[05:15] -queuebot:#ubuntu-release- Unapproved: accepted calamares-settings-ubuntu [source] (noble-proposed) [1:24.04.21]
[05:53] <vorlon> the archive publisher has succeeded
[05:53] <vorlon> and noble is unfrozen
[06:01] <Bashing-om> < vorlon> the archive publisher has succeeded >> and noble is unfrozen \o/.
[06:07] <vorlon> libmtdev1t64 exists. qt is installable.  libext2fs2 is priority: optional.  Kicking off image rebuilds.
[07:19] <fossfreedom_> ubuntu-archive during a local sbuild of budgie-desktop it is failing to install v5.6 of xz-utils ... looking at the publishing history this was deleted.  Is the archive in the middle of cleaning up forced package removals? https://launchpad.net/ubuntu/+source/xz-utils/+publishinghistory
[07:49] <juliank> Ah good current britney takes ages because it schedules glibc test runs
[07:49] <juliank> Retried glibc/armhf build
[07:52] <juliank> schopin: glibc/armhf build started now, dependencies installed succesfully
=== guiverc2 is now known as guiverc
[07:56] <doko> arraybolt3: please also look at unity-api, and probably re-enable gtest
[07:56] <juliank> And that means we're back from 20 min britney runs to 2 hour runs
[08:03] <juliank> ubuntu-desktop continues being removed, now apt wants to remove gstreamer1.0-plugins-good for no reason
[08:05] <juliank> So `apt dist-upgrade ubuntu-desktop gstreamer1.0-plugins-good` works fine
[08:06] <juliank> Probably one could tweak `gstreamer1.0-plugins-good` by adding Conflicts on the pre-t64 libraries it uses to tilt the balance; I recorded a solver log and can investigate next week
[08:08] -queuebot:#ubuntu-release- New binary: freerdp2 [amd64] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:08] -queuebot:#ubuntu-release- New binary: freerdp2 [i386] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:10] -queuebot:#ubuntu-release- New binary: freerdp2 [s390x] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:15] -queuebot:#ubuntu-release- New binary: freerdp2 [arm64] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:15] -queuebot:#ubuntu-release- New binary: freerdp2 [ppc64el] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:16] -queuebot:#ubuntu-release- New binary: freerdp2 [armhf] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:17] <juliank> Please change Archive piece of topic back to Archive: Feature Freeze (doko, you op?)
=== juliank changed the topic of #ubuntu-release to: Released: 23.10 Mantic Minotaur, 22.04.4 Jammy Jellyfish | Archive: Feature Freeze | Highlight ubuntu-archive for archive admin help | We accept payment in cash, cheque or scotch | melius malum quod cognoscis | infinity, you are missed
[08:18] <juliank> Ooh I am op
[08:18] <juliank> I did not know
[08:25] <doko> since yesterday night I get "Temporary failure resolving 'archive.ubuntu.com'", fetching the InRelease files. same for de.archive. uk.archive seems to work
[08:26] <juliank> doko: possibly an issue with your DNS server
[08:26] <juliank> Otherwise you could check with IS
[08:27] <doko> no, works from another machine in the same network, with mantic. let me check another noble upgrade
[08:43] <doko> ahh, no. the network-manager was removed during the uprade ...
[08:45] <juliank> doko: ah yes that happened yesterday because it got published before libnetplan1 which it depends on
[08:50] <adrien> how can we know when everything is published?
[08:50] <juliank> adrien: should be fine now, glibc appeared that schopin uploaded yesterday
[08:52] <adrien> juliank: thanks!
[08:54] -queuebot:#ubuntu-release- New binary: freerdp2 [riscv64] (noble-proposed/main) [2.11.5+dfsg1-1] (desktop-core, i386-whitelist)
[08:57] <lotuspsychje> juliank: in some cases (like mine) ubuntu-desktop was purged in that same network-manager bug
[08:57] <lotuspsychje> should we reinstall ubuntu-desktop or wait updates?
[09:02]  * ricotz proceeds with the dist-upgrade
[09:05] -queuebot:#ubuntu-release- Unapproved: ovn (jammy-proposed/main) [22.03.3-0ubuntu0.22.04.3 => 22.03.7-0ubuntu0.22.04.1] (no packageset)
[09:05] -queuebot:#ubuntu-release- Unapproved: ovn (mantic-proposed/main) [23.09.0-1ubuntu1 => 23.09.3-0ubuntu0.23.10.1] (no packageset)
[09:05] <juliank> lotuspsychje: reinstall it
[09:05] <lotuspsychje> ok lets c
[09:09] <lotuspsychje> looking good juliank network-manager back and gnome-snapshot back gone
[09:10] <lotuspsychje> tnx
[09:31] <doko> last upgrade worked better. not upgradable: gnome-remote-desktop, gstreamer libcamera, yaru-theme-gnome-shell
[09:32] <lotuspsychje> those are phased yeah
[09:33] <doko> no, stuck in -proposed
[09:33] <juliank> My reports are updated now, dose-distcheck continued getting OOM killed
[09:33] <juliank> But I increased the swap size now
[09:34] <juliank> So kswapd and dose-distcheck are taking a CPU each
[09:34] <juliank> Need to move this to a Canonical-hosted machine with 4GB of RAM
[09:34] <juliank> I need 1 core and 4GB RAM :D
[09:34]  * juliank only has 2GB
[09:36] <juliank> Possibly it might need 6GB of RAM
[09:37] <juliank> I mean it uses 2GB RAM + 2GB swap file + some right now
[09:37] <juliank> Hence I bumped swap to 4GB
[09:37] <juliank> I don't think it ever wrote to the physical swap though, it just compressed pages with zstd
[09:38] <juliank> I switched zswap to lz4 now, hopefully this improves things a bit
[09:39] <juliank> And niced the cronjob to keep the web server (and this weechat) responsive
[12:26] <cjwatson> wgrant: ah, that was always a possibility I guess
[12:28] <cjwatson> vorlon: I think that was less "lost track of" and more "it turned out to be too difficult to express it after the DB connectivity changes we needed to make when charming the services", but I guess it was inevitable that the shortcut would bite us sooner or later
[13:05] <juliank> Queues are full, rejoice
[13:05] <juliank> armhf mass reference tests to be scheduled later
[13:06] <juliank> About 5000 tests for glibc and friends; and then about 11k reference tests
[13:06] <juliank> Dashboard for the migration references is https://autopkgtest.ubuntu.com/releases/noble, note that it doesn't have many yet
[13:07] <juliank> Next week I'll build a diff vs mantic health
[13:07] <juliank> Then we see inter-release regressions :)
[13:08] <juliank> N/A is good though it means it never failed so never needed retrying
[13:09] <juliank> Albeit now we retry everything that we tried in the past month since the time_t transition started
[13:09] <juliank> Like I said earlier, the script creating the list runs every 30 mins, so we can also delete items, let other stuff pass, and then wait for the next script run and it will produce a new list
[13:10] <juliank> Or some transitions could be moved to the normal queues
[13:10] <juliank> We should have added a migration-reference queue though :D
[13:12] <juliank> This also almost ticks the continuous baseline retesting goal we had years ago, in that you can easily change the script to expire tests older than X days, and then it just queues new migration-reference tests all the time
[13:12] <juliank> (at which point we really want a separate queue ofc)
[13:15] <jbicha> doko: I wish we had gotten mutter to migrate before your upload. There are installability problems for Ubuntu Desktop now :(
[13:38] <doko> jbicha: feel free to ignore the test failure again
[13:40] <jbicha> doko: is it possible to restore the 46.0-1ubuntu3 build since it almost passed its autopkgtests?
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [amd64] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [armhf] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [ppc64el] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [s390x] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted nvidia-graphics-drivers-550-server [arm64] (noble-proposed) [550.54.15-0ubuntu0]
[14:19] -queuebot:#ubuntu-release- New: accepted nvidia-imex-550 [arm64] (noble-proposed) [550.54.15-0ubuntu0]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [arm64] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [riscv64] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted nvidia-imex-550 [amd64] (noble-proposed) [550.54.15-0ubuntu0]
[14:19] -queuebot:#ubuntu-release- New: accepted libevent [i386] (noble-proposed) [2.1.12-stable-9ubuntu1]
[14:19] -queuebot:#ubuntu-release- New: accepted nvidia-graphics-drivers-550-server [amd64] (noble-proposed) [550.54.15-0ubuntu0]
[14:45] -queuebot:#ubuntu-release- Unapproved: apparmor (focal-proposed/main) [2.13.3-7ubuntu5.3 => 2.13.3-7ubuntu5.4] (core, i386-whitelist) (sync)
[14:45] -queuebot:#ubuntu-release- Unapproved: apparmor (jammy-proposed/main) [3.0.4-2ubuntu2.3 => 3.0.4-2ubuntu2.4] (core, i386-whitelist) (sync)
[14:48] <doko> jbicha: restored
[14:49] <schopin> doko: are you handling the NCR uploads for libevent or do you want me to?
[14:52] <doko> schopin: you can do if you want. just make sure that you can see it / download it before doing these
[14:52] <doko> do you want to do freerdp2 as well? then I'll accept that too
[14:54] <schopin> Sure.
[14:57] <doko> schopin: ohh, and
[14:57] <doko> /<<PKGBUILDDIR>>/gl
[14:57] <doko> ibc-2.39/build-tree/sh4-libc/signal/sigrelse.o
[14:57] <doko> {standard input}: Assembler messages:
[14:57] <doko> {standard input}:846: Error: symbol `__sigtimedwait64' is already defined
[14:57] <doko> c-t-b-p
[14:58] <schopin> ...
[14:59] <mdeslaur> FYI I have put some apparmor uploads into the focal and jammy -proposed upload queues if someone could please release them to -proposed
[14:59] <mdeslaur> (we want more testing before releasing this as a security update)
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [amd64] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [armhf] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [ppc64el] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [s390x] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [arm64] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [riscv64] (noble-proposed) [2.11.5+dfsg1-1]
[15:08] -queuebot:#ubuntu-release- New: accepted freerdp2 [i386] (noble-proposed) [2.11.5+dfsg1-1]
[15:47] <arraybolt3> doko: will do, thanks for the heads-up
[15:50] <schopin> It's weird, I can see the new libevent packages on ports.ubuntu.com but not on archive.ubuntu.com.
[15:54] <doko> same here. maybe wait a cycle
[16:02] <doko> jbicha: glib2.0 gained a dependency on sysprof (universe)
[16:02] <schopin> wtf?
[16:03] <doko> juliank: could you requeue all armhf tests triggered by glib2.0? there were run too early yesterday
[16:07] <jbicha> doko: yes, I updated Extra-Exclude for it. There's a lot of -dev packages that will need to be demoted to universe now (see the component-mismatches reports)
=== chris14_ is now known as chris14
[16:23] <juliank> doko: I can only queue them in the normal queue either via http. I plan to issue retries for failures there end of tomorrow or so
[16:23] <juliank> Such that there's some quiet in the archive
[16:24] <juliank> doko: but I guess could do that now
[16:24] <juliank> Did my retry-autopkgtest-regressions MP get merged?
[16:25] <vorlon> yes
[16:25] <juliank> Very good
[16:25] <vorlon> did I push it? double-checking
[16:25] <bdmurray> juliank: I'm happy to put things in the huge queue again.
[16:25] <vorlon> I did push it
[16:26] <juliank> bdmurray: If you want to do that, you can retry-autopkgtest-regressions --blocks=glib2.0 in huge and then queue the reference runs in https://people.canonical.com/~jak/mass-reference-tests.txt
[16:26] <vorlon> ok heads up to everyone, since it's now public: we had a compromized liblzma in the noble archive for a month. https://www.openwall.com/lists/oss-security/2024/03/29/4
[16:27] <juliank> bdmurray: but it's probably best to do glib2.0 in the normal queue
[16:27] <juliank> Otherwise it gets stuck behind glibc
[16:27] <juliank> So let me just queue those
[16:27] <vorlon> as soon as we were notified of the issue we removed it from the archive but we are still working through understanding if there is any impact on other packages in the archive that were built while this was present in the build environment
[16:27] <vorlon> this may impact beta timeline
[16:28] <vorlon> more communication forthcoming when we're able, but I wanted to let people know this ASAP
[16:28] <juliank> bdmurray: Queuing glib2.0 in the normal queue now, such that tests run concurrently with glibc from huge; but please then do the reference runs :)
[16:29] <juliank> I see a handful of new passes on i386 which is nice
[16:30] <juliank> bdmurray: Probably between release and 24.10 opening, could add a 'reference' queue to autopkgtest such that we can run reference tests in there it would be nice
[16:31] <juliank> Your team can enhance my script and run it from autopkgtest-web leader to get continous baseline retesting; just need to calculate the data in the NOT IN clause rather than using a fixed one :D
[16:32] <juliank> It's now trivial, I have hardcoded it to not run migration-references if we have run one since Mar 28 13:00 UTC
[16:32] <juliank> but if you calculate another day and say "1 month" ago, you get a rolling list of tests to run
[16:33] <juliank> doko: glib2.0 requeued
[16:34] <juliank> doko: And I think they are already all running which is surprising
[16:36] <juliank> Not sure what's going on there
[16:37] <juliank> Let me try to schedule again
[16:37] <juliank> New session cookie
[16:39] <juliank>  89s dpkg: error processing archive /var/cache/apt/archives/libglib2.0-0t64_2.80.0-5_armhf.deb (--unpack):
[16:39] <juliank>  89s  trying to overwrite '/usr/lib/arm-linux-gnueabihf/glib-2.0/gio-launch-desktop', which is also in package libglib2.0-0:armhf 2.79.2-1~ubuntu1
[16:39] <juliank> I think libglib2.0-0t64 is buggy?
[16:40] <juliank> OK maybe just cache-amqp is super slow now
[16:41] <doko> jbicha: ^^^
[16:42] <juliank> bdmurray: did you notice arm64 has 60-80% abnormal failure rate?
[16:42] <juliank> Like averaging it over 30 mins
[16:43] <juliank> Also sorry to say, my dose-distcheck reports continue crashing
[16:46] <doko> jbicha: all main -> universe component mismatches addressed
[16:46] <doko> vorlon: why is libjansson-dev:amd64 installed in the binutils (build) autopkg test?
[16:46] <juliank> Hopefully this is fixed now
[16:47] <vorlon> doko: I don't have to look at and answer this right now
[16:47] <vorlon> doko: in crisis mode wrt the abovementioned backdoor that was in noble
[16:47] <juliank> I'm disabling the workaround to use end of 27th proposed in my reports to fix packages disappearing between proposed and release pocket
[16:48] <bdmurray> juliank: I'll have a look.
[16:49] <jbicha> vorlon: can we sync the xz-utils revert from Debian?
[16:50] <cjwatson> AIUI this was only briefly in noble release, right, from yesterday?  but it was in noble release for a short while
[16:51] <vorlon> cjwatson: correct
[16:51] <vorlon> and it was in noble-proposed for a month
[16:51] <vorlon> and we are not currently confident that it didn't include other payload vectors besides sshd that could have propagated into other built binaries
[16:51] <cjwatson> rightly so
[16:54] <bdmurray> juliank: we were at capacity in bos03 and got our quota bumped a wee bit ago so that might help
[16:55] <vorlon> jbicha: do you mean bumping the version number?  We wouldn't normally do so, given that it was only in the release pocket briefly at a time when things were massively not-upgradeable; we can do a revert but currently I'm not viewing that as the priority
[16:55] <juliank> We're also doing an infra stresstest right now with all our non-stop testing for weeks :)
[16:57] <doko> schopin: libevent is now visible
[16:58] <doko> vorlon: it has 5.6.1+really5.4.5-1, so safe to sync
[16:59] <schopin> doko: erm, what? I saw it much earlier?
[16:59] <vorlon> doko: please feel free to go ahead
[17:00] <doko> synced
[17:00] <schopin> doko: damnit. I guess I pulled the trigger too soon.
[17:02] <schopin> Cancelling all builds, and reuploading.
[17:02] <doko> schopin: did you look at an armhf build?
[17:02] <schopin> Yes, even those haven't picked it up.
[17:02] <jbicha> juliank: I'm bumping glib's Breaks/Replaces in Debian and plan to sync it over later. I can ping you then to move the autopkgtests to the right queue
[17:02] <doko> you said you were able to see it on the porter archiive?
[17:03] <doko> jbicha: please can you fix it now in noble? or say what needs to be fixed?
[17:03] <juliank> jbicha: to move stuff you'll want to ping bdmurray :)
[17:03] <juliank> jbicha: I have no special privileges myself :(
[17:05] <schopin> doko: nevermind. I guess I looked at the only package for which the upload was rejected? (and so the build was an old one)
[17:05] <doko> schopin: hmm, don't see any emails on changes yet
[17:07] <doko> schopin: looks like at least apt-cacher-ng is ok on amd64 and armhf, both having t64 deps
[17:08] <schopin> assuming the builds are picked in FIFO order we should be good.
[17:08] <juliank> I assume glibc is good?
[17:08] <juliank> It only depends on itself, no?
[17:09] <schopin> juliank: good in what sense? It built fine, but now it's the usual -proposed dance.
[17:09] <doko> glibc doesn't have a dependency on libevent
[17:10] <schopin> oh yeah I definitely wouldn't have uploaded a rebuild of glibc without telling anyone :D
[17:11] <juliank> schopin: Good in whatever misbuilds your are discussing is not a concern
[17:11] <juliank> It did not build on armhf until this morning when I retried it :)
[17:11] <schopin> thanks for that
[17:11] <juliank> (it was in depwait)
[17:13] <schopin> the excuses section of glibc doesn't look too bad (yet)
[17:13] <doko> vorlon: should we copy the new xz-utils directly to the release pocket once it's built?
[17:13] <vorlon> doko: it should be fine to let it migrate on its own IMHO
[17:13] <doko> ok
[17:18] <doko> vorlon: for the priority mismatches, did you just follow the proposed changes on priority-mismatches.txt?
[17:18] <vorlon> doko: schopin, juliank: ok heads up here's the current plan.  Since we don't know we can trust any amd64 binaries built in noble-proposed for the past month, we are preparing to: 1) do archive surgery to upload pre-compromise amd64 binaries to -proposed for all affected packages; 2) delete all untrusted amd64 binaries from the release pocket; 3) issue no-change rebuilds of all affected packages
[17:18] <jbicha> doko: ok, manually synced early https://launchpad.net/ubuntu/+source/glib2.0/2.80.0-6
[17:18] <juliank> eek
[17:18] <vorlon> doko: yes but I haven't finished yet because there are some promotions on that list that are not straight-swaps of t64 vs not so I want to look at them more closely. Please leave this with me to follow through
[17:19] <juliank> So liblzma ran backdoors at build time and potentially they spread rto other binaries?
[17:19] <juliank> horrible
[17:19] <vorlon> doko, schopin, juliank: analysis of the payload is ongoing and we may decide that this was unnecessary, but in the event it IS necessary it's important that we start on it sooner rather than later
[17:19] <juliank> sure
[17:19] <vorlon> juliank: we don't *know* if it did and therefore we are taking the safe path
[17:19] <schopin> vorlon: I'm sorry, I have a hard EOD in 10mn, although I can conceivably come back later today if you need me.
[17:19] <schopin> well, tonight
[17:20] <vorlon> schopin: mostly I'm informing you so that you don't spend further time worrying about autopkgtests of the current upload which will be superseded one way or another
[17:20] <schopin> Oh, right. Thanks then:)
[17:20] <juliank> So, move beta a week later?
[17:20] <juliank> I fear 24.04 will become 24.05
[17:20] <vorlon> it would *also* help to finish flushing as much of the packages currently into -proposed to the release pocket as possible (any outstanding t64 transitions)
[17:21] <juliank> Well we can't trust any of them?
[17:21] <vorlon> juliank: move beta> this is a possible outcome, but no decision is made yet pending the payload analysis efforts
[17:22] <doko> indep is also built on amd64 :-/
[17:22] <vorlon> juliank: can't trust any of them> we know the payload is only injected into liblzma in RPM and deb builds on amd64; so we can publish those t64 packages to the release pocket and then remove the amd64 binaries there, leaving noble-proposed clear for upload of the previous version without compromising armhf bootstrap
[17:23] <vorlon> doko: yes but we can do static analysis of indep packages to make sure they haven't injected any binaries onto the library load path
[17:23] <vorlon> an indep package that adds a binary payload that isn't anywhere that our builds will execute it is, for these purposes, uninteresting
[17:23] <vorlon> afk now for a bit
[17:24] <juliank> What about i386
[17:24] <doko> xz-utils was removed 2024-03-29 07:10:17 CET from noble-proposed, did enter proposed 2024-02-29 06:42:25 CET
[17:25] <vorlon> juliank: the build-time check is for "x86_64" https://www.openwall.com/lists/oss-security/2024/03/29/4
[17:26] <doko> hmm, but we're running on x86_64 kernels?
[17:28] <vorlon> the build env uses the jammy kernel
[17:28] <cjwatson> I don't think that matters - $build there should be a config.guess/config.sub check I think
[17:28] <vorlon> oh, you mean wrt the build-time check
[17:28] <vorlon>     if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then
[17:28] <cjwatson> if it were x86_64 there then all i386 builds would be considered cross-compiles, which I don't believe is the case
[17:28] <cjwatson> though I haven't looked at the context
[17:28] <vorlon> ^ that's a configure gnu-type check yeah
[17:29] <doko> yes, System type:
[17:29] <doko> checking build system type... i686-pc-linux-gnu
[17:29] <doko> checking host system type... i686-pc-linux-gnu
[17:33] <juliank> I want to say this is a great example for how open source improves security, it only took about 1 month since committing the backdoor to it being detected and people starting to fix it
[17:34] <juliank> I see a blog post about how well our supply chain security works
[17:35] <doko> vorlon: do you want a particular order of uploads? if yes, then I would upload binutils now, followed by compilers and glibc
[17:35] <juliank> This is much better than when you go consult with proprietary software companies that are being coerced by goverment agencies to add backdoors in their binary deliveries to you :)
[17:35] <doko> then probably everything which is in the buildd chroots
[17:35] <doko> good that they are buildable now =)
[17:40] <juliank> Do you actually need to upload anything and not just copy the binaries from older versions?
[17:40] <juliank> Where do you have trusted pre-Feb-29th binaries from?
[17:41] <doko> a pity that liblzma-dev is not in the buildd chroot. there would something positive about not updating the buildd chroots...
[17:48] <juliank> So I have a different wild idea where we put the buildds on a snapshot of the release pocket of the 25th and then upload no-change uploads to proposed
[17:48] <juliank> Because we have that fancy snapshot service
[17:49] <juliank> But the coordination is annoying
[17:53] <arraybolt3> So... should I actually be doing any build and uploads right now? I was going to work on the unity-api package but now I'm a bit skittish :P
=== arraybolt3 is now known as arraybolt3-dange
=== arraybolt3-dange is now known as arraybolt3-cmp
[18:17] <mkukri> did anyone analyze what this actually does? does it just backdoor sshd? or is  there any exposure of e.g. key material stored on machines where the malicious code might have been executed
[18:19] <juliank> WIP
[18:19] <bdmurray> correct
[18:21] <doko> fyi, noble-changes ML is currently not working. so there might be communication lag about current uploads (maybe other MLs as well)
[18:41] <vorlon> doko: please open an RT about the list not working
[18:42] <vorlon> doko order of uploads> please hold, sil2100 is still working on the full list of packages and nothing can be uploaded until we've removed all the tainted amd64 binaries
[18:42] <vorlon> doko: I'm fine to special-case binutils/gcc/glibc; after that my plan was to upload all packages shipping runtime libraries, then all other packages
[18:42] <vorlon> arraybolt3: it's fine to do uploads
[18:43] <vorlon> doko: anyway, maximum parallelism is to our advantage here
[18:44] <arraybolt3> awesome
[18:45] <arraybolt3> mkukri: we don't fully know yet, looks like an sshd backdoor but could be so much worse. I'm treating it as a total system compromise just for good measure, which means probably several hours of my day consumed changing passwords and auth info
[18:46] <mkukri> i thankfully didnt have a noble machine with my private keys exposed
[18:46] <mkukri> but probably had vms and chroots and maybe my arm build machine, so still thinking of it like a nasty network compromise
[18:48] <arraybolt3> yeah, I had Noble VMs with total access to my GPG and important SSH keys, so... having a mild freak-out here :P
[18:48] <arraybolt3> probably will change my builder strategy from now on so that the Noble VMs only build packages and I sign them on my physical machine.
[18:53] <waveform> mkukri, so far looks like amd64 only -- arm boxes may not be affected
[18:54] <mkukri> ah yes, you are right, the injected code is a precompiled amd64 object file
[18:54] <mkukri> i still have amd64 chroots with the affected liblzma.so sadly tho
[18:59] <doko> vorlon: no ticket yet, IS is actively working on it. the MM
[19:00] <doko> vorlon: please consider having a buildd chroot milestone. follow-up builds will benefit after a buildd chroot update
[19:06] <juliank> arraybolt3: time for a smartcard
[19:07] <juliank> I use the YubiKey with tap to sign
[19:07] <arraybolt3> juliank: heh, not a bad idea, though I don't have a true Yubikey, I only have the FIDO security keys.
[19:09] <juliank> arraybolt3: you could get one :)
[19:10] <arraybolt3> ERR:TOO_EXPENSIVE_FOR_ME_RIGHT_NOW
[19:10] <juliank> So I have no critical secrets in a sense, everything important is 2 factor secured
[19:11] <juliank> The 2nd factor is on arm64
[19:11] <mkukri> also the author of the changes seems extremely concerning with no other online presence outside github, and nonsensical PRs into many projects going back years.... this might go beyond just this one thing
[19:11] <juliank> I'm very annoyed 😠
[19:12] <arraybolt3> everyone here is
[19:15] <juliank> Ah yes the primary key is on the machine, in another user home directory, but it hasn't been unlocked in months
[19:16] <juliank> I guess I gotta load an old live image, grab the key, and then move it to three openpgp smartcards
[19:16] <doko> @vorlon please consider hinting xz-utils not waiting for autopkg tests. queues are busy
[19:16] <mkukri> im probably fine, but i think it's a good opportunity to rotate my 2048 gpg key to a 4096 bit one and put it in a smartcard
[19:17] <doko> mkukri: update in -proposed
[19:18] <mkukri> how far back do we trust XZ and liblzma? it kinda feels like this person has been building the infrutructure for this for years with the ifunc stuff
[19:18] <juliank> We can make the queues not busy
[19:39] <vorlon> doko: queues should be dumped to make them un-busy
[19:43] <vorlon> doko: I think a further modification to my earlier suggestion about build ordering: we should upload binutils, gcc, glibc, then all libs seeded on flavors, then all other packages seeded on flavors, THEN all other libs, then all other packages
[19:47] <doko> juliank, bdmurray: queues ^^^
[19:48] <vorlon> I can take care of it
[19:48] <vorlon> (doing so)
[19:48] <vorlon> (and Julian can't)
[19:49] <vorlon> ... somewhat delayed by this having fallen out of the command history, though
[19:51] <vorlon> doko: eh only the huge queues are loaded though?
[19:51] <vorlon> I'll check on arm
[19:51] <vorlon> oh I see, xz-utils tests are also in the huge queue
[19:51] <vorlon> but they can just be requeued to the tiny queue
[19:51] <vorlon> doing now
[19:52] <vorlon> not going to bother flushing duplicates from the huge queue because the huge queue is likely to be dumped en masse later
[20:01] <vorlon> doko: I've done the priority swaps for the t64 libraries; after a regeneration of https://ubuntu-archive-team.ubuntu.com/priority-mismatches.html what remains is unrelated to t64 so should be given closer scrutiny to make sure we aren't accidentally and unintentionally bloating the base system
[20:07] <doko> vorlon: uploaded binutils and gcc-13 with a b-d. need to delay gcc-14. again, please consider to have an extra step to build everything to get the buildd chroors updated. this will pay off later
[20:18] <vorlon> doko: b-d is insufficient
[20:19] <vorlon> doko: but I've killed the binutils build on amd64 and we can retry it once the archive is rejiggered
[20:19] <vorlon> doko: the problem is your b-d is currently satisfied so we'd be rebuilding binutils with the untrusted gcc + binutils currently in the archive
[20:27] <vorlon> does anyone here want to look into why debootstrap is currently failing?
[20:27] <vorlon> https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/noble/lubuntu/+build/597770 points the finger at libtirpc3t64 but there's no output here explaining why
[20:28] <vorlon> libtirpc3 is *not* in the log
[21:09] <wxl> vorlon: right at the top, libtirpc3 is being removed and libtirpc3t64 is being installed
[21:09] <vorlon> wxl: that's the build environment and not the chroot being bootstrapped so not relevant
[21:09] <vorlon> someone needs to reproduce this and see what the log says since we don't output that log in livecd-rootfs
[21:09] <vorlon> (someone could also fix that in livecd-rootfs)
=== dbungert1 is now known as dbungert
[22:17] <ItzSwirlz> this is probably the worst release cycle ever in the history of ubuntu
[22:20] <arraybolt3> meh, the *cycle* has been awesome, some of the unexpected bumps on the other hand have been pretty awful.
[22:21] <arraybolt3> but even most of those weren't so bad. Having an upstream dev try and hack you on the other hand...
[22:23] <cjwatson> I mean it hasn't been delayed yet
[22:23] <cjwatson> so it's no 6.06 :)
[22:25] <bdmurray> 🙈
[22:27] <arraybolt3> meh, if we're delayed, no big deal, I think "well upstream literally tried to hack us" is a good excuse for a late release.
[22:28] <arraybolt3> (tried and in some ways succeeded)
[22:30] <vorlon> so here's what's happening now
[22:30] <vorlon> we're still finalizing the list of packages that would need rebuilt, but right now we have 7207 packages whose current amd64 binaries were built after xz-utils publication
[22:31] <ItzSwirlz> ow
[22:31] <vorlon> I'm in the process of copying old, good versions of these packages (one by one) to noble-updates
[22:31] <ItzSwirlz> vorlon, how can i help
[22:31] <vorlon> as they publish, I will be removing the *current* amd64 binaries from noble+noble-proposed
[22:31] <vorlon> that is slowest, because launchpad doesn't like binary removals
[22:32] <vorlon> and only once that's all done we will start uploading no-change rebuilds
[22:34] <bdmurray> ItzSwirlz: You could see if ubuntucinnamon-meta needs updating / uploading.
[22:35] <ItzSwirlz> i dont think it has xz-utils in it in noble. I think it did have in older versions but not in noble
[22:35] <ItzSwirlz> Also I don't have upload perms
[22:35] <bdmurray> ItzSwirlz: I meant for the Beta
[22:35] <ItzSwirlz> Build has been broken for about a month
[22:52] -queuebot:#ubuntu-release- Unapproved: ubuntucinnamon-meta (noble-proposed/universe) [24.04.3 => 24.04.3.1] (no packageset)
[23:01] <bdmurray> vorlon: I'm gonna self approve these meta package uploads okay?
[23:02] -queuebot:#ubuntu-release- Unapproved: edubuntu-meta (noble-proposed/universe) [24.04.12 => 24.04.12.1] (no packageset)
[23:02] <vorlon> bdmurray: I think it's best to not build any packages in noble right now
[23:02]  * Eickmeyer jumps
[23:02] <vorlon> bdmurray: the chances of something contaminating a metapackage as a payload is slim, but it's still built using untrusted binaries
[23:03] <bdmurray> vorlon: ack
[23:14] -queuebot:#ubuntu-release- Unapproved: ubuntu-meta (noble-proposed/main) [1.537 => 1.538] (core)
[23:22] -queuebot:#ubuntu-release- Unapproved: ubuntukylin-meta (noble-proposed/universe) [0.56 => 0.57] (ubuntukylin)
[23:31] -queuebot:#ubuntu-release- Unapproved: ubuntu-mate-meta (noble-proposed/universe) [1.294 => 1.294.1] (ubuntu-mate)
[23:35] -queuebot:#ubuntu-release- Unapproved: ubuntustudio-meta (noble-proposed/universe) [24.04.17 => 24.04.18] (ubuntustudio)
[23:42] -queuebot:#ubuntu-release- Unapproved: ubuntu-unity-meta (noble-proposed/universe) [0.16 => 0.17] (no packageset)
=== Guest55 is now known as arraybolt3
[23:45] -queuebot:#ubuntu-release- Unapproved: xubuntu-meta (noble-proposed/universe) [2.260 => 2.261] (xubuntu)
[23:48] <arraybolt3> I know I'm in the wrong channel, but while I'm in hyper-paranoia mode, can someone verify for me that this is the authentic Ubuntu CD image signing key? 843938DF228D22F7B3742BC0D94AA3F0EFE21092
[23:51] <cjwatson> yes
[23:51] <cjwatson> if you look on the keyserver it's signed by e.g. me and vorlon
[23:51] <arraybolt3> thank you kindly
[23:52] <arraybolt3> aha, nifty, *files that trick in the back of mind for next time*
[23:52] <cjwatson> (and a bunch of other people who have no particular justification for signing it ...)
[23:53] <cjwatson> we generated that key at UDS-Q in Oakland IIRC