[01:52] vorlon: i sent email too but i think the glibc autopkgtest failures are an upstream testsuite bug: https://sourceware.org/bugzilla/show_bug.cgi?id=25652 [01:52] sourceware.org bug 25652 in build "testroot.pristine misses /lib64/ld-linux-x86-64.so.2" [Normal,Unconfirmed] [02:11] (i think this one gets filed under "shell is bad") [02:11] so bad [02:18] i assume https://sourceware.org/git/?p=glibc.git;a=commit;h=7db1fe38de21831d53ceab9ae83493d8d1aec601 is to blame without any particular evidence === pieq_ is now known as pieq [03:50] mwhudson: thanks. I don't want to migrate glibc with failing autopkgtests however, and it needs another upload still anyway to get the predepends on libcrypt1; would you want to take a stab at fixing the missing ld, or should I just XFAIL these for now? [03:51] (fwiw I tried to reproduce the failures and haven't gotten as far as managing to get the invocation correct so that I can even reproduce the failure outside of a full 'make check') [04:20] doko, xnox: soooo libxcrypt doesn't build multilib variants [04:28] doko: hmm I'm not sure what to make of the i386 dbus-glib autopkgtest regression, but it looks somehow related to glibc despite the log showing no packages being pulled from -proposed https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/i386/d/dbus-glib/20200310_223456_7d94d@/log.gz [06:21] doko, mwhudson: I have a glibc -0ubunut4 staged here and ready for upload, but I want to let the autopkgtests for -0ubuntu3 complete on all archs so we get a clear picture before I upload (provided -0ubuntu3 is good wrt autopkgtests, I will skip all the tests with 0ubuntu4 aside from glibc itself). there are still some autopkgtest regressions listed on [06:21] https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses_by_team.html#foundations-bugs that I don't have a handle on - clearcut, gnudatalanguage, libseccomp, rtags [06:44] rbasak: hey, do you think you could have a look at the SRU of snapd in unapproved? [06:46] vorlon: dbus-glib is becase cross-toolchain-base already migrated, having mismatched glibc versions for cross/native. did you retrigger with glibc from -proposed? [06:47] I've been working on the regressions triggered by glibc yesterday. will continue today [06:48] vorlon, xnox: multilib, are you worried that we don't drop the glibc crypt code for the multilib variants? [06:52] given back dbus-glib with the glibc trigger [07:05] vorlon: please ignore the gcc-arm-none-eabi/i386 test failure [07:06] and the force-reset-test binutils-arm-none-eabi/13 hint can be removed, fixed [08:22] could someone remove the glslang-dev/-tools binaries from s390x, it's not supposed to build on big-endian anymore as spirv is broken there [08:22] also spirv-tools should be removed [08:22] from s390x.. [08:29] tjaalton, [08:29] $ reverse-depends -b glslang-dev [08:29] Reverse-Build-Depends [08:29] ===================== [08:29] * libplacebo [08:29] * renderdoc [08:30] * vulkan-validationlayers [08:30] tjaalton, the first and third build on s390x [08:31] tjaalton, also https://paste.ubuntu.com/p/62n3D4vPDQ/ for tools [08:31] seb128: remove those too [08:31] I'll fix mesa [08:32] vulkan-* are in proposed [08:32] meh [08:32] or just ignore that it's broken are restore the build.. dunno [08:32] I would rather do that... [08:34] in any case for removal I think a bug would be best since it's not trivial [08:34] right [09:13] -queuebot:#ubuntu-release- Unapproved: xf86-input-wacom (bionic-proposed/main) [1:0.36.1-0ubuntu1 => 1:0.36.1-0ubuntu1.1] (desktop-core, xorg) [09:34] -queuebot:#ubuntu-release- Unapproved: zfs-linux (eoan-proposed/main) [0.8.1-1ubuntu14.3 => 0.8.1-1ubuntu14.4] (core) [09:53] -queuebot:#ubuntu-release- Unapproved: rejected grub2 [source] (xenial-proposed) [2.02~beta2-36ubuntu3.24] [10:00] -queuebot:#ubuntu-release- Unapproved: rejected lz4 [source] (xenial-proposed) [0.0~r131-2ubuntu2.1] [10:01] -queuebot:#ubuntu-release- Unapproved: rejected lz4 [source] (bionic-proposed) [0.0~r131-2ubuntu3.1] [10:24] -queuebot:#ubuntu-release- Unapproved: accepted postgresql-9.5 [source] (xenial-proposed) [9.5.21-0ubuntu0.16.04.1] [10:29] -queuebot:#ubuntu-release- Unapproved: accepted quassel [source] (xenial-proposed) [0.12.2-0ubuntu1.16.04.1] [11:24] apw, can I please have coq removed in some architectures? it follows a debian removal bug. it might come back, but maybe not in time for focal https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953408 [11:24] Debian bug 953408 in ftp.debian.org "RM: coq [armel armhf i386 mipsel mips64el s390x] -- ROM; FTBFS (mostly on 32bit architectures, or where ocaml has only a bytecode compiler)" [Normal,Open] [11:25] the list and reverse-deps is on that bug, basically coq armhf i386 s390x, with also prooftree why3 and libaac-tactics-ocaml [11:26] -queuebot:#ubuntu-release- New binary: linux-signed-oracle [amd64] (xenial-proposed/main) [4.15.0-1035.38~16.04.1] (kernel) [11:40] -queuebot:#ubuntu-release- New: accepted linux-signed-oracle [amd64] (xenial-proposed) [4.15.0-1035.38~16.04.1] [11:53] -queuebot:#ubuntu-release- Unapproved: drbd-utils (bionic-proposed/main) [8.9.10-2 => 8.9.10-2ubuntu0.1] (ubuntu-server) [13:36] -queuebot:#ubuntu-release- Unapproved: accepted libvirt [source] (xenial-proposed) [1.3.1-1ubuntu10.30] [13:52] -queuebot:#ubuntu-release- Unapproved: accepted sg3-utils [source] (bionic-proposed) [1.42-2ubuntu1.18.04.2] [13:53] -queuebot:#ubuntu-release- Unapproved: accepted sg3-utils [source] (eoan-proposed) [1.44-1ubuntu1.1] [14:48] -queuebot:#ubuntu-release- Unapproved: adobe-flashplugin (bionic-proposed/partner) [1:20200211.1-0ubuntu0.18.04.1 => 1:20200311.1-0ubuntu0.18.04.1] (no packageset) [14:48] -queuebot:#ubuntu-release- Unapproved: adobe-flashplugin (xenial-proposed/partner) [1:20200211.1-0ubuntu0.16.04.1 => 1:20200311.1-0ubuntu0.16.04.1] (no packageset) [14:48] -queuebot:#ubuntu-release- Unapproved: adobe-flashplugin (eoan-proposed/partner) [1:20200211.1-0ubuntu0.19.10.1 => 1:20200311.1-0ubuntu0.19.10.1] (no packageset) [15:42] doko: cross-toolchain-base vs dbus-glib> ah thanks, I hadn't tried triggering against -proposed because I didn't understand why it would help [15:42] doko: gcc-arm-none-eabi, I'll have a look [15:44] doko: gcc-arm-none-eabi/i386 hinted, thanks [16:06] vorlon: before you upload glibc, I'm testing a fix for 1861353 [16:06] doko: ok [16:07] -queuebot:#ubuntu-release- Packageset: Added wpebackend-fdo to i386-whitelist in focal [16:09] doko: any ideas on the failing autopkgtests that I flagged last night? There's a collection of failures on armhf which concern me that we may have arch-specific regressions in glibc [16:11] no, I gave back now gnudatalanguage with empty test queues. let see if that helps [16:11] yes, but the only tests ignored for arm were the soft-float tests [16:22] -queuebot:#ubuntu-release- Packageset: Added mozjs68 to i386-whitelist in focal [16:35] vorlon, xnox: LP: #1861353 has a patch [16:35] Launchpad bug 1861353 in glibc (Ubuntu Focal) "libc6-dev:amd64 is not co-installable with libc6-dev:s390x" [Undecided,New] https://launchpad.net/bugs/1861353 [16:40] doko: looks good, i'm not sure if we want current glibc to migrate or upload again?! [16:42] steve wants to upload afaik [16:43] xnox: upload again, getting the pre-depends, the autopkgtest fix, and the multiarch fix [16:44] xnox: but I am still looking to bottom out on the remaining revdep autopkgtest regressions [16:44] xnox: and when I upload -0ubuntu4 I will flush the irrelevant re-tests [16:46] rtags is weird, I can reproduce it but when I run autopkgtest with -s and then re-run the test from a shell, it succeeds [16:54] vorlon: i don't see cd build logs https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/focal/?C=M;O=D but forexample there were images built on the 11th of march [16:54] any idea why build-logs are not getting updated? [16:56] no idea, but looking [16:57] xnox: because someone landed a move from python2 to python3 and mirror-image-build-logs is not python3-clean [16:58] ;) [16:58] (and not covered by unit tests) [16:59] xnox: doing a one-off run under python2 now [17:00] -queuebot:#ubuntu-release- Unapproved: grub2 (focal-proposed/main) [2.04-1ubuntu22 => 2.04-1ubuntu22] (core) [17:01] xnox: and fixed [17:04] universal_newlines=True and str would be better style, but OK :) [17:04] (IMO anyway, not that it matters much) [17:17] ok I'm going to turf the rtags/amd64 autopkgtest regression, given that I can't reproduce it in an interactive shell I don't believe it points to an actual regression in glibc but is a buggy test of some sort [17:24] -queuebot:#ubuntu-release- Unapproved: grub2 (focal-proposed/main) [2.04-1ubuntu22 => 2.04-1ubuntu22] (core) [18:00] -queuebot:#ubuntu-release- Unapproved: sosreport (xenial-proposed/main) [3.6-1ubuntu0.16.04.4 => 3.9-1ubuntu0.16.04.1] (ubuntu-desktop, ubuntu-server) [19:32] glibc uploaded [19:32] (a half hour ago) [19:32] if we can figure out what's up with those remaining arm failures and let this in, great [19:32] if not and we have to reupload, not much time wasted [19:32] (after jettisoning the autopkgtest queues) [19:36] if it's build, please retrigger libseccomp with the one from -proposed [19:36] built even [19:38] vorlon: the arm64 autopkg tests triggered by perl don't seem to make any progress [19:45] doko: why do you expect a different result for a libseccomp retrigger? [19:47] doko: perl/arm64> I had dumped all of those tests from the queue after analyzing the root cause of the libc preinst failure. it seems LocutusOfBorg has re-queued them and I see progress being made now. [19:54] I sent a message about the autopkgtest testbed earlier in #ubuntu-devel but got no answer, let me copy and paste it here to see if someone can help me [19:54] hi, I am investigating the gem2deb/1.0.5 autopkgtest failure on armhf and it is complaining (reprotest actually) about the kernel version: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/armhf/g/gem2deb/20200311_020802_c6e4e@/log.gz [19:54] if you check the log you will find: autopkgtest [01:38:52]: testbed running kernel: Linux 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:21:09 UTC 2019 [19:55] is linux 4.15.0 expected in an ubuntu focal testbed? [19:56] is glibc/yt/arm64 ooming? [19:57] would it make sense for autopkgtest artifacts to include syslog [19:57] kanashiro: yes, because the armhf autopkgtest runners are containers, running on an LTS host [19:58] complaining about the kernel version: that sounds like a wrong test [19:58] no userspace packages in focal should fail with the bionic kernel [19:58] so a wrong test or a right test exposing wrong code [20:02] mwhudson: so what's surprising to me is that yt should oom consistently on arm64 but not on other archs, given that we allocate the same size VM for all [20:02] but we could try adding it to the 'huge' config and see if that fixes it? [20:02] vorlon: seems worth a shot [20:06] vorlon: er i think yt _does_ oom on other arches [20:06] is it badtested? [20:07] eg https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/y/yt/20200310_112622_9fadd@/log.gz [20:08] ubuntu-release:force-reset-test yt/3.5.1-3/amd64 [20:08] that version has passed though, so i don't know why it's marked "always failed" in excuses [20:08] ANYWAY i need to get on the bus [20:11] vorlon, ack, I'll see what I can do about it [20:18] vorlon, it wasn't my intention, stuff like llvm-9 had tests "running" but not really running, queues were empty, so I rescheduled only running tasks that were lost. In case something is missing, please reschedule them :) [20:19] LocutusOfBorg: ok [20:19] mwhudson: yt passed on armhf and on s390x; armhf would not have the memory limit, but s390x would [20:20] -queuebot:#ubuntu-release- Unapproved: util-linux (bionic-proposed/main) [2.31.1-0.4ubuntu3.5 => 2.31.1-0.4ubuntu3.6] (core) [20:20] -queuebot:#ubuntu-release- Unapproved: util-linux (eoan-proposed/main) [2.34-0.1ubuntu2.3 => 2.34-0.1ubuntu2.4] (core) [21:00] -queuebot:#ubuntu-release- Unapproved: grub2 (bionic-proposed/main) [2.02-2ubuntu8.14 => 2.02-2ubuntu8.15] (core) [21:01] vorlon, any idea? https://launchpad.net/ubuntu/+source/puma/3.12.4-1ubuntu2/+build/18828540 [21:01] chroot problem due to glibc? [21:01] same arm64 ppc64el s390x [21:02] -queuebot:#ubuntu-release- Unapproved: grub2-signed (bionic-proposed/main) [1.93.15 => 1.93.15ubuntu1] (core) [21:04] Fetched 69.0 MB in 1s (81.4 MB/s) [21:04] E: This installation run will require temporarily removing the essential package libc6:ppc64el due to a Conflicts/Pre-Depends loop. This is often bad, but if you really want to do it, activate the APT::Force-LoopBreak option. [21:04] E: Internal Error, Could not early remove libc6:ppc64el (2) [21:04] doko: vorlon: juliank: ^ [21:07] vorlon: yeah, not saying it makes sense [21:09] xnox: weeeeeeeeeee [21:09] xnox: Did we add libc6 pre-depends libcrypt1? [21:09] and is that the result? [21:09] y [21:09] at least that's what the .changes said [21:10] ok I guess we gotta revert [21:10] yay another glibc upload? [21:10] wondering why that happens [21:11] exactly, another glibc upload [21:11] Or figuring out the other side of the loop and removing that if possible [21:11] but meh [21:11] idk [21:12] I suggested removing the libc6 depends from libcrypt1 [21:12] it's going to take an aa to remove glibc from proposed before anything can build now isn't it? [21:12] ubuntu-archive: halp [21:12] yes ^^ doko [21:22] wgrant: are you up already? [21:22] wgrant: rm glibc from focal-proposed [21:22] wgrant: and copy in the previous one? [21:23] xnox: context? [21:23] vorlon: see highlight above [21:23] vorlon: cannot install libc6 [21:23] xnox: yes, where is that quoted from? an autopkgtest log, a build log? [21:23] Build log [21:23] https://launchpad.net/ubuntu/+source/puma/3.12.4-1ubuntu2/+build/18828540 [21:24] also why does it say conflicts when there are not [21:24] ah [21:24] Meh who says messages are accurate anyway [21:24] >_< [21:25] https://launchpad.net/ubuntu/+builds?build_text=&build_state=all [21:25] * wgrant isn't quite awake yet [21:25] (from which we can see the order the various architectures got published in) [21:26] ok, removing [21:27] xnox, juliank: is the Breaks: strictly required? should it be only Replaces: instead? then there would be no loop [21:27] we don't know [21:27] I guess [21:27] you probably need it for partial upgrades, but we don't support those [21:28] juliank: Breaks: + Replaces: is generally advisable so that if one installs the replacing package, then removes it again, one doesn't have files go unexpectedly removed from the filesystem [21:28] however that seems unlikely here [21:28] and I don't see any other reason to need it [21:28] you just can't have libcrypt1 migrated before libc6 [21:28] so the question is whether we would prefer dropping the Breaks: and allowing someone to manually install+remove libcrypt1 without upgrading libc6 and thereby breaking their syste [21:28] m [21:29] or if we would prefer dropping the Pre-Depends: and leaving open the possibility of wrong ordering on upgrade [21:29] vorlon: libcrypt1 is priority required, thus i don't think one can "simply" remove it? [21:29] apt doesnt care [21:29] apt only cares about essential / xb-important [21:29] i thought apt wants one to type "Do as i say" [21:29] hm [21:30] is libcrypt1 essential yet? [21:30] no, and runtime libraries never are [21:30] presumably removing a dependency of essential: yes is non-trivial? [21:30] they're only "virtually essential" [21:30] well from apt's side it is essential once libc6 depends on it [21:30] mwhudson: correct, but that only helps you /after/ you've upgraded libc [21:30] apt's understanding of essential is transitive [21:30] :D [21:31] if you shoot yourself in the foot by installing and removing libcrypt without upgrading libc, it doesn't protect you [21:31] yeah, cause i don't see essential set on libc6 itself [21:31] heh just mark it xb-important: yes :D [21:31] vorlon: if the libc on disk doesn't depend on libcrypt1 how does that matter? or is this about what happens during upgrades? [21:32] mwhudson: it's the fact that it's taking over files that previously belonged to libc [21:32] then remove the bit in focal+1 [21:32] that's why we want the pre-depends in the first place [21:32] ah [21:32] mwhudson: we are splitting libc6 into libc6 & libcrypt1, with libcrypt1 taking over file from libc6 [21:32] on upgrade [21:32] we got it a bit wrong 3 times now =) [21:32] xnox: i think i had missed the fact that libcrypt1.so was already on disk [21:33] mwhudson: right, it used to be provided by glibc (they had their own implementation) but libxcrypt implementation is better [21:33] yeah i get all that, but i didn't realize the soname was the same [21:33] vorlon: juliank: can we just use tarball component of libxcrypt and build it as part of glibc source build and avoid this mess? =) [21:33] should have just added libxcrypt to glibc source package as extra tarball and called it ad ay [21:33] or like Built-Using, and make libc6 copy-in and ship libxcrypt's libcrypt1 =) [21:34] but to be more serious, my understanding from the last discussion is that it's unlikely for Depends to go wrong ordering wise, and the only thing we need to fix is the file path [21:34] riht [21:35] I'd kind of prefer us to have the same issues debian has over making up our own ones [21:35] and we thought we needed predepends, whilst autopkgtest VMs, had wrong version strings in breaks/replaces which were incorrect for ubuntu [21:35] and it migrated by accident [21:35] hence I was slightly surprised that we did the pre-depends upload after all [21:35] but then I don't have a straight view of the timeline [21:36] and I'm not exactly fully conscious at the moment [21:38] vorlon: so without breaks the failure mode is something like: [21:38] person running eoan updates apt sources to focal [21:38] they upgrade libcrypt1 only from focal [21:38] then apt remove libcrypt1 will not complain but also will break their system [21:40] right, or focal to focal more likely [21:40] we'vce got a ton of people on there now [21:40] making the version of libcrypt1 in focal un-removable would be a bit of a hack but close this hole [21:40] right [21:41] i.e. essential: yes or whatever xb-important is :) [21:41] Still I'd prefer to do what Debian does, because it _should_ be fine, and I don't want to get into a different mess than they'd be in [21:41] juliank: does debian have an actual plan here? [21:41] mwhudson: debian thinks they're approach is fine [21:41] because i was wondering if debian is watching our flailing here [21:41] juliank: which is? [21:41] mwhudson: oh we told them [21:42] mwhudson: libcrypt1 breaks/replaces old libc6, libc6 depends on libcrypt1, perl preinst gets changed to ignore libcrypt1 loading error IIRC [21:42] i.e. sdome debconf questions were made optional [21:42] uh huh ok [21:43] The assumption is that it's highly unlikely that they'll end up in a broken situation [21:43] because you know, apt will configure libc6 immediately anyway [21:43] there's no real difference to a pre-depends _in practice_ [21:44] but it avoids having to break the loop which apt barfso n [21:45] but anyway I'm too tired to think much [21:46] i am conscious that i've never really understood this end of things [21:47] I might understand it if I was actually awake [21:47] i am confused why libcrypt1 doesn't have Depends libc6 >= 2.31 [21:47] Depends: libc6 (>= 2.25) [21:47] Breaks: libc6 (<< 2.31) [21:47] Replaces: libc6 (<< 2.31) [21:47] is bogus [21:48] juliank: no difference in practice> even if other virtually-essential packages were also unconfigured at the time and apt were to try to decide to configure them before libc6? [21:48] vorlon: that seems unlikely to happen, as they'd be all immediately configured or apt would tell you it can't immediate-configure them [21:48] AFAIUI [21:48] don't trust a sleepy juliank [21:48] hmm [21:49] xnox: makes no difference [21:49] does immediately-configured imply one at a time? [21:49] i think so [21:49] * xnox goes to watch the latest episode of "keeping up with coronavirus" [21:49] xnox: Depends and Breaks both other the same thing in practice [21:50] hm i wonder if predepends is impossible [21:50] because we cannot configure libcrypt1, only unpack it?! [21:50] that is correct, there is a loop with Pre-Depends [21:50] dpkg will break a pre-depends/depends cycle [21:51] it will configure it despite the dep being unsatisfied [21:51] dpkg's loop breaking might be irrelevant in essential [21:51] because apt will break the loop itself, or try to and fail [21:51] because the whole immediate configure thing [21:51] E: This installation run will require temporarily removing the essential package libc6:ppc64el due to a Conflicts/Pre-Depends loop [21:52] I think that's the reason it says that [21:53] A {Pre-,}Depends-Unpacked would solve such issues and simplify a lot of stuff [21:58] juliank: does apt not implement the same loop-breaking logic as dpkg then, regarding pre-depends+depends loops? [22:00] vorlon: i don't know [22:00] so I could test this [22:01] by uploading libxcrypt, dropping the Breaks [22:01] and reviving glibc [22:01] and seeing what happens [22:04] sure [22:04] in any case, I'll try to get some sleep, maybe I get lucky [22:05] a good night's sleep is about the same level of difficulty as this library split [22:21] were you talking about E: This installation run will require temporarily removing the essential package libc6:amd64 due to a Conflicts/Pre-Depends loop. This is often bad, but if you really want to do it, activate the APT::Force-LoopBreak option. ? [22:24] ahasenack: yes [22:27] * doko hates debugging broken autopkg tests [23:07] juliank, xnox, mwhudson: libxcrypt without Breaks: unblocks us, so I'd like to proceed from here [23:08] vorlon: yay [23:10] vorlon: is britney going to try to schedule 1 million autopkgtests again? [23:10] yes [23:10] and then I'm going to murder them all [23:10] and manually reschedule the ones we want [23:11] vorlon: and did you think at all about ways to cheat and make libcrypt1 unremovable? [23:11] (seems low priority though) [23:11] mwhudson: no; I think that particular corner case is unlikely [23:11] https://launchpad.net/ubuntu/+builds?build_text=&build_state=chrootwait <- i guess i'll retry the ones of these from today [23:12] unless chroot problems get autoretried somehow? [23:12] mwhudson: they don't afaik [23:12] hm wait someone already did this? [23:12] https://launchpad.net/ubuntu/+source/puma/3.12.4-1ubuntu2/+build/18828540 is successful now [23:12] yeah it was really only puma that landed in the window afaik [23:13] vorlon: or did you retry some to test things were ok? [23:13] and I retried them thinking it would be a test case [23:13] and they built with glibc 2.30 instead [23:13] so that was a fail :P [23:13] ha [23:13] * mwhudson retries the others [23:19] /32 [23:19] grr [23:21] ok they all built now [23:22] well bar one but that'll be fine i'm sure [23:58] -queuebot:#ubuntu-release- Unapproved: ubuntu-dev-tools (bionic-proposed/universe) [0.164 => 0.175~18.04.1] (no packageset)