| tsimonq2 | Let me know if there's anything else I need to do with those SRUs | 00:00 |
|---|---|---|
| vorlon | sil2100: as you pointed out, some of these nvidia packages are on the UbuntuStudio image, so it would be more than just livecd-rootfs impacted | 00:00 |
| vorlon | sil2100: what about taking a second snapshot for just UbuntuStudio? | 00:01 |
| Eickmeyer[m] | vorlon: That's strange, we don't seed nvidia because Calamares has no facility to install them via ubuntu-drivers, unlike ubiquity. | 00:02 |
| sil2100 | vorlon: hm, I guess that's an idea | 00:02 |
| vorlon | oh, when he said desktop I assumed he meant all of tem | 00:02 |
| tsimonq2 | Old seed leftovers/recommends? | 00:03 |
| * Eickmeyer[m] checks recent germinat | 00:03 | |
| sil2100 | Well, I checked the .list for studio and didn't see any nvidia packages, but I just did a quick look earlier. Just saw those on te ubuntu ones | 00:03 |
| Eickmeyer[m] | Ok, nvm then. | 00:03 |
| vorlon | sil2100: an additional snapshot is comparatively cheap | 00:04 |
| sil2100 | But still, I like the idea of a separate snapshot for studio. I think it's relatively clean too - sure, it's not something we do usually, but still not bad | 00:04 |
| sil2100 | Okay, let's get things moving then! vorlon did you review tsimonq2's MP? | 00:05 |
| sil2100 | The livecd-rootfs is already building in bileto, so we can bin-copy it if it looks good to you as well | 00:05 |
| tsimonq2 | https://code.launchpad.net/~tsimonq2/ubuntu/+source/livecd-rootfs/+git/livecd-rootfs/+merge/427786 | 00:05 |
| vorlon | I have reviewed it now and it looks sane to me | 00:05 |
| vorlon | unnecessary quoting of a literal string within [] and unnecessary use of bash as a shell but that's minor | 00:06 |
| tsimonq2 | Do you guys want me to handle the Bileto handling and let you take it from there with the archive portion or do you have the inclination to do it all? | 00:06 |
| tsimonq2 | bash as a shell is personal preference for CYA and the lack of quotations made vim mad :P | 00:07 |
| tsimonq2 | (I know you can't always assume bash is everywhere but I saw it in another hook so :P) | 00:07 |
| * Eickmeyer[m] bashes all the things, including his head against the wall for the pun of it. | 00:09 | |
| vorlon | tsimonq2: this is the wrong git history, see Vcs-Git for the source package | 00:09 |
| tsimonq2 | vorlon: No wonder I was so confused... thanks for the heads-up, will recreate | 00:09 |
| sil2100 | hm, does anyone know how useful autopkgtests for livecd-rootfs are in this case? Since we're already ekhm, fast-tracking, I'd be willing to just let it go straight to -proposed *and* -updates | 00:10 |
| tsimonq2 | I mean, I literally copied the test script to build my changes, so I think it would be fine to skip :P | 00:10 |
| sil2100 | Once the riscv64 binaries build I'll double check what it built with and then do the copies | 00:12 |
| tsimonq2 | Sounds good! | 00:12 |
| sil2100 | Ok, actually those are built but need to publish, and I'll prefer to wait so that not to re-trigger those | 00:12 |
| vorlon | livecd-rootfs has good autopkgtests but if there's a failure mehhh we can fix it in the next SRU, the point is whether ubuntustudio builds correctly with it | 00:13 |
| vorlon | sil2100: I need to interleave picking up dinner for the family at some point in the next hour, puts me out for ~20 minutes; is now a good time? | 00:14 |
| sil2100 | vorlon: yes! I'll keep an eye out for the builds to publish, do the copies and start pre-publishing in the meantime | 00:14 |
| vorlon | sil2100: ok | 00:14 |
| tsimonq2 | vorlon, sil2100, Eickmeyer: https://code.launchpad.net/~tsimonq2/livecd-rootfs/+git/livecd-rootfs/+merge/427788 | 00:17 |
| sil2100 | Argh, mvo! | 00:22 |
| sil2100 | cp: cannot create hard link 'www.prev/full/ubuntu-core/22/stable/current' to 'www/full/ubuntu-core/22/stable/current': Operation not permitted | 00:22 |
| sil2100 | When doing the backup copy. Because mvo created current as the 'mvo' user | 00:22 |
| sil2100 | btw. it's scary how our images got bigger actually, hm hmm | 00:29 |
| sil2100 | Diffing manifest didn't show any new packages pulled in, but I didn't think about diffing the .list files | 00:29 |
| sil2100 | Ah, okay, more nvidia drivers, that might be it | 00:32 |
| sil2100 | Okay, binaries published, let me do two copy-package calls | 00:34 |
| -queuebot:#ubuntu-release- Unapproved: lxcfs (bionic-proposed/main) [3.0.3-0ubuntu1~18.04.2 => 3.0.3-0ubuntu1~18.04.3] (edubuntu, ubuntu-server) | 00:36 | |
| sil2100 | Copied with binaries to jammy-proposed and jammy-updates, waiting for publishing | 00:38 |
| tsimonq2 | Thank you very much! | 00:38 |
| sil2100 | In the meantime I pre-published the server and desktop images | 00:39 |
| sil2100 | tsimonq2: thanks for working on this! | 00:39 |
| sil2100 | I'll also get the git MP formally approved | 00:39 |
| tsimonq2 | Of course! Feels good to be back :) | 00:39 |
| sil2100 | Merged and tagged | 00:41 |
| tsimonq2 | 🎉 | 00:42 |
| sil2100 | Ouch, okay, need to re-copy the jammy-updates binaries, since apparently LP doesn't like when I copy the same binaries to two pockets at once | 00:42 |
| vorlon | back | 00:47 |
| sil2100 | o/ | 00:49 |
| sil2100 | Oh, darn, forgot that we basically ship one OEM metapackage in the pool now | 00:50 |
| sil2100 | Need to reach out to the OEM QA team to do some testing | 00:50 |
| sil2100 | I'll do some nvidia test install tomorrow | 00:50 |
| jbicha | I found a major bug with the OEM install: bug 1983528 | 00:52 |
| ubottu | Bug 1983528 in snapd (Ubuntu) "snaps fail to launch after OEM install" [Undecided, New] https://launchpad.net/bugs/1983528 | 00:52 |
| tsimonq2 | jbicha: Is this a regression from 22.04? | 00:54 |
| sil2100 | jbicha: ok, that sounds scary. But yeah, could someone try the old 22.04 just-in-case? | 00:55 |
| * tsimonq2 sees if I can reproduce | 00:55 | |
| sil2100 | Since if it's broken and reproducible by others, there's little chance of fixing it before .1 | 00:55 |
| jbicha | I'm testing the OG 22.04 now :) | 00:55 |
| * arraybolt3[m] tests | 00:55 | |
| sil2100 | Thanks o/ | 00:55 |
| jbicha | right | 00:55 |
| arraybolt3[m] | Which should I test, 22.04 or 22.04.1? | 00:56 |
| sil2100 | arraybolt3[m]: can you check 22.04.1? Since I'd like to see if that's reproducible by other testers | 00:57 |
| sil2100 | It's the OEM install test case | 00:57 |
| arraybolt3[m] | Nice, booting into OEM mode now. | 00:57 |
| vorlon | sil2100: do you want to hand things off wrt ubuntustudio so you can get some sleep? | 01:05 |
| sil2100 | vorlon: I still want to start drafting the release announcement, but I'd love if you could pick this up regardless | 01:06 |
| sil2100 | We need livecd-rootfs to be fully published (rmadison still doesn't see it), then snapshot the archive to like some -ubuntustudio variant on snakefruit and then getting new studio images building | 01:07 |
| vorlon | right | 01:07 |
| vorlon | I have 'point-release-snapshot jammy jammy.1.ubuntustudio-security-updates-snapshot' ready to go | 01:07 |
| sil2100 | And of course poking our excellent testers to hack on testing the studio images ASAP | 01:08 |
| sil2100 | (after checking the manifest) | 01:08 |
| vorlon | https://launchpad.net/ubuntu/+source/livecd-rootfs/+publishinghistory still has it pending for -updates | 01:08 |
| sil2100 | oh, and if you build the studio images, you don't have to backup the old images just-in-case, since those are in www.prev/ due to my publishing | 01:08 |
| vorlon | ack | 01:09 |
| vorlon | sil2100: I can take it from here then | 01:09 |
| sil2100 | s/publishing/pre-publishing | 01:09 |
| sil2100 | Thank you! | 01:09 |
| tsimonq2 | While Jeremy checks 22.04 and Aaron checks 22.04.1, I'm checking 22.04 upgraded to 22.04.1 :P | 01:10 |
| jbicha | I'm going to check one more time but I did not get the bug with original 22.04 just now | 01:14 |
| tsimonq2 | Me neither | 01:15 |
| vorlon | does the non-OEM test case cover launching snaps? are we sure it's not broken there too? | 01:16 |
| tsimonq2 | Bug report seems to indicate that non-OEM was fine which seems to indicate AppArmor weirdness to me... | 01:18 |
| tsimonq2 | (Wow, me and repeating myself today, jeez.) | 01:18 |
| jbicha | I believe I ran Firefox successfully after a normal non-OEM install. We should add visit ubuntu.com in Firefox to our other test cases though | 01:18 |
| sil2100 | This is uh not good news | 01:19 |
| sil2100 | arraybolt3[m]: any luck with running the OEM install? | 01:19 |
| arraybolt3[m] | sil2100: Almost done. | 01:19 |
| arraybolt3[m] | I just hit the Firefox button. | 01:19 |
| tsimonq2 | I'll volunteer to git bisect snapd for a beer IOU, sil2100 ;) | 01:19 |
| arraybolt3[m] | Reproduced. This is, indeed, a bug. | 01:20 |
| * arraybolt3[m] uploaded an image: (441KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/QqXzlGUVmpQcvwBPTgawPAwv/image.png > | 01:20 | |
| tsimonq2 | arraybolt3 @arraybolt3:matrix.org: Please ensure the bug report has clear instructions for reproducing | 01:20 |
| tsimonq2 | Literally right after your first reboot or second? | 01:21 |
| sarnold | maybe grab dmesg while you're there | 01:21 |
| sarnold | is snap expected to work without a reboot? | 01:21 |
| tsimonq2 | very good idea, I think sarnold heard apparmor ;) | 01:21 |
| sil2100 | This is terrible. Basically .1 images will not be useful for OEM installs, which I think is a release blocker | 01:21 |
| arraybolt3[m] | tsimonq2: Second reboot. | 01:22 |
| sil2100 | Since this means they might get systems that they can't really use firefox in, which is a core application | 01:22 |
| arraybolt3[m] | I finished the entire OEM setup process as instructed by the test case. | 01:22 |
| arraybolt3[m] | So that means the system the end-user is faced with has a b0rked Firefox out of the box. Yeah, that seems... just shy of fatal. | 01:23 |
| sil2100 | This smells like .1 delay | 01:24 |
| arraybolt3[m] | And this is in a GNOME Boxes VM with EFI + Secure Boot - I think that means it's not hardware-specific. | 01:24 |
| sil2100 | Or again doing the pathetic .1.0 dance as we did already in the past | 01:24 |
| sil2100 | I think we really need to get better with testing our images earlier | 01:25 |
| arraybolt3[m] | Hopefully when that openQA instance goes up all of this will be a thing of the past. | 01:25 |
| sil2100 | I'm sure this was the same case in our dailies, but simply no one tests OEM installs besides during isotesting, since it's not a typical use-case | 01:25 |
| jbicha | I haven't done an oem install in many years before today :/ | 01:27 |
| jbicha | yeah, I duplicated the bug 2x in a row now on 22.04.1 inside GNOME Boxes and avoided the bug 2x in a row on 22.04 (without installing non-security updates in between) | 01:28 |
| arraybolt3[m] | Well fine. I have a "bad" 22.04.1 install active in my VM, any commands you want me to throw at it? | 01:29 |
| tsimonq2 | As sarnold said earlier, try to get the output of dmesg | 01:32 |
| arraybolt3[m] | https://termbin.com/o8g7m | 01:33 |
| vorlon | livecd-rootfs .8 published | 01:34 |
| tsimonq2 | \o/ | 01:34 |
| vorlon | rebuilding ubuntustudio | 01:35 |
| sil2100 | vorlon: heey! | 01:35 |
| sil2100 | vorlon: come visit uuus | 01:35 |
| arraybolt3[m] | Tons of "DENIED" AppArmor stuff with Firefox trying to access stuff. But what really worries me is missing shared libraries within the Snap. That almost sounds like the Snap itself is corrupted somehow. | 01:35 |
| sil2100 | vorlon: come visit uuuus in google meeeeet | 01:36 |
| bdmurray | The non-OEM test case launches firefox to read the release notes but that's in the live environment | 01:37 |
| arraybolt3[m] | Also worthy of note, even "firefox --help" doesn't work, and it fails with what looks like the exact same problem. | 01:38 |
| arraybolt3[m] | (Er, same error message.) | 01:38 |
| tsimonq2 | ...what other snaps does Ubuntu proper ship by default? | 01:38 |
| bdmurray | snap store | 01:39 |
| tsimonq2 | Is there a different GUI application to test to see if it's limited to Firefox? | 01:39 |
| * arraybolt3[m] has to go afk right now, BRB | 01:39 | |
| arraybolt3[m] | Simon Quigley: snap-store fails also. | 01:39 |
| jbicha | yeah, it's just firefox and snap-store for Ubuntu Desktop. But Ubuntu MATE ships their welcome app for instance as a snap | 01:39 |
| jbicha | Budgie too | 01:39 |
| tsimonq2 | I think that eliminates it as a snap-specific issue... | 01:41 |
| vorlon | did you mean Ubuntu-specific? | 01:41 |
| tsimonq2 | No, I meant firefox-snap-specific | 01:42 |
| bdmurray | http://iso.qa.ubuntu.com/qatracker/milestones/437/builds/254990/testcases/1305/results why did Kubuntu succeed? | 01:43 |
| bdmurray | Could somebody confirm that? | 01:43 |
| tsimonq2 | On it | 01:44 |
| * guiverc is doing a ubuntu-mate install currently; running ubuntu-mate-welcome from terminal gives error, but firefox runs | 01:46 | |
| tsimonq2 | oO | 01:46 |
| arraybolt3[m] | Testing on my end too | 01:48 |
| tsimonq2 | arraybolt3: You back for a while? I'm being poked to go afk irl | 01:49 |
| arraybolt3[m] | Yep, should be back for quite a bit. | 01:49 |
| tsimonq2 | Sounds good. I'll let you take the lead | 01:50 |
| arraybolt3[m] | I can also give Budgie a whirl, though I will have to pull that one's ISO which may take a bit. | 01:51 |
| guiverc | arraybolt3[m], you can zsync from a like iso to reduce bandwidth time (time to calc diff gets added though) | 01:52 |
| arraybolt3[m] | guiverc: Yeah, I do that, but .1 is so new it's only an ~40% match and my speeds are about 4 MiB/s at best. | 01:52 |
| jbicha | I am also able to duplicate the bug with 22.04 OEM install if I do these steps in oem mode: sudo apt update; sudo apt dist-upgrade; reboot; snap refresh | 01:54 |
| arraybolt3[m] | Kubuntu 22.04.1 OEM install in progress | 01:54 |
| bdmurray | jbicha: snap refresh of something specific or ? | 01:54 |
| jbicha | snap refresh to update everything including snapd | 01:55 |
| -queuebot:#ubuntu-release- Builds: Ubuntu Studio DVD amd64 [Jammy 22.04.1] has been updated (20220804) | 02:05 | |
| Eickmeyer[m] | There we go. | 02:05 |
| bdmurray | A targetted refresh of snaps on a 22.04 OEM install might be helpful | 02:09 |
| -queuebot:#ubuntu-release- Builds: Xubuntu Desktop amd64 [Jammy 22.04.1] has been marked as ready | 02:13 | |
| arraybolt3[m] | Firefox launches out of the box on a 22.04.1 ISO of Kubuntu. | 02:16 |
| arraybolt3[m] | OEM install thereof. | 02:17 |
| arraybolt3[m] | And this is after the second reboot. | 02:17 |
| vorlon | it is apparmor-related. /var/lib/snapd/apparmor/profiles/snap.firefox.firefox is completely different before and after refresh | 02:18 |
| amurray | hmm I would argue that is snapd related - what is the output of snap connections firefox before and after the refresh? | 02:19 |
| sil2100 | Good luck everyone, see you in the morning o/ | 02:19 |
| arraybolt3[m] | sil2100: 👋 | 02:19 |
| * amurray is currently download the ISO to try and replicate + debug locally too... | 02:19 | |
| vorlon | amurray: will have to reinstall to answer that - I agree it's likely snapd that's getting it wrong and wanted to look at connections based on what I'm seeing here | 02:20 |
| vorlon | $ sudo diff -uNR /var/lib/snapd/apparmor/profiles/snap.firefox.firefox /tmp/apparmor.bak/profiles/snap.firefox.firefox | wc -l | 02:21 |
| vorlon | 2343 | 02:22 |
| bdmurray | that's a lot of lines | 02:22 |
| arraybolt3[m] | Meanwhile I'm gonna attack Budgie and see the situation there in case it will be helpful. | 02:23 |
| kenvandine | sil2100: does anyone have an oem install that can be used for debugging? | 02:42 |
| arraybolt3[m] | kenvandine: Me over here. | 02:42 |
| arraybolt3[m] | kenvandine: I kept the VM around for just such an occasion. | 02:42 |
| kenvandine | jamesh suggested checking snap changes | 02:43 |
| arraybolt3[m] | (Ubuntu Budgie also has broken Snap Firefox.) | 02:43 |
| arraybolt3[m] | So now I have two "bad" VMs ready for action. | 02:43 |
| kenvandine | hey jamesh glad to see you are here :) | 02:43 |
| jamesh | arraybolt3[m]: if it is the same boot where snapd tries to seed the snaps, you might see evidence in "snap changes" | 02:43 |
| vorlon | jamesh: hi, https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1983528/comments/10 | 02:44 |
| ubottu | Launchpad bug 1983528 in snapd (Ubuntu) "seeded snaps fail to launch after OEM install" [Critical, Confirmed] | 02:44 |
| arraybolt3[m] | https://termbin.com/faqk <- output of "snap changes" on bad Budgie VM | 02:44 |
| jamesh | arraybolt3[m]: could you also provide "snap change 1" and "snap change 2"? I'm not sure whether it'll actually have anything helpful, but it can't hurt. | 02:46 |
| arraybolt3[m] | http://termbin.com/3ur0 | 02:47 |
| arraybolt3[m] | http://termbin.com/vanz | 02:47 |
| arraybolt3[m] | First one is "snap change 1", second one is "snap change 2", both on Budgie. | 02:47 |
| vorlon | jamesh: see my comment on the bug; there are no new snap changes when it breaks, and fwiw what I'm looking at here shows the last snap change happening before snap.firefox.firefox gets mangled | 02:47 |
| vorlon | "Ready today at 19:36/19:37 PDT", but the file changed at 19:39 PDT | 02:48 |
| arraybolt3[m] | (I also just booted the Ubuntu VM and... hey here's something new. Firefox still won't launch on the bad Ubuntu VM, but after a reboot the error message has changed entirely.) | 02:48 |
| * arraybolt3[m] uploaded an image: (649KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/rARiJFonqGLcZCGgbmcNtfud/image.png > | 02:48 | |
| arraybolt3[m] | incase the screenshot isn't helpful: https://pastebin.com/Raf0QvVF | 02:48 |
| arraybolt3[m] | output of "firefox" on the bad Ubuntu VM | 02:48 |
| kenvandine | that would still be the lack of connected interfaces | 02:49 |
| arraybolt3[m] | The original error I believe, and the error I still get in Budgie is as follows: | 02:49 |
| kenvandine | arraybolt3: what does snap connections firefox | 02:49 |
| arraybolt3[m] | https://termbin.com/dib4 <- snap connections firefox from Ubuntu | 02:50 |
| jamesh | vorlon: if the interface connections have disappeared, I wonder if /var/lib/snapd/state.json is getting corrupted somehow? | 02:50 |
| arraybolt3[m] | The same command on Budgie returns nothing but a hard return. | 02:50 |
| vorlon | jamesh: state.json has a matching timestamp to the broken apparmor profile; it looks jsonish in content and firefox is listed | 02:51 |
| vorlon | should I copy that file out and attach to the bug? | 02:52 |
| jamesh | vorlon: may as well. If it doesn't mean anything to me, it might to some of the snapd folk | 02:54 |
| vorlon | jamesh: sent | 02:54 |
| vorlon | afk for a bit | 02:55 |
| bdmurray | arraybolt3[m]: to be clear you did an OEM install of Budgie and it has the snap issue? | 03:00 |
| tsimonq2 | Does anyone know if this is a thing on Kinetic? | 03:00 |
| arraybolt3[m] | bdmurray: Yes. OEM install, this is after the second reboot. | 03:00 |
| bdmurray | tsimonq2: No, that'd be a good test | 03:00 |
| bdmurray | arraybolt3[m]: Could you update the ISO tracker so the release team has a record of what images would need fixing. | 03:00 |
| arraybolt3[m] | So OEM install, boot in, do update and dist-upgrade, reboot, do the thing that preps it for the end user, reboot, then setup and then Firefox is broke. | 03:01 |
| arraybolt3[m] | Yes. | 03:01 |
| arraybolt3[m] | (In response to ISO tracker update) | 03:01 |
| bdmurray | Thanks | 03:01 |
| jamesh | vorlon: there's no implicit system connections in that state.json file, which is very weird | 03:02 |
| amurray | the journalctl output of snapd shows a lot of errors once in this state - looks like the snaps are not mounted or somesuch when snapd is starting - https://paste.ubuntu.com/p/xNY66Rt9DM/ | 03:04 |
| amurray | (but that is just a guess - since I can see them now via ls) - I wonder if a systemctl restart snapd would be sufficient... | 03:06 |
| tsimonq2 | Does restarting the snapd service do anything to help? | 03:07 |
| tsimonq2 | Ah, you just said that :) | 03:07 |
| kenvandine | those errors look like seeded failed | 03:08 |
| vorlon | it does not | 03:08 |
| vorlon | NB oem-config interrupts the normal boot sequence | 03:08 |
| kenvandine | but snap changes should show that | 03:08 |
| vorlon | it's possible snapd has started but some other units have not been run due to wrong dependencies | 03:08 |
| vorlon | I'm going to finish the oem install and try another reboot to see if that fixes anything | 03:09 |
| vorlon | if so, "have you tried turning it off and on again" might be release-noteable | 03:09 |
| arraybolt3[m] | Don't forget I still have both my broken-Firefox VMs over here for experimentation. | 03:10 |
| bdmurray | I thought arraybolt3[m] said that didn't work | 03:10 |
| arraybolt3[m] | A VM reboot did not fix the problem, but it did change the error message dramatically. | 03:10 |
| amurray | restarting snapd didn't help | 03:11 |
| arraybolt3[m] | Testing UKylin... | 03:11 |
| vorlon | right, after a reboot 'snap connections' is no longer empty but it lists only content interfaces | 03:12 |
| arraybolt3[m] | Also, I just remembered something - this bug does not occur on the Raspberry Pi image, despite the fact that it appears to be a preinstalled OEM installation. | 03:13 |
| vorlon | and there are no longer error messages since reboot about unmounted snaps | 03:13 |
| kenvandine | the flow on the pi is very different, iirc | 03:14 |
| vorlon | the snap mount units use WantedBy=multi-user.target | 03:14 |
| vorlon | and when we boot oem mode we're using a different target | 03:14 |
| amurray | systemctl shows multi-user.target is active for me | 03:20 |
| -queuebot:#ubuntu-release- New binary: meshsdfilter [amd64] (kinetic-proposed/universe) [1.0+1gitb81411-1] (no packageset) | 03:24 | |
| -queuebot:#ubuntu-release- New binary: specreduce [amd64] (kinetic-proposed/universe) [1.0.0-2] (no packageset) | 03:24 | |
| -queuebot:#ubuntu-release- New binary: meshsdfilter [s390x] (kinetic-proposed/universe) [1.0+1gitb81411-1] (no packageset) | 03:24 | |
| -queuebot:#ubuntu-release- New binary: meshsdfilter [ppc64el] (kinetic-proposed/universe) [1.0+1gitb81411-1] (no packageset) | 03:25 | |
| jamesh | vorlon: yep. That looks to be the problem from snapd's logs: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1983528/comments/13 | 03:30 |
| ubottu | Launchpad bug 1983528 in snapd (Ubuntu) "seeded snaps fail to launch after OEM install" [Critical, Confirmed] | 03:30 |
| jamesh | is the different systemd target new? | 03:30 |
| arraybolt3[m] | Ugh, just found an OEM bug in Ubuntu Kylin, totally different this time - the slideshow played during the final user config step is from Ubuntu, not UKylin's slideshow. | 03:33 |
| bdmurray | That sounds familiar | 03:34 |
| tsimonq2 | Meh, not a release blocker but still file the bug please ;) | 03:35 |
| bdmurray | It's likely already reported so let me have a look first | 03:35 |
| arraybolt3[m] | However, Firefox is working out of the box on UKylin. | 03:36 |
| bdmurray | arraybolt3[m]: bug 1842047 might be the bug I'm remembering and the one which you are encountering | 03:41 |
| ubottu | Bug 1842047 in ubiquity-slideshow-ubuntu (Ubuntu Impish) "Flavors should use their slideshows when shipping to the end user" [Undecided, Triaged] https://launchpad.net/bugs/1842047 | 03:41 |
| Eickmeyer[m] | bdmurray: I think I remember that one too. Recent memory says it's just a gray box with a progress bar at the moment after shipment. | 03:43 |
| arraybolt3[m] | Yep, looks like the same bug to me. | 03:43 |
| tsimonq2 | Eickmeyer @eickmeyer:matrix.org: So... your ISO? | 03:43 |
| Eickmeyer[m] | Simon Quigley: No, that's more with my Kubuntu hat on. | 03:44 |
| tsimonq2 | No... the earlier issue... | 03:44 |
| arraybolt3[m] | oh my gosh I totally forgot Studio in this whole flurry. OK, testing Secure Boot now. | 03:44 |
| arraybolt3[m] | (I updated my ISO.) | 03:44 |
| Eickmeyer[m] | Simon Quigley: In the words of Fraser Crane, "I'm listening." | 03:45 |
| -queuebot:#ubuntu-release- New binary: meshsdfilter [arm64] (kinetic-proposed/universe) [1.0+1gitb81411-1] (no packageset) | 03:52 | |
| vorlon | jamesh, amurray: oem-config-prepare sets the default target to oem-config.target. So on the first user boot, we are not booting to multi-user.target, until after oem-config runs, then we 'systemctl set-default graphical.target'. Before that, multi-user.target has not been required. and this has not changed since release on the ubiquity/oem-config side so the regression seems like it must be on | 03:57 |
| vorlon | the snapd side | 03:57 |
| bdmurray | A 'snap remove firefox' and 'snap install firefox' fixed it for me but that doesn't seem like a good workaround | 03:57 |
| vorlon | agreed | 03:58 |
| amurray | systemd-analyze plot - https://people.canonical.com/~amurray/systemd.svg | 03:59 |
| vorlon | snapd.service is also WantedBy=multi-user.target, but snapd.socket is of course WantedBy=sockets.target which always happens by default | 03:59 |
| jamesh | and the only ordering between the snap .mount units and snapd.service is that the mount units have Before=snapd.service | 04:01 |
| vorlon | amurray: indeed, that confirms the snapd mount units are only starting after oem-config finishes. But what causes snapd itself to start up and break things? | 04:01 |
| amurray | yep and what is even odder is I can't see snapd.service in that plot output... | 04:02 |
| jamesh | if we're not actually going into multi-user.target, they won't be scheduled to start and there's nothing to stop snapd.service running first | 04:02 |
| jamesh | vorlon: we've got snapd-desktop-integration installed now: a user session service that would be pinging snapd's rest socket and starting it | 04:03 |
| jamesh | that's probably it | 04:04 |
| vorlon | aha, that definitely would do it | 04:04 |
| jamesh | it'd be running in the oem-config desktop session | 04:04 |
| vorlon | and bdmurray that also explains why it doesn't affect other flavors ^^ | 04:04 |
| vorlon | (i.e. you were right :) | 04:04 |
| vorlon | so how can we fix this | 04:05 |
| jamesh | make snapd create the mount units so they're wanted by oem-config.target? | 04:05 |
| vorlon | I was wondering if WantedBy=multi-user.target default.target would dtrt | 04:06 |
| amurray | can the mount units specify that they run before snapd.socket? | 04:06 |
| vorlon | only if they also do something to get themselves pulled into the current boot graph | 04:07 |
| vorlon | (systemd only looks at ordering of the set of units that are wanted to be started) | 04:08 |
| vorlon | I think the Before=snapd.service is correct (even though snapd.service doesn't show in the boot graph because it's started via the socket) | 04:08 |
| vorlon | So wantedBy=multi-user.target default.target should fix this | 04:09 |
| jamesh | right. Before= only imposes an order if both units are to be started | 04:09 |
| amurray | ah ok | 04:09 |
| vorlon | someone could test this out by doing an oem install boot and editing /etc/systemd/system/snap-firefox-*.mount to change WantedBy, before rebooting to oem-ocnfig | 04:10 |
| * arraybolt3[m] does what vorlon just said | 04:10 | |
| vorlon | (not me at the moment, I have to run to the store) | 04:10 |
| amurray | why not just WantedBy=default.target (ie. is having multi-user.target redundant / unnecessary since we want snapd to always be wanted right?) | 04:10 |
| * amurray is likely showing his systemd ignorance | 04:11 | |
| vorlon | amurray: ah I was thinking in terms of ordering but you're right | 04:11 |
| vorlon | (because default.target never points to multi-user.target, it points to graphical.target; and graphical.target is later than multi-user.target; but since snapd itself is WantedBy=multi-user.target it doesn't matter in practice) | 04:12 |
| vorlon | maybe we want to be explicit and consistent with snapd.service, maybe not | 04:12 |
| jamesh | the only problem with this hypothesis is: how does snap-desktop-integration run if it isn't mounted? | 04:14 |
| jamesh | could it have been started before all the mount units were stopped? | 04:16 |
| -queuebot:#ubuntu-release- New binary: meshsdfilter [riscv64] (kinetic-proposed/universe) [1.0+1gitb81411-1] (no packageset) | 04:27 | |
| bdmurray | I changed /etc/systemd/system/snap-firefox-*.mount to "WantedBy=default.target" before rebooting into oem-config (by mounting the qcow2 img, one I'd saved before rebooting, and editing the systemd service) and that did fix firefox launching. | 04:31 |
| bdmurray | s/did/did not/ | 04:31 |
| amurray | I think we would also need to run systemctl daemon-reload though to install the right symlinks | 04:33 |
| arraybolt3[m] | vorlon: I tried that trick you talked about (modifying the Firefox snap line in systemd to read WantedBy=multi-user.target default.target), and it didn't work - I still get an "Error: can't open display: :0" message when I try to run Firefox in the terminal on a fresh OEM install of Ubuntu 22.04.1. | 04:34 |
| vorlon | bdmurray, arraybolt3[m]: did it at least fix the journalctl output? | 04:53 |
| vorlon | you may also need to do it for more snaps than just firefox | 04:53 |
| vorlon | e.g. snapd which provides the many interfaces? | 04:54 |
| arraybolt3[m] | vorlon: By journalctl output do you mean the dmesg log? | 04:56 |
| vorlon | snapd.apparmor.service is also supposed to start before snapd but is being skipped | 04:56 |
| arraybolt3[m] | I can termbin either one if you'd like. | 04:56 |
| vorlon | arraybolt3[m]: no, I mean 'sudo journalctl -lu snapd.service' | 04:56 |
| arraybolt3[m] | https://termbin.com/x5e9 | 04:57 |
| vorlon | Aug 03 23:27:51 oem-Standard-PC-Q35-ICH9-2009 snapd[796]: snapmgr.go:363: cannot read snap info of snap "firefox" at revision 1635: cannot find installed snap "firefox" at revision 1635: missing file /snap/firefox/1635/meta/snap.yaml | 04:57 |
| vorlon | that's in your output, so evidently the snap was still not mounted before snapd started | 04:57 |
| arraybolt3[m] | OK I guess then I'll change the other snap things, didn't realize that was necessary. | 04:58 |
| vorlon | well the above output shows your change to the firefox mount unit was insufficient | 04:58 |
| vorlon | and I currently don't know why | 04:58 |
| vorlon | we're on the general right track, anyway | 04:59 |
| arraybolt3[m] | I changed every single Snap mount file in /etc/systemd/system to have default.target added to the line and I still get the same error message. | 05:01 |
| arraybolt3[m] | https://termbin.com/la3c <- journalctl -lu snapd.service output after latest change | 05:02 |
| arraybolt3[m] | The full error from Firefox: https://pastebin.com/u67NktzB | 05:03 |
| vorlon | arraybolt3[m]: and if you also set /lib/systemd/system/snapd.apparmor.service to use default.target? | 05:08 |
| arraybolt3[m] | One moment... | 05:09 |
| vorlon | more than a moment I suspect unless you're much more sensible than me and are working from image snapshots | 05:09 |
| arraybolt3[m] | Ah, if only... I'm just changing things in the fully installed image at this point. | 05:10 |
| arraybolt3[m] | I'll probably have to do another reinstall at some point. | 05:10 |
| vorlon | ok that doesn't work | 05:10 |
| vorlon | you need to make these changes before oem-config runs | 05:10 |
| arraybolt3[m] | :( | 05:10 |
| * arraybolt3[m] proceeds to scrap entire VM and start from scratch | 05:10 | |
| amurray | but after running 'Prepare for shipping to end user' so the default target it already set to oem-config.target - I am trying this now too FWIW | 05:11 |
| amurray | once you have edited the mount unit file you'll also need to run `systemctl enable snap-xxxx-nnn.mount` so that the symlink gets installed | 05:12 |
| vorlon | oh hah I missed that part didn't I | 05:13 |
| arraybolt3[m] | vorlon: So just so I'm clear, I install in OEM mode, I boot into the OEM user, I mess with the files, I reboot, I run oem-config, I reboot again, I do the configuration, then I'm in the end-user system and try to launch Firefox? | 05:13 |
| vorlon | install in oem mode; boot into the oem user; use the 'prepare' link; mess with the files (and also run systemctl enable, per amurray); reboot; run oem-config; then you can log in as the user you've just created | 05:14 |
| amurray | vorlon: ok so I can confirm that setting "WantedBy=multi-user.target default.target" in each of the mount units, then doing a systemctl enable for each of them is sufficient to fix this | 05:16 |
| arraybolt3[m] | Ah OK. | 05:16 |
| vorlon | amurray: \o/ sweet so this should be fixable with a targeted snapd change | 05:17 |
| vorlon | amurray: do you mind dumping this finding into the bug? | 05:17 |
| amurray | I do wonder if perhaps snapd.socket may better than default.target as I am hoping this would then mean the mount units get started as the socket is setup | 05:17 |
| amurray | sure will update the bug | 05:18 |
| * amurray schoolrun | 05:22 | |
| arraybolt3[m] | Success, got Firefox and snap-store working "out of the box" on Ubuntu 22.04.1 with vorlon's instructions. | 05:37 |
| arraybolt3[m] | OEM install obviously, just to be clear. | 05:38 |
| sil2100 | o/ | 07:34 |
| arraybolt3[m] | sil2100: o/ We pinpointed the issue and appear to have a working fix! | 07:50 |
| sil2100 | arraybolt3: I saw! | 07:51 |
| sil2100 | Thanks everyone for the fast turnaround! | 07:51 |
| sil2100 | I'll be discussing next steps, as since the fix is in snapd I don't think it's feasible to still go live with .1 today | 07:51 |
| mardy | jamesh: hi! Do you still have the OEM VM around? It might be helpful to see the full journald logs (maybe also those from the user session) | 07:53 |
| arraybolt3[m] | Yep, I got it. | 07:54 |
| arraybolt3[m] | Do you want the Budgie one with the original error message, or the plain Ubuntu one with the other error message? | 07:54 |
| arraybolt3[m] | (Actually, I'll just do both.) | 07:54 |
| jamesh | mardy: sure. | 07:54 |
| arraybolt3[m] | mardy: Just realized you're not talking to me, but I also have OEM VMs. | 07:55 |
| arraybolt3[m] | (For some reason I thought you were jamesh...) | 07:55 |
| jamesh | mardy: system journal: https://paste.ubuntu.com/p/XcySs8tJmb/, and user journal: https://paste.ubuntu.com/p/RgSNcJYMG5/ | 07:58 |
| mardy | jamesh: thanks! arraybolt3[m]: thanks anyway :-) | 08:03 |
| mardy | BTW, do we know why it started to fail? Has the failure been triggered by the addition of snapd-desktop-integration? | 08:05 |
| mardy | jamesh: so, indeed all the snap mount units where successfully performed initially, and then they've been undone | 08:09 |
| jamesh | mardy: there's a reboot in the middle of that log | 08:09 |
| mardy | jamesh: ah! | 08:11 |
| jamesh | mardy: in essence, the system installs and boots into a throw-away user account where the OEM can perform additional configuration. They then run this script to prepare the system for first boot by the user: https://git.launchpad.net/ubiquity/tree/bin/oem-config-prepare | 08:12 |
| jamesh | The system boots into this second target to peform first boot setup, then sets the default target back to graphical.target and completes the boot | 08:15 |
| mardy | jamesh: I'm a bit confused, I see on the first boot, "started snapd/2.56.2"; then, immediately after the reboot, it's 2.55.5+22.04 (a deb, I guess), and later it's 2.56.2 again | 08:26 |
| mardy | so, the fix that we are making, should it be released as 2.56.3, or should it be delived as a patch to 2.55.5? | 08:27 |
| -queuebot:#ubuntu-release- Builds: Kubuntu Desktop amd64 [Jammy 22.04.1] has been marked as ready | 08:28 | |
| jamesh | mardy: I think an updated snapd snap would be sufficient. The mount units are all being created by 2.56 | 08:32 |
| jamesh | mardy: we only see snapd 2.55.2 run in oem firstboot because that's the deb packaged version, and it can't re-exec to 2.56 due to it not being mounted | 08:32 |
| jamesh | that's my reading of the logs, at least. | 08:33 |
| mardy | jamesh: thanks, makes sense | 08:37 |
| sil2100 | seb128: o/ | 08:40 |
| seb128 | hey sil2100! | 08:43 |
| sil2100 | seb128: welcome back! Did you hear about the OEM install woes? | 08:43 |
| seb128 | sil2100, thanks. I've read that and saw in the morning backlog that the snapd team is having a fix? | 09:02 |
| sil2100 | Yes, but it won't make it for .1 today, it's a snapd change and that's the last thing we can fast-track in just a few hours | 09:03 |
| mardy | jamesh: I think your theory is right, and the reason why the snapd-desktop-integration service is able to start is that it declares a "Requires:" on its mount unit (but you can double-check this by looking at its service file in /etc/systemd/system/) | 09:20 |
| -queuebot:#ubuntu-release- Unapproved: adsys (focal-proposed/main) [0.8~22.04 => 0.9.2~20.04] (no packageset) | 09:20 | |
| -queuebot:#ubuntu-release- Unapproved: rejected adsys [source] (focal-proposed) [0.9.2~20.04] | 09:22 | |
| -queuebot:#ubuntu-release- Unapproved: adsys (jammy-proposed/main) [0.8.5~22.04 => 0.9.2~22.04] (no packageset) | 09:25 | |
| -queuebot:#ubuntu-release- Unapproved: adsys (focal-proposed/main) [0.8~22.04 => 0.9.2~20.04] (no packageset) | 09:32 | |
| arraybolt3[m] | OK, I'm signing off for now. I'll still have my OEM VMs open in case they come in handy for stuff later. Thanks all! | 09:36 |
| jamesh | mardy: it actually doesn't have the Requires= line. It's a user unit rather than a system unit | 09:44 |
| jamesh | mardy: I do wonder if "snap run" is hitting the REST socket before failing to start the app. | 09:45 |
| mardy | jamesh: there are lines like this in the user's journal log: Aug 04 11:11:44 james snapd-desktop-integration.snapd-desktop-integration[34750]: /snap/snapd-desktop-integration/14/snap/command-chain/desktop-launch: line 261: /home/james/.config/user-dirs.dirs: Permission denied | 10:19 |
| mardy | that seems to show that the snap was mounted, doesn't it | 10:19 |
| mardy | ? | 10:19 |
| jamesh | mardy: I think that's from after OEM firstboot setup has finished and I logged in. It comes after snapd has deleted all the interface connections and rebuilt broken AppArmor profiles for the snaps though, hence the Permission Denied error | 10:29 |
| jamesh | The line "Aug 04 11:09:41 oem-Standard-PC-Q35-ICH9-2009 snapd[880]: helpers.go:268: removed stale connections: ...." message in the system log is where everything breaks. | 10:30 |
| mardy | jamesh: right... do you have a user journal log before 11:11:43? | 10:34 |
| jamesh | mardy: not sure how to retrieve that. It'd be for a user that doesn't exist any more | 10:34 |
| jamesh | ah. the file's still there so I should be able to read it | 10:37 |
| -queuebot:#ubuntu-release- Unapproved: sudo (jammy-proposed/main) [1.9.9-1ubuntu2 => 1.9.9-1ubuntu2.1] (core, i386-whitelist) | 10:40 | |
| jamesh | mardy: here's the oem user's journal: https://paste.ubuntu.com/p/8fpGmf99r3/ | 10:41 |
| -queuebot:#ubuntu-release- New binary: lcrq [amd64] (kinetic-proposed/none) [0.0.1-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: lcrq [s390x] (kinetic-proposed/none) [0.0.1-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: unilog [ppc64el] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: lcrq [ppc64el] (kinetic-proposed/none) [0.0.1-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: unilog [s390x] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: unilog [amd64] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:19 | |
| -queuebot:#ubuntu-release- New binary: lcrq [riscv64] (kinetic-proposed/universe) [0.0.1-2] (no packageset) | 11:31 | |
| -queuebot:#ubuntu-release- New binary: unilog [riscv64] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:33 | |
| -queuebot:#ubuntu-release- New binary: unilog [arm64] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:49 | |
| -queuebot:#ubuntu-release- New binary: unilog [armhf] (kinetic-proposed/universe) [2.5-2] (no packageset) | 11:49 | |
| -queuebot:#ubuntu-release- New binary: lcrq [arm64] (kinetic-proposed/universe) [0.0.1-2] (no packageset) | 11:50 | |
| -queuebot:#ubuntu-release- New binary: lcrq [armhf] (kinetic-proposed/universe) [0.0.1-2] (no packageset) | 11:50 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [amd64] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:12 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [i386] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:13 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [s390x] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:13 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [ppc64el] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:14 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [arm64] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:18 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [armhf] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 12:18 | |
| -queuebot:#ubuntu-release- Unapproved: nfs-utils (jammy-proposed/main) [1:2.6.1-1ubuntu1 => 1:2.6.1-1ubuntu1.1] (core, i386-whitelist) | 12:22 | |
| -queuebot:#ubuntu-release- New binary: cryptsetup [riscv64] (kinetic-proposed/main) [2:2.5.0-1ubuntu1] (core, i386-whitelist) | 13:04 | |
| === plars_ is now known as plars | ||
| sil2100 | :' ) | 14:42 |
| -queuebot:#ubuntu-release- New: accepted lcrq [amd64] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted lcrq [armhf] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted lcrq [riscv64] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted meshsdfilter [amd64] (kinetic-proposed) [1.0+1gitb81411-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted meshsdfilter [ppc64el] (kinetic-proposed) [1.0+1gitb81411-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted meshsdfilter [s390x] (kinetic-proposed) [1.0+1gitb81411-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted orage [armhf] (kinetic-proposed) [4.16.0-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted specreduce [amd64] (kinetic-proposed) [1.0.0-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [arm64] (kinetic-proposed) [2.5-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [ppc64el] (kinetic-proposed) [2.5-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted lcrq [arm64] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted lcrq [s390x] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted meshsdfilter [riscv64] (kinetic-proposed) [1.0+1gitb81411-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted rust-tokio-openssl [riscv64] (kinetic-proposed) [0.6.3-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [armhf] (kinetic-proposed) [2.5-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [s390x] (kinetic-proposed) [2.5-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted lcrq [ppc64el] (kinetic-proposed) [0.0.1-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted orage [arm64] (kinetic-proposed) [4.16.0-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [riscv64] (kinetic-proposed) [2.5-2] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted meshsdfilter [arm64] (kinetic-proposed) [1.0+1gitb81411-1] | 15:01 | |
| -queuebot:#ubuntu-release- New: accepted unilog [amd64] (kinetic-proposed) [2.5-2] | 15:01 | |
| jbicha | bdmurray: is it intentional that amd64 is the only arch for 21.10 final at https://old-releases.ubuntu.com/releases/21.10/ ? | 15:22 |
| bdmurray | jbicha: yes | 15:22 |
| jbicha | and there's no other official archive elsewhere for others arches, right? | 15:22 |
| bdmurray | jbicha: I'm in the middle of a meeting but could explain later or I heard the reasoning from vorlon so he might explain it better | 15:22 |
| jbicha | I only noticed while checking the osinfo-db details | 15:23 |
| bdmurray | Right, I'd seen your comment in Jira which is why it was dicussed. | 15:24 |
| -queuebot:#ubuntu-release- Unapproved: livecd-rootfs (jammy-proposed/main) [2.765.8 => 2.765.9] (desktop-core, i386-whitelist) | 15:26 | |
| tjaalton | sil2100: now that the image is postponed, could linux-firmware be released to updates? | 15:59 |
| -queuebot:#ubuntu-release- Packageset: Added libxs-parse-keyword-perl to i386-whitelist in kinetic | 16:15 | |
| -queuebot:#ubuntu-release- Unapproved: accepted livecd-rootfs [source] (jammy-proposed) [2.765.9] | 16:18 | |
| tsimonq2 | sil2100, vorlon: Lubuntu 22.04 updates we staged for .2> if we have these ready and we're postponing the release anyway, can we include another Cala fix too or is it much too late? | 16:50 |
| sil2100 | I would like to keep the changes minimal, so for now I only opened up the gates for security | 16:51 |
| arraybolt3[m] | sil2100: The fix Simon Quigley is referencing is a fix for a LibreOffice localization bug - we prepped the SRU for before the usual .1 release but didn't manage to actually get it applied to the bug report in mind. The bug breaks a core feature of the installer and makes LibreOffice always display in English no matter what language the user selects. | 16:51 |
| tsimonq2 | Just asking - if we must punt then we must. | 16:52 |
| bdmurray | I'm not following how does LibreOffice break a core feature of the installer? | 16:56 |
| arraybolt3[m] | bdmurray: It's not. There's a bug in our conifg that makes it so that the installed system doesn't have Internet access during installation. | 16:57 |
| arraybolt3[m] | bdmurray: And since the LibreOffice localization packages are downloaded dynamically at install time, they never get installed as a result. | 16:57 |
| arraybolt3[m] | LP: #1970270 | 16:58 |
| ubottu | Launchpad bug 1970270 in calamares-settings-ubuntu (Ubuntu Kinetic) "LibreOffice localization is not working out of the box" [High, Confirmed] https://launchpad.net/bugs/1970270 | 16:58 |
| tsimonq2 | I would not strictly call it a release blocker since it doesn't look like a regression, more of a major annoyance for non-English speaking users (of which we have quite a few). If it's too late, tell me and I'll accept that, + get it into the queue for when you guys are ready. However, I'm 100% confident we could do full SRU verification on this prior to even cutting it close | 16:58 |
| tsimonq2 | sil2100: I understand I've already well, kind of pushed it this release, and I'm sorry for that. Trying to ask ahead of time before I upload to make sure we're on the same page first (+ to be better] | 17:00 |
| tsimonq2 | <arraybolt3[m]> "LP: #1970270" <- arraybolt3 @arraybolt3:matrix.org: I need a Kinetic patch like, yesterday. :P | 17:05 |
| ubottu | Launchpad bug 1970270 in calamares-settings-ubuntu (Ubuntu Kinetic) "LibreOffice localization is not working out of the box" [High, Confirmed] https://launchpad.net/bugs/1970270 | 17:05 |
| * tsimonq2 roasts arraybolt3 @arraybolt3:matrix.org elsewhere ;) | 17:05 | |
| arraybolt3[m] | tsimonq2: I gave it to you a week ago. :) | 17:05 |
| arraybolt3[m] | Check the #lubuntu-devel Matrix scrollback. | 17:05 |
| tsimonq2 | Hah. I'll need it again. Thanks | 17:06 |
| -queuebot:#ubuntu-release- New binary: svt-av1 [i386] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 17:22 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [amd64] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 17:24 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [ppc64el] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 17:35 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [s390x] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 17:41 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [riscv64] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 18:09 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [armhf] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 18:17 | |
| -queuebot:#ubuntu-release- New binary: svt-av1 [arm64] (kinetic-proposed/universe) [1.2.0+dfsg-2] (i386-whitelist) | 18:21 | |
| vorlon | hey folks, so because fixing Ubuntu Desktop OEM install requires updating snapd, this means all the point release images are going to need to be respun. We don't have the snapd fix in yet, but a number of other security updates have been accepted in the meantime, so we think it's useful to have refreshed candidate images for further testing. These are rebuilding now, please test! | 18:32 |
| vorlon | (please test once available :) | 18:32 |
| vorlon | jbicha, bdmurray: old-releases has only ever included images that were published to releases.u.c. The rationale is that if they weren't important enough (== widely downloaded enough) to be hosted on releases.u.c and its mirrors while supported, they are also not important enough to be published in, and take up space in, the online historical archive | 18:34 |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [amd64] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [armhf] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [ppc64el] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [s390x] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [arm64] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [riscv64] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| -queuebot:#ubuntu-release- New: accepted svt-av1 [i386] (kinetic-proposed) [1.2.0+dfsg-2] | 19:06 | |
| vorlon | chmod: cannot access 'config/hooks/100-ubuntustudio-dkms.chroot': No such file or directory | 19:08 |
| vorlon | that doesn't look good | 19:08 |
| vorlon | tsimonq2: ^ well, your chmod was outside the if block and I didn't notice it, so livecd-rootfs currently works *only* to build ubuntustudio | 19:10 |
| vorlon | fixing | 19:10 |
| tsimonq2 | vorlon: already on it? | 19:11 |
| tsimonq2 | oh that's amazing | 19:11 |
| vorlon | sil2100: ^^ looks like it would've been a good idea to wait for livecd-rootfs autopkgtests after all | 19:11 |
| -queuebot:#ubuntu-release- Unapproved: livecd-rootfs (jammy-proposed/main) [2.765.9 => 2.765.10] (desktop-core, i386-whitelist) | 19:13 | |
| vorlon | bdmurray: ^^ fast track fix for livecd-rootfs which has a regression in -updates, can you review please? (this is also why we got all the build failure emails) | 19:14 |
| bdmurray | vorlon: looking | 19:17 |
| -queuebot:#ubuntu-release- New binary: libreoffice-dictionaries [amd64] (kinetic-proposed/main) [1:7.4.0~rc2-4] (ubuntu-desktop) | 19:20 | |
| bdmurray | vorlon: well the "where problems could occur" section of bug 1983521 missed something then | 19:21 |
| ubottu | Bug 1983521 in livecd-rootfs (Ubuntu Jammy) "[SRU] Manually blacklist DKMS modules for Ubuntu Studio builds" [Critical, Fix Released] https://launchpad.net/bugs/1983521 | 19:21 |
| vorlon | sure did | 19:22 |
| vorlon | but we also don't usually skip the autopkgtests | 19:23 |
| sil2100 | vorlon: oh well, this only confirms that rushing things in is usually a bad idea! I missed that as well somehow, not sure how | 19:23 |
| tsimonq2 | Lesson learned | 19:24 |
| vorlon | is it though? some of us are experienced enough to have learned this lesson several times over ;) | 19:24 |
| bdmurray | Why isn't this in Kinetic? That's not documented in the bug either. | 19:26 |
| bdmurray | Or not well "This bug does not affect 22.10+" | 19:27 |
| bdmurray | Oh because the seed was modified before the release. | 19:27 |
| vorlon | yes | 19:27 |
| -queuebot:#ubuntu-release- Builds: Ubuntu Studio DVD amd64 [Jammy 22.04.1] has been updated (20220804.1) | 19:28 | |
| -queuebot:#ubuntu-release- Unapproved: accepted livecd-rootfs [source] (jammy-proposed) [2.765.10] | 19:29 | |
| bdmurray | I forget was there any recent discussion about bug 1921862? | 19:30 |
| ubottu | Bug 1921862 in livecd-rootfs (Ubuntu Jammy) "Call germinate with -updates" [High, Triaged] https://launchpad.net/bugs/1921862 | 19:30 |
| vorlon | we pointed out in passing that this would've fixed the root issue | 19:30 |
| tsimonq2 | not the first time I've learned such a lesson I'm sure (I was away for a year or two as well), but I will note the various stop-gaps and chances for review... we definitely should not have skipped the autopkgtest | 19:31 |
| tsimonq2 | This will at least stay fresh in my memory and I won't make the same mistake anytime soon. More verbose bug reports, don't skip the autopkgtests :) | 19:31 |
| tsimonq2 | vorlon: Thank you very much for identifying the issue and following up on it. | 19:32 |
| bdmurray | I think Steve's "experienced enough" comment might have been addressed at the release team | 19:32 |
| tsimonq2 | Ah, re-reading it I got the same message. Even in that case, I still feel some responsibility as the patch owner and apologize for the extra work. | 19:42 |
| -queuebot:#ubuntu-release- New binary: nginx [amd64] (kinetic-proposed/main) [1.22.0-1ubuntu1] (ubuntu-server) | 19:50 | |
| vorlon | jchittum: ^^ your regression is the same regression that triggered the upload of .10 | 20:24 |
| vorlon | bdmurray: any idea why the SRU bot is in a loop on LP: #1983599? | 20:28 |
| ubottu | Launchpad bug 1983599 in livecd-rootfs (Ubuntu Jammy) "RISC-V cloud images FTBFS after recent StarFive/Nezha changes" [Critical, Fix Committed] https://launchpad.net/bugs/1983599 | 20:28 |
| jchittum | vorlon: ah, i'm seeing it in scrollback now. thanks | 20:36 |
| jchittum | vorlon: i see the package, but i don't see the commit in the git source? | 20:38 |
| vorlon | really? I'm sure I pushed | 20:39 |
| vorlon | jchittum: apparently I didn't actually. corrected now | 20:39 |
| jchittum | thanks! | 20:39 |
| bdmurray | vorlon: no and I think that's sil_2100's bot | 21:27 |
| -queuebot:#ubuntu-release- New binary: bleak [amd64] (kinetic-proposed/none) [0.14.3-1] (no packageset) | 23:29 | |
| -queuebot:#ubuntu-release- New binary: python-railroad-diagrams [amd64] (kinetic-proposed/none) [1.1.1-1] (no packageset) | 23:29 | |
| -queuebot:#ubuntu-release- New binary: gosop [amd64] (kinetic-proposed/none) [0.0~git20220512.966ec01-1] (no packageset) | 23:29 | |
| -queuebot:#ubuntu-release- New binary: archlinux-keyring [amd64] (kinetic-proposed/none) [0~20220727-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [ppc64el] (kinetic-proposed/none) [0.4.4-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gamescope [amd64] (kinetic-proposed/none) [3.11.32-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gamescope [s390x] (kinetic-proposed/none) [3.11.32-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gosop [ppc64el] (kinetic-proposed/none) [0.0~git20220512.966ec01-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [amd64] (kinetic-proposed/none) [2.1.1-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: plymouth-theme-mobian [amd64] (kinetic-proposed/none) [1.0] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [amd64] (kinetic-proposed/none) [0.2+git20220718-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [s390x] (kinetic-proposed/none) [0.2+git20220718-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: sqlalchemy-utc [amd64] (kinetic-proposed/none) [0.14.0-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [amd64] (kinetic-proposed/none) [0.4.4-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gamescope [ppc64el] (kinetic-proposed/none) [3.11.32-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gosop [s390x] (kinetic-proposed/none) [0.0~git20220512.966ec01-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: python-astropy-affiliated [amd64] (kinetic-proposed/universe) [2.1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: sop-java [amd64] (kinetic-proposed/none) [4.0.0-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [s390x] (kinetic-proposed/none) [0.4.4-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [s390x] (kinetic-proposed/none) [2.1.1-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: golang-github-shenwei356-kmers [amd64] (kinetic-proposed/none) [0.1.0-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [ppc64el] (kinetic-proposed/none) [0.2+git20220718-1] (no packageset) | 23:30 | |
| -queuebot:#ubuntu-release- New binary: gosop [arm64] (kinetic-proposed/none) [0.0~git20220512.966ec01-1] (no packageset) | 23:31 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [ppc64el] (kinetic-proposed/none) [2.1.1-1] (no packageset) | 23:31 | |
| -queuebot:#ubuntu-release- New binary: gosop [armhf] (kinetic-proposed/none) [0.0~git20220512.966ec01-1] (no packageset) | 23:33 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [armhf] (kinetic-proposed/none) [0.4.4-1] (no packageset) | 23:35 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [arm64] (kinetic-proposed/none) [0.4.4-1] (no packageset) | 23:36 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [arm64] (kinetic-proposed/none) [2.1.1-1] (no packageset) | 23:38 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [armhf] (kinetic-proposed/none) [2.1.1-1] (no packageset) | 23:38 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [riscv64] (kinetic-proposed/none) [0.2+git20220718-1] (no packageset) | 23:39 | |
| -queuebot:#ubuntu-release- New binary: eg25-manager [riscv64] (kinetic-proposed/universe) [0.4.4-1] (no packageset) | 23:42 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [armhf] (kinetic-proposed/universe) [0.2+git20220718-1] (no packageset) | 23:42 | |
| -queuebot:#ubuntu-release- New binary: gamescope [arm64] (kinetic-proposed/universe) [3.11.32-1] (no packageset) | 23:43 | |
| -queuebot:#ubuntu-release- New binary: rmtfs [arm64] (kinetic-proposed/universe) [0.2+git20220718-1] (no packageset) | 23:43 | |
| -queuebot:#ubuntu-release- New binary: gamescope [armhf] (kinetic-proposed/universe) [3.11.32-1] (no packageset) | 23:43 | |
| -queuebot:#ubuntu-release- New binary: gamescope [riscv64] (kinetic-proposed/universe) [3.11.32-1] (no packageset) | 23:49 | |
| -queuebot:#ubuntu-release- New binary: gosop [riscv64] (kinetic-proposed/universe) [0.0~git20220512.966ec01-1] (no packageset) | 23:50 | |
| -queuebot:#ubuntu-release- New binary: phosh-antispam [riscv64] (kinetic-proposed/universe) [2.1.1-1] (no packageset) | 23:53 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
