[00:09] <kgiii> I don't have time to file bugs and dig into it, but today's the 2nd day in a row that the desktop doesn't show up at least in a VM. You get to the boot screen, the one with the bird, and then it turns black and stays black. I've not had time to check on real hardware as real life just happens to coincide with this turn of events. I'll have time again on Monday, for what it's worth. The VM is multiple versions of VIrtualBox. 
[00:09] <guiverc> thanks for letting us know kgiii, appreciated.  (and also :(  I've not booted lunar last couple of days so wasn't aware)
[00:11] <kgiii> I was trying to do a quick test and bumped into this. If it's still an issue on Monday, I'll go ahead and try on real hardware. I want to eliminate me as the problem, but it's multiple versions of VirtualBox on two separate bits of hardware. So, that mostly eliminates me - mostly...
[00:13] <guiverc> I'll zsync a lunar & do a quick boot on bios & uefi boxes shortly kgiii 
[00:16] <arraybolt3> RikMills: Sorry, was gone for a while, back now.
[00:17] <arraybolt3> RikMills: It's xdg-desktop-portal-lxqt that needs help.
[00:17] <arraybolt3> Lemme see if it still needs help or not...
[00:17] <arraybolt3> Yeah, it does.
[00:18] <arraybolt3> RikMills: https://git.lubuntu.me/Lubuntu/xdg-desktop-portal-lxqt-packaging It's just a no-change rebuild, but I don't have the upload rights.
[00:25] <kgiii> Sounds like a plan to me guiverc. I'll at least try to check chat.
[00:26] <arraybolt3> Currently testing for Secure Boot problems myself. kgiii: What version of VBox are you on, 6 or 7?
[00:26] <kgiii> Tested in both @arraybolt3.
[00:26] <arraybolt3> Now I see where you said that :P Crud.
[00:27] <arraybolt3> Also, what host OS?
[00:27] <kgiii> Lubuntu 22.04 and LMDE 4.
[00:27] <arraybolt3> Heh, guess what. Exact same problem happening in Virtual Machine Manager (QEMU).
[00:27] <arraybolt3> Host Ubuntu Unity 22.10. I'm gonna say this is very bad.
[00:28] <arraybolt3> I wonder if yesterday's LXQt stuff caused this.
[00:28] <kgiii> Oh, good. It's not just me. It has been like this for two days, but I lacked time to even really commit to commenting yesterday.
[00:29] <arraybolt3> Hmm, if it's been two days of this, then I *bet* it's not LXQt. It dawns on me all the LXQt work is going to be stuck in -proposed.
[00:29] <arraybolt3> OK, I can get to a TTY, that's good.
[00:29] <kgiii> Yesterday's build and today's build. 
[00:29] <arraybolt3> Tar, I can't even sign into the TTY XD
[00:30] <arraybolt3> Usually you can do something like "lubuntu" as the username and a blank password and it will let you in. No such luck here.
[00:30] <kgiii> LOL It's above my pay grade!
[00:30] <RikMills> arraybolt3: https://launchpad.net/ubuntu/+source/xdg-desktop-portal-lxqt/0.3.0-0ubuntu4
[00:31] <arraybolt3> RikMills: Thank you kindly!
[00:31] <RikMills> np
[00:32] <arraybolt3> kgiii: Can reproduce in both UEFI and BIOS modes using QEMU+VMM and raw QEMU. We have ourselves a bug. Guess it's reporting time.
[00:33] <arraybolt3> Might be interesting to know if anything *other* than Lubuntu suffers from this.
[00:34] <arraybolt3> zsyncing now, but my Internet isn't the speediest so I may be a while before managing to report results.
[00:34]  * guiverc is still waiting for ISO to complete write to thumb-drive...
[00:34]  * guiverc means the first lubuntu iso too ^
[00:34] <kgiii> I am not set up to do so and really need to get back to family. Thus my not digging into it yesterday or today. I just figured I'd better mention something today.
[00:34] <arraybolt3> kgiii: No problem, thanks for letting us know!
[00:35] <guiverc> you go back to family kgiii , you've reported it here & we'll follow up .  Thanks again !! for reporting it !
[00:35] <arraybolt3> May your day/night/whatever it is over there go well!
[00:35] <kgiii> I can try swinging back by after everyone's gone to bed. (I don't get that much time with my kids. They're adults and whatnot now.)
[00:35] <tsimonq2> What if anything caches the wallpaper? And how?
[00:35] <tsimonq2> This is not our 1st time having transient issues with that
[00:35] <arraybolt3> tsimonq2: Hmm? Not sure how that would affect things, it's not just the wallpaper failing to show up, it's the whole entire desktop.
[00:35] <tsimonq2> Thanks kgiii :)
[00:36] <arraybolt3> No panel, no notifications, no icons, nothing. Just black and frozen.
[00:36] <tsimonq2> Wait what lol
[00:36] <arraybolt3> The ISO is bricked :D
[00:36] <tsimonq2> Maybe we needed an NCR after all...
[00:36] <tsimonq2> Is there any change after upgrading to the packages you rebuilt?
[00:37] <arraybolt3> tsimonq2:Not sure, but all of the packages are stuck in -proposed.
[00:37] <arraybolt3> None of the LXQt work should be able to cause this.
[00:38] <arraybolt3> Also I think kgiii was explaining that this started happening before I uploaded things.
[00:38] <tsimonq2> But likely after I Uploaded things
[00:39] <arraybolt3> oh
[00:39] <tsimonq2> arraybolt3 @arraybolt3:libera.chat:  Do you have the capacity to drive this to a fix right now or should I stop what I'm doing?
[00:40] <arraybolt3> I don't even know what the problem is for sure so I can't say.
[00:40] <arraybolt3> I know the symptoms, but not the underlying cause.
[00:40] <arraybolt3> Right now I'm debugging. If I can figure out the cause, then yeah, I can probably handle this, but idk for sure.
[00:40] <arraybolt3> (Actually rn I'm waiting for a download to help with debugging.)
[00:42] <arraybolt3> tsimonq2: Does lxqt-build-tools do anything except help with building? Like is it installed on the main system?
[00:42] <arraybolt3> I wouldn't expect it to be able to cause this.
[00:43] <tsimonq2> I would have to check to see if it is installed on the system, but mostly it is used for building
[00:43] <arraybolt3> Currently my plan is to test an Ubuntu Desktop ISO and see if it works. If so, then the next step is to get logs, but since I can't even log into a TTY, that means the next step will be to (yikes) customize the ISO to plant a backdoor.
[00:44] <arraybolt3> (read: replace /etc/shadow with something that gives me a known root password)
[00:44] <arraybolt3> (Obviously I'll be doing this purely locally, no uploads will happen to put anything like that in public. It's just so that I can get in and get logs.)
[00:46] <tsimonq2> ...why can't you log into a TTY?
[00:46] <tsimonq2> Also, we would have heard about it already if it affected Ubuntu Desktop I think :)
[00:47] <arraybolt3> Dunno. I can get to a TTY, but it refuses anything I put into it.
[00:47] <arraybolt3> Usually I type "lubuntu" as the username, give a blank password, and I'm in. Now it just says "Login incorrect" every time I do that.
[00:47] <arraybolt3> And no other username:password combos are doing the trick (at least none of the ones I've tried).
[00:47] <tsimonq2> Uh... what?
[00:47] <arraybolt3> I tried using lubuntu or ubuntu as the user, "ubuntu", "", "lunar", etc. as the password, none of them let me log into the ISO.
[00:47] <tsimonq2> That points to a change in Casper
[00:48] <arraybolt3> I'm very much wondering if the two are related.
[00:48] <tsimonq2> That is severe breakage 
[00:48]  * arraybolt3 looks at Casper in LP
[00:48] <tsimonq2> I'll stop what I'm doing to come help - either someone PEBKACed hard or something is screwed up
[00:49] <arraybolt3> Last change in Casper was in early December...
[00:49] <arraybolt3> I still have an older Lunar ISO that was working that I can test out and see if it has the login problem.
[00:50] <arraybolt3> tsimonq2: OK, so the older Lunar ISO I have boots into a desktop just fine, *and* lets me log into a TTY with "lubuntu" as the user and a blank password.
[00:51] <arraybolt3> I think these two things are very likely related. Probably the live system isn't logging on, is my suspicion.
[00:52] <arraybolt3> (Regressions are fun!)
[00:56] <arraybolt3> Eickmeyer: If I don't get to it first, check the Studio ISO at your earliest convenience and see if you're affected by the above black-screen-and-cant-log-into-TTY bug.
[00:56] <arraybolt3> I wonder if maybe the difference between ISO login with Ubiquity and ISO login with Calamares is coming into play here.
[00:58] <arraybolt3> (DL of Ubuntu Desktop is about half done over here.)
[01:00] <arraybolt3> OK, so... let's see what all changed between "worked" and "doesn't work" in the manifest...
[01:02] <arraybolt3> Shoot. Last for-sure good ISO was the 26th, which no longer exists :(
[01:02] <arraybolt3> And according to kgiii, the 27th iso is likely bad.
[01:03] <arraybolt3> I guess I'll use my still-good ISO as a source then.
[01:03] <arraybolt3> (btw we might should have someone always download the newest ISO and *keep* the old ones so that we can refer to them. I might start doing that, keeping them a week or even a month back.)
[01:04] <arraybolt3> My good ISO appears to be from the 25th.
[01:04] <kgiii> Effected builds 20230127 and 20230128. And, I figured this one was important enough to report to chat. I couldn't find a way by the black screen.
[01:05] <arraybolt3> kgiii: K. btw, does anyone happen to have a 20230126 ISO? It's the last good one according to the testing tracker, but I don't have a copy and neither does the cdimage mirror.
 arraybolt3:  I have `2.8G Jan 24 17:56 lunar-desktop-amd64.iso.old`  (jammy of same date), it's not 0126 though sorry  (times likely my local & not UTC)
[01:11] <arraybolt3> Meh. That's OK. I have a 25th ISO, so hopefully that will be close enough.
[01:13]  * guiverc finally notices my lunar ISO write finished to thumb-drive... looking at times ^ it probably finished awhile ago now..
[01:16] <arraybolt3> Very interesting. The Ubuntu Desktop ISO makes me create a user account... to log into the ISO?
[01:17] <RikMills> wtf?
[01:17] <arraybolt3> Once I finish that, *then* I'm able to get into a live session. After making a user account and creating a password. Only after that do I get the installer screen asking if I want to try or install Ubuntu.
[01:17] <arraybolt3> I think we have our bug.
[01:18] <arraybolt3> So now the question... can I log into a TTY before doing the setup? And what about after?
[01:18] <arraybolt3> I cannot log into a TTY before doing the user account setup!
[01:19] <arraybolt3> https://imgur.com/a/akWJe8u <-- the setup screen
[01:20] <guiverc> yeah I can't login to lubuntu live on uefi boot (bare metal).. let me know bug when filed please arraybolt3  (in case I miss it)
[01:20] <RikMills> what date for the ubuntu iso?
[01:20] <arraybolt3> 20230128.
[01:21] <arraybolt3> (From "strings lunar-desktop-amd64.iso | less".)
[01:21] <RikMills> so not the one in the 'current' folder
[01:21] <arraybolt3> Once I finish the setup process, "ubuntu" still won't let me log in, but if I use the username and password made during the setup process, I can get in through a TTY.
[01:21] <arraybolt3> Today is the 28th.
[01:21] <arraybolt3> Over here at least.
 nah it's the 29th @arraybolt3  :)
 /me was too slow for your last correction...
[01:22] <arraybolt3> @guiverc: Timezone differences then. It's 7:22 PM here, 28th.
[01:22]  * arraybolt3 hates time :P
[01:23] <RikMills> http://cdimage.ubuntu.com/ubuntu/daily-live/current/
[01:23] <guiverc> be glad we don't have a blue box and can thus travel in time... and have to deal with the timey-wimey nature of space/time....
[01:23] <RikMills> not even this year in 'current'
[01:23] <arraybolt3> RikMills: It might not be the one in the current folder, since the current folder says it was last modified on the 18th and the pending folder says it was last modified on the 28th.
[01:23]  * guiverc assumes you watch, or have watched doctor who...
[01:24] <arraybolt3> guiverc: Heh, never seen it actually, but I've definitely seen time-travel related movies before, it's a mess :P
[01:24] <RikMills> arraybolt3: the actual iso in there is timestamped 2022-12-09 08:17 
[01:24] <RikMills> eek
[01:24] <arraybolt3> RikMills: Yikes.
[01:25]  * arraybolt3 wishes I had gigabit fiber so I could test stuff this quickly... maybe it's about time I start using a cloud machine for some of this stuff
[01:25] <RikMills> the current symlinks only get updates when some automatic QA passes IIRC
[01:25] <RikMills> which is enforced for the main ubuntu iSO
[01:25] <RikMills> but skipped for flavours
[01:26] <arraybolt3> OK so it's quite obvious that something is dreadfully wrong with the login system on the ISO, so I guess it's manifest comparison time.
[01:26] <RikMills> have fun
[01:26]  * RikMills heads to sleep
[01:27] <arraybolt3> RikMills: o/
[01:27] <arraybolt3> tsimonq2, guiverc: Guess it's up to the three of us to save the world :D
[01:28]  * guiverc isn't a dev [thankfully]; so its up to the two of you :)
[01:29] <arraybolt3> :P You think we'd survive for even a week without testers? Were it not for kgiii I'm certain none of us would have noticed this until we were in the thick of something else.
[01:29]  * guiverc still has to learn about SHIM & booting; my new boxes' lunar won't boot :(
[01:29] <arraybolt3> And I have no clue what's wrong with shim since I couldn't even boot the lunar ISO :D
[01:30] <guiverc> & yep regards kgiii & QA-testers !  (testing is mostly @Leokolb & kgiii these days; I'm mostly elsewhere & occasional tester)
[01:31] <guiverc> thanks arraybolt3 (shim issue isn't lubuntu specific; 15.7 change/upgrade)
[01:41] <guiverc> likely obvious; but same failure on metal with a BIOS box as before (sony vaio uEFI & hp dc7700 BIOS); lunar iso cannot login & isn't usable
[01:43] <arraybolt3> Shoot. apt list --installed gives me data in a different order than the manifest :D
[02:01] <arraybolt3> tsimonq2: OK, so I got a list of everything interesting that changes that I could see (albeit my way of detection was extremely rudimentary and wonky), and I got the following list of possible suspects: fwupd (unlikely), lsb-base, snapd, systemd, sysvinit-utils, and udev. These all changed versions between the 25th ISO and the 27th ISO.
[02:01] <arraybolt3> I currently am guessing it's either systemd or I missed something.
[02:02] <Eickmeyer> arraybolt3: Studio ISOs are failing to build because Debian introduced a packaging bug which has been resolved, but now it's failing to migrate as it's become a main inclusion issue, so cannot test.
[02:03] <arraybolt3> Eickmeyer: No problem, it's not Lubuntu-specific (it's affecting the main Desktop ISO too).
[02:03] <Eickmeyer> Packaging bug was in Pipweire.
[02:03] <arraybolt3> Something got horribly messed up with login. I think I should file a bug pronto and pop my head into -release.
[02:03] <guiverc> arraybolt3, let me have bug ID & i can add to QA-test reports & close em...
[02:04]  * guiverc wasn't sure what to file under, was hoping you'd work that out :)
[02:10] <kgiii> LOL I was kinda hopeful that Leokolb would notice and report it. They did not. I'd have no clue what to file this under and I figured it'd take me a few hours to figure all that out.
[02:11] <Eickmeyer> livecd-rootfs
[02:11] <Eickmeyer> At least, that's where I'd start.
[02:11] <guiverc> kgiii, if unsure, just file against ubuntu (filed online) & state you don't know, it'll get changed once someone is aware of a correct (or nearer) package
[02:12]  * guiverc hasn't as I'm aware arraybolt3 was looking at it..  will file if i need to, just to close reports as failures
[02:13] <Eickmeyer> I'm pretty sure the package that needs to be filed against is livecd-rootfs.
[02:13] <kgiii> Yeah, I probably could have done that. I'll check chat tomorrow to see if there's a big ID. You can't file a failed report without a bug ID, so I filed no report yesterday. I filed no report today, either.
[02:15] <guiverc> (a ubuntu bug filed incorrectly won't likely gain much attraction, but when linked to the ISO tracker - it'll get noticed soon enough my reasoning..)
[02:16] <arraybolt3> guiverc: https://bugs.launchpad.net/ubuntu/+bug/2004092
[02:16] -ubot93:#lubuntu-devel- Launchpad bug 2004092 in Ubuntu "Live ISO login is utterly broken, package causing problem is currently unknown" [Undecided, New]
[02:17] <arraybolt3> I'll report this against the Ubuntu Desktop ISO.
[02:17] <arraybolt3> In the QA tracker.
[02:17] <Eickmeyer> arraybolt3: Did you see that it needs to go against livecd-rootfs?
[02:17] <tsimonq2> Rik covered my initial suspicion, which is "hey, did it pass the automated testing?"
[02:17] <guiverc> thanks arraybolt3 ; greatly appreciated (& filed against Desktop will get the most attention; the lubuntu additions adding more weight..)
[02:17] <arraybolt3> Eickmeyer: No, I did not.
[02:17] <Eickmeyer> (filing against Ubuntu itself is never the answer, but a temporary holding space)
[02:17] <arraybolt3> I had no idea livecd-rootfs could be the problem, lemme check my diff...
[02:18] <tsimonq2> arraybolt3: If this is affecting the main ISO, I will start there. Everyone on the Desktop Team has a weekend right now.
[02:18] <tsimonq2> Eickmeyer: The last update to livecd-rootfs was on the 10th, that doesn't make a whole lot of sense.
[02:18] <guiverc> (Eickmeyer, that was my intention against filing against ubuntu; better than ignoring my point)
[02:18] <tsimonq2> Also, live user handling is done in Casper.
[02:19] <Eickmeyer> tsimonq2: Good point.
[02:19] <tsimonq2> ...which was updated at an even older date. Mmkay.
[02:19] <arraybolt3> Eickmeyer: Thanks, changed to livecd-rootfs.
[02:19] <Eickmeyer> Either way, filing against Ubuntu itself is *never* the solution.
[02:20] <Eickmeyer> tsimonq2: That's when it was updated, but when did it migrate? Things have been taking forever to migrate lately.
[02:21] <tsimonq2> Eickmeyer: https://launchpad.net/ubuntu/+source/livecd-rootfs/2.796/+publishinghistory
[02:21] <tsimonq2> 10th is when it migrated
[02:21] <tsimonq2> Technically, it was uploaded on the 9th.
[02:21]  * arraybolt3 wonders what /etc/shadow looks like on a borked ISO
[02:22] <Eickmeyer> That means that the earliest this would've been in production would've been the 12th GMT since that publication date is based on PST.
[02:23] <arraybolt3> In Ubuntu Desktop, there isn't even an "ubuntu" user in /etc/shadow.
[02:23] <tsimonq2> Well, we've already established that it's a red herring.
[02:23] <tsimonq2> (As for what the actual cause is? No clue yet, just looking now.)
[02:24] <Eickmeyer> FWIW, ubuntu-desktop-installer is a snap.
[02:25] <arraybolt3> tsimonq2: What creates the initial "ubuntu" user or equivalent on the ISOs?
[02:26] <tsimonq2> For the third time, Casper. ;)
[02:26] <arraybolt3> Hmm. Wonder when Casper migrated.
[02:26] <tsimonq2> Already been there.
[02:26] <Eickmeyer> Casper has been sitting for over a month.
[02:29] <arraybolt3> Anything that would cause a Casper bug to have this sort of delayed response (something akin to a y2k bug)?
[02:30] <tsimonq2> adduser is reporting that GID 999 already exists
[02:31] <arraybolt3> There's an "ubuntu" folder in /home.
[02:35] <arraybolt3> Hey... I can look at the logs since I'm in the Ubuntu Desktop!
[02:40] <tsimonq2> adduser fails because something else exists with GID 999
[02:40] <tsimonq2> and systemd uses 995, 996, 997, and 998
[02:40] <arraybolt3> I can't see anything weird in /var/log/syslog.
[02:41] <arraybolt3> I see where I create the "user" user account, but nothing about casper and user account creation weirdness before that.
[02:41] <arraybolt3> Oh wait... casper.log has interesting things.
[02:42] <tsimonq2> I'm guessing it has that adduser --uid 999 ubuntu fails because 999 is already being used :P
[02:42] <arraybolt3> https://termbin.com/48l8h
[02:42] <arraybolt3> Yep, looks like.
[02:42] <arraybolt3> So something's creating UID 999 before casper gets a chance.
[02:42] <tsimonq2> What I'd like someone to tell me (before I figure it out myself) is where 999 is being used. :P
[02:43] <arraybolt3> :P
[02:43]  * arraybolt3 cracks knucles and tries to to a recursive `stat -t`
[02:45] <arraybolt3> /etc/apparmor.d/abstractions/p11-kit
[02:45] <arraybolt3> tsimonq2: ^
[02:45] <arraybolt3> Check the UID on that file.
[02:45] <tsimonq2> Looking.
[02:46] <arraybolt3> Meh, nevermind, I looked at the wrong field.
[02:46] <arraybolt3> It's the 5th field, not the 2nd that I need to look at :P
[02:49] <arraybolt3> OK this is going to take a *while* to run (turns out 'stat' takes a significant amount of runtime).
[02:49] <kgiii> To make this abundantly clear (skimming the text while I was away) - this was *not* a problem until yesterday's daily build - on *my* end.  I have no idea when it was introduced, but I am 100% certain that this was not a problem in the daily build from Jan 26th.
[02:49] <tsimonq2> systemd-jounrla
[02:50] <tsimonq2>  * systemd-journal
[02:50] <tsimonq2> That's what uses 999
[02:50] <arraybolt3> tsimonq2: OK, how did you figure this out, I could not find it for the life of me.
[02:50] <tsimonq2> This is a systemd bug, updating the BR as such
[02:50] <tsimonq2> cat /etc/group | grep 999
[02:50] <tsimonq2> :P
[02:51] <arraybolt3> Huh. It's not in /etc/passwd or /etc/shadow :P
[02:52] <tsimonq2> Self-assigning. I have upload access to casper, I'll take the blame for this change, heh.
[02:52] <arraybolt3> Oh. "adduser: The **GID** 999 is already in use."
[02:52] <arraybolt3> I was looking for UID the whole time.
[02:52] <tsimonq2> yes :)
[02:53] <arraybolt3> Heh, so it *was* systemd! That was one of my potential candidate packages.
[02:53] <arraybolt3> And the one I suspected the most.
[02:54] <guiverc[m]> kgiii: you're quoted on https://bugs.launchpad.net/casper/+bug/2004092 as saying "Effected builds 20230127 and 20230128." which matches your 'yesterday's daily build' last comment here ^
[02:54] -ubot93:#lubuntu-devel- Launchpad bug 2004092 in casper (Ubuntu) "casper fails to add any users because GID 999 is already taken" [Undecided, Confirmed]
[02:54] <arraybolt3> I wonder how the GID change from 999 to 1000 is going to make thngs behave. Does anything depend on the UID being 999?
[02:55] <tsimonq2> nothing reasonable should
[02:56] <arraybolt3> Also, would that change possibly influence the created UID inside a newly installed system? Just me thinking of where things could go wrong.
[02:56] <kgiii> The missus and I won the game of pinochle. So, there's that. And thanks for letting me know, guiverc[m]. Do you have a bug ID at this time - and thanks for filing it.
[02:56] <arraybolt3> (I'm guessing the answer is probably "no" since the UID in question won't be in the squashfs since casper makes it automatically.)
[02:56] <arraybolt3> kgiii: \o/
[02:57] <tsimonq2> don't touch the BR description, I'm editing it ;)
[02:57] <arraybolt3> +1
[02:57] <tsimonq2> +1 or ack?
[02:57] <tsimonq2> :P
 nah Aaron filed it, bug ID was used ^  (I quoted you in my comment on bug report)   https://bugs.launchpad.net/casper/+bug/2004092
[02:57] -ubot93:#lubuntu-devel- Launchpad bug 2004092 in casper (Ubuntu) "casper fails to add any users because GID 999 is already taken" [Undecided, Confirmed]
[02:58] <kgiii> I see it now... D'oh... I'm going to go file today's testing reports - so it'll end up confirmed and all that. Gimme a few minutes.
 sorry i intended to paste 200409 only kgiii
[02:58] <arraybolt3> tsimonq2: I meant thumbs-up, so I guess ack :)
[02:58] <arraybolt3> As in, me no touch bug report :P
[03:02] <kgiii> I see guiverc[m]  already confirmed 'em, but I've tossed two more reports into the mix. I did not touch the bug report. 
[03:02] <guiverc[m]> thanks kgiii 
[03:03] <kgiii> In the background, I have the 24 hour Daytona race on the TV. The fireworks just went off, so it's 22:00...
[03:04] <kgiii> And, no worries @guiverc[m]. When nobody else picked up on it, I figured I'd better at least make time to point it out in chat. It's a bit of a show-stopper kinda bug. 
[03:05]  * guiverc smiles ...   (at daytona in background)..   quoting you was for 'credit' sake; you discovered issue !
[03:05] <guiverc> (the ISO detail is also VERY USEFUL diagnostic detail!)
[03:08] <kgiii> Bathurst wasn't all that long ago. I watched that one flag to flag, like I usually do. 
[03:08] <kgiii> Anyhow, I don't do it for credit. I do it 'cause someone's gotta do it.
 I watch imsa after thursaday on the youtube channel so please no spoilers
[03:10] <kgiii> I will not spoil it - except to say it's well worth watching, so far. I use a VPN and watch live. I pay for NordVPN but only use the browser extension aspect of it. I just pick a server in the EU and watch at imsa.tv while it's live. But, there will be no spoilers from me - except to say it's a good race so far and you'll love it when you do watch it.
[03:12] <tsimonq2> https://launchpad.net/ubuntu/+source/casper/1.478
[03:13] <arraybolt3> Wow, quick turnaround!
[03:13] <arraybolt3> Gonna spin a new ISO for us maybe?
[03:13] <kgiii> Will that be in tomorrow's build?
[03:14] <tsimonq2> Actually, I forgot a line. 1.479 is the best one
[03:14] <tsimonq2> Well, my intention is to stick around (or at least be pingable) until this migrates.
[03:15] <arraybolt3> nice, will test when ready if I'm awake, which I will try to be.
[03:16] <guiverc[m]> & kgiii it was you that started that 'ball rolling' for the fix; well done & thanks (again)   .. 
[03:16] <kgiii> Everyone else is headed to bed, more or less. So, I can stick it out for a bit. 
[03:18] <kgiii> @guiverc[m] When it was here in a second build and nobody had reported it, I figured I better say something in the dev chat. I don't have the *hours* it'd have taken me to figure out what to file against, at least not until Monday. Fortunately, @tsimonq2 managed to figure it out quickly and seemingly has a fix in the works.
[03:20] <tsimonq2> Note: it could take up to four hours for this to migrate, unfortunately. (And another couple of hours before an ISO is ready.)
[03:20] <tsimonq2> arraybolt3: Do you have access to rebuild ISOs?
[03:20] <guiverc> & thanks also to Eickmeyer, RikMills, & others including arraybolt3 for much of the 'heavy lifting', tsimonq2 for the uploading etc... Thanks all
[03:20] <arraybolt3> tsimonq2: I have no clue, how would I find out?
[03:21] <guiverc> i can kick of rebuild (if around)
[03:21]  * guiverc clarifies, rebuild is vague, kick of iso build
[03:21] <tsimonq2> guiverc: Please add arraybolt3 here, which gives him this access: https://launchpad.net/~lubuntu-product-managers/+members\
[03:21] <tsimonq2>  * guiverc: Please add arraybolt3 here, which gives him this access: https://launchpad.net/~lubuntu-product-managers/+members
[03:22] <tsimonq2> (Even if you can get it this time, there will come a time where he'll need to.)
[03:22] <arraybolt3> Thanks, will use rarely and carefully.
[03:23] <guiverc> I'll only add rights on other Council members agreement (ideally 2 others)
[03:23] <tsimonq2> ¯\_(ツ)_/¯ fair enough :)
[03:24]  * arraybolt3 pings te
[03:24] <tsimonq2> Speaking of that, we're about a month overdue for an LC election
[03:24] <arraybolt3> So that was enter, not shift.
[03:24] <tsimonq2> hah
[03:25] <tsimonq2> In any case, the Britney logs are here: https://people.canonical.com/~ubuntu-archive/proposed-migration/log/lunar/2023-01-29/
[03:25] <tsimonq2> Here's the issue... ubiquity is a reverse dependency. Which means we're waiting on those autopkgtests to pass.
[03:26] <tsimonq2> Which means we're waiting on ubiquity to have working autopkgtests...
[03:26] <tsimonq2> Which means it could be a Monday thing. :(
[03:26] <arraybolt3> Eh, we'll live.
[03:27] <arraybolt3> We just won't live ISO.
[03:27] <tsimonq2> Aaaaaaaaaaaactually... it looks like ubiquity migrated 5 hours ago, hopefully with working autopkgtests :)
[03:27] <tsimonq2> ha ha ha.
[03:27] <tsimonq2> https://autopkgtest.ubuntu.com/packages/u/ubiquity/lunar/amd64
[03:28] <tsimonq2> Disregard my earlier comments re: ubiquity's working autopkgtests (or lack thereof)
[03:28] <kgiii> I doubt I'll be here that late. So, I can test it in tomorrow's build - unless I do stay awake.
[03:28] <tsimonq2> So, let's do the math...
[03:29] <tsimonq2> This Britney run should take us until 5-5:30 UTC, assuming autosync doesn't bring new presents (read: many new autopkgtests)
[03:29] <tsimonq2> Which means this will be picked up on the next run, sometime between 5:30-6 UTC, and that run will finish about 8:00 UTC
[03:29] <tsimonq2> May need an additional run after that if the autopkgtest queues are bogged down/Britney doesn't care about them in that run. Which means about 10 UTC
[03:29] <tsimonq2> For us Central US people, that's about 4 AM :P
[03:30] <tsimonq2> so go to bed :P
[03:30] <arraybolt3> And then two hours later the ISO build?
[03:30] <kgiii> Ol' KGIII is gonna be in bed long before that!
[03:30] <tsimonq2> ah yes, that too heh
[03:30] <arraybolt3> 4 AM I can do. 6 AM? ...might go to bed.
[03:30] <kgiii> I'm EST - located in Maine, USA. I'll be snuggled up with the wife long, long before that.
[03:31] <arraybolt3> CST here.
[03:31] <arraybolt3> I think same timezone as tsimonq2.
[03:32] <tsimonq2> I'm in Green Bay, WI
[03:32] <tsimonq2> So, Central, yeah
[03:32] <kgiii> You could watch the IMSA race. It's nighttime, so the racing is either boring or really interesting. I'll say it is interesting - as spoilers are right out of the question.
[03:32]  * tsimonq2 is really hating the 20 degree weather right now :/
[03:32] <arraybolt3> Ugh, same here, same here.
[03:32] <kgiii> LOL I just sent a few dollars via PayPal to a buddy in GB. They didn't have any beer money and were out of toilet paper.
[03:33] <tsimonq2> Green Bay is the one city where you could probably open a drivethrough bar. :P
[03:33]  * arraybolt3 shouts at the weather and tells it to behave itself the way I want it to
[03:33] <kgiii> I'm covered in snow and the temps are below freezing. I, on the other hand, absolutely love it. We average 12' of snow per year. I've seen just over 21' of snow in a season.
[03:33]  * guiverc is away; off walking dog looking for birbs etc.
[03:33] <arraybolt3> kgiii: 12 **foot**?!?!
[03:34] <arraybolt3> That sounds nightmarish. I have a hard enough time wading through a few inches.
[03:34] <tsimonq2> I would love it more if I didn't have a sedan :D
[03:34] <kgiii> If you go to Florida, they have drive through margarita and daquari stands - and drive through liquor stores.
[03:34] <tsimonq2> hahahahahahahaha
[03:34] <arraybolt3> kgiii: AKA instant death.
[03:34] <tsimonq2> that is so Florida Man
[03:34] <kgiii> I have a big ol' plow truck with a broadcast sander in the bed. Then, I have a tractor. I love the snow! I live here on purpose.
[03:35]  * arraybolt3 makes note to never live in Florida
[03:37] <kgiii> It's kinda amusing. They all have like concrete or metal poles as barriers and you can see where drunk people have smashed into them. Seriously, you can drive up and get a margarita - in like a 62 oz cup. It's not an 'open container' because they either didn't put the straw in it or because they left some of the paper on the straw that covers where you'd drink through.
[03:38] <arraybolt3> Yeah that sounds about like what I would imagine.
[03:38] <kgiii> 32 oz cup* (though they might have 64 oz cups that I've not noticed).
[03:38] <tsimonq2> Take all of those people, put them in one parking lot, turn the temperature down 40 degrees, and that's called the Lambeau Field parking lot on game day bahahahahaha
[03:39] <kgiii> They also have daquari stands. They're like an ice cream shop sorta thing, where you walk up and order from the outside.
[03:40] <arraybolt3> How do you generate a random string? Put a laptop with Vim running on it in the middle of Florida and wait for someone to try and close it.
[03:40] <kgiii> And, sorry 'bout your team in the playoffs. I'm a 49ers fan, so it has been an exciting year for me - and I don't actually normally watch too much football. 
[03:42] <tsimonq2> Yeah, I was hoping for it, we'll have to see if our quarterback stays the same next year. :/
[03:42] <kgiii> He's getting up there in age. I can't picture him playing for anyone else. 
[03:45] <tsimonq2> Honestly, they said the same thing about Brett Favre, nobody knows for sure
[03:47] <tsimonq2> Anyway... back on topic lol
[03:51]  * arraybolt3 cracks open The List
[03:51] <kgiii> LOL I believe the topic is waiting to see if the problem is fixed. I'm pretty sure I'm not gonna last that long. Someone on the other side of the planet (eg. guiverc) will know before I know, I suspect.
[03:51] <arraybolt3> I should probably verify the Focal lubuntu-update-notifier bugfix so that we can let it through on the 31st.
[03:51] <tsimonq2> I'm seeing like, one actionable Python 3.11 blocker yet, so I'll go ahead and fix that up
[03:51] <tsimonq2> That will make Britney runs much faster ;P
[03:51] <arraybolt3> Tell qtbase-opensource-src to get lost too, will ya?
[03:52] <tsimonq2> Unfortunately, they're linked together
[03:52] <arraybolt3> Also, are there any notes we should add for the Flavor Sync Meeting on the 30th?
[03:53] <arraybolt3> Currently the only flavor to leave notes other than Ubuntu itself is Cinnamon Remix.
[03:53] <arraybolt3> (Which isn't a flavor yet.)
[03:53] <kgiii> If you want a list, there are a couple of long-lived bugs. The software-properties-qt still wants to use kURL. Then, there's the lock screen. I don't suppose tonight's bug is related to the lock screen failing?
[03:54] <arraybolt3> Not the lock screen almost certainly. kURL, adding that to the list.
[03:55] <arraybolt3> After the bug verification, I guess the next thing I should be working on is banishing KDE System Settings and kde-config-updates from the Lubuntu ISO.
[03:56] <arraybolt3> Which I think the only good way to do that is with a Conflicts in lubuntu-meta since blacklisting won't work.
[03:56] <kgiii> LOL Yeah, those are probably more pressing. Though I'd say the lock-screen bug is kinda important, but maybe not. After all, how many people lock the screen in the live environment?
[03:56] <arraybolt3> It's an annoyance, but a possibly major one for sure.
[03:58] <kgiii> I should check to see if it works on an installed system. What I do is the *daily live* testing. That's my 'job'. I seldom test anything except that and I test that pretty much twice a day - every day. It was a hole that needed filling when I started participating. So, I have no idea if the lock screen works properly on an installed system.
[04:00] <arraybolt3> The lock screen has always worked on an installed system for me (on all Lubuntu 20.04 and higher systems I've used)
[04:01] <arraybolt3> oh my gosh I just remembered what led me to this whole ordeal, which was a Secure Boot dual-boot test. Glad I still have a working ISO for that.
[04:01] <kgiii> Ah... In the live environment, you can't unlock it.
[04:01] <arraybolt3> Also, unless you guys think it's pointless, I intend to start mirroring all of the daily images from cdimage on a daily basis and keeping them up to four weeks back (unless space constraints keep me from doing so). That way we always have access to manifests and whatnot, which may have helped speed things along here.
[04:02] <arraybolt3> I have a 12 TB drive that sits mostly unused except for backups.
[04:02] <arraybolt3> *all of the *development* daily images that is
[04:02] <kgiii> I use the manifest to check which files have changed. I simply keep a set of two and use diff on them both. I never have more than two, if I'm doing it right.
[04:03] <arraybolt3> Yeah we were missing a manifest for the last working ISO over here (26th), which caused me at least some Big Problems :P
[04:04] <arraybolt3> I guess though maybe keeping all the ISOs is overkill, perhaps just the manifests is enough.
[04:05] <arraybolt3> Wow. So somehow Ubuntu Desktop was set to use Japanese as the main language when I first booted it.
[04:06] <arraybolt3> At least it looked like that :P Doesn't actually seem to make typing difficult in English though.
[04:07] <arraybolt3> Anyway, interesting possible bug, but whatever, other things to test.
[04:08] <kgiii> That does sound unusual.
[04:32] <tsimonq2> arraybolt3: kde-config-updates> Honestly, after mulling it over we should probably ask Rik to demote it to a Suggests and deal with the broken buttons at a later date.
[04:32] <tsimonq2> (That is, unless the thing completely segfaults if that doesn't exist.)
[04:32] <tsimonq2> Erich will also have to add kde-config-updates to his seed as well.
[04:38] <arraybolt3> tsimonq2: If that sounds like the best course of action to you, then great.
[04:38] <arraybolt3> guiverc: Oddly, while I did run into os-prober related problems with the Jammy+Kinetic dual boot, I did **not** run into any shim issues.
[04:38] <arraybolt3> I can still boot into both operating systems after installing and fully upgrading both.
[04:39] <arraybolt3> Only tricky part is that I have to force my way into a GRUB shell and specify that the configuration file from Lunar. (Not kinetic above, Lunar. Gah.)
[04:39] <arraybolt3> Wow. I can't type.
[04:39] <arraybolt3> * Only tricky part is that I have to force my way into a GRUB shell and specify that the configuration file from Lunar should be used, then I can boot Lunar again.
[04:42] <arraybolt3> ?! somehow Jammy didn't have shim installed.
[04:43] <arraybolt3> Do newly installed Ubuntu systems fail to install shim if they detect an existing Ubuntu system with shim installed on it?
[04:44] <arraybolt3> Hmm. It wasn't installed on the Lunar one either.
[04:45] <arraybolt3> Ugh. shim and shim-signed are different. Nevermind.
[04:50] <arraybolt3> guiverc: Anyway, disregard the above noise from my confusion and typos. Currently, I have two fully-updated Lubuntu installs in one VM (one Lunar, one Jammy), with Secure Boot enabled, and both are able to be booted.
[11:54] <arraybolt3> Nearly 6 AM here and casper *just* migrated.
[12:00] <guiverc> i forget where the cron job file is that tells you when daily jobs are kicked off (ie. if there isn't much time till a normal build, it's a waste of resources..), but FYI on another box I booted lunar, updated & on reboot it started without issue..  (I wasn't willing to boot that boxes jammy & re-test its update; maybe tomorrow)
[12:39] <arraybolt3> Verification mostly passed for the lubuntu-update-notifier bugfix in Focal, just need to make sure usual updates still work, which I'll tackle on the 30th and 31st once there's hopefully some new updates.
[17:09] <tsimonq2> Does anyone have confirmation either way that the Casper update works on a Ubiquity-based flavor?
[17:09] <tsimonq2> (Have any ISOs besides ours been built in the last 6 hours?)
[17:48] <tsimonq2> Trying Ubuntu Budgie
 https://launchpad.net/ubuntu/+source/plasma-discover/5.26.5-0ubuntu2
[20:40] <kgiii> It's nice to see the quick response last night resulted in a bug fix that was effective the first time.
[21:59] <tsimonq2> That's for sure :)