[07:54] <schopin> @pilot in
[08:18] <schopin> Oh well. The sponsoring reports are down, so I'll return to my regularly scheduled program.
[08:23] <schopin> The error seems to situated between keyboard and chair. Thanks ginggs :)
[08:24] <ginggs> "something loose in cockpit"
[08:25] <schopin> It's a bird, it's a plane, it's human error.
[10:08] <nteodos> I was installing a base-files from Jammy in my Mantic installation, at the end Apt failed with some message about rm not being found (sorry, should have saved that full error) and then no command would be found. Rebooted and now get kernel panic, can't find /sbin/init though inspecting the disk from another OS proves it's there, as is everything else. Chroot also fails with ‘/bin/bash’: No such file or directory.
[10:19] <cjwatson> nteodos: installing a base-files from jammy on mantic sounds thoroughly unsupported to begin with - why?
[10:19] <cjwatson> nteodos: my guess would be that you've done something horrible with respect to the /usr move
[10:21] <nteodos> cjwatson, that was just to test front-ends to Ubuntu Pro as they are only interesting in LTS. I did it many times before without issue...
[10:21] <cjwatson> nteodos: regarding /sbin/init, it should be a symlink to "../lib/systemd/systemd" I think, so check that /lib/systemd/systemd exists; or that error might also indicate that some of its dynamically-linked libraries are missing
[10:21] <nteodos> It is a symlink to that, yes.
[10:22] <nteodos> Which exists.
[10:22] <cjwatson> nteodos: in future I would recommend just temporarily tweaking /usr/lib/os-release if you just want it to present as a different version for testing purposes.  base-files does a bunch of other stuff
[10:23] <nteodos> Thanks, after a kernel panic I'll sure stick to your suggestion.
[10:23] <cjwatson> nteodos: pretty certain that you have a missing library somewhere.  /bin/bash is an easier place to start since it has simpler linkage - it needs /lib64/ld-linux-x86-64.so.2, /lib/x86_64-linux-gnu/libc.so.6, and /lib/x86_64-linux-gnu/libtinfo.so.6
[10:23] <cjwatson> (this is from noble, but I think that should be true on mantic too)
[10:24] <nteodos>  But if nothing was uninstalled during the operation why would libraries suddenly go missing?
[10:24] <cjwatson> I could speculate about various possibilities but I prefer not to do that before knowing exactly what's missing
[10:25] <cjwatson> and without the apt output it's also pretty difficult
[10:27] <nteodos> In the chroot I can't find /lib64
[10:27] <cjwatson> lrwxrwxrwx   1 root   root      9 Mar 12 07:43 lib64 -> usr/lib64
[10:28] <cjwatson> So maybe just put that in place and try again
[10:28] <nteodos> Aha! Chroot works now.
[10:28] <nteodos> Thanks a lot Colin, let me see if I get past the kernel panic now.
[10:28] <cjwatson> I don't see why a base-files downgrade would have done that; perhaps apt also did something else
[10:28] <cjwatson> you might find evidence in /var/log/apt/term.log
[10:30] <cjwatson> and yes, not having the system dynamic linker will mean just about nothing works :)
[10:32] <nteodos> So this is term.log: https://termbin.com/2vbl
[10:32] <nteodos> :)
[10:34] <cjwatson> nteodos: 13ubuntu7 is in noble.  you said you were running mantic
[10:34] <cjwatson> nteodos: so did you install noble's base-files first and then jammy's?
[10:34] <nteodos> Ugh my bad I was indeed on Noble
[10:34] <cjwatson> nteodos: in that case that's due to the /usr move, indeed
[10:35] <cjwatson> there's stuff in /var/lib/dpkg/info/base-files.postrm related to this which I am not very much inclined to spend time unpicking given how unsupported this is in the first place
[10:36] <cjwatson> downgrading past noble's base-files is dangerous and mustn't be done
[10:36] <nteodos> Well lesson learned :P Thanks again, will reboot to Ubuntu now, hopefully with success.
[10:39] <cjwatson> though it's interesting that base-files has a postrm in Ubuntu but not in Debian
[10:41] <cjwatson> ah, because https://bugs.debian.org/1064459 hasn't landed yet
[10:41] -ubottu:#ubuntu-devel- Debian bug 1064459 in base-files "base-files: install aliasing symlinks for merged-/usr DEP17" [Normal, Open]
[10:41] <nteodosio> cjwatson, can confirm, lib64 was all that was needed
[10:46] <cjwatson> though when I took the latest Debian patch there, built Debian's base-files with it, and upgraded then downgraded a scratch container, things seemed fine - no missing symlinks
[10:46] <cjwatson> so that's probably about as much time as I have to root-cause this, sorry
[10:48] <nteodosio> I might have lost some messages during reboot but I'll check in the log later.
[10:48] <cjwatson> just me saying "though it's interesting that base-files has a postrm in Ubuntu but not in Debian"
[10:48] <cjwatson> but then I got further
[11:43] <bastif> I don't know if this was missed, could anyone look at this RFS please? https://bugs.launchpad.net/ubuntu/+source/google-android-installers/+bug/2056080
[11:43] -ubottu:#ubuntu-devel- Launchpad bug 2056080 in google-android-installers (Ubuntu) "Update version in noble to 1707406511ubuntu2" [Undecided, New]
[12:46] <schopin> @pilot off
[12:47] <schopin> @pilot out
[14:40] <bluca> heads up on https://bugs.launchpad.net/ubuntu/+source/linux-meta-azure-6.5/+bug/2038777 - as open source projects start to use KVM on Github Actions CI jobs (it was made available last month), they'll hit that crash due to a missing bugfix in the jammy azure kernel image
[14:40] -ubottu:#ubuntu-devel- Launchpad bug 2038777 in linux-meta-azure-6.5 (Ubuntu Jammy) "UBSAN: array-index-out-of-bounds (drivers/net/hyperv/netvsc.c)" [Undecided, Confirmed]
[14:40] <bluca> so that's likely going to get hotter
[14:40] <bluca> we just hit that in the systemd CI on github
[16:40] <bandali> fossfreedom_, alrighty i'll add that to my list
[17:50] <fossfreedom_> bandali: many thanks.  Hope that the debdiff on that issue is useful for you.
[18:22] <philroche> rbasak: As SRU team member on duty - would you be able to have a look at focal livecd-rootfs v 2.664.53 @ https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=livecd-rootfs ?
[18:23] <bandali> fossfreedom_, cheers, and thanks to you :) i'm sure it will be
[18:43] <rbasak> philroche: sorry I'm beyond EOD now.
[18:43] <rbasak> waveform: FYI, bug 1833322. I'm not sure what package that should belong to.
[18:43] -ubottu:#ubuntu-devel- Bug 1833322 in cloud-images "Please consider no more having irqbalance enabled by default (per image/use-case/TBD)" [Undecided, New] https://launchpad.net/bugs/1833322
[18:44] <rbasak> Sorry, wrong bug. Bug 2057822 is what I meant.
[18:44] -ubottu:#ubuntu-devel- Bug 2057822 in ubuntu-meta (Ubuntu) "Removing irqbalance disables power button on Raspberry Pi 5" [Undecided, New] https://launchpad.net/bugs/2057822
[18:48] <jbicha> rbasak: I was thinking the ubuntu-server-raspi & ubuntu-desktop-raspi metapackages unless the issue can be fixed in the kernel or something
[18:48] <jbicha> Eickmeyer: cc ^
[19:00] <tjaalton> moving to tbird snap broke openpgp, claims it can't find the private key
[19:02] <kenyon> I've been trying to mirror ddebs.ubuntu.com <https://wiki.ubuntu.com/Debug%20Symbol%20Packages> with apt-mirror, and discovered that it basically doesn't work at all due to severe rate-limiting on ddebs.ubuntu.com. any way we can get that rate limiting config changed, or document what the rate limiting config is so that we can work around it?
[19:04] <jbicha> tjaalton: could you file a bug on Launchpad with details?
[19:04] <tjaalton> jbicha: against the deb?
[19:05] <jbicha> tjaalton: so that it shows up at https://bugs.launchpad.net/ubuntu/+source/thunderbird/
[19:05] <tjaalton> ok
[19:08] <tjaalton> 2057835
[19:19] <jbicha> tjaalton: there is a snap permission turned off by default named gpg-keys. Is that turned on for you? You can enable it in GNOME Settings > Apps > Thunderbird Mail
[19:21] <tjaalton> jbicha: ok, didn't fix it though
[19:22] <jbicha> tjaalton: are you using a smart card or other device to store your gpg keys?
[19:22] <tjaalton> no
[19:22] <jbicha> could you add those notes to the bug report? :)
[19:25] <tjaalton> done
[19:25] <tjaalton> not a huge deal for me atm, but the popups are annoying :)
[19:26] <jbicha> I don't know anything about that, I use Evolution :)
[19:26] <jbicha> actually I use Gmail but sometimes I use Evolution
[19:29] <rbasak> jbicha: sure. I guess it depends on the root cause which isn't obvious to me.
[19:30] <rbasak> It seems like a bug to me that it doesn't work without irqbalance. Maybe a firmware/kernel/dtb thing?
[19:37] <Eickmeyer> jbicha: I don't see why it can't be fixed in the metas. FWIW, I can't test, my case doesn't have a power button.
[19:39] <Eickmeyer> jbicha, rbasak: I can see a use case for not having irqbalance on server, but for desktop I can see a use case. But on the other hand, a kernel bug perhaps? Something seems off about needing it.
[19:40] <Eickmeyer> Not that it matters for rpi as it's out-of-scope, but (Studio hat on) irqbalance can mess with lowlatency processes. But (Edubuntu hat on) it doesn't really matter, but is it necessary? Maybe a stopgap until a kernel bug is fixed?
[19:41]  * Eickmeyer is thinking out-loud
[19:42] <rbasak> Eickmeyer: have you seen bug 1833322? There's extensive discussion on pros/cons of irqbalance there.
[19:42] -ubottu:#ubuntu-devel- Bug 1833322 in cloud-images "Please consider no more having irqbalance enabled by default (per image/use-case/TBD)" [Undecided, New] https://launchpad.net/bugs/1833322
[19:42] <Eickmeyer> rbasak: I'll admit, I skimmed it. I definitely need to read it more in-depth.
[20:01] <Eickmeyer> rbasak, jbicha: Reading all of that and reading bug 2057822 through that lens, I'm highly skeptical of irqbalance being the culpret, more likely coincidental with a kernel bug. I will have to test for the same kernel message here.
[20:01] -ubottu:#ubuntu-devel- Bug 2057822 in ubuntu-meta (Ubuntu) "Removing irqbalance disables power button on Raspberry Pi 5" [Undecided, New] https://launchpad.net/bugs/2057822
[20:45] <vorlon> jbicha, Eickmeyer: anyway, reassigning to ubuntu-raspi-settings, which is seeded in both Ubuntu and Edubuntu
[20:46] <Eickmeyer> vorlon: I might be reassigning it to linux if I can't reproduce.
[20:46] <vorlon> no
[21:15] <Eickmeyer> rbasak, jbicha, vorlon: Unable to reproduce: https://bugs.launchpad.net/ubuntu/+source/ubuntu-raspi-settings/+bug/2057822/comments/1
[21:15] -ubottu:#ubuntu-devel- Launchpad bug 2057822 in ubuntu-raspi-settings (Ubuntu) "Removing irqbalance disables power button on Raspberry Pi 5" [Undecided, Incomplete]
[21:35] <rbasak> Eickmeyer: thank you for investigating!
[21:35] <Eickmeyer> rbasak: My pleasure. :)
[21:45] <arraybolt3> Who is the person who most frequently works on GRUB? I just ran into a bug that I think is pretty severe that only occurs when /boot is on BTRFS that I would like to address (will be filing a bug report soon).
[21:45] <arraybolt3> TL;DR: GRUB_TIMEOUT in /etc/default/grub is ignored and the timeout is always set to 30 if /boot is on BTRFS and the system is UEFI-based.
[21:46] <arraybolt3> Meaning the user always sees a boot menu on every single boot.
[21:46] <arraybolt3> This has been happening for a long time, but only just now is it something I'm pursuing fixing because it is about to affect an OEM I work with (KFocus).
[21:47] <arraybolt3> anyway, if there's someone I can ping about this, please let me know.
[21:49] <Eickmeyer> arraybolt3: I think you might want juliank, if not he can point you in the right direction.
[21:49] <vorlon> arraybolt3: juliank and mkukri.  But the issue there is that grub doesn't have read-write support for btrfs, therefore it cannot save state to a grub env file, therefore we cannot guarantee that the user will ever be able to get to the boot menu in the case of a problem unless we always display it with a timeout
[21:50] <arraybolt3> ahh
[21:50] <vorlon> we should respect timeout /values/, but we should not let grub get into a state where the menu is not displayed
[21:51] <vorlon> because UEFI can't detect a shift key being held down
[21:51] <arraybolt3> UEFI can detect a rapid press of Esc during bootup though?
[21:51] <arraybolt3> That's how I usually get to the GRUB menu.
[21:51] <vorlon> it's racy
[21:51] <mkukri> arraybolt3 i believe the issue is detecting modified keys specifically
[21:51] <arraybolt3> as in a code bug, or just "it's hard to hit"?
[21:52] <Eickmeyer> vorlon: ime, it doesn't display *at all* unless a) there's another OS installed, or 2) it didn't complete a previous boot.
[21:52] <vorlon> "it's hard to hit"
[21:52] <vorlon> Eickmeyer: when /boot is on a filesystem we can write to, yes
[21:52] <mkukri> i think it's hard to hit. also sometimes you hit it at the wrong time and you get the console instead of the menu
[21:52] <mkukri> and there is like a 100ms window
[21:52] <arraybolt3> that makes sense. You do have to hit it immediately after the firmware screen vanishes.
[21:52] <vorlon> Eickmeyer: but if it's btrfs there's no way to detect "it didn't complete a previous boot"
[21:52] <Eickmeyer> I see.
[21:52] <arraybolt3> Also it only drops to a console if you mash Esc over and over.
[21:53] <arraybolt3> If you tap it once at just the right time, it works. It took some training before I figured out how to get it to work though.
[21:53] <mkukri> i dont think "grub training" should be a requirement for using ubuntu :)
[21:53] <arraybolt3> agreed
[21:53] <juliank> We have bug reports for this and they're closed won't fix I believe
[21:54] <mkukri> i guess if someone really cares maybe we could use the uefi varstore for storing this information instead of the grubenv?
[21:55] <vorlon> is there support for that currently in grub?
[21:55] <vorlon> I do like the idea
[21:56] <arraybolt3> There's also the possibility of putting the GRUB environment file on the ESP perhaps?
[21:56] <mkukri> i dont think there is but grub surprising me wouldnt be the first time
[21:56] <arraybolt3> not sure how practical that is but it might be a thought
[21:58] <arraybolt3> I bet I can test it here, is there any immediately obvious reason why that would not work?
[22:00] <vorlon> I don't remember if there was a reason we previously ruled that out
[22:03] <arraybolt3> I'll see what I can make happen here, and maybe provide a merge proposal.
[22:11] <kenyon> any possibility of getting ddebs.ubuntu.com available over rsync like archive.ubuntu.com is? this would avoid the rate limit problem on http
[23:37] <arraybolt3> vorlon, mkukri, juliank: got it working
[23:38] <arraybolt3> I just patched the recordfail save line back into grub.cfg and then added some stuff for finding the ESP, grabbing the path to the grubenv that I had saved under EFI/ubuntu/grubenv, and then using the --file switch on all load_env and save_env calls to point to the right file.
[23:38] <arraybolt3> I now have working recordfail on a BTRFS-on-/boot system.
[23:39] <arraybolt3> Now, valid question, will the bootloader always be under EFI/ubuntu? There's a chance that the user might have a buggy machine and reconfigure it so that GRUB is saved straight to EFI/BOOT, and that will probably need special casing. But, the general proof-of-concept functions right.
[23:39] <arraybolt3> so now I have to find the right repo to submit an MP to, and write the real code as opposed to hacking grub.cfg directly
[23:44] <arraybolt3> er, ahem, correction - it *almost* works. Somehow recordfail isn't ever becoming unset, and I can't actually see where the ext4-based code ever unsets recordfail either.
[23:46] <arraybolt3> is the unsetting of recordfail done after the boot process? It sort of looks like it.
[23:48] <arraybolt3> ah, indeed it is.
[23:58] <arraybolt3> alright, now I actually have it working.