/srv/irclogs.ubuntu.com/2024/04/24/#lubuntu-devel.txt

-queuebot:#lubuntu-devel- Unapproved: lubuntu-installer-prompt (noble-proposed/universe) [24.04.5 => 24.04.5.1] (lubuntu)05:29
mkukrihello, i am investigating LP: #2060624 , and i do wonder, in the calamares based installer, what is/should be responsible for configuring grub-pc/efi's debconf entries during/after installation12:22
-ubottu:#lubuntu-devel- Launchpad bug 2060624 in grub2 (Ubuntu) "lubuntu/xubuntu reinstall (& install) on dual boot system, grub does appear & offer OS choice" [Undecided, New] https://launchpad.net/bugs/206062412:22
mkukrisubiquity has logic to this itself, and there is also a cloud-init module that suppossed to do this (although it's less complete and more buggy)12:23
mkukris/subiquity/curtin in this case just be clear, it applies to both desktop isntaller and server one12:24
* guiverc can't help there sorry.. you need a developer. I did get same response on ubuntu-desktop-installer (xubuntu) BEFORE the format was forced.12:34
mkukrithan this issue might strictly be unrelated but it seems like calamares based ubuntu installs do not fill the grub-pc/efi debconf fields correctly12:59
mkukriinterestingly grub2/enable_os_prober is still false in debconf, even in installations were the alternative OS shows up in the menu13:12
mkukrihmm in fact regenerating grub debconf with update-grub after the os booted reproduces the bad behavior13:14
mkukriguiverc, i seem to have reproduced this on the desktop installer too, and i have a hunch what's going on but i am gonna do a few more tests before proclaiming it for sure13:41
guiverc:|  at creating it with ubuntu-desktop-installer (I like confirmation of what i experiened, but :( at it still being possible as the forced-format b/c of another issue I thought prevented it appearing with ubuntu-desktop-installer...14:03
tsimonq2mkukri: There's debconf prompts for that?14:04
mkukrigrub-{efi,pc}/install-devices needs to be filled in, otherwise grub is not reinstalled on package update14:05
mkukrithis is what curtin does https://github.com/canonical/curtin/blob/204562c277b096229ca20ba546149deef1d85e38/curtin/commands/curthooks.py#L63214:06
mkukritsimonq2 ^14:06
mkukrithe os-prober one i think im wrong about14:06
mkukriit exists, but there is an ubuntu patch that should force it enabled if the existing grub.cfg was generated with os-prober enabled14:07
mkukriand btw definitely not release critical, there were similar bugs (grub not reinstalling) in cloud images for years. but it should eventually be fixed14:08
tsimonq2mkukri: Ah okay, I see, thanks :)14:11
tsimonq2If I had to guess off the top of my head, this is where the modification should be made: https://git.lubuntu.me/Lubuntu/calamares-settings-ubuntu/src/branch/ubuntu/noble/common/modules/before_bootloader_context.conf14:12
tsimonq2Either that, or a dedicated contextualprocess to set the debconf prompts14:13
* tsimonq2 would be interested in seeing what entries this results in *exactly* for the main Desktop Installer14:13
mkukritsimonq2 i can give you an example from a simple install, the more annyoing case is raid where you have to put multiple things into the multiselect14:58
mkukriso for example, a simple *BIOS* qemu VM results in *grub-pc/install_devices: /dev/disk/by-id/ata-QEMU_HARDDISK_QM0000115:02
mkukrialso i am not sure if lubuntu uses cloud-init at all, but the following recent bug might be relevant too https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/206069515:05
-ubottu:#lubuntu-devel- Launchpad bug 2060695 in cloud-init (Ubuntu) "24.04 grub-pc cannot upgrade on mirrored software RAID root disk" [Medium, Triaged]15:05
mkukritl;dr cloud-init's cc_grub_dpkg module can mess things up on first boot if enabled15:06
tsimonq2mkukri: ack looking thanks :)15:51
tsimonq2mkukri: I don't believe Lubuntu uses cloud-init, no15:52
mkukriseems to be my observation as well because, ive done some lubuntu installs to (try to) track down the above bug and grub-pc/install-devices was always empty15:55
mkukriso yeah i assume it should be set by your installer15:55
tsimonq2mkukri: Calamares doesn't support RAID yet, interesting16:05
arraybolt3I just recently did a side-by-side install of Kubuntu (which also uses Calamares) and Windows 10 and os-prober did the right thing...16:06
arraybolt3does having the debconf fields not filled in result in an intermittent issue in this area?16:07
tsimonq2arraybolt3: To my understanding, when the grub package is updated it looks for those debconf config values to install GRUB to the right place16:07
tsimonq2postinst or similar16:07
arraybolt3right16:07
mkukriarraybolt3, yeah the os-prober issue is seperate from this16:08
arraybolt3I've seen very strange GRUB prompts during bootloader updates (or possibly even just kernel updates) that looked like something was awry with debconf16:08
arraybolt3and that is NOT a good UX16:08
mkukrii just noticed that this _whle_ looking at tthat16:08
arraybolt3so I'd say this is worth calling a release blocker in the event we can get a fix for it16:08
mkukrithose should go away if debconf is set16:08
mkukrior there is grub-pc/cloud_style_installation but i recommend not setting that outside cloud images, that uses grub-probe instead16:08
arraybolt3right16:08
tsimonq2arraybolt3: What kind of errors did you see? just for the sake of justifying the fix16:09
mkukrii dont think this should give you errors, just debconf questions16:09
tsimonq2ah okay that follows16:09
mkukriand in non-ineractive mode it should just silently leave the old grub installed on disk16:10
arraybolt3yep, debconf questions16:10
arraybolt3so you'd upgrade and then see "Where eddo you want to install GRUB?"16:10
arraybolt3(or something like that)16:10
arraybolt3s/eddo/do/16:10
arraybolt3(lag is horrible right now because I'm zsync'ing an ISO)16:10
mkukrithe reason this didnt come up in 23.10 btw is because grub-pc was *never* updated due to a multi year workaround....16:10
mkukrithat  i finally removed in 24.0416:10
arraybolt3aha16:11
arraybolt3that makes a lot of sense16:11
mkukriand that's why cloud_style_installation was added, and why the cloud-init issue with subiquity was discovered too...16:12
mkukrithings seem okay in noble, but i still have many discussions to have with various folks over the issues that were papered over for years16:12
tsimonq2mkukri: Can I ask why you think this may not be a 24.04 release blocker?16:12
tsimonq2(If I'm reading your earlier messages correctly, that is.)16:13
mkukrii guess i didnt have the debconf questions popping up in mind earlier16:13
mkukrii was only thinking of noninteractive upgrades16:13
tsimonq2arraybolt3: oh... wait... does Kubuntu's update notifier even support debconf prompts?16:14
arraybolt3Kubuntu's does not, Lubuntu's does.16:14
arraybolt3(because I made very sure that it did :P)16:14
tsimonq2I'd definitely call it a release blocker for Kubuntu then, what do you think?16:14
arraybolt3Even on Kubuntu though this issue should trigger if the user upgrades via apt, which I did for a while and still do sometimes.16:14
mkukriis kubuntu and lubuntu the only flavours not using the new desktop installer?16:15
arraybolt3Ubuntu Unity also.16:15
tsimonq2mkukri: and Ubuntu Unity16:15
tsimonq2arraybolt3: jinx :)_16:15
arraybolt3The main difficulty I'm seeing is, how do we get Calamares to tell us what device the bootloader was installed to?16:15
* arraybolt3 digs in scripts16:15
tsimonq2arraybolt3: already there16:15
tsimonq2globalstorage16:15
arraybolt3right but I mean in a contextual process16:16
arraybolt3Calamares doesn't expose everything to the shell script thingies16:16
arraybolt3we could write a module in Python...16:16
tsimonq2Calamares exposes all globalStorage values to contextualprocesses :)16:16
arraybolt3but let's exhaust the other options first for sanity's sake16:16
arraybolt3oh it does? Perfect.16:16
mkukrishould i file a bug about this. and what package is appopriate?16:17
tsimonq2mkukri: calamares-settings-ubuntu and yes please16:18
arraybolt3tsimonq2: I see a contextualprocess can decide what code to run based on the value of a global storage key, but does it let you use the value of a global storage key as a variable?16:20
tsimonq2...oh?16:20
arraybolt3The thing is we need to be able to say "take the global storage key, find the corresponding by-id device, stuff it in this debconf entry".16:21
mkukrihttps://bugs.launchpad.net/ubuntu/+source/calamares-settings-ubuntu/+bug/206335416:22
-ubottu:#lubuntu-devel- Launchpad bug 2063354 in calamares-settings-ubuntu (Ubuntu) "Calamares based flavours do not set grub-{efi,pc}/install_devices debconf" [Undecided, New]16:22
tsimonq2arraybolt3: hey guess what, the bootloader module itself is written in Python :P16:22
tsimonq2mkukri: Thank you!16:22
arraybolt3tsimonq2: heh, well there you go16:22
tsimonq2arraybolt3: do we want to move to Matrix?16:23
arraybolt3tsimonq2: mkukri is here so I think this is OK here for now16:23
tsimonq2arraybolt3: ok cool :)16:24
mkukriim end of day currently, but if you need help please tag my nick, ill check on IRC periodically16:24
arraybolt3will do, thank you for your help!16:25
arraybolt3tsimonq2: want me to tackle the patch or are you already doing it?16:26
tsimonq2arraybolt3: I'm on it, will check in16:26
tsimonq2arraybolt3: In the meantime any chance you could update RT and the bug report on what's happening?16:26
tsimonq2(probs release blocker for all three flavors, eh?)16:27
arraybolt3sure thing16:27
tsimonq2thank you :)16:27
arraybolt3and yes, probably a release blocker16:27
mkukrii assume the desktop installer based ones are fine?16:27
mkukriand maybe talk to the release team please16:27
arraybolt3I do not know if the desktop installer based ones are fine or not. If they don't set the debconf prompt, they're probably not fine.16:29
arraybolt3er, the debconf variable - you know what I mean16:29
mkukricurtin sets it16:29
arraybolt3Either the user will get unsightly debconf prompts, OR GRUB won't end up updating most likely, which would leave a gaping security hole and opportunity for Secure Boot bypasses and potential failure to boot if the SBAT key is rotated out.16:30
mkukriubuntu desktop itself is definitely fine16:30
mkukri(ubuntu server too) i do test test thoseregularly16:30
arraybolt3kk16:30
mkukriand i think i tested xubuntu too yesteday, but dont hold me to that16:31
* mkukri is going afk for a bit but will be back later16:31
tsimonq2arraybolt3: At this point, just wondering if setting it to /dev/disk/by-id/SOMETHING is the right thing compared to just /dev/sda1?16:48
tsimonq2thoughts?16:48
arraybolt3tsimonq2: /dev/sdXY are unstable and *will* change if the user changes drives (by, e.g., booting with a USB drive inserted).16:48
arraybolt3Learned this the hard way some time ago, and it's pretty common advice on the Interwebs.16:48
arraybolt3Figuring out the link between /dev/whatever and /dev/disk/by-id/SOMETHING should be easy enough in theory since the /dev/disk/by-id thingamajigs are just symlinks.16:49
tsimonq2That follows, now just wondering how it works with the main desktop installer, if there are multiple /dev/disk/by-id entries that match to one disk16:49
tsimonq2(can happen, from what I'm reading)16:49
arraybolt3https://www.tutorialspoint.com/python/os_readlink.htm16:49
arraybolt3oh really? That's not good...16:50
arraybolt3mmm, I see that I have that situation on *this* system I'm typing on.16:50
tsimonq2Bright side, the debconf config value is of type "multiselect"16:50
arraybolt3for me nvme-eui.00253857214071d7 and nvme-Samsung_SSD_970_EVO_Plus_1TB_S6S1NJ0T713093F point to the same partition16:50
arraybolt3er, the same NVME namespace I mean16:50
arraybolt3gosh I wish that there weren't NVME namespaces *and* partitions :P16:50
arraybolt3tsimonq2: maybe by-uuid would work too/instead? Just a thought.16:51
tsimonq2arraybolt3: the other thing is that in this virtual machine I can only see one by-id mapping to /dev/sda116:52
tsimonq2Do you have test hardware? Would be great to see what those entries are actually set to in such a case with the Ubuntu Desktop installer16:53
arraybolt3tsimonq2: https://termbin.com/8n2f16:53
arraybolt3hrm... I can make the hardware I have test hardware.16:53
tsimonq2If you're okay with that, go for it16:54
arraybolt3I lost my two best testing laptops, one to video card failure, and one to whatever power mess it suffers from, so I'm stuck using one desktop for everything test-wise :P16:55
arraybolt3but anyway, I was going to test Xubuntu Minimal anyway so I'll give it a shot there16:55
tsimonq2awesome, thanks16:55
arraybolt3and maybe also try it in BIOS mode so we get a good picture16:55
tsimonq2BIOS and UEFI would be great for full testing coverage, probably looking for `sudo debconf-show grub2` on first boot16:57
arraybolt3nifty16:57
tsimonq2Testing some code now which just sets an example value for that debconf setting - to test syntax before we have actual values to work with :)17:07
tsimonq2arraybolt3: `sudo debconf-show grub-pc` not `sudo debconf-show grub2` my bad :)17:11
tsimonq2I just did a `sudo dpkg-reconfigure grub-pc` on an installed system which gave me some ideas, would still like to see the output of that for both GRUB and EFI to double check though17:17
arraybolt3tsimonq2: Might take me a bit, I'm just now starting to install Xubuntu for testing.17:21
arraybolt3Also I'm multitasking between work duties and also Matrix Council activity17:21
arraybolt3Got the installation going now.17:23
tsimonq2arraybolt3: sweet thanks17:26
arraybolt3tsimonq2: alright, so, from an EFI installation:17:28
arraybolt3uh, `sudo debconf-show grub2` on Xubuntu Minimal outputs nothing.17:28
arraybolt3Not even a newline.17:28
tsimonq2arraybolt3: 17:11 < tsimonq2> arraybolt3: `sudo debconf-show grub-pc` not `sudo debconf-show grub2` my bad :)17:28
tsimonq2:)17:29
arraybolt3oh lol, alrighty17:29
arraybolt3ok, grub-pc shows this:17:29
arraybolt3uh, lemme pastebin17:29
arraybolt3https://termbin.com/bn4o17:30
tsimonq2can I get the whole thing?17:30
arraybolt3I have the ls -l from /dev/disk/by-id coming17:30
arraybolt3and that is the whole thing17:30
tsimonq2wait, the whole output?17:30
arraybolt3That's literally the result of `sudo debconf-show grub-pc | nc termbin.com 9999`17:30
arraybolt3so yes, that's the whole output17:30
tsimonq2oh wow this is interesting, ok, yes please what is the /dev/disk/by-id output?17:31
arraybolt3frustratingly it looks like this particular physical machine with a SATA drive does NOT have multiple /dev/disk/by-id entries that point to the same device!17:31
arraybolt3https://termbin.com/y01a17:31
arraybolt3I didn't expect that17:31
arraybolt3And I cannot safely do this test on my primary laptop.17:31
tsimonq2arraybolt3: On your primary laptop, any chance you could just run `sudo dpkg-reconfigure grub-pc`, make sure all the entries are correct, then get the debconf-show output from that?17:33
tsimonq2also looking on my end17:33
arraybolt3grr, well... https://termbin.com/xiuu now you don't get any device at all... because this is a 22.04 installation on my main laptop.17:35
tsimonq2aaaaaaaHA17:35
tsimonq2ok it's only one device17:35
tsimonq2but it picks the first from the list17:35
arraybolt3ok good whatever works :P17:36
tsimonq2arraybolt3: I appreciate ya :)17:36
arraybolt3sorry I wasn't much help17:36
arraybolt3I tried *really hard* though17:36
tsimonq2you helped more than you think - only one debconf setting for that package is a good one17:36
arraybolt3ah ok17:40
mkukrimake sure to test cryptsetup, LVM and those types of things if you support it too. ive seen weird bugs elsewhere with that before17:50
arraybolt3mkukri: we have a pretty immense checklist for the Cala-based flavors that covers almost all those weird edge cases (except for LVM which Doesn't Work(TM) with Calamres yet).17:54
arraybolt3tsimonq2: you still need the BIOS install output? I'm about to reinstall in BIOS mode for that.17:54
arraybolt3oh fun, Dell's BIOS has arbitrarily decided to gray out my legacy boot option for who-knows-why17:55
arraybolt3oh there we go, had to enable legacy option ROMs first17:55
tsimonq2arraybolt3: That would help :)17:58
arraybolt3kk, in progress...17:58
arraybolt3(man this installer takes like time and eternity to load)17:59
tsimonq2Prototype fix:17:59
tsimonq2+def get_disk_id(device_name):17:59
tsimonq2+    by_id_path = "/dev/disk/by-id"17:59
tsimonq2+    device_path = os.path.realpath(device_name)17:59
tsimonq2+17:59
tsimonq2+    for entry in os.listdir(by_id_path):17:59
tsimonq2+        full_entry_path = os.path.join(by_id_path, entry)17:59
tsimonq2+        if os.path.realpath(full_entry_path) == device_path:17:59
tsimonq2+            return full_entry_path17:59
tsimonq2[...]17:59
tsimonq2+        dev_id_path = get_disk_id(boot_loader["installPath"])17:59
tsimonq2+        check_target_env_call(["echo 'grub-pc grub-pc/install_devices multiselect " + dev_id_path + "' | debconf-set-selections"])17:59
arraybolt3Install in progress18:00
arraybolt3I don't get the details of what that function does but it looks vaguely right to me :)18:01
tsimonq2the one thing that's verylikely to cause issues is check_target_env_call18:02
tsimonq2and yep just ran into it not DTRT - coooooooooool18:02
arraybolt3needs sudo rights perhaps?18:03
arraybolt3or are you not chroot'd in?18:03
arraybolt3mmm, you probably need to run that command from within a chroot environment18:04
arraybolt3tsimonq2: alright a BIOS system has some significantly more interesting keys: https://termbin.com/qixa18:05
arraybolt3not sure which ones you care about (probably things like grub2/kfreebsd_cmdline_default are ignorable :P), but there you go18:06
arraybolt3and also ls -l from /dev/disk/by-id: https://termbin.com/15h818:06
tsimonq2check_target_env_call means "call this in the target environment and check the result" :)18:07
tsimonq2arraybolt3: looking thanks!18:07
arraybolt3ah shoot then you *are* chroot'd in it sounds like18:07
arraybolt3well, I'm sure you'll figure it out, good luck and I'll be around to try and help18:08
tsimonq2thanks - also that BIOS output helps thank you :)18:08
tsimonq2arraybolt3: I have working code for BIOS, testing for EFI18:44
arraybolt3awesome18:52
tsimonq2arraybolt3: can I rubber ducky off you real quick?18:59
tsimonq2BIOS is good, EFI is the last hurdle here19:00
tsimonq2It seems like in the bootloader module it's being passed as /boot/efi, not the actual partition name19:00
arraybolt3oooh, ok19:00
tsimonq2So when grabbing the device ID for EFI, it should grab the ID corresponding to the partition mounted at /boot/efi yes?19:01
arraybolt3In that instance figure out what partition is mounted at /boot/efi and then use that?19:01
arraybolt3jinx19:01
arraybolt3so yeah, that sounds right19:01
tsimonq2sounds good :) finding some coffee19:01
* genii twitches19:01
mkukriinto install_device you need to put the actual disks19:36
mkukritsimonq2, arraybolt3. grub-multi-install internally finds all the ESPs from that and mounts them19:36
mkukri/boot/efi is just a fallback and that mountpoint is ignored when you set install devices19:37
mkukriand grub-pc also expects the bare disks, but that part i think is obvious19:37
tsimonq2mkukri: Oh, just the actual disks?19:39
tsimonq2mkukri: Nothing under like /dev/disk/by-id or /dev/disk/by-uuid, just e.g. /dev/sda1 ?19:40
mkukriseems like i am wrong sorry, i checked a desktop installer install and it seems to have the ESP device node https://imgur.com/a/nVSnPGj19:40
mkukrithe by-id and by-uuid part is still there yeah. i was just trying to say that it wasnt expecting the partition but it looks like it is19:40
tsimonq2No problem! Quick question though...19:41
tsimonq2Does by-uuid work just as well as by-id in this case?19:41
mkukriwell, i do want to switch to by-uuid *eventually* but it's not tested yet19:41
mkukriand just to be clear this is the debconf options i get https://imgur.com/a/MvCEVlQ and the above image is the result in debconf db19:42
tsimonq2mkukri: The current roadblock I have is, for some reason I can only query by-uuid on EFI and by-id on BIOS19:42
tsimonq2Otherwise I have working code now19:43
mkukriby-id should work on EFI19:44
mkukrihow are you trying to query it?19:44
tsimonq2$ ls -l /dev/disk/by-id/19:44
tsimonq2total 019:44
tsimonq2lrwxrwxrwx 1 root root 9 Apr 24 13:45 ata-QEMU_DVD-ROM_QM00001 -> ../../sr019:44
mkukrialso if you are using a if=virtio disk in qemu you have to give it a serial number on the command line or it wont show up19:45
tsimonq2mkukri: Simple ls of /dev/disk/by-id19:45
tsimonq2oooh19:45
tsimonq2mkukri: I am using QEMU, how does that part work?19:45
mkukriwhat command line flag are you using to specify your hard disk?19:45
mkukrior is it virt manager? or LXD?19:45
tsimonq2virt-managere19:45
tsimonq2s/managere/manager/19:45
mkukriah i am not sure what that does, but i am guessing it doesnt give the disk a serial number19:46
mkukrithis is a problem with regular ubuntu as well in that case the debconf wont be filled either19:46
mkukribut the fallback install via /boot/efi still works19:46
mkukriif you want to test it similar to real hardware, try to specify a disk serial number for your VM in the options somewhere19:46
mkukrig19:46
mkukriapparently you can put <serial>some-arbitrary-serial</serial> in the vm's xml inside the <disk> tag19:47
tsimonq2ok looking then thanks19:47
tsimonq2mkukri: So to be clear, as for right now you'd recommend against going with by-uuid?19:48
mkukriif you want to have the same bugs as us then yes :)19:49
mkukriby uuid is the future plan but maybe it's better if we all switch at once in the 24.10 or 25.04 cycle19:49
tsimonq2cool, when we do get to that point it'll be a one-liner :)19:50
mkukrithere are problems like UUID's not actually always being unique :/19:50
* mkukri looks at our cloud images in disappointment.....19:51
tsimonq2oh wowww, okay, didn't know that19:51
mkukrithe issue is, the images are obviously pre generated, so you'd need to replace the uuid with a random one on first boot, but no one has bothered yet19:52
tsimonq2ahh okay19:53
tsimonq2mkukri: yes that <serial>ABCDGEF</serial> fix worked \o/19:53
tsimonq2ok doing a final round of testing before uploading, should be in the clear in the next hour here19:54
mkukriim glad to hear, that took me many many hours of headbanging to figure out once19:54
mkukriso you said you dont have LVM or RAID. but do you have cryptsetup?19:54
mkukriand maybe ZFS?19:54
mkukridid you test those?19:55
tsimonq2I can do a round of testing with those, yeah. For now I have just been concerned with making sure regular non-encrypted BIOS and EFI work19:57
tsimonq2We don't do ZFS, RAID, or LVM at this current point in time, but encrypted installs definitely19:57
mkukriokay cool im just trying to be exhaustive19:58
tsimonq2no you're okay :) thanks for that19:58
mkukriand amd64 bios and uefi are your only GRUB based platforms?19:58
tsimonq2yes19:58
mkukriokay i think, i guess that's all the cases than19:59
tsimonq2cool cool :)19:59
-queuebot:#lubuntu-devel- Unapproved: calamares (noble-proposed/universe) [3.3.5-0ubuntu3 => 3.3.5-0ubuntu4] (lubuntu, ubuntustudio)20:30
-queuebot:#lubuntu-devel- Unapproved: rejected lubuntu-installer-prompt [source] (noble-proposed) [24.04.5.1]20:42
-queuebot:#lubuntu-devel- Unapproved: lubuntu-installer-prompt (noble-proposed/universe) [24.04.5 => 24.04.6] (lubuntu)20:45
-queuebot:#lubuntu-devel- Unapproved: accepted lubuntu-installer-prompt [source] (noble-proposed) [24.04.6]20:50

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!