[10:22] <xnox> realtime-neil:  linux-image-generic is invalid selections.
[10:22] <xnox> realtime-neil:  you get the kernel that's available on the iso..... the isos are built with hwe kernels.
[10:23] <xnox> realtime-neil:  if you want 4.15 kernel; you do need to rebuild your iso to be with the GA kernel available. And/or have it in the pool.
[10:23] <xnox> realtime-neil:  if we can't find a kernel asked for, we install a kernel we have.
[10:24] <xnox> realtime-neil:  5.4 is the same kernel as in focal btw. and will be supported for the rest of the bionic lifetime.
[12:38] <realtime-neil> xnox: understood. Could you explain how one "asks for" a different kernel? I was under the impression that this was impossible with ubiquity.
[12:57] <realtime-neil> xnox: Yesterday, cjwatson explained that, with the introduction of ubiquity, the generally accepted means to install a desired kernel image is to include it in the filesystem.squashfs. I have been able to demonstrate that this works --- but the problem is that the hwe kernel (the one with which the isos are built) is installed _also_. I could tolerate the presence of two kernels if I could force grub to default to the one I put in the fil
[12:57] <realtime-neil> esystem.squashfs.
[13:28] <xnox> realtime-neil:  in focal, we added support to correctly select hwe or oem kernel; and uninstall hwe kernel, when a different one is installed.
[13:28] <xnox> realtime-neil:  that was not backported to bionic ubiquity i don't think.
[13:30] <realtime-neil> xnox: How was this implemented?
[13:35] <xnox> realtime-neil:  i think it's these 2020 commits https://git.launchpad.net/ubiquity/log/scripts/check-kernels
[13:37] <realtime-neil> xnox: I saw that script in the Bionic ubiquity source package --- I figured I could just overwrite it with logic of my own choosing. The problem I'm having is that I don't know where it "lands" inside the iso.
[13:38] <xnox> it's weird. it's in the ubiquity.deb itself in the filesystem.squashfs
[13:38] <xnox> i think
[13:39] <xnox> realtime-neil:  but there are matching things in python code too!
[13:40] <xnox> there is this https://git.launchpad.net/ubiquity/tree/ubiquity/components/check_kernels.py
[13:40] <xnox> but also
[13:40] <realtime-neil> xnox: do I understand correctly that you're suggesting I override the _python_ script that calls the shell script `check-kernels`?
[13:40] <xnox> https://git.launchpad.net/ubiquity/log/scripts/plugininstall.py
[13:40] <xnox> which also has matching 2020 changes for the kernels replacement (install new one, remove old one)
[13:41] <xnox> and be careful there are multiple plugininstall.py files
[13:42] <xnox> realtime-neil:  not sure. but when i looked at it in april. it was not possible to correctly install a different kernel flavour from the iso pool and/or the internet, instead of the one that the image itself is built with (i.e. the hwe one by default)
[13:42] <xnox> realtime-neil:  and i had to fix all of those pieces to make it work correctly. Especially with/without nvidia; with/without secureboot.
[13:46] <realtime-neil> xnox: Ick. Might I be able to stub it out altogether? Effect some kind of `d-i base-installer/kernel/image string none`? I already have the kernel I want in the filesystem.squashfs.
[14:18] <xnox> realtime-neil:  trace things with print() and/or sh -x, and yeah find things you don't want, and skip them.
[14:18] <xnox> realtime-neil:  but be careful, because you want linux-generic installed (not just linux-image-generic)
[14:19] <xnox> realtime-neil:  and at the end of things, on the installed system it must be marked as installed manually (with like apt-mark manual) because otherwise ubiquity might try to remove it at the end.
[14:19] <xnox> or like user could run $ apt autoremove and get it auto removed too, if it's isntalled auto.
[14:20] <xnox> realtime-neil:  why are you using 4.15 kernel, which has less support, for new laptops/desktops/VM/hypervisor when compared to the v5.4 kernel?
[14:20] <realtime-neil> xnox: understood ... Can you give me some guidance as to where the "action" starts? I have a unpacked iso in front of me, but I don't know what (or where) the "entrypoint" to the ubiquity things is.
[14:20] <realtime-neil> xnox: I'm using the 4.15 kernel because $DAYJOB says $VENDOR only supports LTS kernel
[14:21] <realtime-neil> xnox: point taken regarding linux-generic vs. linux-image-generic
[14:21] <xnox> realtime-neil:  v5.4 is the LTS kernel!
[14:22] <realtime-neil> I mean for Bionic, not Forcal
[14:22] <xnox> both everyone, canonical, ubuntu, and kernel.org
[14:22] <realtime-neil> I mean for Bionic, not *focal*
[14:22] <xnox> v5.4, is LTS on both *focal* _and_ *bionic* =)
[14:22] <xnox> we always offer next lts kernel, on lts-1.
[14:22] <xnox> with the same lifecycle support.
[14:23] <xnox> so $VENDOR is telling lies =)
[14:23] <realtime-neil> okay, maybe I'm mixing terminology --- what's the "common ubuntu" name for the thing pointed to by the package `linux-image-generic`?
[14:24] <xnox> usually we refer to it as GA kernel, meaning it's the same major series at release time, as it will be for the lifetime of the series.
[14:24] <xnox> which minority of our installs use
[14:24] <realtime-neil> "General Availability"?
[14:25] <xnox> yeah
[14:25] <realtime-neil> Okay. And it's unpopular because why?
[14:25] <xnox> ubuntu desktop defaults to hwe kernel; oem certified laptops use rolling linux-oem kernel; Public Clouds, use cloud speicific rolling kernel i.e. linux-aws
[14:26] <xnox> it gets old, doesn't support newer hardware, and is slower on newer hardware => because newer kernel series squeeze more performance out of existing hw because they are coded to do better.
[14:26] <xnox> thus 5.4 vs 4.15 on the same hardware is faster, uses less power, supports more hw.
[14:27] <xnox> thus you installing 4.15 on ubuntu desktop, deviates from what Ubuntu Desktop installs do, by default, in 2020. of the bionic release.
[14:27] <xnox> because it's not April 2018 anymore.
[14:28] <xnox> also for example as virtualbox / vmware / public-cloud /hyperv host platforms improve they typically need newer kernels to work properly.
[14:28] <realtime-neil> point taken --- your advice is to just take the hwe kernel that the Bionic Desktop installer wants to give me?
[14:29] <realtime-neil> I also note that this looks promising:
[14:29] <realtime-neil> ```$ apt-cache policy linux-oem
[14:29] <realtime-neil> linux-oem:
[14:29] <realtime-neil>   Installed: (none)
[14:29] <realtime-neil>   Candidate: 4.15.0.1097.101
[14:29] <realtime-neil>   Version table:
[14:29] <realtime-neil>      4.15.0.1097.101 500
[14:29] <realtime-neil>         500 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
[14:29] <realtime-neil>         500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
[14:30] <realtime-neil>         500 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 Packages
[14:30] <realtime-neil>         500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
[14:30] <realtime-neil>      4.15.0.1004.2 500
[14:30] <realtime-neil>         500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
[14:30] <realtime-neil>         500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
[14:30] <xnox> realtime-neil:  do not use linux-oem => that's kernel tuned to work on certified laptops, with specific quirks for particular laptop models.
[14:31] <realtime-neil> xnox: understood ... I figure it's not by accident that it's based on the GA kernel.
[14:31] <xnox> realtime-neil:  it's not for general consumption, and may be better or worse. we have explicit metadata to only install it when needed; and roll-off to the hwe-kernel as soon as all the fixes are srued.
[14:31] <xnox> there is linux-oem-5.6 => based on 5.6 which matches none of the GA/hwe kernels from any series.
[14:31] <xnox> etc.
[14:32] <xnox> there is linux-oem-osp1 too
[14:32] <xnox> there are many different oem kernels for specific SKUs.
[14:32] <xnox> only one of them is v4.15
[14:32] <realtime-neil> is there any documentation or catalog with these listed?
[14:33] <xnox> no, because that's private with the hw vendors. as we cannot referenced unreleased SKU names until they are public and for sale, yet we enable them before hand.
[14:33] <realtime-neil> got it
[14:33] <xnox> ubiquity in focal; automatically picks and installs the right thing.
[14:33] <xnox> which for ubuntu desktop, the default is the linux-hwe-MM.YY metapackage. eventually all oem kernels roll onto that.
[14:34] <xnox> (but on per sku basis, because raisins)
[14:34] <realtime-neil> and we've established that this behavior --- at least in bionic --- is not easy to override
[14:34] <realtime-neil> back to my question about the ubiquity "entrypoint" in the installer iso .... where does it start in earnest?
[14:36] <realtime-neil> I've looked at the init scripts within the casper/initrd, but I got lost trying to follow the invocations.
[14:37] <xnox> ubiquity.service no?
[14:38] <xnox> but you can also boot to live session, and launch the ubiquity.desktop file from the desktop if one is interactive
[14:38] <xnox> so the answer is, it has many =)
[14:48] <realtime-neil> xnox: Oh, boy! This whole "multiple entrypoints", "infinite wrappers" seems to be a recurring theme in this software game. :D
[15:04] <xnox> only slightly better than d-i with 110 source packages.
[15:04] <xnox> i guess it's best to clone the git repo and/or bzr branch
[15:04] <xnox> and run update, because then you know this is the installer bits all on this directly
[15:05] <xnox> because image has a lot of ubuntu-desktop stuff which is not quite always related
[15:05] <xnox> (casper is though)
[15:32] <realtime-neil> xnox: I want to revisit this "run update" of which you speak, but what I really want to know is how the ubiquity artifacts map to iso contents. Is that detailed in the source I just cloned out (https://git.launchpad.net/ubuntu/+source/ubiquity) ?
[15:35] <xnox> realtime-neil:  for bionic, focal, or groovy? did you get the right branch & a matching repo?
[15:35] <xnox> it's years of development between them. most things don't stay the same in terms of repos/layout/etc.
[15:36] <realtime-neil> I've checked out ubuntu/bionic
[15:38] <realtime-neil> ```
[15:38] <realtime-neil> $ git remote --verbose
[15:38] <realtime-neil> origin	https://git.launchpad.net/ubuntu/+source/ubiquity (fetch)
[15:38] <realtime-neil> origin	https://git.launchpad.net/ubuntu/+source/ubiquity (push)
[15:38] <realtime-neil> $ git status --short --branch
[15:38] <realtime-neil> ## ubuntu/bionic...origin/ubuntu/bionic
[15:38] <realtime-neil> $ git grep check_kernels.py
[15:38] <realtime-neil> debian/ubiquity.install-any:ubiquity/components/check_kernels.py usr/lib/ubiquity/ubiquity/components
[15:40] <realtime-neil> xnox: And yet:
[15:40] <realtime-neil> $ find ubuntu-18.04.5-desktop-amd64.d/ -name '*ubiquity*'
[15:40] <realtime-neil> ubuntu-18.04.5-desktop-amd64.d/pool/main/u/ubiquity
[15:40] <realtime-neil> ubuntu-18.04.5-desktop-amd64.d/pool/main/u/ubiquity-slideshow-ubuntu
[15:40] <realtime-neil> xnox: I'm clearly missing something obvious, but I have no idea what that is.
[15:41] <xnox> realtime-neil:  i don't know waht ubuntu-18.04.5-desktop-amd64.d is
[15:42] <realtime-neil> it's the unpacked contents of http://cdimage.ubuntu.com/releases/bionic/release/ubuntu-18.04.5-server-amd64.iso
[15:42] <xnox> realtime-neil:  ubiquity is a source package. it builds many .debs. Some of them are optional. Some of them are installed inside filesystem.squashfs
[15:42] <xnox> ubiquity installer is not used on server.iso at all
[15:42] <xnox> it only is used on the desktop iso
[15:42] <realtime-neil> whoops, my bad, I meant the desktop iso
[15:43] <xnox> (some of them are in the pool, and some of them are not anywhere at all! or like are flavour specific, i.e. the kde/qt frontend is shipped on kubuntu iso)
[15:43] <xnox> inside that bionic ubiquity tree you should be able to do
[15:43] <xnox> ./debian/rules update
[15:43] <realtime-neil> xnox: you're saying the ubiquity things are already in the filesystem.squashfs ? That's where I should be making changes (if any)?
[15:43] <xnox> to download vendored pieces
[15:43] <xnox> realtime-neil:  that's where the installer runs from, yes. it's preinstalled in the filesystem.squashfs as it must start and run on boot, and it does.
[15:44] <xnox> cause that's how it works.
[15:44] <xnox> if you are patching ubiquity you might want to patch source, and build new debs, chroot into filesystem.squashfs and install updated debs there.
[15:44] <xnox> or just patch/hack files inline up to you.
[15:44] <xnox> i'm busy.
[15:45] <realtime-neil> xnox: let me see if I understand --- ubiquity runs from the flilesystem.squashfs, but the filesystem.squashfs is also installed to the disk partition that the ubiquity installer prompts the user to choose?
[15:45] <realtime-neil> sorry
[15:45] <realtime-neil> I'll bother you some other day
[15:46] <xnox> realtime-neil:  yes!
[15:46] <xnox> initrd on the iso, mounts the iso, then mounts filesystem.squashfs as /cdrom then also mounts it with overlayfs tmpfs.
[15:46] <realtime-neil> I have much to study; thank you very much
[15:46] <xnox> during install it copies /rofs contents to target disk; and cleans up things, like self-removing ubiquity
[15:47] <xnox> because you don't want installed system, to try to boot into isntaller again (doh)
[15:47] <xnox> hence it's a very much 2-in-1 thing, both the installer and what gets installed into target
[15:47] <realtime-neil> that's either brilliant or crazy
[15:47] <xnox> hence you can find in ubiquity sourcecode a bit of a harakiri, where it supposedly removes itself, but it does it in the chroot /target => (aka in-target) instead of the running system.
[15:48] <xnox> think of it as a virus, that infects all the things with itself =)
[15:48] <realtime-neil> I wrote a self-destructing systemd unit like that once; I think I felt similarly about it :D
[15:48] <xnox> and it tries to be smart about things, i.e. it tries not to copy things over which will be deleted later anyway.
[22:56] <realtime-neil> Why is it adding `linux-image-generic-hwe-18.04` to the  `casper/filesystem.manifest-remove` in the Ubuntu Bionic Desktop installer isoaccomplishes nothing; i.e., the installed system still has the unwanted `linux-image-generic-hwe-18.04`?