[15:26] <xnox> realtime-neil:  because it uses launchpad builders to schedule building livefs via launchpad api, which is a priviledged operation.
[15:27] <xnox> realtime-neil:  and expects +livefs objects to be available in launchpad.
[15:27] <xnox> realtime-neil:  see for example https://launchpad.net/~ubuntu-cdimage/+livefs
[15:27] <xnox> realtime-neil: you might want to search for "ubuntu-image classic" or "ubuntu old fashioned builder" if you want to figure out a way to run livecd-rootfs builder locally without launchpad.
[15:28] <xnox> realtime-neil:  there are etc/ config settings as well to "ssh into a builder" and build there. But that has not been used for years and years now, not sure if those "manual" builders still work.
[15:29] <xnox> realtime-neil:  it's best to monkey patch existing iso / squashfs, instead of trying to rebuild them from scratch. Because for example, you might be able to replicate the official build. But will not be able to correctly sign the pool with keys that are trusted => rending the .iso non-installable.
[15:29] <realtime-neil> xnox: okay, I understood; I recently discovered the `simple-cdd` way of building Debian installation media --- any overlap there? Anything I can use for Ubuntu?
[15:30] <xnox> realtime-neil:  the plan was to have "ubuntu-image" snap do it all by now. Both in our infrastructure and locally. Alas, those grand plans have not been done yet.
[15:30] <xnox> realtime-neil:  if you don't need to change squashfs, you can run cron.daily without '--live' it means it will simply download the currently built squashfs, without rebuilding them.
[15:31] <xnox> but that still will only produce half-baked thing.
[15:33] <realtime-neil> xnox: Using the http://cdimage.ubuntu.com/releases/bionic/release/ubuntu-18.04.4-server-amd64.iso, I did try to put my proprietary debs in the filesystem.squashfs, and it mostly worked. I then attempted to _also_ install ubuntu-desktop in there, which did not play nice with the subsequent `pkgsel` operations.
[15:33] <xnox> realtime-neil:  why are you trying to shoe-horn two different products?
[15:34] <realtime-neil> xnox: because there's no desktop image on that index page
[15:35] <xnox> realtime-neil:  because they are on releases.ubuntu.com..... (obviously)
[15:35] <xnox> realtime-neil:  http://releases.ubuntu.com/
[15:36] <xnox> http://releases.ubuntu.com/bionic/ & http://releases.ubuntu.com/focal/
[15:36] <xnox> note that desktop does not use neither pure d-i, nor pure subiquity.
[15:36] <xnox> it uses casper + ubiquity (which wraps portions of d-i)
[15:36] <realtime-neil> xnox: Yep, there it is --- the reason I discarded that image.
[15:36] <xnox> hence for example, ubiquity preseed looks like d-i one, but supports a different subset of keys, and/or slightly simplified config
[15:37] <xnox> but one can use ubiquity to automatically preseed installation, which works across all releases
[15:37] <xnox> together with nfsroot network boot
[15:37] <xnox> realtime-neil:  are you after customized desktop? server? or both? in the end?
[15:38] <xnox> https://wiki.ubuntu.com/UbiquityAutomation
[15:38] <realtime-neil> xnox: this implies I can translate every d-i directive I have into ubiquity-speak --- a process you mentioned was not strictly possible
[15:38] <xnox> network, partitioning, etc all works.
[15:39] <xnox> w.r.t. package selections things are a lot easier, as ubiquity simply copies all of the filesystem.squashfs. Simply modify filesystem.squashfs and install anything you like into it, and ensure it is marked as installed.
[15:39] <xnox> and that's it.
[15:39] <xnox> no need to make a pool, sign it, etc.
[15:40] <realtime-neil> xnox: I'm after a customized desktop with private debs; are you saying this can be made to work via ubiquity and the ubuntu-18.04.4-desktop-amd64.iso image?
[15:40] <xnox> realtime-neil:  i do not expect you to have that complicated d-i preseed
[15:40] <xnox> realtime-neil:  yeah. trivially, and in the same manner acroos bionic/focal/groovy.
[15:40] <xnox> xenial too.
[15:41] <xnox> unpack desktop iso, chroot into filesystem.squashfs, (bind mount dev proc, bring etc/resolv.conf), add any repositories you need, install any packages you need), update iso with the new bigger filesystem.sqaushfs => nfsroot boot that, complete install.
[15:41] <realtime-neil> xnox: Okay, and this works because --- using ubiquity --- we fundamentally do NOT `pkgsel` after the filesystem.squashfs is extracted to the rootfs filesystem?
[15:41] <xnox> if you are after desktop, you do want to start from the desktop iso.
[15:42] <xnox> realtime-neil:  ubiquity does NOT use pkgsel at _all_, and why should you?
[15:42] <realtime-neil> xnox: force of habit, I guess. Okay; I understood everything except the NFS bit. Why do I need that?
[15:42] <xnox> it copies filesystem.squashfs verbantim to target, and then like adds/removes a few things to make it look nicer.
[15:43] <xnox> realtime-neil:  because currently xenial/bionic/focal/groovy casper in desktop iso do not quite support url=http:///path-to.iso ip=dhcp network boot with the desktop iso.
[15:43] <xnox> i think url= boot should now work on focal in desktop.iso but i did not test that.
[15:43] <xnox> but the NETBOOT=nfs nfsroot= should work across all releases with ubiquity desktop iso
[15:44] <realtime-neil> xnox: this assumes I want to boot via network, but If I'm okay losing that functionality, I can boot the resquashed iso directly, yes?
[15:44] <xnox> realtime-neil: there is toram option too. I.e. if you can place installer onto disk somehow, it can be copied to ram, and then running from ram, it can isntall back onto the drive one is booted from.
[15:44] <xnox> realtime-neil:  if you repack it, in a similar way we packed => yeah it should be fine to boot in uefi/bios cd/usb like regular iso.
[15:45] <xnox> if you update md5sums.txt it will be indistinguishable from regular isos
[15:45] <realtime-neil> xnox: okay, cool; what kind of kernel params should I be changing in the /boot/grub/grub.cfg, if at all?
[15:46] <xnox> realtime-neil:  if you rebuild .iso with updated md5sums.txt & filesystem.squashfs, for local boot, you will not need to change grub.cfg at all.
[15:46] <realtime-neil> xnox: excellent; where is this checksum file to which you refer?
[15:47] <xnox> realtime-neil:  it's on the top level of the .iso you can either rebuild/update it, or you can just remove it.
[15:47] <xnox> realtime-neil:  all our build logs are public
[15:47] <xnox> so if you navigate to https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu/groovy/daily-live-20200805.log
[15:48] <xnox> you will see ubuntu (which desktop), groovy build for today with full xorriso command used to recreate the iso
[15:48] <xnox> /srv/cdimage.ubuntu.com/scratch/ubuntu/groovy/daily-live/debian-cd/amd64/groovy-desktop-amd64.raw -J -joliet-long -l -b boot/grub/i386-pc/eltorito.img -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info --grub2-mbr cd-boot-images/usr/share/cd-boot-images-amd64/tree/boot/grub/i386-pc/boot_hybrid.img -append_partition 2 0xef
[15:48] <xnox> cd-boot-images/usr/share/cd-boot-images-amd64/tree/boot/grub/efi.img -eltorito-alt-boot -e boot/grub/efi.img -no-emul-boot -partition_offset 16 cd-boot-images/usr/share/cd-boot-images-amd64/tree CD1
[15:48] <xnox> you might need to adjust the paths to things, if you unpack things into different places.
[15:48] <xnox> realtime-neil:  or, if you have the cdimage setup already, you will have tmp/scratch directory.
[15:48] <xnox> realtime-neil:  if you run the build without --live everything should be there.
[15:49] <realtime-neil> Oh, i've become quite the xorriso user since we last spoke; I'm fine executing it directly with the proper bootloader paths
[15:49] <xnox> realtime-neil:  then it's trivial to unpack actually daily iso in those direcotreis, unpack/chroot/customize filesystem.squashfs/mksquashfs/replace, rerun xorriso => done.
[15:49] <xnox> realtime-neil:  cool. Just direct xorriso is best.
[15:50] <realtime-neil> xnox: agreed; I've got some scripts that probably duplicate most of what's in the ubuntu-cdimage python scripts, but my python is bad, so...
[15:50] <xnox> realtime-neil:  also if you have some generic script, that does this.... i wouldn't mind like shipping it in the cd-boot-images package directly which effectively will be "rebuild .iso with ` apt install path/to/debs/*.deb` in filesystem.squashfs"
[15:51] <xnox> realtime-neil:  or yeah stuff for ubuntu-cdimage. There are some remains of "update iso" but we probably ship something like "update squashfs" script there. as it would be very useful for customizations.
[15:52] <xnox> realtime-neil:  it's trivial for me to talk about these things, as i do work on improving things
[15:52] <realtime-neil> xnox: heh. I call it `edsquashfs` and it uses docker to mutate the filesystem.squashfs
[15:52] <xnox> but for the production builds; rather than the "down the line, tweak/customize"
[15:52] <xnox> realtime-neil:  hahhahahhahahhhhaa cute.
[15:53] <realtime-neil> xnox: hey it saves me mount/umount mistakes, so that's something
[15:53] <xnox> yeah, probably docker / lxd / multipass is nice for this to "isolate it" and still have "i want network access, and please repeat this"
[15:53] <realtime-neil> If I can adapt this it to use raw lxd (maybe?), then I might have something worth sharing
[15:55] <xnox> realtime-neil:  if there are secrets you can share it privately, and i can work on cleaning it up.
[15:56] <realtime-neil> no secrets; lemme polish what I have and paste it
[15:56] <xnox> realtime-neil:  i am mostly after desires & requirements, and rough implementation. Because i really don't know, what issues people face, who are in your position.
[16:00] <realtime-neil> xnox: https://paste.ubuntu.com/p/smgdTNrwPY/
[16:04] <realtime-neil> xnox: why do all ubuntu-18.04.4-server-amd64.iso  `/boot/grub/grub.cfg` menu entries that have a `file=` parameter point to /cdrom/preseed/ubuntu.seed --- a file which definitely looks like a d-i preseed? I thought you said it was using ubiquity?
[16:05] <realtime-neil> also, it's doing `tasksel/first multiselect ubuntu-desktop`. Am I using the wrong image? What am I misunderstanding?
[16:07] <xnox> realtime-neil: in bionic; server uses just d-i; desktop uses just ubiquity
[16:08] <xnox> realtime-neil:  in bionic; server.iso "becomes a full-server iso" because of that file= parameter, that customizes the behaviour of d-i to be like 'Ubuntu Server'.
[16:08] <realtime-neil> whoops, this is in the ubuntu-18.04.4-desktop-amd64.iso also
[16:08] <xnox> realtime-neil:  because d-i is flexible, and we used to build "desktop alternative images" that also used d-i / console installer, yet installed desktops like ubuntu/lubuntu/xubuntu/etc => but all of those got discountinued over the years.
[16:08] <xnox> realtime-neil:  so desktop
[16:10] <xnox> realtime-neil:  yes desktop iso also have file= parameter to slightly configure ubiquity too..... despite most of it
[16:10] <xnox> being actually redundant
[16:11] <xnox> i've tried to clean up / remove file= in groovy
[16:11] <xnox> because yeah, it's sad and pointless.
[16:11] <xnox> but also
[16:11] <realtime-neil> xnox: Okay, and there's spooky ubiquity magic that will pick up, what, a `seedfrom=` directive?
[16:11] <xnox> a good place for you to hook in ;-)
[16:11] <xnox> realtime-neil:  there is no cloud-init on desktop isos
[16:12] <xnox> realtime-neil: .... unless you do install cloud-init in your customized filesystems.squashfs.... because then it will use that directive.
[16:12] <xnox> realtime-neil:  but note, ubuntu-desktop do not use autoinstall/cloud-init
[16:12]  * xnox realizes what a mess all of this is
[16:12]  * xnox sighs a lot
[16:13]  * xnox feels like i should document all of our conversations
[16:13] <realtime-neil> xnox: I'm so confused; I thought I was avoiding the pkgsel for good by using the ubuntu-18.04.4-desktop-amd64.iso ; are you now saying that the pkgsel is inevitable, no matter what?
[16:14] <xnox> realtime-neil:  it feels like "take stock iso, and have hello.deb preinstalled" is like a usecase which is painfully hard across all series server & desktop
[16:15] <xnox> realtime-neil:  desktop iso, mounts filesystem.squashfs and does cp -r of it.
[16:16] <xnox> realtime-neil:  live-server, mount .squashfs and uses autoinstall/curtin, to cp -r them
[16:16] <xnox> realtime-neil:  "old/classic" server, uses pkgsel to apt install all the things.
[16:18] <xnox> realtime-neil:  i'm saying that no modern images use pkgsel anymore.
[16:18] <xnox> (specifically live-server & desktop, in any series)
[16:18] <realtime-neil> xnox: the bionic desktop does: https://paste.ubuntu.com/p/XRxKXzqJDq/
[16:25] <xnox> realtime-neil:  which is actually has no effect at all.
[16:26] <xnox> realtime-neil:  it's obsolete remains form the old "d-i ubuntu-desktop alternate iso" which we no longer produce.
[16:26] <xnox> realtime-neil:  it's literary missleading =) sorry about that.
[16:26] <realtime-neil> xnox: okay, okay; I'm starting to thing we should be documenting, like, all of this.
[16:26] <xnox> extras.ubuntu.com is dead for decade now
[16:26] <xnox> tasksel is unused
[16:27] <xnox> keep-installed => yes, it is used, but those packages listed there do not exist in ubuntu for decades now
[16:27] <xnox> and altmeta should be use, but actually doesn't actually (because i found bugs in it this year, which were there from the begineeing of time)
[16:27] <realtime-neil> xnox: I believe all of this, but I do want to know the mechanism that's preventing these very-much-extant files from having any effect.
[16:30] <xnox> hahahhaha right
[16:30] <xnox> they are kind of noops
[16:30] <xnox> realtime-neil:  and i guess they do get in the way when _you_ try to use preseed
[16:30] <xnox> realtime-neil: horum.
[16:31] <xnox> realtime-neil:  also i'm in meetings at the moment, so sorry for slow responses
[16:32] <realtime-neil> xnox: no problem answer as your schedule dictates.
[16:34] <realtime-neil> xnox: So, if i have a custom filesystem.squashfs ready to go into custom ubuntu-18.04.4-desktop-amd64.iso, _AND_ I want to quash all the debconf prompts, then my only recourse is to, what, a bunch of kernel params?
[16:35] <xnox> realtime-neil:  if you want to make it auto installable one has to do a few things.
[16:35] <xnox> 1) change grub.cfg to do auto | noninteractive boot 2) supply preseed with more answers.
[16:36] <xnox> to acheive those things i would steal the preseeds from ubuntu-qa automated test cases
[16:36] <xnox> and either pack the preseed into the initramfs + customize grub.cfg
[16:36] <xnox> or customze that file=ubuntu.cfg thing in place with my pressed, and append auto to the kernel cmdline
[16:37] <xnox> let me try to find all of that
[16:37] <realtime-neil> xnox: Okay, I'll try that; got a link to something that shows me the "ubuntu-qa automated test cases"?
[16:37] <xnox> yeah trying to find that.
[16:38] <xnox> it does all sorts of things like connect to libvirt, create vms, take iso, make it auto pressedable, boot it.
[16:38] <realtime-neil> I need all this goodness in my life
[16:38] <xnox> i would not call it goodness
[16:38] <xnox> nobody likes touching it, even in full PPE and barge poll
[16:39] <realtime-neil> xnox: I have some rather uncommon interests, if you haven't noticed already :D
[17:36] <xnox> har har har
[17:37] <xnox> realtime-neil:  sorry i'm running out of time. i will be back later and/or tomorrow with pointers to qa-tests
[17:37] <realtime-neil> xnox: okay, see you then
[21:45] <xnox> realtime-neil:  so
[21:45] <xnox> we have jenkins, which is configured using jenkins job builder, using yaml syntax.
[21:46] <xnox> which uses "utah" which downloads isos, statically validates isos, create preseeds to autoinstall them, uses libvirt/virsh to boot things, install them, ssh into them, test them, shutdown, collect test results
[21:46] <xnox> the test results are public. I.e. focal, desktop, default "smoke-test" install is at https://platform-qa-jenkins.ubuntu.com/view/Focal/job/ubuntu-focal-desktop-amd64-smoke-default/
[21:46] <xnox> which is defined from https://git.launchpad.net/qa-jenkins-jobs/tree/jobs/iso-testing/jobs.yaml
[21:47] <xnox> which uses a lot of "utah" which is
[21:47] <xnox> https://git.launchpad.net/utah/tree/
[21:47] <xnox> which has this little gem
[21:47] <xnox> https://git.launchpad.net/utah/tree/conf/utah/default-preseed.cfg
[21:48] <xnox> which is enough to autoinstall ubiquity
[21:48] <xnox> based images
[21:48] <xnox> and like you can drop that on the iso, and change grub.cfg to point at that file
[21:48] <xnox> or like utah does, you can append it to the initrd
[21:48] <xnox> note i think that preseed might be slightly incomplete
[21:48] <xnox> as i think utah injects some ephemeral things - i.e. unique hostname or some cuhs.
[21:49] <realtime-neil> xnox: dear lord
[21:49] <xnox> https://git.launchpad.net/utah/tree/conf/utah/default-vm.xml i think is the default virsh template for the smoke-test VM
[21:49] <xnox> which again i think is a bit incomplete as it dynamically insers the right iso
[21:49] <xnox> also setting up utah is a nightmare https://git.launchpad.net/utah/tree/utah_howto.txt and i don't think you need or want it.
[21:49] <xnox> it's literary an acient hacked up thing to do iso testing
[21:50] <xnox> https://git.launchpad.net/utah/tree/utah/preseed.py => is some of the hackery that it does on per-run basis
[21:51] <xnox> https://git.launchpad.net/utah/tree/utah/provisioning/provisioning.py is a nightmare and deals with "funny" powerpc, and "funny" s390x etc.
[21:51] <xnox> realtime-neil:  so when you said "i want all of these goodness" i did giggle.
[21:52] <realtime-neil> xnox: this is probably going to take me a while to get through, but the preseed.cfg is already helping
[21:52] <xnox> because like even looking at this now, i'm not sure where it repacks the preseed!
[21:52] <xnox> let me see if i can actually pull the "final full" preseed.cfg that it embed for the install
[21:52] <xnox> as that might be more helpful
[21:52] <xnox> and like cmdline args to boot it unattended
[21:53] <xnox> do you see
[21:53] <xnox> realtime-neil:  https://platform-qa-jenkins.ubuntu.com/view/Focal/job/ubuntu-focal-desktop-amd64-smoke-default/165/artifact/log/utah-99555-focal-amd64-preseed.cfg/*view*/ ?
[21:53] <xnox> this looks like the full preseed with hostname, user password, user and success/fail commands.
[21:54] <xnox> realtime-neil:  in success/failure commands, one normally "calls home" or executes extra stuff "at the end of the install", the system to install is mounted in /target
[21:54] <xnox> and one can do things like "in-target apt install -y hello"
[21:54] <realtime-neil> yep, that's how I was writing my late_command scripts
[21:55] <xnox> in https://platform-qa-jenkins.ubuntu.com/view/Focal/job/ubuntu-focal-desktop-amd64-smoke-default/165/artifact/log/utah-99555-focal-amd64/installer/
[21:55] <xnox> you see collected logs from installation
[21:55] <xnox> from https://platform-qa-jenkins.ubuntu.com/view/Focal/job/ubuntu-focal-desktop-amd64-smoke-default/165/artifact/log/utah-99555-focal-amd64/installer/syslog/*view*/ you can see that "to boot unattended" it did
[21:55] <xnox> Kernel command line: automatic-ubiquity noprompt netcfg/get_hostname=utah-99555-focal-amd64 log_host=192.168.124.1 log_port=0 boot=casper keyboard-configuration/layoutcode=us
[21:55] <realtime-neil> I can't even ping that jenkins host
[21:55] <xnox> and i think it embedded the preseed.cfg inside the initrd.
[21:56] <xnox> hm
[21:56] <xnox> maybe i am on vpn
[21:56] <xnox> there must be these publically too
[21:56] <xnox> =(((((((
[21:56] <xnox> they used to be
[21:56] <xnox> yeap without vpn it's dead
[21:56] <xnox> none of this is secret though i wonder if i can repastebini it
[21:56] <realtime-neil> okay, those command line params looks awfully similar to the one that finally worked for me
[21:57] <xnox> but like log_host is for remote syslog? so optional.
[21:57] <realtime-neil> yep; didn't need it, so it's not part of my tweaks
[21:58] <xnox> and instead of appending preseed.cfg to the initrd, i guess it is best to just modify the exisitng one pointed with file= thing, or like make your own on the top level of thecd, and do file=/cdrom/realtime-neail-preseed.cfg
[21:59] <xnox> realtime-neil:  https://paste.ubuntu.com/p/BBH3FpVYV7/
[21:59] <realtime-neil> wow; all of this is definitely similar to my `preseedify` script
[22:00] <xnox> so that's the full preseed for default "all in one disk, the only one, plain ext4, no crypto, no lvm, no raid"
[22:00] <xnox> without even picking a disk, cause only one internal one.
[22:01] <xnox> and it uses serial console for logging
[22:01] <xnox> but it does show things like automated reboot, and stuff
[22:01] <xnox> but note, this preseed is "universal" for desktop & d-i old server
[22:01] <xnox> i.e.
[22:01] <xnox> d-i pkgsel/include string openssh-server python-yaml bzr git gdebi-core => i'm not sure that does anything
[22:01] <xnox> maybe it does? who knows
[22:02] <xnox> so use that as a guide, with a pinch of salt, and adjust to taste.
[22:02] <xnox> realtime-neil:  is any of above useful at all? and or need time to process?
[22:04] <realtime-neil> it's immediately useful as a sanity check; with time to examine, I'm sure it will be directly useful with my own work; thank you very much