=== chris14_ is now known as chris14
=== chris14_ is now known as chris14
=== justache is now known as justache_Test
=== justache_Test is now known as justache
=== JanC is now known as Guest8325
=== JanC_ is now known as JanC
=== kostkon_ is now known as kostkon
=== esv_ is now known as esv
=== JanC is now known as Guest1241
=== xispita_ is now known as xispita
=== vince_ is now known as vince
[16:24] <MTecknology> dbungert: I heard a rumor that maybe you know some things about the autoinstall stuff? I still can't figure out how to make the installer grab an autoinstall file.
[16:25] <dbungert> MTecknology: I do.  Thanks for sharing your autoinstall in progress.
[16:25] <dbungert> I'm in a meeting for the moment so my responses might be slow
[16:25] <dbungert> the two general plans are to build the autoinstall into the ISO or to serve it over cloud-init, which one sounds better for your case?
[16:26] <MTecknology> I've been at this for a couple weeks, so slow is still faster than me.
[16:27] <p3lim> MTecknology: I use kernel args with nocloud, e.g: `linux /casper/vmlinuz autoinstall ds=nocloud\;s=http://myhost/autoinstall.yaml ---`
[16:28] <p3lim> if you've packaged the autoinstall file with the ISO or have a separate disk you can get it from that instead, e.g. `s=/cdrom/mypath/autoinstall.yaml`
[16:29] <MTecknology> My original goal was to bake it into the iso, but I also need to support remote re-installation so I'm trying to use those same "nocloud" arguments. I thought that the thing needed to end in a / so that it could find the user-data and meta-data files and then read the autoinstall data from the cloud-init config file
[16:29] <p3lim> docs: https://cloudinit.readthedocs.io/en/17.2/topics/datasources/nocloud.html
[16:29] <p3lim> I don't use separate user-/meta-data that way so can't answer that for you
[16:31] <MTecknology> That's what's shown on https://ubuntu.com/server/docs/install/autoinstall and https://canonical-subiquity.readthedocs-hosted.com/en/latest/howto/autoinstall-quickstart.html#write-your-autoinstall-configuration
[16:31] <p3lim> question: how can I validate the autoinstall/curtin match targets for storage? trying to match by 'model', which should be ID_VENDOR but it just fails to match one
[16:32]  * MTecknology goes off and tries with a full path to that specific file.
[16:32] <p3lim> debug output doesn't really help me here, so just wondering if there are any tools to validate/match that
[16:34] <MTecknology> The attempt I just tried -> kvm -no-reboot -m 2048 -drive file=image.img,format=raw,cache=none,if=virtio -kernel linux -initrd initrd.gz -append 'autoinstall ds=nocloud-net\;s=https://deb.domain.tld/ubuntu2004/autoinstall.yaml'
[16:35] <p3lim> MTecknology: don't do it on initrd, do it on linux
[16:36] <MTecknology> "on initrd"?
[16:36] <p3lim> (not too familiar with raw kvm parameters)
[16:36] <p3lim> I assume `-append` appends it to `-initrd` here?
[16:37] <MTecknology> nope, it's the appended arguments in grub
[16:37] <MTecknology> well, not "in grub", but what's handed to the kernel
[16:38] <MTecknology> I can see the arguments listed in the kernel command line, but they don't appear to have any impact on the installer.
[16:38] <p3lim> e.g. you see them in /proc/cmdline?
[16:38] <MTecknology> I also can't find any logs to suggest any web activity was attempted.
[16:38] <MTecknology> yep
[16:38] <p3lim> show us exactly what /proc/cmdline contains please
[16:40] <p3lim> mask your s= if you want :P
[16:41] <MTecknology> heh ... I don't know how to take a screenshot from this.  It's "autoinstall ds=nocloud-net;s=https://deb.corp.tld/ubuntu2004/user-data"   where "corp.tld" is a typed replacement. Everything else is char4char.
[16:42] <p3lim> I made a mistake in my paths above, I had some part of the ISO builder that did something, I'll explain it here:
[16:43] <p3lim> my /proc/cmdline contains "BOOT_IMAGE=/casper/vmlinuz autoinstall ds=nocloud;s=/cdrom/nocloud/foo ---"
[16:43] <p3lim> within /cdrom/nocloud/foo directory I have two files; meta-data and user-data
[16:43] <p3lim> user-data is the autoinstall.yaml file
[16:45] <p3lim> in your case you should point to https://deb.domain.tld/ubuntu/2004 which contains those two files
[16:45] <dbungert> ^
[16:45] <dbungert> was just about to say the same, thanks p3lim
[16:45] <MTecknology> That's what I was trying first, before you mentioned autoinstall.yaml ;)
[16:45] <p3lim> also keep in mind that https _might_ be problematic depending on how you've signed the certificate on that domain
[16:46] <MTecknology> > I thought that the thing needed to end in a / so that it could find the user-data and meta-data files and then read the autoinstall data from the cloud-init config file
[16:47] <dbungert> https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html#nocloud - the examples are showing a trailing slash, I don't know if it matters
[16:47] <p3lim> you might need to end your url/path with "/" indeed, I know I've done that before
[16:47] <MTecknology> There's a spot in documentation somewhere that claims the trailing slash must be there
[16:47] <p3lim> use it to be safe :)
[16:48] <MTecknology> I have been
[16:48] <p3lim> dbungert: do you know how I can verify/test the "match" keypair from curtin without running the entire autoinstall process? can't seem to get a match against "model" (which is supposed to be ID_VENDOR in udev according to the docs)
[16:49] <MTecknology> I guess ... I'm sharing the arguments that are absolutely definitely being passed to the installer, so the real question is why the installer seems to do nothing with that.
[16:49] <MTecknology> I'm not sure if it's trying and not being logged, but netstat suggests it's not being tried at all
[16:49] <MTecknology> tcpdump*
[16:50] <p3lim> are you actually using this for 20.04?
[16:50] <MTecknology> yep ... I get to target 18.04, 20.04, 22.04, and there's soon to be a fourth in that list.
[16:51] <dbungert> p3lim: I'm not sure we have a great way to verify match directives, that sounds like a nice feature improvement.  I'm jotting that down as a feature request.
[16:51] <p3lim> dbungert: you're on the curtin team?
[16:52] <dbungert> I own the Subiquity stack, so all of Subiquity/Curtin/Probert, but I'm not the sole decision maker
[16:52] <p3lim> ah cool
[16:52] <p3lim> please jot down a feature request for configuring btrfs subvolumes too then :D
[16:53] <p3lim> anyways, I cannot for the life of me tell what curtin is actually trying to match against, because it just fails whenever I supply the `match.model: foo` field
[16:53] <dbungert> MTecknology: if you're testing newer ISOs, there will be a log line in /var/log/installer/subiquity-server-debug.log load_autoinstall_config
[16:54] <dbungert> if subiquity sees the autoinstall, that will have useful information, or maybe just a 'None' if it didn't make it that far
[16:54] <p3lim> to be specific, I'm using Dell EMC hardware and ID_VENDOR is "DELL     " and I've set `match.model: DELL*` (tried without the glob too) and it doesn't work
[16:54] <MTecknology> there's no /var/log/installer on this one
[16:54] <dbungert> p3lim: I actually have a branch started but it's not a committed feature at this point
[16:54] <p3lim> cool :)
[16:55] <dbungert> p3lim: (with respect to btrfs and subvolumes)
[16:55] <p3lim> if there are steps I can take to test the btrfs branch let me know, would gladly help once it's kinda-ready for that
[16:55]  * MTecknology screams ... this is a horrible mess that is driving me absolutely bonkers. How the frick is it so dang difficult just to feed a configuration file to an installer?!!
[16:56] <p3lim> MTecknology: try using http instead of https
[16:56] <p3lim> certificates can pose a problem here
[16:56] <p3lim> and ensure it can actually resolve the address (or just use IP)
[16:56] <MTecknology> It's not making a web request ... cart in front of the horse
[16:56] <p3lim> can you wget/curl the file from the installation image?
[16:57] <MTecknology> yep, I *just* tried that, actually.
[16:57] <p3lim> 🤷
[16:57] <dbungert> MTecknology: ok, so do you have /var/log/cloud-init.log ?  That should have information on the attempt to process the nocloud-net datasource
[16:59] <MTecknology> heh.. there's also no /var/log/cloud-init.log.  In fact, the only thing at all in /var/log is syslog
[17:02] <dbungert> that one is beyond me, something very wrong is going on in the ephemeral install environment
[17:02] <dbungert> MTecknology: what install ISO are you looking at?
[17:03] <MTecknology> I'm using the ubuntu-server-20.04 iso to try to install a desktop of the same version
[17:03] <dbungert> MTecknology: which one?  20.04.6?
[17:04] <MTecknology> yep, ubuntu-20.04.6-live-server-amd64.iso / sho256: b8f31413336b9393ad5d8ef0282717b2ab19f007df2e9ed5196c13d8f9153c8b
[17:04] <MTecknology> sha*
[17:06] <p3lim> super weird
[17:07] <p3lim> I just tested that exact same image in libvirt just now, although using localpath, worked just fine
[17:09] <p3lim> dbungert: has there been any thought of simplifying the build tooling for ISOs? I currently have 300+ lines mess of a bash script around xorriso, and I feel like it could break at any second :(
[17:09] <p3lim> feel free to finish helping MT before answering that
[17:09] <MTecknology> lol ... I think canonical is trying to abstract the abstraction as much as abstractly possible
[17:12] <MTecknology> I get the impression any attempts to help me have been exhausted without having control of my keyboard. I'm back to the mercy of documentation ... and soon to be comments.
[17:13] <p3lim> I still think it's your cmdline being incorrect :/
[17:13] <p3lim> considering the info provided
[17:14] <p3lim> have you tried booting the ISO and manually typing in the cmdline in interactive grub?
[17:14] <MTecknology> I assumed the backslash was specifically for grub escaping, so by the time the system is booting, the "\;" will be escaped to a ;
[17:14] <MTecknology> If it is truly supposed to remain a \; into cmdline, then I have only accomplished this once.
[17:14] <p3lim> it is, see the above message from me where I dumped my /proc/cmdline
[17:14] <p3lim> it's supposed to be just ";" between ds=nocloud and s=
[17:15] <dbungert> MTecknology: where did the kernel and initrd come from?  It's best to match them to the one from the iso, and can be found at casper/{vmlinuz,initrd}
[17:15] <MTecknology> dbungert: I grabbed them from the iso, per some other instructions that I linked to but can't find again
[17:16] <dbungert> ok.  I think at this point you need to understand why the install environment seems to not be coming up normally.  journalctl may have errors.
[17:19] <dbungert> p3lim: ubuntu-image has that as a goal, but it's not supporting ISO creation yet
[17:20] <p3lim> so the official images is also built using a messy bash script? :D
[17:27] <dbungert> you may or may not be interested in https://pad.lv/c/livecd-rootfs, p3lim
[17:28] <dbungert> but I think you already know the answer
[17:28]  * MTecknology is curious what's behind that link
[17:30] <p3lim> I just repackage the ISO with the nocloud files
[17:30] <p3lim> can't host anything in our prov environment
[17:32] <dbungert> p3lim: don't worry about it if you have a working solution, I'm just pointing to some of the build software in question
[17:32] <p3lim> it's working now, might not work tomorrow :D
[17:32] <p3lim> that's why I'm asking :)
[17:33] <p3lim> once it breaks I'll just see if either ubuntu-image has been improved, or look at the actual tooling ya'll use
[17:34] <MTecknology> p3lim: If you can tolerate static images, live-build is pretty nifty
[17:35] <dbungert> MTecknology: I need to be afk, you're welcome to ping or leave comments here and I'll check later
[17:40] <MTecknology> dbungert: Thanks; It occurred to me that I'm probably trying to rely too heavily on inaccurate and missing documentation, and it appears to not exist because it's always changing
[17:41] <p3lim> no wonder I can't get anything to work
[17:42] <p3lim> the docs are incorrect
[17:43] <p3lim> in the docs, disk selection extensions says model=ID_VENDOR, but it's actually ID_MODEL
[17:43] <p3lim> https://github.com/canonical/subiquity/blob/server/jammy/subiquity/models/filesystem.py#L999-L1001
[17:44] <p3lim> vendor=ID_VENDOR isn't even documented
[17:46] <MTecknology> Did I just ... are rotational and ssd the only options? what about tape? :P
[17:46] <MTecknology> s/Did I just .../the answer was no and I was going to delete it
[17:50] <MTecknology> ya know ... I /do/ have ipxe readily available. I was using this earlier and it provided absolute proof of network connectivity, since that's where the two payloads came from. It just means I also have to go back inside because I don't want to make a network cable long enough for me to play with that outisde. :/
[17:51] <MTecknology> #ITproblems :P
[17:54] <MTecknology> does ubuntu not have build-essential?
[17:56] <MTecknology> oooooh, lol ... there's no sources.list present. That must be one of the final setup things offloaded to cloud-init on the first boot.
[17:57] <MTecknology> dbungert: Is there any chance you can make this part easy for me and tell me what things cloud-init is actually responsible for on the first startup? So far it seems to be sources.list, one of the lists of package installs, and the first user
[18:09] <dbungert> p3lim: new subiquity stable snaps are "backported" to newer ISO builds, so that jammy branch is relevant for 22.04.0 ISOs but https://github.com/canonical/subiquity/blob/ubuntu/mantic/subiquity/models/filesystem.py#L1533 would be relevant instead for 22.04.4
[18:12] <p3lim> good to know, but docs are incorrect regardless
[18:12] <p3lim> I've reported the docs issue on launchpad
[18:20] <dbungert> MTecknology: user creation does happen at first boot of the target system.  the `packages` directive of autoinstall is the same.  /etc/apt/sources.list really should be present.  It might be interesting to add `while [ ! -f /run/finish ] ; do sleep 1 ; done` to autoinstall.late-commands, which will halt the install just before reboot to the target system and let you poke around to see before
[18:20] <dbungert> first-boot stuff runs.
[18:22] <MTecknology> Can I provide that (autoinstall.late-commands) as a boot argument?
[18:23] <dbungert> no, unfortunately
[18:24] <MTecknology> oh ... this isn't doing an automatic reboot. I haven't figured out the autoinstall, but I also need a mental break from it so I'm moving on to doing a manual install that does the same steps.
[18:24] <MTecknology> so "late-commands" is just what I type after it says updates are done, and early-commands is before giving the installer any input.
[18:26] <dbungert> right, late-commands are at install time but before first boot, one of the last steps.  early-commands are before anything has really started.
[18:30] <MTecknology> Oh! Is there any way to get input from the user? I'd like to prompt for a password and dump that to a text file for another late-command to pick up ... or is it possible to let the script in the late-command directly request the password?  (the command invoked is gpg - https://github.com/MTecknology/teckhost/blob/master/bootstrap#L225-L226)
[18:31] <MTecknology> for debian, I use https://github.com/MTecknology/teckhost/blob/master/iso/debconf_early
[18:42] <dbungert> MTecknology: for prompting for supported things like the first user password or the LUKS passphrase, that can be done, but for arbitrary inputs like your gpg case I have no answer.
[18:43] <MTecknology> oh ...
[18:44] <MTecknology> I guess that's something that would have to be offloaded to firstboot and written to interact with the boot process, similar to cryptsetup
[18:45] <MTecknology> I don't actually have any "firstboot" in my personal deployment. As soon as the installer shuts down at the end, it's completely configured and ready to ship (usually to family). I can even walk family members through doing a remote installation with any hardware they pick up at some cheap box store
[18:56] <MTecknology> dbungert: Oh .. how can I use "curtin" from another tty? I assumed that it would behave similar to in-target, but that doesn't seem to be the case.
[18:57] <MTecknology> I guess that's confusing since there's "curtin in-target" vs. "in-target" .. I was using the latter
[18:59] <dbungert> on recent ISOs you can get to curtin with `subiquity.curtin`, so for example you can do `subiquity.curtin in-target -i -t /target -- /bin/bash` to get a live shell chrooted in, in exactly the same mount setup
[19:06] <MTecknology> I saw lots of warnings against using different releases for different installs. Does that mean I'm stuck without for this?
[19:08] <MTecknology> It's mostly the removal of (I think) sg3-utils that causes a problem when run inside of chroot at the end of the install, because it needs /dev to be remounted.
[19:13] <MTecknology> lol ... apparently "subiquity-shell" is not what I thought it was
[19:13] <MTecknology> that seems to be the user interface bit?
[19:24] <dbungert> MTecknology: you can refresh the snap to the latest version if you want newer features.  The subiquity snap does not refresh itself by default, or you can say `refresh-installer: { update: true }` if you do want that.
[19:25] <dbungert> and yes subiquity-shell is internals stuff, I doubt it's very helpful when directly invoked
[19:25] <MTecknology> I had the impression that updating the snap was only an option for newer releases
[19:25] <MTecknology> like, update the snap, and get updated ubuntu
[19:35] <dbungert> they are separate items - if you use a 22.04.x iso you are getting Ubuntu 22.04.  Refreshing the snap helps if you want to get newer features or bugfixes but it doesn't change the OS version being installed.
[19:36] <MTecknology> gotcha, thanks for clarifying
[19:48] <MTecknology> "Also, please note that software in universe WILL NOT receive any review or updates from the Ubuntu security team."  ... but ... isn't that enabled by default?
[19:56] <MTecknology> dbungert: Which is actually canonical; security.ubuntu.com or archive.ubuntu.com foo-security?
[19:58] <dbungert> archive.ubuntu.com can be a different mirror, security updates hit security.ubuntu.com first.  This has more details:  https://wiki.ubuntu.com/SecurityTeam/FAQ#What_repositories_and_pockets_should_I_use_to_make_sure_my_systems_are_up_to_date.3F
[19:58] <MTecknology> thanks
[21:26] <MTecknology> dbungert: oh ... I'm finally getting some logs. Now it seems to be failing because vendor-data is missing. Is that a required-empty file?
[21:28] <MTecknology> https://dpaste.com/F7KUQ44MZ
[21:29] <dbungert> AIUI vendor-data is optional, I see that also in successful cases.
[21:32] <MTecknology> That's the only thing standing out to me right now. It seems like cloud-init is finally trying to grab the "cloud-config" files, but it still seems like there's no impact to the installation
[21:33] <MTecknology> oooh ... 2024-03-11 21:21:32,837 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 33 column 49: "mapping values are not allowed here
[21:34] <MTecknology> - printf "network:\n  version: 2\n  renderer: NetworkManager" >/target/etc/netplan/nm.yaml
[21:34] <MTecknology> that should be wrapped entirely in quotes
[21:38] <MTecknology> oooh next fun challenge
[21:40] <MTecknology> I guess "install to largest disk" is the best strategy here
[23:40] <minimal> note the WARNING about "netplan apply" failing, that doesn't look good