[16:52] <falcojr> blackboxsw: since you authored #1607, do you have strong opinions on https://bugs.launchpad.net/cloud-init/+bug/1999680 ?
[16:52] -ubottu:#cloud-init- Launchpad bug 1999680 in cloud-init "Fix remove machine-id" [Undecided, New]
[16:53] <falcojr> truncating the machine-id sounds reasonable to me, but I'm not sure if there are specific use cases that require us to remove the id
[17:02] <blackboxsw> falcojr: thanks for the ping I was going through that now trying to figure out why on our LXD images things are working fine .
[17:02] <blackboxsw> Bionic++
[17:03] <blackboxsw> I just want to find out why it may *not* be working on this other image.
[18:43] <falcojr> blackboxsw: I can see it on a bionic vm. Launch ubuntu:bionic --vm , rm /etc/machine-id, reboot
[20:19] <blackboxsw> strange as it does work on containers 
[20:26] <blackboxsw> I'm seeing /etc/machine-id created prior to journal being started up. maybe by dbus: /etc/init.d/dbus via a call to dbus-uuidgen --ensure. But that only resets /var/lib/dbus/machine-id. it doesn't 'publish' the new value up to /etc/machine-id of it's own accord. something else is doing that.... like systemd-machine-id-setup, but I don't see that invoked by any systemd service outside of systemd-machine-id-commit.service
[20:41] <blackboxsw> on both container and kvm, a trunated /etc/machine-id 'works' and gets regenerated as a new uuid. So, sure I think that is fine for us to do. I'm going to continue poking at the reason why bionic kvm images for lxc  don't t regenerate uuids but bionic containers still do. the timing of the generation of /etc/machine-id is before journald is started on containers, so I'm wondering if lxd is invoking something 
[20:41] <blackboxsw> I'll also test on other platfroms/clouds
[21:09] <meena> blackboxsw: i wonder how 11:56 <meena> oh ho! https://freshbsd.org/freebsd/src/commit/62a149bf621947fb7475c64b1ff04fe19fe16b29 — will affect FreeBSD in the future
[21:10] <meena> MFC one week, that was two weeks ago
[21:14] <blackboxsw> thanks a lot of the link on that meena and watching out for the conversation. Right, if we opted to allow for a truncated /etc/machine-id file, it'd break BSD which would leave that empty file in tact. 
[21:15] <blackboxsw> original context on the 'bug' in ubuntu from 8 years ago https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1508766 
[21:15] -ubottu:#cloud-init- Launchpad bug 1508766 in systemd (Ubuntu) "/etc/machine-id not created if missing" [Low, Won't Fix]
[21:15] <holmanb> blackboxsw: falcojr: truncating makes it not a "first boot" on newer systemd versions
[21:16] <holmanb> from machine-id(5): 3. If /etc/machine-id exists and is empty, a boot is not considered the first boot.
[21:16] <blackboxsw> not considered a first boot (which would likely invoke systemd-firstboot in older cases).
[21:16] <holmanb> which is a behavior I don't think we want
[21:17] <blackboxsw> *older ubuntu series* cases
[21:18] <blackboxsw> generally the cloud-init clean --machine-id is intended for folks trying to generate golden images which should perfrom all system setup from scratch as if it were a "greenfield" (or first time)" vm launch
[21:18] <holmanb> since that would cause units with ConditionFirstBoot=yes to not be run
[21:19] <holmanb> that's what I'm saying - truncating instead of deleting would make --machine-id change to not run those ConditionFirstBoot=yes units on newer systemd versions
[21:20] <falcojr> so sounds like our newer bug should be invalid/won't fix and somebody hitting that bug can just manually clear the machine id
[21:21] <holmanb> falcojr: I'd give a +1 to that approach
[21:21] <blackboxsw> yes a workaround would be for someone to `sudo truncate /etc/machine-id --size 0`. but this has to be peformed by some systemd unit that is not active in bionic kvm images but 'is' active in container images on bionic
[21:22] <holmanb> blackboxsw: I don't think that a systemd unit should be required for golden image creation.
[21:22] <blackboxsw> so I was fruitlessly trying to dig into what is actually triggering the re-generation of /etc/machine-id on boot, as it looks to be either dbus or systemd-machine-id-commit (yet the latter seems inert because it is only invoked with /etc/machine-id is a mount point)
[21:22] <holmanb> maybe I misunderstand 
[21:24] <blackboxsw> no a systemd unit shouldn't be required for the golden image creation, but I'd like to be able to explain in that won't fix bug why lxc container bionic images properly re-generate the /etc/machine-id across reboot, but bionic kvm images  do not. 
[21:24] <blackboxsw> so that we can explain to the bug filer what symptom is affecting their environment
[21:24] <holmanb> I see
[21:25] <falcojr> blackboxsw: that feels beyond our scope
[21:40] <blackboxsw> fair, though this does affect VMs of both bionic (systemd 237) and focal (systemd 245) it looks like, It doesn't affect jammy vms on (sytsemd 249)
[21:41] <blackboxsw> so we'll need to be aware that this workaround of trucating is necessary on older systemd versions.
[21:49] <blackboxsw> holmanb: and right. I confirmed behavior from systemd as far as expectations of 'first boot' https://github.com/systemd/systemd/blob/main/src/core/main.c#L2096-L2113
[21:50] <blackboxsw> so from this terraform case, it is a 'dirty boot' if folks are truncating to 0 length and expecting machine-id to be regenerated
[21:54] <blackboxsw> whereas v245 or less systemd doesn't support the "uninitiailized" value for machine-id but treats absence of the file as the same (first boot) schenario https://github.com/systemd/systemd/blob/v245/src/core/main.c#L1873-L1882
[22:14] <blackboxsw> and random https://systemd.io/BUILDING_IMAGES/ suggests the same 
[22:16] <blackboxsw> anyhow too much time spent on this. I think removing is intended for golden image creation. If someone has an environment that will be reusing those images without a full/first boot, then they likely will either have to `sudo truncate /etc/machine-id --size 0` or `sudo systemd-machine-id-setup` to generate a new uuid on the image just before booting