[15:15] <meena> this is ready to review https://github.com/canonical/cloud-init/pull/4334
[15:15] -ubottu:#cloud-init- Pull 4334 in canonical/cloud-init "cc_mounts: xfs is a Linux only FS" [Open]
[15:50] <blackboxsw> Thanks meena, we'll unpack from last week's event and get through the review queue.
[15:51] <blackboxsw>  [community notice] We are working toward the next upstream quarterly release of cloud-init 23.3. If there are critical bugs or pull requests that you would like to see resolved in 23.3. Please raise them for review/inclusion and we'll do our best to make sure we get eyes on those concerns.
[15:52] <blackboxsw> we're spending today scrubbing our integration test runner logs to ensure we are in good shape prior to making a 'cut' from tip of main. Expectation is that we are a couple days out.
[16:02] <blackboxsw> in scrubbing integration tests runners, we've got some integration tests timing out on Ubuntu Mantic that need resolution, cloud-init status --wait never returns. I'll peek at that right now and propose a fix and will report any other  integration test issues that would block 23.3 a bit later today
[16:05] <minimal> looking for some help regarding a testcase. I had a test working find, then as I changed functionality of the module that it is testing I had to modify the testcase to use FilesystemMockingTestCase, however now that testcase is failing as the caplog is no longer handled
[16:05] <minimal> I seeing a "TypeError: TestKeyboard.test_alpine_linux_ignore_model() missing 1 required positional argument: 'caplog'
[16:18] <blackboxsw> minimal: if that unit test declares a @pytest.param it typically means that one of the named params or pytest.mocks decorating the function were not provided earlier than, caplog (as caplog is typically the last fixture used in a unittest)
[16:19] <blackboxsw> ahh I see but you've shifted to FilesystemMockingTestCase (and away from pytest and fixtures)
[16:21] <blackboxsw> ok let's have a look at the unittest you have to see if there is somewhat to avoid FilesystemMockingTestCase (as that's the directly we ideally want to go anyway long-term) if not, there's always setting up `with_logs = True` on the test instance and calling self.logs to process the logged messages
[16:22] <minimal> To be honest I'm not familiar with all this testing stuff. I switched to FilesystemMockingTestCase as I'm using helper.populate_dir
[16:24] <minimal> let me create a gist with the test file
[16:24] <blackboxsw> 'tis okay. There is a lot of historic use of FilesystemMockingTestCase throughout the codebase and that ends up deriving from CITestCase (which predated cloud-init's use of pytest and fixtures etc)  I'm sure we can get eyes on the work-in-progress test to see what can be done to get the test feature and/or fixture you need to make that work.
[16:25] <blackboxsw> long-term goal: I think we want to get rid of CITestCase if we can, but that's a ways out.
[16:25] <blackboxsw> especially given the need for some of the more complex FileSystemMockingTestCase uses
[16:28] <minimal> https://gist.github.com/dermotbradley/fe9ff6ce8c2e10a25838387f544a209c
[17:34] <blackboxsw> just came out of the woods on a strange symptom I'm seeing on Mantic where cloud-config.service doesn't report `exited` in systemd https://github.com/canonical/cloud-init/issues/4344
[17:34] -ubottu:#cloud-init- Issue 4344 in canonical/cloud-init "Ubuntu 23.10: cloud-init status --wait blocks indefinitely on cloud-config.service when APT keys in user-data" [Open]
[17:34] <blackboxsw> looking over the gist now
[17:56] <minimal> thanks
[17:57] <ShaneAH> Hello All, I am new to using cloud-init and I'm wondering if what I'm trying to do is advisable given the issues I'm having.  I am attempting to create a golden image VM (Ubuntu in Azure) with pre-defined mounts according to CIS/STIG requirements.  I'm using packer+cloud-init to create, format and mount the volumes and build an image that goes
[17:57] <ShaneAH> into our gallery.  All that seems to work fine, the problem comes in when I create a new VM from the image the mounts are removed from /etc/fstab.  It seems to be due to code in cc_mounts.py that wants to remove fstab entries with comment=cloudconfig in them.  Am I trying to use cloud-init incorrectly?
[17:58] <minimal> ShaneAH: I think you'd need to provide more information regarding your issue
[17:58] <minimal> what do those entries in /etc/fstab look like?
[17:59] <minimal> are you seeing any errors in /var/log/cloud-init.log?
[18:03] <ShaneAH> The fstab entry in question is.
[18:03] <ShaneAH> I see in the log file that when I build the image with packer the fstab entry is added but when I build a VM from the image it's being removed.  The cloud.config file I'm being used is not being passed to the VM build process.
[18:10] <ShaneAH> Well, that didn't post.
[18:11] <ShaneAH> dev/disk/azure/scsi1/lun0-part1        /storage        auto    defaults,nofail,comment=cloudconfig 0       2
[18:11] <ShaneAH> that's the fstab line that is added by my packer build.
[18:11] <ShaneAH> When I create a VM that line is removed.
[18:17] <meena> holmanb: thanks for the feedback. I think i need to rethink this patch. util.is_Linux() doesn't make sense at that point in the code.
[18:17] <holmanb> meena: sounds good
[18:18] <meena> holmanb: fstype = util.get_mount_info(swap_dir)[1] is the function returning xfs, or btrfs, so we can only be on linux. So it's just the tests that need to reflect that fact.
[18:20] <holmanb> =1
[18:20] <holmanb> +1
[18:24] <ShaneAH> This is my mounts entry
[18:24] <ShaneAH> mounts:
[18:24] <ShaneAH>  - [ "/dev/disk/azure/scsi1/lun0-part1", "/storage", "auto", "defaults,nofail", "0", "2" ]
[18:27] <minimal> ShaneAH: I don't understand what the actual problem is that you're having though
[18:27] <minimal> I assume that you ran packer on Azure and so when building the image cloud-init added the entry to /etc/fstab
[18:28] <minimal> so why is it's removal upon VM boot a problem?
[18:31] <ShaneAH> Yes, running packer to create a VM image in Azure and cloud-init correctly adds the entry to FS tab.  I can use -debug in packer and see that the entry exists.  But when I build a VM from that image cloud-init runs again but this time with no configuration (as it was done via packer).  There is code in cc_mounts.py that looks like it is scanning
[18:31] <ShaneAH> every line in fstab for comment=cloudconfig and removing any line that matches that.
[18:32] <ShaneAH> line 452 in cc_mounts.py.
[18:35] <ShaneAH> 2023-08-12 21:33:02,717 - cc_mounts.py[DEBUG]: Changes to fstab: ['+ /dev/disk/azure/scsi1/lun0-part1 /storage auto defaults,nofail,comment=cloudconfig 0 2', '+ /dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,_netdev,comment=cloudconfig 0 2']
[18:35] <ShaneAH> 2023-08-14 18:32:25,911 - cc_mounts.py[DEBUG]: Changes to fstab: ['- /dev/disk/azure/scsi1/lun0-part1 /storage auto defaults,nofail,comment=cloudconfig 0 2']
[18:35] <ShaneAH> Note the 21:33:02 line that is packer running and it adds the fstab line.  The 18:32 is a VM I built today from that image and you can see it is removing the line.
[18:37] <ShaneAH> removal on VM boot is going to become a problem because once I figure this out my plan is to replace /var /var/log /home and a few other folders with mount points.
[18:43] <minimal> AFAIK cloud-init is removing that entry as it is a cloud-init "managed" entry and as you didn't specify the mount in your user-data
[18:44] <ShaneAH> I guess what I'm expecting is that I can build a base golden image using cloud-init, and then othere can build on top of that image also using cloud-init w/o having to reference my original cloud-config file.  Is that an unreasonable assumption?
[18:44] <blackboxsw> minimal: https://paste.opendev.org/show/bRonpunIoLDwBbfXUPor/ for a quick 'fix' using FileSystemMockingTest case . sry for delay
[18:46] <meena> holmanb: done.
[18:48] <minimal> ShaneAH: so why not put whatever config you need in /etc/cloud/cloud.cfg.d/ then?
[18:51] <ShaneAH> hmm, so instead of handing packer the config file add the configuration I want to /etc/cloud/cloud.cfg.d and put it in the right place from an order standpoint?
[18:53] <minimal> ShaneAH: when you run packer and pass user-data that user-data only applies to the run of cloud-init, I expect at the end of your packer run you do clean up cloud-init properly, which then makes cloud-init on next boot "forget" about any previous user-data
[18:57] <ShaneAH> I'm not running the cleanup yet, I did see files in instances that looked like they needed to be cleaned up just hadn't gotten there yet.  I guess the trick would be to get the files on to the VM early enough for cloud-init to pick them up?
[18:59] <minimal> ShaneAH: each time cloud-init runs at boot one of the first things I does is determine if this boot is a new instance or an existing instance and it behaves differently for those cases. When creating a "cloud image" you should be cleaning up so whenever the image is booted it treats that 1st boot as a new instance
[19:03] <ShaneAH> that make sense, I couldn't determine how to persist my changes.  Most of the docs and examples seem to lean to a single run scenario.
[19:04] <ShaneAH> I just added a packer provisioner to copy my config file into cloud.cfg.d.
[19:10] <minimal> blackboxsw: Thanks for the help, the seems to fix things (it is failing for an unrelated reason now)
[19:25] <esv> hey folks, I am trying to read the page for https://cloudinit.readthedocs.io/en/20.2/reference/index.html but I get a 404 error, I got the link from selecting the 20.2 version on the cloudinit.readthedocs.io page 
[19:27] <esv> is user-data support added after 20.2 ? 
[19:27] <ShaneAH> @esv https://cloudinit.readthedocs.io/en/20.2/
[19:28] <esv> thnx
[19:38] <holmanb> meena: thanks
[19:38] <holmanb> merged
[19:44] <blackboxsw> @esv  that 404 looks like comes from the restructuring that took place to group all "reference" material like module documentation/user-data/instance-data etc under a new top-level "references/index.html" subdirectory in cloud-init version 22.4.3. so it looks like clicking into the version tab in readthedocs from that specific page  will incorrectly redirect to the specific sub-route 20.2/reference/index.html which is absent
[19:45] <dbungert> in a nocloud scenario, what does the local-hostname actually do?
[19:50] <meena> anyone got input on https://github.com/canonical/cloud-init/issues/4332#issuecomment-1677412596 ? I'm not sure how silly this is
[19:50] -ubottu:#cloud-init- Issue 4332 in canonical/cloud-init "util.py: get_proc_env() doesn't work on FreeBSD" [Open]
[19:52] <blackboxsw> esv: we can setup redirects that'll push you through to the top-level of 20.2 version docs for version 20.2 (or any version prior to 22.4.3, so you don't see the 404 on that index page, but beyond that I'm not sure what else we can do here.
[19:53] <ShaneAH> minimal I think you're suggestion is working.  I need to do it in 2 parts.  Continue passing my cloud.init to get the disk partitioned and formatted and mounted in the VM for packer to touch, but then also add a config file to /etc/cloud/cloud.cfg.d that will add the fstab entry when the image is actually used to create a VM.
[19:56] <esv> I think I can remember to do it myself next time, if it is not something that comes too often, I don't see the value in it.
[19:57] <blackboxsw> dbungert: it allows cloud-init to invoke write out /etc/hostname and invoke `hostname YOUR_NAME` on the system to ensure the ephemeral hostname is set. 
[19:58] <dbungert> blackboxsw: I'm trying to imagine a scenario where that would make it to the target system and I can't see one, unless subiquity started looking for that value and passing it along.  The above is what I was assuming, thanks.
[20:04] <blackboxsw> dbungert: additionally the  init-local stage of cloud-init also sets the hostname early (before system network config is officially up) to ensure that any DHCP requests that are sent out by the VM contain the "send host-name" request which then affects DDNS setups. See: https://github.com/canonical/cloud-init/commit/133ad2cb327ad17b7b81319fac8f9f14577c04df for that context.
[20:04] -ubottu:#cloud-init- Commit 133ad2c in canonical/cloud-init "set_hostname: When present in metadata, set it before network bringup."
[20:04] <blackboxsw> dbungert: for subiquity, if /var/lib/cloud/seed/nocloud-net/meta-data contains a local-hostname key that'd provide that that content potentially.
[20:06] <dbungert> would it always be /var/lib/cloud/seed/nocloud-net/meta-data ? You're in the process of merging nocloud and nocloud-net, iiuc?  so it might also be at /var/lib/cloud/seed/nocloud/meta-data ?
[20:07] <blackboxsw> dbungert: also if kernel cmdline has either `ds=nocloud;h=my-hostname`  or `ds=nocloud;local-hostname=my-hostname` then it'd push that content into the nocloud datasource read of metadata
[20:12] <blackboxsw> dbungert: it could be at either /var/lib/cloud/seed/nocloud*/meta-data and could be either ds=nocloud-net or ds=nocloud on the kernel cmdline.
[20:12] <dbungert> blackboxsw: got it, thanks!
[20:14] <blackboxsw> though dbungert , nocloud will exit on the first path containing both the files meta-data and user-data in /var/lib/cloud/seed/nocloud, /var/lib/cloud/seed/nocloud-net. So, /v/l/c/seed/nocloud-net could get ignored if a /v/l/c/seed/nocloud/(meta-data|user-data) exists
[20:18] <dbungert> ok, so I guess we check nocloud first and fallback to nocloud-net
[20:23] <blackboxsw> dbungert: that aligns with ordering. I'm not certain why subiquity would be checking those files. But, to be clear, the files /v/l/cloud/seed/nocloud/ & nocloud-net are inputs to cloud-init for nocloud datasource detection, not output artifacts based on cloud-init discovering/identifying DataSourceNoCloud.
[21:23] <meena> let's see if i can fix all unit tests on FreeBSD before my python patches get merged
[22:07] <meena> and another one: https://github.com/canonical/cloud-init/pull/4346
[22:07] -ubottu:#cloud-init- Pull 4346 in canonical/cloud-init "cc_wireguard: make tests pass on FreeBSD" [Open]
[22:30] <meena> holmanb: i think that's the easy ones out of the way, the rest is gonna be gnarly
[22:44] <meena> the current set of failures: https://im.eena.me/uploads/117b1b4ff5000414/py3.8-tox.log 
[22:44] <meena> (please ignore the fact that I'm using Python 3.8…)
[22:45] <meena> (it's the last version i patched and built and tested, and stuck with it for today. The failures are the same up to 3.13)
[23:38] <meena> not sure this is the right way to fix this: https://github.com/canonical/cloud-init/pull/4347
[23:38] -ubottu:#cloud-init- Pull 4347 in canonical/cloud-init "analyze: fix (unexpected) timestamp parsing" [Open]
[23:38] <meena> but it works™
[23:43] <meena> works on FreeBSD lol