=== esv_ is now known as esv
[06:37] <dch> https://www.irccloud.com/pastebin/D07fSXRI/userdata
[06:38] <dch> I've tried both sending this at VM invocation, and also manually in `cloud.cfg.d/90_userdata.cfg` and it doesn't seem ever to get started
[06:38] <dch> https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg
[06:40] <dch> this is the cloud.cfg from the same system, which does correctly install its own ssh keys, but my userdata one never gets run
[06:40] <dch> `cloud-init schema --system --annotate` says that `Valid cloud-config: user-data` is fine
[06:41] <dch> running `# cloud-init analyze show` looks like userdata is run, but awfully fast https://www.irccloud.com/pastebin/O5n4kMSo/analyse-show
[06:41] <dch> which is suspicious
[07:17] <meena> dch: morning
[07:17]  * dch waves
[07:18] <meena> dch: which part doesn't work? 
[07:19] <dch> I can see that the userdata is correctly fetched, `cloud-init query userdata` post boot shows me the expected file
[07:19] <dch> but it doesn't get run
[07:19] <dch> I mean, the yaml "commands" in the userdata file aren't executed
[07:20] <meena> I would kill -SIGALRM $RC_PID (or whatever the variable name i documented)
[07:21] <dch> meena: I don't even think it gets that far
[07:22] <dch> I replaced that with a simple `touch /var/tmp/bootcmd_was_here` and will re-try
[07:27] <dch> nope, nothing happens
[07:28] <dch> dagnabdit
[07:37] <dch> I can see the userdata being fetched to the appropriate file
[07:37] <dch> but I don't see anywhere userdata being *merged in* to the existing data
[07:37] <dch> which I would expect to see
[07:38] <dch> maybe this assumption of mine is wrong
[07:44] <dch> https://www.irccloud.com/pastebin/FpY3A8MZ/setup-datasource-and-consume-user-data
[07:46] <dch> the merged file /var/lib/cloud/instances/11cabbbf-8dd7-446f-91bc-a1ec48b17801/cloud-config.txt appears valid (has the 984 bytes in it)
[07:47] <dch> SUCCESS: reading and applying user-data
[07:47] <dch> by this point I would expect it to know the new keys/values introduced
[07:49] <dch> https://www.irccloud.com/pastebin/4hhwWwVb/scripts-user-module
[07:50] <dch> all modules run successfully tho
[07:54] <aciba> Does the output of `cloud-init query userdata` contain your user-data?
[07:57] <dch> aciba yes it does
[07:58] <aciba> you have manually modified `cloud.cfg.d/90_userdata.cfg`, and some modules only run on first boot, did you perform a `cloud-init clean` after rebooting?
[07:59] <aciba> s/after/before/
[08:00] <dch> yes
[08:00] <dch> in fact I am now starting from a fresh slate each time
[08:03] <aciba> assuming this is your userdata:  https://www.irccloud.com/pastebin/D07fSXRI/userdata, could you please share `/var/log/cloud-init.log` or the output of `cloud-init collect-log`, making sure that any sensitive information is redacted?
[08:05] <aciba> what parts of the userdata are not getting done? write_files, run_cmd ?
[08:08] <dch> sure, let me re-summon a box to get a fresh log (I just annihilated the last one)
[08:08] <dch> none of it is run
[08:10] <meena> dch: do you have logging enabled
[08:10] <dch> yep, always
[08:17] <dch> woop https://usercontent.irccloud-cdn.com/file/o02bSYTv/cloud-init.log
[08:18] <dch> aciba: full log 
[08:18] <dch> note it fails the first time through because network isn't up, and then is restarted when network is up
[08:19] <dch> this restart issue is addressed in next release from cloud provider but not yet live
[08:23] <aciba> I see
[08:23] <dch> I amended userdata slightly, dropping the pkill and just using a touch command
[08:23] <aciba> cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: (DataSourceEc2)
[08:24] <dch> aciba: right, this is expected the first time around, because there's no network available yet
[08:24] <dch> the next time, it works ok
[08:24] <dch> https://www.irccloud.com/pastebin/D07fSXRI/userdata has been refreshed
[08:24] <aciba> the datasource is not detected because the network is not up, thus the user-data is not pulled
[08:25] <dch> and the 2nd run the network is up, the datasource is there, the userdata is fetched, we can see it being merged, 
[08:25] <dch> 2023-10-19 08:14:45,812 - util.py[DEBUG]: Attempting to load yaml from string of length 1028 with allowed root types (<class 'dict'>,) ...
[08:25] <dch> 2023-10-19 08:14:45,815 - handlers.py[DEBUG]: finish: init-network/consume-user-data: SUCCESS: reading and applying user-data
[08:26] <dch> and checking /var/lib/cloud/instances/bbabee48-2061-41bb-80ee-0c7fbde96f33/cloud-config.txt has the expected data in it
[08:27] <aciba> in between first and next time, can you perform a `cloud-init clean` ? I think cloud-init does not fully execute your userdata because of the state created on the first boot.
[08:28] <aciba> and that second boot logs are not included in the file you shared
[08:28] <dch> I can't modify this part its on the vendor side
[08:29] <dch> yes the 2nd run is in the log
[08:29] <dch> see e.g. line 217
[08:30] <dch>     217 2023-10-19 08:14:43,492 - util.py[DEBUG]: Cloud-init v. 22.4-28-gaecdcbf8 running 'init' at Thu, 19 Oct 2023 08:14:43 +0000. Up 146.85823392868042 seconds.
[08:30] <dch> lines 267-~300 show successful metadata fetching
[08:30] <dch> so I can cloudinit clean now and re-run
[08:32] <dch> v2 run, not fresh boot, but after cloudinit clean & removing stale logs https://www.irccloud.com/pastebin/fPXcz5bB/cloud-init.txt
[08:33] <dch> same thing, userdata actions aren't being run
[08:36] <dch> with latest cloudinit-23.3, same issue (this is 22.4 by default)
[08:36] <dch> and as this system has already booted the metadata is returned immediately
[08:36] <dch> so we don't have a 2nd run attempt
[08:38] <aciba> I see that `Cloud-init v. 22.4-28-gaecdcbf8 running 'modules:final'` is present but there is not an entry corresponding to 'modules:config'.
[08:39] <aciba> https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg is interesting, it has fewer modules activated than usually
[08:41] <aciba> what modules do you see when running `cloud-init query combined_cloud_config.cloud_config_modules` and `cloud-init query combined_cloud_config.cloud_final_modules` ?
[08:43] <dch> https://www.irccloud.com/pastebin/d4HPBHsX/show-modules
[08:43] <aciba> That is the issue, only those modules are allowed to run.
[08:45] <dch> aciba: is scripts-user not the thing we need?
[08:46] <dch> I had a look this morning already through https://cloudinit.readthedocs.io/en/latest/reference/modules.html but didn't see anythiing different that should be used
[08:49] <aciba> what do you mean by scripts-user ?
[08:50] <aciba> https://canonical-cloud-init.readthedocs-hosted.com/en/latest/explanation/format.html#user-data-script ?
[08:50] <dch> aciba: so this has been super helpful. the issue then is that the modules defined in cloud.cfg need to be extended
[08:51] <dch> I added `- bootcmd` to cloud.cfg and now the bootcmd section runs
[08:51] <dch> which is great, but not so important for me
[08:51] <aciba> sweet!
[08:51] <dch> but we are on the right path
[08:51] <dch> I will need to find each failing section, and add them in
[08:52] <dch> this file is inherited from many years ago, from the cloud vendor, and I've honestly not looked at it
[08:53] <aciba> sounds good, reach out if encounter any problem :)
[08:54] <dch> presumably I can *add* settings via a new `cloud.cfg.d/99_my_stuff.cfg` 
[08:55] <dch> 🔥
[08:55] <dch> looking great
[08:55] <dch> ✅ user and groups created
[08:55] <dch> ✅ bootcmd works
[08:55] <dch> ✅ write_files works
[08:56] <aciba> awesome!
[08:59] <dch> ✅ sudoers.d
[10:08] <meena> dch: who packaged your cloud-init??
[10:10] <dch> meena: literally nobody knows, it was done back on 10.x I think and there isn't anybody left who knows who did it
[10:11] <dch> it has been handed down like a sacred scroll, mis-interpreted at each stage until ... today
[10:15] <meena> dch:  I would take a look at the current (dev) package config files and diff them. every release i try to improve things, and fix modules
[10:16] <dch> meena: yeah I think the best thing is to dump the existing config file completely and just use the .sample one you provide
[10:18] <dch> meena: does adding packages work in your latest versions?
[10:19] <dch> `packages:` "module"
[10:25] <meena> that's been working forever 
[10:25] <meena> i don't think i had to even do anything
[10:25] <meena> I would like to break it, by adding multi repo support, but one thing at a time
[10:50] <dch> didn't work here, but then who knows what's borked
[10:51] <dch> I don't have ssh_import_id: gh:dch working either
[10:51] <dch> even after merged in all the missing module names from the std install
[10:55] <meena> yeah, i think  ssh_import_id is Debian / Ubuntu specific 
[10:56] <dch> ok so a good haul, finally I am missing:
[10:57] <dch> ☠️ runcmd, gh:$USER, packages
[10:57] <meena> https://github.com/canonical/cloud-init/blob/9a401b4a21ea179452143ef8c97f261c1655c539/cloudinit/config/cc_ssh_import_id.py#L22
[10:57] <dch> and of those only packages is relevant, and at that, for freebsd, not very important
[10:57] <dch> good to work through these finally
[10:57] <meena> if it's good enough for alpine, it should be good enough for freebsd 
[10:58] <meena> https://www.freshports.org/security/ssh-import-id/
[10:58] <meena> it's just python
[10:59] <meena> i can see adding support for that
[10:59] <dch> +1 for that
[10:59] <meena> it should literally be four lines in four files 
[10:59] <dch> I'm calling this a day, thanks meena  and aciba for your endless help and patience
[10:59] <dch> good progress
[10:59] <dch> curl | tee
[11:00] <meena> it'd be cool if we had optional dependencies in FreeBSD, but failing that, I think it's no harm adding a single python module with no dependencies 
[11:03] <dch> ack
[11:03] <dch> by the time we add python itself, its already 100mb+
[11:27] <minimal> meena: yeah it wasn't hard adding ssh-import-id for Alpine
[12:16] <meena> minimal: cool, thanks.
[12:17] <meena> hrm, i was wrong about no dependencies: New packages to be INSTALLED:
[12:17] <meena>         py39-distro: 1.8.0_2 [FreeBSD]
[12:17] <meena>         py39-ssh-import-id: 5.11 [FreeBSD]
[13:20] <meena> dch: https://github.com/canonical/cloud-init/pull/4533 let's see if netbsd and dragonfly and openbsd also have that package, if so, I'll extend it to those, too.
[13:20] -ubottu:#cloud-init- Pull 4533 in canonical/cloud-init "cc_ssh_import_id: Extend support to FreeBSD" [Open]
[14:28] <meena> i see some discrepancy between distros = ["all"] and "distros": [ALL_DISTROS],
[14:57] <minimal> yupe and I believe I mentioned in the past it might make sense to also have ALL_LINUX_DISTROS and ALL_BSD_DISTROS (or similar)
[15:02] <meena> *nod*
=== sergiodj_ is now known as sergiodj