=== esv_ is now known as esv [06:37] https://www.irccloud.com/pastebin/D07fSXRI/userdata [06:38] I've tried both sending this at VM invocation, and also manually in `cloud.cfg.d/90_userdata.cfg` and it doesn't seem ever to get started [06:38] https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg [06:40] this is the cloud.cfg from the same system, which does correctly install its own ssh keys, but my userdata one never gets run [06:40] `cloud-init schema --system --annotate` says that `Valid cloud-config: user-data` is fine [06:41] running `# cloud-init analyze show` looks like userdata is run, but awfully fast https://www.irccloud.com/pastebin/O5n4kMSo/analyse-show [06:41] which is suspicious [07:17] dch: morning [07:17] * dch waves [07:18] dch: which part doesn't work? [07:19] I can see that the userdata is correctly fetched, `cloud-init query userdata` post boot shows me the expected file [07:19] but it doesn't get run [07:19] I mean, the yaml "commands" in the userdata file aren't executed [07:20] I would kill -SIGALRM $RC_PID (or whatever the variable name i documented) [07:21] meena: I don't even think it gets that far [07:22] I replaced that with a simple `touch /var/tmp/bootcmd_was_here` and will re-try [07:27] nope, nothing happens [07:28] dagnabdit [07:37] I can see the userdata being fetched to the appropriate file [07:37] but I don't see anywhere userdata being *merged in* to the existing data [07:37] which I would expect to see [07:38] maybe this assumption of mine is wrong [07:44] https://www.irccloud.com/pastebin/FpY3A8MZ/setup-datasource-and-consume-user-data [07:46] the merged file /var/lib/cloud/instances/11cabbbf-8dd7-446f-91bc-a1ec48b17801/cloud-config.txt appears valid (has the 984 bytes in it) [07:47] SUCCESS: reading and applying user-data [07:47] by this point I would expect it to know the new keys/values introduced [07:49] https://www.irccloud.com/pastebin/4hhwWwVb/scripts-user-module [07:50] all modules run successfully tho [07:54] Does the output of `cloud-init query userdata` contain your user-data? [07:57] aciba yes it does [07:58] you have manually modified `cloud.cfg.d/90_userdata.cfg`, and some modules only run on first boot, did you perform a `cloud-init clean` after rebooting? [07:59] s/after/before/ [08:00] yes [08:00] in fact I am now starting from a fresh slate each time [08:03] assuming this is your userdata:  https://www.irccloud.com/pastebin/D07fSXRI/userdata, could you please share `/var/log/cloud-init.log` or the output of `cloud-init collect-log`, making sure that any sensitive information is redacted? [08:05] what parts of the userdata are not getting done? write_files, run_cmd ? [08:08] sure, let me re-summon a box to get a fresh log (I just annihilated the last one) [08:08] none of it is run [08:10] dch: do you have logging enabled [08:10] yep, always [08:17] woop https://usercontent.irccloud-cdn.com/file/o02bSYTv/cloud-init.log [08:18] aciba: full log [08:18] note it fails the first time through because network isn't up, and then is restarted when network is up [08:19] this restart issue is addressed in next release from cloud provider but not yet live [08:23] I see [08:23] I amended userdata slightly, dropping the pkill and just using a touch command [08:23] cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: (DataSourceEc2) [08:24] aciba: right, this is expected the first time around, because there's no network available yet [08:24] the next time, it works ok [08:24] https://www.irccloud.com/pastebin/D07fSXRI/userdata has been refreshed [08:24] the datasource is not detected because the network is not up, thus the user-data is not pulled [08:25] and the 2nd run the network is up, the datasource is there, the userdata is fetched, we can see it being merged, [08:25] 2023-10-19 08:14:45,812 - util.py[DEBUG]: Attempting to load yaml from string of length 1028 with allowed root types (,) ... [08:25] 2023-10-19 08:14:45,815 - handlers.py[DEBUG]: finish: init-network/consume-user-data: SUCCESS: reading and applying user-data [08:26] and checking /var/lib/cloud/instances/bbabee48-2061-41bb-80ee-0c7fbde96f33/cloud-config.txt has the expected data in it [08:27] in between first and next time, can you perform a `cloud-init clean` ? I think cloud-init does not fully execute your userdata because of the state created on the first boot. [08:28] and that second boot logs are not included in the file you shared [08:28] I can't modify this part its on the vendor side [08:29] yes the 2nd run is in the log [08:29] see e.g. line 217 [08:30] 217 2023-10-19 08:14:43,492 - util.py[DEBUG]: Cloud-init v. 22.4-28-gaecdcbf8 running 'init' at Thu, 19 Oct 2023 08:14:43 +0000. Up 146.85823392868042 seconds. [08:30] lines 267-~300 show successful metadata fetching [08:30] so I can cloudinit clean now and re-run [08:32] v2 run, not fresh boot, but after cloudinit clean & removing stale logs https://www.irccloud.com/pastebin/fPXcz5bB/cloud-init.txt [08:33] same thing, userdata actions aren't being run [08:36] with latest cloudinit-23.3, same issue (this is 22.4 by default) [08:36] and as this system has already booted the metadata is returned immediately [08:36] so we don't have a 2nd run attempt [08:38] I see that `Cloud-init v. 22.4-28-gaecdcbf8 running 'modules:final'` is present but there is not an entry corresponding to 'modules:config'. [08:39] https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg is interesting, it has fewer modules activated than usually [08:41] what modules do you see when running `cloud-init query combined_cloud_config.cloud_config_modules` and `cloud-init query combined_cloud_config.cloud_final_modules` ? [08:43] https://www.irccloud.com/pastebin/d4HPBHsX/show-modules [08:43] That is the issue, only those modules are allowed to run. [08:45] aciba: is scripts-user not the thing we need? [08:46] I had a look this morning already through https://cloudinit.readthedocs.io/en/latest/reference/modules.html but didn't see anythiing different that should be used [08:49] what do you mean by scripts-user ? [08:50] https://canonical-cloud-init.readthedocs-hosted.com/en/latest/explanation/format.html#user-data-script ? [08:50] aciba: so this has been super helpful. the issue then is that the modules defined in cloud.cfg need to be extended [08:51] I added `- bootcmd` to cloud.cfg and now the bootcmd section runs [08:51] which is great, but not so important for me [08:51] sweet! [08:51] but we are on the right path [08:51] I will need to find each failing section, and add them in [08:52] this file is inherited from many years ago, from the cloud vendor, and I've honestly not looked at it [08:53] sounds good, reach out if encounter any problem :) [08:54] presumably I can *add* settings via a new `cloud.cfg.d/99_my_stuff.cfg` [08:55] 🔥 [08:55] looking great [08:55] ✅ user and groups created [08:55] ✅ bootcmd works [08:55] ✅ write_files works [08:56] awesome! [08:59] ✅ sudoers.d [10:08] dch: who packaged your cloud-init?? [10:10] meena: literally nobody knows, it was done back on 10.x I think and there isn't anybody left who knows who did it [10:11] it has been handed down like a sacred scroll, mis-interpreted at each stage until ... today [10:15] dch: I would take a look at the current (dev) package config files and diff them. every release i try to improve things, and fix modules [10:16] meena: yeah I think the best thing is to dump the existing config file completely and just use the .sample one you provide [10:18] meena: does adding packages work in your latest versions? [10:19] `packages:` "module" [10:25] that's been working forever [10:25] i don't think i had to even do anything [10:25] I would like to break it, by adding multi repo support, but one thing at a time [10:50] didn't work here, but then who knows what's borked [10:51] I don't have ssh_import_id: gh:dch working either [10:51] even after merged in all the missing module names from the std install [10:55] yeah, i think ssh_import_id is Debian / Ubuntu specific [10:56] ok so a good haul, finally I am missing: [10:57] ☠️ runcmd, gh:$USER, packages [10:57] https://github.com/canonical/cloud-init/blob/9a401b4a21ea179452143ef8c97f261c1655c539/cloudinit/config/cc_ssh_import_id.py#L22 [10:57] and of those only packages is relevant, and at that, for freebsd, not very important [10:57] good to work through these finally [10:57] if it's good enough for alpine, it should be good enough for freebsd [10:58] https://www.freshports.org/security/ssh-import-id/ [10:58] it's just python [10:59] i can see adding support for that [10:59] +1 for that [10:59] it should literally be four lines in four files [10:59] I'm calling this a day, thanks meena and aciba for your endless help and patience [10:59] good progress [10:59] curl | tee [11:00] it'd be cool if we had optional dependencies in FreeBSD, but failing that, I think it's no harm adding a single python module with no dependencies [11:03] ack [11:03] by the time we add python itself, its already 100mb+ [11:27] meena: yeah it wasn't hard adding ssh-import-id for Alpine [12:16] minimal: cool, thanks. [12:17] hrm, i was wrong about no dependencies: New packages to be INSTALLED: [12:17] py39-distro: 1.8.0_2 [FreeBSD] [12:17] py39-ssh-import-id: 5.11 [FreeBSD] [13:20] dch: https://github.com/canonical/cloud-init/pull/4533 let's see if netbsd and dragonfly and openbsd also have that package, if so, I'll extend it to those, too. [13:20] -ubottu:#cloud-init- Pull 4533 in canonical/cloud-init "cc_ssh_import_id: Extend support to FreeBSD" [Open] [14:28] i see some discrepancy between distros = ["all"] and "distros": [ALL_DISTROS], [14:57] yupe and I believe I mentioned in the past it might make sense to also have ALL_LINUX_DISTROS and ALL_BSD_DISTROS (or similar) [15:02] *nod* === sergiodj_ is now known as sergiodj