| === esv_ is now known as esv | ||
| dch | https://www.irccloud.com/pastebin/D07fSXRI/userdata | 06:37 |
|---|---|---|
| dch | I've tried both sending this at VM invocation, and also manually in `cloud.cfg.d/90_userdata.cfg` and it doesn't seem ever to get started | 06:38 |
| dch | https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg | 06:38 |
| dch | this is the cloud.cfg from the same system, which does correctly install its own ssh keys, but my userdata one never gets run | 06:40 |
| dch | `cloud-init schema --system --annotate` says that `Valid cloud-config: user-data` is fine | 06:40 |
| dch | running `# cloud-init analyze show` looks like userdata is run, but awfully fast https://www.irccloud.com/pastebin/O5n4kMSo/analyse-show | 06:41 |
| dch | which is suspicious | 06:41 |
| meena | dch: morning | 07:17 |
| * dch waves | 07:17 | |
| meena | dch: which part doesn't work? | 07:18 |
| dch | I can see that the userdata is correctly fetched, `cloud-init query userdata` post boot shows me the expected file | 07:19 |
| dch | but it doesn't get run | 07:19 |
| dch | I mean, the yaml "commands" in the userdata file aren't executed | 07:19 |
| meena | I would kill -SIGALRM $RC_PID (or whatever the variable name i documented) | 07:20 |
| dch | meena: I don't even think it gets that far | 07:21 |
| dch | I replaced that with a simple `touch /var/tmp/bootcmd_was_here` and will re-try | 07:22 |
| dch | nope, nothing happens | 07:27 |
| dch | dagnabdit | 07:28 |
| dch | I can see the userdata being fetched to the appropriate file | 07:37 |
| dch | but I don't see anywhere userdata being *merged in* to the existing data | 07:37 |
| dch | which I would expect to see | 07:37 |
| dch | maybe this assumption of mine is wrong | 07:38 |
| dch | https://www.irccloud.com/pastebin/FpY3A8MZ/setup-datasource-and-consume-user-data | 07:44 |
| dch | the merged file /var/lib/cloud/instances/11cabbbf-8dd7-446f-91bc-a1ec48b17801/cloud-config.txt appears valid (has the 984 bytes in it) | 07:46 |
| dch | SUCCESS: reading and applying user-data | 07:47 |
| dch | by this point I would expect it to know the new keys/values introduced | 07:47 |
| dch | https://www.irccloud.com/pastebin/4hhwWwVb/scripts-user-module | 07:49 |
| dch | all modules run successfully tho | 07:50 |
| aciba | Does the output of `cloud-init query userdata` contain your user-data? | 07:54 |
| dch | aciba yes it does | 07:57 |
| aciba | you have manually modified `cloud.cfg.d/90_userdata.cfg`, and some modules only run on first boot, did you perform a `cloud-init clean` after rebooting? | 07:58 |
| aciba | s/after/before/ | 07:59 |
| dch | yes | 08:00 |
| dch | in fact I am now starting from a fresh slate each time | 08:00 |
| aciba | assuming this is your userdata: https://www.irccloud.com/pastebin/D07fSXRI/userdata, could you please share `/var/log/cloud-init.log` or the output of `cloud-init collect-log`, making sure that any sensitive information is redacted? | 08:03 |
| aciba | what parts of the userdata are not getting done? write_files, run_cmd ? | 08:05 |
| dch | sure, let me re-summon a box to get a fresh log (I just annihilated the last one) | 08:08 |
| dch | none of it is run | 08:08 |
| meena | dch: do you have logging enabled | 08:10 |
| dch | yep, always | 08:10 |
| dch | woop https://usercontent.irccloud-cdn.com/file/o02bSYTv/cloud-init.log | 08:17 |
| dch | aciba: full log | 08:18 |
| dch | note it fails the first time through because network isn't up, and then is restarted when network is up | 08:18 |
| dch | this restart issue is addressed in next release from cloud provider but not yet live | 08:19 |
| aciba | I see | 08:23 |
| dch | I amended userdata slightly, dropping the pkill and just using a touch command | 08:23 |
| aciba | cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: (DataSourceEc2) | 08:23 |
| dch | aciba: right, this is expected the first time around, because there's no network available yet | 08:24 |
| dch | the next time, it works ok | 08:24 |
| dch | https://www.irccloud.com/pastebin/D07fSXRI/userdata has been refreshed | 08:24 |
| aciba | the datasource is not detected because the network is not up, thus the user-data is not pulled | 08:24 |
| dch | and the 2nd run the network is up, the datasource is there, the userdata is fetched, we can see it being merged, | 08:25 |
| dch | 2023-10-19 08:14:45,812 - util.py[DEBUG]: Attempting to load yaml from string of length 1028 with allowed root types (<class 'dict'>,) ... | 08:25 |
| dch | 2023-10-19 08:14:45,815 - handlers.py[DEBUG]: finish: init-network/consume-user-data: SUCCESS: reading and applying user-data | 08:25 |
| dch | and checking /var/lib/cloud/instances/bbabee48-2061-41bb-80ee-0c7fbde96f33/cloud-config.txt has the expected data in it | 08:26 |
| aciba | in between first and next time, can you perform a `cloud-init clean` ? I think cloud-init does not fully execute your userdata because of the state created on the first boot. | 08:27 |
| aciba | and that second boot logs are not included in the file you shared | 08:28 |
| dch | I can't modify this part its on the vendor side | 08:28 |
| dch | yes the 2nd run is in the log | 08:29 |
| dch | see e.g. line 217 | 08:29 |
| dch | 217 2023-10-19 08:14:43,492 - util.py[DEBUG]: Cloud-init v. 22.4-28-gaecdcbf8 running 'init' at Thu, 19 Oct 2023 08:14:43 +0000. Up 146.85823392868042 seconds. | 08:30 |
| dch | lines 267-~300 show successful metadata fetching | 08:30 |
| dch | so I can cloudinit clean now and re-run | 08:30 |
| dch | v2 run, not fresh boot, but after cloudinit clean & removing stale logs https://www.irccloud.com/pastebin/fPXcz5bB/cloud-init.txt | 08:32 |
| dch | same thing, userdata actions aren't being run | 08:33 |
| dch | with latest cloudinit-23.3, same issue (this is 22.4 by default) | 08:36 |
| dch | and as this system has already booted the metadata is returned immediately | 08:36 |
| dch | so we don't have a 2nd run attempt | 08:36 |
| aciba | I see that `Cloud-init v. 22.4-28-gaecdcbf8 running 'modules:final'` is present but there is not an entry corresponding to 'modules:config'. | 08:38 |
| aciba | https://www.irccloud.com/pastebin/tHfN2nMc/cloud.cfg is interesting, it has fewer modules activated than usually | 08:39 |
| aciba | what modules do you see when running `cloud-init query combined_cloud_config.cloud_config_modules` and `cloud-init query combined_cloud_config.cloud_final_modules` ? | 08:41 |
| dch | https://www.irccloud.com/pastebin/d4HPBHsX/show-modules | 08:43 |
| aciba | That is the issue, only those modules are allowed to run. | 08:43 |
| dch | aciba: is scripts-user not the thing we need? | 08:45 |
| dch | I had a look this morning already through https://cloudinit.readthedocs.io/en/latest/reference/modules.html but didn't see anythiing different that should be used | 08:46 |
| aciba | what do you mean by scripts-user ? | 08:49 |
| aciba | https://canonical-cloud-init.readthedocs-hosted.com/en/latest/explanation/format.html#user-data-script ? | 08:50 |
| dch | aciba: so this has been super helpful. the issue then is that the modules defined in cloud.cfg need to be extended | 08:50 |
| dch | I added `- bootcmd` to cloud.cfg and now the bootcmd section runs | 08:51 |
| dch | which is great, but not so important for me | 08:51 |
| aciba | sweet! | 08:51 |
| dch | but we are on the right path | 08:51 |
| dch | I will need to find each failing section, and add them in | 08:51 |
| dch | this file is inherited from many years ago, from the cloud vendor, and I've honestly not looked at it | 08:52 |
| aciba | sounds good, reach out if encounter any problem :) | 08:53 |
| dch | presumably I can *add* settings via a new `cloud.cfg.d/99_my_stuff.cfg` | 08:54 |
| dch | 🔥 | 08:55 |
| dch | looking great | 08:55 |
| dch | ✅ user and groups created | 08:55 |
| dch | ✅ bootcmd works | 08:55 |
| dch | ✅ write_files works | 08:55 |
| aciba | awesome! | 08:56 |
| dch | ✅ sudoers.d | 08:59 |
| meena | dch: who packaged your cloud-init?? | 10:08 |
| dch | meena: literally nobody knows, it was done back on 10.x I think and there isn't anybody left who knows who did it | 10:10 |
| dch | it has been handed down like a sacred scroll, mis-interpreted at each stage until ... today | 10:11 |
| meena | dch: I would take a look at the current (dev) package config files and diff them. every release i try to improve things, and fix modules | 10:15 |
| dch | meena: yeah I think the best thing is to dump the existing config file completely and just use the .sample one you provide | 10:16 |
| dch | meena: does adding packages work in your latest versions? | 10:18 |
| dch | `packages:` "module" | 10:19 |
| meena | that's been working forever | 10:25 |
| meena | i don't think i had to even do anything | 10:25 |
| meena | I would like to break it, by adding multi repo support, but one thing at a time | 10:25 |
| dch | didn't work here, but then who knows what's borked | 10:50 |
| dch | I don't have ssh_import_id: gh:dch working either | 10:51 |
| dch | even after merged in all the missing module names from the std install | 10:51 |
| meena | yeah, i think ssh_import_id is Debian / Ubuntu specific | 10:55 |
| dch | ok so a good haul, finally I am missing: | 10:56 |
| dch | ☠️ runcmd, gh:$USER, packages | 10:57 |
| meena | https://github.com/canonical/cloud-init/blob/9a401b4a21ea179452143ef8c97f261c1655c539/cloudinit/config/cc_ssh_import_id.py#L22 | 10:57 |
| dch | and of those only packages is relevant, and at that, for freebsd, not very important | 10:57 |
| dch | good to work through these finally | 10:57 |
| meena | if it's good enough for alpine, it should be good enough for freebsd | 10:57 |
| meena | https://www.freshports.org/security/ssh-import-id/ | 10:58 |
| meena | it's just python | 10:58 |
| meena | i can see adding support for that | 10:59 |
| dch | +1 for that | 10:59 |
| meena | it should literally be four lines in four files | 10:59 |
| dch | I'm calling this a day, thanks meena and aciba for your endless help and patience | 10:59 |
| dch | good progress | 10:59 |
| dch | curl | tee | 10:59 |
| meena | it'd be cool if we had optional dependencies in FreeBSD, but failing that, I think it's no harm adding a single python module with no dependencies | 11:00 |
| dch | ack | 11:03 |
| dch | by the time we add python itself, its already 100mb+ | 11:03 |
| minimal | meena: yeah it wasn't hard adding ssh-import-id for Alpine | 11:27 |
| meena | minimal: cool, thanks. | 12:16 |
| meena | hrm, i was wrong about no dependencies: New packages to be INSTALLED: | 12:17 |
| meena | py39-distro: 1.8.0_2 [FreeBSD] | 12:17 |
| meena | py39-ssh-import-id: 5.11 [FreeBSD] | 12:17 |
| meena | dch: https://github.com/canonical/cloud-init/pull/4533 let's see if netbsd and dragonfly and openbsd also have that package, if so, I'll extend it to those, too. | 13:20 |
| -ubottu:#cloud-init- Pull 4533 in canonical/cloud-init "cc_ssh_import_id: Extend support to FreeBSD" [Open] | 13:20 | |
| meena | i see some discrepancy between distros = ["all"] and "distros": [ALL_DISTROS], | 14:28 |
| minimal | yupe and I believe I mentioned in the past it might make sense to also have ALL_LINUX_DISTROS and ALL_BSD_DISTROS (or similar) | 14:57 |
| meena | *nod* | 15:02 |
| === sergiodj_ is now known as sergiodj | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!