=== cpaelzer_ is now known as cpaelzer
=== janitha75 is now known as janitha7
=== jrm2 is now known as jrm
[21:27] <blackboxsw> meena: sorry for the question out of left field. Do you know if kenv kernel env var names are synonymous across freebsd, openbsd, netbsd, dragonflybsd etc? I ask because I'm wondering if this check should actually be is_BSD() https://github.com/canonical/cloud-init/blob/main/cloudinit/dmi.py#L161
[22:36] <dch> I'm trying unsuccessfully to get userdata scripts to execute.
[22:37] <dch> - `cloud-init query --all | jq .userdata` returns "#cloud-config\nruncmd:\n ...' as expected
[22:38] <dch> - passing a shell script also produces the expected result `loud-init query --all | jq .userdata "#!/bin/sh\...`
[22:38] <dch> but logs and filesystem don't show these as executing
[22:45] <dch> there is no `allow_userdata: false` key, and AFAICT I'm following https://cloudinit.readthedocs.io/en/latest/topics/format.html#user-data-script
=== SirScott6 is now known as SirScott
=== kenyon_ is now known as kenyon
[23:11] <falcojr> dch: can you provide a pastebin of /var/log/cloud-init.log and/or your userdata?
[23:11] <dch> hmmm `cloud-init status` returns "running" but no processes remain
[23:12] <blackboxsw> systemctl list-units --type=service --state=active # probably will also show you someting is hung or incomplete etc.
[23:13] <dch> this is FreeBSD 13.1-RELEASE amd64 with py39-cloud-init-22.3.4 btw
[23:14] <dch> I don't have a cloudinit.log for some reason, but here is the various userdata etc
[23:14] <dch> userdata https://git.sr.ht/~dch/cloud-confused/tree/main/item/var/lib/cloud/instances/5f05e50c-79bc-4d26-841c-4176dd41566d/user-data.txt
[23:15] <dch> falcojr: btw this repo is the snapshot of /var/lib/cloud and the config /usr/local/etc/cloud/ off the system post-cloudinit running
[23:15] <blackboxsw> hrm, ok BSD. `sudo cloud-init schema --system` should check that user-data provided is right format and header
[23:15] <dch> status.json https://git.sr.ht/~dch/cloud-confused/tree/main/item/var/lib/cloud/data/status.json shows no errors AFAICT
[23:16] <blackboxsw> ahh sorry cloud-init schema --system is only for #cloud-config specifically :/
[23:16] <blackboxsw> script looks fine to me
[23:16] <dch> # cloud-init schema --system Error: Cloud config schema errors: format-l1.c1: File None needs to begin with "#cloud-config"
[23:16] <dch> "computer says No"
[23:16] <blackboxsw> heh
[23:17] <blackboxsw> too true now that I know your user-data isn't #cloud-config  different set of tools :/
[23:17] <dch> let me restart one and get the console log, it has all the cloudinit logs that we expect to see
[23:17] <blackboxsw> your  https://git.sr.ht/~dch/cloud-confused/tree/main/item/var/lib/cloud/data/status.json also shows that none of the other boot stages ran 
[23:18] <dch> interesting. I'm not yet familiar enough with all these files to know what I should expect
[23:19] <dch> I'll stick with a basic shell script atm, in case #cloud-config has bsd-related bugs
[23:19] <blackboxsw> I'd expect to see both a start and finished time for modules-config, modules-final and init-local. So those separate rc.d services are not being started in your image for some reason
[23:19] <blackboxsw> it looks like the only boot stage that ran was 'init' stage in this image in order to discover the Ec2 datasource. modules-(config|final) is where all the user-data is acted upon
[23:20] <blackboxsw> per https://canonical-cloud-init.readthedocs-hosted.com/en/latest/topics/boot.html
[23:21] <blackboxsw> typically these stages are skipped in Linux if there is a system ordering dependency cycle that forces cloud-init's separate systemd init scripts from being run in the current boot target.
[23:22] <dch> ok so once this next fresh boot finishes, I will re-run cloudinit locally without rebooting, this should be a bit easier to deal with
[23:22] <dch> blackboxsw: btw I can't read that rtd-hosted URL, it requires GH auth and then 404
[23:23] <dch> https://cloudinit.readthedocs.io/en/latest/topics/boot.html is presumably close enough?
[23:25] <blackboxsw> oops. my bad, we just started setting up a paid offering without RTD adds and not up for public consumption yet. It is exactly the same, sorry about mis-link there
[23:25] <blackboxsw> both are a direct mirror of each other.
[23:27] <blackboxsw> yes dch, there should be four rc.d scripts to get cloud-init functioning properly (and those init scripts should run the following in order
[23:27] <blackboxsw> sudo cloud-init init --local
[23:27] <blackboxsw> sudo cloud-init init
[23:27] <dch> this is annoying, it was working a few months ago, and our new release broke something... hope to track bug down
[23:27] <blackboxsw> sudo cloud-init modules --mode=config
[23:27] <blackboxsw> sudo cloud-init modules --mode=final
[23:28] <dch> ok, now with logs! https://git.sr.ht/~dch/cloud-confused/tree/main/item/console.log
[23:28] <dch> https://git.sr.ht/~dch/cloud-confused/tree/main/item/console.log#L630
[23:28] <dch> L764 we got metadata ok
[23:29] <dch> L816 we skip network deliberately
[23:30] <blackboxsw> what I see in that boot log is https://git.sr.ht/~dch/cloud-confused/tree/main/item/console.log#L631  only the 'init' stage was run, no 'init-local'  'modules:config' or 'modules:final' stage log header when starting each boot stage
[23:31] <blackboxsw> +1 on proper meta-data detected for Ec2 datasource discovery in L 764, yeah just nothing seems to have called `sudo cloud-init modules --mode=config` or --mode-final for the final two boot stages
[23:32] <blackboxsw> +1 on proper meta-data detected for Ec2 datasource discovery in L 764, yeah just nothing seems to have called `sudo cloud-init modules --mode=config` or `--mode-final` `for the final two boot stages
[23:33] <dch> hmm
[23:33] <dch> "normally" on FreeBSD we just use `cloudinit_enable=YES` and that's all that's required to run the full gamut
[23:34] <blackboxsw> Here are our systemd services templates for modules:config stage[1]  and modules:final (each is separate in Linux)
[23:34] <blackboxsw> 1.  https://github.com/canonical/cloud-init/blob/main/systemd/cloud-config.service.tmpl
[23:34] <blackboxsw> 2. https://github.com/canonical/cloud-init/blob/main/systemd/cloud-final.service.tmpl
[23:34] <dch> blackboxsw: thanks for your diag help here, is there a log of a working one you can point me towards?
[23:34] <blackboxsw> I'd expect something in rc.d for each of those
[23:34] <dch> doesn't have to be FreeBSD, I just need to see what line we're missing
[23:34] <blackboxsw> sure. and cloud-init analyze show typically organizing this output based on boot stages. one sec
[23:34] <dch> on bsd here I have cloudinit, cloudinitlocal cloudfinal, cloudconfig
[23:35] <dch> for example, this config is what we deploy in Oracle Cloud. Let me check if that still works...
[23:39] <dch> ok, `cloud-init analyze blame|show|dump` all blow up
[23:39] <dch> https://www.irccloud.com/pastebin/Zy4Osq8R/cloudinit_analyze_dump
[23:43] <dch> presumably this is because there's no log file to digest
[23:44] <blackboxsw> sorry crashed my browser: https://paste.opendev.org/show/bQuuSdQMl12zyPz9DQtT/
[23:46] <blackboxsw> on linux: absence of /var/log/cloud-init.log isn't a trace
[23:46] <blackboxsw> root@dd-j:~# cloud-init analyze show
[23:46] <blackboxsw> Cannot open file /var/log/cloud-init.log
[23:46] <dch> ok!
[23:46] <dch> we got it
[23:46] <blackboxsw> but that could be just a newer commit 
[23:47] <dch> you are (all) right
[23:47] <dch> cloudfinal needs to be run
[23:47] <dch> this also causes `cloudinit status` to be "done"
[23:47] <blackboxsw> ok so I'm curious when you find out what's off if you can give feedback on which rc.d script wasn't running or why
[23:47] <blackboxsw> just for our context
[23:48] <dch> yeah I think we can improve the port by adding a short message "enable these things or you will fail"
[23:49] <blackboxsw> yeah good note: also `cloud-init status --format json`should give more information about the reasons we are in waiting state (it'll be in cloud-init version 22.4 )
[23:50] <blackboxsw> doesn't help today, but we could raise a bigger warning message if we are blocked on different stages not yet run
[23:50] <blackboxsw> https://paste.opendev.org/show/biPxW0DUiq8xbx6lZkIl/
[23:54] <dch> ok now we have this all figured out, I think there are 3 things:
[23:54] <dch> 1. the FreeBSD port should be clear that you need to run all the commands now
[23:55] <dch> 2. it shouldn't put stuff in /run/cloud-init/ but use /var/run/cloud-init/ like other BSD stuff
[23:55] <dch> 3. it should (by default) produce logs. they are useful.
[23:55] <dch> 4. the existence of the 4 boot rc.d scripts should be clarified.
[23:56] <dch> I think all of these can be addressed downstream
[23:56] <dch> 5. I owe you all some sort of beverage. Perhaps I should come to FOSDEM for the first time.
[23:56] <blackboxsw> sounds like a great plan, and anything that makes sense upstream we would like to incorporate
[23:57] <blackboxsw> Boom FOSDEM'd be nice that'd make two of us for first timers.
[23:58] <blackboxsw> -> have to head out for dinner EndofDay. Take care thx for the help on BSD
[23:58] <dch> I checked, I copypasted my cloudinit stuff from the FreeBSD openstack deployment. 
[23:58] <dch> I guess none of these worked fully.
[23:59] <dch> although, reading my OCI patch, I definitely tested this and had it working. 
[23:59] <dch> https://reviews.freebsd.org/D34746
[23:59] <dch> that was ~ a year ago maybe?