=== EugenMayer5968 is now known as EugenMayer596
[20:45] <tiaz> Hi. I'm messing around with #include directives. Right now it's pulling a trivial config that just adds a pair of CAs. However, it appears to be stuck in `running`, even though there are no processes named anything like 'cloud' running anymore.
[20:46] <tiaz> Finding that `cloud-init status` examines /run/cloud-init/status.json, I checked that out ... the errors arrays are empty, and "v1" and "init-local" both have start and finish times, but modules-{init,config,final} all have neither
[20:46] <tiaz> er, sorry, the "init" and "init-local" (rather than v1 and init-local)
[20:47] <tiaz> anyway, there's also a stage: null key/value. I don't see anything in the logs indicating where it failed or gave up. Can anyone advise as to where I might dig deeper to find out where this broke?
[20:47] <tiaz> I really don't want to restart cloud-init through strace ;)
[20:51] <blackboxsw> tiaz: what distribution are you on? That makes me think something in your image has potentially ejected cloud-init.service, cloud-config.service and cloud-final.service from the systemd boot goals due to systemd ordering cycles/incompatibilities with cloud-init's other boot stages.  Each bootstage of cloud-init is a separate systemd unit that could be dropped if the ordering of services at initial boot is unable to resolve.
[20:52] <tiaz> blackboxsw: I'm running ubuntu:focal through lxc almost exactly like the quickstart/demo recommends
[20:52] <tiaz> except instead of #cloud-config/runcmd: it's #include/someurl. logs show it pulled someurl successfully, and it applied the CAs I expect
[20:53] <tiaz> cloud-init is active/exited, -config is inactive/dead, -final is inactive/dead
[20:53] <falcojr> tiaz: any problems from "systemctl list-units | grep cloud" or "systemctl list-units --failed"?
[20:55] <tiaz> there are failed units: e2scrub_reap, systemd-hostnamed, systemd-remount-fs. grep cloud w/o failed shows active/active for cloud-config.target, then as described (init active/exited, init-local active/exited, config,final inactive/dead, none failed)
[20:56] <tiaz> -and cloud-init.target inactive/dead
[20:56] <tiaz> oh, I have an idea, just a moment
[20:58] <tiaz> ah, no joy (thought it might be remove_defaults:true to make it easier to see it'd been applied, but I started it without that and no change)
[21:00] <tiaz> https://pastebin.com/T5UawKKa is the cloud-config that's being #include-d (censored, ofc, the certs are valid in the real one). as you can see it's pretty trivial
[21:00] <blackboxsw> tiaz also cloud-init analyze show can quickly confirm whether cloud-init boot stages got skipped by showing you what boot stages were actually run on the latest boot
[21:02] <tiaz> blackboxsw: https://pastebin.com/ayBGrffe doesn't seem to say anything about skipped, are there other values I should expect in this output?
[21:21] <blackboxsw> alrighty, well that confirms what your results.json file states. no modules-config or modules-final stage. in latest boot. What does `systemctl status cloud-config.service` and `systemctl status cloud-final.service` say?   also journalctl -b 0 | grep "ordering cycle"`  to ensure those systemd units are active/health.
[21:58] <tiaz> blackboxsw: both inactive/dead but enabled: https://pastebin.com/tZ05CEwG , no joy on journalctl
[22:00] <tiaz> truncated the next shell line by accident. 'ordering cycle' (insensitive) does not occur in journalctl output
=== sam_ is now known as sam
=== sam is now known as sam_