/srv/irclogs.ubuntu.com/2022/02/11/#cloud-init.txt

blackboxsworndorffgrant: if you  are doing any UA testing for systemd daemon, add-apt-repository ppa:cloud-init-dev/daily on Jammy will get you /run/cloud-init/cloud-id-*00:29
blackboxswfor tomorow00:30
blackboxsws/r/rr 00:30
blackboxswwoo hoo thx for the reviews folks  Jammy update is accepted for tip of cloud-init [ubuntu/jammy-proposed] cloud-init 21.4-119-gdeb3ae82-0ubuntu1~22.04.1 (Accepted)03:34
orndorffgrantblackboxsw: Great! Yes that will make testing much simpler14:10
apple-corps[m]blackboxsw: I had removed the sudo commands from my cloud-init script. I tried adding a sleep statement regarding the filesystem initialization concerns. I'm seeing the same behavior. From the cloud-init.log it looks like the runcmd is written, but not executed as shown previously. Then I assume I will need to collect the logs to file a bug report.19:39
blackboxswapple-corps[m]: yes I would have expected your logs to contain ""Running command ['/var/lib/cloud/instance/scripts/runcmd'] with allowed return codes [0] (shell=False, capture=False)""19:42
blackboxswto show that cloud-init actually ran it. Has your system actually completed running cloud-init? You can check via `cloud-init status --wait --long`19:42
blackboxswapple-corps[m]: I'm also presuming that you are using nitro system and maybe IPv6 only networking?19:43
apple-corps[m]blackboxsw: when I run `sudo cloud-init status --wait --long` it appears I get repeated periods ... displayed so does that suggest it has not completed?19:44
blackboxswapple-corps[m]: yeah, it suggests cloud-init hasn't been able to get far enough to complete on the system. probably because something is happening in networking that is preventing cloud-init from moving to the next stage where it actually runs the runcmd19:45
apple-corps[m]Regarding nitro system, I am not aware regarding nitro-system, then I think not. Running a z1d.large ec2 instance type19:46
blackboxswapple-corps[m]: runcmd module is only executed at the end of the "config" boot stage https://cloudinit.readthedocs.io/en/latest/topics/boot.html#config. So we we don't get that far on your system you'll never see that command execute.19:47
apple-corps[m]blackboxsw: should I go ahead and provide the log and file a bug report then? The last lines of the log. util.py[DEBUG]: cloud-init  mode 'modules' took 4.127 seconds(4.12)19:48
blackboxswapple-corps[m]: what's /run/cloud-init/status.json should show a start and end time for each boot stage cloud-init has completed. If you have any stages which don't show an end time you know you are 'stuck' there. `cloud-init analyze show` might also help discern how far in boot you have gotten19:48
apple-corps[m]handlers.py[DEBUG]: finish: modules-config: SUCCESS: running modules for config19:49
blackboxswapple-corps[m]: +1 on bug filing. if you have the ability to get /var/log/cloud-init*log and or `cloud-init collect-logs` it'd make this way easier19:49
blackboxswif you had network connectivity from the instance I'd say run `ubuntu-bug cloud-init`19:49
apple-corps[m]I probably need to scrub the logs of any sensitive information.19:50
blackboxswapple-corps[m]: +1 on a scrub, generally watch out for the included ./run/cloud-init/instance-data-sensitive.json ./run/cloud-init/instance-data.json files. The cloud-init.log itself should have "REDACTED" passwords from user-data for instance 19:52
apple-corps[m]I see finished and start times on 3 modules. The other modules look like they have null values for both.19:52
blackboxswmy example status.json has start/end times for init, init-local, modules-config, modules-file19:53
blackboxswmy example status.json has start/end times for init, init-local, modules-config, modules-final19:53
blackboxswif you don't get across modules-final something is holding things up. `cloud-init analyze show` or `cloud-init analyze blame` may help when looking at your first boot on the system19:54
apple-corps[m]mine has init, init-local, modules-config all show start / end times. modules-final and modules-init are both null19:54
apple-corps[m]for start / end times19:55
apple-corps[m]Ok, I will work on collecting and scrubbing the logs then.19:56
falcojrI believe systemd-analyze critical-chain should tell you if something if blocking the service from starting19:56
blackboxswone other thing to double check on your image is whether cloud-init's systemd targets have all executed (sometimes systemd "ordering cycles" could have been introduced with other software on the node resulting in cloud-init system units being dropped/disabled from the boot target.19:58
blackboxswfor unit in cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service; do systemctl status $unit; done19:58
apple-corps[m]I'll take a look. I can see at least some of the modules are running. The script was shellified by cloud-init, etc19:58
blackboxswconfirm they were all run. and confirm you don't see "Breaking ordering cycle" in journalctl -b 019:58
apple-corps[m]When I run systemd-analyze critical-chain it says Bootup is not yet finished. Please try again later19:59
blackboxswand `systemctl status network-online.target`19:59
blackboxswas that'd block a number of cloud-init jobs20:00
blackboxswI'm also thinking of checking active services still running/blocking with `systemctl --type=service --state=running` or something20:01
apple-corps[m]cloud-finlal service is loaded, enabled and dead from the for loop20:01
apple-corps[m]cloud-init.local.service doesn't exist. The others are enabled and active20:02
blackboxswok. good and it is a typo cloud-init.local.service should be cloud-init-local.service20:03
blackboxswbut yes given your status.json, I'd expect init-local to be "healthy"20:04
blackboxsw`systemctl list-jobs` may also point to something unfinished on your system boot20:06
blackboxswhrm my cloud-final.service is      Active: active (exited) since Thu 2022-02-10 01:50:08 UTC; 1 day 18h ago20:07
blackboxswagain I think minimally your cloud-init.log will help us deduce what's happened here if possible20:07
apple-corps[m]Looking at cloud-init-local.service "my bad", I see it's active, enabled. From the journalctl logs for this I see cloud-init warning. dhcp.py[WARNING]: dhclient did not produce expected files: dhclient.pid, dhclient.leases . I see some logging related to a dhclient process. But I do see dhclient provided the leases20:08
apple-corps[m]looks like there were some permissions errors creating dhcp.eleases and dhclient.pid. But it's not clear that is blocking the service or not.20:09
apple-corps[m]I think if we expect cloud-final.service to execute the runcmd command, because it is dead and it's last run was much before today, then we know there is an issue there20:10
apple-corps[m]collecting the logs then and will file20:28
apple-corps[m]atteched to bug 196067821:18
ubottuBug 1960678 in cloud-init "cloud-config script not executing" [Undecided, New] https://launchpad.net/bugs/196067821:18

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!