=== cpaelzer_ is now known as cpaelzer [15:53] I'm experiencing some weird issues on a SLES 15 SP4 machine I'm trying to configure with cloud-init 21.4, perhaps someone can help, please. I'm sending over user-data and config such as hostname, users, groups and write_files are all working as expected, however nothing from runcmd seems to be executed. /etc/cloud/cloud.cfg does have entry for [15:53] runcmd in cloud_config_modules. In runcmd I have 1 entry with "- sh /usr/local/custom_scripts/runme.sh" but I've even tried to echo to a file but with no joy [15:53] Weirdly, the exact same user-data works on a different distro (Ubuntu) [15:59] thaddeus: have you tried setting cloud-init logging to debug? That way you'll see what each module is doing [16:02] thaddeus: to collect even more information: https://cloudinit.readthedocs.io/en/21.4/topics/cli.html#collect-logs [16:09] Looks like debug logging is enabled. I've collected those logs and I can see it notes |`->config-runcmd ran successfully @2230.17900s +00.00300s [16:11] I can also see util.py[DEBUG]: Writing to  /usr/local/custom_scripts/runme.sh &  Changing the ownership of /usr/local/custom_script/install.sh to 0:0 [16:12] thaddeus: I was thinking more of looking in cloud-init.log (with debug) to see what happened when it logged "Running module runcmd" [16:13] that would be the write_file module logging rather than runcmd module logging though... [16:13] "Running module runcmd" doesn't even appear in cloud-init.log :blink: [16:14] ok, search for "runcmd" in general then in the log [16:14] Nothing [16:15] that doesn't make sense then as you said "- runcmd" was present in your /etc/cloud/cloud.cfg file's "cloud_config_modules:" section [16:15] There's 14 instances of running module $something, but not runcmd [16:15] That's correct [16:17] could you run cloud-init query merged_cfg.cloud_config_modules ? [16:17] and see if runcmd is in the output [16:17] you'd either see "Running module runcmd" or "Skipping modules 'runcmd' because not applicable config is provided." if runcmd was enabled [16:17] sounds like a problem with your cloud.cfg file [16:17] cloud-init query merged_cfg.cloud_config_modules shows runcmd [16:18] Although I've not made any manual changes to cloud.cfg [16:21] what's the 1st line of your install.sh script (the shebang line) ? [16:22] thaddeus: given that you can see /usr/local/custom_scripts/runme.sh we know minimally part of your user-data is correct (maybe per write-scripts. can you try running 'sudo sh /usr/local/custom_scripts/runme.sh` from a terminal on your system. I'm guessing we are erroring out or something and the output of that failure is in /var/log/cloud-init-output.log [16:24] I'm wondering if the shebang line is something like "#!/bin/bash" and "sh" on the system points to /bin/dash or similar [16:24] #!/bin/sh [16:24] ok, and "sh" is /bin/sh also? "which sh" [16:25] "/usr/bin/sh" [16:25] hmm, and are /bin/sh and /usr/bin/sh the same file? (probably at least 1 is a softlink) [16:25] if I run sh /usr/local/custom_script/install.sh it works as expected [16:26] Correct minimal [16:26] ls -l /bin/sh /usr/bin/sh ? [16:27] "/bin/sh -> /usr/bin/sh & /usr/bin/sh -> bash" [16:28] on the working ubuntu system they both point to dash though [16:28] on Ubuntu AFAIK /bin/sh is dash, not bash [16:28] snap! ;-) [16:28] Indeed [16:28] That wouldn't be why runcmd couldn't run though? [16:29] hmm, seems the opposite way around to what I expected, I'd guessed your script has some Bashisms that "normal" shell might not handle [16:29] thaddeus: on your system did cloud-init write out /var/lib/cloud/instance/scripts/runcmd ? that should be the "shellify function in couldinit would manipulate your run command and wrap it in a shell script [16:30] take a look at /var/lib/cloud/instance/scripts/runcmd and also see if you can run that successfully directly [16:30] It did. /var/lib/cloud/instance/scripts/runcmd contains "#!/bin/sh" on the first line and "sh /usr/local/custom_script/install.sh" on the second [16:31] And that works fine [16:33] ok and that runs fine for me too. and runs fine for me as well with silly sample config like this: https://paste.opendev.org/show/bYcTj6vpb01BOwi6jtVO/ [16:34] Yeah I get no issues with debian based systems like Ubuntu, can you try that with opensuse / centos, please? [16:41] thaddeus: is your shellscript doing anything "funky"? [16:43] Define "funky" [16:43] :D [16:43] something that has a 50/50 chance of working across various distros with different shells ;-) [16:45] Hah, no. Weirdly it has worked in the past on SLES but I rebuilt my VM image and I know it's running a slightly outdated cloud-init binary compared to Ubuntu, but going through the changelogs I couldn't find anything that may explain why [16:45] Ignoring the script I can't even do things like echo to a file from runcmd [16:48] I wonder if it's some apparmour / selinux shenanigans [16:53] It's like cloud_init_modules all run, but nothing from cloud_config_modules does [16:54] I haven't been near SLES for several years, I forget any specifics about it [16:54] thaddeus: which cloud-init stages are running? [16:54] redhat got some modifications when cloud-init runs in comparison fo debian/ubuntu [16:55] you can check stages run on latest boot w/ `cloud-init analyze show` or cat /run/cloud-init/status.json [16:55] yeah, are all the cloud-init init.d/service files enabled? [16:58] cloud-init analyze show only shows init-local and init-network being executed [16:58] ahh nice, ok so config-modules is skipped. thx waldi [16:59] thaddeus: it might be a systemd ordering cycle issue with cloud-init systemd services vs something 'new' installed in your opensuse image [17:00] typically you can check journalctl -b 0 | grep "ordering cycle" to see if systemd found a funky unresolvable dependency chain and punted a cloud-init service out of the boot target/goal [17:00] https://unix.stackexchange.com/questions/193714/generic-methodology-to-debug-ordering-cycles-in-systemd for context [17:04] No ordering cycle issues from the journal, going to compare module configs between ubuntu and sles [17:11] thaddeus: just validated on opensuse 15.3 (on lxc images I had to: lxc launch images:opensuse/15.3 -c user.user-data="$(SAMPLE .YAML)" ypper install cloud-init; systemctl enable cloud-init.service; systemctl enable cloud-init-local.server; systemctl enable cloud-config.service; systemctl enable cloud-final.service; cloud-init clean --reboot --logs) [17:12] version: /usr/bin/cloud-init 21.4-150100.8.58.1 [17:13] Interesting, if I fun systemctl status cloud-config.service it shows as disabled Loaded: loaded (/usr/lib/systemd/system/cloud-config.service; disabled; vendor preset: disabled) [17:13] I've never had to manually enable them in the past, wasn't aware they were individual cloud-init unit files for different stages [17:13] Going to rebuild my template adding your systemctl enable commands blackboxsw [17:15] thaddeus: generally you shouldn't have to enable those services as they *should* have been enabled in the stock distrubution cloud image [17:16] I only suggested that as you had mentioned "custom image" which carries a lot of baggage for me about the potential of a derivative image which has been toyed with a bit. [17:18] Yeah it's enabled on my ubuntu image. I'm going to modify my packer script and see if that works, will take a few mins to rebuild and I'll report back. But thank you blackboxsw, minimal, waldi, aciba for all your help so far, much appreciated. [17:52] Yup, that was it blackboxsw! Thank you! [17:54] currently looking into network config v2 as I'm wanting to set some IPv6 related settings in eni such as "privext", "request_prefix", "autoconf". Seems this is currently only defined/catered for via netplan pass-through [17:55] So before I start work on a PR to add eni support for this sort of stuff wanted to get some thoughts/input from people