=== cpaelzer_ is now known as cpaelzer
thaddeusI'm experiencing some weird issues on a SLES 15 SP4 machine I'm trying to configure with cloud-init 21.4, perhaps someone can help, please. I'm sending over user-data and config such as hostname, users, groups and write_files are all working as expected, however nothing from runcmd seems to be executed. /etc/cloud/cloud.cfg does have entry for15:53
thaddeusruncmd in cloud_config_modules. In runcmd I have 1 entry with "- sh /usr/local/custom_scripts/runme.sh" but I've even tried to echo to a file but with no joy15:53
thaddeusWeirdly, the exact same user-data works on a different distro (Ubuntu)15:53
minimalthaddeus: have you tried setting cloud-init logging to debug? That way you'll see what each module is doing15:59
acibathaddeus: to collect even more information: https://cloudinit.readthedocs.io/en/21.4/topics/cli.html#collect-logs16:02
thaddeusLooks like debug logging is enabled. I've collected those logs and I can see it notes |`->config-runcmd ran successfully @2230.17900s +00.00300s16:09
thaddeusI can also see util.py[DEBUG]: Writing to  /usr/local/custom_scripts/runme.sh &  Changing the ownership of /usr/local/custom_script/install.sh to 0:016:11
minimalthaddeus: I was thinking more of looking in cloud-init.log (with debug) to see what happened when it logged "Running module runcmd"16:12
minimalthat would be the write_file module logging rather than runcmd module logging though...16:13
thaddeus"Running module runcmd" doesn't even appear in cloud-init.log :blink:16:13
minimalok, search for "runcmd" in general then in the log16:14
minimalthat doesn't make sense then as you said "- runcmd" was present in your /etc/cloud/cloud.cfg file's "cloud_config_modules:" section16:15
thaddeusThere's 14 instances of running module $something, but not runcmd16:15
thaddeusThat's correct16:15
acibacould you run cloud-init query merged_cfg.cloud_config_modules ?16:17
acibaand see if runcmd is in the output16:17
minimalyou'd either see "Running module runcmd" or "Skipping modules 'runcmd' because not applicable config is provided." if runcmd was enabled16:17
minimalsounds like a problem with your cloud.cfg file16:17
thaddeuscloud-init query merged_cfg.cloud_config_modules shows runcmd16:17
thaddeusAlthough I've not made any manual changes to cloud.cfg16:18
minimalwhat's the 1st line of your install.sh script (the shebang line) ?16:21
blackboxswthaddeus: given that you can see  /usr/local/custom_scripts/runme.sh  we know minimally part of your user-data is correct (maybe per write-scripts. can you try running 'sudo sh /usr/local/custom_scripts/runme.sh` from a terminal on your system. I'm guessing we are erroring out or something and the output of that failure is in /var/log/cloud-init-output.log16:22
minimalI'm wondering if the shebang line is something like "#!/bin/bash" and "sh" on the system points to /bin/dash or similar16:24
minimalok, and "sh" is /bin/sh also? "which sh"16:24
minimalhmm, and are /bin/sh and /usr/bin/sh the same file? (probably at least 1 is a softlink)16:25
thaddeusif I run sh /usr/local/custom_script/install.sh it works as expected16:25
thaddeusCorrect minimal16:26
minimalls -l /bin/sh /usr/bin/sh ?16:26
thaddeus"/bin/sh -> /usr/bin/sh & /usr/bin/sh -> bash"16:27
thaddeuson the working ubuntu system they both point to dash though16:28
minimalon Ubuntu AFAIK /bin/sh is dash, not bash16:28
minimalsnap! ;-)16:28
thaddeusThat wouldn't be why runcmd couldn't run though?16:28
minimalhmm, seems the opposite way around to what I expected, I'd guessed your script has some Bashisms that "normal" shell might not handle16:29
blackboxswthaddeus: on your system did cloud-init write out /var/lib/cloud/instance/scripts/runcmd ? that should be the "shellify function in couldinit would manipulate your run command and wrap it in a shell script 16:29
blackboxswtake a look at /var/lib/cloud/instance/scripts/runcmd   and also see if you can run that successfully directly16:30
thaddeusIt did. /var/lib/cloud/instance/scripts/runcmd contains "#!/bin/sh" on the first line and "sh /usr/local/custom_script/install.sh" on the second16:30
thaddeusAnd that works fine16:31
blackboxswok and that runs fine for me too. and runs fine for me as well with silly sample config like this: https://paste.opendev.org/show/bYcTj6vpb01BOwi6jtVO/16:33
thaddeusYeah I get no issues with debian based systems like Ubuntu, can you try that with opensuse / centos, please?16:34
minimalthaddeus: is your shellscript doing anything "funky"?16:41
thaddeusDefine "funky"16:43
minimalsomething that has a 50/50 chance of working across various distros with different shells ;-)16:43
thaddeusHah, no. Weirdly it has worked in the past on SLES but I rebuilt my VM image and I know it's running a slightly outdated cloud-init binary compared to Ubuntu, but going through the changelogs I couldn't find anything that may explain why16:45
thaddeusIgnoring the script I can't even do things like echo to a file from runcmd16:45
thaddeusI wonder if it's some apparmour / selinux shenanigans16:48
thaddeusIt's like cloud_init_modules all run, but nothing from cloud_config_modules does16:53
minimalI haven't been near SLES for several years, I forget any specifics about it16:54
waldithaddeus: which cloud-init stages are running?16:54
waldiredhat got some modifications when cloud-init runs in comparison fo debian/ubuntu16:54
blackboxswyou can check stages run on latest boot w/ `cloud-init analyze show` or cat /run/cloud-init/status.json16:55
minimalyeah, are all the cloud-init init.d/service files enabled?16:55
thaddeuscloud-init analyze show only shows init-local and init-network being executed16:58
blackboxswahh nice, ok so config-modules is skipped. thx waldi 16:58
blackboxswthaddeus: it might be a systemd ordering cycle issue with cloud-init systemd services vs something 'new' installed in your opensuse image16:59
blackboxswtypically you can check journalctl -b 0 | grep "ordering cycle" to see if systemd found a funky unresolvable dependency chain and punted a cloud-init service out of the boot target/goal17:00
blackboxswhttps://unix.stackexchange.com/questions/193714/generic-methodology-to-debug-ordering-cycles-in-systemd for context17:00
thaddeusNo ordering cycle issues from the journal, going to compare module configs between ubuntu and sles17:04
blackboxswthaddeus: just validated on opensuse 15.3 (on lxc images I had to: lxc launch images:opensuse/15.3 -c user.user-data="$(SAMPLE .YAML)" ypper install cloud-init; systemctl enable cloud-init.service; systemctl enable cloud-init-local.server; systemctl enable cloud-config.service; systemctl enable cloud-final.service;  cloud-init clean --reboot --logs) 17:11
blackboxswversion: /usr/bin/cloud-init 21.4-150100.8.58.117:12
thaddeusInteresting, if I fun systemctl status cloud-config.service it shows as disabled Loaded: loaded (/usr/lib/systemd/system/cloud-config.service; disabled; vendor preset: disabled)17:13
thaddeusI've never had to manually enable them in the past, wasn't aware they were individual cloud-init unit files for different stages17:13
thaddeusGoing to rebuild my template adding your systemctl enable commands blackboxsw17:13
blackboxswthaddeus: generally you shouldn't have to enable those services as they *should* have been enabled in the stock distrubution cloud image17:15
blackboxswI only suggested that as you had mentioned "custom image" which carries a lot of baggage for me about the potential of a derivative image which has been toyed with a bit.17:16
thaddeusYeah it's enabled on my ubuntu image. I'm going to modify my packer script and see if that works, will take a few mins to rebuild and I'll report back. But thank you blackboxsw, minimal, waldi, aciba for all your help so far, much appreciated.17:18
thaddeusYup, that was it blackboxsw! Thank you!17:52
minimalcurrently looking into network config v2 as I'm wanting to set some IPv6 related settings in eni such as "privext", "request_prefix", "autoconf". Seems this is currently only defined/catered for via netplan pass-through17:54
minimalSo before I start work on a PR to add eni support for this sort of stuff wanted to get some thoughts/input from people17:55

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!