[06:39] <holmanb> dean: hello o/ can you please be more specific about the problem? 
[16:37] <faiqus> hi folks im currently working on a cloud-init datasource for cluster-api-provider-aws. some background here when CAPA would provision a machine it would take the initial user-data, and then download more "secure" userdata from aws ssm. after downloading that it would restart cloud-init.
[16:39] <faiqus> the code for this new datasource is here now https://github.com/kubernetes-sigs/image-builder/pull/158 and everything seems to be populated in the right place. the file /var/lib/cloud/instances/i-<instance-id>/cloud-config.txt has all the correct contents
[16:39] <faiqus> but nothing actaully runs. would someone be able to help me figure out what's going wrong 
[16:57] <dean> Does '/etc/cloud/cloud-init.disabled' exist?
[17:02] <faiqus> ls: cannot access '/etc/cloud/cloud-init.disabled': No such file or directory
[17:32] <faiqus> the data also isn't templated by jinja either 
[17:32] <faiqus> i feel like i need to do this with a boothook?
[17:33] <faiqus> for example i see this 
[17:33] <faiqus>     name: '{{ ds.meta_data.local_hostname }}'
[17:46] <connor_k> I'm experimenting with including a shell script in vendor data, and it seems to work on fresh instances (gets written and executed) when I set the Content-Type to text/cloud-boothook or text/x-shellscript. But when I try text/x-shellscript-per-instance it gets written under /var/lib/cloud/scripts/vendor/per-instance/ but I do not see it executed in /var/log/cloud-init.log on a fresh instance like I do with the other content types. Does anyone have 
[17:46] <connor_k> any ideas for further debugging this?
[17:47] <connor_k> "scripts_vendor" is listed under cloud_final_modules. I noticed there's also "scripts_per_once", "scripts_per_boot", "scripts_per_instance". Are there vendordata analogs to those I need to factor in?
[18:08] <minimal> faiqus: was ds-identify updated to know about/detect this new DataSource?
[19:41] <holmanb> faiqus: the link you shared doesn't appear to be relevant to the issue that you described
[19:43] <holmanb> connor_k: are you sure that it is really a fresh instance? if you run `cloud-init clean` and try rebooting does it work with the -per-instance script?
[19:44] <connor_k> holmanb: yeah, I've tried using cloud-init clean, but no avail. I also created brand-new VMs where using the x-shellscript-per-instance and didn't observe the "Running command ... " line in /var/log/cloudinit.log like I did with the other content types
[19:45] <holmanb> connor_k: err, is looking for identical logs the only way that you're checking if it is worki9ng?
[19:47] <minimal> holmanb: in the gh pull he referenced there's a comment from "faiq" which has a link to the DS code in question
[19:48] <minimal> https://github.com/kubernetes-sigs/image-builder/compare/main...faiq:image-builder:custom-ami-ds
[19:51] <holmanb> weird, for me the link redirects to this issue which doesn't appear relevant: https://github.com/kubernetes-sigs/image-builder/issues/158
[19:51] -ubottu:#cloud-init- Issue 158 in kubernetes-sigs/image-builder "Ensure the SSM agent is installed and running in AWS images" [Closed]
[19:52] <minimal> yeah that happened to me also earlier
[19:53] <minimal> ah, the original url was truncated, there's a missing "3"
[19:53] <minimal> #1583
[19:53] <holmanb> gotcha, thanks
[19:54] <minimal> I don't understand why that DS is trying to restart the cloud-init service...
[19:58] <connor_k> holmanb: no, the script should `echo hello >> /tmp/helloworld.txt` -- I was mainly referencing the cloud-init logs since I thought it would be forensically useful when I noticed that the Running command ... appeared in the logs for each content-type that worked for me but was absent when I switched the content-type to text/x-shellscript-per-instance
[20:05] <holmanb> minimal: it's not trying to restart the cloud-init service
[20:05] <holmanb> it's commenting out the restart in the cloud-init service
[20:06] <holmanb> it's convoluted
[20:06] <minimal> ah ok
[20:09] <holmanb> faiqus: Pastbin the logs, please.
[20:12] <holmanb> connor_k: you too
[20:23] <faiqus> hi folks sorry i had to step out. i will paste bin the logs momentarily 
[20:25] <faiqus> https://pastebin.com/Du9A8ds4
[20:26] <holmanb> faiqus: also /var/log/cloud-init.log 
[20:27] <faiqus> that is entirely empty
[20:27] <holmanb> faiqus: ahh right, if memoryy serves correctly you have a custom log configuration
[20:27] <faiqus>  wc -l /var/log/cloud-init.log 0 /var/log/cloud-init.log
[20:28] <faiqus> so yeah this /var/lib/cloud/instances/i-<>/user-data.txt has all the bits but there are two issues with it. 1. the jinja templates aren't rendered 2. it doesn't run
[20:29] <holmanb> faiqus: you'll need to figure out where those logs are going or switch to the default log configuration temporarily
[20:30] <faiqus> how would i do that? 
[20:30] <holmanb> faiqus: under /etc/cloud/cloud.cfg.d there is a file
[20:30] <holmanb> probably 05_logging.cfg
[20:31] <holmanb> that should have the log configuration
[20:31] <holmanb> this is the upstream default: https://github.com/canonical/cloud-init/blob/main/config/cloud.cfg.d/05_logging.cfg
[20:32] <faiqus> it looks almost identitical to the upstream one 
[20:33] <faiqus> https://pastebin.com/NuBGupMM
[20:34] <holmanb> faiqus: well it's not identical, the term which defines the log file is different
[20:35] <faiqus> oh sorry - the "almost" was doing a lot of work in the last message. what should i change?
[20:36] <holmanb> I don't know all the details of your setup
[20:37] <holmanb> but I'd try just using the upstream default configuration
[20:37] <faiqus> ok i will copy and paste that.
[20:38] <faiqus> im gonna build a new ami and run it - stay tuned. and thank you again for the help 
[21:10] <faiqus> holmanb: modifiying the log file to the upstream has given me a lot of logs. let me prepare a pastebin for you 
[21:12] <faiqus> holmanb: https://pastebin.com/a5HqKVsR
[21:15] <faiqus> the cloud-init-output.log looks about the same as the previous paste 
[21:24] <holmanb> faiqus: is the secret user-data always a jinja configuration?
[21:25] <faiqus> yeah, it will have jinja configuration. 
[21:26] <faiqus> i think almost always because it needs the host name 
[21:33] <faiqus> i couldn't find any convience functions to template that data in other datasources 
[21:41] <faiqus> yes, any time cloud-init will process the "secret" userdata there will be jinja configuraiton
[22:16] <connor_k> holmanb: I think I found a lead: https://github.com/canonical/cloud-init/commit/42f2cffadabfefb0469ade2f1f1c3ce5edabc9fa
[22:16] -ubottu:#cloud-init- Commit 42f2cff in canonical/cloud-init "remove vendor-scripts-per-{boot,instance,once}"
[22:41] <faiqus> im trying to parse the data but i can't seem to do that either. i imported this function -- render_jinja_payload_from_file. but its complaining that neither of these files: /run/cloud-init/instance-data-sensitive.json /run/cloud-init/instance-data.json is present