JoBbZhi, I'm not familiar with cloud init so I apologize in advance if I'm missing something basic. :) We're currently migrating systems from Amazon Linux 2 to Amazon Linux 2023.  However, I've run into a problem were the data payload I define in terraform on the autoscaling group to be passed to cloud init on startup no longer gets created or executed when an instance built on its AMI is brought online. I suspect it's a configuration difference between them wit15:54
JoBbZany pointers appreciated!15:58
minimalJoBbZ: not sure what you mean. How exactly do you expect a autoscaling group to relate to cloud-init? A ASG is AWS infrastructure, not anything to do with a VM's OS that cloud-init would configure16:16
JoBbZvia https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/cloudinit_config16:40
JoBbZand sorry, I mispoke.. it's part of the launch template16:41
JoBbZnot ASG :) 16:41
JoBbZ(although the launch template is included in the ASG definition, but point being, that's the flow above)16:49
minimalJoBbZ: ok, so that's just specifying the user-data to provide to the instance16:55
minimalso what exactly is the problem?16:55
minimalhave you checked the cloud-init logs on a new instance to verify it sees the specified user-data?16:56
blackboxswto check cloud-init user-data on the system in question `sudo cloud-init query userdata`17:03
blackboxswthx minimal for all the discussion fielding :)17:03
minimalcould do with some help for a testcase I'm writing. The c-i module calls "which" twice but I'm not sure how to check the result of the *2nd* call to "which" (I have cloudinit.subp.which mocked)17:07
minimalDoing m_which.return_value = "xyz" matches for the 1st "which" call17:10
minimalDoing m_Which.return_value = None matches for the 3rd situation (neither "which" call gives any result)17:10
JoBbZblackboxsw: will do that.. ty.  17:14
JoBbZminimal: yeah so I don't know a lot about this... I just know that my script 'ansible.cfg' ends up existing on the old systems for cloud init to execute, and doesn't on the new ones, but the terraform side is unchanged and I ran with with debug logging in terraform to ensure it's all generated correctly17:15
JoBbZok, I see a base64 encoded? result on both servers with that query17:16
minimalJoBbZ: so either looking at /var/logs/cloud-init.log (if debugging is enable) or running the command that blackboxsw suggested will show you the user-data that cloud-init is using.17:16
JoBbZminimal: yes, the command comes back with encoded and/or encrypted data...17:20
=== NightMonkey_ is now known as NightMonkey
JoBbZi've read through /var/logs/cloud-init.log, but I don't see where the user data bit comes into play exactly.  Basically the user data is supposed to create an executable called ansible.cfg, that exists in /var/lib/cloud/instance/scripts/ on the old AL2 systems, but it isn't created on the new AL2023 systems17:21
JoBbZok, grepping for userdata in the log I see17:24
minimalwhat is the content-type of the user-data you are specifying in the terraform file and what is the contents of that user-data given to terraform?17:24
JoBbZ[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'H4sIAAA...17:25
JoBbZand that string matches what's coming back from my query about user data17:25
JoBbZand that message does *not* exist on the other system17:25
JoBbZso... guessing that's the issue?17:26
minimalcan you show the relevant portions of your terraform file?17:26
JoBbZand I can get the debug logs from terraform one sec17:30
minimalJoBbZ: where is the part when you create an instance and set a user_data_base64 value for it?17:31
JoBbZyou can see that here in the debug log: https://dpaste.org/u8tmL17:35
JoBbZin the rendered: section17:35
JoBbZThe value in the rendered: section is what I get back from the command `sudo cloud-init query userdata`17:36
JoBbZas well17:36
minimalJoBbZ: I don't see any reference to a "aws_instance" resource to create a VM and pass it user-data17:37
JoBbZit's part of a launch template... so it gets put into the system when it comes up... regardless, as I noted, the user data that is rendered there is what cloud init is showing on both systems17:39
JoBbZit's just not having any effect on the AL2023 system, and there's the warning about (text/x-not-multipart) userdata not being handled on the AL2023 system17:40
JoBbZwhich is not the case on the AL2 system17:40
minimaland the 2 systems are running which cloud-init version(s)?17:40
JoBbZ19.3-46 amazon2.0.1 on the AL2 system17:41
JoBbZ22.2.2 on the AL2023 system17:41
minimalcould be a difference in behaviour between those 2 versions17:45
minimallooking at current code, text/x-not-multipart is used when "a message is not multipart and it doesn't contain its own content-type"17:46
minimalhowever you have specified "text/x-shellscript" as content-type17:47
JoBbZok.. I figured it must be a difference between the two releases as well, but not sure how/why17:50
minimalJoBbZ: have you tried *not* base64 encoding it to see if that makes any difference?17:52
JoBbZI've not, I can definitely try17:53
JoBbZactually... got something..18:01
JoBbZin /var/lib/cloud/instance/ there is a user-data.txt.i file18:02
JoBbZon the old systems, it's seen as multipart/mixed with a boundary18:02
JoBbZon the new systems, it has that, but then it's followed by Content-Type: text/x-not-multipart18:02
minimalhmm, but if you're only providing a single text/x-shellscript then why would it be treated as multipart?18:02
JoBbZso seems to be some sort of issue there with handling the generated MIME data in 2.22.2 that didn't exist in 19.3-4618:03
minimalon the old systems I mean18:03
JoBbZno, it's multipart... shell script and a #cloud-init config as well18:03
minimalso on the old systems, which are the content-type for each of the parts?18:03
JoBbZyou can see both parts in this: https://dpaste.org/u8tmL18:04
JoBbZfirst part is the #cloud-config part, with content type text/cloud-config18:04
minimalah........not sure if that is valid, I though for user-data you EITHER provided cloud-config OR a shell script, not both18:04
JoBbZsecond part is the script, with content type text/x-shellscript18:04
JoBbZno, you can combine them18:04
JoBbZthat's documented :)18:04
JoBbZit's just not extracting correctly with 22.2.2 for some reason hm18:04
minimal"User data that will be acted upon by cloud-init must be in one of the following types."18:06
minimalat first glance the "in one of" implies ONLY one type18:06
JoBbZnah it says both can be used https://cloudinit.readthedocs.io/en/latest/explanation/format.html#mime-multi-part-archive18:07
JoBbZ"For example, both a user data script and a cloud-config type could be specified."18:07
JoBbZwhich is exactly what I'm doing18:07
minimalah, further down, "For example, both a user data script and a cloud-config type could be specified."18:07
minimalright, just saw that18:08
JoBbZso something's changed in mime handling between the two releases I guess18:08
JoBbZI'll have to dig into that18:08
minimaljust wondering tho if you need to specify "text/cloud-config" when using multipart18:09
minimalBegins with: #cloud-config or Content-Type: text/cloud-config when using a MIME archive."18:09
minimalah, you are specifying that18:10
JoBbZyup :/18:10
JoBbZand again works fine with the older cloud init version... everything seems to match the docs18:10
JoBbZhm.. so I just took /var/lib/cloud/instance/user-data.txt18:13
JoBbZbase64 -d user-data.txt > file.gz18:14
JoBbZgunzip file.gz18:14
JoBbZand I get exactly what it is supposed to contain18:14
JoBbZit appears that the difference is the version fails to gunzip it18:14
minimaltry not gzip-ing the user-data in terraform and see if that changes things?18:17
JoBbZyeah, going to do that next18:18
JoBbZI'm only 4k, well below the 16k gzip requirement18:18
minimalI'm not seeing any multipart content-type in your Terraform doc. I'd assume it should be present next to "base64_encode": true and "gzip": true18:20
JoBbZterraform is supposed to do that automatically 18:21
JoBbZterraform creates the final mime document.  everything on the terraform side is the same between both systems, it's cloud init that's different between them18:22
minimalI guess I'm just assuming it would appear in the JSON you provided (which includes stuff like "rendered" value18:23
JoBbZsure understood :)18:23
minimalI'm just wondering if current code is "stricter" in terms of MIME spec18:23
JoBbZmaybe, but then I'd expect a lot of people's existing stuff to break18:23
JoBbZI'm on a very recent version of terraform as well (1.4.6)18:24
JoBbZwonder if amazon patched cloud init and broke it, always possible too18:24
JoBbZor redhat18:24
JoBbZ(since AL2023 is based on fedora/redhat)18:24
minimalso when you did the base64 decode and gunzip do you see a MIME document?18:26
JoBbZI see exactly what it should be18:27
minimalso it has a multipart/mixed content type?18:29
JoBbZI found a bug report for this issue18:29
minimalc-i bug?18:29
JoBbZsame exact issue I'm seeing18:30
minimalwhat's the number?18:30
-ubottu:#cloud-init- Issue 3712 in canonical/cloud-init "gzipped and base64 encoded user-data leads to failure" [Closed]18:31
JoBbZwhich is interesting, because it says fixed in 20.318:31
JoBbZand I'm on 22.2.218:31
JoBbZbut same exact problem18:31
minimalI though you'd tried without base64 and gzip?18:34
JoBbZI'm trying w/o gzip now18:34
JoBbZtakes an hour to rebuild18:34
JoBbZso I'll know more in about 40 minutes if removing gzip resolves it18:34
minimalgood luck!18:43
user1001Hello comunity, what format is the "Fingerprint (sha256)" in cloud-init logs shown e.g. in "Authorized keys from /home/ubuntu/.ssh/authorized_keys for user ubuntu" ? I need to compare that "Fingerprint (sha256)" with fingerprints of my local public-private key pair. How can I compare them?18:45
minimaluser1001: isn't it in sha256 format? ;-)18:50
minimaluser1001: you use the relevant ssh tool to display your public key's fingerprint18:55
user1001yeah, if I generate from my public key I get "SHA256:SH8HjfHJdpu3MiAEpTuF4YrLNislS6GD+meUh11z44g" or "MD5:2e:00:ca:99:c4:6f:fb:78:2f:26:b4:f9:c4:84:83:78", but in the cloudinit logs I see a much longer fingerprint: "48:7f:07:8d:f1:c9:76:9b:b7:32:20:04:a5:3b:85:e1:8a:cb:36:2b:25:4b:a1:83:fa:67:94:87:5d:73:e3:88" (according to logs should be18:59
user1001"Fingerprint (sha256)" )18:59
minimaluser1001: on which version of cloud-init are you seeing this in the logs? I only see it on the console (but not logs) with 22.219:22
minimaloops, s/22.2/23.2/19:22
JoBbZwell just turning off gzip didn't do it... turning off base64 too19:29
user1001 @minimal  Cloud-init v. 22.2-0ubuntu1~22.04.319:29
minimaluser1001: ok. I think this behaviour was changed to not log it. Anyway, what exactly is the underlying problem you are having?19:30
user1001minimal I am booting an jammy-server-cloudimg-amd64 image in OpenStack, and OpenStack injects a ssh key into "/home/ubuntu/.ssh/authorized_keys", but I cannot make a ssh conecttion - so I try to verify that the ssh key was injected correctly (already tried everything else)19:35
minimalcat /home/ubuntu/.ssh/authorized_keys ?19:42
meenaI was off by a month for the summit19:45
meenawhew 19:45
minimalmeena: panicing about travel plans? ;-)19:46
meenaminimal: i can't travel until my partner is fully licenced. but i would like to prepare stuff19:49
meenait would be nice if that was the case next month… but that's not something i can actually bank on 19:52
user1001minimal I have no access to the VM, as OpenStack only booted the source image so I can start installing Ubuntu, but I can't, because normal login does not work, and ssh access cannot verify my ssh key...19:53
JoBbZminimal: OK, it works after disabling both base64 encoding and gzip20:36
JoBbZI've filed a bug with cloud-init20:36
user1001if someone stumbles on this problem (xkcd.com/979 ;-) solution: the cloud-init log for the authorized_keys file is produced in cc_ssh_authkey_fingerprints.py by the function _gen_fingerprint(), which generates the long "Fingerprint (sha256)" for the public ssh key (the first function argument), so you can generate and compare the fingerprint of21:49
user1001your local public ssh key with the fingerprint reported in cloud-init log21:49
=== not_phunyguy is now known as phunyguy
=== not_phunyguy is now known as phunyguy

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!