[05:06] <socket-> Hey all, any idea why my cloud-init config is not creating the ssh_host keys for me in this CFT? https://imgur.com/a/5t86kS7
[05:07] <socket-> instead it does the normal random generation at the start up... Generating public/private ecdsa key pair. Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key. Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub. The key fingerprint is: SHA256:sILmAwj+6f1qIJTvw0R1QGegrX75uY+q9bc5yLIniRI root@ip-10-0-1-129
[05:09] <socket-> https://apaste.info/X8x0
=== juergh_ is now known as juergh
[12:44] <rharper> socket-: the cc_ssh module will write host keys if you provide them in user data,   you'll need to lay out your config, like so https://cloudinit.readthedocs.io/en/latest/topics/modules.html#host-keys
[12:45] <rharper> socket-: if are are including the host keys in the expected form and it's not writing them out, please file a bug and include the tarball output from cloud-init collect-logs
[16:50] <socket-> Hmm, so i think the problem is that I am trying to use cloud-init code in 'AWS::CloudFormation::Init:'. I think i need to figure out where is this AMI the cloud-init cofig is stored and add the ssh stuff there
[16:51] <socket-> AWS::CloudFormation::Init only supports packages, groups, users, sources, files, commands and services
[16:56] <rharper> huh
[17:09] <socket-> im saying cloud-init is different from cloudformation init.  I thought one just called the other, but it only has a subset of the commands
[17:11] <socket-> So it looks like I can pass a cloud-config in the EC2 instances userdata like you said.  I currently have shell code being passed to the user data. Do you know if they can be mixed, or if i have to change all the shell code to be cloud-config runcmds?
[17:18] <socket-> rharper: do you know if i can do something like this? https://apaste.info/EteD
[17:43] <Odd_Bloke> socket-: You would either need to convert your shell script to a runcmd stanza (which can be strings, so that's not impossible by any means), or you'd need to look at multipart MIME user-data.  (Let me look up references for those options now.)
[17:44] <Odd_Bloke> runcmd documentation is https://cloudinit.readthedocs.io/en/latest/topics/modules.html#runcmd, MIME multipart is https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive.
[18:02] <rharper> Odd_Bloke: socket-  cloud-init/tools/make-mime.py is quite useful here for constructing a multi-part mime
[18:05] <Odd_Bloke> rharper: Hmm, seems like we should probably point to that instead of the inline helper script.
[18:10] <Odd_Bloke> I'll fix that now.
[18:28] <socket-> so iv learned some more. AWS user data allows only one content type. You can make this content type be multipart/mixed though.  So iv been testing out a simple mix https://apaste.info/c4HT
[18:29] <socket-> I havent got it working yet, but so far i can confirm that cloud-init-output says it ran the final_modules, and also the shell code at the bottom was run, and the file was created
[18:29] <socket-> but none of the packages are being installed, its like cloud-init is skipping that section
[18:31] <Odd_Bloke> socket-: Can you pastebin what your latest config looks like?
[18:31] <socket-> Odd_Bloke: lol thanks just saw your message
[18:31] <socket-> https://apaste.info/c4HT
[18:32] <socket-> thats what im sending to aws userdata
[18:33] <socket-> https://apaste.info/WtST
[18:34] <socket-> ^ is the cloud-config.txt attachment aws uploads
[18:36] <Odd_Bloke> socket-: What does `cloud-init query -a` give you?
[18:37] <socket-> lots of meta data about the instance
[18:37] <socket-> do you wanna see the whole thing?
[18:38] <socket-> Odd_Bloke: https://apaste.info/GYGV
[18:39] <socket-> i removed the crypto but the rest is intact
[18:41] <Odd_Bloke> OK, I was hoping that would display the user-data after we'd unpacked it, so that didn't help. :p
[18:41] <Odd_Bloke> socket-: Do you have any WARN lines in /var/log/cloud-init.log?
[18:42] <socket-> i think it just writes 2 files to disk, the cloud-init.txt and userdata.txt
[18:42] <socket-> so i have access to both of those
[18:45] <socket-> checking
[18:46] <socket-> nothing for grep -i warn /var/log/cloud-init.log
[18:51] <socket-> Odd_Bloke: if i take out cloud_final_modules ... then it installs the packages defined
[18:51] <socket-> but it doesn't run the shell script
[19:17] <Odd_Bloke> socket-: Could you try using make-mime.py to create your input file?  That b64 encodes stuff, which might work if there's something strange about the input that's breaking things?
[19:25] <socket-> havent tried that yet, but in my latest test I noticed..
[19:25] <socket-> 2020-04-28 19:22:20,990 - __init__.py[WARNING]: Unhandled unknown content-type (text/plain) userdata: 'b'MIME-Version: 1.0'...'
[19:26] <socket-> when using https://apaste.info/Ji5n
[19:31] <socket->  python mime.py -a user.txt ... ValueError: not enough values to unpack (expected 2, got 1)
[19:31] <socket-> whats the syntax
[20:43] <Odd_Bloke> socket-: filename:mimetype (where mimetype will be cloud-config for one and x-shellscript for the other)
[20:43] <Odd_Bloke> Something like `./tools/make-mime.py -a config.yaml:cloud-config -a script.sh:x-shellscript`
[21:14] <Odd_Bloke> blackboxsw: Would really appreciate your review on https://github.com/canonical/cloud-init/pull/335, I put that together in the background while I was lurking in that meeting earlier.
[21:14] <Odd_Bloke> (Obviously has a couple of TODOs, but would like to know how that looks as a technical schema as far as you're concerned.)
[21:15] <Odd_Bloke> (And I'd like to know what testing you generally do, too. :)
[21:23] <blackboxsw> will do Odd_Bloke
[21:26] <blackboxsw> haha nice Odd_Bloke
[21:26] <blackboxsw> I thought I heard a bit of typing then :)
[21:26] <blackboxsw> Odd_Bloke: rharper rick_h just drew up an initial spec for review for cloud-init a json schema artifact
[21:32] <rharper> blackboxsw: nice
[21:40] <blackboxsw> Odd_Bloke: couple of minor comments on https://github.com/canonical/cloud-init/pull/335/files
[21:41] <blackboxsw> rharper: Odd_Bloke I think we have to think a bit about how we handle values that come in from /etc/cloud/cloud.cfg https://github.com/canonical/cloud-init/pull/335/files#r416940582
[21:51] <rharper> blackboxsw: I don't think so, args is not set via user-data , no ?
[21:51] <rharper> if you wrote your own etc/cloud/cloud.cfg.d/XXX file *and* you override the modules_stage list, *and* you pass args on that
[21:55] <blackboxsw> rharper: I think args is only set via /etc/cloud/cloud.cfg.d/XXX by overriding the modules_list and providing - <config_name> <frequency>   <n-args>
[21:57] <rharper> right
[21:57] <rharper> you can provide user-data that overrides the list, but it's really awkward to use
[21:57] <rharper> I'd rather log deprecated and avoid exposing it more
[22:00] <blackboxsw> yeah it feels gross/wrong.
[22:00] <blackboxsw> and undocumented
[22:35] <blackboxsw> rharper: Odd_Bloke here's the undocumented image-based module config  I was talking about https://pastebin.ubuntu.com/p/qjkVYVq5Vg/
[22:35] <blackboxsw> provide a list for the config module and you'd have to know the specifics of how that module reacts to additional args
[22:36] <blackboxsw> so yeah we can chalk up an item to deprecate that