[14:03] <jgomo3> Hi! I'm writing my cloud-config file, with some values I would like to parametrize. E.g. A software version. What would you recommend me to do to achieve that parametrization? The idea I have for now is to process the file with a template system like Jinja2 before launching the instance, but I'm wondering if there is some "parameters" service implied by cloud-init that I'm missing out.
[14:58] <falcojr> jgomo3: cloud-config supports jinja templating and can make use of the instance metadata. https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html#using-instance-data
[14:59] <falcojr> I think templating based on things beyond instance metadata is outside the scope of cloud-init
[15:56] <holmanb> Any Neovim users here? Here's a blog post that describes cloud-init configuration using yaml-language-server and the userdata schema for tab completion / key hinting.
[16:02] <blackboxsw> holmanb: #ENOLINK
[16:03] <blackboxsw> aciba: thanks for the wrap up on the alpine cC_set_passwords warnings. landed. I'm finishing your other PR for this release that'll help subiquity and the auto-installer and should be able to land that in the netx ~15
[16:03] <blackboxsw> *next 15 min
[16:05] <holmanb> @blackboxsw: ha!
[16:05] <holmanb> https://phoenix-labs.xyz/blog/setup-neovim-cloud-init-completion/
[16:11] <aciba> blackboxsw: thank you, it was a pleasure. Let me know if there is something I can help with for the remaining PR or whatever!
[16:19] <blackboxsw> aciba: actually one more nit PR if you get a chance. Can you add 'text/plain' to an allowed enum value  of "write_files.encoding" in json schema  It's documented as "default" value when encoding is not specified, but I think we may want to change cc_write_files to allow the value "text/plain" since schema auto-completion tools (like vscode) will auto-inject those default values in auto-completion.
[16:20] <aciba> blackboxsw: sure, let me check it
[16:21] <blackboxsw> esposem: hiya (sorry to IRC pounce) thanks for the Redhat related PRs, there are a couple of open questions do you know if that is something you'll have time to poke at this week?
[16:21] <blackboxsw> ooh looks like you may have already responded. reading now
[16:21] <esposem> blackboxsw: yes :)
[16:22] <jgomo3> falcojr: Thank you very much! I'll check that.
[16:25] <esposem> blackboxsw: feel free to ping me also here if you have any other question. In general, I was not sure what the additional services/module were doing, and since our downstream package did not use them, was afraid that something would break. Buf for the After=X thing there should be no problem, and modules are supporting "rhel".
[16:26] <blackboxsw> +1 esposem I was just confirming the specific systemd docs on After=X only triggering if that systemd unit X is present (and ignoring when absent) which should give us the behavior we want. NOOP when hv-kvm not available and order after when it is.
[16:28] <blackboxsw> esposem: the only "risky" thing I see once we drop those conditionals which redacted is the NetworkManager restart logic in cloud-init-final. trying to get more context on that behavior now 
[16:32] <esposem> blackboxsw: do you want to drop conditionals also for the "ExecStartPost" line in cloud-init-final?
[16:34] <esposem> blackboxsw: That was actually proposed by someone else from ubuntu contributing to cloud-init, the RHBZ contained already the fix. Maybe we could ask him if it's the case of having this fix upstream or not
[16:35] <blackboxsw> esposem: I'm wondering about dropping that ExecStartPost part from this PR temporarily so that we can put this in upstream 22.2 that we'd like to cut for today. but still reviewing now to be sure
[16:36] <blackboxsw> holmanb: or falcojr or aciba did you have any opinions on https://github.com/canonical/cloud-init/pull/1431/files#diff-eb94d50ed1a0a76032438b4bc5bccb389e9cc03d6766d7881463d8ee75f21f8fR22-R27 
[16:37] <blackboxsw> specifically lines 22-27 above
[16:38] <esposem> blackboxsw: or maybe leave it under conditionals? I don't know, but I think it will cause regression if we don't add that to rhel. And if we drop it completely then files are not aligned and we still need to resolve to hardcoded files
[16:39]  * holmanb looking
[16:39] <blackboxsw> ahh ok I see the originator here of the patch here :) https://bugzilla.redhat.com/show_bug.cgi?id=1897616
[16:40] <blackboxsw> interesting. and this looks like it was contributed by smoser :) given the RHEL/centos for quite some time 
[16:40] <blackboxsw> ok this makes me feel a bunch better about the intent.
[16:48] <aciba> blackboxsw: PR for cc_write_files' schema created here: https://github.com/canonical/cloud-init/pull/1460
[16:49] <holmanb> blackboxsw: +1 concur no objections as-is
[16:49] <holmanb> specifically the 22-27 lines
[16:50] <blackboxsw> thx holmanb . aciba I think there might need to be a change in cc_write_files.py to allow for a value of "text/plain" at runtime
[16:50] <blackboxsw> sorry the 2nd half of that messsage was only for aciba per #1460
[16:51] <holmanb> all good, figured :)
[16:51] <blackboxsw> aciba: I believe the handle function will reject a value of "text/plain" but it shouldn't
[16:52] <esposem> blackboxsw: holmanb: thank you :) If all is good, I will resolve the conversations. I am still waiting from our QA to perform regression tests just to be sure all is working as expected, once I get that I will ping blackboxsw or falcojr to merge everything :)
[16:55] <holmanb> esposem: know how long QA will take?
[16:56] <esposem> holmanb: hopefully not too long :) They already did some initial tests and it looks good. any specific reason?
[17:01] <aciba> blackboxsw: verifying it
[17:02] <holmanb> esposem: 22.2 release is today
[17:04] <esposem> holmanb: then it won't make it for sure
[17:13] <holmanb> esposem: I think this counts as relatively low risk and there are a couple of other PRs getting merged last minute as well (assuming tests pass) today so I think we were open to including in 22.2 depending on the timeline
[17:13] <holmanb> esposem: if you'd rather hold off, however, happy to oblige
[17:14] <esposem> holmanb: folks from QA are in China, so it's at least 7-8 hours before they read my request for updates
[17:14] <esposem> let's see
[17:35] <jgomo3> Hi. Problem: specify something like a version in my cloud-config file as a parameter. Knowing that I could use Jinja for accessing the instance metadata and use that information in the cloud-config user data, I'm imagining the posibility to add my own data source so I could write a template like this example: http://ix.io/3Y16
[17:35] <jgomo3> (... cont) But reading about Datasources, it looks like is something for the cloud provider, not for the users to define.
[17:36] <jgomo3> Is that the case? Or is it possible to add a custom datasource in a cloud provider, let's say AWS?
[17:37] <jgomo3> Beside: I understand the chicken egg problem, and the fact that the "user data" IS the place to let the users to add parameters to control the cloud init process.
[17:38] <aciba> blackbosw: cc_write_files updated. It did write the file but logged a wrong warning. Updated and test added.
[17:40] <holmanb> jgomo3: "datasource" at a high level defines "where do cloud-init inputs come from"
[17:41] <holmanb> not sure I understand the use case - why not just update the cloud-config when you want to specify a different version? presumably as a user for your parametrization you would have to update custom metadata keys anyways, why complicate it?
[17:43] <jgomo3> (... cont) But some capability like the example I'm provided would be "nice to have", because the other alternative to achieving that is to write an script that prepare the context before installing the package (eg. http://ix.io/3Y1a). But you can see, it is sad I can't use the `packages` key, and have to do that scripting instead.
[17:43] <blackboxsw> thanks aciba merged https://github.com/canonical/cloud-init/pull/1440
[17:45] <jgomo3> holmanb: Yes, that's the other way to do it. That step before launching, where I generate the new cloud-config file, is the one I would like to remove.
[17:47] <jgomo3> holmanb: That I could hide automating it. But I'm just digging in the cloud-init capabilities and docs just to check if there is support for such a parametrization as explained, so I'm not doing the unnecesary automation step (in other words, I'm researching if it is actually unnecesary or not, so far, it is necessary).
[17:51] <jgomo3> holmand: and when, one thing lead to the other one and I'm just Imagining this capability (if doesn't exist already, who knows). Anyways, you say "datasource" define "where do cloud-init inputs come from", and that sounds like what I'm looking for: Tell cloud-init to get inputs from an arbitrary source. How could I add my own arbitrary source in a cloud (beside the userdata, which I understand IS the normal way to inject arbitrary
[17:51] <jgomo3> data)? Or is it something that only the cloud owners can do?
[17:59] <blackboxsw> jgomo3: trying to get the meat of your feature request. Generally each cloud platform (read as Datasource) provides metadata, vendordata and userdata values to cloud-init in a specific format. The metadata,vendordata and userdata is base structured configuration which cloud-init ends up storing in /run/cloud-init/instance-data.json  (to give you the params you are sourcing in templates).
[18:01] <blackboxsw> it sounds like you are looking to either define your own datasource to supply more detailed parameterized package installs which can be used or possibly finding a way that you can supplement /run/cloud-init/instance-data.json with more known parameters that you could source in #cloud-config during initial install?
[18:04] <blackboxsw> in your AWS example, you can't adapt/extend the metadata provided by AWS datasource directly as it only sources configuration from their instance metadata service which lives at 169.254.169.254/... https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceEc2.py#L512  you can see that cloud-init is told both metadata and user-data from various routes on the datasource. 
[18:05] <jgomo3> blackboxsw: Cool, let me study that, thank you.
[18:05] <blackboxsw> But, if your "cloud" were to use NoCloudNet datasource which sources a URL of your choosing or NoCloud (which looks for seed files in /var/lib/cloud/seed/nocloud-net/(meta-data|vendor-data|user-data)) then you'd be able to provide any YAML metadata your want
[18:07] <blackboxsw> jgomo3: adding a custom datasource is not too hard to get upstream if needed and other existing datasources don't quite surface what you need. We can help you with that as you get your bearings. https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html
[18:08] <blackboxsw> jgomo3: also if you have access to setup lxd in your environment I recommend James' tutorial https://cloudinit.readthedocs.io/en/latest/topics/tutorial.html as it'll leave you with a LXC container that you can poke around with (and see how /var/lib/cloud/seed files affect the underlying system)
[18:08] <blackboxsw> LXC uses the NoCloud datasource for what it's worth
[18:09] <jgomo3> blackboxsw: Awesome! Thank you very much. I'll dig into that.
[18:15] <jgomo3> And just out of curiosity: can I add my own modules to be used when launching instances in AWS? 1 way to do it given the idea you just gave me about LXD, would be launching an EC2 instance, configure LXD in it and playing with the cloudinit at that level, where I think I can do anything I want. But, is there any other way without that extra "virtual" level?
[18:40] <falcojr> jgomo3: you can run LXD locally on linux or in a VM. cloud-init runs directly in the container and doesn't need to be on a cloud instance like EC2. Is that what you're asking?
[18:47] <jgomo3> falcojr: Hi, no. I mean, in the end, I'm writing cloud-config files to be used in AWS. But I know LXD locally in my machine gives me more flexibility around cloud-init than what I can adapt in AWS. My last question was only if I can write my own module so I can use it in a cloud-config file for launching an AWS EC2 instance.
[18:49] <jgomo3> falcojr: and as an aside to my question, I was describing an idea of how I would try to do it if it is not possible: installing LXD in the EC2 instance and launching LXC containers there. I think that way makes sense to talk about creating modules myself.
[19:10] <jgomo3> Hi! How would you install a private ssh key in an AWS instance? The idea is that the launched instance can git pull from some repository.
[19:11] <jgomo3> Oh, I see, rsa_private. Is that secure?
[19:42] <holmanb> jgomo3: re: Is that secure? <- uhhh, maybe? depends? what's your threat model?
[19:42] <holmanb> jgomo3:  The privkey would be accessible via the imds, so anything that can access the IMDS can access your privkey.
[19:52] <holmanb> "Is that secure?" is a loaded question in many ways (who wants to say "yes" in one context and be proven wrong in another with no chance of rebuttal)
[19:53] <holmanb> so official answers are typically "ask your security person(s)"
[19:54] <holmanb> lots of security folks probably agree with statements like "don't copy privkeys from one machine to another"
[19:57] <holmanb> jgomo3: Sorry for the non-answer. I typically generate my keys on system after boot. That allows me to set the password on the ssh key and specify the keysize/algo/etc.
[21:15] <jgomo3> holmanb: Thank you very much!. Beside the secure part, I get now your way of installing a private key in the first booted server. But the challenge now is that that server should be able to interact via git with a repo, where a public key is assigned to an user. Generating the keys after launching the instance is not an option for a fully automated solution, unless there is an API I could use to assign from the new instance the newly
[21:15] <jgomo3> created public key. But then we have a chicken egg problem because probably the new instance would need credentials to use that API. So, we know that `rsa_private` is an option. What other options are there, how does people launch an instance ready to clone a git repository?
[21:16] <jgomo3> TLDR; How would you use cloud-init to launch an instance ready to clone a git repository via SSH?
[22:27] <holmanb> jgomo3: "Unless there is an API I could use...." -> The phone home module might be able to do what you're describing https://cloudinit.readthedocs.io/en/latest/topics/modules.html#phone-home