[11:34] <sirhopcount> Quick question related to datasources. The 18.3 documentation has reference on how to configure them but not on how to use them. I'm trying to set the availability zone in the content of a file. I tried using the syntax of the most docs: Availability_zone: {{ v1.availability_zone }} but for some reason the v1.availability_zone isn't interpreted. Does 18.3 support the use of datasources and if so what am I doing wrong?
[14:37] <meena> sirhopcount: "I'm trying to set the availability zone in the content of a file. I tried using the syntax of the most docs:" where you setting that
[14:38] <sirhopcount> I'm creating a file using 'write_files:` and I use that in the 'content: |' section.
[14:39] <meena> i am suddenly more confused than ever
[14:40] <meena> for what is this configuration? where are you writing that file?
[14:40] <sirhopcount> It's a cloud-config file for AWS.
[14:46] <sirhopcount> meena: This is an example of what I'm trying to do: https://gist.github.com/sirhopcount/6873745c1de6a830cb40ab0b26e402d9
[15:55] <rharper> sirhopcount: you need tag the file with the jinja template key,  ## template: jinja
[15:55] <rharper> that needs to be the very first line in your cloud-config file
[15:55] <rharper> this tells cloud-init to run your config through the template renderer which expands the {{ }} keys
[15:56] <rharper> https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html   *search for jinja*
[16:04] <sirhopcount> rharper: thanks!
[16:06] <rharper> sirhopcount: yw
[16:17] <sirhopcount> rharper: I might be mistaken but that feature doesn't seem available in 18.3
[16:26] <rharper> sirhopcount: yes, 18.3 was tagged in June 2018, feature landed in Sept 2018, 18.4 tagged Oct 2, 2018;
[16:26] <rharper> sirhopcount: so you'll need 18.4 or newer
[16:26] <sirhopcount> thats to bad, Debian Buster ships with 18.3 and I don't think there is an easy way for me to upgrade.
[16:27] <sirhopcount> thanks for the help anyway :)
[16:27] <rharper> sirhopcount: sure;  is there a newer release in downstream in testing ? or unstable ?
[16:28] <sirhopcount> Bullseye seems to have 19.4-2
[16:31] <sirhopcount> but the buster backports repo doesn't seem to contain it.
[16:35] <sirhopcount> Debian might but be considered stable but the facts they deploy with "old" software versions has more then once bitten me. I prefer Ubuntu over Debian but unfortunately its not up to me in this case.
[16:39] <rharper> sirhopcount: I see, bummer
[16:42] <sirhopcount> mm seems the debian apt repo does contain a cloud-init_19.4-2_all.deb which seems to install just fine. Going to test with that.
[19:20] <blackboxsw> rharper: yaml on kernel commandline is not as easy as we thought as we are parsing that cmdline spliting on spaces. valid yaml for {'config': 'disabled'} is actually "config: disabled" with a space otherwise yaml.load interprets "config:disabled" as a single concatonated key which gets loaded as {"config:disabled" : None}
[19:20] <rharper> blackboxsw: yes, the space is required, is the parsing code eating the space ?
[19:21] <rharper> I see the tok is done with split()
[19:21] <blackboxsw> rharper:  no, yaml.load needs that space. and if we present a kernel cmdline with that space, then our handling in tok split breaks
[19:22] <rharper> blackboxsw: ok, then maybe we want to regex for the disabled only
[19:22] <blackboxsw> but I'm thinking generally about kernel cmdline parsing if allowing spaced in cmdline values could break other things
[19:22] <blackboxsw> *spaces
[19:22] <blackboxsw> rharper: I think possibly, but seeing if we can be more flexible than that
[19:23] <rharper> we also have cc this is cloud config parsed end_cc;
[19:24] <rharper> blackboxsw: see util.read_cc_from_cmdline
[19:25] <rharper> so, the proper cmdline would be cc: network-config: {config: disabled} end_cc
[19:25] <blackboxsw> rharper: yeah that makes more sense
[19:26] <blackboxsw> good suggestion
[19:26] <rharper> >>> from cloudinit import safeyaml
[19:26] <rharper> >>> safeyaml.load(util.read_cc_from_cmdline("foobar cc: network-config: {config: disabled} end_cc wark"))
[19:26] <rharper> {'network-config': {'config': 'disabled'}}
[19:27] <blackboxsw> and good  fix suggestion :)
[19:28] <rharper> and then lastly, we need to decide, if we support the explicit:  network-config={config: disabled}
[19:28] <rharper> I think doc-wise, we should check that disable_string is in the cmdline
[19:29] <rharper> if 'network-config={config: disabled}' in cmdline:  then return {'network-config': {'config': 'disabled'}}
[19:29] <blackboxsw> rharper: if we are supporting a one-off, non-yaml string. why not just network-config=disabled
[19:29] <rharper> I think that closes the bug
[19:29] <rharper> because we documented it the other way ?
[19:29] <rharper> I'm open to suggestion  here
[19:30] <rharper> clearly no one is using it
[19:30] <rharper> as it doesn;'t work
[19:30] <rharper> so, we could update the docs with all methods that work now (without any changes) for example the cc: .... end_cc
[19:31] <blackboxsw> hrm I'm on the fence. I like simple ways to globally disable something, but writing an example cc: end_cc may advertise that more flexible functionality
[19:32] <blackboxsw> which folks could better leverage for bigger c-config changes
[19:32] <rharper> since the current doc is broken as it is;  do we support that documented string ?
[19:33] <rharper> I'm find with saying , no; that's broken;  instead use:  network-config=disabled;  and support that;
[19:33] <rharper> separately we can file a bug on documenting use of cc: / end_cc;  and figure out if cloudinit/net/cmdline.py should be employing read_cc_from_cmdline() (try this first, followed up by search for network-config) etc
[19:34] <rharper> I think the non-encoded yaml only works if it's wrapped in cc: end_cc
[19:34] <rharper> as you mention, the spaces are needed
[19:35] <blackboxsw> I not even sure we can/should be supporting the documented string  "network-config={config: disabled}" with the space because that type of kernel commandline I don't think conforms to typical k=v seen on kernel commandlines. So other cmdline parsing tools may break as a result if cloud-init *did* support that.
[19:36] <blackboxsw> but I'm good with documenting a simple network-config=disabled  or more complex cc... end_cc options too
[19:36] <rharper> blackboxsw: yeah; I think you're right there as well
[19:36] <rharper> yes
[19:36] <blackboxsw> rharper: I just thought about the cc end_cc  approach we use too, might also break those same cmdline parsing tools that are out there anyway
[19:37] <rharper> so to fix the bug, 1) update docs to metion network-config=disabled 2) add this to the cmdline.py
[19:37] <blackboxsw> but, that's a separate issue
[19:37] <rharper> blackboxsw: well, they'll ignore it
[19:37] <blackboxsw> +1 on 1 and 2
[19:37] <rharper> cool
[19:37] <rharper> and we can follow up with documenting cc/end_cc separately
[19:37] <rharper> and the refactor around b64/gzip stuff;  low-hanging fruit cleanup
[19:37] <blackboxsw> 3) would you like cc/end_cc docuemented option for disable in the rtd docs too?
[19:38] <blackboxsw> +1 on your statements up until now
[19:39] <blackboxsw> I'm good with document 3 too. and we can followup later if this is a breaking behavior for others in cloud-init that we need to address (but as you said, it probably isn't something that breaks cmdline parsing folks as they'll ignore things they don't care about)
[19:39] <rharper> I dunno about (3);  I suppose we should test whether that works as well (it should, I know I've tested a cc: some cloud config here end_cc for ssh_import_id
[19:40] <rharper> we can hold off on (3) I think, as you mentioned, then folks are wondering in general about cc: end_cc; and I'd rather document that fully else where and then we could add a disable network cc:end_cc and refer to the cc:end_cc docs for more details
[19:41] <blackboxsw> yeah and for 3 we do that for https://github.com/canonical/cloud-init/blob/master/doc/examples/kernel-cmdline.txt
[19:41] <blackboxsw> yeah something separate
[19:41] <blackboxsw> ok
[19:41] <blackboxsw> got it will post this PR within the hour
[19:44] <rharper> cool
[20:04] <blackboxsw> rharper: question at this point is if we will ultimately support cc ... end_cc for network-config on the commandline. As it is currently, merged config dict looks like it might ignore the network-config=base64-stuff and only rely on util.read_cc_from_cmdline. so we may have a gap there too in terms of merged network config.
[20:05] <blackboxsw> I think there is a separate bug here that would be helpful to address when we address 3) adding cc ... end_cc documented option for network-config cmdline setup
[20:05] <rharper> blackboxsw: I suspect because if cc/endcc being under-advertised that we've got merge gaps; so let's file a bug for that and we can fix that separately
[20:05] <blackboxsw> even more reason to address separately
[20:05] <rharper> ack
[20:05] <blackboxsw> yeah
[20:13] <blackboxsw> rharper: so, do we drop the documented yaml support https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#default-behavior ?
[20:15] <rharper> it's supported but only if it's b64 encoded
[20:15] <rharper> it was the line about, you can disable this with network-config={config: disabled} on kernel cmdline;  that gets moved to network-config=disabled, and we need to *bold*/*note* the yaml stablement that it requires b64 encodingin
[20:15] <rharper> blackboxsw: ^
[20:17] <blackboxsw> rharper: I was wondering about the text
[20:17] <blackboxsw> ip= or network-config=<YAML config string>
[20:17] <blackboxsw> rharper: I was wondering about the text     ip= or network-config=<YAML config string>
[20:17] <blackboxsw> we can't really support YAML config string currently
[20:18] <rharper> s/YAML config string/Base64 encoded YAML config string
[20:18] <blackboxsw> roger
[20:18] <blackboxsw> thanks
[20:48] <blackboxsw> rharper: https://github.com/canonical/cloud-init/pull/232
[20:48] <rharper> thanks
[21:11] <Goneri> @blackboxsw, the NetBSD PR is ready for review https://github.com/canonical/cloud-init/pull/62 :-)
[21:13] <blackboxsw> rharper: thoughts on this for focal? https://trello.com/c/wepeWcKf/1109-ec2-support-secondary-ip-addressses-on-eth0
[21:13] <blackboxsw> Goneri: +1. will spend get you an update today.
[21:14] <Goneri> :-) awesome, I will rebase the OpenBSD branch this weekend if this one get merged this week
[21:17] <rharper> blackboxsw: +1, we need FFE and I left review on it; haven't looked again, but generally wondered about whether we also need the route-metric and dhcp*-override like we have on azure
[21:18] <blackboxsw> ok rharper so I have 2 FFEs: instance-data to include merged_cfg and and ec2 secondary ip address support
[21:19] <rharper> yeah, I think so
[21:21] <blackboxsw> ok I fixed ci for https://github.com/canonical/cloud-init/pull/230 I found an httpretty.last_request() that *is* available on xenial httpretty v 0.9.6
[21:22] <blackboxsw> rharper: so that first bug fix for ec2 redact should be good to go
[21:22] <rharper> blackboxsw: let me re-review
[21:22] <rharper> I had one nit on docstring
[21:22] <blackboxsw> and I'm teasing out the bug-fix from feature-request for instance-data top-level keys @ https://github.com/canonical/cloud-init/pull/214
[21:22] <ahosmanMSFT> blackboxsw:
[21:22] <blackboxsw> will push a separate BR for the bugfix
[21:23] <blackboxsw> hi ahosmanMSFT
[21:23] <ahosmanMSFT> Hey blackboxsw what would be the best way to keep our repo of cloudinit up to date with upstream and have the debian salsa packages included, without having to do it manually?
[21:26] <blackboxsw> ahosmanMSFT: not sure I understand the question, you mean how can you get latest cloud-init in a debian salsa based image?
[21:28] <blackboxsw> or are you looking for how best to install salsa debs on an ubuntu system?
[21:28] <ahosmanMSFT> something like this https://salsa.debian.org/cloud-team/cloud-init
[21:31] <ahosmanMSFT> blackboxsw: We have a branch of cloud-init with some changes, I was wondering what the best way to get debian salsa packages on there and also keep everything up to date
[21:31] <ahosmanMSFT> currently we have a manual approach
[21:32] <blackboxsw> ahosmanMSFT: I think that project is owned by debian devs to get cloud-init updated/maintained in debian proper. I see the committers are names that don't participate in cloud-init upstream as far as I can tell
[21:33] <blackboxsw> you might be able to touch base with Noah or Thomas listed as debian salsa maintainers there https://salsa.debian.org/cloud-team/cloud-init/-/commits/master/
[21:33] <blackboxsw> ahosmanMSFT: since we don't control when specific distributions pick up cloud-init upstream changes, each distro differs (besides Ubuntu which we *do* control)
[21:34] <ahosmanMSFT> blackboxsw: thanks that helps, I'll try getting in contact with them
[21:35] <blackboxsw> ahosmanMSFT: you might be able to trigger your own deb package builds via using cloud-init's packages/bddeb in our source tree to roll your own cloud-init upstream deb if that helps too
[21:35] <ahosmanMSFT> That might be better, or we would have to rely on debian releases to update our cloud-init
[21:36] <blackboxsw> ahosmanMSFT: you could also try using cloud-init's ./tools/run-container --package  debian/10 #  to build a deb from source on a debian container
[21:37] <rharper> blackboxsw: one challenge is not using Debian's debian/ packaging dir
[21:37] <blackboxsw> that tool should create a binary deb for you from whatever your latest commit is in your local directory
[21:37] <rharper> what you really want is git checkout cloud-init.git; git merge /path/to/debian/cloud-init/git/branch/with/debian-dir-changes
[21:37] <rharper> which is like our daily-ppa for our release branches
[21:38] <rharper> that'd be cloud-init-daily-debian
[21:38] <rharper> which is what I think would be desired
[21:40] <ahosmanMSFT> Would that also have the debian salsa packages or is it jsut a cloud-init met to work on debian
[21:43] <ahosmanMSFT> I think I see what you mean rharper, If we merge latest cloudinit and debian upstream changes we can get the best of both
[21:43] <ahosmanMSFT> That might just work
[21:43] <blackboxsw> thanks rharper fixed docstr https://github.com/canonical/cloud-init/pull/230
[21:43] <rharper> blackboxsw: ok
[21:43] <blackboxsw> also closed 221 as that was my first cut of 230 that we didn't merge last week
[21:45] <rharper> blackboxsw: approved, I'll leave to use to wait on CI and squash/merge
[21:49] <blackboxsw> thanks
[21:50] <blackboxsw> rharper: I forgot what did you suggest for top-level instance-data key for merged config instead of 'ci_cfg'
[21:51] <rharper> we discused merged_cfg or merged_config
[21:51] <rharper> something indicating it was combined
[21:51] <blackboxsw> ok will go with merged_cfg
[21:51]  * blackboxsw saving those keystrokes one step at a time