[06:42] <cxdeviit> How can I update cloud-init in AWS?
[08:58] <meena> https://github.com/canonical/cloud-init/pull/42#discussion_r353283759
[08:58] <meena> duh.
[11:23] <meena> i've started a patch here: https://github.com/canonical/cloud-init/pull/621 hope i can get it into the next release… along with my /other patch/
[11:26] <meena> this needs a better title than the title of the first commit
[13:22] <Odd_Bloke> cxdeviit: cloud-init generally comes from your distro vendor as part of the images you launch, and is generally updated via standard upgrade mechanisms or via new distro images being generated.  What distro are you using, and what's the motivation for needing cloud-init upgraded?
[14:17] <cxdeviit> One of my AWS instance has Amazon Linux 1 and Cloud-init version 0.7.6 which doesn't seem to support devel analyze etc , that's where I am stuck. If there was a way to patch it .
[14:23] <Odd_Bloke> cxdeviit: Aha, right.  I would be hesitant to recommend upgrading cloud-init in any way other than distro packages: that could break your machine in ways that are difficult to recover from.  So I don't know if you have a good path to a newer cloud-init.  That said, if you're looking to use `analyze`, it has a `--infile` parameter, so you could pull your log files to a newer cloud-init installation
[14:23] <Odd_Bloke> (locally or in a different distro instance in the cloud) and perform your analysis that way.
[14:25] <cxdeviit> Aha, will try that. Another issue is regarding requests dependency of urllib3 to be 1.21.1 , not sure if pip/python is messing here
[14:33] <minimal> Odd_Bloke: Hi. Re: Mac Addreses, as Netplan files are YAML also then they should also have all-numeric MAC addresses quoted, right?
[14:41] <Odd_Bloke> minimal: Presumably, yeah.
[14:42] <minimal> so then I've found a bug in the netplan renderer - when I fixed an all-numeric MAC address in a testcase then it failed due to the macaddress in the generated netconf file having no quotes
[14:43] <Odd_Bloke> Can you pastebin the error?
[14:53] <minimal> Odd_Bloke: https://gist.github.com/dermotbradley/bb7353c00bf505bbf2f1ce0eb2b56c17
[14:54] <minimal> its the 5th MAC address
[15:13] <Odd_Bloke> minimal: That's test_render_output_supports_both_grat_arp_spelling?  (I'm heading into a meeting, will look more after.)
[15:14] <minimal> Odd_Bloke: yupe, that's the one
[15:54] <meena> smoser:  that sure is one way to say, focus on  the other PR
[15:54] <smoser> meena: ? sorry... didnt' mean to be a pita.
[16:00] <smoser> oh. funny.
[16:00] <smoser> wow. total fail
[16:01] <smoser> meena: updated that comment https://aci-github.cisco.com/atom/atomix/pull/1269
[16:04] <meena> smoser, nah, it's cool. what you're doing is useful and important
[16:15] <meena> smoser, I'm getting address not found for that link
[16:15] <smoser> man. i totally cannot paste
[16:16] <smoser> https://github.com/canonical/cloud-init/pull/622
[16:16] <smoser> yeah, that other is my iternal github
[16:32] <Odd_Bloke> minimal: So I think that MAC address _doesn't_ need quoting because it has values that exclude it from being a timestamp (99 > 59, in the seconds and hour fields): https://paste.ubuntu.com/p/tddYNXWrbk/
[16:37] <minimal> Odd_Bloke: hmm, the YAML "timestamp" is actually whether something is treated as a base 60 number and according to https://yaml.org/type/int.html the definition for this is |[-+]?[1-9][0-9_]*(:[0-5]?[0-9])+
[16:40] <minimal> so you're right, I'll have to rewrite my PR docs change to be even more specific about when quoting is required ;-)
[16:57] <Odd_Bloke> minimal: OK, right, I was trying to figure out why I couldn't find reference to it in the spec; didn't occur to me that it was a base 60 number!  Good find, and I agree that more specificity is needed.
[16:58] <Odd_Bloke> (At least we know that anyone running cloud-init in ancient Sumeria will experience no problems. :p)
[17:05] <minimal> Odd_Bloke: search YAML docs to find that ref gave me a headache, lol
[17:25] <momousta> Hi, we would like to land this PR (https://github.com/canonical/cloud-init/pull/614) before the code cutoff for the next SRU, Can someone help taking a look at it?
[17:34] <minimal> Odd_Bloke: PR created for the MAC address docs. Any chance you could review it and PR 618 as well?
[17:35] <minimal> for PR 618 the question of how to test it is still open - /etc/shadow is normally only readable by root user, so guess it would need to be mocked
[18:48] <meena> wait, wait wait
[18:48] <meena> YAML, which is a superset of JSON knows integers, but JSON does not
[18:49] <meena> that's hilarious, i didn't realise how lossy the conversion between the two could be
[18:56] <minimal> meena: with JSON strings are quoted so there's no room for confusion, with YAML you don't have to quote (most) strings and magically the app reading the YAML works out if a value is a number or a string...
[18:58] <minimal> except in this case in YAML a MAC value of 11:22:33:44:55:66 (without quotes) will be treated as a Base 60 number whereas a value of 11:22:33:44:55:6a (without quotes) will be treated as a string
[19:07] <minimal> does anyone on here from Canonical look after cloud-utils? Am confused that https://launchpad.net/cloud-utils has 0.31 dated 10/12/2018 as latest release but that https://github.com/canonical/cloud-utils/ has no 0.31 and has 0.30 dated 10/07/2019 (i.e. dated far newer than 0.31 on the other site)
[19:09] <Odd_Bloke> minimal: It's getting even better: the YAML 1.2 spec doesn't include base 60 (which is why I couldn't find it!) but PyYAML only supports YAML 1.1 which _does_.
[19:10] <minimal> minimal: any cloud-init's network config uses with YAML schema? (Also didn't notice mention of YAML version on netplan's website)
[19:10] <minimal> s/minimal/Odd_Bloke/
[19:11] <Odd_Bloke> We (implicitly) use YAML 1.1, because we use PyYAML.
[19:12] <minimal> Odd_Bloke: any idea on what I pointed out about cloud-utils above?
[19:13] <meena> minimal, yeah, we've seen this issue in puppet / hiera
[19:13] <Odd_Bloke> minimal: I believe the tag in GitHub was created on that much later date, but if you look at the commit it points to it's older.
[19:14] <Odd_Bloke> (In fact, there are a whole bunch of version tags that were created on that same day, so that was presumably an import to GH?)
[19:15] <minimal> ok. I maintain the Alpine package for cloud-utils and 0.31 is downloaded from Launchpad........just wondered if I was using out of date code. Any idea if a 0.32 is going to happen any time soon? The last release has been out a while
[19:17] <Odd_Bloke> I know some cloud-utils things have been happening, but I haven't kept on top of them.
[19:19] <minimal> for example in "master" there a growpart LVM commit from July that's obviously significant for cloud-init users
[19:23] <Odd_Bloke> minimal: I'll look into it.
[19:24] <minimal> Thanks
[19:24] <powersj> heh smoser released 0.31 end of 2018 when he moved it to git
[19:25] <minimal> powersj: yeah found the 0.31 tag hidden on the 2nd (or 3rd?) Releases page in git. Have just downloaded from both Launchpad and Github and compared to verify both 0.31 files are identical
[19:26]  * meena quotes herself quoting someone else she heard saying it: release early release often
[19:26] <meena> also: why y'all afraid of 1.0
[19:26] <meena> what's up? why the 0.?
[19:26] <minimal> meena: as its a prime number? They're scary!
[19:35] <meena> you can skip primes
[19:40] <Odd_Bloke> Every number is prime if you define division the right way: https://paste.ubuntu.com/p/rcHw2M5pVf/
[19:43] <meena> Odd_Bloke: na na na. that means we're then skipping all numbers, which brings us back to where we started
[19:47] <minimal> Odd_Bloke: for the Ansible/Jinja2 templates I'm using to create network-config-v2.yaml adding "| to_yaml" to the macaddress variables sorts the automagic YAML string quoting for me
[19:57] <Odd_Bloke> OK, good, that's what I'm hoping will be the case for most generated YAML: the libraries will DTRT.  And most networking YAML will be generated in some way, because it's handled by the platform (which launches many instances) rather than the user (who may only launch a few and not bother investing in automation).
[19:57] <Odd_Bloke> That said, I wonder if YAML 1.2 libraries do the correct quoting for values which are strings in 1.2 but ints in 1.1.
[19:58] <minimal> To be honest I think YAML is trying to be too smart - they should have just required strings to always be quoted and been done with it
[19:59] <Odd_Bloke> Agreed.
[20:02] <meena> wait_for_physdevs_netcfg() is a terrible name for a function that's no longer in that class
[20:02] <meena> but it looks generally useful
[20:05] <minimal> Odd_Bloke: your command about ".. include:", does this support including from another part of the same document? or only from a separate file?
[20:05] <minimal> s/command/comment/
[20:06] <Odd_Bloke> minimal: I think we'd need a separate file with just that snippet.
[20:06] <Odd_Bloke> Which is perhaps overkill, but I also wouldn't be surprised if we find this info needs updating again in future.
[20:06] <Odd_Bloke> So I could go either way.
[20:10] <Odd_Bloke> I come bearing bad news: ruamel.yaml supports YAML 1.2, and `yaml.load(ruamel.yaml.dump({"x": "11:22:33:44:55:59"}))` gives `{'x': 8846009759}` (whereas dumping with PyYAML round-trips correctly).
[20:14] <minimal> where is ruamel.yaml used? A quick grep doesn't show in anywhere in cloud-init
[20:14] <Odd_Bloke> It isn't, but it's a library that could be used to generate the networking YAML that is fed to cloud-init.
[20:15] <Odd_Bloke> And as it is a YAML 1.2 library, it does not quote base60 ints properly (because in YAML 1.2 base60 ints aren't a thing, so their unquoted form is unambiguously a string).
[20:16] <Odd_Bloke> This is in reference to "That said, I wonder if YAML 1.2 libraries do the correct quoting for values which are strings in 1.2 but ints in 1.1."
[20:16] <minimal> It describes itself as a YAML 1.2 loader/dumper, whereas you'd indicated earlier that network-config-v1.yaml and network-config-v2.yaml are YAML 1.1
[20:17] <minimal> though I guess that's just because c-i uses PyYAML and not stated anywhere in the c-i "specification" for those YAML files
[20:18] <Odd_Bloke> Yeah, that's the gap I'm worried about.
[20:18] <Odd_Bloke> Until today, I hadn't really thought too much about YAML 1.1 vs 1.2, and I strongly suspect I'm not alone in that.
[20:18] <minimal> same applies to Netplan - did'nt see any YAML version mentioned there either
[20:20] <minimal> now you'll have the fun of determining what each of the DataSources vendors (AWS, GCE, etc) do and try and get them to agree on YAML 1.1, doh!
[20:22] <Odd_Bloke> Practically speaking, if any of the large platforms could generate incorrect YAML in this way, we'd have heard about it, I think: this isn't a recently introduced "bug", we've always used PyYAML and therefore YAML 1.1.
[20:22] <Odd_Bloke> (And they obviously get through a lot of MAC addresses. :p)
[20:24] <minimal> Time to hurriedly insert "These files are in YAML 1.1 format" into the docs in big flashing letters eh? ;-) (damn, Mozilla decommissioned the HTML blink tag)
[20:24] <Odd_Bloke> I'm sure we can bring it back with some custom JS.
[20:25] <Odd_Bloke> But yeah, I think we do want to document this.
[20:25] <minimal> I'm guessing a all-numeric MAC address where all of the component part is 00-59 is statistically unusual
[20:26] <Odd_Bloke> Yeah, and it is possible they might be using a prefix (that's a MAC address thing, right, vendor prefixes?) that excludes all of their MACs from triggering this.
[20:32] <Odd_Bloke> minimal: https://bugs.launchpad.net/netplan/+bug/1901084
[20:32] <ubot5> Ubuntu bug 1901084 in netplan "netplan should specify the version of YAML that it takes as input" [Undecided,New]
[20:35] <minimal> I'd already noticed that the page https://netplan.io/reference/ has at least 1 unquoted MAC address that needs to be quoted
[20:36] <minimal> Odd_Bloke: If you search online for OUI each hardware vendor that registers get a prefix to use for network cards they manufacture. Likewise a MAC beginning "52:54" is KVM (from memory). There's a OUI.txt file online managed by IETF that is regularily updated
[20:39] <minimal> in my case as I'm testing on KVM I'm using made up MACs within the KVM prefix and happened to use all numeric addresses.
[20:40] <minimal> The other PR I raised a few days ago about user-data YAML for user settings - I'd noticed that expirydate was being misinterpreted without quotes and what was being put into the field in /etc/shadow was incorrect
[20:41] <minimal> so potentially the impact is on more than just MAC addresses
[20:58] <meena> https://github.com/canonical/cloud-init/pull/617#issuecomment-714758289 ds-identify is a trash fire
[21:08] <meena> maybe reading disk labels doesn't work for bsd
[22:05] <minimal> Odd_Bloke: FYI https://en.wikipedia.org/wiki/Sexagesimal#YAML