[04:02] <jam> zeestrat: hi, I'm John from the bug list, and just wanted to reach out about your upgrade issues.
[05:54] <zeestrat> jam: hey John. Thanks for reaching out. How can I help.
[06:31] <jam> zeestrat: hi. If you can describe the steps you do to reproduce it, ideally we'd get a local reproduction and then be able to fold it into our ongoing testing infrastructure
[06:39] <zeestrat> jam: all the steps are in the bug besides the bundle and maas setup. You want those too?
[13:59] <zeestrat> Hey folks, does juju support upgrading straight from 2.1.2 to 2.3.2 by setting --agent-version, or does it only support step-wise upgrades such as 2.1.2 -> 2.2.x -> 2.3.x?
[14:01] <rick_h> zeestrat: go in one jump
[14:05] <zeestrat> rick_h: Thanks. Not working too great here in staging on MAAS and LXD when upgrading HA controllers. Let me do my homework and make sure I can reproduce it before I bother y'all more.
[14:05] <rick_h> zeestrat: ok, yea let us know. I know we agreed long ago that going step by step was madness
[14:08] <zeestrat> rick_h: Cool. Nice opportunity to test out that snippet of yours to build 2.1.2 :)
[14:09] <rick_h> zeestrat: :) but also might try that snap route cory_fu mentioned at the end there
[14:09] <rick_h> might be fewer steps but haven't tried it yet
[14:10] <zeestrat> Yeah, that sounded very nice.
[16:42] <ryebot> hml expected to be around today?
[16:48] <ryebot> hml o/
[16:48] <ryebot> thanks for your help yesterday
[16:49] <hml> ryebot: so it worked?
[16:49] <ryebot> hml partly! 2.3.1 controller was part of my problem
[16:49] <ryebot> I can get packages to work now :)
[16:49] <ryebot> here's what I'm hitting now: https://gist.github.com/wwwtyro/6afde641c5c3db038b6b28106345251a
[16:49] <hml> ryebot: i’ll have to double check and maybe adjust some data, i thought that all of the code was in 2.3.1
[16:51] <ryebot> hml: sounds good; packages was just a test for me though, if that makes a difference. What I'm really driving towards is ca-certs.
[16:51] <hml> ryebot: hmm…
[16:51] <hml> ryebot: can you ssh to the machine?
[16:51] <ryebot> hml the one stuck in pending?
[16:52] <ryebot> hml or the controller?
[16:52] <hml> ryebot: brain fart, sorry, i doubt it’s up.
[16:52] <ryebot> np
[16:52] <hml> ryebot: let me look at something.
[16:52] <ryebot> hml great, thanks!
[16:53] <hml> ryebot: i had a heck of a time getting the darn yaml files correct when working on this.  :-/
[16:53] <ryebot> hml: yeah, I'm hoping I'm just getting the yaml wrong. it does seem pretty persnickety.
[16:57] <hml> ryebot: trying to reproduce, you’re at juju 2.3.2 and the new machines are using xenial correct?
[16:57] <ryebot> hml: yes
[17:02] <hml> ryebot: I was able to reproduce, now to find out why :-)
[17:03] <ryebot> hml: awesome thankyou. if it's the yaml, let's fork juju and do everything in http://haml.info :D
[17:03] <hml> ha
[17:17] <hml> ryebot: part of the challenge here, is that we’re reading the yaml is as a string, then converting to yaml later.
[17:18] <ryebot> alright
[17:21] <ryebot> hml: if it helps, here's one way I validated that yaml: https://gist.github.com/wwwtyro/38a5d94179fa31fd2c54a54f652260c2
[17:22] <hml> ryebot: I think I have the config.yaml correct… but something isn’t working as expected
[17:23] <ryebot> hml: I hear you. really glad I don't have to deal with yaml and a static language!
[17:23]  * ryebot shivers
[17:31] <hml> ryebot: small progress, https://paste.ubuntu.com/26542164/, this at least gets by the EOF failure… but the write_files doesn’t happen… not sure about the ca-certs, though both end up in ls /var/lib/cloud/instance/user-data.txt.i
[17:31] <ryebot> \o/
[17:31] <hml> ryebot: the two packages do get installed, so half battle perhaps?
[17:31] <hml> Dmitrii-Sh: ping
[17:32] <Dmitrii-Sh> hml: 1
[17:32] <ryebot> hml yeah this looks promising :)
[17:33] <hml> Dmitrii-Sh: have you had good luck formating the config.yaml for cloudinit-userdata?  ryebot  is having fun trying to get ca-certs formatted for that model-config var
[17:34] <Dmitrii-Sh> cat userdata.yaml
[17:34] <Dmitrii-Sh> cloudinit-userdata: |
[17:34] <Dmitrii-Sh>   apt_proxy: http://10.10.10.88:3128
[17:34] <Dmitrii-Sh> juju model-config userdata.yaml
[17:34] <Dmitrii-Sh> it works like that
[17:34] <Dmitrii-Sh> as a PoC
[17:34] <Dmitrii-Sh> haven't tried certs though
[17:34] <hml> Dmitrii-Sh: we’ve gotten a few easy things working, but ca-certs were giving and EOF
[17:34] <hml> and wouldn’t provision
[17:35] <Dmitrii-Sh> hmm, let me check some yaml example I had for bundles
[17:35] <Dmitrii-Sh> https://gist.github.com/dshcherb/a5f8dd27af74b38d483c1715994caed3
[17:36] <hml> Dmitrii-Sh: i think the challenge is formatting the yaml such that juju can read as a string, then, spit back out as yaml
[17:36] <Dmitrii-Sh> hml: try the install-keys example from the gist above
[17:41] <hml> Dmitrii-Sh: that doesn’t work,
[17:41] <Dmitrii-Sh> hml: hmm
[17:42] <Dmitrii-Sh> hml: could also try
[17:42] <Dmitrii-Sh> https://www.irccloud.com/pastebin/Rf96y0FC/
[17:42] <hml> Dmitrii-Sh:  the contents of cloudinit-userdata isn’t real yaml to juju, it’s a string….
[17:42] <Dmitrii-Sh> hml: this should be treated as a multi-line string
[17:43] <Dmitrii-Sh> well, at least by a yaml parser
[17:43] <hml> Dmitrii-Sh: that’s the problem, it’s not seen by a yaml parser first
[17:43] <hml> Dmitrii-Sh: I got it to work by adding a | to “variables: |”
[17:44] <hml> Dmitrii-Sh: where can i verify the installed key on the new machine?
[17:44] <Dmitrii-Sh> hml: just a CA cert or a gpg key?
[17:45] <hml> gpg key?
[17:46] <Dmitrii-Sh> for packages
[17:46] <Dmitrii-Sh> I think you are talking about ca certs though
[17:46] <Dmitrii-Sh> I'll create a gist
[17:46] <hml> what ever was done by adding the variable: thing   :-)
[17:47] <Dmitrii-Sh> ah, wait
[17:47] <Dmitrii-Sh> hml: let me give you a better example
[17:47] <Dmitrii-Sh> those were not for cloud-init - just yaml examples with complex content
[17:49] <Dmitrii-Sh> hml: https://gist.github.com/dshcherb/c03d080f9f721d16050a65210f3a05e2 - first, this
[17:49] <Dmitrii-Sh> hml: /etc/ssl/certs mainly
[17:49] <hml> Dmitrii-Sh: k - thx
[17:50] <Dmitrii-Sh> hml: http://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates
[17:50] <Dmitrii-Sh> and this is for ca certs
[17:51] <Dmitrii-Sh> hml: another example is pgp keys for apt packages: http://cloudinit.readthedocs.io/en/latest/topics/modules.html http://cloudinit.readthedocs.io/en/latest/topics/examples.html?highlight=pgp#install-and-run-chef-recipes
[17:51] <ryebot> hml I'm starting to suspect some copy paste character encoding mistake on my part
[17:52] <ryebot> I have what appears to be two identical bodies of text, one works and one doesn't
[17:52] <ryebot> still investigating
[17:52] <hml> ryebot: it’s tricky
[17:52] <hml> ryebot: we may be able to make it simplier in the near future
[17:55] <Dmitrii-Sh> hml, ryebot: I think it's safe to assume that we need to treat this option as yaml, parse it, do a transformation for pre- and post- and then render that as yaml again
[17:55] <Dmitrii-Sh> that way we can also validate that this is valid yaml
[17:56] <ryebot> hml ah, here's the diff between yours and mine: line 10 from your paste has a | that maybe shouldn't be there?
[17:56] <hml> ryebot: i think it has to be…
[17:56] <hml> ryebot: but try without.
[17:57] <hml> Dmitrii-Sh: it’s yaml, but that yaml must be read as a string…. no so straight forward
[17:57] <ryebot> hml I did, which is when I get the eof - wait, are you saying ca-certs is also a string that is converted to yaml?
[17:57] <ryebot> ahhhh
[17:57] <ryebot> sorry, a little slow on the uptake there
[17:57] <ryebot> okay, I think I finally get this :)
[17:57] <hml> ryebot: everything after “cloudinit-userdata:” Is read as a string, that we parse to yaml
[17:58] <hml> it’s confusing,
[17:58] <hml> so some yaml works, some doesn’t depending on whether juju can’t read it all as 1 string
[17:58] <hml> a pita
[17:58] <Dmitrii-Sh> hml: ah, right. So if you have config.yaml that option is then nested yaml
[17:59] <Dmitrii-Sh> read config.yaml, parse it, read that option, parse it as yaml
[17:59] <ryebot> except for some, like packages, which is processed elsewhere?
[17:59] <hml> Dmitrii-Sh: yes - though everything after the key is read as a string…. not yaml, it’s a string that’s parsed to yaml later
[18:00] <Dmitrii-Sh> hml: I think I used it somewhere like that
[18:00] <Dmitrii-Sh> well, in LXD
[18:00] <Dmitrii-Sh> for user-data it's the same problem
[18:00] <hml> ryebot: not sure what you’re asking
[18:01] <ryebot> hml for example in your paste, packages is presented as a raw list (it's not packages: |)
[18:01] <Dmitrii-Sh> here's an example of nested yaml, albeit simple https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html#breaking-ground
[18:02] <Dmitrii-Sh> user.user-data option
[18:02] <hml> ryebot: i’m not sure why it works, but it does… from packages to the end of the file is read as 1 string
[18:02] <hml> ryebot: the glories of yaml perhaps?  :-)
[18:02] <ryebot> hahaha
[18:03] <ryebot> alright, I'm gonna try again with this new information - thanks!
[18:07] <ryebot> hml: https://paste.ubuntu.com/26542336/ line 23, gets passed in as a string and not yaml
[18:09] <ryebot> results in traceback on line 630 here: https://paste.ubuntu.com/26542344/
[18:09]  * ryebot snaps fingers
[18:19] <hml> ryebot: there’s a trick to using the pipe, i’m just not sure of the rules
[19:56] <ryebot> hml: I think I'm going to throw in the towel for now. I've opened a bug here: https://bugs.launchpad.net/juju/+bug/1748283
[19:56] <mup> Bug #1748283: Unable to pass ca-certs field in cloudinit-userdata model config <juju:New> <https://launchpad.net/bugs/1748283>
[19:56] <ryebot> hml: thanks for looking at that with me!
[19:57] <hml> ryebot: k
[20:01] <thumper> morning
[20:02] <thumper> hml: wanna chat about that?
[20:02] <thumper> hml: there is a function we probably need to use :)
[20:06] <hml> thumper: sure
[20:06] <thumper> hml: 1:1 HO
[20:06] <hml> thumper: omw
[20:55] <zeestrat> rick_h: here's that upgrade issue in case you'd like to give it a spin yourself. #1748294
[20:55] <mup> Bug #1748294: juju-upgrade from 2.1.2 to 2.3.2 fails <juju:New> <https://launchpad.net/bugs/1748294>
[21:30] <hml> thumper: you called it, ConformYAML did it - and ca-certs were good on the other end too .  TY.  :-)
[21:35] <thumper> sweet
[22:44] <hml> small PR review anyone?  https://github.com/juju/juju/pull/8354