[04:02] zeestrat: hi, I'm John from the bug list, and just wanted to reach out about your upgrade issues. [05:54] jam: hey John. Thanks for reaching out. How can I help. [06:31] zeestrat: hi. If you can describe the steps you do to reproduce it, ideally we'd get a local reproduction and then be able to fold it into our ongoing testing infrastructure [06:39] jam: all the steps are in the bug besides the bundle and maas setup. You want those too? === frankban|afk is now known as frankban [13:59] Hey folks, does juju support upgrading straight from 2.1.2 to 2.3.2 by setting --agent-version, or does it only support step-wise upgrades such as 2.1.2 -> 2.2.x -> 2.3.x? [14:01] zeestrat: go in one jump [14:05] rick_h: Thanks. Not working too great here in staging on MAAS and LXD when upgrading HA controllers. Let me do my homework and make sure I can reproduce it before I bother y'all more. [14:05] zeestrat: ok, yea let us know. I know we agreed long ago that going step by step was madness [14:08] rick_h: Cool. Nice opportunity to test out that snippet of yours to build 2.1.2 :) [14:09] zeestrat: :) but also might try that snap route cory_fu mentioned at the end there [14:09] might be fewer steps but haven't tried it yet [14:10] Yeah, that sounded very nice. [16:42] hml expected to be around today? [16:48] hml o/ [16:48] thanks for your help yesterday [16:49] ryebot: so it worked? [16:49] hml partly! 2.3.1 controller was part of my problem [16:49] I can get packages to work now :) [16:49] here's what I'm hitting now: https://gist.github.com/wwwtyro/6afde641c5c3db038b6b28106345251a [16:49] ryebot: i’ll have to double check and maybe adjust some data, i thought that all of the code was in 2.3.1 [16:51] hml: sounds good; packages was just a test for me though, if that makes a difference. What I'm really driving towards is ca-certs. [16:51] ryebot: hmm… [16:51] ryebot: can you ssh to the machine? [16:51] hml the one stuck in pending? [16:52] hml or the controller? [16:52] ryebot: brain fart, sorry, i doubt it’s up. [16:52] np [16:52] ryebot: let me look at something. [16:52] hml great, thanks! [16:53] ryebot: i had a heck of a time getting the darn yaml files correct when working on this. :-/ [16:53] hml: yeah, I'm hoping I'm just getting the yaml wrong. it does seem pretty persnickety. [16:57] ryebot: trying to reproduce, you’re at juju 2.3.2 and the new machines are using xenial correct? [16:57] hml: yes [17:02] ryebot: I was able to reproduce, now to find out why :-) [17:03] hml: awesome thankyou. if it's the yaml, let's fork juju and do everything in http://haml.info :D [17:03] ha [17:17] ryebot: part of the challenge here, is that we’re reading the yaml is as a string, then converting to yaml later. [17:18] alright [17:21] hml: if it helps, here's one way I validated that yaml: https://gist.github.com/wwwtyro/38a5d94179fa31fd2c54a54f652260c2 [17:22] ryebot: I think I have the config.yaml correct… but something isn’t working as expected [17:23] hml: I hear you. really glad I don't have to deal with yaml and a static language! [17:23] * ryebot shivers [17:31] ryebot: small progress, https://paste.ubuntu.com/26542164/, this at least gets by the EOF failure… but the write_files doesn’t happen… not sure about the ca-certs, though both end up in ls /var/lib/cloud/instance/user-data.txt.i [17:31] \o/ [17:31] ryebot: the two packages do get installed, so half battle perhaps? [17:31] Dmitrii-Sh: ping [17:32] hml: 1 [17:32] hml yeah this looks promising :) [17:33] Dmitrii-Sh: have you had good luck formating the config.yaml for cloudinit-userdata? ryebot is having fun trying to get ca-certs formatted for that model-config var [17:34] cat userdata.yaml [17:34] cloudinit-userdata: | [17:34] apt_proxy: http://10.10.10.88:3128 [17:34] juju model-config userdata.yaml [17:34] it works like that [17:34] as a PoC [17:34] haven't tried certs though [17:34] Dmitrii-Sh: we’ve gotten a few easy things working, but ca-certs were giving and EOF [17:34] and wouldn’t provision [17:35] hmm, let me check some yaml example I had for bundles [17:35] https://gist.github.com/dshcherb/a5f8dd27af74b38d483c1715994caed3 [17:36] Dmitrii-Sh: i think the challenge is formatting the yaml such that juju can read as a string, then, spit back out as yaml [17:36] hml: try the install-keys example from the gist above === frankban is now known as frankban|afk [17:41] Dmitrii-Sh: that doesn’t work, [17:41] hml: hmm [17:42] hml: could also try [17:42] https://www.irccloud.com/pastebin/Rf96y0FC/ [17:42] Dmitrii-Sh: the contents of cloudinit-userdata isn’t real yaml to juju, it’s a string…. [17:42] hml: this should be treated as a multi-line string [17:43] well, at least by a yaml parser [17:43] Dmitrii-Sh: that’s the problem, it’s not seen by a yaml parser first [17:43] Dmitrii-Sh: I got it to work by adding a | to “variables: |” [17:44] Dmitrii-Sh: where can i verify the installed key on the new machine? [17:44] hml: just a CA cert or a gpg key? [17:45] gpg key? [17:46] for packages [17:46] I think you are talking about ca certs though [17:46] I'll create a gist [17:46] what ever was done by adding the variable: thing :-) [17:47] ah, wait [17:47] hml: let me give you a better example [17:47] those were not for cloud-init - just yaml examples with complex content [17:49] hml: https://gist.github.com/dshcherb/c03d080f9f721d16050a65210f3a05e2 - first, this [17:49] hml: /etc/ssl/certs mainly [17:49] Dmitrii-Sh: k - thx [17:50] hml: http://cloudinit.readthedocs.io/en/latest/topics/examples.html#configure-an-instances-trusted-ca-certificates [17:50] and this is for ca certs [17:51] hml: another example is pgp keys for apt packages: http://cloudinit.readthedocs.io/en/latest/topics/modules.html http://cloudinit.readthedocs.io/en/latest/topics/examples.html?highlight=pgp#install-and-run-chef-recipes [17:51] hml I'm starting to suspect some copy paste character encoding mistake on my part [17:52] I have what appears to be two identical bodies of text, one works and one doesn't [17:52] still investigating [17:52] ryebot: it’s tricky [17:52] ryebot: we may be able to make it simplier in the near future [17:55] hml, ryebot: I think it's safe to assume that we need to treat this option as yaml, parse it, do a transformation for pre- and post- and then render that as yaml again [17:55] that way we can also validate that this is valid yaml [17:56] hml ah, here's the diff between yours and mine: line 10 from your paste has a | that maybe shouldn't be there? [17:56] ryebot: i think it has to be… [17:56] ryebot: but try without. [17:57] Dmitrii-Sh: it’s yaml, but that yaml must be read as a string…. no so straight forward [17:57] hml I did, which is when I get the eof - wait, are you saying ca-certs is also a string that is converted to yaml? [17:57] ahhhh [17:57] sorry, a little slow on the uptake there [17:57] okay, I think I finally get this :) [17:57] ryebot: everything after “cloudinit-userdata:” Is read as a string, that we parse to yaml [17:58] it’s confusing, [17:58] so some yaml works, some doesn’t depending on whether juju can’t read it all as 1 string [17:58] a pita [17:58] hml: ah, right. So if you have config.yaml that option is then nested yaml [17:59] read config.yaml, parse it, read that option, parse it as yaml [17:59] except for some, like packages, which is processed elsewhere? [17:59] Dmitrii-Sh: yes - though everything after the key is read as a string…. not yaml, it’s a string that’s parsed to yaml later [18:00] hml: I think I used it somewhere like that [18:00] well, in LXD [18:00] for user-data it's the same problem [18:00] ryebot: not sure what you’re asking [18:01] hml for example in your paste, packages is presented as a raw list (it's not packages: |) [18:01] here's an example of nested yaml, albeit simple https://dshcherb.github.io/2017/12/04/qemu-kvm-virtual-machines-in-unprivileged-lxd.html#breaking-ground [18:02] user.user-data option [18:02] ryebot: i’m not sure why it works, but it does… from packages to the end of the file is read as 1 string [18:02] ryebot: the glories of yaml perhaps? :-) [18:02] hahaha [18:03] alright, I'm gonna try again with this new information - thanks! [18:07] hml: https://paste.ubuntu.com/26542336/ line 23, gets passed in as a string and not yaml [18:09] results in traceback on line 630 here: https://paste.ubuntu.com/26542344/ [18:09] * ryebot snaps fingers [18:19] ryebot: there’s a trick to using the pipe, i’m just not sure of the rules [19:56] hml: I think I'm going to throw in the towel for now. I've opened a bug here: https://bugs.launchpad.net/juju/+bug/1748283 [19:56] Bug #1748283: Unable to pass ca-certs field in cloudinit-userdata model config [19:56] hml: thanks for looking at that with me! [19:57] ryebot: k [20:01] morning [20:02] hml: wanna chat about that? [20:02] hml: there is a function we probably need to use :) [20:06] thumper: sure [20:06] hml: 1:1 HO [20:06] thumper: omw [20:55] rick_h: here's that upgrade issue in case you'd like to give it a spin yourself. #1748294 [20:55] Bug #1748294: juju-upgrade from 2.1.2 to 2.3.2 fails [21:30] thumper: you called it, ConformYAML did it - and ca-certs were good on the other end too . TY. :-) [21:35] sweet [22:44] small PR review anyone? https://github.com/juju/juju/pull/8354