=== blackboxsw is now known as blackboxsw_away [01:26] cherylj: got bootstrap to work, hitting another issue trying to deploy an instance https://bugs.launchpad.net/juju-core/+bug/1595360 [01:26] Bug #1595360: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[01:27] jhobbs: try restarting jujud on the controller [01:27] the "lease manager stopped" error has been seen recently and usually a restart of jujud brings things back [01:33] Bug #1595360 opened: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[01:38] cherylj: that made the lease manager stopped errors go away, but it's still failing. It looks like it's trying to go back to the internet for the image stream instead of using the image-metadata-url i bootstrapped with [01:39] cherylj: do i have to set the image-metadata-url for non bootstrap stuff some other way? === redir_afk is now known as redir [01:42] jhobbs: it may be falling back because it can't access the streams from the client [01:42] and by client I mean machine [01:42] Bug #1595360 changed: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[01:42] let me look at the output again, one sec [01:45] Bug #1595360 opened: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[01:52] jhobbs: can you attach /var/log/cloud-init-output.log for one of the machines that's in "waiting for agent initialization"? [01:54] cherylj: there is no machine [01:54] bleh, sorry [01:56] anastasiamac: would you be able to help jhobbs with bug 1595360? I need to eod [01:56] Bug #1595360: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[01:57] cherylj: jhobbs: sure.. gimme 30 mins to catch up :) [01:57] thanks, anastasiamac! [02:00] cherylj: anastasiamac: thanks, i need to EOD too, if you comment on the bug i will pick it up tomorrow [02:02] jhobbs: will do \o/ [02:23] jhobbs: is there any chance, you could also attach the other file from machine-0 log directory? [02:23] jhobbs: and if u get a chance, could you pleas also send `juju status --format=yaml` output? [02:25] anastasiamac: http://paste.ubuntu.com/17727508/ [02:27] * anastasiamac wondering how jhobbs got machine-0 log if there are no machines according to status :D [02:28] jhobbs: have u been able to bootstrap juju 1.25 on the same cloud? [02:32] anastasiamac: ah well that's for the model i added, here is the juju status for the controller model http://paste.ubuntu.com/17727730/ [02:32] anastasiamac: i did bootstrap with 1.25 (and 2.0 now) but I didn't try deploying anything after bootstrapping on 1.25 [02:34] oO do i need to set image-metadata-url as config on the model i added maybe? [02:35] jhobbs: i think it would help [02:36] jhobbs: atm m not seeing images being picked at all and we are falling throught to trying to pull images from streams-canonical and getting gpg error - signature differences.. [02:36] can't juju set-model-config image-metadata-url, get "WARNING key "image-metadata-url" is not defined in the current model configuration: possible misspelling [02:38] hmm [02:38] jhobbs: m not sure that it will help on already created model.. I would have thought it makes a difference at bootstrap.. [02:39] anastasiamac: it did make a difference at bootstrap - i wasn't able to bootstrap without passing that as a config option [02:39] it's like it's forgotten about it after bootstrap though [02:39] jhobbs: it shouldn't have, these images (metadata) would have been put into controller db [02:40] jhobbs: is there another file in logs directory that u could attach to the bug? [02:40] jhobbs: besides the machine-0 one [02:40] ? [02:40] logsink.log [02:40] yep [02:42] anastasiamac: https://bugs.launchpad.net/juju-core/+bug/1595360/+attachment/4688820/+files/logsink.log [02:42] Bug #1595360: 2.0beta9.1: Can't deploy an instance after bootstrapping against OpenStack

[02:42] jhobbs: is there anyway, u could do a db dump before u eod so that I can have a look at what u have there? [02:42] jhobbs: tyvm for logsink :) [02:43] anastasiamac: yeah, how do i do a db dump? [02:44] jhobbs: gimme a sec, I'll dig it up :) [02:45] jhobbs: I'll email u details :) [02:45] anastasiamac: k [03:45] wallyworld: looks like my neighbours are getting their power line sunk today, so if I go AWOL it's because I've got no power [03:46] no worries [04:04] fuckity fuckity fuck fuck [04:04] * thumper sighs [04:26] axw: the current destroy semantics are to nuke the model to which the incoming connection belongs. thus destroy-model -m foo establishes a connection to model foo and kills it. introducing a bulk api sounds ok but does it make sense to allow models to be destoryed over a different connection. for whatever reason, sadly model create is not bulk either :-( [04:27] thumper: do you know why model create api is not bulk? [04:27] reasons [04:27] wallyworld: true, but I think you're more likely to destroy multiple models at once rather than create multiple at once... [04:27] and probably not good ones [04:27] * wallyworld sighs [04:28] wallyworld: doesn't seem like a huge change, but if it is, defer [04:28] axw: let me land it as is, and i'll follow up with a change to look at making create and delete bulk [04:28] at least it will allow the gui to be updated [04:28] wallyworld: thanks [04:29] wallyworld: my upcoming cloud rejiggering changes will conflict with yours I think, so the sooner it lands the better anyway [04:29] yep [04:32] thumper: why were you headesking? [04:33] wallyworld: converting apiserver/params structs with embedded charm.v6-unstable structures to things that don't use those structures [04:33] would have been so much easier if we caught it when they were created [04:33] thumper: should we change the charm.v6 stuff in that repo then [04:33] it is still unstable [04:33] no [04:34] we shouldn't be using external structs [04:34] fullstop [04:34] too easy for things to change without us catching it [04:34] yes true [04:36] it is just boring work [04:36] work we should do [04:36] but boring [04:39] well... it builds now [04:39] * thumper wonders how many tests fail [04:39] bored [04:39] bored [04:39] bored [04:39] * thumper waits [04:39] Boredom is a delicious luxury; use it wisely. [04:40] Bug #1571477 changed: juju 1.25.3: juju-run symlink to tmpdir

[04:40] :) [04:47] * thumper is trashing this laptop's cpu [04:48] load 20, cpu 100% [04:49] engineer reports 105% on the reactor is possible, but not recommended [04:58] wallyworld: http://reviews.vapour.ws/r/5148/ [04:58] looking [04:58] munging types [04:58] mostly trivial but annoying [05:07] thumper: i had one thing to fix and a suggestion or 2 [05:09] k === urulama|__ is now known as urulama [06:43] wallyworld: FYI: http://reviews.vapour.ws/r/5149/ [06:43] no rush I guess, probably need to rebase on your changes [06:44] ok [06:44] we have meeting soon then i have soccer [06:44] wallyworld: I've gotta go get Charlotte soon, will miss the beginning of the meeting [06:46] axw: ok, btw we gotta hide those certs http://paste.ubuntu.com/17734075/ [06:46] yup [06:46] wallyworld: I think we can just add them to SecretAttrs for now, I'll go ahead and do that [06:46] axw: maybe also authorized keys [06:46] not secret but annoying [06:47] we don't need to see those in get-config [06:47] wallyworld: yeah. I kinda think they shouldn't be in config, since they're not managed through config... but not sure [06:47] maybe that will break stuff though === frankban|afk is now known as frankban [08:32] babbageclunk: ping [08:32] poong [08:33] voidspace: ^^ [08:34] babbageclunk: I sent you a PM [08:38] voidspace, babbageclunk: morning :) if you can, please take a look at http://reviews.vapour.ws/r/5134/ [08:40] frobware: ^^ [08:50] dimitern: morning! looking now. [08:50] babbageclunk: thanks! [09:01] dimitern: taking a look [09:02] frobware, jam, fwereade, dooferlad: standup? [09:30] dimitern: LGTM. thanks [09:30] frobware: tyvm [09:41] <_thumper_> night all === _thumper_ is now known as thumper [09:46] axw, http://reviews.vapour.ws/r/5151/ should be trivial if you have a moment [09:55] hello to my former colleagues ;) [10:01] TheMue: o/ ;) [10:04] dimitern: heya, are all our British colleagues voting? I hope they do vote #remain. [10:06] TheMue: yeah, same here! today's a bit quiet, not completely though :) [10:08] dimitern: btw, my try to return to Canonical doesn't look so good. currently extreme few potential matching positions [10:09] TheMue: yeah? :/ well, I'm sure something will come up sooner or later! [10:09] babbageclunk: review poke? :) [10:10] dimitern: I'll keep up looking, but I've already got other interesting contacts. so we'll see. [10:10] TheMue: good luck though! [10:12] dimitern: thanks [10:34] could anyone merge it ? http://reviews.vapour.ws/r/4924/ [10:34] I think now the CI is not locked down http://juju.fail/ [10:47] hoenir: done - it should land soon, I'll keep an eye on it [10:51] dimitern, thanks ! [10:51] np :) [10:54] dimitern: Sorry! Got distracted! Doing it right now. [10:55] babbageclunk: no worries, ta! [11:01] dimitern: done, sorry for the holdup [11:01] babbageclunk: thanks! [11:04] dimitern: just bootstrapping, noticed the gui installation message - does that mean that the gui is available straight away in a newly bootstrapped controller? [11:04] dimitern: How do I get to it? [11:05] babbageclunk: haven't tried, but there is 'juju help gui' you can check [11:06] dimitern: ooh, will try it once this bootstrap finishes. [11:07] babbageclunk: YMMV though, as the bundled gui might still be broken due to thumper's recent API rename changes [11:07] dimitern: oh, of course [11:11] dimitern: Yup, you're right: "unknown object type: Client". Pretty slick up until then though - will be a really nice experience once it works! ;) [11:13] babbageclunk: it works if you switch to a commit earlier than https://github.com/juju/juju/commit/769aeb4b45068389a88f9fef34f3d61915d28562 [11:14] dimitern: thanks - mostly just curiosity after seeing the message scrolling past in a few bootstraps.. [11:31] macgreagoir: cherylj: standup? [11:47] babbageclunk: my headphones arrivedd [11:48] babbageclunk: they're huuuuuuge. Headphone quality and build quality seem reasonable though. [11:48] voidspace: ooh, exciting [11:48] babbageclunk: they're very white [11:52] * babbageclunk goes for a run. === mthaddon` is now known as mthaddon [12:17] fwereade: sorry for the delay, LGTM [12:29] Bug #1595514 opened: destroy-controller fails [12:35] axw, ta [12:59] Bug #1595514 changed: destroy-controller fails [13:01] frobware: sync? [13:09] cherylj: ping? [13:31] dimitern: power just restored [13:31] frobware: we're still in the call btw [13:31] dimitern: omw [13:54] babbageclunk: what up [13:56] cherylj: oh hey - I just wanted to check about the email about blocking master you sent the other day. [13:56] I was surprised no one responded :) [13:56] did you have a specific question? [13:57] cherylj: I've got something that I'd like to get into the rc - is it legit to JFDI it? [13:57] babbageclunk: yes, provided you have done manual testing, run with -race, etc. [13:57] cherylj: (it's the state changes for workload version) [13:57] babbageclunk: yeah, that's fine [13:58] cherylj: ok, great - that's how I was interpreting the email, just checking. [13:58] np, thanks for asking :) [13:58] :) [14:29] Bug #1581157 changed: github.com/juju/juju/cmd/jujud test timeout on windows

[14:29] Bug #1591290 changed: serverSuite.TestStop unexpected error

[14:36] cherylj: whatchoo got for me? [14:36] cherylj: and go easy on me, it's my birthday ;) [14:37] natefinch: in that case, let me see what the worst one is! [14:37] ;) [14:37] hey [14:38] I've shared a model with someone [14:38] but they can't ss [14:38] ssh* [14:38] cherylj: haha... as long as you wait until my next birthday for the next really bad one ;) [14:38] how do I get their keys into the system [14:38] natefinch: did you see anastasia's email about 1595155 [14:38] bug 1595155 [14:38] Bug #1595155: new systemd and dbus dependencies are broken

[14:38] cherylj: oh yeah, I'll jump on that, forgot about that. [14:40] marcoceppi: is it a perm denied error? or a timeout? [14:40] cherylj: perm denied [14:40] tvansteenburgh: ^^ [14:40] cherylj: I ran: `juju import-ssh-keys lp:tvansteenburgh` after I added tvansteenburgh as a user and shared a model with him [14:40] cherylj: he can juju status, but not ssh [14:41] fwereade_: can you go back to your usual nickname so I can privmsg you without having 2 different logs plz? [14:45] cherylj: i have lots of ssh keys, how does `juju ssh` know which one to use? is there a way to tell it? === fwereade_ is now known as fwereade [14:53] perrito666, sorry :) [14:55] tvansteenburgh: it uses a lot of keys. You can see the actual ssh command if you run: [14:55] JUJU_LOGGING_CONFIG="=TRACE" juju ssh [14:56] tvansteenburgh: you'll see something like this: [14:56] 2016-06-23 14:55:22 TRACE juju.utils.ssh ssh_openssh.go:131 running: ssh -o "StrictHostKeyChecking no" -o "ProxyCommand /usr/bin/juju ssh --proxy=false --pty=false 52.10.231.18 nc %h %p" -o "PasswordAuthentication no" -o "ServerAliveInterval 30" -t -t -o "UserKnownHostsFile /dev/null" -i /home/cherylj/.juju/ssh/juju_id_rsa -i /home/cherylj/.ssh/id_rsa ubuntu@172.31.46.40 [14:57] (this is a 1.25 example, but the same should work for 2.0) [15:08] cherylj: http://pastebin.ubuntu.com/17749178/ [15:09] tvansteenburgh: sorry, you need --debug [15:09] JUJU_LOGGING_CONFIG="=TRACE" juju --debug ssh [15:10] http://pastebin.ubuntu.com/17749251/ [15:10] cherylj: ^ [15:11] oh... it's not even getting to running the ssh command [15:11] there have been changes to store keys in state [15:13] tvansteenburgh: I'm on a call right now, but will dig into this after [15:13] cherylj: thanks! [15:18] voidspace, frobware: review please? http://reviews.vapour.ws/r/5152/ [15:20] natefinch: oh happy birthday! just saw that :) [15:25] ericsnow: lol sorry [15:25] ericsnow: i heard "oh w-" [15:26] katco: was just going to see about a ship-it on that patch [15:26] ericsnow: gave it one [15:26] ta [15:26] ericsnow: provisional to other issues i opened [15:27] katco: yep [15:31] elmo: hey re. bug 1425808, what version were you experiencing this problem on? [15:31] Bug #1425808: OpenStack provider doesn't try another AZ if the scheduler fails to find a valid host

[15:32] katco: err, it's been a while since we tried so probably early 1.2x's [15:34] elmo: i did some work on that in early april; would anyone have bandwidth to test with 2.0 and provide fresh logs? [15:35] katco: unfortunately it's a little hard to test - if we create a private host aggregate on prodstack, it'll break all juju 1.2x users :) [15:35] Bug #1595600 opened: bundle parsing rejects strings containing hex values [15:36] katco: we'd have to spin up a test cloud - but if you think it's fixed in 2.x, I'll ask someone to do that [15:37] elmo: i just updated the bug with relevant info/code. it does look like we try and handle that properly; just a flaw in some error checking probably [15:37] btw natefinch Happy Bday! [15:38] elmo: but we need to reconfirm that this is still happening. [15:44] alexisb: thanks :) [15:44] Bug #1595600 changed: bundle parsing rejects strings containing hex values [15:48] natefinch: yeah, happy happy happy [15:52] happy bday natefinch [15:57] Bug #1292156 changed: add-machine help docs don't mention kvm

[15:57] Bug #1595600 opened: bundle parsing rejects strings containing hex values [15:57] thanks everybody [15:58] ericsnow: got a minute for a HO? [15:58] redir: sure [15:58] moonstone [15:58] ? [16:09] natefinch: happy birthday! [16:09] babbageclunk: thanks :) [16:13] natefinch: Oh, just seen it, Happy Bday also from the other side of the Atlantic. o/ [16:44] Bug #1595617 opened: Subordinate charm dies when deployed with --storage option [16:52] oh yay [16:52] cloud-images.ubuntu.com is down [16:52] cherylj: yes, and some others are noting other urls [16:52] so probably something DC related [16:53] wish I hadn't destroyed my previous env :( === frankban is now known as frankban|afk [17:15] perrito666: ping? [17:19] cherylj: pong, gimme a sec otp [17:32] natefinch: any luck with bug 1595155? [17:32] Bug #1595155: new systemd and dbus dependencies are broken

[17:41] Bug #1595636 opened: Cannot kill manual cloud controller [17:44] cherylj: sorry, phone was not letting me [17:44] cherylj: here now [17:45] cherylj: no... it's weird, because the original bug was reported before that change was made :/ [17:47] hey perrito666 - just wanted to see if the message in this bug is improved with your ACL work: bug 1594440 [17:47] Bug #1594440: juju gives weird errors about macaroons when a read-only user [17:48] cherylj: checking [17:49] cherylj: nope, I dont think so :( (at least not conciously) [17:53] Bug #1595636 changed: Cannot kill manual cloud controller [18:02] Bug #1595636 opened: Cannot kill manual cloud controller [18:21] man, I hate that I have to remember to pass in configuration values every time I bootstrap now [18:22] natefinch: good thing there's work to allow you to set it in the cloud settings [18:22] rick_h_: oh thank goodness [18:33] hey natefinch, sorry I didn't get back to you earlier - was on a call [18:33] natefinch: regarding the HA bug [18:33] natefinch: the original problem was something else [18:34] natefinch: and that godbus / systemd bug from anastasia was raised as something they hit when trying to verify the fix we gave them for the HA issue [18:34] does that make sense? [18:35] anyone free for a review? http://reviews.vapour.ws/r/5153/ [18:52] cherylj: trying to understand it. Sounds like there was a bug, and that got fixed, but then the deps update I did somehow re-broke it? [18:53] natefinch: it didn't re-break it. It broke things in a different way such that we can't confirm the HA fix works on xenial [18:54] cherylj: Maybe there's something special about MAAS? It works fine on Azure and LXD, at least. I've tested those in the last couple days. [18:54] natefinch: it could be [18:54] er... at least on master, not 1.25 [19:06] sinzui, mgz: do we have CI tests that test HA on MAAS? [19:08] natefinch: No. It will not take much effort to add one...when our mass comes back [19:08] lol [19:08] classic [19:08] natefinch: the outage took out all of the canonical host machines [19:08] natefinch: munna and silcoon are still not operational [19:09] sinzui: ouch [19:10] natefinch: is maas 2.0 or maas 1.9 the substrate that needs the test? [19:10] sinzui: did we run it on maas before? we've had the terminate code for maas for agest [19:11] mgz: several bundles use HA. but we never had a regular test for ha recovery on maas [19:11] sinzui: the bug I'm looking at is using maas 1.9 [19:11] mgz: this related to the current pressure to stop testing on all substrates [19:11] bbl [19:12] natefinch: okay, when munna returns, I will add the test [19:12] why is there pressure to stop testing on all substrates? [19:12] that seems like the opposite of what we want [19:12] natefinch: testing is fast if we only test on lxd...but there are two many substrate nuances for that to be realistic [19:12] the same is true for arch and series [19:13] sinzui: in theory they should all be able to run in parallel, right? So the only problem is the one slowest test. Which, granted, will be slower than LXD, but the benefits would seem to outweigh the problems [19:13] natefinch: exactly what I wrote today :) [19:25] * redir goes for lunch [19:29] natefinch: the test is added. We are just waiting for the host to comeback to life [19:44] sinzui: thanks [19:58] thumper, do you know when menn0 will be around today? [19:59] alexisb: normally sometime in the next hour [19:59] thumper, ok [19:59] natefinch, how long on you on todya? [20:01] alexisb: one more hour... I could make time for a meeting later, but it's both my birthday and my wife's birthday, so working late before the kids go to bed is a no-go. [20:01] you and your wife have the same bday? [20:01] alexisb: yep :) and my twin sister :) [20:02] lol [20:02] makes it harder to forget :) [20:02] natefinch, nevermind [20:02] enjoy your bday [20:02] alexisb: met my wife on a dating service, I got her to notice me by mentioning we had the same birthday ;) [20:02] heh [20:03] I leaned some interesting finch trival today [20:03] :) [20:03] makes for fun when we all go out to eat and they ask whose birthday it is :) [20:03] natefinch: what is the plural of finch? finches? finchii? [20:03] katco: finches :) [20:04] natefinch: happy birthday to the finches [20:04] btw the collective noun is a charm of finches :) [20:04] lol [20:05] * natefinch doesn't know who makes up this crap. [20:05] katco: and thanks :) [20:12] Bug #1595636 changed: Cannot kill manual cloud controller [20:12]