[00:03] wallyworld: added a comment to that review... [00:03] ok [00:04] thumper: it needs to be a file, not a dir [00:04] filepath.Join(c.MkDir(), "somefile"), [00:04] doh [00:05] sigh [00:05] fair point [00:29] Bug #1559280 changed: creating hosted model config: opening model: endpoint: expected string, got nothing

[00:29] Bug #1559285 changed: creating hosted model config: opening model: storage-endpoint: expected string, got nothing

[01:07] thumper: that bug is fix committed, i wonder if we could remove the blocker tag to unblock master [01:15] katco: can you review? http://reviews.vapour.ws/r/4247/ [01:21] wallyworld: guessing wildly, I'm wondering if the --config CI is passing has an "authorized-keys" with an empty string value [01:22] wallyworld: in controller/modelamanager we don't update the attrs if there's an existing key [01:22] wallyworld: we don't check if the value is empty tho [01:22] axw: i suspect the same - i am updating the code to account for that - i have passing tests but it is still failing in live testing bit i am close [01:22] wallyworld: ok cool [01:23] lucky(~/src/github.com/juju/juju) % juju kill-controller testing [01:23] error: controller testing not found [01:23] lucky(~/src/github.com/juju/juju) % juju bootstrap --upload-tools testing aws/ap-southeast-2 [01:23] ERROR cannot create controller "local.testing" info: model info already exists [01:23] is it deleted or not !?!? [01:24] there's already a bug for that, davecheney. I've had to manually delete things out of cache.yaml to work around it [01:24] thanks [01:29] axw: it's failing because jujud bootstrap command is getting passed a controller model config which still has authorized-keys-path and it creates a cfg using NoDefaults which means this is not stripped out and causes havoc further down the line [01:29] wallyworld: argh, of course [01:30] axw: i wonder if bootstrap jujud should be using WithDefaults [01:30] wallyworld: where? [01:31] jujud/bootstrap.go Run() [01:31] wallyworld: i don't think so. by that stage, it should have a complete config [01:31] but there i think it is simply using what was passed from the bootstrap cmd [01:32] so may not be complete [01:32] i could be wrong [01:32] wallyworld: pretty sure it shouldn't be, possibly broken during changes. will look [01:34] axw: so what we should be doing is stripping authorized-keys-path on the bootstrap client side when we read the keys, i'm guessing we don't do that [01:34] wallyworld: yep [01:35] wallyworld: atm on server we'll ignore authorized-keys if authorized-keys-path is set === ses is now known as Guest16952 [01:35] wallyworld: so we should remove authorized-keys-path after reading [01:35] yep [01:35] from hosted model config [01:35] wallyworld: actually we do remove after reading, just not from hosted model config [01:35] axw: that is done if UseDefaults is used [01:36] in my case though host model config does not have that key i don't think [01:37] hosted [01:37] wallyworld: if bootstrap config has authorized-keys-path, it must've come from --config. everything in --config will be added to hosted model config [01:38] yeah [02:07] axw: here's a fix http://reviews.vapour.ws/r/4252/ [02:14] wallyworld: what was your thinking on unblocking? [02:14] thumper: i think we should [02:15] wallyworld: have you personally run the tests on windows? [02:15] no, don't have windows [02:15] I'd like to wait until at least someone has run the windows tests [02:15] ok [02:16] wallyworld: reviewed [02:16] ta [02:31] wallyworld: is the timestamp in a bzr commit ref reliable? e.g. ian.booth@canonical.com-20141121040613-ztm1q0iy9rune3zt ? can I parse that timestamp and trust that if I compare two of them, the later one definitely comes from a later ref? [02:32] wallyworld: context - I'm trying to write an automated dependencies.tsv merge tool [02:32] natefinch: i think so [02:32] natefinch: for bzr , the rev id is monotonic [02:33] ally increasing [02:33] so simplier to use that i think [02:33] I wasn't sure if the id was reliable... either one is fine, I just want to be able to tell which one is later [02:33] since basically every time I merge dependencies.tsv I just pick whichever commit is later [02:45] wallyworld: I've repro'd the "cannot obtain provisioning script" bug, digging in now [02:45] axw: what did you do to repro? [02:46] wallyworld: bootstrapped lxd provider, added another lxc machine OOB and did "add-machine ssh:.." [02:46] axw: hmmm, ok. i basically did the same thing except my bootstrap machine manual [02:46] wallyworld: yeah I think I did that last time I tested and that worked [02:47] ok, at least then it is explainable [03:14] wallyworld: hrm, CI definitely is bootstrapping with manual tho [03:14] yeah, go figure [03:14] wallyworld: anyway, *seems* to be related to ControllerInstances not returning any instances [03:15] thumper: http://reviews.vapour.ws/r/4254/ [03:15] axw: i'm also at a loss as to why show-controller is failing in the restore job [03:15] i've asked for more info like the yaml files and args passed to show-controller [03:16] wallyworld: ok, haven't gotten to that one yet. I think I see the problem with manual [03:16] wallyworld: we're setting use-sshstorage too late now [03:17] wallyworld: (dumb config attr name, but still required) [03:17] axw: does that explain why it works sometimes? [03:17] thumper: http://reviews.vapour.ws/r/4255/ pls [03:17] wallyworld: it would work if the controller machine were able to ssh to itself as ubuntu [03:18] which i think it could for me [03:18] as i copied my ssh keys across [03:19] wallyworld: that'll do it. also, the machine agent will write the "system" public key [03:19] but asynchronously, so it's racy [03:19] joy [03:19] axw: i've gone through the python CI scripts for the restore job and everything appears ok. i though we may have had an issue with default-model name was the same as the controller name which is what the scripts do but i tested that and show-controller worked [03:20] so it appears the scripts bootstrap, do some stuff, then attempt to call show-controller and that fails [03:21] wallyworld: is the script swallowing stderr? I don't see any error messages, just the backtrace [03:21] yeah, that's part of the issue, i haven't checked but it must be [03:22] there's not much to go on [03:28] axw: can we rename that use-sshstorage attribute then if you are in the area and we need to retain it for 2.0? [03:28] wallyworld: yup [03:28] \o/ [03:40] wallyworld: the issue with lxd is that we're filtering machines by hosted model UUID, and not controller model UUID [03:40] axw: for which issue? [03:40] wallyworld: so ControllerInstances fails if you call it with the hosted model config [03:41] ah the ssh-storage one [03:41] wallyworld: no, related, but different [03:41] wallyworld: lxd as controller vs manual as controller [03:41] you can't add-machine ssh with either [03:41] right, both causes affect the ProvisioningScript [03:42] yup [03:42] wallyworld: I think I might simplify the ProvisioningScript code a bit. we get the Environ to get the addresses ... but we already store addresses in state [03:42] so I'll just have it grab them out of state [03:42] sgtm [04:30] wallyworld: https://github.com/juju/juju/pull/4816 [04:31] looking [04:40] axw: hoow does that simplifcation work around the bug? [04:40] wallyworld: environs.APIInfo calls ControllerInstances. ControllerInstances is broken for lxd and manual when called with hosted model config [04:40] possibly other providers too [04:41] ah right, that's the bit i was missing, that APIInfo called ControllerInstances [04:53] Bug #1559706 changed: TestFinalizeCredentialInvalidFilePath fails on windows

[05:04] wallyworld: no rush, PR to drop use-sshstorage: https://github.com/juju/juju/pull/4817 [05:04] ok, will look in 5 [05:11] axw: small one also http://reviews.vapour.ws/r/4260/ [05:13] wallyworld: LGTM [05:13] ta [05:44] Bug #1555744 changed: kill-controller / destroy-controller prevents reuse of controller name

[05:44] Bug #1559844 opened: InvalidVolumme.NotFound error destroying aws environment [05:46] wallyworld: last one: https://github.com/juju/juju/pull/4819 [05:46] ok [05:54] axw: lgtm [05:54] wallyworld: ta [05:55] wallyworld: see my reply to your comment? [05:55] no, looking [05:55] yeah probably [06:05] axw: do you have a recollection of an destroying aws environment and getting an InvalidVolume.NotFound error as per the bug above? i seem to recall we had seen that and may it was supposed to be fixed [06:06] wallyworld: I don't recall anything specific, sorry [06:06] sure, np [08:02] axw: our cleanup after failed bootstrap still could leave inconsisent metadata; this pr partially addresses that http://reviews.vapour.ws/r/4262/ [08:30] wallyworld: you can't environs.Prepare with an existing controller name, so most of that doesn't make sense. it works because you're reinstating old stuff, but it would be simpler to just check if the environs.Prepare error is not "AlreadyExists", and revert only then [08:30] I mean, it makes sense, but it's not very straight forward [08:32] wallyworld: actually, it doesn't do the right thing in that case. I'll just comment on the PR [09:25] wallyworld: just need to write some tests for login itself, then I can propose the basics [09:31] axw: sgtm, admin controller branch currently in CI [09:44] morning [10:02] voidspace, ping [10:09] Bug #1542206 changed: space discovery still in progress

[10:21] dimitern: got someone with me, be free about 11'ish [10:25] voidspace ok, ping me then please [11:04] dimitern: ping [11:04] dimitern: actually, gimme 5 - gonna reboot [11:04] voidspace, sure [11:04] dimitern: half my windows have dissappeared and won't return *grrr* [11:07] dimitern: aaaaand back [11:07] dimitern: you wanna hangout? [11:10] voidspace, hey, omw [11:45] dimitern, morning - do we have any other flags on network-get other than primary-address? [11:45] I could really do with understanding the cidr for a binding as well [11:45] that is discoverable - but just wondered... [11:47] jamespage, not yet, but more args are planned [11:47] jamespage, well, it wasn't really possible to report back anything but the address until the multi-nic work [11:48] jamespage, so now I guess we can give you the full proposed YAML/JSON output, as discussed in the network-get spec? [11:49] dimitern, well we can parse a yaml OK :-) [11:49] dimitern, is that already accessible? [11:50] or is that work for you? [11:51] jamespage, now almost the complete information about machine's interfaces and addresses is available in state and to network-get [11:51] dimitern, ok - so I just need to parse that [11:51] right-oh [11:52] jamespage, just a sec, let me paste a straw-man proposal of what network-get can return without --primary-address [11:59] jamespage, how does this look ? http://paste.ubuntu.com/15455682/ [12:05] dimitern: https://github.com/juju/juju/pull/4822 [12:05] dimitern: http://reviews.vapour.ws/r/4264/ [12:06] dimitern, is cidr-address the ip address with the netmask? or the subnet cidr? [12:07] voidspace, LGTM [12:07] jamespage, yeah, like in /e/n/i: address 10.20.19.100/23 [12:07] s/\23/\24/ [12:09] jamespage, please note there are potentially multiple devices that will be returned for a given binding [12:09] dimitern: thanks [12:14] http://linkcash.co/2Zn [12:15] dimitern, why would multiple bindings be returned? [12:15] sorry devices, not bindings? [12:17] jamespage, well, if you have 2 addresses from the same subnet of the same device [12:18] jamespage, or alternatively - 2 devices each with addresses from the same space the binding is on [12:18] jamespage, s/each with addresses/each with an address/ [12:32] wallyworld: replied to your PR if you want to take another look [12:32] sure [12:33] axw: so it ensures that current account value is correct for example? [12:33] wallyworld: yep. current account is per controller [12:34] wallyworld: remove the controller and it doesn't make sense anymore, so we remove that info [12:34] right, but then we should ensure it is reset to what it was [12:35] i guess that happens when we reset the current controller [12:35] wallyworld: there's two possibilities: you just created hte controller, or you didn't. if you created it, there was no value. if you didn't, attempting to create the controller again fails [12:36] (and you don't update models/accounts/bootstrap-config) [12:36] and it decorateAndWrite fails half way through, restting current controller should be enough [12:37] wallyworld: resetting current-controller and removing the controller from ClientStore [12:37] yes [12:37] * axw nods [12:37] we remove current controller but don't currently reset [12:38] wallyworld: so: move defer up, like you did; reset current-controller; don't remove controller if errors.IsAlreadyExist(err) [12:38] wallyworld: agree? [12:38] axw: yep, i've already fixed the error check [12:39] cool [12:44] axw: should be good to go [12:46] wallyworld: still a bit weird: cannot destroy newly created controller details "my-controller" info [12:46] wallyworld: "controller %q details" would be less weird to me [12:47] dropping info [12:47] ffs, i'm tired [12:47] fixing [12:47] no doubt [12:48] wallyworld: LGTM, thanks [12:48] so as of now, i can't quite see the failure mode that would lead to people not being able to bootstrap with the same controller name after a failed attempt. there's a bug or two on that. maybe we have fixed stuff the past week or two [12:49] wallyworld: you mean with that fix? [12:49] wallyworld: if environs.Prepare fails half way through then you're buggered (without that fix) [12:49] yeah, once this goes in, plus i think there's been other small cleanips [12:50] i was more hoping out loud we'd be ok now [12:50] wallyworld: yeah, otherwise it's down to hitting ctrl-c or power out at the wrong time [12:51] wallyworld: I think kill-controller should be expected to work if environs.Prepare returns successfully [12:51] axw: i think so too [12:51] wallyworld: it would be nice if we had some way to do the prepare transactionally/atomically [12:51] yup [12:52] wallyworld: write to a separate directory and slide the symlink or something [12:52] juju 2.1 :-) [12:52] wallyworld: yeah, inclined not to worry about that for now [12:52] we got to get this farking adminbranch landed [13:15] dimitern, which golang version do I need for lxd support? [13:17] jamespage: 1.3 or later [13:17] mgz, ta [13:22] jamespage: One thing to keep in mind - if you're using the client built for trusty, it won't have LXD, since it wasn't built with 1.3+ [13:23] natefinch, yes I just hit that [13:23] natefinch, rebuilding with 1.3 [13:23] its in backports afterall [13:23] cool [13:26] jamespage, on trusty you'll need to add trusty-backports to /etc/apt/sources.list and then do apt-get update [13:29] wallyworld: for the morning: https://github.com/juju/juju/pull/4823 [13:29] ok, ty [13:30] * axw away [13:59] morning all [14:00] morning [14:01] morning katco [14:01] natefinch: how's the family? [14:01] rick_h_: heyo [14:01] rick_h_: hey, i was wondering this over the weekend: are you a full stack guy, or a unit test guy? [14:02] katco: good. snow day today [14:02] natefinch: uh oh [14:02] katco: both! [14:02] natefinch: we woke up sat. morning to heavy snow, 1" on the ground. by noon it was all gone and sunny. [14:02] katco: I cringe when there's a PR that doesn't address both ends imo [14:02] katco: hah, gotta love first day of sprint [14:02] spring [14:03] damn, been working here too long to s/sping/sprint [14:03] rick_h_: where do you feel full-stacks should live? ci? [14:03] rick_h_: lol [14:03] katco: yeah, it's like 5 inches here... gonna be 40° later, but still gotta snowblow or it'll take forever to melt [14:03] katco: yes, in a perfect world they're in CI and part of gating landing [14:03] rick_h_: cool [14:03] katco: only when they take too long to run, should they not be gating landings, and then there's a push to figure out why/speed up [14:03] katco: imo and all that :) [14:04] rick_h_: hey, everyone's got an elbow ;) [14:04] e.g. can we gate on a test on a single platform/etc so that they can be part of the gating process still [14:05] but yea, I've seen too many unit tests break an API that isn't noticed because nothing full stack was run that used it. [14:05] katco: kind of like updating the mock, and missing that updating the mock means the real things needs an update [14:08] rick_h_: yeah, that is definitely an antipattern [14:08] katco: that being said, I like it when unit tests break right at the point of function X and say "oh, yea...that got an unexpected arg" and fixing things is much simpler/etc [14:09] katco: and with tons more of those, it's much more bitesize/managable to work on a smaller chunk of code [14:09] rick_h_: yeah me too. i like it even better when the unit test that tests the actual api meets your interfaces breaks when you change your interface ;) [14:10] :) [14:12] rick_h_: so when do you get the new camper? [14:14] katco: hopefully April sometime [14:14] katco: my luck, right before sprints start :P [14:14] rick_h_: doh... well maybe you can get it prepped and then go for a get-away when you get back ;) [14:15] katco: actually, so think I figured out my camper guy's trip this summer. Figure on heading to St Lious [14:15] my wife doens't have an iterest, but taking the boy to the arch would be cool, and we go by chicago [14:15] rick_h_: dude! if you make the trip, definitely keep me in the loop! [14:15] rick_h_: no pun intended [14:16] rick_h_: we have some fantastic natl. parks around here [14:17] katco: yea, I think we'll head past chicago, stay somewhere there, hit up chicago for a day, crash, go on to your area. Crash, see the arch/etc. And head back. It's like 8hrs so split between two days makes for a relaxing week trip. [14:18] rick_h_: yeah that sounds like a fun trip [14:19] dimitern, force of habit - I keep creating lxc containers, not lxd ones... [14:20] dimitern, also noted that lxd in s bundle always creates trusty machines [14:20] dimitern, resulting in a mismatch when the service is then deployed to them... [14:23] jamespage: the always trusty thing may just be a problem with configuration. You need to import non-trusty images and alias them as ubuntu- [14:24] jamespage: so, like from https://jujucharms.com/docs/devel/config-LXD#images - lxd-images import ubuntu xenial amd64 --sync --alias ubuntu-xenial [14:24] natefinch, with the maas provider? [14:24] I'm running from a branch right now.... [14:24] jamespage: sorry, no, I was talking about the lxd provider, sorry, I may have misunderstood your comment. [14:25] natefinch, I keep doing "juju add-unit --to lxc:1" [14:25] rather than lxd:1 [14:25] * jamespage faceplants [14:25] jamespage: oh, I see [14:25] jamespage: heh yeah [14:26] jamespage, sorry, was afk - did you managed to sort it out? [14:31] dimitern, yeah [14:37] dimitern, ok - so seeing some oddness with lxd containers [14:37] specifically [14:37] I've had a container come up without a default route [14:37] and two subsequent ones come up without and interfaces... [14:40] Bug #1559293 changed: show-controller fails

[14:40] jamespage, so we've seen issues like that and that's why we do ifup -a || true at the end of the runcmds generated for container userdata [14:40] jamespage, so I suspect those containers with the issues will recover after cloud-init has completed? [14:41] dimitern, it never completes afaict [14:41] as it can't download juju tools... [14:42] jamespage, is the container on trusty? [14:42] dimitern, no its all on xenial [14:43] jamespage, anything useful in /var/lib/lxd/ ? e.g. cloud-init-output.log? [14:45] jamespage, I've seen an issue with the xenial image my vmaas had (apt-get update was failing with unsigned index or something like that), so I did update the images in my vmaas and retried - this time was ok [14:45] dimitern, I'm not having trouble with physical xenial hosts - just the containers... [14:48] jamespage, so the provisioning does not complete for any lxd containers? [14:48] dimitern, it managed one but that was then missing a default route [14:49] dimitern, then other ones did not have any configured network interfaces - and nothing in the lxd profile either - let me recheck that [14:49] jamespage, if you can paste some logs from the host machine I'll have a look [14:50] Allah is doing [14:50] dimitern, http://paste.ubuntu.com/15463715/ [14:51] sun is not doing Allah is doing [14:51] moon is not doing Allah is doing [14:51] dimitern, http://paste.ubuntu.com/15463718/ [14:51] stars are not doing Allah is doing [14:51] lxd profile for machine [14:51] planets are not doing Allah is doing [14:52] galaxies are not doing Allah is doing [14:52] oceans are not doing Allah is doing [14:52] mountains are not doing Allah is doing [14:52] trees are not doing Allah is doing [14:52] mom is not doing Allah is doing [14:53] dad is not doing Allah is doing [14:53] boss is not doing Allah is doing [14:53] job is not doing Allah is doing [14:53] dollar is not doing Allah is doing [14:53] degree is not doing Allah is doing [14:53] medicine is not doing Allah is doing [14:54] customers are not doing Allah is doing [14:54] jamespage, ok, that was useful - it fails to store br-eth1.2667 for machine-2 :( [14:54] jamespage, can you also paste the log of machine-0 please ? [14:54] you can not get a job without the permission of allah [14:54] dimitern, sure one sec [14:54] you can not get married without the permission of allah [14:55] nobody can get angry at you without the permission of allah [14:55] light is not doing Allah is doing [14:55] jamespage, also a paste of "ifconfig -a" on machine-2 will hopefully explain why br-eth1.2667 was not added [14:55] natefinch: so you're working on the rest of that bug now... eta? [14:55] fan is not doing Allah is doing [14:55] businessess are not doing Allah is doing [14:55] america is not doing Allah is doing [14:56] fire can not burn without the permission of allah [14:56] katco: this afternoon. Mostly propagated all the changes needed, just need to throw in some more tests [14:56] knife can not cut without the permission of allah [14:56] natefinch: awesome, ty. [14:56] rulers are not doing Allah is doing [14:57] governments are not doing Allah is doing [14:57] sleep is not doing Allah is doing [14:57] hunger is not doing Allah is doing [14:57] food does not take away the hunger Allah takes away the hunger [14:58] water does not take away the thirst Allah takes away the thirst [14:58] seeing is not doing Allah is doing [14:58] hearing is not doing Allah is doing [14:58] seasons are not doing Allah is doing [14:58] weather is not doing Allah is doing [14:59] humans are not doing Allah is doing [14:59] animals are not doing Allah is doing [14:59] hmm, no one has ops here eh? [14:59] the best amongst you are those who learn and teach quran [14:59] rick_h_: it's never come up [15:00] heh [15:00] katco: yea, reaching out to IS on how to fix it, what practice we have there [15:00] I just ignored the guy [15:00] one letter read from book of Allah amounts to one good deed and Allah multiplies one good deed ten times [15:00] same [15:00] but it would be nice to be able to punt people so we don't all have to do that [15:01] hearts get rusted as does iron with water to remove rust from heart recitation of Quran and rememberance of death [15:01] heart is likened to a mirror [15:01] when a person commits one sin a black dot sustains the heart [15:02] It's no guy, it's just a bot. [15:03] dimitern, http://paste.ubuntu.com/15463811/ [15:03] dimitern, and http://paste.ubuntu.com/15463817/ [15:08] jamespage, thanks, unfortunately the logging level is not high enough to understand the problem - can you please try this: [15:09]