[00:40] I think I need someone to explain the TestSystemKillCallsEnvironDestroyOnHostedEnviron test in featuretests [00:44] because it looks like it (1) adds a hosted model (2) blocks that model's destruction (3) creates an undertaker that can't connect and silently fails in the background (4) checks that some dummy environment was destroyed... and I really can't see how it all fits together [00:47] ...yeah, it all works fine if I just delete the code that starts the undertaker [00:47] ohhhhh [00:47] wait [00:47] are we still running an undertaker in JujuConnSuite or something? [00:48] ...apparently not === psivaa_ is now known as psivaa [00:49] waigani_, does TestSystemKillCallsEnvironDestroyOnHostedEnviron ring a bell? === mpontillo_ is now known as mpontillo [00:49] waigani_, ISTM that it runs an undertaker that doesn't do anything [00:50] waigani_, because it still passes if I don't run it === thumper-dogwalk is now known as thumper [00:54] Bug #1555391 opened: MachineSuite failed during unit test

[00:54] fwereade: waigani_ is not around for a while due to paternity leave [00:54] fwereade: so he won't answer :) [00:55] waigani_, ignore me, you have important things to do [00:57] fwereade, thumper: it's true, I'm not working - had a quick look though. Yes I wrote it, it did fail when not called. I see it has changed a bit, also there is the dep engine work? Sorry, nothing jumps out :( [01:38] Is anyone else seeing the tests from github.com/juju/juju/cmd/jujud/reboot timeout all the time? [01:41] thumper, I haven't, fwiw [01:41] if anyone's will to live is inconveniently strong: http://reviews.vapour.ws/r/4110/ [01:42] * thumper peeks [01:43] fwereade: looks pretty freaking big to me === JoseeAntonioR is now known as jose [02:21] thumper: wat> import cycle detected: github.com/juju/juju/api/backups -> github.com/juju/juju/apiserver [02:21] ?? [02:21] nm [02:22] i found the problem [02:22] the amount of coupling between the api server and client is facepalm [02:33] thumper: here is the fix [02:33] https://github.com/juju/juju/pull/4673 [02:33] it's tiny [02:34] how did we get an import cycle? [02:34] see yesterdays rant about dummy importing the apiserver [02:34] in doing some other refactoring I hit this [02:34] have a look at the branch [02:35] shipit [02:36] thanks [02:41] axw: tiny if you have a sec https://github.com/juju/bundlechanges/pull/17 [02:44] wallyworld: LGTM [02:44] ty [02:48] wallyworld, axw: is it a known issue that if you bootstrap 2 local (lxd) controllers at the same time, one of the them will fail? [02:49] menn0: don't think I've tried it, not sure if it's known by others. what happens? [02:49] wallyworld, axw: there seems to be some reliance on the controller being bootstrapped being the "current" controller [02:49] * menn0 pastebins [02:49] axw: here: http://paste.ubuntu.com/15339061/ [02:49] axw: the 2 bootstraps were running at the same time [02:51] well looks like a bug there :-( [02:51] menn0: thanks, looks like a bug in the client code. can you please file a bug? [02:51] menn0: why can't you go get a cup of coffee while waiting like everyone else :-P [02:52] I'll take a look after I finish this up [02:53] wallyworld: when trying out migration functionality it's super nice being able to quickly spin up 2 controllers at once :) [02:53] i know, just yanking your chain :-) [02:53] wallyworld: i'm just happy it's even possible now with the LXD provider [02:54] yep, it is so much better [02:54] otherwise i'd be using AWS or something all the time [03:08] lucky(~/src/github.com/juju/juju/apiserver) % pt api/private [03:08] resource.go: [03:08] 14: internalserver "github.com/juju/juju/resource/api/private/server" [03:08] the apiserver depends on a private type (if there is such a thing) from the resource api [03:08] wat [03:30] axw: here's the charm repo update https://github.com/juju/charm/pull/197 [03:35] wallyworld: done [03:35] ta [04:11] wallyworld, axw: the ticket for that lxd provider, concurrent bootstrap issue is here: https://bugs.launchpad.net/juju-core/+bug/1555430 [04:11] Bug #1555430: LXD bootstrap fails current controller is switched during bootstrap

[04:11] ta [04:11] menn0: thanks [04:11] wallyworld, axw: it also happens if you use switch while a lxd bootstrap is in progress [04:11] (same root cause I suspect) [04:12] menn0: yeah, I think we just need to set the current model name using the canonical form, so it doesn't try to resolve the current controller [04:12] axw: sounds feasible [04:15] Bug #1555430 opened: LXD bootstrap fails current controller is switched during bootstrap

[04:21] Bug #1555430 changed: LXD bootstrap fails current controller is switched during bootstrap

[04:32] axw: last one, no rush, can't land yet anyway http://reviews.vapour.ws/r/4114/ [04:33] Bug #1555430 opened: LXD bootstrap fails current controller is switched during bootstrap

[05:17] axw: will i double up with you if i implement bootstrap maas/ [05:17] wallyworld: nope [05:17] ok, i'll do that [05:18] wallyworld: I'm probably going to have to add CredentialsStore to ClientStore after all... all model commands will need it to pass to juju/api.go [05:18] ok [05:18] wallyworld: sorry for the extra work before, didn't see that coming [05:18] np, i thought i did but wasn't sure [05:18] worth trying to keep it separate [05:19] wallyworld: it's still possible, but more trouble than it's worth I think. all the command tests would then need to provide multiple mocks. if it was just bootstrap and a couple of commands, it would be okay [05:20] but since it's all of them... [05:20] yeah [05:49] axw: a small one http://reviews.vapour.ws/r/4115/ [05:52] wallyworld: I don't think this is useful without the MAAS client profile reading code? you'll still need to record the credentials, so you haven't really gained much [05:53] axw: agreed, but what you have gained is not having to manually edit *two* yaml files [05:53] and in our getting started, we talk about credentials [05:54] so being able to bootstrap maas right after that is a win [05:54] the getting started was just added to the release notes [05:54] wallyworld: sure, I suppose so. it'd be nice if we could just use the CLI profiles, but I guess that can still be done later. [05:54] axw: yeah, i don't even know what the CLI profile format or locaton is tbh [05:59] wallyworld: it would appear to be an sqlite3 database [05:59] ~/.maascli.db [05:59] hmmm, ok. i have asked them about providing a .json file [06:00] wallyworld: we may just be able to use the "maas" CLI directly, ther eis a command to list [06:00] not sure if it has the right details tho [06:00] we need the 3 parts to the apikey - token key, token secret, client key [06:00] looks like it does [06:01] it'll print out the profile name, URL and creds [06:01] axw: wow, ok, i didn't realise we had that. what generates that sql db? [06:01] wallyworld: "maas login" [06:01] because i've only ever logged in to the web ui [06:02] axw: how hard woud it be to whip up a credential detector for maas then? [06:02] wallyworld: shouldn't be hard at all, but I don't have a MAAS set up to test with atm [06:02] damn, ok [06:02] was thinking it would be nice to sneak into the beta [06:28] wallyworld: just FYI, I'm changing the interface of EnvironProvider again, so this may take a little while yet. [06:28] "this" being the bootstrap config changes [06:29] ok, np [06:30] axw: i pushed fixes to your review, there's a couple of issues i've commented on and/or dropped, feel free to query. bbiab, school pickup [06:30] wallyworld: ok, thanks. will look a little bit later [08:13] Bug #1555475 opened: "juju bootstrap ... lxd" fails if lxd listening to some but not all addresses

[08:32] morning [09:00] dimitern: nope, /e/n/i/ does not get additional iface stanzas created if you apply a profile during lxc init. [09:02] frobware, right, that's not so surprising actually, as lxc typically ignores /e/n/i when configuring networking on boot [09:33] dimitern: I was just merging master. You recently changed github.com/juju/schema and github.com/juju/testing, we want the maas-spaces2 version in depds.tsv, correct? [09:34] frobware, yeah, unless in master those are using more recent revs [10:01] frobware, dooferlad, sorry guys, I'll be 5m late for standup [10:55] fwereade, ping? [11:11] frobware, voidspace, dooferlad, http://reviews.vapour.ws/r/4117/ - fixes bug 1554584, please have a look [11:11] Bug #1554584: TestAddLinkLayerDevicesInvalidParentName in maas-spaces2 fails on windows

[11:11] dimitern: looking [11:12] managed to get the state tests to pass on windows [11:12] dimitern: that's great. thx [11:13] dimitern: what was the problem with ".." on windows? [11:14] dimitern: anyway, the change LGTM [11:16] dimitern: frobware: dooferlad: reinstalled maas 2 with controller with only one nic - looks like the same issue commisioning nodes so trying reconfiguring the world [11:16] voidspace, thanks [11:16] Bug #1554584 opened: TestAddLinkLayerDevicesInvalidParentName in maas-spaces2 fails on windows

[11:17] voidspace, the problem is linux and windows consider different NIC names as valid - in linux's case "." and ".." are not allowed [11:17] dimitern: right [11:18] rack controller had wrong address, region controller had the right one it seems [11:19] still 400 error on commission [11:19] dimitern, voidspace, dooferlad: merge master into maas-spaces2, http://reviews.vapour.ws/r/4118/ [11:19] frobware, looking [11:19] dimitern, voidspace, dooferlad: unit tests passed on my machine [11:20] frobware, how did you update the dependencies.tsv? [11:20] frobware, plese use godeps github.com/juju/juju/... instead of just replacing revs [11:21] dimitern: ah... so that's how I've done it in the past. [11:21] frobware: looks like that doesn't have discoverspaces worker changes - so they haven't landed on master [11:21] fwereade must have done them in a branch then, rather than on master [11:21] so it's fine [11:21] (from my perspective) [11:24] frobware, so which files had conflicts? [11:24] dimitern: listed in the commit: dependencies.tsv, provider/maas/bridgescript_test.go and provider/maas/environ.go [11:25] frobware, ah, sorry - navigating a giant merge is not well supported on github ui [11:26] voidspace, yeah, it's on MADE-model-workers [11:28] frobware, LGTM [11:29] dimitern: since I did the godeps update and rebuild make check, unit tests now fail. [11:30] frobware, oh - what failed? [11:30] fwereade: cool, thanks [11:30] dimitern: FAIL: client_macaroon_test.go:58: clientMacaroonSuite.TestAddLocalCharmSuccess [11:31] frobware, hmm.. I've pulled your branch and trying the same [11:31] dimitern: the branch you're pulling doesn't have the update to deps...tsv [11:32] frobware, ok, so I might have mislead you [11:32] frobware, just a sec.. [11:33] dimitern: why is using godeps better than selectively choosing between A and B during a merge conflict? [11:36] frobware, so, this is what I did: [11:37] frobware, git checkout frobware/maas-space2-merge-master-at-0582881f (I have rev 193...) [11:38] frobware, godeps -u dependencies.tsv & go install -v github.com/juju/juju/... [11:39] frobware, then godeps -N github.com/juju/juju > new-dependencies.tsv && diff -u dependencies.tsv new-dependencies.tsv - revs are the same, but a few dates got updated, while some probably no longer used repos got dropped [11:40] frobware, finally, rm -fr $GOPATH/pkg ; mv new-dependencies.tsv dependencies.tsv ; godeps -u dependencies.tsv (passed ok) ; go install -v github.com/juju/juju/... (all ok still) - now running make check [11:40] frobware, this is the diff - http://paste.ubuntu.com/15340794/ [11:43] so far I test failure, flaky as re-runs pass: http://paste.ubuntu.com/15340800/ [11:43] s/far I/far 1/ [11:47] dimitern: let's HO, I get different results to your diff [11:47] frobware, so I guess the steps above yielded the same set of deps more or less as in your case, but with the added benefit of also having the correct timestamps and sort order [11:47] frobware, ok, omw to standup HO in a moment [12:07] dimitern: I pushed the updated deps [12:09] frobware, cheers [12:10] frobware, apparently virsh have changed and now "list" does not show inactive domains, unless you pass --all [12:10] frobware, so not sure if maas is broken due to that or what.. doing dist-upgrade on the maas machine just to be sure [12:11] Bug #1555585 opened: Master dumps a "client.crt" and "client.key" file in CWD when copying images

[12:12] dimitern: that 'virsh list --all' was always a thing for me [12:13] frobware, I'm unable to start any kvm even manually on my machine - hitting https://bugzilla.redhat.com/show_bug.cgi?id=680821 apparently :/ [12:14] dimitern: that's an old bug and kernel [12:18] frobware, yeah, this is more recent: http://stackoverflow.com/questions/6307950/virt-install-error [12:21] although uncommenting `user="root"` and `group="root"` in /etc/libvirt/qemu.conf and restarting libvirt-bin didn't help [12:42] voidspace: did I misunderstand your recent change around space id vis-a-vis names: I get this from status: ... interfaces=peer:space=1;reverseproxy:space=1;website:space=2 zone=default [12:43] frobware, interesting.. I learned something today: http://www.agix.com.au/kvm-error-ioctlkvm_create_vm-failed-16-device-or-resource-busy/ [12:44] dimitern: aha, yes. :) [12:44] frobware, apparently you can't run a virtualbox vm at the same time as a kvm machine [12:44] :) [12:44] dimitern: in that scenario what does kvm-ok report? [12:44] dimitern: you want lxd anyway. :-D [12:45] frobware, reports as expected [12:45] frobware, INFO: /dev/kvm exists [12:45] KVM acceleration can be used [12:45] frobware, bootstrap is now going [12:51] frobware, it still takes ~4-5m for the bridge script to complete (on a node with 2 phys nics and 3 vlan nics each) [12:51] frobware, I can see that with ps xawwf after ssh-ing into the node as bootstrap still goes [12:52] frobware, so we might need to add a shorter maxwait stanza (or whatever it is) [12:58] frobware, alternatively, this slowdown can be due to this: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/246203 (or something like this, still related to ntpdate dead-locking) [12:58] Bug #246203: ntp init script started multiple times - causing 180 seconds wait on boot [13:00] frobware, I can confirm the bundle deployment worked as expected, so let's merge it [13:15] frobware: that sounds right [13:15] frobware: oh, from *status* [13:15] frobware: status should report name [13:15] frobware: looks like there's somewhere still using ProviderId instead of name then [13:15] frobware: I'll look into it [13:28] voidspace: thanks [13:32] tych0: dooferlad: if you want to do a followup review of the LXD patches: http://reviews.vapour.ws/r/4082/ [13:32] should handle bug #1555585 [13:32] Bug #1555585: Master dumps a "client.crt" and "client.key" file in CWD when copying images

[13:33] as well let us set "image-stream: daily" so when you bootstrap you get the current daily image [13:36] frobware: ^^ looks like you are OCR for today. [13:37] jam: yep [13:40] jam: releases vs released. isn't the stream releases? [13:42] the string is cloud-images.ubuntu.com/releases, but we use "ImageStream: released" internally [13:42] If you look at environs/config/config.go the default is "released" [13:42] frobware: https://github.com/jameinel/juju/blob/master/environs/config/config.go#L1103 [13:43] jam: was asking because I set up a local mirror yesterday and somewhat intuitively did `mkdir images/released' only to notice the URL used releases. [13:48] frobware: yeah. We have a bit of code for other providers that does "val = stream; if val == released: val = "releases" somwhere in the code [13:48] frobware: I believe in the simplestream metadata itself it is "released" [13:48] frobware: http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:aws.sjson [13:48] noticed the "released" in the URL [13:49] and "released" is part of the content, as well. [13:49] Yay for consistency. [13:49] frobware: anyway, review appreciated, will be walking the dog for 30 min or so. [13:50] jam: looks good to me [14:23] frobware: I think I addressed your comments, but I had some questions, can you clarify? [14:24] frobware: how were you getting that status line? [14:24] frobware: is that from a deployed service - and was it deployed with constraints/bindings ? [14:26] jam: looking [14:26] jam: 15 min downstairs, 15 min upstairs? ;) [14:27] voidspace: I was using the dooferlad's maas-spaces.py mediawiki demo, so yes to constraints. [14:27] TheMue: 5 min down, walk the dog for ~20, 5 min back [14:27] jam: I'm impressed there's no diff between down and up. :) [14:27] jam: which floor? photos always seem to be very high [14:27] frobware: elevators. I don't usually walk 61 flights. [14:27] ouch, 61 [14:28] TheMue: I've walked it down when there was smoke once. About 20 min down [14:28] jam: good for physics, but no must have for each day [14:29] TheMue: would be better for me if I did :) [14:29] Jam stairs are a great exercise [14:29] perrito666: I've been tempted to try it up once, but I don't think it would go super well. [14:29] jam: oh yeah, in my case too. but already sharpening myself a bit. [14:29] You get used to :p [14:31] perrito666: I had it for some time with 6 floors for parking and then 4 into the office, has been fine, but please not more than these [14:31] Lol i think i could try it once a day for cardio [14:31] frobware: if it's just showing constraints then that's "correct" [14:32] frobware: as in, that's what we pass through to maas [14:32] The thing is , i have fear of heights so i might not like to go 61 floors up [14:32] perrito666: currently I have to admit I'm sitting in the fourth floor and take the elevators. so it's a good hint, will take the stairs down later. [14:33] perrito666: fear of heights in buildings or also when flying? [14:33] voidspace: let me scrollback to see if I can find the output... [14:34] frobware: anand what format are you using for status [14:34] frobware: I'm not seeing anything like that in default status [14:34] Themue just buildings [14:34] voidspace: http://paste.ubuntu.com/15341413/ [14:34] frobware: that's not "status"... [14:35] perrito666: Always insteresting how different we react on buildings or flights. [14:35] voidspace: fair enough. using old terminology. [14:35] frobware: those constraints are being returned by maas [14:36] frobware: so that is the constraints we passed to maas - it is "correct" [14:36] frobware: that doesn't explain the 409 conflict though [14:36] frobware: to fix it we'd have to parse the error strings [14:36] btw, how has the last sprint been? afaik in Oakland? [14:36] (i.e. to replace space ids with space names) [14:36] thanks for the reviews dooferlad and frobware [14:36] voidspace: and that's why I ... yep, exactly. [14:36] Themue: my current theory is that I only have this issue with bing windows [14:37] frobware: the change I made recently was to report the space name if we discover a conflict *before* we pass it to maas [14:38] perrito666: some do have it when standing outside on high towers or bridges [14:38] frobware: this conflict error from maas indicates that it doesn't have a node matching those constraints [14:38] frobware: which we can't detect in advance [14:38] voidspace: which is odd because the script goes and makes everything the same... [14:38] frobware: but for example if you had a binding to space "foo" and specified a constraint "^foo" then you'd have a conflict we can detect [14:39] voidspace: "before we pass it to mass" <- thanks [14:39] frobware: right, I've not used jame's script so I don't really know what it's trying to do [14:39] np [14:39] we still detect the conflicts using provider id - we just change them to names in the error messages [14:39] voidspace: was just trying to understand the bits and pieces that go whizzing by [14:40] voidspace: however, as an end-user how do I understand and resolve the error message I got back [14:40] frobware: well - the error is that maas has no machine available that meets your requirements [14:41] frobware: hopefully as an end user you understand your requirements [14:41] voidspace: :) but UX. [14:42] frobware: the error message is relatively clear "No available node matches constraints" [14:43] frobware: we're directly surfacing a provider error though [14:43] voidspace: and "space=2" means... ? [14:44] frobware: if you really don't know what spaces you asked for then "juju list-spaces" does tell you ids [14:44] frobware: but this is an error direct from maas and we *can't* depend on the error format (i.e. parse it) [14:44] voidspace: all I'm really questioning is the user experience. and whether we can have names [14:44] frobware: not in the errors that maas gives us back, no [14:45] voidspace: our only option would be to parse the string and interpolate as we can. true? [14:45] frobware: that would require the error string from maas to be of a fixed format - and the format of error strings is *not* part of the api [14:45] we *should not* start to depend on it [14:45] IMO [14:46] voidspace: understood. Just wary that we also create a problem for ourselves with bug triage, explaining to customers, et al. [14:47] frobware: I'm confident it will be less effort explaining this to users than it will be fixing it (and then refixing it every time the error message format changes) [14:47] frobware: we *could* catch 409 errors specifically and build a pretty-printed constraints string ourselves as well as showing the maas error (we don't want to mask the underlying error - the error may have nothing to do with spaces) [14:47] frobware: that wouldn't be too much work I guess [14:47] voidspace: I think this is still "status" though. You can still get show-machines output using juju status --format=yaml [14:48] heh [14:48] fair enough [14:48] voidspace: I like your suggestion ^^ [15:02] dimitern: still want to sync today? [15:02] ericsnow: morning. natefinch will be out today... your reviews will take precedent! [15:04] frobware, about the devices creation? [15:11] ericsnow: ping [15:15] cherylj, ping? [15:15] mattyw: what up? [15:20] dimitern: yep [15:22] frobware, well, I'm quite deep in code changes now, and it will be better to do it tomorrow I guess [15:22] frobware, as I've already changed twice what I though we need :) [15:23] dimitern: ok, I'll continue to look at the LXD side [15:23] frobware, sounds good [15:24] cherylj: I looked at the master failure today, seems like containers are created OK since I landed my change, but didn't really draw any conclusions about the timeout [15:24] frobware: the os deployer test? [15:29] Bug #1555585 changed: Master dumps a "client.crt" and "client.key" file in CWD when copying images

[15:48] dimitern: http://reviews.vapour.ws/r/4002/ is ready for another review [15:49] dooferlad, cheers, looking [15:49] * dooferlad goes to refresh his coffee [15:51] frobware: ping [15:52] voidspace: pong [15:52] dooferlad: looking [15:59] Bug #1555694 opened: help text for juju remove-relation needs improving

[16:04] dooferlad, reviewed === JoseeAntonioR is now known as jose [16:11] cherylj: yes [16:11] dooferlad: reviewed [16:11] ha, I was like "yes what?" [16:11] frobware, dimitern: thanks! [16:11] I had completely forgotten what we were talking about [16:12] frobware: I don't think the os deployer tests are related to your change [16:13] cherylj: it's been the same test (maas-1_9-OS-deployer) for a while though. but as you say, might be the same test but a different issue [16:14] frobware: we actually disabled that test for a while, and just recently introduced it again [16:14] frobware: it was using juju deployer and mgz had to do some work to get the bundle to work with the native deployer [16:14] frobware: so it really hasn't been run on 2.0 before [16:14] and was just blocked by that one multi-nic bug before [16:15] frobware: the test wasn't working for a while while we had the nic bug you just fixed [16:32] frobware: dimitern: dooferlad: is the spaces sync on? [16:32] voidspace: I thought I was optional, which is why I haven't joined. [16:32] jamespage, thedac, sync call? [16:33] yep :) [16:33] oops yep, omw [16:56] dimitern: on you 2 nic, 3 vlan setup, are they configured as dhcp or static? [16:56] frobware, all static === bcsaller_ is now known as bcsaller [17:02] fwereade__, ping? [17:03] dimitern: whoa, the boot time is a bit sucky when we bridge the world. [17:03] frobware, yeah [17:08] dimitern: it's quicker to bounce the machine than wait [17:09] frobware, how? just restart it while it's bootstrapping? [17:10] dimitern: yep. need to see if cloud-init allows you to do that. [17:12] dimitern: do you see the same delay on subsequent boots? [17:14] frobware, I haven't tried, but will do [17:15] dimitern: does for me [17:20] mattyw, o/ [17:20] Bug #1555727 opened: juju remove-ssh-key can remove Juju's own keys

[17:50] Bug #1555744 opened: kill-controller prevents reuse of controller name