[01:32] does anyone know where the code that fakes out the simple streams is ? [01:32] [LOG] 0:00.023 DEBUG juju.environs.simplestreams read metadata index at "file:///tmp/check-6283502795135149108/15/tools/streams/v1/index2.sjson" [01:32] ^ the one that generates this local file [01:32] for some reason the ec2 tests are different to all the other providers [01:42] nope [01:42] not me [02:09] davecheney: ToolsFixture?.. [02:10] anastasiamac: thanks [02:10] i'll try to figure out where that is hooked up [02:10] and why it is special in the ec2 provider tests [02:16] davecheney: i think test roundtripper is what delivers this stuff in tests :D [02:16] davecheney: have fun [02:35] oh boy, i think i've crackd it [02:36] fixed the ec2 test [02:36] imagetesting "github.com/juju/juju/environs/imagemetadata/testing" is the magic import [02:44] menn0: anastasiamac thumper https://github.com/juju/juju/pull/4957 [02:44] could I get a second look [02:44] this change is bigger than it started because upgrading the testing dependency hit a lot of places [02:44] i'm 90% confident that I've backported all the AddCleanup fixes from master [02:45] i'd be more confident, but the local tests haven't finished running for me [02:49] man that embedded test suite stuff is wicked error prone [02:52] davecheney: hmm... what changed with the AddSuiteCleanup code? [02:52] I have vague recollections... [02:57] thumper: I believe it's just that there's no more separation between test cleanup and suite cleanup. AddCleanup does the right thing depending on when it's called [02:58] thumper: since we found there were placing we were calling the wrong one, and it was causing problems. [02:58] * thumper nods [02:59] thumper: https://github.com/juju/testing/blob/master/cleanup.go#L59 [03:04] thumper: there are a few changes here [03:04] 1. updaed the testing dependency to spot suite mistakes [03:04] 2. updated testing itself to remove the deprecated AddSuiteCleanup method [03:04] this is already comitted to master [03:04] I backported this fix to 1.25 [03:04] then adjusted the code to avoid calling AddSuiteCleanup [03:05] and backported all of jam's fixes for various suite failures to 1.25 [03:08] davecheney: lgtm [03:08] Bug #1566024 changed: The juju GCE error message references the wrong key name [03:53] * thumper grumbles [04:47] Bug #1564163 changed: environment name in credentials file is not a tag

[04:47] Bug #1564165 changed: Credentials file displays unhelpful message for syntax errors

[04:47] Bug #1566130 opened: awaiting error resolution for "install" hook [05:15] func (s *UpgradeSuite) getAptCmds() []*exec.Cmd { s.aptMutex.Lock() defer s.aptMutex.Unlock() return s.aptCmds [05:15] } [05:15] ^ note, doesn't actaully prevent a race [05:15] unless you're only appending to that slice [05:15] , maybe [05:22] anastasiamac: review please http://reviews.vapour.ws/r/4431/ [05:27] menn0: looking \o/ [06:50] dummy needs some love http://reviews.vapour.ws/r/4433/ [07:46] davecheney: FWIW I think the "p := &providerInstance" thing was just to make it more convenient to use. There's nothing wrong about it per se. [07:49] anyone know if dimitern is gonna be around today? [07:49] rogpeppe: tomorrow [07:51] frobware: i'm interested to inquire about an apparent networking bug with multiple models in a controller. do you know who else might know about the networking stuff? [07:52] rogpeppe: try us "me, dooferlad, voidspace" [07:53] frobware, dooferlad, voidspace: ok, so we got a controller to start an instance in another model (with different provider creds) yesterday, and the machine was network-isolated from the controller [07:54] frobware, dooferlad, voidspace: i.e. it couldn't connect to the API server or ping the controller machine [07:54] frobware, dooferlad, voidspace: this was in juju 1.25.4 [07:54] frobware, dooferlad, voidspace: i was hoping this might be a known issue that's been fixed in 2.0 [07:56] rogpeppe: when you network-isolated, by how much? at the risk of stating the obvious, no network connectivity ... [07:56] frobware: we could ssh to it [07:56] frobware: (only directly, but that's another bug :-\) [07:57] frobware: i didn't test whether it could dial out to the outside world [07:57] rogpeppe: still a bit confused. is one instance running 1.25? [07:58] frobware: everything was running 1.25 [08:00] rogpeppe: otp, back in a bit... [09:02] voidspace, fwereade, jam: hangout? [09:03] dooferlad: omw [09:12] dooferlad: frobware: review please http://reviews.vapour.ws/r/4425/ [09:15] http://reviews.vapour.ws/r/4403/diff/1/?file=324207#file324207line52 [09:22] http://reviews.vapour.ws/r/4342/ [09:31] babbageclunk: hey, hi [09:31] voidspace: hi! [09:34] babbageclunk: should be in the office around 10am tomorrow... trains permitting. [09:34] voidspace: hmm, my irc connection died because I've dropped off ethernet, and the wifi is a bit spotty in the office. [09:34] babbageclunk: ok [09:34] frobware: great! I'll be in well before then. [09:34] frobware: so, will you and babbageclunk be pairing tomorrow? (and the rest of the week?) [09:34] frobware: sounds like a good plan [09:36] voidspace: oh nice, restarting network-manager doesn't actually drop the connection. [09:36] babbageclunk, voidspace: I guess... I think there's some general stuff to go over. There's also the bug I'm looking at. And there's a couple of bugs to shift to dimiter. But, yes, in principle... [09:36] ah yes, dimiter returns tomorrow - yay [09:37] voidspace, frobware: we should try to get to a point where there are parallelisable bits of work to do on the maas provider. [09:38] babbageclunk: yep. and I'm going to need an intro to what's been done too. :) [09:38] babbageclunk: frobware: it's easily parallelisable now [09:39] babbageclunk: frobware: we need Instances, AvailabilityZones and acquireNode implementing for MAAS 2 [09:39] those are the next steps, all can be tackled separately and all already have support in gomaasapi [09:39] we have the basics of test infrastructure in place too [09:40] frobware: babbageclunk *should* be able to show you what we've done [09:40] voidspace, frobware: I can have a go [09:40] frobware: it's pretty straightforward - the diff of maas2 against master isn't too huge and shows it [09:40] babbageclunk: frobware: we could topic it in standup [09:40] or just a hangout [09:41] it's actually it took us a week to get here, but the code isn't hard to understand I don't think [09:41] nor is the path ahead [09:41] * frobware needs to sort his craptop out... [09:42] :-) [09:44] oh, we'll need Spaces too [09:45] we also have the endpoint for that already written [10:06] dooferlad: can I steal some of your time? h/w too. :) [10:06] frobware: sure. 2 mins. [10:07] dooferlad: 5. coffee. [10:07] * fwereade amusing typo: synchorinses for synchronises [10:12] dooferlad: ready whenever works for you. In standup HO. [10:39] Bug #1566237 opened: juju ssh doesn't work with multiple models [10:54] frobware: were you investigating bug #1565461 [10:54] Bug #1565461: deploy Ubuntu into an LXD container failed on Xenial

[10:54] ? [10:55] jam: not actively. sidetracked by bug #1565644 [10:57] jam: but I added a comment to bug #1565461 just in case it was significant. [10:57] Bug #1565461: deploy Ubuntu into an LXD container failed on Xenial

[10:57] k [10:57] jam: I haven't looked to see if the failure on xenial is related to juju not creating any network devices for the container [10:58] frobware: fwiw I saw it on Trusty as long as you --upload-tools from Master. [10:58] frobware: it might be the same bug, I didn't get a ip addr show from inside the container when I tested last. [10:58] jam: and there did you take a look at the LXD network profile that gets created? [10:59] frobware: I'll go bootstrap now and do some debugging. I'll let you know when it is up and running. [10:59] jam: first pass validation since we did the multi nic support would be to check the LXD profile [11:30] frobware: well I would be testing but it seems jujucharms.com is broken right now. [11:31] jam: do you need a charm? can you just add-machine lxd:0 in this case? [11:35] frobware: fair point === urulama__ is now known as urulama [11:45] fwereade: ping [11:45] fwereade: we're fixing a theoretical panic case in maasEnviron.Instances (that we hit in testing our new implementation for MAAS 2) [11:46] fwereade: in the case that MAAS returns instances with different ids to the ones you requested you can get a slice of nil instances back [11:46] fwereade: because the code that builds the map of id -> instance doesn't check the id is actually in the map when fetching them back [11:47] fwereade: fixing that to return ErrPartialInstances when an id you requested isn't returned causes an existing test to fail [11:48] fwereade: because it assumes that even in the case of an error return that the returned partial results will be present [11:48] hmm... we've found a better way that makes this a non issue I think [11:48] Bug #1566268 opened: poor error when "jujucharms.com" is down

[11:48] Bug #1566271 opened: It is hard to open juju API if you're not implementing a ModelCommand [12:03] frobware: so I see a "juju-machine-0-lxd-0-network' profile that contains nothing [12:03] the machine itself seems to only have a "lo" [12:03] but somehow it came up correctly... [12:04] frobware: ok, no it did not come up correctly [12:05] machine-status is "running" but juju-status is "pending" [12:05] jam: can you try bouncing the node and adding a new container [12:05] frobware: so it does seem to be bug #1564395 [12:05] Bug #1564395: newly created LXD container has zero network devices

[12:05] frobware: restarting the host or just the container? [12:05] jam: yep, was suspicious (to me at least). [12:05] jam: just the node hosting the container [12:06] jam: the first container will still fail but I'm expecting subsequent containers to work correctly [12:06] frobware: are we not detecting networking correctly without a reboot? [12:07] jam: bug introduced post March 21st... is as far as I got. [12:07] jam: sometimes... (rarely) we get eth0 added to the container. so, a timing issue IMO [12:12] frobware: juju-machine-0-lxd-1-network is *also* empty [12:13] jam: sigh [12:15] jam: let me try again [12:15] frobware: this is with a Trusty controller, but we'll want it to work there ,too. [12:15] jam: my tip commit is probably behind master (a084e423e0586d2348963e6ba91aa3d2454997dd) [12:16] jam: any chance you could validate against xenial? [12:16] 23 commits [12:16] frobware: sure [12:16] frobware: any reason to keep trusty around? [12:17] jam: not for me. :) [12:18] jam: just bootstrapping, back in 10 [12:21] jam: who does babbageclunk ping to get added to the juju team calendar? [12:21] frobware: bootstrap is the new complie step? [12:21] voidspace: I believe all team leads should be admin, but I can go do i [12:21] it [12:21] jam: thanks! [12:23] voidspace: he should be able to add items now [12:23] have babbageclunk check to make sure I did it right [12:25] jam: heh, I bootstrapped to a node which ... will fail ... because that's configured to try and fix another issue ... [12:25] apt-get is a bit slow today [12:26] jam: swings. roundabouts. repeat. [12:26] jam: and then I run into: ERROR some agents have not upgraded to the current model version 2.0-beta4.1: machine-0-lxd-0 [12:26] jam: wow, today. ffs. [12:26] not everything got to the broken version so you can't upgrade to the fixed one... [12:26] frobware: maybe you can 'juju destroy-machine --force' [12:27] ? [12:27] jam: I upgraded to MAAS 1.9.1 and no DNS running there anymore... [12:27] frobware: no DNS at all? or they just moved where DNS is running? [12:27] jam: not sure if my 1.9.0 > .1 borked maas-dns. Either way named is not running anymore. [12:28] jam: could be because I futzed with my MTU setting to try and fix bug #1565644 [12:29] frobware: xenial is up and running [12:29] trying now [12:30] jam: lxd-images is no longer a thing... ? [12:31] frobware: no, "lxc image ubuntu:" [12:31] frobware: but juju should handle those for you [12:31] but you can do" [12:31] "lxc launch ubuntu:trusty" [12:31] it reads simplestreams directly [12:32] frobware: on xenial juju-machine-0-lxd-0-network is empty [12:32] trying reboot and 0/lxd/1 [12:32] fingers crossed for at least one thing to kind-of work today. [12:33] I have too many things which are broken. [12:35] voidspace, sorry! [12:35] voidspace, I think ErrPartialInstances is a bit different [12:36] jam: whee... error: Get https://cloud-images.ubuntu.com ....... i/o timeout. [12:36] voidspace, if maas tells us extra instances, that is annoying, but I think the reactions there are either to ignore or to go nuclear -- maas is making no sense, give back no instances and ErrMaasInsane [12:37] frobware: juju-machine-0-lxd-0 is shown as being on the lxdbr0 bridge [12:37] with an address [12:38] voidspace, if we ignore that, which is reasonable, we return the instances we got (so long as we got at least one we asked for) and ErrPartialInstances if any are missing [12:38] jam: is it possible my current tip is no longer compatible with lxd as installed on xenial? [12:38] frobware: well, the agent still didn't come up for 0-lxd-0 [12:39] and cloud-init-output.log looks very truncated [12:39] checking if 0-lxd-1 comes up [12:39] jam: I cannot import any images [12:39] frobware: 0-lxd-1 comes up with the *same* IPV4 address as 0-lxd-0 not a great sign [12:39] 10.0.3.1 for both [12:40] jam: well, I knew it was a bug... just not working on it atm... :( [12:40] frobware: juju-machine-0-lxd-1-network is also empty [12:48] jam: trying with tip of master now [12:48] jam: my maas named conf had duplicate entries [12:49] jam: (not sure how!) [12:49] fwereade: I think we've fixed it in a sane way [12:49] voidspace, cool [12:49] fwereade: I like the idea of ErrMaasInsane [12:50] fwereade: although the tempation would be just to return that for everything [12:50] haha [12:54] fwereade: on the maas2 work bootstrap now gets into StartInstance, which is nice tangible progress [12:56] frobware: dooferlad: if you get a chance we now have two PR ready: http://reviews.vapour.ws/r/4425/ [12:57] frobware: dooferlad: plus one that didn't make it onto reviewboard (yet?) https://github.com/juju/juju/pull/4995 [12:57] ericsnow: this PR hasn't appeared on reviewboard: https://github.com/juju/juju/pull/4995 [13:01] jam: so I'm not going entirely bonkers: http://pastebin.ubuntu.com/15629100/ [13:02] jam: but there's a new issue now. /e/n/i in the container is not what it used to be. [13:02] frobware: you coming to the maas meeting? [13:02] jam: http://pastebin.ubuntu.com/15629133/ [13:02] voidspace: nope. too much entropy. [13:02] frobware: sure [13:03] voidspace: I can if needed, but I guess you'll have more to discuss anyway [13:04] frobware: we're fine [13:05] frobware: and we're done [13:05] babbageclunk: so, lunch [13:06] jam: https://bugs.launchpad.net/juju-core/+bug/1564395/comments/3 [13:06] Bug #1564395: newly created LXD container has zero network devices

=== perrito667 is now known as perrito666 [13:18] Bug #1566303 opened: uniterV0Suite.TearDownTest: The handle is invalid

=== xnox_ is now known as xnox === psivaa_ is now known as psivaa [13:26] voidspace, oops, missed that too: awesome! [13:42] aghh this kill controller thing is ridiculous [13:43] perrito666: what now? [13:44] I bootstraped an lxd controller yesterday and now I cannot destroy it, and I am pretty sure nothing changed betweenyesterday and today [13:44] perrito666: :/ [13:45] * perrito666 takes the killbyhand hammer [13:46] its a good thing I actually love lxd so I dont mind playing with it === Guest77267 is now known as jcastro_ [13:47] I had to kill the containers from underneath it [13:48] apparently the lxc container is not there [13:48] and then blow away most of ~.local/config/juju [13:48] but juju thinks otherwise [13:48] yeah, look in ~/.local/config/juju [13:48] I removed everything but my creds from there and that got juju to stop lying to itself [13:49] jcastro_: that IS a bug though [13:49] oh I agree 100% [13:49] that is exactly what kill controller should do [13:49] I am worried this goes beyond the current ongoing discussion about what is not working in kill controller [13:50] lxc list | grep -E 'juju-(.*)-machine(.*)' | awk '{print $2}' | xargs lxc stop [13:50] I am extra worried that this did not blow in someone' s face in CI [13:50] mbruzek: lxc says no running container [13:50] lxc list | grep -E 'juju-(.*)-machine(.*)' | awk '{print $2}' | xargs lxc delete [13:50] its clearly juju [13:50] perrito666: Then juju needs to be more forceful when I tell it to KILL [13:52] perrito666: This happened to me yesterday which is why I had those commands in my history. [13:52] perrito666: I could not kill-controller it just sat there in a loop [13:52] mbruzek: tx anyway, I suspect this are going to be useful to me soon [13:53] perrito666: I also helped out one of our partners (IBM) who was having problems with the local lxd provider [13:54] yesterday [13:56] perrito666: mbruzek: if it's not listed here, please file a bug: https://blueprints.launchpad.net/juju-core/+spec/charmer-experience-lxd-provider [13:57] oooh, can we add a bug to this? [13:58] katco: https://bugs.launchpad.net/juju-core/+bug/1565872 [13:58] Bug #1565872: Juju needs to support LXD profiles as a constraint

[13:58] jcastro: sure... do you have a "link a bug report" button or is that just the owner? [13:58] I appear to not have a link button [13:58] but it's the bug mbruzek just linked [13:58] jcastro_: k np just toss me the # [13:59] mbruzek: linked your bug [14:00] Thanks katco [14:09] heh, it feels like we need a --force flag for kill-controller :) [14:09] I've also hit the "kill spins in a loop" error. [14:09] heh [14:09] maybe if it doesn't complete in a certain amount of time, it falls back to just destroying through the provider [14:10] ? [14:12] well that is actually what it should be doing [14:15] perrito666: yeah, I'm going to open a bug [14:15] not to dogpile on, it appears kill-controller is also not removing storage, nor is it removing units which had storage volumes attached at the time of issuing kill-controller. Working on getting a bug for you about this now [14:15] bbl [14:16] Bug #1565991 changed: juju commands don't detect a fresh juju 1.X user and helpfully tell them where to find juju 1.X [14:16] Bug #1564622 opened: Suggest juju1 upon first use of juju2 if there is an existing JUJU_HOME dir

[14:16] Bug #1566332 opened: help text for juju remove-credential needs improving

[14:16] Bug #1566339 opened: `juju run` needs a --all-machine, --all-service, --all-unit

[14:16] hey jam, I see you made this statement in a PR: "We're currently broken on AWS which makes it hard for me to evaluate this" - what's going on with AWS? [14:21] cherylj: at a guess, related to https://launchpad.net/bugs/1564395 [14:21] Bug #1564395: newly created LXD container has zero network devices

[14:22] hmm, fun [14:23] cherylj: the trouble is nobody working on it. :( [14:23] cherylj: although I was but now preempted [14:24] frobware: alas, we are interrupt driven :| [14:29] katco: crap, gotta help my wife with our accountant.... she's there now and needs my help. I'll almost certainly miss the standup. Sorry [14:29] katco: I should be back by 15-30 minutes after standup is scheduled to start, though. [14:30] natefinch: let's talk when you get back [14:30] katco: yep === natefinch is now known as natefinch-taxes [14:34] Bug #1566345 opened: kill-controller leaves instances with storage behind [14:40] Bug #1566345 changed: kill-controller leaves instances with storage behind [14:41] dooferlad: ping - any chance of using one of your NUCs? If not I'll look elsewhere [14:43] frobware: you need a nuc? what else do you need around it? [14:43] frobware: there's two free in http://maas.jujugui.org/MAAS/#/nodes [14:44] rick_h_: ability to change the MTU for jumbo frames [14:44] rick_h_: might affect the rest of your network :-D [14:44] rick_h_: needs two physical NICs [14:45] jam: yup, adding me on the calendar worked - thanks! [14:46] rick_h_: can I grab an account on there anyway? [14:46] Bug #1566345 opened: kill-controller leaves instances with storage behind [14:46] frobware: sure thing, what's your LP username [14:47] rick_h_: frobware [14:48] rick_h_: if it has two NICs I'll try anyway [14:48] frobware: yes [14:48] frobware: sec [14:52] hey lazyPower, regarding bug #1566345 - are you getting errors that your rate limit has been exceeded? [14:52] Bug #1566345: kill-controller leaves instances with storage behind [14:52] cherylj - negative, i think what happened is the volumes weren't unmounted during the storage-detaching hook (we haven't implemented this) and everything was left behind [14:52] lazyPower: I'm wondering if your bug is another side effect of bug 1537620 [14:52] but no log errata regarding rate limits [14:52] Bug #1537620: ec2: destroy-controller blows the rate limit trying to delete security group - can leave instances around <2.0-count>

[14:52] lazyPower: ah, ok [15:01] ericsnow: standup time [15:03] natefinch-taxes: are you here today, or are you too heavily taxed? [15:03] natefinch-taxes: you're OCR :-) [15:10] voidspace: he's here [15:10] voidspace: will be back momentarily [15:11] katco: cool, thanks [15:11] ericsnow: ping [15:11] voidspace: hey [15:11] voidspace: thanks for getting that poster stuff sorted out [15:12] ericsnow: no problem [15:12] ericsnow: we have a PR that didn't make it's way onto reviewboard [15:14] ericsnow: https://github.com/juju/juju/pull/4995 [15:14] voidspace: k [15:14] ericsnow: looks like you're lumbered with reviewboard issues for life... :-) [15:14] voidspace: mwahaha [15:16] Bug #1566362 opened: help text for juju add-credential needs improving