[00:03] wallyworld: tyvm for the review. You're right about the logging/error handling. I'll improve it. [00:03] menn0: bp, i got distracted and didn't let you know i had done ir, sorry [00:03] *np [00:04] wallyworld: np, I was busy with the next thing anyway [01:19] wallyworld: are you working on any of those bugs? just don't want to double up [01:20] axw: i had good intentions but haven't started yet [01:20] are they legit issues? [01:20] wallyworld: nps. yes, I think so. [01:20] ok [01:28] wallyworld: actually, create-model would not work on master for manual either [01:28] wallyworld: but meh, I can fix it on admin-controller-model with one line change [01:28] axw: that was the point i tried to make and they were not wanting to ack that [01:28] ok that would be good [01:32] axw: there is also a bug for create-model on azure bug 1558769 [01:32] Bug #1558769: unable to create-model in azure [01:33] axw: is it related and will b fixed by the one line change above? :D [01:33] anastasiamac: I saw, thank you. I think wallyworld already fixed it, but I'll double check [01:33] \o/ [01:33] anastasiamac: nah, the manual one is specific to manual [01:33] axw: thnx :D [01:43] wallyworld: sorry, actually, the problem is worse on admin-controller-model. you can't even bootstrap, because we try to create a hosted model automatically [01:44] axw: understood. but root cause comes form master [01:44] the test would have seem it if they had existed [01:44] seen [01:44] wallyworld: you can't create-model on master, you can't even bootstrap on this branch. yes, we would have caught it earlier if we tested create-model in all substrates [01:46] axw: indeed. but i want to snure the bug is correctly targetted - i am tired of pushback to fix issues in feature branches which blocks work where the root cause comes from master [01:47] oh yeah i found https://tracker.debian.org/pkg/golang-golang-x-tools yesterday i think [01:48] whoops [02:38] wallyworld: just a few more small things on the PR [02:38] ok [02:39] axw: the maas agent-name thing is because we grab the controller model and end up calling PrepareForCreateEnvironment() which doesn't like the agent name being there [02:39] i need to rework it a bit [02:39] wallyworld: okey dokey [02:40] wallyworld: I thought we didn't copy across attrs if they were restricted? [02:40] or ... non restricted [02:41] axw: that could be - but it seems we do now [02:41] likely a bug from me supporting new create model [02:44] wallyworld: possibly also in the code I added to agent/agentbootstrap [02:44] yeah, it won't taje much to fix i don't think [03:13] axw: http://reviews.vapour.ws/r/4222/ [03:14] wallyworld: looking [03:14] wallyworld: I'm looking at the other bug btw [03:14] ta [03:15] wallyworld: I think your change means the credentials won't be copied across [03:15] wallyworld: and things from --config [03:16] isn't the stuff in --config in HostedModelConfig arg? [03:16] I don't think so, but I'll check [03:16] i may need to do credentials though [03:16] wallyworld: nope, it's not [03:16] damn, what is in that arg? [03:17] wallyworld: model name, uuid [03:17] wallyworld: it's set in cmd/juju/commands/bootstrap.go [03:17] hmmm, ok. may need to add the --config to that arg then maybe [03:17] wallyworld: and credentials... [03:17] yup [03:18] wallyworld: problem is how to separate credentials from things like maas-agent-name [03:18] on bootstrap we don't need to worry right? [03:19] and it's taken care of in create model that is used by create-model command [03:20] maas-aganr-name is add in PrepareForCreateEnvironment. on bootstrap, it won't be there in passed in config, or will error if it is. in create-model we create a skeleton config [03:20] wallyworld: how are you going to get the credential attrs? [03:21] need to look into that [03:21] wallyworld: rhetorical question. the answer is (currently) to call EnvironProvider.BootstrapConfig [03:22] wallyworld: that adds the credential attrs to the config given a cloud.Credential [03:22] ok [03:22] wallyworld: ... but it also adds maas-agent-name [03:22] ah damn ok [03:23] wallyworld: I think what we could is this: call RestrictedConfig, make sure we include those, and delete anything that's not passed in via --config [03:23] except for credentials maybe [03:24] wallyworld: I was thinking they're restricted... but yeah, they're not [03:24] le sigh [03:24] sigh indeed, i found out the same thing [03:36] axw: it's all good. all i need to do is include --config in the hosted model config arg. (c ModelConfigCreator) NewModelConfig does the right thing with credentials [03:38] wallyworld: ah, I see [03:39] wallyworld: uhm, although, it's kinda wrong :/ [03:39] wallyworld: those credential attributes are not supposed to map 1:1 [03:40] they sort of do atm in common usage [03:40] but they don't have to that's true [03:40] wallyworld: they do atm, but it's an abstraction breakage, and means we're buggered if we change something [03:40] yep [03:41] wallyworld: I think we may need something on EnvironProvider to identify which things to carry across [03:41] was a quick win, we need to fix the tech deby [03:41] solves the common case for admins creating new models [03:41] wallyworld: or we update PrepareForCreateEnvironment to take a Credential too [03:41] that might be better [03:42] wallyworld: or ... maybe separate BootstrapConfig even further to convert Credential to attrs [03:42] or that [03:43] wallyworld: main problem is that some cloud.Credentials only make sense at the client (e.g. ones that refer to files) [03:43] indeed [03:43] tho we should convert them [03:43] we need to think it through [03:44] wallyworld: https://bugs.launchpad.net/juju-core/+bug/1558803/comments/1 [03:44] Bug #1558803: Manual deploy on ppc64el wants wrong package and agents

[03:44] the file values are parsed server side, need to do that client side [03:45] did you find an issue? [03:45] wallyworld: a pretty minor one - see comment [03:50] wallyworld: I've regargeted and lowered importance [03:50] axw: sorry, got pinged still to look [03:51] axw: it's not even relevant to admin controller branch is it [03:51] wallyworld: no, I've removed that [03:51] ah cool [03:51] wallyworld: and set the medium, and removed regression tag [03:51] * wallyworld hits refresh [03:52] axw or wallyworld: http://reviews.vapour.ws/r/4223/ please [03:52] menn0: i can look in a bit, just wip atm [03:53] wallyworld: thanks [03:53] in that case, I'll go get lunch [03:54] wallyworld: quick ping. Do you remember how to tell a test what tools to use? It looks like EC2 tests were relying on side-effects from some other test to set up tools [03:58] jam: that has recently changed, we were relying on cloud (provider storage) which is gone. rthere are helpers to uplod tools to state [03:58] i can look up the methods [03:59] wallyworld: so with my recent change to fix SetUp stuff, provider/ec2 is not finding tools. It looks like the FakeVersionNumber is 1.99.0 but it only sees 2.0-beta3 tools in the faked out location. [03:59] looks like something is faking out tools before we fake out the version number [03:59] trying to sort out where that may be [04:00] jam: i'll take a look, i am not deeply familiar as i didn't see the final code [04:00] environs/jujutest/livetests.go SetUpTest looks like it is doing the right thing (calls FakeVersionNumber before it calls UploadFakeTools) [04:00] wallyworld: k, I have the feeling this is old semi-rotten code [04:00] axw: pushed new changes to that maas fix [04:00] ah, maybe its me [04:00] I moved one of the PatchValue calls [04:01] ah [04:01] because it was happening before SetUp [04:01] joy [04:01] which is unsafe, because we haven't called IsolationSuite.SetUp yet [04:01] a lot of the tests rely on patching version [04:01] Bug #1558901 opened: TestAddLocalCharmSuccess read has been closed

[04:01] so we haven't told it what test we're about to run. I think I can just patch for the lifetime of the Suite and we'll be ok [04:02] i think that souuds ok [04:02] nope... :( [04:03] wallyworld: we had a lot of tests that were doing "SetUpSuite() { base.SetUpTest() }" [04:03] those were interesting. [04:03] wot? [04:03] wow [04:03] i hope i didn't write any [04:04] wallyworld: FakeXDGHomeDir was one of them [04:04] apparently broken since 2014 according to blame. [04:04] huh. thats actually old code renamed [04:04] wallyworld: yeah, a bunch of our base infrastructure was actually wrong. [04:04] at yeast it will be right now :-) [04:05] wallyworld: BaseSuite.SetUpSuite() was calling PatchValue(utils.OutsideAccess) [04:05] lol yeast. i can' type [04:05] which would be reset on the first test that ran [04:05] except we leaked it somewhere early [04:05] so it was always false [04:05] oh dear [04:05] wallyworld: axw: before I can bootstrap master tip on my openstck, I need to add-cloud and i guess add-credential [04:05] so it was Correct, but for the wrong reasons. [04:05] anyway, provider/ec2 is the last bastion of failing tests, so I'm close. [04:06] where do I put auth-url? in my-cloud.yaml or my-credential.yaml? [04:06] anastasiamac: that's a credential attribute [04:06] k [04:06] anastasiamac: if you have a novarc and master, it will auto detect [04:06] wallyworld: auth-url isn't that what URL to hit which is a cloud attribute? [04:06] (souds like a per-region attribute) [04:07] jam: we have endpoint url which i think is different but i haven't look specifically at openstack for a bit so could be misremembering [04:08] jam: but i think you may be right [04:08] wallyworld: I have novarc but do I need to have it somehwere in a special place? [04:09] ~/.novarc [04:09] :D [04:09] so in this case, I do not need to add-credentials? [04:10] Can I get a review? https://github.com/juju/juju/pull/4783 [04:10] (this is for the restore failure on master) [04:11] jam: yes, i looked at the code and confirmed you are right [04:11] cherylj: looking [04:12] thanks, wallyworld. I had to shuffle things around to be able to mock out stuff for testing [04:12] turns out that there really aren't tests for restore :( [04:15] wallyworld: ... provider/ec2/ LocalLiveTests embeds LiveTests but doesn't call LiveTests.SetUpTest() [04:15] win [04:16] anastasiamac: if you have ~/.novarc, or if you just source your novarc in your shell, you can do "juju bootstrap openstack" [04:16] cherylj: only ci tests it seems [04:16] yeah [04:16] wallyworld: eating lunch, will look again soon [04:17] np, thanks [04:17] cherylj: why is there a "fakeEnviron" in live code? [04:18] jam: for mocking out the environ in the test. We only set it in tests [04:18] wallyworld: actually I can review that while eating :) [04:18] LGTM [04:18] axw: i have both ~/.novarc and sourced it in the shell, but when I "juju bootstrap openstack" I get "ERROR missing auth-url not valid [04:18] " [04:19] anastasiamac: that would suggest you're missing OS_AUTH_URL from the file [04:19] anastasiamac: where did you get the novarc file from? [04:19] cherylj: have you done manual testing of an environment that really hasn't ever existed? [04:19] axw: it would... but I don't [04:19] it's in the file [04:20] that was supplied [04:20] oh, that's odd [04:20] anastasiamac: does it have "export OS_AUTH_URL" in it per chance? [04:20] anastasiamac: wallyworld fixed a bug yesterday to do with that [04:20] :q [04:20] oops [04:21] axw: yes, everything in the file has an export... and m running from master tip as of 1hr ago :D [04:22] cherylj: i've made a request to alter things slightly to better set up for testing [04:22] anastasiamac: ok then, I dunno. sounds like a bug [04:23] axw: anastasiamac: it's a bug that's been there for a while [04:23] anastasiamac: try moving the file away from ~/.novarc and sourcing it, and see if that makes a difference [04:23] DetectRegions() only calls CredentialsFromEnv() [04:23] cherylj: reviewed. [04:23] not the new DetectCredentials() code [04:23] wallyworld: anastasiamac said she sourced it as well tho [04:24] don't believe it :-) [04:24] since that should have worked [04:24] wallyworld: right.... [04:24] unless there's a 3rd problem [04:25] i think openstack has been used with beta2 which is where sourcing the novarc would have been required [04:25] wallyworld: m so calling u for that! [04:35] jam: any idea why SetInstanceStatus is so slow? it's all local, so should be super fast :( [04:35] axw: I don't know specifically, but IIRC the spec wanted to rate limit charms calling it [04:35] given how accurate it is to 1.0s [04:36] I think its just a time.Sleep somewhere. [04:37] jam: ok, I'll dig. we should probably use the ratelimit package with a token bucket [04:38] well, it could be that, too. I just get the feeling it is explicitly limiting itself, which interacts poorly with fast downloads and a set number of events [04:54] wallyworld: and now the test suite "passes" but a test is taking 60s, investigation looks like it is accessing my EC2 credentials and launching a real instance... [04:55] (ec2.localLiveSuite) not so local [04:55] jam: localLive tests, from memory, are run both local and live [04:55] that distinction goes waaaaaay back [04:56] i wish we dropped live tests, we don't need them now [04:56] i guess they were added for juju 0.1 before we had CI [05:06] wallyworld: Can you take another look? http://reviews.vapour.ws/r/4224/ [05:07] wallyworld: Also, I think that the backup / restore commands should be controller commands, not model commands [05:07] but that's outside the scope of this PR [05:09] cherylj: looking [05:13] cherylj: thanks for that fix. ideally we'd not have the func as an arg to NewRestoreCommand. There'd be a NewRestoreCommandForTest() in export_text. see detectcredentials.go in juju/cmd/juju/cloud and also the NewDetectCredentialsCommandForTest in export_test.go [05:15] wallyworld: ok, I see. Give me a few and I'll update it [05:22] wallyworld: okay, maybe 3rd time's the charm [05:22] :-) [05:22] looking [05:26] cherylj: very small issue, lgtm [05:26] ha, sorry it's taking so long. I'm so very tired [05:29] wallyworld: hm, looks like it isn't talking to EC2, just that we have a 60s timeout waiting for a security group that is being used to be released [05:30] maybe a setup was supposed to be changing that timeout [05:30] (just ran in offline mode and had the same result) [05:30] could be [05:30] we have so many tests with clocks to fix [05:30] cherylj: indeed, you should not be at work [05:31] and with that... bedtime [05:31] see ya [05:32] axw: also, i did end up needing to remove all the CompatSalt stuff yesterday, as bootstrap still relied on it and it needed to be killed [05:32] wallyworld: cool [05:32] wallyworld: does it need re-reviewing? [05:32] axw: nah, was 99% cut [05:32] i tested live [05:33] wallyworld: sounds good [05:33] i tested i could log into mongo console from machine 0 as well as a deployment [05:54] axw: awesome, the interactive add credential branch doesn't appear to compile on go 1.2 [05:55] wallyworld: ? :/ [05:55] compiles and runs locally [05:55] http://juju-ci.vapour.ws:8080/job/github-merge-juju/6912/console [05:55] i'll have to get go 1.2 to repro [05:56] unless i'm being dumb [05:57] wallyworld: nothing obvious to me [05:57] sigh [05:59] wallyworld: k. there is definitely a problem with 60s to destroy, but it exists in Master as well, so my code is better than previous [06:00] jam: i've found and fixed a few bad tests lately too. our code has many :-( [06:08] Bug #1558924 opened: provider/ec2 localLiveSuite.TestGlobalPorts localLiveSuite.TestStartStop takes 60s

[07:25] axw: am merging master into admin-controller-model... soooo many conflicts \o/ [07:25] will have to finish after soccer [07:25] wallyworld: thank you [07:26] wallyworld: my current status is refactoring modelcmd/juju code in between bouts of sneezing [07:26] need to refactor so I can pass in alternative credentials [07:26] ok [07:27] i still need to install go 1.2 and resolve that other issue also [08:07] axw: is this ur juju allergies making u sneeze? :P [08:08] anastasiamac: naw, got a sleep-deprivation induced cold I think [08:10] axw: :( sounds awful [09:22] frobware: ping [09:22] voidspace: pong [09:22] voidspace: I pushed your branch [09:22] frobware: yeah, I see it [09:23] frobware: no CI run yet though [09:23] voidspace: as upstream/drop-maas-1.8-support-from-juju2 [09:23] voidspace: nope... [09:23] frobware: I'm writing up a high-level sketch of the maas 2 work [09:23] voidspace: but then again our multi-nic branch only only started running this morning [09:24] voidspace: thanks! \o/ [09:24] frobware: the gomaasapi test server needs not far short of a full rewrite, so that's not a small chunk of work [09:24] :-/ [09:24] frobware: I think a new one, with some shared code, will be cleaner than a single implementation or an interface based approach [09:26] voidspace: given the time remaining is that at all feasible? Or even desirable? [09:28] frobware: I think it's the only reasonable approach [09:28] frobware: the TestServer has about thirty attributes, most of which won't be used for 2.0 [09:29] frobware: so a monster with 60 fields and branches in every method will be impossible to work on [09:29] frobware: that's what I think [09:29] frobware: a lot of the code can be shared (all the endpoint stuff) [09:29] frobware: but maybe there's a middle ground somewhere [09:30] wallyworld: https://github.com/juju/juju/pull/4761 but for some reason rb didnt pick it, perhaps the conflict? [09:30] voidspace: realistically I think we should consider that middle ground, otherwise the potential of adding new bugs is a concern [09:31] frobware: I don't think hugely branching code is less likely to introduce bugs [09:31] that's my worry [09:32] anyway [09:32] morning [09:32] TheMue: o/ [09:42] voidspace, frobware, dimitern, dooferlad: would one of you please check the AddressAllocation test changes in jujud/agent, in http://reviews.vapour.ws/r/4110/diff/5/?page=2 ? [09:43] ashipika, by the way, I am deeply suspicious of what github thinks about that branch -- I am planning to just apply the correct diff to MADE-model-workers in a new branch -- will that inconvenience you horribly? [09:44] fwereade: nah, not really.. just give me the new branch and i'll rebase :) [09:45] ashipika, cool, just a heads up :) [09:45] fwereade: thanks! [09:45] fwereade: when do you expect this to land? [09:46] ashipika, I have a ship it, so hopefully today [09:46] fwereade: \o/ [09:47] ashipika, although that's just onto MADE-model-workers, which needs to be updated with latest trunk and then make CI happy [09:48] fwereade, looking [10:03] frobware: stdup? [10:03] voidspace: oops, sidetracked. omw [10:10] dimitern, for reference: I dropped the skipped tests, and added one test that enables the feature flag and checks there's an address-cleaner worker running alongside all the usual ones [10:10] dimitern, I called it address-cleaner because that seemed to be its main job, let me know if I misunderstood anything [10:33] fwereade, that sounds good to me [10:34] dimitern, ok, cool, will be submitting it as RB4225 in a few seconds then :) [10:37] fwereade, go for it :) [10:37] frobware, voidspace, there's the fix for those panics: http://reviews.vapour.ws/r/4226/ [10:58] dimitern: will stash the lxd changes and try your PR now [10:58] frobware, cheers [11:00] dimitern: do you know why in the container the device are not ifup'd? [11:01] dimitern: I now have 3 nics, but without an ifup they have no addrs [11:01] frobware, hmm no idea - it works like that for lxc [11:01] frobware, maybe something is blocking the networking job to finish bringing them up? ntpdate lock contention? [11:02] dimitern: will take a look after trying your changes [11:02] frobware, ok [11:03] frobware, btw I've realized we're lagging behind 2 major feature branch merges into master [11:03] dimitern: I think we need to decide whether it makes sense to create one singular network profile, or one profile per container [11:04] dimitern: this is an entropy judgement call [11:04] frobware, one per container seems to be the simplest option, considering the hwaddr needs to match [11:04] dimitern: in my mind we should get the multi-nic branch back to the state of maas-spaces2, CI-wise. [11:04] frobware, I'm looking into how bad the conflicts are [11:05] dimitern: then look at merging master [11:06] frobware, that's a good point [11:06] frobware, but if we lag too much behind it will only get harder [11:07] dimitern: I know. but equally adding the unknown is ... [11:07] dimitern: I think there's benefit in giving the OS folks a known state today [11:08] dimitern: yes, the hwaddr forces that decision. [11:09] frobware, I wouldn't rush to do a merge today exactly for that matter [11:09] http://paste.ubuntu.com/15413949/ [11:11] dimitern: so entropy level at lines 1..3624. :) [11:12] frobware, that's BS though as I did reset --hard FETCH_HEAD before, redoing it properly now [11:15] frobware, it's entirely not that bad: http://paste.ubuntu.com/15413972/ [11:17] dimitern: well, that's a lot better [11:17] frobware, it even builds, running make check now and a live test with the bundle [11:17] dimitern: I think we need to decide how to spend the rest of the day; polish the branch and give it to OS folks, or merge and polish [11:20] frobware, I think those two are not necessarily mutually exclusive [11:21] frobware, OS folks can test the maas-spaces-multi-nic-containers + my proposed fix and the workaround bouncing the MA of machine-0 [11:21] dimitern: ah, you still need to bounce the MA [11:21] frobware, yeah [11:22] frobware, if you're deploying containers on machine 0 [11:22] dimitern: right [11:23] dimitern: we can merge master; we can also give the OS folks a specific commit for testing. that would allow us to move on. thoughts? [11:23] frobware, we can give them a binary tarball to try, while we sync up with master (and assuming that does not cause a regression for multi-nic stuff) [11:24] dimitern: thedac seemed happy with checkout a commit id last time; it's also easier if we want them to move to a different commit [11:24] frobware, fair enough [11:24] frobware, but otherwise I think that sounds like the plan for today? [11:25] dimitern: great [11:25] dimitern: ah... one other thing... [11:25] dimitern: if we can first resolve the missing mac addr that would allow lxd end-to-end [11:26] frobware, right! [11:26] dimitern: can you look at that next? [11:26] frobware, I'll look into that when I'm back ~1h [11:26] dimitern: thanks! [11:26] ashipika, MADE-model-workers is up to date with the model-agent-integration work [11:26] and I'll leave the live test and make check going in the mean time [11:27] dimitern: I have to say this is why I don't want to merge master: http://reports.vapour.ws/releases/3776 [11:27] dimitern: multi-nic came from the tip of maas-spaces2 and that had only 2 failures [11:27] dimitern: we should really get back to that state first [11:29] frobware, well, we touched a lot of places in the multi-nic branch [11:30] frobware, voidspace, btw a review on http://reviews.vapour.ws/r/4226/ will be appreciated :) [11:30] * dimitern is out for ~1h [11:30] fwereade: that's a feature branch i presume? [11:32] dimitern: no issues with what we've changed; just for our own sanity bringing new stuff may make it harder to differentiate root cause analysis [11:34] ashipika, yeah [11:36] dimitern: getting closer with lxd: http://pastebin.ubuntu.com/15414021/ [11:38] dimitern: cannot login using juju ssh but can ordinarily. error fetching address for machine "0/lxd/0": private no address [11:41] voidspace: CI run 3777 is the drop-1.8 support. [12:01] frobware: ah, cool [12:02] frobware: lots blocked and a couple of build-binary failures [12:02] frobware: I'm looking at the failures [12:04] frobware: problems with lxc-start in the build binary [12:04] voidspace: log? link? [12:05] frobware: http://data.vapour.ws/juju-ci/products/version-3777/build-binary-wily-amd64/build-884/consoleText [12:05] frobware: same problem on build-binary-wily-arm64 [12:05] doesn't *look* like a problem with the code [12:05] the logs are "terse" though [12:06] voidspace: check status of master too [12:06] looking [12:06] frobware: they succeeded on master [12:07] frobware: my guess is that other tests will be gated on the binary build being successful [12:07] mgz: ping [12:07]