[00:17] http://reports.vapour.ws/releases/3811/job/run-unit-tests-trusty-ppc64el/attempt/4913 [00:17] so need to get off gccgo [00:18] thumper: around now [00:19] thumper: master merge, dat bug fixed I see bug 1561023 [00:19] Bug #1561023: charmstore v5.WillIncludeMetadata gccgo build failure

[00:20] rick_h_: hey, wondering if you wanted to chat about maas2 or not [00:20] rick_h_: I forwarded you my sentiments [00:22] thumper: yea, saw that while at the boy's school. Haven't given it a careful read yet. [00:22] rick_h_: so... chat or not? [00:22] thumper: sure [00:22] https://plus.google.com/hangouts/_/canonical.com/tim-rick?authuser=1 [00:38] axw_: we should probably land that rename Server->ResolvedAPIEndpoints so it is locked in [00:43] wallyworld: axw_: so I want to have 2 controllers, each with a hosted model [00:43] wallyworld: axw_: so that models in both have the same name [00:43] anastasiamac: juju now creates a hosted model when bootstrapping [00:43] wallyworld: axw_: (to see if I can kill one and still connect to the other)... [00:43] it defaults to the name "default" [00:43] imaginative right [00:44] but you can also use the --default-model arg [00:44] wallyworld: exactly what I need! [00:44] the controller model is always called "admin" [00:44] wallyworld: but I cannot really add more models right as we r updating create-model? [00:44] but you need latest master [00:44] no, you can also use create model [00:44] yep, the tip :D [00:44] create model has been enhance [00:45] it now no longer *requires* credentials if you are admin [00:45] if you are admin the new model inherits the credentials from the admin model [00:45] wallyworld: reall?! even better :D [00:45] but you can also specify different ones [00:46] wallyworld: i'll give it a spin right now! tyvm [00:46] you used to *have* to always specify credentials each time [00:46] and non admons still need to [01:05] thumper: this is something that came out of a discussion I had with fwereade this morning: https://github.com/nylas/stress-tester [01:05] not that :) [01:05] this: http://reviews.vapour.ws/r/4332/ [01:06] thumper: ^ [01:06] oh hai [01:06] wow, that means I hadn't copied anything into the clipboard since last night :) [01:09] menn0: a question [01:10] thumper: yes? [01:10] on the pr [01:10] well, review [01:10] ok [01:12] thumper: responded [01:12] shipit [01:12] cheers [01:15] anastasiamac: what's the status of bug 1536792 [01:15] Bug #1536792: Some providers release wrong resources when destroying hosted models [01:15] wallyworld: this is what m testing now... I think it's all good but need to confirm [01:16] I suspect most of it would have been fixed with tag fixes in joyent :D [01:16] anastasiamac: awesome, i'll mark as resolved in the release notes [01:16] wallyworld: not yet [01:16] i'm optimistic [01:16] wallyworld: i'll mark as resolved when m 100% sure [01:16] gimme an hr or two? :P [01:30] wallyworld: i have 2 controllers, both with default models [01:30] i have added a machine to both, so each default model has a machine 0 [01:30] now i try to [01:30] juju remove-machine 0 -m=default [01:30] ERROR model local.tags:admin@local:=default not found [01:30] if i do not set -m, the command runs fine [01:30] but I can still see the machine in status :( [01:31] ok, we'll need to test and see if there's an issue [01:31] i wonder if it is getting confused because of two model names the same on different controllers [01:31] Bug #1558333 changed: juju's logging the literal "$cmd" instead of value of $cmd

[01:31] oh and m on joyent.. so it could be that joyent resources, the bug u quoted aboce is not quiet there yet.. [01:32] axw_: have you seen the above issue recently? ^^^^^^ [01:32] machine in nether controller is removed... [01:32] let me kill one controller... [01:34] however, mayb is non-issue as both machines are shown as "pending" still.. [01:39] so i think that this a joyent issue... [01:39] i've killed one controller [01:39] and now, i can still list-controllers [01:39] but both status and list-models hangs.. [01:39] it may be that the current controller is still set to the one killed? [01:39] wallyworld: so please do not mark this bug as resolved - it clearly isnt [01:39] no, i've switched controllers once one was successfully killed! [01:39] depends - may just be joyent [01:40] we can put the caveart on there [01:40] we'll need to test with other providers to be sure [01:40] wallyworld: i will assume it's joyen and will work on this bug 1536792 [01:40] Bug #1536792: Some providers release wrong resources when destroying hosted models [01:40] ok, tyvm [01:40] maybe worth a test on aws to be sure [01:40] in case we need to fix modelcmd or something [01:42] k. i'll run the same on aws just to make sure [01:53] axw_: ping? [01:55] wallyworld: well... in a relatively short time I have a vmaas setup with one machine comissioned [01:55] with xenial and maas2 [01:55] awesome sauce [01:55] thumper: uses those notes? [01:55] I annotated the doc with my findings [01:55] yeah [01:56] pretty trivial [01:56] thumper: well that's another thing - you can't annotate bloody videos [01:56] :) [01:56] i'll have to do the same [02:04] anastasiamac: probably already on your todo list - i've add a heading in What's New, can you remember to add a section further down with details? New Openstack machines can be provisioned based on virtualisation type [02:10] wallyworld: sure, so r release notes ready from ur perspective? [02:10] more or less [02:10] still need to do a little more [02:10] wallyworld: excellent. tyvm [02:43] Bug #1561293 opened: remove machine model flag [02:57] axw: hey [03:00] wallyworld: hey [03:00] axw: there's a section already added to release notes on CLI login (for SSO). can you add the local macaroon stuff there or in a subsection as appropriate? [03:00] wallyworld: got the link to notes handy? [03:00] https://docs.google.com/document/d/1ID-r22-UIjl00UY_URXQo_vJNdRPqmSNv7vP8HI_E5U/edit# [03:01] axw: also see if i've forgotten something. we've added a lot [03:01] wallyworld: sure [03:04] axw: lgtm also, much nicer, ty [03:04] wallyworld: thanks [03:04] axw: did you want to do that rename as a drive by? [03:04] so it lands together in one go [03:04] wallyworld: can do [03:05] would be good i think [03:05] one less thing to worry about later [03:07] axw: and if you land soon, it will get included in the next CI run, current one looking ok so far except that log rotation still broken [03:08] wallyworld: ok. can you please take a look at the paragraph I added to "command-line login"? [03:08] sure [03:09] axw: lgtm, ty [03:10] wallyworld: I forget what the suggestion was. UnresolvedAPIAddresses and APIAddresses? [03:10] ResolvedAPIEndpoints (instead of Servers) I think [03:10] resolved-api-endpoints for yaml [03:10] wallyworld: Servers is the unresolved set [03:10] and then we also still have api-endpoints for the ip addresses [03:11] oh, did i get that wrong way around [03:11] wallyworld: I think I'd rather say Unresolved for that, since it's the one that should typically not be used [03:11] sgtm [03:12] yep, just re-read my notes, i got it backwards [03:14] Bug #1561293 changed: remove machine model flag [03:14] Bug #1561300 opened: juju debug-log does honor current model aside from the admin model

[03:25] wallyworld: we currently render both servers and api-endpoints in show-controller. do you think we should remove the unresolved ones? [03:25] I don't think they have any value to the user [03:25] axw: we can do and put them back if there's complaints, now is the time [03:26] wallyworld: ok, doing that now [03:47] Bug #1561315 opened: Ensure availability uses wrong constraints [03:53] wallyworld: can you take a quick look at the PR again? [03:53] please [03:53] yup [03:58] axw: lgtm, let's hope we beat the next CI run :-) === Spads_ is now known as Spads [04:30] axw: did you see the test failure? [04:56] wallyworld: went to make lunch, not yet [04:56] doh [04:56] np, list a small one [04:56] just [05:02] wallyworld: there's new simplestreams in the works for azure already? does it include windows and centos? [05:02] axw: yup [05:02] not far off being ready [05:02] wallyworld: ubuntu too? [05:02] didn't want to distract :) [05:03] axw: so yeah, believe so. the windows and centos streams will be maintained by us on streams.canonical.com, the ununtu streams on cloud-images [05:03] simplestreams search will look in 2 search paths [05:03] wallyworld: ok [05:04] axw: i added the extra search path and signing key back in dec at oakland [05:04] has taken till now to get the streams done [05:04] * axw nods [05:04] all needs to be tested etc [05:04] might be issues, who knows [05:06] allwatcher_internal_test.go:3066: tw.c.Assert(tw.NumDeltas(), jc.GreaterThan, 0) [05:06] ... obtained int = 0 [05:06] ... expected int = 0 [05:06] golf clap [05:14] Bug #1561339 opened: environs/sync: test failure [05:32] menn0: https://github.com/juju/juju/pull/4887 [05:32] ^ urgent [05:39] axw: https://github.com/juju/juju/pull/4887 [05:46] davecheney: LGTM [05:47] danka, I'll hulk smash that in [05:48] axw: as mwhudson noted, Juju's compiler breaking bug came early this development season [05:48] heh :) [05:49] fwiw, I don't think working around that bug made the code any worse [05:49] in fact, I think it made it better [05:49] davecheney: agreed [05:49] but I make no comment on the name 'processedStatus' [05:49] i was going to change to to just status [05:49] but then we'd have shit like [05:49] status.Status.Status = [05:49] heh [05:50] and I couldn't see through the tears to keep typing [05:50] juju/juju/juju/Status.Status.Status [05:50] go 1.6 is in trusty-proposed [05:51] https://launchpad.net/ubuntu/+source/golang-1.6/1.6-0ubuntu1~14.04 [05:51] 16:50 < axw> juju/juju/juju/Status.Status.Status [05:51] mwhudson: woohoo, thank you :) [05:52] mwhudson: fantastic [05:52] I think that deserves a toot of your horn on juju-dev@ [05:55] one thing to note is that this package doesn't install /usr/bin/go [05:55] (rather it's /usr/lib/go-1.6/bin/go) [05:56] ah yeah [05:58] axw: doh, next CI run started without your change [05:59] indeed [06:00] anastasiamac: my theory is correct [06:00] https://apidocs.joyent.com/cloudapi/ [06:00] \o/ [06:00] m fixing now :D [06:00] axw: the reason joyent is farked up - we need to prefix tags with "tag." [06:00] we pass in tags from ResourceTags() [06:00] eg model uuid etc [06:00] ah, so it is a special prefix [06:00] appears so [06:00] okey dokey [06:01] i just had an educated guess, i think it's corretc [06:01] we'll find out soon enogu, but the api doc appears to confirm it [06:01] wallyworld: yeah, `An arbitrary set of tags can be set at provision time, but they must be prefixed with "tag."` [06:02] axw: but what i did see was an extraordinary long time between instance start up and the log entry "spaces discovery complete, client connections now allowed" [06:02] 10 minutes or thereabouts [06:02] yikes [06:03] and also worker machines not able to talk back to start server [06:03] but one thing at a time - we'll get the tags fixed first [06:15] anastasiamac: looks like it worked :-) [06:15] yes.. m just running couple of other tests a nd then that's it! [06:22] 16:55 < mwhudson> one thing to note is that this package doesn't install /usr/bin/go [06:22] 16:55 < mwhudson> (rather it's /usr/lib/go-1.6/bin/go) [06:22] ^ this is a good thing, it's why we have update-alternatives [06:23] i guess you can use that too [06:23] the package could even include an alternative, but it doesn't [06:23] alternatives are a bit iffy for things that end up as build-depends [06:24] (alternatives for go went away in xenial yesterday) [08:20] Bug #1561375 opened: state: TestRegisterNoSecretKey unreliable [10:04] voidspace: sorry, just noticed the time. On my way to the hangout. [10:04] dooferlad: me too [10:04] no babbageclunk yet [10:04] voidspace, he's in there already [10:05] fwereade: ah, just not on irc [10:23] babbageclunk: hey, hii [10:23] babbageclunk: so you're on maas 2.0 for the morning then [10:23] voidspace: yup yup [10:24] voidspace: might drop off the standup then [10:24] babbageclunk: yeah [10:25] babbageclunk: my merge of master onto the "drop-1.8" branch landed and I'm now landing that onto our maas2 branch [10:25] babbageclunk: which we still haven't done anything with yet... [10:30] voidspace: should be an easy merge then! [10:35] babbageclunk: :-) [10:39] so, we nearly got a blessed master, one test failure off. then admin-controller-model merged. [10:56] mgz: I had a load of "unknown" failures on my drop-1.8-support branch, all seemingly joyent related [10:56] mgz: http://reports.vapour.ws/releases/3811 [10:57] mgz: plus a couple of test timeouts and a couple with known bug numbers [10:57] mgz: is the joyent problem just a spurious thing, known, or something else? (do you think) [10:57] does anyone here understand how the new model/controller/account hierarchy works? [10:59] voidspace: what version of master did that branch last merge... ah, my rev, so includes admin-controller [11:02] mgz: ah, they're all failing on master as well [11:03] mgz: so my branch has consistently been "no worse than master"... :-) [11:03] voidspace: three things [11:03] voidspace: one, you have the new regressions from master (timeout etc across a lot of tests) [11:03] voidspace: two, you mis-resolved conflicts in dependencies.tsv and dropped a bug fix [11:04] mgz: misresolved dependencies.tsv? I didn't resolve a conflict there [11:04] mgz: and dropped a bug fix - you mean calling the test setup twice was a bug fix? [11:04] voidspace: three, the joyent config is borked for reasons unclear [11:05] unless that was a previous conflict resolution [11:05] voidspace: well, it's not clear how exactly, and talking in shas makes life complicated [11:05] mgz: can you point me to the dependencies.tsv issue [11:05] but the rev you merged includes the fix for bug 1561023 [11:05] Bug #1561023: charmstore v5.WillIncludeMetadata gccgo build failure

[11:05] mgz: also the bugfix I dropped - I'll fix those [11:05] but the dependencies.tsv in your branch at that rev doesn't have the charmstore dep bump [11:06] so... something happened [11:06] it's gone on master [11:06] I wonder if the joyent cred issue is similar? [11:07] I'm tempted to blame git, until someone can prove me wrong... [11:07] voidspace: well, it's certainly not wrong to blame git [11:07] :-) [11:08] mgz: I do have that old version in dependencies.tsv - but I don't have anything that would have conflicted with it [11:08] mgz: that's an easy fix though [11:08] mgz: you said screwed up dependencies.tsv *and* dropped a bug fix [11:08] mgz: was that two things or one thing? [11:09] voidspace: merging that branch into master produces a sane diff at least [11:09] so, nothing to borked happened git-wise [11:10] voidspace: one thing, and I'm not actually sure what's up [11:10] mgz: yeah, I just wonder how that dependencies.tsv line got reverted [11:10] mgz: I'll update it manually in my branch - it's just concerning if things are missing [11:11] mgz: the diff doesn't show it as a change against master, so it looks like it just hasn't been pulled into that branch [11:11] voidspace: the rev tested doesn't include the change from 1abf825dcb71f198c3895e24fecc99537454c64e ... which predates rev 9e2a02b17a92c90b524b33938b5b32bb74451538 which... aha [11:11] voidspace: is *not* the rev you merged [11:12] voidspace: so actually, just merging master again would resolve that, and likely the joyent creds issue [11:12] mgz: right, and pull in new problems instead :-) [11:12] mgz: maybe I'll wait until there's a bless and pull that in [11:12] voidspace: that would be reasonable [11:12] the timeouts are worrying but I don't *think* they're consistent against test runs on my branch [11:12] I'll check [11:13] if they are I'll need to look at them [11:13] I certainly don't see any failures that are obviously from changes in that branch [11:14] the changes are all in the maas provider so it would be "interesting" if it caused failures elsewhere [11:21] voidspace: what joyent creds issue are you having? [11:23] wallyworld: ERROR cmd supercommand.go:448 validating "credentials" credential for cloud "joyent": manta-user: expected string, got nothing [11:23] and btw joyent has been quite sick - they were upgrading their data centre, and now we have issue which are still being diagnosed, maybe network. one instance took 10 or more minutes from agent start up to get the "maas spaces all discovered" message [11:23] network spaces i mean [11:23] mgz: you need to remove the manta items [11:23] they are no longer needed [11:23] wallyworld: thanks [11:24] we added the instructions to the release notes, but have not advertised yet [11:24] didn't think anyne besides Qa was using joyent :-) [11:25] wallyworld: this is just a sync issue between different versions of the code and those changes [11:25] ah ffs {\\\"code\\\":\\\"NotAuthorized\\\",\\\"message\\\":\\\"QuotaExceeded: [11:25] that's the latest reason for joyent failures [11:25] sigh, we're having no luck [11:25] wallyworld: yeah, that's some of them [11:26] mgz: the joyent provider was also misusing tags [11:26] that was done in master a while back i think [11:26] fixed in latest CI run [11:27] the machines were not being tagged properly with uuid [11:27] so ControllerInstances() was all messed up [11:27] as was bootstrap finalisation [11:30] voidspace: ok, I've got a maas2 cluster going, I think. I might try doing the setup to get power addresses working so that I can try dooferlad's script [11:30] voidspace: unless there's something else I should do? [11:48] mgz: voidspace: i sent an email - we have a routing issue with joyent machines, nfi what's wrong [11:50] wallyworld: thank you [11:50] mgz: any ideas welcome :-) [11:51] wallyworld: did you also look at the windows test run? [11:51] the unit test failure? [11:51] with admin-controller it's now hitting timeouts [11:52] i have only seen one test failure in the builds i have looked at, and it says there's an existing file handles bug [11:52] i'll look at the latest run [11:53] wallyworld: that does sometime happen anyway, but last three runs have all hit [11:53] *** Test killed: ran too long (10m0s). [11:53] FAIL github.com/juju/juju/api 609.327s [11:54] mgz: nothing has changed inthe api package at all lately, and the last time in the admin comtroller branch it was the file hadnles bug [11:55] i'll look at the logs though, maybe something changed i don't know about [11:55] but i would have expected the same pass or fail now in master as in the latest admin controller runs [11:55] wallyworld: also http://reports.vapour.ws/releases/3813/job/run-unit-tests-race/attempt/1216#highlight [11:55] as what was run before the merge into master [11:56] i have not looked at the races at all [11:56] they have been happening for a while in master, have not been ob the radwr for us [11:57] wallyworld: http://reports.vapour.ws/releases/3810 <- this is my baseline [11:57] wallyworld: that's just before admin-controller-model merged, followed by a trivial win fix [11:57] mgz: right, but in the latest admin controller model runs we are not seeing the same issues [11:58] the only windows test failure i recall was the file handle one' [11:58] compare 3812 3813 3814 with the merge in [11:58] mgz: here's the latest admin controller run http://reports.vapour.ws/releases/3809 [11:58] all known issues from master [11:59] which all timeout on the windows tests, and have a new data race [11:59] ? [11:59] bug 1521699 is not a tineout [11:59] Bug #1521699: windows unit tests fail because handles are not available

[11:59] that's the only windows test failure [12:00] in that last admin controller run [12:00] what are you referring to when you say timeout? [12:01] so just before we merged admin controller into master - we got a clean run apart from a few known inherited issues, so we got agreement from qa team to nerge [12:01] one of the issues is a know CI script issue - the log rotation one [12:01] wallyworld: http://juju-ci.vapour.ws/job/run-unit-tests-win2012-amd64/buildTimeTrend [12:02] the azure arm deploy is also known - an lxc issue on xenial [12:02] master was failing (with one small issue) in 30 mins [12:02] it's now 2 hrs [12:02] wallyworld: anyway, lunch... [12:03] right so we need to root cause which it is wrong to straight away blame admin controller model when that branch was clean [12:03] before merging [12:21] mgz: is the machine under load perhaps, there's things like this Panic: cannot create index for logs collection: WSARecv tcp 127.0.0.1:55607: i/o timeout (PC=0x41597B) [12:21] that's quite low level, can't immediately see the issue is caused by juju [12:39] babbageclunk: ah sorry, missed your tweet [12:39] babbageclunk: where are you at now? [12:39] voidspace: no worries, unless you're going to say "Nooooo, you should have been working on something else!" [12:40] voidspace: juuuuust about got the last piece of the power management setup done - will try using the kvm_maas script to add nodes. [12:41] babbageclunk: cool [12:41] babbageclunk: I'm not sure how much you can help on the API spelunking stuff I'm doing [12:42] babbageclunk: getting some stuff deployed with juju on maas 1.9 might be useful [12:42] babbageclunk: get familiar with the juju command line and concepts [12:43] Ok, I'll start on that once I do this and get it written up. [12:43] voidspace: (Also, just about to pop out and pick up some monitors.) [12:49] babbageclunk: ok, cool [13:27] Bug #1561526 opened: api/usermanager: no way to find if a user doesn't exist [13:46] morning, everyone. [13:46] * cherylj reads backscroll [13:47]