[00:17] <thumper> http://reports.vapour.ws/releases/3811/job/run-unit-tests-trusty-ppc64el/attempt/4913
[00:17] <thumper> so need to get off gccgo
[00:18] <rick_h_> thumper: around now
[00:19] <mgz> thumper: master merge, dat bug fixed I see bug 1561023
[00:19] <mup> Bug #1561023: charmstore v5.WillIncludeMetadata gccgo build failure <ci> <gccgo> <ppc64el> <juju-core:Fix Committed by gz> <https://launchpad.net/bugs/1561023>
[00:20] <thumper> rick_h_: hey, wondering if you wanted to chat about maas2 or not
[00:20] <thumper> rick_h_: I forwarded you my sentiments
[00:22] <rick_h_> thumper: yea, saw that while at the boy's school. Haven't given it a careful read yet.
[00:22] <thumper> rick_h_: so... chat or not?
[00:22] <rick_h_> thumper: sure
[00:22] <thumper> https://plus.google.com/hangouts/_/canonical.com/tim-rick?authuser=1
[00:38] <wallyworld> axw_: we should probably land that rename Server->ResolvedAPIEndpoints so it is locked in
[00:43] <anastasiamac> wallyworld: axw_: so I want to have 2 controllers, each with a hosted model
[00:43] <anastasiamac> wallyworld: axw_: so that models in both have the same name
[00:43] <wallyworld> anastasiamac: juju now creates a hosted model when bootstrapping
[00:43] <anastasiamac> wallyworld: axw_: (to see if I can kill one and still connect to the other)...
[00:43] <wallyworld> it defaults to the name "default"
[00:43] <wallyworld> imaginative right
[00:44] <wallyworld> but you can also use the --default-model <name> arg
[00:44] <anastasiamac> wallyworld: exactly what I need!
[00:44] <wallyworld> the controller model is always called "admin"
[00:44] <anastasiamac> wallyworld: but I cannot really add more models right as we r updating create-model?
[00:44] <wallyworld> but you need latest master
[00:44] <wallyworld> no, you can also use create model
[00:44] <anastasiamac> yep, the tip :D
[00:44] <wallyworld> create model has been enhance
[00:45] <wallyworld> it now no longer *requires* credentials if you are admin
[00:45] <wallyworld> if you are admin the new model inherits the credentials from the admin model
[00:45] <anastasiamac> wallyworld: reall?! even better :D
[00:45] <wallyworld> but you can also specify different ones
[00:46] <anastasiamac> wallyworld: i'll give it a spin right now! tyvm
[00:46] <wallyworld> you used to *have* to always specify credentials each time
[00:46] <wallyworld> and non admons still need to
[01:05] <menn0> thumper: this is something that came out of a discussion I had with fwereade this morning: https://github.com/nylas/stress-tester
[01:05] <menn0> not that :)
[01:05] <menn0> this: http://reviews.vapour.ws/r/4332/
[01:06] <menn0> thumper: ^
[01:06] <thumper> oh hai
[01:06] <menn0> wow, that means I hadn't copied anything into the clipboard since last night :)
[01:09] <thumper> menn0: a question
[01:10] <menn0> thumper: yes?
[01:10] <thumper> on the pr
[01:10] <thumper> well, review
[01:10] <menn0> ok
[01:12] <menn0> thumper: responded
[01:12] <thumper> shipit
[01:12] <menn0> cheers
[01:15] <wallyworld> anastasiamac: what's the status of bug 1536792
[01:15] <mup> Bug #1536792: Some providers release wrong resources when destroying hosted models <juju-core:In Progress by anastasia-macmood> <https://launchpad.net/bugs/1536792>
[01:15] <anastasiamac> wallyworld: this is what m testing now... I think it's all good but need to confirm
[01:16] <anastasiamac> I suspect most of it would have been fixed with tag fixes in joyent :D
[01:16] <wallyworld> anastasiamac: awesome, i'll mark as resolved in the release notes
[01:16] <anastasiamac> wallyworld: not yet
[01:16] <wallyworld> i'm optimistic
[01:16] <anastasiamac> wallyworld: i'll mark as resolved when m 100% sure
[01:16] <anastasiamac> gimme an hr or two? :P
[01:30] <anastasiamac> wallyworld: i have 2 controllers, both with default models
[01:30] <anastasiamac> i have added a machine to both, so each default model has a machine 0
[01:30] <anastasiamac> now i try to
[01:30] <anastasiamac> juju remove-machine 0 -m=default
[01:30] <anastasiamac> ERROR model local.tags:admin@local:=default not found
[01:30] <anastasiamac> if i do not set -m, the command runs fine
[01:30] <anastasiamac> but I can still see the machine in status :(
[01:31] <wallyworld> ok, we'll need to test and see if there's an issue
[01:31] <wallyworld> i wonder if it is getting confused because of two model names the same on different controllers
[01:31] <mup> Bug #1558333 changed: juju's logging the literal "$cmd" instead of value of $cmd <juju-log> <logging> <juju-core:Invalid> <https://launchpad.net/bugs/1558333>
[01:31] <anastasiamac> oh and m on joyent.. so it could be that joyent resources, the bug u quoted aboce is not quiet there yet..
[01:32] <wallyworld> axw_: have you seen the above issue recently? ^^^^^^
[01:32] <anastasiamac> machine in nether controller is removed...
[01:32] <anastasiamac> let me kill one controller...
[01:34] <anastasiamac> however, mayb is non-issue as both machines are shown as "pending" still..
[01:39] <anastasiamac> so i think that this a joyent issue...
[01:39] <anastasiamac> i've killed one controller
[01:39] <anastasiamac> and now, i can still list-controllers
[01:39] <anastasiamac> but both status and list-models hangs..
[01:39] <wallyworld> it may be that the current controller is still set to the one killed?
[01:39] <anastasiamac> wallyworld: so please do not mark this bug as resolved - it clearly isnt
[01:39] <anastasiamac> no, i've switched controllers once one was successfully killed!
[01:39] <wallyworld> depends - may just be joyent
[01:40] <wallyworld> we can put the caveart on there
[01:40] <wallyworld> we'll need to test with other providers to be sure
[01:40] <anastasiamac> wallyworld: i will assume it's joyen and will work on this bug 1536792
[01:40] <mup> Bug #1536792: Some providers release wrong resources when destroying hosted models <juju-core:In Progress by anastasia-macmood> <https://launchpad.net/bugs/1536792>
[01:40] <wallyworld> ok, tyvm
[01:40] <wallyworld> maybe worth a test on aws to be sure
[01:40] <wallyworld> in case we need to fix modelcmd or something
[01:42] <anastasiamac> k. i'll run the same on aws just to make sure
[01:53] <wallyworld> axw_: ping?
[01:55] <thumper> wallyworld: well... in a relatively short time I have a vmaas setup with one machine comissioned
[01:55] <thumper> with xenial and maas2
[01:55] <wallyworld> awesome sauce
[01:55] <wallyworld> thumper: uses those notes?
[01:55] <thumper> I annotated the doc with my findings
[01:55] <thumper> yeah
[01:56] <thumper> pretty trivial
[01:56] <wallyworld> thumper: well that's another thing - you can't annotate bloody videos
[01:56] <thumper> :)
[01:56] <wallyworld> i'll have to do the same
[02:04] <wallyworld> anastasiamac: probably already on your todo list - i've add a heading in What's New, can you remember to add a section further down with details? New Openstack machines can be provisioned based on virtualisation type
[02:10] <anastasiamac> wallyworld: sure, so r release notes ready from ur perspective?
[02:10] <wallyworld> more or less
[02:10] <wallyworld> still need to do a little more
[02:10] <anastasiamac> wallyworld: excellent. tyvm
[02:43] <mup> Bug #1561293 opened: remove machine model flag <juju-core:New> <https://launchpad.net/bugs/1561293>
[02:57] <wallyworld> axw: hey
[03:00] <axw> wallyworld: hey
[03:00] <wallyworld> axw: there's a section already added to release notes on CLI login (for SSO). can you add the local macaroon stuff there or in a subsection as appropriate?
[03:00] <axw> wallyworld: got the link to notes handy?
[03:00] <wallyworld> https://docs.google.com/document/d/1ID-r22-UIjl00UY_URXQo_vJNdRPqmSNv7vP8HI_E5U/edit#
[03:01] <wallyworld> axw: also see if i've forgotten something. we've added a lot
[03:01] <axw> wallyworld: sure
[03:04] <wallyworld> axw: lgtm also, much nicer, ty
[03:04] <axw> wallyworld: thanks
[03:04] <wallyworld> axw: did you want to do that rename as a drive by?
[03:04] <wallyworld> so it lands together in one go
[03:04] <axw> wallyworld: can do
[03:05] <wallyworld> would be good i think
[03:05] <wallyworld> one less thing to worry about later
[03:07] <wallyworld> axw: and if you land soon, it will get included in the next CI run, current one looking ok so far except that log rotation still broken
[03:08] <axw> wallyworld: ok. can you please take a look at the paragraph I added to "command-line login"?
[03:08] <wallyworld> sure
[03:09] <wallyworld> axw: lgtm, ty
[03:10] <axw> wallyworld: I forget what the suggestion was. UnresolvedAPIAddresses and APIAddresses?
[03:10] <wallyworld> ResolvedAPIEndpoints (instead of Servers) I think
[03:10] <wallyworld> resolved-api-endpoints for yaml
[03:10] <axw> wallyworld: Servers is the unresolved set
[03:10] <wallyworld> and then we also still have api-endpoints for the ip addresses
[03:11] <wallyworld> oh, did i get that wrong way around
[03:11] <axw> wallyworld: I think I'd rather say Unresolved for that, since it's the one that should typically not be used
[03:11] <wallyworld> sgtm
[03:12] <wallyworld> yep, just re-read my notes, i got it backwards
[03:14] <mup> Bug #1561293 changed: remove machine model flag <juju-core:Invalid> <https://launchpad.net/bugs/1561293>
[03:14] <mup> Bug #1561300 opened: juju debug-log does honor current model aside from the admin model <conjure> <juju-core:New> <https://launchpad.net/bugs/1561300>
[03:25] <axw> wallyworld: we currently render both servers and api-endpoints in show-controller. do you think we should remove the unresolved ones?
[03:25] <axw> I don't think they have any value to the user
[03:25] <wallyworld> axw: we can do and put them back if there's complaints, now is the time
[03:26] <axw> wallyworld: ok, doing that now
[03:47] <mup> Bug #1561315 opened: Ensure availability uses wrong constraints <juju-core:Triaged> <https://launchpad.net/bugs/1561315>
[03:53] <axw> wallyworld: can you take a quick look at the PR again?
[03:53] <axw> please
[03:53] <wallyworld> yup
[03:58] <wallyworld> axw: lgtm, let's hope we beat the next CI run :-)
[04:30] <wallyworld> axw: did you see the test failure?
[04:56] <axw> wallyworld: went to make lunch, not yet
[04:56] <axw> doh
[04:56] <wallyworld> np, list a small one
[04:56] <wallyworld> just
[05:02] <axw> wallyworld: there's new simplestreams in the works for azure already? does it include windows and centos?
[05:02] <wallyworld> axw: yup
[05:02] <wallyworld> not far off being ready
[05:02] <axw> wallyworld: ubuntu too?
[05:02] <wallyworld> didn't want to distract :)
[05:03] <wallyworld> axw: so yeah, believe so. the windows and centos streams will be maintained by us on streams.canonical.com, the ununtu streams on cloud-images
[05:03] <wallyworld> simplestreams search will look in 2 search paths
[05:03] <axw> wallyworld: ok
[05:04] <wallyworld> axw: i added the extra search path and signing key back in dec at oakland
[05:04] <wallyworld> has taken till now to get the streams done
[05:04]  * axw nods
[05:04] <wallyworld> all needs to be tested etc
[05:04] <wallyworld> might be issues, who knows
[05:06] <davecheney> allwatcher_internal_test.go:3066: tw.c.Assert(tw.NumDeltas(), jc.GreaterThan, 0)
[05:06] <davecheney> ... obtained int = 0
[05:06] <davecheney> ... expected int = 0
[05:06] <davecheney> golf clap
[05:14] <mup> Bug #1561339 opened: environs/sync: test failure <juju-core:New> <https://launchpad.net/bugs/1561339>
[05:32] <davecheney> menn0: https://github.com/juju/juju/pull/4887
[05:32] <davecheney> ^ urgent
[05:39] <davecheney> axw: https://github.com/juju/juju/pull/4887
[05:46] <axw> davecheney: LGTM
[05:47] <davecheney> danka, I'll hulk smash that in
[05:48] <davecheney> axw: as mwhudson noted, Juju's compiler breaking bug came early this development season
[05:48] <axw> heh :)
[05:49] <davecheney> fwiw, I don't think working around that bug made the code any worse
[05:49] <davecheney> in fact, I think it made it better
[05:49] <axw> davecheney: agreed
[05:49] <davecheney> but I make no comment on the name 'processedStatus'
[05:49] <davecheney> i was going to change to to just status
[05:49] <davecheney> but then we'd have shit like
[05:49] <davecheney> status.Status.Status =
[05:49] <axw> heh
[05:50] <davecheney> and I couldn't see through the tears to keep typing
[05:50] <axw> juju/juju/juju/Status.Status.Status
[05:50] <mwhudson> go 1.6 is in trusty-proposed
[05:51] <mwhudson> https://launchpad.net/ubuntu/+source/golang-1.6/1.6-0ubuntu1~14.04
[05:51] <davecheney> 16:50 < axw> juju/juju/juju/Status.Status.Status
[05:51] <axw> mwhudson: woohoo, thank you :)
[05:52] <davecheney> mwhudson: fantastic
[05:52] <davecheney> I think that deserves a toot of your horn on juju-dev@
[05:55] <mwhudson> one thing to note is that this package doesn't install /usr/bin/go
[05:55] <mwhudson> (rather it's /usr/lib/go-1.6/bin/go)
[05:56] <mwhudson> ah yeah
[05:58] <wallyworld> axw: doh, next CI run started without your change
[05:59] <axw> indeed
[06:00] <wallyworld> anastasiamac: my theory is correct
[06:00] <wallyworld> https://apidocs.joyent.com/cloudapi/
[06:00] <anastasiamac> \o/
[06:00] <anastasiamac> m fixing now :D
[06:00] <wallyworld> axw: the reason joyent is farked up - we need to prefix tags with "tag."
[06:00] <wallyworld> we pass in tags from ResourceTags()
[06:00] <wallyworld> eg model uuid etc
[06:00] <axw> ah, so it is a special prefix
[06:00] <wallyworld> appears so
[06:00] <axw> okey dokey
[06:01] <wallyworld> i just had an educated guess, i think it's corretc
[06:01] <wallyworld> we'll find out soon enogu, but the api doc appears to confirm it
[06:01] <axw> wallyworld: yeah, `An arbitrary set of tags can be set at provision time, but they must be prefixed with "tag."`
[06:02] <wallyworld> axw: but what i did see was an extraordinary long time between instance start up and the log entry "spaces discovery complete, client connections now allowed"
[06:02] <wallyworld> 10 minutes or thereabouts
[06:02] <axw> yikes
[06:03] <wallyworld> and also worker machines not able to talk back to start server
[06:03] <wallyworld> but one thing at a time - we'll get the tags fixed first
[06:15] <wallyworld> anastasiamac: looks like it worked :-)
[06:15] <anastasiamac> yes.. m just running couple of other tests a nd then that's it!
[06:22] <davecheney> 16:55 < mwhudson> one thing to note is that this package doesn't install /usr/bin/go
[06:22] <davecheney> 16:55 < mwhudson> (rather it's /usr/lib/go-1.6/bin/go)
[06:22] <davecheney> ^ this is a good thing, it's why we have update-alternatives
[06:23] <mwhudson> i guess you can use that too
[06:23] <mwhudson> the package could even include an alternative, but it doesn't
[06:23] <mwhudson> alternatives are a bit iffy for things that end up as build-depends
[06:24] <mwhudson> (alternatives for go went away in xenial yesterday)
[08:20] <mup> Bug #1561375 opened: state: TestRegisterNoSecretKey unreliable <juju-core:New> <https://launchpad.net/bugs/1561375>
[10:04] <dooferlad> voidspace: sorry, just noticed the time. On my way to the hangout.
[10:04] <voidspace> dooferlad: me too
[10:04] <voidspace> no babbageclunk yet
[10:04] <fwereade> voidspace, he's in there already
[10:05] <voidspace> fwereade: ah, just not on irc
[10:23] <voidspace> babbageclunk: hey, hii
[10:23] <voidspace> babbageclunk: so you're on maas 2.0 for the morning then
[10:23] <babbageclunk> voidspace: yup yup
[10:24] <babbageclunk> voidspace: might drop off the standup then
[10:24] <voidspace> babbageclunk: yeah
[10:25] <voidspace> babbageclunk: my merge of master onto the "drop-1.8" branch landed and I'm now landing that onto our maas2 branch
[10:25] <voidspace> babbageclunk: which we still haven't done anything with yet...
[10:30] <babbageclunk> voidspace: should be an easy merge then!
[10:35] <voidspace> babbageclunk: :-)
[10:39] <mgz> so, we nearly got a blessed master, one test failure off. then admin-controller-model merged.
[10:56] <voidspace> mgz: I had a load of "unknown" failures on my drop-1.8-support branch, all seemingly joyent related
[10:56] <voidspace> mgz: http://reports.vapour.ws/releases/3811
[10:57] <voidspace> mgz: plus a couple of test timeouts and a couple with known bug numbers
[10:57] <voidspace> mgz: is the joyent problem just a spurious thing, known, or something else? (do you think)
[10:57] <rogpeppe1> does anyone here understand how the new model/controller/account hierarchy works?
[10:59] <mgz> voidspace: what version of master did that branch last merge... ah, my rev, so includes admin-controller
[11:02] <voidspace> mgz: ah, they're all failing on master as well
[11:03] <voidspace> mgz: so my branch has consistently been "no worse than master"... :-)
[11:03] <mgz> voidspace: three things
[11:03] <mgz> voidspace: one, you have the new regressions from master (timeout etc across a lot of tests)
[11:03] <mgz> voidspace: two, you mis-resolved conflicts in dependencies.tsv and dropped a bug fix
[11:04] <voidspace> mgz: misresolved dependencies.tsv? I didn't resolve a conflict there
[11:04] <voidspace> mgz: and dropped a bug fix - you mean calling the test setup twice was a bug fix?
[11:04] <mgz> voidspace: three, the joyent config is borked for reasons unclear
[11:05] <voidspace> unless that was a previous conflict resolution
[11:05] <mgz> voidspace: well, it's not clear how exactly, and talking in shas makes life complicated
[11:05] <voidspace> mgz: can you point me to the dependencies.tsv issue
[11:05] <mgz> but the rev you merged includes the fix for bug 1561023
[11:05] <mup> Bug #1561023: charmstore v5.WillIncludeMetadata gccgo build failure <ci> <gccgo> <ppc64el> <juju-core:Fix Committed by gz> <https://launchpad.net/bugs/1561023>
[11:05] <voidspace> mgz: also the bugfix I dropped - I'll fix those
[11:05] <mgz> but the dependencies.tsv in your branch at that rev doesn't have the charmstore dep bump
[11:06] <mgz> so... something happened
[11:06] <mgz> it's gone on master
[11:06] <mgz> I wonder if the joyent cred issue is similar?
[11:07] <voidspace> I'm tempted to blame git, until someone can prove me wrong...
[11:07] <mgz> voidspace: well, it's certainly not wrong to blame git
[11:07] <voidspace> :-)
[11:08] <voidspace> mgz: I do have that old version in dependencies.tsv - but I don't have anything that would have conflicted with it
[11:08] <voidspace> mgz: that's an easy fix though
[11:08] <voidspace> mgz: you said screwed up dependencies.tsv *and* dropped a bug fix
[11:08] <voidspace> mgz: was that two things or one thing?
[11:09] <mgz> voidspace: merging that branch into master produces a sane diff at least
[11:09] <mgz> so, nothing to borked happened git-wise
[11:10] <mgz> voidspace: one thing, and I'm not actually sure what's up
[11:10] <voidspace> mgz: yeah, I just wonder how that dependencies.tsv line got reverted
[11:10] <voidspace> mgz: I'll update it manually in my branch - it's just concerning if things are missing
[11:11] <voidspace> mgz: the diff doesn't show it as a change against master, so it looks like it just hasn't been pulled into that branch
[11:11] <mgz> voidspace: the rev tested doesn't include the change from 1abf825dcb71f198c3895e24fecc99537454c64e ... which predates rev 9e2a02b17a92c90b524b33938b5b32bb74451538 which... aha
[11:11] <mgz> voidspace: is *not* the rev you merged
[11:12] <mgz> voidspace: so actually, just merging master again would resolve that, and likely the joyent creds issue
[11:12] <voidspace> mgz: right, and pull in new problems instead :-)
[11:12] <voidspace> mgz: maybe I'll wait until there's a bless and pull that in
[11:12] <mgz> voidspace: that would be reasonable
[11:12] <voidspace> the timeouts are worrying but I don't *think* they're consistent against test runs on my branch
[11:12] <voidspace> I'll check
[11:13] <voidspace> if they are I'll need to look at them
[11:13] <mgz> I certainly don't see any failures that are obviously from changes in that branch
[11:14] <voidspace> the changes are all in the maas provider so it would be "interesting" if it caused failures elsewhere
[11:21] <wallyworld> voidspace: what joyent creds issue are you having?
[11:23] <mgz> wallyworld: ERROR cmd supercommand.go:448 validating "credentials" credential for cloud "joyent": manta-user: expected string, got nothing
[11:23] <wallyworld> and btw joyent has been quite sick - they were upgrading their data centre, and now we have issue which are still being diagnosed, maybe network. one instance took 10 or more minutes from agent start up to get the "maas spaces all discovered" message
[11:23] <wallyworld> network spaces i mean
[11:23] <wallyworld> mgz: you need to remove the manta items
[11:23] <wallyworld> they are no longer needed
[11:23] <voidspace> wallyworld: thanks
[11:24] <wallyworld> we added the instructions to the release notes, but have not advertised yet
[11:24] <wallyworld> didn't think anyne besides Qa was using joyent :-)
[11:25] <mgz> wallyworld: this is just a sync issue between different versions of the code and those changes
[11:25] <wallyworld> ah ffs {\\\"code\\\":\\\"NotAuthorized\\\",\\\"message\\\":\\\"QuotaExceeded:
[11:25] <wallyworld> that's the latest reason for joyent failures
[11:25] <wallyworld> sigh, we're having no luck
[11:25] <mgz> wallyworld: yeah, that's some of them
[11:26] <wallyworld> mgz: the joyent provider was also misusing tags
[11:26] <wallyworld> that was done in master a while back i think
[11:26] <wallyworld> fixed in latest CI run
[11:27] <wallyworld> the machines were not being tagged properly with uuid
[11:27] <wallyworld> so ControllerInstances() was all messed up
[11:27] <wallyworld> as was bootstrap finalisation
[11:30] <babbageclunk> voidspace: ok, I've got a maas2 cluster going, I think. I might try doing the setup to get power addresses working so that I can try dooferlad's script
[11:30] <babbageclunk> voidspace: unless there's something else I should do?
[11:48] <wallyworld> mgz: voidspace: i sent an email - we have a routing issue with joyent machines, nfi what's wrong
[11:50] <mgz> wallyworld: thank you
[11:50] <wallyworld> mgz: any ideas welcome :-)
[11:51] <mgz> wallyworld: did you also look at the windows test run?
[11:51] <wallyworld> the unit test failure?
[11:51] <mgz> with admin-controller it's now hitting timeouts
[11:52] <wallyworld> i have only seen one test failure in the builds i have looked at, and it says there's an existing file handles bug
[11:52] <wallyworld> i'll look at the latest run
[11:53] <mgz> wallyworld: that does sometime happen anyway, but last three runs have all hit
[11:53] <mgz> *** Test killed: ran too long (10m0s).
[11:53] <mgz> FAIL	github.com/juju/juju/api	609.327s
[11:54] <wallyworld> mgz: nothing has changed inthe api package at all lately, and the last time in the admin comtroller branch it was the file hadnles bug
[11:55] <wallyworld> i'll look at the logs though, maybe something changed i don't know about
[11:55] <wallyworld> but i would have expected the same pass or fail now in master as in the latest admin controller runs
[11:55] <mgz> wallyworld: also http://reports.vapour.ws/releases/3813/job/run-unit-tests-race/attempt/1216#highlight
[11:55] <wallyworld> as what was run before the merge into master
[11:56] <wallyworld> i have not looked at the races at all
[11:56] <wallyworld> they have been happening for a while in master, have not been ob the radwr for us
[11:57] <mgz> wallyworld: http://reports.vapour.ws/releases/3810 <- this is my baseline
[11:57] <mgz> wallyworld: that's just before admin-controller-model merged, followed by a trivial win fix
[11:57] <wallyworld> mgz: right, but in the latest admin controller model runs we are not seeing the same issues
[11:58] <wallyworld> the only windows test failure i recall was the file handle one'
[11:58] <mgz> compare 3812 3813 3814 with the merge in
[11:58] <wallyworld> mgz: here's the latest admin controller run http://reports.vapour.ws/releases/3809
[11:58] <wallyworld> all known issues from master
[11:59] <mgz> which all timeout on the windows tests, and have a new data race
[11:59] <wallyworld> ?
[11:59] <wallyworld> bug 1521699 is not a tineout
[11:59] <mup> Bug #1521699: windows unit tests fail because handles are not available <ci> <intermittent-failure> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1521699>
[11:59] <wallyworld> that's the only windows test failure
[12:00] <wallyworld> in that last admin controller run
[12:00] <wallyworld> what are you referring to when you say timeout?
[12:01] <wallyworld> so just before we merged admin controller into master - we got a clean run apart from a few known inherited issues, so we got agreement from qa team to nerge
[12:01] <wallyworld> one of the issues is a know CI script issue - the log rotation one
[12:01] <mgz> wallyworld: http://juju-ci.vapour.ws/job/run-unit-tests-win2012-amd64/buildTimeTrend
[12:02] <wallyworld> the azure arm deploy is also known - an lxc issue on xenial
[12:02] <mgz> master was failing (with one small issue) in 30 mins
[12:02] <mgz> it's now 2 hrs
[12:02] <mgz> wallyworld: anyway, lunch...
[12:03] <wallyworld> right so we need to root cause which it is wrong to straight away blame admin controller model when that branch was clean
[12:03] <wallyworld> before merging
[12:21] <wallyworld> mgz: is the machine under load perhaps, there's things like this Panic: cannot create index for logs collection: WSARecv tcp 127.0.0.1:55607: i/o timeout (PC=0x41597B)
[12:21] <wallyworld> that's quite low level, can't immediately see the issue is caused by juju
[12:39] <voidspace> babbageclunk: ah sorry, missed your tweet
[12:39] <voidspace> babbageclunk: where are you at now?
[12:39] <babbageclunk> voidspace: no worries, unless you're going to say "Nooooo, you should have been working on something else!"
[12:40] <babbageclunk> voidspace: juuuuust about got the last piece of the power management setup done - will try using the kvm_maas script to add nodes.
[12:41] <voidspace> babbageclunk: cool
[12:41] <voidspace> babbageclunk: I'm not sure how much you can help on the API spelunking stuff I'm doing
[12:42] <voidspace> babbageclunk: getting some stuff deployed with juju on maas 1.9  might be useful
[12:42] <voidspace> babbageclunk: get familiar with the juju command line and concepts
[12:43] <babbageclunk> Ok, I'll start on that once I do this and get it written up.
[12:43] <babbageclunk> voidspace: (Also, just about to pop out and pick up some monitors.)
[12:49] <voidspace> babbageclunk: ok, cool
[13:27] <mup> Bug #1561526 opened: api/usermanager: no way to find if a user doesn't exist <juju-core:New> <https://launchpad.net/bugs/1561526>
[13:46] <cherylj> morning, everyone.
[13:46]  * cherylj reads backscroll
[13:47] <mgz> cherylj: wotcha
[13:48] <cherylj> hey mgz, how are things?  :)
[13:48] <cherylj> so ,looks like we need a bug for the github.com/juju/juju/worker/environ data race failure
[13:49] <cherylj> and we need to get someone on the stringforwarder failure for ppc
[13:50] <cherylj> and someone on the windows test timeouts
[13:50] <cherylj> and the joyent networking stuff
[13:51] <cherylj> I'll open a bug for the environ data race, unless you already did mgz ?
[13:51] <mgz> cherylj: not yet
[13:51] <rogpeppe1> anyone know of a handy mongodb key sanitizer function? nice if it was reversible.
[13:51] <cherylj> k, I'll do it now
[13:51] <cherylj> katco: do you have someone who can look at bug 1560203?
[13:52] <mup> Bug #1560203: stringForwarderSuite.TestRace sometimes fails <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1560203>
[13:52] <cherylj> katco: right now it's blocking our ability to release
[13:52] <cherylj> we seem to hit it every time
[13:52] <mgz> ah, 3814 is done now, ace
[13:53] <cherylj> blergh, katco is out
[13:53] <cherylj> natefinch, ericsnow, can either of you take bug 1560203?
[13:53] <mup> Bug #1560203: stringForwarderSuite.TestRace sometimes fails <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1560203>
[13:53] <mgz> good to bug jam on that too as it's his code :)
[13:54] <cherylj> cmars: can you spare people to help us get the release out?
[13:54] <mgz> I couldn't get the test to fail on its own, but seems as part of a full run on ppc64 it comes up a lot
[13:54] <natefinch> cherylj: katco is out today and tomorrow btw
[13:54] <mgz> so I tuink t
[13:54] <mgz> *think the 'race' assumptions are just not conservative enough
[13:56] <natefinch> cherylj: oops, sorry, I see you realized that already
[13:56] <cherylj> :)
[13:56] <mgz> cherylj: the dumb option is just skip the test
[13:57] <cherylj> hey dooferlad, how are things going with the proxy bug?  (I'm sure it will come up in the cross team call this morning)
[13:58] <natefinch> cherylj: that's some gnarly code
[13:58] <dooferlad> cherylj: moving along. Have had some useful discussions with fwereade on how to fix it properly and have some code that I am reasonably happy with.
[13:58] <cherylj> dooferlad: excellent.  Is early next week still a sane target?
[13:58] <cherylj> like Monday / Tuesday?
[13:59] <mgz> cherylj: 3814/joyent-deployer-bundle failed due to routing issues when trying to fetch tools from the state server
[13:59] <dooferlad> cherylj: yes
[13:59] <natefinch> gah, why is this code under juju/juju?
[13:59] <mgz> could be our manual firewall cleanup rules not working any more?
[13:59] <dooferlad> cherylj: though remember that Friday and Monday are public holidays (at least in the UK)
[13:59] <cherylj> dooferlad: d'oh, that's right
[14:00] <cherylj> thanks :)
[14:00] <dooferlad> cherylj: no problem :)
[14:00] <cherylj> dooferlad: you'll want to touch base with mgz to discuss CI testing with proxies for 1.25.5
[14:00] <cherylj> dooferlad: make sure that we can get a test set up for it
[14:00] <mgz> yeah, I've looked at that a little this week
[14:00] <dooferlad> cherylj: sure.
[14:01] <cherylj> muchas gracias
[14:01] <cherylj> natefinch: are you looking at that stringforwarder bug?
[14:01] <natefinch> cherylj: sort of.
[14:01] <cherylj> heh
[14:02] <natefinch> cherylj: both my managers are out, so I'm a little unclear on whether or not I should take this bug over what I was otherwise working on
[14:03] <natefinch> cherylj: on the upside (for you), it looks interesting and the code seems to need some cleanup, so it makes me want to fix it.
[14:03] <cherylj> sweet!
[14:04] <rick_h_> natefinch: what are you working on today?
[14:05] <natefinch> rick_h_: getting the resources juju code to actually support channels
[14:06] <natefinch> rick_h_: it's a "bug" that we don't :D
[14:07] <alexisb> natefinch, today we need help with bugs
[14:07] <alexisb> everyone
[14:07] <rick_h_> natefinch: ok, +1 ^
[14:07] <natefinch> alexisb: ok, thanks for the clarification
[14:07] <rick_h_> :)
[14:08] <cherylj> I'm making the bugs that are currently blocking us actual blockers, so they'll show up on juju.fail
[14:12] <natefinch> juju.fail is probably my favoritest thing that marcoceppi has ever done :)
[14:12] <cherylj> yeah, it's nice :)
[14:12] <marcoceppi> natefinch: I also have juju.qa - still waiting for a good idea to come to me
[14:13] <mgz> cherylj: I'm unsure what to do over the win2012 unit test timeout,
[14:13] <mgz> ian is right that the last run of the feature branch didn't have issues
[14:13] <mgz> but nothing else in master particularly affects that
[14:13] <cherylj> mgz: the last run of the feature branch may have been masking it
[14:13] <mgz> cherylj: my best bet may be restart the windows slave and rerun?
[14:13] <cherylj> because of the setuptest
[14:14] <mgz> that's one small test later in the process than where the timeout happens
[14:14] <cherylj> I'm suspicious of this because of the change could have brought in, namely around utils.OutgoingAccessAllowed
[14:15] <cherylj> change it could have brought in
[14:17] <natefinch> Oh man, that is so confusing. StringForwarder has a Receive() method that actually sends a message. :/
[14:17] <cherylj> lol
[14:18] <natefinch> I think I'll rename it "Forward"... since that's what it's doing
[14:21] <cherylj> bogdanteleaga: would you be able to help with bug 1561566?
[14:21] <mup> Bug #1561566: Many windows tests fail with "WSARecv tcp 127.0.0.1:53731: i/o timeout (PC=0x41593B)" <blocker> <ci> <juju-core:Triaged> <https://launchpad.net/bugs/1561566>
[14:26] <alexisb> ericsnow, ping
[14:28] <bogdanteleaga> cherylj, looks really weird, but I have a feeling I've seen it before, I think it was transient though
[14:28] <bogdanteleaga> I'll ask gabriel too
[14:29] <cherylj> thanks, bogdanteleaga!
[14:29] <mgz> cherylj: I may have some good news on that
[14:29] <cherylj> oh?
[14:29] <cherylj> that would be awesome
[14:29] <mgz> cherylj, bogdanteleaga: http://paste.ubuntu.com/15487526/
[14:31] <bogdanteleaga> mgz, if those were still around you probably wanna check the tempdir folder as well
[14:31] <mgz> yeah, rm -rf $TMP/* running
[14:32] <mgz> bogdanteleaga: what do I run to restart the (cloud) machine cleanly?
[14:33] <mup> Bug #1561555 opened: Data race in github.com/juju/juju/worker/environ <blocker> <ci> <juju-core:Triaged> <https://launchpad.net/bugs/1561555>
[14:33] <mup> Bug #1561566 opened: Many windows tests fail with "WSARecv tcp 127.0.0.1:53731: i/o timeout (PC=0x41593B)" <blocker> <ci> <juju-core:Triaged> <https://launchpad.net/bugs/1561566>
[14:35] <bogdanteleaga> mgz, shutdown -r -t 0 should do
[14:37] <wallyworld> cherylj: a small one https://github.com/juju/juju/pull/4889
[14:39] <ericsnow> alexisb: pong
[14:40] <alexisb> ericsnow, we have a push on bugs to get a very important beta3 out
[14:40] <alexisb> ericsnow, can you please work with cherylj on any needs she may have of you today
[14:41] <wallyworld> ericsnow: a +1 of this small date race fix would be great https://github.com/juju/juju/pull/4889
[14:41] <ericsnow> alexisb: k
[14:41] <natefinch> wallyworld: you should go to bed.  it's after midnight on the first day of your vacation
[14:42] <wallyworld> natefinch: soon, just one more fix :-)
[14:42] <ericsnow> alexisb: keep in mind that we still have to land the implementation of resources in the charm store in short term
[14:42] <alexisb> ericsnow, yes I know
[14:42] <ericsnow> cherylj: keep me posted on how I can help
[14:42] <natefinch> alexisb: you should make wallyworld take an extra day of vacation for every hour he works past midnight, that would teach him ;)
[14:43] <alexisb> natefinch, agreed ;)
[14:43] <ericsnow> natefinch, alexisb: lol
[14:44] <alexisb> ericsnow, natefinch this is just to support beta3 over the next 2 days
[14:44] <ericsnow> wallyworld: ship-it
[14:44] <wallyworld> tyvm
[14:44] <alexisb> ericsnow, natefinch I will work with katco and wallyworld on adjusting schedules with the 2 day impact
[14:44] <alexisb> but we need to get beta3 out
[14:45] <ericsnow> alexisb: the nice thing is that the charm store patches don't have quite the same deadlines :)
[14:45] <alexisb> ericsnow, yep :)
[14:45] <alexisb> and sorry for the shift in priority but this one is important
[14:45] <ericsnow> alexisb: not that we can affort to be complacent about them!
[14:45] <ericsnow> alexisb: np
[14:46] <wallyworld> ericsnow: ok, that should be landing, can you monitor for me? i've asked abentley to pause CI pending any more short term work to address any remaining issues; we'll need to keep him inthe loop
[14:46] <ericsnow> wallyworld: k
[14:47] <ericsnow> wallyworld: should I ping him once that's merged?
[14:47] <abentley> ericsnow: yes, please.
[14:47] <ericsnow> abentley: will do
[14:47] <wallyworld> ericsnow: just need to check what else there is - mgz did we think a windows slave restart would help the windoes tests?
[14:47] <wallyworld> there's also a potential log rotation fix
[14:48] <mgz> wallyworld: I belive so, I'm doing that now
[14:48] <wallyworld> haven't looked at that
[14:48] <wallyworld> i think the only other issue is joyent?
[14:48] <wallyworld> and its networking issue
[14:49] <wallyworld> ericsnow: mgz: it's possible TestAddUserAndRegister has an issue on windows cleaning up the logsink.log file at the end of the test. nfi why
[14:49] <mgz> wallyworld: the other big thing is joyent routing
[14:49] <wallyworld> there's nothing special in that test
[14:49] <wallyworld> so maybe we c.Skip() on windows with a todo for now
[14:50] <wallyworld> mgz: yeah, i don't know enough to fix joyent routing issues
[14:50] <mgz> http://data.vapour.ws/juju-ci/products/version-3814/joyent-deploy-trusty-amd64/build-1965/machine-0.log.gz
[14:52] <mgz> that's not actually the most helpful
[14:52] <mgz> basically... the provisioned machines can't talk back to admin:machine-0 to get tools
[14:53] <mgz> this is similar to what we added hack firewall cleanup rules to work around
[14:53] <wallyworld> mgz: yep, but only if they use the cloud address
[14:53] <mgz> it's possible those hacks need updating to the new world of admin controllers
[14:53] <wallyworld> the public address is fine
[14:53] <wallyworld> mgz: admin controller should make 0 difference
[14:53] <wallyworld> it's just a controller
[14:54] <mgz> the hack being:
[14:54] <mgz> $RELEASE_TOOLS/joyent-curl.bash /cpcjoyentsupport/fwrules | sed -e 's/[\[\{]/\n\0/g;' | grep $JOB_NAME | sed -e 's/.*"id":"\([^"]*\)".*/\1/' | xargs -I{} $RELEASE_TOOLS/joyent-curl.bash /cpcjoyentsupport/fwrules/{} -X DELETE || true
[14:54] <wallyworld> i'll take your word for it :-)
[14:54] <mgz> but given all the joyent downtime and other misc fallout from the last few days it could also be something else
[14:55] <wallyworld> ericsnow: mgz: one difference in that testAdduserAndRegister test is that it does a c.Assert(api.Close(), jc.ErrorIsNil) at the end whereas other tests just do a api.Close(). we could try that or just skip the test for now
[14:55] <wallyworld> but that's just clutching at straws
[14:55] <wallyworld> trying to find something wrong
[14:55] <wallyworld> the test itself doesn't do anything with logsink.log
[14:55] <wallyworld> it's all setup by JujuConnSuite
[14:55] <wallyworld> under the covers
[15:06] <mgz> windows machine restarted. rerunning 3814 win2012 unit tests.
[15:14] <abentley> wallyworld, mgz: cherylj and I think the joyent issues are caused by the admin model and the default model being in different regions.
[15:15] <wallyworld> hmmm, that may explain it
[15:16] <wallyworld> would need to check the code to be sure
[15:16] <ericsnow> abentley: in case you didn't notice, this merged: https://github.com/juju/juju/pull/4889
[15:16] <mgz> abentley: actual different regions? not just different network zones (or whatever joyent calls that)
[15:16] <cherylj> mgz: yeah
[15:16] <cherylj> mgz: I see that my controller is in us-east-3
[15:16] <ericsnow> abentley: first try too <wink>
[15:17] <cherylj> mgz: but when I deploy to the default model
[15:17] <wallyworld> AFAIR, there's no attempt to be in different regions, but also no attempt to be in the same region
[15:17] <abentley> wallyworld: Should we test now or hold for more fixes?
[15:17] <cherylj> mgz: it gets put in us-east-1
[15:17] <abentley> wallyworld: We specify region in the bootstrap config, so we'd expect the same one to be used for both.
[15:18] <cherylj> mgz, abentley, deploying a machine in the admin model properly puts the machine in the same region as the controller
[15:18] <wallyworld> abentley: depends on whether that is passed through, i'd need to check. but have we looked at the joyent instances to confirm?
[15:18] <cherylj> wallyworld: yes
[15:19] <wallyworld> so seems like we need to explicitly constrain the region of any hsted models
[15:19] <cherylj> wallyworld: yeah, seems to be
[15:19] <wallyworld> i'd need to check the code
[15:19] <wallyworld> can't recall exactly the setup behaviour ottomh
[15:19] <cherylj> wallyworld: I can take this, go to bed!
[15:20] <wallyworld> let me take a quick look
[15:20] <wallyworld> abentley: maybe just hold off on a new run for  bit longer
[15:20] <abentley> wallyworld: Okay, let me know.
[15:21] <abentley> wallyworld: The functional-log-rotation-unit issue also seems to be the same joyent issue.  I re-ran it on AWS and it passed.
[15:21] <wallyworld> abentley: oh good :-)
[15:21] <abentley> cherylj: ^^
[15:21] <wallyworld> so really one more issue
[15:21] <cherylj> yay
[15:27] <abentley> wallyworld: Does this cross-region apply to other providers?  We saw a bunch of admin machines all by themselves in aws earlier this week.
[15:27] <abentley> Maybe their default model machines were in a different region.
[15:27] <cherylj> ahhhh, I can check on AWS
[15:28] <cherylj> that reminds me I need to check my dashboard for other regions!!
[15:28] <wallyworld> abentley: not sure, from what i can see, the region for joyent comes from the sdc url which is the same for both admin and hosted
[15:28] <cherylj> what's the default amazon region?
[15:28] <wallyworld> us-east-1
[15:29] <abentley> wallyworld: On bootstrap, we're not allowed to pass in sdc-url.  Only region.
[15:29] <wallyworld> abentley: SDC-URL COMES FROM CREDENRIALS
[15:29] <wallyworld> caps lock fail
[15:29] <wallyworld> sorry
[15:29] <ericsnow> natefinch: if you're working on bug #1560203, would you mind assigning it to yourself?
[15:29] <cherylj> I was like woah!
[15:29] <mup> Bug #1560203: stringForwarderSuite.TestRace sometimes fails <blocker> <ci> <intermittent-failure> <test-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1560203>
[15:29] <cherylj> didn't know you felt so strongly about SDC-URL
[15:29] <wallyworld> lol
[15:29] <wallyworld> 'A' key too close to caps lock
[15:30] <natefinch> ericsnow: oops, yep, thanks
[15:30] <cherylj> lol
[15:32] <natefinch> wallyworld: you should remap caps lock to something useful... I made it the cmopose key so I can easily make things like ™ :)
[15:32] <wallyworld> :-)
[15:33] <wallyworld> cherylj: https://github.com/juju/juju/blob/master/provider/joyent/environ_instance.go#L99
[15:34] <wallyworld> the region for start instance appears to be the same for all models based on https://github.com/juju/juju/blob/master/provider/joyent/config.go#L170
[15:35] <wallyworld> and sdc-url comes from credentials
[15:35] <wallyworld> which is the same for both admin and default model
[15:35] <wallyworld> with create-model  the user can pass in different credentials so could shoot themselves in the foot
[15:36] <natefinch> ahhh.... stupid gocheck
[15:37] <natefinch> always put your func TestPackage(t *testing.T) { gc.TestingT(t) } in the internal package, not _test package, otherwise it doesn't actually run any internal tests :/
[15:37] <natefinch> some might call that a feature ;)
[15:38] <wallyworld> cherylj: so it seems confusing based on the above how machines in hosted models can be in different regions for joyent
[15:38] <cherylj> wallyworld: I'm not specifying sdc-url and I see this ssue
[15:39] <wallyworld> cherylj: let me see where sdc-url comes from
[15:39] <cherylj> wallyworld:  I'm using bootstrap joyent joyent/us-east-3
[15:40] <wallyworld> cherylj: right, so it comes from the clouds.yaml file; will be the same for all models
[15:41] <wallyworld> so i don't see off hand how machines get created in different regions
[15:41] <cherylj> There's nothing in my clouds.yaml for joyent
[15:41] <wallyworld> cherylj: fallback-public-clouds.yaml
[15:41] <wallyworld> in the code base
[15:42] <cherylj> ah
[15:42] <cherylj> wallyworld: I'll keep looking into this
[15:43] <wallyworld> ok, it annoying me that the regions are different and they shouldn't be
[15:43] <wallyworld> according to the code as i see it, but i need sleep
[15:43] <wallyworld> i'll find out i guess at the release standup :-)
[15:44] <wallyworld> cherylj: the only other thing is to consider skipping that 1 failing test on windows as per backscroll a bit ago
[15:47] <wallyworld> cherylj: func (joyentProvider) RestrictedConfigAttributes() []string {
[15:47] <wallyworld> add sdc-utl
[15:47] <wallyworld> sdc-url
[15:47] <wallyworld> to the result
[15:47] <wallyworld> that should fix it
[15:48] <wallyworld> that will force histed model config to share the sdc-utl value
[15:48] <wallyworld> with the admin model
[15:48] <cherylj> nice, thanks wallyworld
[15:49] <wallyworld> you may need to add additional ones, i think that's all that's needed
[15:49] <cherylj> ok, will test
[15:49] <wallyworld> tl;dr; - add any attrs there that must be duplicated between admind and host model
[15:49] <wallyworld> hosted
[15:50] <wallyworld> cherylj: eg for cloudsigma, the result is return []string{"region"}
[15:50] <wallyworld> so for joyent return []string{"sdc-url"}
[15:50] <wallyworld> hopefully a one line fix :-D
[15:50] <cherylj> awesome, thanks :)
[15:51] <wallyworld> yep, also "region" for ec2 :-)
[15:51] <wallyworld> joyent provider is the red headed step child
[15:51] <wallyworld> that no one loves
[15:52] <rick_h_> wallyworld: fyi, cancelled our call. Obviously you're here late and off so want to make sure you don't expect to have it :)
[15:52] <wallyworld> rick_h_: i don't mind, i need to be up 30 miuntes before hand for the relese standup
[15:52] <wallyworld> happy to still have it
[15:52] <rick_h_> wallyworld: nope, I'm not :P
[15:53] <wallyworld> rick_h_: ok, hopefuly my status email filled you in
[15:53] <rick_h_> wallyworld: rgr
[15:53] <rick_h_> wallyworld: email me if you need anything
[15:53] <wallyworld> will do
[15:53] <wallyworld> i am happy i think we got joyent sorted out
[15:53] <wallyworld> beta3 will rock
[15:54] <wallyworld> ttyl
[15:56] <wallyworld> abentley: one last thing before i go - we think there's a one line fix for joyent - cherylj will ping you when  landed
[15:59] <redir> morning
[16:05] <natefinch> someday reviewboard will pick this up: https://github.com/juju/juju/pull/4890
[16:05] <natefinch> if anyone wants to review it
[16:10] <abentley> cherylj: Here is a list of attrs that may occur in environments.yaml that we have blacklisted from the bootstrap config in 2.0: https://pastebin.canonical.com/152723/
[16:15] <cherylj> ericsnow: can you review natefinch's PR?  https://github.com/juju/juju/pull/4890
[16:15] <ericsnow> cherylj: will do
[16:15] <jam> mgz: bug me about what?
[16:16] <mup> Bug #1561611 opened: Joyent machines deployed to hosted models use wrong region <blocker> <ci> <juju-core:Triaged by cherylj> <https://launchpad.net/bugs/1561611>
[16:16] <mgz> jam: look at nate's pr ^
[16:16] <mgz> bug you about that
[16:16] <cherylj> mgz: I see the windows tests are still failing :(
[16:16] <mgz> cherylj: well, good news bad news...
[16:16] <mgz> right, that
[16:17] <mgz> go tests to run again, still super unreliable compared to before on master
[16:17] <mgz> two things under apiserver hitting 600s timeout, strongly suggests actual deadlock
[16:18] <mgz> bogdanteleaga: can you normally get a clean test run locally? if so, can you try with current master?
[16:19] <jam> natefinch: so the first thing that jumps out at me is that if you call 'Stop()' twice you'll get a panic. Its often hostile to not make cleanup actions reentrant
[16:21] <natefinch> jam: ahh you're here, great
[16:21] <bogdanteleaga> mgz, I stopped trying to run the whole thing on windows a while ago, they seem to be way slower on windows
[16:21] <natefinch> jam: I wanted to talk to you about my change, but figured you were out
[16:22] <bogdanteleaga> I remember hitting something like 299.x seconds with a 300s timeout on a suite
[16:22] <bogdanteleaga> but I'll give it a shot
[16:22] <mgz> bogdanteleaga: yeah, can do it on 30 mins on big cloud machine but it's not much use for narrowing down problems
[16:22] <mgz> bogdanteleaga: apiserver seems the interestng package
[16:22] <natefinch> jam: the reason I did it is to make it more obvious that it's not thread-safe to call stop multiple times. When I first looked at the code for Stop, I figured the nil check was an attempt to make it threadsafe (which obviously it's not)
[16:22] <natefinch> er not thread safe to call stop from multiple threads
[16:23] <mgz> master as-of before admin controller and HEAD, if they behave differently
[16:23] <bogdanteleaga> mgz, I'll try head first, you're saying apiserver times out?
[16:23] <natefinch> jam: and given that we don't actually need to call stop multiple times in production, that seems ok
[16:24] <bogdanteleaga> also what go version are you using?
[16:24] <mgz> bogdanteleaga: yeah
[16:24] <jam> natefinch: So i think it isn't all that uncommon to want to do something like defer (Stop()), and then end up with logic that might also Stop early.
[16:24] <jam> generally re-entrant cleanups are going to play nicer
[16:24] <jam> I'd rather just put a Mutex in there
[16:24] <natefinch> jam: yeah, the defer and then also call stop is true
[16:25] <mgz> bogdanteleaga: urk, that thar is a reasonable question, not sure if we've updated the window build to 1.6 yet
[16:25] <mgz> # C:/go/bin/go version
[16:25] <mgz> go version go1.2.2 windows/amd64
[16:26] <mgz> soo... that's also on our list to get done
[16:26] <bogdanteleaga> 2507 files changed, 160673 insertions(+), 117711 deletions(-)
[16:26] <bogdanteleaga> :)
[16:26] <natefinch> jam: I just kind of hate writing code for conditions that we don't actually need or care about
[16:26] <mgz> bogdanteleaga: :)
[16:27] <bogdanteleaga> mgz, are the binaries used for deployment built using 1.6 though?
[16:28] <natefinch> jam: anyway, I can add the mutex if you think that's the right fix.
[16:29] <jam> natefinch: done vs stopch is fine, New vs NewFoo is good, and I'm happy with therest
[16:29] <jam> natefinch: yeah, most is just spelling that I'm agnostic about, I'd just like Stop() to be reentrant
[16:29] <natefinch> jam: thanks... sorry for stomping all over your code
[16:29] <jam> natefinch: if its clearer for someone that isn't me, then its all good
[16:30] <mgz> bogdanteleaga: not yet I think, cross builds also on go 1.2
[16:30] <mgz> we're about to move all the remaining bits to go 1.6 though
[16:35] <natefinch> jam: do you agree with the changes to TestRace?  That's the one that was failing.  It seemed like checking that the goroutine stops before running out of runway was not actually the point of the test.
[16:35] <mgz> bogdanteleaga: basically, I'm fine signing off windows test failures for the release until we've done some of that upgrade work,
[16:36] <mgz> if you can confirm that master as of now is not vastly more borked than it was a few days ago
[16:39] <cherylj> jam, do you think that this PR could have anything to do with these windows failures?   https://github.com/juju/juju/pull/4798
[16:42] <voidspace> alexisb: you back?
[16:42] <jam> cherylj: yes, with the caveat that it is exposing brokenness in the test suite.
[16:42] <jam> cherylj: we were doing stuff like not calling SetUpSuite
[16:43] <jam> or calling PatchValue before calling SetUpTest
[16:43] <jam> which would cause the patched value to never be cleaned up.
[16:43] <jam> cherylj: IIRC mgz had a patch that changed at least one of them to correctly call SetUpSuite
[16:44] <alexisb> voidspace, I am and will be free shortly
[16:44] <alexisb> will ping
[16:44] <voidspace> alexisb: cool
[16:54] <cherylj> natefinch, ericsnow, can I get a review? http://reviews.vapour.ws/r/4340/
[16:54]  * ericsnow reviews
[16:55] <ericsnow> cherylj: LGTM
[16:57] <natefinch> jam, ericsnow: the reason I made loop into a stanadalone function is that it's a goroutine... it doesn't have any state its storing, and it shouldn't be implied that it is.  There's two separate concerns, a goroutine looping over channel, and a type that can send messages to that channel.  linking them to the same object is conceptually incorrect, and unnecessary.  Plus it means the goroutine has receive-only halves of done and messages, making it
[16:57] <natefinch> clear that it should not (and cannot) be the one closing them.
[16:59] <ericsnow> natefinch: my point was that it is confusing that way
[17:04] <natefinch> ericsnow: hmm, ok.
[17:06] <natefinch> ericsnow: my default is to make a goroutine a standalone function, since that's what it is.  I actually found it confusing that it was a method, since it didn't really need to be :)
[17:06] <natefinch> ericsnow: I can hit a middle ground I think... just share the done and messages channels in the struct
[17:07] <ericsnow> natefinch: even putting that loop function inside New() as a closure would help, though jam's point about other loop methods is still correct
[17:31] <natefinch> ericsnow: what was your oops comment about?
[17:31] <ericsnow> natefinch: you had pushed up some typos
[17:31] <natefinch> ericsnow: oh, did I fix it?
[17:31] <ericsnow> natefinch: apparently you caught them anyway :)
[17:31] <natefinch> heh
[17:33] <natefinch> ericsnow: also, the fix was actually just removing the limited for loop inside TestRace, and removing the error that happened if the goroutine went through the for loop before the test called stop
[17:34] <ericsnow> natefinch: k
[17:46] <bogdanteleaga> mgz, it was successful, but it almost timed out with several packages over 550s
[17:47] <mgz> bogdanteleaga: thanks
[17:47] <mgz> cherylj: ^I'm fine signing off windows test for a beta3
[17:47] <bogdanteleaga> mgz, trying 1.6 now
[17:48] <redir> cherylj: review please http://reviews.vapour.ws/r/4341/
[17:48] <bogdanteleaga> mgz, oh and it was apiserver only :)
[17:51] <mgz> bogdanteleaga: that should do, other things timed out in CI as well but seems like we just made juju slower on windows tests
[18:02] <natefinch> W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/source/Sources  Hash Sum mismatch
[18:02] <natefinch> cute
[18:03] <natefinch> team meeting anyone?
[18:03] <cherylj> alexisb: you going to make the is call?
[18:08] <alexisb> cherylj, sorry looks like I missed it
[18:08] <cherylj> yeah, there wasn't anything to talk about anyway.  Unless you had something
[18:15] <alexisb> nope
[18:25] <mgz> natefinch: we failed...
[18:27] <natefinch> mgz: indeed
[19:37] <redir> ericsnow, natefinch review please: http://reviews.vapour.ws/r/4341/
[19:38] <ericsnow> redir: will do
[19:55] <ericsnow> redir: done
[20:03] <redir> ericsnow: tx, I'll have a couple questions -- after a reboot.
[20:03] <ericsnow> redir: k
[20:15] <natefinch> redir: I reviewed as well.
[20:16] <natefinch> ericsnow: it's just you and me for the standup today, mind if we meet early?
[20:16] <ericsnow> natefinch: sounds good
[20:16] <ericsnow> natefinch: now?
[20:18] <natefinch> ericsnow: yep
[20:21] <redir> holler when you're off ericsnow
[20:21] <ericsnow> redir: k
[20:49] <alexisb> cherylj, I am finding my first issue testing out beta3
[20:50] <alexisb> I had a controller I created with beta2
[20:50] <alexisb> trying to kill it with beta3 fails
[20:50] <alexisb> but when I kill it with the old beta2 it works
[20:50] <cherylj> alexisb: that doesn't surprise me
[20:51] <cherylj> there have been problems like that between the other betas
[20:51] <cherylj> the expectation is that you create and destroy with the same versionj
[20:51] <alexisb> we should note it in known issues though, dont you think?
[20:51] <alexisb> or somewhere in the release notes
[20:55] <cherylj> alexisb: yeah, could say something about the different betas not guaranteed to be compatible with eachother
[20:56] <alexisb> cherylj, if you are good with saying something I can add
[21:00] <cherylj> alexisb: yeah, probably worth noting.  Thanks :)
[21:15] <wallyworld> alexisb: beta3 is not compatible with beta2
[21:16] <wallyworld> we make no guarantees of compatability
[21:16] <wallyworld> the releases notes will say as much when i update them
[21:17] <alexisb> wallyworld, you are not allowed to correct me on your vacation day
[21:17] <wallyworld> alexisb: will disappear after release standup
[21:18] <wallyworld> alexisb: we changed the format ofr controllers.yaml
[21:18] <wallyworld> better to do to now than after releases
[21:19] <alexisb> wallyworld, agreed,  I am just trying to be a 'new' user
[21:19] <wallyworld> sure np, we just need to be clear that folks need to "start again" between betas
[21:25] <redir> Am I supposed to 'resolve' issues in review board when I fix them or leave them for the original reviewer to resolve?
[21:25] <redir> somebody say new user?
[21:34] <ericsnow> redir: feel free to mark them as resolved
[21:34] <redir> ericsnow: one outstanding
[21:34] <ericsnow> redir: if you aren't sure if you've satisfied the reviewer though, it sometimes pays to leave it alone until you get more feedback
[21:35] <redir> ericsnow: your first one, I am not sure...
[21:41] <ericsnow> redir: I've dropped that one
[21:41] <redir> OK.
[21:41] <ericsnow> redir: so you should be ready to go :)
[21:41] <redir> So then does it automatically merge?
[21:41] <redir> sorry this is my first bug
[21:41] <redir> so first time through.
[21:41] <redir> Or do I merge it myself?
[21:41] <ericsnow> redir: you have to add a comment in the PR with $$merge$$ in it
[21:42] <redir> k. tx.
[21:42] <redir> And the buildbots don't run the tests before the merge?
[21:43]  * redir wonders where the build dashboard is.
[21:49] <ericsnow> redir: there's a merge bot that adds a link to the PR for the merge request and then runs the tests and does the merge for you
[21:50] <redir> awesome. so if the tests fail the merge should too. thanks ericsnow
[21:50] <ericsnow> redir: congrats on your first patch merged :)
[21:50] <ericsnow> redir: yep
[21:50] <ericsnow> redir: np
[21:50] <redir> ...insert fancy ascii art in here...
[22:08] <alexisb> i would like to point out that wallyworld did not ask about mongo
[22:08] <alexisb> in the release call
[22:08] <wallyworld> alexisb: no point :-( i saw the email