[00:09] <davecheney> menn0: fair enough
[00:09] <davecheney> there is an option for running tests within a package in parallel
[00:09] <davecheney> but
[00:09] <wallyworld__> axw: trivial review when you have a moment http://reviews.vapour.ws/r/825/
[00:09] <davecheney> a. it's not the default (nobody writes tests taht can be run concurrently)
[00:09] <davecheney> b. you have to specifically enable it on a per tets by test basis
[00:10] <davecheney> c. it wouldn't matter because gocheck makes everything look like one test case anyway
[00:11] <menn0> davecheney: thanks
[00:16] <menn0> thumper, wallyworld__ : possible fix to the CI blocker. http://reviews.vapour.ws/r/826/
[00:19] <thumper> menn0: how confident are you that it fixes it?
[00:19] <thumper> it certainly looks like a candidate
[00:20]  * thumper is munching then free for the call
[00:20] <thumper> ah shoot
[00:20] <thumper> menn0: I forgot about my chat with wallyworld__
[00:20] <thumper> wallyworld__: what is your schedule like?
[00:20] <wallyworld__> i can wait
[00:20] <menn0> thumper: well I can't replicate the problem locally, it's timing dependent. it only seems to fail for i386 and ppc in CI.
[00:20] <thumper> cool cool
[00:20] <wallyworld__> ping when ready
[00:20] <jw4> menn0: I can try it on my slow ec2 instance
[00:21] <menn0> thumper: but there was definitely a problem which this PR fixes and I can see how the bug could lead to the problems we're seeing.
[00:21] <jw4> menn0: although after reviewing the fix my error might be unrelated
[00:21] <menn0> jw4: that would be great
[00:22] <menn0> jw4: the error you added to the bug could definitely be related to this fix
[00:22] <jw4> menn0: cool - I'll spin it up and try
[00:22] <menn0> jw4: thank you
[00:53] <jw4> menn0: hmm - first test run without your change did not fail - I'll try again
[01:03] <thumper> wallyworld: chat?
[01:03] <menn0> jw4: I would expect the problem to be somewhat intermittent
[01:03] <jw4> yup
[01:04] <axw> wallyworld: can you just change the timeout to be based on the bootstrap-timeout?
[01:04] <axw> wallyworld: it's configurable
[01:16] <thumper> wallyworld: I'm in the hangout, but I'm going to make a coffee
[01:29] <jw4> menn0: still not repro'ing it - i'm gonna try with an i386 image
[01:30] <menn0> jw4: ok thanks
[01:31] <jw4> menn0: otherwise we'll just have to watch for it in the CI servers
[01:31] <menn0> jw4: the CI unit test runs with the fix in are about to finish so that might tell us more as well
[01:31] <jw4> menn0: +1
[01:32] <menn0> jw4: this is the key one to watch: http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-i386/1327/console
[01:32] <menn0> jw4: the jujud tests have already passed which is a good sign
[01:33] <jw4> sweet
[02:06] <thumper> wallyworld: missed you, I'm off to the vet, bbl
[02:06] <wallyworld> thumper: np, i'm off to buy a new farking network adaptor
[02:06] <wallyworld> can't stand these disconnects
[02:39] <menn0> wallyworld: how common is it for the jujud test runs to time out on the (apparently slower) i386 CI hosts?
[02:39] <menn0> wallyworld: because a lot of runs just did
[02:40] <wallyworld> not sure, but often i suspect
[02:40] <wallyworld> our tests timeout on my laptop
[02:49] <wallyworld> axw: pr updated
[02:50] <axw> wallyworld: just noticed. LGTM
[03:05] <axw> wallyworld: your default size change is ineffective, sorry I didn't pick up on it earlier. your change just alters a temporary
[03:05] <axw> I'll fix it in my branch
[03:08] <wallyworld> ah ball ok, thanks
[03:08] <wallyworld> that's what i get for rushing
[03:12] <thumper> wallyworld: you back?
[03:13] <wallyworld> yep
[03:13] <wallyworld> haven't left yet
[03:13] <thumper> chat?
[03:13] <wallyworld> sure
[03:39] <axw> wallyworld: https://github.com/wallyworld/juju/pull/21
[03:39] <wallyworld> looking
[03:46] <wallyworld> axw: gh not letting me comment. but i think we need allcons[name] = cons after cons.Pool = defaultPool
[03:49] <axw> wallyworld: ah yeah, I'll fix it
[03:49] <wallyworld> ta
[03:51] <axw> wallyworld: updated
[03:51] <wallyworld> thanks, merging
[04:43] <thumper> night all
[04:57] <wallyworld_> menn0: in case i don't catch you later, see you in capetown
[04:58] <wallyworld_> axw: small problem to add to todo list - storage constraints reject pool names with "_" eg fast_ebs
[04:58] <axw> wallyworld_: yeah, we need to decide what's a valid name. I just copied what was valid for service names
[04:59] <wallyworld_> np, i'll add to spreadsheet
[05:25] <wallyworld_> axw: i can't get ebs volumes with provisioned iops to work - the ec2 instance doesn't attach any ebs volumes and doesn't finish coming up in AWS
[05:26] <axw> wallyworld_: just about to propose UML changes, will take a look in a sec
[05:26] <wallyworld_> that's using volume-type = provisioned-iops
[05:26] <wallyworld_> and iops = 2000 or whatever
[05:26] <wallyworld_> i blame aws
[05:26] <mattyw> folks - is it possible to change the juju admin secret in a deployed environment?
[05:26] <mattyw> also - morning
[05:27] <wallyworld_> not sure
[05:27] <axw> wallyworld_: there is a ratio between size and IOPS that you need to adhere to, could be that
[05:27] <wallyworld_> ah ok
[05:28] <axw> wallyworld_: http://reviews.vapour.ws/r/828/
[05:31] <axw> morning mattyw. I don't think so, but not 100% sure
[05:36] <axw> wallyworld_: I'm going to $$__JFDI__$$ for hopefully obvious reasons :)
[05:36] <wallyworld_> yeah, np
[05:36] <wallyworld_> i ned to land the maas fix also
[05:37] <wallyworld_> i'm off for a bit - going to computer shop to get network card
[05:38] <axw> later
[05:39] <axw> will let you know how I go with IOPS
[05:49] <mattyw> axw, morning - I was in the process of bringing the env up - so I decided to just destroy and start again
[06:39] <wallyworld_> axw: anastasiamac: i think we should enhance storage list to display the provider type
[06:39] <anastasiamac> axw: wallyworld_: sounds good but let me merge my stuff 1st
[06:39] <anastasiamac> axw: wallyworld_: i can add it next :D
[06:41] <wallyworld_> sure :-)
[06:42] <axw> wallyworld_: provider type, or pool?
[06:42] <axw> wallyworld_: I was thinking pool...
[06:42] <wallyworld_> axw: both maybe. as a user, i'd like to see provider type as well
[06:43] <axw> wallyworld_: I'd hope the pools are named to make it obvious which provider is involved
[06:43] <wallyworld_> that's up the the user :-)
[06:43] <axw> well if you know what pool to select when you deploy... you should also know what it is when you list them
[06:44] <wallyworld_> supporting "_" would help
[06:44] <axw> that only holds if the person listing and deploying are the same tho
[06:44] <axw> wallyworld_: why? you can use "-"
[06:44] <wallyworld_> i agree in principal, but yes, as you just said
[06:44] <wallyworld_> true
[06:44] <wallyworld_> i guess i'm an old python guy at heart
[06:44] <axw> :)
[07:22] <axw> wallyworld_: https://github.com/wallyworld/juju/pull/23
[07:23] <axw> wallyworld_: did you need any other docs or anything for next week, or is that UML update sufficient?
[07:24] <axw> wallyworld_: btw I just confirmed that with a pool having iops=3000, I can deploy with a 100GiB volume
[07:24] <axw> otherwise the block device mapping seems to get silently eaten up...
[07:24] <axw> that branch just beefs up the validation
[07:40] <mattyw> davecheney, ping?
[08:15] <davecheney> mattyw: ack
[11:16] <dimitern> voidspace, jamestunnicliffe, TheMue, fix for bug 1416134 - please take a look http://reviews.vapour.ws/r/830/
[11:16] <mup> Bug #1416134: Unable to override network-bridge if container type is kvm (local provider) <cloud-installer> <config> <lxc> <network> <regression> <cloud-installer:Confirmed
[11:16] <mup> for adam-stokes> <juju-core:In Progress by dimitern> <juju-core 1.21:Triaged> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1416134>
[11:33] <dimitern> voidspace, http://reviews.vapour.ws/r/831/
[11:43] <dimitern> TheMue, http://reviews.vapour.ws/r/832/
[11:45] <dimitern> thanks for the reviews voidspace and TheMue!
[11:45] <TheMue> dimitern: yw
[12:11] <perrito666> morning
[12:58] <perrito666> wallyworld_: still here
[12:58] <perrito666> ?
[13:00] <wallyworld_> hi, about to sleep, i catch a taxi soon
[13:01] <perrito666> wallyworld_: the short answer is tounit <-- testing purposes
[13:02] <wallyworld_> sure, but i think the function signature is wrong
[13:02] <perrito666> you mean my findEntity?
[13:02] <wallyworld_> yeah - why not have it return a StatusSetter
[13:03] <perrito666> wallyworld_: because it needs to satisty an interface to be used by common statusetter?
[13:03] <wallyworld_> right, so change that
[13:03] <wallyworld_> the operation is to call SetStatus
[13:03] <wallyworld_> the different is how the thing to set status on is found
[13:04] <wallyworld_> for machines and units, shared logic can be used
[13:04] <wallyworld_> for unit agents, the Agent() method is used
[13:04] <wallyworld_> anyways, have a think about it, i gotta get a few hours sleep before i catch th plane
[13:05] <perrito666> go, have a nice trip
[13:05] <wallyworld_> i might be off base too, so see if it makes sense looking at the code
[13:05] <perrito666> we can re-discuss this via mail whenever you are more awake, cheers
[13:05] <wallyworld_> ty, see you later
[14:04] <voidspace> anyone know anything about the proxyupdateworker?
[14:11] <voidspace> wallyworld_: ping if you're still awake at this horrendous hour
[14:13] <fwereade_> voidspace, might do a little, I touched it semi-recently
[14:13] <fwereade_> voidspace, I am 80% convinced that it's fundamentally dangerous
[14:14] <fwereade_> voidspace, but it was a reduce-duplication driveby, not a make-tings-right
[14:22] <voidspace> fwereade_: so this worker updates the .juju-proxy file with the http(s) (etc) settings from the config
[14:22] <fwereade_> voidspace, yeah
[14:22] <voidspace> fwereade_: so these values are propagated from environments.yaml into .juju-proxy and /home/ubuntu/.profile is updated to source this file
[14:22] <fwereade_> voidspace, it's the env vars that that http module uses that worry me
[14:22] <voidspace> fwereade_: however the environment variables are never set in the jujud process
[14:23] <voidspace> fwereade_: so http.DefaultClient (which uses these environment variables) does not see the proxy settings
[14:23] <voidspace> fwereade_: so if they are needed (which is usually why they are set...) everything fails
[14:23] <voidspace> fwereade_: so my intention is that the proxyupdater should also set the environment variables for the process
[14:24] <voidspace> fwereade_: https://bugs.launchpad.net/juju-core/+bug/1403225
[14:24] <mup> Bug #1403225: charm download behind the enterprise proxy fails <cloud-installer> <deploy> <proxy> <sync-tools> <cloud-installer:Confirmed for adam-stokes> <juju-core:Triaged by mfoord> <https://launchpad.net/bugs/1403225>
[14:24] <fwereade_> voidspace, proxyupdater.go:151?
[14:24] <perrito666> lol, just found in the code
[14:24] <perrito666> 	// TODO(fwereade) GAAAAAAAAAAAAAAAAAH this is LUDICROUS.
[14:24] <fwereade_> :)
[14:24] <voidspace> fwereade_: ah right, but only the first time - right
[14:25] <voidspace> fwereade_: interesting. I can attach to jujud and see that they are *not* set.
[14:25] <fwereade_> voidspace, I think that says "always on first call or if they've changed since last time"?
[14:27] <fwereade_> voidspace, best guess is that jujud changes somehow stopped it from being started properly?
[14:27] <voidspace> fwereade_: hmmm... indeed. Well, it doesn't seem to be "working". I'll do some debugging. Thanks.
[14:27] <fwereade_> voidspace, I thought I had tests for that, but possibly I screwed them up
[14:28] <voidspace> no problem, I know where I'm looking now - so it should be easy from here ;-)
[14:28] <sinzui> jam, dimitern: We have a licensing bug to address. bug 1416425 which has an easy fix I think
[14:28] <mup> Bug #1416425: src/bitbucket.org/kardianos/osext/LICENSE is wrong <licensing> <packaging> <juju-core:Triaged> <juju-core 1.21:Triaged> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1416425>
[14:28] <sinzui> jam dimitern: Can you ask someone to address the licensing bug
[14:30] <fwereade_> voidspace, (also: it's started in both the unit agent and the machine agent, please check both)
[14:30] <voidspace> fwereade_: yep
[14:31] <sinzui> also dimitern , or anyone, why did the license to juju/cmd change?
[14:31] <dimitern> sinzui, i'll have a look
[14:32] <dimitern> sinzui, so is this about using rev 44140c5 in dependencies.tsv for bitbucket.org/kardianos/osext - in 1.23, 1.22, and 1.21 ?
[14:35] <sinzui> dimitern, yes, tip has the fix
[14:36] <perrito666> fwereade_: ftr, the ensuing line is indeed ludicrous
[14:36] <perrito666> cmd := exec.Command("go", "build", "github.com/juju/juju/cmd/jujud")
[14:36]  * fwereade_ knows :(
[14:40] <dimitern> sinzui, ok, I'll prepare a fix for it
[14:50] <dimitern> oh boy.. I found a bug with retry-provisioning
[14:51] <dimitern> using it on a failed container in EC2 caused a new *instance* to be provisioned for "0/lxc/0"
[14:52] <dimitern> how do you like this http://paste.ubuntu.com/9957564/
[15:12] <sinzui> dimitern, Sorry, several more licensing bugs were found. Can you find people to look into the issues. they all affect 1.23, 1.22, and 1.21 https://bugs.launchpad.net/juju-core/+milestone/1.21.2
[15:13]  * sinzui return to the broken publish-revision job
[15:14] <dimitern> sinzui, I can fix the osext one, but unfortunately we're close to finishing the sprint here and somebody else can take over
[15:16] <dimitern> alexisb, hey, are you around?
[15:27] <coreycb> does anyone have any tips on getting around a "virbr0": net: no such interface error on bootstrap of local provider on 1.21.1?
[15:28] <dimitern> coreycb, do you have virbr0 on your machine?
[15:28] <coreycb> dimitern, nope
[15:28] <coreycb> dimitern, I know I could create it, but wondering more about if my juju config is wrong
[15:28] <dimitern> coreycb, and you're using container: kvm ?
[15:28] <coreycb> dimitern, yes
[15:29] <dimitern> coreycb, looks like you're affected by bug 1416134 which I just fixed this morning :)
[15:29] <mup> Bug #1416134: Unable to override network-bridge if container type is kvm (local provider) <cloud-installer> <config> <lxc> <network> <regression> <cloud-installer:Confirmed for adam-stokes> <juju-core:In Progress by dimitern> <juju-core 1.21:Fix Committed by dimitern> <juju-core 1.22:Fix Committed
[15:29] <mup> by dimitern> <https://launchpad.net/bugs/1416134>
[15:29] <coreycb> dimitern, sweet, I think :)
[15:30] <coreycb> dimitern, thanks
[15:30] <dimitern> coreycb, in the mean time you can either create a virbr0 or use "network-bridge" setting but lxcbr0 there won't work due to that bug
[15:30] <coreycb> dimitern, ok, thanks
[15:31] <dimitern> np :)
[15:33] <dimitern> voidspace, http://reviews.vapour.ws/r/833/, http://reviews.vapour.ws/r/834/, http://reviews.vapour.ws/r/835/ PTAL :)
[15:46] <bodie_> natefinch around?
[15:46] <perrito666> bodie_: no natefinch today
[19:42] <menn0> sinzui: ping
[19:42] <sinzui> hi menn0
[19:43] <menn0> sinzui: so it's hard to know if I've fixed the CI blocker because we seem to be hitting test timeouts for the i386 and ppc unit test runs
[19:43] <menn0> sinzui: I know my recent changes added a few seconds to the cmd/jujud/agent test run (on my machine at least)
[19:44] <menn0> sinzui: I wonder if we were close to the 20min threshold on the i386 and ppc CI machines and I've just tipped it over?
[19:45] <sinzui> menn0, I am not user. but looking at the FAIL: in the tests, these are very different from what we usually see
[19:46] <sinzui> menn0, and the ppc test are run very quickly
[19:47] <menn0> sinzui: run-unit-tests-trusty-ppc64el and run-unit-tests-precise-i386 seem to be hitting timeouts I don't see fails there
[19:48] <sinzui> menn0: the fact that we are still seeing panic means something is wrong. We know that some tests will fail, but pass when retried, but we don't see panics
[19:48] <menn0> sinzui: since the last fix was committed the only panics I see are about tests taking too long
[19:48] <sinzui> menn0, the 386 tests  ridiculously slow. we cannot get  a fast cloud images
[19:49] <menn0> sinzui: can you point me at another panic?
[19:49] <sinzui> menn0, http://data.vapour.ws/juju-ci/products/version-2286/run-unit-tests-trusty-ppc64el/build-2281/consoleText
[19:49] <menn0> sinzui: that's because "panic: test timed out after 20m0s"
[19:50] <menn0> sinzui: go test triggers a panic because the timeout set for the test run was exceeded
[19:50] <sinzui> menn0, but about that 386. We have a fast trusty 386 machine, but the tests don't pass because of two issues. I reported a bug about this. If the two tests are fixed or *skipped* you will see a passs http://data.vapour.ws/juju-ci/products/version-2286/run-unit-tests-trusty-i386/build-164/consoleText
[19:51] <sinzui> bug 1408459
[19:51] <mup> Bug #1408459: pingerSuite tests consistenly fail on trusty 386 <ci> <i386> <intermittent-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1408459>
[19:53] <menn0> sinzui: ok, I see that and we need to fix that. but that's an unrelated issue to the current CI blocker.
[19:53] <menn0> sinzui: what I'd like is if we could bump up the test timeout to 25 min or something to see if the panics related to the CI blocker have been resolved.
[19:54] <sinzui> menn0,the machine is healthy
[19:54] <sinzui> and I can see it doesn't need a updated
[19:54] <sinzui> menn0, and we can see that 1.21 and 1.22 pass
[19:55] <sinzui> menn0, so I think master + gccgo have a problem that 1.21 doesn't
[19:55] <menn0> sinzui: I'm not saying there's anything wrong with the machine. I think that the addition of extra tests in trunk recently has pushed us over the timeout 20min timeout for the machine agent tests.
[19:57] <sinzui> menn0, the whole suite when it was passing take 37 to 44 minutes to run
[19:58] <sinzui> menn0, this is the last health mast run http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/run-unit-tests-trusty-ppc64el/2263/
[19:59] <sinzui> disregard the tar timestamp issue. I fixed that this morning
[20:02] <menn0> sinzui: ok, that's useful
[20:02]  * menn0 checks how the timeout works
[20:02] <sinzui> oh good, because I am running out of thinks to look for
[20:07] <jw4> sinzui: are there any plans to get the tests working on windows and osx ?
[20:08] <menn0> sinzui: ok... so i'm pretty sure the timeout is per Go package and it's looking like on i386 and ppc64 the cmd/jujud/agent package is now taking over 20mins
[20:09] <sinzui> jw4 yes We test the client ever revision and we run windows units every revision http://reports.vapour.ws/releases/2286
[20:09] <menn0> sinzui: that's a massive jump from the last successful runs
[20:09] <menn0> sinzui: so it's not just tests taking a little bit longer... something is stuck
[20:09] <menn0> sinzui: but these tests pass fine on amd64
[20:09] <sinzui> menn0, yep
[20:09] <menn0> sinzui: am I ok to use those hosts later on today
[20:09] <menn0> ?
[20:10] <menn0> sinzui: I think I still have the details to access the ppc64 host that you gave me some time ago
[20:10] <sinzui> jw4, when there is a way to for juju to provision a windows machine and deploy an agent we will add that test to ever revision
[20:11] <jw4> sinzui: cool.  That link doesn't work with my ubuntu credentials, probably because I'm not a cougar (<-- is that even a thing ? ;) )
[20:11] <sinzui> menn0, oh, you my have visited a different machine even if you had. these tests are run on stilson-08. I can add your ssh keys if needed
[20:11] <jw4> sinzui: I'm just curious how they're being run because on the surface they're not even close to running on my windows and osx machines
[20:12]  * menn0 checks
[20:12] <sinzui> jw4, only canonical staff can see the results, but the raw results are public
[20:12] <jw4> sinzui: kk - tx
[20:12] <menn0> sinzui: that's what I have access for and I can still get in
[20:13] <sinzui> jw4, the last 2 tests are windows client and units http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/
[20:13] <sinzui> fab
[20:13] <menn0> sinzui: access to the i386 host might be useful too if that's possible
[20:13] <sinzui> menn0, we spin up an instance for that
[20:13] <jw4> sinzui: sweet - thanks again
[20:14] <menn0> sinzui: ok, I can spin up my own then.
[20:14] <sinzui> menn0, but I can get you acesss to the machine we want to run tests on. again, about that 386 bug. when those tests pass, we will remove the old 386 job
[20:14] <menn0> sinzui: ok
[20:15] <jw4> sinzui: hmm the last 'successful' run of win-client-deploy doesn't seem to be actually successful : http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/win-client-deploy/1381/console
[20:16] <sinzui> jw4, looks like a successful bootstrap to me
[20:16] <jw4> sinzui: it probably is - I just see a strange error about 5 lines up from the bottom
[20:17] <sinzui> jw4, The SUCCESS was on the win machine the other lines are from ssh and the test running. we don't what them cursing the win success
[20:17] <sinzui> and ssh did bugger up again
[20:18] <jw4> sinzui: I see - also I noticed that the run-unit-tests-win2012-amd64 is getting the same '/usr/lib/juju/bin/mongod' file does not exist error that I'm getting on my windows tests
[20:19] <menn0> sinzui: do you have any details or a script for spinning up the precise and trusty i386 test hosts?
[20:19] <menn0> sinzui: I want to make sure I replicate as closely as possible
[20:21] <sinzui> menn0, juju-ci-tool/run-unit-tests m1.medium ami-81dee0e8
[20:21] <menn0> sinzui: perfect. thanks. i've got a few things to take care of right now but I'll jump back on this today.
[20:22] <sinzui> menn0, aws is the only provider that permits 386 and they limit it to a slow machine, and aws is phasing out 386, which is why we need to switch to our special machine
[20:22] <menn0> sinzui: got it