/srv/irclogs.ubuntu.com/2015/01/30/#juju-dev.txt

davecheneymenn0: fair enough00:09
davecheneythere is an option for running tests within a package in parallel00:09
davecheneybut00:09
wallyworld__axw: trivial review when you have a moment http://reviews.vapour.ws/r/825/00:09
davecheneya. it's not the default (nobody writes tests taht can be run concurrently)00:09
davecheneyb. you have to specifically enable it on a per tets by test basis00:09
davecheneyc. it wouldn't matter because gocheck makes everything look like one test case anyway00:10
menn0davecheney: thanks00:11
menn0thumper, wallyworld__ : possible fix to the CI blocker. http://reviews.vapour.ws/r/826/00:16
thumpermenn0: how confident are you that it fixes it?00:19
thumperit certainly looks like a candidate00:19
* thumper is munching then free for the call00:20
thumperah shoot00:20
thumpermenn0: I forgot about my chat with wallyworld__00:20
thumperwallyworld__: what is your schedule like?00:20
wallyworld__i can wait00:20
menn0thumper: well I can't replicate the problem locally, it's timing dependent. it only seems to fail for i386 and ppc in CI.00:20
thumpercool cool00:20
wallyworld__ping when ready00:20
jw4menn0: I can try it on my slow ec2 instance00:20
menn0thumper: but there was definitely a problem which this PR fixes and I can see how the bug could lead to the problems we're seeing.00:21
jw4menn0: although after reviewing the fix my error might be unrelated00:21
menn0jw4: that would be great00:21
menn0jw4: the error you added to the bug could definitely be related to this fix00:22
jw4menn0: cool - I'll spin it up and try00:22
menn0jw4: thank you00:22
jw4menn0: hmm - first test run without your change did not fail - I'll try again00:53
thumperwallyworld: chat?01:03
menn0jw4: I would expect the problem to be somewhat intermittent01:03
jw4yup01:03
axwwallyworld: can you just change the timeout to be based on the bootstrap-timeout?01:04
axwwallyworld: it's configurable01:04
thumperwallyworld: I'm in the hangout, but I'm going to make a coffee01:16
jw4menn0: still not repro'ing it - i'm gonna try with an i386 image01:29
menn0jw4: ok thanks01:30
jw4menn0: otherwise we'll just have to watch for it in the CI servers01:31
menn0jw4: the CI unit test runs with the fix in are about to finish so that might tell us more as well01:31
jw4menn0: +101:31
menn0jw4: this is the key one to watch: http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-i386/1327/console01:32
menn0jw4: the jujud tests have already passed which is a good sign01:32
jw4sweet01:33
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
thumperwallyworld: missed you, I'm off to the vet, bbl02:06
wallyworldthumper: np, i'm off to buy a new farking network adaptor02:06
wallyworldcan't stand these disconnects02:06
menn0wallyworld: how common is it for the jujud test runs to time out on the (apparently slower) i386 CI hosts?02:39
menn0wallyworld: because a lot of runs just did02:39
wallyworldnot sure, but often i suspect02:40
wallyworldour tests timeout on my laptop02:40
wallyworldaxw: pr updated02:49
axwwallyworld: just noticed. LGTM02:50
axwwallyworld: your default size change is ineffective, sorry I didn't pick up on it earlier. your change just alters a temporary03:05
axwI'll fix it in my branch03:05
wallyworldah ball ok, thanks03:08
wallyworldthat's what i get for rushing03:08
thumperwallyworld: you back?03:12
wallyworldyep03:13
wallyworldhaven't left yet03:13
thumperchat?03:13
wallyworldsure03:13
=== kadams54 is now known as kadams54-away
axwwallyworld: https://github.com/wallyworld/juju/pull/2103:39
wallyworldlooking03:39
=== kadams54-away is now known as kadams54
wallyworldaxw: gh not letting me comment. but i think we need allcons[name] = cons after cons.Pool = defaultPool03:46
axwwallyworld: ah yeah, I'll fix it03:49
wallyworldta03:49
axwwallyworld: updated03:51
wallyworldthanks, merging03:51
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
thumpernight all04:43
=== kadams54 is now known as kadams54-away
wallyworld_menn0: in case i don't catch you later, see you in capetown04:57
wallyworld_axw: small problem to add to todo list - storage constraints reject pool names with "_" eg fast_ebs04:58
axwwallyworld_: yeah, we need to decide what's a valid name. I just copied what was valid for service names04:58
wallyworld_np, i'll add to spreadsheet04:59
wallyworld_axw: i can't get ebs volumes with provisioned iops to work - the ec2 instance doesn't attach any ebs volumes and doesn't finish coming up in AWS05:25
axwwallyworld_: just about to propose UML changes, will take a look in a sec05:26
wallyworld_that's using volume-type = provisioned-iops05:26
wallyworld_and iops = 2000 or whatever05:26
wallyworld_i blame aws05:26
mattywfolks - is it possible to change the juju admin secret in a deployed environment?05:26
mattywalso - morning05:26
wallyworld_not sure05:27
axwwallyworld_: there is a ratio between size and IOPS that you need to adhere to, could be that05:27
wallyworld_ah ok05:27
axwwallyworld_: http://reviews.vapour.ws/r/828/05:28
axwmorning mattyw. I don't think so, but not 100% sure05:31
axwwallyworld_: I'm going to $$__JFDI__$$ for hopefully obvious reasons :)05:36
wallyworld_yeah, np05:36
wallyworld_i ned to land the maas fix also05:36
wallyworld_i'm off for a bit - going to computer shop to get network card05:37
axwlater05:38
axwwill let you know how I go with IOPS05:39
mattywaxw, morning - I was in the process of bringing the env up - so I decided to just destroy and start again05:49
=== Guest8948 is now known as jam
wallyworld_axw: anastasiamac: i think we should enhance storage list to display the provider type06:39
anastasiamacaxw: wallyworld_: sounds good but let me merge my stuff 1st06:39
anastasiamacaxw: wallyworld_: i can add it next :D06:39
wallyworld_sure :-)06:41
axwwallyworld_: provider type, or pool?06:42
axwwallyworld_: I was thinking pool...06:42
wallyworld_axw: both maybe. as a user, i'd like to see provider type as well06:42
axwwallyworld_: I'd hope the pools are named to make it obvious which provider is involved06:43
wallyworld_that's up the the user :-)06:43
axwwell if you know what pool to select when you deploy... you should also know what it is when you list them06:43
wallyworld_supporting "_" would help06:44
axwthat only holds if the person listing and deploying are the same tho06:44
axwwallyworld_: why? you can use "-"06:44
wallyworld_i agree in principal, but yes, as you just said06:44
wallyworld_true06:44
wallyworld_i guess i'm an old python guy at heart06:44
axw:)06:44
axwwallyworld_: https://github.com/wallyworld/juju/pull/2307:22
axwwallyworld_: did you need any other docs or anything for next week, or is that UML update sufficient?07:23
axwwallyworld_: btw I just confirmed that with a pool having iops=3000, I can deploy with a 100GiB volume07:24
axwotherwise the block device mapping seems to get silently eaten up...07:24
axwthat branch just beefs up the validation07:24
mattywdavecheney, ping?07:40
davecheneymattyw: ack08:15
dimiternvoidspace, jamestunnicliffe, TheMue, fix for bug 1416134 - please take a look http://reviews.vapour.ws/r/830/11:16
mupBug #1416134: Unable to override network-bridge if container type is kvm (local provider) <cloud-installer> <config> <lxc> <network> <regression> <cloud-installer:Confirmed11:16
mupfor adam-stokes> <juju-core:In Progress by dimitern> <juju-core 1.21:Triaged> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1416134>11:16
dimiternvoidspace, http://reviews.vapour.ws/r/831/11:33
dimiternTheMue, http://reviews.vapour.ws/r/832/11:43
dimiternthanks for the reviews voidspace and TheMue!11:45
TheMuedimitern: yw11:45
perrito666morning12:11
=== lazyPower is now known as lp|Metrics
=== liam_ is now known as Guest51703
perrito666wallyworld_: still here12:58
perrito666?12:58
wallyworld_hi, about to sleep, i catch a taxi soon13:00
perrito666wallyworld_: the short answer is tounit <-- testing purposes13:01
wallyworld_sure, but i think the function signature is wrong13:02
perrito666you mean my findEntity?13:02
wallyworld_yeah - why not have it return a StatusSetter13:02
perrito666wallyworld_: because it needs to satisty an interface to be used by common statusetter?13:03
wallyworld_right, so change that13:03
wallyworld_the operation is to call SetStatus13:03
wallyworld_the different is how the thing to set status on is found13:03
wallyworld_for machines and units, shared logic can be used13:04
wallyworld_for unit agents, the Agent() method is used13:04
wallyworld_anyways, have a think about it, i gotta get a few hours sleep before i catch th plane13:04
perrito666go, have a nice trip13:05
wallyworld_i might be off base too, so see if it makes sense looking at the code13:05
perrito666we can re-discuss this via mail whenever you are more awake, cheers13:05
wallyworld_ty, see you later13:05
voidspaceanyone know anything about the proxyupdateworker?14:04
voidspacewallyworld_: ping if you're still awake at this horrendous hour14:11
fwereade_voidspace, might do a little, I touched it semi-recently14:13
fwereade_voidspace, I am 80% convinced that it's fundamentally dangerous14:13
fwereade_voidspace, but it was a reduce-duplication driveby, not a make-tings-right14:14
voidspacefwereade_: so this worker updates the .juju-proxy file with the http(s) (etc) settings from the config14:22
fwereade_voidspace, yeah14:22
voidspacefwereade_: so these values are propagated from environments.yaml into .juju-proxy and /home/ubuntu/.profile is updated to source this file14:22
fwereade_voidspace, it's the env vars that that http module uses that worry me14:22
voidspacefwereade_: however the environment variables are never set in the jujud process14:22
voidspacefwereade_: so http.DefaultClient (which uses these environment variables) does not see the proxy settings14:23
voidspacefwereade_: so if they are needed (which is usually why they are set...) everything fails14:23
voidspacefwereade_: so my intention is that the proxyupdater should also set the environment variables for the process14:23
voidspacefwereade_: https://bugs.launchpad.net/juju-core/+bug/140322514:24
mupBug #1403225: charm download behind the enterprise proxy fails <cloud-installer> <deploy> <proxy> <sync-tools> <cloud-installer:Confirmed for adam-stokes> <juju-core:Triaged by mfoord> <https://launchpad.net/bugs/1403225>14:24
fwereade_voidspace, proxyupdater.go:151?14:24
perrito666lol, just found in the code14:24
perrito666// TODO(fwereade) GAAAAAAAAAAAAAAAAAH this is LUDICROUS.14:24
fwereade_:)14:24
voidspacefwereade_: ah right, but only the first time - right14:24
voidspacefwereade_: interesting. I can attach to jujud and see that they are *not* set.14:25
fwereade_voidspace, I think that says "always on first call or if they've changed since last time"?14:25
fwereade_voidspace, best guess is that jujud changes somehow stopped it from being started properly?14:27
voidspacefwereade_: hmmm... indeed. Well, it doesn't seem to be "working". I'll do some debugging. Thanks.14:27
fwereade_voidspace, I thought I had tests for that, but possibly I screwed them up14:27
voidspaceno problem, I know where I'm looking now - so it should be easy from here ;-)14:28
sinzuijam, dimitern: We have a licensing bug to address. bug 1416425 which has an easy fix I think14:28
mupBug #1416425: src/bitbucket.org/kardianos/osext/LICENSE is wrong <licensing> <packaging> <juju-core:Triaged> <juju-core 1.21:Triaged> <juju-core 1.22:Triaged> <https://launchpad.net/bugs/1416425>14:28
sinzuijam dimitern: Can you ask someone to address the licensing bug14:28
fwereade_voidspace, (also: it's started in both the unit agent and the machine agent, please check both)14:30
voidspacefwereade_: yep14:30
sinzuialso dimitern , or anyone, why did the license to juju/cmd change?14:31
dimiternsinzui, i'll have a look14:31
dimiternsinzui, so is this about using rev 44140c5 in dependencies.tsv for bitbucket.org/kardianos/osext - in 1.23, 1.22, and 1.21 ?14:32
sinzuidimitern, yes, tip has the fix14:35
perrito666fwereade_: ftr, the ensuing line is indeed ludicrous14:36
perrito666cmd := exec.Command("go", "build", "github.com/juju/juju/cmd/jujud")14:36
* fwereade_ knows :(14:36
dimiternsinzui, ok, I'll prepare a fix for it14:40
dimiternoh boy.. I found a bug with retry-provisioning14:50
dimiternusing it on a failed container in EC2 caused a new *instance* to be provisioned for "0/lxc/0"14:51
dimiternhow do you like this http://paste.ubuntu.com/9957564/14:52
sinzuidimitern, Sorry, several more licensing bugs were found. Can you find people to look into the issues. they all affect 1.23, 1.22, and 1.21 https://bugs.launchpad.net/juju-core/+milestone/1.21.215:12
* sinzui return to the broken publish-revision job15:13
dimiternsinzui, I can fix the osext one, but unfortunately we're close to finishing the sprint here and somebody else can take over15:14
dimiternalexisb, hey, are you around?15:16
coreycbdoes anyone have any tips on getting around a "virbr0": net: no such interface error on bootstrap of local provider on 1.21.1?15:27
dimiterncoreycb, do you have virbr0 on your machine?15:28
coreycbdimitern, nope15:28
coreycbdimitern, I know I could create it, but wondering more about if my juju config is wrong15:28
dimiterncoreycb, and you're using container: kvm ?15:28
coreycbdimitern, yes15:28
dimiterncoreycb, looks like you're affected by bug 1416134 which I just fixed this morning :)15:29
mupBug #1416134: Unable to override network-bridge if container type is kvm (local provider) <cloud-installer> <config> <lxc> <network> <regression> <cloud-installer:Confirmed for adam-stokes> <juju-core:In Progress by dimitern> <juju-core 1.21:Fix Committed by dimitern> <juju-core 1.22:Fix Committed15:29
mupby dimitern> <https://launchpad.net/bugs/1416134>15:29
coreycbdimitern, sweet, I think :)15:29
coreycbdimitern, thanks15:30
dimiterncoreycb, in the mean time you can either create a virbr0 or use "network-bridge" setting but lxcbr0 there won't work due to that bug15:30
coreycbdimitern, ok, thanks15:30
dimiternnp :)15:31
dimiternvoidspace, http://reviews.vapour.ws/r/833/, http://reviews.vapour.ws/r/834/, http://reviews.vapour.ws/r/835/ PTAL :)15:33
bodie_natefinch around?15:46
perrito666bodie_: no natefinch today15:46
=== kadams54 is now known as kadams54-away
=== benji_ is now known as benji
=== kadams54-away is now known as kadams54
=== lp|Metrics is now known as lazyPower
menn0sinzui: ping19:42
sinzuihi menn019:42
menn0sinzui: so it's hard to know if I've fixed the CI blocker because we seem to be hitting test timeouts for the i386 and ppc unit test runs19:43
menn0sinzui: I know my recent changes added a few seconds to the cmd/jujud/agent test run (on my machine at least)19:43
menn0sinzui: I wonder if we were close to the 20min threshold on the i386 and ppc CI machines and I've just tipped it over?19:44
sinzuimenn0, I am not user. but looking at the FAIL: in the tests, these are very different from what we usually see19:45
sinzuimenn0, and the ppc test are run very quickly19:46
=== kadams54 is now known as kadams54-away
menn0sinzui: run-unit-tests-trusty-ppc64el and run-unit-tests-precise-i386 seem to be hitting timeouts I don't see fails there19:47
sinzuimenn0: the fact that we are still seeing panic means something is wrong. We know that some tests will fail, but pass when retried, but we don't see panics19:48
menn0sinzui: since the last fix was committed the only panics I see are about tests taking too long19:48
sinzuimenn0, the 386 tests  ridiculously slow. we cannot get  a fast cloud images19:48
menn0sinzui: can you point me at another panic?19:49
sinzuimenn0, http://data.vapour.ws/juju-ci/products/version-2286/run-unit-tests-trusty-ppc64el/build-2281/consoleText19:49
menn0sinzui: that's because "panic: test timed out after 20m0s"19:49
menn0sinzui: go test triggers a panic because the timeout set for the test run was exceeded19:50
sinzuimenn0, but about that 386. We have a fast trusty 386 machine, but the tests don't pass because of two issues. I reported a bug about this. If the two tests are fixed or *skipped* you will see a passs http://data.vapour.ws/juju-ci/products/version-2286/run-unit-tests-trusty-i386/build-164/consoleText19:50
sinzuibug 140845919:51
mupBug #1408459: pingerSuite tests consistenly fail on trusty 386 <ci> <i386> <intermittent-failure> <juju-core:Triaged> <https://launchpad.net/bugs/1408459>19:51
menn0sinzui: ok, I see that and we need to fix that. but that's an unrelated issue to the current CI blocker.19:53
menn0sinzui: what I'd like is if we could bump up the test timeout to 25 min or something to see if the panics related to the CI blocker have been resolved.19:53
sinzuimenn0,the machine is healthy19:54
sinzuiand I can see it doesn't need a updated19:54
sinzuimenn0, and we can see that 1.21 and 1.22 pass19:54
sinzuimenn0, so I think master + gccgo have a problem that 1.21 doesn't19:55
menn0sinzui: I'm not saying there's anything wrong with the machine. I think that the addition of extra tests in trunk recently has pushed us over the timeout 20min timeout for the machine agent tests.19:55
sinzuimenn0, the whole suite when it was passing take 37 to 44 minutes to run19:57
=== kadams54-away is now known as kadams54
sinzuimenn0, this is the last health mast run http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/run-unit-tests-trusty-ppc64el/2263/19:58
sinzuidisregard the tar timestamp issue. I fixed that this morning19:59
menn0sinzui: ok, that's useful20:02
* menn0 checks how the timeout works20:02
sinzuioh good, because I am running out of thinks to look for20:02
jw4sinzui: are there any plans to get the tests working on windows and osx ?20:07
=== kadams54 is now known as kadams54-away
menn0sinzui: ok... so i'm pretty sure the timeout is per Go package and it's looking like on i386 and ppc64 the cmd/jujud/agent package is now taking over 20mins20:08
sinzuijw4 yes We test the client ever revision and we run windows units every revision http://reports.vapour.ws/releases/228620:09
menn0sinzui: that's a massive jump from the last successful runs20:09
menn0sinzui: so it's not just tests taking a little bit longer... something is stuck20:09
menn0sinzui: but these tests pass fine on amd6420:09
sinzuimenn0, yep20:09
menn0sinzui: am I ok to use those hosts later on today20:09
menn0?20:09
menn0sinzui: I think I still have the details to access the ppc64 host that you gave me some time ago20:10
=== kadams54-away is now known as kadams54
sinzuijw4, when there is a way to for juju to provision a windows machine and deploy an agent we will add that test to ever revision20:10
jw4sinzui: cool.  That link doesn't work with my ubuntu credentials, probably because I'm not a cougar (<-- is that even a thing ? ;) )20:11
sinzuimenn0, oh, you my have visited a different machine even if you had. these tests are run on stilson-08. I can add your ssh keys if needed20:11
jw4sinzui: I'm just curious how they're being run because on the surface they're not even close to running on my windows and osx machines20:11
* menn0 checks20:12
sinzuijw4, only canonical staff can see the results, but the raw results are public20:12
jw4sinzui: kk - tx20:12
menn0sinzui: that's what I have access for and I can still get in20:12
sinzuijw4, the last 2 tests are windows client and units http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/20:13
sinzuifab20:13
menn0sinzui: access to the i386 host might be useful too if that's possible20:13
sinzuimenn0, we spin up an instance for that20:13
jw4sinzui: sweet - thanks again20:13
menn0sinzui: ok, I can spin up my own then.20:14
sinzuimenn0, but I can get you acesss to the machine we want to run tests on. again, about that 386 bug. when those tests pass, we will remove the old 386 job20:14
menn0sinzui: ok20:14
jw4sinzui: hmm the last 'successful' run of win-client-deploy doesn't seem to be actually successful : http://juju-ci.vapour.ws:8080/view/Juju%20Revisions/job/win-client-deploy/1381/console20:15
sinzuijw4, looks like a successful bootstrap to me20:16
jw4sinzui: it probably is - I just see a strange error about 5 lines up from the bottom20:16
sinzuijw4, The SUCCESS was on the win machine the other lines are from ssh and the test running. we don't what them cursing the win success20:17
sinzuiand ssh did bugger up again20:17
jw4sinzui: I see - also I noticed that the run-unit-tests-win2012-amd64 is getting the same '/usr/lib/juju/bin/mongod' file does not exist error that I'm getting on my windows tests20:18
menn0sinzui: do you have any details or a script for spinning up the precise and trusty i386 test hosts?20:19
menn0sinzui: I want to make sure I replicate as closely as possible20:19
sinzuimenn0, juju-ci-tool/run-unit-tests m1.medium ami-81dee0e820:21
menn0sinzui: perfect. thanks. i've got a few things to take care of right now but I'll jump back on this today.20:21
sinzuimenn0, aws is the only provider that permits 386 and they limit it to a slow machine, and aws is phasing out 386, which is why we need to switch to our special machine20:22
menn0sinzui: got it20:22
=== kadams54 is now known as kadams54-away
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!