[00:12] ericsnow: if you're around, looks like review board isn't picking this up? https://github.com/juju/juju/pull/1931 [00:12] ericsnow: oh shoot and i know why. [00:12] ericsnow: i recently changed my github username: katco- to kat-co [00:13] katco: that would do it :) [00:13] ericsnow: any way to move my account and get that PR up? [00:13] katco: I should be able to do that real quick [00:14] ericsnow: you are a gentleman and a programmer. [00:16] katco: done; hopefully it works ;) [00:17] ericsnow: account looks good, but PR still isn't there? [00:17] rbt post is asking for a pw [00:26] ericsnow: and the instructions you sent out last year aren't working for me =/ [00:28] axw: wallyworld_: https://github.com/juju/juju/pull/1931 until it gets onto RB. i have to go take care of some things. [00:28] ty [00:34] boom, gccgo bug is fixed upstream [00:34] now ... PAPERWORK! [00:39] davecheney: \o/ [00:40] yeah, lynn's patch was submitted this morning [00:53] ok, that joy was short lived, our other fix that went into gccgo 4.9 needs to be forward ported to 5.0 [00:53] i'll raise another bug [01:06] win12 [01:22] wallyworld_: got a minute? [01:22] sure [01:22] 1:1 hangout? === kadams54 is now known as kadams54-away [02:08] holy crap, that mostly works [02:09] right, going to bed [02:22] katco: if you're still around, try killing that PR, go to RB, log out, log back in, and then re-submit the PR over on github [04:29] thumper: ping [04:38] wallyworld: FYI, I've proposed the environ-storageprovisioner and dynamic EBS branches -- http://reviews.vapour.ws/r/1258/ and http://reviews.vapour.ws/r/1259/ [04:38] wallyworld: I guess I'll work on volume-backed filesystems now [04:41] wallyworld: FYI, I've proposed the environ-storageprovisioner and dynamic EBS branches -- http://reviews.vapour.ws/r/1258/ and http://reviews.vapour.ws/r/1259/ [04:41] wallyworld: I guess I'll work on volume-backed filesystems now [04:42] axw: great ty, will look soon. volume backed fs sounds good [05:19] wallyworld: does this sounds familiar (relative to state log pruning): "failed to retrieve log counts: no such cmd: scale" [05:47] anyone know what the error "juju.rpc server.go:554 error writing response: EOF" in machine-0.log means? the juju environment isn't very healthy, juju sets aren't triggering the appropriate config-changed hooks [06:45] wallyworld: the sessions added, alexisb notified. [06:55] axw: off to soccer for a couple of hours, reviews done [06:56] wallyworld: thanks [06:56] later [07:17] Bug #1436191 was opened: gce: bootstrap instance has no network rule for API

[07:24] morning all o/ [08:30] axw, ping [08:31] axw, I'm +100 on moving replicaset out of the main repo, but will this reduce the tests run time for core? [08:59] dimitern: hey sorry, was afk. this change will not. this change is just preparing to move it out of core [09:00] and to generate discussion [09:00] axw, right, so a step in the right direction :) [09:00] yup [09:03] dimitern: in case I misunderstood - the change on RB atm won't reduce test time, but moving replicaset out of core will [09:03] unless of course CI starts running tests for all the github.com/juju packages [09:04] axw, yep, got it [09:04] axw, well, that happening at some point certainly won't hurt [09:05] it'll hurt the run time ;) I think we should do it, not sure about as part of the merge gating, maybe just as a voting CI job [09:05] axw, no no - definitely not part of the merge gating :) [09:22] wallyworld, dimitern, axw, anyone else: I'm getting a "no reachable servers" trying to connect to state, which then causes all the workers to shut down... but jujud hangs around, and doesn't restart or retry or anything. Why is that? Shouldn't it retry? This is on a machine that I've converted from a regular server to a state server in a replicaset. [09:22] It's a bummer, because I think it's just that mongo is slow, and if I kill jujud, it restarts and is able to connect. [09:22] natefinch: not sure off hand, I'll have a look at the code and see if I can think of a reason... [09:22] natefinch, I'm not quite sure why, but it should retry [09:23] natefinch, maybe some worker / runner is hanging instead of exiting [09:23] dimitern: ahh, hmm, yeah, that's a good point [09:31] natefinch: only reason I can see that would explain that is if the state-serving info in the agent config changed [09:31] i.e. from having it to not having it [09:32] not sure if that's even possible [09:33] wow that's a first - http://paste.ubuntu.com/10676289/ - build failure in the net-cli branch - anyone seen something like that with the gc compiler? [09:34] dimitern: did you update your go compiler? [09:34] I've seen that sort of thing when there's version mismatch [09:34] axw, no - still go 1.2.1 amd64 from trusty [09:36] try deleteing your gopath/pkg directory and rebuild [09:36] weird.. so I've wiped out my /tmp and $GOPATH/pkg/* dirs, rebuilt and now it's fine [09:36] yep [09:36] natefinch, yep, cheers [09:37] some weird mismatch.. maybe gccgo or something? [09:37] I did have both gccgo and gc binaries/object files in pkg [09:38] in separate subdirs, but still [09:38] oh right forgot they're in separate dirs... well, I dunno, I've had it happen once in a blue moon [09:39] yeah [09:40] (as they say quite often lately around here - "putin is most likely behind this" :D) [09:40] lol [09:40] haha [09:42] Anyone around who could help with a test failure due to an unexpected receive? http://paste.ubuntu.com/10676354/ [09:42] This seems to happen if I am running the entire suite sometimes, but never if I just run that one test. [09:42] dooferlad, oh, I know that one [09:42] dooferlad, it's known to be flaky, esp. under load [09:43] dimitern: *sigh* [09:43] dooferlad, I did try fixing it, but gave up because it wasn't easy without refactoring the way the filter uses watchers [09:43] I love our tiny logs. ;) [09:47] dooferlad, our SpaceTag caused quite a stir apparently [09:47] dimitern: do you remember what caused the problem? (filter tests) [09:48] dimitern: yea, I expect that will be renamed at some point! [09:48] network-group maybe [09:48] is spacetag anything like freeze tag? sounds like fun ;) [09:49] dooferlad, the problem is synchronization between the apiserver, api, and state watchers [09:49] natefinch, it's more like a space cake I guess :D [10:00] dooferlad, TheMue, standup? [10:01] dimitern: yes. *lol* [10:39] dimitern: so I fixed the first test problem (only a new worker sees the broken provider) [10:40] dimitern: for the second test issue I've pushed a failing test [10:40] dimitern: when we start the worker we have two dead addresses 0.1.2.4 and 0.1.2.6 [10:40] dimitern: I'm expecting to see two ReleaseAddress calls, one for each [10:40] dimitern: what I *actually* see is *three* calls [10:40] dimitern: two for the first address and one for the second [10:40] dimitern: I'm looking into it [10:40] voidspace, hmm weird [10:41] dimitern: I have a test that reliably passes asserting those three calls... [10:42] voidspace, do the first 2 calls have the same args? [10:42] dimitern: well, the OpReleaseAddress we get is the same [10:42] so yes [10:42] same subnetid etc [10:42] they're the same address [10:43] dimitern: I'll add some logging to see why [10:43] voidspace, yeah [10:43] voidspace, it might be the last call comes from Handle() [10:43] voidspace, oh, no actually.. if the first 2 are the same maybe the first one does [10:44] voidspace, make sure you check the addresses you get in Handle are both dead and not gone [10:46] dimitern: I check that [10:46] dimitern: I think I found a bug, not sure how it can be the cause [10:46] dimitern: or how the code worked at all [10:47] dimitern: I have a select that removes the dead addresses - but it is a single select it doesn't loop [10:47] dimitern: so it should only do the first address [10:49] so how I'm seeing three releases is a real mystery... [10:49] running with logging [10:49] voidspace, good catch then [10:49] dimitern: right, I see two attempts for each address [10:49] from the logging [10:50] voidspace, I guess one from SetUp and one from Handle ? [10:50] let me add to Handle to check that [10:50] dimitern: it's possible that the removal triggers the watcher too - but inside Handle I'm specifically checking if the address exists [10:50] fwereade: you around? [10:50] dimitern: if it's *already* been removed we should fail to fetch it from State [10:51] wallyworld, heyhey [10:51] dimitern: it looks like the removal succeeds, triggering the watcher and then we *successfully* pull the address back out of state [10:51] dimitern: race condition due to transactions? [10:51] is that possible [10:51] voidspace, hmm [10:51] voidspace, it is possible [10:51] fwereade: if you had time, i'd love a quick pre-impl chat [10:51] dimitern: I'm going to confirm that this is the case (one attempt from Handle and one from SetUp) [10:51] voidspace, that's the thing with JujuConnSuite [10:51] wallyworld, will be in our hangout in 5 mins [10:52] voidspace, we have 2 states there - State and BackingState - the latter is the real one used by the apiserver, the former is the state used by the api client [10:52] sure [10:52] voidspace, and they're not synced wrt watchers [10:53] voidspace, try calling s.BackingState.StartSync() before each assert on the watcher [10:56] dimitern: what i'm seeing on start is Handle called with every address [10:56] those were exactly the same issues that lead to flaky tests [10:56] dimitern: so the initial SetUp is trying to remove them at the same time as Handle [10:56] voidspace, yeah, that's what I suspected [10:57] I thought State and BackingState were the same these days, we fixed it by just having one State? [10:57] I'll try the extra sync anyway [10:57] voidspace, so you might not need to do the out-of-bound goroutine in setup perhaps? [10:57] dimitern: maybe not [10:58] dimitern: I read a docstring saying "Initial" was only the alive entities for a lifecyclewatcher [10:58] that maybe out of date [10:58] dimitern: I'll read the lifecycle watcher code [10:58] I maybe able to just delete the SetUp [10:58] well, most of it anyway [10:59] voidspace, I think it has to include dying ones as well and frequently the dead ones [10:59] voidspace, consider something like a provisioner [10:59] voidspace, it needs to know about its dead machines so it can remove them [10:59] fwereade: "frequently"... I'd quite like a deterministic result :-) [11:00] fwereade: right, I was just going off some docstring that said "initial()" was called with only alive entities [11:00] voidspace, ok, lifecycle watchers should return everything in their initial results, including dying and dead [11:00] voidspace, fwereade, so the lifecycle watcher is smarter than I thought :) [11:00] fwereade: I'll dig it out [11:00] that's great [11:00] removes some code [11:00] voidspace, cool [11:01] The first event emitted will contain the ids of all non-Dead entities [11:01] state/watcher.go line 136 [11:03] fwereade: so in lifecycleWatcher.initial it fetches the whole collection, *stores* the ids of "non-dead" entities, but returns *all* ids [11:03] I'll update the docstring [11:03] dammit, I should no better never to trust a docstring [11:03] voidspace, yeah, that sounds right [11:03] voidspace, thanks [11:03] fwereade: we need to go back to calling them lies... [11:04] *know better* [11:06] dimitern: so if I just delete that code in SetUp the test passes... [11:06] dimitern: which is good [11:07] voidspace, only one comment on your review [11:07] voidspace, great! [11:07] voidspace, I'll wait for the changes around SetUp and will re-review it [11:08] dimitern: that select is gone [11:08] dimitern: ah, no it's not [11:08] dimitern: that's iin the test, fine I'll fix that [11:08] dimitern: SetUp change pushed [11:09] voidspace, looks nice and red :) [11:09] :-) [11:12] * dimitern steps out for ~2h [12:17] ericsnow: here's a trivial fix for an intermittent failure on ppc64 http://reviews.vapour.ws/r/1261/ [12:43] ocr" could anyone please take a look at http://reviews.vapour.ws/r/1263/ ? thanks! [12:51] * dimitern is back [13:09] * fwereade omw uk for a bit, will be around again later but was up until 3 last night so might be a bit relaxed about it [13:39] voidspace, hey [13:40] voidspace, the releaser looks much better now, thanks! [13:40] voidspace, you have a review with a provisional "ship it!" and mostly minor changes suggested. [13:45] jw4: ping [13:45] ericsnow: ola [13:46] jw4: could you take a look at this log from CI: http://data.vapour.ws/juju-ci/products/version-2478/run-unit-tests-vivid-amd64/build-289/consoleText [13:47] ericsnow: that should be fixed? [13:47] jw4: it shows a failure that you fixed the other day with the log pruning stuff [13:47] right [13:47] did that error occur recently? [13:47] jw4: I'm trying to figure out why [13:47] jw4: Tuesday [13:48] (yesterday) [13:48] ericsnow: ah, look at the documentation for the mgo http://godoc.org/labix.org/v2/mgo#Database.Run [13:49] jw4: right, but I thought you fixed this already [13:50] ericsnow: yeah, I'm looking to see if there's another call to Database.Run [13:50] jw4: good point [13:50] ericsnow, jw4, I did see that patch replacing bson.M with bson.D land, but maybe the dependencies.tsv was not updated after? [13:51] dimitern: there should be no dependencies.tsv update [13:51] jw4, oh, right - that's in state only [13:52] ericsnow: weird I don't see any other call [13:53] ericsnow: I see that run was testing a version before my fix landed: Testing gitbranch:master:github.com/juju/juju 0752a4bc [13:54] jw4: ah, never mind then :) [13:54] :) [13:54] (I was just checking that) [13:54] * ericsnow promises himself not to try to reason about problems late in the evening [13:54] hehe [13:55] (but aren't you in Mountain Time? 8 AM now?) [13:58] well, ha --to 1,2 works now, though I have to use a hack to get jujud to restart after the StateWorker errors out from not being able to connect to state. I gotta figure out why the damn workers aren't shutting down. [14:02] dimitern: cool, thanks [14:02] dimitern: you want me to use the envtesting package in the worker itself? [14:03] dimitern: in fact, I don't think envtesting.ShortAttempt exists, which is why I was using the other one in the tests [14:03] voidspace, ah [14:03] dimitern: I'm pretty sure you've hallucinated its existence [14:03] voidspace, let me double check [14:03] dimitern: and I didn't want a dependency on a test package in the production code [14:04] voidspace, oh, you're right [14:05] voidspace, so then use common.ShortAttempt in both the worker and the tests? [14:06] dimitern: it's in the provider package, but ok [14:06] well, provider/common [14:06] voidspace, yeah [14:06] seems like a weird dependency [14:06] dimitern: and I dropped one of your issues - you slightly misunderstood the test [14:06] dimitern: the rest are good and I'll fix them, thanks [14:07] dimitern: after a break... [14:08] voidspace, ok, cheers [14:12] Yeeeeeeeeeeeeeeeeeeeeeeeeeeeeeehaw! Local vMAAS acquiring works. *phew* [14:13] TheMue, awesome! [14:13] Needed some patches/changes in their hack, but now the first node started as wanted. [14:14] TheMue, does it stop as well? :) [14:14] dimitern: Yeah, will pass this information to the MAAS team. Maybe they are interested in adding it too. [14:14] dimitern: Oh, good question. one moment. [14:16] dimitern: Hehe, yeah, it does. *dancingThroughTheRoom* [14:17] TheMue, great! then you can have a look at that bug 1427814 :) [14:17] Bug #1427814: juju bootstrap fails on maas with when the node has empty lshw output from commissioning

[14:18] TheMue, in order to reproduce it, you'll need to remove the lshw output for a node (or all nodes) from maas db [14:22] dimitern: shit, I should have been quiet *lol* [14:22] TheMue, :) [14:23] dimitern: so, assigned to me and will add a card. but let me provision my fresh nodes first [14:23] TheMue, sure, np [14:23] TheMue, there's already a card btw [14:24] dimitern: ah, ok, then I'll grab it [14:24] TheMue, cheers, ping me if anything is unclear [14:24] dimitern: yep, will do [14:43] ericsnow, I have bug 1435974 into something we can do. In short, the license files don't have copyrights...the owner of the licence is not clear the owner of the code [14:43] Bug #1435974: Copyright information is not available for some files

[14:45] sinzui: ah, that makes sense [14:45] sinzui: so does that mean we should have a separate top-level COPYRIGHT file? [14:46] ericsnow, no it means stop cargo-culting licenses. just put the copyright and meaningful paragraph in the LICENCE file. see my comment int the bug [14:48] sinzui: got it; I'll take a look [14:52] sinzui: should I get someone to write up patches or were you planning on it? [14:53] ericsnow, I think core staff are best suited to it since they have to change the project, then the juju 1.22.1 dependencies.tsv [14:53] sinzui: sounds good [14:58] ericsnow, hey [14:59] dimitern: hi [15:00] ericsnow, any idea why I'm getting this with GCE ? Bootstrap failed, cleaning up the environment. [15:00] 2015-03-25 14:58:45 ERROR juju.cmd supercommand.go:430 there was an issue examining the environment: retrieving auth token for : Invalid Key [15:00] dimitern: I'm guessing your key is invalid [15:00] ericsnow, I've followed the instructions, registered for a gce free trail, generated a client id as instructed [15:01] ericsnow, and the gce provider is not liking it for some reason - triple checked I pasted it ok [15:01] dimitern: it may be, as axw said, that it's simply not clear what to put into environments.yaml [15:01] ericsnow, sinzui: yeah, just slapping a copyright line on the license seems to be the easiest way to do it [15:02] dimitern: you probably have to put it in quotes [15:02] ericsnow, should the private-key setting be a patch to the json file I downloaded? [15:02] natefinch: as sinzui indicates in the bug, we should also simplify the license file [15:02] ericsnow, s/patch/path/ [15:02] dimitern: nope, though that is basically what axw suggested (good idea too) [15:03] ericsnow, ah, so it needs to be the verbatim contents of the file then? [15:03] dimitern: you have to copy-and-paste stuff out of that JSON file [15:03] I think QA can change the tarball scripts to check for copyrights in LICENSE/LICENCE file so we don't get these surprises from Ubuntu [15:03] ericsnow, oh, well, I'll try this [15:04] dimitern: also, I've put better instructions in the 1.23 release notes [15:05] ericsnow, I'll check there as well then, thanks [15:05] dimitern: something like this: https://pastebin.canonical.com/128327/ [15:05] (replace with your actual key) [15:05] dimitern: BTW, where did you get that JSON file? [15:06] ericsnow: google's site creates it for you [15:06] natefinch: right, but at what point? during the project creation process? I don't remember [15:07] ericsnow, from the GCE web ui - API & auth [15:07] dimitern: k [15:07] dimitern: I remember now, thanks [15:13] hi devs [15:13] I have a question about charm distros [15:13] I need to be free in what distro I launch with my charms: CentOS, Kali, etc... can I actually do that with juju? [15:13] dimitern: thanks for looking at the bug, BTW [15:14] ericsnow, np, looked like a relatively easy fix and a chance for me to try the gce awesomeness :) [15:14] dimitern: haha [15:19] Muntaner, currently juju supports deploying Ubuntu and Windows workloads [15:19] Muntaner, we will soon have support of Centos [15:19] but it is not currently available [15:20] alexisb, so I'm not able to choose the distro in my charm? [15:21] alexisb, maybe by writting my own metadatas? [15:22] Muntaner: you definitely can create your charm for a specific distro... it just has to be a version of Ubuntu or Windows right now [15:26] Muntaner: charm distro isn't specified by metadata.yaml, it's defined by where you push the charm to lp or to a local repo. So if you create ~/charms/win2012r2/charm-name then juju deploy --repository ~/charms local:win2012r2/charm-name Juju will attempt to create a VM with Windows Server 2012r2 [15:26] Muntaner: but that series, which defines distro (trusty, precise, win2012r2, etc) [15:27] Bug #1436390 was opened: GCE provider config should support extracting auth info from JSON file

[15:27] Bug #1436397 was opened: map-order sensitive test in md/juju/storage needs to be fixed

[15:27] marcoceppi_, there is no way that allows me to use other distros atm? [15:28] Muntaner: no, because Juju has to know how to handle that distro and that distro needs to have an image spun up with cloudinit running. Juju is doing the machine setup so if it can't speak to that distro it won't work. [15:29] \o/ gce is bootstrapping now [15:29] Muntaner: We're adding support for other distros all the time though, the more distros we add support for the easier juju becomes to port to other platforms [15:29] marcoceppi_, where can I find a list of the usable distros? [15:30] Muntaner: right now it's Ubuntu and Windows, with support for CentOS underway [15:30] sinzui: FYI, this last run of 1.23 (2482) had only 1 unit test fail :) [15:30] Muntaner: CentOS support will be submitted pretty soon, in a matter of weeks. As for current distros, it's just Ubuntu (precise or later) and Windows... I would have ot look to see which versions of windows we support. Just the server versions for the most part, and I think 2012 or later. [15:32] ericsnow, \0/ [15:34] Does anyone know the status of this critical bug? https://bugs.launchpad.net/juju-deployer/+bug/1421315 [15:34] Bug #1421315: subordinate service not appearing in watch output

[15:39] ericsnow, bootstrap fails now -I did 2 attempts, the first one waited for a while, then I stopped it, the second failed right away with RESOURCE_NOT_READY when trying to start the instance [15:41] dimitern: do you see an instances or other resources in the GCE [web] console for the project? [15:41] probably because I'm using region europe-west1 [15:41] folks - is there a way I can open an api connection acting as the state server in JujuConnSuite? [15:41] ericsnow, I did see a deprecation warning for zone europe-west1-a [15:41] ^^ seems like there must be a way - but I can't find it [15:42] ericsnow, now I've switched to us-central1, and so far no errors, it's trying to connect with ssh [15:42] mattyw, you need to log in as a machine with JobManageEnviron [15:43] Muntaner, did you get your question answered? [15:43] ericsnow, it worked this time - bootstrapping is mostly done [15:43] dimitern, yeah, I'm trying to access an api that is looked to machines with JobManageEnviron [15:44] dimitern, but I can't create a new machine with OpenAPIasNewMachine because I'm not allowed to start machines with that job [15:44] dimitern: oh good :) [15:44] mattyw, what do you mean not allowed? you're getting an error from state.AddMachine? [15:45] dimitern, correct [15:45] dimitern, I can do a quick screen share if you like [15:45] alexisb, yes, I don't know what to do now ;( [15:45] I should instantiate automatically subnets with different OS [15:45] could I get a volunteer to work on lp:1436407? [15:46] mattyw, hmm, let me look in a few places first [15:46] it's a blocker for 1.23 [15:46] connected between them... and I wanted to use juju bundles [15:46] #1436407 [15:46] Bug #1436407: certSuite.TestNewDefaultServer failed on vivid

[15:46] cmars, if you sill have someone available ^^ [15:46] ericsnow, I do believe one of the patches you've reviewed should fix that - I've left a comment [15:47] dimitern: ah, cool [15:47] mattyw, so is the machine you're adding the first one? [15:48] dimitern, I tried s.OpenAPIAsNewMachine(c, state.JobManageEnviron) [15:48] dimitern: yep, http://reviews.vapour.ws/r/1261/ [15:48] dimitern, but that fails because I'm not allowed [15:48] mattyw, ok, let me have a look at your code? [15:48] ericsnow, that's the one [15:49] dimitern: ah, can't use worker.AssertStop as worker.Worker doesn't implement Stop [15:49] dimitern: thanks for remembering :) [15:49] ericsnow, lucky catch I guess :) [15:49] statetesting.AssertStop I mean [15:49] voidspace, well, how about defer worker.Stop(w) then ? [15:51] dimitern: don't need the func call around it as it's a single liner I guess [15:51] all, we also need someone to work on #1435974, which should be pretty simple but will involve patches to most of our repos [15:51] Bug #1435974: Copyright information is not available for some files

[15:51] cherylj, are you working anything critical?? [15:51] that one is needed before Ubuntu will take 1.22, so it is pretty important [15:52] if not can you take a look at 1435974 [15:57] dimitern: what's the goal of the revision updater? is it just for juju status? [15:59] dimitern: nope, needs wrapping [15:59]