[01:45] I'm finally happy with the tests around create environment now [01:45] on to manual testing [01:47] * thumper crosses fingers [02:15] menn0, ping? [02:23] mattyw: pong [02:48] menn0: https://github.com/juju/juju/pull/1731 [02:48] menn0: waiting for RB to pick it up [02:49] thumper: RB doesn't seem to want to [02:50] no... [02:50] thumper: shall I just review on GH [02:50] ? [02:50] sure [03:15] mwhudson: the output from 7g -g is a total dogs breakfast [03:16] it also appears this particular dog has barfed up the response, twice [03:16] for good effect [03:16] is 6g -g any better? [03:16] probaly not [03:16] i've narrow the current crapping out [03:16] to a nil check [03:16] x := *y [03:16] will generate a nil check [03:16] and the encoding on that is screwed [03:17] menn0, http://reviews.vapour.ws/r/1063/ when you have a moment [03:18] mattyw: looking [03:19] mattyw: ship it! [03:20] menn0, thanks very much [03:28] wallyworld: https://bugs.launchpad.net/juju-core/+bug/1420049 -- can you please see my latest comments on this [03:28] Bug #1420049: ppc64el - jujud: Syntax error: word unexpected (expecting ")")

[03:29] sure [03:34] axw: i think it might be reasonable to say that when deploying containers, we match the machine arch. even if they bootstrap with an env arch, and start a machine with a different explicitly set arch for that machine, it doesn't seem useful to go and use the env constraint when deploying a container without an arch when what we really want is just to match the host arch for lxc [03:34] deploying containers = lxc containers [03:35] wallyworld: I suppose they only get into this scenario when using placement, so in that sense it's okay [03:35] yeah, i think so [03:35] wallyworld: I'm worried about cornering ourselves though, where we might want to start adding containers to non-empty machines [03:35] hmmm [03:36] in that case it would be up to the placement code to match the constraint... [03:36] even in that case, add lxc container - arch should match host [03:36] okay [03:36] if they explicitly speify an arch when deploying that is incompatible, then we error [03:36] but if we are just falling back to env arch, then we should "do the right thing" [03:37] eg, for kvm thy can specify different arch [03:38] o/ howdy [03:38] + addl response on bug 1420049 [03:38] Bug #1420049: ppc64el - jujud: Syntax error: word unexpected (expecting ")")

[03:39] beisner: thanks. sorry, I glanced over the repro the first time (I can't easily repro since I don't have a multi-arch MAAS) [03:39] axw, np. understandable. the attached file count is growing on that bug ;-) [03:43] axw, feel free to ping me if we should try something different. also, if/when you have a build to test, please point me at whatever ppa or CI/jenkins job has the debs? [03:44] beisner: will do, thanks. the package building is a black box to me, but I'll keep an eye out for when they're available [03:44] axw, could also set up access for you, though that would take an RT -> vpn acl addition. from there, we could coordinate access/creds. [03:45] axw, wallyworld & co., appreciate your work on this bug. [03:45] mainly axw :-) [03:45] beisner: no worries, sorry it didn't get fixed the first time [03:45] it's been interesting because we don't normally deploy the same way [03:46] but it's a good use case to know about [03:46] axw - well actually it appears to really be 2 different bugs. you got the 2-for-1 special. i think one bug was fixed, revealing another. but go is a blackbox to me. ;-) [03:47] first issue being: that funky syntax error [03:47] I just shifted the bug :) [03:47] that was fixed [03:48] 2nd issue: juju doesn't wanna use the same tools it just used to successfully deploy a unit onto the same dang box. ;-) [03:49] wallyworld, yeah this is also an edge case for us. normally everything is 1 arch. [03:50] for sure a customer will ht the issue too at some point [03:50] better we find it than them [03:55] yep indeed [04:05] axw: remind me, what's that lib for printing human readable volume sizes [04:06] wallyworld: github.com/dustin/go-humanize [04:07] axw: ta, doesn't look like we use that in master? [04:07] wallyworld: nope, I added it in the feature branch [04:07] ok, i'll look to add it [04:30] menn0, still around? [04:31] mattyw: yep [04:53] wallyworld: how would we surface an error like that? in the provisioner, we don't know the origin of the constraints [04:54] axw: at add-machine or deploy time we have the info [04:54] wallyworld: placement explicitly overrides constraints though, so I don't think it makes sense? [04:54] so we can return the error from apiserver layer [04:55] ah [04:55] yes, i forgot that [04:56] what abut add machine [04:56] will placement override constraints there too [04:57] wallyworld: I'm under the impression that placement always overrides constraints [04:57] so juju add-machine --constraints mem=2G lxc [04:58] will ignore the constraints? [04:58] i guess there it doesn't matter [04:58] as a new machine will be created [04:58] so arch will not skew [04:59] i'll revert my comment [05:00] wallyworld: ignore is too strong a word I guess. it may override aspects of them, or all of them [05:01] wallyworld: e.g. add-machine ssh:... will ignore the env constraints (or should) [05:02] ok, i was just concerned about surprising behaviour [05:02] axw: thanks for fixing the lxc tools selection [05:02] ie if the user specifies an arch constraint of one thing and the lxc container comes up on another arch [05:02] thumper: no worries [05:03] wallyworld: that was my complaint before. I'm not sure how to resolve it [05:03] yeah [05:03] you could prevent "deploy --constraints= --to lxc:N" [05:04] doesn't seem worthwhile though [05:07] maybe something to fix to improve UI, but not for now [05:08] axw: at first read the TestLxcContainerUsesConstraintsArch() seems wrong? host arch set to ppc64, expect arch amd64? [05:09] wallyworld: I just named the tests back-to-front [05:09] will fix [05:09] ta, i thought i was going mad [05:09] i misse dthe mrthod param [05:15] thumper: cmd/juju/environment/create.go:128: no formatting directive in Errorf call [05:29] wallyworld: have you pulled master today? are the tests in environs/manual failing for you? [05:29] looks upstart/systemd-related [05:30] axw: i'll check [05:31] axw: this? FAIL: provisioner_test.go:47: provisionerSuite.TestProvisionMachine [05:31] wallyworld: yes [05:31] le sigh [05:31] thanks [05:31] how the fark did that get past the bot [05:34] wallyworld: do you know why the build didn't fail when the verify.bash script got the "no formatting directive in Errorf call" ? [05:34] wallyworld: I'm nervous it was due to my IGNORE_VET_WARNINGS change from a couple weeks ago [05:34] jw4: i don't think we run vet on the bot [05:34] or i seem to recall we used not to [05:35] wallyworld: hmm - I see the warning in the last build output [05:35] http://juju-ci.vapour.ws:8080/job/github-merge-juju/2372/console [05:35] maybe we just don't fail if it exits non-zero [05:35] yes, i think that may be what we do :-( [05:36] wallyworld: well I feel better if I didn't break it :) [05:36] i don't think you did [05:36] FWIW, I've been getting a lot of provisioner test failures too [05:36] which package? [05:36] manual? [05:36] yeah [05:36] seems they started today [05:36] for me it seems to be related to OpenSSH and ubuntu@lxc [05:37] I've been getting it for at least a week [05:37] NFI how the bot let any breakage through [05:37] oh i see [05:37] wallyworld: there's a map [05:37] linuxExecutables in service/service.go [05:37] well that would explain it [05:38] i'm on go 1.3 [05:38] the bot runs 1.2 [05:38] le sigh indeed [05:38] I'm on 1.4 :) [05:38] I'll fix [05:38] yeah, I'm on 1.4 too [05:38] my first recorded error there was on 2/28 fwiw [05:50] wallyworld: small one please: https://github.com/juju/juju/pull/1738 [05:50] sure [05:52] axw: LGTM. i have to go get kid, could you please send email to juju-dev highlighting what happened and cc curtis and wes. We need to change how we land to stop this happening again [05:52] we will likely gate landings on gccgo tests passing [05:53] this will add another reason for doing that [05:53] wallyworld: ok [05:53] tyvm [06:09] jw4: what was the problem that let to IGNORE_VET_WARNINGS again? [06:10] axw: I've brought up go vet warnings from newer versions of go several times [06:10] axw: most times the answer is... go 1.2.1 is the official version [06:10] axw: to make it easy for folks to use newer versions of go and still have the pre-push hook [06:10] axw: I added that envar so that at least bypassing it is explicit rather than implicit [06:10] jw4: I see. and we can't modify the rules to work with both 1.2 and 1.4? [06:11] axw: I'm not sure I understand - as I understand it the vet rules are largely determined by the version of go you're using [06:12] axw: we have flags we pass in but those don't seem to be useful for this sort of thing [06:12] jw4: you can customise the functions that are considered printf-style, as in scripts/verify.bash [06:12] ok [06:12] axw: yeah - tweaking those hasn't addressed the issue in the past === Murali_ is now known as Murali [06:13] jw4: I don't see any vet warnings/errors, I'm on 1.4. do you still need to do this at the moment? [06:13] or was the source changed? [06:13] axw: the source was changed [06:13] axw: I think it makes sense to have that guard in there anyway [06:14] jw4: yep, it's fine since it's explicit - just wanted to understand [06:14] thanks [06:14] yw, thank you [06:26] menn0, are you done for the day? [06:56] axw: thanks for email, was well written [07:09] wallyworld: cool, nps [08:35] fwereade: when you're around say hi [09:00] dimitern, ping? [09:00] tasdomas, hey [09:01] dimitern, mattyw is afk at the moment, but he asked me to ask you to take a look at http://reviews.vapour.ws/r/994/ [09:01] tasdomas, yes, I know - will get to it at some point [09:01] dimitern, ok - no problem [09:01] tasdomas, cheers [09:02] dimitern, another question - with the laster master of juju, I'm getting test failures in api/networker and apiserver/networker [09:03] is this a known issue? [09:03] tasdomas, oh - can you paste them? [09:03] dimitern, sure, one sec [09:03] Morning! o/ [09:04] dooferlad, morning! [09:04] dooferlad, did you know that it's valid in go to do: x, y = y,x ? [09:04] dimitern: I had forgotten. Not a language feature I am used to. [09:05] by the way, is there a recommendation as to which version of go should be used for juju development ? [09:05] tasdomas, 1.2.1 - the version in precise and trusty [09:06] tasdomas, we need to make sure it works on 1.2.1 that is, it's fine if you use a more recent version otherwise [09:07] dooferlad, yeah, I was looking at that PR of yours and noticed a place where you swap two volume items in a deeply nested set of slices [09:08] dimitern: I think other than rebasing it to the head of trunk the code is OK now. Just need to get code.google.com/p/gcfg ok as a dependency. [09:09] jam, I know we discussed a policy about having 2 team leads at least approve adding a new external dependency, but how about if it's only used for tests? [09:10] morning o/ [09:11] dimitern: thanks for retry my merge. just wanted to do it and wondered where it is. :D [09:11] TheMue, morning, np :) [09:11] dimitern: probably the biggest thing for me is that if it is just for testing, then it can't be a strong enough dependency that we should actually need it [09:12] dimitern: in this case, we have a pretty strong preference for YAML in our codebase (or possibly JSON) [09:12] jam, it's just for testing [09:12] jam, and it's a about parsing INI-style files (like systemd uses) [09:13] jam, the alternative is to implement an INI parser ourselves [09:13] dimitern: that's definitely something I see creeping past "just testing" very quickly [09:13] I'm not saying we don't want it, just that it should follow the normal review for dependencies [09:14] jam, this is the dependency in question - code.google.com/p/gcfg and the PR that uses it is this https://github.com/juju/juju/pull/1718 [09:20] dimitern: "some tests don't generate valid ini files" doesn't sound like a good mix with nondeterministic ordering [09:21] at best it feels like the test is "more likely to pass" but it doesn't actually get us to determinism [09:21] jam: it isn't about generating the files, it is parsing them. The order of items in those files isn't deterministic. [09:22] dooferlad: I fully understand that part. [09:22] but your fix is that "sometimes we'll parse them because sometimes the data can't be parsed" [09:22] and what happens when the "data that can't be parsed" ends up the one that is non-deterministic order [09:22] dooferlad, yeah, that statement is a bit worrying [09:23] dooferlad: offhand I'd rather we just made the way config files are *written* deterministic [09:23] even if it means sorting the map keys before iterating them [09:23] jam: it compares the strings because sometimes we don't output an ini file during the test. Sometimes we have "a\nb\nc" for example. [09:23] dooferlad: put another way. It sounds like a problem if our config writer is iterating over maps as much as it is a problem if our tests do [09:23] dimitern, http://paste.ubuntu.com/10524916/ and http://paste.ubuntu.com/10524897/ [09:25] tasdomas, this looks like a map ordering issue - which version of go are you using? [09:25] dimitern, 1.4 [09:25] dimitern, I can try switching to 1.2 [09:26] tasdomas, yeah, it will pass with 1.2, but the test needs to be fixed nevertheless [09:26] jam: agreed. I would rather that the configs were outputted with sorted keys just from a user POV - I would rather we could diff our ini files to see what changed. [09:26] tasdomas, would you do me a favor and file a bug for it if it doesn't exist yet? [09:28] dimitern: where to investigate next regarding networking? [09:29] dimitern, sure [09:29] TheMue, I'll have a review for you shortly [09:29] tasdomas, thanks! [09:29] dimitern: great, pressing refresh in dashboard every second now *lol* [09:30] dimitern: dooferlad: so aside from the specifics of these tests, being able to easily parse the files we're writing sounds useful, and reasonable to have as a dependency so we don't have to reimplement it ourselves [09:30] one small concern [09:30] is that gcfg doesn't have any sort of guarantee that it parses the way systemd does. does it? [09:31] jam: No, ini files are not standardised. [09:32] TheMue, :) it's a long one let me warn you, but it does 99% of what's left around container addressability [09:33] dimitern: it's ok, can imagine what to expect [09:33] TheMue, I was live testing it (and fixing some places as a result) on both EC2 and MAAS for the past 2 days [09:34] * dimitern well now! isn't this just beautiful: http://paste.ubuntu.com/10525017/ [09:35] dimitern: :-D [09:35] dimitern: ouch, neat, that looks cool [09:42] TheMue, dooferlad, there it is http://reviews.vapour.ws/r/1072/ [09:43] *click* [09:43] * dimitern looks at the diff, which is slightly shy of 1000 lines [09:45] jam, dimitern: Oops. We already have go-systemd, which has a unit Deserialise function. Will switch to that. I will still need to work around unparsable strings, but it won't add a dependency. [09:45] dooferlad, oh, that sgtm then [09:46] dimitern: you are the reason why http://www.internetslang.com/ exists :-) [09:47] dooferlad, I knew it :) [09:50] dimitern: FYI there were some changes to worker/provisioner for LXC today, may need a minor merge if you haven't already done so [09:50] the pastebin looks nice :) [09:50] axw, I did - fortunately small conflict [09:51] axw, it does, doesn't it? :) and here's the one for maas: http://paste.ubuntu.com/10525174/ [09:52] (some stuff needs ironing out it seems) [09:54] very cool, glad to see it come to fruition [09:57] :) thanks axw [10:00] dooferlad, TheMue, I'm omw in ~2m [10:00] kk [10:05] Hi, I'd like to start prodding at the leadership election function. I had thought it was in 1.23-alpha1.1 but it seems not. Is there anywhere I can grab an alpha version of juju that has it in ? Failing that is there a bug I can track? [10:33] natefinch: is there anyone on your team who knows about mongo replica sets who can look at bug 1346597? the relevant error seems to be "not authorized for query on local.system.replset" [10:33] Bug #1346597: cannot get replset config: not authorized for query on local.system.replset

[10:33] this is currently blocking 1.22 and 1.23 [10:41] wallyworld: sure [10:41] tyvm [10:41] wallyworld: usually it's just that we' [10:41] we're not running as admin somehow [10:41] we've fixed or are fixing the other 2 so are close to being able to release 1.22 [10:42] natefinch: it's on oil so maybe a deployment or setup issue? [10:45] hey, there is not much worth of me as I am about to go back to bed and feel bad, but https://bugs.launchpad.net/juju-core/+bug/1346597 after comment 4 looks all kinds of invalid to me [10:45] Bug #1346597: cannot get replset config: not authorized for query on local.system.replset

[10:46] wallyworld: it could be a bug... looking at the logs now [10:46] ok [10:46] 862473 [10:46] perrito666: get well soon :-) [10:47] wallyworld: tx, I think soup and wather should get me on my feet soon and in the mean time getting some more sleep could help [10:47] indeed. [10:47] anyway, comment 4 on taht bug says that the mongo was a leftover from a dirty machine [10:47] perrito666: i could run your status-get hook tool to poll your health :-) [10:48] * perrito666 imagines getting paper notes with rpc calls every couple of seconds [10:49] perrito666: i *think* they said the latest occurrence was on a fresh instance, but am not sure [10:49] wallyworld: that bug means you are neither admin nor the machine-0 tag user (which is odd) [10:50] perrito666: yeah, it may pay to ping jason on irc to ask some questions about their set up [10:51] I'm assuming any comments on that bug from earlier than this month are unrelated [10:57] you might notice in the logs that all credentials are empty in the dial info [10:58] anyway, I would ask for the agent.conf as it might shed some light, ill go to sleep before I die on my laptop bbl [11:05] Bug #1428074 was filed: api/networker and apiserver/networker tests fail [11:06] Bug #1428074 changed: api/networker and apiserver/networker tests fail [11:16] Bug #1428074 was filed: api/networker and apiserver/networker tests fail [11:17] dooferlad, btw I've discovered that (for maas at least) in addition to enabling ip forwarding we need to enable proxy_arp as well, otherwise containers on another host cannot resolve the state server's ip [11:24] dimitern: you've got a review [11:24] TheMue, thanks! [11:24] dimitern: yw [11:28] aaargh, once again a map ordering issue [11:36] dimitern, re the bug I filed - I can still reproduce it using go 1.2.1 (built from source) [11:37] tasdomas, oh really? that's worse than I though then [11:38] dimitern, the fact that nobody else has reproduced this makes me think that I'm doing something wrong [11:38] tasdomas, let me try [11:39] dimitern: bug 1428074 - IMHO any test failure related to map ordering should be no lower than high, even critical [11:39] Bug #1428074: api/networker and apiserver/networker tests fail [11:39] tasdomas, why don't you try using golang-go package from the trusty main archive? [11:40] dimitern, that's what Im about to do [11:40] they will fail on gccgo and for those of us running go > 1.2 (there are many) [11:40] and they are easy to fix [11:40] wallyworld, I'm investigating [11:40] ok [11:40] ty [11:40] wallyworld, will change as needed [11:40] i just saw the triage to medium :-) [11:41] wallyworld, I was not aware of that decision (about map ordering issues needing to be no less than high) [11:42] tasdomas, I can't reproduce it with 1.2.1 gccgo, which means it's not a map ordering issue then [11:43] dimitern, it is or it isn't ? [11:43] dimitern: it's my opinion - i think it's a fundamental code flaw that shold be fixed as it is almost guaranteed to fail unit tests [11:43] tasdomas, :) sorry - it is not [11:43] dimitern: I have a fix without a new library. It does involve a change to juju/testing/checkers/checker.go though. [11:43] dooferlad, I'll have a look shortly [11:43] deterministically failing tests are critical, especially those we can fix [11:43] dooferlad, why the change to checkers? [11:44] wallyworld, fair enough [11:44] dimitern: so I can optionally dereference pointers in checkSameContents [11:44] dimitern: ty. it means that gccgo and those of us running > 1.2 can get clean test runs [11:44] dimitern: created a new check: SameContentsDeref [11:44] and avoid nasty tech debt for later [11:45] when we upgrade go toolchain [11:45] wallyworld, +1 that sgtm [11:46] tasdomas, bad news - I can't reproduce it with 1.4.2 either [11:46] dimitern, great, thanks [11:46] tasdomas, I'll comment on the bug [11:47] dimitern, ack [11:52] dooferlad, have you pushed the changes? [11:52] dimitern: just doing so now [11:52] ok [11:53] dimitern: pushed [11:53] dimitern: https://github.com/juju/testing/compare/master...dooferlad:master-deref is the testing branch change [11:54] dimitern: Oh, and I need to remove the change to dependencies.tsv. That will arrive shortly. [11:55] dooferlad, can you expand a bit why the Indirect was needed? [11:57] dimitern: because comparing pointers wasn't useful :-) Basically, we were comparing two lists of pointers and reflect.DeepEqual wasn't following them, only examining their value. [11:57] dimitern: or at least that seemed to be the case. [11:57] dimitern: without the change, the test failed. With the change it passed 100 times in a row. [11:57] Bug #1428074 changed: api/networker and apiserver/networker tests fail [11:58] dooferlad, ok, great [11:59] dooferlad, this change seems useful enough to be unconditionally applied to SameContents checker I think [11:59] dimitern: do I propose the juju/testing branch in the same way as juju/juju? [11:59] dooferlad, but it needs more tests as well in juju/testing [12:00] dimitern: seems reasonable [12:00] dooferlad, however, I'd like you to scan our juju/* repos for places SameContents is used and see if making the change will break anything [12:00] dooferlad, there should be a very few places indeed [12:02] dooferlad, as for how to propose it - the usual way - fork juju/testing to your account, wipe out your $GOPATH/src/github.com/juju/testing/ repo, clone your fork in the same place, add upstream remote to the master repo, etc. [12:09] dimitern: goamz v3 regions is missing 2 that i can see... is it possible or m i looking in the wrong place? [12:09] anastasiamac, which ones? [12:09] dimitern: EU (Frankfurt) eu-central-1 [12:10] anastasiamac, that one was never added [12:10] IIRC [12:10] dimitern: i was looking at prices and there is a price for it... [12:10] dimitern: another one that comes up is AWS GovCloud in US [12:10] dimitern: shall i just ignore these 2 then?... [12:11] anastasiamac, prices? [12:11] anastasiamac, you mean in juju source? [12:11] dimitern: http://aws.amazon.com/ec2/pricing/ [12:11] dimitern: and putting updating them in juju for bug 1427840 [12:11] Bug #1427840: ec2 provider unaware of c3 types in sa-east-1 [12:12] anastasiamac, oh, of course - they are indeed available and used (not sure about the gov one - you should be working for the us gov perhaps) [12:12] anastasiamac, but since both are fairly new goamz doesn't yet have them [12:13] anastasiamac, there was even a bug filed somewhere that eu-central-1 is missing [12:13] dimitern: k, so m put n them in on our side intoec2/instancetype [12:13] anastasiamac, yes, unless you need them I think you can safely ignore them :) [12:14] anastasiamac, that was too concise for me to understand, sorry :) [12:14] dimitern: i think that i could get away with putting them into struct as string... and then goamz will need to be updated... [12:15] dimitern: m adding the prices for these 2 regions into ec2/instancetype as they are specified by string :D [12:15] dimitern: but on goamz side they r still missing as regions :D [12:15] anastasiamac, ah, so you do need them? [12:16] dimitern: if the pricing is on aws website, do i need to have it in juju?... I would have thought that the answer is "yes" [12:16] dimitern: unless our point of truth is goamz and not aws source... [12:16] dimitern: no? [12:17] anastasiamac, well, think about it this way: we only support what we have tested, so unless you need to test stuff on those regions and need the pricing for them [12:18] anastasiamac, then yes, add them but also update goamz and test it works on the new regions [12:18]