[00:11] thumper: alexisb email sent [00:11] tell me what you think [00:11] certainly more than 6 weeks work :) [00:22] sinzui: i am blocked [00:23] something is fucked up on batuan and I cannot pull source from github [00:23] [rt.admin.canonical.com #72748] [00:23] <_mup_> Bug #72748: Crash while exiting [00:23] going to talk to #is now [00:24] davecheney, I have copied files to a machine inside the network, hackes /etc/hosts to point to that machine. I have also created certs that claim to be the other site [00:24] * sinzui doesn't accept no [00:26] sinzui: i'll try to debig this on amd64 for the moment [00:30] bodie_: review of PR 141 finished for now [00:39] sinzui: crisis averted [00:39] connectivity restored [00:44] sinzui: interesting, I cnanot reproduce the build failure on my pcc64 machine [00:44] we saw it several times in ci http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/ [00:45] dave the machine is stilson-07, do you want to visit it? [00:46] davecheney, ^ [00:50] davecheney, ssh ubuntu@10.245.67.135. Your keys are there. maybe you can find a older or newer version of gcc there [00:53] davecheney, oh, 10.245.67.136 saw the same error. so both the unittest machine and the packaging machine hate juju [00:58] sinzui: intersting [01:01] sinzui: trying stilson now [01:08] sinzui: stilson-07 is running the outdated version of gccgo [01:08] sinzui: are you permitted to update it ? [01:18] sinzui: wtf, stilzon-07 doesn't even have bzr in the path ... [01:18] that's like saying the machine is actually running fedora [01:19] menn0: thanks for the review [01:19] wallyworld: can you also take a look at https://github.com/juju/juju/pull/148/ ? there's some changes to the instance-type constraint handling that you added [01:20] davecheney, We try to keep the machines clean. tests purge the left overs, they tests often export all the locations needed [01:20] axw: oh hi, i had looked, was waiting for you to come online [01:21] thumper: added some comments on the doc, heading your way [01:21] sinzui: short version [01:21] axw: i don't quite see the purpose from the covering letter - the azure provider already calls into common instance matching functionality [01:21] you need to upgrade the compiler [01:22] hurray [01:22] to the one that is (hopefully) in trusty updates [01:22] wallyworld: I'll find the code in question for you, just a sec [01:22] bug is fixed in that version [01:22] wallyworld: would've been more obvious with a branch preqreq'ing ;) [01:22] sinzui: I don't know how you can reconcile that with the requirements of the lp builders [01:22] yup [01:23] wallyworld: https://github.com/juju/juju/blob/master/provider/azure/instancetype.go#L19 [01:23] all this preferredTypes stuff can go [01:23] and be replaced with a call to MatchingInstanceTypes [01:23] axw: also, this line "if len(itypes) == 0 && cons.Mem != origCons.Mem {" <-- the semantics seem slightly different from the original origCons.Mem != nil [01:24] davecheney, the unittest run called this to get the compiler [01:24] sudo apt-get install -y build-essential bzr distro-info-data git-core mercurial zip rsyslog-gnutls juju-mongodb gccgo-4.9 gccgo-go [01:24] wallyworld: the idea is that above it'll only try again if it tried above with implied mem=1G [01:24] sinzui: the update has not landed [01:25] it's still stuck in whatever process is blocking it [01:25] i had to install the compiler from ppa [01:25] The builder called this sudo apt-get install -y build-essential fakeroot dpkg-dev debhelper bash-completion gccgo-4.9 gccgo-go [01:25] yes, you said [01:25] but it looks liek the compiler has not made it out of the -propose pocket [01:25] the original ppc bug is not marked fixed-released [01:25] wallyworld: I could do "!cons.HasInstanceType() && origCons.Mem == nil", but I thought this was clearer [01:26] sinzui: https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/1304754 [01:26] <_mup_> Bug #1304754: gccgo has issues when page size is not 4kB

[01:27] davecheney, more interestingly. all machines got their compilers from ppa, but the ppa [01:27] what ppa ? [01:27] davecheney, the machines use ubuntu-toolchain-r-ppa-trusty.list per your instructions [01:27] axw: so you plan on changing the selectInstanceTypeAndImage() method? [01:28] davecheney, stilson-7 is missing it though [01:28] wallyworld: yes, but I think selectMachineType is more relevant [01:28] wallyworld: actually it could probably just be replaced [01:29] wallyworld: I could just propose my other branch [01:29] as a wIP [01:29] axw: i'm thinking that both openstack and ec2 providers call into FindInstanceSpec() and their code handles instance type constraints etc as is - can we just tweak azue to use the same code? [01:29] without exporting anything? [01:29] wallyworld: maybe... I'll see what they do [01:30] sinzui: ppa is not installed on stilson-07 [01:30] apt-get update confirms it [01:30] I am adding it now. I think a test replaced it... [01:30] wallyworld: do they allow you to explicitly specify the image name? [01:30] axw: i'm all for refactoring if it improves the code, so if it's necessary, go for it. but i also think we'd want azure, ec2, openstack, joyent etc to be consistent [01:30] ah, imagename [01:30] davecheney, the machine with the right ppa failed the same way [01:30] sorry [01:31] i was getting confused with instance type [01:31] wallyworld: this is caused by having to deal with force-image-name [01:31] hmmm. we specifically removed image name selection from ec2 etc [01:31] ie the image-id config was removed [01:31] because people could do bad things [01:32] like forcibly specify an image that didn't match tools [01:32] the fact that azure added support for it is unfortunate [01:33] i'm almost inclined to see if we shouldn't look at deprecating it [01:33] axw: np [01:33] it may have been useful early on when azure images were changing rapidly and we didn't have simplestreams [01:34] wallyworld: can do, but I think we need to give people notice. is exposing this function really that bad? [01:34] sinzui: can I see the failure from that machine ? [01:36] davecheney, http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/606/consoleFull [01:36] axw: yeah, it would be deprecated over a release cycle. in which case i guess we could expose the function. but any time we expose previously internal stuff is potentially an issue we'd prefer to avoid. but in this case we could do at and hide it again when azure is changed [01:37] yep, that sounds fine to me [01:37] ok, let me take another look at the pr [01:37] ta [01:37] davecheney: arm email looks good to me [01:38] axw: also, i'd be interested in your thoughts on review board as per martin's email [01:38] davecheney: thanks [01:38] wallyworld: yeah taking a look now [01:39] sinzui: that machine is also lacking the ppa [01:39] oh no [01:39] i'm sorry [01:39] davecheney, ubuntu-toolchain-r? [01:39] i guess [01:39] i cannot see from that output [01:40] all i see is [01:40] Get:3 http://ppa.launchpad.net trusty/main ppc64el Packages [11.9 kB] [01:40] sinzui: can you add a [01:40] gccgo -v to the build script, just before doing go build [01:40] that will settle it [01:41] davecheney, that is a little tricky since that would error on golang [01:41] eh ? [01:41] davecheney, http://pastebin.ubuntu.com/7693053/ [01:41] wallyworld: are you able to get onto reviewboard atm? it's refusing my connection, just wondering if I've set up the tunnel right or not [01:41] interesting [01:41] that is the correct version [01:42] that matches winton-07 [01:42] that matches winton-09 [01:44] wallyworld_: are you able to get onto reviewboard atm? it's refusing my connection, just wondering if I've set up the tunnel right or not [01:44] axw: yeah, worked for me [01:45] hrm [01:45] did you set up proxy in browser? [01:45] wallyworld_: seems busted. I can't even telnet to 8080 on the machine. [01:45] oh, it's 80 isn't it [01:46] axw: all i did was run the ssh -D command, and then that logged me into a session. then i changed the browser to use a socks proxy and it worked [01:46] i used port 8080 in the socks proxy config [01:47] wallyworld_: I was trying to be clever and just forward the port, but I chose the wrong destination port. never mind :) [01:47] ok :-) [01:48] davecheney, stilson-07 with the correct ppa failed the same way http://juju-ci.vapour.ws:8080/job/publish-revision/546/console [01:48] davecheney, I am still deploying a hack to print the compiler to all the slaves [01:49] gccgo -v :: [01:51] sinzui: confirmed [01:51] same compiler [01:51] virtually same kernel [01:51] different results :( [01:51] sinzui: do you know what the policy would be in terms of deprecating and removing a juju feature which shipped in trusty? can we do that over a 1.20->1.22 release, or are we stuck with it forever? [01:54] wallyworld_, jamespage's proposal is co-installable version, so we can deprecate in devel and obsolete next devel. We will continue to state modern client works with older server 1 stable behind === wallyworld_ is now known as wallyworld [01:59] davecheney, I can reboot? I have rebooted stilson-08 a lot actually because the ppc bug past week cause the disk and ram to fill up [02:00] wallyworld: how did you create a review from a PR in reviewboard? [02:00] axw: i clicked on the new review link and it showed the recent branches [02:00] wallyworld: did you just get the diff and upload it? [02:00] i then clicked on the branch [02:00] huh ok [02:00] I don't get that... [02:01] no New Review menu? [02:01] wallyworld: it just shows me the juju/juju repo and master branch [02:01] hmmm [02:01] so you have the menu but it doesn't show any branches? [02:02] wallyworld: I clicked "New Review Request" at the top, and the only repo is "juju" which is juju/juuju [02:02] juju* [02:02] no obvious way to add my own [02:04] hmm, let me re setup the tunnel etc and try [02:04] sinzui: sure [02:05] sinzui: stilson-07 looks fucking sit [02:05] dmesg shows continual crashes [02:08] axw: just emailed a screenshot showing what i get when i click "New Review Request" [02:08] s/sit/sick [02:08] wallyworld: yeah that's what I see too. that's merged commits. [02:09] oh balls, it is too [02:09] i didn't look too closely [02:09] so looks like the rbt tool is needed after all [02:10] yeah, which is a bit crap IMO [02:10] yep :-( [02:10] i wonder if gerrit is any better [02:10] waigani: you're ocr today ? [02:10] wallyworld: everyone says gerrit is the one to go for [02:11] davecheney: we'll take a look at it. trouble is, they're all crap compared to lp [02:11] wallyworld: i'll agree to disagree with you there [02:12] no inline commenting on lp was a non starter for me [02:12] davecheney: what don't you like about lp? it has the best review queue, can mark stuff as wip, is *not* patch based, supports pre-req branches etc etc [02:12] none of the others do that very well at all [02:12] wallyworld: lack of inline commenting on reviews [02:12] meh [02:12] lp has that now anyway [02:13] damn, too late [02:13] since maybe about a month [02:13] like i say, we don't have to agree on this point [02:13] sure :-) [02:36] sinzui: did you reboot stilson ? [02:36] davecheney, 7 and 8 are rebooted. 7 is back [02:37] and 8 is back [02:37] [ 5.238055] init: plymouth-upstart-bridge main process ended, respawning [02:37] [ 7.117990] init: pollinate main process (797) terminated with status 1 [02:37] this machine is sick [02:45] davecheney, we could backout your change for a half a day to get a pass rev we can release. or may rewrite the offending code to be friendlier to gccgo [02:47] sinzui: yup, will revert [02:47] does anyone know how to do that ? [02:49] wallyworld: can you help revert a merge ? [02:49] which one? [02:49] https://github.com/juju/juju/pull/144 [02:49] i think you just reverse cheery pick and propose a pr [02:49] wallyworld: i have _never_ done this before on git [02:50] me either :-) [02:50] it's guarenteed i'll scrwe it up [02:50] might be easier to work around the compiler bug [02:50] davecheney, I don't have confidence in that myself. [02:50] davecheney: it that what breaks the compiler? [02:53] sinzui: wallyworld I have a workaround for the compiler crash [02:53] pls hold [02:53] ok [02:53] \o/ [02:54] hang on [02:54] shit is weird [02:55] my local working copy of master does not match what i'm seeing on stilson [02:55] fuck, forgot to pull master [02:57] ok, running tests [02:57] PR comming ASAP [03:07] wallyworld: did you see the review request for gwacl? [03:07] no, i'll look [03:07] wallyworld: https://code.launchpad.net/~axwalk/gwacl/rolesizes-update/+merge/224103 [03:09] axw: i know, let's just write a html page scraper, WCPGW [03:10] wallyworld: heh :) [03:10] sinzui: wallyworld https://github.com/juju/juju/pull/151 [03:11] davecheney, jam made a similar fix last week [03:13] davecheney: we could leave the authEntityTag alone and just do if names.NewMachineTag(parentId).String() == authEntityTag.String() { ?? [03:14] wallyworld: sure, [03:14] just a suggestion to preserve more of the original branch [03:14] but i've tested this version [03:15] ok, i was just thinking that we'd want to keep various getAuth functions consistent [03:15] this isn't forever [03:15] ok, +1 [03:17] davecheney, there is a cursed branch in front of yours. github-merge-juju will be available in 30 minutes [03:17] sinzui: ok [03:17] i don't know what you mean [03:17] but ok [03:40] I think this is ready to go -- https://github.com/juju/juju/pull/140 [03:40] would appreciate a lgtm / lbtm :) [03:49] bodie_: I'll take a look [03:49] davecheney: do you want to get this merged or would you prefer that the tests I suggested get done first? https://github.com/juju/juju/pull/127 [03:50] menn0: i'm not fussed [03:51] davecheney: let's just get it in. it's got my LGTM. [03:57] cool, thanks [03:58] panic: runtime error: invalid memory address or nil pointer dereference [03:58] [signal 0xb code=0x1 addr=0x1 pc=0x40761f] [03:58] goroutine 1438 [running]: [03:58] runtime.panic(0xed70c0, 0x1f9c7c8) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6 [03:58] github.com/juju/juju/state/apiserver.func��009(0x0, 0x0, 0xc210256c60, 0x30941b60a0, 0x414361, ...) /home/ubuntu/juju-core_1.19.4/src/github.com/juju/juju/state/apiserver/root.go:169 +0x5ed [03:58] github.com/juju/juju/state/apiserver.(*srvCaller).Call(0xc21033cb40, 0x0, 0x0, 0x0, 0x0, ...) /home/ubuntu/juju-core_1.19.4/src/github.com/juju/juju/state/apiserver/root.go:101 +0x3f [03:58] github.com/juju/juju/rpc.(*Conn).runRequest(0xc210859500, 0x7f9e941e1b18, 0xc21033cb40, 0x129fb20, 0x12, ...) /home/ubuntu/juju-core_1.19.4/src/github.com/juju/juju/rpc/server.go:533 +0xd5 [03:58] created by github.com/juju/juju/rpc.(*Conn).handleRequest /home/ubuntu/juju-core_1.19.4/src/github.com/juju/juju/rpc/server.go:462 +0x671 [03:58] err [03:58] didn't jam make a fix for this ? [03:59] r.objectCache[objKey] = objValue [03:59] return objValue, nil [03:59] the hell [03:59] that means r.objectCache is nil [04:00] could use a bit more review on this as well -- menn0 if you're interested -- jcw4 https://github.com/juju/juju/pull/141 [04:02] davecheney: he has a fix but he hasn't merged it yet [04:02] davecheney: PR 146 [04:02] cool, thanks [04:02] i'll check it out [04:03] davecheney: it's fairly sporadic [04:04] bodie_: that extension to the tests for #141 looks good. I had some other comments too. [04:05] thumper: did you see this? https://github.com/juju/juju/pull/108#issuecomment-46822183 [04:06] thumper: fwereade is right that the implementation is probably not quite right given the plans in the identity spec. most of the work is still useful though. I wonder if it should merged but disabled. [04:07] menn0: urgh, shit, lots of races in cmd/jujud [04:07] davecheney: where? what? [04:08] menn0: pls hold [04:09] sinzui: that gccgo fix landed [04:09] \o/ [04:09] how do I kick off the ppc build ? [04:10] menn0: http://paste.ubuntu.com/7693425/ [04:10] as suspected there is a race on the api server root hashmap [04:10] davecheney, you can't CI already sees there is a new revision http://juju-ci.vapour.ws:8080/ [04:10] sinzui: ok [04:11] davecheney, Both publish-revision and run-unit-tests-trusty-ppc64el will start in 5 minutes [04:12] sinzui: https://bugs.launchpad.net/juju-core/+bug/1333513 [04:12] <_mup_> Bug #1333513: state/apiserver: data race on on apiserver method hashmap

[04:12] ^ blocks 1.19.4 i'm afraid [04:12] :( [04:13] i can try jam's fix and see if that helps [04:14] menn0: hey, back at my desk now [04:14] menn0: let me look [04:15] thumper: k [04:16] davecheney: that fail is due to the thing that jam has fixed but not merged. I believe there was discussion about the best way to fix it which might be why there's a delay. [04:22] menn0, I had a few replies for your comments -- I need confirmation from jcw4 on at least one of them since 141 is actually an API for the Watcher, which itself is tested elsewhere -- I'm not entirely sure all the error cases need testing, since many of them are things from other packages [04:22] the API should really be simple, I think, but I'm open to being wrong about that :) [04:23] bodie_: give me a bit, I'm in the middle of your other review [04:24] sure thing, it's way past EOD for me here so I'll be reviewing in the morning -- hopefully we can sync effectively :) I really appreciate the comments [04:57] bodie_: no problems [05:01] sinzui: ppc build passed! [05:01] w00t [05:06] menn0: what shold we do [05:06] 1. release with this know issue ? [05:06] 2. wait a few hours for jam ? [05:06] 3. try to fix it ourselves ? [05:07] davecheney: I don't think we can release. This problem will happen in actual use. [05:07] ok [05:07] i agree [05:07] there are also a shitload of other races in that paste [05:07] davecheney: it looks like jam has a valid fix that several people approved [05:07] i only made an issue for the first one [05:07] menn0: hit it with $$merge$$ [05:08] davecheney: there was quite a bit of discussion though so I wonder if jam1 was planning on doing something better [05:09] davecheney: menn0: the fix for *my* code is just fine, the ancillary fix for other stuff that showed up with -race is in question, but I'm looking into it right now. [05:09] :) [05:10] davecheney: maybe just go with what jam has right now and he can always change it later. [05:10] i agree [05:10] hitting merge now [05:12] davecheney: trying to use "go test -compiler=gccgo" I got this error: http://paste.ubuntu.com/7693569/ [05:12] thoughts? [05:12] jam1: fix in trunk [05:12] running it a second time and it passed... [05:12] ah, nm, still breaks [05:12] landed a few minutes ago [05:13] I forgot I run the regular tests first [05:13] davecheney: thanks [05:13] jam1: did you catch that davecheney and I just hit $$merge$$ on your objectCache PR? [05:13] davecheney: is that a fix in "juju" trunk, or a fix in gccgo trunk [05:13] ? [05:13] jam1: workaround in juju [05:13] menn0: yeah, though I did *just* push up the alternative fix [05:14] jam1: crap. [05:14] jam1: shall we cancel the merge then? [05:14] menn0: meh we can just submit it again, I think if we missed something. It is noise but not terrible [05:17] menn0: there is a queue 3 deep already and maybe the bot doesn't track the tip version that was voted on, in which case it will get the updated one. We'll see. [05:18] jam1: I was wondering the same thing. [05:19] tarmac was careful about it, because you could be approving 3rd party proposals, though it often was annoying to have to go and reapprove since most of our branches were actually trusted. [05:19] I don't know the new bot nearly as well. [05:20] davecheney: I just send off that status API PR for merging too [05:20] thanks [05:20] sinzui: still there ? [05:21] when do we have to stop screwing around so you can cut a release ? [05:50] jam1: did you still want to catch up? [05:54] OMG... lots of addressing document comments, but no other work [05:54] * thumper feels exhausted [05:54] it is all rick_h_'s fault [05:54] yes rick_h_, read this when you wake up and know that you did this to me :-P [05:54] * thumper will add more tomorrow [05:54] night all [06:02] wallyworld: axw http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/ [06:02] ^ can you push go on this build please [06:02] sure [06:02] ta [06:02] davecheney: what revision? tip? [06:05] wallyworld: tip is fine [06:05] ta, i started it with the sha of your latest commit [06:05] yup [06:05] that is the one i want to check [06:05] make sure I havn't just screwed things again === vladk|offline is now known as vladk [06:16] fwereade: if you are around and have time for a 5-minute chat about a tasteful way to share common code I'd like to run some ideas by you. [06:19] jam1, sure [06:19] fwereade: https://plus.google.com/hangouts/_/canonical.com/juju-sapphire [06:43] mornin' all [07:09] wallyworld: playing with wercker [07:09] it looks like we can setup custom build environments [07:09] trying to get the juju tests to pass [07:09] cool [07:09] wallyworld: you get pre commit tests [07:09] on a per branch basis [07:10] the werker bot participates on the PR and says if the build passes or not [07:12] i've probably screwed my working copy [07:12] doing all this [07:17] https://app.wercker.com/#buildstep/53a925f9770aadd70b0f1944 [07:19] well [07:19] that didn't work [07:19] and i'm out of fucks for today [07:22] fwereade: so I've made a couple comments, but now Google is telling me "file not found" anytime I click on the doc. [07:23] fwereade: It comes to mind that if we do the "this hook is only called if the other hook doesn't exist" model, then we could call it "unhandled-changes" [07:23] jam1, that's just the "missing" hook, and it'll get called 10000 times, isn't it? [07:23] jam1, (fwiw loading it in a new tab works for me) [07:23] fwereade: well it could be "missing" but have the semantics that it is only called once for a sequence of hooks [07:26] jam1, it's not quite working for me, but maybe I just need to think it through more [07:26] fwereade: so I mentioned in my comments that either semantic would be possible [07:26] "always queue one of these" vs "only queue if the hook wasn't there" [07:27] fwereade: I still get it to load and show the first time, but clicking anywhere says there is a problem. Can you share the doc directly with me? [07:28] jam1, shared [07:29] fwereade: thanks, it seems happier [07:30] fwereade: I just brought it up because I think it might change the name of the thing, and naming it seems problematic right now :) [07:32] jam1, I dunno, calling it once in response to any sequence of unimplemented hooks feels quite different to calling it when stable (and occasionally if it looks like we're not stabilising any time soon) [07:33] jam1, tying it to other hooks feels like complexity for no benefit -- maybe I'm just not getting the value though [07:33] jam1, you can implement install, you can implement relation-broken, etc etc [07:34] fwereade: so tying it to other hooks is because existing charms are going to have this symlink farm, and it will mean that it actually gets called significantly more often [07:34] jam1, having it so if you implement r-b you basically always see s-c, r-b, s-c, r-b, s-c, r-b, s-c, r-b, s-c when leaving a few relations -- vs r-b, r-b, r-b, r-b, s-c [07:34] jam1, ok, but the symlink farm is stupid and broken [07:35] jam1, getting rid of that is almost the whole point [07:35] fwereade: sure, getting there from here is something to be aware of. [07:36] jam1, I don't follow [07:36] fwereade: juju certainly doesn't have the concept of "I can deploy version X of charm Y", but not the latest. [07:36] jam1, symlink farms keep working until they delete everything [07:36] jam1, er, yes it does [07:36] so charms that *just implement* the new hook still need the symlink farm [07:36] jam1, ahhh ok [07:36] jam1, sorry, yes, I think we need charm feature flags much sooner than later [07:38] jam1, i reverted the network model doc to before domas changes [07:38] fwereade: so the thing with even feature flags is that Juju-1.18 could still install version 10 of the charm, it is just version 11 that it can no longer install, but juju has no way of knowing/representing that to the user. [07:38] dimitern: is that the giant paste in the header? [07:38] jam1, yep [07:38] jam1, expand please? [07:39] jam1, the charm store needs to know about feature flags too, it is true [07:39] jam1, and getting a version of 1.18 out that understands and uses them is important [07:39]