[00:03] dannf: yeah, seems to be working for me now. i think it was just so slow yesterday that i gave up. i also tried building juju from src and i gave up after 30 minutes. maybe the vms are ust slow [00:13] thumper: wallyworld__ on my ppc systems I see the provisioning constantly (every 300 sec) polling the charm store [00:13] this fails because FIREWALLS! [00:14] but i wonder why does it poll at all ? [00:14] davecheney: to see if charms are out of date [00:14] so that can be shown in status [00:14] wallyworld__: ok, so i should raise an RT to get access to charm store enabled [00:14] looks like the proxy is blocking it [00:14] yeah that would be good [00:15] wallyworld__: ill fix [00:15] davecheney: so when you run status, it says "hey you have mysql version 10 installed but version 12 is available" [00:40] ahh nice [00:40] https://bugs.launchpad.net/juju-core/+bug/1305365 [00:40] <_mup_> Bug #1305365: juju 1.18.0 environment unusable after bootstrap [00:40] ^ what the hell [00:41] it bootstrapped and deployed fine [00:41] why does status have a whinge [01:00] thumper, i'm trying to capture output that's been written with logger.Errorf (from within in a coretesting.RunCommand). is there a way to hook into loggo like this? [01:00] i've tried coretesting.Stdout(ctx), and Stderr, but nothing there [01:08] cmars: hey [01:08] hi [01:08] cmars: chances are you are using a LoggingSuite [01:08] it caputres the logging [01:09] it is in the gocheck *gc.C thingy [01:09] as some test log [01:09] normally we don't test logging [01:09] what are you after exactly? [01:10] hi axw [01:10] hey thumper [01:10] axw: was asked to hit you up about a critical bug [01:10] bug 1303735 [01:10] ok [01:10] <_mup_> Bug #1303735: public-address change to internal bridge post juju-upgrade

[01:10] thumper, the recent error message I've added. i think that'll do it, thanks [01:11] apparently you were talking with rob about it [01:11] ah, was chatting with rog about that last night... [01:11] cmars: there are some new methods on the context object [01:11] cmars: if you are testing the error [01:11] then you should check the error response from Run [01:12] cmars: testing.RunCommand returns a context and an error [01:12] the error is what you want to be checking [01:13] axw: do you know what the issue is there with the addresses [01:13] ? [01:13] I recall work going on there [01:13] but I don't know exactly what changed [01:14] thumper: it looks like the openstack environment has no public addresses, only cloud-local and/or unknown; the SelectPublicAddress code chooses the last cloud-local/unknown address in the list... so it gets one of the unknown addresses that the machiner records [01:15] hmm... [01:15] so... how do we go about fixing it? [01:15] thumper: so, we should probably prefer the provider addresses over machine addresses if they're all cloud-local/unknown [01:16] are we able to tell the difference [01:16] ? [01:16] yes, they're recorded in separate lists [01:17] thumper: it would be ideal if the machiner could decide which addresses were machine-local, but it's not a straightforward thing to do [01:17] apart from localhost addresses of course... [01:18] right [02:08] thumper: davecheney has LGTMed https://codereview.appspot.com/85710043/ did you want to take a look before I land it? [02:08] yes please === alexlist` is now known as alexlist [02:15] ah, I should remove loggingSuite from environ_test.go [02:33] https://bugs.launchpad.net/juju-core/+bug/1305386 [02:33] <_mup_> Bug #1305386: state/apiserver: multiple data races [03:03] davecheney: what causes these and do you have any suggestions to fix? [03:06] thumper: not sure yet [03:06] i'm hoping its something we're doing (sharing a mgo conn) not a driver bug [03:06] its not clear if these are cosmetic or serious [03:12] thumper: awesome, found out why go install was hanging. trunk of code.google.com/p/go.crypto has changed and juju-core no longer compiles but that causes go install to just hang unless you compile with the -v option [03:12] how good is thst [03:12] wut? [03:12] hah [03:12] so we had better not update go.crypto [03:12] why? [03:12] no idea [03:12] wallyworld__: so you should run godeps, yeah? [03:13] seems so [03:13] but i mean wtf [03:13] it should have told me there was a compile error [03:13] i would have thought so [03:14] i just thought it was slow cause other stuff has been slow [03:15] wallyworld__: is the compiler spinning ? [03:15] agl and hanwen landed that big branch today [03:15] davecheney: not sure, how do i tell? [03:15] i didn't see any discussion about it [03:15] top ? [03:15] that was rather unorthodox [03:16] davecheney: strangely using the -v flag which prints aeach package as it is compiled printed out the errors also and then exited when done [03:16] wallyworld__: paste [03:16] ? [03:18] davecheney: https://pastebin.canonical.com/108166/ shows first attempt hanging, and then -v showing errors [03:18] wallyworld__: can you use pastebin it [03:18] my 2fa is downstairs [03:19] ok [03:19] http://pastebin.ubuntu.com/7229253/ [03:20] wallyworld__: can't tell if a hang, or just took a long time [03:20] try [03:20] rm -rf $GOPATH/pkg [03:20] davecheney: it ran for over an hour before i hit ^C [03:20] wallyworld__: did you look at top ? [03:20] no, not at the time [03:20] bummer [03:20] as for tip of ssh being broken [03:20] yes, that is poor form [03:20] but we're not blameless here [03:21] the top of gomaasapi is unusable isn't it [03:21] or is it gwacl ? [03:21] not sure [03:21] davecheney: i ran it again and that time it exited with the compile errors even without the -v flag [03:22] i don't -v would have any impact on a hang [03:22] -v is for cmd/go [03:22] which just forks the compiler [03:22] its hard to say uless you can get it to happen again [03:22] i'd 100% believe go get hanging [03:22] i read in help that -v is for verbose [03:22] wallyworld__: yes, that is what it does [03:23] go get can get confused when hg/bzr/git fire off some program to deal with merge conflicts as the pty they run against isn't a terminal [03:23] ok [03:25] davecheney: right, after running godeps, install now appears to be spinning [03:26] which process is spinning [03:26] cmd/go [03:26] yep [03:26] or the compiler ? [03:26] kill cmd/go with SIGQUIT [03:27] capture the output [03:27] great [03:27] this is some pretty shit [03:27] 11th hour and everyting is breaking, including the toolchain [03:27] wonderful [03:28] davecheney: http://pastebin.ubuntu.com/7229265/ [03:28] something about stack unavailable [03:29] nuking pkg fixes it [03:31] wallyworld__: whos' machine is this ? [03:31] its running gccgo [03:39] thumper: \o/ and now i have it compiled i can't test anyway because of bug 1304742 FML [03:39] <_mup_> Bug #1304742: version reports "armhf" on arm64

[03:39] i guess i should fix that [03:39] yeah... guess so [03:40] * wallyworld__ sighs heavily [03:45] * axw joins in [03:45] HP cloud doesn't like my change [03:49] supportedArchitectures isn't an ordered list, right ? https://bugs.launchpad.net/juju-core/+bug/1305397 [03:49] <_mup_> Bug #1305397: provider/common: test failure

[03:50] * davecheney goes to lunch [03:50] in the rain [04:04] davecheney: another one http://pastebin.ubuntu.com/7229318/ [04:21] faaaark. only took 1000 attempts but finally got the go compiler to build without panicing === vladk|offline is now known as vladk [05:17] wallyworld__: eerk, that looks simlar to the bug we see on 64k ppc64 kernels [05:18] davecheney: yuk. i'm not having much luck with juju per se either. machine 0 agent won't start properly [05:18] wallyworld__: more nil pointers ? [05:18] doesn't seem to get past starting the start server [05:19] not sure yet if it's because it can't see the db or something else [05:20] wallyworld__: using lxc ? [05:20] nope, arm vms [05:20] manual provisioning [05:20] ok [05:21] lxc is broken atm [05:23] :-( [05:24] wallyworld__: i know [05:25] i'm running from trunk and it seems like it's dying inside the ensure ha stuff but there's not enough logging to know for sure [05:25] :( [05:25] :emoji crying: [06:29] davecheney: i farking give up. takes 1000 retries to get a compile. then complains about missing libgo.so.5. so find that i need extra compile flags. 1000 retries later get new jujud. still libgo.so.5 missing. so i install deb package on target machine just to get that shared lib. rinse and repeat. still complains. sigh [06:32] wallyworld__: see https://docs.google.com/a/canonical.com/document/d/1m9R2n6LPLNLGjdopcNkQYVG8D5V4FTyvc1vvn-9ZifM/edit [06:32] at the bottom [06:32] the gb alias [06:32] wallyworld__: can you paste me anything intersting from dmesg [06:32] davecheney: i used those flags [06:32] i want to compare with the ppc64 problems [06:33] if it's complaining that libgo.so.5 is missing [06:33] binaries were about 30% bigger after [06:33] then it didn't work [06:33] wallyworld__: i think you should cut your losses [06:33] so then i tried installing the lib directly on the target machine [06:33] looks like juju doesn't work on arm64 [06:33] i don't know if anyone is working on that atm [06:34] we're close [06:34] dmesg doesn't have anything that interesting [06:35] wallyworld__: ldd $(which jujud) [06:35] ubuntu@ms01a:~/juju/src/launchpad.net/juju-core/cmd/jujud$ ldd $(which jujud) [06:35] linux-vdso.so.1 => (0x0000007fafbdd000) [06:35] libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007fafba7000) [06:35] libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007fafb07000) [06:35] libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007fafae4000) [06:35] libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007faf999000) [06:35] /lib/ld-linux-aarch64.so.1 (0x0000005595496000) [06:36] hmm, so libgo.so.5 is missing [06:36] i built with [06:36] go install -a -v -gccgoflags -static-libgo launchpad.net/juju-core/... [06:36] ah missing = [06:37] fark [06:37] wallyworld__: nope [06:37] you got it right [06:37] if you got it wrong [06:37] there would be a line showing [06:37] libgo.so.5 => (missing) [06:37] ok [06:38] but you should probably use the = [06:39] used the =, same result [06:40] mornin' all [06:41] in this case it will be [06:41] but if you wanted to pass a second option [06:41] you'd need = and to quote it [06:41] rogpeppe2: o/ [06:41] davecheney: yo! [06:44] morning rogpeppe2 [06:45] axw: hiya [06:45] rogpeppe2: I've got a CL for the addresses bug, about to propose (just finishing live test) - got time to have a look? [06:45] axw: sure [06:46] rogpeppe2: I made the changes we talked about, but also changed instance.NewAddress derive scope from IPv4 address range [06:46] so 10.*, 192.168.*, etc. are recorded as cloud-local [06:47] axw: that seems reasonable to me [06:49] axw: although that's not really true in the case of this bug [06:49] axw: the 192.168.* addresses were machine-local [06:50] rogpeppe2: I think that's okay, because we'll still choose provider cloud-local over machine cloud-local [06:50] machiner* [06:51] axw: yeah, it should be ok [06:51] * axw taps finger, waits for lbox [06:54] davecheney: success. i blew away *everything* and started again. bootstrapped manual provider on arm64. gotta run to soccer, but will add another machine later and deploy a charm [06:59] rogpeppe2: https://codereview.appspot.com/85590044/ [07:00] mgz: would appreciate a review from you too, in case there's some subtlety I've missed [07:01] rogpeppe2: on a completely unrelated note, I was thinking about the EnsureAvailability task some more this morning. Is there a reason why we shouldn't move a non-voting state server back to voting if it becomes accessible again? [07:02] axw: absolutely not. [07:02] axw: i think we should do that [07:02] goodo [07:02] that's what I thought too [07:02] axw: that's why we keep 'em around === vladk is now known as vladk|offline [07:21] anyone know anything about the replicaset failure in https://code.launchpad.net/~fwereade/juju-core/uniter-relation-states/+merge/215003 ? [07:21] are we seeing it a lot? [07:27] fwereade: wrt your previous comment [07:27] http://paste.ubuntu.com/7229693/ [07:28] i am not sure if this is a real race, or an instrumentatoin error [07:28] davecheney, I will take a look at that, thanks [07:29] davecheney, although at first glance... sync.WaitGroup *ought* to be used that way, oughtn't it? [07:30] fwereade: you cannot call Add() while another goroutine calls Wait() [07:30] only Done() [07:31] axw: reviewed [07:31] ta [07:32] rogpeppe2: heh, I kept writing IPv4 in my new code and then changed it to match the existing :/ [07:32] I will fix it another time- this is going to need to be backported, don't want to create unnecessary hardship :) [07:32] axw: yeah, it's definitely worth changing at some point [07:32] axw: fair enough [07:33] http://golang.org/pkg/sync/#WaitGroup.Add [07:34] fwereade: ah, i think i know what that might be to do with [07:35] fwereade: i think perhaps the replicaset tests were relying on the fact that Set put the session into monotonic mode. [07:35] rogpeppe2: likewise for NewAddress arg order if you don't mind - that will create a lot of noise [07:35] axw: ok [07:36] fwereade: (i mean Initiate, not Set) [07:37] davecheney, indeed, I see; but I'm having some difficulty mapping it onto the code [07:37] fwereade: yes [07:37] davecheney, the code looks odd fwiw, I don't see why we'd wait twice [07:37] i'm having trouble understandig if that is a real race [07:37] or a bug in the race detector [07:37] i'd use 1.3, but thre are heaps of bugs at tip [07:37] heh [07:37] as usual juju is leading the way in showig bugs in go tip [07:37] go us! [07:38] fwereade: hmm, it doesn't look as if that is actually the case [07:39] davecheney: it may well be a real race [07:39] rogpeppe2: \o/, i guess [07:39] davecheney: something similar came up on golang-nuts recently [07:39] rogpeppe2: indeed [07:40] and the semantics may change [07:40] but under 1.2 [07:40] the old semantics apply, you can't call Add(positive) after someone else has called Wait() [07:40] davecheney, hey, I think it is a race, but I'm still only at single-coffee levels of incisiveness [07:41] fwereade: based on that assesment, I shall raise a bug [07:41] i have found many other races [07:41] some in the mongo driver [07:41] davecheney, cheers, we can always close it if it isn't [07:42] davecheney: looks like a trivial fix for that one at least [07:42] davecheney: though i'm not sure it will actually fix the bug we're seeing [07:42] rogpeppe2: they normally are [07:42] true [07:43] davecheney: it may do though, thinking about it [07:43] davecheney: get that Add in! [07:46] https://code.launchpad.net/~dave-cheney/juju-core/128-environs-sync-tempdir-prefix/+merge/215087 [07:46] if anyone has two seconds [07:46] no [07:46] ignore, that one landed [07:46] https://code.launchpad.net/~dave-cheney/juju-core/127-fix-lp-1305397/+merge/215077 [07:46] this one is also trivial [07:46] rogpeppe2, that failure in my MP has an *awful* lot of "attempting Set got error: replSetReconfig command must be sent to the current replica set primary." lines before the no-reachable-servers failure [07:47] fwereade: yeah, i don't know why [07:57] rogpeppe2: worker/peergrouper/worker_test.go :318 [07:58] i see test failures when the servers is not the first entry in expectedAPIHostPorts(3) [07:58] davecheney: yeah, agreed, it's dubious. weren't we going to sort the servers slice? [07:59] rogpeppe2: i am trying [07:59] but how can I sort a set of []instance.APIPorts [08:00] davecheney: fairly easily [08:00] do tell [08:00] davecheney: it's not hard to define an ordering between two []instance.HostPort values [08:01] davecheney: (compare each element in order) [08:01] rogpeppe2: thta isn't what you told me to do [08:01] davecheney: you can probably just compare address value [08:01] you said that []instance.HostPort is already sorted [08:01] davecheney: it is [08:01] wright [08:02] rifht [08:02] davecheney: we're sorting a [][]instance.HostPort [08:02] so I need to sort a [][]instance.HostPort [08:02] yup [08:02] davecheney: so to do that, you need to compare two []instance.HostPorts [08:02] rogpeppe2: ok, got it [08:02] davecheney: cool [08:29] anyone know how to find out what code a given CI test is actually running? [08:29] (the CI test itself, that is) [08:31] rogpeppe2, I think it's lp:~juju-qa/juju-core/ci-cd-scripts2 [08:32] fwereade: i just found that, but it doesn't seem to have all the scripts in it (e.g. aws-upgrade) [08:33] fwereade: but i'm not sure what the correspondence is between that branch and the test names we see in jenkins [08:44] rogpeppe2: https://codereview.appspot.com/86400043/ [08:44] darn, i've just realised that we really need to support upgrading to HA [08:44] your thoughts sir [08:44] opps [08:44] two secs [08:46] davecheney: that doesn't look quite right [08:46] davecheney: i don't think that Less function is commutative [08:47] https://codereview.appspot.com/86400043/ [08:47] sorry, have another look [08:47] davecheney: i think you want: if len(a) != len(b) { return len(a) < len(b) } [08:49] ok [08:53] davecheney: in fact, i think the other test isn't right either [08:53] yeah, that was tricky [08:53] davecheney: i think it needs to compare the values for equality, and only if they're not equal should it compare the less-ness of them [08:54] yeah, that makes sense [08:54] davecheney: i've got that wrong before [08:54] davecheney: so i always look closely at Less functions now :-) [08:55] rogpeppe2: this one is tricky because in our case the ports are always the same [08:55] so a < b is always false [08:55] davecheney: i don't think it's that tricky [08:55] davecheney: one mo, i'll paste a suggestion [08:58] davecheney: reviewed (with suggested code in the review) https://codereview.appspot.com/86400043/ [08:58] rogpeppe2: ta === rogpeppe2 is now known as rogpeppe [09:00] rogpeppe: thanks, that is much more straight forward [09:00] davecheney: np [09:02] davecheney: just sent one other trivial comment [09:02] mgz: ping [09:03] afk [09:03] rogpeppe: et al, http://code.google.com/p/go/issues/detail?id=7749 [09:03] juju really breaks go 1.3 at the moment [09:11] axw: hey [09:14] mgz: hey, would you please take a look at https://codereview.appspot.com/85590044/ when you have a moment? [09:14] I've changed some address logic, and I'm a bit nervous about breaking everything :) [09:14] :) [09:18] axw: had planned the pivate range stuff, that looks fine, will have a proper poke through before the standup and see if any of the subtle logic bits have got lost [09:21] mgz: great, thanks [09:22] axw: only thing that jumps out as a risk is canonistack and similar setups [09:23] were we don't actually *have* a public address generally, [09:23] but currently juju can lie, and return the NetworkUnknown 10. address which will work with sshuttle [09:25] right, I need to change location now [09:25] mgz: I did live test with canonistack (and HP); the 10. address is now recorded as cloud-local, and usable as both public and internal [09:25] davecheney: :-( [09:25] davecheney: can you reproduce that reliably? === natefinch-afk is now known as natefinch [09:30] morning all [09:31] morning natefinch === vladk|offline is now known as vladk [09:43] natefinch: good morning. extra early for you, isn't it ? [09:45] jam1: yeah, been trying to get up early to get some more work done on HA. Also, yesterday I lost a good bit of the working day due to some emergency beekeeping tasks that came up. [09:55] good morning [09:57] just mentioning, standup in 3 min [10:19] natefinch: you have bees? [10:24] axw: just wondering about 11.0.0.0/8 [10:24] axw: it is technically public [10:24] axw: but actually private [10:24] axw: I wonder if anyone tries to use it [10:24] axw: it is the ipaddress of the us military group [10:25] and is a disconnected internet type network [10:25] I think the current code is good, and if someone is stupid enough to use this [10:25] then it can be on their head :-) [10:25] thumper: that would be an interesting problem to have :) [10:26] * axw thinks [10:26] it will be no worse than it is now, as cloud-local will be used for public/internal as well as unknown [10:26] oh wait [10:26] public.. hrm [10:27] well they would be inside the cloud anyway, so it would be fine I think [10:27] inside the cloud -> inside the network [10:28] axw: I do like the 'less unknowns' :-) [10:38] hey fwereade this is a wip https://codereview.appspot.com/86430043 for some reason https://codereview.appspot.com/86430043/patch/1/10003 is getting empty unit networks, care to take a quick look? (I did not upload work for the machine not changed checks to avoid clutter on what I am trying to debug) [10:43] thumper: yeah, three hives. bees are awesome :) [10:44] perrito666, actually do you want to pop on, I'm not quite following everything there [10:45] pop? [10:45] perrito666, sorry, I mean, quickly reenter the team meeting hangout [10:45] I certainly can [10:46] i've got two branches up for review. would much appreciate if someone could take a look: https://codereview.appspot.com/86200043/ https://codereview.appspot.com/85600044/ [10:52] axw: properly going over address branch now, I see rog has already looked through it [10:57] c7z: I assume that's mgz; yes he has, just thought I'd get your thoughts too, because you did the original work I think? [10:58] bbs [10:59] axw: yup, though there's a lot more complexity than the first version unfortuanately [11:04] rogpeppe: looking [11:05] oh noooo, daylight savings I missed the meeting [11:05] hahaha [11:06] I was like, why is there no one here??? [11:06] yep, daylight savings is annoying [11:06] sigh, reading notes [11:06] fwereade, sorry - bug 1305780 [11:06] <_mup_> Bug #1305780: juju-backup command fails against trusty bootstrap node [11:07] waigani: didn't miss much [11:07] hey, I'm in the notes! [11:08]