[00:16] wallyworld_: o/ [00:29] hi davecheney [00:32] wallyworld_: how can I help with that ppc bug ? [00:32] davecheney: fix the compiler :-) [00:32] wallyworld_: link [00:33] it broke between -10 and -12 [00:33] * wallyworld_ looks up link [00:33] https://bugs.launchpad.net/bugs/1365480 [00:33] Bug #1365480: new compiler breaks ppc64el unit tests in many ways

[00:34] the part that hooks in the monkey patch is broken [00:34] i guess because the call stack has changed somehow [00:34] let me know if i can provider more info, thanks for looking [00:34] np [00:34] sorry i couldn't help on fiday [00:34] i have many hats atm [00:36] np [00:36] i got the landing unblocked [00:36] cause i managed to help prove it was a compiler bug, not juju [00:37] excellent [00:42] oh crap [00:42] i forgot python on winton-09 was screwed [00:42] hmm === Guest4804 is now known as wallyworld [01:15] hmm, mercurial on stilson-07 is also scrwed [01:40] axw_: can I get a few quick reviews? [01:40] https://github.com/juju/juju/pull/692 [01:40] thumper: hey. sure [01:40] https://github.com/juju/names/pull/25 [01:40] https://github.com/juju/juju/pull/693 [01:40] axw_: all related to changes due to land soon where the initial user isn't called "admin" any more [01:40] okey dokey [01:41] I'm trying to break it up into small understandable bits [01:46] wallyworld_: i cannot reprodue the error [01:46] hmmm, that doesn't make sense [01:47] very little does these days [01:47] so the affected tests using the COntrolHook stuff all pass on -12? [01:47] wallyworld_: that isn't what I see in https://bugs.launchpad.net/bugs/1365480 [01:47] Bug #1365480: new compiler breaks ppc64el unit tests in many ways

[01:48] i see a repl test failyure and a linking failure [01:48] https://bugs.launchpad.net/juju-core/+bug/1365480/comments/3 [01:49] man, there are three indepdent bugs onthat issue [01:49] i can't confirm that failure either [01:50] from the attached log [01:50] local_test.go:244: [01:50] c.Assert(err, gc.ErrorMatches, "(.|\n)*cannot allocate a public IP as needed(.|\n)*") [01:50] ... value = nil [01:50] ... regex string = "" + [01:50] ... "(.|\n" + [01:50] ... ")*cannot allocate a public IP as needed(.|\n" + [01:50] ... ")*" [01:50] ... Error value is nil [01:50] that is the symptom of the compiler issue [01:50] forget the replicaset failure [01:50] that fails all the time [01:50] i cannot confirm that failure [01:51] i've been able to reproduce the linking fialure now [01:51] the local test failure? [01:51] /tmp/go-build604963151/github.com/juju/juju/api/provisioner/_test/github.com/juju/juju/api/libprovisioner.a(provisioner.o): In function `github_com_juju_juju_api_provisioner.State$equal': [01:51] /home/ubuntu/src/github.com/juju/juju/api/provisioner/provisioner.go:10: multiple definition of `github_com_juju_juju_api_provisioner.State$equal' [01:51] /home/ubuntu/pkg/gccgo_linux_ppc64/github.com/juju/juju/api/libprovisioner.a(provisioner.o):/home/ubuntu/src/github.com/juju/juju/api/provisioner/provisioner.go:10: first defined here [01:51] trivial for someone: https://github.com/juju/juju/pull/694 [01:52] davecheney: so with -12, you don't get FAIL: local_test.go:227: TestBootstrapFailsWhenPublicIPError.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite ??? [01:53] i think that failure is because of the one above [01:53] it's really hard to tell [01:53] so many things are broken on ppc [01:53] python has stopped working [01:53] no, it's due to my comment in the ug [01:53] bug [01:53] i think [01:53] maybe not [01:53] but all the failures point to the control hook not being run [01:54] ok [01:54] "all the failures" = the ones hwere a control hook is used but not run [01:54] and a slightly bigger one: https://github.com/juju/juju/pull/695 [01:55] wallyworld_: which package [01:55] provider/openstack ? [01:55] davecheney: provider/openstack [01:55] yep [01:56] there's maybe 4 or 5 tests which use the control hook stuff [01:56] and i think they all fail [01:56] but pass with -10 [01:56] ok, confirmed with -12 [01:56] thanks [01:59] great, ty [02:27] axw_: i called the handleFailure func because i thought we'd need to ensure that the interrupt notify APIs were called. do we not need to do that? [02:28] 1 sec [02:29] wallyworld_: no we don't, that's just there to add feedback if the user hits ctrl-c while it's tearing down [02:29] ah, rightio [02:29] i'll fix, thanks [02:29] cheers [02:30] axw_: speaking of that, have you seen this one? https://bugs.launchpad.net/juju-core/+bug/1365665 [02:30] Bug #1365665: ^C doesn't stop bootstrap

[02:31] wallyworld_: hmm, nope, hadn't [02:31] np, keep it in mind as a background task, might be a simple fix [02:54] thumper: https://github.com/juju/juju/pull/697 [02:58] thumper: wallyworld_ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750444 [02:58] mercurial blow up bug when http_proxy is set [02:59] http://bugs.python.org/issue7776 [02:59] broken in both python2 and python3 [02:59] yay [03:00] davecheney: but fixed in hg 3.0.1? [03:00] we don't ship that ersion in trusty [03:00] ii mercurial 2.8.2-1ubuntu1 ppc64el easy-to-use, scalable distributed version control system [03:00] ah, i'm running utopic [03:00] ii python 2.7.8-1~ubuntu1 ppc64el interactive high-level object-oriented language (default version) [03:00] only happens if you set http_proxy [03:00] which we do on ppc machines in our lab [03:00] ok [03:00] :-( [03:02] maybe i can just hulk smash in the utopic version to get thi8gns going [03:38] thumper: re my comment [03:39] we (juju) have several admin users [03:39] yes... [03:39] well, for now [03:39] right [03:39] and currently they also all have the same password [03:39] but in the future they wont [03:39] so please make it very obvious which "admin" we're talking about [03:39] sure... [03:39] ie, does the mongo admin user have to be called "admin" [03:40] id' prefer it if it wasnt [03:40] AFAICT, which is why I added `mongo.AdminUser` [03:40] thumper: my concern is we have two many sets of credentials called admin [03:40] we have the mongodb username/password [03:40] we have the state first user [03:41] we have the details we pass over the api. etc [03:41] can they all have differtn usernames to make it super easy when one is being transposed [03:41] otherwise it'll just be "admin": login fialed, wrong password [03:41] where it would be clearer if [03:41] mongodb: "environment-admin-user": login failed, wrong password [03:41] well, for now the initial user is still called "admin", but the branch I'm landing soon changes that for all our tests [03:42] I've had to tease it apart [03:42] ok, so if you tell me it'll get change in a followup [03:42] davecheney: RSN, the only "admin" user will be the mongo admin user [03:42] i withdraw my nitpicking [03:42] ok [03:42] but when that happens [03:42] can the mongo admin user not be called admin [03:42] "mongo-admin-user" [03:42] "master" [03:42] "root" [03:42] "scott" [03:42] anything [03:42] just not admin [03:42] I'm not sure we can... can we? [03:43] maybe... [03:43] ok, if mongo is hard coded to call the admin user admin [03:43] then we cannot use that word anywhere else [03:43] it'll be the only one [03:43] ie, on the api [03:43] ok then [03:43] I've changed our tests to have the env owner as: [03:43] "initialize-admin" [03:43] "factory-admin" [03:43] "dummy-admin" [03:43] etc [03:44] EXCELLENT [03:44] davecheney: it's coming [03:44] very soon now [03:44] ... [04:16] davecheney: https://github.com/howbazaar/juju/commit/26e038ab7d7c0ebb47bae2b598903b1aa5e8cd5c for https://github.com/juju/juju/pull/695 [04:21] thumper: does userTag.Name() and userTag.Id() do the same thing ? [04:22] there are too many subtly named methods on that type [04:24] davecheney: no [04:24] davecheney: Id() returns what it was constructed with... [04:25] Name() returns the first part of the string [04:25] so... [04:25] NewUserTag("dave"), id == "dave", name == "dave", provider == "local" (but internally it is "") [04:25] NewUserTag("dave@local"), id == "dave@local", name == "dave", provider == "local" [04:26] the Id needs to be able to be fed back into the type to make something equal [04:26] the tag round-tripping bit [04:26] do Id and Username do the same thing ? [04:26] no [04:26] NewUserTag("dave"), id == "dave", username == "dave@local" [04:27] NewUserTag("dave@local"), id == "dave@local", username == "dave@local" [04:27] wow, subtle [04:27] yes, had to be for the transition period [04:27] there are six bullets in that footgun [04:27] I've managed to keep it pretty sane in most places [04:27] which is why I don't generally use Id() for name tags [04:28] good plan [04:29] davecheney: I'm starting a branch that you'll like :) [04:33] wallyworld_: FYI, https://github.com/juju/juju/pull/700 [04:33] looking [04:34] wallyworld_: there's no upgrade steps to move tools from provider storage yet, but it won't adversely affect anything until we drop provider storage [04:34] searching/fetching tools will still go to provider storage for now, it's just behind the scenes [04:34] sure, sounds ok [04:35] going to work on getting toolstorage to remove old tarballs now [04:35] easy review pls: https://github.com/juju/juju/pull/699 (thumper?) [04:35] then will move onto charms I think [04:35] * thumper looks at menn0's branch [04:36] thumper: this is just something in the support infrastructure that came up today [04:36] thumper: the big PR is Coming Soon [04:36] * thumper nods [04:52] thumper: cheers. I've changed to uses backticks throughout those tests where it makes sense [04:52] kk [05:17] davecheney: are you happy enough with https://github.com/juju/juju/pull/695 ? [05:17] yup [05:18] go fot it [05:21] kk [05:27] * thumper got sad [05:28] seeing usernames passed around as tags, because tags are strings and usernames are strings, so that's all good, right? [05:28] but not string versions of those tags, oh no, just names [05:44] wallyworld_: i'm not going to have a ppc fix for you today [05:44] i've not gotten to the bottom of the issue [05:44] davecheney: no worries, CI has been tweaked to avoid having the failing ppc tests block landings [05:44] when we fixed this last time we made it automagiclaly figure out the stack call depth [05:44] and that bit is working now [05:44] well it is working [05:45] so something broke ebtween -10 and -12? [05:45] absolutelty [05:45] -12 is broken [05:45] but I can't tell yo what [05:45] or how to fix it yet [05:46] are there tests that should be writen to avid this again? [05:46] avoid [05:46] unknown [05:46] ok [05:46] thanks for looking at it [05:46] np [05:46] i'll keep looking [05:46] ok [05:51] * thumper closes irc and plans to work on his KiwiPyCon talk tonight [05:52] good luck === uru_ is now known as urulama [06:08] wallyworld_: data point: the registerserverpoint thinggy doesn't appear to be run [06:09] know that is what you said [06:09] but putting a panic in the function doesn't do jack ... [06:09] i wonder what is going on [06:09] davecheney: i added a panic too, and it didn't get called [06:10] i haven't looked at the code in a while, but from memory it looks at the call stack to figure out when to run the patched code [06:10] what the firetruck [06:10] ok [06:10] i'll keep looking [06:18] wallyworld_: look what I found [06:18] // This is very fragile. fullName will be something like: [06:18] // launchpad.net_goose_testservices_novaservice.removeServer.pN49_launchpad.net_goose_testservices_novaservice.Nova [06:18] // so if the number of dots in the full package path changes, [06:18] // this will need to too... [06:19] \o/ [06:19] what file is that in? [06:20] service_gccgo.go [06:20] hook/ [06:21] gotta go get kid, bbiab [06:25] wallyworld_: tweaked constants, things look better [06:25] wallyworld_: btw, we already have tests for all of this [06:25] they are in the goose repo [06:25] can goose be tested on ppc reguarly ? [06:40] davecheney: i can ask that to happen [06:40] davecheney: will the changes break with -10? [06:41] wallyworld_: not sure [06:41] i've just proposed a branch [06:41] which should try to figure it out automatically [06:41] ok [06:41] the bug was one of theose 'how the hell did that work in the first place' errors [06:42] lbox propose --- ah memories [06:43] FARK [06:43] my branch was screwde [06:43] wallyworld_: can you figure out how to merge this ? https://codereview.appspot.com/31890043 [06:44] davecheney: there's a prereq branch? [06:45] looks like the prereq has been merged already [06:47] not sure what you've done there. did you bzr branch off lp:goose to create a new branch? and then push that? and then propose? [06:53] my goose was very old [06:53] i just proposed it on whatever [06:54] i've forgotten almost all of the old ways [06:54] davecheney: i can propose if you like, but i have to go to the doctor in a bit so can do it after dinner [06:55] ok [06:55] let me see if i can figure it out [06:55] i'll also let curtis know so we can get a goose ppc test set up [06:56] ok, i'll check back a bit later, gotta leave in about 5 mins [07:02] dimitern: 1:1 ? === rogpeppe2 is now known as rogpeppe [07:47] morning [07:52] dimitern: I changed PR 626 after your review. mind another look? [07:54] TheMue, morning, sure - looking [07:58] dimitern: thanks [08:04] wallyworld_: i got that branch landed [08:04] https://code.launchpad.net/~go-bot/goose/trunk [08:04] need to land a followup to udpate depdencies.tsv [08:11] wallyworld_: https://github.com/juju/juju/pull/703 [08:15] morning TheMue [08:16] jam: heya [08:20] jam: your emails are interesting [08:20] jam: I noticed the "socket.Close()" issue [08:20] jam: I assumed that the test author knew what they were doing and the defered session.Close() was late bound [08:21] (i.e. the newly opened session would be closed to) [08:21] I don't think that can be the cause of the spurious connect (log lines connecting to ipv4 address) [08:32] davecheney: thanks for fix, I'll let curtis know [08:33] not sure how/when/if to backport to 1.20 [08:33] we could do, i can update the deps file [08:33] wallyworld_: i reckon if it aint broke, don't touch it [08:33] it's not clear what compiler 1.20 is being built with on trusty [08:34] not sure, i'll find out [08:35] heya voidspace [08:36] jam: morning [08:36] voidspace: so I'm not sure if session.Close is the issue, as it seems to be doing a syncServer inside the 'mgo' driver [08:36] and I wonder if the fact that we are calling inst.Destroy() isn't causing the cluster sync to get cleaned up. [08:37] jam: I'd like to see the mongo logs [08:37] jam: do you know where they go to? [08:37] jam: I'm seeing the tests die sometimes with unreachable servers - so it looks like mongo has fallen over altogether [08:37] voidspace: they are supposed to be on stdout, such that if you kill mongo somehow you'll get them in the test log [08:37] github.com/juju/testing/mgo.go is where that is set up [08:37] you can hack it to pass "--logfile" to mongo [08:37] jam: ah... we catch standard out when we start mongo [08:37] and then debug them from there [08:38] jam: and we parse the log to tell when mongo has started [08:38] voidspace: yes [08:38] jam: so you can't redirect it to a logfile [08:38] ah, we might not be able to redirect [08:38] someone did that in the past, you're right [08:38] maybe I can tee that handle [08:38] and send it to standard out as well [08:38] I really need those logs [08:39] jam: using CurrentStatus and checking member state *plus* uptime works *most of the time* for telling when the replicaset is healthy [08:39] "most of the time" lovely [08:39] sigh [08:39] jam: however it often dies with unreachable servers - and can't recover within five minutes [08:40] voidspace: have you tried looking in the mongo source ? [08:40] to see what it uses ? [08:40] jam: nope [08:40] jam: if I see the original errors in the mongo log then I can follow from there [08:40] voidspace: I'm not sure if it is worth it, but at this point its where I would go next if we can't get something reliable [08:41] right [08:41] voidspace: no reachable servers might be better to focus on, [08:41] jam: the thing is that the attemptLoop wasn't dying like this [08:41] and that sounds strange given we know mongo has started from its stdout right? [08:41] voidspace: I have the feeling it was failing in its own way that was a similar root cause but a retry caused it to get papered over [08:42] jam: right, but I have session.Refresh inside the healthy check - so I don't know (yet) what's different about the way attemptLoop was retrying and this is retrying [08:42] that's my next point of attack [08:42] but I'd like to see the logs to see why mongo thinks it's fallen over [08:44] jam: the main difference is that before we would retry the operation multiple times (with the same session) [08:44] e.g. Set() (which is usually the one that dies) [08:44] whereas now we just retry CurrentStatus multiple times [08:44] voidspace: I'm pretty sure we iterate over the output from mongo, you can probably line-by-line that to a logger.Debugf() call [08:44] jam: yep [08:45] have juju actions happened yet? [08:47] mwhudson: TheMue is working with the guys who are doing it. I don't think it is something you can put in a charm yet [08:48] ok [08:48] is there a spec? [08:48] i remember some discussion on the list, but that was a while ago [08:49] jam: so we do log the lines - at Trace level [08:49] mwhudson: I believe it was part of the "mega planning" sheet that was put together a while back, I'm not sure if there is a focused line, TheMue would probably know better when he notices these pings [08:49] heh [08:50] mwhudson: https://docs.google.com/a/canonical.com/document/d/14W1-QqB1pXZxyZW5QzFFoDwxxeQXBUzgj8IUkLId6cc/edit exists, last edit was 2 weeks ago, so it is probably up to date [08:50] * jam goes to make coffee, brb [08:51] mwhudson: so far I'm collecting the infos about all docs. one interesting source may be https://github.com/binary132/juju/blob/actions-doc/doc/actions.md [08:59] TheMue, you've got a review [09:00] dimitern: yep, already seen first feedback. thanks. will discuss one point with you after my 1:1, got a question here. [09:00] TheMue, sure [09:00] TheMue: thanks, looks appropriate for what i want to do, but for now will go for "juju ssh unit/0 random-shell-script.sh" :) [09:01] mwhudson: ;) [09:02] mwhudson: I think it is "juju scp unit/0 random-shell-script.sh . && juju ssh unit/0 random-shell-script.sh" right? [09:03] jam: the charm will provide random-shell-script.sh [09:04] mwhudson: SGTM [09:50] voidspace: I need to take my dog out, so I might be a little late for our 1:1, but hopefully not too late [09:50] jam: no problem [10:07] voidspace: I'm here now [10:09] jam: cool [10:09] jam: sapphire hangout? [10:09] voidspace: I linked it to the calendar event: https://plus.google.com/hangouts/_/canonical.com/john-michael [10:11] jam: found it :-) [10:45] jam, voidspace, TheMue, standup? [10:46] dimitern: just finishing up our 1:1 be there in a sec [10:46] jam, sure, np [11:12] jam: when it fails (the IPv6 test), I see this in the logs: [11:12] [LOG] 0:43.834 DEBUG juju.testing tls.Dial(127.0.0.1:37874) failed with EOF [11:12] jam: which mirrors what you were saying and is I believe "a clue" [11:12] using the non IPv6 address [11:12] tasdomas, ping [11:23] Ah, *actually* switched on the mongo log [11:23] That's better [11:25] voidspace: so... when the bus comes and says "we need someone at the door immediately because we can't just park here", my response is "don't show up 10 minutes early without letting me know"... [11:25] jam: it's unfortunate the number of people who think their mistake is your problem... [11:26] voidspace: so I have to work on homework, but then I'd be willing to pair debugging if mongo is being a pain [11:26] jam: yep, their should be a kind of automatism to signal the approach of your kid [11:26] jam: sure [11:27] Hah, and of course after switching on logging the test refuses to fail [11:27] :-) [11:27] I'm sure it will happen shortly [11:28] voidspace: turning on logging changes timings... [11:28] mongo is thinking "they're watching what I'm doing, I'd better behave now" [11:28] jam: I've only changed the log level in our code, so it shouldn't *actually* [11:28] vo [11:28] but yeah, definitely timing [11:28] writing to stdout is slow [11:29] 4 passes in a row, it was failing every time a minute ago :-) [11:33] voidspace: and now if you turn off stdout logging, they will pass every time [11:33] it was a phase of the moon thing [11:36] morning [12:01] lunch [12:01] jam: I'm taking a break - going to post office so may be a bit [12:19] dimitern, pong [12:40] voidspace: np. The homework for today seems completely inappropriate (almost all stuff that I know he didn't learn last year), so its going rather slowly. I'm not sure if it is "pre test" and this is what he'll be learning, or whether somehow we're completely off track... [12:42]