/srv/irclogs.ubuntu.com/2014/09/08/#juju-dev.txt

davecheneywallyworld_: o/00:16
wallyworld_hi davecheney00:29
davecheneywallyworld_: how can I help with that ppc bug ?00:32
wallyworld_davecheney: fix the compiler :-)00:32
davecheneywallyworld_: link00:32
wallyworld_it broke between -10 and -1200:33
* wallyworld_ looks up link00:33
wallyworld_https://bugs.launchpad.net/bugs/136548000:33
mupBug #1365480: new compiler breaks ppc64el unit tests in many ways <ci> <ppc64el> <regression> <juju-core:Triaged> <gcc-4.9 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1365480>00:33
wallyworld_the part that hooks in the monkey patch is broken00:34
wallyworld_i guess because the call stack has changed somehow00:34
wallyworld_let me know if i can provider more info, thanks for looking00:34
davecheneynp00:34
davecheneysorry i couldn't help on fiday00:34
davecheneyi have many hats atm00:34
wallyworld_np00:36
wallyworld_i got the landing unblocked00:36
wallyworld_cause i managed to help prove it was a compiler bug, not juju00:36
davecheneyexcellent00:37
davecheneyoh crap00:42
davecheneyi forgot python on winton-09 was screwed00:42
davecheneyhmm00:42
=== Guest4804 is now known as wallyworld
davecheneyhmm, mercurial on stilson-07 is also scrwed01:15
thumperaxw_: can I get a few quick reviews?01:40
thumperhttps://github.com/juju/juju/pull/69201:40
axw_thumper: hey. sure01:40
thumperhttps://github.com/juju/names/pull/2501:40
thumperhttps://github.com/juju/juju/pull/69301:40
thumperaxw_: all related to changes due to land soon where the initial user isn't called "admin" any more01:40
axw_okey dokey01:40
thumperI'm trying to break it up into small understandable bits01:41
davecheneywallyworld_: i cannot reprodue the error01:46
wallyworld_hmmm, that doesn't make sense01:46
davecheneyvery little does these days01:47
wallyworld_so the affected tests using the COntrolHook stuff all pass on -12?01:47
davecheneywallyworld_: that isn't what I see in https://bugs.launchpad.net/bugs/136548001:47
mupBug #1365480: new compiler breaks ppc64el unit tests in many ways <ci> <ppc64el> <regression> <juju-core:Triaged> <gcc-4.9 (Ubuntu):Confirmed> <https://launchpad.net/bugs/1365480>01:47
davecheneyi see a repl test failyure and a linking failure01:48
wallyworld_https://bugs.launchpad.net/juju-core/+bug/1365480/comments/301:48
davecheneyman, there are three indepdent bugs onthat issue01:49
davecheneyi can't confirm that failure either01:49
wallyworld_from the attached log01:50
wallyworld_local_test.go:244:01:50
wallyworld_    c.Assert(err, gc.ErrorMatches, "(.|\n)*cannot allocate a public IP as needed(.|\n)*")01:50
wallyworld_... value = nil01:50
wallyworld_... regex string = "" +01:50
wallyworld_...     "(.|\n" +01:50
wallyworld_...     ")*cannot allocate a public IP as needed(.|\n" +01:50
wallyworld_...     ")*"01:50
wallyworld_... Error value is nil01:50
wallyworld_that is the symptom of the compiler issue01:50
wallyworld_forget the replicaset failure01:50
wallyworld_that fails all the time01:50
davecheneyi cannot confirm that failure01:50
davecheneyi've been able to reproduce the linking fialure now01:51
wallyworld_the local test failure?01:51
davecheney/tmp/go-build604963151/github.com/juju/juju/api/provisioner/_test/github.com/juju/juju/api/libprovisioner.a(provisioner.o): In function `github_com_juju_juju_api_provisioner.State$equal':01:51
davecheney/home/ubuntu/src/github.com/juju/juju/api/provisioner/provisioner.go:10: multiple definition of `github_com_juju_juju_api_provisioner.State$equal'01:51
davecheney/home/ubuntu/pkg/gccgo_linux_ppc64/github.com/juju/juju/api/libprovisioner.a(provisioner.o):/home/ubuntu/src/github.com/juju/juju/api/provisioner/provisioner.go:10: first defined here01:51
thumpertrivial for someone: https://github.com/juju/juju/pull/69401:51
wallyworld_davecheney: so with -12, you don't get FAIL: local_test.go:227: TestBootstrapFailsWhenPublicIPError.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite ???01:52
davecheneyi think that failure is because of the one above01:53
davecheneyit's really hard to tell01:53
davecheneyso many things are broken on ppc01:53
davecheneypython has stopped working01:53
wallyworld_no, it's due to my comment in the ug01:53
wallyworld_bug01:53
wallyworld_i think01:53
wallyworld_maybe not01:53
wallyworld_but all the failures point to the control hook not being run01:53
davecheneyok01:54
wallyworld_"all the failures" = the ones hwere a control hook is used but not run01:54
thumperand a slightly bigger one: https://github.com/juju/juju/pull/69501:54
davecheneywallyworld_: which package01:55
davecheneyprovider/openstack ?01:55
wallyworld_davecheney: provider/openstack01:55
wallyworld_yep01:55
wallyworld_there's maybe 4 or 5 tests which use the control hook stuff01:56
wallyworld_and i think they all fail01:56
wallyworld_but pass with -1001:56
davecheneyok, confirmed with -1201:56
davecheneythanks01:56
wallyworld_great, ty01:59
wallyworld_axw_: i called the handleFailure func because i thought we'd need to ensure that the interrupt notify APIs were called. do we not need to do that?02:27
axw_1 sec02:28
axw_wallyworld_: no we don't, that's just there to add feedback if the user hits ctrl-c while it's tearing down02:29
wallyworld_ah, rightio02:29
wallyworld_i'll fix, thanks02:29
axw_cheers02:29
wallyworld_axw_: speaking of that, have you seen this one? https://bugs.launchpad.net/juju-core/+bug/136566502:30
mupBug #1365665: ^C doesn't stop bootstrap <bootstrap> <joyent-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1365665>02:30
axw_wallyworld_: hmm, nope, hadn't02:31
wallyworld_np, keep it in mind as a background task, might be a simple fix02:31
waiganithumper: https://github.com/juju/juju/pull/69702:54
davecheneythumper: wallyworld_ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=75044402:58
davecheneymercurial blow up bug when http_proxy is set02:58
davecheneyhttp://bugs.python.org/issue777602:59
davecheneybroken in both python2 and python302:59
davecheneyyay02:59
wallyworld_davecheney: but fixed in hg 3.0.1?03:00
davecheneywe don't ship that ersion in trusty03:00
davecheneyii  mercurial                          2.8.2-1ubuntu1         ppc64el                easy-to-use, scalable distributed version control system03:00
wallyworld_ah, i'm running utopic03:00
davecheneyii  python                             2.7.8-1~ubuntu1        ppc64el                interactive high-level object-oriented language (default version)03:00
davecheneyonly happens if you set http_proxy03:00
davecheneywhich we do on ppc machines in our lab03:00
wallyworld_ok03:00
wallyworld_:-(03:00
davecheneymaybe i can just hulk smash in the utopic version to get thi8gns going03:02
davecheneythumper: re my comment03:38
davecheneywe (juju) have several admin users03:39
thumperyes...03:39
thumperwell, for now03:39
davecheneyright03:39
davecheneyand currently they also all have the same password03:39
davecheneybut in the future they wont03:39
davecheneyso please make it very obvious which "admin" we're talking about03:39
thumpersure...03:39
davecheneyie, does the mongo admin user have to be called "admin"03:39
davecheneyid' prefer it if it wasnt03:40
thumperAFAICT, which is why I added `mongo.AdminUser`03:40
davecheneythumper: my concern is we have two many sets of credentials called admin03:40
davecheneywe have the mongodb username/password03:40
davecheneywe have the state first user03:40
davecheneywe have the details we pass over the api. etc03:41
davecheneycan they all have differtn usernames to make it super easy when one is being transposed03:41
davecheneyotherwise it'll just be "admin": login fialed, wrong password03:41
davecheneywhere it would be clearer if03:41
davecheneymongodb: "environment-admin-user": login failed, wrong password03:41
thumperwell, for now the initial user is still called "admin", but the branch I'm landing soon changes that for all our tests03:41
thumperI've had to tease it apart03:42
davecheneyok, so if you tell me it'll get change in a followup03:42
thumperdavecheney: RSN, the only "admin" user will be the mongo admin user03:42
davecheneyi withdraw my nitpicking03:42
davecheneyok03:42
davecheneybut when that happens03:42
davecheneycan the mongo admin user not be called admin03:42
davecheney"mongo-admin-user"03:42
davecheney"master"03:42
davecheney"root"03:42
davecheney"scott"03:42
davecheneyanything03:42
davecheneyjust not admin03:42
thumperI'm not sure we can... can we?03:42
thumpermaybe...03:43
davecheneyok, if mongo is hard coded to call the admin user admin03:43
davecheneythen we cannot use that word anywhere else03:43
thumperit'll be the only one03:43
davecheneyie, on the api03:43
davecheneyok then03:43
thumperI've changed our tests to have the env owner as:03:43
thumper"initialize-admin"03:43
thumper"factory-admin"03:43
thumper"dummy-admin"03:43
thumperetc03:43
davecheneyEXCELLENT03:44
thumperdavecheney: it's coming03:44
thumpervery soon now03:44
thumper...03:44
thumperdavecheney: https://github.com/howbazaar/juju/commit/26e038ab7d7c0ebb47bae2b598903b1aa5e8cd5c for https://github.com/juju/juju/pull/69504:16
davecheneythumper: does userTag.Name() and userTag.Id() do the same thing ?04:21
davecheneythere are too many subtly named methods on that type04:22
thumperdavecheney: no04:24
thumperdavecheney: Id() returns what it was constructed with...04:24
thumperName() returns the first part of the string04:25
thumperso...04:25
thumperNewUserTag("dave"), id == "dave", name == "dave", provider == "local" (but internally it is "")04:25
thumperNewUserTag("dave@local"), id == "dave@local", name == "dave", provider == "local"04:25
thumperthe Id needs to be able to be fed back into the type to make something equal04:26
thumperthe tag round-tripping bit04:26
davecheneydo Id and Username do the same thing ?04:26
thumperno04:26
thumperNewUserTag("dave"), id == "dave", username == "dave@local"04:26
thumperNewUserTag("dave@local"), id == "dave@local", username == "dave@local"04:27
davecheneywow, subtle04:27
thumperyes, had to be for the transition period04:27
davecheneythere are six bullets in that footgun04:27
thumperI've managed to keep it pretty sane in most places04:27
thumperwhich is why I don't generally use Id() for name tags04:27
davecheneygood plan04:28
thumperdavecheney: I'm starting a branch that you'll like :)04:29
axw_wallyworld_: FYI, https://github.com/juju/juju/pull/70004:33
wallyworld_looking04:33
axw_wallyworld_: there's no upgrade steps to move tools from provider storage yet, but it won't adversely affect anything until we drop provider storage04:34
axw_searching/fetching tools will still go to provider storage for now, it's just behind the scenes04:34
wallyworld_sure, sounds ok04:34
axw_going to work on getting toolstorage to remove old tarballs now04:35
menn0easy review pls: https://github.com/juju/juju/pull/699 (thumper?)04:35
axw_then will move onto charms I think04:35
* thumper looks at menn0's branch04:35
menn0thumper: this is just something in the support infrastructure that came up today04:36
menn0thumper: the big PR is Coming Soon04:36
* thumper nods04:36
menn0thumper: cheers. I've changed to uses backticks throughout those tests where it makes sense04:52
thumperkk04:52
thumperdavecheney: are you happy enough with https://github.com/juju/juju/pull/695 ?05:17
davecheneyyup05:17
davecheneygo fot it05:18
thumperkk05:21
* thumper got sad05:27
thumperseeing usernames passed around as tags, because tags are strings and usernames are strings, so that's all good, right?05:28
thumperbut not string versions of those tags, oh no, just names05:28
davecheneywallyworld_: i'm not going to have a ppc fix for you today05:44
davecheneyi've not gotten to the bottom of the issue05:44
wallyworld_davecheney: no worries, CI has been tweaked to avoid having the failing ppc tests block landings05:44
davecheneywhen we fixed this last time we made it automagiclaly figure out the stack call depth05:44
davecheneyand that bit is working now05:44
davecheneywell it is working05:44
wallyworld_so something broke ebtween -10 and -12?05:45
davecheneyabsolutelty05:45
davecheney-12 is broken05:45
davecheneybut I can't tell yo what05:45
davecheneyor how to fix it yet05:45
wallyworld_are there tests that should be writen to avid this again?05:46
wallyworld_avoid05:46
davecheneyunknown05:46
wallyworld_ok05:46
wallyworld_thanks for looking at it05:46
davecheneynp05:46
davecheneyi'll keep looking05:46
wallyworld_ok05:46
* thumper closes irc and plans to work on his KiwiPyCon talk tonight05:51
davecheneygood luck05:52
=== uru_ is now known as urulama
davecheneywallyworld_: data point: the registerserverpoint thinggy doesn't appear to be run06:08
davecheneyknow that is what you said06:09
davecheneybut putting a panic in the function doesn't do jack ...06:09
davecheneyi wonder what is going on06:09
wallyworld_davecheney: i added a panic too, and it didn't get called06:09
wallyworld_i haven't looked at the code in a while, but from memory it looks at the call stack to figure out when to run the patched code06:10
davecheneywhat the firetruck06:10
davecheneyok06:10
davecheneyi'll keep looking06:10
davecheneywallyworld_: look what I found06:18
davecheney        // This is very fragile.  fullName will be something like:06:18
davecheney        // launchpad.net_goose_testservices_novaservice.removeServer.pN49_launchpad.net_goose_testservices_novaservice.Nova06:18
davecheney        // so if the number of dots in the full package path changes,06:18
davecheney        // this will need to too...06:18
wallyworld_\o/06:19
wallyworld_what file is that in?06:19
davecheneyservice_gccgo.go06:20
davecheneyhook/06:20
wallyworld_gotta go get kid, bbiab06:21
davecheneywallyworld_: tweaked constants, things look better06:25
davecheneywallyworld_: btw, we already have tests for all of this06:25
davecheneythey are in the goose repo06:25
davecheneycan goose be tested on ppc reguarly ?06:25
wallyworld_davecheney: i can ask that to happen06:40
wallyworld_davecheney: will the changes break with -10?06:40
davecheneywallyworld_: not sure06:41
davecheneyi've just proposed a branch06:41
davecheneywhich should try to figure it out automatically06:41
wallyworld_ok06:41
davecheneythe bug was one of theose 'how the hell did that work in the first place' errors06:41
davecheneylbox propose --- ah memories06:42
davecheneyFARK06:43
davecheneymy branch was screwde06:43
davecheneywallyworld_: can you figure out how to merge this ? https://codereview.appspot.com/3189004306:43
wallyworld_davecheney: there's a prereq branch?06:44
wallyworld_looks like the prereq has been merged already06:45
wallyworld_not sure what you've done there. did you bzr branch off lp:goose to create a new branch? and then push that? and then propose?06:47
davecheneymy goose was very old06:53
davecheneyi just proposed it on whatever06:53
davecheneyi've forgotten almost all of the old ways06:54
wallyworld_davecheney: i can propose if you like, but i have to go to the doctor in a bit so can do it after dinner06:54
davecheneyok06:55
davecheneylet me see if i can figure it out06:55
wallyworld_i'll also let curtis know so we can get a goose ppc test set up06:55
wallyworld_ok, i'll check back a bit later, gotta leave in about 5 mins06:56
jamdimitern: 1:1 ?07:02
=== rogpeppe2 is now known as rogpeppe
TheMuemorning07:47
TheMuedimitern: I changed PR 626 after your review. mind another look?07:52
dimiternTheMue, morning, sure - looking07:54
TheMuedimitern: thanks07:58
davecheneywallyworld_: i got that branch landed08:04
davecheneyhttps://code.launchpad.net/~go-bot/goose/trunk08:04
davecheneyneed to land a followup to udpate depdencies.tsv08:04
davecheneywallyworld_: https://github.com/juju/juju/pull/70308:11
jammorning TheMue08:15
TheMuejam: heya08:16
voidspacejam: your emails are interesting08:20
voidspacejam: I noticed the "socket.Close()" issue08:20
voidspacejam: I assumed that the test author knew what they were doing and the defered session.Close() was late bound08:20
voidspace(i.e. the newly opened session would be closed to)08:21
voidspaceI don't think that can be the cause of the spurious connect (log lines connecting to ipv4 address)08:21
wallyworld_davecheney: thanks for fix, I'll let curtis know08:32
davecheneynot sure how/when/if to backport to 1.2008:33
wallyworld_we could do, i can update the deps file08:33
davecheneywallyworld_: i reckon if it aint broke, don't touch it08:33
davecheneyit's not clear what compiler 1.20 is being built with on trusty08:33
wallyworld_not sure, i'll find out08:34
jamheya voidspace08:35
voidspacejam: morning08:36
jamvoidspace: so I'm not sure if session.Close is the issue, as it seems to be doing a syncServer inside the 'mgo' driver08:36
jamand I wonder if the fact that we are calling inst.Destroy() isn't causing the cluster sync to get cleaned up.08:36
voidspacejam: I'd like to see the mongo logs08:37
voidspacejam: do you know where they go to?08:37
voidspacejam: I'm seeing the tests die sometimes with unreachable servers - so it looks like mongo has fallen over altogether08:37
jamvoidspace: they are supposed to be on stdout, such that if you kill mongo somehow you'll get them in the test log08:37
jamgithub.com/juju/testing/mgo.go is where that is set up08:37
jamyou can hack it to pass "--logfile" to mongo08:37
voidspacejam: ah... we catch standard out when we start mongo08:37
jamand then debug them from there08:37
voidspacejam: and we parse the log to tell when mongo has started08:38
jamvoidspace: yes08:38
voidspacejam: so you can't redirect it to a logfile08:38
jamah, we might not be able to redirect08:38
jamsomeone did that in the past, you're right08:38
voidspacemaybe I can tee that handle08:38
voidspaceand send it to standard out as well08:38
voidspaceI really need those logs08:38
voidspacejam: using CurrentStatus and checking member state *plus* uptime works *most of the time* for telling when the replicaset is healthy08:39
jam"most of the time" lovely08:39
jamsigh08:39
voidspacejam: however it often dies with unreachable servers - and can't recover within five minutes08:39
jamvoidspace: have you tried looking in the mongo source ?08:40
jamto see what it uses ?08:40
voidspacejam: nope08:40
voidspacejam: if I see the original errors in the mongo log then I can follow from there08:40
jamvoidspace: I'm not sure if it is worth it, but at this point its where I would go next if we can't get something reliable08:40
voidspaceright08:41
jamvoidspace: no reachable servers might be better to focus on,08:41
voidspacejam: the thing is that the attemptLoop wasn't dying like this08:41
jamand that sounds strange given we know mongo has started from its stdout right?08:41
jamvoidspace: I have the feeling it was failing in its own way that was a similar root cause but a retry caused it  to get papered over08:41
voidspacejam: right, but I have session.Refresh inside the healthy check - so I don't know (yet) what's different about the way attemptLoop was retrying and this is retrying08:42
voidspacethat's my next point of attack08:42
voidspacebut I'd like to see the logs to see why mongo thinks it's fallen over08:42
voidspacejam: the main difference is that before we would retry the operation multiple times (with the same session)08:44
voidspacee.g. Set() (which is usually the one that dies)08:44
voidspacewhereas now we just retry CurrentStatus multiple times08:44
jamvoidspace: I'm pretty sure we iterate over the output from mongo, you can probably line-by-line that to a logger.Debugf() call08:44
voidspacejam: yep08:44
mwhudsonhave juju actions happened yet?08:45
jammwhudson: TheMue is working with the guys who are doing it. I don't think it is something you can put in a charm yet08:47
mwhudsonok08:48
mwhudsonis there a spec?08:48
mwhudsoni remember some discussion on the list, but that was a while ago08:48
voidspacejam: so we do log the lines - at Trace level08:49
jammwhudson: I believe it was part of the "mega planning" sheet that was put together a while back, I'm not sure if there is a focused line, TheMue would probably know better when he notices these pings08:49
mwhudsonheh08:49
jammwhudson: https://docs.google.com/a/canonical.com/document/d/14W1-QqB1pXZxyZW5QzFFoDwxxeQXBUzgj8IUkLId6cc/edit exists, last edit was 2 weeks ago, so it is probably up to date08:50
* jam goes to make coffee, brb08:50
TheMuemwhudson: so far I'm collecting the infos about all docs. one interesting source may be https://github.com/binary132/juju/blob/actions-doc/doc/actions.md08:51
dimiternTheMue, you've got a review08:59
TheMuedimitern: yep, already seen first feedback. thanks. will discuss one point with you after my 1:1, got a question here.09:00
dimiternTheMue, sure09:00
mwhudsonTheMue: thanks, looks appropriate for what i want to do, but for now will go for "juju ssh unit/0 random-shell-script.sh" :)09:00
TheMuemwhudson: ;)09:01
jammwhudson: I think it is "juju scp unit/0 random-shell-script.sh . && juju ssh unit/0 random-shell-script.sh" right?09:02
mwhudsonjam: the charm will provide random-shell-script.sh09:03
jammwhudson: SGTM09:04
jamvoidspace: I need to take my dog out, so I might be a little late for our 1:1, but hopefully not too late09:50
voidspacejam: no problem09:50
jamvoidspace: I'm here now10:07
voidspacejam: cool10:09
voidspacejam: sapphire hangout?10:09
jamvoidspace: I linked it to the calendar event: https://plus.google.com/hangouts/_/canonical.com/john-michael10:09
voidspacejam: found it :-)10:11
dimiternjam, voidspace, TheMue, standup?10:45
jamdimitern: just finishing up our 1:1 be there in a sec10:46
dimiternjam, sure, np10:46
voidspacejam:  when it fails (the IPv6 test), I see this in the logs:11:12
voidspace[LOG] 0:43.834 DEBUG juju.testing tls.Dial(127.0.0.1:37874) failed with EOF11:12
voidspacejam: which mirrors what you were saying and is I believe "a clue"11:12
voidspaceusing the non IPv6 address11:12
dimiterntasdomas, ping11:12
voidspaceAh, *actually* switched on the mongo log11:23
voidspaceThat's better11:23
jamvoidspace: so... when the bus comes and says "we need someone at the door immediately because we can't just park here", my response is "don't show up 10 minutes early without letting me know"...11:25
voidspacejam: it's unfortunate the number of people who think their mistake is your problem...11:25
jamvoidspace: so I have to work on homework, but then I'd be willing to pair debugging if mongo is being a pain11:26
TheMuejam: yep, their should be a kind of automatism to signal the approach of your kid11:26
voidspacejam: sure11:26
voidspaceHah, and of course after switching on logging the test refuses to fail11:27
voidspace:-)11:27
voidspaceI'm sure it will happen shortly11:27
jamvoidspace: turning on logging changes timings...11:28
voidspacemongo is thinking "they're watching what I'm doing, I'd better behave now"11:28
voidspacejam: I've only changed the log level in our code, so it shouldn't *actually*11:28
jamvo11:28
voidspacebut yeah, definitely timing11:28
voidspacewriting to stdout is slow11:28
voidspace4 passes in a row, it was failing every time a minute ago :-)11:29
jamvoidspace: and now if you turn off stdout logging, they will pass every time11:33
jamit was a phase of the moon thing11:33
perrito666morning11:36
voidspacelunch12:01
voidspacejam: I'm taking a break - going to post office so may be a bit12:01
tasdomasdimitern, pong12:19
jamvoidspace: np. The homework for today seems completely inappropriate (almost all stuff that I know he didn't learn last year), so its going rather slowly. I'm not sure if it is "pre test" and this is what he'll be learning, or whether somehow we're completely off track...12:40
perrito666jam: teacher having a papers mixup?12:42
jamperrito666: it certainly looks like stuff he might learn this year, so I'm not really sure.12:42
jamhe's going into second grade, and they're having him do stuff like "friends" vs "friend's"12:43
jamhe's a bit more at the speed of being able to read the word "friend"12:43
perrito666jam: well just in case you should not help him (In case its a level finding test)12:43
jamperrito666: the rules from earlier in the week that they sent home are that we should help if they ask for help12:44
jamand there is only 1 line for each type of thing12:44
jamanyway, I'm certainly going to be asking about it.12:44
perrito666jam: I dont think is customary for teachers to handle their phone numbers there, right?12:45
jamxI have her email address.12:45
perrito666teachers and emails are not as fast as one would expect :p12:46
* perrito666 will be a very annoying parent someday12:46
jamperrito666: :012:46
jam:)12:46
wallyworld_hazmat: question about bug 1363971 if you are online12:57
mupBug #1363971: add-machine containers should default to latest lts <14.10> <juju-core:Triaged> <juju-core 1.20:In Progress by wallyworld> <https://launchpad.net/bugs/1363971>12:57
hazmatwallyworld_, shoot13:00
hazmatwallyworld_, the issue is the container default series logic is shared i think between local and other providers.13:01
hazmatwallyworld_, the issue is the fallback is basically  a hardcoded string of precise13:01
hazmatwallyworld_, i'm in the uk this week fwiw13:01
hazmatwallyworld_, in terms of what the proper behavior would be.. default to latest lts or in non local provider case host series both sound reasonable.13:02
voidspacejam: :-/13:10
voidspacejam: good news for me though - text and email to say internet is on13:11
voidspacejam: about to find out if this is true...13:11
voidspaceno, it seems like a lie13:46
voidspaceno internet on the dsl line13:46
voidspacethe router can detect a dsl line, but no internet...13:47
voidspaceHow do I see Trace logs in tests? -gocheck.vv doesn't do it and there's no -gocheck.vvv13:53
voidspacenatefinch: do you know the trick for turning on verbose mgo logging?14:19
voidspacenatefinch: dimiterm mentioned it last week and foolishly I didn't write it down14:19
natefinchvoidspace:  nope14:19
voidspaceheh14:19
voidspacenatefinch: I'm pretty sure we don't write mongo logs to disk by the way14:20
voidspacenatefinch: we deliberately send to stdout and capture them so we parse them to know when mongo has started14:20
natefinchvoidspace: possible during the tests I guess14:20
voidspacenatefinch: we log them at Trace level14:20
perrito666voidspace: hey, you are from uk :)14:20
voidspacenatefinch: I couldn't see how to show them, but I've just changed them to Debug for the moment14:20
voidspaceperrito666: I am...14:20
perrito666voidspace: priv14:21
voidspacenatefinch: annoyingly I see lots of mongo heartbeat messages whilst still getting a "no reachable servers" error14:21
voidspacenatefinch: and increasing that timeout (it's in the Dial function we use as far as I can see) doesn't fix it...14:21
natefinchvoidspace: heh yeah14:21
voidspaceso as far as mongo is concerned everything is ok as far as I can tell14:21
voidspacethat's why I want those mgo logs14:21
voidspacethe timeout is in cluster AcquireSocket14:22
sinzuialexisb, I take the silence regarding inserting a new development milestone means I can release 1.21alpha114:30
natefinchjam: I think you mentioned something about someone working on making juju resolved --retry into just juju retry?  Is that work that is currently underway?14:34
voidspacedimitern: ping15:23
dimiternvoidspace, hey15:23
voidspacedimitern: last week you suggested a way to get more logging out of mgo15:23
voidspacedimitern: which I didn't write down...15:23
dimiternvoidspace, right :) so take a look at mgo.SetLogger() and mgo.SetDebug(true)15:24
dimiternvoidspace, SetLogger can take *gc.C I think, as it needs something with .Output(..) method, so you can have both mgo and other logs in the same place15:24
voidspacedimitern: ok, I'll try tinkering with those15:25
voidspacedimitern: thanks15:25
natefinchin case no one else noticed.... there's a built-in side by side diff on github now15:25
dimiternvoidspace, you'll need to call these early, like in SetUpSuite, although depending on what you need just setting them up in SetUpTest or the TestXX case itself could be enough15:26
TheMuedimitern, jam: *phew* I think I found a nice way testing an API for all existing versions taking into account the individual changes15:26
TheMuedimiterm, jam: you'll see it with the next push15:26
dimiternnatefinch, cool!15:26
dimiternTheMue, nice, I'll have a look15:26
TheMuedimitern: I'll ping you then15:27
TheMuenatefinch: hey, yes, simply press "Split". that's cool, thanks for the hint15:27
voidspacedimitern: wow15:28
voidspacedimitern: that's screenful after screenful of logs...15:28
voidspacelike, constant15:28
dimiternvoidspace, it is a lot, and it takes some time to understand how to read it :)15:32
voidspacedimitern: this is interesting15:34
voidspacedimitern: for mongo we have to *not* use square brackets for ipv6 addresses15:35
voidspacedimitern: i.e. :::port15:35
voidspacedimitern: but mgo uses net.DialTimeout15:35
voidspacedimitern: and *it* specifies15:35
voidspaceIf host is a literal IPv6 address or host name, it must be enclosed in square brackets as in "[::1]:80"15:35
voidspaceso, technically the address form we're using for ipv6 in the tests are invalid for net.DialTimeout15:36
voidspacenow that I have logs, the specific error in mgo is15:36
voidspace[LOG] 2:22.633 SYNC Failed to start sync of ::1:41050: failed to resolve server address: ::1:4105015:36
voidspacewhilst trying to sync servers15:36
voidspace[LOG] 7:21.167 SYNC Synchronization was partial (cannot talk to primary).15:36
dimiternvoidspace, hmm.. interesting15:37
voidspacebut it's not deterministic15:37
voidspacebut it maybe that ipv6 addresses of that form are incompatible with *part* of mgo15:37
dimiternvoidspace, IPv6 localhost address + port could be specified as [::1]:port, but it depends on the way the thing is written I guess15:38
voidspacedimitern: no, mongo fails to parse that15:38
voidspacedimitern: we specifically can't use that format15:39
dimiternvoidspace, fails to parse "::1:12345" or "[::1]::12345" ?15:40
voidspacedimitern: the second15:40
voidspacethe correct one :-)15:40
dimiternvoidspace, hmm, but it still works sometimes?15:42
voidspacedimitern: yeah, which is *possibly* due to the test only failing if we cause primary renegotiation15:42
voidspacedimitern: so it has to redial15:42
voidspacedimitern: and hits this function which can't handle our ipv6 address format15:42
TheMuedimitern: it's pushed. the interesting one is apiserver/machine/machiner_test.go. the suite is now run twice, once for v0, once for v1. when the version is lower than 1 the special tests for v1 (today only one) are skipped. this concept should also work when we get more versions.15:43
voidspacedimitern: I'm going to try locally and see if it does work with addresses in that format15:43
dimiternvoidspace, so the ::1:port is only needed when passing args to mongod, or?15:43
voidspacedimitern: yep15:43
voidspacedimitern: but the address we pass to mongo is the address used by the dial function15:43
voidspacedimitern: I'd have to hack round it if that is the problem15:43
voidspacedimitern: I'm going to see if net.DialTimeout does work with these addresses despite claiming not to15:44
voidspaceif it does then the resolve error must be due to something else15:44
dimiternTheMue, it's run twice due to you calling gc.Suite() twice with for the same type with different constructor args15:46
voidspacedimitern: not looking good15:48
voidspacedimitern: dial udp: too many colons in address ::1:8015:48
voidspacedimitern: but mgo discards the actual error and we just see "failed to resolve"15:48
voidspacedimitern: so for ipv6 I think that's the problem15:48
voidspacejust confirming added logging of the actual error and re-running15:49
dimiternvoidspace, weird..15:49
dimiternvoidspace, what if you try with "[::1]:80"15:49
voidspacedimitern: in the test?15:49
voidspacedimitern: mongo will fail to start15:49
voidspacethat's what will happen15:49
voidspaceI'll need to hack round it (if I even can) to mix the forms15:50
voidspaceif I can confirm that this is the problem I'll do that15:50
voidspaceI just need the test to fail...15:50
voidspaceit fails about 50/5015:50
dimiternvoidspace, right, sounds good15:51
voidspacedimitern: *damn* - got it to fail, but I logged the address not the error!!!15:53
voidspacenow to try again...15:53
voidspacedimitern: confirmed15:58
voidspacedimitern: when the IPv6 mongo replicaset fails, the root cause is this error inside mgo15:58
voidspacedimitern:15:58
voidspace[LOG] 1:26.513 SYNC failed to connect: dial udp: too many colons in address ::1:4211315:58
voidspaceduring a syncServers15:59
voidspacedimitern: I'll look at a workaround15:59
voidspacejam: I've found the cause of the ipv6 failures - but it isn't a general issue for the other tests15:59
voidspacejam: we use the form ::1:port for starting mongo on ipv6 because mongo doesn't understand [::1]:port16:00
voidspacejam: but if the Set operation causes a primary renegotiation mgo has an internal call to net.DialTimeout16:00
voidspacejam: and net.DialTimeout doesn't work with ::1:port format16:00
voidspacejam: mgo logging doesn't log the original error so it was only visible as "no reachable servers"16:00
voidspacejam: the actual error is16:01
voidspace[LOG] 1:26.513 SYNC failed to connect: dial udp: too many colons in address ::1:4211316:01
voidspacejam: so we need to be able to start mongo with the "bad" ipv6 address format, and then use the "good" format elsewhere16:01
dimiternvoidspace, sweet! there's progress then :)16:03
=== Ursinha is now known as Ursinha-afk
=== urulama is now known as urulama-afk
wwitzel3ericsnow: do you know how to use rbtools with our OAuth login on the rb instance?16:17
ericsnowwwitzel3: yuck16:20
ericsnowwwitzel3: I hadn't thought of that16:20
ericsnowwwitzel3: I'll take a look16:20
wwitzel3ericsnow: yeah, we will have to extend rbtools to support it16:23
wwitzel3ericsnow: looking at the code now16:23
ericsnowwwitzel3: that or some kind of wrapper16:24
ericsnowwwitzel3: I'm also going to look for a git plugin for rbt16:24
wwitzel3ericsnow: yeah, or one you oauth with git, make the user able to set a password and login either way?16:25
wwitzel3ericsnow: but only allow creation through the inital oauth16:25
ericsnowwwitzel3: yeah, maybe16:25
ericsnowwwitzel3: depends on how easy the OAuth approach is16:26
=== Ursinha-afk is now known as Ursinha
ericsnowwwitzel3: try this for your password: oauth:wwitzel@github17:04
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
voidspaceLooks like home internet works!17:48
voidspaceThis box not yet on it17:48
voidspaceBig switchover shortly17:48
voidspaceStill a crappy 3.6Mbps line though, which is really annoying17:48
perrito666voidspace: wow, I have more, that is not normal17:49
voidspaceperrito666: it used to be ok, about four months ago it got really crappy17:50
voidspaceperrito666: I hoped changing provider might fix that17:50
voidspaceapparently not17:50
voidspaceall changing provider did is leave me without internet at all for ten days...17:50
perrito666voidspace: I have a 10 or 12 Mbps line, lousy upload and terrible lag but at least has more bw :p17:50
voidspacehah17:51
voidspace:-p17:51
=== Ursinha is now known as Ursinha-afk
wwitzel3ericsnow: that worked! :)18:07
ericsnowwwitzel3: awesome18:08
gsamfiravoidspace, perrito666: no worries, Google Fiber will soon take over the world via project Loon :P18:08
ericsnowwwitzel3: now if only I could get rbt post to do what I want18:08
perrito666gsamfira: I live outside of the world :p18:09
gsamfira=))18:09
wwitzel3ericsnow: what do you want it to do?18:09
ericsnowwwitzel3: actually work :)18:09
ericsnowwwitzel3: it's not generating the right diff18:09
wwitzel3ericsnow: hrmm .. mine worked just fine, are you trying to do something specific?18:11
wwitzel3ericsnow: or just the default behavior of diff vs. master18:11
ericsnowwwitzel3: yep, just that18:11
ericsnowwwitzel3: I'm pretty sure my branches have confused it18:11
ericsnowwwitzel3: so diffs are reflecting only the last commit in the branch18:13
wwitzel3ericsnow: if none of the other commits are merged in to master, that is really weird18:14
ericsnowwwitzel3: yep18:15
wwitzel3ericsnow: the first step is figuring out how to generate the right diff using git diff --full-index and if you can get the diff you want there, then you can usually get rbt to do what you want.18:15
ericsnowwwitzel3: looks like rbt can't handle it when my branch is based on a stale master18:23
katcoericsnow: to be fair, would you ever really want to review something like that?18:55
ericsnowkatco: no :)  however, rbt happily did the wrong thing18:56
ericsnowkatco: well, the wrong thing for me :)18:57
katcoericsnow: ah i see :p18:57
perrito666ericsnow: did you get your bkp code landed?18:59
ericsnowwwitzel3: FYI, I made a comment on your test review request :)18:59
=== Ursinha-afk is now known as Ursinha
ericsnowperrito666: still waiting on reviews19:00
natefinchericsnow: sorry... I'll look at it after my current meeting19:00
ericsnownatefinch: thanks!19:01
natefinchor during the meeting if alexisb is held up ;)19:01
perrito666wwitzel3: you have been reviewed20:12
natefinchericsnow: you have an LGTM20:17
ericsnownatefinch: thanks20:17
perrito666'o/20:18
perrito666\o/20:19
perrito666my previous happy guy had no right arm apparently20:19
ericsnowperrito666: the result of a terrible waving accident20:20
perrito666ericsnow: the result of spending a life using a en_US keyboard and now type in an es_ES20:20
ericsnowperrito666: ah, it's worse than I thought ;)20:21
perrito666ericsnow: since I moved to my wife's laptop I am stuck with an es kb until my next US trip (this one is far easier to change though :p)20:21
natefinchperrito666: no more ripping out keyboards via brute force?20:28
ericsnowperrito666: I've reviewed your restore-mode patch.  Basically LGTM.20:28
perrito666natefinch: its a thinkpad, its as close as it gets to being made out of lego20:28
perrito666ericsnow: thanks20:28
perrito666ericsnow: merge streak20:33
ericsnowperrito666: don't forget that my LGTM doesn't count yet :)20:34
perrito666ericsnow: well you just merged my patch20:34
ericsnowoh shoot20:35
perrito666s/oo/i/20:35
ericsnowperrito666: no wonder it didn't go for the one I thought I had done20:35
ericsnowis there a way to cancel the $$merge$$?20:37
perrito666ericsnow: I presume it can be done by hand20:37
perrito666I might be able to break the patch so it wont merge, but I would rather not20:38
ericsnownatefinch: you know how to tell the bot to cancel a merge?20:38
natefinchericsnow: I don't think it's possible20:47
natefinchericsnow: at least not easily20:48
ericsnowwell, it passed CI and merged20:49
ericsnownatefinch: should we revert it?20:49
natefinchericsnow: what's the PR? Maybe I can do a retro-active review :)20:52
ericsnownatefinch: :)20:52
ericsnownatefinch: https://github.com/juju/juju/pull/67820:52
wwitzel3perrito666: thanks, that else was just an artifact of refactoring, nothing actually goes there, I will remove it20:55
natefinchperrito666: I don't see anything glaringly wrong, but I haven't really had time to thoroughly review it.21:01
natefinchperrito666: I say leave it21:02
natefinchperrito666: only minor fidly stuff, no logic problems.21:02
ericsnownatefinch, perrito666: yeah, that's the way I saw it too21:02
natefinchperrito666, ericsnow: gotta run.  Good luck!21:04
ericsnowperrito666: FYI, that backups patch has landed now21:07
ericsnowperrito666: https://github.com/juju/juju/pull/708 adds the top-level backups abstraction21:07
* perrito666 jumps like an anime little girl21:08
ericsnowperrito666: lol21:08
wwitzel3perrito666, ericsnow: what do you guys need reviews of21:14
ericsnowwwitzel3: https://github.com/juju/juju/pull/70821:14
wwitzel3besides that21:14
waigani_thumper: morning, https://github.com/juju/juju/pull/702 I've got an LGTM but wanted to double check with you21:14
wwitzel3j/k :P21:14
ericsnowwwitzel3: :)21:14
thumpermorning21:15
ericsnowwwitzel3: 702 is a much smaller patch, not the fat one that just landed21:15
thumperwaigani_: fine with me21:15
ericsnowwwitzel3: 708 I mean21:15
lazyPower-sprintkwmonroe: whats your google+ link you want in the circle?21:31
lazyPower-sprintoops wrong chan21:31
thumperrick_h_: can we reschedule the call at the end of the week re:mess?21:51
rick_h_thumper: when works for you?22:05
rick_h_thumper: I really want to chat. I've not heard what's up since Germany really and want to make sure we've got a good path on that22:05
thumperrick_h_: can you do a day earlier?22:05
rick_h_thumper: put something on the calendar and we can make it fit22:06
thumperrick_h_: any changes I make only appear on my calendar22:06
thumperthe event creator needs to change it22:06
rick_h_thumper: ok, so the same time the day before?22:07
thumperyep22:07
rick_h_thumper: or a diff time as well?22:07
thumperI can make that time22:07
rick_h_thumper: ok cool22:08
rick_h_thumper: alexisb has stuff at that time but will move so we can get caught up22:08
rick_h_thumper: and we can report to urulama-afk and alexisb if we need then22:08
rick_h_thumper: ok, moved and such. Let me know if you don't get the notice. Dinner time here, biab22:10
menn0thumper: meta-review pls: https://github.com/juju/juju/pull/67822:11
wallyworld_katco: hiya, back now if you are free22:14
katcowallyworld_: i am, one sec22:14
thumpermenn0: ack, otp22:14
menn0thumper: no rush22:14
katcowallyworld_: ok ready?22:18
wallyworld_yup22:18
=== jherouxz is now known as jheroux_away
perrito666menn0: thanks a lot dude22:22
menn0perrito666: np22:22
perrito666wow github lied to me blattantly22:23
perrito666wow, thumper uses emoticons in his reviews, we are on a whole new level here22:35
perrito666I wonder if I can get davecheney to do the same with mine :p22:35
thumperheh22:36
thumperdoesn'22:36
thumperI don't think that github supports the unicode "pile of poo"22:36
* perrito666 feels thumper wants to tell him something22:42
* thumper whistles22:42
perrito666thumper: look out, you might have an accident while visiting a foreign country like say... belgium?22:43
thumperhmm... waffles...22:44
perrito666tell memore about waffles22:45
perrito666if someone where to read this conversation without context none of us would pass a turing test22:46
thumperperrito666: and that is the way it should be22:49
thumperunpredictable22:49
ericsnowwallyworld_: FYI, I've added you and the other team leads who have logged in as reviewboard admins22:59
ericsnowsorry thumper22:59
davecheneysinzui: ping23:00
wallyworld_ericsnow: awesome, thank you, on call, will look soon23:03
ericsnowwallyworld_: no worries23:04
ericsnowwallyworld_: I'm pretty close to EOD, but I'm sure I'll pop in a time or two before tormorrow23:04
perrito666ericsnow: why is there an interface for Backups?23:07
perrito666ericsnow: I am pretty sure you already explained this to me at some point23:08
ericsnowperrito666: it's to hide away the implementation details of backups23:09
ericsnowperrito666: so that there is a single concise implementation in terms of the larger low-level code23:09
ericsnowperrito666: without the low-level details (as much as possible)23:10
perrito666ericsnow: I take there is more to the implementation than what I see in 708?23:10
wallyworld_hazmat: sorry i missed you, i fell asleep after pinging you. i will change the hard coded "precise" string. what i was curious about was why the distro-info command failed which caused the hard coded default to be used23:11
wallyworld_hazmat: we already default to the host series for non-local from what i can see; i'll double check that's the case23:12
ericsnowperrito666: there's the create() implementation from the previous PRs23:13
ericsnowperrito666: the code in 708 just has to call the low-level create() method23:13
ericsnowperrito666: basically what you already have implemented for restore would be the low-level implementation of it23:13
wallyworld_sinzui: hi, there's been a change to goose to work around the ggc compiler change which broke ppc tests on CI. juju has had the dependencies updated so those failures are not happening anymore. but there still looks like a linker error with the compiler causing other failures23:19
wallyworld_sinzui:  can a CI test be set up for goose on ppc - this will flag any future breakages like the one we just saw?23:20
waigani_davecheney, thumper: I blew away my go/pkg dir and the pre-push hook passed23:31
davecheneywaigani_: did you run ./all.bash ?23:32
menn0thumper, waigani_: here it is: https://github.com/juju/juju/pull/70923:32
davecheneyit will have given you a message explaining exactly what has changed23:32
waigani_davecheney: no...23:32
waigani_oh23:32
waigani_menn0: ta da23:33
davecheneywaigani_: http://paste.ubuntu.com/8294496/23:33
waigani_davecheney: handy, thanks23:34
davecheneywaigani_: please remember that the official compiler version we use to compile juju is the one that ships with ubuntu23:34
davecheneyif you want to use tip23:34
davecheneythat's cool, it often turns up interesting bugs23:34
davecheneybut it's your responsibility to own your environment and deal with the tools we've written that may not expect to be run against tup23:34
davecheneytip23:34
waigani_yep, understood I'm using go that ships with Ubuntu23:35
davecheney09:31 < waigani_> davecheney, thumper: I blew away my go/pkg dir and the pre-push hook passed23:36
davecheney^ what does this mean ?23:36
davecheneyoh23:36
davecheneyyou mean $GOPATH/pkg23:37
davecheneyright23:37
waigani_sorry, yeah23:37
davecheneynope, my mistake23:37
waigani_next time I'll use ./all.bash and remove just the offending pkg23:38
davecheneysinzui: ping23:38
davecheneywaigani_: please don23:38
davecheneyplease disregard all the advice I fave you23:38
davecheneyit is only relevant if you are running tip23:38
waigani_oh right. consider it disregarded!23:39
waigani_thumper: quick look - https://github.com/juju/juju/pull/702/files I've added three lines23:39
davecheneywallyworld_: can you kick this build job http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el-devel/23:44
davecheneyfor me23:44
thumperwaigani_: my gut reaction is that we should only need one factory line not two...23:44
wallyworld_ok23:44
thumperwaigani_: but since an env user isn't necessarily local23:44
thumperwaigani_: it seems that we should add something to the makeUser method23:44
thumperwaigani_: so it can also create the env user23:44
waigani_thumper: yeah, I know what you mean, had the same initial thought23:44
thumperwaigani_: in fact, we probably want the default behaviour of MakeUser to be one where the env user is created23:45
waigani_thumper: sure, shall I add that to this branch23:45
thumperwaigani_: and we need to explicitly say "don't make an env user for this user"23:45
thumperwaigani_: yeah, should only be a few lines23:45
* thumper looks at the params23:45
waigani_yep, no problem23:45
wallyworld_davecheney: what do you need done?23:46
thumperwaigani_: since go defaults to zero values, I feel we are going to have a double negative...23:47
thumperwaigani_: so "if !params.NoEnvUser { make the env user }@23:47
davecheneywallyworld_: i'd like to know if the fix to goose has made the ppc situation any better23:47
davecheneywallyworld_: can you push the build button on that job for me ?23:47
thumperwaigani_: that way we could create a user, and set {  NoEnvUser: true, ... } in the params23:47
wallyworld_davecheney: it has - you can see from the job runs overnight. but there's still alinker issue23:47
thumperwaigani_: that way the default will be the most expected behaviour23:48
davecheneywall fuk23:48
davecheneyerr23:48
davecheneyok23:48
thumperwaigani_: make sense?23:48
wallyworld_davecheney: eg23:48
wallyworld_/tmp/go-build476839839/github.com/juju/juju/api/provisioner/_test/github.com/juju/juju/api/libprovisioner.a(provisioner.o): In function `github_com_juju_juju_api_provisioner.DistributionGroup.pN44_github_com_juju_juju_api_provisioner.Machine':23:48
wallyworld_/home/ubuntu/juju-core_1.21-alpha1/src/github.com/juju/juju/api/provisioner/machine.go:161: multiple definition of `github_com_juju_juju_api_provisioner.DistributionGroup.pN44_github_com_juju_juju_api_provisioner.Machine'23:48
davecheneywallyworld_: got it23:49
wallyworld_davecheney: ty23:49

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!