/srv/irclogs.ubuntu.com/2013/03/20/#juju-dev.txt

davecheney_thumper_: seen mramm ?00:09
_thumper_davecheney: yeah00:10
* davecheney waits on mramm for our 1-on-100:10
=== _thumper_ is now known as thumper
bigjoolsdavecheney: I'm trying to do a go get on github.com/andelf/go-curl and go says "cannot find package", yet it seems to work for others.  Any ideas?02:33
bigjoolsok we worked it out02:35
bigjoolsFFS02:35
bigjoolsmy GOPATH started with a :02:35
bigjoolsridiculous02:35
davecheneybigjools: O_O!!03:38
davecheneyahh, there is a long story about that03:39
davecheneywhich isn't (all) go's fault03:39
bigjoolsdavecheney: I see :)03:40
davecheneyfor example, did you know that PATH=$PATH:: is short hand for03:41
davecheneyPATH=$PATH:.03:41
bigjools!03:42
bigjoolsdavecheney: so when I do a perfectly reasonable line like this in my .bashrc, I am screwed then: export GOPATH="$GOPATH":/my/path03:43
davecheneyyup, that isn't awesome03:43
davecheneythat says GOPATH=.:/somethign/else03:44
davecheneywhere . is the first element, and will be the target for go get03:44
davecheneyif it is of any consolation, i have fixed many of these usability problems in 1.103:44
davecheneybigjools: thumper do you two have time to talk about the mongo problem ?03:45
bigjoolssire03:45
bigjoolssure even03:45
davecheneyso, latest i have is everyone is working really hard to get mongo into at least raring by 13.0403:46
davecheneybut I think we need a backup03:46
davecheneybigjools: what I would like to do is move from the tarball to your ppa for getting mongo for the bootstrap nodes03:47
bigjoolsthe current raring packaging has a bug when it tries to build with ssl, fwiw03:47
bigjools\o/03:47
bigjoolshow about an official PPA03:47
bigjoolsI just did mine as I don't trust unsigned binaries :)03:48
davecheneybigjools: sounds fine by me, i'd really like your guidence on this03:48
bigjoolsok03:48
davecheneythe trick is we need P, Q and R, amd64 as a must03:48
davecheney386 and arm as a maybe03:48
bigjoolsthumper was saying that the 2.2.3 build didn't work for him03:48
davecheneyshitter03:48
bigjools2.2.2 was defo ok03:48
bigjoolsI built 2.2.3 in the same way so I dunno what's up03:48
davecheneybigjools: how do I, or you03:49
davecheneyi think you'd be faster03:49
davecheneyget a PPA location setup so I can then reference that in the cloud init scripts ?03:49
bigjoolsactivate a new PPA for the juju devs?03:50
davecheneybigjools: full disclosure, i'm a launchpad numpty03:50
davecheneyyou'll have to walk me through this03:50
bigjoolsheh03:50
bigjoolsnp03:50
bigjoolswhich team should own the ppa?03:50
davecheneyi think htere is a juju team now03:50
davecheneyit used to be ~gophers03:50
bigjoolsperfect03:50
davecheneybut that was changed recently03:50
* davecheney checks03:50
davecheneybigjools: if you are not a member03:51
davecheneyi can fix that03:51
bigjoolsso if you are an admin of the team you can activate a ppa for it03:51
bigjoolsonce you do that you can't rename the team03:51
bigjoolsnot sure I need to be a member03:51
* davecheney goes to look03:51
davecheneyhttps://launchpad.net/~juju03:51
davecheney^ i can create a new ppa03:51
bigjoolson the left mid way down, yes03:52
bigjoolsso I suggest making one called devel, one called staging and one called stable03:52
bigjoolsor perhaps s/devel/experimental/03:53
bigjoolsthen we can test packages and copy them later to the stable ppa03:53
davecheneybigjools: sounds reasonable03:53
davecheneyi'm goig to be adding this ppa to the cloudinit script03:53
bigjoolsthe forms on LP are pretty self-explanatory I hope03:53
davecheneyhow will having dev/stage/stable play into this03:54
bigjoolsyeah then it does need a production PPA that is stable :)03:54
davecheneyok, done devel and stable03:54
davecheneythat'll do for the moment03:54
davecheneybigjools: also, because i'm a lp numpty, i'm not setup with all the pkg signing aparatus03:55
bigjoolseasy to sort03:55
davecheneyok03:55
bigjoolson the cmd line just type "add-apt-repository ppa:account/ppa-name"03:55
davecheneyoh yeah, i've done that side03:56
bigjoolsit won't work until the 1st package is in the PPA though03:56
davecheneybut the publishing part will probably involve more steps03:56
bigjoolspackaging is tricky, yes03:56
bigjoolsI can help there03:56
davecheneysweet, thanks03:56
bigjoolsI suggest that once you have the experimental PPA up, you copy my 2.2.3 package into there03:56
davecheneyok, experimental created03:57
davecheneyi copied your ppa once before03:57
* davecheney looks03:57
bigjoolsso visit my PPA's packages page03:57
bigjoolsand click copy packages03:58
bigjoolsthen your experimental PPA will be in the destination drop down03:58
bigjoolscopy to same series and copy source + binary03:58
davecheneyhttps://launchpad.net/~julian-edwards/+archive/mongodb04:00
davecheneywhy is it always a sonofabitch to find the copy link ?04:00
bigjoolsclick "view packages"04:00
bigjoolsit's on that page04:00
davecheneygotcha04:00
davecheneyok, wheels are grinding now04:02
davecheneywhile that is baking i shall make a branch to use it04:02
bigjoolswill be interesting to see why it fails04:05
bigjoolsor at least thumper said it fails :)04:05
davecheneybigjools: can I use copy package to change the series to quantal, etc ?04:05
bigjoolsdavecheney: not a good idea to go backwards04:05
davecheneyok04:05
bigjoolsdavecheney: I did a 2.2.2 for quantal04:05
bigjoolsit works fine04:05
davecheneythat is ok04:06
davecheneyonce we get _one_ working mongo04:06
davecheneyi can use default-series to boot an environment that matches04:06
bigjoolsyou can promote 2.2.2 from quantal to raring04:06
davecheneybigjools: is there anyone I should talk to about using ppa's in cloudinit ?04:06
bigjoolssmoser04:06
davecheneyof course04:07
davecheneyda man04:07
davecheneybigjools: are you on juju-dev ?04:31
davecheneyML04:31
bigjoolsdavecheney: yes I think so04:31
davecheneyok, just announced my intentions for this packaging bollocks there04:31
davecheneyplease comment and/or throw fruit04:32
bigjoolsseems ok04:32
davecheneydidnt' expect it to be contraversial04:32
bigjoolsI don't understand juju well enough to comment really04:32
bigjoolswriting the python maas provider was enough for me :)04:33
davecheneyi can give you more background04:33
davecheneybut you probably don't want to know04:33
davecheneythumper: seen this test failure ?04:34
davecheney... obtained *net.OpError = &net.OpError{Op:"write", Net:"tcp", Addr:(*net.TCPAddr)(0xc200245d50), Err:0x68} ("write tcp [::1]:46014: connection reset by peer")04:34
davecheney... expected *rpc.ServerError = &rpc.ServerError{Message:"transformed: message", Code:"transformed: code"} ("server error: transformed: message (transformed: code)")04:34
thumperumm...04:59
thumperperhaps05:00
thumperI do still get some intermittant failures05:00
davecheneyhappens 100% of the time05:00
davecheneywill bisect in a few mins05:00
thumperhmm05:00
thumperI ran tests just before I went out05:01
thumperand all passed05:01
davecheneytrying to improve my jenkins builder to run our tests05:01
thumperlet me check my trunk revision05:01
davecheneythumper: which revno05:01
thumperr102405:01
thumperdavecheney: what do you have?05:01
davecheneyhmm, how can I check05:01
davecheneythis branch has local changes05:01
thumperbzr revno05:02
davecheneyyeah, checking 1024 now05:02
davecheneythumper: yup, totally repeatable @102405:03
davecheneyif you can confirm the failure I will log a bug about it05:03
thumperhmm... I don't get it on r102405:03
davecheneywe've seen failures like this before05:03
thumperlet me triple check05:03
davecheneyraces between the client reading the reseponse and the server closing the connection05:04
thumperok  launchpad.net/juju-core/rpc0.186s05:04
thumperthat is with r1024 of trunk05:04
davecheneythumper: can you please try GOMAXPROCS=2 go test launchpad.net/juju-core/rpc05:15
thumperdavecheney: tried three times, succeeded each time05:16
* thumper is trying to land the upload tools tweaks05:16
davecheneywith GOMAXPROCS=2 (or any number other than 1)05:16
davecheneyfair enough05:16
thumperOMG there were a lot of conflicts with trunk05:16
thumperdavecheney: yes, with set to 205:17
davecheneythat is important, gets the review queue unblocked05:17
thumperexactly05:17
davecheneycarry on05:17
* thumper nods05:18
davecheneybigjools: https://launchpad.net/~juju/+archive/experimental/+packages05:24
davecheneylooks like it worked05:24
bigjoolsdavecheney: cool05:24
bigjoolsso hax0r away05:24
thumperbigjools: have you fixed mongo?05:30
thumperdavecheney: branch landed05:30
bigjoolsrofl05:30
* thumper is done05:30
bigjoolsthumper: it's not a packaging problem AFAICS therefore it's a juju problem05:31
bigjoolsor mongo.... take your pick. either way, it's not *my* problem :)05:32
thumperheh05:32
bigjoolsyou said you saw libssl linked?05:32
bigjoolsperhaps 2.2.3 has something that breaks juju05:32
thumpermaybe... NFI05:33
* thumper goes to help make dinner05:33
davecheneyLFTM, ldd $(which mongod) linux-gate.so.1 =>  (0xb76ef000) libssl.so.1.0.0 => /lib/i386-linux-gnu/libssl.so.1.0.0 (0xb7690000)05:36
davecheneymongo is stupid05:36
davecheneythey require the elephant that is libboost05:36
davecheneybut they only need it for the headers05:36
davecheneythumper: ok, the rpc test failure is real05:40
davecheneyjust verified under 1.0.3 as well05:40
davecheneyit fails under my stress test script05:41
davecheneywill raise a bug05:41
davecheneyhttps://bugs.launchpad.net/juju-core/+bug/115755305:43
_mup_Bug #1157553: rpc: tests are unreliable <juju-core:New> < https://launchpad.net/bugs/1157553 >05:43
davecheneyjam1: question about juju cli for windows ?08:21
=== jam1 is now known as jam
jamyes davecheney?08:21
davecheneydid you get a cross compile working, or did you do it directly on win32 ?08:21
jamdavecheney: just did it directly08:22
davecheneyfair enough08:22
davecheneythat is what I tought08:22
davecheneythought08:22
davecheneyas you were08:22
jamwell, I need to actually run it there anyway, right ? :)08:22
davecheneyyes08:22
davecheneysorry, was answerin a question from another channel08:22
jamnp08:22
jamI did recompile go, but I probably could have just downloaded the 32-bit version. The big key was just needing 'gcc' for cgo.08:23
jamThough maybe I would need a "close" match between the gcc for go and the one for goyaml08:23
jamso maybe I would need to recompile go08:23
jamC usually isn't as version specific as C++08:23
davecheneyi would really like to remove the cgo requirement for goyaml08:24
davecheneywhich is a medium sized job08:24
davecheneybut might be helpful if we have to cross compiile a lot in the future08:25
jamwhen I mentioned it in the past (1yr?) gustavo claimed that he didn't want to implement the mess that was yaml parsing.08:25
davecheneyie, I don't see lp supporting a win32 target any ti,e in the future08:25
davecheneyyeah, that is very true08:25
davecheneygoyaml just wraps the python yaml c code08:25
jamright08:25
davecheneyso it has the same compatibilty as the python version08:25
davecheneya decent compromise08:25
jamfrom what kapil said, yaml parsing speed becomes a bottleneck at large scale, so having a slow parser might also be bad08:26
jambut it could be *really* useful if the API was compatible08:26
jamso you could use "A" or "B" yaml libsd08:26
jamlibs08:26
jamand then we could have native go  yaml for stuff like win3208:26
davecheneyi don't think yaml parsing speed is that importnat08:27
jamdavecheney: do you know why cgo doesn't support cross-compiling? You certainly have cross-compiling gcc08:27
davecheneywe don't have the topology node08:27
davecheneyjam: it's not that it doesnt' support it08:27
davecheneybut the moving parts required especially to go from elf -> PE08:27
davecheneyare substantial08:27
davecheneyif you did all the compilation steps by hand, and had the right cross compiler and cross linker, as well as headers and .so's08:28
davecheneyyou could do it08:28
davecheneybut it was just too hard to autoate with the simple go tool08:28
jamsure. So for Bazaar, we just created an EC2 instance that we bundled, and then brought up when we did a release to build the next binaries.08:28
jamIt wouldn't be hard to set that up again.08:29
rogpeppemornin' all08:29
jamhi rogpeppe, hope your day has started well so far08:29
davecheneymorning rogpeppe08:29
rogpeppejam: the shower was hot :-)08:29
rogpeppedavecheney: hiya08:29
davecheneyjam: i think that is a good suggestion08:29
davecheneylets see how it pans out08:29
davecheneymaybe someone will kick in the resources for this machine on a native platform08:30
davecheney:P08:30
jamdavecheney: so in the short-term, I have no problem doing the compiling on win32, but I imagine we'll want a more automated system so we can detect issues pre-release.08:30
dimiternmorning08:31
davecheneyjam: smells like a jenkins shaped problem08:31
jamhi dimitern08:31
jamdavecheney: it does, indeed. Though my personal jenkins-fu is quite rusty.08:31
davecheneyjam: i've setup a jenkins CI build now that i have the right i386 mongo08:31
jamI've used it, I've played with it, but not allt hat much.08:31
jamdavecheney: you have a mongod that runs on Windows? Or is that Linux-i386?08:32
davecheneyjam: jenkins is easy, imagine a hammer that has been worn down by repeatedly bashing on hard thins08:32
davecheneythat is jenkins08:32
davecheneyjam: i386, see mail to juju-dev about backstop plan for how to get away from using a tarball for mongo08:32
jamvila from our old squad had jenkins stuff set up. I think his biggest trouble was trying to create a reproducible save state (so if needed someone else could bring it up similarly). It wasn't terrible to configure and sort of hack it together.08:33
jamdavecheney: yeah, I saw the "start at a ppa"08:33
jamthough IIRC the ppa build breaks the test suite.08:33
jamIf you run with '-gocheck.vv' I *think* rogpeppe and I (mostly him at my request) made it so the failing mongod would at least get logged.08:33
davecheneyjam: i think we fixed that08:33
dimiternsetting up jenkins is painful, but at least takes less than a day, even from scratch - did it twice :)08:36
rogpeppedavecheney: i saw your TestTransformErrors failure yesterday08:36
rogpeppedavecheney: it only happens in go tip08:36
rogpeppedavecheney: and i haven't narrowed it down yet08:36
davecheneyrogpeppe: it happens with 1.0.308:39
davecheneyi checked08:39
rogpeppedavecheney: orly?08:39
davecheneyit's timing related08:39
davecheneyrogpeppe: http://play.golang.org/p/lSPW2BO2Tk08:39
davecheneycd $PKG08:39
davecheneybash stress.bash08:39
rogpeppedavecheney: i'm sure it never happened before on go tip, so i'm surprised about 1.0.308:39
davecheneyfails in a few seconds08:39
dimiternfwereade: https://codereview.appspot.com/7497047/ please?08:40
rogpeppedavecheney: ah, i've managed to reproduce the problem on 1.0.2 (and i also saw a log-target-race related panic)08:46
davecheneyit's less prevelent under the old 1.0.x scheduler08:47
davecheneybut the race is there08:47
rogpeppedavecheney: yeah (this time i saw the issue in a different test, BTW, but still "connection reset by peer")08:48
rogpeppedavecheney: (when reading a response, but writing the request though - it might be a different issue but i suspect not)08:49
rogpeppes/but/not/08:49
davecheneyrogpeppe: indeed, the faliure modes are always subtly different but it appears to be the client and the server racing to read the final message08:49
davecheneyrogpeppe: didn't dimeter fix a very similar bug in Atlanta ?08:49
fwereadedimitern, sorry, on it right now08:49
dimiternfwereade: cheers!08:49
rogpeppedavecheney: ah, i *think* i might see the problem08:50
dimiternhow can i subscribe to canonical-juju ?08:51
davecheneydimitern: how much cash do you have ?08:51
dimiterndavecheney: :D08:51
davecheneydimitern: https://lists.canonical.com/mailman/listinfo/canonical-juju08:53
rogpeppedavecheney: yup, fixed it, i think08:53
dimiterndavecheney: as a BG rapper sang once "i own tens of bucks"08:53
dimiterndavecheney: cheers08:53
rogpeppedavecheney: https://codereview.appspot.com/791904308:56
* davecheney looks08:57
davecheneyrogpeppe: that'll do it every time08:58
rogpeppedavecheney: yup :-)08:58
davecheneyi'm surprised that ever work08:58
davecheneyworked08:58
rogpeppedavecheney: yup.08:58
davecheneywhat is the history on that file08:58
davecheneyi have a worrying feeling that we tried to 'fix' it before08:59
davecheney(poorly)08:59
rogpeppedavecheney: i guess it was almost always running the new goroutine first08:59
rogpeppedavecheney: i'm not sure. i don't remember doing so08:59
davecheneyi have a vague memory from atlanta09:00
rogpeppedavecheney: i might've missed it, being involved with gui stuff09:01
* davecheney checks09:01
davecheneythe reason i mentoin it is we had a very similar bug that week09:01
rogpeppedavecheney: well, there are no fixes of that kind in the rpc package at any rate09:04
rogpeppedavecheney: this produces only ian's recent changes to log:  for(i in *.go) {echo $i; bzr blame $i | grep -v '^ +\|' | grep -v roger.p }09:04
davecheneysomeone showed me09:05
davecheneybzr log --show-diffs09:05
davecheneybut I cna't make it work09:05
davecheneywhat am I doing wrong ?09:05
rogpeppedavecheney: i just checked that noone other than me has made any changes to the files in rpc. :-)09:05
davecheneymaybe it was a similar error in another package09:06
davecheneyscrew it09:06
davecheneyget one more LGTM and job done09:06
davecheneyyup, i'm happy we are not reverting another fix09:07
fwereadedimitern, qualified LGTM, ping me if there's any uncertainty09:11
dimiternfwereade: tyvm09:12
dimiternfwereade: no, i'm not getting the thing about using the wrong charm url in unit.go, please explain09:14
fwereadedimitern, the service config we care about from the perspective of the unit is the config that is known to match the *unit*'s charm url09:15
* TheMue likes failing tests *hmpf*09:15
dimiternfwereade: exactly what i'm doing isn't it?09:16
dimiternfwereade: getting u.doc.CharmURL + u.doc.Service for the key09:16
dimitern(well, vice versa, but the code is correct i think)09:16
fwereadedimitern, looking up the service's charm url is defeating the point of doing all the work in the first place09:17
fwereadedimitern, not entirely ofc, because you do the right thing in watcher.go09:17
fwereadedimitern, when watching the unit's service config09:17
dimiternfwereade: are you talking about WatchServiceConfig() ?09:17
fwereadedimitern, WatchServiceConfig is Correct09:17
fwereadedimitern, ServiceConfig is not09:17
dimiternfwereade: aah, got you :)09:17
fwereadedimitern, they should be using the same document09:18
dimiternfwereade: right, sorry - i'll fix it09:18
fwereadedimitern, np, I didn't catch that first time round anyway09:18
fwereadedimitern, but, sorry, wait a mo09:19
fwereadedimitern, no, we're good09:20
fwereadedimitern, thanks :)09:20
dimiternfwereade: i'm a bit divided what comments to put around the ops09:21
fwereadedimitern, which ones? :)09:21
dimiternfwereade: this case you described about inc ops with a single upgraded unit in error state getting reverted09:21
dimiternfwereade: should we document this somehow, or make the "first use" comment more correct09:22
fwereadedimitern, I think that the explanatory line should be either made strictly accurate, or maybe even dropped; as it is, it clearly contradicts the preceding if/else chain09:23
fwereadedimitern, I'm not sure what explanatory power a strictly accurate version has though09:24
dimiternfwereade: ok, i'm dropping it :) can be bothered to wrap my mind around commenting all cases in a sentence09:24
fwereadedimitern, how about "The document might need to be created"?09:24
fwereadedimitern, I think that is accurate and high-level helpful09:24
fwereadedimitern, "A new settings document might need to be created"?09:25
dimiternfwereade: that's like saying the water is wet09:25
dimiternfwereade: we're creating it anyway09:25
fwereadedimitern, ha, sorry, "a new refcount document might need to be created"?09:26
dimiternfwereade: hmm... let me think a bit09:26
dimiternfwereade: what we're actually doing there is either creating a refcount=1 or just refcount++09:27
fwereadedimitern, honestly, it is documented anyway is settingsIncRefOp09:27
dimiternfwereade: yeah, i think it makes sense, although not saying when we might do it... ah right - just take a look at the settingsIncRefOp09:28
fwereadedimitern, on service creation we just blat it out directly because we don't need the complexity of an incref, but in this case we need it so we use it09:28
dimiternfwereade: ok, i'll change both inc and dec comments so09:28
fwereadedimitern, maybe the comments are best couched in terms of "the unit adds a reference to the new settings doc" and "the service drops its reference to its old settings doc"?09:29
dimiternfwereade: even better! 10x09:29
fwereadedimitern, cool09:30
TheMueHa, killed the next failure. *g*09:32
* fwereade cheers at TheMue09:32
TheMuefwereade: After changes the hard coded references to $HOME/.juju there are now still some hidden glitches in the tests. ;)09:34
fwereadeTheMue, yeah, no doubt -- good luck with them09:34
TheMuefwereade: thx09:34
dimiternthe update manager this morning suggested to do a "partial distribution upgrade" - never seen this before, i did it and /etc/issue still reports 12.10 - anyone seen this?09:44
dimiternfwereade: done with the changes - would you like one last check (mostly for wording in comments) before I submit?  https://codereview.appspot.com/7497047/09:46
fwereadedimitern, sure09:59
davecheneygood night gentlemen10:22
davecheneydon't forget, release on friday10:22
davecheneyi'll do the release notes dance tomorrow10:23
dimiterndavecheney: good night!10:23
davecheneynight y'all10:23
jamnight davecheney10:23
jamfwereade: so we had a question come up recently about Series vs Arch. Specifically, one seems to be a conf parameter, and one is meant to be a --constraint (from what I could tell).10:24
jamI noticed because both are a bit borked on Windows10:24
jamwhere it wants <unknown> i386, and we don't have those charms available :)10:24
jamI can change the code so that if series is unknown, it returns the default series from conf, but where do I overload the arch? (--constraint was the recommended method from mgz, IIRC)10:25
jammgz, dimitern: can either of you try bootstrapping on HP? I'm getting weird inability to write to swift, but I don't know if it is something about me.10:26
jamI would ask wallyworld, but he is obviously away right now.10:26
fwereadejam, heyhey10:26
fwereadejam, usage of version.Current.Series will hopefully be evaporating imminently: tim's in the process of clearing up the environs api to accept series instead of tools10:27
jamalso, fwereade, https://code.launchpad.net/~fwereade/juju-core/bootstrap-constraints-3a/+merge/153725 landed, so I think your "EC2 should choose" kanban card can go into merged?10:27
jamfwereade: yeah, I was hoping I wouldn't have to do much about series, and could let Tim do the work for me there.10:28
jamBut Arch is also important in this context, and it was unclear how that happens.10:28
jamAt the very least, it seemed both Arch and Series should be configured in the same manner10:28
jam(environments.yaml, or --constraints, but not both)10:28
jamwell, even both, but not one one way and the other differently10:29
mgzjam: can check10:29
jamthanks mgz, good morning, btw10:29
jamI hope your Muselix is tasty today10:29
fwereadejam, I don't consider my card really done until I've merged -6 -- it was originally meant to include that too10:29
fwereadejam, but I am working on merging it all today10:29
jamk10:29
fwereadejam, wrt series and arch I think matters are a little more subtle10:29
mgzeaten long ago, the fun of getting on the 7:45 bus...10:30
jammgz: I thought colocation was Thurs?10:30
mgzat least I have a net connection up to doing audio+ssh today, not sure what was up with the crazy packet loss at home yesterday10:30
jamyeah, that got pretty crazy at the end.10:30
mgzit is, I'm casually in town10:30
jamI could hear you fine, but your ssh lag was pretty bad10:31
mgzand irc stayed up...10:31
mgzbut incoming audio was being mangled, and screen was paaainful10:31
jammgz: so how much to get Fi-OS to your house? :010:32
mgzwell, there have been promises from the local authority...10:32
jamwe promise to make you pay through the nose10:32
jamto get 1Mbit10:32
fwereadejam, would you like a quick hangout?10:32
jamI guess T3 was 1.5Mbit?10:32
jamfwereade: sure10:32
jamfwereade: you want to set it up or me?10:33
mgzit's mostly a service reliablity and cost thing these days rather than speed, which is actually pretty reasonable10:33
fwereadejam, I was getting water but will do so now10:33
jamnp10:33
dimiternfwereade: ping10:38
fwereadedimitern, pong10:41
dimiternfwereade: does it look ok?10:41
fwereadedimitern, dammit sorry10:41
dimiternfwereade: np, i'm already half though the upgrade-charm cmd anyway :)10:42
rogpeppedimitern: one last question on your review https://codereview.appspot.com/7497047/10:44
dimiternrogpeppe: just answered :)10:44
rogpeppedimitern: hmm, but settingsRefsDoc doesn't *have* an id field. that was what was confusing me.10:45
dimiternrogpeppe: it has an implied id10:45
dimiternrogpeppe: if not specified mongo creates one, afaik10:46
rogpeppedimitern: oh, i see. we're relying on a mongo-created _id?10:46
dimiternrogpeppe: yeah10:48
rogpeppedimitern: so how do we find out that key to increment the ref count?10:49
dimiternrogpeppe: we use the serviceSettingsKey() to construct the key, and then Find/FindId10:49
rogpeppedimitern: oh, i see, we do specify the id when creating the document10:50
dimiternrogpeppe: exactly10:50
rogpeppedimitern: we just don't have in the doc10:50
rogpeppedimitern: hmm, i think that's a bit confusing. all the other docs contain their id fields.10:50
dimiternrogpeppe: yeah, if seemed a bit weird at first, but fwereade explained it and it made sense10:50
fwereadedimitern, rogpeppe: no, we specify the key in the txn.Op10:51
rogpeppefwereade: yes, i saw that. but don't all the other doc types have the key in the struct too?10:51
dimiternrogpeppe: well, saves a bit of typing: settingsRefsDoc{1} instead of {Id:x, RefCount:1}10:52
fwereadedimitern, rogpeppe: it ends up cleaner imo, because then we can just use the doc directly in update operations without mgo whining about _id not neing sane to update10:52
fwereaderogpeppe, and since it's not actually required anywhere else, meh, why include it?10:52
rogpeppefwereade: ok. maybe a comment next to the settingsRefDoc would be appropriate then10:52
fwereaderogpeppe, +1 to that10:53
dimiternrogpeppe: i'll add a comment10:53
rogpeppefwereade: because it wasn't obvious to me at any rate10:53
mgzjam: bootstrap on hp worked btw, want to compare config?10:53
fwereaderogpeppe, yeah, good observation10:53
jammgz:  my hp config: https://pastebin.canonical.com/87223/10:54
mgzbasically, pulled juju-core trunk, `go install ./...`, sourced creds, `~/go/bin/juju bootstrap --upload-tools` and `~/go/bin/juju status`10:55
dimiternfwereade, rogpeppe: thanks for the reviews. i'll include your suggestions and submit it shortly10:55
fwereadedimitern, cool, thanks10:55
jammgz: and the failure: https://pastebin.canonical.com/87224/10:57
jamI can't --upload-tools on Windows10:57
jambut that isn't where it is failing10:57
mgzjam: https://pastebin.canonical.com/87225/10:58
mgzjam: check your OS_REGION_NAME10:58
jammgz: export OS_REGION_NAME="az-3.region-a.geo-1"10:59
jamI checked that10:59
jamand the URL for the swift service looks right10:59
jamit is going to: https://region-a.geo-1.objects.hpcloudsvc.com/v1/AUTH_3...10:59
jambut it is getting an EOF11:00
jamwhich if I then 'swift list' after changing OS_REGION_NAME, I don't see the bucket mentioned in the bootstrap debug11:00
jamwhile I *do* see your hpgztesting bucket11:00
mgzI'll try changing the bucket name to see if creation is borked somehow, I would have had that bucket before11:02
jammgz: I would have thought 'destroy-environment' would delete the bucket.11:05
mgznope, that works as well11:06
mgzjam: you'd have thought...11:06
jammgz: (that can also be tested :)11:07
jammgz: I don't see your old bucket, and I do see a "hpgztesting-new" bucket11:07
mgz$ swift list11:09
mgzNo handlers could be found for logger "keystoneclient.v2_0.client"11:09
mgzEndpoint for object-store not found - have you specified a region?11:09
mgz...thanks python-swiftclient11:09
mgz...is that actually the bucket name?11:09
mgzah, no, it should have my hex bit as well11:10
jammgz: you have to change OS_REGION_NAME for swift list11:14
jambecause it doesn't match the nova region11:14
mgzah, ta11:17
jam(note that you have to use the regular one for 'nova list' but you have to change it for 'swift list'... sign)11:21
jamsigh11:21
mgzjam: so yeah, sorry, works for me, and nothing's obviously wrong with your setup in comparison11:22
dimiternrogpeppe, fwereade: is it ok for both of you if I just drop the last sentence of settingsRefsDoc comment (the one about the last unit upgrading should drop the old settings doc) and leave the rest? that is already mentioned in the service.go inline11:25
fwereadedimitern, I think I'd still favour dropping all but the first, for the same reasons, but I think I should defer judgment to roger who's in a better position to know how the code looks when it's unfamiliar11:26
dimiternfwereade: ok, so to make peace, I'll leave it then :)11:27
fwereadedimitern, what? this means WAR!11:28
fwereadedimitern, (no, that's fine ;))11:28
dimiternfwereade: :D11:28
jamWAR? This means PEACE!11:28
jamwait...11:28
rogpeppefwereade: is there anywhere else that has a more coherent comment about the overall strategy for managing settings docs?11:28
fwereaderogpeppe, there is not11:28
dimiternrogpeppe: not in one place, not11:28
fwereaderogpeppe, it would be a very good thing for us to write such a document, but in my personal queue it comes behind a how-to-write-transactions document11:29
rogpeppefwereade: in which case, i think it's good to have that comment there (and to perhaps fix it if it's not accurate), rather than inferring the strategy from scattered comments11:29
jamwallyworld_: I didn't mean to scare you away, please come back :)11:29
fwereaderogpeppe, yep, +111:29
wallyworld_jam: stupid mumble is freezing :-(11:29
rogpeppe fwereade: i really think it's nice to have comments in the code (perhaps in addition to other external docs)11:29
rogpeppefwereade: or at the least a pointer to the external doc within the code.11:30
fwereadedimitern, for pedantry's sake would you mention that the service decrefs the settings doc for its old url on change please?11:30
dimiternfwereade: sure, +111:30
rogpeppefwereade: when the service config settings change?11:31
fwereaderogpeppe, no, when the service's charm url changes11:31
rogpeppefwereade: how is that different from upgrading?11:31
fwereaderogpeppe, it's not, but the decref is not mentioned -- only the incref to the new one11:32
dimiternrogpeppe, fwereade: there it is - take it or leave it :) http://paste.ubuntu.com/5630902/11:32
rogpeppefwereade: "11:32
rogpeppeWhen a unit upgrades to the new charm,11:32
rogpeppe 717 // the old service settings ref count is decremented11:32
rogpeppe"11:32
rogpeppeisn't that what you're talking about?11:32
fwereaderogpeppe, ok, I would appear to be lacking in perception11:33
fwereadedimitern, go for it11:33
rogpeppefwereade: ah, i thought i was missing something crucial :-)11:33
dimiterncheers11:33
jamwallyworld_: no love for mumble today?11:33
wallyworld_trying11:34
wallyworld_jam: mumle won't connect, wanna do a hagout?11:36
jamwallyworld_: unfortunately, mgz is on his chromebook, and there are not binaries for G+ on it... :( but I can hangout with you and proxy for martin)11:37
jamwallyworld_: can you hear us?11:37
jamI see you in mumble11:37
wallyworld_no :-(11:37
jamif you can hear, then you can "speak" in the channel maybe?11:37
wallyworld_mumble locks the cpu11:37
jamjoy joy... :(11:37
wallyworld_i guess we can start a hangout11:38
jamwallyworld_: https://plus.google.com/hangouts/_/261ed219337d5a6c8fb184b90b243b7a06b746f3?authuser=2&hl=en11:39
jamdimitern: mgz ^^11:39
mgzwill be there shortly11:39
wallyworld_jam: can you hear me?11:40
jamwallyworld_: we can hear you11:40
jamyou can't hear us now?11:40
jammgz: no audio11:43
=== teknico__ is now known as teknico
TheMueAhhh! *jump* *jump* *jump* It seems I've found it.12:14
TheMue*tschakka* Yeah, got it. *big-smile*12:20
* fwereade_ cheers at TheMue again12:24
fwereade_rogpeppe, does http://paste.ubuntu.com/5631012/ look like something you've seen before?12:24
jammgz: it might have been the tools that are in the bucket on hp-cloud (in the shared account). When I added public-bucket-url, it seems to be working...12:24
jam(maybe we have a bug if you download from swift and then try to upload it)12:25
jamdoes swift do anything with uploading identical content?12:25
jamlike tell you "I don't need you to send any bytes because you already have a file matching that sha1" ?12:25
rogpeppefwereade_: no - AddRelation is new in the api. it's almost certainly something to do with the way that the AddRelation operation is being undone (or not...)12:25
rogpeppefwereade_: at least, i *think* AddRelation is new12:26
fwereade_rogpeppe, yeah, that's the latest commit12:27
fwereade_teknico, I'm seeing a consistent failure in trunk: http://paste.ubuntu.com/5631012/12:31
teknicofwereade_, what did I break? looking12:32
dimiternfwereade_: i have the same issue12:32
dimiternjust run the tests12:32
dimiternon trunk12:32
fwereade_teknico, if it's not happening for you, that's not immediately clear :)12:33
=== BradCrittenden is now known as bac
dimiternfwereade_: could it be because of the go version? i'm on 1.0.312:34
fwereade_dimitern, I'm 1.0.212:34
fwereade_teknico, assuming it's not so trivial as to take only a couple of minutes, would you back it out please? I'd be happy to help investigate on another branch if it turns out to be something tricky12:35
fwereade_teknico_, what was the last thing you saw?12:37
teknico_fwereade_, my own question at 13:33:5512:37
=== bcsaller__ is now known as bcsaller
fwereade_teknico_, I saw nothing from you before 13:3212:38
=== teknico_ is now known as teknico
fwereade_teknico, I have http://paste.ubuntu.com/5631037/12:38
teknicofwereade_, thanks, here's what I said:12:40
teknico[13:33:55] <teknico> fwereade_, is it the test on line 1952? it did pass here before I submitted12:40
teknicochecking right now12:40
fwereade_rogpeppe, fwiw it is not totally wonderful that a test failure seems to cascade into a panic; would it be simple to address this?12:46
teknicofwereade_, yes, it's happening here too, I'm not clear on why, so I'm reverting the whole branch12:47
fwereade_teknico, no worries, let me know when it's reverted12:48
rogpeppefwereade_: it's because opClientAddRelation returns a nil function pointer. i can't see how that could ever have worked12:48
rogpeppefwereade_: i don't believe the tests were run before submitting12:48
fwereade_rogpeppe, ah, jolly good12:48
fwereade_rogpeppe, thanks12:49
benjifwereade_: I am about to submit this branch https://codereview.appspot.com/7600044/ and it was suggested that you might want to know it is coming because it may cause merge conflicts with your current work12:51
teknicofwereade_, reverting branch is lp:~teknico/juju-core/revert-add-relation-commit12:54
teknicofwereade_, shall I directly "lbox submit", or do I still need to "lobx propose"?12:54
fwereade_teknico, I think you need to do both12:55
teknicofwereade_, https://codereview.appspot.com/7719050 , wanna have a look or shall I just submit it?12:59
fwereade_teknico, LGTMed on trust for form's sake13:00
TheMuerogpeppe: ping13:01
rogpeppeTheMue: pong13:01
dimiternbenji: please wait before submitting - trunk is currently broken13:01
TheMuerogpeppe: just updated and made a test ./...13:01
benjidimitern: thanks for the heads-up13:01
TheMuerogpeppe: now I've got this http://paste.ubuntu.com/5631081/13:02
TheMuerogpeppe: any idea?13:02
dimiternTheMue: that's the problem we have in trunk, pending a revert on last commit13:03
rogpeppeTheMue: yes, the recently merged branch broke trunk13:03
TheMuerogpeppe: ah, thx, so it's well known13:04
teknicofwereade_, landed, sorry about that13:04
teknicoand everyone too :-)13:04
fwereade_teknico, np, I'm pretty sure we've all done it, thanks for cleaning it up swiftly13:04
dimiternteknico: it happened to me recently, i know the feeling13:05
rogpeppefwereade_: i replied to the "Synchronizing: watchers, GUI, and API" thread BTW. you might want to have a look and weigh in.13:15
fwereade_rogpeppe, looks sane to me -- I don't feel any particular need to add to it, because I've shunted those details off into my mental "oakland" bucket13:17
rogpeppefwereade_: cool, thanks13:17
* rogpeppe goes for some lunch13:18
=== wedgwood_away is now known as wedgwood
rogpeppeback13:51
niemeyerGreetings al13:58
niemeyerl13:58
niemeyermthaddon: Heya13:58
niemeyermthaddon: Can we do a quick store deploy today?13:58
mthaddonniemeyer: sure, anyone from webops can do that - can you file an RT with the details?14:00
niemeyermthaddon: Sure14:00
mthaddonthx14:00
niemeyermthaddon: Doing that right now14:00
=== frankban_ is now known as frankban
niemeyermthaddon: #60244 is up14:24
_mup_Bug #60244: install problem <ubiquity (Ubuntu):New> < https://launchpad.net/bugs/60244 >14:24
niemeyer_mup_: no no no14:24
mthaddonthx14:25
niemeyermthaddon: Thank you!14:25
mthaddonniemeyer: https://pastebin.canonical.com/87252/ - has the branch location changed?14:30
niemeyermthaddon: Hmm14:31
niemeyermthaddon: It has, kind of14:31
niemeyermthaddon: It's still at lp:juju-core14:32
niemeyermthaddon: BUt the trunk owner has changed14:32
niemeyermthaddon: To ~juju14:32
niemeyermthaddon: If you had lp:juju-core, it should theoretically just work14:32
niemeyermthaddon: If you must use the explicit reference for some reason, then lp:~juju/juju-core/trunk should do14:33
mthaddonbzr remember resolves the full URL I believe, so I'm not sure if that's an option - will adjust manually14:33
mthaddonok, happier now14:34
niemeyersweet14:34
mthaddonniemeyer: deployed14:35
niemeyerwoot, thanks!14:35
rogpeppedimitern: i just saw this in trunk; any ideas?: http://paste.ubuntu.com/5631407/15:22
rogpeppedimitern: hmm, maybe not trunk actually, but i'm pretty sure the branch hasn't got a problem there.15:22
dimiternrogpeppe: hmm, haven't seen this15:22
dimiternrogpeppe: let me look closer15:23
dimiternrogpeppe: does it happen every time?15:24
rogpeppedimitern: i'll let you know when this test finishes15:24
dimiternrogpeppe: if it appears hung, just kill it - this happens sometimes on failure in the uniter tests15:25
rogpeppedimitern: it just passed15:25
rogpeppedimitern: i think it may just be a timeout that's too short15:25
dimiternrogpeppe: so it's intermittent15:25
rogpeppe(i hate those friggin' arbitrary timeouts)15:26
frankbanrogpeppe, dimitern: that seems similar to the 8 failures in trunk a mentioned yesterday, I was able to make the suite pass increasing timeouts in uniter_test15:26
rogpeppefrankban: yes, that's what i was thinking15:26
dimiternrogpeppe: no this seems something else15:27
dimiternrogpeppe: the unit's charm never gets installed for some reason15:27
dimiternfwereade_: ideas? ^^15:27
=== racedo` is now known as racedo
dimiternfrankban: what were these failures and what did you change?15:33
fwereade_dimitern, rogpeppe: looking15:39
rogpeppefwereade_: the most significant difference i can see is that the broken version goes into "awaiting error resolution for "start hook"" after "loading uniter state" where the ok version says "charm is not deployed"15:41
fwereade_rogpeppe, yeah, I am staring at that in complete bafflement15:41
frankbandimitern: I still see these failures in trunk -> http://pastebin.ubuntu.com/5631469/15:42
dimiternfrankban: at trunk tip?15:43
frankbandimitern: tests passed when I locally increased timeouts timeouts in uniter_test.go (from 5 to 20 seconds and from 50 to 200 milliseconds)15:43
frankbandimitern: revno 103415:43
dimiternfrankban: hmm... i'm getting trunk and running tests now15:43
dimiternfrankban: have you seen these before? r1034 is gustavo's store CL - not related to the uniter15:45
frankbandimitern: yes, 2 days ago, 1034 is just the revno of my last test run in trunk15:46
frankbandimitern: I think it was 1013.15:46
rogpeppefwereade_: can i run something by you please?15:48
fwereade_rogpeppe, sure15:48
dimiternfrankban: that's really strange - i run all tests on trunk at least 10 times a day and haven't seen these (apart from temporary failures when I was working on the uniter itself before proposing)15:49
rogpeppefwereade_: i'm looking at line 1353 on https://codereview.appspot.com/7598043/diff2/31001:88001/state/state_test.go15:49
rogpeppefwereade_: and wondering why we're calling Initialize on the state when all we want is to write the environ config15:49
rogpeppefwereade_: there are quite a few other places that do that too. and Initialize is not in fact called in the usual case in tests!15:49
rogpeppefwereade_: (AFAICS)15:50
rogpeppefwereade_: i'm thinking that we should always call state.Initialize, apart from the few places we actually want to test Initialize.15:51
fwereade_rogpeppe, +1 to that15:51
dimiternfrankban: all tests pass for me @1036 - what's your go version?15:51
rogpeppefwereade_: i'll leave frankban's changes to go in and then we'll fix it in a later branch15:52
fwereade_rogpeppe, cool, thanks15:52
frankbanrogpeppe: cool15:52
frankbandimitern: 1.0.215:52
frankbandimitern: it seems that my wall clock is faster than your, i.e. for some reasons, my local machine is slower15:53
dimiternfrankban: i'm on 1.0.3, but this shouldn't be the problem really.. are you using HDD or SSD drive?15:53
frankbandimitern: hybrid drive15:53
fwereade_rogpeppe, so, the only scenario I can come up with when looking at the code in trunk is that there's some weird stale uniter state somewhere15:54
dimiternfrankban: which timeouts in uniter_test you needed to change?15:54
fwereade_rogpeppe, but I don't see how that's possible either15:54
dimiternfrankban: i'm on ssd, but i think this is not the most common in the team, so also probably not an issue15:54
frankbandimitern: I changed all, just to check that time was the problem there15:55
rogpeppefwereade_: i haven't looked at the code yet, i'm afraid. i'm just running all tests again to see if it happens again.15:55
rogpeppefrankban: LGTM15:55
frankbandimitern: I think Makyo reproduced those 8 failures too, in his local env15:55
frankbanrogpeppe: \o/15:55
rogpeppefrankban: thanks a lot for your patience with this branch!15:56
Makyofrankban, dimitern.  Yes.  I have a log saved, I think.15:56
dimiternfrankban: with changed timeouts how long does it take to run all tests? these are my timings: http://paste.ubuntu.com/5631522/15:56
frankbanrogpeppe: thank you for your help15:56
dimiternMakyo: does it happen with the current trunk tip as well? also timeouts perhaps..15:57
fwereade_TheMue, am I being dense? I can't see what happens when $JUJU_HOME is not set15:58
rogpeppefwereade_: i just saw the same failure again. not in trunk though - perhaps my megawatcher changes are making a difference. that wouldn't surprise me entirely, as we've got the allWatcher that might be affecting things.15:59
dimiternfwereade_: i have some questions about how to properly test upgrade-charm cmd and a wip CL, if you have the time?15:59
fwereade_dimitern, I am very keen to take a look but chat with TheMue takes priority, if he's available16:00
fwereade_dimitern, send me the link and I'll let you know my thoughts16:01
dimiternfwereade_: sure, np - https://codereview.appspot.com/7927043/16:01
Makyodimitern, checking.16:01
TheMuefwereade_: if it's not set it will fallback to $HOME/.juju.16:01
TheMuefwereade_: and if $HOME is also not set it will panic with an according message.16:02
TheMuefwereade_: dimitern asked for a test, i'll see how I can do it. i already tested it by hand and it's working.16:03
dimiternTheMue: PanicMatches checker?16:03
rogpeppedimitern, fwereade_: and another failure: http://paste.ubuntu.com/5631542/16:04
fwereade_TheMue, hold on, we panic if we don;t have a $HOME? why should we need a $HOME to load the config package?16:04
TheMuedimitern: yep, remove both envs before and then call Restore... with the PanicMatches.16:04
rogpeppedimitern: i think the allWatcher must be interfering with the uniter operation somehow, but i'm not sure how.16:04
TheMuefwereade_: if we don't have a home, how do you now where to locate .juju?16:04
dimiternrogpeppe: wow.. I never seen this one - it's falling apart :)16:05
fwereade_TheMue, why do we need to know that in order to manipulate a config file16:05
TheMuefwereade_: we have a lots of places in the code where $HOME is read and taken, even if it is unset. Getenv returns "" then.16:05
fwereade_dimitern, that latest one smells of races and luck in the change you made actually ;p16:06
TheMuefwereade_: where do you read environments.yaml from if $HOME isn't set?16:06
dimiterngents, have to step out for about half an hour, bbiab16:06
fwereade_dimitern, having hit start-failed is not sufficient reason to believe we've run the subsequent upgrade16:06
fwereade_dimitern, ttyl16:06
fwereade_TheMue, but this panics on package load if $HOME is not set, right?16:07
fwereade_TheMue, I don't think that's a reasonable requirement16:07
TheMuefwereade_: yep, but I also can live with a different default for the juju home if $JUJU_HOME and $HOME aren't set. so what do you want JujuHome() to return then?16:08
fwereade_TheMue, JujuHome can panic, that's fine16:09
TheMuefwereade_: so we would panic a bit later. ;)16:09
TheMuefwereade_: when the first one accesses it.16:09
rogpeppedimitern: wow, *everything* is failing now: http://paste.ubuntu.com/5631564/16:10
fwereade_TheMue, right, and if that happens it's usually an error16:10
rogpeppedimitern: and that's without the allWatcher running.16:10
rogpeppedimitern: but this still isn't in trunk.16:10
rogpeppedimitern: so it may well be a problem i've introduced16:10
TheMuefwereade_: so not JH() string but JH() (string, error)? no pro, please write it into the review.16:11
fwereade_TheMue, I'm thinking about it16:11
fwereade_rogpeppe, I'm baffled by the jump into start mode, but that other one seems like a real problem to me16:12
fwereade_TheMue, I think the situation is usually better communicated by a panic than an error -- it generally indicates that one of us is not thinking properly and is calling the wrong code in the wrong context16:13
rogpeppefwereade_: ha, yes, i just saw that failure in trunk16:14
rogpeppedimitern: ^16:14
fwereade_TheMue, but I also think we should have no risk of panic in the actual client: even if they don't have either set for whatever crazy reason, a goroutine trace is not a nice way to say hello16:14
rogpeppedimitern: so i don't think the issue is my fault, so i'm going to submit the allWatcher integration branch16:14
fwereade_TheMue, in addition, even if $HOME is set, that is not enough reason to set JH16:15
benjiI think the tests are leaving un-cleaned-up files in /tmp/.  Is that a known issue?16:15
TheMuefwereade_: IMHO it's a panic situation, but you're right about the info to the user.16:15
rogpeppedimitern: here's the latest failure i've seen, in trunk: http://paste.ubuntu.com/5631582/16:16
fwereade_benji, it is not, they should in general be being cleaned up16:16
benjifwereade_: thanks; I'll verify that it is and file an issue.16:16
fwereade_benji, thanks16:16
TheMuefwereade_: so I could switch from an automatic init() to a manual Init() error that has to be called in commands. and also a panic by JujuHome() if the init hasen't been done.16:17
rogpeppedimitern: it also has the same "[LOG] 15.63869 INFO: worker/uniter: awaiting error resolution for "start" hook" error16:17
fwereade_rogpeppe, I'm sure that's a straight-up bug16:18
rogpeppefwereade_: yeah, i think it must  be16:18
rogpeppefwereade_: i was worried it was provoked by my allWatcher stuff, but i don't think so16:18
fwereade_rogpeppe, well, actually, I don't know what it *is*, it still defies my understanding16:18
rogpeppefwereade_: well, i can reproduce the issue reasonably consistently16:19
fwereade_rogpeppe, are you getting test droppings in your temp dir as well as benji?16:19
rogpeppefwereade_: in /tmp ?16:19
fwereade_rogpeppe, yeah, but I forget where exactly16:20
fwereade_rogpeppe, the code seems very clear that it *has* loaded a state file in a sane format, and that the state file told it what to do16:20
fwereade_rogpeppe, which makes me wonder whether that one is a test isolation issue16:20
fwereade_rogpeppe, although it's strange that it never came up before16:21
rogpeppefwereade_: it seems quite possible. perhaps the uniter isn't being cleaned up synchronously or something?16:21
fwereade_rogpeppe, quite possible, I shall poke at it16:21
rogpeppefwereade_: i don't see any recent test droppings in /tmp16:21
fwereade_rogpeppe, ok, cool, that is good to know16:22
rogpeppefwereade_: you might see if you can reproduce the issue with GOMAXPROCS=5 go test16:22
rogpeppefwereade_: also, i'm running go tip, which has a completely different scheduler now, which might be the trigger16:22
Makyodimitern, current trunk tests for me: http://paste.ubuntu.com/5631605/16:23
fwereade_TheMue, I think so -- set it to panic-on-call by default, and make cmd/juju call something to make it safe16:23
TheMuefwereade_: yep, will do so. you'll note it in the review too, please?16:24
fwereade_TheMue, sure, I'm just soliciting opinions live before I do the review proper16:24
TheMuefwereade_: yeah, sure, I only want to have it noted afterwards, in case I'm not remembering it correctly. I'm getting older. :D16:25
fwereade_TheMue, sent16:38
TheMuefwereade_: great, thank you16:43
TheMuefwereade_: you ask why the juju home is set in the hook context. exactly there, during a hook execution, i had the case of no $JUJU_HOME and no $HOME.16:45
rogpeppefwereade_: you've got a review of the state dev docs branch, BTW16:46
TheMuefwereade_: i sadly don't have the log anymore, but it has been the install hook.16:46
dimiternback16:46
fwereade_TheMue, my point is that I think agent code *should* fail if it even tried to look at JUJU_HOME16:47
fwereade_TheMue, it's not its business at all16:47
fwereade_TheMue, it's no more meaningful than trying to hit the ec2 metadata service from your own laptop16:48
TheMuefwereade_: yeah, the change we discussed will lead to a removal of this setting.16:48
TheMuefwereade_: good catch.16:49
fwereade_rogpeppe, thanks, good comments16:50
dimiternfwereade_: if you still feel like it, take a peek at my CL as well please16:51
fwereade_dimitern, sorry, a load got added to the stack16:52
fwereade_dimitern, fwiw I think I know the problem in the uniter tests16:52
dimiternfwereade_: oh, what is it?16:52
dimiternfwereade_: np about the cmd CL, i'll leave it for tomorrow16:53
fwereade_dimitern, two things; :572 is asserting something it should be waiting for, and the tests aren't properly cleaning up after themselves16:53
fwereade_dimitern, there's a RemoveAll(s.unitDir) that is not run quite as often as it should be if tests fail16:54
fwereade_dimitern, if I check your review would you see if you can repro rog's failures against go tip?16:54
dimiternfwereade_: sorry, i'm confused now16:56
dimiternfwereade_: the 2 things are about uniter failures?16:57
fwereade_dimitern, yeah, rog sent a number of pastes while you were away16:57
dimiternfwereade_: these were test execution dumps, not code right?16:58
fwereade_dimitern, yeah16:58
dimiternfwereade_: and :572 is where?16:58
fwereade_dimitern, line 572, the second forced upgrade error test iirc16:58
dimiternfwereade_: let me check16:59
fwereade_dimitern, it's a verifyCharm{1} that depends on unwarranted assumptions16:59
fwereade_dimitern, then when that fails it takes out all subsequent tests because it doesn't clean up16:59
dimiternfwereade_: is this the possible case of tests hanging on failure sometimes?17:00
dimitern*cause17:00
=== deryck is now known as deryck[lunch]
dimiternfwereade_: right! the RemoveAll in runUniterTests?17:01
* TheMue is AFK for some time, bbl17:01
fwereade_dimitern, hanging forever is unrelated, is there a bug for that?17:01
fwereade_dimitern, yeah17:01
dimiternfwereade_: don't think so - but I've only seen it twice over a month probably and it was always when some other test fails first17:02
dimiternfwereade_: so that RemoveAll should be deferred as well probably17:03
fwereade_dimitern, can't remember what happens if RemoveAll fails17:04
fwereade_dimitern, but yeah, I think that would od it17:04
dimiternfwereade_: I *cannot* reproduce any of these failures against tip - tried 5 or 6 times already17:04
fwereade_dimitern, hum, that is annoying17:04
dimiternfwereade_: trying again now17:05
fwereade_dimitern, quick hangout re upgrade-charm when you're free? o maybe over a beer later?17:15
dimiternfwereade_: +1 for beer17:15
dimiternfwereade_: still no joy - all tests pass17:15
fwereade_ok: dimitern, rogpeppe, I need to visit the shops with some haste17:16
fwereade_dimitern, if you make a test fail deliberately you can verify the effect of a deferred RemoveAll; if that's shown to work it'll get a swift LGTM on its own when I return17:17
fwereade_dimitern, rogpeppe: but in the large I think this is a situation for skip-the-test, add-critical-bug17:18
rogpeppefwereade_: sounds reasonable17:19
fwereade_dimitern, rogpeppe: I'm pretty sure I know what's going on but it won;t be fixed tonight and I don't *think* in this case that it is better to back out the change17:19
fwereade_ttyl all17:19
rogpeppedimitern: have you tried running against go tip, with GOMAXPROCS > n ?17:20
dimiterni'll add the quickfix for RemoveAll and try17:20
dimiternrogpeppe: no17:22
rogpeppedimitern: that's where i see the problem (the go tip scheduler is different)17:22
dimiternrogpeppe: i have go tip at hand, but haven't updated recently17:22
rogpeppedimitern: "hg sync; cd src; ./all.bash" and bob's yer uncle17:22
dimiternrogpeppe: hg pull you mean?17:23
rogpeppedimitern: yeah, same thing. i think "sync" is a synonym they introduced that also copes with CL changes.17:23
rogpeppedimitern: unless you're contributing to the go project, there's no need to use it.17:23
dimiternrogpeppe: haven't seen it, it might be a plugin (reports an error here)17:24
rogpeppedimitern: yeah, it's a plugin17:24
rogpeppedimitern: just use hg pull; hg update tip instead17:24
dimiternrogpeppe: done, building now17:24
dimiternrogpeppe: what's n? ^^ 2 or more17:26
rogpeppedimitern: i used 5, one more than the number of cores in my machine17:26
dimiternrogpeppe: ok17:27
dimiternrogpeppe: how do i run it with 5 ?17:27
rogpeppedimitern: GOMAXPROCS=5 go test17:28
dimiternrogpeppe: running all tests now, first without GOMAXPROCS17:29
rogpeppedimitern: i hope you've changed GOROOT and GOPATH17:30
rogpeppedimitern: otherwise you might be running the wrong binaries17:30
dimiternrogpeppe: ofc, even run go version: go version devel +8cc853b84f89 Wed Mar 20 09:06:33 2013 -0700 linux/amd6417:30
rogpeppedimitern: cool17:31
rogpeppedimitern: i have a separate GOPATH dir for the alternative go compiler, so there's no chance of confusion17:31
dimiternrogpeppe: actually, do I have to wipe out pkg/* as well as changing GOROOT?17:31
rogpeppedimitern: probably not, as everything will have changed, but in general yes17:31
rogpeppedimitern: that's why i have a separate copy of everything in its own GOPATH17:32
dimiternrogpeppe: so you have 2 separate bzr working trees?17:33
rogpeppedimitern: yup17:33
rogpeppedimitern: and a command (go-alt-pull) that pulls the current branch in my usual working dir into the alternative branch17:33
dimiternrogpeppe: with a corresponding magical shortcut, beginning with ",x.." ? :)17:34
rogpeppedimitern: (most of the time i work with go tip, but i test against 1.0.2 before submitting17:34
rogpeppe)17:34
dimiternall tests pass with trunk/tip and go/tip, now will try GOMAXPROCS=517:34
rogpeppedimitern: nah, i just type go-alt-pull17:34
rogpeppe:-)17:34
rogpeppedimitern: ,x just edits.17:35
dimiternrogpeppe: ahaa17:35
rogpeppedimitern: ("," is short for "0,$" and "x" says do something for everything matching the following regexp)17:36
dimiternrogpeppe: apart from some weird warnings in decode/cgo something no other issues17:36
rogpeppedimitern: hmm.17:36
rogpeppedimitern: try it a few times :-)17:36
dimiterni have some failures now17:37
rogpeppedimitern: cool17:38
rogpeppedimitern: the same ones?17:38
dimiternin rpc though http://paste.ubuntu.com/5631798/17:38
dimiternfwereade_: riak testing charm surprisingly has rev 717:39
dimiternrogpeppe: nope, no errors, will run it a couple of times again17:39
dimiternrogpeppe: (apart from the rpc panic)17:39
rogpeppedimitern: ah, i think i know why that happens, and it should be fixed by dfc's upcoming branch17:40
dimiternrogpeppe: the logging stuff?17:40
dimiternrogpeppe: on second run (n=5) rpc passed17:41
rogpeppedimitern: yeah, the race setting log.Target17:41
dimiternrogpeppe: it failed now, but with a different issue (SIGSEGV) http://paste.ubuntu.com/5631814/17:43
rogpeppedimitern: hmm, i've not seen that befoe17:45
rogpeppere17:45
dimiternrogpeppe: how often do the uniter tests fail like that for you?17:45
rogpeppedimitern: most times17:45
rogpeppedimitern: i can reproduce by just running the uniter tests too17:47
dimiternrogpeppe: rpc failed again (the same issue), but the uniter tests passed (i noticed with n=5 they are slightly faster - about 10-15%)17:47
dimiternrunning again without n=517:48
fwereade_rogpeppe, does my sketchy judgment that it looks like (1) a genuinely flaky test that fails in verifyCharm, and (2) poor isolationcausing all subsequent test cases to fail17:48
fwereade_rogpeppe, ...match your own observations?17:48
dimiternfwereade_: imo 2 is definitely happening17:48
rogpeppefwereade_: i haven't been delving into it as much as dimitern17:48
dimiternfwereade_: alas, not for me17:48
dimiternthat segfault is troubling though17:49
rogpeppefwereade_: (i figured both you and dimitern are closer to the problem than me, and will likely find the fix much sooner)17:49
fwereade_dimitern, while I have concern about the things you're observing I am focused pretty hard on the uniter tests17:49
fwereade_rogpeppe, ofc -- that was just something tossed into the air in case it were to get shot down with an instant "well no I've also seen X, Y, Z" :)17:50
fwereade_dimitern, agreed17:50
fwereade_dimitern, 8ish @cuba?17:50
dimiternwow, new panic now (second run, n=1 go tip): http://paste.ubuntu.com/5631831/17:50
dimiternfwereade_: sgtm17:51
dimiternI've seen this before once17:51
dimiternand it's a pass (except the panic)17:52
rogpeppefwereade_: looking at the errors, yes, i think your analysis seems likely17:52
dimiternfwereade_: so my thoughts about the cleanup: there are 2 approaches - extract the loop body of runUniterTests and add a defer RemoveAll there; OR just add an additional defer RemoveAll in runUniterTests17:53
rogpeppedimitern: hmm, i can't see how that could happen17:55
dimiternrogpeppe: what?17:55
rogpeppedimitern: the panic in randomHexToken17:55
dimiternrogpeppe: today's the day for obscure test failures it seems :)17:56
rogpeppedimitern: unless... what revision of goose are you using?17:56
dimiternmaybe it's biorhythms or full moon or smth17:57
dimiternrogpeppe: haven't updated recently, that could be it, lemme check17:57
rogpeppedimitern: yup, that's it17:57
rogpeppedimitern: if you're before r80, that's the reason17:57
dimiternrogpeppe: i was, now @80, running again with n=5/tip17:58
dimiternnow the uniter tests failed..goroutine 4574  wtf?!18:02
dimiternhttp://paste.ubuntu.com/5631858/18:02
dimiternmaybe mgo has hard time coping with a lot of concurrent procs?18:03
fwereade_dimitern, whoa, that definitely deserves a bug, probably against mgo... niemeyer? does a theory leap out at you?18:04
fwereade_actually, I have to be off for supper18:05
dimiternfwereade_: so see you around 8 @cuba then18:06
fwereade_rogpeppe, would you skip that  test and add a critical bug/card for it please?18:07
rogpeppefwereade_: the uniter test?18:07
fwereade_rogpeppe, if nobody's around to approve it I pre-LGTM a single line that just skips the one that fails for you :)18:07
fwereade_rogpeppe, please, if you have time18:07
rogpeppefwereade_: ok, will do18:07
fwereade_rogpeppe, awesome, tyvm18:07
fwereade_later all18:08
dimiterni'll call it a day as well18:13
dimiternnight y'all18:13
rogpeppedimitern: g'night18:14
hazmatseems odd that a co of juju-core gets .   submit branch: /home/rog/src/go/src/launchpad.net/juju-core/juju/.bzr/cobzr/go-state-units-under-service18:21
=== deryck[lunch] is now known as deryck
rogpeppehazmat: that is really weird18:39
rogpeppehazmat: i think that branch dates from when i did see some weirdness around there18:39
hazmatrogpeppe, just some cached metadata in the branch18:39
hazmatnot an issue, just odd18:39
rogpeppehazmat: it may have happened when i did a push --overwrite, possibly18:40
rogpeppeanyway, eod here18:40
hazmatrogpeppe, cheers18:40
rogpeppehazmat: cheers to you too18:40
rogpeppeand g'night all others18:40
niemeyerfwereade_: I can't see how there could be a nil reference in that line.. mongoServers is part of the cluster by value20:09
niemeyerfwereade_: Do you know what Go version this is running on?20:10
thumpermoring folks20:22
* thumper does the conflict resolution dance again21:03
robbiewlol21:11
* thumper sighs21:23
thumpernever just as simple as resolving conflicts21:23
* thumper fixes test failures, import errors and other boring bits21:23
* thumper wishes go had a build_tests21:26
thumperso I could check all the tests built before running them...21:26
mrammthat would be nice21:28
davecheneythumper: any final comments on https://codereview.appspot.com/752404622:35
davecheneyi don't think this bike shed will stand up to any more coats of paint22:35
* thumper looks22:37
thumperdavecheney: morning22:37
davecheneymornig22:39
davecheneygutter cleaners are here today in the complex22:39
davecheneyand they have bought the australia strategic reserve of petrol powered leaf blowers22:39
thumperdavecheney: defer log.SetTarget(log.SetTarget(c))22:40
thumperdoes the defer command evaluate all the args at the call time?22:40
thumperdavecheney: I'm assuming that log.SetTarget(c) is evaluated prior to the defer stashing the method away for later22:41
davecheneythumper: yes22:41
davecheneyso the inner log.SetTarget(c) sets the target to c and reutrns the previous value22:42
davecheneythat value is captured by the defer and passed to the outer log.SetTarget when run22:42
thumperdavecheney: got time for a hangout chat?22:46
davecheneysure22:51
davecheneylemmie go upstairs22:51
thumperok22:51
=== wedgwood is now known as wedgwood_away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!