/srv/irclogs.ubuntu.com/2013/09/10/#juju-dev.txt

davecheneywallyworld_: why is bootstrap doing sync tools ?00:01
davecheneythis is ec200:01
wallyworld_not sure. it should only do that if it can't find any. i'll take a look00:01
wallyworld_davecheney: well, it was looking for 1.1500:02
wallyworld_which it couldn't find00:02
wallyworld_so you need --upload-tools00:02
davecheneysure, but 1.12 exists in the s3 bucket00:02
wallyworld_otherwise it will sync00:02
wallyworld_but different minor version00:03
wallyworld_12 = 1500:03
wallyworld_!=00:03
davecheneywallyworld_: is that important00:03
davecheney1.12.0-precise-amd64 tools exist in the public s3 bucket00:03
davecheneythis is an ec2 environment00:03
davecheneythis is a regression00:03
wallyworld_no - we can't guarantee 12 is compatible with 1500:04
wallyworld_we talked about this remember?00:04
wallyworld_you asked the change be held till after 1.13 was released00:04
davecheneyyes, we talked00:04
davecheneybut i don't see how this is related00:04
davecheneyi am using 1.15 client00:04
wallyworld_you are bootstrapping using juju 1.15 right?00:04
davecheney1.12 tools exist in the public bucket00:05
davecheneywhy is bootstrap syncing the 1.12 tools00:05
davecheneyshouldn't it refuse to run saying there are no 1.15 tools ?00:05
wallyworld_sync tools is dumb00:05
wallyworld_it needs work00:05
davecheneyshould I raise a bug ?00:05
wallyworld_it is doing what it has always done and grab whatever tools it can find - the algorithm needs work00:05
davecheneythe problem is that folks in CTS will be using the devel version and they won't understand why their bootstrap times on ec2 have blown out00:06
wallyworld_sure, raise a bug - it's being worked on as we speak00:06
wallyworld_as part of the tools work00:06
davecheneywallyworld_: i thought the change was exact match or TFO00:06
wallyworld_TFO?00:06
wallyworld_bootstrap requires an exact match, but if it can't find one, it does a sync tools. and sync tools needs some love. it's all in progress00:08
wallyworld_i guess the expectation was that people running from source ie dev version, should always know to do an upload-tools00:09
wallyworld_upload-tools = problem solved00:09
davecheney/s/FTO/GTFO00:10
davecheneyok, upload-tools it is00:11
wallyworld_sorry about that - upload tools is a consequence of exact version match00:12
wallyworld_i think it should do it automatically for dev versions00:12
wallyworld_with an appropriate message00:12
davecheneywallyworld_: 2013-09-10 00:15:00 INFO juju.environs.tools tools.go:131 filtering tools by series: precise00:17
davecheney2013-09-10 00:15:00 INFO juju.environs.tools tools.go:53 no architecture specified when finding tools, looking for any00:17
davecheney^ why does it say that00:17
davecheneyamd64 is the default00:17
wallyworld_davecheney: the tools finding code it calling "findtools" without specifying an architecture - that's how it's always been. but now, with the simplestreams metadata, tools info is keyed of arch and series, so it logs that message00:18
wallyworld_so the code which calls FindTools needs to be looked at to see if it can get access to an arch to pass in00:19
wallyworld_davecheney: where is the default arch specified - i can't recall right now00:19
wallyworld_and if amd64 is not found, doesn't it default back to i386?00:20
wallyworld_in which case passing in nothing for arch is ok? since it will look for tools matching any arch?00:20
wallyworld_bigjools: ffs. gwacl panics when asked for the url of a non-existent file :-(00:23
davecheneywallyworld_: dunno, sounds wrong00:24
davecheneythe default is precise/amd6400:24
davecheneyand the tools lookup clearly shows it is scoping by precise00:24
bigjoolswallyworld_: \o/00:24
davecheneyit shold also scope by amd6400:24
wallyworld_davecheney: probs. but i'm not sure how arm or i386 would be specified00:25
bigjoolswallyworld_: I hear you're an expert in Go, you could fix it!00:25
wallyworld_bigjools: have i told you today?00:25
bigjoolslol00:25
bigjoolsactual lol00:25
davecheneywallyworld_: ok00:25
davecheneygood point00:25
wallyworld_bigjools: can you add me the the gwacl-hackers team, and +1 this? https://code.launchpad.net/~wallyworld/gwacl/storage-error-not-panic/+merge/18470401:06
bigjoolswallyworld_: yes yes01:06
wallyworld_bigjools: thanks, and we use the bot is assume?01:06
wallyworld_ie just set to approved and the bot will work01:06
bigjoolswallyworld_: I hope so.01:06
wallyworld_we'll find out :-)01:07
bigjoolswallyworld_: it requires one +1 review01:07
wallyworld_juju-core does now too01:07
wallyworld_for a little while01:07
bigjoolswallyworld_: you need to run " make format"01:10
wallyworld_bigjools: i ran go fmt and it fucked everything01:10
wallyworld_i had to revert and start again :-(01:10
bigjoolswallyworld_: yes, use make format :)01:10
wallyworld_WHY???01:10
wallyworld_what's wrong with go fmt01:10
bigjoolsbecause01:10
bigjoolsbecause tabs are fucking evil01:10
davecheneythat is the problem with standards; everyone wants their own01:10
bigjoolsthere are so many to choose from01:11
wallyworld_yeah agree. wish i had known that01:11
davecheney... the Aristocrats!01:11
wallyworld_before if*cked myself01:11
bigjoolsbzr has this really useful "commit" thing01:11
wallyworld_bigjools: yes, but one does not expect to have to run it before doing a go fmt01:12
wallyworld_changes pushed01:12
bigjoolswallyworld_: I  never ever trust tools01:13
bigjoolsever01:13
wallyworld_bigjools: that's why i don't trust you01:13
bigjoolsand he's here ALL WEEK01:13
wallyworld_yep :-D01:13
bigjoolswallyworld_: anyway I already told you once we used 4 space indents in gwacl and you said that was awesome01:17
wallyworld_bigjools: and you expect me to remember that after many bottles of red have passed under the bridge?01:18
wallyworld_and months of "go fmt" muscle memory01:18
bigjoolshahaha01:18
bigjoolswe left our present01:18
davecheneybrb, someone just rang my doorbell01:19
davecheneyhey wait, why is that bag on fire01:20
bigjoolssnork01:20
thumpero/01:23
* thumper makes a smoothie for lunch01:23
axwdavecheney: you wanna LGTM this?   https://codereview.appspot.com/13620043/01:27
davecheneyI wanna01:30
davecheneydone01:30
davecheneyLGTM01:30
davecheneyfire when ready01:30
axwmany thanks01:31
davecheneythank you sir01:32
thumperwallyworld_: ping02:13
wallyworld_hi02:13
thumperwallyworld_: got a few minutes for a hangout?02:14
wallyworld_thumper: sure, just give me 502:14
wallyworld_thumper: ok, free now02:20
thumperok02:20
thumperwallyworld_: https://plus.google.com/hangouts/_/6b1e2ea0df710aa62ee610909253df3e89de4c9b?hl=en02:21
axw_somehow my branch revision just disappeared :(03:21
* axw_ starts the null provider again03:21
thumper?!03:23
thumperaxw_: disappeared how?03:23
axw_nfi.03:23
axw_I committed03:23
axw_then did a go build03:23
axw_then my files are all gone, and the revision is back where it was before03:23
axw_fortunately all the hard stuff is already uploaded03:24
thumperweird03:33
=== axw__ is now known as axw
thumperaxw: ing04:22
thumperp04:22
axwthumper: pong04:24
thumperaxw: are you backporting bug 1222664 to the 1.14 branch?04:24
_mup_Bug #1222664: maas provider's instance is not a Stringer <juju-core:In Progress by axwalk> <juju-core 1.14:In Progress by axwalk> <juju-core trunk:In Progress by axwalk> <https://launchpad.net/bugs/1222664>04:24
axwnot this very moment, but I will04:24
thumperif you are busy04:24
thumperI can do it04:24
thumperwon't take long at all04:24
thumperthen you can ack it04:24
axwif you don't mind, trying to unbugger my stream04:25
thumperkk04:25
thumperwill do, right now04:25
axwthanks04:25
jamhappy birthday thumper! Today is also my son's birthday.04:27
thumperjam: thanks, and happy birthday to your son :)04:28
axwouted! happy birthday thumper04:28
davecheneyyou sly bugger04:29
thumperaxw, davecheney: https://code.launchpad.net/~thumper/juju-core/backport-bug-1222664-to-1.14/+merge/184718 I've kept it out of lbox for simplicity04:33
axwthumper: approved, thank you04:33
thumpernp04:33
* thumper looks at the last issue04:34
thumperdavecheney: ping04:44
davecheneythumper: looking04:44
thumperdavecheney: I don't need you to look04:44
thumperI need to talk to you04:44
davecheneyack04:44
thumperdavecheney: this one with the race to start04:44
davecheneyyup04:44
thumperyou talk about the provisioner, but I don't see that in the logs04:44
thumperdo you mean the state connection?04:44
davecheneythis is the machiner on machine 004:45
davecheneywhich has the provisioner worker04:45
davecheneymy terminilogy is old04:45
thumperyou mean the machine agent04:45
thumperthe provisioner is a task, as is the machiner04:45
* thumper thinks how to solve this sanely04:46
davecheneythumper: the agent ?04:46
davecheneyis trying to conncet to a service that it hosts itself04:46
davecheneyhence, a gordian knot04:46
* thumper nods04:46
thumperI think I have a plan04:47
* thumper tries it04:47
davecheneyi guess that plan doesn't include making the api server a different process ?04:47
davecheneyhttps://bugs.launchpad.net/juju-core/+bug/121832904:48
_mup_Bug #1218329: Update default environment.yaml for Azure to use Precise for default-series <juju-core:Fix Committed by axwalk> <juju-core 1.14:Fix Committed by axwalk> <https://launchpad.net/bugs/1218329>04:48
davecheney^ is this really fix committed on 1.14 ?04:48
davecheneyi see no branch04:48
axwI didn't branch04:49
thumperdavecheney: yes it is there04:49
thumperI checked the last commit04:49
thumperI'll help axw understand the process later04:49
thumperwhen I'm done04:49
davecheneyok04:49
davecheneycool04:49
thumperperhaps we should write up a backporting howto04:49
davecheneyjust checking04:49
thumperand shove it in the tree04:49
thumperdavecheney: I see the problem with the machine agent04:49
* thumper thinks04:49
jamdavecheney: bug #1218329 was landed, let me see if I can dig up the branch04:52
_mup_Bug #1218329: Update default environment.yaml for Azure to use Precise for default-series <juju-core:Fix Committed by axwalk> <juju-core 1.14:Fix Committed by axwalk> <https://launchpad.net/bugs/1218329>04:52
davecheneyjam: i can see the commit, it'll do04:53
jamdavecheney: https://code.launchpad.net/~axwalk/juju-core/lp1218329-azure-released-images/+merge/18338104:53
jamit is pretty small :)04:53
davecheneyjam: that is not the commit04:54
davecheneythat is the trunk commit04:54
jamdavecheney: I don't quite understand your "that is not the commit". The specific fix for bug 1218329 is just to change those 3 lines.04:55
_mup_Bug #1218329: Update default environment.yaml for Azure to use Precise for default-series <juju-core:Fix Committed by axwalk> <juju-core 1.14:Fix Committed by axwalk> <https://launchpad.net/bugs/1218329>04:55
jamit is about having a good "juju init" value for azure, which was delayed until we had proper images upload to azure04:56
davecheneyjam: i am lookin for the mp for the merge onto the 1.14 branch04:57
jamdavecheney: it hasn't ever landed on 1.1404:58
jamFix Committed is landed-in-trunk04:58
axwjam: it's there, I just didn't do whatever the normal procedure is04:59
davecheneyjam: ok, i'm trying to get to the bottom of why it says '<juju-core 1.14:Fix  Committed by axwalk>04:59
jamdavecheney: http://bazaar.launchpad.net/~juju/juju-core/1.14/revision/173804:59
davecheneythank you04:59
davecheneyi cannot find a mp for that05:00
davecheneyso i cannot link it to the issue05:00
jamdavecheney: there isn't one05:00
davecheneythat was my problem05:00
jamaccording to axw05:00
* axw nods05:00
jamone problem with making the series branches not owned by the bot, is that people end up with direct commit access.05:00
davecheneyjam: how can we fix that ?05:01
jamdavecheney: we can change the owner and I can add the bot to controlling that branch05:01
davecheneyexcellent, sgtm05:01
jamthumper: is there a way to change the owner of a branch to a person where you aren't the direct owner?05:03
jamdavecheney: worst case, I create a bot branch, and just change what "lp:juju-core/1.14" points to05:03
davecheneysgtm05:03
thumperjam: not sure I understand05:04
jambut right now, I only have groups *I'm* in as potential owners, and the bot is intentionally not in ~juju so he can't touch things he hasn't been given direct access to.05:04
jamthumper: I want the go-bot to own an existing branch05:04
thumperand the go-bot is a person?05:04
jambut he isn't in ~juju, and I'm not the bot if I'm ~jameinel05:04
jamthumper: yes05:04
thumperjam: you need to get an lp-admin to do that05:04
thumperyou can't05:04
jamthumper: then I'll just create another branch05:04
jamyou *used* to be able to hand branches to people05:04
thumperhowever05:04
jambut I think that was considered a csecurity hole05:04
thumperyou can pass it through an intermediate team05:04
thumperif you are both a member of the same team05:04
thumperyou change it to the team05:05
thumperthey change from the team to themselves05:05
thumperyes, it was a security hole05:05
thumperwe closed it05:05
jamthumper: I thought the plan was to just have confirmation by both parties05:06
jamso someone gives it to you, but you have to 'accept' it.05:06
jambut meh05:06
thumpernah05:06
thumpertoo much work05:06
thumperdavecheney: how do I ask if a channel is closed?05:14
davecheneythumper: you cannot05:14
davecheneywe normally use a tomb to provide that05:14
thumperselect on a closed channel succeeds immediately yes?05:15
davecheneyyes05:18
davecheneyselect {05:18
davecheneycase c, ok := <- ch: ; if !ok // closed05:18
davecheneyyou can also do05:18
davecheneyc, ok := <- ch outside select and it will block until ch is closed05:19
davecheneybut there is no isClosed(ch) builtin05:19
thumperc, ok := <- ch will return if closed or something on the channel, right?05:20
davecheneyy05:20
jamdavecheney: https://code.launchpad.net/juju-core/1.14 is now pointing at ~go-bot/juju-core/1.1405:27
jamthumper: I deleted the ~juju/juju-core/1.14 branch, so it deleted your MP against it. But I really didn't want to have 2 1.14 branches that might cause confusion05:28
thumperok05:28
thumperit had landed05:28
jamthumper: yeah, if you want it for posterity, you can submit it against the new branch and mark it merged :)05:28
thumpernah, don't care05:29
jamdavecheney: so the change from ian was explicitly requested from fwereade_ about "juju bootstrap" will only support exact Major.Minor versions.05:33
jamso for dev releases you have to use '--upload-tools'05:33
jambecause of the chance that we break bootstrap between minor versions (which we've done)05:34
jamdavecheney: I'm happy to have you in the conversation (I'm more on the "lets make it easier to be cross version compatible, rather than stricter"). And it doesn't make a *huge* difference for finally released things.05:34
jambut it does mean you can't use dev to bootstrap stable05:35
jammgz: when you're up and around, I need to get some info from you to control the go-bot config. (You can't access the root bot with just the admin password, you have to have the cert.pem files as well)05:41
jammgz: also, I can't just test CACerts by renaming my own certs, because what we need to test is the bootstrap on the remote machine and "wget" *there* not on my local machine.05:41
jam(I can set up a test-suite HTTPS server with self-signed certs pretty easily, so I have 'local' testing done)05:42
* thumper gives up on subtlety05:51
* thumper adds a sleep05:58
jamthumper: how did you arrive at 3s ?06:02
thumper:)06:02
jamwould it be easier to just retry 1 time to connect rather than sleeping?06:02
jamas in, try to connect for 3s06:02
jamrather than sleep06:02
thumperhave you looked at the code..06:02
thumperhmm...06:03
thumperperhaps06:04
jamthumper: line 14706:04
thumperhonestly, I didn't think of it from that side :)06:04
jamwhen it says "failed to connect" error, just retry 1 time, or retry for 3s or whatever06:04
thumperand EOD now06:04
thumperan exercise for the reader06:04
rogpeppe1mornin' all07:50
axwmorning07:51
TheMuemorning07:51
TheMuethumper not around anymore? hmm, ok, have to congratulate him later.07:52
rogpeppe1jam: ping08:00
rogpeppe1axw: do you know anything about the background for this, by any chance? https://codereview.appspot.com/1341204708:03
axwrogpeppe1: I think there's a race between the API server starting up, and something trying to connect to it; the connection fails, and brings down the process08:04
axwor something like that.08:04
rogpeppe1axw: that's what the comment in the CL seems to imply, but i can't quite see how it can actually happen08:04
* axw looks08:04
rogpeppe1axw: AFAICS if the API worker fails to connect to the API, it will finish with a non-fatal error, and be restarted after a little while by the top level runner08:05
rogpeppe1axw: that's how i *intended* it to work anyway :-)08:06
jamhey rogpeppe108:08
rogpeppe1jam: hiya08:08
rogpeppe1jam: see my question to axw above08:08
jamrogpeppe1: right now, when the APIWorker fails it restarts everything08:08
jpdshttps://bugs.launchpad.net/ubuntu/+source/charm-tools/+bug/1223225 - this needs fixing.08:08
jamthere is a bug on it08:08
_mup_Bug #1223225: charm-tools needs to stop recommending juju <charm-tools (Ubuntu):New> <https://launchpad.net/bugs/1223225>08:08
rogpeppe1jam: really?08:09
rogpeppe1jam: if the connection to the API fails, it should not restart everything08:09
jamrogpeppe1: bug #122002708:09
_mup_Bug #1220027: worker/provisioner: cannot restart cleanly due to hard dependency on api server <papercut> <juju-core:In Progress by thumper> <juju-core 1.14:In Progress by thumper> <juju-core trunk:In Progress by thumper> <https://launchpad.net/bugs/1220027>08:09
rogpeppe1jam: the top level runner does not use allFatal08:09
jamrogpeppe1: the openAPIAs code has DialTimeout of 008:09
jamso it always restarts08:09
rogpeppe1jam: that should be fine, because the top level runner does *not* exit when one of its tasks exits08:10
rogpeppe1jam: so the APIWorker task should be restarted until the API server is available, no?08:10
jamrogpeppe1: the issue is the "provisioner" is triggering restarts because it can't connect to the API08:11
jamthat is what the bug report claims08:11
rogpeppe1jam: why is the provisioner trying to connect to the API?08:11
jamrogpeppe1: I haven't debugged this08:11
jamI'm just conveying the context I have so far08:12
rogpeppe1jam: shouldn't it be using the API connection that is opened by openAPIState?08:12
rogpeppe1jam: sure, thanks08:12
axwrogpeppe1: I see in worker.Runner.run code that does check isFatal08:12
davecheneyrogpeppe1: state.OpenState (something) returns both an api and state connection08:12
axwam I looking at the right thing?08:12
davecheneyeverything needs a connection to the api08:12
davecheneymaybe also the api08:12
jamaxw: isFatal is slightly-less than allFatal08:12
axwjam: well, it then goes and calls killAll(workers)08:12
rogpeppe1davecheney: yeah, the API also (initially) needs a connection to the API, except on the bootstrap node, obviously08:12
rogpeppe1axw: see the definition of isFatal in jujud/agent.go for which errors are considered fatal at the top level08:13
davecheneyrogpeppe1: this creapt in late in the 1.11.x cycle08:13
davecheneyit is present in 1.12.008:14
rogpeppe1davecheney: what crept in, sorry?08:14
jamrogpeppe1: the race condition triggering restarts08:14
axwrogpeppe1: ah sorry, I misread your statement before - I thought you said it didn't call isFatal08:14
jammgz: poke08:15
jammgz08:15
rogpeppe1jam, davecheney: i'm not sure that https://bugs.launchpad.net/juju-core/+bug/1220027 is a bug at all08:15
_mup_Bug #1220027: worker/provisioner: cannot restart cleanly due to hard dependency on api server <papercut> <juju-core:In Progress by thumper> <juju-core 1.14:In Progress by thumper> <juju-core trunk:In Progress by thumper> <https://launchpad.net/bugs/1220027>08:15
rogpeppe1jam, davecheney: i *think* it's working as intended08:16
jamIn case you are actually around now:08:16
jam(9:41:32) jam: mgz: when you're up and around, I need to get some info from you to control the go-bot config. (You can't access the root bot with just the admin password, you have to have the cert.pem files as well)08:16
jam(9:42:00) jam: mgz: also, I can't just test CACerts by renaming my own certs, because what we need to test is the bootstrap on the remote machine and "wget" *there* not on my local machine.08:16
jamrogpeppe1: "but it causes extended delays" sounds like it is more than a 3s08:17
jamdelay08:17
rogpeppe1jam: it does, yes. but i don't see more than a 3 second delay in the bug report08:17
rogpeppe1davecheney: you've marked the bug as "papercut" - could you explain a bit more about why it's a particular problem?08:18
jamrogpeppe1: so davecheney ran into this at someone's site (hence the papercut issue) so I'm guessing there could be more background. I agree that the particular log snippet only takes 3s from start to finish, but note the first line is "restarting "state"08:18
jamrogpeppe1: I think the first line is pretty key08:18
jamsomething caused the "state" worker to restart08:19
jam1:46:09 is restarting state in 3s, then 1:46:11 is restarting api in 3s, then 1:46:12 is starting "state" again.08:19
davecheneyrogpeppe1: papercut is (one of the many) bugs used by various parts of the company to indicate that this bug affects customers08:19
davecheneythe name is the least describtive, but comes from SABFDL08:20
rogpeppe1davecheney: yeah, i'm aware of the name - i just wondered how this was affecting customers08:20
davecheneyrogpeppe1: the process will eventually start up correctly, as the orer of job manage environ jobs is not specified08:20
jamTheMue: as you're on call today, can you look at my goose branches? https://codereview.appspot.com/13379047/  and https://codereview.appspot.com/13396048/08:20
rogpeppe1jam: yeah, that's odd, and something which probably isn't addressed by the proposed fix AFAICS08:21
TheMuejam: sure08:21
jamTheMue: thanks08:21
rogpeppe1davecheney: have you got the full log from when the problem happens?08:21
jamdavecheney: I think what rogpeppe1 is getting at, is that the bouncing provisioner might be a symptom rather than a cause.08:21
jamhowever rogpeppe1: if the provisioner fails to start, won't it bounce the "state" worker,08:21
jamcausing the API server to bounce08:21
jamcausing them to get into a restart dance?08:22
jamso the issue is that something that *isn't* being run by the APIWorker is depending on the API, which then kills the API servere08:22
jamserver08:22
jamso it can't start itself.08:22
TheMuejam: could you please take a look at https://codereview.appspot.com/13522043/ for me? need a lgtm here too. ;)08:22
rogpeppe1jam: ah, that seems wrong08:23
rogpeppe1jam: i thought the provisioner only talked to state08:23
davecheneyrogpeppe1: you can generate one for yourself08:23
davecheneyjust deploy a few services on ec208:23
davecheneyand watch the machine-0 log08:24
rogpeppe1davecheney: it happens every time you deploy a service?08:24
jamTheMue: so that one is "support empty strings locally as meaning "set this to the empty strings", but preserve the API path which means "empty strings ==> default value" ?08:24
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: TheMue | Bugs: 9 Critical, 122 High - https://bugs.launchpad.net/juju-core/
TheMuejam: yep08:25
davecheneyrogpeppe1: pretty reliably at the moment08:26
rogpeppe1davecheney: hmm, weird - i will take a deeper look.08:27
davecheneyrogpeppe1: ta08:27
TheMuejam: both reviewed08:30
jamfwereade_: rogpeppe1, TheMue: I'm not going to make the standup today (it is my son's 6th b-day party at school). so you can make dimitern crack the whip if you want.08:34
TheMuejam: ok08:34
rogpeppe1jam: ok, have sweet-cake fun :-)08:34
TheMuejam: and enjoy the b-day08:34
axwahasenack: ping08:39
ahasenackaxw: hi08:39
axwahasenack: hi. stupid question - how did the /etc/init/juju* files get into your clean container?08:39
ahasenackaxw: oh, that is run inside the container?08:39
ahasenackaxw: I ran that on my host08:39
axwahh08:39
axwyes, on the container08:40
ahasenackI'll have to run again then08:40
ahasenackaxw: so my env is bootstrapped using lxc08:40
axwahasenack: ok, that'll be a problem. you can't add-machine an already bootstrapped machine08:40
ahasenackaxw: I will bring up a new container with lxc-create and lxc-start, one that was never touched by juju08:40
axwcool08:40
ahasenackaxw: should that work?08:40
ahasenackon my laptop:08:40
ahasenackjuju bootstrap (lxc)08:40
TheMuejam: thx for the review, will add some tests08:41
ahasenacklxc-create && lxc-start08:41
ahasenackthen juju add-machine ssh:that-new-container-I-created-manually08:41
axwahasenack: yes if the container is clean, it should work08:41
ahasenackok, that's what I thought I did, but let me do it carefully again08:41
axwahasenack: thanks. I'm doing the same now08:42
ahasenackaxw: got the same error08:50
ahasenackaxw: inside the container there is no /etc/init/juju*08:51
ahasenackaxw: I created the container fresh, precise08:51
ahasenackstarted it08:51
ahasenackjuju bootstrapped (lxc), juju status shows only bootstrap08:51
ahasenackran add machine08:51
ahasenackand got that error08:51
axwok. that doesn't make any sense to me :(08:52
axwI just did all that and it worked up until apparmor got in my way08:52
ahasenacklet me paste what I did08:53
ahasenackaxw: http://pastebin.ubuntu.com/6087049/08:56
axwahasenack: thanks, I'll try with your configuration and see what I get08:58
ahasenackaxw: that ls /etc/init/ test, it's run inside the container really?08:58
ahasenackbecause I do have such files in my host, of course08:59
ahasenack/etc/init/juju-agent-andreas-local.conf and /etc/init/juju-db-andreas-local.conf08:59
axwahasenack: yes, it'll ssh and execute ls /etc/init/juju*08:59
axw"ls /etc/init/ | grep juju.*\\.conf || exit 0" to be precise09:00
ahasenackit checks the output of that, or the exit code?09:07
ahasenackanyway,09:07
ahasenack$ ssh 10.0.3.230 "ls /etc/init/ | grep juju.*\\.conf || exit 0"09:07
ahasenackWarning: Permanently added '10.0.3.230' (ECDSA) to the list of known hosts.09:07
ahasenackandreas@nsn7:~/canonical/source/landscape/production$ echo $?09:07
ahasenack009:07
ahasenack$09:07
rogpeppe1jam, davecheney: how about this as an alternative fix? https://codereview.appspot.com/1364004309:37
rogpeppe1jam, davecheney: i haven't decided on a good way of testing it yet09:38
ahasenackaxw: I don't understand this, this will always return 009:50
ahasenackconst checkProvisionedScript = "ls /etc/init/ | grep juju.*\\.conf || exit 0"09:50
ahasenackah, you look for the output09:50
ahasenack    return len(strings.TrimSpace(string(out))) > 0, nil09:50
davecheneywhat is wrong with09:51
ahasenackI wonder if it's my ssh warning that is messing with the output09:51
davecheneyls /etc/init/juju*conf ?09:51
ahasenack"Warning: Permanently added '10.0.3.230' (ECDSA) to the list of known hosts."09:52
ahasenackI get that09:52
ahasenackor, to be more specific09:52
ahasenack$ ssh 10.0.3.230 "ls /etc/init/ | grep juju.*\\.conf || exit 0"09:52
ahasenackWarning: Permanently added '10.0.3.230' (ECDSA) to the list of known hosts.09:52
ahasenackthat "Warning" line is making len(string(out)) > 0 be true09:52
ahasenack    out, err := cmd.CombinedOutput()09:53
ahasenackbecause of that probably09:53
ahasenackcombined09:53
ahasenackaxw: ^^^09:53
davecheneyyeah, we dont want combined09:53
davecheneythat is wrong09:53
rogpeppe1davecheney: your thoughts on https://codereview.appspot.com/13640043 would be appreciated09:55
davecheneyrogpeppe1: i looked at it09:56
davecheneyi'm not qualified to comment09:56
davecheneysounds like it would just be easier to have the api server ignore other errors09:57
rogpeppe1davecheney: the API server isn't seeing any errors09:57
davecheneythis is why i am not qulified to review this patch09:58
rogpeppe1davecheney: it's being taken down because another state worker is going down09:58
rogpeppe1davecheney: np09:58
davecheneyrogpeppe1: i think that is the error I was talking about09:59
rogpeppe1davecheney: ah - but that's a provisioner error, no?09:59
davecheneybut the solutoin I would like to see is job manage environe ensure a startup order09:59
davecheneywe know the allfatal/shutdown/restart path works10:00
davecheneyi'd rather not change it10:00
davecheneyall that needs to be done is ensure the api server is running first10:00
rogpeppe1davecheney: we could do that, but i'd prefer to fix the underlying problem, which is that we don't need to take the api server down at all10:05
rogpeppe1davecheney: fwereade_ has been asking me to fix the allFatal thing for ages10:06
rogpeppe1fwereade_: your thoughts on https://codereview.appspot.com/13412047/ would be good to have, please10:06
rogpeppe1davecheney: the jujud logic needs to be able to cope with an API server that is only sporadically available anyway10:07
* fwereade_ looks10:07
davecheneyrogpeppe1: then making the api server connection retry sounds like the best approach10:07
rogpeppe1fwereade_: oops, wrong CL10:07
rogpeppe1fwereade_: https://codereview.appspot.com/13640043/10:07
davecheneythe not trying behavior that currently exists sounds like it is too chummy with other workers in its process10:07
rogpeppe1davecheney: it does retry currently10:08
axwahasenack: ahh10:08
axwthank you :)10:08
davecheneyrogpeppe1: i don't think that is correct, but am not in a position to comment authoratively10:09
rogpeppe1davecheney: the retrying is what prints the "worker: restarting "api" in 3s" log msgs10:10
davecheneyrogpeppe1: i was not clear enough10:10
davecheneynot restarting the api process10:10
davecheney/s/process/jo10:10
davecheneyjob10:10
davecheneybut retrying the connection _to_ the api10:11
rogpeppe1davecheney: i'm not sure i see the distinction10:11
rogpeppe1davecheney: the first thing APIWorker does is try to connect to the API10:11
rogpeppe1davecheney: and when it fails, the outer runner will wait for 3 seconds, then try again10:11
davecheneyrogpeppe1: ok, then something must be wrong, because looking at the log, if the connection fails, allFatal is triggered and everyone shuts down10:12
davecheneyif what I just wrote is not true10:12
davecheneythen close the bug10:12
davecheneyit's not a bug10:12
rogpeppe1davecheney: allFatal is not used in the outer level runner10:12
davecheneyi don't think i can add anything more useful here10:13
davecheneyi don't know the code10:13
davecheneyi can only describe what I see in the log file10:13
rogpeppe1davecheney: it would be clearer with the full log file, i think10:13
fwereade_rogpeppe1, https://codereview.appspot.com/13640043/ looks pretty sane, I think10:13
rogpeppe1fwereade_: thanks10:13
rogpeppe1fwereade_: any ideas for a good way to test it?10:14
fwereade_rogpeppe1, custom mgo test runner that exposes a kill method..?10:14
davecheneyrogpeppe1: https://bugs.launchpad.net/juju-core/+bugs?field.searchtext=restart&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=10:14
davecheneybest we have10:14
fwereade_rogpeppe1, would be nicest if we didn't have to have the whole stack in place ofc, but I don't see a clear way to get there from here in a reasonable amount of time...10:15
axwahasenack: FYI, https://bugs.launchpad.net/juju-core/+bug/122327710:17
_mup_Bug #1223277: manual provisioning should ignore ssh warnings when detecting presence of /etc/init/juju* <juju-core:New> <https://launchpad.net/bugs/1223277>10:17
ahasenackaxw: right10:17
mgzplan b!10:50
dimiternyep10:50
TheMuestanding10:51
dimiternlet's go in this order: dimitern, TheMue, wallyworld, jam, mgz, rogpeppe1, fwereade_ ?10:51
dimitern:)10:51
TheMue+110:52
rogpeppe1dimitern: what's the problem?10:52
rogpeppe1dimitern: i'm in the standup10:52
TheMuerogpeppe1: no sound10:52
dimiternhangout is not working10:52
rogpeppe1dimitern: ok10:52
dimiternso I'll go first then10:52
TheMuedimitern: jam cannot participate10:52
dimiternah? ok10:53
TheMuedimitern: b-day of his 6yo10:53
dimiternoh, jam congrats then!10:53
TheMue...oooOOO( like thumper today, only older *smile* )10:53
rogpeppe1i'm succeeding in talking to nate on the hangout10:53
natefinchhey it seems like it's better in the hangout now10:53
dimiternstandup: so, I've landed a couple of prereq branches in the uniter api yesterday, and stated working on migrating the uniter code from state to the api, still working on it10:54
dimiternstandup: I think I'll have something to propose later today. that's me - TheMue?10:54
rogpeppe1dimitern: try joining the standup?10:55
rogpeppe1dimitern: sorry, hangout10:55
TheMuedimitern: we've got sound now10:56
jamrogpeppe1: wrt https://codereview.appspot.com/13640043/, would that actually solve the problem? One issue here is that we *can't* connect to the API so calling .Ping() will still say "unable to connect" which will cause us to restart.11:03
rogpeppe1jam: it might be easier to chat directly; perhaps we could go through this some time after the standup?11:05
=== TheRealMue is now known as TheMue
rogpeppe1jam: i think it would solve the problem, which is caused by workers under StateWorker dying and causing their peers to die -  that will ping the mongo state, not the API state, so the API server should remain up11:44
rogpeppe1jam: i changed the allFatal in APIWorker for symmetry - it doesn't actually help to fix the bug we're trying to address11:45
jamrogpeppe1: apiserver is a StateWorker11:48
jamah, I think I see your point11:48
jamit will notice it still has a connection to mongo11:48
rogpeppe1jam: yeah11:48
=== gary_poster|away is now known as gary_poster
dimiternrogpeppe1: https://codereview.appspot.com/1351204912:26
dimiternrogpeppe1: that's the apiuniter package, as discussed on the standup12:26
rogpeppe1dimitern: LGTM12:28
dimiternrogpeppe1: thanks!12:33
dimiternrogpeppe1: I think I've beaten my previous 8000-line trivial diff record by far :D12:33
rogpeppe1lunch12:38
=== marcoceppi_ is now known as marcoceppi
natefinchrogpeppe1: you all set with that StateWorker testing?14:21
rogpeppe1natefinch: i'm going to propose a branch that adds this to worker/runner.go: http://paste.ubuntu.com/6088175/14:22
rogpeppe1natefinch: and we can layer testing onto that14:23
rogpeppe1fwereade_: how does that sound to you?14:23
* fwereade_ looks14:23
fwereade_rogpeppe1, lgtm14:24
fwereade_rogpeppe1, I might even make it a Thing not a DebugThing, and implement one that did the logging for normal usage14:24
rogpeppe1fwereade_: that's an interesting idea actually14:24
rogpeppe1fwereade_: although in that case i'd probably make it an argument to NewRunner14:25
rogpeppe1fwereade_: i'm not sure actually - there are more log messages than just those events, and i'm not sure i want to either lose log messages or turn them *all* into events14:27
natefinchrogpeppe1: that looks good to me.  I agree that making it a Thing is a good idea.  I'd change Start to Starting, though.  Also, I agree that we don't want everything to be an event. These are the major events, obviously. If we need more we can make more.14:30
rogpeppe1natefinch: Starting sgtm14:31
rogpeppe1natefinch: the problem i have with making it a Thing and not including all the events is that different events on the same runner will be logged in quite different ways.14:32
rogpeppe1natefinch: which may well make the log messages less obvious (and they're not that obvious as it is)14:32
natefinchrogpeppe1: Why does the logging have to change at all?  Can we not just add this and leave the actual logging as-is?14:33
rogpeppe1natefinch: mostly because i can't think of a decent use for this that doesn't involve testing or debugging - i don't want our code to start relying on it in subtle and hard-to-reason-about ways.14:34
rogpeppe1natefinch: if we leave the logging as it is, i think i'd be happier with it left as a DebugThing, so people don't see it later and start leveraging it for evil witchery14:35
natefinchrogpeppe1: thats a valid concern.14:36
natefinchrogpeppe1: maybe keep it as debug only for now, and if it turns out we need it for something later (probably unlikely), we can always promote it to non-debug.  And since promoting it is non-trivial, it'll take more thought than a casual call to it14:37
natefinchrogpeppe1: do you need the full pointer to the runner? That kind of invites people to twiddle with the runner, rather than just passing an identifier that can't be used for witchcraft14:38
natefinch(or at least makes witchcraft more difficult)14:39
rogpeppe1natefinch: i could pass fmt.Sprintf("%p", runner) i suppose.14:39
rogpeppe1natefinch: but given that it's only for testing, i'm ok sending the whole pointer, i think14:40
natefinchrogpeppe1: fair enough14:41
natefinchjam, fwereade_: other things I should be working on?14:50
fwereade_natefinch, I would be most grateful if you would take a quick look at https://bugs.launchpad.net/juju-core/+bug/1218616 and see what we need to do14:52
_mup_Bug #1218616: all-machines.log is oversized on juju node <juju-core> <juju-core:Triaged> <https://launchpad.net/bugs/1218616>14:52
fwereade_natefinch, I don't recall the CL that reportedly added log rotation, but I may just be ignorant14:52
natefinchfwereade_: sure thing, I'll take a look.14:54
fwereade_natefinch, there may or may not be a related issue with machine-0.log contents being repeatedly spammed into all-machines.log14:55
fwereade_natefinch, I'm not sure whether that one ever made it to the top of anyone's queue14:56
natefinchfwereade_: that bug you linked to links to this one, which has a fix.... not sure if it actually rolls the log... doesn't look like it from a casual glance: https://bugs.launchpad.net/juju-core/+bug/119522314:56
_mup_Bug #1195223: juju all-machines.log is repetitive and grows unbounded <juju-core:Fix Released by wallyworld> <https://launchpad.net/bugs/1195223>14:56
fwereade_natefinch, there was definitely an issue with repeated unit-log spam into all-machines.log, but that one was resolved14:56
fwereade_natefinch, yeah, that may be the one14:58
fwereade_natefinch, rsyslog stuff happens both at cloudinit time for machine logs, and in a deployer context for unit logs14:58
natefinchfwereade_: there are a bunch of bugs about not rotating logs... I don't think it's been fixed, but I'll double check in the code15:00
fwereade_natefinch, thanks15:02
dimiternfwereade_: so it's how I feared15:43
fwereade_dimitern, that sounds ominous15:44
dimiternfwereade_: finally everything builds with the api, but the tests will need some refactoring15:44
fwereade_dimitern, I presume they'll need to use backend syncs across theboard15:44
dimiternfwereade_: mocking will really help to separate api parts from state parts15:44
dimiternfwereade_: that's that perhaps (haven't even reached that far)15:44
dimiternfwereade_: but the problem is a whole lot of things fail due to permission denied15:45
fwereade_dimitern, ok, but I'm inclined to imagine they're just bugs15:45
dimiternfwereade_: we have to re-login to the api at some places to be able to pass the test as written15:45
fwereade_dimitern, that's an issue on the api side not refreshing auth when required, surely? there's the delayed getAuthFunc malarkey to address that scenario, I think15:46
dimiternfwereade_: well, at least I can now say, save for 2 CLs which I discovered are missing, the API is indeed complete15:46
dimiternfwereade_: I can propose these today, and continue on the struggle15:47
dimiternfwereade_: no, no - it's just trying to access freshly added unit15:47
fwereade_dimitern, the login failures sound like they're detecting api server bugs, and fixes for those will be great candidates for individual CLs before you land the big one15:47
dimiternfwereade_: and the api assumes we are working with a specific unit the entire sesson15:48
dimiternfwereade_: not like create this - try it out - scrap it - create a new one - do that..15:48
fwereade_dimitern, ahh... yeah, I see15:48
fwereade_dimitern, so we actually should be logging in again in those cases15:49
dimiternfwereade_: yeah15:49
fwereade_dimitern, thanks15:49
dimiternfwereade_: it's possible there would be some lurking bugs as well15:49
dimiternfwereade_: for example I discovered the really weird way filter_test.go is written - never noticed before it's not in uniter_test package, but in uniter15:49
fwereade_dimitern, quite so15:50
dimiternfwereade_: hence, the suites, etc. are exported15:50
fwereade_dimitern, I forget precisely the combination of forces that led me to believe that was the least evil thing to do at the time15:50
dimiternfwereade_: or maybe not15:50
fwereade_dimitern, huh, that shouldn't happen in a _test.go file15:50
dimiternfwereade_: yeah15:50
fwereade_dimitern, it's one of those things that's crying out for its own package, but the winds were not favourable at the time15:51
fwereade_dimitern, and go does tend to make package contents a bit hard to extract15:51
fwereade_dimitern, one for our Copious Free Time15:51
dimiternfwereade_: the major issues I'm having is that all public or intra-package types need to use the api types, not the state types15:51
dimiternfwereade_: and the tests use only the state types to setup scenarios, etc.15:52
fwereade_dimitern, cath's made mushroom risotto; when you can't stand it any more, come over and help us eat it?15:52
dimiternfwereade_: and I need to bring up the full facade just to get an apiRelUnit instance to give to the thing that needs it for example15:52
fwereade_dimitern, understood; tedious :(15:53
fwereade_dimitern, if you spot opportunities to ease your burden with interfaces, take them :)15:53
dimiternfwereade_: mm sgtm, but I'll perhaps give up in an hour or so and lay down15:53
dimiternfwereade_: I need a freash head for that - but I might have spotted some places15:53
fwereade_dimitern, don't exhaust yourself15:54
dimiternfwereade_: I'll just extract these two CLs from the whole mess of like 30 changed files and propose them, so I can scratch the API completely15:54
fwereade_dimitern, perfect15:55
rogpeppe1natefinch, fwereade_, TheMue: worker/runner introspection - a prereq to doing the testing right in the earlier branch: https://codereview.appspot.com/1349304415:56
natefinchrogpeppe1: looking15:56
natefinchrogpeppe1: I'm really not a fan of having a testing-only exported type in the system.  Can you put the type in a _test file so it won't be exposed outside of test-time?16:00
rogpeppe1natefinch: nope, i'm afraid not16:00
rogpeppe1natefinch: that won't be available to cmd/jujud16:00
rogpeppe1natefinch: i'm not a fan either16:00
fwereade_rogpeppe1, reviewed16:01
rogpeppe1natefinch: i suppose an alternative might be to mock worker.NewRunner inside cmd/jujud16:01
TheMuerogpeppe1: looking16:03
fwereade_rogpeppe1, or to make the hooks a first-class feature of Runner, and just mock out the DRA we pass in in the jujud tests16:03
rogpeppe1fwereade_: DRA?16:04
fwereade_rogpeppe1, DebugRunnerActions16:04
natefinchrogpeppe1: that's a possibility. There has to be a better way, even if they're also suboptimal..16:04
rogpeppe1fwereade_: i think making the hooks a first class feature is just as bad16:05
rogpeppe1fwereade_: there's really no reason to use them other than for testing/debugging16:05
rogpeppe1fwereade_: and by making them a first-class feature we invite abuse16:06
natefinchrogpeppe1: It's going to stab me in the pancreas every time I see DebugRunnerActions in the godoc16:06
fwereade_rogpeppe1, it means you don't need to expose funky global config in distant packages in order to test jujud16:06
fwereade_natefinch, that's another reason to make it Proper Code imo16:06
rogpeppe1natefinch: i kinda want it to be funky, because it *is* funky16:06
rogpeppe1s/natefinch/fwereade_/16:06
fwereade_rogpeppe1, the globalness contributes a pretty serious amount to the funkness16:07
natefinchfwereade_: I'd rather it be real code that invites abuse than funky code that we assume people won't use, but invites the same abuse.  But I think there must be a third way16:07
fwereade_natefinch, maybe we could implement AOP for go by way of promiscuous code generation :)16:08
natefinchfwereade_: ha!  tempting.... (not really :)16:09
rogpeppe1fwereade_: how about mocking worker.NewRunner in cmd/jujud instead?16:10
rogpeppe1fwereade_: i.e. var newRunner = worker.NewRunner16:11
rogpeppe1fwereade_: and replace it for testing16:11
rogpeppe1fwereade_: actually var newRunner workerRunner = worker.NewRunner16:12
fwereade_rogpeppe1, so you just check that appropriate StartWorker and StopWorker calls are made?16:14
rogpeppe1fwereade_: yeah16:14
fwereade_rogpeppe1, I'd be fine with that16:14
rogpeppe1fwereade_: and we can also check that the functions are called at the right time too16:15
rogpeppe1fwereade_: we've got just as much of a handle on things as worker/runner.go, tbh16:15
rogpeppe1fwereade_: i'd still have the original worker.Runner inside the type - we'd just wrap it so we know what's going on16:15
yolandahi, question: i'm finding that every time i destroy an environment and want to bootstrap a new one, i get the error of "no tools available", and i have to run the juju sync-tools command every time, that takes ages16:36
yolandais there any way to avoid it?16:36
fwereade_yolanda, are you on maas?16:44
yolandafwereade_ , no, canonistack16:44
fwereade_yolanda, blast, I can't find it16:46
fwereade_yolanda, I think there's a public-bucket you should be able to use16:46
yolandafwereade_, where should i look at? it's a waste of time to always sync tools16:52
mgzyolanda: hm, you should just pick up the cloud-wide sumple streams stuff I'd have thought16:53
dimiterntrivial review anyone? https://codereview.appspot.com/1338104516:54
mgzlooking16:55
mgzdimitern: seems fine as a stopgap. what's the end api we're actually envisioning?16:58
dimiternmgz: that's pretty much it - I'm preparing the last CL now16:58
dimiternmgz: I have a huge branch about the uniter api migration I'm splitting up16:59
dimiternmgz: this and the following CL are the only needed api changes16:59
rogpeppe1fwereade_, mgz, dimitern, natefinch, jam: a new testing helper - there are lots of places in the tests that could be simplified by using this: https://codereview.appspot.com/1365104317:04
rogpeppe1i put it in checkers because it has no dependencies, but it could be argued that it would be better in testing17:04
dimiternrogpeppe1: can you paste some examples how it simplifies things?17:05
rogpeppe1dimitern: ok, i'll propose a branch that uses it in some places17:06
natefinchrogpeppe1: what happens if you screw up and try to set a value to something that isn't assignable to it?17:09
rogpeppe1natefinch: you get a panic17:09
rogpeppe1natefinch: (which i think is reasonable in a testing context)17:09
natefinchrogpeppe1: can you throw in a test that makes sure that's true?17:09
natefinchrogpeppe1: absolutely17:09
rogpeppe1natefinch: good point, will do17:10
natefinchfwereade_: I never said explicitly, but we're definitely not rolling the all-machines log. I'm new to rsyslog, so coming up to speed on how to translate our current configuration to one that rolls the logs.  Also, any suggestions as to rolling policies are welcome, otherwise I'll just make something up :)17:19
dimiternanother really small review? https://codereview.appspot.com/13648044/17:24
dimiternmgz: ?17:25
rogpeppe1dimitern: reviewed17:28
rogpeppe1dimitern: here's the example you asked for: https://codereview.appspot.com/1351205117:28
dimiternrogpeppe1: thanks, looking17:28
dimiternrogpeppe1: that's nice17:29
rogpeppe1dimitern: thanks. i've been thinking about doing it for ages, but a change i was making today finally persuaded me it was worth doing17:30
dimiternrogpeppe1: LGTMed17:30
rogpeppe1dimitern: thanks17:30
* dimitern needs to stop for now17:31
dimiterng'night all!17:32
rogpeppe1dimitern: see ya17:37
rogpeppe1natefinch: reproposed with test for panic when not assignable17:37
rogpeppe1natefinch: i'm presuming it LGTY?17:38
natefinchrogpeppe1: yeah lemme look real quick17:38
rogpeppe1natefinch: thanks17:40
natefinchrogpeppe1: LGTM :)17:40
rogpeppe1natefinch:17:40
rogpeppe1natefinch: thanks!17:40
* rogpeppe1 must go now17:46
rogpeppe1g'night all17:46
natefinchrogpeppe1: gnight!17:46
marcoceppio/ X-warrior18:16
X-warrior\o18:16
X-warriorwell I will be leaving soon, anyway I will drop the idea so you guys can think about it a little bit and we can discuss it later.18:17
X-warriorI'm willing to add Elastic IP control to juju, but the question is is elastic ip control inside juju scope? Does it sound like a good feature?18:20
X-warriorI'm leaving now, talk to you later guys :D18:26
thumpermorning20:54
thumpermorning21:01
thumpertaking daughter to the doctor shortly21:01
bigjoolsmorning22:39
thumpero/22:41
bigjoolsthumper: how's the daughter?22:42
thumperantibiotics for ear infection22:42
thumperhoping it doesn't rupture22:42
thumperdoc said if it was going to, it would do so in the next 24 hours22:42
bigjoolseugh, my son just had that too22:42
thumperbut nothing they can do about it really22:42
bigjoolsruptured mine once when I blew my nose22:43
thumpernot nice22:43
bigjoolsnot much too it really, just get a bang and hearing goes funny.  Fixes itself in no time22:44
bigjoolss/too/to/22:44
* thumper headdesks23:14
thumperdavecheney, wallyworld: https://code.launchpad.net/~thumper/juju-core/task-race-on-api/+merge/18472623:34
thumperhttps://codereview.appspot.com/13412047 has the diff wrong23:34
thumperthe lp review has it right23:34
thumperfor some reason lbox chooses an incorrect parent to do the diff from23:34
rogpeppe1thumper: ping23:34
thumperhi rogpeppe123:35
thumperyou just missed a post23:35
thumper<thumper> davecheney, wallyworld: https://code.launchpad.net/~thumper/juju-core/task-race-on-api/+merge/18472623:35
thumper<thumper> https://codereview.appspot.com/13412047 has the diff wrong23:35
thumper<thumper> the lp review has it right23:35
thumperrogpeppe1: does a retry in the agent open bit23:35
rogpeppe1thumper: i've got 5 minutes before my lovely partner finishes her TV crime program - fancy a quick hangout about it?23:35
rogpeppe1thumper: or happy to talk here whatever23:36
rogpeppe1thumper: i saw your post (that's kinda why i dropped in)23:36
thumperwhich ever is easiest23:36
thumperhappy to hangout23:36
rogpeppe1thumper: higher bandwidth might be good (but a couple beers down, so excuse me in advance :-])23:37
thumper:)23:37
thumperrogpeppe1: https://plus.google.com/hangouts/_/40d498494100dea75532416a284f2bd890dd7a29?hl=en23:37
thumperdavecheney,wallyworld: unping re review, I'll take it down as unneeded, rogpeppe1 is going to fix23:45
wallyworldok23:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!