bigjoolsanyone know how the vlan work is going on, particularly testing against maas?00:57
thumperbigjools: dimiter is leading that AFAIK00:57
thumperbigjools: but, no, I have no idea00:58
bigjoolsthumper: ok thanks.  I had not heard anything and, well, it's kinda late to make changes now,00:58
* thumper nods00:59
davecheneyAH HA!00:59
davecheneyoh, no mwhudson here00:59
davecheneyi've figured out why some tests fail to compile on gccgo00:59
thumperdavecheney: mwhudson is on leave for the start of this week00:59
* davecheney works on smaller repro00:59
davecheneynah, it's a bug when compiing a package for test that has both internal, package agent, and external, package agent_test suites01:01
davecheneywell, that is my hypothesis01:01
davecheneynow to make a smaller repro01:01
davecheneyit's really hard to pin down01:02
davecheneydelete either bootstrap_test.go -or- package_test.go in juju-core/agent01:02
davecheneyand the test binary builds01:03
davecheneythe common thing between them is they are both external test files01:03
davecheneythumper: strong correlation between duplicate symbols and those with coverage01:45
davecheneyi need to take a walk and think about this01:45
* thumper goes to make a coffee02:29
davecheneyi have a reproductoin02:40
* davecheney blushes02:41
davecheneythumper: https://code.google.com/p/go/issues/detail?id=762702:54
axwthumper: one-liner (excluding test) https://codereview.appspot.com/79620043/03:11
axwincluding, two-liner ;p03:11
thumperaxw: and I assume you have tested locally?03:29
axwthumper: yup03:29
axwthumper: modified sudoers to prevent setting env vars, and ensured it failed then worked03:30
thumperaxw: I almost have a test for the mongo service cleanup03:30
axwthumper: cool03:30
axwyikes, I chewed through 7GB yesterday installing MAAS03:32
axwforgot to tell it to not download the Internet03:32
davecheneyaxw: yikes, was any of that unmetered ?03:35
davecheneyfrom a local mirror ?03:35
axwdavecheney: nope. but it's okay, I use very little of my quota03:36
axwsuggested daily usage: 24G ;)03:36
=== vladk|offline is now known as vladk
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
thumperaxw: store test failed :-(03:45
axwle sigh03:45
davecheneyfucking kick that thing out03:46
davecheneywhy do we keep having this discussion03:46
davecheneymonth after month03:46
thumperdavecheney: because we like punishing ourselves04:00
davecheneythumper: is this the right room for an argument04:07
thumperdavecheney: it depends04:07
thumperargument about what?04:08
davecheneythumper: doing things many times to keep seeing if they hurt04:08
thumperaxw: just pushing the test branch now04:15
axwthumper: thanks04:16
thumperI'll go and help make dinner, and come back to check on the review a little later04:16
thumperafter I have proposed that is04:16
thumperaxw: and yours merged \o/04:16
axwthumper: I'm going to block SIGINT right up in cmd/juju/bootstrap.go - but now I need to make it possible to cancel tools upload04:17
axwthumper: hooray04:17
axwcos otherwise you could cancel during upload and you'll also have a broken .jenv file04:18
thumperwhy do we create an empty .jenv so early?04:19
lazyPower^ i have asked myself that question.04:19
davecheneyaxw: don't forget you can't unhook a signal handler04:19
lazyPoweralso, sorry for butting in. Just saw something that made me extremely happy.04:20
thumperaxw:  https://codereview.appspot.com/7969004304:20
axwdavecheney: don't need to in this case, but you can ...? os/signal.Stop04:20
axwthumper lazyPower: yeah I kinda think we should not write it out until things are working04:20
axwhowever we do want to still catch the interrupt so we can clean up04:21
lazyPowerI dont think its a problem if there's a cleanup on exception. Thats the part that gets me, it leaves traces behind that prevent normal users from bootstrapping and they dont understand why they have to delete it.04:21
davecheneyespecially now we tell them they can use ^C to undo a partial boostrap04:22
lazyPowerthe inverse of that - the more I think about it - is what exception is ok to remove it? I can see someone trying to bootstrap an env and wiping out their jenv of a working deployment..04:22
lazyPowerwhich is exponentially worse04:23
davecheneylazyPower: i don' think it works like that04:23
davecheneyif .jenv exits04:23
axwlazyPower: .jenv is only removed if created for the bootstrap04:23
davecheneythen the environment is bootstrapped04:23
axwwhat davecheney said04:23
davecheneycogito ego bootstrap04:23
thumperaxw: I haven't done the 'remove the rsyslog file if it exists', perhaps I should add that...04:23
thumperany way, dinner cooking time04:23
axwthumper: would be good. I will review what you've got so far anyway.04:24
=== thumper is now known as thumper-cooking
lazyPowerahhh ok. talking about completely different phases of the bootstrap process.04:25
lazyPowerSome day i'll dig through the juju-core source and try to make sense of this "go" thing you guys are having so much fun with04:25
=== vladk is now known as vladk|offline
rogpeppemornin' all07:49
rogpeppejam1: i'm not quite sure how your Worker.Running suggestion solves the problem07:57
rogpeppejam1: AFAICS there's a dependency between two workers - we must only start the API worker after the localstorage worker is ready.07:58
rogpeppedimitern: hiya08:00
fwereaderogpeppe, jam is proposing a mechanism for communicating that dependency08:01
jamfwereade: right08:01
fwereaderogpeppe, jam, I'm not really that keen though -- do we have use cases for that that *aren't* localstorage? because localstorage is a bit of a hack08:01
jamrogpeppe: right now, we don't have a way to detect that the local storage worker is ready08:01
jamfwereade: we shouldn't start the environ provisioner until the API server is ready08:01
jamwe do it08:01
jamand just bounce08:02
jamfirewaller, etc.08:02
jamwe have a "startAfterUpgrader"08:02
jamthat is our hack to handle doing upgrades before running the other workers.08:02
jambut it isn't generic in any fashion.08:02
jamfwereade: in that particular case we want to start after the other one is stopped08:02
rogpeppei don't really see how the proposed mechanism would work for communicating the dependency, though i'm probably just being stupid here08:02
jambut it is still a case of "don't start all Workers simultaneously and hope for the best"08:02
jamrogpeppe: so it doesn't actually do the dependency08:03
jambut right now, we don't even have visibility08:03
jamrogpeppe: we know when something has been "started" but not when it would actually be able to respond to requests08:03
rogpeppejam: i don't see how you'd use the mechanism in this case to achieve the desired goal08:03
jamrogpeppe: so in my first cut, it would be that Bootstrap can wait until all services claim to be Running()08:04
rogpeppejam: Bootstrap?08:04
jamrogpeppe: so it doesn't actually handle dependencies, except that all dependencies must be met if everything is running.08:04
jamrogpeppe: "juju bootstrap" would only return once we determine all services have gotten to Running()08:04
jamso we could add an API for "are you Running()"08:04
jamwhich reports that info08:04
jambootstrap starts, ssh's in, sets everything up, then waits to connect to the API, and waits for Running() to return true08:05
jamClient.AllServicesRunning() or whatever.08:05
fwereadejam, I worry that that would be both tricky and brittle08:05
rogpeppejam: oh, i see, this is just about bootstrap08:05
jamfwereade: worse than the race conditions we have today ?08:05
jamrogpeppe: well, *today* we have a clear race condition08:05
jambut we have a general pattern08:06
rogpeppejam: but this could be a problem at other times too - any time the machine agent bounces08:06
jamof assuming that "if I start X, it is ready"08:06
jamwhich isn't true.08:06
fwereadejam, I don't think that's ever really a sane assumption to make, and I'm not sure that waiting for self-reported readiness makes it much better08:06
jamfwereade: so alternatives are that we put retries at all places where we think something might be starting/restarting/etc.08:07
jamwhich is a lot of whack-a-mole (from what I can tell)08:07
rogpeppejam: alternatively we could add explicit support to localstorage for determining when that it's running08:08
jamrogpeppe: sure, but we have the same thing with provisioner and api worker08:09
jamso it isn't *that* special of a case08:09
rogpeppejam: do we?08:09
jamrogpeppe: environ provisioner bounces until the API server is up08:09
jamwhich is "ok" I guess08:09
jamits better than when we restarted the whole thnig08:09
rogpeppejam: i don't think that's user-visible08:09
jamso you kept racing until it won08:09
jamrogpeppe: note, I'm not strictly suggesting that this is high priority, but it does feel like we are missing a pattern of being able to express and take action based on dependencies08:10
rogpeppejam: i see how your suggestion allows bootstrap itself to wait until everything has started, but i don't see how it could be used in a more general way08:10
jamrogpeppe: so if you wanted to express "don't start the API Server until local storage is ready"08:11
jamhow do you know when local storage is ready08:11
jamif it doesn't give a way to tell you ?08:11
jamdon't start env provisioner until api server is ready08:11
jamyou can do it bespoke each time08:11
jamso the env provisioner just tries to connect, and bounces until it succeeds08:11
jamthe api server would try to connect to port 8040 and bounce until it can08:12
rogpeppejam: i think i would add a mechanism (not in worker.Runner) to individual workers to allow them to express readiness08:12
rogpeppejam: part of the problem is that there's not a good definition of "readiness" - for example the API server could come up and then promply go down again08:13
jamrogpeppe: that is true, being able to handle transient failures is nice, though having the common cases just work is a good thing, too.08:14
=== vladk|offline is now known as vladk
jamwe *know* about start-the-world08:14
jamit happens at least as often as things dying out of band08:14
rogpeppejam: but i think we could work around that in an approximate way, by saying that we'll start as soon as we know it's been up very recently08:14
vladkgood morning08:14
jammorning vladk08:14
jamrogpeppe: so it is entirely possible to fix the current race by just adding a retry in AddCharm if Put fails, or even put it in the local Storage implementation that retries a Put() that fails.08:18
jamI was bringing up the larger discussion because we've had a couple cases where workers depend on eachother08:18
jamin ways that it feels would be good to actually model.08:18
rogpeppejam: yeah, i think it's a good discussion to bring up08:18
rogpeppejam: i'm just putting together a possible way that it might be done08:19
jamrogpeppe: so "add to individual workers" doesn't seem that different from having some workers implement a Running() method.08:20
jam(or Ready(), Started(), whatever else we would want to call it)08:20
jamit is still difficult to define the "I want X to wait for Y"08:20
rogpeppejam: i haven't yet seen how that method allows dependencies between workers08:20
jamrogpeppe: necessary but not sufficient.08:20
jamas in, you have to know when something is ready before you could react to it.08:21
rogpeppejam: because workers don't have access to the Worker types created for other workers08:21
fwereadejam, rogpeppe: I am in general in favour of the idea that workers should themselves wait for their dependencies, rather than complicating the decisions about when to start workers08:21
jaman alternative is a generic event bus and things starting up fire events that others can chose to listen to.08:21
jamfwereade: in a Poll and bounce method, or in a "wait for an event" style ?08:21
rogpeppefwereade: i'm not sure about that08:22
fwereadejam, I think it's probably situational08:22
rogpeppefwereade: i don't think that, for example, the API server should know about the possibility of a localstorage worker that it maybe has to wait for08:22
dimiternfwereade, hey08:22
jamrogpeppe: right, I would want the localstorage to indicate the API server should wait, rather than the other way around.08:22
fwereaderogpeppe, well we shouldn't have a localstorage worker anyway, which is why I'm nervous about this -- the motivating case doesn't seem like a good argument for anything08:23
fwereadedimitern, heyhey08:23
jamfwereade: don't we end up with *something* listening even if we have gridfs storage?08:23
dimiternfwereade, re your comment on https://codereview.appspot.com/79250044/diff/20001/state/compat_test.go - do you mean readNetworks should return []string{nil}, []string{nil}, nil (networks and error) when mgo reports NotFound ?08:23
fwereadejam, certainly we do, but doesn't that want to be part of the API server, like the charm put api etc?08:24
fwereadedimitern, I think so, yes08:24
jamfwereade: we have UploadTools and AddCharm that now both sit on a more standard HTTP POST URL, presumably object storage would also be just-another HTTP PUT/POST/etc08:24
fwereadejam, yeah, that's my thought08:24
fwereadejam, although probably not even that -- all we need to expose is GET, surely?08:25
jamfwereade: well, fwiw, it seems really strange to have the API server be POSTing back to itself.08:25
dimiternfwereade, I fail to see the point in that though08:25
rogpeppejam, fwereade: here's how i see that it might work: http://paste.ubuntu.com/7150179/08:25
fwereadejam, yeah, exactly08:25
jamfwereade: so localstorage today is a way to make local look like other providres.08:25
dimiternfwereade, what if we do want to know if there are no networks set and there are nil ones set08:25
dimiterns/know if/distinguish between/08:26
jamrogpeppe: anyway, I don't mean to distract us from the rest of our work too much. I'm not sure that fixing this bug is worthwile yet anyway08:26
rogpeppeactually, this is a little better: http://paste.ubuntu.com/7150180/08:26
fwereadedimitern, restate use case please? my model has been hat we shouldn't be able to tell the difference between a machine/service started with no networks set, and one created before it was possible to set networks08:26
fwereadedimitern, when do we need to distinguish those cases? because that's basically what that NotFound does for us, and I think it's a disadvantage08:26
rogpeppejam: yes, i don't think i mind a little instability at bootstrap time *that* much08:27
fwereaderogpeppe, honestly I do08:27
dimiternfwereade, hmm.. ok then08:27
fwereaderogpeppe, bootstrap weirdness is disproportionately painful for new users08:28
dimiternfwereade, mgo.ErrNotFound will be ignored in readNetworks08:28
rogpeppefwereade: yeah, consider me convinced08:28
fwereadedimitern, plus an explanatory comment, and we're good08:28
jamfwereade: so this *particular* race requires you to be able to download a charm faster than the localstorage can start08:28
fwereadejam, rogpeppe: so what's the actual rate we're seeing08:29
jamfwereade: but I'm perfectly happy that if this takes precedence over other work, fixing it with retry code in LocalStorage.Put/Get is the most expedient fix08:30
jamfwereade: CI has to do a retry loop in their local provider bootstrap && deploy test08:30
jamfwereade: I've never seen it in the wild, and had to put a Sleep() in the code to trigger it myself08:30
dimiternfwereade, something in the line of "We treat the non-existence of networks for a service or machine as a valid case and return nil error" ?08:30
fwereadedimitern, "because pre-1.17.7"08:31
fwereadedimitern, but yeah08:31
dimiternfwereade, ok, cheers08:31
fwereadejam, if it's really only hurting CI that that's a bit less of a stress for me -- and makes me all the more reluctant to build a general worker-dependency mechanism vs just implementing in-env storage08:32
jamfwereade: so if you are using the local provider on a machine with low latency to launchpad and slow CPU, you could probably trigger it, that would generally be exercising the local provider on Canonistack08:32
jamfwereade: the worker-dependency was never the fix for the bug08:33
jamit was a "we have this pattern that seemed to come up that we aren't really handling"08:33
fwereadejam, ok, I see08:33
jamfwereade: it seems the outcome of the discussion is that Retry loops handle more cases08:35
fwereadejam, that fits my instinct too08:35
jamfwereade: the one thing I *did* think we also wanted08:35
jamwas a way to have Bootstrap know when things are actually ready08:35
jamwe can trivially hook into something like "workersStarted" and some sort of global flag the API server can see.08:36
jamthat Bootstrap could then request "have you started all workers"08:36
jamthough functionally that is probably the same as "is the API Server up"08:36
fwereadejam, that's definitely interesting08:36
fwereadejam, I'm a little suspicious of the quantity of gubbins we'd need but I'm not against the idea ;)08:37
jamfwereade: so I already added a "workersStarted" channel to have the test suite be able to wait until it knew the workers were actually running, but I was personally aware that "started != running"08:38
jamand we have no channel today to signal that08:38
rogpeppejam: i think it would be more like "are all workers currently up" than "have you started all workers"08:39
jamrogpeppe: sure, but we *do* have code today that knows when all workers were started, we *don't* have a way to poll all the workers to know if they are running08:41
jam(running as in able to respond to requests)08:41
jamwe could certainly phrase it in "is everything running"08:42
jamand just return the best answer we have today08:42
jamwhich is everything was started08:42
jamand is not currently Dead08:42
rogpeppejam: i can only think of two workers like that - localstorage and API08:43
fwereaderogpeppe, zero one infinity ;)08:43
rogpeppejam: none of the others respond to external requests AFAICS08:43
jamfwereade: :)08:43
rogpeppefwereade: istm that it would be more appropriate to make those two special cases than to change the whole worker mechanism around them08:44
rogpeppehence my suggestion above08:44
rogpeppejam: what did you think of that as an idea, BTW?08:45
fwereaderogpeppe, I'm not sure that what jam proposes is as much of a change as you seem to think08:46
fwereaderogpeppe, sorry, I can't parse the important parts out of that paste08:46
rogpeppefwereade: ah, sorry, i'll trim08:46
fwereade(jam, rogpeppe: there *is* some legitimate fuzziness in between "zero one infinity" and "three strikes and you refactor")08:47
rogpeppefwereade: in fact, it's easier than that - the important lines all contain the string "localStorageReady"08:47
jamfwereade:  Ithink it is essentially "NewWorkerReady" is a way to signal, you pass it into LocalStorage if you create it, otherwise just set ready immediately, and API Server waits on it.08:48
rogpeppejam: yeah. NewDependentWorker provides a way to gate the starting of one worker on the readiness of one or more ready signals08:49
rogpeppeif the dependency graph got more complex, you could make it more table-driven08:50
fwereadejam, rogpeppe: that does seem like a reasonable cut at it, but I do worry about that sort of mechanism's sanity in the face of restarting workers08:50
rogpeppefwereade: i think it can work ok, as far as it's possible to make it work at all08:51
rogpeppefwereade: in particular, once a worker has started, you have no idea that it remains up for any period of time08:51
fwereaderogpeppe, you speak eloquently of the core of my objections :)08:51
rogpeppefwereade: but you have to do something some time...08:52
rogpeppefwereade: so the idea behind NewDependentWorker is that it waits for readiness and then starts the worker.08:52
rogpeppefwereade: but readiness is not set-once08:52
rogpeppefwereade: when a worker goes down, it should reset its ready flag08:52
fwereaderogpeppe, and more mechanism creeps in to iteratively refine the quality of the guess... but we still need to be prepared for it failing to work08:53
rogpeppefwereade: another possibility i suppose is that if the thing you're dependent on goes down, then you should go down too08:53
rogpeppefwereade: sure, it can certainly fail to work - we should be prepared for workers to fail at any time08:54
rogpeppefwereade: but if we can't take a worker's readiness as a signal that we can start its dependents, then i'm not sure we can *ever* start its dependents :-)08:55
fwereaderogpeppe, if the dependents can be resilient in the face of the occasional absences of heir dependencies, I think we sidestep the whole issue08:56
rogpeppefwereade: that's a reasonable point08:57
rogpeppefwereade: though i hate arbitrary retries :-)08:57
fwereaderogpeppe, intellectually, so do I, but they often quite useful in practice08:58
rogpeppefwereade: you've certainly convinced me that this is the best solution at this time to this problem08:59
* fwereade is happy then :)08:59
* rogpeppe reluctantly throws away the nascent DependentWorker implementation08:59
axwwe should probably just use gridfs for storage, so there's no need for another service (though I realise worker dependency is bigger than this one issue)09:01
jamfwereade: unfortunately goose has a very nice retrying HTTP Client that is not factored into a 3rd party lib, and is slightly Openstack specific (because of the Retry-after:  header not being everywhere)09:02
jamthough I don't think the goose one retries if it fails to connect completel09:03
jamvs getting a Retry response09:03
rogpeppeaxw: +109:14
dimiternmgz, hey09:17
dimiternmgz, you should really set up some alerts on the (other) machine you're using irc to notify you when there's a message :)09:21
fwereadeaxw, +100 to that09:23
dimiternfwereade, one last look at https://codereview.appspot.com/79250044/ before I submit it for landing?09:23
* fwereade looks09:24
fwereadedimitern, LGTM09:26
dimiternfwereade, thanks!09:27
fwereadedimitern, would you take a look at https://codereview.appspot.com/79730043/ please? mostly moves, only significant change is using uniter/charm.Bundle instead of charm.Bundle09:30
dimiternfwereade, sure, looking09:30
fwereadedimitern, and abit of preparatory test gubbins that will be used in a followup09:30
dimiternfwereade, reviewed09:45
dimiternhey wwitzel309:45
jam1dimitern: I have to miss the standup today, do you care to crack the Whip ?10:07
dimiternjam1, sure, my pleasure :)10:07
wwitzel3rogpeppe: did you push your changes from last night?10:10
fwereadedimitern, reproposing10:10
rogpeppewwitzel3: checking10:11
dimiternfwereade, will look when it appears10:11
fwereadedimitern, cheers10:11
perrito666good morning everyone10:12
rogpeppewwitzel3: pushed10:12
rogpeppevoidspace: hiya10:13
rogpeppevoidspace: hope you're feeling a bit better today10:13
wwitzel3hi perrito66610:13
rogpeppeperrito666: hiya10:13
wwitzel3hey voidspace , you getting along better today?10:13
voidspacehey guys10:15
wwitzel3rogpeppe: does that build for you?10:16
voidspacewwitzel3: rogpeppe: definitely better - I wouldn't recommend coming to visit but I got a good night's sleep10:16
rogpeppewwitzel3: no, it's WIP10:16
voidspacewhich makes a lot of difference :-)10:16
rogpeppevoidspace: yeah10:16
wwitzel3rogpeppe: k, just checking :)10:16
dimiternfwereade, LGTM10:16
wwitzel3rogpeppe: want to join the standup hangout, I'd like to talk about some of the changes?10:18
rogpeppewwitzel3: joining10:18
wwitzel3rogpeppe: I didn't get back from dinner early enough last night to catch you :)10:18
rogpeppewwitzel3: i'm there10:19
wwitzel3rogpeppe: I don't see you ..10:19
wwitzel3rogpeppe: these hangouts are a PITA10:19
rogpeppewwitzel3: i've joined the usual juju-core standup hangout, right?10:20
wwitzel3rogpeppe: yeah, same here :/10:20
rogpeppewwitzel3: https://plus.google.com/hangouts/_/calendar/Y2Fub25pY2FsLmNvbV9pYTY3ZTFhN2hqbTFlNnMzcjJsaWQ5bmhzNEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t.mf0d8r5pfb44m16v9b2n5i29ig10:20
frankbanmorning core devs: I am trying to track down the issue described in bug 1277445. It seems to be a juju-core provisioning issue, could you please take a look?10:28
dimiternfrankban, looking10:38
frankbandimitern: thanks10:38
dimiternfwereade, mgz, standup10:45
fwereadefrankban, I think you're on the right track for that bug10:54
fwereadefrankban, wallyworld is the person most likely to have immediate insight, but I can talk about it myself in a bit, I think10:54
frankbanfwereade: cool thanks. should this be assigned to juju-core?10:54
fwereadefrankban, yes, I think it should10:55
frankbanfwereade: cool thanks10:55
frankbanfwereade: uhm... Proprietary bugs cannot affect multiple projects. do you have a team to which assign this bug?10:56
wallyworldi can look at the bug a little later10:57
fwereadefrankban, maybe assign it to wallyworld then :)10:58
frankbanfwereade, wallyworld: thank you!10:58
* wallyworld has no idea off hand about root cause but will try and figure it out10:58
rogpeppevoidspace: separate hangout? (unless you particularly want to follow the topics)11:10
voidspacerogpeppe: sure, you start one and I'll grab coffee11:13
rogpeppewwitzel3, natefinch: i've pushed my WIP branch to lp:~rogpeppe/juju-core/030-MA-HA/11:13
rogpeppevoidspace: k11:13
wwitzel3rogpeppe: ok, thank you11:13
natefinchwwitzel3: did it work?  I'm Z. Nate Finch noew, but of course I'm still all the way right for myself11:14
wwitzel3natefinch: you're still in the same spot, you probably have to rejoin11:14
natefinchwwitzel3: I did. Boo.11:15
natefinchwwitzel3:  wonder if making it an initial instead of a name was a mistake11:16
rogpeppevoidspace: https://plus.google.com/hangouts/_/76cpijifqckugj18ncu09812oc?authuser=1&hl=en11:16
voidspacerogpeppe: currently failing to join, trying again11:28
wwitzel3natefinch: ready for my call?11:30
natefinchwwitzel3: not exactly.  Tuesday & Thursday, Lily, my older daughter, goes to preschool, and 7:30-8:30 is where I help get her up and dressed etc.  Around 8:30 I'll be able to call, but I'll have ZoĆ«, the baby, until about 9:15 or so, when my wife gets back from dropping off Lily.11:32
natefinchwwitzel3: and now you know more than you ever wanted to know about my weekly schedule :)11:32
wwitzel3natefinch: ok, I will look in to the upgrade stuff we discussed during standup then11:33
natefinchwwitzel3:  cool11:33
wwitzel3and just ping me when you're ready, I've merged rogers latest code and pushed to my branch11:33
jamnatefinch: I need more nate, I need to know it all. :)11:34
jambut the cameras will be installed next week, so we'll be cool then.11:34
natefinchjam:  haha... we make boring TV, it's all poopy diapers and finger paints :)11:35
natefinch(and hopefully not vice versa)11:35
jamnatefinch: poopy paints and finger diapers ? Doesn't sound too bad11:36
* perrito666 discover a whole new world by having hangout in a completely different device than work11:38
jamwwitzel3, natefinch, rogpeppe: so how is HA looking? Did we manage to sort out actually bring up a replica cluster?11:44
rogpeppejam: we actually got EnsureMongoServer working yesterday11:45
jamrogpeppe: I thought Wayne's summary is that you still needed keyfile to have everything work together, but that would be a follow up patch11:45
rogpeppejam: we still do need keyfile, yes11:45
rogpeppejam: there should probably be an API to allow the machine agent to retrieve the keyfile data11:46
rogpeppejam: i've been considering whether we could just use a hash of the state server private key11:46
jamrogpeppe: it is just a big password, right? I'd rather it just be a unique string that we hand to all the state server agents, otherwise there is bits of "why is it the hash(X)" rather than just "this is the string"11:47
rogpeppejam: if we use a hash of some existing data, we don't need any new API entry points11:48
rogpeppejam: and i can't *really* see why one might want to give a server the private key but not the keyfile11:49
rogpeppejam: or vice versa11:49
rogpeppejam: but...11:49
jamrogpeppe: while that is True, I'm guessing we don't have 100% the API we need for everything today, do we? In which case we are changing the API anyway.11:49
rogpeppejam: i don't mind much either way. i was just wondering if we could save ourselves some work11:49
rogpeppejam: that's true, and i think that reminds me of some tickets we don't have on the board11:50
rogpeppejam: we need API entry points for retrieving the API server private key11:50
jamrogpeppe: so I would be fine extending that API to be "give me the stuff I need to be a state server" and it hands the private key and the keyfile content and anything else we find is relevant.11:51
rogpeppejam: sgtm11:51
wwitzel3jam: +111:52
* perrito666 wonders why our ssh command doesn't prepend ubuntu@ to the host12:02
jamperrito666: it does in cmd/juju/ssh.go12:11
perrito666jam: I was talking about the utils one, ssh.Command12:12
axwperrito666: because the utils/ssh one is meant to be less opinionated than that12:17
axwi.e. it should allow ssh'ing to hosts as other users  - that is required for the manual provider12:18
perrito666axw: I see, thank you for the info :D12:23
axwperrito666: nps12:23
bodie_morning all12:34
rogpeppeaxw: ping12:35
axwrogpeppe: yo12:35
rogpeppeaxw: if you wanted to deliberately inject a bad environ config into the state (for a test), how would you go about it?12:36
rogpeppeaxw: i am being thwarted by the Policy12:36
axwrogpeppe: reopen state with a nil policy12:36
rogpeppeaxw: doh! of course.12:36
rogpeppeaxw: ta12:36
axwrogpeppe: if you grep for state.Policy(nil) you'll find an example somewhere12:36
rogpeppeaxw: presumably you could just use nil rather than state.Policy(nil)12:37
rogpeppeaxw: although it *was* useful to grep for :-)12:37
axwrogpeppe: yeah that's just for documentation12:38
rogpeppefairly trivial review, anyone? (just moving code) https://codereview.appspot.com/7689004913:00
mgzrogpeppe: lgtm13:03
rogpeppemgz: ta!13:03
=== Ursinha is now known as Ursinha-afk
jam1rogpeppe: natefinch: did you guys get the EnsureMongoServer patch put up for review?13:15
rogpeppejam1: i pushed the changes i'd made, but it wasn't anything like ready for review13:15
rogpeppejam1: i'm currently working with michael on getting the instance-ids published in the environ13:16
natefinchjam1: I think we got the last of the unknowns out of the way.  The code needs some cleanup, and I think the only thing left to do is implement upgrading existing environments. which is what we were talking about this morning in the standup13:17
jam1natefinch: so if we didn't support upgrade, but we did support just working in 1.18, I'd be happier working on the follow up patches.13:17
natefinchjam1: I think we can get that ready for review by EOD, then13:18
jam1that is, if upgrading to 1.18 doesn't break, just doesn't give you the ability to do HA13:18
jam1mgz: what's up with the destroy-environment empty .jenv stuff? We'd like to do a release today to get closer to 1.1813:19
natefinchjam1: we won't have actual HA today, just a replicaset of 113:19
jam1natefinch: sure13:19
jam1perrito666: what's up with https://code.launchpad.net/~hduran-8/juju-core/1295650_rsyslog_on_restore/+merge/212207 ?13:19
jam1Is it something we still want to have for 1.18 ?13:19
perrito666jam1: definitely13:20
jam1perrito666: well, we'd like to do a release today13:20
perrito666jam1: I am applying the suggested changes from the review13:20
perrito666and will re submit-it in a moment13:20
natefinchjam1: we'll have to test upgrade13:21
=== Ursinha-afk is now known as Ursinha
* perrito666 types faster13:22
jam1perrito666: I mostly just wanted to know if there was a blocker and if there is anything I can do to unblock it13:29
jam1but it sounds like you're active on it13:29
axwjam1: 1.17.7 is planned for today?13:30
jam1axw: we would like to if there isn't any remaining blockers13:30
jam1the milestones look possible13:30
jam1axw: do you know of other things ? https://launchpad.net/juju-core/+milestone/1.17.713:31
axwcool, looks good to me13:31
perrito666jam1: nope, I got a little bit delayed on that one bc of the time it takes to run the suite on some situations13:31
axwjam1: nope. there's the empty jenv thing, but tbh I think it could wait13:31
jam1axw: yeah, I was trying to poke mgz to see where that was at13:31
jam1rogpeppe: are you and voidspace able to pair again today?13:35
rogpeppejam1: that's what we're doing, yes13:35
jam1rogpeppe: sounds great13:37
axwnight all13:37
mgzjam1: I may be able to get it in before the release13:45
jammgz: so it doesn't seem critical, but it really feels like we need to make a decision whether we want to try to get it in.13:47
fwereadejam, mgz: I would be most grateful if we did, *but* I was wondering again13:49
jamfwereade: certainly I'd rather have  a release than wait on it (IMO), but it does seem like it should be a small amount of work.13:51
fwereadejam, mgz: do we not namespace our local environments per actual user anyway? I don't see why we need to claim space with an empty jenv... the cost appears not to match the benefit, of defending against a user bootstrapping the sameenvmultiple times in parallel13:51
fwereadejam, mgz: have I missed some nuance?13:51
mgzfwereade: JUJU_HOME13:51
mgzapart from that, no13:51
fwereademgz, explicitly shared JUJU_HOME across multiple users?13:52
mgzyeah, or other per-role account stuff13:52
rogpeppefwereade: the particular nuance that i'm thinking of is if the config store is actually a network-shared resource13:52
mgzeg, I ssh into a box to do server stack things, there can be someone else doing the same thing13:52
rogpeppefwereade: (which something that i've tried to keep as a possibility)13:53
mgzrogpeppe: hoping that locking scheme actually *works* on a network-shared resource is probably a bit much :)13:54
fwereaderogpeppe, so two users storing config for their local dev environments remotely? surely the answer is not "try to act to block collisions" but "don't collide in the first place"13:54
rogpeppemgz: the locking scheme isn't defined by the interface13:54
mgzit's a create-a-file, error if it exists one, no?13:54
fwereaderogpeppe, and I'm not sure it really helps for remote environments either13:55
natefinchwwitzel3: back finally.  Want to get on a call?13:56
fwereademgz, we are aiming for a model where, if you need to communicate with juju, you have your own identity and are expected to use that13:56
jamfwereade: rogpeppe: So I see "bootstrap.CreateStateFile" but I don't see anything calling it.13:57
rogpeppefwereade: i'm thinking of a situation where there's a group of users that have a shared environment name space13:57
fwereademgz, I'm not really interested in the multiple-admins-debugging-a-system-with-the-systems-own-identity situation13:57
rogpeppefwereade: so that one of them might destroy an environment and re-bootstrap it, and others can still connect to it13:57
rogpeppefwereade: this corresponds closely with the way that some existing users are working13:58
fwereademgz, if they aren't sufficiently well-coordinated that they can avoid having two admins fixing the same problem at the same time, we are unlikely to be able to help them13:58
rogpeppemgz: well, it does require that there is *something* which the various clients can rendezvous on13:59
mgzI think we have a variant of this issue even if we drop the empty file creation anyway13:59
rogpeppejam: i don't think CreateStateFile is pertinent here, is it?13:59
fwereademgz, ok, I do not want to confuse the issue and distract you from a simple fix that solves the problem anyway13:59
rogpeppeFWIW i have actually got a web-server implementation of configstore.Storage lying around somewhere14:00
dimiternmgz, if you're having trouble with that AddService branch I can do it btw14:01
rogpeppefwereade: the real question is: do we expect users to be able to share an environments name space?14:01
mgzdimitern: I shall be hitting propose ina sec14:02
rogpeppefwereade: and the failure case i'm concerned about is orphan environments14:02
dimiternmgz, ta!14:02
rogpeppeanyway, i must lunch14:02
* rogpeppe lunches14:03
* fwereade will think on rogpeppe's words14:03
wwitzel3natefinch: yep, sounds good14:11
jamfwereade: rogpeppe: so I think we have a specific case where the chances of creating an empty file causing problems * the actual problem is much higher than the chance of failing because of a real race14:19
fwereadejam, agreed14:20
frankbancore devs: I am encountering an error while trying to deploy a service in trusty using a local env: see http://pastebin.ubuntu.com/7151411/ Is this a known issue?14:26
mattywfwereade, got a moment?14:46
mgzdimitern: https://codereview.appspot.com/79800043 - want anything else for merging with your bits?14:55
mgzI was faffing with making the passing around use a shared struct, but don't think you need that right now, so have deferred it for later14:56
dimiternmgz, looking14:56
dimiternmgz, reviewed15:00
mgzah yeah, forgot nil is just fine for a slice for some reaso15:02
mgzdimitern: done, and landing15:06
rogpeppejam: i definitely agree that the chances of a real race is pretty low to non-existent in most realistic cases15:09
rogpeppejam: but that might not be the case in some future scenarios15:10
perrito666rogpeppe: hey, I re submitted https://codereview.appspot.com/78870043/patch/20001/30002 would you bash-review it for me? :)15:11
rogpeppeperrito666: looking15:12
rogpeppeperrito666: please could you use tabs where the code is already using tabs15:12
perrito666rogpeppe: certainly, sorry, I dragged the python config from my editor15:13
wwitzel3perrito666: which editor?15:13
perrito666wwitzel3: vi15:13
perrito666I just set tabs-> spaces for all files intead of only python15:14
rogpeppeperrito666: best to avoid any tab<->space conversion in Go15:15
rogpeppeperrito666: reviewed, BTW15:16
perrito666rogpeppe: t15:16
perrito666I just re-set the editor to avoid mixups in the future15:16
wwitzel3perrito666: also running gocmt on save will fix any tab/space issues as well15:16
=== hatch__ is now known as hatch
wwitzel3perrito666: though I still haven't got gofmt to highlight my error line in vim, so if you get that going let me know so I can steal it :)15:18
perrito666wwitzel3: Ill take a look during the weekend, I could certainly use that15:18
rogpeppemgz: remind me how to kick the gobot into action again, please15:20
rogpeppemgz: (it hasn't done anything for a couple of hours now)15:21
mgzrogpeppe: depends on the kind of stuck, generally ssh in and look at the logs15:21
mgzand borked mongo tends to be the blame, so kill some processes and see15:21
wwitzel3mgz: how does one rename the *(default) branch in bzr?15:22
mgzbeing a bit careful to distingush between test juju ones, and actual on the machine juju ones :)15:22
rogpeppemgz: that was it15:22
mgzwwitzel3: to setup colo, you do `bzr switch -b NAME` and NAME becomes trunk15:22
rogpeppemgz: ta!15:22
wwitzel3mgz: thanks :)15:22
perrito666hey, I am suddenly getting this while trying to fetch status http://pastebin.ubuntu.com/7151665/15:25
perrito666any ideas?15:25
mgzperrito666: looks like a testing cert, in a non-testing env15:28
mgzsomehow the client doesn't have the self-signed bit client side15:28
wwitzel3mgz: and then to get a remote branch from lp .. bzr switch -b NAME lp:BRANCH ?15:28
mgzmaybe you're trying to talk to an older env with a fresh env?15:28
mgzwwitzel3: `bzr branch --switch lp:BRANCH co:NAME`15:29
mgzotherwise you bind, which is probably not what you want15:29
wwitzel3mgz: thank you15:41
mgzah, pants, conflict15:46
perrito666mgz: what are your pants conflicting with?15:54
mgzmy socks maybe?15:54
rogpeppefwereade: what do you know about the Characteristics field inside bootstrap.BootstrapState?15:59
rogpeppefwereade: ISTM that it's now redundant15:59
fwereaderogpeppe, I was just about to say I think it is redundant15:59
rogpeppefwereade: and even if it isn't, it was only ever used at bootstrap time, right?16:00
fwereadeeven if we do use it we should have easier ways of getting the data in16:00
fwereaderogpeppe, concur16:00
rogpeppefwereade: all this leads to: i don't think it matters if publishing state server instance ids overwrites it16:00
fwereaderogpeppe, agreed, but please make sure it really is unused (or guaranteed no longer used by the time it could be overwritten)16:01
fwereaderogpeppe, and file a tech-debt bug for doing it right if we don't already16:01
rogpeppefwereade: it seems like we do it right already - i deleted16:07
* fwereade cheers16:07
rogpeppethe field16:08
rogpeppeand all the code still compiles16:08
natefinchdeleting code is always my favorite thing to do16:08
voidspacerogpeppe: https://codereview.appspot.com/7982004316:22
perrito666ah, it was the succesful restore of the backup that was breaking the dev certificate... price of success16:24
rogpeppesimple review anyone? https://codereview.appspot.com/7982004316:25
rogpeppefwereade, natefinch, jam: ^16:27
fwereaderogpeppe, meetings I'm afraid16:27
rogpeppefwereade: np16:27
perrito666if I have a bzr branch, lets call it B, originally from branch A, then I want to start depending on branch C instead of A, is as simple as a merge and tell lbox that I depend on C?16:29
rogpeppeeven more trivial review anyone? https://codereview.appspot.com/79830043/16:30
=== tvansteenburgh is now known as tvansteenburgh-l
rogpeppeperrito666: it depends if you want launchpad to know about the prereq16:31
rogpeppeperrito666: in general, prereqs are a pain16:31
=== tvansteenburgh-l is now known as tvanlunchburgh
rogpeppenatefinch, wwitzel3: how's it going?16:33
=== vladk is now known as vladk|offline
hazmatis there a known issue on trunk and issues bootstrapping.. seems like the ssh connection stalls out, even though things are potentially done.16:36
rogpeppe hazmat: not that i'm aware of16:48
natefinchrogpeppe: going good.  Merged with trunk, had a few conflicts to fix... tests are compiling and almost passing now16:55
rogpeppevoidspace: lp:~rogpeppe/juju-core/531-SetStateInstances16:55
wwitzel3rogpeppe: same as nate, I'm just trying to get a set of test failures passing, not sure why your message before didn't hilight16:56
wwitzel3rogpeppe: all the cmd/jujud bootstrap tests are compling about no --env-config being passed in, but it is clearly being passedi n.16:56
voidspacemgz: so, using the branches workflow you taught me at the sprint17:01
voidspacemgz: "bzr switch -b new_branch_name"17:01
mgz...which I should still send to the list17:01
voidspacemgz: always creates a branch from *the current branch*17:02
mgzvoidspace: yup17:02
voidspacemgz: what I inevitably (usually) want is "create a branch from trunk" instead17:02
mgzjust `bzr switch trunk` first for normal tings17:02
voidspacemgz: is there a handy shortcut for that17:02
voidspaceheh, ok17:02
voidspaceI usually forget...17:02
voidspaceI wondered if there was an alternative17:02
mgzvoidspace: `alias bzrnew="bzr switch trunk&&bzr switch -b"17:05
voidspacemgz: yeah, I guess so :-) thanks17:05
mgzvoidspace: what I hate most is forgetting to switch before doing a big merge...17:11
mgzas they can't be carried over, unlike general changes17:11
voidspaceall good fun17:12
natefinchmgz: the thing I hate is doing a merge when I have a bunch of local changes (or switching branches when I have local changes)17:12
natefinchmgz: I really want bzr to complain that I have local changes and make me add a flag or confirm that I really want to muck with my local changes on this branch first17:13
mgzhm, yeah17:15
perrito666hey could someone remind me the process to get something merged once it got reviewed and approved?17:21
mgzperrito666: `bzr rv-submit` is the easy way17:21
mgzI think I walked you through setting that up17:21
perrito666mgz: it is set up17:21
perrito666:D I had forgot the command I was not sure if there was anything else to do there17:22
=== tvanlunchburgh is now known as tvansteenburgh
voidspaceSimple review for someone: https://codereview.appspot.com/7982004317:31
perrito666jam: my blocking proposal has been submited for merge17:35
perrito666anyone could hint me the syntax for merging two lightweight branches???17:40
wwitzel3perrito666: you mean in bzr?17:41
mgzvoidspace: reviewed17:41
voidspacemgz: thanks, looking17:41
mgzperrito666: generally want to care about which one is the left hand side, so which branch you're merging into the other17:42
voidspacemgz: on your first point, I'll add a comment17:42
mgzso, switch/goto that branch, then `bzr merge LOCATION_OF_OTHER_BRANCH`, then resolve conflicts, then commit17:42
voidspacemgz: on your second, they're actually being published separately17:42
voidspacemgz: so if we bundle them together we'll just need to unbundle them17:43
voidspacemgz: so bundling as a struct not really appropriate17:43
voidspacemgz: I'll add the comment and push a new revision17:43
mgzvoidspace: fair enough17:43
voidspacemgz: if you're happy with my excuse I'll just approve after that17:43
perrito666mgz: I want to sort of run an integration test between two branches of mine17:43
mgzperrito666: okay. good special case17:43
mgzbest thing for that, is go to trunk17:44
mgzmerge the first branch17:44
rogpeppemgz: to add to that: address is orthogonal to instance id - we can potentially have an instance id but no address yet17:44
mgzmerge --force the second branch17:44
mgzthen run tests17:44
mgz(then throw away the changes on trunk with revert or similar)17:44
perrito666bzr merge just fails stating that it cant find the path17:44
mgzrogpeppe: was just an api neatness comment, if there are good reasons to think of them as seperate values like that, then it makes sense as is17:45
mgzyou need to address the other locations correctly17:45
mgz`bzr info LOCATION` should work, for bzr merge to understand it17:45
voidspacemgz: rogpeppe: so I've updated the comment that we need to actually use instanceIds17:46
mgzso, you'd need a co: for the colo workflow, or some leading ../../ for seperate trees layout17:46
voidspacemgz: rogpeppe: ok to approve?17:46
perrito666mgz: tx17:46
rogpeppemgz: there *was* a TODO comment two lines up from your "at least a TODO comment" remark :-)17:47
mgzyeah, but not one I parsed apparently :)17:48
mgzI did giggle at the diff17:48
voidspacemgz: thanks17:48
rogpeppemgz: a review of this would be appreciated too, if poss (deleting code only) https://codereview.appspot.com/79830043/17:49
mgzhm, I kind of wanted Ian to look at that17:49
voidspaceI guess we all knew C++ templates were turing complete, but did you know that Magic the Gathering rules are turing complete?17:52
bloodearnestvoidspace: I think Mt Gox was written in MTG17:53
voidspaceapparently mediawiki templates, sendmail configuration and apache rewrite rules are all turing complete17:53
mgzI guess I knew there were infinite loops in MtG17:53
natefinchwait, did I miss a conversation about Magic the Gathering, or what?17:54
mgzrewrite rules are a special bit of hell17:54
perrito666can I mark my own bugs as fixed after merged?17:54
mgzperrito666: the bot should actually do it for you17:55
voidspacenatefinch: apparently the rules are complex enough to be turing complete17:55
mgzif you linked the bug correctly to the branch17:55
natefinchvoidspace: I can believe it17:55
perrito666mgz: the branch is linked https://code.launchpad.net/~hduran-8/juju-core/1295650_rsyslog_on_restore and says there it is meged17:56
=== vladk|offline is now known as vladk
mgzperrito666: I've manually flipped the bug, in case the bot was just having a holiday17:58
rogpeppemgz: have you got a few moments to chat?17:59
mgzrogpeppe: sure17:59
mgzstandup hangout?18:00
rogpeppemgz: https://plus.google.com/hangouts/_/76cpijifqckugj18ncu09812oc?authuser=1&hl=en18:00
natefinchdamn, something in agent/bootstrap tests isn't properly cleaning up behind itself :/18:00
mgzrogpeppe: can you do the invite thing to that?18:01
rogpeppemgz: invited18:01
perrito666mgz: tx18:02
voidspaceg'night all18:27
voidspacesee you tomorrow18:28
rogpeppemgz: there's one wrinkle still: what about old clients talking to the new version? i'm not sure what degree of compatibility guarantees we provide for that.18:30
rogpeppemgz: even if we leave the original instance id there, since it won't be updated, once the original instance dies, old clients won't have the fallback-to-environment option.18:31
wwitzel3rogpeppe: so I'm stuck, when the cmd/jujud bootstrap_test calls selectPreferredStateServerAddress, the inst.Addresses() is returning nil19:01
wwitzel3rogpeppe: can't figure out how that is even that case19:01
rogpeppewwitzel3: are you in a hangout?19:01
rogpeppewwitzel3: link?19:01
wwitzel3rogpeppe: https://plus.google.com/hangouts/_/76cpitlq3utehe1ddn4giaao8s19:01
rogpeppei.addresses = instance.NewAddresses([]string{string(i.id) + ".dns"})19:08
rogpeppewwitzel3: http://paste.ubuntu.com/7152689/19:09
rogpeppewwitzel3: http://paste.ubuntu.com/7152710/19:14
=== Ursinha is now known as Ursinha-afk
mrammhey, anybody from the juju core team got 30 min to work on a "special project" with me?19:24
mrammwwitzel3, vladk, mgz, natefinch, jam, fwereade, hazmat, cmars -- any of you available?19:27
natefinchmramm: sure19:27
vladkmramm: hi19:27
natefinchmramm: oh, missed the 30 min requirement... not really19:27
=== Ursinha-afk is now known as Ursinha
vladkwhat is "special project"?19:29
mrammnatefinch: understandable, I know you are on the HA stuff19:29
mrammvladk: I'll ping you in private19:29
natefinchmramm: yep.  Too bad.  It's close though.19:29
perrito666:D amazon restored19:44
perrito666the whole thing worked19:44
wwitzel3natefinch: pushed the fixes for cmd/jujud/bootstrap_test.go to my HA branch.20:02
wwitzel3EOD for me but I'll be floating around20:02
thumpersinzui: re bug 129730620:13
_mup_Bug #1297306: local juju bootstrap fails, cannot talk to db 37017 <local-provider> <lxc> <mongodb> <ppc64el> <trusty> <juju-core:Triaged> <https://launchpad.net/bugs/1297306>20:13
thumpersinzui: I'm guessing either mongo issue or no-proxy/http-proxy issue20:13
thumpersinzui: do we have access to the machine?20:14
sinzuithumper, I suspect the mongo 64k page size bug is the issue20:14
thumpercheck status of juju-db service20:14
sinzuithumper, It is not ours20:14
thumperthis line "sudo: unable to resolve host frb-juju" is suspect20:14
rogpeppemramm: if you'd pinged me, i might've been up for a bit of light relief :-)20:40
mrammrogpeppe: sorry, missed your name in the IRC list20:44
rogpeppemramm: np. i'm better doing this anyway...20:44
=== vladk is now known as vladk|offline
perrito666hey did .7 got released?22:43
thumperperrito666: not yet AFAIK22:44
waiganithumper: plugins boilerplate is ready for review: https://code.launchpad.net/~waigani/juju-core/suspend-resume-local-plugin/+merge/21274323:57
davecheneythumper: === RUN Test23:58
davecheneyOK: 16 passed23:58
davecheney--- PASS: Test (0.06 seconds)23:58
davecheneyok      launchpad.net/juju-core/agent   0.360s23:58
davecheneypassed with gccgo23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!