/srv/irclogs.ubuntu.com/2012/12/05/#juju-dev.txt

hazmatmramm, welcome back01:14
mrammhazmat: thanks01:14
TheMueMorning07:00
rogpeppe1TheMue_, davecheney, fwereade: morning!07:47
TheMue_rogpeppe1: Heyhey07:47
fwereaderogpeppe1, TheMue_, davecheney: heyhey07:52
TheMue_fwereade: Also for you a good morning07:52
jamTheMue_: good morning08:11
TheMue_jam: Hiya08:11
=== TheMue_ is now known as TheMue
fwereadeTheMue, rogpeppe1: do either of you have any context on what the deal is with those vast terrifying incomprehensible tables in machine_test.go?08:48
fwereadeTheMue, rogpeppe1: ISTM that they should be completely rewritten and could probably be at least 50% smaller08:48
=== rogpeppe1 is now known as rogpeppe
* rogpeppe looks08:49
fwereadeTheMue, rogpeppe: but I don't know that for sure, and maybe one of you will be able to point out some way this would be a foolish move08:49
rogpeppefwereade: you're talking about machinePrincipalsWatchTests, right?08:50
fwereaderogpeppe, all the machine watchy tests really08:50
rogpeppefwereade: if you've got a nice way of making them prettier and more maintainable, i'm definitely all for that08:50
rogpeppefwereade: it would be nice if all the test cases were independent, right enough08:51
rogpeppefwereade: although i fear that we might slow down the testing considerably by doing that08:52
rogpeppefwereade: i'd prefer to avoid another test suite that takes 2 minutes to run08:52
fwereaderogpeppe, yeah, it's work I don't really *want* to do, but since I need to mess with those tests destructively anyway (because they're doing totally nonsensical things with subordinates)08:52
fwereaderogpeppe, indeed, I shall try to figure out a way to avoid setting up and tearing down all of state in one go08:53
fwereaderogpeppe, incidentally, I would appreciate comments on https://codereview.appspot.com/6845120/ if you have a moment08:54
rogpeppefwereade: yes, sorry, i've been fully focusing on getting an API proof-of-concept spike going the last couple of days. (i'm hoping to have something for you to have a look at later this morning)08:56
fwereaderogpeppe, awesome08:56
TheMuefwereade: I'll take a look.08:59
* TheMue is puzzled about his firewaller test which is hanging after the last change09:01
fwereadeTheMue, cheers09:05
TheMuefwereade: Didn't forget you, just stuck in the tests. ;) So far OK, I'm only a bit unhappy with Context.09:36
fwereadeTheMue, if yu can think of a beeter name for it I'm all ears -- or are there deeper worries?09:37
TheMuefwereade: No, it's exactly the name. Somehow too generic. I'm trying to find a better one, but so far I've got none. :/09:38
TheMuefwereade: Where do you see the future implementations of the Context interface? At the providers?09:39
fwereadeTheMue, I was considering Target but that wasn't quite right09:39
fwereadeTheMue, no, I expect an LXC implementation and that's it09:39
fwereadeTheMue, I am only aware of those two deployment styles, but I guess there may be others one day?09:40
TheMuefwereade: Hmm, here I'm missing the big picture. I would like to stand in front of a whiteboard with you now. :)09:40
fwereadeTheMue, ok, I'll try to do the quick version09:40
fwereadeTheMue, principal is to machine as subordinate is to principal09:41
fwereadeTheMue, ie the responsibilities of a machine (wrt deployment) are the same as those of a prinicpal: the aspects that differ are (1) what setof units are relevant and (2) how those units should be deployed09:42
fwereadeTheMue, at the moment, I plan to add deployers for both machine agents and for prinicpal unit agents, each of which will use the SimpleContext09:42
fwereadeTheMue, once we have LXC available, the machine's deployer just swaps out the Context, and we're done09:43
TheMuefwereade: Thanks, makses it clearer.09:44
fwereadeTheMue, cool09:44
TheMuefwereade: Sadly all the names that come into my mind now are generic too.09:44
TheMuefwereade: Maybe workspace, that matches to your method names.09:46
fwereadeTheMue, hmm, interesting, I think I might like that09:47
fwereadeTheMue, it makes me wonder whether the state info should be on the context/workspace or the Deployer09:48
TheMuefwereade: Cheers09:48
fwereadeTheMue, with "context" that made sense, with "workspace" maybe less so09:49
fwereadeTheMue, but maybe the interface should be demanding the deployer name in every method, and state info for dpeloying with09:49
TheMuefwereade: What exactly do you refer with "state info" to?09:49
fwereadeTheMue, *state.Info?09:49
TheMuefwereade: Aargh, ok.09:50
TheMuefwereade: I would see a Workspace implementation only responsible for that Workspace, but everything regarding the state should be handled by the Deployer.09:50
TheMuefwereade: But for shure those informations could be passed to the Workspace by the Deployer initially.09:51
fwereadeTheMue, yeah, I'll think about it :)09:52
TheMuefwereade: Cheers++09:52
TheMuefwereade: AddUnitSubordinateTo(unit) is on Service and adds the Unit as subordinate to the Service?09:57
fwereadeTheMue, it is currently, but I am changing that ATM10:04
TheMuefwereade: What will the new name be?10:04
fwereadeTheMue, I want RelationUnit.EnsureSubordinate()10:04
TheMuefwereade: What we talked about, ok. Thought that this would not come. Alternatively AddUnitAsSubordinate() would also be ok, not as elegant as Ensure…, but ok. :)10:05
fwereadeTheMue, this is unwieldy from a testing perspective, but I think right from every other perspective10:06
fwereadeTheMue, sorry, what type would that method be on?10:06
fwereadeTheMue, and what args?10:06
TheMuefwereade: Eh, as I've seen it here in the tests on Service. But you do more than just a renaming, so I have to wait and see.10:07
fwereadeTheMue, my issue with putting it on Service is that, at the point we need to add a subordinate, we have a RelationUnit but no Service; and it seems silly to do the whole service-getting dance outside state, just so we can use a method with lots of opportunities for crackful usage, rather than to keep the fiddly details inside state and have a single method that can only be used when there's a reasonable chance of the operation being sane10:09
fwereadeTheMue, the tests are riddled with nasty things like principals with N subordinates of the same service, and subordinates existing without having relations with their principal10:10
fwereadeTheMue, EnsureSubordinate involves more work with the tests, but at least ensures (ha) that we can only add subordinates when it's sane to do so10:11
TheMuefwereade: Thanks for explanation, I only stumbled uppon the name.10:11
fwereadeTheMue, ofc fixing all those tests is no fun, every time I look at machine_test.go my brain thinks of something else I have to do that is terribly important10:12
fwereadeTheMue, but I'm getting there :)10:12
TheMuefwereade: *rofl*10:13
rogpeppefuck me, it works10:29
rogpeppe:-)10:29
fwereaderogpeppe, yay!10:33
rogpeppefwereade: here's the spike: https://codereview.appspot.com/6878052/10:40
rogpeppefwereade: i'm very interested to know what you think. in particular, i *think* the rpc package is quite nice, but mileages might vary significantly!10:41
rogpeppeTheMue: likewise, i'd like your reaction10:44
rogpeppethe docs for the rpc package can be viewed here: http://go.pkgdoc.org/launchpad.net/~rogpeppe/juju-core/176-rpc-spike/rpc10:44
TheMuerogpeppe: I'll take a look.10:44
Arammoin.10:45
rogpeppeAram: hiya10:46
rogpeppeAram: if you fancy it, you might wanna take a look at my API spike branch. https://codereview.appspot.com/6878052/10:47
TheMueAram: Moring10:48
TheMues/Moring/Morning/10:48
Aramrogpeppe, right away.10:50
AramTheMue, cheers.10:50
TheMuerogpeppe: First look is impressive, need more time to get it. Will look again after found my own problem here. ;)10:57
rogpeppeTheMue: thanks10:57
rogpeppehere comes the snow...11:02
TheMuerogpeppe: Here we have a fine white cover since last night. Not very much, but white (and cold, -6° this morning).11:09
rogpeppeTheMue: it's just started snowing quite hard... i wonder how long it'll keep going11:10
TheMuerogpeppe: Those are the day I like working from home. ;)11:10
rogpeppeTheMue: me too!11:14
fwereadewhoops, gotta dash, post office about to close, bbiab11:22
niemeyerMorning all11:39
rogpeppeniemeyer: hiya11:46
dimiternjam, wallyworld_ here's the first part of the nova double https://codereview.appspot.com/687705412:03
wallyworld_cool12:03
dimiternjam: ping12:16
dimiternjust a quick question - so I have that branch I proposed, but I want to keep working on the same feature, is the correct process like this: 1) create a new branch while I'm in the same branch I proposed, 2) continue working there, 3) once ready - propose it with a prerequisite=the old branch?12:19
jamdimitern: it depends what "keep working" means, but generally I would say yes. I'm not sure if lbox supports prerequisite branches or not, though.12:21
dimiternjam: yes it does12:22
jam--req =12:22
dimiternyep12:22
dimiternjam: keep working means continue on the same feature, even though the first branch is not merged yet12:23
jamdimitern: right, so it is possible that keep-working is tweaking the already proposed branch, such that you would just push up a new version. Or it can be "building stuff on top of the branch" which would be creating a new branch and using prereq12:24
jamMost likely what you want is a prereq12:24
dimiternjam: ok, so I'll do that (with the new branch of the old one) - I ask because once the proposed branch is merged, you cannot propose it again - it has to be a new one12:25
jamdimitern: I think 'lbox propose' will refresh the current proposal12:25
jamcertainly you can resubmit in the lp ui12:25
jamI'm not 100% sure what the reitveld bits are.12:26
dimiternjam: not unless it's submitted (it will reopen the CL, but the merge won't work if you try submitting again)\12:26
Aramlbox propose refreshes both rietveld and launchpad, yes.12:30
Arambasically the workflow you describe dimitern is sane, that's what we use.12:30
Aramdo something, propose, branch into something else, propose with prereq, repeat.12:31
dimiternAram: ok, thanks, good to know I'm doing it right :)12:31
niemeyerGosh12:39
niemeyerI was trying to build a C++ program last night.. it's interesting how biased we get after we taste sane builds12:39
niemeyerNot only it took several hours and overheated the laptop, but it also killed my disk space12:40
niemeyer(which is why I'm reminding of it just now)12:40
Aramwhat the hell did you try to build?12:40
jamniemeyer: what were you building?12:40
niemeyerjam: LLVM12:40
Aramah, yes.12:40
Aramheh.12:40
niemeyer7GB on the build directory so far12:41
niemeyerHasn't finished, so I don't know how much it actually takes12:41
niemeyerand some people complain Go binaries are large12:41
Aramit's interesting. Solaris was 20 million lines of C code. It took two hours to build on T410, but LLVM is significantly smaller and takes longer just because it is C++ not C.12:42
Aramoh, and Solaris was built twice in this time (one with gcc and once with sunpro).12:43
rogpeppeniemeyer: i've made a spike branch as a proof of concept around the API stuff. i'd like to know what you think. it's sketchy in many places. https://codereview.appspot.com/6878052/12:46
niemeyerrogpeppe: Cool, I'll just finish this test I'm writing and we can talk12:46
rogpeppeniemeyer: cool12:47
niemeyerWhy is env.Destroy hanging on TearDown of the dummy live tests?12:47
rogpeppeniemeyer: it works for me12:49
niemeyerThe lack of space probably destroyed my state12:50
niemeyerNope.. looks like something is actually wrong with the way dummy is managing MongoDB12:56
niemeyerrogpeppe: We should go back to that idea of using a single binary at some point13:07
rogpeppeniemeyer: agreed13:07
rogpeppeniemeyer: it'll be quite simple to change i think. the only wrinkle is setting up the symlinks13:07
niemeyerrogpeppe: How's it different?13:08
rogpeppeniemeyer: we'd still need binaries called "juju", "jujud", and "jujuc" even if they all point to the same thing, no?13:08
niemeyerrogpeppe: I don't think so13:08
rogpeppeniemeyer: ah ok - we just have the same binary act as all things?13:09
niemeyerrogpeppe: jujuc is already just a symlink target by itself13:09
niemeyerrogpeppe: jujud is the only one that might require some tweaking13:09
rogpeppeniemeyer: so the user types "juju help" and they get jujud subcommands too?13:09
niemeyer[LOG] 81.27788 JUJU state: connection failed: dial tcp 107.21.168.241:37017: connection refused13:10
niemeyer[LOG] 81.27794 JUJU state: connecting to 107.21.168.241:3701713:10
niemeyer[LOG] 81.46296 JUJU state: connection failed: dial tcp 107.21.168.241:37017: connection refused13:10
niemeyer[LOG] 82.02849 JUJU state: connecting to 107.21.168.241:3701713:10
niemeyer[LOG] 82.23336 JUJU state: connection failed: dial tcp 107.21.168.241:37017: connection refused13:10
niemeyer[LOG] 82.23343 JUJU state: connecting to 107.21.168.241:3701713:10
niemeyerrogpeppe: Btw, we should slow that down a bit, I thnk13:10
rogpeppeniemeyer: i agree. perhaps an option in mgo.Info ?13:10
niemeyerrogpeppe: A sane default might be enough13:10
rogpeppeniemeyer: yeah, true13:11
niemeyerWe're punching the server/services too hard it seems13:12
rogpeppeniemeyer: is it causing problems?13:12
niemeyerrogpeppe: It creates a fast-scrolling log, without much in return13:13
rogpeppeniemeyer: agreed.13:13
niemeyerrogpeppe: and I just got a "unauthorized access" error13:13
niemeyerrogpeppe: Any hints when that happens?13:13
rogpeppeniemeyer: did it cause a test failure?13:13
niemeyerrogpeppe: Yeah13:13
rogpeppeniemeyer: hmm. i thought i'd fixed that13:13
niemeyerrogpeppe: What was the cause?13:14
rogpeppeniemeyer: can you paste the test log?13:14
rogpeppeniemeyer: the race between bootstrap-state and the client connection13:14
niemeyerrogpeppe: The log is really uninteresting.. a long tail of attempts to connect followed by unauthorized13:14
rogpeppeniemeyer: which test was it? and at which point in the test was it trying to connect? the log might still be useful to show that.13:15
rogpeppeniemeyer: this was in the live tests?13:15
niemeyerrogpeppe: It's the test I'm writing, and it's trying to connect after bootstrap13:15
niemeyerrogpeppe: Yes13:15
niemeyerrogpeppe: Maybe it's my fault then.. I'll have a deeper look13:16
rogpeppeniemeyer: is it using juju.NewConn ?13:16
niemeyerrogpeppe: Yeah13:16
niemeyerAwww13:16
niemeyerIt kills the machine doesn't it13:16
rogpeppeniemeyer: i bet bootstrap-state never ran13:17
niemeyerrogpeppe: Yeah, possibly.. it'd be easy to figure it out with logs13:17
niemeyerI mean, if I had access to the machine13:17
niemeyerHmm.. I wonder if I can still check the console logs13:18
rogpeppeniemeyer: run the test, wait until it's started the machine, then ^C the test and ssh to the machine13:18
niemeyerYep!13:18
rogpeppeniemeyer: do you actually see anything useful in the console logs?13:18
niemeyerrogpeppe: All good, it's probably me doing something silly on the test13:18
niemeyerrogpeppe: It couldn't find jujud13:19
niemeyerrogpeppe: Which is most likely me13:19
rogpeppeniemeyer: ah, that could explain it :-)13:19
niemeyerrogpeppe: I'll keep the symptom in mind :)13:19
rogpeppeniemeyer: there is one outstanding issue to do with unauthorized access, but i think that only happens when several clients are trying to connect concurrently.13:19
rogpeppeniemeyer: yeah, it would be nice to give a more helpful error.13:20
rogpeppeniemeyer: the only problem is that if your password is genuinely wrong, then that error message is actually correct.13:21
niemeyerrogpeppe: Yeah, it's a bit tricky to be helpful in this case13:21
niemeyerrogpeppe: Really, *anything* could go wrong13:21
niemeyerThere I go again13:24
niemeyerAwesome, the failing test was entirely related to the fact the test was *supposed* to fail.13:29
* rogpeppe goes for some lunch13:30
niemeyerrogpeppe: Enjoy13:30
niemeyerI shall not be long either13:30
niemeyermramm: ping13:48
rogpeppeback13:48
niemeyerhttps://codereview.appspot.com/686807013:58
niemeyerI'll step out for lunch13:58
niemeyermramm: The meeting you scheduled overnight is over my lunch time.. I hope it's okay to move it forward.13:59
andrewdeaneniemeyer, rogpeppe Hello. I'm just going through the data available. Is it possible to detail the certificate(s) associated with an environment, machine, service, unit? Thanks.14:50
rogpeppeandrewdeane: from what point of view are you asking?14:50
andrewdeaneWhen recording the current state I'd like to see the details of the 'person' that invoked the item.14:52
mrammniemeyer: sure, we can move it, let's move it back though.14:55
mrammjust ping me when you get back from lunch14:55
rogpeppeandrewdeane: i don't quite understand what you're saying there. what do you mean by "recording the current state" ?15:00
rogpeppeandrewdeane: (BTW i generally won't see remarks on IRC unless they mention my IRC nick)15:00
rogpeppefwereade: ping15:00
fwereaderogpeppe, pong15:00
rogpeppefwereade: i'm trying to understand what you're trying to protect against with the DeployerName check15:01
rogpeppefwereade: when might it be triggered?15:01
fwereaderogpeppe, sorry, which check? the purpose of DeployerName is to distinguish between different contexts that happen to be using the same init dir15:02
rogpeppefwereade: when would that happen?15:02
andrewdeanerogpeppe, Sorry. If I do a juju status is there a way to see who (certificates?) started the items?15:02
fwereaderogpeppe, any agent that runs will have an upstart job for itself in that dir that it is not responsible for15:02
rogpeppeandrewdeane: not currently, no.15:03
andrewdeanerogpeppe, any plans to, or to get the information out some other way?15:04
fwereaderogpeppe, and since we don't currently have LXC, we'll have (1) the machine agent which should not be removed; (2) N principals which should be touched by the machine agent only and (3) M subordinates which should each be touched only by their principal15:04
fwereaderogpeppe, all in the same init dir15:04
rogpeppefwereade: but surely the deployer's unit watcher won't see the wrong units?15:04
rogpeppefwereade: (which is what we're trying to guard against, presumable)15:05
rogpeppey15:05
rogpeppeandrewdeane: currently there are no notions of "ownership" in the state. we might move in that direction in the future, but maybe not for a while15:05
rogpeppeandrewdeane: anyone with admin rights (the administrator password) can currently change anything in the state.15:06
fwereaderogpeppe, hmm, I suppose we could just drop the ability to list deployed services15:07
rogpeppefwereade: or we could perhaps tag the upstart job name with the entity that's responsible.15:08
fwereaderogpeppe, that is what you're complaining about, isn't it?15:08
andrewdeanerogpeppe, is it true that on bootstrap we know which cert is being used? Is there a way to id those with admin rights? Incoming IP, say?15:08
rogpeppefwereade: i'm mumbling because i don't understand the logic in changed() and why it's there.15:09
rogpeppeandrewdeane: with the API will come the ability to know who's doing what in the state.15:09
rogpeppeandrewdeane: then it will be easy enough to tag items with the entity that touched them15:09
rogpeppeandrewdeane: but first i've got to implement it!15:10
andrewdeanerogpeppe, :) I wont hold you to anything but do you have ball park timescales?15:10
rogpeppeandrewdeane: currently there are only two certificates in the system - the server's certificate and the CA certificate.15:10
rogpeppeandrewdeane: the API is due for 13/04, i think15:10
fwereaderogpeppe, I'm not sure how to answer that -- changed() makes sure that the local deployed state is what it should be, for any given unit, and recalls anything that shouldn't be there even if by some poor luck it was not mentioned in an event15:11
fwereaderogpeppe, eg on unit reassignment15:11
fwereaderogpeppe, if a unit is assigned away from a machine while the agent is down, it won't know about the unit at all when it comes back up, except by examining local state15:11
fwereaderogpeppe, I know we never unassign from machines in real code at the moment15:12
andrewdeanerogpeppe, ok. thanks. I can put placeholder fields in, in the meantime.15:12
fwereaderogpeppe, but since it's possible, I want to be sure it's handled sanely, even if it's kinda incomplete15:12
fwereaderogpeppe, ie I'm trying to handle what's currently possible sanely, not to make the whole assign/unassign functionality sane and usable at this stage15:13
fwereaderogpeppe, sorry gtg a mo15:13
rogpeppefwereade: k15:14
rogpeppefwereade: i'm trying to understand how a DeployedUnits can return the names of any units that weren't deployed by the context's deployer entity, given that the upstart job's name is tagged with that entity name.15:18
rogpeppefwereade: i.e. i think we could lose the "responsible" logic entirely, and we would never see any difference in behaviour.15:19
niemeyermramm: Hi15:19
mrammhey15:19
niemeyermramm: Back and forward are interesting when talking about time15:20
niemeyermramm: I tend to think that "moving forward" is making it later15:20
niemeyermramm: But that's obviously subjective :)15:20
mrammniemeyer: forward feels like pulling it towards me15:21
mrammand back like pushing it away15:21
niemeyermramm: TIme machines that move back in time tend to go to the past :)15:21
mrammniemeyer: true enough15:21
AramI never understood forward and backward when talking about time. it's especially confusing during daylight changes. "how did the clock move?", "one hour forward", "wtf".15:21
niemeyerAram: Yeah, I'm totally lost on those too15:22
niemeyerThere might be some cartesian thinking there too.. I tend to think of the T axis as having positive values on the right-hand side15:23
fwereaderogpeppe, sorry, I forgot I was talking to you :/15:35
fwereaderogpeppe, ok, what should happen when a unit is unassigned while the machine agent is down?15:35
rogpeppefwereade: when it comes back up, it should remove it.15:36
fwereaderogpeppe, how do we do that without the responsible logic?15:36
rogpeppefwereade: SimpleContainer doesn't return any units that weren't deployed by the responsible entity15:37
rogpeppefwereade: so, AFAICS, the responsible logic is redundant15:37
fwereaderogpeppe, I don't understand how that can be the case -- I could rearrange how the deployer is written, but I still need to handle that case, don't I?15:39
fwereaderogpeppe, and I still need to be able to get just the units I've deployed15:39
rogpeppefwereade: that's what SimpleContainer does, no?15:39
fwereaderogpeppe, precisely how I reconcile is open for discussion, but I picked what I did because it seemed simplest15:39
rogpeppefwereade: (by including the deployer name in the upstart job name)15:39
rogpeppefwereade: i may well have got the wrong end of the stick!15:40
fwereaderogpeppe, ah! got you! sorry, no, I did15:40
fwereaderogpeppe, how can we guarantee that a unit reported by WatchPrincipalUnits is still assigned to the machine that reported it?15:41
fwereaderogpeppe, all we know is that something changed wrt that unit15:42
rogpeppefwereade: the responsible logic isn't checking the assigned machine AFAICS15:42
fwereaderogpeppe, `responsible = deployerName == d.ctx.DeployerName()`?15:43
rogpeppefwereade: how does that tell us anything about the assigned machine?15:44
fwereaderogpeppe, what is deployerName?15:44
rogpeppefwereade: ah, gotcha.15:45
rogpeppefwereade: but if the unit *has* been assigned to a different machine, surely we'd want to kill it locally anyway>15:46
rogpeppe?15:46
fwereaderogpeppe, we want to recall it, but not to do anything else15:46
fwereaderogpeppe, if it's installed and we're not responsible, we recall it15:47
fwereades/installed/deployed/15:47
rogpeppefwereade: i don't like the "recall" name btw15:47
rogpeppefwereade: i think "kill" might be better15:47
fwereaderogpeppe, whoa please no15:47
fwereaderogpeppe, total sematic collision with ensuredead etc15:47
rogpeppefwereade: recall sounds like something fairly non-destructive15:47
niemeyerrogpeppe: Replied to the comments, thanks for the review15:48
fwereaderogpeppe, mmmaybe, although I think it is conceptually non-destructive -- put your mind 6 months in the future and consider, if the unit is unassigned and recalled, surely any persistent storage it uses will still be around ready for its new assignment15:50
rogpeppefwereade: rm -rf is pretty destructive, no matter how you look at it.15:50
fwereaderogpeppe, won't that actually just be "unmount" in the future though?15:51
niemeyerfwereade: I think unassignment is the same as a Dead unit, for all purposes of the deployer, except it shouldn't remove the unit from the state15:51
niemeyerfwereade: I don't think we even need a noun for that.. "the unit was unassigned" sounds fine15:52
fwereadeniemeyer, is there some way in which what I have implemented differs from that?15:52
niemeyerfwereade: I have no idea.. I'm commenting on top of the conversation above15:52
niemeyerfwereade: I'll try to review the branch today still, though15:52
niemeyerfwereade: I see you saying "we want to recall it", for example, and whether "kill" or "recall" is beter15:53
niemeyerbetter15:53
niemeyerfwereade: I think we want to remove the container, in the same way we do for a dead unit15:54
fwereadeniemeyer, yes15:54
fwereadeniemeyer, that is what I do15:54
fwereadeniemeyer, "recall" is the opposite of "deploy", in english, and it seems sensible to use words in juju to mean what they mean in english where possible15:55
niemeyerfwereade: Cool, I'm probably just misunderstanding the context then15:55
fwereadeniemeyer, the semantic intent is precisely "opposite of deploy"15:55
fwereadeniemeyer, this is *currently* a destructive operation15:55
niemeyerfwereade: That's what I was referring to15:57
niemeyerfwereade: I don't *think* we need a new noun for "remove the container"15:57
fwereadeniemeyer, but I don't think that "undeploying" a unit is necessarily always going to be so15:57
niemeyerfwereade: juju deploy is something pretty different15:57
fwereadeniemeyer, ok, we agreed that "deploy" was the term to use for "install a unit agent"15:57
fwereadeniemeyer, I implemented an Installer you told me was badly named, sight unseen, and now you're telling me that deploy is the wrong verb?15:58
niemeyerfwereade: Erm15:58
niemeyerfwereade: Sorry, please ignore me15:59
fwereadeniemeyer, np -- I think we do have terminology problems though, in hindsight you had no way of knowing what sort of "deploy" we were talking about :)15:59
niemeyerfwereade: I think we have too much terminology.. I don't know what "recall" actually means, and I was hoping I wouldn't have to learn.16:00
fwereadeniemeyer, it is the opposite of "deploy", and is used entirely within the deployer package to indicate the operation that reverses a deploy16:01
niemeyerfwereade: Okay16:02
fwereadeniemeyer, I'm not emotionally attached to the name, but "undeploy" just seemed overwhelmingly sucky so I spent some time with a dictionary :)16:02
fwereadeniemeyer, always happy to hear improvements :)16:03
rogpeppefwereade: remove?16:04
fwereaderogpeppe, collides with other remove operations in the package16:04
fwereaderogpeppe, words involving death collide with death-related operations used in the package16:04
rogpeppefwereade: RemoveDeployment ? :-)16:04
rogpeppeniemeyer: replied https://codereview.appspot.com/6868070/16:05
rogpeppeniemeyer: if you've got a little while at some point, i'd like to have a chat about the API stuff16:06
fwereaderogpeppe, is life *so* boring that we need to overload terminology to keep ourselves on our toes?16:06
niemeyerfwereade: recall is fine16:06
fwereaderogpeppe, (and what makes you think that recalling intrinsically "removes" anything? ;))16:07
fwereadeniemeyer, cheers16:07
niemeyerfwereade: I mainly wondered if we could avoid the term altogether.. maybe we can't16:07
niemeyerfwereade: (rather than suggested there was another term that would fit better)16:07
rogpeppefwereade: "os.RemoveAll(agentDir)" :-)16:07
fwereadeniemeyer, it's not something that anyone needs to know unless they're directly working withing the deployer package16:07
fwereaderogpeppe, and?16:08
fwereaderogpeppe, that is the only possible implementation?16:08
fwereaderogpeppe, we have no persistent storage features planned?16:08
rogpeppefwereade: we do, but they might not be always active.16:08
rogpeppefwereade: "Recall" sounds like it's fetching something16:09
rogpeppefwereade: but it's actually removing local state pertaining to the deployed unit16:09
fwereaderogpeppe, my point is that all your suggestions carry a "this is a destructive operation" payload, and I don't think that's necesarily appropriate16:09
niemeyerReally, recall is fine.. the discussion I was trying to avoid is already happening :)16:09
fwereadeniemeyer, ha, yeah, I will be quiet :)16:10
niemeyerWe all know what "recall" means now :)16:10
rogpeppeniemeyer: ok, if you're ok with it, i'll go with it too :-)16:10
fwereadeniemeyer, haha16:10
fwereaderogpeppe, cheers16:10
fwereadehuh, cath has done a super-early supper, I'll be back in a sec16:10
mrammI'm heading out to a lunch meeting, will be back in a bit.16:21
mrammif you need me in the meantime my phone is operational again, so feel free to call/text16:22
rogpeppeniemeyer: any chance of that chat some time today? (sorry to poke, but it would be nice to have an idea if i'm totally off in the wrong direction)16:50
niemeyerrogpeppe: Oh yeah, thanks for poking, I forgot actually16:50
niemeyerrogpeppe: What's up?16:50
rogpeppeniemeyer: i did a spike branch, just to see if some ideas would work out16:50
rogpeppeniemeyer: it's only a sketch in quite a few places, but it does give an idea of how it might work, end-to-end: https://codereview.appspot.com/6878052/16:51
rogpeppeniemeyer: crucial pieces are:16:51
rogpeppehttp://bazaar.launchpad.net/+branch/~rogpeppe/juju-core/176-rpc-spike/view/head:/state/api/serve.go16:51
rogpeppehttp://go.pkgdoc.org/launchpad.net/~rogpeppe/juju-core/176-rpc-spike/rpc#NewServer16:52
rogpeppeniemeyer: the former is how the API surface is implemented at by the server. the latter is the piece of magic that makes it possible.16:52
rogpeppes/at by/by/16:53
niemeyerrogpeppe: Well, there are thousands of lines in there.. I can't really comment on it without sitting down and reading what's going on16:53
rogpeppeniemeyer: that's why i pointed you at the NewServer docs16:54
rogpeppeniemeyer: which should give an idea, even without looking at the implementation16:54
niemeyerrogpeppe: A Server with an Accept method.. yeah, that sounds fine :-)16:54
rogpeppeniemeyer: the idea is that a single API implementation can provide access to both http and connection-based RPC interfaces.16:55
rogpeppeniemeyer: i've modelled the rpc package quite strongly on net/rpc where appropriate16:56
rogpeppes/on/after/16:56
niemeyerrogpeppe: That kind of stuff is a bit dubious: "If a path element contains a hyphen (-) character, the method's argument type T must be string, and it will be supplied from any characters after the hyphen."16:58
rogpeppeniemeyer: that's so you can use a path like: /State/Machine-0/InstanceId16:59
niemeyerrogpeppe: There seems to be quite a bit of generality built on that interface16:59
niemeyerrogpeppe: I'd feel more comfortable with an API that was highly tailored for our specific problem16:59
niemeyerrogpeppe: with one encoding, that is known sensible for all use cases we have in mind17:00
niemeyerrogpeppe: we're not building net/rpc, in that sense17:00
rogpeppeniemeyer: i think we want to provide both http access and RPC-style access17:00
niemeyerrogpeppe: We discussed this over UDS, I think17:00
niemeyerrogpeppe: The strawman was attempt to have https only17:01
niemeyerrogpeppe: and see where that would lead17:01
rogpeppeniemeyer: even if we don't provide access via multiple protocols, i still think this package is a good way to do it17:02
niemeyerrogpeppe: Having stuff like XML there makes it pretty clear that we're over-engineering it, and stumbling upon that while trying to tailor it to our needs17:02
niemeyerrogpeppe: (as the /State/Machine-0/InstanceId edge case indicates)17:02
rogpeppeniemeyer: the XML stuff is like 10 lines of code17:02
niemeyerrogpeppe: It could be 217:02
niemeyerrogpeppe: Or one17:02
niemeyerrogpeppe: It's still XML, and we don't want to use that17:02
rogpeppeniemeyer: sure. i just put it in there because it only took 10s to type17:03
niemeyerrogpeppe: EC2 is a lesson in that sense.. there are two interfaces17:03
niemeyerrogpeppe: And no body but the heaviest Java pundits use that17:03
rogpeppeniemeyer: http://paste.ubuntu.com/1412851/17:03
niemeyerrogpeppe: You see my underlying point, though?17:04
niemeyerrogpeppe: It took 10 seconds to put it in there because there's a *LOT* of logic to make that possible17:04
rogpeppeniemeyer: so do you think we should build it as a standard web server, with no websockets?17:04
rogpeppeniemeyer: i hadn't thought we'd decided that at UDS.17:05
niemeyerrogpeppe: Erm.. how did websockets get into the above context?17:05
rogpeppeniemeyer: well, there's a significant difference between serving GET requests and serving messages from a websocket17:05
niemeyerrogpeppe: I bet.. still.. !?17:06
rogpeppeniemeyer: i'd be happy if we could do both.17:06
niemeyerrogpeppe: Ah, I see17:06
niemeyerrogpeppe: Rather than designing two entirely different API mechanisms at once, it feels like we could try to get one of them right17:07
rogpeppeniemeyer: this package means we can serve them both with the same implementation17:08
niemeyerrogpeppe: Understood, the comments above take that into account17:08
rogpeppeniemeyer: agreed. i'd start with the websocket implementation, in that case.17:08
rogpeppeniemeyer: and i *still* think this would be a useful package for that.17:09
rogpeppeniemeyer: because net/rpc is not sufficient17:09
niemeyerrogpeppe: I'm not sure.. I'd start with the http package, and the websocket package17:09
rogpeppeniemeyer: so where do we deal with the message processing?17:09
rogpeppeniemeyer: this rpc package is my way of trying to abstract that ou17:10
rogpeppeout17:10
rogpeppeniemeyer: in the same way that net/rpc does, but a little more general17:10
rogpeppeniemeyer: in that it allows rpc calls to know the context of the connection that's making the call17:10
rogpeppeniemeyer: i *think* it makes for a very natural style of implementation of the API, while keeping all the RPC gunge out of the way.17:13
niemeyerrogpeppe: I'm trying to understand what that would mean, but I can't see that yet17:14
rogpeppeniemeyer: did you look at the server implementation? http://bazaar.launchpad.net/+branch/~rogpeppe/juju-core/176-rpc-spike/view/head:/state/api/serve.go17:14
niemeyerrogpeppe: It seems that most of whta is there is a generic mechanism to plug different codecs17:14
niemeyerrogpeppe: Without any correlation to the needs of the underlying infrastructure17:14
rogpeppeniemeyer: you could delete all the codec stuff and it would still be useful17:14
rogpeppeniemeyer: for instance we could hard-code JSON17:15
rogpeppeniemeyer: that would be fine17:15
niemeyerrogpeppe: What are we getting out of it then?17:16
niemeyerrogpeppe: http://gary.beagledreams.com/page/go-websocket-chat.html17:16
niemeyerrogpeppe: This is where I would start17:16
rogpeppeniemeyer: i've written something that sends and receives messages over websockets already17:17
niemeyerrogpeppe: It seems a few orders of magnitude more straightforward than the logic that is in that rpc package17:17
niemeyerrogpeppe: I'm not saying that stuff is impossible to do with your implementation.. I bet you can do anything with it. :-)17:17
rogpeppeniemeyer: but we don't want to manually marshal and unmarshal messages for every single call in the API17:18
niemeyerrogpeppe: No, we don't.. we also don't want to implement a generic net/rpc package that is even more generic than net/rpc17:18
niemeyerrogpeppe: There's a sweet spot in between, and hopefully that sweet spot is actually trivial to implement17:18
rogpeppeniemeyer: well, it *needs* to be more generic than net/rpc17:19
rogpeppeniemeyer: because net/rpc doesn't provide what we need17:19
niemeyerrogpeppe: It doesn't have to be generic at all, IMHO.. it has to solve our problem, our very specific problem.. not anyone else's problem17:19
rogpeppeniemeyer: i thought it was quite neat :-(17:19
niemeyerrogpeppe: It seems super neat, and a lot of people will love that I think..17:20
niemeyerrogpeppe: But we have a smaller issue to solve17:20
rogpeppeniemeyer: agreed, but our API surface is fairly broad. i was trying to think of something just sufficiently broad to solve our problem.17:20
niemeyerrogpeppe: Okay, *that* is exactly the problem17:21
niemeyerrogpeppe: I suggest we start by *having* the problem, and walking up from there17:21
rogpeppeniemeyer: where do you think i should start then?17:21
niemeyerrogpeppe: By implementing one tiny API call that actually works17:22
niemeyerrogpeppe: Without significant infrastructure to support everything we think we may need for everything we think we'll do17:22
rogpeppeniemeyer: ok, so don't use net/rpc at all then.17:23
niemeyerrogpeppe: Right17:23
rogpeppeniemeyer: ok, i'll start there. and i still think that what i've just made will be a good fit for what we'll need, but i take your point that it's perhaps overly general.17:24
rogpeppeniemeyer: i built something that would make building what we need fairly trivial, which i thought was a reasonable approach.17:25
niemeyerrogpeppe: I reserve my right to doubt that this net/rpc replacement makes building what we need fairly trivial.17:26
rogpeppeniemeyer: but still, only three days lost.17:26
niemeyerrogpeppe: We have enough track record about "fairly trivial" changes by now :)17:27
rogpeppeniemeyer: yeah, "fairly trivial" means "i can see a clear path" :-)17:27
niemeyerrogpeppe: Exactly, a path to somewhere.. but given that we don't have a working API, and are not even close to having one, we don't know where that path really leads17:28
niemeyerrogpeppe: There's a lot of value in putting stuff to actually work, and learning about what it takes to do that17:28
rogpeppeniemeyer: well, that branch does include at least one test from the state package that passes.17:28
rogpeppeniemeyer: which seems not a million miles away from having a working API17:29
niemeyerrogpeppe: In 20 minutes I can code a websocket that talks across the wire to the state package17:30
niemeyerrogpeppe: I'd still consider myself completely ignorant about what it takes to make all the juju communication work across the wire17:30
niemeyerrogpeppe: But that's just me.. perhaps I'm just pessimistic about myself.17:31
rogpeppeniemeyer: in a few days, i reckon i could get most of the non-watcher state tests passing. but perhaps that would still be "not even close".17:36
niemeyerrogpeppe: That's great to hear.. maybe we can have the API working even before christmas then17:39
rogpeppeniemeyer: without some equivalent of net/rpc, i'm not sure.17:39
rogpeppeniemeyer: oh, sorry, that was sarcasm, right?17:43
niemeyerrogpeppe: Not at all, I was seriuos17:47
niemeyerrogpeppe: If you can get everything all of state in three days, it doesn't feel off-the-park to have most of it in 2.5 weeks17:47
rogpeppeniemeyer: i could only do that because my rpc package makes it straightforward17:48
niemeyerrogpeppe: heh17:48
rogpeppeniemeyer: because all the bookkeeping is hidden away17:48
niemeyerrogpeppe: I'm only suggesting doing what we actually need, rather than what we need and what we don't17:48
niemeyerrogpeppe: There's nothing about websockets there, there's nothing about auth there, there's nothing about a lot of stuff there17:49
rogpeppeniemeyer: websockets are trivial to add, and so is auth17:49
rogpeppeniemeyer: i designed it with both in mind17:49
niemeyerrogpeppe: Can you explain how websockets is trivial to add there and is so hard without it?17:50
niemeyerrogpeppe: Because this: http://gary.beagledreams.com/page/go-websocket-chat.html, is surprisingly similar to what we want17:50
niemeyerrogpeppe: and it has a fraction of the size17:50
rogpeppeniemeyer: most of the code that we will be writing will be working out what call has been made, marshalling and unmarshalling parameters, and checking authorisation.17:51
rogpeppeniemeyer: not dealing with websockets directly17:51
rogpeppeniemeyer: that example has only a single message type17:51
rogpeppeniemeyer: and no authorisation17:52
rogpeppeniemeyer: we've got about 60 calls in our API used outside of state17:53
niemeyerrogpeppe: http://go.pkgdoc.org/code.google.com/p/go.net/websocket17:53
niemeyervar JSON = Codec{jsonMarshal, jsonUnmarshal}17:53
niemeyertype T struct {17:53
niemeyerMsg string17:53
niemeyerCount int17:53
niemeyer}17:53
niemeyerrogpeppe: You see the similarity in terminology?17:53
niemeyertype Codec struct {17:54
niemeyer    Marshal   func(v interface{}) (data []byte, payloadType byte, err error)17:54
niemeyer    Unmarshal func(data []byte, payloadType byte, v interface{}) (err error)17:54
niemeyer}17:54
niemeyerfunc (cd Codec) Send(ws *Conn, v interface{}) (err error)17:54
rogpeppeniemeyer: it can encode and decode json messages... and?17:54
niemeyerrogpeppe: and!?17:55
rogpeppeniemeyer: that's a fraction of what we need to do17:55
niemeyerrogpeppe: Exactly, that's all pretty close to what that package implements17:55
niemeyerrogpeppe: Codecs, marshalling, unmarshalling17:55
niemeyerrogpeppe: There's missing logic.. let's add it17:55
rogpeppeniemeyer: yes, and none of that is implemented by my rpc package17:55
rogpeppeniemeyer: it just uses codecs from outside17:55
niemeyerrogpeppe: Okay, sorry, if you think state is going to be ready before the end of the week, please go for it.17:56
niemeyerrogpeppe: We're clearly talking across each other without much benefit17:56
niemeyerrogpeppe: If state isn't ready by the end of the week, we can talk again17:56
rogpeppeniemeyer: i will implement a direct websocket implementation to show where we might be headed without something like net/rpc.17:56
TheMueYiiiiiiiiehaaa! After a wrong direction and fiddling why that breaks I think the firewaller is stable now. I'll add some explicitely stressing tests tomorrow, but it looks good. *phew*17:57
rogpeppeniemeyer: and then hopefully it'll be more clear what the advantages are17:57
niemeyerrogpeppe: Please don't do that.. I'm not interested in you coding logic to prove how bad it will look like17:57
rogpeppeniemeyer: in which case i don't know what to do17:57
rogpeppeniemeyer: i'd try to make it look as nice as possible17:58
niemeyerrogpeppe: I was hoping to collaborate to a working prototype, without creating massive infrastructure which is obviously necessary, despite the fact we have no idea17:58
niemeyerrogpeppe: The freaking thing supports XML for god's sake17:58
rogpeppeniemeyer: only because it was ultra-trivial to do17:58
rogpeppeniemeyer: (and actually it doesn't work :-])17:58
niemeyerrogpeppe: Yeah, it's trivial to put a car within an airplane, once you have the airplane17:59
* rogpeppe should have known that typing the letters X-M-L was a mistake17:59
niemeyerrogpeppe: But this isn't being productive.. there's no point in you working on stuff you're unhappy about17:59
rogpeppeniemeyer: i think that putting in the infrastructure for the websocket stuff from the bottom up would actually be a good thing18:00
niemeyerrogpeppe: My personal opinion is that the current super-net/rpc package is a pretty trivial part of the problem we have to solve, and I'd prefer to focus on the problem itself rather than trying to put yet another layer we don't even understand18:00
niemeyerrogpeppe: But there's an easy way out: you said it's going to be ready in three days, so go for it.18:01
rogpeppeniemeyer: then when we've sent a few messages back and forth, we can see if something like my rpc package might be a good fit.18:01
niemeyerrogpeppe: Won't be much time spent, and will be a learning exercise anyway.18:01
niemeyerrogpeppe: I think there's a non-trivial number of pieces missing, that will guarantee thousands of additional lines18:01
rogpeppeniemeyer: it's a spike. even if i get things working, it's still a spike.18:01
rogpeppeniemeyer: out of interest, what pieces are you thinking of particularly?18:02
rogpeppeniemeyer: i'm wondering where you see the blocking points18:02
niemeyerrogpeppe: Well, it's not a code spike.. a spike is something we code to understand a problem. This is a pretty polished API that is documented and that you're attached to already18:03
rogpeppeniemeyer: it really was code to try to understand how i might solve the problem18:03
rogpeppeniemeyer: even if i did write some doc comments18:03
rogpeppeniemeyer: (which i actually wrote before i wrote the code, so i had an idea of what i was trying to do)18:04
rogpeppeniemeyer: and it's true i think it turned out quite nice, but there y'go.18:05
rogpeppeniemeyer: anyway, gotta go.18:05
rogpeppeniemeyer: see you tomorrow18:05
niemeyerrogpeppe: See ya18:05
niemeyerThis reminds me of Launchpad's API all over again18:06
niemeyerrogpeppe: That sounds like a side effect of that design, for example:18:15
niemeyer 102         path := fmt.Sprintf("/Machine-%s/SetInstanceId", m.Id)18:15
niemeyer 103         return m.state.c.Call(path, instId, nil)18:15
niemeyerrogpeppe: The natural design on that is to have the equivalent of {"method": "set-instance-id", "machine": m.Id, "instance-id": instId}18:16
niemeyer(disconsidering marshalling issues.. that's a map in whatever format we decide to us)18:17
niemeyeruse18:17
niemeyerPossibly with an additional "request-id" key too18:18
niemeyerI guess I suck at explaining.. I should try to do what I do best and code it18:24
* niemeyer => doc appointment.. back later18:49
fwereadeniemeyer, ping21:44
fwereadeniemeyer, in case you come back, please consider what the Dying state might mean for a Machine; I'm starting to believe it's meaningless (setting a machine to Dying should not, I think, affect its units; so a Dying machine generally just sits there forever (or at least until all its units are unassigned; but having a long-running machine which cannot have further units assigned feels like crack, as does the alternative of allowing units to be assigned t23:10
fwereadeo Dying machines23:10
fwereade)23:10
fwereadeniemeyer, grar, just thought of something: there's an additional constraint on subordinate relations, which is that the principal and the subordinate charms must have the same series23:13
* fwereade will stop dumping random thoughts into the channel and go to bed23:14

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!