/srv/irclogs.ubuntu.com/2012/09/10/#juju-dev.txt

davecheneyshitballs, I deleted by ssh branch before pushing it03:26
davecheneycrap -- have to start again03:26
wrtpdavecheney: aw shit, i've managed to avoid doing that so far, but i've come close06:36
wrtpdavecheney: morning BTW!06:37
wrtpfwereade: hiya06:37
fwereadewrtp, heyhey06:38
wrtpdavecheney: i wanted to use that branch too06:38
fwereadedavecheney, morning06:38
fwereadewrtp, sorry I haven't reviewed it properly yet, I'll do that right now06:39
wrtpfwereade: np. that would be great, ta!06:39
davecheneywrtp: s'ok, rewritten already06:41
wrtpfwereade: is there are reason that state.Initialize doesn't take a *config.Config ?06:41
wrtps/are/a/06:41
davecheneywrtp: maybe that is my failt06:41
davecheneyfault06:41
fwereadewrtp, I don't think so, no06:41
davecheneywrtp: also, since I patched that issue in the TLS lib06:41
fwereadewrtp, ie I think it probably should06:41
davecheneynot a single EOF from etc06:41
wrtpfwereade: i was just looking at if len(c.EnvConfig) == 0 and thinking that it should probably be checking that the config was accepted by the config package06:42
davecheneyalso, if you are following tip, cgo is broken again, http://codereview.appspot.com/6501101/06:42
davecheney^ apply this for general satisfaction06:42
wrtpdavecheney: i haven't followed tip for a while now, as i haven't been working on go core stuff. there is something i've been meaning to do though, and i may have some time in the next few days as carmen's away for a bit.06:44
davecheneywrtp: some awesome imrpovements to 6g coming06:45
davecheneyat least 15% improvement across the board06:45
davecheneyand substantial reduction in stack size06:45
wrtpdavecheney: which bit are you referring to?06:45
wrtpdavecheney: (i've been following golang-dev, but skimming it and missing quite a bit)06:45
davecheneyhttp://codereview.appspot.com/6501110/06:46
davecheneyhttp://codereview.appspot.com/6494107/06:46
davecheneyand so forth06:46
davecheneyfiltering out all the LEAQ's06:46
davecheneyissue 191406:46
wrtpdavecheney: great. yeah, there's so much potential optimisation.06:47
wrtpdavecheney: i'm really looking forward to russ's Grand Optimisation branch whenever it might arrive06:47
davecheneywrtp: this one is nice too http://codereview.appspot.com/6506089/06:47
davecheneythe big improvements are in stack usage06:49
davecheneyreflect.Value.call consumes a kilobyte of stack on x64!06:49
wrtpdavecheney: that will help a lot06:49
wrtpdavecheney: woah06:49
davecheneyremy has shaved 1/4 off that06:49
davecheneybut it's stupid large06:49
davecheneyso any time that method comes near you, it's a stack split for sure06:50
davecheneyand an expensive one06:50
wrtpdavecheney: i'm not too surprised - it's a complex piece of code06:50
davecheneynah, it's just long06:50
wrtpdavecheney: things will be better when the runtime knows what size of stack is likely to be necessary.06:50
davecheneyif it was split up06:50
davecheneythe stack could be reused because it works top to bottom06:50
davecheneywrtp: i hope that is more than a pipe dream06:51
wrtpdavecheney: i don't see why it shouldn't work. a best-case stack size calculation should not be out of reach06:51
davecheneywrtp: lp:~dave-cheney/juju-core/093-cmd-juju-ssh/06:51
wrtpdavecheney: another optimisation i thought would be nice is to cache the function value from the itab when an interface value doesn't change.06:53
davecheneywrtp: daniel morosing has been working on that one for a while06:53
wrtpdavecheney: that could save quite a bit in tight loops calling interfaces06:53
davecheneywrtp: http://codereview.appspot.com/6351090/ ?maybe?06:54
wrtpdavecheney: i don't think so06:55
wrtpdavecheney: that's about interface type conversion; i'm just talking about interface calling06:55
davecheneywrtp: does taht require runtime support ?06:55
davecheneyi'm guessing it does, but for some reason I can't think of where it is in the runtime06:56
wrtpdavecheney: i don't think so06:56
wrtpdavecheney: the compiler generates the code to call methods06:56
davecheneywhich must take the interface type, and look it up in a table to get the func addr of the i impl06:57
davecheneyanyone heard from mark lately ?06:58
fwereadewrtp, sorry, missed you; you may be right, I was trying to avoid getting sidetracked into fixing too much in one CL though :)06:59
fwereadewrtp, reviewed https://codereview.appspot.com/6501106/06:59
wrtpdavecheney: yeah, probably best in another CL06:59
wrtpfwereade: thanks!07:00
wrtpfwereade: i used VarDir because it was niemeyer's suggestion originally ("all directories are juju directories")07:00
fwereadewrtp, you probably won't agree with it all07:00
wrtpfwereade: but i'd personally be very happy to use JujuDir.07:00
wrtpfwereade: in fact i agree it's a better name07:01
fwereadewrtp, cool :)07:01
fwereadewrtp, we'll have to see how it does then :)_07:01
fwereadewrtp, JujuRoot, perhaps?07:01
wrtpfwereade: how about Machiner.simpleContainer ?07:02
fwereadewrtp, (btw, sorry, I feel the review comments skewed rather negative, the "lots to like" is the overriding impression though)07:02
wrtpfwereade: we can't stick with container.Simple as a global07:02
wrtpfwereade: as we need to be able to pass in VarDir07:03
fwereadewrtp, well, outside of testing, when do we need to do that?07:03
wrtpfwereade: always07:03
wrtpfwereade: there's no global VarDir any more07:03
wrtpfwereade: and you can invoke commands with --juju-dir07:04
wrtpfwereade: (which works now, BTW)07:04
fwereadewrtp, hmm, I think I have an odd perspective on this, but I did a bunch of agent stuff over the w/e, and I'm (say) 90% sure that an Agent type is (1) a good thing and (2) the appropriate home for JujuDir07:05
fwereadewrtp, so ISTM that it's not actually the container's responsibility (but that initDir and logDir are)07:05
wrtpfwereade: i'm not sure what you mean by "the ... home for JujuDir"07:06
wrtpfwereade: it's just a parameter07:06
fwereadewrtp, it's a parameter used in several places, all of which IMO become nicer once an Agent type is added07:07
wrtpfwereade: so an Agent would be a parameter to the Container?07:08
fwereadewrtp, perhaps it would help if I propose -wip my current branch (which is very very unfinished but I hope instructive)07:08
fwereadewrtp, I *think* so, yes, it comes out quite nice IMO07:08
wrtpfwereade: i'll maybe give you a glance at another unfinished branch of mine to get your thoughts too07:10
fwereadewrtp, cool -- I'm a little worried that we may be about to collide actually, a lot of the ugliness in https://codereview.appspot.com/6500095 will evaporate once your VarDir branch is merged07:11
fwereadewrtp, the underlying insight that led me in this direction is that all the agents really *are* the same in that they should all be named, because they do in fact all correspond to a single state entity, and that messing around with --machine-id and --unit-name and not-having-one-for-provisioning is actually a side-effect of missing this insight07:15
fwereadewrtp, ofc it may, as always, be crack07:15
wrtpfwereade:  i'm absorbing it07:16
wrtpfwereade: i was thinking about something a little similar in some ways actually07:16
wrtpfwereade: i wondered if we could have a single "jujud agent <agent-name>" command07:16
fwereadewrtp, I've been keeping that out of mind07:17
fwereadewrtp, because I think it is *probably* right but that we are not quite there yet07:17
fwereadewrtp, once we have upgraders for everything, and (if it passes muster) agent.Agent, it will I think be a good time to move the upgrading task out of the Kind-specific tasks, and make it all happen at the Agent level07:18
wrtpfwereade: the one thing i'm not sure about is having a single agent.Run for all the agents.07:19
fwereadewrtp, ISTM that the list-of-tasks abstraction is an excellent one07:19
wrtpfwereade: i think that's the kind of direction i was heading with my "runner" package and it was deemed crackful07:19
fwereadewrtp, bah07:19
fwereadewrtp, my feeling is that list-of-tasks is *exactly* the thing that differentiates multiple agents07:20
wrtpfwereade: it depends whether we think that *all* agents *always* will be a simple set of concurrent tasks07:20
fwereadewrtp, that STM to be the assumption underlying the worker package, and it seems to have served us well so far07:21
wrtpfwereade: i'm not sure that's true actually07:21
wrtpfwereade: the workers can be used in any way07:21
wrtpfwereade: they *happen* to implement the same interface, but there's no requirement that they do so07:22
wrtpfwereade: i do like the the factoring-out of the UpgradedError logic though.07:24
wrtpfwereade: BTW why does this code deserve its own package? it seems like it would still live well in cmd/jujud07:24
wrtpfwereade: ah! but you want to pass Agents around to other packages.07:25
fwereadewrtp, yep, exactly07:25
fwereadewrtp, like I say, might all be crack, but it seems like a small amount of code that is useful in several places07:25
wrtpfwereade: i'm not sure. i think that simply passing JujuDir as a parameter work well for many of the methods07:26
wrtps/work/works/07:26
wrtpfwereade: and agent name when appropriate, i guess07:27
fwereadewrtp, and agent kind...07:27
fwereadewrtp, and frequently state info...07:27
wrtpfwereade: when do we need agent kind?07:27
fwereadewrtp, any time we want to get a tools dir (according to the agent-foo-123, provisioning-whatever naming scheme I thought we discussed on friday)07:28
wrtpfwereade: that comes from the agent name, no?07:28
fwereadewrtp, ah, ok, sorry, I misunderstood which name you were talking about07:28
wrtpfwereade: i'm thinking that every agent has a unique name07:29
fwereadeer sorry s/agent-foo/unit-foo/07:29
wrtpfwereade: i think i'd be happier if the agent package had stuff for agent identification and location only, and all the *actual* agent logic lived elsewhere.07:30
fwereadewrtp, I am also thinking that, but I've called it "badge" because ISTM that the *name* is the name of the attached state entity, and the agent itself is somewhat different07:30
wrtpfwereade: when would you need "name" instead of "badge"?07:30
fwereadewrtp, if one takes the Agent abstraction seriously it becomes useful to pass a *Agent into (eg) uniter, instead of the name/dir bits, and then the unit name is directly accessible from there07:32
wrtpfwereade: but the uniter knows the *Unit? and the name comes from that, no? or perhaps the name is something else and i'm misunderstanding07:32
fwereadewrtp, no, the Uniter has never been started with a *Unit07:33
wrtpfwereade: why not, out of interest?07:33
wrtpfwereade: it would seem logical07:33
fwereadewrtp, there's no reason to?07:33
fwereadewrtp, we have available a state and a name, and we need the state anyway07:34
fwereadewrtp, it's just smearing the state setup across the uniter and its client to no apparent benefit07:35
fwereadewrtp, s/it's/pregetting the unit is/07:35
wrtpfwereade: i think that when the uniter does upgrades, the caller of the uniter will need to get the unit anyway07:36
wrtpfwereade: because it needs to be passed into the upgrader.07:36
wrtpfwereade: same as the MA07:36
fwereadewrtp, sure, but the MA takes the wrong params itself07:36
fwereadewrtp, where's the state info?07:36
fwereadewrtp, (another Agent field, you'll notice :))07:37
wrtpfwereade: i'm not sure. these are things that individual agents need, but it feels you're making them into universal parameters, and i'm not sure that's necessary.07:38
wrtpfwereade: and, um, i think that's a red herring in fact.07:38
wrtpfwereade: we'll still need to get the *Unit before calling the Uniter07:39
fwereadewrtp, go on?07:39
wrtpfwereade: so we may as well pass the *Unit into the uniter07:39
fwereadewrtp, is it just that we want one to create the upgrader?07:40
wrtpfwereade: yes. but if we've already got one, passing it into the Uniter seems like a fine thing. why pass in a name when you've already got the thing itself?07:40
wrtpfwereade: BTW with your current arrangement, you *can't* pass a *agent.Agent into the Uniter07:42
fwereadewrtp, ISTM like a nicer interface to pass a *State and a name than a *State and a *Unit07:42
fwereadewrtp, you just talking about package dependencies?07:42
wrtpfwereade: yeah07:42
fwereadewrtp, yeah, doesn't feel insurmountable to me07:42
wrtpfwereade: why a name nicer than the thing it's naming?07:43
wrtpfwereade: i *think* it's inherent to this approach. the Agent calls the worker, which needs the Agent. cyclic dependency.07:43
wrtps/why a/why is a/ :-)07:44
fwereadewrtp, surely it's just a matter of re-extracting an agent conf type and passing that around?07:44
wrtpfwereade: so then we have *another* package, just to contain that type?07:45
fwereadewrtp, which probably helps to address the run-mixed-with-info concerns07:45
wrtpfwereade: why not factor out all the run stuff, and make the agent type *solely* concerned with agent storage?07:45
wrtpfwereade: then there's no problem07:45
fwereadewrtp, I think that's the type that holds most of the methods -- it's just Run and Stop that move elsewhere07:46
wrtpfwereade: agreed07:46
wrtpfwereade: in the end, i think you've got something like: type Agent {Name string; JujuDir string}07:47
fwereadewrtp, what about kind and state info?07:47
wrtpfwereade: (i *think* putting the StateInfo in there is mixing concerns)07:47
fwereadewrtp, name me an agent that doesn't need a state info07:47
wrtpfwereade: the firewaller :-)07:48
wrtpfwereade: i know it's not its own agent, but it could be07:48
fwereadewrtp, so is your contention that if 4 things use something, and a 5th doesn't, it is apropriate to write 5 separate code paths rather than suffer the shame and indignity of an unused param? ;p07:49
wrtpfwereade: it's not really a matter of "which agent's don't need it" but "why is it living in this package, which is actually only to do with agent storage?"07:49
wrtpfwereade: i don't see that this would require any extra code paths07:49
fwereadewrtp, ah, hmm, I don;t think it is just concerned with agent storage, I think Conf is an imortant part07:49
wrtpfwereade: Conf?07:50
fwereadewrtp, Agent.Conf, which means we can write a Deploy wethod that works07:50
fwereadewrtp, well, in concert with other things, it does07:50
fwereadewrtp, rather than having the crazy doesn't-even-work-and-is-not-consistent-with-cloudinit Deploy we currently have07:51
wrtpfwereade: an explicit StateInfo param to Deploy seems quite reasonable to me07:51
wrtpfwereade: i'm already most of the way through fixing that07:51
fwereadewrtp, and to the machine agent, and to the unit agent, and (I presume) to the provisioning agent as well?07:52
wrtpfwereade: absolutely.07:52
fwereadewrtp, sorry s/agent/worker/g07:52
wrtpfwereade: i don't think an extra parameter is a problem07:52
wrtpfwereade: especially as it makes it obvious that this worker connects to the state07:53
wrtpfwereade: it would *not* be a parameter to the upgrader or to the firewaller07:53
fwereadewrtp, what? that's not why we pass it at all07:53
wrtpfwereade: no?07:53
wrtpfwereade: why do we pass it?07:53
fwereadewrtp, we pass it to the things that themselves need to set up new agents one way or another07:53
wrtpfwereade: indeed - things that need to connect to the state07:54
fwereadewrtp, the *State is handled outside and shared by many worker, and that really does need to be passed to everything07:54
fwereadewrtp, an MA doesn't connect to the state07:54
fwereadewrtp, sorry a Machiner07:54
wrtpfwereade: it starts things that do07:54
fwereadewrtp, yeah, exactly07:55
wrtpfwereade: so, indirectly, yes, it does.07:55
fwereadewrtp, I would like it if we used a definition of "connects to the state" that involved, y'know, opening a connection to the state ;p07:55
wrtpfwereade: lol07:55
fwereadewrtp, but yes, anyway, I see your perspective07:56
wrtpfwereade: i think of it like a capability07:56
fwereadewrtp, this is kinda by the by anyway07:57
fwereadewrtp, I'm explicitly *not* proposing a common worker creation interface taking an agent, because I felt it would lead to derails ;)07:58
wrtpfwereade: i'm just objecting to the fact we're stuffing StateInfo inside the agent package, when *nothing* in the agent package uses it. it's solely to avoid us passing an extra parameter.07:58
fwereadewrtp, Conf uses it...07:58
fwereadewrtp, and IMO it's a really nice thing to be able to take the exact same object and run it, or install it, or generate the scripts required to install it07:59
fwereadewrtp, it feels like that's what an agent "is"07:59
fwereadewrtp, you need exactly the same information to run one, and to generate a conf for it07:59
wrtpfwereade: i'm not sure that Conf lives inside agent.08:00
wrtpfwereade: i think it lives inside container.08:00
wrtpfwereade: i've had some thought as to how container should work08:00
fwereadewrtp, interesting08:01
wrtpfwereade: that's what i was planning to work on this morning08:01
wrtpfwereade: but your thought of "to be able to take the exact same object and run it, or install it, or generate the scripts required to install it" is exactly where i was coming from.08:02
fwereadewrtp, if your position is that Container is a good place for this then I am very happy to stand back and let you get on with it08:02
wrtpfwereade: i was planning to filch a pattern of yours from the unit agent testing, which i rather liked08:02
fwereadewrtp, because my spidey-sense kept saying "use container in cloudinit", but I couldn't figure out how to08:03
wrtpfwereade: i *think* it is08:03
wrtpfwereade: that is my plan08:03
fwereadewrtp, sweet08:03
wrtpfwereade: the plan is for container to be able to generate shell scripts as well as running Deploy08:03
fwereadewrtp, ok, I just need to figure out what I can do that doesn't conflict with you08:03
wrtpfwereade: ok, sorry, i didn't realise you were so deep into this area08:04
wrtpfwereade: the global VarDir was a prelude08:04
fwereadewrtp, because, well, I really want to make the UA just run a Uniter, but the prospect of essentially writing more duped tests inside jujud bugged me enough to go looking for abstractions :)08:04
fwereadewrtp, I think we have actually been for a couple of days, but I think we're coming at the problem from opposite ends so it hasn't been entirely apparent08:05
wrtpfwereade: yeah, i know what you mean. but i think we can write shared tests without shoehorning them all into the same code.08:05
fwereadewrtp, it is extremely reassuring to me that we seem to be in broad agreement about the general problems depsite our differing perspectives :)08:06
wrtpfwereade: agreed08:06
wrtpfwereade: like two ends of an inductive proof coming together...08:07
fwereadewrtp, yeah :)08:07
wrtpfwereade: now we just have to make the terms match08:07
fwereadewrtp, the trouble is that getting a running Uniter is blocked on getting Container to run a unit agent08:09
wrtpfwereade: ok, i'll try and get it out very soon08:09
fwereadewrtp, I guess I can at least just write a really dumb unit agent + test, on top of your VarDir branch08:09
fwereadewrtp, that should be pretty independent08:10
wrtpfwereade: sounds like a reasonable way to go08:10
fwereadewrtp, cool08:10
fwereadewrtp, I look forward to seeing what you do with container08:11
Arammoin.08:13
wrtpfwereade: here's the likely crackful branch i alluded to earlier. it gives us PA upgrading. but at what cost? what d'ya think? https://codereview.appspot.com/6493101/08:13
fwereadewrtp, haha, I'll take a look08:18
fwereadewrtp, I think I'm -1 on that, it feels like too much special-casing in state08:24
fwereadewrtp, I think it would be better to either charm the provisioner (which I don;t think is a good use of our time right now) or to build the provisioner-deploying directly into machiner08:25
fwereadewrtp, I thought it was agreed a while ago that we'd be adding a provisioner field to state.machine somehow, and I was expecting to use that (in the absence of a nice charmy way to do it)08:26
fwereadewrtp, (and *that* then makes me think that, hell, the MA itself should probably just run the PA tasks if it's configured to be a provisioner machine, and drop the whole idea of a separate agent08:27
fwereade)08:27
wrtpfwereade: i wanted to go in that kind of direction, but niemeyer thinks that the PA should have its own entity in state08:28
fwereadewrtp, foiled again :(08:28
wrtpfwereade: which implies loads more mechanism08:28
fwereadewrtp, indeed08:28
wrtpfwereade: which i'm really reluctant to do, because it's actually *identical* to what Unit does08:29
wrtpfwereade: except there's no charm for an agent08:29
wrtpfwereade: hence my AgentService08:29
fwereadewrtp, yeah, I understand, I just think it's basically inferior to a `provisioner bool` field in state.Machien08:30
fwereadewrtp, or at least the underlying state if not that type08:30
wrtpfwereade: except it's much more flexible than that08:30
wrtpfwereade: it means PAs are independent of MAs08:30
wrtpfwereade: ... well...08:31
wrtpfwereade: an MA must start a PA, as with any unit08:31
wrtp(non-subordinate unit)08:31
fwereadewrtp, I'm not sure -- a separate provisioning state entity would be, but I'm suspicious that the AgentService thing feels unhelpfully cross-cutting08:31
wrtpfwereade: you're probably right.08:32
wrtpfwereade: the other thing it offers is the capability to add any new agent of our choice for free.08:32
wrtpfwereade: *currently* we only have three agent types, but that may well change.08:32
wrtpfwereade: i would be happy to just add a bool to a Machine for now.08:33
fwereadewrtp, sure, I'm just not convinced that we can predict the circumstances that might lead us to change well enough to call it right08:33
fwereadewrtp, that would be my choice, indeed, but I guess it's niemeyer's call08:33
wrtpfwereade: i'm not sure i'll ever show him this branch08:34
fwereadewrtp, but I would like it most if we were able to drop the conecpt of a PA entirely, I really don't see why the MA shouldn't run those tasks if so configured08:34
wrtpfwereade: i agree. it seems fine for a machine-level task.08:35
wrtpfwereade: ah, there is one issue08:35
fwereadewrtp, ah bother, there's usually something ;p08:35
wrtpfwereade: we won't give PA-like authority to all MAs08:36
wrtpfwereade: but given that everything has all authority currently, i'm not sure it's a problem for the time being08:36
fwereadewrtp, I don't *think* that's a serious problem... I presume the magic secure API layer will be able to grant/revoke appropriately08:37
fwereadewrtp, yeah, exactly08:37
wrtpfwereade: yeah08:37
fwereadewrtp, ok, well, I think we know what we're doing, I wish you luck :)08:38
wrtpfwereade: thanks! toi aussi08:38
wrtpfwereade: i'm thinking along these kinds of lines for the container API: http://paste.ubuntu.com/1196267/08:45
fwereadewrtp, +-0, I'm not sure I know enough to judge yet08:46
fwereadewrtp, I think I'll need to see some actual use :)08:47
wrtpfwereade: ok. i think that gives enough for use by both cloudinit and agents, but we'll see.08:47
fwereadewrtp, fwiw, it crosses my mind that agent commands should probably start their Run methods with `a.Conf.JujuDir = ctx.AbsPath(a.Conf.JujuDir)`, even if that's frequently a no-op09:47
wrtpfwereade: interesting point.09:48
wrtpfwereade: or should that happen where something passes the jujudir to something that might change directory?09:48
fwereadewrtp, I *think* that we should never be depending on working directory internally anyway09:49
fwereadewrtp, and I also think it's largely moot because we'll basically always be passing absolute paths anyway, but I think it will be more technically correct and mildly convenient for testing09:50
fwereadewrtp, (it's feeling very hard to write this code because I forsee it changing significantly, but I'm still not quite sure in what direction :))09:51
wrtpfwereade: yeah, maybe. i don't really have a feel for how --juju-dir might be used in practice09:51
fwereadewrtp, IMO it should just always be passed explicitly09:52
wrtpfwereade: that sounds reasonable09:52
wrtpfwereade: so we should make it a required flag?09:52
wrtpfwereade: well, tbh defaulting to /var/lib/juju seems ok too09:52
wrtpfwereade: we could just give an error if the path was *not* absolute.09:53
fwereadewrtp, I'm inclined to just drop the defaults -- if anyone's ever running itby hand i think we want it to be explicit, and it doesn't cost us much to explicitly set it when generating upstart scripts09:53
wrtpfwereade: but AbsPath seems like a reasonable thing to do too09:53
wrtpfwereade: indeed09:53
fwereadewrtp, all it means is that you *can* run it with a relative path and have it do the right thing in all situations09:54
fwereadewrtp, although, maybe it won't be able to find the tools necessarily09:55
wrtpfwereade: true. i'm just wondering when we'd ever actually want to run an agent from the command line09:55
wrtpfwereade: it doesn't need to find the tools09:55
fwereadewrtp, when things are weird and we're trying to figure out what is going on :)09:55
fwereadewrtp, unit agent does09:55
wrtpfwereade: good point.09:56
wrtpfwereade: i think if things are weird, it's easy for us to pass an absolute pathname :-)09:56
fwereadewrtp, yeah, maybe just requiring abs is the right way to go09:57
fwereadewrtp, cheers09:58
wrtpfwereade: np09:58
wrtpAram: morning10:32
Aramhello there.10:32
wrtpfwereade: i'm looking at the upstart package and wondering whether it might be best folded into container11:30
wrtpfwereade: it's juju specific, and the actual code is fairly trivial.11:30
wrtpfwereade: and in doing so, i wondered: is there any time we actually care about Service.{Start,Remove} idempotency?11:31
fwereadewrtp, hmmm, +0.9 to folding it into container11:49
fwereadewrtp, not sure about start/remove but it's not like it's heavily used, so probably not11:50
wrtpfwereade: thanks11:50
fwereadewrtp, is there anything I should have done to induce a charm in a dummy env to be downloadable?12:17
wrtpfwereade: i don't *think* so. it should just work, assuming it's pushing to storage.12:17
fwereadewrtp, ah, why might it not push to storage? do I have to tell it to expect puts/gets?12:18
wrtpfwereade: i don't think so12:18
wrtpfwereade: without seeing what you're doing, i'm not sure i can help much12:19
fwereadewrtp, I'm doing http://paste.ubuntu.com/1196552/ and expecting that a uniter will be able to download the result12:20
fwereadewrtp, Get http://127.0.0.1:38234/dummyenv/private/local_3a_series_2f_dummy-1: dial tcp 127.0.0.1:38234: connection refused12:20
fwereadewrtp, I can investigate myself, I'm just hoping for a shortcut to enlightenment, don't spend time on it :)12:21
wrtpfwereade: i think it *should* work.12:22
fwereadewrtp, cool, that is useful data :)12:22
fwereadewrtp, cheers12:22
wrtpfwereade: i *think* the juju deploy tests are testing this case12:23
wrtpfwereade: perhaps you're doing a Reset by accident?12:23
fwereadewrtp, hmm, I will poke around, that sounds quite plausible12:24
fwereadewrtp, cheers12:24
fwereadewrtp, bah, I trashed pkg and now it works12:26
fwereadewrtp, well it fails differently actually but in a much more scrutable way :)12:26
wrtpfwereade: good. i wonder what went on there12:26
fwereadewrtp, no idea, but I have found "trash pkg" to be a useful troubleshooting step every so often12:27
fwereadewrtp, couldn't remotely say what it's correlated with12:27
wrtpfwereade: i almost never do that.12:27
wrtpfwereade: i wonder how your setup differs12:27
fwereadewrtp, it's probably related to my bloody-minded insistence on keeping separate source dirs and swapping them around12:27
wrtpfwereade: lol12:27
wrtpfwereade: i'm impressed you manage to do that12:28
fwereadewrtp, meh, it fits my brain better and it costs my fingers little12:28
wrtpfwereade: ah, yes, i understand why you have the problems now12:28
fwereadewrtp, I presume something is checking for newer-than12:29
wrtpfwereade: you're moving source directories, but none of the source files change mtime12:29
fwereadewrtp, indeed12:29
wrtpfwereade: yeah.12:29
fwereadewrtp, the amazing thing honestly is that it works so well so much of the time ;p12:29
wrtpfwereade: you should trash the pkg directory each time you change source dirs12:29
wrtpfwereade: or touch all the .go files :-)12:29
fwereadewrtp, yeah, it's really just that actual adverse consequences from failing to do so are surprisingly rare, and so I sometimes forget :/12:30
wrtpfwereade: fair enough.12:31
fwereadewrtp, separate question: is a JujuConnSuite meant to be already bootstrapped? I thought it was but can't see where it's done12:34
wrtpfwereade: it is, i believe12:34
wrtpfwereade: search for Bootstrap on juju/testing/conn.go12:35
wrtps/on/in/12:35
* fwereade suspects he searched for Bots instead of Boots :/12:36
fwereadewrtp, thanks12:36
wrtpfwereade: we don't have bots yet :-)12:36
fwereadewrtp, for some reason "botstrap" seems to be one of my muscle-memory typos12:36
Aramwrtp: do you understand the purpose of this function? https://codereview.appspot.com/6503086/diff/7002/mstate/watcher/watcher.go#newcode16812:48
wrtpAram: yeah12:48
wrtpAram: it's for using in tests12:48
Aramhmm.12:48
wrtpAram: so that we can have shorter timeouts when waiting for nothing to happen12:49
Aramok, but then we don't need to export it publicly?12:49
wrtpAram: no, it's for any tests that use watchers12:49
Aramit can be in export_test.go12:49
wrtpAram: the idea is that you can change something in the state, call Sync, then you know that the watcher will have triggered any sends that might happen12:50
wrtpAram: no it can't12:50
wrtpAram: that would be ok if we only wanted to use in tests of the watcher package itself12:50
wrtpAram: i can't say i'm enormously keen on the idea, tbh. i'm not sure what particular advantage you get from having the done channels.12:58
wrtpniemeyer: yo!12:59
Aramwrtp: interesting, I thought the trick of exporting something for a pkg_test package from a pkg package _test file worked for every package used in a test, not only for pkg_test.12:59
wrtpAram: no indeed not12:59
niemeyerHello!12:59
fwereadeniemeyer, heyhey12:59
Aramhi.12:59
* wrtp gets a bite of lunch13:01
niemeyerAram: Just reproposed the branch13:04
niemeyerAram: I think I've fixed the spurious error with Sync13:04
niemeyerOkay, I'm starting my morning by implementing a watcher, probably the EnvironConfig one, to get an idea if the infrastructure is working well for real in an end-to-end case, and will push that for review13:07
niemeyerAfter that I'm back in review mode, perhaps for the rest of the week13:07
Aramniemeyer: I'm running the test in a loop.13:08
niemeyerAram: Cool13:08
niemeyerAram: Any more errors?13:08
Aramnothing yet13:08
niemeyerAram: Superb13:08
niemeyerAram: It was a race.. the test is asserting very defined behavior, and with StartSync() we can actually move on without the watcher having done anything13:09
=== wrtp is now known as rogpeppe
fwereadeniemeyer, I am becoming fretful about how service configs change when charms are upgraded... is this something we've already thought about in detail, that I've missed?14:52
niemeyerfwereade: Hmm14:54
niemeyerfwereade: I'm afraid to not know the context14:54
fwereadeniemeyer, well, just that units running an old version will not necessarily be able to properly understand a new config, and vice versa14:55
fwereadeniemeyer, this may just be a matter of "write your service configs carefully"14:56
niemeyerfwereade: No, you're right, I don't think we have given the problem proper consideration yet14:56
fwereadeniemeyer, ok -- well, I kinda need to stop for a while now, but I will try to think it through a little14:57
fwereadeniemeyer, just wanted to check there wasn't anything that sprang to mind :)14:57
niemeyerfwereade: It's probably easy to do something sane14:58
niemeyerfwereade: E.g. do not run the hook while service config doesnt14:58
niemeyer't validate properly14:58
niemeyerfwereade: with the current charm14:58
niemeyerfwereade: I'd be happy for us to discuss this, yet postpone the solution until a second point, though14:59
niemeyerfwereade: Just so you don't get blocked on this for too long14:59
niemeyerfwereade: UNless the solution is trivial, of course14:59
niemeyerfwereade: (which it might be, given the above)15:00
hazmatfwereade, you mean incompatible stored value with new schema?15:14
hazmatfwereade, unset values with defaults should switch out to new defaults/types cleanly15:15
niemeyerhazmat: Configuration options may also have disappeared, and the removal is only handled properly on the new hook, for example15:21
niemeyerhazmat: It's worth pondering about the edge cases more carefully at some point15:21
hazmatdefinitely15:22
niemeyerMachineWatcher tests pass!15:22
hazmatniemeyer, fwiw i found out that the whole yaml speed thing, was because pyyaml needs a different calling convention to actually use the c ext.15:23
niemeyerhazmat: I think there's a distinction between the loader and the parser15:23
niemeyerhazmat: The C extension is used at all times for certain tasks, IIRC15:24
hazmatniemeyer, there is its a two part combo.. with callbacks15:24
niemeyerhazmat: Go and Py were doing the same things in C and the same things in native lang15:24
niemeyerhazmat: Or the same layer, anyway15:24
hazmatbut dump/load wouldn't use the c ext opportunistically without changing the parameters15:24
niemeyerhazmat: Sure, as I understand it you can remove the higher level so it's all in C too15:25
niemeyerhazmat: So the stuff goyaml does in Go, can be done in C15:25
hazmatits messier to do but sure..  with the change it basically halves the test time, and triples the speed of status on large envs.15:26
niemeyerhazmat: Yeah, C is fast :-)15:26
hazmat:-)15:26
niemeyerhazmat: Wonder how things would look like in that old scale check15:26
hazmatwe're going to have simulatenous juju sprints on different continents15:26
niemeyerhazmat: Hah, nice :)15:27
hazmatniemeyer, i've got a simulator now for scale testing large envs.. specifically for the other proj15:27
hazmatand  everyone does dev with it15:27
hazmaton the principal that the best way to ensure scaling is to incorporate it into dev pratice15:28
niemeyerhazmat: What does "everyone does dev with it" mean?15:29
niemeyerAram, rogpeppe: A real watcher now: https://codereview.appspot.com/649711015:33
AramI'll take a look in a moment.15:33
niemeyerAram: I'll apply rogpeppe's comments to the foundation, and then I think I'll have to step out from impl for a while to clean up reviews15:34
niemeyerAram: Actually, I'll do one more cleanup on presence to bring it in line with watcher before I do that, but then it's back to you15:35
niemeyerSo two more branches on my plate.. will handle those right away15:35
Aramniemeyer: cheers.15:35
niemeyerrogpeppe: s/Non-existing/Non-existing/ !? :-)15:37
niemeyerrogpeppe: Non-existent?15:37
rogpeppeniemeyer: yeah!15:37
rogpeppeniemeyer: oops, sorry15:37
niemeyerrogpeppe: Cheers :)15:37
niemeyerrogpeppe: np15:37
niemeyerWatch helpers is up for review16:37
niemeyerI'll resend machine watchers again after lunch16:37
niemeyerbiab16:37
rogpeppefwereade: ping16:48
rogpeppeniemeyer: ping16:57
niemeyerrogpeppe: Yo16:57
rogpeppeniemeyer: i'm looking at fixing container, and i'm going around in circles a little bit16:58
rogpeppeniemeyer: i wonder if i could run some ideas past you16:58
niemeyerrogpeppe: Hmm, ok16:58
niemeyerrogpeppe: Sure16:58
niemeyerrogpeppe: What's broken there?16:58
rogpeppeniemeyer: so... container doesn't work at all currently16:58
rogpeppeniemeyer: it doesn't give the right flags to jujud etc16:59
niemeyerrogpeppe: Aren't we using it in the real-world tests that are run?16:59
rogpeppeniemeyer: i'm hoping it might be possible to make container use the same mechanism for installation as cloudinit17:00
rogpeppeniemeyer: no17:00
rogpeppeniemeyer: not yet17:00
niemeyerrogpeppe: Hmm, ok17:00
rogpeppeniemeyer: everything that runs currently is started by cloudinit17:00
rogpeppeniemeyer: here's an idea i've had: http://paste.ubuntu.com/1197028/17:00
rogpeppeniemeyer: oops, one crucial line missing: http://paste.ubuntu.com/1197030/17:01
niemeyerrogpeppe: What is changing and why?17:02
rogpeppeniemeyer: the idea is to replace the container package with the agent package.17:02
rogpeppeniemeyer: then environs/cloudinit can use that package to generate its cloudinit scripts17:03
rogpeppeniemeyer: and agents can use that package to start other agents17:03
rogpeppeniemeyer: it would start agents in new containers if required17:03
niemeyerrogpeppe: In the last couple of weeks we've had three different versions of what an Agent is17:03
rogpeppeniemeyer: yes, i think we're trying to find out :-)17:04
rogpeppeniemeyer: Agent may not be a good name here17:04
niemeyerrogpeppe: Yeah, but we have Agent today17:04
rogpeppeniemeyer: after all, it's just some information about an agent17:04
niemeyerrogpeppe: They exist already.. we can't give the same name to two different things17:04
rogpeppeniemeyer: i'm not sure why not. this is just one package's idea of an agent. different namespace.17:05
niemeyerrogpeppe: Our brains have a single namespace..17:05
niemeyerrogpeppe: It sucks to say "an agent" and having no idea about what it is17:05
rogpeppeniemeyer: agent.Info ?17:05
niemeyerrogpeppe: container?17:06
niemeyer:)17:06
niemeyerrogpeppe: What are we trying to fix?17:06
niemeyerrogpeppe: There's no problem statement yet that I can correlate to17:06
rogpeppeniemeyer: we're trying to put the upstart generation stuff in one place17:06
niemeyerrogpeppe: We have that.. that's container17:06
rogpeppeniemeyer: ok, so let's call this package "container". and give it a similar API.17:07
rogpeppeniemeyer: (to the one i've proposed)17:07
niemeyerrogpeppe: I'm not arguing for that even.. I'm asking you to tell me what I'm trying to fix :)17:07
rogpeppeniemeyer: currently the container package can't deploy a machine agent17:07
niemeyerrogpeppe: Sounds sane.. it's used by the machine agent17:08
rogpeppeniemeyer: i'd like to be able to use it from environs/cloudinit17:08
niemeyerrogpeppe: Hmm17:10
rogpeppeniemeyer: we've got these two pieces of code that are similar but different: http://paste.ubuntu.com/1197050/17:10
rogpeppeniemeyer: rather than copying all the logic from the latter to the former, i'd like to make both places use the same mechanism.17:11
niemeyerrogpeppe: What is similar among them?17:11
rogpeppeniemeyer: they should both be almost identical.17:11
niemeyerrogpeppe: Not really, they are managing independent commands, that need independent info, in very different circumstances17:12
rogpeppeniemeyer: except one runs the action there and then; the other generates a shell script to do the same on the remote machine17:12
niemeyerrogpeppe: So what is actually similar?17:12
rogpeppeniemeyer: everything up to InstallCommands vs Install17:13
niemeyerrogpeppe: Do we need a MachineConfig to deploy a unit?17:13
rogpeppeniemeyer: should be the same except that jujud agent arguments are different, but i think that need not be the case.17:13
rogpeppeniemeyer: we need a state info. and we need a VarDir. that's all it's used for.17:14
niemeyerrogpeppe: Do we need a MachineConfig to deploy a unit?17:14
rogpeppeniemeyer: i wasn't suggesting that we did.17:14
niemeyerrogpeppe: it is a question17:14
niemeyerrogpeppe: "No" is a fine answer17:14
niemeyerrogpeppe: I'm arguing that they are doing different things, and asking for the similarities, and you're saying that they are pretty much exactly the same17:15
rogpeppeniemeyer: no. a MachineConfig is a concept unique to environs/cloudinit.17:15
niemeyerrogpeppe: I don't see that17:15
niemeyerrogpeppe: and I'm showing you why that doesn't seem to be the case17:15
rogpeppeniemeyer: addAgentScript doesn't need a MachineConfig either17:15
niemeyerrogpeppe: We already have packages: container, upstart, cloudinit, environs/cloudinit, ...17:16
rogpeppeniemeyer: i was thinking of merging the upstart package into container - it's pretty trivial and not actually that helpful.17:16
niemeyerrogpeppe: If we're adding another layer, it must be clear what that layer is17:16
niemeyerrogpeppe: I'm not feeling we know that, given the line of thinking so far17:16
rogpeppeniemeyer: i was not proposing adding a layer, but changing an existing layer17:16
niemeyerrogpeppe: The "agent" package is a new layer, apparently17:17
niemeyerrogpeppe: It doesn't address the needs of container17:17
rogpeppeniemeyer: doesn't it?17:17
rogpeppeniemeyer: i was proposing it to replace container17:17
niemeyerrogpeppe: I don't see the word "LXC" there17:17
niemeyerrogpeppe: Nor the word unit17:18
rogpeppeniemeyer: should there be the word LXC there? or might it actually be ok to have that be an implementation detail of container?17:18
rogpeppe(or agent)17:18
niemeyerrogpeppe: It can be anything, but we need to know about what it is17:19
niemeyerrogpeppe: The proposal has to consider it, because that's exactly the reason why the container package exists17:19
rogpeppeniemeyer: likewise, does the mechanism for starting a unit agent need to know the *state.Unit? or might it be ok just to give it the info it actually needs?17:19
niemeyerrogpeppe: We can't obsolete the package without telling how it's going to work17:19
rogpeppeniemeyer: currently all we need to start a new unit in a container is the info provided in the proposal above.17:20
rogpeppeniemeyer: it's easy for the machine agent to derive that info from the *state.Unit and use that to call agent.Deploy (or container.Deploy)17:21
niemeyerrogpeppe: Again, it's not about "currently", it's about how we handle the problem being addressed by "container"17:21
rogpeppeniemeyer: ok. so let's see. what *is* the problem being addressed by "container"?17:21
niemeyerrogpeppe: We spent a lot of time thinking why we need that interface, I'd appreciate hearing your thoughts about how these problems we talked about will be handled17:22
* rogpeppe goes back to look at those discussions.17:23
niemeyerrogpeppe: Jun 14th17:25
rogpeppeniemeyer: i'm there17:25
rogpeppeniemeyer: i'm looking at this (http://paste.ubuntu.com/1040898/) and wondering what the container package actually wants from the *state.Unit value17:29
niemeyerrogpeppe: It wants to know what to deploy17:30
rogpeppeniemeyer: does it actually need to know any more than the args that need to be passed to jujud?17:31
niemeyerrogpeppe: How do we destroy a container given a list of arbitrary arguments to jujud?17:32
niemeyerrogpeppe: It feels like the thinking is very incipient17:32
niemeyerrogpeppe: I see 14 lines in addAgentScript, where most of those lines are already based on abstractions17:33
rogpeppeniemeyer: i'm not suggesting that the arbitrary args be a parameter to Deploy. but actually, we could easily make the jujud arguments uniform for all agents.17:33
niemeyerrogpeppe: The differences in the abstractions are exactly the things you're referring to17:33
niemeyerrogpeppe: Like a description for the agent, the information used to build the command line, etc17:33
niemeyerrogpeppe: I'm concerned that we're reinventing another wheel at this stage without even having the current wheels working17:34
rogpeppeniemeyer: ok, i'll make it work, then we'll see if it's worth abstracting17:35
rogpeppeniemeyer: i feel that it might be, but i agree that perhaps it's hard to tell at this stage.17:35
niemeyerrogpeppe: The right abstraction will likely take code out, rather than adding new layers such as Action, etc17:35
rogpeppeniemeyer: yeah. the difficulty is we've got these actions that we can either perform here and now, or remotely. Action was a way of trying to make both work uniformly.17:36
rogpeppeniemeyer: one other thing17:37
rogpeppeniemeyer: i'm been thinking about what new stuff we need to create to make the PA work in state.17:37
rogpeppeniemeyer: fwereade suggested earlier that we could just add a bool param to *state.Machine to say "run provisioning worker". so the MA would also be the PA when that's set.17:38
rogpeppeniemeyer: that would make lots of things easier (we'd get everything for free) but perhaps it's a bad idea. what do you think?17:38
niemeyerrogpeppe: Hmm17:39
niemeyerrogpeppe: I can't see any bad sides either17:40
rogpeppeniemeyer: great!17:40
rogpeppeniemeyer: that brings full upgrades about a week forward.17:40
niemeyerrogpeppe: Well, and that's a huge good side :)17:41
rogpeppeniemeyer: definitely.17:41
niemeyerreceived document: bson.M{"ok":0, "errmsg":"collection already exists"}17:46
niemeyerHow unfortunate.. no error codes whatsoever17:46
rogpeppeniemeyer: guess you'll just have to string match17:47
niemeyerYeah, sucks17:47
niemeyerWill file a bug upstream17:47
niemeyerLovely missed pre-reqs..18:13
niemeyerAlright, mstate/presence is polished19:29
niemeyerI'm done on the coding side for the moment, I think19:29
niemeyerI need to visit a friend at the hospital now.. back later19:30
mrammniemeyer: I hope your hospital trip goes well.   Good luck.19:36
mrammniemeyer: if there is anything I can do to help, let me know.19:36
wrtpmramm: hiya19:39
mrammwrtp: hey!19:40
niemeyermramm: Thanks, all good there.. his dad was making a delicate heart procedure23:37

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!