fwereadehazmat, niemeyer: been trying to figure out the details of the unit agent for upstartification12:20
niemeyerfwereade: Cool12:21
fwereadehazmat, niemeyer: there's talk of in-memory state we need to persist; is that *just* UnitLifecycle._relations, which is all I can obviously see, or do you know of more things I should be worrying about?12:21
fwereadeniemeyer, I don't really expect you to reread all the code tbh, I'm just hoping for a quick yea/nay or just a "figure it out yourself :p"12:23
niemeyerfwereade: One of the most interesting bits is in HookScheduler12:26
niemeyerfwereade: You're right that the relation is a problem as well, though12:27
fwereadeniemeyer, hm, I'd missed that, I'll take a look12:27
fwereadethanks :)12:27
niemeyerfwereade: Due to the use of ephemeral nodes12:27
fwereadeniemeyer, ahhh12:27
niemeyerfwereade: When we restart the agent for whatever reason, ideally the underlying connectivity shouldn't go away12:28
niemeyerfwereade: Which raises further questions about how to handle it12:28
niemeyerfwereade: The timeout should probably be moved into application logic rather than trusting on ephemeral nodes for that12:28
niemeyerfwereade: That said, it's probably a step you can care about at a later point12:29
niemeyerfwereade: The HookScheduler I'm mentioning will simply yield weird faults if not handled properly12:29
niemeyerfwereade: Which is worse than having a relation restarted12:29
fwereadeniemeyer, not entirely following you, but I think I just need to spend a bit more time on the code12:30
fwereadeniemeyer, haven't formed a really good model of how it all fits together yet12:30
niemeyerfwereade: What part?12:30
niemeyerfwereade: I mean, which part aren't you following? Can I explain something else, or would you rather spend some time looking at the code?12:31
fwereadeniemeyer, all the parts I've looked at make sense; I just don't have instant recall of how it all fits together12:31
fwereadeniemeyer, I think I'll learn best by just diving in and breaking stuff12:31
fwereadeniemeyer, I'll certainly hassle you if I need help with something specific12:31
niemeyerfwereade: I easily get lost as well, to be honest, which is something I'd like to fix12:32
fwereadeniemeyer, I'm not sure whether that's reassuring or alarming :p12:32
fwereadea bit of both mayeb :)12:32
niemeyerfwereade: ;)12:35
hazmatfwereade, g'morning13:09
* hazmat has been watching lbox manipulate the reitveld queue13:09
fwereadeheya hazmat13:09
hazmatfwereade, re persistence for the unit agent and upstart, i think their different problems13:10
fwereadehazmat, hmm; I got the impression that UA persistence was a prereq13:10
hazmatfwiw workflow states are recorded to zk, the memory state is the seen nodes / watches that input into the hook queue / scheduler13:10
fwereadehazmat, I'd figured that much out at least :)13:11
niemeyerhazmat: Hm?13:11
hazmatfwereade, its not a pre-req, their both problems, solving the restart problem fully means doing both13:11
fwereadehazmat, indeed, I'd seen "upstartify" as meaning "upstartify and ensure it's actually useful"13:12
hazmatfwereade, upstartification applies to lots of things, all the agents, and likely additional things like local provider processes13:13
niemeyerhazmat: How's lbox manipulating the rietveld queue?13:13
hazmatfwereade, its useful byitself13:13
hazmatniemeyer, oh.. i guess you where using it by hand?13:13
niemeyerhazmat: I don't know.. what are you referring to? :-)13:13
hazmatniemeyer, http://codereview.appspot.com/5372097/13:13
niemeyerhazmat: This was created after your initial comment, which makes me wonder where is this showing?13:14
niemeyerhazmat: This is the rietveld Go package.. almost thre13:14
hazmatniemeyer, i saw one a few hrs ago (2.5)13:14
niemeyerhazmat: Where13:15
hazmatniemeyer, same url same content13:15
niemeyerhazmat: I mean.. is there a feed somewhere that you're watching?13:15
hazmatwell maybe not same url, but same content13:15
hazmatniemeyer, no just hitting the front page13:15
niemeyerhazmat: Hah :-)13:16
fwereadehazmat, I have the impression that upstartifying the UA without persistence is... of verify limited utility, if not necessarily and more harmful than the current state13:17
fwereadeMA/PA I don't see issues with upstrartifying, agreed13:18
niemeyerhazmat, fwereade, rog: http://goneat.org/lp/goetveld/rietveld13:19
niemeyerNow, to tweak lbox so it sends the delta on propose13:19
hazmatfwereade, unit agent sans persistence, will invoke relation hooks redundantly, thats not the end of the world13:24
fwereadehazmat, ah, I had a feeling it might be able to miss them as well13:24
* hazmat checks the codz13:25
hazmatfwereade, it should just restart the relation.RelationUnitWatcher which will re-feed to the hook queue based on current state13:29
hazmatfwereade, a change event while down will get mutated into a joined event13:29
hazmatactually it still gets a changed event since we have expansion of joined to (joined and changed)13:30
fwereadehazmat, hmm, ok, shall I just look at upstartification for now then?13:31
fwereadehazmat, hopefully the UA stuff will bed into my brain for a bit while I work on something else ;)13:31
hazmatfwereade, sounds good13:31
fwereadehazmat, ok, cool :)13:31
hazmatfwereade, incidentally there was some concern about lost events, in the case that the session was active, and we restablished the process hooked up to the same session, but the recent tests in txzk demonstrate that watches are still delivered when the client does reconnect to the extant session13:38
fwereadehazmat, ah cool, perhaps that was what I was thinking of13:43
rogniemeyer: cool!13:49
rogniemeyer: is there anything rietveld-specific about rietveld.Auth ?13:51
niemeyerrog: Not sure about what's the underlying question13:52
rogniemeyer: just wondered if it would be worth putting in its own package, so it can be used by other people talking to google apps13:52
niemeyerrog: Maybe..13:53
niemeyerrog: But I'm happy with it for now13:53
rogniemeyer: just a though13:54
niemeyerrog: Yeah, I understand.. just saying the actual use case right now is Rietveld13:54
niemeyerrog: People can freely factor it out, though13:54
rogniemeyer: that was the reason for my original question - because it's in the rietveld package, it's not clear whether it has rietveld dependencies. i don't mind - it was just my first reaction on seeing it.13:56
niemeyerrog: I understand.. :-)13:56
uksysadminhi all14:10
uksysadmingot a quick q - my bzr set up seems to have gone screwey... think I'm ok now btu I've got a situation now where my charms dir where I have all my charms checked out using "charm getall oneiric" I get the following:14:11
uksysadminmr update: /home/kevinj/cloud/charms/hadoop-slave14:11
uksysadminbzr: ERROR: No location specified or remembered14:11
uksysadminmr update: command failed14:11
hazmatuksysadmin, i try just blowing away the hadoop-slave charm dir.. and let it refetch, mr can get wedged if things change on it14:12
uksysadminnp - will have another go14:13
uksysadminhazmat, that's done it14:14
mcclurmchi all14:53
mcclurmci've heard that there is an openstack service provider for juju, but it's not in the main source repository. can anyone point me to it?14:53
hazmatmcclurmc, the ec2 provider works with openstack14:58
hazmatniemeyer, got a moment?14:58
mcclurmchazmat: ah, of course. thanks15:00
mcclurmchazmat: actually, this might be a better question for #openstack, then, but how does it handle storage? is there an S3 api for openstack as well?15:01
hazmatmcclurmc, there is, juju ec2 provider been tested with the simple s3 objectstore thats distributed with openstack/nova, it should work with swift if its got the s3 middleware enabled but that hasn't been tested by the juju team yet15:03
mcclurmchazmat: great, thanks15:03
hazmatjcastro, ping15:04
_mup_juju/local-repo-log-broken-charm r414 committed by kapil.thangavelu@canonical.com15:58
_mup_unify metadata/config error handling, distinguish config definition errors from value errors, repository find notes errors and location/path on charms.15:58
mplhazmat: thx for the reply on the list.16:21
hazmatmpl, np16:22
mplhazmat: the reason I'm not working from local containers is I intend to play with the go code, and gustavo said I'd be better off directly on an ec2 for that.16:27
hazmatmpl, not sure why that would be the case, is the code something likely to overwhelm a single machine.. your working with camelstore ?16:30
hazmatmpl, local provider isn't perfect, its got some known issues (doesn't survive restarts or suspends)16:31
jcastrohazmat: pong16:31
hazmatbut as long as what your building isn't going to more than the underlying machine can handle it should be fine.. if your looking for large memory instances or large cpu .. then going native in ec2 from the start makes ense16:32
hazmatniemeyer, ^?16:32
mplhazmat: iiuc, the go port atm is only working with ec2 and focusing on that. that may be the reason.16:33
hazmatjcastro, hi.. i wanted to catch up on charm stuff16:33
jcastrohazmat: yeah sure, gimme 5?16:34
hazmatjcastro, sounds good16:35
niemeyerhazmat: Yo17:08
niemeyerSorry, I'm just up from a nice, long, restful nap..17:08
niemeyermpl: That's right17:09
niemeyermpl, hazmat: That was indeed the reason..17:09
rogniemeyer, fwereade: any response to my thoughts on william's review of the initial ec2 stuff? particularly point [1].17:18
niemeyerrog: Sorry, I haven't been watching reviews much in the hopes we can move them over to Rietveld ASAP, but let me check that one17:18
rogniemeyer: that's fine, i thought as much. this is a central (and interesting) point though, and worth thinking about.17:19
niemeyerrog: I agree with fwereade in general17:21
niemeyerrog: The Machine should probably be an interface rather than a concrete type17:21
niemeyerrog: and it should definitely not expose a ec2 instance in its interface17:21
rogniemeyer: it is already an interface17:21
niemeyer694+type Machine struct {17:21
niemeyer695+ MachineId string17:21
niemeyer696+ *ec2.Instance17:21
niemeyer697+ Reservation *ec2.Reservation17:21
niemeyerrog: That's not an interface17:21
rogniemeyer: it's just an implementation of juju.Machine17:21
rogniemeyer: i could avoid exporting it though17:21
niemeyerrog: Right, that's what I meant17:22
rogniemeyer: the renaming i'm proposing is to rename juju.Machine to juju.Interface17:22
niemeyerrog: juju.Machine sounds good17:22
rogniemeyer: but the problem is that, as fwereade points out, that there's not necessarily a direct correspondence between instances and machines17:23
rogniemeyer: at least, i *think* that's his point.17:23
niemeyerrog: It's not17:24
niemeyerrog: The point is that the instance underneath can change17:24
rogniemeyer: doesn't that imply what i just said?17:24
fwereadeniemeyer, rog: that said, juju.Instance might be a better name than juju.Machine, because it's less likely to occupy the same mental register as "machine" (as in machine state)17:25
rogniemeyer: i.e. a machine can correspond to different instances over time17:25
rogfwereade: that's what i'm thinking17:25
fwereadeniemeyer, rog: OTOH it's quite ec2-specific17:25
niemeyerrog: It can, but there's a direct correspondence between a machine and an instance, which is what you implied to not be the case17:25
rogniemeyer: hmm. i think i'm probably missing something. how does it happen that an instance underneath can change?17:26
niemeyerfwereade, rog: Hmm.. maybe.. let me see the code again, please17:26
niemeyerrog: I'll let fwereade explain while I have a quick look at the code17:27
* rog is all big flappy ears17:27
fwereadeniemeyer, rog: I *think* you actually agree but are using incompatible words :)17:27
rogwouldn't be the first time!17:28
niemeyerfwereade: I certainly don't disagree with anything right now.. :-)17:28
niemeyerJust trying to sort out what we're after..17:28
TheMueniemeyer: Did you followed my mail correspondence with Robbie?17:28
rogbut it was a genuine question: how *can* an instance change when the juju machine itself remains the same?17:28
fwereadeniemeyer, rog: I would be happy with a juju.Instance, so long as it didn't have any *machine*-specific data tacked on17:28
fwereaderog: if I were to go and terminate-instance, juju would notice the instance was gone and bring up a new one17:29
fwereaderog: that's ec2 terminate-instance, that juju is unaware of17:29
rogfwereade: that's what i thought. but it needs to be possible, i think, to tag a machine (as ec2 does with the security group)17:29
fwereaderog: what are we missing if we don't have that?17:29
niemeyerfwereade, rog: I think you guys are onto something for sure.. just want to see the code to ensure I'm getting it too17:30
rogfwereade: good question.17:30
rogfwereade: i just saw that that's what the code does, and haven't seen how it's actually used.17:30
niemeyerrog: I think it's not part of the actual ProviderMachine interface, though17:31
fwereaderog: as I see it, if an instance is associated with a machine, we can find that out from ZK; we might want to interact with i-foobar because it corresponds to machine/2717:31
niemeyerrog: So this again holds up your/William's suggestion that an Instance would make more sense17:31
fwereaderog: but I can't think of a case where we only have i-foobar and need to find out what juju machine it corresponds to17:32
rogniemeyer: what isn't? the tag?17:33
niemeyerrog: The information used to tag a machine isn't part of the ProviderMachine interface17:33
rogniemeyer: you mean the juju.Machine interface?17:34
rog(as currently, or juju.Instance as proposed)17:34
niemeyerrog: I mean the ProviderMachine interface17:34
fwereadeguys, sorry, dinner is ready and apparently it's important that it be eaten hot17:34
niemeyerrog: We're still defining how it looks like in juju, so that might be ambiguous17:34
fwereadeI'll pop back on when I can17:34
niemeyerrog: I can correctly state that about the current code, though17:35
niemeyerrog: Sorry, still defining in the Go port17:35
rogfwereade: okeydokey17:35
niemeyerfwereade: Cheers man17:35
niemeyerrog: Ok, in general, I'm happy with yours/William's suggestion of using juju.Instance17:35
niemeyerrog: Where today we have ProviderMachine17:35
rogniemeyer: by "ProviderMachine" you mean some as-yet-to-be-defined thing?17:35
rogah, the python code!17:36
niemeyerrog: Yeah, that forgotten piece of code that happens to be the critical reference!17:36
rogsorry, living in two worlds17:37
niemeyerrog: So, for the docs of juju.Instance: it needs to be Machine agnostic, and Provider agnostic, which is interesting17:37
rogniemeyer: yeah. that seems ok to me.17:38
rogniemeyer: it's already close to that. except that it mentions machine id17:38
niemeyerrog: So juju.Instance is the internal representation of that thing that a provider calls a machine :-)17:38
niemeyerrog: Yeah, that's the bit worth cleaning up17:38
rogniemeyer: definitely. that's good.17:39
niemeyerrog: Awesome, thanks for the suggestion.. will be a nice mental clean up :)17:40
rogniemeyer: that's kind of how i thought of it before, but i hadn't realised the loose relationship between instance and machine.17:40
rogright, Machine -> Instance it is. and instance id becomes defined by Instance rather than being given to StartMachine.17:40
rogniemeyer: and for now, i won't provide the ability to tag a machine on startup. that can be added back later.17:41
niemeyerrog: I think there may be some confusion taking place still17:41
rogniemeyer: quite probably!17:42
niemeyerrog: start_machine takes a machine id, and it needs it17:42
rogniemeyer: ah. why's that?17:42
niemeyerrog: It should probably be called start_instance, though17:42
TheMuebeside the wiki, do we have some diagrams showing how the software is organized and the components play together?17:42
niemeyerrog: But it needs a machine id still, not an instance id17:42
niemeyerTheMue: Sorry, I missed your earlier message17:42
TheMueniemeyer: nopro17:43
niemeyerTheMue: I've followed a few threads, but I'm not sure we're referring to the same one17:43
TheMueniemeyer: regarding a talk on the parallel 2012 in May here in Germany17:43
niemeyerrog: I suggest following through in the Python code so you understand more of that and see the separation of concepts that already exists17:43
rogniemeyer: how does the machine id get used by the provider?17:43
TheMueniemeyer: should not conflict with the next summit17:44
niemeyerrog: Despite the name, MachineProvider is already not the same as a state machine17:44
niemeyerTheMue: Ah, yeah17:44
rogniemeyer: ?17:44
niemeyerTheMue: It should be fine for sure17:44
niemeyerrog: !17:44
niemeyerSee, that's my job.. ;-D17:44
rogniemeyer: did i somehow imply that it was?17:45
niemeyerrog: We're just evolving an idea.. not every comment has to be  disagreement :-)17:45
niemeyerrog: Check the code out.. it'll be enlightening17:46
rogniemeyer: eek! i wasn't disagreeing! i was trying to find out what you understood by what i said17:46
TheMueniemeyer: maybe we'll find a good topic matching how we use the concurrency of go in juju.17:47
niemeyerTheMue: Sounds good.. even though I hope our use of concurrency is simplistic enough to not be a fantastic highlight :-)17:47
* hazmat finishes lunch17:49
hazmatTheMue, greetings17:50
TheMueniemeyer: my other topic would be concurrent design with message based communication as a more natural approach for software architectures17:50
TheMuehazmat: hi17:50
hazmatthat reminds of the start of the project :-)17:50
niemeyerTheMue: My first reaction to the topic is that it feels a little bit overcovered already17:51
rogniemeyer: AFAICS it looks like the machine-specific security group is just used for opening ports. am i missing something?17:51
TheMueniemeyer: ok, maybe from a too high level. i like the idea of event-driven architectures.17:52
hazmatTheMue, state based approaches have some advantages over message systems, esp. as one considers faults scenarios17:52
hazmatTheMue, i'm a fan as well.. but i think the state based approach is feature of ensemble and makes reasoning about the system much more succinct17:53
niemeyerrog: No, that's right.. the machine id in start_machine/instance is used for bootstrapping the node too, though17:53
niemeyerrog: So we can't get rid of it17:53
rogniemeyer: ah, i must have missed that bit17:54
niemeyerhazmat: TheMue is referring to internal architectures17:54
niemeyerrog: Otherwise the starting machine has no idea of its purpose in the world (poor guy :-)17:54
TheMuehazmat: event-driven and state based is no opposite for me17:54
rogniemeyer: ah, i thought it knew from its boot arguments.17:55
niemeyerrog: It does!17:55
niemeyerrog: Where do the boot arguments come from? ;-)17:55
rogniemeyer: argument to StartInstances?17:56
niemeyerrog: Right17:56
rogniemeyer: so, i must be missing something. where does the machine id come in there?17:56
rogBTW i have to go in 1 minute!17:56
niemeyerrog: Check the MachineProvider in the ec2 backend17:57
niemeyerrog: start_machine method17:57
hazmatTheMue, i'm missing a context then of what you mean by event driven architecture then17:57
rogniemeyer: ah yes. i guess it could be some other unique string but then we wouldn't know how to remove it.17:59
hazmatthe state system is observable and the observations could be decomposed into an event based dispatch driving application logic, so they are compatible, but its hard to reason about the usefulness in the abstract, the observation pattern against state is a non concurrent  one18:02
niemeyerrog: Btw, the _machine_ id is a number18:03
niemeyerhazmat: TheMue was really talking about concurrent programming within a process, rather than across processes, I believe18:03
TheMueback again, had to bring my daughter to driving school18:09
TheMuehazmat: i've learned about eda in large systems, with cep for a large number of events in realtime18:10
hazmatTheMue, nice.. custom cep engine? or something like esper?18:10
TheMuehazmat: langs like go and erlang allow to create the same insde one process18:10
TheMuehazmat: would have liked to learn more about esper, only read about it. we tried to establish it as a part of a soa. our part has been done in smalltalk, for a telco18:12
hazmatTheMue, effectively in process eda using coroutine pools  for concurrency for a given event dispatch?18:13
TheMuehazmat: my own Tideland ECA will be a framework for go internal event-processing, but not publish/subscribe based. more like connected cells with inputs and outputs18:13
TheMuehazmat: no explicit concurrency controlled by us, only by GemStone/S as the backend18:15
TheMuehazmat: thankfully no high load yet, project is still in progress. but shall be used later for the analysis of outages18:16
TheMueso, have to stop here for a moment and finish my (print) article on Dart18:19
niemeyerhazmat: ping18:30
raphinkhi there :-)19:31
raphinkhow is juju different from zookeeper (or other similar products)?19:31
marcoceppiraphink: Well, Juju uses zookeeper - so that's something to think about19:44
raphinkah ok19:46
hazmatraphink, greetings.. zookeeper is just a coordination storage layer.. it doesn't really do much by itself.. its what the application chooses to do with it thats interesting..  juju is an application that uses zk (mostly as an implementation detail) to provide a language neutral mechanism for defining a service and its dependencies in a reuseable fashion.  its like apt for the clouds.. solving dependencies on the network19:49
hazmatjuju differs from most of the similar configuration management products in that it focuses on orchestration and encapsulation/reuse of a service definition.. ie. its focused on service orchestration not configuration management19:50
raphinkhazmat: and it makes use of other configuration management products (like puppet/cfengine/chef) under the hood, right?20:17
hazmatraphink, actually it allows a charm author  (the definition of the reusable bit) to use whatever other products their comfortable with in whatever language their comfortable with, juju provides communication channels (exposed via clis) and guarantees on when things are called (orchestration).. juju core doesn't have an internal dependency on other CM tools20:18
raphinkI see20:19
raphinkjust wondering20:20
raphinkwhy was the product renamed?20:20
hazmatraphink, for that i have no good answer ;-).. it suffices to say it was done20:52
raphinkhehe I guess it was haakon_20:53
raphinksorry, hazmat20:53
hazmatniemeyer, is there any reason the wtf tests can't also be run against the local provider?21:07
_mup_Bug #890449 was filed: metadata.yaml needs contact fields <juju:New> <juju Charms Collection:New> < https://launchpad.net/bugs/890449 >22:06
_mup_Bug #890453 was filed: metadata.yaml needs a category / classifier field <juju:New> < https://launchpad.net/bugs/890453 >22:20
niemeyerhazmat: No intrinsic reasons22:58
niemeyerhazmat: I've not tried to set them up against the local provider, but as long as we can put the system back on a clean state I guess it should work22:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!