[12:20] hazmat, niemeyer: been trying to figure out the details of the unit agent for upstartification [12:21] fwereade: Cool [12:21] hazmat, niemeyer: there's talk of in-memory state we need to persist; is that *just* UnitLifecycle._relations, which is all I can obviously see, or do you know of more things I should be worrying about? [12:23] niemeyer, I don't really expect you to reread all the code tbh, I'm just hoping for a quick yea/nay or just a "figure it out yourself :p" [12:26] fwereade: One of the most interesting bits is in HookScheduler [12:27] fwereade: You're right that the relation is a problem as well, though [12:27] niemeyer, hm, I'd missed that, I'll take a look [12:27] thanks :) [12:27] fwereade: Due to the use of ephemeral nodes [12:27] niemeyer, ahhh [12:28] fwereade: When we restart the agent for whatever reason, ideally the underlying connectivity shouldn't go away [12:28] fwereade: Which raises further questions about how to handle it [12:28] fwereade: The timeout should probably be moved into application logic rather than trusting on ephemeral nodes for that [12:29] fwereade: That said, it's probably a step you can care about at a later point [12:29] fwereade: The HookScheduler I'm mentioning will simply yield weird faults if not handled properly [12:29] fwereade: Which is worse than having a relation restarted [12:30] niemeyer, not entirely following you, but I think I just need to spend a bit more time on the code [12:30] niemeyer, haven't formed a really good model of how it all fits together yet [12:30] fwereade: What part? [12:31] fwereade: I mean, which part aren't you following? Can I explain something else, or would you rather spend some time looking at the code? [12:31] niemeyer, all the parts I've looked at make sense; I just don't have instant recall of how it all fits together [12:31] niemeyer, I think I'll learn best by just diving in and breaking stuff [12:31] niemeyer, I'll certainly hassle you if I need help with something specific [12:32] fwereade: I easily get lost as well, to be honest, which is something I'd like to fix [12:32] niemeyer, I'm not sure whether that's reassuring or alarming :p [12:32] a bit of both mayeb :) [12:35] fwereade: ;) [13:09] fwereade, g'morning [13:09] * hazmat has been watching lbox manipulate the reitveld queue [13:09] heya hazmat [13:10] fwereade, re persistence for the unit agent and upstart, i think their different problems [13:10] hazmat, hmm; I got the impression that UA persistence was a prereq [13:10] fwiw workflow states are recorded to zk, the memory state is the seen nodes / watches that input into the hook queue / scheduler [13:11] hazmat, I'd figured that much out at least :) [13:11] hazmat: Hm? [13:11] fwereade, its not a pre-req, their both problems, solving the restart problem fully means doing both [13:12] hazmat, indeed, I'd seen "upstartify" as meaning "upstartify and ensure it's actually useful" [13:13] fwereade, upstartification applies to lots of things, all the agents, and likely additional things like local provider processes [13:13] hazmat: How's lbox manipulating the rietveld queue? [13:13] fwereade, its useful byitself [13:13] niemeyer, oh.. i guess you where using it by hand? [13:13] hazmat: I don't know.. what are you referring to? :-) [13:13] niemeyer, http://codereview.appspot.com/5372097/ [13:14] hazmat: This was created after your initial comment, which makes me wonder where is this showing? [13:14] hazmat: This is the rietveld Go package.. almost thre [13:14] there [13:14] niemeyer, i saw one a few hrs ago (2.5) [13:15] hazmat: Where [13:15] ? [13:15] niemeyer, same url same content [13:15] hazmat: I mean.. is there a feed somewhere that you're watching? [13:15] well maybe not same url, but same content [13:15] niemeyer, no just hitting the front page [13:16] hazmat: Hah :-) [13:17] hazmat, I have the impression that upstartifying the UA without persistence is... of verify limited utility, if not necessarily and more harmful than the current state [13:18] MA/PA I don't see issues with upstrartifying, agreed [13:19] hazmat, fwereade, rog: http://goneat.org/lp/goetveld/rietveld [13:19] Now, to tweak lbox so it sends the delta on propose [13:24] fwereade, unit agent sans persistence, will invoke relation hooks redundantly, thats not the end of the world [13:24] hazmat, ah, I had a feeling it might be able to miss them as well [13:25] * hazmat checks the codz [13:29] fwereade, it should just restart the relation.RelationUnitWatcher which will re-feed to the hook queue based on current state [13:29] fwereade, a change event while down will get mutated into a joined event [13:30] actually it still gets a changed event since we have expansion of joined to (joined and changed) [13:31] hazmat, hmm, ok, shall I just look at upstartification for now then? [13:31] hazmat, hopefully the UA stuff will bed into my brain for a bit while I work on something else ;) [13:31] fwereade, sounds good [13:31] hazmat, ok, cool :) [13:31] cheers [13:38] fwereade, incidentally there was some concern about lost events, in the case that the session was active, and we restablished the process hooked up to the same session, but the recent tests in txzk demonstrate that watches are still delivered when the client does reconnect to the extant session [13:43] hazmat, ah cool, perhaps that was what I was thinking of [13:43] ta [13:49] niemeyer: cool! [13:51] niemeyer: is there anything rietveld-specific about rietveld.Auth ? [13:52] rog: Not sure about what's the underlying question [13:52] niemeyer: just wondered if it would be worth putting in its own package, so it can be used by other people talking to google apps [13:53] rog: Maybe.. [13:53] rog: But I'm happy with it for now [13:54] niemeyer: just a though [13:54] t [13:54] rog: Yeah, I understand.. just saying the actual use case right now is Rietveld [13:54] rog: People can freely factor it out, though [13:56] niemeyer: that was the reason for my original question - because it's in the rietveld package, it's not clear whether it has rietveld dependencies. i don't mind - it was just my first reaction on seeing it. [13:56] rog: I understand.. :-) [14:10] hi all [14:11] got a quick q - my bzr set up seems to have gone screwey... think I'm ok now btu I've got a situation now where my charms dir where I have all my charms checked out using "charm getall oneiric" I get the following: [14:11] mr update: /home/kevinj/cloud/charms/hadoop-slave [14:11] bzr: ERROR: No location specified or remembered [14:11] mr update: command failed [14:12] etc [14:12] uksysadmin, i try just blowing away the hadoop-slave charm dir.. and let it refetch, mr can get wedged if things change on it [14:13] s/i/i'd [14:13] np - will have another go [14:14] hazmat, that's done it [14:14] ta [14:14] np [14:23] Phew.. [14:53] hi all [14:53] i've heard that there is an openstack service provider for juju, but it's not in the main source repository. can anyone point me to it? [14:58] mcclurmc, the ec2 provider works with openstack [14:58] niemeyer, got a moment? [15:00] hazmat: ah, of course. thanks [15:01] hazmat: actually, this might be a better question for #openstack, then, but how does it handle storage? is there an S3 api for openstack as well? [15:03] mcclurmc, there is, juju ec2 provider been tested with the simple s3 objectstore thats distributed with openstack/nova, it should work with swift if its got the s3 middleware enabled but that hasn't been tested by the juju team yet [15:03] hazmat: great, thanks [15:04] jcastro, ping [15:58] <_mup_> juju/local-repo-log-broken-charm r414 committed by kapil.thangavelu@canonical.com [15:58] <_mup_> unify metadata/config error handling, distinguish config definition errors from value errors, repository find notes errors and location/path on charms. [16:21] hazmat: thx for the reply on the list. [16:22] mpl, np [16:27] hazmat: the reason I'm not working from local containers is I intend to play with the go code, and gustavo said I'd be better off directly on an ec2 for that. [16:30] mpl, not sure why that would be the case, is the code something likely to overwhelm a single machine.. your working with camelstore ? [16:31] mpl, local provider isn't perfect, its got some known issues (doesn't survive restarts or suspends) [16:31] hazmat: pong [16:32] but as long as what your building isn't going to more than the underlying machine can handle it should be fine.. if your looking for large memory instances or large cpu .. then going native in ec2 from the start makes ense [16:32] niemeyer, ^? [16:33] hazmat: iiuc, the go port atm is only working with ec2 and focusing on that. that may be the reason. [16:33] jcastro, hi.. i wanted to catch up on charm stuff [16:34] hazmat: yeah sure, gimme 5? [16:35] jcastro, sounds good [17:08] hazmat: Yo [17:08] Sorry, I'm just up from a nice, long, restful nap.. [17:09] mpl: That's right [17:09] mpl, hazmat: That was indeed the reason.. [17:18] niemeyer, fwereade: any response to my thoughts on william's review of the initial ec2 stuff? particularly point [1]. [17:18] rog: Sorry, I haven't been watching reviews much in the hopes we can move them over to Rietveld ASAP, but let me check that one [17:19] niemeyer: that's fine, i thought as much. this is a central (and interesting) point though, and worth thinking about. [17:21] rog: I agree with fwereade in general [17:21] rog: The Machine should probably be an interface rather than a concrete type [17:21] rog: and it should definitely not expose a ec2 instance in its interface [17:21] niemeyer: it is already an interface [17:21] 694 +type Machine struct { [17:21] 695 + MachineId string [17:21] 696 + *ec2.Instance [17:21] 697 + Reservation *ec2.Reservation [17:21] 698 +} [17:21] rog: That's not an interface [17:21] niemeyer: it's just an implementation of juju.Machine [17:21] niemeyer: i could avoid exporting it though [17:22] rog: Right, that's what I meant [17:22] niemeyer: the renaming i'm proposing is to rename juju.Machine to juju.Interface [17:22] oops [17:22] juju.Instance [17:22] rog: juju.Machine sounds good [17:23] niemeyer: but the problem is that, as fwereade points out, that there's not necessarily a direct correspondence between instances and machines [17:23] niemeyer: at least, i *think* that's his point. [17:24] rog: It's not [17:24] rog: The point is that the instance underneath can change [17:24] niemeyer: doesn't that imply what i just said? [17:25] niemeyer, rog: that said, juju.Instance might be a better name than juju.Machine, because it's less likely to occupy the same mental register as "machine" (as in machine state) [17:25] niemeyer: i.e. a machine can correspond to different instances over time [17:25] fwereade: that's what i'm thinking [17:25] niemeyer, rog: OTOH it's quite ec2-specific [17:25] rog: It can, but there's a direct correspondence between a machine and an instance, which is what you implied to not be the case [17:26] niemeyer: hmm. i think i'm probably missing something. how does it happen that an instance underneath can change? [17:26] fwereade, rog: Hmm.. maybe.. let me see the code again, please [17:27] rog: I'll let fwereade explain while I have a quick look at the code [17:27] brb [17:27] * rog is all big flappy ears [17:27] niemeyer, rog: I *think* you actually agree but are using incompatible words :) [17:28] wouldn't be the first time! [17:28] lol [17:28] fwereade: I certainly don't disagree with anything right now.. :-) [17:28] Just trying to sort out what we're after.. [17:28] niemeyer: Did you followed my mail correspondence with Robbie? [17:28] but it was a genuine question: how *can* an instance change when the juju machine itself remains the same? [17:28] niemeyer, rog: I would be happy with a juju.Instance, so long as it didn't have any *machine*-specific data tacked on [17:29] rog: if I were to go and terminate-instance, juju would notice the instance was gone and bring up a new one [17:29] rog: that's ec2 terminate-instance, that juju is unaware of [17:29] fwereade: that's what i thought. but it needs to be possible, i think, to tag a machine (as ec2 does with the security group) [17:29] rog: what are we missing if we don't have that? [17:30] fwereade, rog: I think you guys are onto something for sure.. just want to see the code to ensure I'm getting it too [17:30] fwereade: good question. [17:30] fwereade: i just saw that that's what the code does, and haven't seen how it's actually used. [17:31] rog: I think it's not part of the actual ProviderMachine interface, though [17:31] rog: as I see it, if an instance is associated with a machine, we can find that out from ZK; we might want to interact with i-foobar because it corresponds to machine/27 [17:31] rog: So this again holds up your/William's suggestion that an Instance would make more sense [17:32] rog: but I can't think of a case where we only have i-foobar and need to find out what juju machine it corresponds to [17:33] niemeyer: what isn't? the tag? [17:33] rog: The information used to tag a machine isn't part of the ProviderMachine interface [17:34] niemeyer: you mean the juju.Machine interface? [17:34] (as currently, or juju.Instance as proposed) [17:34] rog: I mean the ProviderMachine interface [17:34] guys, sorry, dinner is ready and apparently it's important that it be eaten hot [17:34] rog: We're still defining how it looks like in juju, so that might be ambiguous [17:34] I'll pop back on when I can [17:35] rog: I can correctly state that about the current code, though [17:35] rog: Sorry, still defining in the Go port [17:35] fwereade: okeydokey [17:35] fwereade: Cheers man [17:35] rog: Ok, in general, I'm happy with yours/William's suggestion of using juju.Instance [17:35] rog: Where today we have ProviderMachine [17:35] niemeyer: by "ProviderMachine" you mean some as-yet-to-be-defined thing? [17:36] ah, the python code! [17:36] rog: Yeah, that forgotten piece of code that happens to be the critical reference! [17:36] doh [17:37] sorry, living in two worlds [17:37] rog: So, for the docs of juju.Instance: it needs to be Machine agnostic, and Provider agnostic, which is interesting [17:38] niemeyer: yeah. that seems ok to me. [17:38] niemeyer: it's already close to that. except that it mentions machine id [17:38] rog: So juju.Instance is the internal representation of that thing that a provider calls a machine :-) [17:38] rog: Yeah, that's the bit worth cleaning up [17:39] niemeyer: definitely. that's good. [17:40] rog: Awesome, thanks for the suggestion.. will be a nice mental clean up :) [17:40] niemeyer: that's kind of how i thought of it before, but i hadn't realised the loose relationship between instance and machine. [17:40] right, Machine -> Instance it is. and instance id becomes defined by Instance rather than being given to StartMachine. [17:41] niemeyer: and for now, i won't provide the ability to tag a machine on startup. that can be added back later. [17:41] rog: I think there may be some confusion taking place still [17:42] niemeyer: quite probably! [17:42] rog: start_machine takes a machine id, and it needs it [17:42] niemeyer: ah. why's that? [17:42] rog: It should probably be called start_instance, though [17:42] beside the wiki, do we have some diagrams showing how the software is organized and the components play together? [17:42] rog: But it needs a machine id still, not an instance id [17:42] TheMue: Sorry, I missed your earlier message [17:43] niemeyer: nopro [17:43] TheMue: I've followed a few threads, but I'm not sure we're referring to the same one [17:43] niemeyer: regarding a talk on the parallel 2012 in May here in Germany [17:43] rog: I suggest following through in the Python code so you understand more of that and see the separation of concepts that already exists [17:43] niemeyer: how does the machine id get used by the provider? [17:44] niemeyer: should not conflict with the next summit [17:44] rog: Despite the name, MachineProvider is already not the same as a state machine [17:44] TheMue: Ah, yeah [17:44] niemeyer: ? [17:44] TheMue: It should be fine for sure [17:44] rog: ! [17:44] See, that's my job.. ;-D [17:45] niemeyer: did i somehow imply that it was? [17:45] rog: We're just evolving an idea.. not every comment has to be disagreement :-) [17:46] rog: Check the code out.. it'll be enlightening [17:46] niemeyer: eek! i wasn't disagreeing! i was trying to find out what you understood by what i said [17:47] niemeyer: maybe we'll find a good topic matching how we use the concurrency of go in juju. [17:47] TheMue: Sounds good.. even though I hope our use of concurrency is simplistic enough to not be a fantastic highlight :-) [17:49] * hazmat finishes lunch [17:50] TheMue, greetings [17:50] niemeyer: my other topic would be concurrent design with message based communication as a more natural approach for software architectures [17:50] hazmat: hi [17:50] that reminds of the start of the project :-) [17:51] TheMue: My first reaction to the topic is that it feels a little bit overcovered already [17:51] niemeyer: AFAICS it looks like the machine-specific security group is just used for opening ports. am i missing something? [17:52] niemeyer: ok, maybe from a too high level. i like the idea of event-driven architectures. [17:52] TheMue, state based approaches have some advantages over message systems, esp. as one considers faults scenarios [17:53] TheMue, i'm a fan as well.. but i think the state based approach is feature of ensemble and makes reasoning about the system much more succinct [17:53] rog: No, that's right.. the machine id in start_machine/instance is used for bootstrapping the node too, though [17:53] rog: So we can't get rid of it [17:54] niemeyer: ah, i must have missed that bit [17:54] hazmat: TheMue is referring to internal architectures [17:54] rog: Otherwise the starting machine has no idea of its purpose in the world (poor guy :-) [17:54] hazmat: event-driven and state based is no opposite for me [17:55] niemeyer: ah, i thought it knew from its boot arguments. [17:55] rog: It does! [17:55] rog: Where do the boot arguments come from? ;-) [17:56] niemeyer: argument to StartInstances? [17:56] rog: Right [17:56] niemeyer: so, i must be missing something. where does the machine id come in there? [17:56] BTW i have to go in 1 minute! [17:57] rog: Check the MachineProvider in the ec2 backend [17:57] rog: start_machine method [17:57] TheMue, i'm missing a context then of what you mean by event driven architecture then [17:59] niemeyer: ah yes. i guess it could be some other unique string but then we wouldn't know how to remove it. [18:02] the state system is observable and the observations could be decomposed into an event based dispatch driving application logic, so they are compatible, but its hard to reason about the usefulness in the abstract, the observation pattern against state is a non concurrent one [18:03] rog: Btw, the _machine_ id is a number [18:03] hazmat: TheMue was really talking about concurrent programming within a process, rather than across processes, I believe [18:09] back again, had to bring my daughter to driving school [18:10] hazmat: i've learned about eda in large systems, with cep for a large number of events in realtime [18:10] TheMue, nice.. custom cep engine? or something like esper? [18:10] hazmat: langs like go and erlang allow to create the same insde one process [18:12] hazmat: would have liked to learn more about esper, only read about it. we tried to establish it as a part of a soa. our part has been done in smalltalk, for a telco [18:13] TheMue, effectively in process eda using coroutine pools for concurrency for a given event dispatch? [18:13] hazmat: my own Tideland ECA will be a framework for go internal event-processing, but not publish/subscribe based. more like connected cells with inputs and outputs [18:15] hazmat: no explicit concurrency controlled by us, only by GemStone/S as the backend [18:16] hazmat: thankfully no high load yet, project is still in progress. but shall be used later for the analysis of outages [18:19] so, have to stop here for a moment and finish my (print) article on Dart [18:30] hazmat: ping [19:31] hi there :-) [19:31] how is juju different from zookeeper (or other similar products)? [19:44] raphink: Well, Juju uses zookeeper - so that's something to think about [19:46] ah ok [19:49] raphink, greetings.. zookeeper is just a coordination storage layer.. it doesn't really do much by itself.. its what the application chooses to do with it thats interesting.. juju is an application that uses zk (mostly as an implementation detail) to provide a language neutral mechanism for defining a service and its dependencies in a reuseable fashion. its like apt for the clouds.. solving dependencies on the network [19:50] juju differs from most of the similar configuration management products in that it focuses on orchestration and encapsulation/reuse of a service definition.. ie. its focused on service orchestration not configuration management [20:17] hazmat: and it makes use of other configuration management products (like puppet/cfengine/chef) under the hood, right? [20:18] raphink, actually it allows a charm author (the definition of the reusable bit) to use whatever other products their comfortable with in whatever language their comfortable with, juju provides communication channels (exposed via clis) and guarantees on when things are called (orchestration).. juju core doesn't have an internal dependency on other CM tools [20:19] I see [20:20] just wondering [20:20] why was the product renamed? [20:52] raphink, for that i have no good answer ;-).. it suffices to say it was done [20:53] hehe I guess it was haakon_ [20:53] sorry, hazmat [20:53] :S [21:07] niemeyer, is there any reason the wtf tests can't also be run against the local provider? [21:07] s/wtf/functional [22:06] <_mup_> Bug #890449 was filed: metadata.yaml needs contact fields < https://launchpad.net/bugs/890449 > [22:20] <_mup_> Bug #890453 was filed: metadata.yaml needs a category / classifier field < https://launchpad.net/bugs/890453 > [22:58] hazmat: No intrinsic reasons [22:58] hazmat: I've not tried to set them up against the local provider, but as long as we can put the system back on a clean state I guess it should work