[03:13]  * niemeyer => bed.. night all!
[12:15] <_mup_> ensemble/resolved-state-api r198 committed by kapil.thangavelu@canonical.com
[12:15] <_mup_> refactor resolved state api to have a more 'friendly' api feel, via relation-set-resolved taking relation-name instead of internal id.
[12:16] <hazmat> niemeyer, on the resolved state refactor.. the relation resolved set now takes unit names, but the inverse, get still returns internal ids, but its primary consumer is the unit agent.
[12:17] <niemeyer> hazmat: Hmm.. that feels a bit icky
[12:18] <hazmat> niemeyer, on the icky front, it also means pulling in some ensemble.unit imports into state.service
[12:18] <niemeyer> hazmat: Ouch
[12:18] <niemeyer> hazmat: That's a big ickiness :)
[12:18] <hazmat> niemeyer, we need to dereference to properly validate
[12:18] <niemeyer> hazmat: How do you mean?
[12:19] <hazmat> niemeyer, well given a rel name by itself, doesn't tell us the workflow state of the  relation, which we need to verify if we want to provide feedback to the user
[12:20] <hazmat> i've got the feedback aspects of the set worked out, and added some additional unit relation accessor to the unit state.
[12:20] <hazmat> but the notion that set/get are not inverses is slightly odd
[12:21] <niemeyer> hazmat: That seems to indicate a more serious problem in our abstractions 
[12:21] <niemeyer> hazmat: ensemble.unit should be using ensemble.state
[12:21] <niemeyer> hazmat: Not the other way around
[12:21] <hazmat> niemeyer, oddly enough no circular imports at the moment, unit.workflow is self contained atm
[12:21] <hazmat> i had to pull in lifecycle for the unit tests though
[12:22] <hazmat> in general i agree state should be foundational layer
[12:22] <niemeyer> hazmat: Maybe it should be moved into the state then..
[12:22] <hazmat> niemeyer, it has no state manipulations.
[12:22] <niemeyer> hazmat: There's stuff in ensemble.hooks which is fiddling with nodes directly too
[12:23] <niemeyer> hazmat: That's another one we have to fix
[12:23] <niemeyer> hazmat: Hmmm
[12:23] <hazmat> niemeyer, the only stuff in hooks that manipulates state is via relationcontext atm
[12:24] <niemeyer> hazmat: Yes
[12:24] <hazmat> niemeyer, i don't see anything wrong with ensemble.hooks consuming a state api
[12:25] <niemeyer> hazmat: Sorry, I'm on crack
[12:25] <niemeyer> hazmat: I was thinking of ensemble/state/hooks.py, which is already in state
[12:26] <hazmat> ah.. right
[12:26] <niemeyer> Anyway, workflow.. hmmm
[12:26]  * hazmat pushes his branch
[12:26] <niemeyer>         yield retry_change(self._client, self.zk_state_path, update_state)
[12:26] <hazmat> niemeyer its at lp:~hazmat/ensemble/resolved-state-api if your curious
[12:27] <niemeyer>             data, stat = yield self._client.get(self.zk_state_path)
[12:28] <niemeyer> Indeed workflow should be better integrated in the state
[12:28] <niemeyer> hazmat: Checking it out
[12:43] <niemeyer> hazmat: Yeah, I think we should talk about some cleaning up there indeed
[12:43] <niemeyer> hazmat: The issues we're seeing in this branch are not a problem in the branch itself
[12:43] <hazmat> niemeyer, i'm concerned as well about meeting the deadlines for the features entailed by the branch
[12:44] <niemeyer> hazmat: It's just a side effect of some pending polishing after some organic growth which took place
[12:44] <hazmat> given some refactoring work to fix things up
[12:44] <hazmat> niemeyer, yeah.. the testing situation in test_service is also need a fix up
[12:45] <niemeyer> hazmat: Yeah, maybe we should just merge your previous version with this specific aspect uncovered, to get the feature itself in, and then polish a bit the organization
[12:45] <hazmat> the relation endpoint stuff never cleaned up some of the tests, so there are like half-dozen ways to set thigns up
[12:45] <niemeyer> hazmat: For instance, it's bizarre that is_relation_running is a method
[12:45] <hazmat> a function you mean and not a method?
[12:45] <niemeyer> hazmat: We have a RelationStateManager, and a RelationState
[12:45] <niemeyer> hazmat: Sorry, yeah
[12:47] <hazmat> niemeyer, so we should have time at uds then to discuss some of these issues
[12:48] <niemeyer> hazmat: I'm not sure there's a lot to debate there.. we just have to integrate that aspect better into the state
[12:48] <niemeyer> hazmat: A UnitState should be aware of its workflow state.. a RelationState should be aware of its workflow state
[12:49] <niemeyer> hazmat: The workflow stuff itself seems nice on itself.. it's just too isolated from the rest
[12:49] <niemeyer> hazmat: The question is: how to merge these concepts nicely?
[12:52] <hazmat> niemeyer,  the workflow just needs a get/set api on the state to push the workflow for zk persistence. There is an analogous api for unit relation settings data (get/set on the unit relation) although in practice that's only used by tests
[12:53] <niemeyer> hazmat: I think it needs a bit more than that.. e.g. "is running" is something we should be able to inquire in a UnitState
[12:54] <niemeyer> hazmat: Otherwise it becomes txzookeeper2
[12:54] <hazmat> niemeyer, in that case we need to fold workflow aspects into state
[13:32] <hazmat> niemeyer, so i need a path forward, i can revert back and tackle the rest review minus this comment or i can push it forward (the set change is largely done).
[13:32] <hazmat> if revert, i'll file an issue regarding this for future
[13:33] <niemeyer> hazmat: The former sounds like a better plan
[13:33] <hazmat> niemeyer, okay.. sounds good
[13:33] <niemeyer> hazmat: If we're compromising somewhere, the id issue sounds relatively minor compared to the other inconsistencies being introduced
[13:34] <niemeyer> hazmat: It'd be good to sort the overall problem soonish, though
[13:34] <niemeyer> hazmat: This will only get worse
[13:35] <hazmat> niemeyer, agreed re id minor to the others
[13:36] <hazmat> niemeyer, service.py/test_service.py are also getting a bit large.. i think the notion of encapsulating the get/set/clear/watch api into some of staterequestprotocol will help clean that up
[13:36] <niemeyer> hazmat: I think we already have a good part of it encapsulated into YAMLState
[13:37] <niemeyer> hazmat: This is effectively read/write/delete/watch
[13:37] <niemeyer> hazmat: read/write are ready
[13:37] <_mup_> Bug #772331 was filed: Incorporate workflow more closely into state <Ensemble:New> < https://launchpad.net/bugs/772331 >
[13:39] <_mup_> Bug #772332 was filed: Use a StateRequestProtocol utilizing yamlstate encapsulate get/set/del/watch apis in unit state <Ensemble:New> < https://launchpad.net/bugs/772332 >
[13:40] <hazmat> niemeyer, agreed, i just want to take a next step that will allow us to move the state based request/response protocols into their own classes. and expose via composition on the unit
[13:41] <niemeyer> hazmat: YAMLState is already its own class
[13:43] <hazmat> niemeyer, yamlstate is a good start and would be used by a protocol impl, but the usage scenarios are different, yaml state represents a concurrent r/w with dict merge, the  protocol implementations are geared more towards request/response.. internally i'm sure they could utilize a yamlstate, but they have additional notions of the api/validation/transformation they want to expose
[13:44] <niemeyer> hazmat: Indeed, and that's what the methods in the respective state classes do
[13:44] <niemeyer> hazmat: Note that the purpose of the state layer is precisely to give semantic meaning to what otherwise is just storage
[13:45] <hazmat> niemeyer, sure, and the protocols implemenation would be in the state package,  part of my reasoning a  concern with some of the practical parts of how big the unit state class and its tests are getting.
[13:45] <niemeyer> hazmat: Yeah, I understand, and I agree there are improvements to be made.. just trying to provide some feedback on them
[13:46] <niemeyer> hazmat: We shouldn't end up with UnitState.resolved.set(data), UnitState.resolved.get() => data
[14:43] <_mup_> ensemble/resolved-state-api r199 committed by kapil.thangavelu@canonical.com
[14:43] <_mup_> revert the last commit, removing relation name args to set relation resolved
[14:55] <_mup_> ensemble/resolved-state-api r200 committed by kapil.thangavelu@canonical.com
[14:55] <_mup_> remove extraneous assertidentical, simplify dict merge, use equality state matching for is_x_running methods.
[17:43] <_mup_> ensemble/resolved-state-api r201 committed by kapil.thangavelu@canonical.com
[17:43] <_mup_> use constant instead of boolean to make the usage clearer for retry_hooks param to unit/relation resolved.
[17:54] <_mup_> ensemble/ensemble-resolved r253 committed by kapil.thangavelu@canonical.com
[17:54] <_mup_> use new retry hook constants
[17:56] <_mup_> ensemble/expose-provisioning r215 committed by jim.baker@canonical.com
[17:56] <_mup_> Collect together requested opened ports by machine for eventual provisioning changes
[18:09] <_mup_> ensemble/ensemble-resolved r254 committed by kapil.thangavelu@canonical.com
[18:09] <_mup_> address syntatic review comments
[18:52] <_mup_> ensemble/refactor-to-yamlstate r199 committed by bcsaller@gmail.com
[18:52] <_mup_> Changes per review
[18:52] <_mup_> Dict copy returned again rather than YAMLState
[18:52] <_mup_> Re-enabled tests of dict copies.
[18:52] <_mup_> Fix remaining set_data as string calls (converted
[18:52] <_mup_>  to dict)
[18:55] <_mup_> ensemble/refactor-to-yamlstate r200 committed by bcsaller@gmail.com
[18:55] <_mup_> merge trunk
[18:58] <niemeyer> bcsaller: re.
[18:58] <niemeyer> """
[18:58] <niemeyer> I allow YAMLState.read(required=True) which will throw an
[18:58] <niemeyer> StateNotFound exception which the higher level API can then translate
[18:58] <niemeyer> """
[18:58] <niemeyer> bcsaller: The point was specifically about the way in which the code is organized
[18:59] <niemeyer> bcsaller: Not about the outside interface
[18:59] <niemeyer> bcsaller: test + get leaves a race condition at the +
[18:59] <niemeyer> bcsaller: The suggestion was to just do try: get except: do whatever
[19:00] <bcsaller> gustavo: YAMLState needed to use the product of that get though and didn't currently take its state via __init__
[19:00] <niemeyer> bcsaller: I don't understand.. just do it, and catch an error in case it fails?
[19:00] <niemeyer> bcsaller: Every try/except involves using the product of the get
[19:01] <bcsaller> I understood it as "check existence or throw an error" to "get state and use it if possible or throw an error". I made it so yamlstate can get the state and use it
[19:02] <bcsaller> the get outside of yamlstate didn't avoid the race if I didn't use that data
[19:02] <bcsaller> and yamlstate was set up to use the data after read
[19:03] <niemeyer> bcsaller:
[19:03] <niemeyer> try:
[19:03] <niemeyer>     state = YAMLState(client, path)
[19:03] <niemeyer> except zookeeper.NoNodeException:
[19:03] <niemeyer>     # not found
[19:03] <niemeyer> bcsaller: ?
[19:03] <bcsaller> YAMLState.__init__ isn't async and doesn't check the node
[19:04] <bcsaller> nor does read normally require it, but there were cases where the existing apis sometimes expect it or check for a specific exception
[19:04] <bcsaller> in those cases this lets read raise an error if the node didn't exist
[19:04] <niemeyer> bcsaller: Ok, sorry..
[19:04] <bcsaller> which for YAMLState isn't normally an issue as it will try to create it on write
[19:04] <niemeyer> state = YAMLState()
[19:04] <niemeyer> try:
[19:04] <niemeyer>     state.read()
[19:05] <niemeyer>     yield state.read() # actually
[19:05] <niemeyer> except zookeeper.NoNodeException: ...
[19:05] <niemeyer> ?
[19:05] <bcsaller> except read didn't throw the error
[19:05] <bcsaller> I made that behavior optional 
[19:05] <bcsaller> write would just try to create/merge to the path when it was done 
[19:05] <niemeyer> bcsaller: Ok, but the equivalent..
[19:05] <bcsaller> so the exception was eaten
[19:06] <niemeyer> bcsaller: Hmmm
[19:06] <niemeyer> bcsaller: I see what you mean, sorry
[19:06] <niemeyer> bcsaller: I misunderstood the comment
[19:07] <niemeyer> bcsaller: Sounds fine
[19:07] <bcsaller> there is a question if that type of test is needed at all now though, if the behavior is to create the node in write 
[19:07] <bcsaller> gustavo: maybe I can make the docstring clearer? if you have a suggestion
[19:07] <niemeyer> bcsaller: Maybe it's not.. the problem was just the race
[19:07] <niemeyer> bcsaller: I haven't seen the code, which is perhaps the issue
[19:08] <niemeyer> bcsaller: (changes were not pushed)
[19:08] <bcsaller> gustavo: pushed
[19:09] <bcsaller> includes a trunk merge, I was rechecking it for any errors
[19:09] <niemeyer> bcsaller: pulling
[19:10] <niemeyer> bcsaller: Yeah, it's fine as it is, sorry for jumping to conclusions
[19:10] <bcsaller> np
[19:11] <bcsaller> keeping me honest ;)
[19:11] <niemeyer> bcsaller: You've mentioned changes in YAMLState and it felt like the actual race wasn't addressed, which clearly is not the case
[19:11] <niemeyer> bcsaller: So, removing the test.. hmmm
[19:11] <niemeyer> bcsaller: Do you recall where the relation data is first initialized?
[19:12] <bcsaller> you mean hook._setup_relation_data or state.relation._add_relation_state?
[19:14] <bcsaller> (I didn't specify that dotpath very well there, left out the class, but I think you know what I mean)
[19:14]  * bcsaller fetches coffee
[19:14] <niemeyer> bcsaller: No, I mean the relation state itself
[19:14] <niemeyer> bcsaller: We're assuming it exists
[19:14] <niemeyer> bcsaller: So someone else must have created it
[19:15] <niemeyer> bcsaller: We can't change that logic without understanding that
[19:15] <niemeyer> bcsaller: Btw, it looks like none of the changes in protocol.py are required
[19:16] <niemeyer> bcsaller: It's probably just left over from the previous version
[19:17] <niemeyer> bcsaller: and the changes to YAMLState are also pending tests
[19:17] <bcsaller> I did add one test for the required flag raising an exception
[19:18]  * bcsaller checks it over again 
[19:20] <niemeyer> bcsaller: Heh, yeah I'm on crack again
[19:21] <_mup_> ensemble/expose-provisioning r216 committed by jim.baker@canonical.com
[19:21] <_mup_> Remove requested opened ports upon a service no longer being exposed
[19:22] <niemeyer> Is it just me or the mouse can't click in Natty every once in a while?
[19:23] <jimbaker>  niemeyer, i have seen something like this - just gets briefly frozen up
[19:23] <jimbaker> in terms of user input
[19:23] <niemeyer> Yeah
[19:24] <jimbaker> timely event loop processing, it's an art ;)
[19:24] <niemeyer> Not just briefly actually
[19:24] <niemeyer> Can barely use the trackpad
[19:26] <jimbaker> for me i have seen pauses up to maybe 15s, especially when i have initially started. pretty rare however. everything else iirc seems to work in terms of normal refreshing
[19:26] <jimbaker> just the input processing
[19:28] <niemeyer> Is there a reset-mouse? :)
[19:28] <_mup_> ensemble/unit-agent-resolved r256 committed by kapil.thangavelu@canonical.com
[19:28] <_mup_> pass transition variables to action.
[19:28] <niemeyer> Switching to the terminal usually works, but not this time
[19:29] <hazmat`> niemeyer, restarting unity from a vterm didn't help?
[19:30] <niemeyer> No, generally kills the whole thing
[19:30] <hazmat`> niemeyer,  its worked for me pretty consistently (vterm, kill compiz & gnome-screensaver, unity --replace, back to desktop) 
[19:31] <jimbaker> hazmat`, that definitely helped with the true freezes. need to attach that on a sticky note to my laptop ;)
[19:31] <niemeyer> hazmat`: Yeah, that seems to have worked
[19:51] <hazmat`> multi az/multi region mongodb replica set http://blog.mongodb.org/post/4982676520/mongodb-on-ec2-best-practices
[19:58] <hazmat`> trunk has surpassed 1k tests, jimbaker you get a prize ;-)
[20:11] <hazmat`> bcsaller, fwiw, i didn't get any errors on current trunk
[20:24] <jimbaker> bcsaller, thanks for fixing refactor-to-yamlstate, that's working for me now
[20:24] <bcsaller> jimbaker: woot
[21:19] <_mup_> ensemble/unit-agent-resolved r257 committed by kapil.thangavelu@canonical.com
[21:19] <_mup_> fold fire_hooks into transition-variables
[21:23] <_mup_> ensemble/unit-agent-resolved r258 committed by kapil.thangavelu@canonical.com
[21:23] <_mup_> unit workflow retry actions take an optional flag for fire hooks, workflow definition specifies alias for retry transitions.
[21:34] <_mup_> ensemble/unit-agent-resolved r259 committed by kapil.thangavelu@canonical.com
[21:34] <_mup_> lifecycle methods sans hook tests
[21:35] <_mup_> ensemble/refactor-to-yamlstate r201 committed by bcsaller@gmail.com
[21:35] <_mup_> revert minor changes and unused import
[21:51] <hazmat`> niemeyer, got a moment?
[22:08] <_mup_> ensemble/unit-agent-resolved r260 committed by kapil.thangavelu@canonical.com
[22:08] <_mup_> allow unit resolved watch to stop watching.
[22:09] <_mup_> ensemble/unit-agent-resolved r261 committed by kapil.thangavelu@canonical.com
[22:09] <_mup_> allow unit resolved watch to stop watching.
[22:16] <_mup_> ensemble/unit-agent-resolved r261 committed by kapil.thangavelu@canonical.com
[22:16] <_mup_> allow unit relation resolved watch to stop watching.
[22:18] <_mup_> ensemble/trunk r214 committed by bcsaller@gmail.com
[22:18] <_mup_> Merge refactor to YAMLState. [r=niemeyer] [f=770448]
[22:18] <_mup_> This branch refactors relation settings to use YAMLState internally. 
[22:18] <_mup_> Changes relation_set commands to take a dict rather than a YAML serialized dict .
[22:20] <niemeyer> hazmat`: Sorry, missed your question, and about to step out for a break with Ale
[22:20] <niemeyer> hazmat`: I'll be back in 1h or so, though
[22:21] <niemeyer> hazmat`: Can we talk then?
[22:59] <_mup_> ensemble/expose-provisioning r217 committed by jim.baker@canonical.com
[22:59] <_mup_> Minimal support for eventual consistency in provider firewalls
[23:00] <hazmat`> niemeyer_bbl, tomorrow
[23:00] <hazmat`> i'm out to a hadoop meetup
[23:03] <jimbaker> hazmat`, have fun
[23:03] <hazmat`> jimbaker, thanks
[23:04] <jimbaker> reminds me, i promised to give a talk on ensemble + zk for our local hadoop/big data meetup in june
[23:21] <niemeyer_bbl> hazmat`: Sounds good, enjoy!