[03:13] * niemeyer => bed.. night all! [12:15] <_mup_> ensemble/resolved-state-api r198 committed by kapil.thangavelu@canonical.com [12:15] <_mup_> refactor resolved state api to have a more 'friendly' api feel, via relation-set-resolved taking relation-name instead of internal id. [12:16] niemeyer, on the resolved state refactor.. the relation resolved set now takes unit names, but the inverse, get still returns internal ids, but its primary consumer is the unit agent. [12:17] hazmat: Hmm.. that feels a bit icky [12:18] niemeyer, on the icky front, it also means pulling in some ensemble.unit imports into state.service [12:18] hazmat: Ouch [12:18] hazmat: That's a big ickiness :) [12:18] niemeyer, we need to dereference to properly validate [12:18] hazmat: How do you mean? [12:19] niemeyer, well given a rel name by itself, doesn't tell us the workflow state of the relation, which we need to verify if we want to provide feedback to the user [12:20] i've got the feedback aspects of the set worked out, and added some additional unit relation accessor to the unit state. [12:20] but the notion that set/get are not inverses is slightly odd [12:21] hazmat: That seems to indicate a more serious problem in our abstractions [12:21] hazmat: ensemble.unit should be using ensemble.state [12:21] hazmat: Not the other way around [12:21] niemeyer, oddly enough no circular imports at the moment, unit.workflow is self contained atm [12:21] i had to pull in lifecycle for the unit tests though [12:22] in general i agree state should be foundational layer [12:22] hazmat: Maybe it should be moved into the state then.. [12:22] niemeyer, it has no state manipulations. [12:22] hazmat: There's stuff in ensemble.hooks which is fiddling with nodes directly too [12:23] hazmat: That's another one we have to fix [12:23] hazmat: Hmmm [12:23] niemeyer, the only stuff in hooks that manipulates state is via relationcontext atm [12:24] hazmat: Yes [12:24] niemeyer, i don't see anything wrong with ensemble.hooks consuming a state api [12:25] hazmat: Sorry, I'm on crack [12:25] hazmat: I was thinking of ensemble/state/hooks.py, which is already in state [12:26] ah.. right [12:26] Anyway, workflow.. hmmm [12:26] * hazmat pushes his branch [12:26] yield retry_change(self._client, self.zk_state_path, update_state) [12:26] niemeyer its at lp:~hazmat/ensemble/resolved-state-api if your curious [12:27] data, stat = yield self._client.get(self.zk_state_path) [12:28] Indeed workflow should be better integrated in the state [12:28] hazmat: Checking it out [12:43] hazmat: Yeah, I think we should talk about some cleaning up there indeed [12:43] hazmat: The issues we're seeing in this branch are not a problem in the branch itself [12:43] niemeyer, i'm concerned as well about meeting the deadlines for the features entailed by the branch [12:44] hazmat: It's just a side effect of some pending polishing after some organic growth which took place [12:44] given some refactoring work to fix things up [12:44] niemeyer, yeah.. the testing situation in test_service is also need a fix up [12:45] hazmat: Yeah, maybe we should just merge your previous version with this specific aspect uncovered, to get the feature itself in, and then polish a bit the organization [12:45] the relation endpoint stuff never cleaned up some of the tests, so there are like half-dozen ways to set thigns up [12:45] hazmat: For instance, it's bizarre that is_relation_running is a method [12:45] a function you mean and not a method? [12:45] hazmat: We have a RelationStateManager, and a RelationState [12:45] hazmat: Sorry, yeah [12:47] niemeyer, so we should have time at uds then to discuss some of these issues [12:48] hazmat: I'm not sure there's a lot to debate there.. we just have to integrate that aspect better into the state [12:48] hazmat: A UnitState should be aware of its workflow state.. a RelationState should be aware of its workflow state [12:49] hazmat: The workflow stuff itself seems nice on itself.. it's just too isolated from the rest [12:49] hazmat: The question is: how to merge these concepts nicely? [12:52] niemeyer, the workflow just needs a get/set api on the state to push the workflow for zk persistence. There is an analogous api for unit relation settings data (get/set on the unit relation) although in practice that's only used by tests [12:53] hazmat: I think it needs a bit more than that.. e.g. "is running" is something we should be able to inquire in a UnitState [12:54] hazmat: Otherwise it becomes txzookeeper2 [12:54] niemeyer, in that case we need to fold workflow aspects into state [13:32] niemeyer, so i need a path forward, i can revert back and tackle the rest review minus this comment or i can push it forward (the set change is largely done). [13:32] if revert, i'll file an issue regarding this for future [13:33] hazmat: The former sounds like a better plan [13:33] niemeyer, okay.. sounds good [13:33] hazmat: If we're compromising somewhere, the id issue sounds relatively minor compared to the other inconsistencies being introduced [13:34] hazmat: It'd be good to sort the overall problem soonish, though [13:34] hazmat: This will only get worse [13:35] niemeyer, agreed re id minor to the others [13:36] niemeyer, service.py/test_service.py are also getting a bit large.. i think the notion of encapsulating the get/set/clear/watch api into some of staterequestprotocol will help clean that up [13:36] hazmat: I think we already have a good part of it encapsulated into YAMLState [13:37] hazmat: This is effectively read/write/delete/watch [13:37] hazmat: read/write are ready [13:37] <_mup_> Bug #772331 was filed: Incorporate workflow more closely into state < https://launchpad.net/bugs/772331 > [13:39] <_mup_> Bug #772332 was filed: Use a StateRequestProtocol utilizing yamlstate encapsulate get/set/del/watch apis in unit state < https://launchpad.net/bugs/772332 > [13:40] niemeyer, agreed, i just want to take a next step that will allow us to move the state based request/response protocols into their own classes. and expose via composition on the unit [13:41] hazmat: YAMLState is already its own class [13:43] niemeyer, yamlstate is a good start and would be used by a protocol impl, but the usage scenarios are different, yaml state represents a concurrent r/w with dict merge, the protocol implementations are geared more towards request/response.. internally i'm sure they could utilize a yamlstate, but they have additional notions of the api/validation/transformation they want to expose [13:44] hazmat: Indeed, and that's what the methods in the respective state classes do [13:44] hazmat: Note that the purpose of the state layer is precisely to give semantic meaning to what otherwise is just storage [13:45] niemeyer, sure, and the protocols implemenation would be in the state package, part of my reasoning a concern with some of the practical parts of how big the unit state class and its tests are getting. [13:45] hazmat: Yeah, I understand, and I agree there are improvements to be made.. just trying to provide some feedback on them [13:46] hazmat: We shouldn't end up with UnitState.resolved.set(data), UnitState.resolved.get() => data [14:43] <_mup_> ensemble/resolved-state-api r199 committed by kapil.thangavelu@canonical.com [14:43] <_mup_> revert the last commit, removing relation name args to set relation resolved [14:55] <_mup_> ensemble/resolved-state-api r200 committed by kapil.thangavelu@canonical.com [14:55] <_mup_> remove extraneous assertidentical, simplify dict merge, use equality state matching for is_x_running methods. === niemeyer is now known as niemeyer_lunch === niemeyer_lunch is now known as niemeyer [17:43] <_mup_> ensemble/resolved-state-api r201 committed by kapil.thangavelu@canonical.com [17:43] <_mup_> use constant instead of boolean to make the usage clearer for retry_hooks param to unit/relation resolved. [17:54] <_mup_> ensemble/ensemble-resolved r253 committed by kapil.thangavelu@canonical.com [17:54] <_mup_> use new retry hook constants [17:56] <_mup_> ensemble/expose-provisioning r215 committed by jim.baker@canonical.com [17:56] <_mup_> Collect together requested opened ports by machine for eventual provisioning changes [18:09] <_mup_> ensemble/ensemble-resolved r254 committed by kapil.thangavelu@canonical.com [18:09] <_mup_> address syntatic review comments [18:52] <_mup_> ensemble/refactor-to-yamlstate r199 committed by bcsaller@gmail.com [18:52] <_mup_> Changes per review [18:52] <_mup_> Dict copy returned again rather than YAMLState [18:52] <_mup_> Re-enabled tests of dict copies. [18:52] <_mup_> Fix remaining set_data as string calls (converted [18:52] <_mup_> to dict) [18:55] <_mup_> ensemble/refactor-to-yamlstate r200 committed by bcsaller@gmail.com [18:55] <_mup_> merge trunk [18:58] bcsaller: re. [18:58] """ [18:58] I allow YAMLState.read(required=True) which will throw an [18:58] StateNotFound exception which the higher level API can then translate [18:58] """ [18:58] bcsaller: The point was specifically about the way in which the code is organized [18:59] bcsaller: Not about the outside interface [18:59] bcsaller: test + get leaves a race condition at the + [18:59] bcsaller: The suggestion was to just do try: get except: do whatever [19:00] gustavo: YAMLState needed to use the product of that get though and didn't currently take its state via __init__ [19:00] bcsaller: I don't understand.. just do it, and catch an error in case it fails? [19:00] bcsaller: Every try/except involves using the product of the get [19:01] I understood it as "check existence or throw an error" to "get state and use it if possible or throw an error". I made it so yamlstate can get the state and use it [19:02] the get outside of yamlstate didn't avoid the race if I didn't use that data [19:02] and yamlstate was set up to use the data after read [19:03] bcsaller: [19:03] try: [19:03] state = YAMLState(client, path) [19:03] except zookeeper.NoNodeException: [19:03] # not found [19:03] bcsaller: ? [19:03] YAMLState.__init__ isn't async and doesn't check the node [19:04] nor does read normally require it, but there were cases where the existing apis sometimes expect it or check for a specific exception [19:04] in those cases this lets read raise an error if the node didn't exist [19:04] bcsaller: Ok, sorry.. [19:04] which for YAMLState isn't normally an issue as it will try to create it on write [19:04] state = YAMLState() [19:04] try: [19:04] state.read() [19:05] yield state.read() # actually [19:05] except zookeeper.NoNodeException: ... [19:05] ? [19:05] except read didn't throw the error [19:05] I made that behavior optional [19:05] write would just try to create/merge to the path when it was done [19:05] bcsaller: Ok, but the equivalent.. [19:05] so the exception was eaten [19:06] bcsaller: Hmmm [19:06] bcsaller: I see what you mean, sorry [19:06] bcsaller: I misunderstood the comment [19:07] bcsaller: Sounds fine [19:07] there is a question if that type of test is needed at all now though, if the behavior is to create the node in write [19:07] gustavo: maybe I can make the docstring clearer? if you have a suggestion [19:07] bcsaller: Maybe it's not.. the problem was just the race [19:07] bcsaller: I haven't seen the code, which is perhaps the issue [19:08] bcsaller: (changes were not pushed) [19:08] gustavo: pushed [19:09] includes a trunk merge, I was rechecking it for any errors [19:09] bcsaller: pulling [19:10] bcsaller: Yeah, it's fine as it is, sorry for jumping to conclusions [19:10] np [19:11] keeping me honest ;) [19:11] bcsaller: You've mentioned changes in YAMLState and it felt like the actual race wasn't addressed, which clearly is not the case [19:11] bcsaller: So, removing the test.. hmmm [19:11] bcsaller: Do you recall where the relation data is first initialized? [19:12] you mean hook._setup_relation_data or state.relation._add_relation_state? [19:14] (I didn't specify that dotpath very well there, left out the class, but I think you know what I mean) [19:14] * bcsaller fetches coffee [19:14] bcsaller: No, I mean the relation state itself [19:14] bcsaller: We're assuming it exists [19:14] bcsaller: So someone else must have created it [19:15] bcsaller: We can't change that logic without understanding that [19:15] bcsaller: Btw, it looks like none of the changes in protocol.py are required [19:16] bcsaller: It's probably just left over from the previous version [19:17] bcsaller: and the changes to YAMLState are also pending tests [19:17] I did add one test for the required flag raising an exception [19:18] * bcsaller checks it over again [19:20] bcsaller: Heh, yeah I'm on crack again [19:21] <_mup_> ensemble/expose-provisioning r216 committed by jim.baker@canonical.com [19:21] <_mup_> Remove requested opened ports upon a service no longer being exposed [19:22] Is it just me or the mouse can't click in Natty every once in a while? [19:23] niemeyer, i have seen something like this - just gets briefly frozen up [19:23] in terms of user input [19:23] Yeah [19:24] timely event loop processing, it's an art ;) [19:24] Not just briefly actually [19:24] Can barely use the trackpad [19:26] for me i have seen pauses up to maybe 15s, especially when i have initially started. pretty rare however. everything else iirc seems to work in terms of normal refreshing [19:26] just the input processing [19:28] Is there a reset-mouse? :) [19:28] <_mup_> ensemble/unit-agent-resolved r256 committed by kapil.thangavelu@canonical.com [19:28] <_mup_> pass transition variables to action. [19:28] Switching to the terminal usually works, but not this time [19:29] niemeyer, restarting unity from a vterm didn't help? [19:30] No, generally kills the whole thing [19:30] niemeyer, its worked for me pretty consistently (vterm, kill compiz & gnome-screensaver, unity --replace, back to desktop) [19:31] hazmat`, that definitely helped with the true freezes. need to attach that on a sticky note to my laptop ;) [19:31] hazmat`: Yeah, that seems to have worked [19:51] multi az/multi region mongodb replica set http://blog.mongodb.org/post/4982676520/mongodb-on-ec2-best-practices [19:58] trunk has surpassed 1k tests, jimbaker you get a prize ;-) === andreas__ is now known as ahasenack [20:11] bcsaller, fwiw, i didn't get any errors on current trunk [20:24] bcsaller, thanks for fixing refactor-to-yamlstate, that's working for me now [20:24] jimbaker: woot [21:19] <_mup_> ensemble/unit-agent-resolved r257 committed by kapil.thangavelu@canonical.com [21:19] <_mup_> fold fire_hooks into transition-variables [21:23] <_mup_> ensemble/unit-agent-resolved r258 committed by kapil.thangavelu@canonical.com [21:23] <_mup_> unit workflow retry actions take an optional flag for fire hooks, workflow definition specifies alias for retry transitions. [21:34] <_mup_> ensemble/unit-agent-resolved r259 committed by kapil.thangavelu@canonical.com [21:34] <_mup_> lifecycle methods sans hook tests [21:35] <_mup_> ensemble/refactor-to-yamlstate r201 committed by bcsaller@gmail.com [21:35] <_mup_> revert minor changes and unused import [21:51] niemeyer, got a moment? [22:08] <_mup_> ensemble/unit-agent-resolved r260 committed by kapil.thangavelu@canonical.com [22:08] <_mup_> allow unit resolved watch to stop watching. [22:09] <_mup_> ensemble/unit-agent-resolved r261 committed by kapil.thangavelu@canonical.com [22:09] <_mup_> allow unit resolved watch to stop watching. [22:16] <_mup_> ensemble/unit-agent-resolved r261 committed by kapil.thangavelu@canonical.com [22:16] <_mup_> allow unit relation resolved watch to stop watching. [22:18] <_mup_> ensemble/trunk r214 committed by bcsaller@gmail.com [22:18] <_mup_> Merge refactor to YAMLState. [r=niemeyer] [f=770448] [22:18] <_mup_> This branch refactors relation settings to use YAMLState internally. [22:18] <_mup_> Changes relation_set commands to take a dict rather than a YAML serialized dict . [22:20] hazmat`: Sorry, missed your question, and about to step out for a break with Ale [22:20] hazmat`: I'll be back in 1h or so, though [22:21] hazmat`: Can we talk then? === niemeyer is now known as niemeyer_bbl [22:59] <_mup_> ensemble/expose-provisioning r217 committed by jim.baker@canonical.com [22:59] <_mup_> Minimal support for eventual consistency in provider firewalls [23:00] niemeyer_bbl, tomorrow [23:00] i'm out to a hadoop meetup [23:03] hazmat`, have fun [23:03] jimbaker, thanks [23:04] reminds me, i promised to give a talk on ensemble + zk for our local hadoop/big data meetup in june [23:21] hazmat`: Sounds good, enjoy!