/srv/irclogs.ubuntu.com/2011/05/04/#ubuntu-ensemble.txt

niemeyerhazmat: I don't follow00:36
niemeyerBut have to step out to pick up Ale now00:36
niemeyerhazmat: Let's talk tomorrow00:36
_mup_ensemble/expose-provisioning r243 committed by jim.baker@canonical.com02:18
_mup_PEP802:18
_mup_ensemble/expose-provisioning r244 committed by jim.baker@canonical.com02:20
_mup_More PEP802:20
_mup_ensemble/expose-provisioning r245 committed by jim.baker@canonical.com02:21
_mup_Merge trunk and resolved conflicts02:21
jimbakerhazmat, it does seem like we are increasingly have random failures due to watches02:30
hazmatjimbaker, there not random02:31
hazmatthey have causes, and tests need to account for it02:31
hazmatjimbaker, if your not clear on why its happening, do a self.capture_output()02:31
jimbakersounds good. so what i'm seeing it is in mostly around cleanup. do you have any good suggestions for avoiding zookeeper.ClosingException: zookeeper is closing exceptions?02:32
hazmatjimbaker, they show up randomly in tests, but a broken test in this context, is pretty much always broken.. 02:32
hazmatjimbaker, it helps to understand which activities cause background operations that need to be synced02:32
hazmatjimbaker, lifecycle.start() is the primary offendor02:32
hazmatjimbaker, you can do a yield self.sleep(0.1) or do it early in your test if has any appreciable time.. or do a yield lifecycle.stop() before the end of the test02:33
hazmatjimbaker, it depends on the context02:33
hazmatjimbaker, on trunk i saw a particular test fail regularly from the endpoint stuff the slow watch callback02:34
jimbakerhazmat, so i'm setting this in a variety of tests: test_watch_exposed_flag_waits_on_slow_callbacks, iirc test_watch_relations_may_defer02:34
hazmater.. not endpoint but exposed02:34
jimbakerhazmat, so we agree on that02:34
jimbakerhazmat, i have been adding sleeps to avoid, and it certainly helps02:35
jimbakerjust not 100%02:35
jimbakercertainly it's much more likely with -u, that's a good way to see them02:36
jimbakerbut if sleep is our best solution for stuff not involved in lifecycles... is that really a solution?02:36
jimbakerhazmat, seems like getting watches right would be very useful for next week02:38
jimbakerhazmat, the other thing is that my current testing of service exposing may have some similarity to the lifecycle testing because it asserts complete cleanup at the end of each test, which seems to be hard with the current watch setup02:40
hazmatjimbaker, there are lots of examples02:42
hazmatjimbaker, i agree 02:42
hazmatjimbaker, its not really a solution, stopping the lifecycle also works, or waiting on a hook execution, it really depends on the context02:43
jimbakerhazmat, agreed on that02:43
_mup_ensemble/expose-dummy-provider r214 committed by jim.baker@canonical.com02:44
_mup_Merged trunk02:44
hazmatjimbaker, getting to niemeyer to understand took much longer than i thought.. i'll see if i can take care of it while we're in budapest, and we can clean up tests incrementally02:44
jimbakerhazmat, sounds like a good plan02:45
_mup_ensemble/expose-dummy-provider r215 committed by jim.baker@canonical.com02:47
_mup_Comments02:47
_mup_ensemble/expose-dummy-provider r216 committed by jim.baker@canonical.com02:55
_mup_Fix upstream changes02:55
_mup_ensemble/expose-provisioning r246 committed by jim.baker@canonical.com02:56
_mup_Merged trunk & resolved conflicts02:56
_mup_ensemble/expose-provisioning r247 committed by jim.baker@canonical.com02:58
_mup_PEP802:58
_mup_ensemble/expose-provisioning r248 committed by jim.baker@canonical.com03:08
_mup_Cleanup03:08
hazmatjimbaker, where you able to fix the tests that failed on trunk in one of those branches?03:39
jimbakerhazmat, i was not03:42
jimbakerhazmat, i do know if i increase the sleep time, i see less failures03:42
jimbakerbut having 2s sleep or whatever seems crazy03:42
hazmatjimbaker, a better way is to find out what the background activity is, and create a test helper that allows you to observe/sync on some post condition of the background activity03:43
hazmatjimbaker, what's it doing in the background in expose watch?03:44
jimbakerhazmat, i have a pretty careful record of the background activity in terms of the logging03:44
jimbakerin terms of removing service units for example, and how that cascades through03:44
jimbakerand as i mentioned, it is more than this new code, it seems common across watches03:45
hazmatjimbaker, try adding a yield callback at the bottom of watch_exposed_flag03:45
jimbakerhazmat, that only makes it worse it seems03:47
hazmatjimbaker, what branch?03:48
hazmatand what test?03:48
jimbakerlp:~jimbaker/ensemble/expose-provisioning, ensemble.agents.tests.test_provision03:49
jimbakerhazmat, taking off for now, but definitely appreciate if you find anything. thanks!03:51
hazmatjimbaker, cool have a good one, fwiw it helps to solve these problems in reverse, at the higher level you get multiple issues nesting, at the lower level you have some hope of sanity building on re-fortified substrates03:52
_mup_ensemble/auto-dependency-resolution r215 committed by kapil.thangavelu@canonical.com03:59
_mup_auto dependency resolver, solves for dependent formulas to be deployed (taking into account what's available in the environment), and new relations that need to be created.03:59
_mup_ensemble/config-get r213 committed by bcsaller@gmail.com05:25
_mup_test config_get communications w and w/o option_name05:25
_mup_ensemble/config-get r214 committed by bcsaller@gmail.com05:30
_mup_cleanup config_set testing method and related config_get->config test streamlining05:30
_mup_ensemble/config-get r215 committed by bcsaller@gmail.com07:35
_mup_test for name/service lookup functions on hook07:35
_mup_ensemble/config-get r216 committed by bcsaller@gmail.com08:40
_mup_prune unused method and exception08:40
_mup_ensemble/config-get r217 committed by bcsaller@gmail.com08:45
_mup_pep808:45
kim0morning everyone09:00
kim0hmm the joined hook is still not in docs ? http://people.canonical.com/~niemeyer/ensemble/formula.html#hooks13:37
hazmatkim0, ugh.. it should be14:30
hazmateveryone's been been heads down implementing14:30
hazmatniemeyer, got a decent auto resolver implementation working14:41
niemeyerhazmat: Oh, sweet!14:45
hazmatniemeyer, there are some broken tests in trunk.. which concern me though14:46
hazmat./test -u ensemble.state.tests.test_service.ServiceStateManagerTest.test_watch_exposed_flag_waits_on_slow_callbacks14:47
hazmatwill end up hanging a terminal hard for me14:47
hazmatstill looking14:47
niemeyerHuh14:47
hazmathmmm.. actually most of the slow watch callbacks can do a hang14:48
_mup_ensemble/auto-dependency-resolution r216 committed by kapil.thangavelu@canonical.com15:00
_mup_test plan, better logging, return formulas objects instead of formula names.15:00
hazmatniemeyer, if your want to take have a look at auto resolve.. its pretty much contained to one file. http://bazaar.launchpad.net/~hazmat/ensemble/auto-dependency-resolution/view/head:/ensemble/formula/resolver.py15:01
niemeyerhazmat: Reading15:09
niemeyerhazmat: {15:11
niemeyer11115:11
niemeyer            "required_by": [(formula_name, dep_name, service_name)],15:11
niemeyer11215:11
niemeyer            "provided_by": None}15:11
niemeyerhazmat: This should be a proper type15:12
hazmatniemeyer, yeah.. there's a todo at the top15:16
hazmatto use named tuples for all the internal data structs15:16
hazmatniemeyer, its very much in an early state ;-)15:16
niemeyerhazmat: Yeah.. :)15:16
niemeyerhazmat: It looks pretty good15:17
niemeyerhazmat: As a hack :)15:17
hazmatits nice to remember what's its like15:17
niemeyerhazmat: Gives an idea of the kind of trouble we're getting into for the full blown implementation15:17
hazmatniemeyer, indeeds, its a very nice exercise for that.15:18
niemeyerhazmat: "depth" provides the wrong idea there15:20
niemeyerhazmat: This is generally used for recursive algorithms15:20
hazmatniemeyer, its tree depth for the resolution15:20
hazmatniemeyer, indeed.. most dep graph solving is done as a dag15:21
niemeyerhazmat: and it's going backwards15:21
hazmatniemeyer, how so?15:22
niemeyer        depth -= 115:22
niemeyer:)15:22
hazmatniemeyer, yeah.. that should be cleaned up.. probably just fix the condition15:23
hazmatand the name15:23
niemeyerhazmat: Nice work man15:24
niemeyerhazmat: It's fantastic we'll have something like that in place for experimenting with15:24
niemeyerhazmat: Does it work? :-)15:25
hazmatniemeyer, indeed it will be fun to show15:25
hazmatniemeyer, i have no idea.. its anti-tdd ;-)15:25
hazmattbd15:25
niemeyerhazmat: :-)15:25
jimbakerkim0, http://people.canonical.com/~niemeyer/ensemble/formula.html is not current against trunk15:26
niemeyerjimbaker: It's not?15:26
jimbakerniemeyer, it's not. the clue for me was seeing "monothonically", which while pythonic in sound, is not a word ;)15:27
jimbaker(i fixed that typo a while ago)15:28
hazmatyeah.. trunk is indeed different 15:29
hazmatby like a several weeks it looks15:30
kim0This hasnt been merged http://bazaar.launchpad.net/~jimbaker/ensemble/sandbox-trunk-r200/revision/20015:30
niemeyerjimbaker: Oops.. :)15:31
jimbakerkim0, not certain what you mean by that... that's a sandbox15:31
niemeyerlet me check that15:31
kim0mm .. then I'm misunderstanding15:31
jimbakerspecifically i just needed something i could deploy on aws, and not knowing how to specify a revision in the bzr url, i just did it that way15:32
jimbakerkim0, i will delete it since we are no longer trying to figure out what happened when we could no longer deploy to aws15:33
niemeyerHmm.. so the branch is up-to-date15:33
niemeyerUgh.. the docs are clearly not15:33
kim0niemeyer: would be great if you'd merge my user docs branch too :)15:34
niemeyerkim0: Yeah, I plan to handle that today still15:34
niemeyerkim0: Thanks for the changes, btw15:34
kim0cool yw15:34
kim0writing a contributing your first formula doc now15:35
kim0is using 'echo' inside formulas an acceptable way to communicate info with user, or should ensemble-log almost always be used ?15:35
kim0coz principia templates use echo, if we don't like that, I'll change them all to ensemble-log15:36
niemeyerjimbaker, kim0: Updated15:43
niemeyerShould continue to update automatically now15:43
niemeyerI've changed it so that it kills the previous version, rather than trying to compile just the difference15:43
niemeyerI think something around that wasn't working properly15:43
kim0why do we have a broken relation, but not established15:47
niemeyerkim0: The best answer is that we haven't missed it15:58
kim0:)15:58
kim0that I'm sure of hehe15:58
niemeyerkim0: In the case of broken, there are obvious things we can do when the service goes disconnected that there's no other place to do15:58
niemeyerkim0: In the case of established, we have good alternatives, such as start15:59
niemeyerkim0: But there's no inherent reason why we shouldn't have it15:59
kim0ok makes sense15:59
niemeyerkim0: If someone comes up with "Oh, I'd like to have established to handle that specific use case."15:59
niemeyerkim0: We can easily add it15:59
* kim0 nods15:59
hazmatthere where handshaking difficulties as i recall, and some questions as to the meaning/utility of established without a join. 16:00
hazmatie. a one sided relation16:00
hazmatwhich join nesc. imparts its not, thus serving as a valid point of establishment for two services to communicate16:01
hazmatit can be added though16:02
hazmattrippy.. ensemble/mine/watching-godot$ make16:10
hazmatYou've just watched the fastest build on earth.16:10
niemeyer:-)16:14
hazmatso gdb shows the hard lock in zk close, looks like something for upstream16:34
niemeyerOur docs have some issues building apparently..16:37
niemeyerLet me look at that16:37
niemeyerhazmat: Hm16:37
niemeyerhazmat: That brings me memories16:38
niemeyerhazmat: I think I've heard about a locked zk close somewhen16:38
niemeyerArgh16:56
niemeyerSphinx is doing pretty weird things :(16:57
niemeyerIt breaks a line with "control-bucket", but not one with "default-instance-type"16:57
niemeyerOk, no warnings anymore17:01
kim0Our meeting kicking in 2 hours .. 17:01
* kim0 rings a little shiny bell17:01
niemeyer:)17:03
* koolhead17 pokes kim017:05
kim0koolhead17: hey o/17:06
kim0koolhead17: you've been hiding lately huh17:06
koolhead17kim0: howdy? 17:06
kim0All going good 17:07
kim0you all good ?17:07
koolhead17kim0: am awesome!!17:07
kim0great :)17:07
koolhead17just trying to not get distracted from real work during working hours in office :D17:08
koolhead17kim0: and yes working on increasing my launchpad karma!! :D17:09
kim0koolhead17: hehe17:09
kim0koolhead17: played with ensemble yet ?17:09
koolhead17kim0: am too occupied with some other stuff natty related. even after office hours. after hitting my head against dhcp server for 48 hours i filed a bug apparmor and stopping execution of dhcpd 17:11
kim0hehe17:11
kim0bugs can sure be fun17:11
koolhead17kim0: yeah when you are deploying something new and you have no documentation supporting you :)17:12
koolhead17hi hazmat TeTeT niemeyer17:12
kim0what were you deploying17:12
koolhead17cobbler :P17:12
kim0a ha 17:12
kim0koolhead17: did that include manual steps ?17:13
koolhead17well am half way mark, able to get PXE install via cobbler17:13
* kim0 wondering if koolhead17 should wrap his knowledge into a cobbler ensemble formula :)17:14
koolhead17few things in default.ks are hardcoded so now have to bang against wall 2morrow figuring out if i have to manually setup local repo and rsync it17:14
kim0while cobbler is not really a cloud service, I can still see value17:14
koolhead17kim0: cobbler is for cloud 4 sure with koan :D17:15
kim0oh! cool :)17:15
kim0at least when openstack is a deploy target for ensemble, it should definitely make sense IMO17:15
niemeyerkoolhead17: Hey!17:15
niemeyerhazmat: Any luck on the lockup problem?17:15
koolhead17kim0: you will be surprized. cobbler has awesome GUI interface for everything but still i simply followed the command line option :D17:17
koolhead17i found GUI too confusing :)17:17
kim0koolhead17: yeah I'm kinda like that as well .. gui is for sissies :)17:18
koolhead17hehe17:18
koolhead17thaks to "zul"  blog17:18
koolhead17*thanks17:19
kim0koolhead17: I've written a largish user level doc for ensemble .. check it out (all the green block text at the end) https://code.launchpad.net/~kim0/ensemble/user-tutorial-and-FAQ/+merge/5886117:19
kim0last time you wanted to get kickstarted I remember17:19
koolhead17kim0: yup17:21
hazmatniemeyer, no, got some lunch17:22
koolhead17kim0: are you going for that developer summit?17:23
koolhead17ubuntu17:23
niemeyerhazmat: Sounds like a good plan, actually17:25
niemeyerI'll get some food too17:25
koolhead17kim0: will be back on this documentation in few hours need to learn some "expect" scripting17:32
kim0koolhead17: yeah I'm going .. should meet Murthy17:41
koolhead17that be great!!17:42
niemeyerOk, actually leaving for lunch now17:43
=== niemeyer is now known as niemeyer_lunch
hazmatinteresting virtualenv seems to strip debugging symbols17:48
=== deryck_ is now known as deryck
=== niemeyer_lunch is now known as niemeyer
jimbakermeeting in #ubuntu-cloud19:00
jimbakerkim0, i assume we are kicking off now, right?19:00
kim0yep19:01
niemeyerhazmat: Do you have a moment for a chat?19:06
hazmatniemeyer, sure19:13
niemeyerSkype or mumble?19:13
jimbakerbcsaller, do you want to join the weekly cloud meeting at #ubuntu-cloud ?19:13
jimbakerjust brought you up19:14
hazmatniemeyer, skype19:14
niemeyerhazmat: Ok19:15
niemeyerhazmat: 19:17
niemeyer+    def do_retry_start(self, fire_hooks=True):19:17
niemeyer+        return self._invoke_lifecycle(19:17
niemeyer+            self._lifecycle.start, fire_hooks=fire_hooks)19:17
hazmatbcsaller, jimbaker standup?19:30
jimbakerhazmat, i was just about to ask the same thing19:30
jimbakerlet me start up skype19:30
bcsallersure19:31
niemeyerjimbaker: map["open"].append({"port": ...})19:46
niemeyerjimbaker: ?19:46
niemeyerjimbaker: 19:52
niemeyerdef expose_port(content, ...):19:52
niemeyer    data = yaml.loads(content)19:52
niemeyer    if not data:19:52
niemeyer        data = {"open": []}19:52
niemeyer    if port not in data["open"]:19:52
niemeyer        data["open"].append(port)19:52
niemeyer    return yaml.dumps(data)19:52
niemeyerjimbaker: the beginning of that should handle empty content as well19:54
niemeyerjimbaker: data = content and yaml.loads(content)19:54
niemeyerjimbaker: That should handle it19:54
_mup_ensemble/watching-godot r216 committed by kapil.thangavelu@canonical.com19:55
_mup_reliable slow watch testing19:55
_mup_ensemble/config-get r218 committed by bcsaller@gmail.com20:03
_mup_docstring cleanups20:03
_mup_ensemble/config-set r215 committed by bcsaller@gmail.com20:05
_mup_resolve merge20:05
niemeyerhazmat: I've added only that single item we discussed to the review of unit-agent-resolved20:11
niemeyerhazmat: It turned out that all of the other issues I had (untested paths, docs missing) are likely going to be rendered irrelevant if you change that20:12
niemeyerbcsaller: Is config-get up for review again, or did I forget to move it to WIP?20:30
bcsallergustavo: its up again, the changes should be pushed20:31
niemeyerbcsaller: Cool, thanks20:32
_mup_ensemble/unit-agent-resolved r270 committed by kapil.thangavelu@canonical.com20:41
_mup_remove passing action transition/state variables20:41
niemeyer"Your trip to Budapest, Hungary is about to begin"20:44
niemeyerTripIt is slightly nervous about trips I see20:44
_mup_ensemble/service-config-unit-lifecycle r217 committed by kapil.thangavelu@canonical.com20:47
_mup_fix up todo comments for post resolved merge work20:47
hazmatso we need a newer version of txaws then is in natty it appears..20:53
hazmatoh.. nm20:53
niemeyerbcsaller: There are still missing tests in config-get20:57
niemeyerbcsaller: E.g.20:57
niemeyer+    def get_formula_state(self):20:57
niemeyerbcsaller: TDD would be very helpful in avoiding that kind of problem20:58
bcsalleryeah, I wrote the tests for the higher levels and then filled in methods to make them pass, I must not have filled in the missing. In reviewing the patch pre-push I didn't even notice though20:59
niemeyerbcsaller: That's not quite TDD21:04
niemeyerbcsaller: TDD is fine-grained.. you have to write several public methods to make a single test pass, the test is probably not fine-grained enough21:05
bcsallergood advise 21:06
niemeyerbcsaller: Whenever you're writing something, and you figure you need support in another area of the application, that other area should *also* be done with TDD21:06
niemeyerbcsaller: Then, before pushing it for review, it's generally good practice to be the reviewer of your own code21:07
niemeyerbcsaller: Actively looking for leftovers, untested areas, etc21:07
niemeyerbcsaller: +        config_state = yield self._setup_config_state()21:07
niemeyer+        yield config_state.write()21:07
niemeyerbcsaller: Untested as well21:07
_mup_ensemble/trunk r217 committed by kapil.thangavelu@canonical.com21:14
_mup_merge service-config-unit-lifecycle [r=niemeyer][f=776596]21:14
_mup_unit lifecycle and workflow work to enable config-changed hooks.21:14
niemeyerbcsaller: config-get reviewed.21:15
bcsallerthanks, I'll look for those places and test them21:16
niemeyerQuick break21:19
_mup_ensemble/watching-godot r217 committed by kapil.thangavelu@canonical.com21:41
_mup_revert changes to watch expose to yield to the first invocation, just fix the tests.21:41
koolhead17kim0: ping21:41
_mup_Bug #777421 was filed: slow watch tests are unreliable <Ensemble:New> < https://launchpad.net/bugs/777421 >21:43
hazmatjimbaker, niemeyer a small branch fix for slow watcher trunk tests is available 21:45
niemeyerhazmat: Super!21:45
hazmatit was probably a trivial21:45
jimbaker hazmat, good to hear21:48
niemeyerTime for a haircut21:51
_mup_ensemble/unit-agent-resolved r271 committed by kapil.thangavelu@canonical.com21:53
_mup_separate transitions for retry with hook21:53
jimbakerniemeyer, now i recall more fully why we want this to be a YAMLState for open ports (and why it will take more time)21:54
jimbakerthe problem is that we want open-port/close-port to participate in the same flush as the overall HookContext21:55
jimbakerthis way we can ensure that hooks that open/close ports have all-or-nothing semantics corresponding to their exit status code21:56
_mup_ensemble/unit-agent-resolved r272 committed by kapil.thangavelu@canonical.com21:58
_mup_merge trunk resolve conflict.21:58
jimbakerhazmat, funny, re bug 777421, i was curious about the whole poke_zk and whether that would be useful or not. not, i guess.22:00
_mup_Bug #777421: slow watch tests are unreliable <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/777421 >22:00
_mup_ensemble/trunk r218 committed by kapil.thangavelu@canonical.com22:03
_mup_merge watching-godo [r=niemeyer][f=777421]22:03
_mup_trivial fix to slow watch callback tests of the service unit api in order22:03
_mup_to reliably run when looped.22:03
_mup_ensemble/trunk-merge r190 committed by kapil.thangavelu@canonical.com22:07
_mup_merge trunk22:07
_mup_ensemble/resolved-state-api r203 committed by kapil.thangavelu@canonical.com22:07
_mup_merge trunk conflict22:07
_mup_ensemble/unit-agent-resolved r273 committed by kapil.thangavelu@canonical.com22:11
_mup_resolve merge conflict22:11
bcsallershort break22:27
_mup_ensemble/unit-agent-resolved r274 committed by kapil.thangavelu@canonical.com22:53
_mup_expand out additional recovery transition actions22:53
_mup_ensemble/expose-provisioning r249 committed by jim.baker@canonical.com23:07
_mup_Do not observe private state directly in tests and fix bad yield23:07
_mup_ensemble/expose-provisioning r250 committed by jim.baker@canonical.com23:08
_mup_Merged trunk23:08
jimbakerhazmat, when is poke_zk still appropriate, given that it is still in the tests? certainly it looks innocuous23:12
hazmatjimbaker, when a round trip to zk is all that's needed to process callbacks, if you have long  running background activity it won't be23:12
jimbakerok, so for ordinary watches, poke_zk is fine, just the specific slow watches cause problems here23:13
hazmatie. using principles of global ordering, we know that there shouldn't be any additional activity.. it might be useful to extend that to a more convoluted poke (watch/set/callback)23:13
hazmatjimbaker, pretty much, but if the watch callback is doing additional zk interaction its dicey23:14
hazmatif its a test contructed callback, its generally fine23:14
jimbakerhazmat, ahh, that certainly limits applicability in my recent work, good way to think about it23:14
jimbakercool, applicable in only one case in my expose-provisioning branch, but that does help23:20
_mup_ensemble/expose-provisioning r251 committed by jim.baker@canonical.com23:21
_mup_Sleep instead of poke for slow watch on openeded ports test23:21
_mup_ensemble/expose-provisioning r252 committed by jim.baker@canonical.com23:50
_mup_Sleep adjustment fu23:50

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!