[00:17] 2011-05-31 16:17:46,019 ERROR Formula %r is the latest revision known [00:18] DOH [00:22] Heh.. munin really is too much for a t1.micro. :-/ [00:23] http://ec2-50-16-148-122.compute-1.amazonaws.com/munin/ [00:23] mmmmmm... graphs [00:51] hazmat: hit the problem again. :( [00:57] SpamapS: nice graphs :) [01:00] bcsaller: I'm besieging the cluster righ tnow.... about to get more interesting! :) [01:00] press record :) [01:01] top - 00:00:59 up 6:56, 1 user, load average: 12.00, 4.46, 1.65 [01:01] 2011-05-31 16:59:30,485 provision:ec2: ensemble.agents.provision INFO: Starting machine id:11 ... [01:01] bcsaller: I'm thinking of writing a "test monster" formula that will crawl and try to destroy a website.. :) [01:02] terrifying [01:02] blammo.. load drops to 4 new box up [01:02] I also wonder if my t1.micro's are getting throttled [01:03] might be interesting to reboot a few as c1.medium [01:04] hmm. I really should be hitting the 10.x address so I can just leave this running all night. ;) [01:31] SpamapS, do you have the relation hook installing munin-node? [01:31] hazmat: yes [01:32] hazmat: will have to discuss later, super late heading out the door.. :-P [01:32] hazmat: any tips though, let me know I'll give it a shot [01:32] SpamapS, me too.. cheers [01:32] SpamapS, i have to rebuild the txzk packages to get that fix pushed out.. i'll have a look at it in the morning [03:03] <_mup_> ensemble/expose-provision-service-hierarchy r255 committed by jim.baker@canonical.com [03:03] <_mup_> Updated watch_exposed_flag (about to be moved into separate branch) [03:05] <_mup_> ensemble/expose-watch-exposed-flag r240 committed by jim.baker@canonical.com [03:05] <_mup_> Changes to watch_exposed_flag moved from expose-provision-service-hierarchy [03:09] <_mup_> ensemble/expose-provision-service-hierarchy r256 committed by jim.baker@canonical.com [03:09] <_mup_> Removed watch_exposed_flag changes in this branch [03:10] <_mup_> ensemble/expose-provision-service-hierarchy r257 committed by jim.baker@canonical.com [03:10] <_mup_> Merged in expose-watch-exposed-flag branch [03:20] <_mup_> ensemble/expose-provision-service-hierarchy r258 committed by jim.baker@canonical.com [03:20] <_mup_> Nonexistent service terminates corresponding watch on exposed flag [03:23] <_mup_> ensemble/expose-provision-service-hierarchy r259 committed by jim.baker@canonical.com [03:23] <_mup_> PEP8, removed unnecessary debug statements [04:27] <_mup_> ensemble/expose-provision-service-hierarchy r260 committed by jim.baker@canonical.com [04:27] <_mup_> Better docstrings + cleanup [05:13] <_mup_> Bug #791035 was filed: removed formula components are not cleaned up when upgrading < https://launchpad.net/bugs/791035 > [05:38] <_mup_> Bug #791042 was filed: *-relation-broken has no way to identify which remote service is being broken < https://launchpad.net/bugs/791042 > [13:42] ERROR Formula %r is the latest revision known [13:42] Is %r supposed to be substitued for by something [14:04] anyone around today [14:15] Good morning all! [14:19] Morning :) [14:21] kim0: Hey! [14:33] * kim0 trying to write a new formula [14:35] hmm .. in debug-hooks, # relation-list --format json [14:35] why am I getting No ENSEMBLE_AGENT_SOCKET/-s option found [14:38] kim0, are you in the window that popped up for the hook or just the default terminal? [14:38] re debug [14:38] the one that poped [14:39] kim0, if you do ENV | grep ENSEMBLE what do you see? [14:39] er. env [14:39] sorry, I had closed the window .. perhaps I shouldn't have [14:40] kim0, no worries, if it happens again let me know, i'd be happy to take a look [14:40] hazmat: does debug-hooks execute the hook + give me shell, or just shell [14:40] kim0, hook + shell [14:40] the interactive shell replaces the hook execution [14:40] the exit of the shell is the considered the exit of the hook [14:41] so the original hook script is NOT executed [14:41] kim0, correct [14:42] hazmat: Good morning [14:42] * hazmat is on his 3rd unity reboot of the last 24hrs. [14:42] niemeyer, g'morning [14:42] hazmat: From the logs, it looks like SpamapS got the same issue again last night [14:43] niemeyer, he did, i did a fix for the top level symptom which is committed on txzookeeper trunk [14:43] niemeyer, i'm still tracking down in the zookeeper client where/why the event occurs, its a session event [14:43] so it hits all the watchers extant [14:43] hazmat: I'm not sure that really fixes it.. right.. it's a session timeout, so it'll continue breaking [14:44] niemeyer, indeed [14:44] hazmat: Btw, the quotes were inconsistent, but that's minor [14:45] niemeyer, i'm wondering if we need some lower level bookeeping in txzk client to be able to attach a handler for session events, what's unclear to me atm is if its a session expired scenario or not ( ie. recreate all watches and ephemerals) [14:45] hazmat: Is the session issue happening before or after the already called error? [14:45] before [14:46] hazmat: SpamapS said something about a micro machine being too small for whatever he was doing.. it may just be locked up completely for so long that it times out [14:46] niemeyer, yeah.. i think we need to up our session timeout values on the zk server, there's a faq on this for ec2 [14:47] and the latencies inherent in that environment [14:47] hazmat: Sounds good, if he's able to reproduce it easily, it would be good to try again under more controlled conditions [14:47] but that doesn't excuse that we need some handling for this behavior [14:47] hazmat: Indeed [14:47] hazmat: In gozk, I redirect session events to a specific channel [14:48] niemeyer, yeah.. i'm going to try and reproduce via manipulation of the ec2 firewall on the bootstrap node [14:48] hazmat: and as an experiment, I'm panicing the application if the session event isn't handled in a given amount of time [14:48] niemeyer, yeah.. that's basically along the lines of what i was thinking for txzk [14:48] hazmat: IOW, if the app isn't acknowledging the fact a session event happened [14:48] use a dedicated session event handler, and route those events there. [14:48] we'd also need to track the outstanding watches if we want to reduce the redunancy from the zk c client [14:48] hazmat: Either way, we'll likely have to kill all the running callbacks [14:48] which notifies on all watchers [14:49] hmm, why does relation-list only result in ["mysql/0"] (no database, user, password ) ..etc This is in db-relation-changed second invocation [14:49] kim0: That's the relation name, not really what you want [14:49] kim0: Check relation-get [14:49] niemeyer, it depends.. does it mean the session is expired.. or not [14:49] hazmat: I think the only thing coming through those channels is up/down notes [14:50] hazmat: But it'd be worth double checking [14:50] relation-get - --format json results in {} [14:50] kim0: That means the other side hasn't relation-set anything [14:50] facing this consistently for the past hour .. I used remove-relation and add-relation many times [14:50] the other side is mysql from examples [14:51] kim0: Hmm [14:51] should I login and check if a DB has been created for my instance (drupal) ? [14:51] kim0, if your debugging both sides interactively [14:51] just one side [14:51] the values aren't flushed for the remote to see till the hook/debug is ended [14:52] er. the debug window for the hook is ended [14:52] hazmat: I'm not debugging mysql side, so that shouldn't be a problem corrrect [14:52] kim0: Kind of.. [14:52] kim0: Try logging out and in again, just to try it out [14:52] kim0: From the debug-hook session [14:52] kim0, are you in the relation join event or the changed event? [14:52] hazmat: in "changed" in its 2nd invocation [14:53] kim0, hmm. that sounds like the right place then [14:53] yeah and I've waited for mysql to create its stuff [14:53] kim0: Try logging out and in [14:53] kim0: From the debug session [14:53] ok [14:53] kim0: and then do the relation-get [14:54] Oh, I think there's a bug, now that you mention it.. [14:54] niemeyer: so I closed the screen window [14:54] opening it again [14:54] kim0: Yeah, I suspect it won't work [14:54] kim0: Our example is doing some magic it shouldn't [14:55] niemeyer: so I should kill mysql and start a new one ? [14:55] kim0: Follow with me so that you can actually fix it and try again [14:55] ok [14:55] kim0: Go to the mysql formula, and look at the joined hook [14:56] opened [14:56] kim0: You see that check under the comment # Determine if ... [14:56] yeah [14:57] kim0: You see, it's bumping out of the hook if the _service_ already has a database created [14:57] which makes sense to me ?! [14:57] what's the problem [14:57] ah [14:57] kim0: The problem is that it bumps out without ever letting the _relation_ to know of the settings [14:57] so it's not relation-set'ing [14:57] kim0: yep [14:57] got it [14:58] doh [14:58] but it must have relation-set'ed the first time [14:58] is that lost ? [14:58] kim0: If you [14:58] kim0: Lost? [14:58] why is the relation-set from first time not available ? [14:58] kim0: Oh, yeah, it seems to be [14:58] ok [14:58] if the service name is reused [14:58] I've just been remove-relation and add-relation [14:59] ie relate worpdress mysql.. destroy wordpress, recreate wordpress and relate [14:59] hazmat: Ugh.. that's bad [14:59] yeah.. i'm not sure this was a problem before with the mysql formula [14:59] hazmat: The service name is not being "reused".. the service is still around [14:59] kim0, are you using the example or principia? [14:59] hazmat: example [15:00] hazmat: It's just another relation [15:00] niemeyer, yes.. its another relation, but the association to the same related service name in the db is still present [15:00] hazmat: Yes, but we shouldn't destroy the database like that no matter what [15:01] i think the principia (based on the original python) doesn't have this issue [15:01] yeah.. it doesn't [15:01] this is a bug in the mysql formula [15:02] in the examples directory [15:02] the original python one would just skip the db creation, but still set the credentials.. the shell script one exits the hook if the db is already present without setting credentials [15:03] iotw. the principia mysql formula should work fine here [15:03] ok I'll destroy everything and start again [15:06] kim0: Thanks for brining this up, we'll have to fix the example somehow [15:06] hazmat: This is another use case for having a name for the actual relation [15:06] hazmat: Or, an identifier [15:06] niemeyer, you mean a generated name? [15:07] hazmat: The identifier we talked about before [15:07] hazmat: relation-N [15:07] right.. to prevent anon relations [15:07] and make them addressable [15:07] hazmat: Yeah.. this would be the right thing to use in this case [15:07] rather than the service name [15:07] then the logic would be right [15:08] "If a database name with the relation name already exists, stop, otherwise create it and set the info" [15:08] But, one thing at a time [15:08] hazmat: What's your plan for the session issue? [15:10] debatable, we've allowed service names to be reusable (as opposed to unit names), so the question is are there scenarios where a service would want to pickup a previously set up database (a/b upgrades perhaps).. fwiw. the logic was right before it was rewritten.. given that we do need relation names to make them addressable in other context that seems like a good choice [15:10] niemeyer, setup a test environment and manipulate the firewall and zk server to reproduce [15:11] might need to send an email to the list as well, i've seen some contradictory information in regards to this [15:11] niemeyer, have a look at http://zookeeper-user.578899.n2.nabble.com/watcher-semantics-for-session-events-in-the-C-client-td6206081.html [15:11] hazmat: The right thing wouldn't be to give the same database for all relations [15:12] hazmat: So the original one wasn't proper either [15:12] the watches aren't freed after the session event by the client [15:12] niemeyer, working vs. proper [15:12] hazmat: ? [15:12] niemeyer, it worked in this exact scenario [15:13] you can add/remove relations all day with the original, and the remote service will get the correct credentials and events [15:13] hazmat: Ok, there are several things we might that that would work [15:13] hazmat: I'm trying to figure how we should actually solve this issue [15:13] niemeyer, which issue session or relation? [15:13] hazmat: Both :) [15:13] well that clarifies ;-) [15:13] hazmat: So.. for relation I'll file a bug [15:13] hazmat: Let's put it aside for now [15:14] niemeyer, sounds good [15:16] hazmat: Interesting indeed [15:16] hazmat: (the link) [15:17] niemeyer, so the ppa will auto update with the new package for txzk since there was a merge to trunk? [15:18] oh.. i need to increment the version i think [15:18] hazmat: It will, but it will be daily [15:18] hazmat: No [15:18] ah.. its got the revno on it [15:18] hazmat: It will use the revision from the repo [15:18] niemeyer, can those builds be triggered by hand? [15:19] yup.. i see it now.. okay requested a rebuild of txzk [15:19] hazmat: When you want to bump up due to a change you've done recently, go to the recipe page, and click on the button [15:19] wow.. the build wait time is much better now [15:19] 4-8m [15:20] is much better than 12-36hrs [15:20] hazmat: Btw, it's good to check if your revision isn't already built [15:20] hazmat: It was, in this case [15:20] txzookeeper - 0.2.1-0ensemble38~oneiric1 [15:21] hazmat: That 38 is the revno, which is the tip atm [15:21] niemeyer, ah good point [15:21] hazmat: Don't use "Request build", in general [15:22] hazmat: When there is a revno unprocessed, there will be a button at the top, right below "Build schedule" [15:22] ic [15:36] hazmat: https://bugs.launchpad.net/ensemble/+bug/791370 [15:36] <_mup_> Bug #791370: We need a relation identifier for hooks < https://launchpad.net/bugs/791370 > [15:36] <_mup_> Bug #791370 was filed: We need a relation identifier for hooks < https://launchpad.net/bugs/791370 > [15:36] We should address this sooner rather than later [15:36] niemeyer, more immediately the formula should probably be fixed as well [15:37] hazmat: It depends a bit on how long we take to get to it [15:37] hazmat: E.g. if Ben is available today, it might be a good brain-break task [15:39] I've assigned the bug to him [15:40] Hah, just in time for bcsaller to be unable to say no! [15:40] oh no [15:40] niemeyer, ;-) here's the diff https://pastebin.canonical.com/48037/ [15:40] what did I walk into? [15:41] bcsaller, a bug in the example mysql formula [15:41] bcsaller: https://launchpad.net/bugs/791370 [15:41] <_mup_> Bug #791370: We need a relation identifier for hooks < https://launchpad.net/bugs/791370 > [15:41] hazmat: How does that solve anything? [15:41] bcsaller, and more generically identifiers for relations [15:42] niemeyer, it allows the formula to work in if add/remove relation is used or the service is destroyed and recreated [15:42] hazmat: How so? [15:42] oh.. i need to remove the exit 0 as well [15:43] niemeyer, it will setup credentials for the service and set them on the relation always this way [15:43] hazmat: Ahhhh, yeah, if you fix it, maybe ;-) [15:44] niemeyer, can i get a +1 on the updated trivial https://pastebin.canonical.com/48038/ [15:44] hazmat: So it'll use a different user per relation? [15:44] niemeyer, yes [15:44] hazmat: Cool [15:45] hmmm.. i wonder what the original did here [15:45] hazmat: +1 if you test it :-) [15:46] this doesn't seem right either [15:46] it shouldn't be resetting the identity every time on join [15:46] it should be checking for the value on the relation [15:46] Yeah, indeed [15:47] niemeyer, oddly though the original (in principia) did this.. and it worked [15:47] for scaling mediawiki [15:47] but i take that as some mysql aberation [15:47] hazmat: Apparently we didn't test non-trivial scenarios, though [15:48] hazmat: Well, or maybe you did with the old one [15:48] i did with the old one.. but i still wonder about this [15:57] <_mup_> Bug #791382 was filed: Session events are not being handled properly < https://launchpad.net/bugs/791382 > [16:01] Lunch time, biab. [16:35] Hmm.. [16:35] so scrolling back a bit.. [16:35] the principia formula right now uses a flag file to know whether or not the relationship has been broken.. [16:36] If its been broken, then it creates a new user [16:36] lp:~ensemble-composers/principia/oneiric/mysql/trunk btw [16:47] hazmat: re the munin server, I had it happen yet again after a full shutdown/bootstrap.. and I tracked it down to heavy network + CPU. When munin was doing its updates and I would add or remove relations, the problem would happen. [16:47] hazmat: so I think your hunch is correct that we probably need to think about longer timeouts for ZK [17:03] <_mup_> ensemble/expose-provision-service-hierarchy r261 committed by jim.baker@canonical.com [17:03] <_mup_> Tests verifying all service units are 'checked' upon destroying a service or unexposing it [17:14] <_mup_> ensemble/expose-provision-service-hierarchy r262 committed by jim.baker@canonical.com [17:14] <_mup_> Doc strings [17:22] <_mup_> ensemble/expose-provision-service-hierarchy r263 committed by jim.baker@canonical.com [17:22] <_mup_> Removed debug logging output [17:38] Yo [17:39] hrm.. augeas lenses have murky copyrights [17:39] * SpamapS is starting to develop copyright policies for formulas [17:40] yuck [17:40] Yeah [17:40] SpamapS: What, seriously? [17:40] SpamapS: What's the copyright like? [17:40] I'm surprised that augeas was allowed into Debian w/ such an inaccurate copyright file [17:41] the copyright file claims that it is Copyright 2007, 2008 Red Hat Inc. [17:41] but that copyright is only in the man pages and tests [17:41] most of it is clearly Copyright 2007-2010 David Lutterkort [17:41] And the lenses have almost no copyrights specified, and there is no blanket copyright specification [17:42] the *license* is very clearly LGPL-2.1 [17:42] but most people seem to get Copyright wrong. [17:43] Does the released column on https://ensemble.ubuntu.com/kanban/dublin.html refer to released in past week ? [17:43] SpamapS, it doesn't need to check a flag file, the relation is different if its broken, just checking for the value should do the trick [17:43] kim0, roughly.. everything since the last milestone, its cumulative. the milestone started roughly right after uds [17:44] hazmat: well my point was simply that the database name should remain the same, as we want to preserve the data most of the time. [17:44] SpamapS, ah.. indeed [17:44] * kim0 rings a little shiny bell [17:45] I actually think even better would simply be to check for the existence of the user.. since I'm also revoking the user perms on broken [17:45] Let's have our irc meeting in 1:15 hours [17:45] Things to talk about [17:45] - Creating the principia distribution and principia-tools project [17:45] - The Ensemble ppa being up [17:45] - Plus any recent development done [17:45] kim0: I'm going to miss it unfortunately.. have to take child to doctor. But next time! [17:45] oh it would have been nice to have you on this one .. [17:45] no problem though [17:46] Yes.. definitely. :( [17:46] just found out about it about 15 min ago [17:46] SpamapS: all the best to our little sick friend [17:46] He's fine.. its just not possible to convince his mother of that without a PhD. ;) [17:47] HAHA [17:47] SpamapS, :-) [17:47] it's amazing women are women everywhere I suppose [17:48] perhaps more specifically.. the nature of a mom to be a mom [17:48] i wish it were true === deryck is now known as deryck[lunch] [17:49] yeah, they're hard wired to care for children more than others can imagine .. [17:51] SpamapS: Ah, that should be fine then [17:51] SpamapS: We care mostly about license details, rather than copyright [17:51] SpamapS: For using Augeas, that is [17:52] SpamapS: +1 on having that cleared up for formulas [17:54] bcsaller: Have an interview now, will push your review to completion after that [17:59] sidnei_! [17:59] yo [17:59] so anyone wrote an ensemble formula for plone yet? :) [18:01] sidnei_, not yet ;-) === sidnei_ is now known as sidnei [18:01] sidnei_, i was thinking it would be a zodb server formula, plone app server formula, varnish formula [18:02] hazmat, yeah, that's what i thought too [18:02] just wondering if i can get off the business of building the installer for windows and provide ensemble formulas instead :) [18:02] That'd be *awesome* [18:02] the varnish is a little wierd, in that its probably passing VCL directly via the service relation [18:03] for more advanced config.. and namespacing across multiple domains and different apps might be a little strange with varnish and service relations.. as its config is effectively code [18:03] but simple stuff should just work ootb [18:04] sidnei, did you see enfold's new plone hosting system? [18:04] hazmat, yup, got early access and all :) [18:05] its insanely fast to setup a new plone with their stuff, i still haven't figure out how they did that [18:05] its way faster than the process start time [18:06] hazmat, i don't think they're setting up a new process, but i could be wrong. possibly shared instance but separate zodb? [18:07] sidnei, that makes sense, although there's some cache thrashing/mem ballooning possible with something like that [18:08] hazmat, indeed. [18:08] could ask alan but he's not online right now [18:14] sidnei, he told me secret sauce last i asked ;-) [18:14] haha [18:42] * hazmat switches into concurrent mode [18:43] <_mup_> ensemble/expose-hook-commands r240 committed by jim.baker@canonical.com [18:43] <_mup_> Hook skeleton for open-port, close-port commands === deryck[lunch] is now known as deryck [18:58] koolhead17: hey [18:58] there ? [18:58] kim0: hello [18:58] hey :) [18:58] koolhead17: can u join the online chat in 2 mins [18:58] and tell us what you've been up to [18:59] yes sure. am very much here. [18:59] cool [19:00] niemeyer: hazmat sidnei jimbaker bcsaller ready for some irc talk [19:00] niemeyer, that problem that SpamapS had seems pretty easy to reproduce.. start client with watch against local zk, restart zk, boom. [19:00] let's hit #ubuntu-cloud [19:00] hazmat: Sweet! [19:00] kim0, rock on [19:00] hazmat: Well, "sweet!" :) [19:00] cool [19:02] COMMUNITY WEEKLY MEETING ROLLING ON #ubuntu-cloud [19:02] ^^^ [19:02] :-)) [19:22] Ok, standup? [19:22] Let's try to do a "real" one.. I have to stop talking and review some code :) [19:23] haha [19:24] niemeyer, sounds good [19:25] hazmat, bcsaller: standup? [19:25] obino: jimbaker hello guys [19:25] niemeyer: mumble [19:25] niemeyer, we're all hanging in mumble [19:46] koolhead17, hi [19:47] <_mup_> Bug #791501 was filed: Ensemble images should use the ppa < https://launchpad.net/bugs/791501 > [20:11] niemeyer, on the call you where saying we shouldn't divert all session events to a separate channel if the session is still alive? [20:11] hazmat: The opposite [20:11] okay.. cool, that's what i thought [20:11] hazmat: We shouldn't divert the event if the session is really dead [20:11] or.. i should say.. yeah.. that makes more sense [20:11] hazmat: Since we have to crash whoever is waiting on the watch [20:12] niemeyer, yeah.. in the case of disconnect we can still divert if the handler is a kill switch [20:12] hazmat: Btw, it's good to double check what I said in the meeting re. the behavior of sessions. That link you posted has someone saying the opposite, but I don't recall this being the case. [20:12] as we want to crash fast in that case, not go through every watcher callback failing [20:12] niemeyer, yeah.. i'm running some tests now [20:13] hazmat: I recall a specific feature announcement saying that "now reestablishment of connections will preserve watches" [20:13] if connecting to a different server? [20:13] hazmat: But this guy's email is from March.. so it's strange [20:13] hazmat: Yeah, given the semantics of zk, switching server is really not a big deal [20:15] hazmat: [20:15] "If the client connects to a different ZooKeeper server, it will send the session id as a part of the connection handshake. [20:15] " [20:15] http://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html [20:15] i wonder how vmware with activestate stealing the thunder of cloudfoundry [20:15] ^feels [20:16] "Another parameter to the ZooKeeper session establishment call is the default watcher. Watchers are notified when any state change occurs in the client. For example if the client loses connectivity to the server the client will be notified, or if the client's session expires, etc..." [20:16] hazmat: The distinction between precisely those two examples is what we were talking about [20:17] niemeyer, from the same page "If you are watching for a znode to come into existance, you will miss the event if the znode is created and deleted while you are disconnected." [20:17] i guess in that particular case its moot [20:17] hazmat: Yeah, sure, but we don't care .. righ [20:17] t [20:18] hmm [20:19] hazmat: re. cloudfoundry, I don't see it as stealing. I think vmware is probably delighted. [20:19] its a separate private cloud foundry installation hawking closed source features [20:22] hazmat: Yeah, but it's still announced as Cloud Foundry.. much better for them than anything else ActiveState pushed. [20:36] niemeyer: I think we are missing a high level architecture doc. I got asked for something and realized we don't really have anything we can point people at that want to eval the overall system design. Unless you know of something I'm not seeing? [20:38] niemeyer, another useful doc http://outerthought.org/blog/435-ot.html [20:43] bcsaller: Indeed [20:43] bcsaller: I started such a doc in the old specifications section in the wiki [20:43] bcsaller: But it's poor and unfinished [20:44] bcsaller: Funny enough, I think the best we'd have today is our glossary [20:44] maybe I'll take a stab at it a little later today [20:44] bcsaller: Sweet! [20:46] bcsaller: https://wiki.canonical.com/Ensemble/Specifications/0010 [20:46] thanks [20:46] bcsaller: Not sure there's anything useful there [20:47] bcsaller: It hasn't been reviewed this century [20:47] understood :) [21:31] <_mup_> ensemble/expose-hook-commands r241 committed by jim.baker@canonical.com [21:31] <_mup_> Completed wiring up hook commands for open-port, close-port === marrusl is now known as marrusl_afk [22:06] looking at some of the callback oriented test code of txzookeeper is like reassembling a puzzle. [22:12] hazmat, indeed, inlineCallbacks do make for a better experience. in talking to guido van rossum at the amazon python day event, he clarified his position on async code - it was the callback oriented code that made his head hurt, not async code per se [23:06] niemeyer: thanks for the review [23:07] bcsaller: No problem, excited to see this close to completion === marrusl_afk is now known as marrusl