[00:00] <niemeyer> hazmat: http://paste.ubuntu.com/668723/
[00:00] <jimbaker> m_3, but my initial feeling is  that the current model has an elegant simplicity to it
[00:01] <niemeyer> Dinner
[00:01] <jimbaker> niemeyer, enjoy what must be a late dinner!
[00:03] <m_3> jimbaker: understand... it might be a design decision to keep the states/events at a certain level
[02:00] <niemeyer> hazmat: ping
[03:42] <hazmat> niemeyer, pong
[03:42] <niemeyer> hazmat: Hey man
[03:42] <niemeyer> hazmat: Just finishing some docs for gozk
[03:42] <hazmat> niemeyer, hola.. long dinner with an out of town friend
[03:42] <niemeyer> hazmat: It's in a pretty good shape I think
[03:42] <niemeyer> hazmat: Nic
[03:42] <niemeyer> e
[03:43] <hazmat> niemeyer, awesome.. 
[03:43]  * hazmat checks out the paste
[03:43] <niemeyer> hazmat: I have a better test, actually..
[03:43]  * niemeyer pastes
[03:44] <niemeyer> hazmat: http://paste.ubuntu.com/668835/
[03:45] <hazmat> niemeyer, interesting
[03:46] <hazmat> looks so you did the diversion of session events off watches, nice
[03:47] <niemeyer> hazmat: Yeah, and there's a non-obvious handy aspect there.. I'm just finishing the docs and will paste it
[03:47] <hazmat> niemeyer, one caveat, perhaps obvious, if the reinit is used to restablish sessions the extant watches against the session are somewhat hosed because the zk c bindings are tracking watches against the handle, and on reinit the handle is hosed though the session restored
[03:47] <hazmat> s/handle is hosed/handle is new
[03:48] <niemeyer> hazmat: Not sure I get how that's an issue
[03:48] <jimbaker> nice simple test
[03:48] <hazmat> niemeyer, its not an issue, just something for a document
[03:48] <hazmat> niemeyer, you where saying there's a handy non-obvious property there?
[03:48] <niemeyer> hazmat: But why does it even have to be in the doc?  It feels like an internal implementation detail
[03:49] <niemeyer> hazmat: Yeah.. events stringify properly through String()
[03:49] <niemeyer> hazmat: and os.Error is actually a interface { String() string }
[03:49] <niemeyer> hazmat: Which means it becomes comfortable to handle problematic situations
[03:50] <niemeyer> hazmat: if !event.Ok { return event }
[03:50] <hazmat> niemeyer, well reconnect/reinit a session, with watches associated to a session, and old extant watches effectively being dead is always obvious.
[03:50] <niemeyer> hazmat: The event itself is the error
[03:50] <hazmat> isn't that is
[03:50] <niemeyer> hazmat: That's exactly what the test I pasted does.. old watches do not die unless the session expires
[03:51] <niemeyer> hazmat: If people get an expired session, the watch will fire
[03:51] <hazmat> niemeyer, in that cases its the same session and same handle
[03:51] <hazmat> if you ReInit with an open watch.. things are different, even though its the same session
[03:52] <niemeyer> hazmat: Sure.. if people obtain a new zk value, it's a new zk value..
[03:52] <hazmat> niemeyer, very cool though.. good to have this 
[03:53] <hazmat> niemeyer, its a c library implementation detail/deficiency, just the association of callbacks to the handle, the watch callbacks should be session associated across reinit boundaries imo
[03:54] <niemeyer> hazmat: I don't feel too strong about this..
[03:54] <niemeyer> hazmat: ReInit is a bizarre detail
[03:55] <niemeyer> hazmat: In practice, zk does the heavy lifting internally
[03:56] <hazmat> niemeyer, this looks pretty good
[03:57] <hazmat> niemeyer, on a separate, unrelated note, i found a gevent / twisted integration, basically a new reactor for twisted that uses gevent and automatically associates twisted protocols to i/o greenlets.. allows intermixing the gevent and twisted style code.. http://wiki.inportb.com/wiki/Projects:Python:Geventreactor
[03:58] <niemeyer> hazmat: Thanks, really glad you like it.. I hope we can sort out some of our issues
[03:58] <niemeyer> hazmat: http://paste.ubuntu.com/668840/
[03:58] <niemeyer> hazmat: This is part of the docs
[03:58] <niemeyer> hazmat: Holy crap
[03:58] <niemeyer> hazmat: This is sick :)
[03:59] <hazmat> super awesome :-)
[04:00] <hazmat> niemeyer, the implementation is pretty clean, i haven't run it against anything yet, i'm curious how well it will handle inlinecallbacks.. but definitely interesting
[04:02] <hazmat> niemeyer, so re docs.. look good.. one point of concern, is that there are numerous session channel events which are effectively transient which mean nothing to the app, it might be worth noting that, there really there to allow apps to respond to connectivity changes for self-throttling.. but on a size limited buffer their effectively no-ops to most apps.
[04:03] <niemeyer> hazmat: This takes two second class citizens in the Python world and put them together.. if you want to create a bomb, this is pretty effective.
[04:03] <hazmat> i guess its captured with the panic if not read from session channel event, but it doesn't really distinguish that most of these events are frivolous to most apps
[04:03] <hazmat> niemeyer, lol
[04:03] <hazmat> niemeyer, if we pool our super powers.. we get cows :-)
[04:03] <niemeyer> hazmat: That milk radioactive acid
[04:04] <hazmat> niemeyer, i guess it does, its just implicit in that non important things aren't delivered to watchers, which implies they are delivered to the session channel
[04:05] <niemeyer> hazmat: Good point re. docs, will add
[04:05] <_mup_> ensemble/expose-cleanup r319 committed by jim.baker@canonical.com
[04:05] <_mup_> Mock completely current tests against shutdown
[04:05] <niemeyer> hazmat: Indeed
[04:06] <hazmat> otoh.. twisted has pluggable reactors.. this is just another pluggable reactor..  a libev one.. the addition of greenlets to it.. is definitely radioactive, but intriguing
[04:07] <niemeyer> hazmat: It's intriguing for sure, but I'm not interested on it in the way you are.. We really can't even think of using something like this for Ensemble
[04:07] <jimbaker> why would inlineCallbacks be impacted by the choice of reactor implementation? other than a slow reactor is going to trampoline slowly?
[04:08] <niemeyer> hazmat: We have to walk towards a platform that enables us to produce very a solid framework
[04:08] <hazmat> jimbaker, it shouldn't be.. it should just work
[04:08] <niemeyer> hazmat: This is going into the exact opposite direction
[04:08] <jimbaker> hazmat, exactly. there's no magic there
[04:09] <niemeyer> hazmat: Made docs more clear
[04:11] <hazmat> niemeyer, cool
[04:12] <hazmat> i'm going to head to bed... have a good night folks
[04:13] <niemeyer> hazmat: Night man.. will just finish this and will head to bed too
[08:11] <TREllis> anyone tried playing with openstack & ensemble yet?
[08:12] <TREllis> just getting a "Connection was refused by other side: 111: Connection refused" upon bootstrapping atm
[12:34] <hazmat> morning
[12:35] <hazmat> TREllis, we found a few issues with it in one of our dependencies, txaws, its definitely a priority to have it working smoothly b4 oneiric
[12:49] <TREllis> hazmat: ah, I saw one Clint put a patch for txaws and applied it manually to check, but I still get problems, perhaps my config is dodgy? lemmie pastebin it
[12:50] <TREllis> hazmat: http://paste.ubuntu.com/669198/ my environments.yaml
[12:51] <TREllis> hazmat: since I replaced the IP with a hostname and the txaws patch I get: http://paste.ubuntu.com/669200/
[12:54] <TREllis> actually, hostname or IP, doesn't make a difference
[12:58] <hazmat> TREllis, yeah.. i think that's about where we got on it
[12:58] <hazmat> TREllis, we're going to need to do some additional debugging 
[13:00] <TREllis> hazmat: ok thanks, added as a comment to the bug
[14:32] <niemeyer> Good morning software lovers
[15:08] <m_3> morning!
[15:09] <heckj> morning west coaster!
[15:24] <kim0> heh any idea how long does our cloudfoundry formula actually take to deploy
[15:27] <niemeyer> kim0: No..?
[15:27] <hazmat> kim0, there's an open bug about it.. there's some env var that the ruby needs that is unknown
[15:28] <hazmat> kim0, it will hang as i recall else.. i thought it was just rabbitmq, but i think that's been resolved
[15:28] <hazmat> i haven't seen any confirmation that its working though
[15:29] <hazmat> but the sympton was it just hanged on install
[15:29] <kim0> botchagalupe is saying it's 25mins min! wanted to know if we beat that
[15:30] <kim0> I know it's mostly packaging, not really an ensemble thing though
[15:30] <kim0> 25mins still sounds too much to me
[15:31] <m_3> kim0: agree, that sounds kind of strange... even if they have to entirely bootstrap a gem env
[15:32] <niemeyer> kim0,m_3: It'd be nice to have more details about what's going on there
[15:32] <niemeyer> It's certainly quite unexpected.. there's no reason for it to take 25 mins
[15:32] <m_3> kim0: you have links to the latest or should we wait for negronjl 
[15:33] <kim0> m_3: haven't looked, but we could dig it up from negronjl's 
[15:33] <kim0> m_3: it seems to have a bug now though
[15:33] <m_3> kim0: ok, I'll schedule some time to branch and play with it this afternoon
[15:33] <kim0> m_3: cool!
[15:34] <kim0> I'll love to show it to the world .. finishing is 5mins or so hopefully :)
[15:34] <m_3> kim0: it's gotten a lot of press lately
[15:34] <kim0> yeah definitely hot
[15:38] <niemeyer> kim0: Exactly.. it'd be nice to expose that
[15:39] <kim0> if the bug is fixed today sometime .. I'll screencast it first thing tomorrow morning 
[15:51] <niemeyer> kim0: That's awesome, thanks a lot
[16:00] <_mup_> ensemble/expose-cleanup r320 committed by jim.baker@canonical.com
[16:00] <_mup_> Mock out sleep
[16:05] <hazmat> fwereade, niemeyer, bcsaller, jimbaker.. g+ meeting?
[16:05] <bcsaller> looking for invite
[16:06] <fwereade> hazmat: invite away
[16:06] <niemeyer> Yep!
[16:07] <jimbaker> meeting sounds good
[16:09] <hazmat> invites out
[16:11] <hazmat> jimbaker, you see invite
[16:11] <hazmat> ?
[16:11] <jimbaker> hazmat, i don't
[16:11] <niemeyer> That's not working..
[16:11] <niemeyer> I'll restart..
[16:11] <jimbaker> i refreshed g+ a few times (which i don't expect to do), nothing yet
[16:20] <_mup_> Bug #828885 was filed: 'relation-broken' hook not firing when relation is set to 'error' state <Ensemble:New> < https://launchpad.net/bugs/828885 >
[16:21] <niemeyer> if !event.Ok { return event }
[16:24] <niemeyer> c.Assert(event, Matches, "ZooKeeper connected; path created: /path")
[17:39] <negronjl> jcastro: ping
[17:40] <jcastro> negronjl: howdy
[17:40] <negronjl> jcastro:  When you get a minute, I'd like to catch up with you re: NoSQL
[17:40] <negronjl> is kim0 also attending ?
[17:40] <jcastro> I have time now
[17:40] <negronjl> G+ ?
[17:40] <jcastro> sure
[17:41] <negronjl> give me a sec
[17:43] <negronjl> jcastro: invite sent
[17:47] <niemeyer> bcsaller, hazmat: I'm available for our call
[17:48] <bcsaller> niemeyer, hazmat: ready when you are
[17:50] <hazmat> niemeyer, bcsaller invitations out
[18:17] <jcastro> lynxman: heya, can you update the ensemble in macports to be a more recent snapshot?
[18:17] <jcastro> we're doing a talk at scale out camp next wednesday and it'd be nice to have something more up to date
[18:59] <_mup_> ensemble/expose-cleanup r321 committed by jim.baker@canonical.com
[18:59] <_mup_> Refactoring of complex mocking
[19:09] <_mup_> ensemble/expose-cleanup r322 committed by jim.baker@canonical.com
[19:09] <_mup_> Finished previous refactoring
[19:35] <_mup_> ensemble/expose-cleanup r323 committed by jim.baker@canonical.com
[19:35] <_mup_> Verify deletion of security group in mocking
[20:44]  * niemeyer breaks for coffee
[20:54] <_mup_> ensemble/expose-cleanup r324 committed by jim.baker@canonical.com
[20:54] <_mup_> Explicit MATCH for mock params, remove debugging
[21:06] <_mup_> ensemble/expose-cleanup r325 committed by jim.baker@canonical.com
[21:06] <_mup_> PEP8 & PyFlakes
[21:18] <_mup_> ensemble/expose-cleanup r326 committed by jim.baker@canonical.com
[21:18] <_mup_> docstrings, comments
[21:20] <_mup_> ensemble/machine-agent-uses-formula-url r314 committed by kapil.thangavelu@canonical.com
[21:20] <_mup_> additional tests for local file urls if the file is not present.
[21:33] <_mup_> ensemble/machine-agent-uses-formula-url r315 committed by kapil.thangavelu@canonical.com
[21:33] <_mup_> replace mocker any with value match functions
[21:38] <fwereade> niemeyer, hazmat: EC2LaunchMachine *theoretically* handles stuff in machine_data like image_release_name, and various other params that get passed through to get_current_ami
[21:39] <fwereade> niemeyer, hazmat: ...but I don't see any mechanism by which they could actually be injected at the moment
[21:39] <fwereade> niemeyer, hazmat: (ok, yes, they get put in machine_data... but nothing *does* put them in machine_data AFAICT, so it all seems to be dead code)
[21:40] <fwereade> niemeyer, hazmat: off the top of your heads, am I missing something?
[21:40] <niemeyer> fwereade: They're probably not being used
[21:40] <niemeyer> fwereade: get_current_ami was built as an experiment, outside of ensemble
[21:41] <fwereade> niemeyer: then I can kill them? :)
[21:41] <fwereade> niemeyer: get_current_ami itself is actually fine, it has (I think) sensible defaults for when you don't specify a default image id in the config
[21:41] <niemeyer> fwereade: If they're working correctly, doesn't feel like a good idea..  we have missing functionality in that area, and this is a utility we can make good use of
[21:43] <fwereade> niemeyer: get_current_ami is used elsewhere, and is useful so far as I know
[21:43] <fwereade> niemeyer: tbh it's not really a hassle to keep them in, it'll just be an unused parameter somewhere
[21:44] <fwereade> niemeyer: but on the other hand it wouldn't be a hassle to add them when we needed it
[21:44] <niemeyer> fwereade: That's fine.. as long as they're not causing problems and not getting on your way to implement something, I suggest keeping things as they are
[21:44] <fwereade> niemeyer: we'd need code changes to actually make use of them now
[21:44] <niemeyer> fwereade: We need to support firing different releases, which touches that area
[21:45] <fwereade> niemeyer: ok, I'll add an unused .image_options or something to the machine_data replacement, and make sure it works for when we get around to using it
[21:46] <niemeyer> fwereade: Ah, I see.. you _are_ in fact touching on that area, and it's getting on your way
[21:46] <niemeyer> fwereade: Sounds fine to remove it then
[21:46] <fwereade> niemeyer: cool :)
[21:47] <fwereade> niemeyer: I promise it won't be any harder to add in the future than it would be right now ;)
[21:49] <hazmat> fwereade, the only injection mechanism atm is via config, we don't have pass through parameters to get machine options from start machine
[21:49] <hazmat> hmm.. or do we
[21:49] <fwereade> hazmat: we do, if we stick the right magic in machine_data
[21:50] <niemeyer> hazmat: Injecting machine details through config is really a hack, though
[21:50] <fwereade> hazmat: but we don't have any mechanism for actually adding them to machine_data, so it's kinda moot
[21:50] <niemeyer> hazmat: Sounds fine to kill it for the moment
[21:51] <hazmat> niemeyer, agreed most of the time folks are asking for it exposed on a per command basis, not an environment default
[21:51] <niemeyer> Right
[21:51] <hazmat> niemeyer, actually things like region are useful for determining ami and are environment settings properly
[21:52] <niemeyer> hazmat: Sure.. that should continue to exist
[21:52] <niemeyer> fwereade: ^
[21:52] <fwereade> niemeyer, hazmat: region remains necessary, and default-image-id remains usable
[21:53] <niemeyer> fwereade: default-image-id is also a hack
[21:53] <fwereade> all we lose is the ability to stick things like image_release_name into machine_data, which we don't do anyway, and should do in a different way when we do come to do it
[21:53] <fwereade> niemeyer: agreed, but it's a hack that people may well be currently using
[21:54] <niemeyer> fwereade: We can support it for now, but it's dying soon
[21:54] <fwereade> niemeyer: sounds good
[21:54] <niemeyer> fwereade: They may be using, but there are features that we're adding very soon that render it unusable
[21:54] <hazmat> niemeyer, to be replaced with?
[21:54] <fwereade> niemeyer: but I think it deserves a targeted death, not just a drive-by
[21:54] <niemeyer> hazmat: To be replaced with release-level selection
[21:55] <niemeyer> hazmat: The way to customize a base image with Ensemble is through formulas, not through custom machine ids
[21:55] <fwereade> niemeyer: I'd been assuming something along those lines
[21:55] <niemeyer> hazmat: People should definitely not be publishingn formulas which require others to tweak their setup to use custom images
[21:55] <fwereade> niemeyer: so, say, mongodb will want a 64-bit arch
[21:56] <hazmat> yeah.. 64-bit image selection ended up being the primary use for default-image-id.. but it should be automatic just based on size
[21:56] <fwereade> niemeyer: and we'll need to have an ensemble format to define these properties anyway, because we'll need to specify them differently on ec2 and on orchestra
[21:56] <hazmat> the only ec2 oddball is tiny with 64bit
[21:57] <fwereade> niemeyer: so the time to thread them through is when we've actually defined what can be specified
[21:58] <niemeyer> fwereade: We'll probably start introducing these bits pretty soon
[21:58] <SpamapS> hrm.. seeing something very strange when trying to bootstrap on canonistack..
[21:58] <niemeyer> fwereade: This is part of the store work
[21:58] <fwereade> niemeyer: sounds good to me
[21:58] <SpamapS> when doing a bootstrap w/ EC2.. it never seems to check for the existence of the s3 bucket..
[21:58] <SpamapS> but with canonistack, it tries to find it, gets a 404, and fails
[21:59] <fwereade> niemeyer: in that case, maybe it would be more sensible to retain the capability, just change how it's done
[21:59] <fwereade> niemeyer: it's just one extra field/parameter after all
[21:59] <niemeyer> SpamapS: I have vague memories of S3 failing in non-obvious ways in some case, with 403.. there's a chance they're behaving differently there
[22:00] <hazmat> SpamapS, i suspect its the error parsing against the file not found
[22:00] <niemeyer> fwereade: Sounds good as well
[22:00] <hazmat> SpamapS, like we expect to find a certain value in the error denoting a file not found
[22:00] <SpamapS> niemeyer: whats confusing is that no such request is even made to ec2
[22:00] <hazmat> and having looked at s3store.py i doubt its compatible on error message content
[22:01] <SpamapS> actually I can't say that for sure..
[22:01] <SpamapS> I just realized the S3 URL is HTTP, not HTTPS
[22:01] <niemeyer> SpamapS: That'd be a serious bug.. it should certainly check
[22:01] <niemeyer> SpamapS: I mean, EC2 not checking the bucket
[22:02] <niemeyer> Ensemble not checking the bucket on EC2, that is
[22:02] <hazmat> SpamapS, specifically line 73 of providers/ec2/files.py i'd stick a pdb in there to verify
[22:03] <hazmat> we check the bucket access on s3  prior to starting any machines on ec2
[22:03] <hazmat> its the error that comes back if the file doesn't exist that seems to be getting not trapped corectly into something ensemble recognizes i suspect
[22:04] <SpamapS> ahh so its looking for 'NoSuchBucket' when a 404 should be sufficient
[22:07] <niemeyer> SpamapS: Man..
[22:07] <niemeyer> SpamapS: This is starting to feel like a can of worms
[22:08] <niemeyer> It's unfortunate that these interactions are entirely different :-(
[22:09] <SpamapS> well there is no published standard for AWS
[22:09] <SpamapS> so it takes somebody reporting that something is different for an implementation to get fixed
[22:09] <SpamapS> its worth noting that nova-objectstore is *not* recommended except for testing/single node setups.
[22:10] <hazmat> SpamapS, i doubt swift does much better  here
[22:10] <niemeyer> SpamapS: When one mimics an implementation, there's only one way to make it look the same..
[22:10] <niemeyer> SpamapS: http://paste.ubuntu.com/669591/
[22:10] <niemeyer> SpamapS: I'd expect people to do that kind of experimentation when implementing it
[22:11] <hazmat> SpamapS, looking at the swift s3 it definitely doesn't..
[22:11] <hazmat> SpamapS, both implementations do however maintain the same http error codes as s3
[22:11] <hazmat> just not the same error content responses
[22:11] <SpamapS> which makes sense
[22:12] <SpamapS> honestly, code > content ;)
[22:12] <hazmat> SpamapS, yeah.. i think switching this to just verifying error codes should be sufficient
[22:13] <SpamapS> it may be worth reporting to swift and nova-objectstore that they should conform to AWS
[22:13] <SpamapS> but we probably should work w/ them before they fix that
[22:15] <SpamapS> big question, why am I not getting a real traceback?
[22:15] <SpamapS> hazmat: that failure is not coming from line 73
[22:15] <niemeyer> SpamapS: Agreed, we should try to make it work regardless
[22:15] <SpamapS> or anywhere in providers.ec2.files :-P
[22:16] <niemeyer> SpamapS: I'm just bitching
[22:16] <SpamapS> niemeyer: yeah, not much we can do unless we want to try and champion AWS compatibility as a standard ;)
[22:16] <SpamapS> anyway, have to run.. ttyl guys
[22:16] <hazmat> SpamapS, re tracebacks.. twisted tends to make that hard.. what traceback do you get?
[22:16] <niemeyer> SpamapS: Cheers
[22:17] <hazmat> SpamapS, ttyl
[22:32] <lynxman> jcastro: sure, it always take a bit to update though
[22:33] <lynxman> jcastro: as a matter of fact, the port I submitted for Ensemble is still not in the public repo :(
[22:33] <lynxman> jcastro: http://www.macports.org/ports.php?by=name&substr=ensemble
[22:43] <niemeyer> hazmat: I'm attempting to force a session expiration without much success..
[22:43] <niemeyer> hazmat: Have you succeeded in testing this on txzk?>
[22:44] <hazmat> niemeyer, hmm.. expiration specifically..
[22:44] <hazmat> niemeyer, i could do closes easily, by connecting another client with the same session, and closing it one, so it closes the other
[22:45]  * hazmat checks for expiration
[22:45] <hazmat> niemeyer, yeah.. that generates an expiration
[22:45] <jcastro> lynxman: oh man that sucks. :-/
[22:46] <hazmat> niemeyer, http://bazaar.launchpad.net/~ensemble/txzookeeper/trunk/view/head:/txzookeeper/tests/test_session.py#L151
[22:46] <niemeyer> hazmat: Yeah, I recall you mentioned the trick
[22:46] <niemeyer> hazmat: I was trying to check a real scenario
[22:47] <lynxman> jcastro: yeah, they're up their elbows on work with the release of Lin :/
[22:47] <niemeyer> hazmat: Reduced the timeout, killed the server
[22:47] <lynxman> jcastro: Lion I mean :)
[22:47] <niemeyer> hazmat: Restarted the server.. session seems reestablished fine
[22:47] <niemeyer> hazmat: It's working too well.. ;)
[22:47] <jcastro> lynxman: ok I'll say "in progress" or something
[22:47] <hazmat> niemeyer, yeah.. so timeout specified on init is not the actual session timeout
[22:47] <hazmat> niemeyer, the timeout is negotiated between client and server
[22:47] <niemeyer> hazmat: Oh, crap.. you're right.. it's even called recvTimeout
[22:48] <niemeyer> Oh well.. I'll see if the trick works
[22:48] <lynxman> jcastro: sounds cool :)
[23:17] <_mup_> ensemble/expose-cleanup r327 committed by jim.baker@canonical.com
[23:17] <_mup_> Add mocking around shutdown taking too long
[23:23] <hazmat> niemeyer, can you confirm that i didn't miss anything from our security discussion in austin.. https://pastebin.canonical.com/51455/
[23:33] <adam_g> anyone bootstrapped an oneiric AMI lately? or at least since monday?
[23:33] <adam_g>   * softwareproperties/ppa.py:
[23:33] <adam_g>     - show PPA description and confirm before adding
[23:33] <adam_g>       (security-o-catch-all spec)
[23:53]  * hazmat tries it out
[23:59] <hazmat> hmm