#ubuntu-ensemble 2011-04-18
<niemeyer> Good morning!
<niemeyer> bcsaller: ping
<SpamapS> goooood morning ensemblers!
 * SpamapS finally has some time to update principia to trunk
<jimbaker> SpamapS, good morning. good to hear about your time for prinicipia!
<SpamapS> hmm.. is trunk broken right now?
<SpamapS> 2011-04-18 15:53:19,044 Machine:3: ensemble.agents.machine ERROR: Error starting unit: wiki-cache/0
<SpamapS>   File "/usr/lib/ensemble/ensemble/ensemble/state/service.py", line 487, in get_formula_id
<SpamapS>     returnValue(data["formula"])
<SpamapS> TypeError: 'NoneType' object is unsubscriptable
<SpamapS> http://paste.ubuntu.com/595562/
<SpamapS> full traceback
<SpamapS> hmmmmmmm
<SpamapS> since ensemble doesn't use a .deb to install itself, it really shouldn't be putting its agent in /usr/lib
<niemeyer> SpamapS: Glad to hear you're back!
 * SpamapS smacks forehead.. PYTHONPATH not PYTHON_PATH
<SpamapS> please ignore backtrace
<niemeyer> SpamapS: Aha, ok.. i was just pondering about what could be wrong there
<niemeyer> SpamapS: I've done something worse before, setting PYTHONPATH to the wrong branch
<niemeyer> SpamapS: Too me ages to figure why it was behaving like crazy
<niemeyer> Took
<SpamapS> niemeyer: I've done this like 3 times with 3 different python projects.. not sure why it always seems to happen
<SpamapS> is it because they don't add '.' to the path automatically where others do?
<niemeyer> SpamapS: Yeah, some also put the binary into "."
<niemeyer> SpamapS: We should do the magic within the tools
<_mup_> ensemble/status-workflow-client r209 committed by jim.baker@canonical.com
<_mup_> More fixed tests for test_status
<bcsaller> gustavo: sorry, missed that
<niemeyer> bcsaller: No worries
<SpamapS> still no --force on deploy?
<SpamapS> so is there no way to get all of the settings for a specific unit anymroe? I see that settings_name is before remote unit name in the positional arguments of relation-get
<SpamapS> (another backward-incompatible change btw)
<SpamapS> one that bcsaller warned me about like 3 weeks ago but I forgot :P
<bcsaller> SpamapS: the order of args in relation-get changed to match the spec
<niemeyer> SpamapS: Yeah, it still works pretty much the same way, just the implementation reflects the spec now
<SpamapS> You guys are such devs.
<SpamapS> ;)
<_mup_> ensemble/yaml-state-node r189 committed by bcsaller@gmail.com
<_mup_> YAMLState(dict) -> YAMLState(UserDict) as requested
<SpamapS> Like.. "what you didn't read all 40 pages of spec before using our tool?" ;-) I am not making fun of you guys, rather just enjoying the irony that what makes ensemble so high quality may not make it easy to use for impatient people like me. ;)
<SpamapS> so is there a special meaning setting name, like '*' to get them all?
<bcsaller> yeah, I think the default was '-'
<SpamapS> I need to read up on the debug hook stuff so I can really get good at formula dev :)
<_mup_> ensemble/status-workflow-client r210 committed by jim.baker@canonical.com
<_mup_> Fixed remaining tests
<SpamapS> this broke almost every formula. Glad I didn't read the spec or I'd have never gotten to play w/ ensemble. ;-)
<bcsaller> its a small change at the cli level 
<SpamapS> I'm not complaining. Don't get me wrong. Its a MASSIVE change across every formula I've written thus far
<SpamapS> seeing as they all silently did the wrong thing
<SpamapS> only one broke because I tried to subscript an empty string
<SpamapS> err, a None rather
<jimbaker> SpamapS, it does make the usage simpler for the common case, but i agree it was a change. it puzzled me the first time when i actually read the docs and was trying to write formulas. and when it was changed :)
<SpamapS> the others just said "no settings, moving on"
<SpamapS> because there was no setting named "wiki-db/0" for the non existant unit "hostname"
<jimbaker> SpamapS, and this despite annotating each usage in my bashified version with an XXX about the bug and its expected change, simply because of the silent quality of how relation-get works
<SpamapS> Is there any reason relation-get should silently fail if a non-existant unit is passed?
<niemeyer> bcsaller: Please actually check the UserDict module and ponder a bit about how to properly to a dict-like class
<niemeyer> s/to a/do a
<bcsaller> gustavo: it passed the cases I needed so far in testing and usage, but I'll look at rounding it out
<niemeyer> bcsaller: It's buggy, as I said in the review
<bcsaller> the version I just reposed with UserDict?
<_mup_> ensemble/status-workflow-client r211 committed by jim.baker@canonical.com
<_mup_> PEP8
<niemeyer> bcsaller: The tests pass because they are incomplete as far as a dictionary interface is concerned
<niemeyer> bcsaller: You simply did s/dict/UserDict
<niemeyer> bcsaller: I believe that, given the current interface of your type, using DictMixin would be a better choice
<niemeyer> bcsaller: Then, you might likely be able to get rid of update, for instance
<niemeyer> bcsaller: It's not hard, but a little bit of attention is needed
<bcsaller> happy to look at it again 
<niemeyer> bcsaller: The current implementation breaks if a user does foo.pop(...), for instance
<bcsaller> yeah, I realized that was possible, but I didn't realize we needed a functional dict proxy at that level. I'll try to make it pass the normal interface a little better
<niemeyer> bcsaller: We certainly don't want a type which looks like a dict and then loses operations randomly
<niemeyer> bcsaller: Then, again, it's trivial to do it right with the UserDict module.. just need to look at it and see what the interface is asking for.
<bcsaller> will do
<niemeyer> bcsaller: Thanks
<_mup_> ensemble/status-workflow-client r212 committed by jim.baker@canonical.com
<_mup_> Removed unused import
<niemeyer> bcsaller: Check DictMixin there.. I believe it matches better what you're doing already
<niemeyer> (better than UserDict.UserDict)
<bcsaller> looking at that now
<niemeyer> hazmat: FWIW, still on your branch since the morning
<niemeyer> hazmat: Looks very good overall, but there's a lot there
<_mup_> ensemble/yaml-state-node r190 committed by bcsaller@gmail.com
<_mup_> make use of dictmixin and verify more basic dict operations
<niemeyer> hazmat: That's delivered now
<niemeyer> bcsaller: Is it ready for another look already, or should I move it to WIP?
<bcsaller> I moved it back to needs review, should be ready
<niemeyer> bcsaller: Cool, checking
<bcsaller> added an additional test of some of the methods we talked about 
<niemeyer> bcsaller: Nice, thanks
<_mup_> ensemble/config-state-manager r194 committed by bcsaller@gmail.com
<_mup_> cover the delete case here (though this is captured in a prereq branch now)
<_mup_> Bug #764938 was filed: Convert example formulas to using augeas <Ensemble:New for jimbaker> < https://launchpad.net/bugs/764938 >
<_mup_> ensemble/yaml-state-node r191 committed by bcsaller@gmail.com
<_mup_> remove update method
<jimbaker> bcsaller, hazmat, niemeyer - standup?
<niemeyer> jimbaker: Yep, let's do it
<bcsaller> ok, on skype now
<niemeyer> hazmat seems to be out-of-action today
<niemeyer> jimbaker can you start the call
<niemeyer> ?
<jimbaker> niemeyer, i will do that
<niemeyer> bcsaller: if foo in dict and gotval == None: del dict[foo]
<hazmat> niemeyer, i'm done out
<hazmat> going to take a sick day today
<niemeyer> hazmat: Aw, sorry to hear that
<bcsaller> hazmat: feel better
<jimbaker> hazmat, take care of yourself and stay hydrated!
<_mup_> ensemble/bashified-wordpress-mysql-examples r186 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<SpamapS> Ok I have an interesting question..
<SpamapS> haproxy has a 'forwardedfor' option ..
<SpamapS> which you definitely want to use if haproxy is the connection point for end users..
<SpamapS> but you definitely don't want to use if it is just a link in a chain of reverse proxies..
<SpamapS> So I need to know if I am related upstream to any other proxy..
<SpamapS> I suppose I can just touch a file .. "/etc/haproxy/upstream.is.proxy" ...
<SpamapS> but I'm more wondering if there is a way to just look at all of my relationships and pull it right out of 'relation-get'
<SpamapS> in thinkign more about it, what I really want to do is identify the relationship between two services that I've been related to..
<hazmat> to find if your endpoint or a middlepoint
<SpamapS> well and I guess more importantly, to route the appropriate upstream to the appropriate downstream...
<SpamapS> this isn't just about load balancers.. there will be plenty of times where middleware is done better with a single service unit
<SpamapS> I think the way to do it is just with settings.
<SpamapS> have the upstream go "I am for X"
 * SpamapS is once again brain dumping into IRC rather than actually doing useful things. ;)
<niemeyer> bcsaller: Review properly delivered now
<bcsaller> gustavo: great, thanks
<niemeyer> SpamapS: If the haproxy is in the middle, it will mean there's a relation established upstream
<niemeyer> SpamapS: Should be easy to just set forwardedfor when that's the case
<niemeyer> bcsaller: np
<niemeyer> """
<niemeyer> This will yield two different states depending on the state of
<niemeyer> things, even though: self[name] is None, or self[name] is non-existent.
<niemeyer> """
<niemeyer> bcsaller: I meant to say "even though it's a deleted entry in both"
<niemeyer> bcsaller: But there was a brain hiccup there
<bcsaller> got it
<SpamapS> niemeyer: its my understanding that you would not want to set forwardedfor if haproxy is a middle-proxy .. only if it is the one that users are actually connecting to
<SpamapS> niemeyer: otherwise you will set the header to the upstream reverse proxy's IP
<niemeyer> SpamapS: Right, so that's what I mean.. should be easy to tell if that's the case or not
<SpamapS> niemeyer: right, so this gives rise to the second part of my babbling above.. which is that there will be times where the proxy is providing services to two unrelated upstream proxies and two unrelated downstream proxies..
<niemeyer> SpamapS: and what's the issue there?
<SpamapS> at the moment the only way to do this is just to have two haproxies
<niemeyer> SpamapS: Why?
<SpamapS> niemeyer: the issue is how to specify routing data. I think I have it solved actually.. the downstreams need to say a bit more about what they're doing, and the upstreams a bit more about what they want.
<SpamapS> niemeyer: because /foo needs to go to   foo-app service, and /bar needs to go to bar-app service
<niemeyer> SpamapS: Ah, yeah
<SpamapS> and /anything-else to foo-app service
<_mup_> ensemble/bashified-wordpress-mysql-examples r187 committed by jim.baker@canonical.com
<_mup_> Removed snarky descriptions from metadata.yaml and replaced it with descriptions of what the example formulas do and how they interact
<SpamapS> and haproxy is *really* good at routing lots of different stuff to different places, so it would be a shame if we can't figure this one out elegantly
<SpamapS> But, I think we can.. just need the "website" relations to specify what URL they prefer to be accessed on.
<niemeyer> SpamapS: Indeed
<niemeyer> SpamapS: We'll need to put some thinking into it indeed
<niemeyer> I'll be right back
<niemeyer> just relocating
<SpamapS> Yeah I think ensemble already handles the case well its just a matter of finding a simple way to make it happen. :)
<niemeyer> bcsaller: Is config-state-manager depending upon YAMLState?
<niemeyer> bcsaller: Just wondering how to properly review it
<bcsaller> gustavo: yes
<_mup_> ensemble/bashified-wordpress-mysql-examples r188 committed by jim.baker@canonical.com
<_mup_> Changed db-relation-changed to db-relation-joined for mysql example formula
<bcsaller> it will change a bit to reflect the last review, we can put it back into WIP
<niemeyer> bcsaller: Ok, can you please put it back in review when that's done?
<bcsaller> the last review means I'll have to revise quite a few tests (if only a little) so its better to wait for those changes to YAMLState
<bcsaller> gustavo: will do
<bcsaller> alright, going to head to lunch now
<niemeyer> bcsaller: Enjoy
<jimbaker> niemeyer, do you want to review my updated summaries/descriptions for the example formulas, or should i merge it in?
<jimbaker> this removes the snarky text in favor of what the formulas actually do
<jimbaker> i was noting this as i was bumping the version numbers in the metadata.yaml files
<niemeyer> jimbaker: No, fine to merge them, thanks
<jimbaker> niemeyer, sounds good
<_mup_> ensemble/trunk r197 committed by jim.baker@canonical.com
<_mup_> merge bashified-wordpress-mysql-examples [r=niemeyer][f=726561]
<_mup_> Converted mysql and wordpress example formulas to use shell scripting,
<_mup_> so as to demonstrate the language neutrality of Ensemble.
<_mup_> ensemble/status-workflow-client r213 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/trunk r198 committed by jim.baker@canonical.com
<_mup_> merge status-workflow-client [r=niemeyer][f=737949]
<_mup_> Adds to the status output the current relation and unit workflow states.
<_mup_> Uses the new StateWorkflowClient support.
#ubuntu-ensemble 2011-04-19
<_mup_> ensemble/new-hook-semantics-5-docs r201 committed by jim.baker@canonical.com
<_mup_> Addressed review points on formula doc
<_mup_> ensemble/new-hook-semantics-5-docs r202 committed by jim.baker@canonical.com
<_mup_> Better intro for formula section
<_mup_> ensemble/yaml-state-node r192 committed by bcsaller@gmail.com
<_mup_> read/write version of YAMLState
<_mup_> ensemble/yaml-state-node r193 committed by bcsaller@gmail.com
<_mup_> test multiple writes with and w/o reads
<_mup_> ensemble/config-state-manager r196 committed by bcsaller@gmail.com
<_mup_> new YAMLState api changes
<_mup_> ensemble/trunk r199 committed by kapil.thangavelu@canonical.com
<_mup_> merge state-machine-enhancements [r=niemeyer][f=755062]
<_mup_> Adds some additional capabilities to the statemachine used for workflows.
<_mup_> Transition aliases, which allowing grouping and addressing of common
<_mup_> transitions from different states. It also introduces success transition
<_mup_> ids that a transition can use to effect an additional transition upon
<_mup_> success.
<niemeyer> hazmat: Morning!
<TeTeT> kim0: hi, my query for the high level overview is primarily motivated by the questions and mis-conceptions I hear from the UEC students, especially when it comes to virtualization vs cloud computing
<niemeyer> hazmat: Feeling better?
<hazmat> niemeyer, a bit, more medications to start my day.
<niemeyer> Hey, morning TeTeT, kim0
<kim0> TeTeT: Yeah .. those believing "put your app on the cloud, and it'll scale like google" :)
<kim0> niemeyer: hazmat TeTeT morning :) o/
<TeTeT> kim0: think we should address this early on to make sure the scenario for use of ensemble is set right, e.g. expectation management from the start
<TeTeT> hi niemeyer 
<TeTeT> morning hazmat 
<niemeyer> hazmat: Ouch.. Did you have that kind of issue often when you were a non-veg?
<TeTeT> kim0: that, and 'oh, this isn't like vmware at all'
<kim0> TeTeT: I agree, Would be awesome if you can start contributing to that too :)
<kim0> it seems you hear lots of feedback
<kim0> aw I can understand someone's disappointment comparing uec to vsphere
<TeTeT> kim0: sure, I'll prepare something. I want to present UEC / ensemble / landscape to our key customer in Germany any time anyway
<hazmat> niemeyer, indeed i did, bad allergies some seasons. mostly a factor of location afaict
<kim0> Awesome
<niemeyer> hazmat: Aw :(
<TeTeT> kim0: so for that introduction I'd better come up with something written anyway, which we can then recycle for the docs, I hope
<kim0> Absolutely .. sounds good
<niemeyer> hazmat: I'll send good energy waves towards you then ;-)
<hazmat> niemeyer, thanks :-)
<kim0> oh can you do that 
 * kim0 points the antenna
<niemeyer> kim0: ;-)
<kim0> I just played with debug-log .. and love it .. except I want to split out text into columns per service unit
<kim0> anyone has a magic one liner
<niemeyer> kim0: I can certainly try, at least
<kim0> if it'll take you more than a couple of mins, I might want to bang my head a bit at it 
<kim0> niemeyer: ideally .. the line order would be preserved by printing empty lines on columns of service units not executing anything
<kim0> i.e. looking at the resulting text table .. you'd easily see the order of events triggering
<niemeyer> kim0: No trivial one-liners that I can think of
 * kim0 wonders if he spotted a missing unix kung foo tool
<kim0> okie .. I'll take a shot then
<kim0> probably needs some custom python
<hazmat> kim0, there's also several filtering/inclusion options if you want to isolate a particular
<hazmat> kim0, do you have an example of what sort of columnization you'd want to see?
<kim0> hmm yeah .. those would be great too .. but I'm really more into looking at the big picture now .. just need a prettier big picture to look at :)
<kim0> Yeah sure
<kim0> cloumn per service uit
<kim0> unit* like
<kim0> mysql/0  | wordpress/0
<kim0> joined | --
<kim0> -- | starting db-relation-changed
<kim0> something like that I guess would be great
<kim0> of course should be tab aligned
<kim0> any way .. I'll try it myself a bit 
<kim0> guess distributed logs weren't that popular 20 years ago :)
<hazmat> yeah.. i think thats possible, but possibly we'll end up going very wide. the stdout of the hooks might also interrupt the visual flow
<kim0> hazmat: yeah, which is another thing
<kim0> currently we have no way to log at different levels from inside hooks right
<hazmat> kim0, there is an ensemble-log command available to hooks, but i don't think we currently utilize it, we just capture stdout/stderr of hooks.. at a set log level
<kim0> ah .. that'd be great 
<kim0> to use :)
<kim0> wonder if I should add that to the examples/ formulas
<hazmat> kim0, sounds good, probably we should also verify its working..
<kim0> hazmat: if ensemble-log is used today, can we filter to only see its output and not stdout/stderr ?
<hazmat> kim0, also i've noticed that install, start hooks aren't commonly reported, this is as per the async watches and creator thread on this list.. after we fix that up, we should be able to use logs to show people hook sequencing/interaction for a tutorial.
 * hazmat checks on ensemble-log
<kim0> hazmat: oh that's what I was doing (log sequencing for a tutorial) :) so it's not working now .. I see
<hazmat> kim0, minus the install/start its correct
<hazmat> looking at the log implementation, it looks like its broken to me
<kim0> yummy
<hazmat> hmm. maybe not.. the test looks correct
<kim0> good then
<kim0> can someone exaplin the workflow for editing (or creating new) formula ..
<kim0> I suppose I deploy, and it breaks horribly
<kim0> how do I fix, and try again
<kim0> I hope I dont have to tear down everything for ever changed line
<kim0> every*
<hazmat> kim0, looks good actually re ensemble-log
<kim0> hazmat: great :) thanks
<kim0> I will start a branch with nice log messages
<hazmat> kim0, we'll have two commands that can assist with this, ensemble formula-upgrade to replace a formula that's in place, and ensemble resolved to correct for any hook errors.
<hazmat> kim0, also ensemble debug-hooks for interactive debugging.
<kim0> yeah I guess debug-hooks helps with what I need .. will play with that too
<hazmat> kim0, there's one missing piece to debug-hook that's still in review... i've kinda let it slide while i was working on upgrades.
<hazmat> kim0, re ensemble resolved.. on a hook error, there's a change to an error workflow state within the unit agent (either at a unit or relation level, depending on the hook that fails).
<hazmat> the ensemble resolved command will transition back to a known good state, optionally refiring the hook
<kim0> hmm that sounds magical :)
<hazmat> formula upgrades are only allowed from a unit that's running (ie. not in an error state).
<kim0> we'll talk more about it in the meeting I guess
<_mup_> ensemble/unit-agent-formula-upgrade r233 committed by kapil.thangavelu@canonical.com
<_mup_> upgrade unit agent tests, remove duplicate test, add some state assertions, clean up doc strings.
<niemeyer> jimbaker: ping
<_mup_> ensemble/trunk-merge r185 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<jimbaker> niemeyer, hi
<niemeyer> jimbaker: nm, already providing a new review
<jimbaker> niemeyer, sounds good, hopefully the reworked intro looks good
<niemeyer> jimbaker: Actually, can you please roll back that big introduction you added?
<niemeyer> jimbaker: It doesn't :(
<niemeyer> jimbaker: There are several issues in there
<jimbaker> niemeyer, introductions are always tough
<niemeyer> jimbaker: and I'm afraid we're getting into a review loop
<jimbaker> so it makes sense that it's problematic
<niemeyer> jimbaker: Can you please roll that back and address just the specific issues mentioned in the review?
<niemeyer> jimbaker: So we can merge
<niemeyer> jimbaker: Feel free to provide another change suggestion later on
<jimbaker> everytime i look at the overall intro for ensemble, i think how much it needs to be reworked too
<niemeyer> jimbaker: Some examples:
<niemeyer> jimbaker: We don't offer total ordering guarantees to the user
<niemeyer> jimbaker: It's not true that machines and service units have a 1-1 relationship
<niemeyer> jimbaker: None of the text in the first bullet point is about scalability
<jimbaker> niemeyer, this makes sense
<niemeyer> jimbaker: It's not true that every event in Ensemble is translated into a hook execution
<niemeyer> jimbaker: Etc
<niemeyer> jimbaker: I'd really like to merge the main changes which this branch was trying to implement at this point, which is fixing the hook documentation
<niemeyer> jimbaker: Rather than stepping through that unrelated review which was introduced post-reviewing
<jimbaker> it sounds like i still wasn't careful enough in the descriptions, despite my attempt to do so
<jimbaker> so definitely needs another branch to make even more precise
<jimbaker> or precise at all
<niemeyer> jimbaker: Cool, feel free to push those into another branch
<niemeyer> jimbaker: It feels like a good direction, but feels bad to have that in mid-review
<jimbaker> ok, i will roll that back. should i simply remove the intro as a whole, just have it start at "this section describes how formulas..."
<jimbaker> ?
<niemeyer> jimbaker: No, put the version you had previously, and apply the review
<niemeyer> jimbaker: There were very simple changes to do only
<jimbaker> niemeyer, ok, i guess i mixed up what you wanted. so you just wanted the one sentence removed, that's it?
<niemeyer> jimbaker: Yes
<_mup_> ensemble/unit-agent-formula-upgrade r235 committed by kapil.thangavelu@canonical.com
<_mup_> doc work, from review comments
<hazmat> niemeyer, on the formula-upgrade point 9, its seems the right solution then is to bypass the machine formula and just have the unit agent download directly to temp space and extract? there's a few other places where the unit agent pokes outside of its directory (for workflow state state files), and perhaps with on disk hook execution queue in the future.
<niemeyer> hazmat: Workflow state should really be within its directory too
<hazmat> niemeyer, i've been trying to segment the unit agent state files from the unit itself, ie outside of the unit. so we're saying we want to have all of these files within the unit?
<niemeyer> hazmat: Hmm.. will the unit agent run within the unit?
<niemeyer> hazmat: Once we have containers
<hazmat> niemeyer, its outside the unit
<_mup_> ensemble/new-hook-semantics-5-docs r203 committed by jim.baker@canonical.com
<_mup_> Removed intro for incorporation into a future branch
<hazmat> niemeyer, i was planning on using lxc-attach to execute hooks within the container
<hazmat> it gave us additional separation from the container, and so i did the same with agent state files.
<niemeyer> hazmat: It's not yet clear if that's a good idea or not
<niemeyer> hazmat: In either case, units shouldn't be poking in the machine agent directory at least
<hazmat> niemeyer, in that case we either have the machine agent download the formula or bypass the formula cache on the machine
<niemeyer> hazmat: Yeah, I think it should bypass
<niemeyer> hazmat: We have no idea about what the machine agent is doing.. it might decide to clear that directory and remove the formula under our feet
<hazmat> niemeyer, it does make the formula-cache on the machine stale.. but if we can live with that..
<niemeyer> hazmat: It's as stale as the machine agent decides it to be
<hazmat> true
<hazmat> niemeyer, sounds good
<niemeyer> hazmat: The point is that this directory is owned by a different process
<niemeyer> hazmat: Also, what happens if two different units decide to download the same formula at the same time?
<niemeyer> I'll head to lunch.. Ale is waiting
<niemeyer> biab
<jimbaker> niemeyer_lunch, i have pushed up the change. this removes the new intro and replaces it with the old intro, cutting the first two sentences (didn't seem to make sense to discuss service units at that point)
<_mup_> ensemble/unit-agent-formula-upgrade r236 committed by kapil.thangavelu@canonical.com
<_mup_> download formula upgrade to temp directory.
 * hazmat lunches
<kim0> when using ensemble-log inside a formula .. do I log at DEBUG level ?
<kim0> ensemble-log --log-level DEBUG "foo bar" ?
<kim0> I hope to be able to get those messages with debug-log *without* getting stdout/stderr from hooks
<kim0> niemeyer: Trying to add proper ensemble-log messages in the example formula .. any comment on ^
<niemeyer> kim0: Hmm.. whether it's DEBUG or not depends on the nature of the message
<kim0> niemeyer: if it's DEBUG, can I still separate it from stdout though
<niemeyer> kim0: Yes, IIRC ensemble-log doesn't print to stdout
<kim0> great .. so how do I get debug-log to filter out stdout/stderr
<niemeyer> kim0: Hmmm.. I see what you want
<niemeyer> hazmat: Do you recall which log-level stdout/err messages get logged as?
<kim0> isn't DEBUG the lowest level .. i.e. I hope there's some other way to filter other than by log level?
<niemeyer> kim0: Yes, it's the lowest/highest level (depending how one looks at it)
<niemeyer> kim0: I don't think I really understand what is the problem you're facing, though
<kim0> niemeyer: well I wanted to use ensemble-log from inside formulas, gather those log messages using debug-log, but do not wish them intermingled with stdout/stderr. If you tell me the way to do this, is to log at a loglevel != DEBUG, that's fine I guess .. was just wondering how to do it
<niemeyer> kim0: Right, that's what you want to do it.. but what's the background on why you want to do it?
<niemeyer> kim0: There's no reason for a hook to expose stdout/stderr besides for debugging/understanding purposes
<_mup_> Bug #766241 was filed: Add dummy provider support for port authorization/revocation <Ensemble:New> < https://launchpad.net/bugs/766241 >
<hazmat> niemeyer, debug
<kim0> yeah .. it's just that "set -eux" is too noisy!
<hazmat> kim0, you can also explicit exclude just the hook output 
<niemeyer> kim0: Hmmm.. we shouldn't be using x I think
<kim0> hazmat: hmm how would I do htat
<kim0> niemeyer: yeah I'd think so 
<hazmat> kim0, ensemble debug-log -x hook-output
<kim0> hazmat: does that include stderr ?
<hazmat> kim0, yes that's all hook output
<kim0> hazmat: oh thanks
<hazmat> kim0, checkout out some of the other options on debug-log --help
<kim0> hazmat: help message doesn't really make it clear what "log channels" I can include/exclude
 * hazmat nods
<hazmat> it needs some more examples
<hazmat> in the help message
<hazmat> and perhaps at least a partial listing of log channels
<jimbaker> kim0, set -x is definitely expedient, and it really makes it possible to see what's going on. but if i could just observe the relation settings change in the debug log, that would probably eliminate a lot of debugging statements
<kim0> jimbaker: yeah, I'm just using ensemble-log at INFO level for now
<jimbaker> hazmat, i think we should also add the relation settings info to ensemble status too, now that we have states there. we can always provide an options flag for what to show
<kim0> jimbaker: are you saying, relation setting changes don't go to logs now 
<kim0> ah yeah
<jimbaker> kim0, i don't believe they are otherwise observable. could be wrong :)
<kim0> nah :)
<hazmat> jimbaker, yeah. it would be nice to have a cli flag to enable that
<jimbaker> kim0, but clearly the relation settings, along with the specific hooks firing, are the key to understanding what actually happens
<hazmat> kim0, er.. that should be debug-log -x hook.output
<kim0> oh ok
<kim0> hazmat: indeed at least popular channels should be listed on the help message please
<kim0> do we need a bug for showing relation-settings changes in logs .. or is someone on that
<jimbaker> kim0, we need a bug for that and in status too i think
<jimbaker> (so two bugs)
<kim0> so "ensemble status" shows relation settings for every service unit, right?
<jimbaker> kim0, it does not yet. but i think it should
<kim0> yeah, I meant that's how you'd want it .. cool .. filing
<jimbaker> the only consideration here is that there security issues
<kim0> ah passwords and stuff
<jimbaker> but i think these be addressed later to mask it out properly. any system auditing always has these issues from what i have seen in the past
<kim0> hmm ok
<jimbaker> not showing it in ensemble status is simply doing it by obscurity anyway
<jimbaker> so the good thing about showing it in ensemble status sooner than later is to keep reminding ourselves of this problem ;)
<jimbaker> kim0, you might be interested in this intro fragment, http://pastebin.ubuntu.com/596140/ - it's not going into the branch lp:~jimbaker/ensemble/new-hook-semantics-5-docs because of its lack of precision is currently misleading
<jimbaker> but there's some useful text in there that we may want to incorporate into the ensemble docs, after some more editing
<kim0> jimbaker: thanks :)
<jimbaker> kim0, re state diagrams - you should be aware of sdedit, if not already - see http://sdedit.sourceforge.net/example/index.html
<jimbaker> there's integration with sphinx too
<jimbaker> (much like the dot integration)
<jimbaker> i could definitely see a script reading the debug log and producing this type of output
<kim0> jimbaker: thanks useful info :)
<jimbaker> (well not loops, but the example is just an indication of what can be done, if there's data to support it)
<_mup_> Bug #766306 was filed: example formulas should use ensemble-log to log useful messages <Ensemble:New> < https://launchpad.net/bugs/766306 >
<niemeyer> jimbaker:   Its purpose is to allow the formula to
<niemeyer> 255	+    clean up established state
<niemeyer> 256	+
<niemeyer> 257	+    The service unit can then clean up any established state. 
<jimbaker> niemeyer, thanks
<niemeyer> jimbaker: No problem, +1 with that
<jimbaker> niemeyer, cool, i will get it merged in shortly!
<kim0> why doesn't the bot here catch merge proposals :)
<jimbaker> kim0, yeah, it's an annoying bug in our mup
 * kim0 kicks _mup_ 
<_mup_> Bug #766315 was filed: status should show relation settings <Ensemble:New> < https://launchpad.net/bugs/766315 >
<niemeyer> kim0: Strange problem indeed
<_mup_> Bug #766317 was filed: debug-log should show relation settings changes <Ensemble:New> < https://launchpad.net/bugs/766317 >
<kim0> filed the two bugs
<kim0> TeTeT: you might enjoy this text too http://pastebin.ubuntu.com/596140/
<TeTeT> kim0: looks great :)
<_mup_> ensemble/new-hook-semantics-5-docs r204 committed by jim.baker@canonical.com
<_mup_> Fixed -relation-broken description
<_mup_> ensemble/trunk r200 committed by jim.baker@canonical.com
<_mup_> merge new-hook-semantics-5-docs [r=niemeyer][f=756581]
<_mup_> Updates the documentation of hooks with respect to the new hook
<_mup_> semantic changes. Copy edits the formula section as a whole to clarify
<_mup_> how hooks actually execute, and under what conditions.
<jimbaker> cool, 200th revision on trunk
<kim0> woohoo :)
<_mup_> ensemble/unit-agent-formula-upgrade r237 committed by kapil.thangavelu@canonical.com
<_mup_> add utility workflow function to tell if a unit is running and its state.
<hazmat> jimbaker, we're closing on 1k unit tests as well
<hazmat> niemeyer, jimbaker, bcsaller standup?
<jimbaker> hazmat, sounds good on all counts!
<bcsaller> yeah
<jimbaker> (so to speak :)
<niemeyer> Yep, let's do it
<hazmat> bcsaller, jimbaker skype?
<bcsaller> I'm on
<niemeyer> Up
<hazmat> bcsaller, cool, false skype presence info
<_mup_> ensemble/unit-agent-formula-upgrade r238 committed by kapil.thangavelu@canonical.com
<_mup_> clear upgrade before flag before retrieving service formula id, remove executor run lock acquisition when starting
<_mup_> ensemble/unit-agent-formula-upgrade r239 committed by kapil.thangavelu@canonical.com
<_mup_> missing test for is_unit_running
<_mup_> ensemble/yaml-state-node r194 committed by bcsaller@gmail.com
<_mup_> simpler YAMLState
<_mup_> reflecting the changes from review this version maintains less internal state
<_mup_> additional test of multiple writes
<niemeyer> hazmat: There are still many appearances of 'cli' in the file.. we should probably try to put acronyms in capitals in user documentation
<hazmat> niemeyer, hmm.. i never considered the possiblity that acronoym would be unknown.
<hazmat> fair enough re caps
<niemeyer> hazmat: Uh.. I'm not saying it's unknown.. it's just an acronym
<niemeyer> hazmat: same way that API is generally capitals
<niemeyer> and AWS, etc
<_mup_> ensemble/ensemble-resolved r240 committed by kapil.thangavelu@canonical.com
<_mup_> ensemble resolved cli
<niemeyer> hazmat: :-)
<hazmat> niemeyer, doh. ;-)
<niemeyer> hazmat: The branch looks good
<hazmat> niemeyer, yeah.. i still need to do an end to end test, but else it should be good.
<niemeyer> hazmat: Was a bit overwhelming, and required some morning-level freshness, but mostly because there's a lot there.. was pretty straightforward
<hazmat> niemeyer, cool. yeah it grew a bit bigger than i wanted in one branch, but it was getting hard to separate out
<hazmat> niemeyer, fwiw, i'm planning on doing the resolved stuff much the same way
<hazmat> unit_state.set_resolved_flag(retry).. unit_state.set_relation_resolved_flag(mapping)
<hazmat> mapping being relation_state_id: retry_boolean
<niemeyer> hazmat: Hmm
<hazmat> and then watches on the agent for the unit, and a watch in the lifecycle for the relation
<niemeyer> hazmat: Why do we need a flag on it?
<hazmat> niemeyer, we don't, probably shouldn't have that in the name
<hazmat> looks like i can deal with the ambiguous anonymous relations without issue here
<niemeyer> hazmat: I mean.. isn't the fact we have an _error state already indicating a non-resolved state, and thus putting the unit back in running would be the action?
<hazmat> niemeyer, i need to signal to the agent that it should take action
<hazmat> the resolved stuff triggers the watch which triggers the agent to do some work
<niemeyer> hazmat: Won't it notice you changing its state?
<hazmat> the workflow transition even sans hook needs to take place on the agent
<hazmat> niemeyer, no it is the owner of that data, directly changing the state, could be arbitrary, without the nesc. lifecycle invocations
<hazmat> i could genericize it via the workflow state client
<niemeyer> hazmat: Ok, sounds good
<hazmat> to allow marking generic firing of transitions and recording that as intent to act upon
<niemeyer> hazmat: Nah, your idea of a semantically rich request actually feels good
<hazmat> niemeyer, cool
<niemeyer> hazmat: Otherwise, you're right in that the agent can't do anything more intelligent with the request
<niemeyer> hazmat: "Oh, I'm resolving, thus ..."
<hazmat> niemeyer, yeah.. or like oh.. my state is stopped now, what does that mean.. where was was i, this is not beautiful house ;-)
<hazmat> ^my
<niemeyer> hazmat: Right :-)
<_mup_> ensemble/ensemble-resolved r241 committed by kapil.thangavelu@canonical.com
<_mup_> add a is_relation_running workflow utility function.
<_mup_> ensemble/yaml-state-node r195 committed by bcsaller@gmail.com
<_mup_> merge trunk
<_mup_> ensemble/unit-agent-formula-upgrade r240 committed by kapil.thangavelu@canonical.com
<_mup_> make sure the upgrade flag presence check is done before we clear the flag.
<hazmat> niemeyer, any thoughts on the async watcher and creators stuff?
<niemeyer> hazmat: Sorry, what's the context again?
<niemeyer> hazmat: Ah, the email on the list
<niemeyer> hazmat: I forgot to reply.. let me look at it again
<hazmat> niemeyer, my mail to the list regarding usage patterns for watchers
<hazmat> niemeyer, thanks
<niemeyer> hazmat: Sorry about that
<hazmat> i'm noticing a lot of tests i'm hacking in fire_transition("stop") to stop the background activity so the test doesn't fail because of the async work
<_mup_> ensemble/trunk r201 committed by bcsaller@gmail.com
<_mup_> Merge yaml-state [r=niemeyer] [f=758583]
<_mup_>  
<_mup_> Adds a reusable utility for managing access to Zookeeper nodes containing YAML serialized state. The YAMLState object proxies as a Python dict for normal operation. Prior to access instance.read() should be invoked and instance.write() will update the Zookeeper state.
<niemeyer> hazmat: Replied
<niemeyer> hazmat: Yeah, that sucks
<niemeyer> hazmat: I have an idea about a better system to make testing this easier
<niemeyer> hazmat: That's the good news.. the bad news is that I don't know how to implement it yet
<hazmat> niemeyer, yeah.. i started poking adding something like this but it was getting a little complicated untop of all the work in the upgrade branch.. there's some subtly there.
<hazmat> there where one or two tests that error'd out with the change, but i think i'd like to approach in a more systematic way.
<niemeyer> hazmat: Like this what, more specifically?
<hazmat> niemeyer, i added support to the unit lifecycle start, that it only fired after the unit relations watcher had fired once.
<niemeyer> hazmat: I think we may come up with a simpler way to test that kind of logic altogether
<niemeyer> hazmat: Which would get rid of all sleeps/waits/etc
<niemeyer> hazmat: Higher bandwidth might be nice for that conversation, though
<hazmat> niemeyer, agreed, let's do it for tomorrow's standup? or before the standup?
<hazmat> i'm making good head way on 'resolved'
<niemeyer> hazmat: Might be good to have it separated
<niemeyer> hazmat: It may be pretty fast, but it could also take  abit
<niemeyer> hazmat: Just as a heads up, when testing we're in control of both the reactor and the zookeeper deployment
<niemeyer> hazmat: We should be able to figure if there's something pending to be executed, or if the system is stale for real
<niemeyer> hazmat: No deferred calls in twisted's reactor after a ping roundtrip to zookeeper means the system is stale
<niemeyer> hazmat: Nothing will happen
<niemeyer> I think we can use that to solve those problems
<hazmat> niemeyer, in general for sequencing it feels like it would be easier  for tests if the background activity was done after the method completed.
<hazmat> at least in a steady state
<niemeyer> hazmat: The background processing is always done after the method completed, so I don't see what you really mean
<hazmat> inspecting the reactor for latent deferreds, and then doing something with that feels kinda of magical as well
<hazmat> niemeyer, well the method could delay its return till the initial invocation of the watcher, at which point its in a steady state, minus any additional test induced state chagnes
<niemeyer> hazmat: Well.. it is much less magical to me than arbitrarily saying the event has to happen in a given timeframe when in fact it doesn't
<niemeyer> hazmat: The initial invocation of the method has no special meaning, as I pointed out in the list
<niemeyer> hazmat: It can be empty, and then can have data 5 milliseconds later
<hazmat> the event happens fairly quickly, just typically not before the test exits... part of the problem is arguably that we need to separate our tests a bit more, and not have tests which have relations that are being setup in the background.
<hazmat> niemeyer, sure, it can, but the test is explicitly setting up a state, ideally the methods its invoking should be done/current with the state when complete. the test is the state driver/manipulator for changes.
<niemeyer> hazmat: Yeah, exactly.. that "ideally" is the problem.. there's no hard constraint about that
<hazmat> a change in data  5ms seconds later in this case is one thats being done by the test, and the test should expect/test for that
<hazmat> which is also arguably why the test shouldn't be subclassing something that sets up a relation if doesn't care about it.
<niemeyer> hazmat: No questions about that.. doesn't affect the points though
<niemeyer> hazmat: The problem is that, e.g., "creation relation" + "set watch on relation" will fire in one way
<niemeyer> hazmat: but "set watch on relation" + "create relation" fires in a different way
<niemeyer> hazmat: Our system was built in such a way that things work well in both cases
<niemeyer> hazmat: Adding special behavior to the first time it fires fails in the second case.
<hazmat> the system always reads the current state, and tries to match it. i agree adding semantics to the first invocation feels a bit iffy perhaps.. but the caller knows then that the async watch creator has completed to matching the current state when its done, and that makes some of the tests a bit easier.. i think its something we can solve in practice for a large set of the cases by having the test setup reflect only what the tes
<hazmat> t needs.
<hazmat> niemeyer, but the problem still exists beyond tests
<hazmat> for example the debug-log and debug-hook stuff are setting up watches, and then going about their business, and some information/behavior is lost
<hazmat> because by the time those watches have finished, the install and start methods have already executed
<niemeyer> hazmat: It makes tests easier and unrealistic..  the first invocation has no special meaning.  It can have state or not, and it should correctly yield the same state with A+B or B+A
<hazmat> niemeyer, a phone call would be better
<niemeyer> hazmat: There are tons of things which right now simply work because of that
<niemeyer> hazmat: Indeed, sorry.. I wanted to give a heads up about the idea and ended up taking your attention
<hazmat> and they'll continue to work that way, but certain cases need to know that the async behavior matches the state before continuing
<hazmat> no worries, we can discuss in more detail tomorrow
<niemeyer> Cool
<niemeyer> I'll push some personal stuff now.. want to get a new mgo release out tonight still.  I'm hoping that after some work over the following nights and on the weekend it should be ready to push the prototype needs.
<_mup_> ensemble/unit-agent-formula-upgrade r241 committed by kapil.thangavelu@canonical.com
<_mup_> a test was incorrectly setting up the workflow state.
<_mup_> ensemble/trunk r202 committed by kapil.thangavelu@canonical.com
<_mup_> merge unit-agent-formula-ugprade [r=niemeyer][f=755062]
<_mup_> This merge completes the upgrade-formula functionality for ensemble,
<_mup_> by implementing the unit agent support for upgrading formulas and
<_mup_> executing formula-upgrade hooks. It changes the unit workflow to add 
<_mup_> a new transition from the started state, that effects an upgrade on
<_mup_> the unit. It also adds the notion of out-of-band hook execution on 
<_mup_> the hook executor for the formula-upgrade-hook to be invoked while 
<_mup_> the executor is stopped but still queuing new hook executions.
<_mup_> ensemble/trunk-merge r186 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
#ubuntu-ensemble 2011-04-20
<hazmat> kim0, i was curious where the log of last weeks cloud-meeting was at
<kim0> hazmat: http://irclogs.ubuntu.com/2011/04/13/%23ubuntu-cloud.html#t19:01 and summary at http://foss-boss.blogspot.com/2011/04/ensemble-cloud-community-meeting_14.html
<hazmat> kim0, awesome thanks!
<_mup_> Bug #766721 was filed: upgrade hook should get access to previous and new versions numbers of the formula <Ensemble:New> < https://launchpad.net/bugs/766721 >
<_mup_> ensemble/config-state-manager r197 committed by bcsaller@gmail.com
<_mup_> merge trunk
<_mup_> ensemble/expose-1-dummy-provider r200 committed by jim.baker@canonical.com
<_mup_> Dummy provider support for exposing ports
<_mup_> ensemble/expose-1-dummy-provider r201 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/expose-1-dummy-provider r202 committed by jim.baker@canonical.com
<_mup_> Test code path on unexposing ports in dummy provider
<hazmat> g'morning
<_mup_> ensemble/natty-image r189 committed by kapil.thangavelu@canonical.com
<_mup_> add a wait to ensure ssh is available, bump default image to natty daily from 20-4-2010
<_mup_> ensemble/natty-image r190 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> Bug #767021 was filed: Use natty machine image <Ensemble:New> < https://launchpad.net/bugs/767021 >
<_mup_> ensemble/resolved-state-api r187 committed by kapil.thangavelu@canonical.com
<_mup_> move is relation running api, up stack
<_mup_> ensemble/config-get r199 committed by bcsaller@gmail.com
<_mup_> config-get synced to current
<_mup_> ensemble/ensemble-resolved r244 committed by kapil.thangavelu@canonical.com
<_mup_> test cases for resolved command line.
<hazmat> bcsaller, g'morning.. still up?
<bcsaller> hazmat: sadly yes
<hazmat> bcsaller, fwiw, i took a stab at getting the yaml-state tests running with with the hook state context, worked beautifully.
<hazmat> er. running with the hook state context tests that is
<bcsaller> ha, I was just playing with that.
<bcsaller> thats great
<bcsaller> state/relation should change as well 
<hazmat> bcsaller, i'm curious what  your game plan is with the hook context, extend it with a yaml state for unit relation settings, and service settings and extend the api to get/set/delete + _relation and get + _service and wire that into the api server?
<hazmat> bcsaller, if you need any help, i've got some extra cycles
<bcsaller> thats pretty much the direction I'm headed. I already went through and did config-get but then I had to wire it into hook.flush to call write and I figured I'd better insert a pipe for those
<bcsaller> other flush calls
<hazmat> bcsaller, even with the service stuff there is still only one yaml state for the purposes of modification by the hook, afaics. the service settings are read only from the hooks
<hazmat> wiring in the initialize/read on the yaml state proved a bit more invasive but minor
<bcsaller> yeah, I imagine you did the same thing, but I just put it in the hooks node cache. 
<bcsaller> and use a single instance var for the service options
<hazmat> hmm.. i ended up bypassing the node cache, and stuck the yaml states as attributes, there's a spot in read which checks if its self unit relation, where i just divert to the instance yaml state
<hazmat> s/read/get_value
<hazmat> i'm sure they both work fine.
<bcsaller> yeah
<_mup_> ensemble/resolved-state-api r188 committed by kapil.thangavelu@canonical.com
<_mup_> new state errors for resolved.
<_mup_> ensemble/resolved-state-api r189 committed by kapil.thangavelu@canonical.com
<_mup_> get/set/clear for unit resolved and relation resolved.
<hazmat> bcsaller, fwiw, i expect the workflow for service change would look like that for an upgrade (only from the running state circular transition)
<hazmat> hmm.. dynamic dependencies
<hazmat> but against a static set of knows
<hazmat> works
<niemeyer> Yo!
<hazmat> niemeyer, top of the morning
<niemeyer> hazmat: Hey man, feeling good today?
<hazmat> niemeyer, yup.. making good progress on resolved
<hazmat> doing a some user oriented docs for it now
<niemeyer> hazmat: Sweet
<hazmat> its hard to talk about sometimes without exposing internals
<_mup_> Bug #767147 was filed: upgrade-formula should have a --force option to allow for upgrading without incrementing the version. <Ensemble:New> < https://launchpad.net/bugs/767147 >
<SpamapS> so.. natty by default? really? ;)
 * SpamapS clings tightly to his beloved LTS
<niemeyer> hazmat: You've answered privately to me on that thread, was that the intention?
<niemeyer> SpamapS: Hehe :)
<SpamapS> hazmat: I think you missed xx-relation-departed in the development update
<hazmat> niemeyer, no it wasn't.. 
<hazmat> niemeyer, i'll resend
<niemeyer> hazmat: Cool, thanks.. will wait for it before responding
<niemeyer> hazmat: Just so people get in the proper order
<niemeyer> hazmat: People will be amazed at how fast I can answer a message, though ;-)
<hazmat> niemeyer, :-)
<SpamapS> ooo I didn't know upgrades were moving along so nicely
<hazmat> niemeyer, sent
<SpamapS> hrm.. nothing in docs/source ??
<hazmat> SpamapS, i forget to move them out of drafts
<niemeyer> hazmat: Oh, we have to move something else too
 * niemeyer tries to remember
<niemeyer> The configuration stuff
<hazmat> niemeyer, looks like it already is for service-config
<hazmat> not sure if its wired into the index
<niemeyer> hazmat: It's in drafts?
<niemeyer> hazmat: Drafts don't have to be manually inserted
<SpamapS> hazmat: indeed. I saw the bug report about passing versions and it immediately made me think that we should be copying debian packaging in this regard..
<hazmat> niemeyer, its not in the drafts.
<niemeyer> hazmat: Right, it should be
<niemeyer> It's not ready yet
<hazmat> ah.. yeah.. i'll do a quick branch to fix that up.
<niemeyer> hazmat: It's also not in the index, but the right thing is to simply move it to the drafts directory
<niemeyer> hazmat: Just cowboy it
<hazmat> SpamapS, i'm all ears.. what's debian do?
<niemeyer> as [trivial]
<hazmat> k
<SpamapS> hazmat: http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
<SpamapS> hazmat: you can probably skip to 6.5
<SpamapS> hazmat: and 6.6 is actually where my mind went
<SpamapS> "Details of unpack phase of installation or upgrade"
<hazmat> hmm
<SpamapS> Now, frankly.. I'd say less than 1% of packages I've touched implement things like 'abort-deconfigure'
<SpamapS> But the useful bits are postinst (upgrade|install) and preinst (upgrade|install) .. and somewhat.. prerm
<hazmat> SpamapS, right now our model is a single hook for the upgrade.. on failure we're going to allow either retry or manual fiddling and resolution.
<SpamapS> hazmat: thats probably ok, because we don't care at all about removal
<hazmat> breaking it out into a half-dozen hooks... 
<hazmat> i dunno it might be real world required.. but i'm not sure debian packaging is the friendly feel that ensemble is going  for 
<SpamapS> I'm thinking more about the flow
<SpamapS> *probably* want to call install again
<SpamapS> but w/ one hook, can be done at maintainer's choice..
<SpamapS> hazmat: hah true dat. ;)
<hazmat> SpamapS, hmmm.. yeah.. i expect the smarts of the upgrade are hopefully encapsulated in the package upgrade.. i'm totally willing to revisit, but it would be nice if it was practice driven.
<hazmat> adding the upgrade version thing otoh, just seems like a no-brainer 
<SpamapS> its useful information for sure
<SpamapS> Did we ever settle on revision number being anything other than an int?
<SpamapS> hazmat: also how are upgrades of units handled? in parallel w/ no coordination?
<hazmat> SpamapS, yes.. there's a separate upgrades doc from the formula doc which explores more advanced upgrade options
<hazmat> SpamapS, the remote sides never see the upgrade with a formula-upgrade atm
<hazmat> it looks like the service is still there..
<hazmat> if they try to talk to it via their relation settings, the response will be delayed till the upgrade is complete
<hazmat> if the upgrade fails, all hooks are queued till the issue is resolved..
<hazmat> how its resolved is the subject of a new command ensemble resolved.. which allows manually correcting for hook errors on units and unit relations,  i'm writing some docs for it atm
<SpamapS> hazmat: well that makes perfect sense. in the upgrade hook I can enumerate and do sets right?
<hazmat> SpamapS, not yet
<SpamapS> ok.. because that would be useful
<hazmat> the only real functionality that the upgrade hook offers atm that's different than normal hooks is that its executed first out of the new formula.
<SpamapS> but no, services should be resilient to transience of their related partners..
<hazmat> SpamapS, yeah.. i did some discussion of that in the futures section of the upgrade-formula doc
<hazmat> re enumerate and set
<SpamapS> hazmat: its kind of a corner case really
<hazmat> SpamapS, the real issue with it at the moment, is that looks we're tending towards some anonymous relations
<hazmat> SpamapS, i definitely think its realistic
<hazmat> maybe a corner, but for advanced upgrading usage, its a pretty important piece.
<hazmat> by tending to anon relations, i mean the mysql formula for example, can have multiple relations with clients named 'db'.. 
<SpamapS> The one I worry about is the new config file that needs to be generated.. need to enumerate and basically rejoin every relation.
<hazmat> the identity of those multiple instances of the same name is a question..
<SpamapS> hazmat: sorry I've gone cross eyed again. huh?
<hazmat> at the moment with another command targeting unit relations, i'm treating them as a set, but if we expose to hooks, they have to have some names
<hazmat> SpamapS, say we have two wordpress blogs/services, and they both connect to mysql
<hazmat> now mysql has two relations instances, but they have the same name.
<SpamapS> two services w/ the same name?
<hazmat> SpamapS, two relations with the same name
<SpamapS> so the services are 'blog0' and 'blog1' .. .right? 
<hazmat> SpamapS, sure
<SpamapS> mysql has seen them as such and there are two databases?
<hazmat> perhaps we should encode the remote service name in user visible portion of the relation to add a qualifier for enumeration
<hazmat> ^of the relation name
<SpamapS> its sort of the natural order of things
<hazmat> niemeyer, ^ re anon relation addressing
<SpamapS> service_name+relation_name == unique relation id
<niemeyer> hazmat: ECONTEXT
<hazmat> SpamapS, yeah.. that sounds good to me
<niemeyer> hazmat: Let me read back
<SpamapS> hazmat: given that one would be able to enumerate their established relations and have something that can be used to pass to tools to say "relation-list blog0-db" ?
<SpamapS> actually blog0:db would imply it is the :db on blog0's side.. but its really the db on mysql's side isn't it?
<hazmat> SpamapS, yeah.. probably as RELATION_NAME=blog0-db relation-list 
<hazmat> yeah.. its the mysql side.
<SpamapS> Well I love the idea of being able to do that
<hazmat> we can reverse the order rel-name/service-name
<niemeyer> hazmat: Ok, in sync again
<niemeyer> hazmat: I don't think we should use the service name anywhere in a relation
<hazmat> niemeyer, even as a transient qualifier?
<hazmat> transient id
<niemeyer> hazmat: It enables people to build custom-logic based on the service name, which is a big killer for the formula concept
<hazmat> hmmm.. it does
<niemeyer> hazmat: We have unique ids for the relation already
<niemeyer> hazmat: The internal ids
<niemeyer> hazmat: They're not semantically meaningful, but that's exactly the idea.. they shouldn't be
<hazmat> niemeyer, are you suggesting we expose them to hooks for the purpose of iteration/transient ids?
<hazmat> niemeyer, cool
<niemeyer> hazmat: Yeah, I think it'd be fine to have, e.g. db-424
<niemeyer> hazmat: Just to enable people to iterate over multiple relations in an upgrade hook or similar
<hazmat> niemeyer, yup.. sounds good.
<SpamapS> Yeah it doesn't matter what the id is, just that a formula can be aware of them on upgrade so that they can list/get all the data out of them.
<_mup_> Bug #767195 was filed: Ensemble should have hook cli apis to enumerate and interact with all the relations of a unit. <Ensemble:New> < https://launchpad.net/bugs/767195 >
<hazmat> transcript in the bug report
<niemeyer> hazmat: Thanks
<niemeyer> SpamapS: Right
<SpamapS> hazmat: when you say the upgrade spec is in drafts.. where is that so I can read it?
<hazmat> SpamapS, its in the docs/source/drafts dir.. it will be online in about 15m.. after i move things around for trunk.
<hazmat> and the doc site rebuilds
<hazmat> niemeyer, any reason we're not using rtfd?
<hazmat> i thought we where, but the links point back to chinstrap
<niemeyer> hazmat: It broke repeatedly
<SpamapS> oh doh :)
<niemeyer> hazmat: So I gave up on them and put a cronjob in place
<hazmat> niemeyer, ugh.. did we get any feedback or insight from them.. i thought it was pretty active dev
<niemeyer> hazmat: It is pretty active.. perhaps too active :)
<niemeyer> hazmat: At one point they said "Oh, sorry, we broke bzr support a couple of weeks ago in PyCon"
<hazmat> its amazing to me after doing tdd, how many branches get merged in openstack with neglible or no tests.
<niemeyer> hazmat: The second time I didn't bother to ask
<hazmat> but their dev is insanely active
<SpamapS> hazmat: so I'd like to see some coordination possibilities. You mention it in the doc a bit..
<niemeyer> hazmat: Yeah, to me it feels a bit like building a bomb :-)
<hazmat> SpamapS, yeah.. those are still open questions, i'd be glad to brain storm with you on it.
<hazmat> what should it look like to a formula author, what should ensemble do to enable it.
<hazmat> i mean we can totally play tick-tac-toe in a formula..  we can probably do a round-robin :-)
<SpamapS> hazmat: basically it would be good to be able to run 'rm -f /srv/code/live && ln -s /srv/code/4481 /srv/code/live' at the exact same time on all machines.
<SpamapS> hazmat: that is a huge reason people are using mcollective right now
<hazmat> SpamapS, its async, but the latency should be comparable to mcollective
<hazmat> perhaps better
<niemeyer> hazmat: Yeah, that's what I was going to say
<niemeyer> hazmat: We're almost certainly better
<niemeyer> hazmat: Due to the live connections
<hazmat> niemeyer, mcollective is pub/sub on amqp.. not parallel ssh, so its pretty comparable
<SpamapS> hazmat: another way people do it is they pick an exact second they want it to happen.
<hazmat> at least that was my understanding of it
<niemeyer> hazmat: Ah, ok, yeah, that may be similar
<hazmat> i found a pretty nifty implementation of func in python.. using zeromq the other day, https://github.com/thatch45/salt
<SpamapS> right mcollective is a two way channel to each box facilitated via AMQP
<hazmat> also uses ruby factr lib to expose to its python scripts.
<hazmat> er. expose machine info
<SpamapS> thats just the default agent
<SpamapS> people write them to do all kinds of weird stuff. :)
<SpamapS> the amorphous nature is one reason we have an opportunity to provide something with more structure and less custom work
<hazmat> niemeyer, here's the delta for the trivial docs update.. https://pastebin.canonical.com/46437/
<niemeyer> SpamapS: Yeah, we have to think a bit more about actual impact, but being able to coordinate execution does sound like something we should be looking at
<niemeyer> hazmat: It shouldn't be in the index
<niemeyer> hazmat: Wait.. is it reversed?
<SpamapS> niemeyer: I think we talked about a relation-cas command before... getting back to what I wanted tho.. relation-lock ;)
<hazmat> niemeyer, the upgrades doc talks about the general update scenarios, and the upgrade-formula one goes into details of the formula upgrade.
<hazmat> niemeyer, into internals/specs?
<SpamapS> either way, thats a nice to have. just being able to upgrade formulas is GREAT right now
<niemeyer> hazmat: They are both being inserted into the index?
<hazmat> SpamapS, should be a few more goodies coming for formula authors before budapest
<SpamapS> settings is, IMO, the last thing before I will start deploying and managing a couple of things on ensemble.
<hazmat> niemeyer, atm yes.
<niemeyer> hazmat: You can just drop the files in drafts/
<SpamapS> Looking specifically at Roundcube for my webmail. :)
<niemeyer> hazmat: Oh, wait, sorry.. have you merged upgrades already?
<hazmat> SpamapS, yeah.. that's key
<hazmat> niemeyer, yes.. the upgrades are merged, i did a few end-to-end tests yesterday
<niemeyer> hazmat: Awesome, fantastic, sorry for the confusion
<niemeyer> hazmat: +1 on the change
<hazmat> niemeyer, cool, thanks
<_mup_> ensemble/trunk r203 committed by kapil.thangavelu@canonical.com
<_mup_> pull docs for committed upgrade functionality out of drafts, move service-config into the drafts dir till it lands. [trivial][r=niemeyer]
<niemeyer> SpamapS: Oh, that's awesome
<niemeyer> hazmat: Regarding the natty branch, can we try to ssh actively rather than just sleeping?
<hazmat> niemeyer, not sure what you mean by active.. fabric uses it but jumps if doesn't work.. 
<niemeyer> hazmat: Active != sleeping
<hazmat> i could try turning up the logging there and see if i can find some additional info out of it.
<niemeyer> hazmat: Nah
<hazmat> but its pretty low priority to me
<niemeyer> hazmat: Just pondering, but that's rare enough
<niemeyer> +1ing
<niemeyer>         parser.add_argument("--log-level",
<niemeyer>                             help="Set a logging level "
<niemeyer>                             "CRITICAL|DEBUG|INFO|ERROR|WARNING",
<niemeyer>                             default=logging.WARNING)
<niemeyer> Hmm
<niemeyer> INFO is the most commonly used default for logging, I think
<niemeyer> Was just reviewing ahmed's branch with --log-level=INFO everywhere
<niemeyer> kim0: ^
<niemeyer> Wait, hmmm..
<niemeyer> It might be more complicated than that.. --log-level means two different things for different commands it seems
 * niemeyer inspects further
<niemeyer> Ah, yes indeed
<hazmat> did it again..
<hazmat> re email send
<hazmat> dog walk.. bbiab
<_mup_> ensemble/expose-1-dummy-provider r203 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/expose-2-hook-commands r203 committed by jim.baker@canonical.com
<_mup_> Initial commit
<_mup_> ensemble/expose-2-hook-commands r204 committed by jim.baker@canonical.com
<_mup_> Merged upstream
<kim0> so let me know whether or not INFO is fine for logging
<niemeyer> kim0: I'm working on a fix for that
<niemeyer> kim0: It is fine for logging
<kim0> cool
<niemeyer> kim0: The issue is that INFO should be the default
<kim0> aha .. yeah agree
<niemeyer> kim0: and it's not because we're currently mixing two different concepts
<kim0> just bec it's the correct level
<niemeyer> kim0: Filtering logs, and logging at a given level
<niemeyer> kim0: I'm fixing that
<kim0> got you .. rock on
<kim0> reminder of our meeting in 3 hours
<niemeyer> kim0: If you want, you can already change that in your branch
<niemeyer> kim0: Just remove all the --log-level INFO from ensemble-log calls
<niemeyer> kim0: Thanks
<kim0> ok .. will clean that and push
<hazmat> this is to change the default level in the code or the command?
<niemeyer> hazmat: The command
<niemeyer> hazmat: It's logging everything as WARNING, because it's incorrectly picking the filtering argument to use as the level to log for
<niemeyer> Almost done with the fix.. will push after lunch
<hazmat> ick
<hazmat> niemeyer, cool
<hazmat> niemeyer, sorry i didn't realize you had posted to me privately.. if that was intentional
<niemeyer> hazmat: It wasn't.. my fault this time
<jimbaker> brb, need to drop off car for service
<kim0> niemeyer_lunch: should I remove x in "set -eux" in formulas ?
<kim0> I think it's too verbose to be the default for anything :)
<niemeyer> kim0: Agreed
<niemeyer> kim0: Yes, please remove it
 * kim0 nods
<hazmat> yipee
<hazmat> off to lunch, bbiab
<kim0> niemeyer: pushed the updated branch "set -eu" and no "--log-level INFO"
<niemeyer> kim0: Awesome, thanks!
<jimbaker> kim0, makes sense to me with the new logging you're doing
<kim0> cool
<kim0> hmm I think I'm hitting some bug
<jimbaker> although i did find -x to be very helpful
<kim0> probably while developing the formula only
<jimbaker> we really need the relation setting logging
<jimbaker> also it would be interesting to have some sort of monitoring of the nodes integrated in some way (not necessarily though logging). need to think of what my wishlist should be
<kim0> hehe
<kim0> can someone please check http://paste.ubuntu.com/596612/
<jimbaker> anyway, round 2 on getting car fixed, my father in law put the wrong tires in the car
<kim0> some twisted errors
<kim0> jimbaker: :) that's what fathers in law are for hehe
<kim0> I think ensemble-log is broken or something
<kim0> exceptions.AttributeError: Logger instance has no __call__ method
<kim0> hmm should I file a bug ?
<jimbaker> kim0, back now. yes, -x is definitely necessary for formula debugging with bash script hooks (at least for this novice bash scripter)
<kim0> jimbaker: any idea why I'm getting those errors http://paste.ubuntu.com/596612/ though
<jimbaker> kim0, yikes that's ugly
<kim0> yeah :)
<kim0> trying to write that user tutorial, but the moon is misaligned :)
<kim0> I wonder what ensemble version runs on the ec2 instances? does it pull trunk
<jimbaker> kim0, it does look like a problem with ensemble-log as you mentioned
<jimbaker> kim0, by default it does, but you can choose another branch instead
<kim0> ok cool
<_mup_> Bug #767364 was filed: ensemble-log logs in wrong level <Ensemble:In Progress by niemeyer> < https://launchpad.net/bugs/767364 >
<kim0> is it realistic to expect those ensemble-log crashes to be fixed soonish .. like today :)
<jimbaker> for an environment in your ~/.ensemble/environments.yaml (or otherwise specified), use something like this:
<jimbaker>     ensemble-branch: https://code.launchpad.net/~jimbaker/ensemble/new-hook-semantics-remove-ensemble-change-env-var
<kim0> thanks
<jimbaker> now i should not be so quick to say it will choose trunk as the desired version... in my environments.yaml file, i do have this
<niemeyer> kim0: That's pretty weird
<jimbaker>  ensemble-branch: http://bazaar.launchpad.net/~ensemble/ensemble/trunk
<niemeyer> bcsaller: Do you have any hints on thta?
<niemeyer> that
<jimbaker> i just comment out as desired
<kim0> jimbaker: thanks :)
<jimbaker> kim0, i suspect someone who doesn't run bleeding edge has more experience on this one aspect to be honest ;)
<bcsaller> hmm
<bcsaller> looks like the setup of passing in the log handle would be wrong there
<jimbaker> kim0, if you ever use ensemble-branch, prepared to be occasionally confused as might get stuff out of sync. ("didn't we fix that? ahhh...")
<kim0> jimbaker: trunk is probably a safe choice for me
<kim0> :)
<jimbaker> kim0, indeed, i think that's the case. it doesn't impact your principal use case, which is formula dev. but if you want to try out a new branch that fixes a problem, then it lets you have immediate access
<kim0> bcsaller: should I file a bug on that logging thing
<bcsaller> kim0: yes, I think its on the server side, sounds misconfigured
 * hazmat catches up
<hazmat> ensemble defaults to trunk if its not specified via 'ensemble-branch'
<hazmat> interesting tracebacks, one in amp and one in the amp server protocol when calling out to the logging
<hazmat> looks like ensemble-log is broken
<hazmat> hook cli api that is
 * hazmat reboots for natty updates
<_mup_> Bug #767391 was filed: ensemble-log causing twisted crashes <Ensemble:New> < https://launchpad.net/bugs/767391 >
<jimbaker> hazmat, thanks for the clarification. what happens when ensemble is installed from a ppa?
<hazmat> jimbaker, trunk on the remote
<jimbaker> hazmat, sounds good for now i think
<_mup_> ensemble/trunk r204 committed by gustavo@niemeyer.net
<_mup_> Merged add-ensemble-log-to-examples branch [a=kim0] [r=niemeyer]
<_mup_> This adds sensible ensemble-log messages to the example formulas, and
<_mup_> removes the excessively verbose -x flag from scripts.
<_mup_> Bug #767405 was filed: Add open-port and close-port hook commands <Ensemble:In Progress> < https://launchpad.net/bugs/767405 >
<_mup_> Bug #767407 was filed: Add "ensemble expose" and "ensemble unexpose" subcommands <Ensemble:New for jimbaker> < https://launchpad.net/bugs/767407 >
<_mup_> Bug #767414 was filed: Modify "ensemble status" to display exposed ports and whether a service is exposed or not <Ensemble:New> < https://launchpad.net/bugs/767414 >
<_mup_> Bug #767418 was filed: Modify the provisioning agent to watch ZK settings for exposed services and makes appropriate firewall changes through the provider <Ensemble:New for jimbaker> < https://launchpad.net/bugs/767418 >
<_mup_> Bug #767420 was filed: Add a provider implementation for EC2 to support port exposing <Ensemble:New for jimbaker> < https://launchpad.net/bugs/767420 >
<kim0> Starting meeting in 1 minute in #ubuntu-cloud : jimbaker hazmat bcsaller niemeyer 
<niemeyer> kim0: Ack
<jimbaker> kim0, over there now
<kim0> yeah the minute just flipped .. let's do it
<_mup_> Bug #767426 was filed: Modify the canonical WordPress formula to demonstrate exposing <Ensemble:New for jimbaker> < https://launchpad.net/bugs/767426 >
<niemeyer> koolhead17: So, welcome
<koolhead17> niemeyer: am still a n00b in coding
 * koolhead17 pokes TeTeT :)
<niemeyer> koolhead17: That's fine
<niemeyer> koolhead17: No one is born knowing it :-)
<koolhead17> :D
<niemeyer> koolhead17: But you'll need quite a bit of patience to push through the basics
<niemeyer> koolhead17: "bzr branch lp:ensemble" will get you the project source
<koolhead17> niemeyer: cool
<TeTeT> hi koolhead17 
 * koolhead17 installs virtualbox
<niemeyer> koolhead17: You'll also need lp:txaws
<hazmat> documentation and other non coding contributions are welcome as well..
<hazmat> and an amazon ec2/aws account
<niemeyer> koolhead17: kim0 can also give you a hand to get your first working environment, since I think he's doing exactly that right now
<niemeyer> kim0: Right?
<koolhead17> hazmat: i will jump to the documentation work soon
<niemeyer> koolhead17: Once you have that in place, you have a world to poke around :0
<niemeyer> :)
<TeTeT> was the s3 problem fixed in txaws?
<koolhead17> nehehe
<kim0> koolhead17: feel free to shoot any questions or problems at me :)
<niemeyer> koolhead17: You can start developing some custom formulas to get a feeling, try to get some small sized improvements going, etc
<kim0> I have a working env already .. but yeah no problem
<niemeyer> kim0: Yeah, the idea was just that if you might be able to easily guide him through the basics of setting up  an environment since you've done that not too long ago
<koolhead17> but i need multi system setup to test my deployment. isn`t it
<koolhead17> ?
<kim0> koolhead17: you probably want to deploy the example wordpress formula .. feel the magic .. love it .. then start writing a new formula .. wow
<niemeyer> kim0: I'll likely skip a lot of steps
<hazmat> TeTeT, niemeyer was working on it re s3 bug
<niemeyer> TeTeT, hazmat: Yes, it's fixed
<hazmat> niemeyer, great!
<TeTeT> awesome :)
<kim0> koolhead17: ensemble basically uses ec2 only for now .. so you don't need any physical machines
<koolhead17> i don`t have any account on ec2
<koolhead17> but i can borrow one from a friend of mine
<kim0> ec2 offers a free account (free tier)
<kim0> you just need to register with a credit card though
<kim0> after that the fun begins :)
<hazmat> niemeyer, how often do the docs update.. i moved the upgrade docs a while ago.
<niemeyer> hazmat: Every 5 minutes
<niemeyer> hazmat: Must be something wrong, checking
<koolhead17> kim0: i hate cc
<koolhead17> am scared of it
<kim0> koolhead17: don't be :)
<koolhead17> kim0: will only use a CC if someone gifts one to me :P
<kim0> hehe you wish
<kim0> it won't really be charged anything (unless you screw up somehow ) so dont worry too much
<koolhead17> is there a apt command which tells you about its depandency with other packages?
<niemeyer> hazmat: Just looking for phones
<niemeyer> Okay, I'm up
<hazmat> koolhead17, apt-cache rdepends will show you reverse deps
<hazmat> apt-cache show will show you deps
<koolhead17> hazmat: cool
<kim0> koolhead17: but since you'd run ensemble from trunk .. it's not too useful :)
<koolhead17> kim0: o.O
<koolhead17> o.0
<kim0> koolhead17: try,    sudo apt-get install python-zookeeper python-txaws
<kim0> those are what I needed
<kim0> koolhead17: there was some python egg too .. can't remember its name
<koolhead17> am on lucid BTW :P
<kim0> lucid .. the land of the ancient
<TeTeT> the land of the corp services team :)
<kim0> hehe
<koolhead17> kim0: am in love with LTS
<koolhead17> until few night my laptop was running on 8.04
<kim0> if you're interesting in running bleeding edge non released software .. lucid is not gonna be too happy
<koolhead17> although netbook has latest one
<koolhead17> :P
<kim0> koolhead17: natty VM then
<kim0> can't wait for natty to be released to upgrade to oneirirc  :)
<koolhead17> kim0: i will install virtualbox cos my hw is not supported. 
<koolhead17> kim0: i had a question i asked the same
<kim0> vbox will work great
<kim0> since all the heavy lifting happens in ec2 .. it should be ok
<koolhead17> why /etc/lsb-release does not mention natty beta2
<koolhead17> it will be a good feature to add
<koolhead17> we just know its development branch or sumthing there
<kim0> just says "development branch" for me ..
<koolhead17> kim0: exactly
<kim0> well it says natty .. so you know it's natty
<_mup_> ensemble/hook-context-uses-yaml-state r203 committed by kapil.thangavelu@canonical.com
<_mup_> hook context using yaml state
<niemeyer> hazmat, bcsaller: Not hearing anything?
<koolhead17> kim0: i installed 3 times today to test the documentation :P
<koolhead17> beta2
<koolhead17> kim0: can i put this in wishlist sumwer?
<hazmat> internet fail
<koolhead17> hazmat: where are you ?
<koolhead17> mine would be worst than most of you guys
<hazmat> koolhead17, washington, dc
<koolhead17> how come it fail there :P
<kim0> hehe
<koolhead17> kim0: i am happy i put my point about beta namings in ubuntu-server :)
<koolhead17> hazmat: where are these launchpad servers hosted? As in data centre? Am i the only one who finds it very slow
<hazmat> koolhead17, london for the most part
<hazmat> koolhead17, where are you based?
<koolhead17> hazmat: india
<hazmat> quite a time zone difference.
<hazmat> koolhead17, they've been working on making them better for everyone, there's some ongoing work on making it regionally better in asia/china region.
<koolhead17> hazmat: i am an insomniac i suppose. :) 
<hazmat> bcsaller, fwiw my yaml-state hook context integration is at lp:~hazmat/ensemble/hook-context-uses-yaml-state
<bcsaller> hazmat: thanks, found it :)
<hazmat> niemeyer, +1 on the log branch
<jimbaker> niemeyer, it seems to me that it would make more sense to use YAML to store the serialization of open port data instead of JSON, especially given the yaml state work
<jimbaker> i guess we could do the equiv for json serialization however
<hazmat> yeah.. would be trivial
<hazmat> abstracting the serialization
<jimbaker> hazmat, actually it may not matter anyway - the current json serialization presumably will require managing conflict below the top key level - that is, the list of port/protocol dicts
<jimbaker> which presumably should be treated as a set for this purpose
<hazmat> jimbaker, indeed
<jimbaker> the current specified format is: {"open": [{"port": 80, "proto": "tcp"}, ...]}
<hazmat> different resolution, would be nice if we could get it for free everywhere, by handling nested data structures for merging, but that's probably too much for the common use case.
<jimbaker> hazmat, yeah, we should probably build a separate solution for now, then see if there's available refactoring
<jimbaker> that's reasonably clean
<jimbaker> the fact that it should be treated as a set is already suggesting some parts not so common
<jimbaker> (of course if we changed the format to be a multi-level dict, with a standard set to dict mapping... anyway, i think the point still stands)
<_mup_> ensemble/resolved-state-api r190 committed by kapil.thangavelu@canonical.com
<_mup_> additional state api tests for resolved.
<niemeyer> jimbaker: Yeah, it'd be fine to have it as YAML too
<niemeyer> jimbaker: The original intention was to have no YAML inside ZooKeeper
<niemeyer> jimbaker: For some reason we forgot about that goal
<niemeyer> jimbaker: So at this point it might even make more sense to have it as YAML than JSON, for consistency
<jimbaker> niemeyer, agreed on being consistent at this point
<_mup_> ensemble/resolved-state-api r191 committed by kapil.thangavelu@canonical.com
<_mup_> add watch apis for unit and unit relation resolved
<_mup_> ensemble/resolved-state-api r192 committed by kapil.thangavelu@canonical.com
<_mup_> more resolved watcher state api tests
<_mup_> ensemble/resolved-state-api r193 committed by kapil.thangavelu@canonical.com
<_mup_> do a final poke in slow callback tests to flush pending watch calls, loops more reliably.
<_mup_> Bug #767762 was filed: Need a unit state api for get/set/del/watch resolved on units and unit relations <Ensemble:New> < https://launchpad.net/bugs/767762 >
<_mup_> ensemble/ensemble-resolved r246 committed by kapil.thangavelu@canonical.com
<_mup_> ensemble resolved cli tests
#ubuntu-ensemble 2011-04-21
<_mup_> ensemble/ensemble-resolved r247 committed by kapil.thangavelu@canonical.com
<_mup_> more resolved cli tests
<_mup_> ensemble/resolved-state-api r194 committed by kapil.thangavelu@canonical.com
<_mup_> use new dict merging utility func, and allow non conflicting updates to relation resolved settings.
<_mup_> ensemble/ensemble-resolved r249 committed by kapil.thangavelu@canonical.com
<_mup_> complete additional tests for resolved cli
<_mup_> Bug #767948 was filed: ensemble resolved command is needed to recover from errors <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/767948 >
<_mup_> Bug #767955 was filed: cli needs beautifcation <Ensemble:New> < https://launchpad.net/bugs/767955 >
<_mup_> ensemble/trunk-merge r187 committed by kapil.thangavelu@canonical.com
<_mup_> add a resolved spec
<_mup_> ensemble/resolved-spec r187 committed by kapil.thangavelu@canonical.com
<_mup_> add resolved spec
<_mup_> Bug #767961 was filed: unit agent needs support for resolving units and unit relations <Ensemble:New for hazmat> < https://launchpad.net/bugs/767961 >
<_mup_> Bug #767964 was filed: ensemble resolved spec/user doc for working with errors <Ensemble:New> < https://launchpad.net/bugs/767964 >
<_mup_> ensemble/refactor-to-yamlstate r196 committed by bcsaller@gmail.com
<_mup_> Hook context's relation settings -> YAMLState
<_mup_> ensemble/config-get r200 committed by bcsaller@gmail.com
<_mup_> YAMLState changes
<_mup_> ensemble/trunk-merge r187 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/resolved-spec r189 committed by kapil.thangavelu@canonical.com
<_mup_> additional doc text regarding --retry
<kim0> hey folks o/
<kim0> I'm trying to launch ensemble now, and every time the ec2 instance fails to boot (becomes terminated) ?!
<kim0> I think a new natty image was pushed yesterday right ? it seems to have problems
<kim0> hazmat: you mentioned the natty image, right
<hazmat> kim0, sorry its not committed
<hazmat> i got caught up in working on ensemble resolved
<hazmat> looking at it now
<hazmat> kim0, which makes your problems more interesting
<hazmat> kim0, actually there is an ec2 problem in the default region
<hazmat> kim0, they where having problems both with ebs and ec2
<hazmat> kim0, for http://www.reddit.com/r/golang/comments/gdn3d/express_go_jit_implementation_of_go/
<hazmat> as an example
<hazmat> http://status.aws.amazon.com/
<hazmat> you can change the default region in the environments.yaml under the provider
<kim0> hazmat: oh thanks
<_mup_> Bug #768311 was filed: txaws deprecation warnings from elementree with ensemble usage <Ensemble:New> < https://launchpad.net/bugs/768311 >
<_mup_> ensemble/unit-agent-resolved r251 committed by kapil.thangavelu@canonical.com
<_mup_> unit and unit relation resolved watch callbacks.
<hazmat> hmm.. lame the txaws only supports two regions
<hazmat> EU and US 
<kim0> yeah 
<kim0> still failing though
<kim0> wonder if  us-east = North vigina
<hazmat> kim0, it does
<hazmat> kim0, hmm. the other problem is we only have images published in that region..
<hazmat> ec2-uri: https://ec2.us-west-1.amazonaws.com
<hazmat> that works as a syntax to talk to other regions but since we're not multi publishing them yet.. its kinda of moot
<kim0> yeah .. so I'll wait a bit I guess
<hazmat> hmmm
<kim0> so much for cloud :)
<hazmat> i can publish an image in another region, but we don't select internally based on region atm
<hazmat> ie. it would need a specification of the image id as well
<_mup_> Bug #768320 was filed: ensemble should support more ec2 regions <Ensemble:New> < https://launchpad.net/bugs/768320 >
<hazmat> kim0, nice article about the other sites down as a result ... http://eu.techcrunch.com/2011/04/21/amazon-ec2-goes-down-taking-with-it-reddit-foursquare-and-quora/
<kim0> 4 hours and the amazon folks are still scrambling .. it's a shame
<hazmat> hmmm.. can't cut new images most of the daily amis  aren't resolvable correctly.
<SpamapS> Have we considered taking over txaws?
<SpamapS> Or at least ... inserting ensemble as a giant influence in its development?
<hazmat> SpamapS, we haven't made the time for it, its really there for anyone who wants it
<hazmat> jamu's been doing some work on it recently and going through old bug reports
<SpamapS> Thats good
<SpamapS> maybe a release soon?
 * SpamapS feels like a broken record
<jimbaker> SpamapS, agreed that having a txaws release would be a good thing
<SpamapS> In Canonical, if you don't have a cadence, you just don't release. ;)
<jimbaker> :)
<hazmat> i've got ensemble images in a few more regions, almost done getting region support working in ensemble
<SpamapS> at this point txaws is a blocker for a) working on natty w/o warnings (my bad for silently fixing that on my box and not patching the package.. oops!), b) working w/ eucalyptus and openstack ... anything else?
<hazmat> SpamapS, b) has been fixed in txaws
<hazmat> the warning thing is being worked on
<kim0> did anyone test ensemble against openstack yet 
<hazmat> kim0, just uec afaik
<SpamapS> hazmat: is that fix released?
<kim0> hazmat: and uec works 
<kim0> ?
<SpamapS> chuck and I tried it against openstack
<SpamapS> same fail as eucalyptus
<kim0> ah so both fail
<SpamapS> s3
<SpamapS> It sounds like there's a patch
<kim0> how hard is it to fix that
<hazmat> SpamapS, no release, its committed to trunk
<SpamapS> hazmat: good enough for me, lets get that into the package. :)
<hazmat> kim0, the issues you where having with txaws are also fixed.
<kim0> hazmat: the logging ?
<hazmat> kim0, no.. the initial setup, where we had to do a non head revision of txaws to get things working.
<kim0> ah ok
<hazmat> kim0, there are branches in review from gustavo on both the logging issues
 * kim0 nods
<TeTeT> so there will be a packaged release of ensemble soon?
<kim0> hazmat: just to be clear, you're saying with txaws trunk, we can deploy ensemble versus uec/openstack sucessfully ?
<hazmat> TeTeT, i think we're going to have a daily ppa. if i remember the plan correctly, and then universe hopefully in 11.10
<_mup_> Bug #768424 was filed: Setup a launchpad daily ppa recipe for ensemble and some deps <Ensemble:New> < https://launchpad.net/bugs/768424 >
<SpamapS> TeTeT: ppa:ensemble/ppa exists now :)
<SpamapS> TeTeT: for lucid and natty
<TeTeT> SpamapS: wow, it works on lucid too?
<TeTeT> great, just installed on a vm, now trying to connect it to the training UEC :)
<TeTeT> how would I configure environment.yaml for my UEC cloud?
<hazmat> TeTeT, ec2_uri and s3_uri
<SpamapS> TeTeT: that won't work until we update txaws in the PPA
<hazmat> but i don't think txaws is in that ppa, of which something recent from trunk is needed.
<SpamapS> maybe we should make that happen.. :)
<hazmat> SpamapS, is that a daily build recipe ppa?
<SpamapS> hazmat: definitely not .. its the one I've tested.
<TeTeT> SpamapS: ah, ok, will postpone the test then
<SpamapS> hazmat: I think an  ensemble/daily would be the place for daily builds
<SpamapS> hazmat: basically the one in the PPA is the demo-able version
<hazmat> SpamapS, sounds good
<hazmat> SpamapS, i'll take a look at the bzr builder stuff after i get the region portability working
<hazmat> hmm. i can launch ensemble in another zone.. but deploying formulas seems to be more problematic.
<hazmat> ah.. the provisioning agent
<koolhead17> hi all
<_mup_> ensemble/ensemble-alternate-regions r205 committed by kapil.thangavelu@canonical.com
<_mup_> region portability
<hazmat> hi koolhead17 
<kim0> koolhead17: o/
<koolhead17> how are things. hazmat amazon had some major issue today
<koolhead17> hi kim0
<kim0> they're still down :/
<koolhead17> yes
<koolhead17> :)
<hazmat> koolhead17, we've  been working on a portability.. we had some code assumptions/dep issues with using alternate regions.
<hazmat> i'm still working on it, i've got it bootstrapping in europe and us-west-1
<hazmat> but the provisioning agent still has thing to work out
<koolhead17> k
<koolhead17> kim0: point me to your blog which you suggested me to start reading from please
<hazmat> internally ensemble is composed of a set of agents, running on a set of machines  in a cloud environment. the provisioning agent is what's responsible for launching new machines for deployed services to live on.. bootstrap means getting a single machine in the environment running at least some of the core agents
<koolhead17> ok
<koolhead17> hazmat: i will focus today and read up on the documentation. i have got a amazon linux machine access from my friend
<koolhead17> i hope that is what you guys meant by EC2
<hazmat> koolhead17, ensemble really needs it own user provided ec2 account, its launching new machines for deployed services
<kim0> koolhead17: you might enjoy http://foss-boss.blogspot.com/2010/10/pointnclick-guide-to-running-ubuntu-in.html
<kim0> to get a basic understanding of ec2
<koolhead17> ok nice
<hazmat> we'll be looking at lxc support for 11.10 to allow people to run ensemble just on their own laptop/desktop hardware
<kim0> you'll also find other cool stuff there :)
<koolhead17> :)
<kim0> hazmat: that would rock :)
<kim0> hazmat: actually openstack now has lxc support ..
<koolhead17> hazmat: okey
<kim0> so hopefully it won't be too much changes for ensemble
<hazmat> kim0, it does, i'm not sure how much its been tested/used though
<hazmat> hopefully its good.. there's a lot of libvirt lxc work the last few months
<koolhead17> hazmat: kim0 apology but i need to know about the lxe full form
<hazmat> koolhead17, what of it? we'll have some attendance at a sprint on lxc in ubuntu with some others in august i think, we'd like to see lxc both within ec2 machines for isolation, and as a machine provider, the latter for desktop support.
<koolhead17> hazmat: what is lxe ?
<hazmat> we've been looking at using libvirt integration, but at the moment raw lxc is a bit easier. openstack which uses libvirt recently added lxc support so that's worth investigating.
<hazmat> koolhead17, http://lxc.sf.net
<koolhead17> thanks
<koolhead17> kim0: we have used libvert only on openstack i suppose
<hazmat> koolhead17, its like a bsd jail or solaris zone.. its an os level containerization of a set of processes such that isolation and resource limits can be put on the process set.
<koolhead17> like Cgroups if am not wrong?
<hazmat> koolhead17, libvirt sits on top of a number drivers for virtualization as an abstraction api, lxc is one of those drivers.
<hazmat> koolhead17, its based on cgroups yes
<koolhead17> okey cool.
<hazmat> cgroups and clone syscall for the namespace isolation  
<koolhead17> waoo.
<hazmat> SpamapS, jamu fixed the txaws warnings on trunk fwiw
<SpamapS> hazmat: yeah seems like txaws trunk is *t3h awesomeness*
<SpamapS> :)
<kim0> any easy way to upgrade to that yet
 * kim0 imagining ec2 engineers pulling out their hair now
<hazmat> kim0, not outside bzr pull | bzr up atm, we can address as part of bug 768424
<_mup_> Bug #768424: Setup a launchpad daily ppa recipe for ensemble and some deps <Ensemble:New> < https://launchpad.net/bugs/768424 >
<koolhead17> kim0: does moinmoin gives you permission to import raw html? Am trying to be little optimist :P
<kim0> I suppose not
<koolhead17> kim0: that angle i would love to explore latex2html will be our key :P
<kim0> koolhead17: a solution might be to write the docs in reST, from which sphinx can generated html for you, PDFs and moinmoin (it seems) can read reST directly (needs testing though)
<kim0> generally however, writing in reST sounds like a good plan to me rather than anything else
<hazmat> support for custom branches on deploy seems to be borked
<koolhead17> kim0: let me think of another angle as well
<koolhead17> :)
<kim0> hazmat: oh I was planning on using gustavo's log fixes branch :)
<jimbaker> moinmoin works well with rst, the only challenge is using sphinx directives not supported by it.  but that's usually pretty minor
<kim0> jimbaker: do you know if the version of wiki.ubuntu.com does 
<hazmat> kim0, it looks like bzr is requiring an lp username and key for the remote machines.
<hazmat> not sure why.. its a public branch
<hazmat> kim0, that's in process of being updated
<hazmat> from ancient 1.6 to 1.9 i think
<jimbaker> kim0, i have only used python.org... but it's a simple plugin for moinmoin and a magic header to specify
<kim0> great
<jimbaker> probably plugin is installed on moinmoin
<kim0> jimbaker: koolhead17 is writing some openstack docs, that needs to be available in html/PDF/moinmoin
<kim0> were not sure what's the best source format
<jimbaker> http://moinmo.in/HelpOnParsers/ReStructuredText
<kim0> it seems reST is a good choice for now
<kim0> just hope wiki.u.c is updated soonish :)
<jimbaker> yeah, i definitely will always recommend rst
<jimbaker> i write everything in it technical that's not code
<kim0> koolhead17: ^ me too :)
<hazmat> and sometimes not even technical ;-)
<jimbaker> kim0, see jythonbook.com, which i coauthored
<jimbaker> we actually produced microsoft word docs (ok open office, but for our publisher, it was ms word) and html from the sames source rst
<jimbaker> ftw
<jimbaker> the other thing i like is rst2pdf. it doesn't support the sphinx directives either, but it's great for precise production of pdfs. i use it for making presentations
<koolhead17> rst2pdf is well enough
<jimbaker> bcsaller, it seems to me that i need to augment HookContext to support open-port/close-port hook commands. in particular, this seems the right place to do it so that flush will properly commit changes
<jimbaker> (in the future, we can take advantage of ZK multinode support too, so that the flush writes all or none of the nodes - https://issues.apache.org/jira/browse/ZOOKEEPER-965
<hazmat> jimbaker, you'll need to use a separate yaml state, and incorporate it into flush
<hazmat> your probably going to end up conflicting with bcsaller's work on service config
<hazmat> there both exposing new hook apis
<bcsaller> I'm thinking that like status HookContext is growing deps on too many interfaces and we'll have to refactor how that works in the future
<jimbaker> hazmat, absolutely because it's on the service unit. not relation
<hazmat> bcsaller, agreed
<bcsaller> HookContext is becoming a kind of dumping ground
<hazmat> bcsaller, well its going to become a facade over many different facilities
<jimbaker> hazmat, that's one reason i'm pinging bcsaller, it seems that his service config stuff must :)
<hazmat> as long as it can manage them all isomerically then hopefully its not too bad.
<hazmat> else we can refactor more
<jimbaker> maybe we can have some sort of plugin mechanism with hookcontext?
<jimbaker> whatever that means
<bcsaller> might be a better way to compose that object though, right now each will have to change flush for example
<jimbaker> we do want one thing to call flush on
<jimbaker> and have an associated client id
<bcsaller> where as we could have a set of strategy objects which get iterated over in said calls
<hazmat> the client id has been a constant string for a while ;-)
<jimbaker> bcsaller, yes, that's what i wanted to call it :)
<jimbaker> hazmat, i did not know that
<hazmat> i think it can wait, but yeah.. it would be simple to do a get/set/del across method names of an abstraction object
<jimbaker> here i am thinking it's supposed to do something
<bcsaller> hazmat: if the debug stuff didn't need it to connect to existing sessions then its kind of a dead horse 
<hazmat> er. strategy plugins.. but at them moment it seems like more trouble than its worth.. somehow its going to conflict whereever it gets configured
<jimbaker> hazmat, but these are touching independent pieces
<jimbaker> we just want to make sure that they can be accessed together, flushed together
<hazmat> jimbaker, to the hook context, are they all just a set of yaml states and methods manipulating them?
<hazmat> ^aren't 
<jimbaker> hazmat, i think they are indeed
<bcsaller> hazmat: agreed for now, but I think the change is simple in the future and it will help keep the various interfaces different but composable 
<hazmat> sounds good, you guys up for a standup bcsaller, jimbaker ?
<bcsaller> yeah
<jimbaker> hazmat, sounds good
<jimbaker> { "port/proto" : True, "port/proto": <other info>...}
<jimbaker> **/units/<internal unit id>/ports**
<jimbaker> { "open": [{"port": 80, "proto": "tcp"}, ...]}
<SpamapS> port isn't required right? there are portless protocols.. ;)
<hazmat> cool working in other regions now
<hazmat> SpamapS, it can be a port range
<hazmat> SpamapS, there's some critical mass of args for modifying accesss on a security group ;-)
<hazmat> which are different depending on your intent/args
<_mup_> ensemble/ensemble-alternate-regions r206 committed by kapil.thangavelu@canonical.com
<_mup_> remove some of the debug helpers
<_mup_> ensemble/ensemble-alternate-regions r207 committed by kapil.thangavelu@canonical.com
<_mup_> verify new ec2 environment schema for regions.
<_mup_> ensemble/ensemble-alternate-regions r208 committed by kapil.thangavelu@canonical.com
<_mup_> remove region alias compatibility with txaws (EU/US) in favor of partial domain names (us-east-1, ap-northeast-1, etc)
<koolhead17> TeTeT: hey
<_mup_> ensemble/ensemble-alternate-regions r209 committed by kapil.thangavelu@canonical.com
<_mup_> failure to find an ssh-key is not fatal to an ec2 environment serialization...
<TeTeT> hi koolhead17 
<koolhead17> TeTeT: Working on the Ubuntu Enterprise Cloud Course :)
<koolhead17> saw your tweet. am curious about it ^^
<TeTeT> koolhead17: right now on a sprint and will present uec persistence in a few minutes
<TeTeT> koolhead17: what are you working on cloud wise these days?
<koolhead17> openstack and trying to understand ensemble 
<koolhead17> :)
<koolhead17> kim0: announcing-project-reddwarf-database-as-a-service intersting :)
<hazmat> bcsaller, jimbaker if you guys have time would you mind looking at the ensemble-alternate-region branch..
<kim0> koolhead17: yep
<hazmat> i'd like to have ensemble working before ec2 is ;-)
<kim0> hazmat: can I play with the branch now on a different ec2 region :)
<koolhead17> hazmat: :P
<hazmat> kim0, yes
<hazmat> kim0, i had forget something basic i needed to do to make that work
<hazmat> like push the  branch to lp
<kim0> hehee :)
<hazmat> well actually i had a name typo.. but same difference ;-)
<koolhead17> hazmat: what prompted the project name ensemble
<kim0> koolhead17: you can direct easy questions at me :)
<hazmat> koolhead17, roots are in group communication systems and paxos, where ensemble is term used to refer to the working set of intercommunicating servers
<hazmat> but that definiteion may change latter ;-)
<kim0> ok that's a sophisticated answer
<kim0> hazmat: I think that should go into a faq
<hazmat> i think gustavo would be a better person to ask if you want a good origin name story
<koolhead17> kim0: hazmat i met a guy this every sweet fellow asking me do u work on Ubunduu
<hazmat> kim0, true
<koolhead17> he pronounced it so differently
<koolhead17> :P
<kim0> Can anyone think of two other questions to put in a faq :)
<kim0> hazmat: so once you push the branch let me know how to play with it (I hope that doesn't conflict with me wanting the instances running ensemble-log fixes branch)
<hazmat> kim0, you mean regarding the alternate region branch?
<TeTeT> kim0: what's the difference between ensemble and <insert favorite config mgmt system>
<kim0> yep
<TeTeT> kim0: when will it be ready for production
<TeTeT> kim0: how to contribute
<hazmat> kim0, if you branch gustavo's and merge mine you should be able to run in a different region, i don't have images up for asia yet, but europe and us (x2) are covered
<hazmat> shouldn't be any conflicts
<kim0> TeTeT: man you're good :)
<TeTeT> kim0: what system do I need to control ensemble from?
<TeTeT> kim0: he he, doing my best, no matter where
 * kim0 starting a faq doc
<koolhead17> kim0: there
<jimbaker> hazmat, i need a txaws branch for your branch to work, right?
<jimbaker> presumably, since it's failing in txaws trunk...
<jimbaker> bcsaller, are you seeing this problem too? http://pastebin.ubuntu.com/597140/
<bcsaller> jimbaker: I haven't tried to run on ec2 in a while
<bcsaller> and haven't seen that 
<jimbaker> bcsaller, just trying out hazmat's new branch for alternative regions and it's not working for me
<hazmat>  jimbaker you also need to specify it as the remote branch
<hazmat> ensemble-branch in the provider
<jimbaker> hazmat, yes, of course :)
<hazmat> the provisioning agent also relies on fixes in the branch
<hazmat> jimbaker, it works with txaws trunk
<jimbaker> all makes sense now... except for the mysterious failure part
<hazmat> jimbaker, there's all kind of wierdness without the provisioning agent support.. we get partially initialized machine states, that status tends to barf on
<hazmat> which arguably it should do better with
 * jimbaker needs to do annual checklist next time... was going to alter environments.yaml, but got left out
<jimbaker> still good experience for yet another bug report :)
<jimbaker> manual, manual
<jimbaker> rebooting really did a number on my mind it seems
<_mup_> Bug #768595 was filed: Status should handle partially initialized machine states, ie. before they been handled by the provisioning agent <Ensemble:New> < https://launchpad.net/bugs/768595 >
<jimbaker> hazmat, where should we report version discrepancies. might be nice if ensemble status did the check, since bootstrap seems too early
<hazmat> jimbaker, we've got an open bug that to record bzr revno/branch info into zk
<hazmat> probably could add it to some sort of common  cli check under verbose mode
<jimbaker> hazmat, yeah. so if we reported that in ensemble status, that could be a good place
<hazmat> jimbaker, i'm getting a little concerned about  how much back and forth traffic ensemble status is doing..
<hazmat> it be nice if we could avoid pulling down the entire zk tree
<hazmat> to do a status listing
<hazmat> and feels like we're going in that direction, i'd rather keep the default and let be do filtering and additional display options to target something than have the default be slow
<jimbaker> sure, but it does seem something that can be filtered on. but dev time is useful too
<hazmat> its a very useful introspection tool
<hazmat> and it would be nice to expand to do targeted display of relations values, ensemble versions, etc.
<jimbaker> a message highlighting "are you really sure" when running two different branches would be valuable
<hazmat> jimbaker, how often do you use the ensemble-branch feature
<jimbaker> hazmat, what's worse is when i leave it turned on :)
<hazmat> most of the time we're i'm just running trunk remotely
<hazmat> jimbaker, true that
<hazmat> that's bit me more than once
<hazmat> we can play with it
<hazmat> i guess its only once to look at the bootstrap node
<jimbaker> which is required by all entry points anyway
<jimbaker> hazmat, your branch looks good. i can't currently run the examples formulas as-is, but that's waiting on the logging changes
<jimbaker> i will try it one more time w/ the older versions of the example formulas
<jimbaker> what's currently on http://ec2-184-72-28-229.us-west-1.compute.amazonaws.com/ demonstrates why the open-port setting needs to be done in the flush, ideally w/ all-or-nothing semantics
<jimbaker> for future ref: http://pastebin.ubuntu.com/597147/
<jimbaker> "it works!" yeah, maybe it's fine, maybe it's really in a bad and unsecured state
<hazmat> jimbaker, thats not the default incidentally, one of the formulas installed apache though
<hazmat> server comes with nothing running except ssh by default as far services
<hazmat> fwiw
<jimbaker> hazmat, sure, we just see a partial configuration
<jimbaker> hazmat, i retried with the example formulas in r200 of trunk (the last time i had run it), and i'm still not able to deploy against us-west-1
<jimbaker> hazmat, i have seen this recur in a couple of attempted deploys - http://pastebin.ubuntu.com/597178/
<jimbaker> it looks like the service relation is not being added, or something related to its workflow, based on what's missing from ensemble status
<jimbaker> i really should look at this more directly in ZK, not certain what's going on here
#ubuntu-ensemble 2011-04-22
<jimbaker> hazmat, this looks provocative. the last line in the formula.log for wordpress/0 - 2011-04-21 23:30:08,060: twisted@ERROR: TypeError: 'NoneType' object is not callable
<hazmat> jimbaker, hmm
<jimbaker> so maybe the relation workflows are running, but they are going into a bad state?
<hazmat> jimbaker, possibly
<hazmat> surprised we don't have anything nicer in the traceback. that's unfortunate
<hazmat> jimbaker, the status looks right
<jimbaker> hazmat, no, it's missing the relation status info
<hazmat> jimbaker, one option is to  have a look at the zkshell.. ensemble ssh 0 && /usr/share/zookeeper/bin/zkCli.sh
<hazmat> jimbaker, hmmm..
<hazmat> jimbaker, i'll have a look in the morning
<jimbaker> compare it against this output: http://pastebin.ubuntu.com/597192/
<jimbaker> (from an earlier run)
<jimbaker> hazmat, sounds good
<jimbaker> hazmat, for later consumption - shouldn't this be more than two? zk: localhost:2181(CONNECTED) 26] ls /units --- [unit-0000000000, unit-0000000001]
<jimbaker> hmmm... maybe not that part after all - i was looking at zk_workflow_identity
<jimbaker> it looks like it uses the same path for both ServiceUnitState and UnitRelationState
<jimbaker> but those are not the same paths
<jimbaker> based on looking at those specific classes
<jimbaker> either i'm confused or ensemble is confused ;)
<hazmat> jimbaker, they are at the same path
<hazmat> all the workflows for a unit-agent are managed on a single node
<jimbaker> hazmat, thanks for the clarification
<jimbaker> this would have caused more issues if it were not the case, i guess
<hazmat> jimbaker np.. sorry i had to run out
<hazmat> jimbaker, yeah.. it seems strange the workflows on the unit are initialized and showing them as running the but the units weren't up or in an error state which seems strange
<hazmat> i can't think of any reason why that would be the case.
<jimbaker> anyway, just curious it's happening in us-west now - one more thing to try is in eu-west (if that's the other region completely set up)
<hazmat> i'll take a look at it tomorrow
<hazmat> jimbaker, it is
<jimbaker> hazmat, have a good night, ttyl
<_mup_> ensemble/refactor-to-yamlstate r197 committed by bcsaller@gmail.com
<_mup_> set not taking any random data, but insisting on a dict in tests (for YAMLState)
<kim0> Morning everyone
<kim0> anyone around o/
<hazmat> kim0, g'morning 
<hazmat> or hello is probably more appropriate
<kim0> hazmat: hey o/
<kim0> team on vacation huh :)
<_mup_> ensemble/merged-alt-region-logging r210 committed by kapil.thangavelu@canonical.com
<_mup_> merge ensemble-log-level
<_mup_> ensemble/merged-alt-region-logging r211 committed by kapil.thangavelu@canonical.com
<_mup_> merge ensemble-log-crash
<jimbaker> kim0, hi
<jimbaker> kim0, we have been trying out the alternative region branch. it's not worked for me with deploying our example formulas. you want to give it a try too?
<jimbaker> it did work w/ kapil however
<kim0> ec2 has recovered it seems .. I'm writing a small user level tutorial now, but if you need someone else to test that branch, sure
<jimbaker> kim0, i think it would be useful for sure
<jimbaker> it's also a good way to play w/ the environments.yaml file
<kim0> jimbaker: cool, any instructions on using that branch ? I'll try it in a few hours though, now right now
<kim0> let me know how do I use it, thanks
<_mup_> Bug #769030 was filed: Enable one control bucket to be used for multiple regions <Ensemble:New> < https://launchpad.net/bugs/769030 >
<jimbaker> you just need to configure two things in your environments.yaml file: region - us-east-1, us-west-1, eu-west-1; and ensemble-branch - https://code.launchpad.net/~hazmat/ensemble/ensemble-alternate-regions 
<kim0> got it
<hazmat> kim0, i also have one with the logging stuff merged in.. lp:~hazmat/ensemble/merged-alt-region-logging
<jimbaker> hazmat, sounds good, then we don't use an earlier formula set
<jimbaker> have to use
<hazmat> jimbaker, it didn't work entirely
<hazmat> jimbaker, i'm trying out trunk in east atm to verify the delta
<jimbaker> hazmat, sounds good, less mystifying then
<jimbaker> we prefer our failures to be consistent ;)
<hazmat> jimbaker, what's odd is that the unit relation state in zk (in west) is good, but status isn't reporting it, and wordpress isn't running
<hazmat> also ensemble-log seems to return 0 even when it fails/errors.
<jimbaker> hazmat, i just got this weird output in a log
<jimbaker> 2011-04-22 09:20:52,519 unit:mysql/0: twisted ERROR: TypeError: 'Port' object is not callable
<hazmat> jimbaker, can you paste the full log to pastebin
<jimbaker> hazmat, will do
<hazmat> thanks
<jimbaker> http://pastebin.ubuntu.com/597493/
<hazmat> jimbaker, is this on the open-port/close-port branches?
<jimbaker> hazmat, no, the alternative region branch, running with trunk r200 formulas
<jimbaker> i'm going to try the new alt region branch w/ logging merge in now
<hazmat> jimbaker, hmm.. there are no port classes in ensemble, only in twisted.
<jimbaker> hazmat, indeed
<jimbaker> are we sure we got good versions of python, twisted, etc built in these amis?
<jimbaker> it's as if we got some version skew going on here
<_mup_> Bug #769035 was filed: Need a top level decorataor on all independent callbacks to do nice error printing <Ensemble:New> < https://launchpad.net/bugs/769035 >
<hazmat> jimbaker, just stock natty
<_mup_> Bug #769036 was filed: Ensemble hook cli api needs to do correct exit codes  <Ensemble:New> < https://launchpad.net/bugs/769036 >
<hazmat> jimbaker, so trunk works for me
<jimbaker> hazmat, trunk formulas, or using trunk with us-east-1?
<hazmat> trunk and us-east-1
<hazmat> trying alt-region with us-east-1
<hazmat> there's nothing in the branch remotely related to units or hooks..
<jimbaker> hazmat, exactly, that's what is so strange here. some other unexpected dependency seemingly
<hazmat> jimbaker, the merged-alt-region-logging branch seems to work okay
<hazmat> in us-east-1
<jimbaker> hazmat, trying it out now
<hazmat> jimbaker, cool, i'm trying it out in a different region and then i do see a problem
<jimbaker> hazmat, doesn't work for me this try. speaking of round trip overhead, now doing "watch ensemble status" ;)
<jimbaker> i had proposed building that in for ensemble status with actual watches, but repeatedly polling like this is the poor man's approach for sure
<jimbaker> and the interesting thing is seeing the relation service state disappear... crazy
<bcsaller>  jimbaker: the plan with status is to have a mode where it blocks on a topo watch and then reissues the status in a loop
<bcsaller> unlike watch it knows when things change
<jimbaker> bcsaller, yes, intelligent watches :)
<jimbaker> bcsaller, good to know it is in the works
<jimbaker> bcsaller, doing "watch ensemble status" is still useful right now
<bcsaller> good
<jimbaker> i think once we have the relation settings added to ensemble status, that's going to be pretty awesome
<hazmat> aha
<hazmat> unit agents are dead
<jimbaker> hazmat, that would make sense
<hazmat> jimbaker, the fact there is nothing in the log is rather frightening
<jimbaker> hazmat, indeed. i'm just about to testing us-east-1 w/ trunk at r200, which is the last good one i observed
<jimbaker> try testing
<hazmat> jimbaker, i was able to get trunk latest from merge-alt-region-logging working on us-east-1
<jimbaker> hazmat, i was unable to get that - i was just getting "it works" plus empty relation service states
<jimbaker> probably because of dead unit agents
<hazmat> jimbaker, no.. i'm actually got populated relation states with dead units agents
<jimbaker> hazmat, crazy
<hazmat> jimbaker, the variations i'm using are trunk, ensemble-alternate-region, merge-alt-region-logging
<hazmat> all with formulas from that are the equivalent of the trunk versions
<hazmat> i've seen trunk and merge-alt-region-logging working on us-east-1
<jimbaker> hazmat, i have tried all of those, both with us-east-1 and us-west-1
<jimbaker> nothing is working end-to-end for me today
<jimbaker> everything starts off fine... then it just mysteriously fails
<jimbaker> hazmat, maybe i should rebuild my buckets, don't know if that's an issue based on bug 769030
<_mup_> Bug #769030: Enable one control bucket to be used for multiple regions <Ensemble:New> < https://launchpad.net/bugs/769030 >
<hazmat> jimbaker, yeah.. i switch my buckets when changing regions atm
<hazmat> they should recover fine
<hazmat> ie. detect dead instance stale file, and create new one
<jimbaker> although given how the control bucket works, i wouldn't expect it to impact
<hazmat> which is what they normally do, or we'd be cleaning it all the time
<jimbaker> bcsaller, hazmat - standup?
<hazmat> jimbaker, sounds good
<bcsaller> I have little to report, but sure
<jimbaker> then it will go fast :)
<_mup_> Bug #769120 was filed: Ensemble status shouldn't report dead units based soley on state, but also on presence. <Ensemble:New> < https://launchpad.net/bugs/769120 >
<hazmat> http://dtrace.org/blogs/bmc/2010/08/30/dtrace-node-js-and-the-robinson-projection/
<hazmat> http://wiki.joyent.com/display/node/Using+Cloud+Analytics
<hazmat> http://dtrace.org/blogs/dap/2011/03/01/welcome-to-cloud-analytics/
<hazmat> allergies miserable.
<jimbaker> hazmat, you really should try hot yoga. i found it really helps clear sinuses and it would seem prevent allergic symptoms too
<hazmat> jimbaker, sadly hot yoga isn't my thing
<hazmat> does anyone understand apport handling of core files?
<hazmat> jimbaker, can you give a hook at this look trivial patch for trunk.. https://pastebin.canonical.com/46611/
<hazmat> not sure if argparse version changed, but i currently have these two tests failing for me on trunk
<hazmat> bcsaller, ^ if you have a moment and could look at the trivial.. i'm waiting on that before doing some merges.
<bcsaller> hazmat: the change to generation happens outside the patch?
<hazmat> bcsaller, yeah.. the error output change cause is not clear, i just matched to what the current production is
<bcsaller> seems like it should have been caught when the change happend and the tests would have broken. 
<bcsaller> I'm fine with the change, but want to understand how it happeded
<bcsaller> happened
<hazmat> bcsaller, yeah.. i'm bisecting the last 5revs now to double check
<hazmat> bcsaller, just went back a month history, still getting the errors, i'd have to guess its an argparse change and rev increment
<bcsaller> maybe, yeah
<bcsaller> thanks for checking
<hazmat> bcsaller, seems to be a change between pypi version of argparse and the builtin 2.7 version
<bcsaller> ahh, natty on 2.7
<bcsaller> makes sense now
<hazmat> not sure if that's it.. also happens with python 2.6 using the distro argparse
<hazmat> but it does work with the pypi argparse 1.2.1
<hazmat> where as the distro version (for 2.6) is 1.1-1
<hazmat> no.. actually it didn't work 1.2.1
<hazmat> i had used a patched trunk to test that one
<_mup_> ensemble/trunk r205 committed by kapil.thangavelu@canonical.com
<_mup_> argparse error output seems to have changed, match tests to match current output [trivial][r=bcsaller]
<_mup_> ensemble/trunk r206 committed by kapil.thangavelu@canonical.com
<_mup_> merge ensemble-log-level [a=niemeyer][r=niemeyer][f=767364]
<_mup_> This fixes a problem with the ensemble-log hook CLI API,
<_mup_> not correctly taking a -lLOG_LEVEL option.
<_mup_> ensemble/trunk r207 committed by kapil.thangavelu@canonical.com
<_mup_> merge ensemble-log-crash [a=niemeyer][r=kapil][f=767391]
<_mup_> This fixes a traceback when attempting to use the ensemble-log
<_mup_> hook CLI API from hooks.
<hazmat> jimbaker, the principia trunk seems to work with trunk, but the trunk formulas don't on natty.
<hazmat> still seeing unit agents die though
<hazmat> hmm
<hazmat> jimbaker, txzookeeper unit tests are segfaulting with default natty it appears.
<hazmat> jimbaker, bcsaller do you run with the package zk or a local build?
<bcsaller> local
<hazmat> yeah.. we've been getting away with not having our own packages
<hazmat> even though there were known issues with the lucid one, it still worked for our uses.
<hazmat> doesn't appear to be the case with natty, we're going to need package trunk (3.4) or backport perhaps
<jimbaker> hazmat, curiously i'm reinstalling stuff now
<jimbaker> is everyone running on python 2.7 at this point?
<hazmat> jimbaker, i am
<hazmat> jimbaker, i test with 2.6 occasionally as well
<jimbaker> i'm now getting some test errors on trunk, just building a new virtualenv to try 2.7 out
<hazmat> jimbaker, what does ./test need in a ZOOKEEPER_PATH ? just the zkServer.sh  script?
<jimbaker> hazmat, iirc, it doesn't use zkServer.sh
<hazmat> hmm
<hazmat> jimbaker, looks like i just need a directory with the jar
<jimbaker> hazmat, sounds about right
<jimbaker> it looks for both dev and prod installs
<jimbaker> which are laid out differently
<hazmat> hmm.. yeah.. it doesn't work just pointing to a directory of jars.. which is what the deb does for install into /usr/share/java
<jimbaker> makes sense
<jimbaker> fortunately easy enough to change in ensemble.tests.common.ManagedZooKeeper
<jimbaker> basically a variant of zkServer.sh
<jimbaker> was that adjusted in the deb package?
<hazmat> jimbaker, just had to fix the test/common get class path to not hardcode src release stuff
<hazmat> jimbaker, debian uses /usr/share/java for java libs
<jimbaker> which was the whole point of that classpath property, so that's cool
#ubuntu-ensemble 2011-04-23
<_mup_> ensemble/config-get r202 committed by bcsaller@gmail.com
<_mup_> remove unused method, tests pass dicts rather than strings to get/set
<_mup_> ensemble/config-get r203 committed by bcsaller@gmail.com
<_mup_> comminucation tests passing
<kim0> hey guys, just pushed a docs branch .. it's lots of text, so your eye balls are highly appreciated :)
<kim0> btw I'm off next week .. longish easter .. have fun everyone
<jimbaker> kim0, enjoy!
<kim0> Thanks :)
<jimbaker> kim0, looking at it now, looks good so far but i will spend some time reviewing it
<kim0> Great!
<jimbaker> kim0, one more thing - will you be at UDS in budapest?
<kim0> It will definitely "grow" a bit with the upgrade/expose stuff ..etc .. just a start for now
<kim0> and yes I should be there
<kim0> woohoo :)
<jimbaker> cool, looking forward to meeting up with you
<kim0> same here
 * kim0 hugs the virtual team
<jimbaker> kim0, thanks, the same :)
<hazmat> kim0, docs look great, thanks, and have a good holiday
<kim0> hazmat: o/ thanks man .. you too
<kim0> I should really sleep now .. it's 2:30am hehe
 * kim0 â /dev/bed
<_mup_> Bug #769286 was filed: Need updated zookeeper packages, else agents segfault <Ensemble:New> < https://launchpad.net/bugs/769286 >
<_mup_> ensemble/expose-ensemble-commands r208 committed by jim.baker@canonical.com
<_mup_> Exposed flag support
