#ubuntu-ensemble 2011-04-04
<kim0> Morning folks
<hazmat> kim0, g'morning
<_mup_> Bug #750193 was filed: unit state api for formula upgrade flag <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/750193 >
<_mup_> ensemble/service-unit-state-upgrade-support r189 committed by kapil.thangavelu@canonical.com
<_mup_> verify set and clear can be used multiple times.
<niemeyer> Good morning!
<niemeyer> Strange.. chat.freenode.net seems to not work on port 7000 for SSL right now
<niemeyer> Had to connect to 7070
<_mup_> ensemble/formula-upgrade-cli r190 committed by kapil.thangavelu@canonical.com
<_mup_> wire in formula upgrade command, use updated upgrade api.
<_mup_> Bug #750304 was filed: formula-upgrade command is needed <Ensemble:New> < https://launchpad.net/bugs/750304 >
<niemeyer> hazmat: Good morning
<hazmat> niemeyer, i've had all kinds of problems with freenode this past week..
<hazmat> niemeyer, seems to require sasl for me
<hazmat> on certain mobile networks, but really none of the clients seem to have good sasl support.. i'm trying a perl script which purports to support it, except deb/ubuntu packaging of xchat prevents it from working :-(
<hazmat> niemeyer, g'morning :-)
<niemeyer> hazmat: It seems to have worked to simply switch to port 7070 somehow
<niemeyer> hazmat: How're things going?
<hazmat> niemeyer, okay.. kinda of slow for me last week, ben and i did some cross reviews on specs which was good, jim had a short week for vacation, i've been doing some hacking on upgrades over the weekend, as i'm going to have a slightly short day today since i'm flying home
<hazmat> niemeyer, the review queue is bursting
<niemeyer> hazmat: Expected.. I'll catch up with it in the next couple of days
<hazmat> niemeyer, also ran into an issue with your suggestion regarding multi-user screen sessions, as it require suid screen support 
<niemeyer> hazmat: Feels great, though.. some good activity has happened
<hazmat> i'd rather just use sudo
<niemeyer> hazmat: Cool, sounds fine
<hazmat> cool
<hazmat> niemeyer, yeah.. overall it seems like we had some good activity, lots of movement on lots of bits
<hazmat> i'm still holding off on merging the formula upgrade spec, till i get a last review from jim, but else its good
<niemeyer> hazmat: Cool
<hazmat> niemeyer, i had some questions regarding service config metadata and getting some additional stuff in there for ui display and validation.. it would be nice to cover in our standup
<niemeyer> hazmat: Have you had a chance to give Jim's branches a look while he was working on them?
<niemeyer> hazmat: Ah, sounds good
<hazmat> niemeyer, i'm also a little wary of the expose spec atm, as noted in review, also nice to cover in standup
<niemeyer> hazmat: Ok
<hazmat> perhaps we can do the standup a little earlier today to cover the extra items
<hazmat> i've got to catch flight out at the tail end of it
<hazmat> assuming it goes an extra hr.
<hazmat> niemeyer, how was the vacation?
<niemeyer> hazmat: Early stand up sounds good
<niemeyer> hazmat: It was amazing indeed
<niemeyer> hazmat: We went to some paradisiac locations
<niemeyer> hazmat: Some natural sea pools
<niemeyer> hazmat: Got the US visa too.. they were waiting for us at home when we came back :)
<niemeyer> 10 years too, which is awesome
<hazmat> niemeyer, nice.. they never called fwiw
<hazmat> niemeyer, i had a look at the some of jim's branches, and offered some input on approach before he started.. i'm happy with them, haven't done full reviews on them yet
<niemeyer> hazmat: The interview was quite straightforward, luckily
<niemeyer> hazmat: Ok, I can do a first pass, and depending on how it goes I may add you for a final review if that's ok
<hazmat> niemeyer, cool.. i imagine they could be stressful.. but at this point.. your a regular
<hazmat> niemeyer, sounds good
<hazmat> niemeyer, i was also thinking we should start in on blueprints usage for ensemble.. it covers our feature development pretty nicely.. ordered sets of bug reports and branches.
<hazmat> only caveat is linking to the feature-spec branch vs. linking to it on trunk.. but i think that's pretty minor
<niemeyer> hazmat: How do you think we might use them?
<hazmat> niemeyer, as artifacts for feature against a release, with links to specs, and relevant bugs against the feature
<hazmat> effectively how they were intended to be use afaik
<niemeyer> hazmat: Sounds good.. the organization would indeed be nice.  Hopefully this won't increase too much the boilerplate we have to go through
<hazmat> niemeyer, true.. if need be we can do some automation against lp.. although what and how isn't clear to me atm having not used blueprints yet, most of the work afaics is just attaching the bug reports. but it would be nice to get into the practice of spec'ing all the bug reports against a feature from the start for the blueprint process
<niemeyer> hazmat: *All* of them is probably too much
<niemeyer> hazmat: There are lots of independent things we do on each cycle
<niemeyer> hazmat: But for big features, would be cool
<_mup_> ensemble/debug-hook-scope-guard r209 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk, resolve conflicts
<hazmat> niemeyer, true.. not saying everything needs one.. but upgrades, expose, service-config, dependency res are the main features for our current cycle that could get blueprints
<jimbaker`> good morning - vacation was great, fantastic skiing with my kids
<hazmat> ie. multi-branch work
<jimbaker`> and now time to catch up :)
<hazmat> jimbaker`, nice
<niemeyer> hazmat: Cool
<niemeyer> jimbaker`: Good morning, and welcome back
<jimbaker`> my daughter did her first waist-deep powder skiing, my son his first double black run
<niemeyer> Hmmm.. must file expense for the last few months
<jimbaker`> niemeyer, same with me... fortunately not too much of a backlog, just some per diem + aws
<niemeyer> What was the perdiem there again?
<niemeyer> per diem
<jimbaker`> Dinner per diem is 150ZAR
<hazmat> we had a few group dinners 
<hazmat> wish shouldn't be counted as they where expensed via corporate card
<niemeyer> Indeed.. how many where them?
<niemeyer> were
<niemeyer> I recall.. 1?
<jimbaker`> i believe it was tues, thur, fri for me
<niemeyer> The corporate dinners?  I only recall the one we drove up to that nice place outside
<niemeyer> Were there any other?
<niemeyer> others
 * niemeyer => lunch
<hazmat> niemeyer_lunch, not that you where at as i recall
<hazmat> i think there was one thursday downtown in the city, and one friday at the winery in addition to the corporate dinner on tuesday
<hazmat> s/downtown/seaside
<hazmat> jimbaker`, do you remember where the dinner on tuesday was?
<hazmat> i'm trying to remember if was the italian place downtown, when clint and i segwayed into the belgium place for nice beer ;-)
<niemeyer_lunch> Okay, I think the expense report is ready
<niemeyer> Ok, I'm done with the receipt delivery marathon.. I'll step out for a moment.
<hazmat> bcsaller, jimbaker` .. i wanted to see if we could start our standup a little bit earlier today, i suspect will go a bit longer, to catch up on items. i was thinking about 30m early. i've got to roll to catch a plane an hr after the normally scheduled standup
<jimbaker`> hazmat, that
<jimbaker`> is fine with me
<_mup_> ensemble/unit-agent-formula-upgrade r190 committed by kapil.thangavelu@canonical.com
<_mup_> unit agent formula upgrade support
<bcsaller> hazmat: thats fine with me
<hazmat> i'm wondering if after an upgrade from a pre-existing error state, we should automatically attempt a retry to proceed to the final state the error was as a result of trying to reach
<_mup_> Bug #750483 was filed: unit agent needs to support upgrades <Ensemble:New> < https://launchpad.net/bugs/750483 >
<jimbaker`> bcsaller, hazmat - standup?
<bcsaller> I'm ready
<jimbaker`> looks like niemeyer is not here, still having trouble i guess connecting to freenode
<hazmat> jimbaker`, yeah.. i'm talking with on canonical irc
<hazmat> jimbaker`, are you online re skype?
<jimbaker`> hazmat, indeed
<jimbaker`> hazmat, yes on skype
<niemeyer> relation-set port=[1,2,3]
<niemeyer> relation-get port
<niemeyer> => "[1,2,3]"
<niemeyer> => "1 2 3"
<niemeyer> config-get my-setting
<niemeyer> for a in $VALUE
<niemeyer> config-get something
<niemeyer> ensemble set key=value
<hazmat> ensemble set < foobar-config.yaml
<niemeyer> ensemble set user=a pwd=b
<hazmat> niemeyer, network/skype?
<niemeyer_> Oh, cheers everyone else I guess.. he was the host :-)
<niemeyer> Oh my :(
<niemeyer> Network is super flaky here
<bcsaller> :(
<niemeyer> So, where do we find details for ugprading to natty
 * niemeyer wondering if at this point upgrade-manager is fine by itself
<niemeyer> https://help.ubuntu.com/community/NattyUpgrades
<bcsaller> upgrade-manager worked for me. 
<niemeyer> http://www.ubuntu.com/testing/natty/beta
<niemeyer> bcsaller: Are you on natty already?
<bcsaller> yeah, as of last week
<niemeyer> bcsaller: Ah, sweet
<niemeyer> bcsaller: How's it going?
<bcsaller> when the beta came out, when ever that was
<bcsaller> I'm not using unity so very little has changed. Some packages were misbehaving but its mostly sorted now
<bcsaller> I guess the biggest dev change is the python 2.7 switch, but even with that very little changed, I only needed to pull in two packages before everything was working again
<niemeyer> Oh my
<niemeyer> (resending) bcsaller: Neat, so does that mean Ensemble is pretty much cool with 2.7?
<bcsaller> seems fine, yeah
<bcsaller> tests run w/o issue for me
 * hazmat digs free airport wifi
<hazmat> bcsaller, you've been running natty?
<bcsaller> just for a few days, but yeah
<hazmat> bcsaller, any problems?
<bcsaller> hazmat: nothing serious, when I first upgraded compiz was crashing a bit, but it seems resolved now
<hazmat> looks like the landscape team just picked up 3 people
<hazmat> bcsaller, cool... i'll give it a whirl this evening
<bcsaller> hazmat: not using Unity either though
<hazmat> yeah.. i might give it a whirl.. i'm thinking my little ec2 indicator might do better as a unity plugin, or at least offer more options that way
<hazmat> whirl.. nice word that ;-)
<niemeyer> Upgrade rolling.. will get some coffee
<niemeyer> bcsaller: Have you had any failure related to ubuntu-desktop?
<bcsaller> gustavo: it just died on me, but the package, no
<niemeyer> It's taking forever and failing with an error about ubuntu-desktop :-(
<niemeyer> Ok, working now
<niemeyer> Had something pinned
<niemeyer> Installing.. wish me luck
#ubuntu-ensemble 2011-04-05
<niemeyer> Quite painless after the initial issue was sorted out
<niemeyer> Man, Unity is *awesome*
<TeTeT> niemeyer: yeah, it's gotten much better through the development cycle. I showed it yesterday and today to LVM
<niemeyer> TeTeT: I'm impressed indeed
<niemeyer> TeTeT: It's not just different.. it feels great to use
<TeTeT> niemeyer: I like the files & folder and applications dialog, very nicely done
<niemeyer> Yeah
<TeTeT> the first versions that I saw just had a search box, not very user friendly
<niemeyer> TeTeT: Yeah, at least for some values of "user" :-)
<niemeyer> TeTeT: I personally love being able to search for apps rather than browsing through menus blindly to find them
<niemeyer> Hmmm.. we have to talk about the whole "expose" plan
<niemeyer> There are some real blockers for using a normal service
<niemeyer> It's feeling like a long and painful path
<niemeyer> hazmat, robbiew: Mornings!
<robbiew> niemeyer: hey
<robbiew> niemeyer: is this the 1st qbr where you don't have to defend ensemble?
<robbiew> lol
<niemeyer> robbiew: I'm not sure.. I may still be invited for a call or something. :-)
<robbiew> heh
<niemeyer> robbiew: How're things going there?
<robbiew> not bad...on an 11.10 Server requirements call right now..with OEM Services
<niemeyer> robbiew: Nice
<jimbaker`> i also like natty. installed it on my laptop, definitely a better ui exp with unity
<jimbaker`> also reported two crashed apps so far - like others i have had problems with compiz
<_mup_> ensemble/expose-services r185 committed by jim.baker@canonical.com
<_mup_> Use iptables for firewall config and make changes via a built-in service
<_mup_> ensemble/expose-services r186 committed by jim.baker@canonical.com
<_mup_> Clarified how ensemble expose adds a relation to the exposer built-in service
 * hazmat restarts to natty
<jimbaker`> hazmat, good luck :)
<jimbaker`> the only thing i really needed to redo for natty was to reinstall distribute, pip, and virtualenvwrapper because of the upgrade to 2.7
<jimbaker`> after that all of my virtualenvs simply ported over w/o a problem
<jimbaker> is everyone else able to run the mysql/wordpress example on ec2 against trunk? 
<jimbaker> it is failing for me in the db-relation-changed hooks
<jimbaker> when i switch to using my new-hook-semantics-4-remove-env-var branch (including environments.yaml change, virtualenv), np
<jimbaker> hazmat, ^^^
<hazmat> jimbaker, i haven't done run them on ec2 in about a week,  i can take a look after the standup
<jimbaker> hazmat, ok
<hazmat> how do you change timezones with unity..
<jimbaker> hazmat, fortunately it got my tz right :)
<hazmat> jimbaker, wait to you switch timezones :-)
<jimbaker> but there have been some changes around the clock for sure
<jimbaker> just click on the clock, you will see time & date settings
<hazmat> jimbaker, sure, but it doesn't actually come up
<hazmat> run the settings dialog that is
<jimbaker> hmm, no problem for me changing the settings
<niemeyer> jimbaker, hazmat: Can we have a call at some point today to talk about expose?
<niemeyer> There are some obvious problems with the idea of using a normal service, and I'm starting to feel like we're digging ourselves a hole
<niemeyer> It'd be nice to go back to an empty table and look at our requirements again, and then think of some simpler/straightforward ways to achieve it
<hazmat> niemeyer, sounds good
<jimbaker> niemeyer, sounds good too
<niemeyer> brb
<_mup_> ensemble/optional-hooks r186 committed by kapil.thangavelu@canonical.com
<_mup_> switch warning to info log level, use failure hooks instead of missing hooks for hook error tests
<jimbaker> hazmat, are you planning that we add get_agent_state(), or is that just out-of-date in the docs?
<hazmat> jimbaker, where's that referenced from?
<jimbaker> hazmat, internals/agent-presence.rst
<hazmat> jimbaker, there's is an agent state node, but its usage and the exposed api is limited to presence only
<hazmat> and only for agents directly associated to a domain context atm (ie. machine and unit, but not provisioning)
<hazmat> the spec does cover context-less agents though
<jimbaker> i'm going through bug 737949 to determine what status info is available for agent state
<_mup_> Bug #737949: ensemble status should take into account unit and relation workflows and unit agent status <Ensemble:In Progress by jimbaker> < https://launchpad.net/bugs/737949 >
<hazmat> jimbaker, wrong tree
<hazmat> jimbaker, they are two different questions
<hazmat> one is there an agent process running for this domain object
<hazmat> which is answered by the api in ensemble/state/agent.py its a mixin of unit, machine, etc.
<hazmat> the second question, which the bug wants to ask, is what is the workflow state of the unit and its relations
<hazmat> which can use the workflow api directly
<jimbaker> hazmat, thanks for the clarification
<hazmat> jimbaker, the information for the workflow is probably better accessed via constructing a unit lifecycle/workflow object like the agent
<hazmat> for read only usage the state directory isn't strictly nesc.
<hazmat> hmm.. i think
<jimbaker> hazmat, right, the only thing i see now that's doing something like that is in test_workflow
<jimbaker> specifically read_persistent_state
<hazmat> also agents/unit.py
<hazmat> jimbaker, so it needs a separate implementation of unitworkflowstate, relationworkflow state that only loads zk state
<hazmat> via modifying load, and probably disabling store
<hazmat> as external modifications will likely need a different mechanism
<jimbaker> so global observability is preserved and tested, but no api as you said
<jimbaker> to actually read it
<hazmat> jimbaker, well there is an api to read it as provided, it just needs an override of _load/_store methods to defer to the correct implementations for an external process usage.
<hazmat> ie. an api that needs a subclass implementation to utilize it for this case
<hazmat> bcsaller, jimbaker, niemeyer_ standup?
<jimbaker> hazmat, sure, just need to connect it to zk
<jimbaker> hazmat, sure
<jimbaker> re standup
<bcsaller> online
<hazmat> niemeyer_, ping
<niemeyer_> hazmat: Hmm.. I'll need some 10~15 minutes, actually, if that's ok
<niemeyer_> ?
<hazmat> fine by me
<jimbaker> ok
<niemeyer_> I'm relocating to Claudio's place just to provide some company.. he's got to stay at home due to an issue in his leg
<niemeyer_> brb
<jimbaker> bcsaller, hazmat, niemeyer - everyone ready?
<niemeyer> Yep
<hazmat> sounds good
<hazmat> niemeyer, you on skype?
<niemeyer> hazmat: Spinning...
<niemeyer> hazmat: Working
<niemeyer> unit-set expose="port[/protocol] [port[/protocol]]"
<niemeyer> export-port port[/protocol]
<niemeyer> open/close-port port[/protocol] + exposed hook
<niemeyer> jimbaker: /units/<internal id>/ports
<hazmat> out of curiosity has anyone on natty gotten the bzr-pipeline plugin working?
<hazmat> i guess i can try modifying the api compatibility by hand and hope it works
<_mup_> ensemble/statemachine-multi-source-and-success-transitions r181 committed by kapil.thangavelu@canonical.com
<_mup_> explicit action support and success transition support
<jimbaker> SpamapS, btw, i really liked the discovery process you mentioned in your email for finding interesting services for ensemble
<SpamapS> jimbaker: yeah, there's tons of helpful info already in the archive. :)
<hazmat> niemeyer, jimbaker so one other benefit to the machine as service was that it made the notion of monitoring much easier as well
<hazmat> which is a topic i think we've sort of left off for perhaps too long
<hazmat> there's service specific monitoring, and then generic machine/container level monitoring
<hazmat> the former we can do via relation hooks between the monitoring service and the other service.. the latter still seems like an open question to me
<niemeyer> hazmat: Monitoring things which depend on a connection to ZooKeeper to function is somewhat easy
<hazmat> niemeyer, not the sort of monitoring i was referencing.. more along the lines of munin/nagios 
<hazmat> ie. hd full/memory/cpu/network
<niemeyer> hazmat: I see.. we haven't really delayed I think.. this is just another service
<hazmat> niemeyer, i'm not so that it doesn't need additional support of some form for the machine level stats
<niemeyer> hazmat: How so?
<niemeyer> jimbaker: The review queue is all yours ;)
<niemeyer> jimbaker: I'll go over that first thing tomorrow
<hazmat> niemeyer, well are we saying we need a unit of the this per service.. in the lxc case that's not quite right.
<niemeyer> hazmat: Why not?
<hazmat> ie. we ensemble deploy munin for example.. we need to install munin-node on each of the machines in the environment
<hazmat> and configure them to point to the munin service
<hazmat> ie. so say we split into two formulas munin-frontend, munin-node.. when we deploy the latter we don't really want it as a separate unit per se, we want it either in the existing unit (which can be tackled by formula specific monitoring support) or at the machine level
<hazmat> what we don't want is creating a unit of the munin-node on arbitrary placement basis, as it will effectively be monitoring an empty container with just itself
<_mup_> ensemble/config-pipeline r182 committed by bcsaller@gmail.com
<_mup_> minor revisions around type validation
<niemeyer> hazmat: I see, indeed there are some details to sort in this area
#ubuntu-ensemble 2011-04-06
<_mup_> ensemble/expose-services r187 committed by jim.baker@canonical.com
<_mup_> Initial draft of service unit setting as the means to describe what ports to be exposed
<SpamapS> kapil_: interesting re the munin-node example...
<SpamapS> kapil_: I have thought of things like this in the past... I think this is where we start stepping all over puppet and chef's toes. ;)
<SpamapS> kapil_: I would call it a "machine formula" rather than a "service formula"
<SpamapS> OR.. you just build this into formulas as a setting
<SpamapS> ensemble deploy mediawiki --extra-package=munin-node
<SpamapS> kapil_: would love to sit down and think through how best to do it.
<_mup_> ensemble/expose-services r188 committed by jim.baker@canonical.com
<_mup_> Additional edits to remove obsolete text and describe the exposed flag and how it is set, removed, and watched
<kapil_> SpamapS, we were having a talk regarding earlier about making machines like units, to enable something unrelated (exposing ports/service outputs), but there's a bit of mismatch mapping them over.
<kapil_> the machine formulas map pretty well to this sort of thing though
<kapil_> this being monitoring
<kapil_> or enabling machine level policies managed by ensemble
<kapil_> be it landscape or what have you
<kapil_> ie. you could deploy landscape and relate to it machine
<kapil_> but say you don't want all your machines managed by landscape..
<kapil_>  a machine agent isn't really  a service unit, so some of the existing useablility breaks down.. if we want to implement machine formulas for machine policy, they'll probably be spelled a little differently.
<kapil_> SpamapS, yeah.. it is very much overlapping with puppet and chef.
<_mup_> ensemble/expose-services r189 committed by jim.baker@canonical.com
<_mup_> Clarification of when the exposed hook runs
<_mup_> ensemble/config-pipeline r183 committed by bcsaller@gmail.com
<_mup_> updates based on niemeyers review
<niemeyer> Morning all!
<niemeyer> Bandwidth is really bad to certain regions :-(
<niemeyer> Can't access gmail.. :/
<niemeyer> Ok, can't access Google either, but Launchpad works fine.. how's that for a change? ;_)
<niemeyer> biab
<niemeyer_> Phew.. ok, reestablishing the ADSL connection seems to have helped
<hazmat> SpamapS, how you liking sup?
<jimbaker> hazmat, were you able to try running the wp example against trunk on ec2?
<hazmat> jimbaker, i wasn't, my plate is overfull atm
<jimbaker> hazmat, np
<hazmat> jimbaker, do you think you could dig into it?
<jimbaker> hazmat, sure
<jimbaker> hazmat, actually i think it's quite simple - r180 of trunk changed the semantics of relation-get, but the examples were not updated at the time
<jimbaker> ok, that feels much better
<hazmat> jimbaker, awesome, thanks for digging into it
<SpamapS> hazmat: I am *loving* sup
<hazmat> SpamapS, out of curiosity what's your setup look like? 
<SpamapS> hazmat: I still have some 40,000 total emails that aren't moved out of my inbox.. but I drop that by 100 or so every day. ;)
<hazmat> SpamapS, i've been trying to figure out if i should use procmail or imapfilter to pre label things
<SpamapS> hazmat: I have offlineimap pulling from my personal and canonical email accounts.
<SpamapS> hazmat: I have just 2 procmail rules to classify bugmail and srumail
<hazmat> SpamapS, cool, how do you get procmail invoked on mail sync'd by offlineimap?
<SpamapS> hazmat: no I'm doing that on the server
<hazmat> ah
<SpamapS> hazmat: specifically so that my iphone doesn't ever see bugmail or sru mail :)
<SpamapS> hazmat: so about "policy formulas" .. or "machine formulas" .. I was thinking how they'd work in the model.. and wouldn't it be cool if the service unit sort of married the policy formula and the main service formula into one that had all the relations from both?
<SpamapS> it would still have to run the install/start/stop for both..
<SpamapS> but otherwise.. they'd work like one formula.    ensemble add-relation munin-master-service myblog:munin-node
<SpamapS> hazmat: and for added fun... 
<SpamapS> ensemble add-relation myblog:website myblog:munin-http-response-graph
<SpamapS> hazmat: that would probably be fairly tricky for you guys to work into the agent, but from a user standpoint, would make the most sense
<hazmat> SpamapS, that almost looks like some sort of formula inheritance.. i guess its more of a containment thing in this context
<hazmat> interesting
<_mup_> ensemble/trunk r183 committed by bcsaller@gmail.com
<_mup_> Merge config-specification [r=niemeyer,kapil] [f=736321]
<_mup_> Outlines the direction of future Service Configuration work. This document defines how services define options, how to modify those settings and how formula authors interact with such a system.
<SpamapS> hazmat: I think this is going to come up rather quickly as people deploy ensemble
<SpamapS> hazmat: they'll do it by manually merging two formulas together.
<_mup_> ensemble/trunk r184 committed by bcsaller@gmail.com
<_mup_> Merge rename-amp [r=niemeyer] [f=746688]
<_mup_> Rename AMP commands used to communicate between hook utilities and the unit agent. These commands used generic names like `get` and `set` which limited clarity when the set of commands expanded to include more than just relation oriented commands. This simple change renames get to relation_get allowing for easy inclusion of things like config_get in a future branch.
<_mup_> ensemble/trunk r185 committed by kapil.thangavelu@canonical.com
<_mup_> merge optional-hooks [r=niemeyer]
<_mup_> This change allows formulas to optionally implement any formula hooks instead
<_mup_> of requiring all hooks to be implemented.
<_mup_> ensemble/trunk-merge r181 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<hazmat> SpamapS, so the only real difference underlying the policy/machine formula is that they live outside of an lxc container, which is effectivley what we do now. That behavior could be
<hazmat> preserved via some formula type distinction, it might have some additional functional distinction later, but for now it would just be a distinction to whether a formula lived outse of a container. 
<hazmat> For service units liviing inside a container, when establishing a relation, the additional syntax could be used to apply a policy formula as a secondary/global relationlookup after the service unit formula relations
<hazmat> are considered. 
<niemeyer> hazmat: When you have a moment, would you mind to have a look at jimbaker's branch which is pending in the review queue?
<niemeyer> hazmat: I had a look at it, but would appreciate a second look from you there
<hazmat> niemeyer, sure
<hazmat>  the issue i see atm with this is that the service's formula authors needs to avoid associating persistent state in service data denoting between the relationship between the two unit instances, unless we're also willing to for ensemble to introduce a unit 2 machine container or a unit 2 unit relations, which is rather tight coupling
<hazmat> niemeyer, jimbaker, bcsaller standup?
<niemeyer> hazmat: Sounds good
<niemeyer> hazmat: I'm up
<hazmat> bcsaller, skype?
<jimbaker> sounds good, just made some hot chocolate here
<jimbaker> uhoh, my natty laptop is unresponsive... need to reboot it
<jimbaker> hazmat, are we having the standup?
<hazmat> jimbaker, just waiting on ben to get a full house
<jimbaker> i noticed no one liked my logical grouping of hooks being executed with respect to the event, although it might have helped if i had made an explicit comment
<hazmat> jimbaker, hmm. true
<hazmat> jimbaker, it is a logical grouping.. a comment would have made it more clear, and the grouping is removed latter in the branch stack isn't it?
<hazmat> except for join/changed
<jimbaker> for departed, yes, but joined has the multiplicity as you mentioned
<hazmat> jimbaker, i'm fine with that.. but you should note the event/change -> hook mapping
<jimbaker> when i was working through it, the grouping was important to understand what was going on
<hazmat> as a comment
<jimbaker> hazmat, agreed, that would have been helpful
<jimbaker> so i will add that in assuming niemeyer is ok with it
<niemeyer> jimbaker: Clearly it didn't serve its purpose.. :-)
<hazmat> jimbaker, niemeyer you guys up for doing a 3 way standup, i'll give the summary to bcsaller  latter.
<niemeyer> hazmat: Yeah, let's do it
<bcsaller> sorry, just got back
<hazmat> bcsaller, skype?
<bcsaller> yeah
<hazmat> niemeyer, any word on logging this channel?
<niemeyer> ubuntulo1: meet hazmat
<hazmat> niemeyer, aweseome
<hazmat> and logs http://irclogs.ubuntu.com/2011/04/06/%23ubuntu-ensemble.txt
<_mup_> ensemble/expose-services r190 committed by jim.baker@canonical.com
<_mup_> Review edits
 * niemeyer steps out for the day
#ubuntu-ensemble 2011-04-07
<_mup_> ensemble/expose-services r191 committed by jim.baker@canonical.com
<_mup_> Further clarifications based on review points
<_mup_> ensemble/expose-services r192 committed by jim.baker@canonical.com
<_mup_> Added missing exposed key-value for ensemble status output
<_mup_> ensemble/expose-services r193 committed by jim.baker@canonical.com
<_mup_> Spell checked
<_mup_> ensemble/expose-services r194 committed by jim.baker@canonical.com
<_mup_> One last minor edit on what is output for ensemble status for showing a service is exposed
<jimbaker> niemeyer, glad to see the expose services spec has been approved. it was a good process
<niemeyer> jimbaker: Indeed
<niemeyer> jimbaker: Great spec
<jimbaker> niemeyer, thanks!
<niemeyer> Lunch, biab
<_mup_> ensemble/expose-services r195 committed by jim.baker@canonical.com
<_mup_> Minor reworking of how ensemble status displays exposed services and their corresponding opened ports
<_mup_> ensemble/expose-services r196 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<jimbaker> hazmat, bcsaller - based on other conversations, i assume you're ok with merging my expose-services branch into trunk. if so, could you please explicitly approve the branch at https://code.launchpad.net/~jimbaker/ensemble/expose-services ? thanks
<hazmat> jimbaker, i'm having a last look over it now
<jimbaker> hazmat, sounds good
<hazmat> jimbaker, +1.. one minor thing.. the ``exposed`` and ``unexposed`` hooks section doesn't have its __ covering the entire statement for valid rst output
<jimbaker>  hazmat, actually rst is fine with that (otherwise it would have been obvious earlier), but it's definitely sloppy. thanks for pointing that out
<hazmat> jimbaker, cool, i've had it barf on me in those cases before, the spec contents looks good though
<_mup_> ensemble/expose-services r197 committed by jim.baker@canonical.com
<_mup_> Fixed section heading for exposed/unexposed hooks
<SpamapS> exposed hooks?
<SpamapS> have you guys gone and changed everything?
<jimbaker> SpamapS, not too much :)
<jimbaker> SpamapS, an exposed hook runs upon a service being exposed, so it's just a way for a formula to respond (if at all, now that hooks are optional)
<SpamapS> is this the implementation of the "virtual formula" idea?
<jimbaker> SpamapS, no
 * SpamapS has had *zero* time to look at anything you guys have done :-/
<jimbaker> SpamapS, it's just supporting the ability for a service to expose it ports publicly. it's not ok for our machine instances to have an open firewall policy
<SpamapS> Sure it is. I think its very dangerous for ensemble to make decisions for formula writers.
<jimbaker> at some point, it was leaning to have more support for formulas being able to do machine configuration, but we decided this was too big of a change - at least as it would have meant for this particular case
<SpamapS> You saw my idea for machine config right? Just merge formulas with policy formulas.
<jimbaker> SpamapS, yes, i saw that. in comparison, this is a very narrow change
<SpamapS> I'm fairly certain that people will hand-merge formulas in their environment to achieve the same thing
<jimbaker> the problem with the exposer service idea is that it would have allowed it to be in a relation with any service. this seemed to be too loose
<jimbaker> but other policy formulas might work out to be better
<SpamapS> firewall was one of the first things I thought of, and the idea of a standard relation thats always available of 'exposed' is quite cool to me
<jimbaker> i agree with the general attractiveness. however, in general, it can be easier to reason about and implement systems without that quality. or potentially even use them. and i think that's the case here
<niemeyer> SpamapS: The expose feature is pretty simple for the moment, purposefully.. it's just a way for the formula author to let Ensemble know of which port it would like to open for the external world to use
<SpamapS> niemeyer: yeah thats cool. I'm concerned that ensemble is going to try and figure out how to implement firewalls tho.. 
<SpamapS> Its like putting apparmor inside dpkg
<niemeyer> SpamapS: Not implementing, but configuring.  We can't avoid that.
<niemeyer> SpamapS: dpkg isn't great at configuring things so they can run out of the box.  That's one of the reasons why Ensemble exists at all.
<jimbaker> bcsaller, please take a look at https://code.launchpad.net/~jimbaker/ensemble/expose-services if you're now +1 on this branch
<bcsaller> ok
<_mup_> ensemble/trunk r186 committed by jim.baker@canonical.com
<_mup_> merge expose-services [r=niemeyer,bcsaller,hazmat][f=736267]
<_mup_>   
<_mup_> Adds a specification and user guide on exposing services publicly.
<_mup_> Introduces new ensemble commands ( ``ensemble expose``, ``ensemble
<_mup_> unexpose``), hook commands (``open-port``, ``close-port``), hooks
<_mup_> (``exposed``, ``unexposed``), and changes to ``ensemble status``,
<_mup_> along with the supporting infrastructure for firewall configuration
<_mup_> and management within ZK.
<kapil_> natty lockup
<jimbaker> kapil_, i got one of those so far
<hazmat> interesting.. http://melor.github.com/poni/
<SpamapS> niemeyer: right. I think Ensemble's job is to enable configuration. Not to do it.
<niemeyer> SpamapS: That's exactly what we're doing.
<SpamapS> I thought so.. but my fear was that it would be built in or something.
<SpamapS> rather than just a default recommended formula that is easy to replace.
<niemeyer> SpamapS: We're solving a real problem in a simple way so that we can move forward.  Any suggestions on how to do this in a different way with similar effort will be welcome.
<SpamapS> niemeyer: I have no suggestions, and no time, so please pardon my interruption. :)
<niemeyer> SpamapS: No problem.. I don't want to discourage you, but we've debated several approaches, faced problems in many of them, and opted for the current one with some ground.
<niemeyer> SpamapS: It's fully documented in the specification which is http://j.mp/ensemble-docs 
<niemeyer> SpamapS: Your ideas and suggestions are always welcome, but we need more concrete feedback to be able to act on it
<SpamapS> Yeah, I have some major platform stuff to tend to the next 10 days.. but I hope to get some time to play with Ubuntu around 4/18
<SpamapS> s/Ubuntu/Ensemble/ ;)
<niemeyer> SpamapS: Sweet
<niemeyer> bcsaller, hazmat, jimbaker: Standup?
<hazmat> niemeyer, sounds good
<jimbaker> one moment - on phone
<jimbaker> ok, ready for standup
<hazmat> bcsaller, ping
<hazmat> spam can be so poetic
<bcsaller> back
<jimbaker> given its origins, how it could not - http://www.youtube.com/watch?v=g8huXkSaL7o ;)
<bcsaller> street cleaning, had to move car
<jimbaker> sup
<jimbaker> http://sup.rubyforge.org/
<niemeyer> Continues to break for me.. :(
<bcsaller> lunch
<_mup_> ensemble/unit-state-with-formula r191 committed by kapil.thangavelu@canonical.com
<_mup_> update parse_formula_id usage by state domain objects with new exception.
 * niemeyer steps out for some exercising
<_mup_> ensemble/trunk r187 committed by kapil.thangavelu@canonical.com
<_mup_> Merge unit-state-with-formula [r=niemeyer][f=748312]
<_mup_> Unit states now track their formula id, which can be get/set on a unit state. Also introduce a utility
<_mup_> function for parsing/validating formula ids.
<_mup_> ensemble/trunk-merge r182 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
#ubuntu-ensemble 2011-04-08
<_mup_> ensemble/new-hook-semantics-1-departed-hook r182 committed by jim.baker@canonical.com
<_mup_> Addressed review points
<_mup_> ensemble/new-hook-semantics-1-departed-hook r183 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/service-unit-state-upgrade-support r194 committed by kapil.thangavelu@canonical.com
<_mup_> merge changes from review
<_mup_> ensemble/trunk r188 committed by kapil.thangavelu@canonical.com
<_mup_> merge service-unit-state-upgrade-support [r=niemeyer][f=750193]
<_mup_> Enable state api methods on units and services to support upgrade, via
<_mup_> an setting, getting, and watching an upgrade flag on unit states.
<_mup_> ensemble/trunk-merge r183 committed by kapil.thangavelu@canonical.com
<_mup_> merge trunk
<_mup_> ensemble/new-hook-semantics-2-joined-hook r183 committed by jim.baker@canonical.com
<_mup_> Merged new-hook-semantics-1-departed-hook and resolved conflicts
<_mup_> ensemble/new-hook-semantics-1-departed-hook r184 committed by jim.baker@canonical.com
<_mup_> PEP8
<_mup_> ensemble/new-hook-semantics-2-joined-hook r184 committed by jim.baker@canonical.com
<_mup_> Addressed review points
<_mup_> ensemble/new-hook-semantics-2-joined-hook r185 committed by jim.baker@canonical.com
<_mup_> Merged new-hook-semantics-1-departed-hook and resolved conflicts
<_mup_> ensemble/new-hook-semantics-2-joined-hook r186 committed by jim.baker@canonical.com
<_mup_> Removed unnecessary parens
<_mup_> ensemble/new-hook-semantics-3-remove-change-hook r184 committed by jim.baker@canonical.com
<_mup_> Merged new-hook-semantics-2-joined-hook and resolved conflicts
<_mup_> ensemble/new-hook-semantics-1-departed-hook r185 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/formula-upgrade-cli r197 committed by kapil.thangavelu@canonical.com
<_mup_> update to latest parse formula id signature.
<_mup_> ensemble/trunk r189 committed by jim.baker@canonical.com
<_mup_> merge new-hook-semantics-1-departed-hook [r=niemeyer,hazmat][f=740569]
<_mup_> Upon a departed event, invokes <name>-relation-changed hook, then
<_mup_> <name>-relation-departed hook.  A subsequent branch merge will remove
<_mup_> the invocation of the <name>-relation-changed hook.
<_mup_> ensemble/new-hook-semantics-2-joined-hook r187 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/trunk r190 committed by jim.baker@canonical.com
<_mup_> merge new-hook-semantics-2-joined-hook [r=niemeyer][f=744723]
<_mup_> Upon a joined event, invokes <name>-relation-joined hook, then
<_mup_> <name>-relation-changed hook.
<_mup_> ensemble/new-hook-semantics-3-remove-change-hook r185 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/trunk r191 committed by jim.baker@canonical.com
<_mup_> merge new-hook-semantics-3-remove-change-hook [r=niemeyer][f=744724]
<_mup_> Upon a departed event, no longer invoke <name>-relation-changed hook.
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r187 committed by jim.baker@canonical.com
<_mup_> Merged trunk and resolved conflicts
<hazmat> ensemble natty image if anyone is curious, ami-d06b96b9
<hazmat> jimbaker, is that fix for the formulas on trunk checked-in?
<_mup_> ensemble/formula-upgrade-cli r198 committed by kapil.thangavelu@canonical.com
<_mup_> add a dry run upgrade option, and a separate exception for newer formulas not found.
<_mup_> ensemble/formula-upgrades-spec r185 committed by kapil.thangavelu@canonical.com
<_mup_> remove per unit relation upgrade discussion and hook-api into new futures section
<_mup_> ensemble/trunk r192 committed by kapil.thangavelu@canonical.com
<_mup_> merge formula-upgrades-spec [r=bcsaller,jimbaker,niemeyer][f=731519]
<_mup_> Spec documentation of an upgrade-formula command, and accompanying
<_mup_> formula support.
<_mup_> Bug #754318 was filed: add register cleanup utility to cli <Ensemble:New> < https://launchpad.net/bugs/754318 >
<kim0> hazmat: Hi what's that ensemble natty ami do ?
<kim0> s/what's/what does/
<niemeyer> kim0: It's the base image in which formulas run
<kim0> niemeyer: that is used for "bootstrp" right
<niemeyer> kim0: It's used to run everything
<kim0> oh I see
<niemeyer> bcsaller: Are you in Canonical's IRC server?
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r188 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<bcsaller> gustavo: not currently
<niemeyer> bcsaller: Can you please join it
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r189 committed by jim.baker@canonical.com
<_mup_> Addressed review points
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r190 committed by jim.baker@canonical.com
<_mup_> Removed hook stubs in light of optional hook support as r185 of trunk
<_mup_> ensemble/trunk r193 committed by kapil.thangavelu@canonical.com
<_mup_> merge formula-upgrade-cli [r=niemeyer][f=750304]
<_mup_> A new ensemble subcommand for upgrading formulas, via local
<_mup_> repository search, formula publishing, and marking service
<_mup_> units for upgrade.
<_mup_> ensemble/bashified-wordpress-mysql-examples r178 committed by jim.baker@canonical.com
<_mup_> Merged trunk and resolved conflicts
<_mup_> ensemble/bashified-wordpress-mysql-examples r179 committed by jim.baker@canonical.com
<_mup_> Updated examples to use new hook semantics
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r191 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<jimbaker> hazmat, r193 broke trunk for me for both natty and meerkat. when i checked out trunk to r192, it runs on both natty and meerkat for me. (fwiw, i also tried rebooting)
<jimbaker> hazmat, this also occurs w/ a fresh checkout of trunk
<hazmat> jimbaker, broke how?
<jimbaker> hazmat, unit test suite
<hazmat> jimbaker, a particular test? 
<jimbaker> first error:    test_file_storage_returns_same_storage ...                          [ERROR] - then cascades from there
<hazmat> jimbaker, thanks.. checking
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r192 committed by jim.baker@canonical.com
<_mup_> Removed references to ENSEMBLE_CHANGE from docs
<hazmat> i just found out that the dc local government also shutsdown on a federal shutdown, 10% local taxes go straight to the federal treasury to be allocated back.. or not. getting back zero on the dollar.
<jimbaker> a good case for dc statehood
<hazmat> jimbaker, it looks like a zk connection exhaustion, esp with the timing
<jimbaker> hazmat, any good options here?
<hazmat> jimbaker, yeah.. i'm making one.. i filed a ticket on it last night.. bug 754318
<_mup_> Bug #754318: add register cleanup utility to cli <Ensemble:New> < https://launchpad.net/bugs/754318 >
<hazmat> jimbaker, basically just a function to register connections generically for cleanup when the cli exits
<jimbaker> hazmat, so we regain connections with a client.close?
<jimbaker> hazmat, should we also play around with maxClientCnxns in the zk config?
<hazmat> jimbaker, i've done the latter a bit earlier.. i think we're up to 32 connections at the moment,  they recently changed the default upstream to increase it. really the solution is to not have tests leak connections, automatic cleanup for the cli, would solve most of it,  perhaps a forsenic tool as well for checking for dangling open connections from a test
<hazmat> most of the other tests manage their own connection
<hazmat> s/most/all
<hazmat> the cli tests do internal construction, so the automatic cleanup registration when getting a connection there should solve it, afaics
<jimbaker> hazmat, i changed maxClientCnxns=0 for ZK; with that the tests run except for ensemble.providers.tests.test_dummy.DummyProviderFileStorageTest.test_file_storage_returns_same_storage - the original failing test for me
<jimbaker> =0 means unlimited of course
<hazmat> jimbaker, yeah.. i fixed that as well
<jimbaker> hazmat, i think the cleanup mech sounds good, but maybe do =0 for now so trunk works
<hazmat> jimbaker, sounds good.. let me put together a minimal trunk diff 
<hazmat> jimbaker, can you give a +1 on this trivial patch.. https://pastebin.canonical.com/45886/
<jimbaker> hazmat, trying it out
<jimbaker> hazmat, +1
<jimbaker> hazmat, we should add to bug 754318 that ideally we can get the max connections down to a small number, just to ensure cleanup in the case you mentioned
<_mup_> Bug #754318: add register cleanup utility to cli <Ensemble:New> < https://launchpad.net/bugs/754318 >
<jimbaker> hazmat,  if that makes sense to you, i will add that in to the text of the bug
<hazmat> jimbaker, sounds good
<_mup_> ensemble/trunk r194 committed by kapil.thangavelu@canonical.com
<_mup_> up max client connections for test zk, to temporarily prevent connection exhaustion, fix typo in a dummy provider test. [r=jimbaker][trivial]
<hazmat> bcsaller, jimbaker, niemeyer standup? 
<bcsaller> yeah
<jimbaker> hazmat, sounds good, let me jump on skype
<niemeyer> hazmat: I would like to skip it if that's ok, unless you have something for debate specifically
<hazmat> i need to go run some errands in a few before the gov shutdown
<niemeyer> I'm working on reviews
<niemeyer> and have to finish that today
<jimbaker> shutdown does seem imminent
<hazmat> niemeyer, i did have one..
<jimbaker> and i was actually planning to go to the nearby national park this weekend :(
<hazmat> niemeyer, but it can wait.. its the notion that upgrade-formula cli interupption is an inconsistent state, i noted on the merge proposal, but it never got discussed
<niemeyer> hazmat: Ok, can we please delay that until Monday?
<bcsaller> thats fine with me
<hazmat> there's a larger discussion on some other things, but i'll kick it off in response to kim0's email
<hazmat> niemeyer, sounds good
<niemeyer> hazmat: I'd like to be able to focus on the problem you'll bring up
<bcsaller> kapil and I talked over some changes on my branch and I have some revisions to do
<niemeyer> hazmat: But don't want to let my current mental state go away
<jimbaker> maybe skip standup then?
<bcsaller> sounds like it will be Monday
<jimbaker> my big issue just got resolved, now i can get back to releasing my branches
<jimbaker> well, release one more branch, plus get another one ready for review :)
<hazmat> sounds good, i'm off to the post office, back in a bitg
<bcsaller> In that case I'm going to get some food. 
<_mup_> ensemble/new-hook-semantics-4-remove-env-var r193 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/trunk r195 committed by jim.baker@canonical.com
<_mup_> merge new-hook-semantics-4-remove-env-var [r=niemeyer][f=740330]
<_mup_> Removes the setting of the ENSEMBLE_CHANGE environment variable.
<SpamapS> Hrm.. I really need to be able to make principia work w/ the current ensemble trunk. :-P
 * SpamapS feels it will take a lot of effort. :-/
<jimbaker> SpamapS, you may want to take a look at lp:~jimbaker/ensemble/bashified-wordpress-mysql-examples, which i just pushed up
<SpamapS> jimbaker: Yeah I will do that.
<jimbaker> this branch incorporates trunk as it is now, including changes to relation-get and new hook semantics. it's actually not too bad
<SpamapS> We're trying to get ensemble to work w/ openstack btw
<jimbaker> SpamapS, now that we are deploying against natty, i'm going to change it to use augtool
<SpamapS> not sure if trunk has this problem, but in the old rev in the PPA you have to hack the code to set the ec2 and s3 uri's
<jimbaker> SpamapS, this bug is still outstanding, not certain if this will have impact on that: bug 725082
<_mup_> Bug #725082: DNS error when specifying a different S3 URI <Ensemble:New> < https://launchpad.net/bugs/725082 >
<SpamapS> jimbaker: yeah we ran into that right at the beginning. :-(
<niemeyer> jimbaker?
<_mup_> ensemble/state-machine-enhancements r199 committed by kapil.thangavelu@canonical.com
<_mup_> statemachine success transition support
<_mup_> ensemble/state-machine-enhancements r200 committed by kapil.thangavelu@canonical.com
<_mup_> add support for transition specified actions
<niemeyer> Heading off for some exercising.. will be back later
<hazmat> niemeyer, got a  moment?
<hazmat> too late?
<hazmat> jimbaker, bcsaller ?
<hazmat> i wanted to bounce an idea regarding upgrade state transitions on error
<_mup_> ensemble/state-machine-enhancements r201 committed by kapil.thangavelu@canonical.com
<_mup_> pep8
<_mup_> Bug #755062 was filed: statemachine needs support for success transitions and custom actions <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/755062 >
#ubuntu-ensemble 2011-04-09
<niemeyer_> hazmat: Here now
<niemeyer> jimbaker?
#ubuntu-ensemble 2011-04-10
<_mup_> Bug #756581 was filed: Documentation for db-relation-changed is out of date <Ensemble:Confirmed> < https://launchpad.net/bugs/756581 >
<_mup_> Bug #756685 was filed: Bootstrap/shutdown should only operate on one environment <Ensemble:New> < https://launchpad.net/bugs/756685 >
