#ubuntu-ensemble 2011-07-04
<niemeyer> Hello!
<hazmat> niemeyer, hallo!
<hazmat> niemeyer, i think most of the state side people are watching fireworks today
<hazmat> er. us
<niemeyer> hazmat: Oh?
<hazmat> niemeyer, 4th of july holiday, us independence day
<niemeyer> hazmat: Ahhh, superb
<niemeyer> hazmat: Congrats then! :)
<hazmat> niemeyer, thanks, dc is a mad house already with security.. i dropped kaleb (my son) off at camp this morning, it was a ghost town besides road closures
<niemeyer> hazmat: Crazy.. I'd expect that would be _the_ place for the celebrations
<hazmat> niemeyer, i'll be hacking today.. how was your trip back?
<niemeyer> hazmat: It was very good.. it was a new experience for me as well to go straight to a conference on arrival
<hazmat> niemeyer, it will be this evening, crazy amount of fireworks in the area
<hazmat> niemeyer, cool, was that FISL?
<niemeyer> hazmat: Yeah
<hazmat> niemeyer, cool, i've always heard good things about that conference
<niemeyer> hazmat: It was pretty nice indeed.. was also very happy to have a talk on Ensemble that people understood, for a change. :-)
<hazmat> niemeyer, indeed, being able to see and touch it, makes a big difference
<hazmat> niemeyer, i've just been in  bug-fixing and support mode the last few days, i'm going to setup deploy to auto add peer relations and take a look at the relation-broken bug
<hazmat> that's on my plate for today, along with trying to push some security stuff
<hazmat> niemeyer, what's the next sprint date?
<niemeyer> hazmat: Monday (!)
<hazmat> niemeyer, ??!
<hazmat> niemeyer, i'm mia till july 23rd
<niemeyer> hazmat: (The list of things sounds awesome)
<hazmat> niemeyer, the review queue definitely needs some love... i think i've got reviews in all the items that i haven't put in
<niemeyer> hazmat: I'll check the queue once I manage to handle some of the key conversations going on
<hazmat> niemeyer, we might need to revisit the 2 reviews per branch.. it sounds like we're going to need more rapid turnaround.. and 50% team buy in per branch might be excessive atm
<hazmat> niemeyer, sounds goodf
<niemeyer> hazmat: Re. the sprint, it'll be about bare metal deployments
<niemeyer> hazmat: In Miami
<hazmat> niemeyer, i'm not going to be able to make any sprints till july 23rd while kaleb is in town.
<niemeyer> hazmat: I'll attend, and will try to have William there
<hazmat> niemeyer, cool re william
<niemeyer> hazmat: No worries.. if you could be somewhat hooked in as you were last week, that'd already be awesome
<hazmat> niemeyer, sounds good... when's the next next sprint ;-)
<hazmat> is that austin in august?
<niemeyer> hazmat: Austin, early next month
<hazmat> cool
<niemeyer> hazmat: Yep
 * hazmat grabs an espresso
<niemeyer> Lunch time
<hazmat>    
<_mup_> Bug #805554 was filed: Ensemble from ppa does not work with nova. <Ensemble:New> < https://launchpad.net/bugs/805554 >
<kim0> Is it ok to file a bug to break the one service-unit per instance assumption?
<niemeyer> Restarting due to kernel change..
<niemeyer> kim0: It certainly is, assuming there's nothing there yet
<_mup_> Bug #805585 was filed: Break the one service-unit per instance assumption <Ensemble:New> < https://launchpad.net/bugs/805585 >
<franciscosouza> niemeyer: ping
<hazmat>   franciscosouza anything i could help you with?
<franciscosouza> last saturday I went to a conference where niemeyer presented an ensemble demo video, do you know where I can get/watch this video?
 * hazmat looks around
<hazmat> franciscosouza, http://www.youtube.com/watch?v=AMHcy63wRL0 and  http://www.youtube.com/watch?v=qxMhKbDSbOw
<hazmat> are the two extant videos that i know of
<franciscosouza> thank you, that's what I was looking for :)
<_mup_> Bug #791042 was filed: *-relation-broken has no way to identify which remote service is being broken <Ensemble:New> < https://launchpad.net/bugs/791042 >
<daker> any formula expert here ?
<daker> or an ensemble expert :)
<daker> my formula has state: install_error, how can i know what's going on with it, debug-log doesn't show any thing useful :/
<hazmat> daker, hmm.. yeah. there's an outstanding issue with debug log and install hooks (the log isn't active till after the debug hook)
<hazmat> er after the install hook
<hazmat> daker, so there are two options, using debug-hooks right after deploying to debug the install hook, or checking the on disk log of the unit agent to see what happened with the failed installed hook
<daker> hazmat, the logs are located in ?
<hazmat> daker, to check the on disk file.. ensemble ssh service_unit_name  && cd /var/lib/ensemble/units/service_unit_name/ && cat formula.log (i think)
<daker> ah ok
<niemeyer> hazmat: It sounds like we should rethink this in the near term
<niemeyer> Something like a rolling log, at least, might be nice
<niemeyer> Maybe even better would be to have easier access to the on-disk log
#ubuntu-ensemble 2011-07-05
<kim0> Morning
<hazmat> sidnei, nice logo ;-)
<kim0> :D
<alienpulse> :)
<kim0> alienpulse: so you were interested in writing an ensemble formula :)
<kim0> welcome 
<kim0> alienpulse: we just had a private chat .. but let's continue talking over here
<alienpulse> kim0, okay no prob.
<kim0> so basically the best way to get started is to start reading the couple of tutoirals
<kim0> https://ensemble.ubuntu.com/docs/user-tutorial.html 
<alienpulse> ill
<kim0> https://ensemble.ubuntu.com/docs/write-formula.html 
<kim0> with getting started
<kim0> maybe checkout the 2 vids we have to get a feel
<kim0> then pick something you'd like to ensemblize 
<kim0> and start hacking
<alienpulse> :)
<alienpulse> thanks a lot ..
<alienpulse> ill work on that ASAP !
<kim0> alienpulse: in every step along the way .. if you have questions ask here in this channel and ping me if you dont get an answer .. I'll always help
<kim0> alienpulse: AWESOME :)
<kim0> \o/
<kim0> woohoo
<alienpulse> kim0, Cool :)
<alienpulse> \m/
<kim0> alienpulse: once you decide which package you'd like to do .. let me know .. rock n roll
<alienpulse> kim0, okay ill do .. i think im gonna go for Drupal
<kim0> alienpulse: nice choice .. I've written the writing a formula tutorial on drupal .. so you can start from there
<kim0> alienpulse: I didn't really link the formula to principia .. so you can do it
<alienpulse> kim0, ohh okay 
<kim0> also, I used drush to keep things too simple .. but you might want to do it differently
<kim0> Have fun
<alienpulse> emm
<alienpulse> ok
<kim0> drupal is a good choice to start with :)
<kim0> alienpulse: what's your LP username
<kim0> alienpulse: I'll create a bug and assign to you so you :)
<alienpulse> kim0, PM :)
<kim0> alienpulse: just assigned :)
<alienpulse> kim0, thanks a lot ...
<alienpulse> :)
<kim0> alienpulse: https://bugs.launchpad.net/principia/+bug/805955
<_mup_> Bug #805955: Formula needed: Drupal <new-formula> <Principia Ensemble:New for abedeltawil> < https://launchpad.net/bugs/805955 >
<kim0> alienpulse: I think the best way, is to assemble the drupal formula from the tutorial, get it working, then start hacking on it to make it a "real" formula :)
<kim0> enjoy
<alienpulse> kim0, okay man ill do my best ,,, thanks again ,
<kim0> Awesome .. and remember for any questions .. we're all here for you
<kim0> rock on
<koolhead17> kim0, am not able to move anywer :(
<kim0> koolhead17: don't feel bad :)
<kim0> koolhead17: you're the best .. are you hitting a problem with phpmyadmin .. or just not enough time ?
<koolhead17> i would say i need to learn bit more about dbconfig-common :)
<kim0> the preseed thing ?
<kim0> I dont really know much about that :)
<koolhead17> hmm. i will try to nail it tonight 
<koolhead17> kim0, BTW people are coming up with nice Ensemble logo`s E for Ensemble :)
<kim0> koolhead17: hehe yeah :)
<_mup_> ensemble/auto-peer r264 committed by kapil.thangavelu@canonical.com
<_mup_> ensemble deploy automatically adds peer relations.
<_mup_> ensemble/test-api r236 committed by kapil.thangavelu@canonical.com
<_mup_> a new test api for construction of environments.
<_mup_> ensemble/auto-peer r265 committed by kapil.thangavelu@canonical.com
<_mup_> riak peer rel test for deploy
<niemeyer> Woot
<hazmat> niemeyer, its in review now as well
<niemeyer> hazmat: Awesome, thanks a lot
<hazmat> niemeyer, for the relation-broken cli api.. i'm thinking just return empty values for relation-list, relation-get (except for self which returns valid values), relation-set is unclear, i think it should error
<niemeyer> hazmat: It depends a bit on the use case
<niemeyer> hazmat: I think Adam had in mind doing actions depending on the remote data, I'd think
<niemeyer> hazmat: It's not clear if should handle this case, or advise to store any _needed_ information locally
<hazmat> niemeyer, hmm.. there is no guarantee that remote data even exists at that point
<niemeyer> hazmat: Yeah, that's the key point to ponder
<hazmat> its assymetric cleanup across the relation endpoints
<niemeyer> hazmat: The question is whether we should really do this or should allow querying of last available state
<hazmat> niemeyer, access to local data is fine, and we could even store it (i'd prefer not to... but it could be valid).. the relation data is effectively  dead, even if the relation is resurrected (it will be as a new relation)
<hazmat> s/store/allow writes to local data
<niemeyer> hazmat: Nah, that'd be weird.. it's a clean up really
<hazmat> niemeyer, agreed
<niemeyer> hazmat: We should talk to Adam to see what he's trying to do
<bcsaller> niemeyer: he wanted to do things like remove grants in the local db when the remote side goes away
<niemeyer> bcsaller: Morning!
<niemeyer> bcsaller: bcsaller: Yeah, that's what I got from the bug description too
<bcsaller> morning
<niemeyer> bcsaller: The question is what is he trying to fetch from remote side
<niemeyer> bcsaller: Why is relation-get relevant in this context?
<niemeyer> Maybe all he needs is a relation id
<bcsaller> yeah, I think he had to know which relationship was broken so he could map the data
<bcsaller> I suspect thats correct
<hazmat> niemeyer, hmm. clint put in a bug last month about wanting to get the service name of the remote side for cases like this
<niemeyer> In which case we'd be fine with the already filed bugs
<hazmat> so he could cleanup local resources for the remote side
<niemeyer> hazmat: Yeah, I've linked the relevant bugs on Adam's bug too
<bcsaller> niemeyer: we didn't talk about it at the rally but we should still talk about the bug where relations should get ids
<niemeyer> hazmat: None of the other bugs require access to the data, though
<hazmat> niemeyer, its unclear if the relation-id would always contain the relevant data
<bcsaller> I wasn't clear on the use case in the sense that I didn't know which tools would take those ids to do anything useful
<niemeyer> bcsaller: Yeah, we have to implement this
<hazmat> niemeyer, ie. if the remote side specifically requested a particular principal/db name etc
<niemeyer> bcsaller: There are a few use cases described in the bug
<niemeyer> hazmat: Well, that'd be weird either way
<niemeyer> hazmat: That kind of cleanup should happen locally
<niemeyer> hazmat: Rather than trusting the remote data (which may have change)
<niemeyer> d
<hazmat> true, but its a communication protocol, i can definitely see cases where local cleanup might want remote state, although i think it could be argued that state should be copied local if its needed for cleanup
<niemeyer> hazmat: Right.. it's a bit strange in that the relation was broken, and one wants to talk to the remote besides that.
<niemeyer> Either way, let's look at the use case
<hazmat> hmm... i think i need to add a generic file manifest to formulas to be able to do clean upgrades ( remove obsolete files)
<niemeyer> hazmat: Interesting idea
<niemeyer> hazmat: We probably don't need an actual file, though
<niemeyer> hazmat: The content of the formula bundle is its own manifest
<hazmat> niemeyer, yeah.. its computed on demand
 * niemeyer => lunch ftw
<daker> hi can someone pls check this http://paste.ubuntu.com/638462/
 * kim0 looks
<kim0> oh more licensing :)
 * hazmat looks as well
<hazmat> daker, unless your working at canonical and doing this for work, i'm pretty sure you need the canonical copyright statement, ie.  is it an original work, on a not for hire basis
<hazmat> er.. you don't need
<daker> hazmat, you mean the first section ?
<hazmat> daker, any of the sections that reference canonical copyright would be dependent on this being done as work for hire, contribution to ensemble itself would require copyright assignment per canonical policy, but formulas are effectively data inputs to ensemble, and are licensed/copyrighted by the author as an independent work. If the author is performing the work as a result of a fiduciary agreement, the terms of such agreement come into play. for mo
<hazmat> st common situations this results in the work being considered a work for hire, with copyright assignment (and thus licensing choice) to the hiring entity. If you do work at canonical, i'm actually unclear what our formula licensing policy is.. ensemble itself is AGPL.
<daker> hazmat, no, i don't do work for canonical 
<hazmat> daker, than your free to claim copyright solely for yourself if its an independent work, and license as you see fit, the canonical copyright statements are nesc in that case.
<hazmat> UGH.. sorry are not nesc in that csae
<hazmat> kim0, we should get this stuff in a FAQ and vetted
<daker> hazmat, good now http://paste.ubuntu.com/638471/ ?
<hazmat> daker, looks good
<kim0> hazmat: thanks .. taking note and acting
<_mup_> ensemble/formula-upgrade-removes-obsolete r264 committed by kapil.thangavelu@canonical.com
<_mup_> dynamically computed manifests for formulas
<hazmat> kim0, also added links to your screencasts to the wiki front page.. i don't see an easy way to embed them directly without a moin plugin.
<_mup_> Bug #806037 was filed: FAQ needs updating for formulas licensing <Ensemble:Confirmed for kim0> < https://launchpad.net/bugs/806037 >
<kim0> hazmat: Yeah doing that one too
<hazmat> kim0, cool, i just finished adding them their near the bottom
<kim0> ah ok
<hazmat> it would be nice to have them embedded closer to the top
<hazmat> but i think that requires some ops support, perhaps niemeyer would know more re the site setup and options for moin plugins
<kim0> yes .. I looked at that before
<kim0> it needs a moin plugin indeed
<kim0> I think we're getting a new site that's more than a wiki
<kim0> so that's probably enough for now
<kim0> I saw people working on a new website during the past sprint
<hazmat> hmm.. i think i'm making this way more complicated than needed (clean formula upgrades), i could just remove the whole formula dir instead of just the stale entries, and extract the new
<kim0> hazmat: does that look good http://paste.ubuntu.com/638479/plain/
<daker> btw you can buy the cloud portal from me :D
<kim0> Mostly punctuating your wording :)
<kim0> daker: lol :)
<kim0> hehee
<hazmat> kim0, looks fine to me, but it would be nice to have this vetted by the org
<kim0> hmm ok
<serge_> jamespage: hey, trying out your new jenkins formulas.  i've probably forgotten how to do it right, but i'm not seeing any plugins available?
<serge_> (wanted to test with git)
<kim0> hmm I tried robbie, but he seems to be OoO till next week .. was wondering who can approve this formula licensing question .. any thoughts are welcome
<m_3> kim0: clint and dustin had some examples
<serge_> (biab)
<kim0> guess I'll just push the faq entry in the review queue for now .. then once vetted can be merged
<kim0> m_3: thanks man
<m_3> kim0: clint's updated mysql copyright... http://pastebin.com/X5EnhCxt
<kim0> SpamapS: If you have a sample formula license for an independant contributor .. let me know so I'd link to it
<kim0> m_3: problem is that it's copyright Canonical
<kim0> m_3: which shouldn't be the case for most people
<kim0> at least that's my current understanding
<kim0> I'm basically trying to get approval over http://paste.ubuntu.com/638479/plain/ .. 
<m_3> kim0: yeah, that was my understanding... dunno who can approve
<niemeyer> hazmat: There's a major revamp of the website front page coming
<niemeyer> kim0 knows about it too
<kim0> Yeah I told him 
<hazmat> niemeyer, nice
<jamespage> serge_: are you just getting an empty list?
<serge_> jamespage: yeah
<serge_> (just commented in the bug)
<jamespage> serge_: its probably unrelated
<jamespage> sometimes the plugin manager can be a bit flakey
<serge_> ok
<serge_> so, i could add a relation, destroy the relation, and i can destroy the jenkins-slave service.
<serge_> BUt what is the ensemble command to just remove one machine from a servcie?
<jamespage> serge_: remove-unit 
<serge_> jamespage: d'oh!  i was looking at the ensemble help output list, and somehow never saw that one
<serge_> thanks!
<serge_> is there an easy to set up distcc plugin?  so i can fire up 3 slaves and get fast kernel compiles?
<serge_> (that's a jenkins q, not ensemble :)
<jamespage> serge_: nope - you would have to setup distcc sep and then call from jenkins
<jamespage> SpamapS: around?
<jamespage> serge_: BTW I added a bit of intelligence into the slave setup - it now calculates the number of executors based on the number of cores the system has
<m_3> jamespage: SpamapS is out this week
<jamespage> m_3: bah
<jamespage> not sure why that did not show on my calendar
<jamespage> oh - it did I just missed it :-)
<m_3> jamespage: dunno... he was gonna have spotty coverage.. in-laws in central europe
<jamespage> its no problem really
<jamespage> I've been working on upgrading the zookeeper package in Debian/Ubuntu and wanted to catchup
<jamespage> serge_: I can't reproduce your issue - I can see plugins just fine.
<serge_> jamespage: excellent
<serge_> jamespage: my new instance has plugins.  yay!  (wonddr what happened before)
<hazmat> jamespage, awesome re zk, it sounded like the debian version was going to  need a new maintainer (the old one bowing out), the latest release of zk is packaged in zk, its a pretty minimal diff to the packaging of the older version in debian
<jamespage> serge_: hmm - sometimes it take a while to scan the update site
<jamespage> hazmat: well I'm the new maintainer for my sins :-) have the package prepped and have done a basic test via PPA
<kim0> jimbaker: hazmat bcsaller niemeyer SpamapS m_3 .. Could you guys please send me your take on Ensemble happenings of past week. Any decisions at the sprint? any new bits, formulas or bugs that you hacked, please send me asap so I can tell the world about it  :)
<kim0> This also goes to anyone in this channel .. let me know cool things you hacked
<hazmat> jamespage, right on.. now i know who to bug ;-)
 * jamespage is looking forward to being bugged already :-)
<jamespage> serge_: hey - one other thing - the python-jenkins package is in the new queue for oneiric
<m_3> kim0: will do
<kim0> m_3: Thanks man :)
<jamespage> I've stuck it in a PPA for natty; but I guess once we can use oneiric AMI's we should update the formula to check
 * kim0 poinst to the dev team :)
<kim0> points even
<serge_> jamespage: cool.  any roadblocks right now to its inclusion?
<jamespage> serge_: I don't think so (fingers crossed :-))
<_mup_> ensemble/formula-upgrade-removes-obsolete r265 committed by kapil.thangavelu@canonical.com
<_mup_> stale formula files are removed on formula upgrade.
<m_3> hazmat: about removing stale formulas...
<m_3> hazmat: what does that do to a rolling migrations?
<hazmat> m_3, its more about removing stale dirents in a formula when upgrading
<hazmat> m_3, nothing
<m_3> s/migrations/migration/
<m_3> ah, ok... misunderstood then
<hazmat> m_3, for rolling upgrades, i'm thinking its something that's more formula implemented with ensemble providing a cli api for something like leader election (which is also useful for other contexts)
<hazmat> although that has implications in that implies potentially long running hooks.
<m_3> hazmat: just was imagining a situation where there are multiple versions deployed at the same time
<m_3> hazmat: yeah, we're still just hitting the surface of the tools (like leader-elect) needed by formulas
<hazmat> m_3, indeed that is possible with the current upgrade scheme, each of the units is upgrade independently although concurrently in practice, we stop hook execution during upgrades atm
<hazmat> m_3, yeah.. i wrote up some docs on more advanced upgrade scenarios, which aren't covered by the simple formula upgrade that's in place right now
<m_3> hazmat: thanks man... still learning and need to test these scenarios
<hazmat> https://ensemble.ubuntu.com/docs/upgrades.html
<m_3> hazmat: cool thanks!
<hazmat> m_3, the current upgrade mechanism is mainly about supporting the formula development/iteration story
<m_3> hazmat: right
<hazmat> at least that was the primary use case i was targeting, simple service upgrades are possible with it, but more complex ones will definitely find it lacking without additional support
<hazmat> i think effectively we'll need to support perhaps some different policies, like rolling migration, or upgrade one with the rest stopped, etc
<m_3> hazmat: yeah, I started thinking about this in your 'Service Upgrades' section
<m_3> hazmat: with a rails app that pulled from source too... so even more complex
<niemeyer> kim0: Maybe put it online somewhere so people can review what you have at the moment
<niemeyer> kim0: I think you were present in most of the key decisions we've made
<m_3> hazmat: agree about policies... until then, we'll try to produce example scenarios
<hazmat> kim0, i've mostly been doing bug fixes, but all my work from last week is still in the review queue, the highlights are more robust handling of transient network issues, ability to use standard ubuntu images (really any cloud-init enabled ubuntu/deb image), hooks executing inside of formula dirs, peer relations automatically added on deploy of a service, fixes for ensemble status with peer relations.. that's all i recall atm
<hazmat> m_3, yeah.. even a simple wordpress service with a db migration runs into this
<negronjl> hazmat:  most of the issues mentioned on your last post affect me.  Any idea when will the changes go through ?
<hazmat> negronjl, as soon as they get reviewed
<negronjl> hazmat:  lol... I guess the question is when will they be reviewed 
<hazmat> negronjl, i
<hazmat> negronjl, btw end of the week at latest i'd assume, they would have gotten reviewed last week except for the sprint/rally i think
<zul> what does this error mean? "2011-07-05 13:44:13,333 ERROR Connection was refused by other side: 111: Connection refused."
<negronjl> zul:  ensemble is not ready for you yet :)
<hazmat> zul, typically it means ssh isn't active on yet on the remote side even though the machine is up
<zul> negronjl: well duh :)
<hazmat> kim0, oh and i wrote a security specification for how we'll be approaching security wrt to zk
<m_3> zul: wait longer
<zul> hazmat: so wait and try agian?
<niemeyer> negronjl: Today
<niemeyer> negronjl: I mean, they'll be reviewed today, at least once
<negronjl> niemeyer:  thx...that's blocking a few things on my side.
<hazmat> zul, yeah.. which command where you running? i think we try to auto-retry most of the time
<zul> hazmat: ensemble bootstrap
<niemeyer> zul: This is a known issue, actually.. it should really be waiting by itself
<hazmat> hmm... that's very odd
<hazmat> bootstrap doesn't talk to any of the ensemble machines.. just the machine provider
<kim0> hazmat: thanks for the list .. that's awesome :)
<hazmat> zul, is this against openstack?
<zul> hazmat: yep but there is somethign wrong with my install though euca-describe-images is taking a long time
<hazmat> ic
<zul> i think i found it
<hazmat> niemeyer, re populating hostname automatically for relations, doing so effectively assumes its static, and if that's the case we might as well populate it directly to the machine state, so the unit agents can avoid having to query the machine provider
<niemeyer> hazmat: Why does it assume it's static?
<hazmat> niemeyer, hmm.. good question.. we don't really capture any event for the unit to correct it upon change
<niemeyer> hazmat: If the hostname in a relation change, things should just work normally.. relation-changed hooks firing, etc
<hazmat> niemeyer, the question is how do we know it changed to update the relation data
<niemeyer> hazmat: That's a separate problem
<hazmat> its effectively an internal detail of the machine provider
<hazmat> afaik most of them give out statics for the lifetime of the instance
<hazmat> subject to reuse upon termination
<niemeyer> hazmat: It's not a detail of the machine provider.. it's a detail of the machine
<niemeyer> hazmat: and the fact IaaS'es don't change it right now doesn't really affect the design
<niemeyer> hazmat: It just feels like a pretty convenient way to get such information
<niemeyer> hazmat: and one that allows for it to change as well, without any effort
<hazmat> niemeyer, agreed, we can propogate it nicely upon change if know about the change, but it sounds like your suggesting we cron to check on the machine.. else we don't have a way of detecting and propogating the change
<niemeyer> hazmat: I'm suggesting a design that allows for us to change it.. I'm not suggesting we address the change right now
<hazmat> or cron internal to the agent
<hazmat> niemeyer, sounds good, so unit agents directly pulling ip address from the machine and populating hostname for relation data when instantiating the unit in the relation
<niemeyer> hazmat: +1
<niemeyer> Man.. I almost feel bad to say that now.. damn Google.
<_mup_> Bug #806098 was filed: fact repository for Ensemble <Ensemble:New> < https://launchpad.net/bugs/806098 >
<hazmat> is there a query syntax for launchpad bugs.. stuff like AND  OR etc?
<hazmat> out to run some errands bbiab
<niemeyer> hazmat: Don't think so
<niemeyer> hazmat: But there are some advance options
<niemeyer> hazmat: See ya
<niemeyer> Got a broken pipe at home.. the joys of assembling a house together.
<kim0> broken pipe .. for a moment I was thinking SIGPIPE :)
<kim0> it's an actual pipe
<niemeyer> kim0: Hehe, yeah
<niemeyer> kim0: A real SIGPIPE
<kim0> hehe :)
<niemeyer> hazmat: ping
<niemeyer> hazmat_: ping
<kim0> Here's our weekly report .. edits are most welcome http://pad.ubuntu.com/ensemble-week-5th-July
<hazmat_> niemeyer: pong
<niemeyer> hazmat_: Hey
<niemeyer> hazmat_: Was reviewing the session-event-handling conversation again
<kim0> I'll publish in an hour .. make sure your edits go there 
<hazmat_> unity seems to have issues for me with multiple screens, i'm currently in a VT on my laptop working on my screen session from my desktop.. sadness
<hazmat_> niemeyer: awesome
<niemeyer> hazmat_: Regarding your questioning on [4], what's your take on it?
<niemeyer> hazmat_: I read your answer on it, but it's not clear what you have in mind to sort out the fact there's background logic that won't complete
<niemeyer> kim0: Cool, thanks
<kim0> great
<niemeyer> kim0: The LXC stuff should probably not be there..
<kim0> too early ?
<niemeyer> kim0: At least not with that wording
<hazmat_> niemeyer: so there are two options, leave it as is.. or on fatal errors (session expired) propogate to both
<niemeyer> kim0: People will want to try that, and it's not integrated yet
<niemeyer> kim0: You can mention the experiment was successful and all
<hazmat_> both being the connection level handler and the extant deferred
<kim0> ok, I'll change to "work has started on ..."
<hazmat_> niemeyer: the thing is we don't really expect the extant watchers to do anything except die, which makes for some additional unhandled error logging, or storm of such when those events happen
<niemeyer> hazmat_: This is the problem..
<niemeyer> hazmat_: The logic simply _stops_. There's no chance for error handling or anything
<niemeyer> hazmat_: Like, oh, if that doesn't work I'll log a warning, or.. if that doesn't work, remove the temporary directory, or anything really
<hazmat_> the connection level handlers allow us to centralize the handling, which i see as a good thing, watch level error handling is going to be more error prone imo and its going to be subject to stampede effects 
<niemeyer> hazmat_: I can accept that as an ugly hack we're putting in place knowing it's a terrible way to program.
<niemeyer> hazmat_: and with a known way out
<hazmat_> niemeyer: how about i put an option to allow for either
<hazmat_> ie. fatal session events propogated to watchers
<hazmat_> those deferreds are effectively dead in that case.. the problem is that its tricky, the fatal error might come in response to a synchronous call not tied to a session event
<niemeyer> hazmat_: But it's important to make it very clear that killing _any_ action is simply not feasible.. it's like crashing, except the application continues running
<hazmat_> er. async call that is
<_mup_> Bug #806184 was filed: peer interface relationship data not propagating <Ensemble:New> < https://launchpad.net/bugs/806184 >
<hazmat_> hmm
<negronjl> Disregard bug: 806184
<negronjl> I marked it invalid. 
<niemeyer> negronjl: Cool
<niemeyer> I've been talking to hazmat_ for something like two hours already.. we'll write down some notes when we're done
<_mup_> Bug #806241 was filed: It should be possible to collocate service units <Ensemble:New> < https://launchpad.net/bugs/806241 >
<adam_g> is there any way of having a service not start after the install hook? other than having a noop install hook?
<hazmat_> adam_g: ensemble is just executing the named hooks, what you do in them is up to you
<hazmat_> adam_g: some packages automatically start the service/daemon after install a pkg
<hazmat_> adam_g: you can switch them to be manually started with an upstart override and set to manual.. or just stop them after install (if its the package)
<hazmat_> probably a few other ways
#ubuntu-ensemble 2011-07-06
<adam_g> hazmat_: right. i was thinking more in terms of deferring executiion of start until some configuration has been completed
<niemeyer> adam_g: Yeah, that'd be fine
<niemeyer> adam_g: I mean, deferring start of the service to another hook
<hazmat_> adam_g: definitely.. i do something similiar in the mongodb-config server, it only starts when there is a relation to a mongos server (else its just a regular mongodb server)
<_mup_> txzookeeper/trunk r41 committed by kapil.foss@gmail.com
<_mup_> merge session-event-handling. Much improved session event and connection error handling with zk cluster tests.
<niemeyer> It's past bed time here.. night all
<koolhead17> kim0, 
<kim0> koolhead17: Morning
<koolhead17> kim0, i don`t want to see some executive in Tie/Coat surfing cloud in the logo :D
<kim0> koolhead17: lol :)
<kim0> it's too early to choose hehe
<koolhead17> cool, then :D
<kim0> yeah
 * kim0 checks Inbox
<koolhead17> hi et_   
<et_> koolhead17: holla
<koolhead17> notthing much 
<kim0> Hi everyone, just letting you know we're having the Ubuntu Cloud Days irc event on the 25th/26th. Everyone is invited to add a session at https://wiki.ubuntu.com/UbuntuCloudDays/Timetable Please add your session as soon as you can, if unsure about the title, just write TBD. Ping me for any details, thanks
<niemeyer> Morning all!
<hazmat> g'morning
<niemeyer> hazmat: Hey man, how're things there
<hazmat> niemeyer, pretty good, just got back from a long drive
<m_3> morning gang
<jcastro> kim0: do I need the PPA for oneiric? In oneiric itself python-txzookeeper is uninstallable
<kim0> jcastro: r u on oneiric again ? 
<kim0> I was contemplating an upgrade :)
<kim0> jcastro: yeah you generally want the ppa
<kim0> jcastro: to get the daily builds
<jcastro> ok
<kim0> m_3: o/
<niemeyer> jcastro!
<niemeyer> jcastro: Rock the world
<niemeyer> fwereade: Welcome, officially! :)
<fwereade> niemeyer: thanks!
<niemeyer> Folks, William is joining the Ubuntu Server team to help developing Ensemble _today_!
 * fwereade waves shyly
<fwereade> still figuring a lot of things out, but looking forward to the sprint
<niemeyer> fwereade: and here comes your first code review
<niemeyer> fwereade: https://code.launchpad.net/~hazmat/ensemble/auto-peer/+merge/66913
<niemeyer> fwereade: Can you please have a look at the change and see what you think of it?
<fwereade> niemeyer: gladly
<niemeyer> fwereade: Then just approve/needs fixing/comment
<niemeyer> fwereade: The change is almost trivial
<niemeyer> fwereade: But I'm sure you'll have to step back a bit and understand the context
<niemeyer> fwereade: Check out the bug (linked from the merge request), and the documentation for more details about what was wrong
<fwereade> niemeyer: yep, it'll probably take me a little while
<fwereade> niemeyer: I'll hassle you if anything seems unduly confusing
<niemeyer> fwereade: Absolutely
<koolhead17> hello fwereade :)
<niemeyer> fwereade: I'm stepping out in a bit for lunch, but I'll be around soon, and others are around to give a hand too
<koolhead17> hi niemeyer kim0 
<niemeyer> koolhead17: Yo!
<kim0> koolhead17: hey :)
<koolhead17> niemeyer, finish up your lunch. 
<fwereade> koolhead17: hello :)
<kim0> fwereade: wow welcome :)
<niemeyer> koolhead17: I have to start it first! ;-)
<fwereade> kim0: thanks :)
<koolhead17> kim0, i have changed content of my pycon talk, it will mainly focus on cloud-init and ensemble now :)
<niemeyer> koolhead17: Wow, that's _awesome_
<koolhead17> niemeyer, :D
<kim0> koolhead17: that sounds good :) If that'll make you work on your first formula LOL
<koolhead17> kim0, yes sir!! 
<kim0> don't sir me man :)
<koolhead17> kim0, :)
<m_3> fwereade: welcome!
<koolhead17> hi m_3 
<fwereade> m_3: thanks :)
<m_3> fwereade: just started a couple of weeks ago... feel free to ping me and I'll share the little I know!
<m_3> fwereade: utlemming's new on the server team too
<fwereade> m_3: awesome, I'll be taking you up on that shortly I think :)
<hazmat> fwereade greetings and welcome
<fwereade> hazmat: and thanks also :)
<fwereade> hazmat: am I right in thinking you'll be in Miami next week?
<niemeyer> bcsaller: I'm assuming your review on ~hazmat/ensemble/status-w-peers was an Approve
<bcsaller> niemeyer: yes, sorry
<niemeyer> bcsaller: Hey!
<hazmat> fwereade, sadly not
<bcsaller> hi :)
<niemeyer> bcsaller: Wasn't sure if you were already around
<niemeyer> bcsaller: Morning :)
<niemeyer> bcsaller: Cool
<bcsaller> barely 
<fwereade> hazmat: ah, shame :( thought I saw you on one of the related pages
<bcsaller> trying to get some coffee down
<hazmat> fwereade, there's always a next time, hope your enjoy your first sprint, i'll be participating remotely
<fwereade> hazmat: cool, I look forward to it
 * niemeyer => lunch
<_mup_> ensemble/status-w-peers r265 committed by kapil.thangavelu@canonical.com
<_mup_> adjust comments per review.
<jcastro> default-instance-type: accepts what as a setting in environments.yaml? I am assuming the instance API name right? t1.micro, etc.
<_mup_> ensemble/trunk r265 committed by kapil.thangavelu@canonical.com
<_mup_> merge status-w-peer [r=bcsaller,niemeyer][f=804120]
<_mup_> Fixes a bug with ensemble status subcommand when displaying
<_mup_> peer relations.
<fwereade> ensemble in particular I guess ;)
<hazmat> jcastro, indeed just that re instance-type.. m1.large, t1.micro.. eg. ec2 api symbols for  instance-type
<hazmat> s/eg/ie
<jcastro> thanks, I'll add a note in the docs
<hazmat> a question for formula authors in the room, which directory would you want your hooks to run in.. The formula hooks directory, the formula directory, or the root of the unit directory (the formula for the unit lives at '/formula' under the unit root)?
<hazmat> jcastro, m_3, jimbaker SpamapS, fwereade, serge_ , kim0  ^
<fwereade> what else lives in the unit root at the moment?
<fwereade> the formula directory feels natural to me but I am basically 100% ignorant
<hazmat> fwereade, with lxc, the unit root will be the fs root "/" of the container
<fwereade> ah ok
<hazmat> at the moment, there's just the service unit log file and the formula directory in the root
<fwereade> ok, the hooks directory feels wrong because... well, it's the formula's interface, and if we run in that dir we encourage people to clutter it up with other stuff
<fwereade> I don't think I have a context to judge the distinction between running in the formula dir and the unit root
<hazmat> fwereade, agreed, authors can include arbitrary library or configuration stuff in the formula, its probably nicer to address that from the root, then including a '..' everywhere or cluttering up the hooks directory
<hazmat> er... s/root/ i mean the formula directory
<fwereade> yep
 * hazmat makes it so
 * niemeyer puts aside his bricklayer cap and goes back to software development
<m_3> hazmat: we've been starting from `pwd`
<m_3> hazmat: i.e., not assuming it's anywhere in particular
<m_3> hazmat: doesn't really matter with anything I've seen so far... pick one
<hazmat> m_3, pwd isn't reliable atm
<hazmat> it will be shortly
<m_3> hazmat: great, we're starting to do more library sorta stuff
<m_3> templates in separate files and such
<adam_g> hazmat: id personally prefer it run in formula hooks directory. in the formulas ive written, im usually keeping some file that contains common stuff that i source in, and currently need to do some extra steps to get the location of that file
<hazmat> adam_g, i think the notion is that what's in the hooks directory is the public interface of the formula, its fine to put that stuff in anywhere in the formula, you could put in the root of the formula, and still continue to source 'common.env' with the hook executing in the formula directory
<hazmat> but perhaps executing it in the hooks directory is the most obvious/intuitive understanding of where it will be executed
<niemeyer> adam_g: With the new approach it will be predictable as well
<niemeyer> adam_g: You'll just have to source hooks/... instead of ./...
<m_3> consistent good
<kim0> my first impression was to be under hooks/ with my templates ..etc .. but sourcing hooks/* is fine as well
<niemeyer> kim0: That's actually one of the reasons we're leaning towards the formula dir as cwd
<niemeyer> kim0: templates should be under templates/, rather than hooks/
<kim0> indeed cleaner +1
<hazmat> but also probably a faq
<hazmat> kim0, niemeyer how's this for an faq entry? http://paste.ubuntu.com/638989/
<niemeyer> hazmat: That's very nie
<niemeyer> nice
<kim0> hazmat: great indeed .. just a question do we have other "resources" other than templates
<kim0> when you mentioned templates should be under templates/ .. I didn't know formulas should have a templates/ directory ?!
<hazmat> kim0, libraries is the other one that comes to mind, but potentially also binaries
<kim0> wonder if formulas should have a directory structures, where authors know where to put stuff
<hazmat> kim0, they don't per spec have one, but formula authors are atm free to organize any resources needed by their hooks how they like
<kim0> ah got it
<hazmat> we might develop more best-practices with time as the formula collection goes, and have those enforced for an official formula distribution
<_mup_> ensemble/debug-with-formula-dir r262 committed by kapil.thangavelu@canonical.com
<_mup_> hooks executed in the formula directory, and a new faq entry regarding
<_mup_> ensemble/hooks-with-formula-dir r262 committed by kapil.thangavelu@canonical.com
<_mup_> merge the debug-with-formula-dir, forget this was setup as a bzr-pipeline
<m_3> kim0: I've been using hooks/templates/...
<m_3> but templates/ works fine too
<kim0> yeah it's just a convention 
<hazmat> simple deb packaging question, how do you go about creating empty directories (like /var/log/ensemble) for a package?
<niemeyer> hazmat: mkdir? :)
<hazmat> niemeyer, ah.. but the magic is where? in such a way that is compatible with the python-distutils packaging helper
<hazmat> niemeyer, just put in in preinst?
<niemeyer> hazmat: I'd say in the binary section of rules
<niemeyer> hazmat: Ah, no.. I'd include it in the package itself
<hazmat> niemeyer, it looks like dh_installdirs is the right place, spec'd in debian/package.dirs
<hazmat> or maybe its just debian/dirs
<niemeyer> hazmat: I belive it's <package>.dirs
<niemeyer> hazmat: I don't claim to know much, though :)
<hazmat> niemeyer, bcsaller regarding [0] on sans-ami branch.. i'm looking at switching ensemble-branch config option to ensemble-install-source with the value being either 'distro' (default for official releases in ubuntu releases), 'ppa:location', or 'lp:~xxx' 
<hazmat> so it would be one option to support default distro packages, ppa packages, or published bzr branches
<niemeyer> hazmat: Hmm
<niemeyer> hazmat: That sounds pretty good
<_mup_> txzookeeper/trunk r42 committed by kapil.foss@gmail.com
<_mup_> [trivial] fix changelog spacing for daily deb builds
<bcsaller> hazmat: nice
<hazmat> niemeyer, bcsaller the only caveat i can think of is if we want to test a branch of txzookeeper in the wild, then the lp location spec is incomplete
<hazmat> as it references the ensemble bzr branch
<dvestal> I really like the concept of Ensemble and am interested in potentially converting some of my AWS EC2 scripts to Ensemble formulas.  Are the principia formulas going to be the best resource for learning how to write them or is there other documentation for creating formulas?
<hazmat> dvestal, and the docs https://ensemble.ubuntu.com/docs/write-formula.html
<hazmat> also https://ensemble.ubuntu.com/docs/formula.html
<dvestal> hazmat: Thank you.
<m_3> dvestal: feel free to ping me with questions while you're writing formulas
<dvestal> Is the target of ensemble primarily for the creation / maintenance of machines as a whole?  More explanation following...
<hazmat> dvestal, its more about automating service deployment and orchestration, as a consequence it also manages machine deployments. regarding it maintenance, it currently handles those tasks as needed for inter service dependencies
<hazmat> particular tasks like service specific maintenance action, and overall maintenance of the machines aren't handled atm (for example managing security updates for packages on a machine).
<jcastro> I made a small doc fix to learn the workflow for ensemble, I don't think it's large enough to like file a bug, etc, so do I just leave it in the merge queue?
<kim0> you probably should still file a bug
<kim0> like "docs need updating to mention changing default instance type"
<jcastro> ok
<kim0> niemeyer: so we talked about how 12px font in the cloud portal is a little too small. I know understand the reason this font size was chosen is because it's the canonical design team's standard size set. As per the design guideline document (http://design.canonical.com/brand/D.%20Ubuntu%20Web%20Guidelines.pdf) page 16 .. they mention paragraphs should be font 12px
<niemeyer> RoAkSoAx: Welcome!
<RoAkSoAx> niemeyer: thank you!
<kim0> I also checked ubuntu.com like (http://www.ubuntu.com/business/cloud/overview) and it's 12px as well .. so I'm little reluctant to change that now
<_mup_> Bug #806638 was filed: Docs need updating to mention what it expects as a value for instance type <Ensemble:New> < https://launchpad.net/bugs/806638 >
<niemeyer> kim0: I'm not a designer.. all I know is that my eyes hurt when I have to read that text
<kim0> obino: o/
<jcastro> niemeyer: can you put me in ~ensemble so I can edit the wiki?
<niemeyer> jcastro: Of course
<jcastro> thanks!
<niemeyer> jcastro: Your Canonical email isn't found in Launchpad?
<jcastro> it should be, my username is "jorge", usually that's enough to add me to a team
<niemeyer> jcastro: Cool
<niemeyer> jcastro: That's done
<niemeyer> jcastro: Might be good to associate your Canonical address there too
<niemeyer> jcastro: I usually look for people by the email, to avoid mixing
<obino> hi kim0
<obino> got your request earlier: yes we would like to do something for cloud days :)
<dvestal> hazmat: Is the individual machine maintenance, etc. something that would be of an interest for the community?  Being able to orchestrate the specific machine maintenance items, as well as app upgrades (Tomcat war's, etc.) is something that I will need to do anyway.
<hazmat> jcastro, if it becomes a common enough activity, it might be worthwhile separating the docs into a separate repo
<kim0> obino: Awesome
<hazmat> it might already be worthwhile
<kim0> obino: I'd love it if you guys would add an event to the table (even if it says TBD) while you discuss what you want to present
<kim0> obino: if eucav3 is ready for a sneak peak, that'd be awesome indeed :)
<kim0> obino: if you think you want 2 sessions .. it's perfectly fine
<kim0> obino: thanks for your interest .. as usual ping me if you need anything
<hazmat> dvestal, its definitely something ensemble will need to grow into fulfill the existing feature set of traditional configuration management, how its implemented (reuse or build) is still tbd
<hazmat> dvestal, at the moment there is a notion that's being explored (in specification form) of co-locating service units. where one of those units could be some sort of policy formula.
<hazmat> dvestal, like installing a separate configuration management system or gui machine management facility, or monitoring, network volume, etc.
<hazmat> although the last is doable now with just relations..
#ubuntu-ensemble 2011-07-07
<m_3> bye baby
<m_3> ok, wrong focus... sorry
<dvestal> m_3 or hazmat: Earlier hazmat mentioned the exploration of the notion of co-locating service units in specification form.  Is that in reference to the upgrade, etc. documentation at https://ensemble.ubuntu.com/docs/ or are there other specs that I could find to see what's already been discussed about the topic?
<kim0> Morning folks
<hazmat> dvestal|away, re co-location not atm re docs
<niemeyer> Hey Ensemblers
<niemeyer> What are you Formulating today? :-)
<m_3> niemeyer: morning
<m_3> niemeyer: I'm debugging nfs this morning
<hazmat> niemeyer, ensemble cli plugins and an ide
<m_3> niemeyer: then planning on expanding node.js and starting on greenplum
<niemeyer> hazmat, m_3: Mornings
<niemeyer> m_3: Sweet
<niemeyer> m_3: Wow :)
<m_3> having a problem with empty endpoints in add-relation at the moment... probably just naming on my end
<hazmat> m_3, awesome, which greenplum product out of curiosity?
<m_3> hazmat: dunno... just popped on Mark's radar from conversations with Barclay's
<m_3> hazmat: that's the first task... ping me with suggestions
<niemeyer> m_3: What are you planning on node.js?
<m_3> niemeyer: well I have a node.js "app" formula written
<niemeyer> m_3: Interesting
<m_3> but it doesn't do much other than deploy a noe.js app from a repo
<niemeyer> m_3: What does it do to deploy the app?
<hazmat> m_3, so greenplum has a couple different product lines, they focus on offering analytics w/ ides, on top of warehouse dbs, be it their own version of pg, hadoop, hive.. the latter is typically an ingestion into the former, they have a fairly sophisticated graphical toolchain that the target analyst market expects
<hazmat> i saw a nice demo of it at a local hadoop meetup
<m_3> niemeyer: just installs nodejs packages and then pulls a node.js test app from github
<m_3> niemeyer: then spins it up
<hazmat> niemeyer, m3 i imagine its the same pattern for any language app server deployment, vcs config details, entry points, url mounting, dependency management, and db and proxy options
<m_3> hazmat: yup
<niemeyer> hazmat: Kind of
<niemeyer> hazmat: It's the same problem, for sure
<m_3> next is to figure out how to bind a db service and make it available to the node app
<niemeyer> hazmat: But I'm curious about how to leverage the environment we're offering
<niemeyer> hazmat: How to connect dbs etc
<m_3> there's a different option too that we should build
<niemeyer> hazmat: We can literally offer a PaaS for people, without having one
<hazmat> niemeyer, wsgi, rails, nodejs are pretty similiar aspects here, i think their different formulas, but really they have the same responsibilities and deploy patterns
<m_3> a node.js _hosting_ service
<m_3> similarly for rails
<hazmat> niemeyer, sure, that's the rather the point of ensemble.
<niemeyer> hazmat: Yep.. that's why I'm asking then.. how can we do more than simply "deploy code in a server" with a formula
<m_3> nodejitsu has an open source framework for hosting node
<m_3> I'll distinguish them by name for now
<m_3> 'rails-app' -vs- 'rails'... similarly for node
<hazmat> m_3, there are literally dozens of the platform specific paas options
<hazmat> although not many are opensource
<m_3> hazmat: yup
<m_3> hazmat: again, please feel free to suggest priorities
<m_3> as I understand, we need a base set of formulas
<m_3> that just keeps building
<hazmat> m_3, your priorities sound good (nodejs & greenplum), i'd like to see more scalable infrastructure components so glusterfs, cassandra, elasticsearch
<m_3> hazmat: yeah, me too
<m_3> hazmat: negronjl has cassandra working I think
<hazmat> hadoop would be interesting to look at as well, especially  in comparison to the python whirr implementation (their project specific ec2 scripts).. the biggest missing thing i see there is better volume management
<hazmat> yeah.. i should check out the  latest principia
<m_3> db service binding was easy in railsapp, there's pretty much a set way to do it in the app
<m_3> harder for node b/c it's a lot more unstructured
<m_3> I'll pick _a_ way to show as an example
<m_3> hadoop's in there too
<m_3> hazmat: I'll check out whirr
<hazmat> m_3, feel to have a formal entry point into the node app, like have a yaml file with this data
<m_3> hazmat: right, one I'll have to structure
<m_3> hazmat: maybe using mongo if the peer stuff's up
<hazmat> m_3, sounds good, which reminds me i should push my mongo formulas
<m_3> hazmat: yes please!
<m_3> yeah, I'm bad about that... I need to push it all to lp
<m_3> branch naming was preventing some of my pushes
<m_3> (i.e., my misunderstanding of branch naming rqmts <grin>)
<m_3> I'll rtfm
<m_3> hazmat: what sort of IDE stuff were you talking about earlier?
<hazmat> m_3, more of a web interface that would allow for interactive formula development with instrumentation
<m_3> hazmat: nic
<m_3> hazmat: I want blocks!  with wires between them!  <grin>
<hazmat> m_3, i think we'll need cli and server plugins first, but the ide itself would likely come after some sort of rest api server
<m_3> hazmat: gotcha
<m_3> hazmat: sounds perfect
<hazmat> m_3, mostly just brainstorming on ideas for the future, i think the cli-plugins are good  first start
<m_3> hazmat: yes
<m_3> hazmat: I was thinking something like a '--block' cli option would allow for formula testing
<m_3> hazmat: right now it'd have to have sleeps put in which is... not ideal
<m_3> but there're all sorts of cool cli ideas
<m_3> love the graphviz status btw
<m_3> dvestal: re your question from last night
<m_3> dvestal: I think the orig idea in colocated services was to provide for 'aspects' like monitoring and log consolidation services to be colocated with a primary service
<m_3> dvestal: think mysql with munin-node
<m_3> dvestal: I'm looking for other 'specs', but I think the ensemble docs are primary atm
<dvestal> m_3: Has there been any discussion on how to handle system configuration things outside of the "service" installation and configuration?
<m_3> dvestal: there are lots of feature requests too (https://bugs.launchpad.net/ensemble)
<m_3> dvestal: yes, lots... I'm not sure the current state of that though
<dvestal> m_3: For example, I have all of my servers authenticating with ldap instead of just the base AWS ubuntu user auth.
<alienpulse> kim0, there ?
<m_3> dvestal: that one might actually work as a service though... "authentication service"
<dvestal> m_3: I'll have to dig through the email list archives to try to catch up different things.
<alienpulse> dvestal, m_3 help plz :)
<niemeyer> dvestal: Oh, that's quite interesting actually
<niemeyer> dvestal: We have a few different things in progress ATM
<m_3> dvestal: there are things that don't fit as a "service"
<niemeyer> dvestal: and this one looks like the concept we're pushing in terms of "co-located" formulas
<m_3> alienpulse: what's up?
<niemeyer> dvestal: The idea is that certain formulas will be able to be deployed within the same container as another formulas
<niemeyer> another formula
<niemeyer> dvestal: To allow tweaking the environment in a sensible way
<niemeyer> dvestal: E.g. distributed logging, monitoring, etc
<niemeyer> dvestal: It sounds like what you're trying to do is related to this as well
<dvestal> niemeyer: Yeah.
<alienpulse> m_3, hello..I'm new with every thing like im clueless ,,in Launchpad ,, i'v my own bug but i don't now how to add formula and nodes and etc on it
<alienpulse> m_3, silly i know ,, but kim0 was my survivor in this kind of things 
<dvestal> niemeyer: I have several small things like that.  ldap auth is just the simplest to explain, and the most widely used.
<niemeyer> dvestal: Ok, do you have another example, just to ensure we're on the same page?
<m_3> alienpulse: no problem... wanna send me a url to the lp bug?
<alienpulse> okay
<alienpulse> m_3, https://bugs.launchpad.net/~abedeltawil
<dvestal> niemeyer: <this is not answering your question yet> What I do now is a scripted snapshotting of a machine so that I have an ami to stand up a new machine from. 
<alienpulse> m_3, so how can i remove Admin links drupal .. ? and how to add a new project to start working on it  and from where 
<niemeyer> dvestal: Ok, this is a fairly common approach, but at the same time it's the kind of trick we're trying to obsolete with Ensemble
<m_3> alienpulse: ah, I think I understand
<alienpulse> m_3, please advice ..
<dvestal> niemeyer:  Basically things like syslog -> mysql logging.  Auto-installation of wars into Tomcat and ears into Glassfish.
<dvestal> niemeyer:  Yeah,  I don't like doing it this way... which is how I found Ensemble. :)
<niemeyer> dvestal: These sound like good examples of co-located formulas
<niemeyer> dvestal: The idea will be quite simple, actually: you'll be able to describe certain relations as co-located, and Ensemble will know that the related services have to live together
<niemeyer> dvestal: They will still be normal relations, though
<niemeyer> dvestal: So, e.g.
<niemeyer> dvestal: Imagine a relation of type "tomcat-war"
<niemeyer> dvestal: WIth colocation on
<niemeyer> dvestal: Ensemble will put both of them in the same container, and will establish the relation
<niemeyer> dvestal: When tomcat notices the relation was established on war-relation-joined,
<niemeyer> dvestal: It does "relation-set war-directory=/path/to/wars"
<niemeyer> dvestal: You see where this is going?
<m_3> alienpulse: I can't remove the associated bug... just please ignore that as noise for now
<m_3> alienpulse: have you started on the formula itself?
<alienpulse> m_3, emm okay from where should i start if i wanna start a project for scratch 
<m_3> alienpulse: what I do to start is copy another formula
<alienpulse> m_3, i'v my bug but i don't now what should i do next ! :/
<m_3> alienpulse: I don't think there's a need to create a new launchpad project for a formula
<m_3> alienpulse: so the first thing to do is get a working directory where your formula will reside
<alienpulse> m_3, how that ?
<alienpulse> how can i do that ?
<m_3> alienpulse: what platform are you working on?
<alienpulse> u mean my project ?
<alienpulse> Drupal !
<niemeyer> dvestal fainted with the explanations.. sorry! ;-)
<m_3> alienpulse: the first thing we need to do is make a local repo for the formula
<m_3> alienpulse: have you used bazaar before?
<alienpulse> no :(
<alienpulse> let say i made a local repo 
<alienpulse> should i upload it ,, right ,, but where on the launchpad should i upload ?
<sladen> rumour has it via the grapevine that somebody has asking for input in doing an Ensemble Logo
<sladen> who would be the best person who might know more details?
<m_3> sladen: kim0 or jcastro maybe?
<m_3> alienpulse: it's not bad... something like http://paste.ubuntu.com/639527/
<jcastro> not me, I only sat in on the web meeting
<sladen> jcastro: that's more than I did.  kim0: any knowledge?
<alienpulse> m_3, and should i copy it and past it where !?
<m_3> alienpulse: so the formula development will be done entirely on your machine
<m_3> alienpulse: once you've gotten it going the way you like
<m_3> alienpulse: you can push your repo up to launchpad
<m_3> alienpulse: (can really push up at any time... the important point is that the development work is done on your local machine)
<m_3> alienpulse: have you installed ensemble on your machine?
<alienpulse> m_3, yeah sorry for that .
<m_3> alienpulse: np
<alienpulse> m_3, okay you where saying should the Formula development will be done on my laptop and with bzr is linked to my bug on launchpad ?
<dvestal> niemeyer:  Sorry for the delayed response.  I had to run upstairs for breakfast with the kids.  Yeah, that's a good explanation.  I like where it's going too.
<dvestal> ^ with the wife and kids...
<niemeyer> dvestal: No worries, was just kidding.. I know how that works too :-)
<m_3> alienpulse: yes, develop the formula first, then we can associate the repo you're using with the lp bug
<niemeyer> dvestal: Nice, so if you have any ideas to suggest or any questions, just pop up and we'll be happy to talk/fix/implement :)
<dvestal> niemeyer:  I definitely will.  This is something that I anticipate contributing to.  I'm sure that it will need to primarily be things that are specific needs of my company, but I talked to our owner about potentially working on this some as I am working toward automating our infrastructure and he seemed to be onboard with it.
<m_3> dvestal: awesome!
<kim0> ~>
<kim0> that was a hung ssh :)
<niemeyer> dvestal: That's great news indeed.. it's a good time to join in too, as you can help shaping the face of it to your needs
<kim0> alienpulse: sorry my irc was hung .. so are starting on the drupal formula
<kim0> alienpulse: did you read my formula writer's tutorial .. it should be easy to follow .. hey it's even for drupal too :)
<alienpulse> kim0, no :( i want to did u blog it ?
<alienpulse> ill read it from your blogs 
 * kim0 getting link
<kim0> alienpulse: just this one https://ensemble.ubuntu.com/docs/write-formula.html
<alienpulse> kim0, okay i was saying i dont know from where should i start ,, i dont want begin wrong ..
<kim0> alienpulse: so basically, I'd start by reading that tutorial, and try to execute it step by step till you have drupal running on ec2
<kim0> alienpulse: once it's running, you can start hacking on it 
<kim0> to make a real formula out of it
<alienpulse> and then i upload it to launchpad ?
<kim0> alienpulse: no need for that .. you can create the formula on localhost .. and use it to deploy .. 
<alienpulse> im almost done with reading the ensemble tutorial ..
<kim0> alienpulse: putting it on LP .. is like the final step
<alienpulse> kim0, aha
<alienpulse> kim0, okay ,,im getting it all now 
<alienpulse> 1st step ..run drupal on ec2 
<koolhead11> okey guys i have filled  a bug finally dbconfig-common and phpmyadin
<kim0> koolhead11: woohoo \o/
<koolhead11> the preseed made me cry
<alienpulse> 2nd step hacking it .. 3rd step create a formula on my machine ,, and finally ill upload it to LP
<alienpulse> kim0, miss any thing ?!
<kim0> let me rephrase .. this is how I would do it
<kim0> 1- copy formula from tutorial into files
<kim0> 2- run it as is, launch drupal on ec2 with it
<kim0> 3- understand what is going on
<kim0> 4- start changing the formula
<kim0> 5- upload to launchpad, share your formula with the world
<kim0> alienpulse: that's all
<alienpulse> kim0, you're a bless from heaven !
<kim0> alienpulse: You know what .. even easier .. I remember now I have pushed my drupal to LP
<kim0> alienpulse: so you can skip step 1 ..  and instead do â  bzr branch lp:~kim0/+junk/drupal
<alienpulse> kim0, okay 
<niemeyer> fwereade: Would you mind to review this merge proposal when you have a moment: https://code.launchpad.net/~hazmat/ensemble/hooks-with-formula-dir
<fwereade> niemeyer: a pleasure
<niemeyer> fwereade: Thanks!
<kim0> sladen: o/
<kim0> sladen: so yeah, a community member is interested in working on a logo+mascot for Ensemble
<kim0> sladen: he does pencil concepts. He's starting to send me concepts drawings, but I'm really not made to judege these things :) 
<kim0> sladen: can I link you two up ? so you start hacking those graphics together /
 * koolhead11 is allowed to ask dumb/stupid questions :P
<niemeyer> hazmat: I don't think the env var should be yanked..  please see the latest comment there
<sladen> kim0: -> sladen@c.com
<niemeyer> hazmat: Just pointing out here so you don't work on this before seeing the comment
<niemeyer> koolhead11: Everyone is! ;_)
<kim0> sladen: awesome .. linking :) thanks man
<koolhead11> niemeyer, :)
<hazmat> niemeyer, k, thanks
<hazmat> niemeyer, could you clarify what you meant by [3]
<niemeyer> hazmat: Just mentioning what is $CWD in the docs
<niemeyer> hazmat: Next to the hooks themselves
<hazmat> niemeyer, ah.. sounds good
<fwereade> couple of questions about what directory we run in
<fwereade> what makes the unit dir better than the formula dir?
<fwereade> I haven't been able to see any serious distinction in this context but presumably there's some reason to prefer the unit dir
<fwereade> hazmat, niemeyer: the docs currently say we run in the formula dir, but it seems we run in the unit dir
<niemeyer> fwereade: I think we're using the formula dir?
<hazmat> fwereade, it does run in the formula dir
<hazmat> it just uses the unit dir as a base reference point
<fwereade> hm, let me reread the code
<fwereade> looked like it was using the unit dir
<niemeyer> In fact, there's an important reason for it to be the formula dir now.. we'll soon have co-located formulas in the same unit dir
<hazmat> fwereade, its at the bottom of invoker where it spawns the process, and there's a corresponding test
<fwereade> hazmat: where it calls with "hook_proto, hook, [hook], env, self._unit_path)"?
<fwereade> hazmat oh damn, wrong rev
<hazmat> fwereade, are you on the right branch? latest rev is 262 of lp:~hazmat/ensemble/hooks-with-formula-dir
<hazmat> cool
<fwereade> forget I said anything 
<fwereade> :)
<hazmat> niemeyer, the implementation of co-location will need to modify the impl as needed. the formula location will no longer just be /formula
<niemeyer> hazmat: Right, that was my point
 * niemeyer => lunch
<kim0> alienpulse: just noticed m_3 did a drupal formula on https://code.launchpad.net/principia .. it's ok, you can still study how things work, then improve on Mark's formula
<alienpulse> kim0, im working on your now ..
<sladen> kim0: can you forward me the actual pencil sketches too
<alienpulse> kim0, but thanks any way
<kim0> alienpulse: yeah don't let that stop you .. you need to understand ensemble .. then you can hack any formula :)
<kim0> sladen: yep .. getting em
<kim0> sladen: emailed 
<hazmat> niemeyer, re formula upgrade and cleanup of files, its not quite read only.. but point taken.. if we assume namespacing in the future, fhs locations in the unit would need qualifications as a result, we can nominate a particular formula directory as inviolate by ensemble and suitable for variable storage (aka /var) or store the manifest at deploy time to compare to the delta, to just remove old formula files but not files created by the formula... 
<sladen> kim0: ta.  Do you have an full copy of the mailing message  https://lists.ubuntu.com/archives/ensemble/2011-June/000199.html  available?
<sladen> kim0: the one in the archive is missing the attachments
<kim0> sladen: it was only links .. no attachments
<kim0> sladen: I emailed you the links as well
<sladen> kim0: nod
<sladen> niemeyer: would you be able to follow-up to  https://bugs.launchpad.net/ubuntu-branding/+bug/807100 ("Ensemble logo (ensemble.ubuntu.com)") with your breathing aperatus quote
<_mup_> Bug #807100: Ensemble logo (ensemble.ubuntu.com) <ubuntu-branding:New> < https://launchpad.net/bugs/807100 >
<niemeyer> sladen: Hey there
<niemeyer> sladen: Sure, what's up?
<niemeyer> hazmat: Yeah, storing old manifest + new manifest + diff and remove might be the most expected outcome
<niemeyer> hazmat: I actually like a bit the var approach too
<niemeyer> hazmat: Makes it easy to move things around
<niemeyer> hazmat: I'm a bit concerned that people won't figure it out until too late
<sladen> niemeyer: Ensemble logo tracking bug
<sladen> niemeyer: I have a quote that kim0 forwarded me, but I'd rather if you attributed and added it yourself :-)
<niemeyer> sladen: Done
<niemeyer> Actually, there's another thread too
<niemeyer> hazmat: Looks like a recent change has broken txzookeeper: https://launchpadlibrarian.net/74683481/buildlog.txt.gz
<sladen> niemeyer: did you mean to copy and re-paste 80% of the content from the bug introduction back into the bug report :-)
<hazmat> niemeyer, yeah.. i'm not sure what the issue is the changelog format looks fine
<hazmat> niemeyer, is there a static check tool ala lintian for checking deb directories
<alienpulse> kim0, how can i remove one of my affects !?
<kim0> alienpulse: what do you mean by affects
<alienpulse> kim0, i want to remove admin links Drupal module
<kim0> alienpulse: don't install it ?
<alienpulse> kim0, in my bug i just assigned by mistake :/
<kim0> alienpulse: give me the bug url
<alienpulse> kim0, https://bugs.launchpad.net/principia/+bug/805955
<_mup_> Bug #805955: Formula needed: Drupal <new-formula> <Admin links Drupal module:New> <Principia Ensemble:In Progress by abedeltawil> < https://launchpad.net/bugs/805955 >
<niemeyer> sladen: Sorry, I didn't notice you had already added one of the links
<niemeyer> sladen: It's not clear to me how can I help there?
<niemeyer> sladen: I've already commented on these graphics in the mailing list thread which I linked there
<kim0> alienpulse: I just set it to invalid on that project .. probably good enough
<niemeyer> sladen, kim0: What sort of action is the bug oriented towards?
<alienpulse> kim0, emm okay thanks ..
<niemeyer> hazmat: There is
<niemeyer> Let me try to find the name
<m_3> kim0, alienpulse: I tried to remove this morning with no luck
<kim0> yeah, I'm no LP expert either :)
<sladen> niemeyer: -> see your private message
<sladen> niemeyer: tracking the logo.  I heard on the grape vine that somebody/something wanted help.  It's now documenteed in one place, with everything that has been said
<niemeyer> hazmat: lintian - Debian package checker
<niemeyer> hazmat: I suspect the date format might be the issue, but not sure
<niemeyer> hazmat: date -R is the standard
 * niemeyer playing with glance
<niemeyer> fwereade: That may be our S3 service for next week ^
<pindonga> SpamapS, you were working on the lxc branch for ensemble, right?
<niemeyer> SpamapS: We should talk about the LXC work, actually
<niemeyer> SpamapS: The first step is probably to put a specification in place detailing what we're going after
<niemeyer> and the impact in the several areas
<m_3> niemeyer: +1 for faster formula development/debugging
<m_3> jcastro: can I the deck you used for your ensemble lightning talk?
<adam_g> hazmat: ping
<adam_g> is bootstrapping stock oneiric ec2 AMIs expected to work?
<m_3> adam_g: I was wondering the same thing when pointing to a euca cloud
<m_3> adam_g: I don't think so... think zk's gotta be there at least
<m_3> adam_g: smoser helped SpamapS bundle up something minimal for lxc
<m_3> adam_g: if he's around
<adam_g> zk looks to be installed via cloud-init
<adam_g> so does the rest of the bootstrapping
<niemeyer> adam_g: I haven't tested this myself.. we've been using Natty images just because it makes it easier to have it on both sides
<niemeyer> adam_g: That said, we should all switch to Oneiric now that we're packing it in 11.10
<niemeyer> adam_g: and I can't see why it'd not owrk
<adam_g> niemeyer: yeah.. i was using the default natty AMIs as well until the archive mirror in us-east-1 decided to fall over
<niemeyer> wokr
<adam_g> W: Failed to fetch bzip2:/var/lib/apt/lists/partial/us-east-1.ec2.archive.ubuntu.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages  Hash Sum mismatch
<niemeyer> Ugh
<Ryan_Lane> howdy
<kim0> Ryan_Lane: o/
<kim0> niemeyer: o/
<kim0> Ryan is from the mediawiki ops team 
<Ryan_Lane> kim0: so, ensemble seems like a really great way of pushing changes into an environment that isn't fully formed, and doesn't have something like chef/puppet
<Ryan_Lane> but it doesn't address long term management of systems
<kim0> MediaWiki has a staging cluster, and I was pinging Ryan to test whether we can get some interest to get them to use Ensemble .. Ryan has some questions which I thought we could better answer here
<Ryan_Lane> the route we are going is to open our puppet repository to everyone
<adam_g> niemeyer: the run_cmd script in cloud-init for bootstrap doesn't succeed on oneiric. and i cant see how it would work on natty, either. are the default AMIs launched by ensemble stock?
<Ryan_Lane> and treat our ops like we treat mediawiki
<kim0> SpamapS: hazmat m_3 your thoughts are all welcome 
<Ryan_Lane> with code review on the puppet manifests, files, and templates
<Ryan_Lane> which will then let us directly deploy these changes to our production hardware cluster
<Ryan_Lane> which isn't cloud based ;)
<kim0> Ryan_Lane: let me try to answer the points you've raised
<kim0> Ryan_Lane: so Ensemble is not really about cloud only 
<Ryan_Lane> lemme bring up the docs so that I can refresh my knowledge of it
<kim0> drivers are being written today to support deploying to LXC containers, as well as hardware machinse and ofcourse openstack private clouds among others
<kim0> ec2 was just the fastest way to get started
<kim0> Ryan_Lane: regarding the first concern, Ensemble operates at a higher level that puppet or chef
<kim0> in the sense that it operates at a "service" level
<kim0> for configuration file management .. you _could_ still use puppet if you really wanted to
<kim0> or you could use something else
<kim0> Ensemble is well suited to long term management of instances
<kim0> it simply focuses more on the concept of deployable "services" over machines
<kim0> umm .. regarding sharing your deployment recipes with the world
<kim0> that is exactly what we're after
<Ryan_Lane> yeah. I'm not concerned about that part
<kim0> We're creating the principia project
<kim0> which should cover hopefully all FOSS soon :)
<kim0> if you have some more questions or exact use cases .. I'd love to hear them as well
<m_3> Ryan_Lane: the thing that really closed ensemble for me was the relationships
<Ryan_Lane> so run me through a basic idea of installing a new mediawiki instance inside of an already configured mediawiki cluster
<kim0> ofc Ensemble is rapidly evolving .. if there's a use case we don't currently support .. things can definitely be tuned
<m_3> Ryan_Lane: ensemble provides events when service relationships are changed
<m_3> Ryan_Lane: take a look at the mediawiki formula hook for db-relation-joined (maybe db-relation-changed)
<Ryan_Lane> puppet/chef does the same
<m_3> Ryan_Lane: the way the events go out it allows for _really_ easy horiz scaling
<kim0> Ryan_Lane: is the rest of the cluster under ensemble management ?
<Ryan_Lane> let's assume I have a memcached cluster, an apache cluster, a database cluster, a squid cluster, a varnish cluster (for bits), a swift cluster (for media), how will ensemble create just a wiki in that?
<Ryan_Lane> without deploying new nodes
<Ryan_Lane> or avoiding too much complexity...
<Ryan_Lane> an apache cluster, and a database cluster
<kim0> are you saying those resources are not under Ensemble's management, or are they
<Ryan_Lane> I get the idea of adding new nodes to the cluster, that's easy enough :)
<Ryan_Lane> under ensemble
<kim0> Very well
<Ryan_Lane> created via the screencast example
<kim0> m_3: care to explain how that works :)
<Ryan_Lane> (this is a specific use case that I need, and don't have a solution for yet)
<kim0> Ryan_Lane: Would deploying a service inside a new LXC container be acceptable for this use case ?
<Ryan_Lane> no
<m_3> Ryan_Lane: yup, creating a pastebin of the hooks we have implemented for mediawiki
<Ryan_Lane> it can't add new instances
<m_3> Ryan_Lane: so here's the metatdata.yaml file for the formula http://pastebin.com/h8HYzfQ5
<m_3> you can see the attachment with mysql, memcache, munin, and the "website" relation
 * Ryan_Lane nods
<m_3> the "Website" relation can be directly accessed or consumed in a relation with a reverseproxy
<m_3> or load balancer
<Ryan_Lane> what's the benefit over puppet?
<m_3> so there are hooks created for each relation
<Ryan_Lane> (or chef)
<m_3> longer question to answer... lots on the topic though
<m_3> lemme get you a list of the hooks here
<m_3> http://pastebin.com/aL36P6A5
<m_3> the basic workflow is as follows
<kim0> Most people find it an eye openeronce they take some time to understand how Ensemble formulas work under the hood
 * kim0 makes way for m_3 
<m_3> http://pastebin.com/i2CPibe1
<m_3> kim0: please participate <grin>
<m_3> the magic happens in the relation
<kim0> :) 
<m_3> the fact that this is the db relation is just an example
<m_3> you'd do the same for memcache, reverseproxy, etc
<m_3> so your "stack" would be more complicated than just the script I pasted
<m_3> but not _much_ more complicated
<m_3> the way the service relation events are fired
<m_3> and organized
<kim0> I think a main benefit, is that Ensemble is a living system. You are no longer thinking on the level of machines and config files. You're thinking in terms of intelligent "services" that know how to respond to the surrounding environment
<m_3> allows you to scale horiz by just 'ensemble add-unit mediawiki'
<m_3> rinse and repeat
<kim0> and configure themselves, consuming other services, and offering their service to other nodes
<m_3> the existing relations between mediawiki and the handful of other services are automatically fired for each new mediawiki unit
<kim0> that simplicity of scaling, is a direct result of the fact that services embed the intelligence and know how to react to events
<adam_g> Ryan_Lane: to answer your question. AFAIK: as it stands right now, when you deploy a new service via ensemble, you're deploying a new instance to host that service. it takes an elastic, cloudy POV that instances are expendable containers of services that will come and go throughout the lifecycle of the service as a whole
<Ryan_Lane> if I change a service, do the current instances update to reflect that?
<m_3> we're working on colocated services within an instance
<m_3> Ryan_Lane: that's formula version
<m_3> there's upgrade-formula
<kim0> Ryan_Lane: yes, you can upgrade formulas on the fly
<m_3> and the ability to do rolling upgrades
<m_3> issues to be worked out here though!
<Ryan_Lane> heh
<Ryan_Lane> so, when upgrading instances, is there some group-based update process?
<m_3> rolling upgrades mean different things to different apps, so that's still in dev/testing
<Ryan_Lane> or do you need to do it per node?
<m_3> lemme show you status output
<m_3> you can address individual service units (instances)
<m_3> but I'm not sure in a upgrade-formula scenario what the options are
<m_3> (just started working on it a couple of weeks ago)
<Ryan_Lane> our staging environment should reflect our production environment as much as possible.
<Ryan_Lane> including how we do ops
<m_3> Ryan_Lane: understood
<kim0> again Ensemble is rapidly evolving, so it can adapt to needs
<m_3> Ryan_Lane: always particularly fun with a cms!  <grin>
<Ryan_Lane> on our production environment, we may have one box doing a couple things
<Ryan_Lane> our software is deployed separately from our os related thingd
<kim0> Ryan_Lane: and it's also a good ground for experimenting with future ops technology stacks right 
<Ryan_Lane> yeah. if you guys would like to work in the environment, we can do that
<m_3> Ryan_Lane: I'd love to talk maybe higher bandwidth or look at docs to learn more about your infra
<Ryan_Lane> for instance, I can give you guys a project where you can try to model our production environment in ensemble
<m_3> yup
<kim0> I think that would be very good
<Ryan_Lane> I think that would be great for both of us
 * kim0 nods
<Ryan_Lane> I like some of the concepts, but it would be very difficult for us to use it now
<m_3> gotta run... giving my first ensemble demo at a local LUG
<m_3> nice to meet you Ryan!
<Ryan_Lane> m_3: you too
<kim0> m_3: thanks for the help
<kim0> Ryan_Lane: Awesome .. we'll both learn a lot by working on this excercise 
<Ryan_Lane> it would be great to see you guys make a similar environment to ours, and show us how we could use it
<Ryan_Lane> and we can work with you guys to show where things are going to run into problems
<kim0> that will surely be needed 
<kim0> great stuff
<kim0> Ryan_Lane: so how do you suggest we get started
<Ryan_Lane> well, I don't have the cluster fully up yet
<kim0> Ryan_Lane: it's openstack right ?
<Ryan_Lane> also, there's a couple things that I'll likely need to add to openstack for this to properly integrate
<Ryan_Lane> I think someone is working on it right now
<Ryan_Lane> the mediawiki extension I wrote adds DNS entries in LDAP when instances are created
<Ryan_Lane> openstack doesn't have support for adding DNS entries right now
<Ryan_Lane> for ensemble to work properly, it needs that ;)
<kim0> well I'm sure we can beat any technical issues together :)
<Ryan_Lane> yeah
<Ryan_Lane> hmm
<kim0> perhaps we could even do the work on ec2 for now
<Ryan_Lane> I may actually be able to give you guys access earlier than others, since you don't need the puppet stuff working
<kim0> since that's the most mature provider
<m_3> exit
<kim0> :)
<kim0> Ryan_Lane: yeah we don't
<m_3> sorry, wrong focus... doh
<kim0> hehe
<Ryan_Lane> could start, on ec2, yeah. would be good to move you guys over as soon as possible though
<kim0> openstack exposes the ec2 api as well, so it should just work
<Ryan_Lane> yeah. I'm using ec2 api in my extension too
<Ryan_Lane> you guys will just need the DNS stuff.
<kim0> might take some hammering .. but that's what we're good at :)
<kim0> Ryan_Lane: so you'll give us access to the cluster, how will we know how your current infrastructure needs to be modelled ?
<kim0> like more info on what needs to be done 
<Ryan_Lane> that's the hard part :)
<kim0> :)
<Ryan_Lane> that may be something you need the puppet repo for
<Ryan_Lane> we have a lot of documentation on wikitech.wikimedia.org
<Ryan_Lane> but it's not a step-by-step guide
<Ryan_Lane> we also have configuration files at noc.wikimedia.org
<Ryan_Lane> for mediawiki/apache
<kim0> Ryan_Lane: In the mediawiki community, would there be ops volunteers interested in helping us get this working ?
<Ryan_Lane> hmm. maybe.
<kim0> Cool .. we'll need to broadcast some messages then
<Ryan_Lane> we are light on ops volunteers. that's one of my goals of the labs project :)
<kim0> hehe yeah I know 
<kim0> very well
<Ryan_Lane> I can talk to you guys about stuff as you work on it
<Ryan_Lane> I can give you an overview of how it works
<Ryan_Lane> you guys located in US? in SF?
<kim0> we're everywhere
<kim0> :)
<Ryan_Lane> heh
<Ryan_Lane> yeah. us too
<kim0> some should be close to that .. just no idea who
<Ryan_Lane> we are going to be doing a hack-a-thon in New Orleans at some point soon
<Ryan_Lane> it'll be labs focused
<Ryan_Lane> so that would be a great time to work together if you guys are interested and can come
<kim0> Ryan_Lane: I think it would be beneficial to have you on the Ensemble mailing list: http://lists.ubuntu.com/mailman/listinfo/Ensemble
<Ryan_Lane> otherwise, I can work with you guys online
<kim0> sure the hack-athon thing can work too
<kim0> I think I'll be sending a summary of this talk to the list, and it would be great to have you over commenting on how to move forward ..etc
<kim0> Ryan_Lane: sounds like a plan ?
<Ryan_Lane> yeah. sounds good
<kim0> Ryan_Lane: Great! thanks a lot Ryan .. I really appreciate your time
<Ryan_Lane> you're welcome. I look forward to working on this with you guys :)
<kim0> Awesome, I am very excited as well :)
<kim0> We all think Ensemble is bringing a big change, and it rocks to be part of it
<kim0> woohoo \o/
<kim0> Okie .. it's getting too late for me now, so I'll catch you guys later
<Ryan_Lane> kim0: later
<kim0> Ryan_Lane: see ya
<_mup_> Bug #807281 was filed: Cannot bootstrap stock ubuntu AMIs <Ensemble:New> < https://launchpad.net/bugs/807281 >
<hazmat> adam_g, not till the sans-ami branch lands, and then it ensemble will work with any cloud-init enabled image ( although we might be dependent on a 10.10 version of cloud-init)
#ubuntu-ensemble 2011-07-08
<hazmat> adam_g, there are also images available in other regions if you'd prefer till such time
<adam_g> hazmat: hmm, well the us-east-1 archive mirror seems to be broken ATM. ive hacked providers/ec2/launch.py a bit so i can bootstrap oneiric from trunk
<adam_g> mirror for natty, that is
<hazmat> g'monring
<niemeyer> Bom dias
<hazmat> niemeyer, greetings
<niemeyer> hazmat: Hey
<hazmat> niemeyer, i think i'm going to switch the sans-ami branch back to what you approved, just addressing the review comments, and put the work for ben's in a separate branch, i'd like to get this feature in this week.
<hazmat> bcsaller, does that sound good?
<niemeyer> hazmat: Sounds good.  How far is it from getting the next few things in?
<niemeyer> hazmat: This will be important for us next week too
<niemeyer> hazmat: I mean, the actual branch, not the further chnages
<hazmat> niemeyer, i'd like to land it today
<hazmat> niemeyer, maybe 20m for the changes from your review
<niemeyer> hazmat: Ah, cool
<hazmat> the additional distro selection, ensemble source selection, i'd think early next week, in review monday
<niemeyer> hazmat: What's the distro selection about?
<niemeyer> hazmat: That wasn't entirely clear for me, because there are a few different perspectives on it
<hazmat> niemeyer, being able to select what release of ubuntu you want to use for your image
<niemeyer> hazmat: Hmm
<hazmat> its an image selection filter
<niemeyer> hazmat: Do we really want that?
<niemeyer> Hmm
<hazmat> defaults to using what's current on the machine running the admin tools, but can be selected per environment 
<hazmat> niemeyer, i think so, the caveat is we probably need a 10.10 selection or higher atm due to cloud-init feature changes (workarounds for which we have removed)
<niemeyer> hazmat: This isn't clear to me
<niemeyer> hazmat: Formulas target a specific release
<niemeyer> hazmat: We shouldnt' be selecting the machine.. we should select the formula
<hazmat> i see what you mean, sounds good, we'll need to capture this release metadata for image selection, i assume we'll be deploying formulas from multiple releases in an environment.
<niemeyer> hazmat: Yeah, we'll have to handle multiple releases
<niemeyer> hazmat: The metadata is in the repository too
<niemeyer> hazmat: Not sure about the proper way to do it
<hazmat> niemeyer, same mechanism i was about to expose to the end user
<niemeyer> hazmat: How do you mean?
<bcsaller> hazmat: I am fine with the other part of that branch coming later
<hazmat> niemeyer, image selection encoded as formula metadata, a machine placement alg, using the formula metadata, via kw arg/image/machine type selection on the launch machine api of the provider interface, the low level is basically already done for ec2. it will be interesting to see what comes out of the sprint next week. we really need some cross-provider way of specing machine size.
<hazmat> i'm curious if that sort of info is exposed via the orchestra api
<hazmat> er. cobbler
<niemeyer> Agreed.. but we need to sort out the distro selection first
<hazmat> bcsaller, cool, could you update the merge proposal to reflect that
<bcsaller> sure
<hazmat> niemeyer, regarding upgrading formulas and removing files not in the new formula, for local state do we want to spec a location within the formula that we ignore, if we allow for formula contained state. 
<niemeyer> hazmat: It feels wise to do so
<hazmat> ie. ignore '/var' dir in a formula, before co-located formulas, i'd have said just stick it in the fhs somewhere.. its not clear that doesn't still work well but if we want to preserve the option
<hazmat> of self-contained state
<niemeyer> hazmat: Right
<niemeyer> hazmat: There's another question regarding whether we should be killing things arbitrarily
<niemeyer> hazmat: On upgrades
<hazmat> realistically anything there installing is already on the machine. if we removal of a co-location, we'll need an uninstall hook perhaps.
<niemeyer> hazmat: It feels like we should be more precise and only remove the files that disappeared from one version to the next
<hazmat> s/we removal/we support removal
<hazmat> niemeyer, we don't kill things for upgrades.. not sure what you mean?
<hazmat> ah.. files
<hazmat> niemeyer, yeah.. that supposes a notion of an install time manifest
<hazmat> the formula dir manifest implementation is a dir walk
<hazmat> ie always current
<niemeyer> hazmat: We might even pick it out of the bundle
<hazmat> niemeyer, sounds good
<niemeyer> hazmat: The zip actually has the manifest already
<niemeyer> hazmat: So it's just a matter of exploring it
<hazmat> niemeyer, yup reusing the  local cache of downloaded formulas would simplify
<niemeyer> hazmat: Hmm
<hazmat> actually that will make it easier to support the upgrading against the same version
<niemeyer> hazmat: I don't think we can trust the cache to be there, otherwise it stops being a cache
<niemeyer> fwereade: Yo
<fwereade> niemeyer, heyhey
<hazmat> niemeyer, hmm. make sense, so we do need an install manifest and checksum recorded out of cache
<fwereade> that was an unexpectedly gruelling many hours
<fwereade> I'd forgotten what london is like
<niemeyer> hazmat: Yeah, I think so
<fwereade> but I think I'm actually ready to fly tomorrow
<hazmat> fwereade, greetings
<fwereade> heya hazmat
<hazmat> fwereade, out of curiosity where are you normally based out of?
<fwereade> hazmat, malta
<hazmat> fwereade, nice
<fwereade> yeah, it's lovely
<hazmat> fwereade, care to help organize a sprint? ;-)
<fwereade> whatever I can do to hlpe :)
<fwereade> I can even spell things sometimes
<fwereade> what do you need from me?
<hazmat> fwereade, not much, but its not in the cards atm, we've got a couple sprint scheduled for the next few months.
<fwereade> hazmat: cool, as long as it's not for Monday :p
<niemeyer> hazmat: So, just to make sure, is the branch going in soonish?
<niemeyer> hazmat: Just asking because Dustin mailed me about it
<fwereade> hazmat: but in principle I'm happy to help however I can, just let me know
<hazmat> niemeyer, i hope to have all my approved branches in by end of day
<hazmat> which should solve most of the immediate issues jcastro brought up last week
<hazmat> niemeyer, what do you think about closing out this milestone and setting up a new one
<niemeyer> hazmat: I'd like to close it in the sprint in Austin, as originally planned
<hazmat> niemeyer, sounds good
<niemeyer> hazmat: There is still very important work that didn't land
 * niemeyer looks at bcsaller
 * bcsaller nods
<niemeyer> hazmat: I'd also hope that simplistic pieces of the security work see the light of the day there
<hazmat> niemeyer, hmm... i could start, but the spec still needs discussion
<hazmat> niemeyer, what do you think about team conf call to discuss
<hazmat> security
<niemeyer> hazmat: Sounds good
<niemeyer> I need lunch now, though
<niemeyer> hazmat: Please feel free to book a time for us today
<hazmat> niemeyer, sounds good cheers
<m_3> Aaargh.... launchpad!
<m_3> SpamapS: I need some repo naming help when you get back
<kim0> o/ I'd like to see many Enmseble related sessions in Ubuntu cloud days â Please add a session to https://wiki.ubuntu.com/UbuntuCloudDays/Timetable
<SpamapS> m_3: pong, got a few minutes now, sup?
<niemeyer> SpamapS: Hey man
<SpamapS> niemeyer: greetings earthling. :)
<niemeyer> SpamapS: Re. the LXC work, we need a specification that describes what is the intended operating semantics and the way we want it to impact the several areas
<niemeyer> SpamapS: Are you interested in championing it?
<SpamapS> niemeyer: yeah thats a good idea. I already threw it together in my head. ;) just need to spit it out in rst :)
<niemeyer> SpamapS: ... and get agreements :)
<SpamapS> niemeyer: Its impact on anything except itself is *minimal*, and mostly involves fixing bugs
<niemeyer> SpamapS: That's awesome
<SpamapS> Though eventually I'm sure the LXCControl class I created would be useful for the separation work that is desired in the unit agents.
<SpamapS> niemeyer: in fact I've gotten it all separated into branches which I'll propose for merging next week along w/ the bugs that are fixed
<niemeyer> SpamapS: I'd appreciate reviewing the high-level problem first
<SpamapS> niemeyer: meanwhile the spec for the provider and such can be subject to review separately
<niemeyer> SpamapS: As we discussed over the sprint last week, there are a few different ways it can be made to work
<SpamapS> niemeyer: the only bugs are things that were implemented in the ec2 provider that should be common to all providers.
<niemeyer> SpamapS: So we should start by agreeing on which way we want it to
<niemeyer> SpamapS: That's certainly neat no matter what
<niemeyer> SpamapS: In fact, we'll have to do exactly that next week
<niemeyer> SpamapS: So if you have well tested code splitting it off, I'd _love_ to have those available next week
<niemeyer> SpamapS: Otherwise we'll be redoing that work again
<SpamapS> niemeyer: sure, I don't have strong feelings about the ways it should work internaly. I just want to have a provider that works w/o amazon, and, preferrably, one that works w/o any net connection at all.
<niemeyer> SpamapS: Right, +1 on that
<niemeyer> SpamapS: The issue is not just internally, though
<niemeyer> SpamapS: as we talked, the initial approach was to have a LXC representing a machine
<niemeyer> SpamapS: We really want to have LXC representing a unit
<SpamapS> niemeyer: yes, I'm perfectly fine with deprecating what I've already done once that work is ready.
<niemeyer> SpamapS: Well, rather than deprecating what you do, I'd prefer to integrate what you do as a goal. :-)
<niemeyer> SpamapS: Which is why, in general, it's important to have a spec and to debate a bit before actually diving in
<SpamapS> niemeyer: that is, IMO, much more important to long term goals, and so, should be done more carefully. This provider is the way forward w/o touching ensemble's core functionality.
<SpamapS> Its my little runt.. cute, but I don't expect it will live as long as its stronger brothers. :)
<SpamapS> niemeyer: anyway, agreed, no more working on it until there's a spec and a gooddiscussion
<SpamapS> niemeyer: anyway I need to run out now. Are any of the ensemble team going to attend the LXC sprint in Austin? I'll be there.
 * SpamapS will read the answer to that later.. 
<niemeyer> SpamapS: Yeah, we'll be there
<SpamapS> ciao!
<niemeyer> SpamapS: Cheers!
<m_3> SpamapS: hey man
<m_3> SpamapS: just having problems pushing stuff to lp
<m_3> SpamapS: I think I've gotten some things pushed into other places coincidentally b/c there are already packages in oneiric with those names
<m_3> SpamapS: but then I saw jamespage's pushes and totally lost any understanding I thought I'd gained
<m_3> lp:~jenkins-formula-dev/principia/oneiric/ensemble/jenkins-slave
<m_3> that one works in both 'principia' and 'ensemble' yet no 'trunk'
<m_3> I'll just put it in +junk for now
<m_3> no biggie
<niemeyer> Borked
<niemeyer> hazmat, bcsaller, jcastro, fwereade: Fail
<hazmat> niemeyer, we're still on, can you rejoin?
<bcsaller> niemeyer: I sent an invite 
<hazmat> doh.. chrome crash
<bcsaller> hazmat: if you can't see the link let me know
<hazmat> bcsaller, dont' see it
<hazmat> bcsaller, go tit
<hazmat> er. got it
<bcsaller> that worked pretty well
<hazmat> bcsaller, agreed, that was fun
<Ryan_Lane> hi guys. what's the license of the project?
<hazmat> Ryan_Lane, AGPL for the core
<Ryan_Lane> any specific license for formulas?
<Ryan_Lane> anything AGPL compatible?
<hazmat> Ryan_Lane, formulas licensed per individual authors wishes.. canonical formulas are like to be GPL afaics
<Ryan_Lane> it isn't very clear in the documentation
<hazmat> Ryan_Lane, indeed, perhaps even propretiary.. its a separate process
<Ryan_Lane> no license is listed on launchpad either.
<hazmat> hmm
<hazmat> fixed re launchpad for ensemble
<Ryan_Lane> great. thanks :)
<pindonga> ac
<_mup_> ensemble/trunk r266 committed by kapil.thangavelu@canonical.com
<_mup_> merge sans-ami [r=bcsaller,niemeyer][f=803852]
<_mup_> Enables the use of ensemble with standard ubuntu amis instead of pre-built images, 
<_mup_> along with installation from the ensemble ppa as the default, while allowing 
<_mup_> option of source install from lp branches.
<adam_g> nice :)
<_mup_> ensemble/trunk r267 committed by kapil.thangavelu@canonical.com
<_mup_> merge hook-with-formula-dir [r=fwereade,niemeyer][f=761015]
<_mup_> Hooks now execute in the formula directory. The formula directory
<_mup_> can be referenced in hooks as environment variable ENSEMBLE_FORMULA_PATH
<adam_g> nice x2 
#ubuntu-ensemble 2011-07-09
<_mup_> ensemble/trunk r268 committed by kapil.thangavelu@canonical.com
<_mup_> merge auto-peer [r=fwereade,niemeyer][f=803855]
<_mup_> ensemble deploy automatically adds peer relations for deployed formulas.
<hazmat> adam_g, x3 maybe ;-)
<adam_g> hazmat: im not totally sure that third one does.. but it might be related to something i havne't complained about yet :P
<hazmat> adam_g, ;-) cool its for things with peer relations ala cassandra/mongodb.. its pretty minor
<adam_g> its the a lp bug that describes it?
<adam_g> er, is there a..
<adam_g> ive found there is a lack of relations being listed in 'ensemble status' for a service unit that relates to more than one peer for a given interface
<adam_g> ie, i have a mysql server that relates to many clients who share the same database. yet ensemble only reports the last added relation. i assume this is expected, but it would be nice to see all units its related to
<hazmat> bug 80385
<_mup_> Bug #80385: Could save some space by omitting /boot/vmlinuz from live filesystem <Ubuntu CD Images:Invalid> <livecd-rootfs (Ubuntu):Fix Released> <ubiquity (Ubuntu):Fix Released by cjwatson> <Baltix:New> < https://launchpad.net/bugs/80385 >
<hazmat> bug 803855
<_mup_> Bug #803855: deploy should add peer relations automatically <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/803855 >
<_mup_> Bug #807792 was filed: Formulas should be able to specify packages & ppas in metadata <Ensemble:New> < https://launchpad.net/bugs/807792 >
<_mup_> Bug #807794 was filed: Ensemble should have cli plugins <Ensemble:New for hazmat> < https://launchpad.net/bugs/807794 >
#ubuntu-ensemble 2011-07-10
<dvestal> Finally getting a chance to start working on it, is the Ensemble app itself supposed to run on Oneiric or will it run on Natty too?  Not necessarily the release, just in its current form.
<dvestal> I can install Oneiric in a vm just as easily as keeping this Natty one on my Macbook.
<SpamapS> m_3: up late? ;)
