#ubuntu-ensemble 2011-08-08
<zodiak> hello guys and gals, strange/stupid question time; can I run/play/experiment with ensemble on local hardware ?
<robbiew> zodiak: hi, I have to go now...but in short "soon".  We've been working on that ability THIS month...bcsaller and RoAkSoAx are the best to speak to
<zodiak> hrm.. good to know .. thanks for the .. oh.. well.. thank you regardless of your presence/non-presence :D
<_mup_> ensemble/expose-provision-machines r295 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/expose-provision-machines-reexpose r304 committed by jim.baker@canonical.com
<_mup_> Merged upstream
<_mup_> ensemble/expose-provider-ec2 r312 committed by jim.baker@canonical.com
<_mup_> Merged upstream and resolved conflict
<adam_g> are default vaules of config options in config.yaml expected to be output via config-get?
<_mup_> ensemble/states-with-principals r322 committed by kapil.thangavelu@canonical.com
<_mup_> resolve conflicts from security-acl merge
<_mup_> ensemble/security-acl r305 committed by kapil.thangavelu@canonical.com
<_mup_> use inline break instead of try/else
<_mup_> ensemble/states-with-principals r323 committed by kapil.thangavelu@canonical.com
<_mup_> resolve conflicts from security-acl merge
<_mup_> Bug #822814 was filed: Ambiguous endpoints error could be more informative <Ensemble:New> < https://launchpad.net/bugs/822814 >
<_mup_> ensemble/security-policy-with-topology r324 committed by kapil.thangavelu@canonical.com
<_mup_> resurrect policy.get_token earlier into branch pipeline, yank policy.client access.
<_mup_> ensemble/security-policy-rules-redux r330 committed by kapil.thangavelu@canonical.com
<_mup_> resolve conflict from security-policy-with-topology merge
<_mup_> ensemble/expose-provider-ec2 r314 committed by jim.baker@canonical.com
<_mup_> Pass machine_id through instead of machine_data bag
<_mup_> ensemble/expose-provider-ec2 r315 committed by jim.baker@canonical.com
<_mup_> Fix dummy and orchestra providers for modified api
<jcastro> negronjl: ping
<jcastro> negronjl: heya, I hear you have a mongo formula
<_mup_> ensemble/expose-provider-ec2 r316 committed by jim.baker@canonical.com
<_mup_> Fix dummy provider to add machine_id param to start_machine
<negronjl> hi jcastro.  Still working out thekinks on it
<negronjl> jcastro:  see my formulas here: https://code.launchpad.net/~negronjl
<jcastro> hey so check this out: http://nosql2011.wilshireconferences.com/agenda.cfm?confid=64&scheduleDay=PRINT
<jcastro> this is in like 2 weeks
<jcastro> and we've signed up for the scale out camp on the wednesday night
<negronjl> jcastro:  checking ..
<negronjl> jcastro:  i see
<jcastro> and we're now thinking, showing mongo scaling like a boss would be awesome there.
<negronjl> jcastro:  The formulas are here: https://code.launchpad.net/~negronjl  pick them out.  I haven't tested them completely yet but, feel free to do so and provide feedback.
<negronjl> jcastro:  I can try and work out the kinks on it
<negronjl> jcastro:  Are you planning on doing this live or prerecorded ?
<jcastro> negronjl: I am convincing your boss that you should copresent with me
<jcastro> this will be live
<negronjl> jcastro: live has it's own set of issues ( time, wifi issues, etc. )  I would sugges that you have a backup plan ( recorded session )
<jcastro> nod
<jcastro> negronjl: m_3 also wants to know how the hadoop is coming along.
<negronjl> jcastro, m_3:  making a blog post about hadoop-master/slave and ensemble right now....will let you all know when ready
<jcastro> negronjl: oh awesome
<m_3> rockin
<_mup_> ensemble/expose-provider-ec2 r317 committed by jim.baker@canonical.com
<_mup_> Fixed remaining tests to support the change to start_machine
<_mup_> ensemble/expose-provider-ec2 r318 committed by jim.baker@canonical.com
<_mup_> Ignore firewall permissions that are ranges greater than 1
<_mup_> ensemble/expose-provider-ec2 r319 committed by jim.baker@canonical.com
<_mup_> Test launch with existing machine security group
<_mup_> ensemble/expose-provider-ec2 r320 committed by jim.baker@canonical.com
<_mup_> Add test to verify scenario that the security group for the env already exists
<_mup_> ensemble/expose-provider-ec2 r321 committed by jim.baker@canonical.com
<_mup_> Doc strings for new tests
<_mup_> ensemble/expose-provider-ec2 r322 committed by jim.baker@canonical.com
<_mup_> Refactored the wrong start_machine in terms of adding machine_id as a param
<_mup_> ensemble/expose-provider-ec2 r323 committed by jim.baker@canonical.com
<_mup_> Better error message upon EC2Error
#ubuntu-ensemble 2011-08-09
<_mup_> ensemble/expose-provider-ec2 r324 committed by jim.baker@canonical.com
<_mup_> Simplified error messages for security group functions in EC2 provider
<kim0> Morning folks
<_mup_> Bug #823157 was filed: Syntax errors in /etc/bash_completion.d/ensemble <Ensemble:New> < https://launchpad.net/bugs/823157 >
<jamespage> morning all - has anyone done any thinking on how formulas might be tested?
<m_3> jamespage: yes... lots, but unfortunately that involves polling status and/or sleeping at the moment
<jamespage> m_3: interesting
<m_3> jamespage: we'll ask for test-related ensemble-cli features
<m_3> jamespage: so far we've got partial solutions using bash/selenium and cucmber/webrat
<jamespage> m_3: I was thinking about 'unit-testing' from within services
<jamespage> rather than externally testing the formula
<m_3> jamespage: oh, gotcha... then nope haven't thought about it :)
<jamespage> m_3: my thinking was that we could do this with a formula-tester service
<m_3> jamespage: formula testing powwow sometime would be great
<jamespage> each formula could understand the concept of being related to it through an optional testing interface
<m_3> jamespage: what have you come up with so far?
<m_3> gotcha
<jamespage> just trying something out at the moment
<m_3> brilliant
<robbiew> speaking of testing...we should also get a daily Ubuntu Cloud test rig setup
<robbiew> to test Orchestra/Ensemble bare-metal deployment
<robbiew> of course, this needs to happen after we sort out what Ubuntu Cloud after install looks like 
<robbiew> heh
<m_3> robbiew: yeah, still need to set up vpn access to the euca canonicloud
<hazmat> jamespage, yeah.. i've thought some about having a test harness for formulas
<_mup_> ensemble/expose-provision-machines r296 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<hazmat> jamespage, basically a two parts per hook, a script to setup the environment, that's run by the admin cli, and then a verification script run after a hook to verify that it succeedd
<hazmat> jamespage, its not happening for 11.10 though
<m_3> hazmat: sounds like the best story so far is a "test environment" that's sort of... well.. live
<_mup_> ensemble/expose-provision-machines-reexpose r305 committed by jim.baker@canonical.com
<_mup_> Merged upstream
<_mup_> ensemble/expose-provider-ec2 r325 committed by jim.baker@canonical.com
<_mup_> Merged upstream
<m_3> jamespage: lemme know... love to go over what you're doing
<jamespage> m_3: sure - just getting it working in a couple of demo formulas and I'll ping it round
<_mup_> ensemble/expose-provider-ec2 r326 committed by jim.baker@canonical.com
<_mup_> Standardized working with EC2Error in ec2 provider tests
<_mup_> ensemble/expose-provider-ec2 r327 committed by jim.baker@canonical.com
<_mup_> PEP8
<_mup_> ensemble/expose-provider-ec2 r328 committed by jim.baker@canonical.com
<_mup_> PyFlakes
<_mup_> ensemble/expose-provider-ec2 r329 committed by jim.baker@canonical.com
<_mup_> Added comment re eventual consistency of security group functions
<_mup_> ensemble/security-agents-with-identity r311 committed by kapil.thangavelu@canonical.com
<_mup_> rework initialize tests to setup security primitives
<_mup_> ensemble/expose-provider-ec2 r330 committed by jim.baker@canonical.com
<_mup_> Better comments of security group functions
<_mup_> ensemble/security-agents-with-identity r312 committed by kapil.thangavelu@canonical.com
<_mup_> set policy owner with a token not a principal, credentials are not always available, like when initializing nodes for the admin cli.
<_mup_> Bug #823352 was filed: ensemble shell <Ensemble:New> < https://launchpad.net/bugs/823352 >
<_mup_> ensemble/security-agents-with-identity r313 committed by kapil.thangavelu@canonical.com
<_mup_> state initialization seeds the token database with the admin identity.
<m_3> negronjl: ping
<niemeyer> http://people.canonical.com/~niemeyer/dublin.html
<negronjl> m_3: pong
<_mup_> Bug #823385 was filed: hard to reuse common features in ec2 provider code <Ensemble:New> < https://launchpad.net/bugs/823385 >
<hallyn> niemeyer: natty ensemble completion typo (missing ';;') : 
<hallyn> bash: /etc/bash_completion.d/ensemble: line 59: syntax error near unexpected token `)'
<hallyn> bash: /etc/bash_completion.d/ensemble: line 59: `        deploy)'
<niemeyer> hallyn: Ugh
<niemeyer> kirkland, bcsaller: ^
<jcastro> negronjl: hey is the mongo formula in decent enough shape so that kim0 can make a screencast?
<kim0> negronjl: oh and thanks for the epic post on the hadoop formulas :)
<eSoul> Hey Ensemble team, supporters, just wanted to join the channel and show my support as well.  Very nice and interesting work.  I want to experiment with this myself.
<negronjl> kim0:  np.  I'll be blogging about all of my formulas....next is cassandra then, mongodb and whatever else I can get my hands on :D
<kim0> negronjl: awesome :)
<kim0> eSoul: howdy
<kim0> eSoul: when did you first hear about ensemble
<kirkland> niemeyer: i think bcsaller dumped my bash completion implementation and used his own, right?
<eSoul> Apparently Facebook picked up on my Ubuntu-liking and saw a few of the posts that featured ensemble, but didn't check it out till the one yesterday
<bcsaller> kirkland: yes, but there was a trivial patch to that already, there was a missing closing case but it was fixed within minutes of it being reported, I don't know why anyone is still seeing it 
<eSoul> Ive been exposed/used linux since 2000 and have used ubuntu over the past few years
<kim0> eSoul: Awesome! So probably the easiest way to get yourself kickstart with Ensemble is checking out â https://ensemble.ubuntu.com/docs/ (especially getting-started, and the user-tutorial)
<kim0> eSoul: you probably wanna checkout the latest 3 vids on http://www.youtube.com/user/ubuntucloud#p/u as well
<eSoul> Thanks kim0.  I've skimmed thru the doc pages and have seen the awesome screencasts.  I just dont have any projects I need to used ensemble for yet.  Im still rooted in using things in-house.
<kim0> eSoul: very soon you'll be able to use ensemble in-house (even on a laptop) :)
<kim0> eSoul: a very good way to start playing with the technology
<kim0> eSoul: is to start writing a formula to address any of the needed ones at
<kim0> eSoul: https://bugs.launchpad.net/principia/+bugs?field.tag=bitesize
<eSoul> I was hoping it would come to that.  I couldnt believe it would stay locked to one virtualization system
<kim0> eSoul: you can deploy to ec2 "micro" instances, which would not cost any money to run
<kim0> eSoul: ec2 is just the most reliable one now .. very soon support will land for openstack, and local LXC containers and others
<kim0> eSoul: special offer â If you'll start hacking on a formula, I'll help you step by step ;)
<eSoul> Thanks kim0, ill try to tinker around with it when I get the chance.  Im afraid im much more of an implementor than a developer
<eSoul> haha, appareciate it
<kim0> eSoul: I know what you mean .. formulas are shell scripts .. so you absolutely do not need to be a developer to write those
<kim0> eSoul: I know because I am not one :) 
<eSoul> agreed
<kim0> eSoul: awsome .. so check things out and let me know what you decide to do .. cheers
<eSoul> But im afraid im still the one ssh-ing in every time and running all the commands (with bash history and tab auto-complete to save time) just to see everything happen one at a time
<eSoul> indeed
<eSoul> have a good one
 * kim0 <- eod .. partially afk o/
<robbiew> jcastro: is anyone keeping the list of hot formulas current? -> https://bugs.launchpad.net/principia/+bugs?field.tag=hot
<robbiew> m_3: SpamapS: ^^
 * robbiew noticed OpenStack is still in "New"....and I know adam_g has that
<m_3> robbiew: nope... need to
<FlipStonE> hello here... i'm trying ensemble... but not working good :-)
<jcastro> what problems are you having?
<FlipStonE> when i use the command 'ensemble bootstrap' i get this error: Could not find AWS_ACCESS_KEY_ID
<FlipStonE> 2011-08-09 21:07:29,617 ERROR Could not find AWS_ACCESS_KEY_ID
<FlipStonE> i'm not finding anything helpful on google...
<jcastro> you need to set your environment first
<jcastro> let me get you a link
<FlipStonE> ok, thanks
<jcastro> https://ensemble.ubuntu.com/docs/getting-started.html#configuring-your-environment
<FlipStonE> ensemble.ubuntu.com/docs also interesting as it seems :-)
<jcastro> start there
<jcastro> and then you can go right into the tutorial
<FlipStonE> ok, i was watching the vids on youtube... this might me more interesting on the website... going to try, thanks
<jcastro> let me know if you need anything!
<jcastro> we should probably link the docs more prominently when publishing the videos
<FlipStonE> hmz, maybe a very stupid question, but in terminal 'bin/ensemble'... gives me nothing?
<jcastro> FlipStonE: if you've installed it from a ppa, you don't need the bin/
<jcastro> just type "ensemble"
<FlipStonE> ensemble: error: too few arguments :-)
<_mup_> Bug #823511 was filed: strange result from find_zookeepers <Ensemble:New> < https://launchpad.net/bugs/823511 >
<_mup_> ensemble/expose-provider-ec2 r331 committed by jim.baker@canonical.com
<_mup_> Added logging on creating machine security group
<_mup_> Bug #823586 was filed: intermittent failure in test_relation_get_format_shell_bad_vars <Ensemble:New> < https://launchpad.net/bugs/823586 >
<_mup_> Bug #823600 was filed: trunk is lagging behind orchestra spike <Ensemble:In Progress by fwereade> < https://launchpad.net/bugs/823600 >
#ubuntu-ensemble 2011-08-10
<basil_kurian_> Hi
<kim0> Nice, we're on http://www.linuxtoday.com/
<adam_g> any ideas why python and apt would be throwing these errors only when executing commands from within an install hook? http://paste.ubuntu.com/662523/
<_mup_> Bug #823866 was filed: ec2metadata fails with traceback within ensemble nodes <cloud-utils:New> <Ensemble:New> < https://launchpad.net/bugs/823866 >
<_mup_> Bug #823898 was filed: When using Ensemble, add-apt-repository no longer functions properly <Ensemble:New> < https://launchpad.net/bugs/823898 >
<SpamapS> adam_g: re bug 816169 , if you run add-apt-repository with 'LANG=C add-apt-repository ...' does that correct the issue?
<_mup_> Bug #816169: When using Ensemble, add-apt-repository no longer functions properly <ensemble (Ubuntu):Confirmed> <software-properties (Ubuntu):Confirmed> < https://launchpad.net/bugs/816169 >
<adam_g> SpamapS: actually, launching a test that messes with the env. right now. willl let you know
<SpamapS> cool
<SpamapS> Daviey: I'm heading down to get some breakfast, and then maybe we can have a quick google hangout / skype chat to figure out a plan for making use of the machines.
<adam_g> SpamapS: same
<Daviey> SpamapS: sure thing
<adam_g> SpamapS: same errors using the exact environment i'd use running it manually
<SpamapS> adam_g: not sure I follow you exactly
<SpamapS> adam_g: did you duplicate the hook environment locally and you got that error?
<adam_g> SpamapS: no, the other way around. duplicated users environment in the hook
<SpamapS> adam_g: so may be related to the lack of controlling terminal maybe
<adam_g> SpamapS: strange. works fine on natty and was working on oneiric up until last week sometime
<RoAkSoAx> fwereade: ping
<fwereade> RoAkSoAx: pong
<fwereade> RoAkSoAx: call ran over, I'll be down shortly
<RoAkSoAx> fwereade: sure, you still want the branch against ensemble? 
<fwereade> RoAkSoAx: don't worry, just in future
<fwereade> RoAkSoAx: it's not such a huge diff I can't handle it ;)
<RoAkSoAx> fwereade: heh, it is really no problem to upload it lol: lp:~andreserl/ensemble/shadow-trunk-sprint
<RoAkSoAx> fwereade: doesn't have any new stuff btw
<fwereade> RoAkSoAx: cheers
<_mup_> ensemble/states-with-principals r324 committed by kapil.thangavelu@canonical.com
<_mup_> resurrect missing security rules module
<niemeyer> +def shutdown_ensemble(provider, kill_machines):
<niemeyer> +    return kill_machines(provider)
<niemeyer> !!!
<fwereade>  niemeyer: ensemble.providers.orchestra.instance is nearly as bad
 * fwereade hangs head
<fwereade> niemeyer: that branch started off as a shutdown branch, at which point the shutdown function included the state-clearing stuff
<niemeyer> fwereade: The whole direction of these alias/helper functions feel a bit dubious.. I'll send a review and we can talk afterwards
<fwereade> niemeyer: cool
<SpamapS> Daviey: talking with RoAkSoAx .. I'm going to try and help him get on the hardware network and get one of the boxes setup w/ the latest orchestra server... which box should he use?
<Daviey> SpamapS: woah, wait
<Daviey> SpamapS: We have done that.
<Daviey> Or i believe we have
<Daviey> cmpedak is the cobbler server.
<Daviey> cempedak*
<SpamapS> Daviey: cobbler or orchestra?
<SpamapS> ;)
<Daviey> SpamapS: I installed orchestra-provisioning-server on Monday
<Daviey> adam_g re-installed it this morning, documenting the steps.
<Daviey> adam_g: what did you isntall
<SpamapS> RoAkSoAx: have you made significant changes since then?
<Daviey> ?
<SpamapS> I know they've been working on stuff to make it more automatic and easier to integrate w/ ensemble.
<SpamapS> Been hard to iterate until today because our downstream bandwidth is crap
<adam_g> the steps i needed to take in the lab were to get dhcp and dns working properly
<adam_g> the only changes to orchestra/cobbler were to modify the kernel options passed via pxe
<adam_g> to look something like http://paste.ubuntu.com/662709/
<SpamapS> Ahh thats exactly what kirkland and RoAkSoAx were working on yesterday.. the DNS and DHCP
<SpamapS> Its possible I'm just adding noise to a situation that is happening naturally.
<adam_g> i think the assumption is that tthese services are there and working ahead of time. they were in our case, but we had to put the orchestra server elsewhere and they weren't reconfigured correctly
<RoAkSoAx> adam_g: that's the simplest thing ever
<RoAkSoAx> that's all controlled by cobbler
<RoAkSoAx> adam_g: but we are currently working on getting the installation working here with the bandwidth limitations we have
<adam_g> ?
 * hazmat yawns
<RoAkSoAx> adam_g: DNS/DHCP
<_mup_> ensemble/security-policy-rules-redux r332 committed by kapil.thangavelu@canonical.com
<_mup_> resolve conflict from states-with-principals
<adam_g> RoAkSoAx: oh, good to know
<adam_g> RoAkSoAx: but surely that'll require some manual steps to get systems entered into a new setup? if instead we use pre-existing servers, we should be able to just install the orchestra server and begin netbooting to automatically enroll
<_mup_> ensemble/expose-provider-ec2 r332 committed by jim.baker@canonical.com
<_mup_> Revised error messages
<_mup_> ensemble/security-policy-rules-redux r333 committed by kapil.thangavelu@canonical.com
<_mup_> update tests in aftermath of removing service state security group accessor.
<_mup_> ensemble/pythonpath-fix-bug-816264 r302 committed by kapil.thangavelu@canonical.com
<_mup_> yank superflous env variable PYTHON
<_mup_> ensemble/expose-provider-ec2 r333 committed by jim.baker@canonical.com
<_mup_> Add testing around a security group is still active and cannot be deleted
<_mup_> ensemble/expose-provider-ec2 r334 committed by jim.baker@canonical.com
<_mup_> PEP8 & PyFlakes
<_mup_> ensemble/expose-provision-machines r297 committed by jim.baker@canonical.com
<_mup_> Merged trunk
<_mup_> ensemble/expose-provision-machines-reexpose r306 committed by jim.baker@canonical.com
<_mup_> Merged upstream expose-provision-machines
<niemeyer> Our Kanban is updating correctly again
<niemeyer> I mean, the official one, rather than the one at people.canonical.com
<niemeyer> Please ping me if it goes crazy again
<_mup_> ensemble/pythonpath-fix-bug-816264 r303 committed by kapil.thangavelu@canonical.com
<_mup_> avoid exposing pythonpath to hooks
<_mup_> ensemble/expose-provider-ec2 r335 committed by jim.baker@canonical.com
<_mup_> Merged upstream & resolve conflicts
<_mup_> ensemble/expose-provider-ec2 r336 committed by jim.baker@canonical.com
<_mup_> PEP8/PyFlakes/leftover text conflict
<niemeyer> robbiew: Do we have consensus around the new name for lp:principia?
<niemeyer> robbiew: Maybe we should get together here in a moment to figure this out and move the renaming on
<robbiew> niemeyer: I think so...sur
<robbiew> sure
 * robbiew digs for the email
<jimbaker> seeing a problem on this test: ensemble.hooks.tests.test_invoker.TestCompleteInvoker.test_relation_get_format_shell_bad_vars
<m_3> negronjl, kim0: note to change hadoop repo links in blog posts
<negronjl> m_3:  will do thx for the reminder :)
<m_3> negronjl: have the dummy packaging trick working... we can push to principia repos that don't already have lp src packages
<negronjl> m_3:  that's cool.  you know ... sharing is caring :)
<m_3> negronjl: ha!... yeah, I'll add it to principia-tools
<negronjl> m_3:  perfect!
<negronjl> m_3:  thx
<m_3> np
<niemeyer> robbiew: Do you have a link for the conversation about the logo at hand?
<niemeyer> robbiew: I'd like to reply to the request about it, but don't know what's the best thread/bug about it at that point
<robbiew> niemeyer: hmm...there was a bug, but I believe they already have that...one sec, let me look
<niemeyer> robbiew: Hmm.. they've requested the logo/conversation about that, so I suspect that if they have it, they forgot abou t it
<robbiew> niemeyer: https://lists.ubuntu.com/archives/ensemble/2011-July/000216.html
<robbiew> there's the start of the thread
<robbiew> I'll get the bug, one sec
<robbiew> niemeyer: https://bugs.launchpad.net/ubuntu-branding/+bug/807100
<_mup_> Bug #807100: Develop Ensemble logo (ensemble.ubuntu.com) <ubuntu-branding:In Progress> < https://launchpad.net/bugs/807100 >
<niemeyer> robbiew: Cheers!
<robbiew> ;)
<niemeyer> robbiew: Sent, you're ccd
<niemeyer> Aram: ping
<robbiew> cool
<_mup_> ensemble/pythonpath-fix-bug-816264 r304 committed by kapil.thangavelu@canonical.com
<_mup_> remove debug/error reproducer from example formula
<jimbaker> there will be a momentary blip in trunk as i merge in 3 branches related to provisioning. the context is that if you're using the ec2 provider (so not testing or working on orchestra), there will not be an available impl of the security group mgmt
<jimbaker> it should be about 15 minutes to go through the 3 merges, starting in about 5 minutes or so
<_mup_> ensemble/trunk r307 committed by jim.baker@canonical.com
<_mup_> merged expose-provision-machines [r=hazmat,niemeyer][f=809007]
<_mup_> Modifies the provisioning agent to manage the firewall on a machine.
<jimbaker> this process has begun
<jimbaker> never mind, 3 commits, one push
<jimbaker> this is a happier process
<_mup_> ensemble/trunk r308 committed by jim.baker@canonical.com
<_mup_> merged expose-provision-machines-reexpose [r=hazmat,niemeyer][f=812619]
<_mup_> Modifies the provisioning agent to enable re-exposing of a service.
<_mup_> ensemble/trunk r309 committed by jim.baker@canonical.com
<_mup_> merged expose-provider-ec2 [r=niemeyer,fwereade][f=767420]
<_mup_> Modified EC2 provider to support firewall management.
<jimbaker> firewall mgmt is now in trunk!
<_mup_> ensemble/formula-state-with-url r307 committed by kapil.thangavelu@canonical.com
<_mup_> ec2 storage has a get_url method that returns a shareable signed url for any resource.
<jimbaker> kim0, we will need to update the include tutorial on writing a formula for drupal to take in account the new support for firewall mgmt
<jimbaker> kim0, i'm currently going through the docs now in light of these changes having landed in trunk
<_mup_> ensemble/formula-state-with-url r308 committed by kapil.thangavelu@canonical.com
<_mup_> get url for dummy provider storage
<_mup_> ensemble/formula-state-with-url r309 committed by kapil.thangavelu@canonical.com
<_mup_> get url for orchestra file storage
<_mup_> ensemble/expose-docs r310 committed by jim.baker@canonical.com
<_mup_> Initial commit
<_mup_> Bug #824218 was filed: Update docs on EC2 firewall mgmt <Ensemble:New for jimbaker> < https://launchpad.net/bugs/824218 >
<_mup_> Bug #824219 was filed: ensemble shutdown and terminate-machine should delete associated security groups <Ensemble:New> < https://launchpad.net/bugs/824219 >
<_mup_> Bug #824222 was filed: ensemble bootstrap should be more robust <Ensemble:New for jimbaker> < https://launchpad.net/bugs/824222 >
#ubuntu-ensemble 2011-08-11
<_mup_> ensemble/expose-docs r311 committed by jim.baker@canonical.com
<_mup_> Corrected example ensemble status output for exposed service in doc
<_mup_> Bug #824279 was filed: Security group functions for EC2 provider should retry <Ensemble:New> < https://launchpad.net/bugs/824279 >
<_mup_> Bug #824403 was filed: ensemble should support openstack as a provider <Ensemble:New> < https://launchpad.net/bugs/824403 >
<kim0> m_3: I updated the hadoop post to point to the updated formula locations
<m_3> kim0: cool... I
<m_3> 'll update
<_mup_> Bug #824557 was filed: Update formula writer tutorial for port exposing functionality <Ensemble:New> < https://launchpad.net/bugs/824557 >
<kim0> hmm, I see the wordpress example uses "open-port" in the db-relation-changed hook
<kim0> Wouldn't the install hook be a better place for this?
<kim0> I'm updating the tutorial to add this, hence the question
<SpamapS> kim0: install would be too early
<SpamapS> kim0: the idea is to open the port when the service is ready to serve users.
<SpamapS> s/users/clients/
<kim0> Yeah .. ok
<RoAkSoAx> fwereade: http://paste.ubuntu.com/663457/
<fwereade> RoAkSoAx: cheers
<RoAkSoAx> fwereade: so yeah, given that thte provisioning agent gets started late, and appears to be after a machine has been obtained for the deploy
<RoAkSoAx> fwereade: it tries to shutdown that machine first
<RoAkSoAx> fwereade: so basically, if the agent gets started when it should, we should not see that issue
<fwereade> RoAkSoAx: yep, I think so
<jimbaker> kim0, the tutorial looks good to me except for the following: we should have one more ensemble status after the ensemble expose
<kim0> jimbaker: mm ok adding
<jimbaker> kim0, cool
<_mup_> ensemble/expose-docs r312 committed by jim.baker@canonical.com
<_mup_> YAML serializes as true, not True
<_mup_> ensemble/trunk r310 committed by jim.baker@canonical.com
<_mup_> merge expose-docs [r=jorge][f=824218]
<_mup_> [trivial] Move expose docs from drafts to main, along with minor
<_mup_> updates, given comprehensive expose functionality now in trunk.
<kim0> jimbaker: ok pushed the one with status at the end
<jimbaker> kim0, just taking a look - the drupal service in that status output doesn't have an exposed: true 
<kim0> jimbaker: yeah .. I didn't want to change machine names, so I'm manually adding what I think should go there
<jimbaker> kim0, no worries, that's the only danger in doing that :)
<kim0> yeah hehe :)
<m_3> jamespage: What are "labels" in the jenkins-slave interface?
<fwereade> RoAkSoAx:             # TODO: Obtain only systems in the management classes
<jimbaker> kim0, anyway, good to have one more lookover, but with that change, it's ready to merge. looks like you're empowered, so please commit when ready
<jamespage> m_3: labels are used by jenkins to control which builds run where; you automagically get some if you don't specify OR you can set them manually for a service
<fwereade> RoAkSoAx: I think it's trying to kill *everything*
<kim0> jimbaker: cool, thanks
<fwereade> RoAkSoAx: I have a fix for that, but it hasn't made it into shadow-trunk yet
<fwereade> RoAkSoAx: consider me on it
<jimbaker> kim0, just add a [trivial] tag, since it's that
<kim0> sure
<m_3> jamespage: cool thanks... (I'm documenting interfaces, authors can update them later)
<jimbaker> kim0, see our bzr log for the formatting on that... rather trivial
<m_3> adam_g: is rabbitmq really two different interfaces?
<m_3> maybe rabbit and rabbit-peer?
<m_3> or rabbit-replication
<fwereade> RoAkSoAx: lp:~fwereade/ensemble/spike-catchup
<adam_g> m_3: AFAIK, yes. one for clients and another for clustered rabbitmq servers
<m_3> adam_g: cool... you good with 'rabbitmq' and 'rabbitmq-node'?
<_mup_> Bug #824654 was filed: no MachineProvider base class <Ensemble:In Progress> < https://launchpad.net/bugs/824654 >
<fwereade> speaking of phones and laptops: http://dilbert.com/2011-08-03/
<hallyn> niemeyer: http://golang.org/doc/go_tutorial.html   uh, what?  'fmt.Printf'?
<hallyn> i can't do that
<niemeyer> LOL
<niemeyer> hallyn: There's println for debugging
<niemeyer> hallyn: Just to relieve you a bit ;)
<hallyn> ok then :)
<niemeyer> hallyn: The uppercasing has some good background, even though it takes a bit to get used to.  Every const/var/method/field/function/etc is public if it's uppercased, and package-private if it's lowercased
<niemeyer> hallyn: Besides the ability to tersely define scope, this has the non-immediately obvious benefit of informing scope not only on declaration, but on every use
<kirkland> to deploy into oneiric machines, do I need to give a specific ami in my environments.yaml, or can I just use "oneiric" as a keyword?
<kirkland> SpamapS said this was under development ...
<SpamapS> jcastro: http://www.reddit.com/r/devops/comments/j8qc1/please_talk_me_out_of_writing_my_own_puppettype/
<_mup_> ensemble/expose-cleanup r311 committed by jim.baker@canonical.com
<_mup_> Spike on wait on terminated instances, then delete security groups
<_mup_> ensemble/formula-state-with-url r310 committed by kapil.thangavelu@canonical.com
<_mup_> add formula bundle url to formula state
<hazmat> kirkland, it looks like it needs an ami, adding support for release name would be easy though
<hazmat> kirkland, and just grab a daily
<hazmat> or release
<hazmat> kirkland, https://ensemble.ubuntu.com/docs/provider-configuration-ec2.html
<_mup_> Bug #824787 was filed: default-ami appears in schema, but default-image-id is used in code <Ensemble:New> < https://launchpad.net/bugs/824787 >
<_mup_> Bug #824792 was filed: cli api to store unit local data <Ensemble:New> < https://launchpad.net/bugs/824792 >
<m_3> kirkland: default-image-id: ami-1335f37a
<kirkland> m_3: thanks, trying .......
<m_3> kirkland: lemme know
<kirkland> m_3: oneiric!!!!!!!
<kirkland> niemeyer: lp:~kirkland/ensemble/byobu-tmux is working nicely :-)
<kirkland> niemeyer: shall I send over a merge proposal?
<_mup_> ensemble/formula-state-with-url r311 committed by kapil.thangavelu@canonical.com
<_mup_> formula publisher now creates formula states with bundle url
#ubuntu-ensemble 2011-08-12
<vciaglia> hi all
<kim0> vciaglia: hey htere
<kim0> vciaglia: I think you've pinged me earlier when I wasn't there
<vciaglia> hi kim0, it was just to say "nice to meet you" ;)
<kim0> hehe same here
<vciaglia> i'm already hacking with ensemble
<kim0> woohoo \o/
<kim0> need any help of some sort
<kim0> vciaglia: are you working on some formula already?
<vciaglia> kim0, no, i'm first discovering the tons of features of AWS
<kim0> yeah that makes sense
<kim0> Alright, I'm here to help .. if you have any questions, just ask away .. Have fun
<vciaglia> really really interesting things ;)
<vciaglia> kim0, ""EC2 error when attempting to delete security group ensemble-sample-0: Error Message: There are active instances using security group 'ensemble-sample-0'" ideas?
<kim0> vciaglia: it seems u have instances running 
<kim0> use    ensemble shutdown
<vciaglia> let's see...
<vciaglia> kim0, maybe a stupid question but ... in the environments.yaml
<vciaglia> the field control-bucket and admin-secret are casual?
<vciaglia> i think it is something related to S3, right?
<kim0> vciaglia: my understanding is that those are randomly generated unique keys for your account
<kim0> I think they're used to create the s3 buckets yeah, but I never cared to look at that part
<vciaglia> oh ok!
<vciaglia> kim0, 2011-08-12 11:12:51,367 INFO 'bootstrap' command finished successfully
<vciaglia> ;)
<kim0> nice :)
<vciaglia> let's see "ensemble status"
<vciaglia> woho, machines:
<vciaglia>   0: {dns-name: ec2-184-73-31-67.compute-1.amazonaws.com, instance-id: i-1984c278}
<vciaglia> services: {}
<vciaglia> 2011-08-12 11:15:14,663 INFO 'status' command finished successfully
<vciaglia> kim0, another question for you
<kim0> shoot
<vciaglia> here i see that the istance-id is: i-1984c278 but in AWS EC2 web panel it is i-dc6955aa: wtf?
<kim0> could you refresh the aws panel
<kim0> sometimes it gets out of sync
<vciaglia> ah, ok!
<vciaglia> now i'm deploying service units
<vciaglia> i'm reading that i need of the .ensemble/example directory, where can i get the service units/repository?
<vciaglia> kim0, ?
<kim0> ah sorry wasn't looking
<kim0> there's two things
<kim0> a built in examples repo inside ensemble .. this is at
<kim0> vciaglia: /usr/share/doc/ensemble/examples
<vciaglia> kim0, great, thanks
<kim0> the 'real' one .. is called principia â 
<kim0> https://ensemble.ubuntu.com/Formulas?action=show&redirect=Principia
<kim0> vciaglia: basically .. you can 'bzr branch xxx' any of those formulas â https://code.launchpad.net/principia
<kim0> vciaglia: if u haven't seen this article yet .. you should check it out http://cloud.ubuntu.com/2011/06/so-what-is-ensemble-anyway/
<vciaglia> kim0, nice
<vciaglia> kim0, no, i'll read it
<vciaglia> kim0, once "ensamble shutdown" i still see in STATUS: Running my istance. Why? Just a sync issue?
<kim0> vciaglia: did you answer 'y' to ensemble ?
<vciaglia> kim0, yes
<kim0> vciaglia: then yeah .. click the little refresh button in AWS
<kim0> you'll see em terminating
<vciaglia> :~$ ensemble shutdown
<vciaglia> Warning, this will destroy all machines and services in the
<vciaglia> environment. Continue [y/N]y
<vciaglia> 2011-08-12 11:43:04,983 INFO Shutting down environment 'sample' (type: ec2)...
<vciaglia> 2011-08-12 11:43:06,275 INFO 'shutdown' command finished successfully
<kim0> cool .. refresh aws then
<vciaglia> kim0, still running
<kim0> that never really happened to me
<kim0> I guess in 5 mins if they're still up .. you could right click and terminate them from aws
<vciaglia> wops, maybe this is the problem: Termination Protection: Enabled
<kim0> ah hehe :)
<vciaglia> kim0, well, the first thing we can do togheter is to add "Do not enable the Termination Protection for your Istance" in the official howto ;)
<kim0> haha
<vciaglia> kim0, umh ... now the status is "Terminated" but how to restart? The fields "Start" and "Stop" in the "Instance Actions menu" are inactive now
<kim0> terminated means 'machine destroyed' .. it is gone
<kim0> you start a new one
<vciaglia> kim0, great! 
<vciaglia> kim0, ping!
<kim0> vciaglia: pong :)
<vciaglia> kim0, i tried to create another istance and bootstrap ensemble but still the same problem: the istance id/dns name showed in Ensemble doesn't match with the AWS Managemente Console. Is it a bug?
<vciaglia> Ensemble:
<vciaglia> machines:
<vciaglia>   0: {dns-name: ec2-174-129-58-219.compute-1.amazonaws.com, instance-id: i-d34504b2}
<vciaglia> AWS Web:
<vciaglia> ec2-46-137-15-64.eu-west-1.compute.amazonaws.com
<vciaglia> and id: i-6e300c18
<vciaglia> wops, kim0 ... problem found!
<kim0> vciaglia: did you configure ensemble to use eu-west
<kim0> by default it uses us-east I think
<vciaglia> you're right! You've found the problem too! ;)
<vciaglia> kim0, how to configure eu-west?
<kim0> vciaglia: is that the region that the lightnin hit :)
<vciaglia> eheh
<vciaglia> i found the solution: add "region" to the environments.yaml
<vciaglia> right?
<kim0> Yes
<kim0> you're fast :)
<kim0> vciaglia: https://ensemble.ubuntu.com/docs/provider-configuration-ec2.html
<vciaglia> thanks!
<vciaglia> kim0, i see the light
<vciaglia>   0: {dns-name: ec2-46-137-8-151.eu-west-1.compute.amazonaws.com, instance-id: i-54281422}
<vciaglia> the right one! ;)
<kim0> \o/
<vciaglia> wow, now it works. Finally deployed wordpress!
<kim0> Yeeha :)
<kim0> vciaglia: did you ensemble expose wordpress ?
<kim0> this is recent functioanlity, so not sure if you've seen it in docs
<vciaglia> kim0, pointing via browser to the wordpress istance i cannot access to the blog ... maybe a security rule problem?
<vciaglia> kim0, i'm reading this doc: https://ensemble.ubuntu.com/docs/user-tutorial.html
<kim0> vciaglia: yeah that's what I meant
<kim0> vciaglia: ensemble expose wordpress
<kim0> this should open up the firewall so you can access it
<vciaglia> kim0, ok!
<kim0> vciaglia: let me know when it works
<vciaglia> kim0, ensemble assumes that Apache is already installed?
<vciaglia> kim0, now it works: http://ec2-46-137-63-5.eu-west-1.compute.amazonaws.com
<vciaglia> ;)
<vciaglia> kim0, you just need to open up the 80/HTTP port on 0.0.0.0/0 to get access
<kim0> vciaglia: what do you mean "open"
<kim0> vciaglia: did you edit AWS security groups manually ?
<vciaglia> kim0, yes
<kim0> vciaglia: ensemble expose should do this for you
<kim0> vciaglia: can u paste its output
<vciaglia> kim0, no i didn't use the "esemble expose" command
<kim0> see I told you to :)
<vciaglia> kim0, oh excuse me :)
<kim0> vciaglia: so now that you get everything running .. start preparing to write your very first own formula .. what do you think :)
<vciaglia> kim0, yes i'm ready! ;)
<vciaglia> kim0, in the next days i'll try to write my first formulas
<vciaglia> kim0, now i have to go. Thank you for your time. Really excited to start working on Ensemble.
<kim0> vciaglia: awesome .. ping me once you're ready to hack again .. enjoy
<vciaglia> kim0, sure! Talk to you very soon
<kim0> same here
<SpamapS> Daviey: last day, how goes the battle?
<_mup_> Bug #825307 was filed: zookeeper connection needs to detect errors differently <Ensemble:New> < https://launchpad.net/bugs/825307 >
<SpamapS> FYI, for those that missed the email, principia is being renamed to 'formulas'
<_mup_> Bug #825326 was filed: Need wrapper around lxc interaction <Ensemble:New for bcsaller> < https://launchpad.net/bugs/825326 >
<heckj> SpamapS: yeah!
<SpamapS> lynxman: ping, whats the status of the openstack deployment you guys were working on?
<lynxman> SpamapS: We just did a demo of an Ensemble deployment with Openstack, didn't have enough time on the sprint to do one in real life servers I'm afraid
<SpamapS> lynxman: ahh, ok.. bummer.
<SpamapS> s
<SpamapS> lynxman: so you were driving openstack w/ ensemble?
<SpamapS> lynxman: as in, using openstack to deploy formulas?
<_mup_> Bug #825398 was filed: Store Ensemble version number in topology node and verify ops <Ensemble:New for jimbaker> < https://launchpad.net/bugs/825398 >
<lynxman> SpamapS: not me, adam_g was the one driving that
<lynxman> SpamapS: I'm with swift upstartify and LXC integration
<adam_g> SpamapS: hey
<adam_g> SpamapS: just got into boston. i can push the swift-storage and swift-proxy, and nova-compute and glance. but need to figure out a way to get the nova-cloud-controller pushed into principia
<SpamapS> adam_g: I found them actually
<adam_g> adam_g: actually, swift-storage needs a way in as well
<SpamapS> adam_g: they can stay in +junk for now.. they're about to fix that bug so you can push directly.
<adam_g> SpamapS: oh, sweet :)
<_mup_> ensemble/verify-version r311 committed by jim.baker@canonical.com
<_mup_> Initial commit
<adam_g> SpamapS: are you putting swift on hardware or ec2?
<SpamapS> adam_g: no but I want to mess with it to run a parallel S3 on canonistack
<adam_g> in either case, i just pushed some changes that you'll probably need
<adam_g> make sure the swift-storage's block-device config matches something usable and free on the instance. 
<SpamapS> ok
<SpamapS> adam_g: do you know if any of the machines are available for use?
<adam_g> SpamapS: no clue. didn't know about canonistack till yestreday
<SpamapS> adam_g: no I mean the machines we had allocated for HW provisioning
<adam_g> SpamapS: oh, they are free from me, dont know about anyone else
<_mup_> ensemble/verify-version r312 committed by jim.baker@canonical.com
<_mup_> Test IncompatibleVersion exception
<_mup_> ensemble/verify-version r313 committed by jim.baker@canonical.com
<_mup_> Doc string and missing new file
<_mup_> ensemble/machine-agent-uses-formula-url r312 committed by kapil.thangavelu@canonical.com
<_mup_> formula downloads via http
<Daviey> SpamapS: It went well for us... far to tired to brief
<SpamapS> Daviey: ;) I bet
<SpamapS> Daviey: didn't realize you guys were done yesterday
<SpamapS> Daviey: been trying to play with canonistack, but the S3 service is basically hosed
<Daviey> SpamapS: better we know now eh? :)
<SpamapS> Daviey: I think its because its using nova-objectstore and not swift
<Daviey> SpamapS: I would be suprised if it plainly doesn't work
<Daviey> if it is using objectstore i mean
<SpamapS> Yeah it gets wedged at 100% CPU often
<SpamapS> I guess they're planning on deploying swift later
<SpamapS> sounds like maybe that needs to be moved up
<Daviey> ah.. i see
<SpamapS> Daviey: when you have time, can you send out a summary of the status of the machines on the nickel network?
<Daviey> I'm just overjoyed it's public facing.. i never thought that would happen! :)
<SpamapS> Me too. :)
<Daviey> SpamapS: Yeah, the old dhcpd from prior deployment was being shared by other teams..
<Daviey> *oops*
<Daviey> but yes, can you jote me a quick email asking for that.. it will drop off my memory otherwise.
<Daviey> my face hurts
<Daviey> -> bed
<SpamapS> toodles
#ubuntu-ensemble 2011-08-14
<fwereade> anyone know anything about ensemble.state.sshclient.SSHClient?
<fwereade> can't figure out why we would make it raise txzookeeper ConnectionTimeoutExceptions instead of ensemble NoConnection ones
<fwereade> (or ensemble ConnectionTimeout ones, or whatever)
<fwereade> niemeyer: ping
<fwereade> hazmat, niemeyer: will lose internets in a few minutes; I hope one of you will see this.please rereview lp:~fwereade/ensemble/hide-instances
<niemeyer> fwereade: Yo
<fwereade> niemeyer: heyhey
<niemeyer> fwereade: Will check it out tomorrow for sure, thanks!
<fwereade> niemeyer: np
<fwereade> niemeyer: got stuck in chicago
<niemeyer> fwereade: I've heard the trip back was a bit bumpy
<niemeyer> fwereade: :(
<fwereade> niemeyer: not to worry, but I am still there
<fwereade> niemeyer: I'll be on for a bit your-morning on monday
<fwereade> but rather jetlagged and probably not v prouctive
<niemeyer> fwereade: Don't worry about it.. get some good rest!
<fwereade> niemeyer: I will :)
<fwereade> niemeyer: I'm a bit unsure about the changes I made on that branch... I *think* they're good but I felt a bit lacking in context so I may have gone off the rails a little
<niemeyer> fwereade: Hmm
<fwereade> niemeyer: and in hindsight I definitely should have kept them in a different branch
<niemeyer> fwereade: Yeah.. it might be a bit hard to review
<niemeyer> fwereade: But will see what I can do about it fiddling with logs
<fwereade> niemeyer: all that had actually changed at the macro level was the precise shape of the error spam during bootstrap
<niemeyer> fwereade: My recommendation would be to try this logic for real
<fwereade> niemeyer: you have already reviewed 80% of it
<niemeyer> fwereade: Attempting to connect to the instances repeatedly right after bootstrap
<niemeyer> fwereade: This will yield all kinds of edge cases
<fwereade> niemeyer: I have, and it seems to me that it's a bit nicer
<niemeyer> fwereade: That must be handled
<niemeyer> fwereade: We already had a few things we needed to improve on this camp.. as long as we're not going back in terms of user interface, it's fine
<niemeyer> fwereade: if you manage to improve on that, though, many bonus points ;-)
<fwereade> niemeyer: well, it was rubbish before, and my changes made it slightly more rubbish, so I tried to fix that
<fwereade> niemeyer: not the best move in hindsight
<fwereade> niemeyer: anyway, I'm boarding in a few minutes
<niemeyer> fwereade: Well, sounds like a good goal.. just good to do in a separate branch to make it easier for people to review
<niemeyer> fwereade: After it's been reviewed once, at least
<fwereade> niemeyer: I hope it's a bearable review... the only unseen bits are providers.common.connect, and state.sshclient
<niemeyer> fwereade: But thanks for the changes in either case.. it'll be awesome to have that improved
<fwereade> niemeyer: there's churn elsewhere but I'mm 99% sure that's all related to the previous review points
<niemeyer> fwereade: Cool
<fwereade> niemeyer: if the sshclient stuff has any problems, would you have any issue with me backing that out and merging the originally-approved stuff?
<fwereade> I wanted to do that originally, but was convinced otherwise :0
<fwereade> ...sorry, time-pressure IRC is a bad time to discuss this
<niemeyer> fwereade: I would..
<niemeyer> fwereade: The change introduced UI problems
<niemeyer> fwereade: It might be split in a separate branch and both merged together
<niemeyer> fwereade: But merging without the UI problem being fixed would be bad
<fwereade> niemeyer: heh, yeah that would have been a better answer
<fwereade> niemeyer: I'd say it mutated UI problems (:p) but point taken
<fwereade> niemeyer, 
<fwereade> niemeyer: cheers; really must go now
<fwereade> niemeyer: happy sunday
<niemeyer> fwereade: Cheers! Have a good flight back
<fwereade> niemeyer: sorry, it looks like I've screwed something up; please don't worry about that branch
<niemeyer> fwereade: Ok, don't worry, we can sort that out
<fwereade> niemeyer: I think we're better off if you take a look at the other branches in the queue: I can revert that one to a focused state, and make a separate descendant that's cleanly reviewable
<fwereade> niemeyer: I'll WIP it for now
<fwereade> niemeyer: later :)
<hazmat> that sucks re getting stuck in chicago, looks there's been weather delays for most of the day
