/srv/irclogs.ubuntu.com/2011/04/14/#ubuntu-ensemble.txt

_mup_ensemble/unit-agent-formula-upgrade r213 committed by kapil.thangavelu@canonical.com00:19
_mup_make lifecycle.start deferred fire only after the initial service relation watch has completed processing, this gives a much better interaction within tests, as the unit relations will also be initialized, and zk communication done after the end of the start lifecylce method for test environments with existing relations (and also works fine with none)00:19
_mup_ensemble/unit-agent-formula-upgrade r214 committed by kapil.thangavelu@canonical.com00:21
_mup_additional upgrade workflow tests.00:21
niemeyerYo01:48
crazedhow does ensemble differ from CloudFormation?03:01
niemeyercrazed: CloudFormation is a service to coordinate AWS specific functionality03:14
niemeyercrazed: So I'd say it differs entirely :-)03:14
niemeyercrazed: Ensemble takes care of the actual service deployment and maintenance03:14
niemeyercrazed: and configuration, etc03:14
crazeddoes it also handle launching aws resources?03:14
niemeyercrazed: It does, yeah03:15
niemeyercrazed: and shutting them down too03:15
crazedi've been dropping stuff into userdata currently to launch chef-solo03:15
crazeddoes ensemble make life easier?03:15
niemeyercrazed: Yeah, absolutely.. that's the primary reason why we're working on ti03:16
niemeyerit03:16
crazedi'm reading through the documentation, don't have much experience with zookeeper unfortunately03:16
niemeyercrazed: No other tool at the moment is really focused on the integration of services the way Ensemble does03:16
niemeyercrazed: You can pretty much ignore it03:17
niemeyercrazed: it's an implementation detail, not necessary for working with it03:17
crazedhow does ensemble handle service specific configurations? for example, i want to use some specific nginx configuration03:18
crazedwhere do you store that config, and can it be populated with data based on the running environment03:18
niemeyercrazed: Yes, to both, if I understand what you mean03:21
niemeyercrazed: Formulas are the basic unit of deployment03:21
niemeyercrazed: You can think about it as a package03:21
niemeyercrazed: Formulas can communicate with each other through simple interfaces03:22
niemeyercrazed: So you can create a frontend app and make it communicate with a postgres app03:22
crazedi understand the higher level parts with the metadata, relations, etc03:23
crazedi'm just trying to wrap my head around the config management part03:23
niemeyercrazed: Ah, ok03:24
niemeyercrazed: This part is one we're actively working on03:24
niemeyercrazed: I think we have a published draft of the specification for you, hold on03:24
crazedi like the relations/formulas a lot, i think that's an aspect missing from CloudFormation 03:25
crazedseems ensemble takes care of that and config management03:25
crazedwhich nothing else currenlty does, you have to string together a bunch of pieces to get that full package03:25
niemeyercrazed: RIght, you get the exact spirit of why we're building it03:26
niemeyercrazed: Sorry about the delay.. the document is misplaced03:27
crazeddefinitely something i'm interested in, i do a lot of ec2 deployments03:27
niemeyercrazed: Here is a useful link, for the moment:03:27
niemeyerhttp://people.canonical.com/~niemeyer/ensemble/service-config.html03:27
niemeyercrazed: Please note this is a specification which is being developed still, so it's not yet working03:28
niemeyercrazed: But that's exactly how it will work03:28
niemeyercrazed: Well, unless we get feedback which makes us change it03:28
crazedi'm going to have to read this over, i'll hang out here though and help out where possible03:31
niemeyercrazed: That's awesome, your EC2 experience on deployments is most welcome.  Thanks a lot.  03:32
_mup_ensemble/yaml-state-node r185 committed by bcsaller@gmail.com04:38
_mup_Provides a dict like interface around a Zookeeper node04:38
_mup_    containing serialised YAML data. The dict provided represents the04:38
_mup_    local view of all node data.04:38
_mup_    `flush` pushes this information into the Zookeeper node.04:38
_mup_    YAMLState(client, path)04:38
_mup_    `client`: a Zookeeper client04:38
_mup_    `path`: the path of the Zookeeper node to manage04:38
_mup_    The state of this object always represents the product of the04:38
_mup_    pristine settings (from Zookeeper) and the pending writes.04:38
_mup_    All mutation to the dict expects the use of inlineCallbacks and a04:38
_mup_    yield. This includes set and update.04:38
_mup_ensemble/yaml-state-node r186 committed by bcsaller@gmail.com05:00
_mup_added revert to return to pristine state05:00
_mup_added sync (as a published name for _combine)05:00
_mup_ensemble/config-state-manager r189 committed by bcsaller@gmail.com05:07
_mup_service.config_get using new YAMLState05:07
kim0Morning folks09:03
hazmatunity multi touch is pretty slick13:39
TeTeTwas there a 'getting started' tutorial somewhere?13:39
TeTeTreading docs/getting-started.rst ...13:42
TeTeTdo I need to get txaws and txzookeeper from source, or are there any packages available?13:46
hazmatTeTeT, source atm13:51
hazmatTeTeT, txzookeeper is in pypi though.. 13:52
TeTeThazmat: just built the txzookeeper package, as it had a debian dir in the branch13:52
hazmatTeTeT, yeah.. i put together some packaging for a ppa a while ago, but we have setup any lp source build recipes to automate it13:53
hazmat^don't13:53
TeTeThazmat: how do I install txaws? python setup.py <argument>? What would be the argument?13:54
hazmatTeTeT, i typically do it in a virtualenv.. 13:54
hazmatTeTeT, actually the txaws python package should be fine for us.. unless your trying to use it with eucalyptus13:54
TeTeThazmat: yep, I do it in a natty vm at the moment13:55
TeTeThazmat: and unfortunately I would love to try it with UEC ;)13:55
hazmatah.. well in that case.. python setup.py develop or install will work.. 'develop' will link that directory to your python installation, install will copy it over.. 'virtualenv' is a python technique to isolate python packages from the system python.13:56
TeTeTone step further: $ bin/ensemble13:56
TeTeTerror: No environments configured. Please edit: /home/ubuntu/.ensemble/environments.yaml13:56
TeTeThazmat: didn't know about virtualenv13:56
hazmatTeTeT, so far so good. it created a sample config for you13:56
TeTeThazmat: so I dumped it in /usr/local ...13:56
hazmatTeTeT, that's fine.. you'll just want keep track in the future in case your upgrading it.. 13:58
TeTeThazmat: ok13:59
TeTeTnow I need to find my amazon credentials first ... somewhere under account they are13:59
hazmatTeTeT, fwiw, i do something like this https://pastebin.canonical.com/46144/14:01
hazmatthat allows me to just update the source directories with bzr and the latest versions are installed14:01
hazmatalternatived you can manipulate PYTHONPATH environment variable to include the source of the each package14:02
hazmatinstead of messing with setup.py14:02
TeTeThazmat: not sure if I see through that yet - I'll have a look. I now added the aws credentials. is there an easy 'print hello world' like example?14:04
hazmatTeTeT, ./bin/ensemble bootstrap  will startup a machine.. you can do ./bin/ensemble status  and ./bin/ensemble deploy --repository=examples mysql after that14:14
kim0TeTeT: there's a "examples/" folder in there14:14
hazmatTeTeT, i stuck a minimal walkthrough in examples/readme.txt14:15
TeTeTkim0 + hazmat : thanks, trying that - how do I inform ensemble on which ssh key to use? I downloaded ensemble-key.pem to the vm, but where to place it?14:15
kim0hazmat: do you know which branch uses the new hooks (joined, departed) ?14:16
hazmatkim0, trunk14:16
kim0hazmat: trunk has docs for that too14:16
kim0?14:16
hazmatkim0, not yet.. that's in the review queue 14:16
hazmatkim0, https://code.launchpad.net/~jimbaker/ensemble/new-hook-semantics-5-docs/+merge/5753614:17
kim0awesome thanks14:17
kim0hazmat: n00b question, why doesn't that branch (new-hook-semantics-5-docs) show in https://code.launchpad.net/ensemble 14:19
hazmatTeTeT, you can put the ssh key into the environments.yaml and reference it by path or putting the contents of the public key there. by default it sniffs for a public key, and barfs it can't find one. the config setting for it in the environment is 'authorized-keys' for the content, or 'authorized-keys-path' to reference it by file path.14:19
hazmatkim0, it does for me14:20
hazmat5th one down after trunk14:20
kim0oh duh .. old page hehe14:22
kim0I am not sure why "provides" must be satisfied14:24
TeTeTso I copied my ensemble-key.pem to id_rsa.pub and I get this error: http://pastebin.ubuntu.com/594014/14:28
crazedsounds like the aws signature is wrong14:30
TeTeTcrazed: does that mean access and secret key?14:31
hazmatTeTeT,  yes.. it sounds like your access/secret key for aws might be wrong14:36
hazmats/sounds/looks14:36
TeTeThazmat: hmm, I use them from s3fox just fine14:39
TeTeThazmat: wouldn't I need to put my account id somewhere as well?14:40
hazmatTeTeT, your access key / secret key effectively are your account id14:40
hazmatTeTeT, do the ec2 cli tools work for you?14:42
TeTeThazmat: I'll give it a try14:43
hazmatTeTeT, we don't use the private key / cert mechanism which is used for the ec2 soap api.14:45
hazmatand the ec2 cli tools14:45
TeTeThazmat: works and apparently ensemble did bootstrap something: http://pastebin.ubuntu.com/594021/14:45
hazmatTeTeT, cool! ./bin/ensemble status will tell you what it did :-)14:45
TeTeThazmat: unfortunately not:14:46
TeTeT$ bin/ensemble status14:46
TeTeTNo Ensemble machines found. Is the environment bootstrapped?14:46
TeTeT2011-04-14 15:46:03,116 ERROR No Ensemble machines found. Is the environment bootstrapped?14:46
hazmatTeTeT, hmm.. that's very strange, 14:54
hazmatit looks like you launched machines,  and now their not visible to ensemble, did the settings change in between bootstrap and status?14:55
TeTeThazmat: yeah, is there a way to get more debugging output I might make available?14:55
hazmatensemble -v subcommand  turns up the cli logging14:55
TeTeThazmat: I changed the credentials one time, but I started another bootstrap14:55
hazmatthere's also distributed logging, but it more about observing system operation.. this problem sounds more fundamental14:56
TeTeThere's bootstrap -v : http://pastebin.ubuntu.com/594030/14:56
hazmatTeTeT, thanks thats helpful14:57
TeTeTand that's status -v immediately thereafter: http://pastebin.ubuntu.com/594032/14:57
hazmatTeTeT, so the problem is that we launch the machine via ec2.. that works fine.. but we get an error saving our 'map' to s3, we use the map to find out where things are in the environment.14:58
TeTeThazmat: the status is saved in the map?14:58
hazmatTeTeT, the underlying problem though looks like an issue with txaws s3 14:58
hazmatTeTeT, the instance-id to the bootstrap machine14:59
hazmator more specifically, the instance-ids of the ensemble zookeeper cluster machines14:59
hazmatTeTeT, its not clear what version of txaws is being used, the traceback shows the package version being used15:00
* hazmat checks for differences15:00
TeTeThazmat: I used rev no 79 from bazaar and did the $ sudo python setup.py install15:01
hazmatTeTeT, cool, yeah. its in /usr/local15:02
_mup_Bug #760719 was filed: ensemble should verify s3 access before launching bootstrap nodes <Ensemble:New> < https://launchpad.net/bugs/760719 >15:05
TeTeThe he, that was quick15:06
hazmatTeTeT, yeah.. i've been using revno 68 of txaws, i'm updating now to try it out.15:07
hazmatstill not clear what the issue with s3 write is15:07
TeTeThazmat: confirm that reverting to 68 and doing an install over it seems to work - still launching a new instance15:10
hazmatTeTeT, confirmed that the latest (rev 80) seems to  have issues15:11
TeTeTstill can't connect due to ssh problems - seems my key gets not injected correctly15:12
hazmathmm.. looks like python2.6 compatibility issues with the latest rev 80 of txaws15:12
hazmatTeTeT, are you using a key from ssh-keygen ?15:13
TeTeThazmat: nah, I generated it via the web console - should I use euca-add-keypair for a change?15:13
TeTeThazmat: I would think natty has 2.7, so not sure if 2.6 is an issue15:13
hazmatTeTeT, its not a provider key/cert.. its just an ssh-keygen key.. preferrably one for which you have the private side in .ssh 15:14
hazmatthen things like `ensemble ssh` just work15:15
hazmatniemeyer, it looks like your merge of s3-bucket-paths to txaws broke ensemble bootstrap15:23
hazmatand jamu's following yours broke python 2.6 compatibility15:23
niemeyerhazmat: Ugh, seriously?15:24
niemeyerhazmat: How did it break it?15:24
hazmatniemeyer, yup15:25
hazmatniemeyer, re s3.. http://pastebin.ubuntu.com/594030/15:25
niemeyerhazmat: I actually tested the changes in practice against S3 with the command line tools within txaws15:25
hazmatniemeyer, cool, it could just be something we have to update in our usage.15:25
niemeyerhazmat: Hmm.. wait.. I think the changes after the review were not tested15:26
niemeyerhazmat: Let me have another look at that15:26
niemeyerhazmat: I'll just finish the review I'm doing and will check it out15:26
hazmatniemeyer, great, thanks15:26
kim0folks .. I've just written my understanding of a trace of how hooks would trigger for the sample wordpress formula .. It's at http://j.mp/fUOgbF .. Would someone please check/correct it ?15:56
niemeyerhazmat: I won't be able to look at this right now, sorry.. the review took some time, and I have to get lunch to attend a meeting in less than an hour16:05
niemeyerhazmat: If you don't get to it first, I can check it out after the meeting, though16:06
hazmatniemeyer, sounds good, i'm trying to get back into the upgrade stuff16:06
hazmatkim0, looking16:06
=== niemeyer is now known as niemeyer_lunch
kim0hazmat: the relation is called "db" in this case16:23
kim0I'm tracing this particular example16:23
hazmatkim0, i can put it back.16:23
kim0not too important16:23
kim0what's the green highlight16:23
hazmatkim0, if you run ensemble debug-log .. before the deploy you'll see it all16:23
kim0hazmat: great!16:24
hazmatkim0, you can filter to just hook execution as well16:24
hazmat--include hook16:24
kim0what's the green highlight16:25
hazmatkim0, those hooks don't exist16:25
kim0hazmat: yeah .. so the sequence is correct .. but the hooks dont exist in this case, right16:26
hazmatyup16:26
kim0great .. are you done with it 16:26
hazmatyup. looks good16:27
kim0hazmat: thanks :)16:27
kim0I feel much more confident now hehe ;)16:28
kim0understanding hook firing sequence is pretty important16:28
kim0hazmat: I think it'd be useful to merge that trace to the readme file .. to me that was about the most confusing thing about ensemble16:34
hazmatkim0, definitely, sounds good to me.16:34
kim0cool .. branching .. proposing16:35
hazmatthe examples can be improved a bit, jim's got some bashified version which may change the sequence16:35
kim0hazmat: I was actually reading from the bashified16:36
hazmatkim0, cool16:36
kim0yeah16:36
_mup_Bug #760818 was filed: debug log is initialized too late in the unit agent startup, to catch install/start hooks <Ensemble:New> < https://launchpad.net/bugs/760818 >16:47
jimbakerkim0, i think it would be good to expand the formula example section with this information17:15
jimbakerhazmat, i don't think the sequence changes in the bashified version17:16
=== niemeyer_lunch is now known as niemeyer_meeting
kim0jimbaker: I just proposed it for merging into your bashified branch17:16
jimbakerkim0, ahh, sounds good17:16
jimbakerkim0, i'm unable to see your branch proposal17:18
kim0woot17:18
jimbakerbut i did get the email about it17:18
kim0jimbaker: it's the next one 17:19
kim0jimbaker: https://code.launchpad.net/~kim0/ensemble/adding-hook-traces-to-readme-for-bashified-example/+merge/5771617:19
jimbakerkim0, i think for your branch proposal you may need to explicitly mention my branch as a dependency, it was hard for to me read because my branch changes were mixed with yours since it was a diff against trunk17:19
kim0because I initially proposed it into lp:ensemble17:19
kim0then deleted that17:19
kim0and proposed into your branch17:19
jimbakerkim0, got it, now that is much clearer17:19
kim0yeah :) first time to propose merging into other branch :)17:20
jimbakerkim0, i still got it messed up a couple of weeks ago with a bunch of stacked branches17:20
jimbakeradd to redo the proposals17:20
jimbakers/add/had17:20
kim0jimbaker: I was actually surprised this always failed:  bzr push lp:~kim0/ensemble/bashified-wordpress-mysql-examples/adding-hook-traces-to-readme17:21
jimbakerlaunchpad is powerful, but nonobvious some times17:21
kim0I concluded it wasn't possible to push 2 level stacks or something ?!17:21
jimbakerkim0, never tried that style, but it looks like something unsupported for sure :)17:21
kim0felt natural to me hehe :)17:22
jimbakerkim0, it would be nice if the merge proposal did catch that you had branched something besides trunk, and so therefore needed that branch dependency in place17:22
jimbakeri think that would be more natural17:22
kim0yeha17:23
jimbakerhazmat, i want to establish something, so to speak; the <relation name>-relation-established hook is not implemented, right? it was discussed in the docs (In formula.rst), which i updated in my docs branch proposal17:26
jimbakerbut unless it's hiding really well, it's not in the code17:27
hazmatjimbaker, we did away with it17:27
jimbakerhazmat, good. because i noticed that kim0 mentions this hook in his trace analysis17:27
jimbakerbut apparently such phantom hooks can leave the impression of being there :)17:27
jimbakerhazmat, i had removed it in my branch proposal, we just need to update the mentioning text in kim0's proposal, and it will be gone forever i hope. (unless we need to actually implement it)17:29
hazmatjimbaker, sounds good17:29
hazmatthere where challenges around implementing it, and utility became a question, since the first hook always executed is relation-join 17:32
hazmats/where/were17:32
jimbakerhazmat, agreed, it seems like -relation-joined is sufficient for this case17:33
_mup_ensemble/unit-agent-formula-upgrade r215 committed by kapil.thangavelu@canonical.com17:47
_mup_revert r213, having start execute return after initial watch callback17:47
=== niemeyer_meeting is now known as niemeyer
_mup_ensemble/unit-agent-formula-upgrade r216 committed by kapil.thangavelu@canonical.com18:20
_mup_upgrade formula error handlers that shutdown hook execution.18:20
niemeyerjimbaker: That's the danger of documenting stuff without a notice about whether it's implemented or not.19:00
jimbakerniemeyer, indeed19:01
_mup_ensemble/unit-agent-formula-upgrade r217 committed by kapil.thangavelu@canonical.com19:04
_mup_redo hook executor prioritize, instead as run immediate hook execution while stopped, bypassing the queue, and allowing us better control of hook executor start/stop status from lifecycle methods19:04
niemeyerhazmat: Interestingly, I can't reproduce the signature failure with a hand-made test case19:08
niemeyerhazmat: Nevermind, reproduced a bug when listing the bucket19:12
hazmatniemeyer, cool19:13
_mup_ensemble/unit-agent-formula-upgrade r218 committed by kapil.thangavelu@canonical.com19:14
_mup_update executor tests with new renamed run priority hook method19:14
hazmatwe're almost at 1k unit tests19:17
niemeyerWow19:19
jimbakerbcsaller, hazmat, niemeyer - standup?19:21
hazmatsoudns good19:21
niemeyerjimbaker: If there are no pressing issues, I'd be happy to skip it today19:21
hazmatniemeyer, i'm curious about the repository discussion19:22
jimbakerniemeyer, sounds good, i can proceed immediately to lunching then :)19:22
niemeyerI've been involved in a bunch of things and don't feel like I've really produced much yet.. I'd like to get this fixed and the rest of branches reviewed19:22
niemeyerWell, cool19:22
niemeyerhazmat, jimbaker: let's do it19:22
niemeyerWe can do an *actual* standup :)19:22
hazmatbcsaller, ping?19:22
jimbakerniemeyer, sounds good, keep us focused19:22
bcsalleryeah, here19:23
hazmatbcsaller, skype?19:24
bcsallerhazmat: with all the desktop crashes with natty I sometimes forget to log back in :(19:24
=== bcsaller1 is now known as bcsaller
bcsallerhazmat: dropped, back now19:38
niemeyerhazmat: https://code.launchpad.net/~niemeyer/txaws/fix-path-with-bucket/+merge/5775319:43
_mup_Bug #761015 was filed: FORMULA_DIRECTORY should be available to hooks <Ensemble:New> < https://launchpad.net/bugs/761015 >19:53
_mup_ensemble/unit-agent-formula-upgrade r219 committed by kapil.thangavelu@canonical.com20:28
_mup_extract formula_download from machine agent into utility function20:28
_mup_Bug #761053 was filed: Formula state should carry a referenced to a download url <Ensemble:New> < https://launchpad.net/bugs/761053 >20:35
_mup_ensemble/unit-agent-formula-upgrade r220 committed by kapil.thangavelu@canonical.com20:35
_mup_note the relevant bug url by the implementation20:35
niemeyerjimbaker: ping20:55
hazmatbcsaller, put some comments on the yaml-state proposal21:19
bcsallerhazmat: still working on it but its pretty close to whats up there, thank you for looking at it :)21:19
jimbakerniemeyer, hi21:29
niemeyerYo21:30
_mup_ensemble/status-workflow r201 committed by jim.baker@canonical.com21:40
_mup_Collect workflow state for service units and their relations21:40
_mup_ensemble/status-workflow r202 committed by jim.baker@canonical.com22:00
_mup_Cleanup22:00
_mup_ensemble/unit-agent-formula-upgrade r221 committed by kapil.thangavelu@canonical.com22:12
_mup_additional agent upgrade tests.22:12
* niemeyer => execrising22:18
* niemeyer => exercising even22:18
=== niemeyer is now known as niemeyer_away
_mup_ensemble/unit-agent-formula-upgrade r222 committed by kapil.thangavelu@canonical.com22:25
_mup_minor prereview cleanups22:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!