/srv/irclogs.ubuntu.com/2011/04/21/#ubuntu-ensemble.txt

_mup_ensemble/ensemble-resolved r247 committed by kapil.thangavelu@canonical.com00:03
_mup_more resolved cli tests00:03
_mup_ensemble/resolved-state-api r194 committed by kapil.thangavelu@canonical.com01:13
_mup_use new dict merging utility func, and allow non conflicting updates to relation resolved settings.01:13
_mup_ensemble/ensemble-resolved r249 committed by kapil.thangavelu@canonical.com01:34
_mup_complete additional tests for resolved cli01:34
_mup_Bug #767948 was filed: ensemble resolved command is needed to recover from errors <Ensemble:In Progress by hazmat> < https://launchpad.net/bugs/767948 >01:36
_mup_Bug #767955 was filed: cli needs beautifcation <Ensemble:New> < https://launchpad.net/bugs/767955 >01:50
_mup_ensemble/trunk-merge r187 committed by kapil.thangavelu@canonical.com02:02
_mup_add a resolved spec02:02
_mup_ensemble/resolved-spec r187 committed by kapil.thangavelu@canonical.com02:04
_mup_add resolved spec02:04
_mup_Bug #767961 was filed: unit agent needs support for resolving units and unit relations <Ensemble:New for hazmat> < https://launchpad.net/bugs/767961 >02:05
_mup_Bug #767964 was filed: ensemble resolved spec/user doc for working with errors <Ensemble:New> < https://launchpad.net/bugs/767964 >02:05
_mup_ensemble/refactor-to-yamlstate r196 committed by bcsaller@gmail.com04:40
_mup_Hook context's relation settings -> YAMLState04:40
_mup_ensemble/config-get r200 committed by bcsaller@gmail.com06:57
_mup_YAMLState changes06:57
_mup_ensemble/trunk-merge r187 committed by kapil.thangavelu@canonical.com12:50
_mup_merge trunk12:50
_mup_ensemble/resolved-spec r189 committed by kapil.thangavelu@canonical.com12:53
_mup_additional doc text regarding --retry12:53
kim0hey folks o/12:54
kim0I'm trying to launch ensemble now, and every time the ec2 instance fails to boot (becomes terminated) ?!12:54
kim0I think a new natty image was pushed yesterday right ? it seems to have problems12:55
kim0hazmat: you mentioned the natty image, right13:06
hazmatkim0, sorry its not committed13:51
hazmati got caught up in working on ensemble resolved13:52
hazmatlooking at it now13:52
hazmatkim0, which makes your problems more interesting13:52
hazmatkim0, actually there is an ec2 problem in the default region13:52
hazmatkim0, they where having problems both with ebs and ec213:52
hazmatkim0, for http://www.reddit.com/r/golang/comments/gdn3d/express_go_jit_implementation_of_go/13:53
hazmatas an example13:53
hazmathttp://status.aws.amazon.com/13:53
hazmatyou can change the default region in the environments.yaml under the provider13:53
kim0hazmat: oh thanks13:55
_mup_Bug #768311 was filed: txaws deprecation warnings from elementree with ensemble usage <Ensemble:New> < https://launchpad.net/bugs/768311 >13:56
_mup_ensemble/unit-agent-resolved r251 committed by kapil.thangavelu@canonical.com13:58
_mup_unit and unit relation resolved watch callbacks.13:58
hazmathmm.. lame the txaws only supports two regions14:02
hazmatEU and US 14:02
kim0yeah 14:04
kim0still failing though14:04
kim0wonder if  us-east = North vigina14:04
hazmatkim0, it does14:07
hazmatkim0, hmm. the other problem is we only have images published in that region..14:07
hazmatec2-uri: https://ec2.us-west-1.amazonaws.com14:07
hazmatthat works as a syntax to talk to other regions but since we're not multi publishing them yet.. its kinda of moot14:07
kim0yeah .. so I'll wait a bit I guess14:07
hazmathmmm14:07
kim0so much for cloud :)14:08
hazmati can publish an image in another region, but we don't select internally based on region atm14:08
hazmatie. it would need a specification of the image id as well14:08
_mup_Bug #768320 was filed: ensemble should support more ec2 regions <Ensemble:New> < https://launchpad.net/bugs/768320 >14:10
hazmatkim0, nice article about the other sites down as a result ... http://eu.techcrunch.com/2011/04/21/amazon-ec2-goes-down-taking-with-it-reddit-foursquare-and-quora/14:26
kim04 hours and the amazon folks are still scrambling .. it's a shame14:34
hazmathmmm.. can't cut new images most of the daily amis  aren't resolvable correctly.15:08
=== deryck is now known as deryck[lunch]
SpamapSHave we considered taking over txaws?16:35
SpamapSOr at least ... inserting ensemble as a giant influence in its development?16:35
hazmatSpamapS, we haven't made the time for it, its really there for anyone who wants it16:38
hazmatjamu's been doing some work on it recently and going through old bug reports16:38
SpamapSThats good16:39
SpamapSmaybe a release soon?16:39
* SpamapS feels like a broken record16:39
jimbakerSpamapS, agreed that having a txaws release would be a good thing16:39
SpamapSIn Canonical, if you don't have a cadence, you just don't release. ;)16:39
jimbaker:)16:40
hazmati've got ensemble images in a few more regions, almost done getting region support working in ensemble16:40
SpamapSat this point txaws is a blocker for a) working on natty w/o warnings (my bad for silently fixing that on my box and not patching the package.. oops!), b) working w/ eucalyptus and openstack ... anything else?16:44
hazmatSpamapS, b) has been fixed in txaws16:51
hazmatthe warning thing is being worked on16:51
kim0did anyone test ensemble against openstack yet 16:52
hazmatkim0, just uec afaik16:52
SpamapShazmat: is that fix released?16:52
kim0hazmat: and uec works 16:52
kim0?16:52
SpamapSchuck and I tried it against openstack16:52
SpamapSsame fail as eucalyptus16:52
kim0ah so both fail16:53
SpamapSs316:53
SpamapSIt sounds like there's a patch16:53
kim0how hard is it to fix that16:53
hazmatSpamapS, no release, its committed to trunk16:53
SpamapShazmat: good enough for me, lets get that into the package. :)16:53
hazmatkim0, the issues you where having with txaws are also fixed.16:54
kim0hazmat: the logging ?16:54
hazmatkim0, no.. the initial setup, where we had to do a non head revision of txaws to get things working.16:54
kim0ah ok16:55
hazmatkim0, there are branches in review from gustavo on both the logging issues16:55
* kim0 nods16:55
TeTeTso there will be a packaged release of ensemble soon?16:55
kim0hazmat: just to be clear, you're saying with txaws trunk, we can deploy ensemble versus uec/openstack sucessfully ?16:55
hazmatTeTeT, i think we're going to have a daily ppa. if i remember the plan correctly, and then universe hopefully in 11.1016:56
_mup_Bug #768424 was filed: Setup a launchpad daily ppa recipe for ensemble and some deps <Ensemble:New> < https://launchpad.net/bugs/768424 >17:00
SpamapSTeTeT: ppa:ensemble/ppa exists now :)17:00
SpamapSTeTeT: for lucid and natty17:00
TeTeTSpamapS: wow, it works on lucid too?17:01
TeTeTgreat, just installed on a vm, now trying to connect it to the training UEC :)17:05
TeTeThow would I configure environment.yaml for my UEC cloud?17:06
hazmatTeTeT, ec2_uri and s3_uri17:06
SpamapSTeTeT: that won't work until we update txaws in the PPA17:07
hazmatbut i don't think txaws is in that ppa, of which something recent from trunk is needed.17:07
SpamapSmaybe we should make that happen.. :)17:07
hazmatSpamapS, is that a daily build recipe ppa?17:07
SpamapShazmat: definitely not .. its the one I've tested.17:07
TeTeTSpamapS: ah, ok, will postpone the test then17:07
SpamapShazmat: I think an  ensemble/daily would be the place for daily builds17:08
SpamapShazmat: basically the one in the PPA is the demo-able version17:08
hazmatSpamapS, sounds good17:09
hazmatSpamapS, i'll take a look at the bzr builder stuff after i get the region portability working17:30
=== deryck[lunch] is now known as deryck
hazmathmm. i can launch ensemble in another zone.. but deploying formulas seems to be more problematic.17:52
hazmatah.. the provisioning agent17:56
koolhead17hi all18:03
_mup_ensemble/ensemble-alternate-regions r205 committed by kapil.thangavelu@canonical.com18:07
_mup_region portability18:07
hazmathi koolhead17 18:08
kim0koolhead17: o/18:08
koolhead17how are things. hazmat amazon had some major issue today18:08
koolhead17hi kim018:09
kim0they're still down :/18:09
koolhead17yes18:09
koolhead17:)18:09
hazmatkoolhead17, we've  been working on a portability.. we had some code assumptions/dep issues with using alternate regions.18:09
hazmati'm still working on it, i've got it bootstrapping in europe and us-west-118:09
hazmatbut the provisioning agent still has thing to work out18:10
koolhead17k18:10
koolhead17kim0: point me to your blog which you suggested me to start reading from please18:11
hazmatinternally ensemble is composed of a set of agents, running on a set of machines  in a cloud environment. the provisioning agent is what's responsible for launching new machines for deployed services to live on.. bootstrap means getting a single machine in the environment running at least some of the core agents18:11
koolhead17ok18:12
koolhead17hazmat: i will focus today and read up on the documentation. i have got a amazon linux machine access from my friend18:13
koolhead17i hope that is what you guys meant by EC218:13
hazmatkoolhead17, ensemble really needs it own user provided ec2 account, its launching new machines for deployed services18:14
kim0koolhead17: you might enjoy http://foss-boss.blogspot.com/2010/10/pointnclick-guide-to-running-ubuntu-in.html18:14
kim0to get a basic understanding of ec218:14
koolhead17ok nice18:14
hazmatwe'll be looking at lxc support for 11.10 to allow people to run ensemble just on their own laptop/desktop hardware18:14
kim0you'll also find other cool stuff there :)18:14
koolhead17:)18:15
kim0hazmat: that would rock :)18:15
kim0hazmat: actually openstack now has lxc support ..18:15
koolhead17hazmat: okey18:15
kim0so hopefully it won't be too much changes for ensemble18:15
hazmatkim0, it does, i'm not sure how much its been tested/used though18:15
hazmathopefully its good.. there's a lot of libvirt lxc work the last few months18:16
koolhead17hazmat: kim0 apology but i need to know about the lxe full form18:16
hazmatkoolhead17, what of it? we'll have some attendance at a sprint on lxc in ubuntu with some others in august i think, we'd like to see lxc both within ec2 machines for isolation, and as a machine provider, the latter for desktop support.18:17
koolhead17hazmat: what is lxe ?18:18
hazmatwe've been looking at using libvirt integration, but at the moment raw lxc is a bit easier. openstack which uses libvirt recently added lxc support so that's worth investigating.18:18
hazmatkoolhead17, http://lxc.sf.net18:18
koolhead17thanks18:18
koolhead17kim0: we have used libvert only on openstack i suppose18:19
hazmatkoolhead17, its like a bsd jail or solaris zone.. its an os level containerization of a set of processes such that isolation and resource limits can be put on the process set.18:19
koolhead17like Cgroups if am not wrong?18:20
hazmatkoolhead17, libvirt sits on top of a number drivers for virtualization as an abstraction api, lxc is one of those drivers.18:20
hazmatkoolhead17, its based on cgroups yes18:20
koolhead17okey cool.18:20
hazmatcgroups and clone syscall for the namespace isolation  18:21
koolhead17waoo.18:22
hazmatSpamapS, jamu fixed the txaws warnings on trunk fwiw18:26
SpamapShazmat: yeah seems like txaws trunk is *t3h awesomeness*18:27
SpamapS:)18:27
kim0any easy way to upgrade to that yet18:28
* kim0 imagining ec2 engineers pulling out their hair now18:29
hazmatkim0, not outside bzr pull | bzr up atm, we can address as part of bug 76842418:29
_mup_Bug #768424: Setup a launchpad daily ppa recipe for ensemble and some deps <Ensemble:New> < https://launchpad.net/bugs/768424 >18:29
koolhead17kim0: does moinmoin gives you permission to import raw html? Am trying to be little optimist :P18:35
kim0I suppose not18:36
koolhead17kim0: that angle i would love to explore latex2html will be our key :P18:39
kim0koolhead17: a solution might be to write the docs in reST, from which sphinx can generated html for you, PDFs and moinmoin (it seems) can read reST directly (needs testing though)18:44
kim0generally however, writing in reST sounds like a good plan to me rather than anything else18:45
hazmatsupport for custom branches on deploy seems to be borked18:47
koolhead17kim0: let me think of another angle as well18:48
koolhead17:)18:48
kim0hazmat: oh I was planning on using gustavo's log fixes branch :)18:49
jimbakermoinmoin works well with rst, the only challenge is using sphinx directives not supported by it.  but that's usually pretty minor18:49
kim0jimbaker: do you know if the version of wiki.ubuntu.com does 18:49
hazmatkim0, it looks like bzr is requiring an lp username and key for the remote machines.18:49
hazmatnot sure why.. its a public branch18:49
hazmatkim0, that's in process of being updated18:50
hazmatfrom ancient 1.6 to 1.9 i think18:50
jimbakerkim0, i have only used python.org... but it's a simple plugin for moinmoin and a magic header to specify18:50
kim0great18:50
jimbakerprobably plugin is installed on moinmoin18:50
kim0jimbaker: koolhead17 is writing some openstack docs, that needs to be available in html/PDF/moinmoin18:51
kim0were not sure what's the best source format18:51
jimbakerhttp://moinmo.in/HelpOnParsers/ReStructuredText18:51
kim0it seems reST is a good choice for now18:51
kim0just hope wiki.u.c is updated soonish :)18:51
jimbakeryeah, i definitely will always recommend rst18:51
jimbakeri write everything in it technical that's not code18:52
kim0koolhead17: ^ me too :)18:52
hazmatand sometimes not even technical ;-)18:53
jimbakerkim0, see jythonbook.com, which i coauthored18:53
jimbakerwe actually produced microsoft word docs (ok open office, but for our publisher, it was ms word) and html from the sames source rst18:54
jimbakerftw18:54
jimbakerthe other thing i like is rst2pdf. it doesn't support the sphinx directives either, but it's great for precise production of pdfs. i use it for making presentations18:55
koolhead17rst2pdf is well enough18:55
jimbakerbcsaller, it seems to me that i need to augment HookContext to support open-port/close-port hook commands. in particular, this seems the right place to do it so that flush will properly commit changes19:00
jimbaker(in the future, we can take advantage of ZK multinode support too, so that the flush writes all or none of the nodes - https://issues.apache.org/jira/browse/ZOOKEEPER-96519:01
hazmatjimbaker, you'll need to use a separate yaml state, and incorporate it into flush19:01
hazmatyour probably going to end up conflicting with bcsaller's work on service config19:01
hazmatthere both exposing new hook apis19:01
bcsallerI'm thinking that like status HookContext is growing deps on too many interfaces and we'll have to refactor how that works in the future19:01
jimbakerhazmat, absolutely because it's on the service unit. not relation19:02
hazmatbcsaller, agreed19:02
bcsallerHookContext is becoming a kind of dumping ground19:02
hazmatbcsaller, well its going to become a facade over many different facilities19:02
jimbakerhazmat, that's one reason i'm pinging bcsaller, it seems that his service config stuff must :)19:02
hazmatas long as it can manage them all isomerically then hopefully its not too bad.19:02
hazmatelse we can refactor more19:02
jimbakermaybe we can have some sort of plugin mechanism with hookcontext?19:03
jimbakerwhatever that means19:03
bcsallermight be a better way to compose that object though, right now each will have to change flush for example19:03
jimbakerwe do want one thing to call flush on19:03
jimbakerand have an associated client id19:03
bcsallerwhere as we could have a set of strategy objects which get iterated over in said calls19:04
hazmatthe client id has been a constant string for a while ;-)19:04
jimbakerbcsaller, yes, that's what i wanted to call it :)19:04
jimbakerhazmat, i did not know that19:04
hazmati think it can wait, but yeah.. it would be simple to do a get/set/del across method names of an abstraction object19:04
jimbakerhere i am thinking it's supposed to do something19:04
bcsallerhazmat: if the debug stuff didn't need it to connect to existing sessions then its kind of a dead horse 19:04
hazmater. strategy plugins.. but at them moment it seems like more trouble than its worth.. somehow its going to conflict whereever it gets configured19:05
jimbakerhazmat, but these are touching independent pieces19:05
jimbakerwe just want to make sure that they can be accessed together, flushed together19:05
hazmatjimbaker, to the hook context, are they all just a set of yaml states and methods manipulating them?19:05
hazmat^aren't 19:06
jimbakerhazmat, i think they are indeed19:06
bcsallerhazmat: agreed for now, but I think the change is simple in the future and it will help keep the various interfaces different but composable 19:06
hazmatsounds good, you guys up for a standup bcsaller, jimbaker ?19:06
bcsalleryeah19:06
jimbakerhazmat, sounds good19:07
jimbaker{ "port/proto" : True, "port/proto": <other info>...}19:24
jimbaker**/units/<internal unit id>/ports**19:25
jimbaker{ "open": [{"port": 80, "proto": "tcp"}, ...]}19:28
SpamapSport isn't required right? there are portless protocols.. ;)19:40
hazmatcool working in other regions now19:44
hazmatSpamapS, it can be a port range19:44
hazmatSpamapS, there's some critical mass of args for modifying accesss on a security group ;-)19:45
hazmatwhich are different depending on your intent/args19:45
_mup_ensemble/ensemble-alternate-regions r206 committed by kapil.thangavelu@canonical.com19:46
_mup_remove some of the debug helpers19:46
_mup_ensemble/ensemble-alternate-regions r207 committed by kapil.thangavelu@canonical.com20:03
_mup_verify new ec2 environment schema for regions.20:03
_mup_ensemble/ensemble-alternate-regions r208 committed by kapil.thangavelu@canonical.com20:05
_mup_remove region alias compatibility with txaws (EU/US) in favor of partial domain names (us-east-1, ap-northeast-1, etc)20:05
koolhead17TeTeT: hey20:06
_mup_ensemble/ensemble-alternate-regions r209 committed by kapil.thangavelu@canonical.com20:07
_mup_failure to find an ssh-key is not fatal to an ec2 environment serialization...20:07
TeTeThi koolhead17 20:08
koolhead17TeTeT: Working on the Ubuntu Enterprise Cloud Course :)20:08
koolhead17saw your tweet. am curious about it ^^20:08
TeTeTkoolhead17: right now on a sprint and will present uec persistence in a few minutes20:09
TeTeTkoolhead17: what are you working on cloud wise these days?20:10
koolhead17openstack and trying to understand ensemble 20:10
koolhead17:)20:11
koolhead17kim0: announcing-project-reddwarf-database-as-a-service intersting :)20:14
hazmatbcsaller, jimbaker if you guys have time would you mind looking at the ensemble-alternate-region branch..20:14
kim0koolhead17: yep20:14
hazmati'd like to have ensemble working before ec2 is ;-)20:14
kim0hazmat: can I play with the branch now on a different ec2 region :)20:15
koolhead17hazmat: :P20:15
hazmatkim0, yes20:15
hazmatkim0, i had forget something basic i needed to do to make that work20:15
hazmatlike push the  branch to lp20:15
kim0hehee :)20:15
hazmatwell actually i had a name typo.. but same difference ;-)20:15
koolhead17hazmat: what prompted the project name ensemble20:16
kim0koolhead17: you can direct easy questions at me :)20:17
hazmatkoolhead17, roots are in group communication systems and paxos, where ensemble is term used to refer to the working set of intercommunicating servers20:17
hazmatbut that definiteion may change latter ;-)20:17
kim0ok that's a sophisticated answer20:18
kim0hazmat: I think that should go into a faq20:18
hazmati think gustavo would be a better person to ask if you want a good origin name story20:18
koolhead17kim0: hazmat i met a guy this every sweet fellow asking me do u work on Ubunduu20:18
hazmatkim0, true20:18
koolhead17he pronounced it so differently20:18
koolhead17:P20:18
kim0Can anyone think of two other questions to put in a faq :)20:18
kim0hazmat: so once you push the branch let me know how to play with it (I hope that doesn't conflict with me wanting the instances running ensemble-log fixes branch)20:20
hazmatkim0, you mean regarding the alternate region branch?20:21
TeTeTkim0: what's the difference between ensemble and <insert favorite config mgmt system>20:21
kim0yep20:21
TeTeTkim0: when will it be ready for production20:21
TeTeTkim0: how to contribute20:21
hazmatkim0, if you branch gustavo's and merge mine you should be able to run in a different region, i don't have images up for asia yet, but europe and us (x2) are covered20:21
hazmatshouldn't be any conflicts20:21
kim0TeTeT: man you're good :)20:21
TeTeTkim0: what system do I need to control ensemble from?20:22
TeTeTkim0: he he, doing my best, no matter where20:22
* kim0 starting a faq doc20:22
koolhead17kim0: there21:25
jimbakerhazmat, i need a txaws branch for your branch to work, right?21:38
jimbakerpresumably, since it's failing in txaws trunk...21:39
jimbakerbcsaller, are you seeing this problem too? http://pastebin.ubuntu.com/597140/21:47
bcsallerjimbaker: I haven't tried to run on ec2 in a while21:48
bcsallerand haven't seen that 21:48
jimbakerbcsaller, just trying out hazmat's new branch for alternative regions and it's not working for me21:49
hazmat jimbaker you also need to specify it as the remote branch21:49
hazmatensemble-branch in the provider21:49
jimbakerhazmat, yes, of course :)21:49
hazmatthe provisioning agent also relies on fixes in the branch21:49
hazmatjimbaker, it works with txaws trunk21:50
jimbakerall makes sense now... except for the mysterious failure part21:50
hazmatjimbaker, there's all kind of wierdness without the provisioning agent support.. we get partially initialized machine states, that status tends to barf on21:50
hazmatwhich arguably it should do better with21:51
* jimbaker needs to do annual checklist next time... was going to alter environments.yaml, but got left out21:51
jimbakerstill good experience for yet another bug report :)21:51
jimbakermanual, manual21:51
jimbakerrebooting really did a number on my mind it seems21:52
_mup_Bug #768595 was filed: Status should handle partially initialized machine states, ie. before they been handled by the provisioning agent <Ensemble:New> < https://launchpad.net/bugs/768595 >21:52
jimbakerhazmat, where should we report version discrepancies. might be nice if ensemble status did the check, since bootstrap seems too early21:54
hazmatjimbaker, we've got an open bug that to record bzr revno/branch info into zk21:55
hazmatprobably could add it to some sort of common  cli check under verbose mode21:55
jimbakerhazmat, yeah. so if we reported that in ensemble status, that could be a good place21:55
hazmatjimbaker, i'm getting a little concerned about  how much back and forth traffic ensemble status is doing..21:55
hazmatit be nice if we could avoid pulling down the entire zk tree21:55
hazmatto do a status listing21:56
hazmatand feels like we're going in that direction, i'd rather keep the default and let be do filtering and additional display options to target something than have the default be slow21:56
jimbakersure, but it does seem something that can be filtered on. but dev time is useful too21:56
hazmatits a very useful introspection tool21:57
hazmatand it would be nice to expand to do targeted display of relations values, ensemble versions, etc.21:57
jimbakera message highlighting "are you really sure" when running two different branches would be valuable21:57
hazmatjimbaker, how often do you use the ensemble-branch feature21:57
jimbakerhazmat, what's worse is when i leave it turned on :)21:58
hazmatmost of the time we're i'm just running trunk remotely21:58
hazmatjimbaker, true that21:58
hazmatthat's bit me more than once21:58
hazmatwe can play with it21:59
hazmati guess its only once to look at the bootstrap node21:59
jimbakerwhich is required by all entry points anyway22:00
jimbakerhazmat, your branch looks good. i can't currently run the examples formulas as-is, but that's waiting on the logging changes22:05
jimbakeri will try it one more time w/ the older versions of the example formulas22:05
jimbakerwhat's currently on http://ec2-184-72-28-229.us-west-1.compute.amazonaws.com/ demonstrates why the open-port setting needs to be done in the flush, ideally w/ all-or-nothing semantics22:06
jimbakerfor future ref: http://pastebin.ubuntu.com/597147/22:07
jimbaker"it works!" yeah, maybe it's fine, maybe it's really in a bad and unsecured state22:08
hazmatjimbaker, thats not the default incidentally, one of the formulas installed apache though23:20
hazmatserver comes with nothing running except ssh by default as far services23:21
hazmatfwiw23:21
jimbakerhazmat, sure, we just see a partial configuration23:42
jimbakerhazmat, i retried with the example formulas in r200 of trunk (the last time i had run it), and i'm still not able to deploy against us-west-123:42
jimbakerhazmat, i have seen this recur in a couple of attempted deploys - http://pastebin.ubuntu.com/597178/23:51
jimbakerit looks like the service relation is not being added, or something related to its workflow, based on what's missing from ensemble status23:53
jimbakeri really should look at this more directly in ZK, not certain what's going on here23:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!