/srv/irclogs.ubuntu.com/2011/06/24/#ubuntu-ensemble.txt

SpamapSSo.. I can deploy and shutdown reliably00:53
SpamapSsomething I've done with the setup.py is borking python-support...00:53
SpamapShttp://paste.ubuntu.com/631507/00:55
SpamapShostname=`curl http://169.254.169.254/latest/meta-data/local-hostname`00:59
SpamapSTHIS is where I'm failing! ;)00:59
SpamapS2011-06-23 17:12:47,535 provision:lxc: ensemble.agents.provision INFO: Starting machine id:2 ...01:14
SpamapSruh roh.. here. we go01:14
hazmatSpamapS, that's a path issue01:16
hazmatSpamapS, the ensemble/bin directory as spec to the hook isn't correct01:16
SpamapSI switched to using 'develop' instead of 'install'01:16
hazmatSpamapS, cool, that's much better for a source installation01:17
hazmatdevelop/symlinks install/copies01:17
SpamapSso..01:21
SpamapSit works01:21
SpamapSthe main problem I have now..01:21
SpamapSwordpress needs the host header to be set..01:21
SpamapSand chrome won't send a non fqdn host header01:22
SpamapSbut like..01:22
SpamapSthey work01:22
* SpamapS does a dance01:22
SpamapSgot dnsmasq working too01:22
SpamapSok its just that the wp formula doesn't make its config the default01:26
SpamapSSo like, I can actually deploy wordpress on LXC01:27
SpamapSI do need to figure out the zookeeper shutdown/bootstrap01:27
m_3SpamapS: cheers... cork popping!01:29
* SpamapS is tempted now to commence with optimizing it so it doesn't take 3 minutes per machine...01:29
m_3ironic, huh01:30
m_3SpamapS: dude, that's awesome.  I definitely wanna set it up in Dublin01:31
SpamapSYeah I hope to have it available as a package in a separate PPA by the end of next week01:31
SpamapSThe refactoring I did needs more testing before its mergable.01:31
hazmatSpamapS, awesome!02:01
hazmatSpamapS, re reset.. this is a simple cli script i used before we had automated zk test setups.. https://pastebin.canonical.com/48970/02:01
hazmatto reset the tree, it could drop into ensemble/state/initialize02:01
SpamapShazmat: I'm thinking it may make more sense to spawn a bastion ZK02:01
SpamapSit doesn't need its own bootstrap node..02:02
SpamapSbut it can just listen on a random port and we can feed that port to all the nodes esaily02:02
SpamapSeasily even02:02
hazmatSpamapS, yeah.. it does.. its one of many reasons that we wanted zk managed separately for tests (ie. don't accidently reset a prod env)02:02
hazmathmm. more context.. yeah a bastion node sounds nice02:02
hazmatbe it just a zk on a random port or.. a container02:03
hazmatisolation is good02:03
SpamapSyeah it could be a container with just ZK actually02:03
SpamapSanwyay, family time02:03
hazmatSpamapS, machine-0 ;-) enjoy02:03
SpamapSI'll try to submit the refactor as a MP tomorrow.. :)02:03
SpamapShazmat: everything is pushed for now02:03
=== daker_ is now known as daker
hazmatg'morning12:27
hazmatbcsaller, the config-set-lifecycle review isn't in the kanban/associated to a bug report13:24
hazmatfixed13:26
* hazmat runs out for a quick errand, bbiab14:50
joerg_herzingerHi everybody. I just read the two parts of the ensemble tutorial and I still don't get it. What exactely is ensemble? Is it some kind of configuration management and if so, then what makes it better than chef or puppet?14:55
joerg_herzingerOr is ensemble just for deploying machines on amazon??14:59
m_3joerg_herzinger: Hi, the channel's pretty quiet right now... big rally next week15:46
m_3joerg_herzinger: I'm still just learning, but I can try to answer15:46
m_3joerg_herzinger: the key eye-opener for me was to look at some examples of relation hooks15:47
joerg_herzingerYes, those relations somehow do something like dependencies do for packages.15:50
joerg_herzingerBut still this is a very very minimal part of what configuration management tools can do...15:50
m_3joerg_herzinger: http://bazaar.launchpad.net/~ensemble-composers/principia/oneiric/wordpress/trunk/view/head:/hooks/db-relation-joined15:50
m_3joerg_herzinger: ensemble is more about service configuration than machine configuration15:50
joerg_herzingeryuck, but that script does everything manually... and it is python.15:51
m_3joerg_herzinger: don't have to care about instances as much.. and only concentrate on services15:52
m_3joerg_herzinger: that's actually a great aspect of ensemble... it's independent of implementation language15:52
m_3joerg_herzinger: you specify the services you want deployed in your infrastructure15:53
joerg_herzingerOk, so if there is a service for wordpress I can use that one, if there is not I have to write one on my own in whatever language I want?15:53
m_3joerg_herzinger: and configure the deps15:54
m_3joerg_herzinger: the idea is similar to apt-get for the cloud, where there will be lots of formulas to use15:54
m_3joerg_herzinger: that you can also fork and customize yourself if necessary15:55
m_3joerg_herzinger: (again, I'm no expert yet)15:55
joerg_herzingerOk, so basically it is a reinvention of the wheel, because all of that can already be done in a much nicer way with chef.15:55
m_3joerg_herzinger: it's fundamentally different than chef though15:56
joerg_herzingerAnd you in chef and puppet your can use ruby instead of ugly python.15:56
m_3joerg_herzinger: different experience when you sit down and use it15:56
joerg_herzingermeaning better of worse?15:56
m_3joerg_herzinger: really pulls you up away from the cloud and hardware and let's you focus on services within your infrastructure15:57
m_3joerg_herzinger: it can even work with chef15:57
m_3joerg_herzinger: different... chef's great at configuring individual machines15:57
joerg_herzingerjust like chef and puppet can work with ensemble... I still don't get what ensemble really can do better or different. I just see what it can't do...15:58
m_3joerg_herzinger: but even opscode recognizes that things are moving towards a higher level of abstraction of the infrastructure15:58
joerg_herzingerOh, so ensemble scales up to several machines of the same type (same services...)15:58
m_3joerg_herzinger: for one it's higher level... "services"15:59
m_3joerg_herzinger: while at the same time it's tighter to the OS15:59
m_3joerg_herzinger: ensemble can take advantage of dpkg15:59
m_3joerg_herzinger: yes16:00
m_3joerg_herzinger: it can deploy a whole cluster at once16:00
m_3joerg_herzinger: it's really cool from that perspective16:00
m_3joerg_herzinger: the key part of ensemble (again my opinion... still learning)16:00
m_3joerg_herzinger: is the relation hooks16:01
m_3joerg_herzinger: I deploy mysql service (won't talk about cluster in this example,but it handles that fine)16:01
joerg_herzingerOk, sorry, but I don't take the "higher level" part. It's a tool like many others and those "services" you mention are just services. They shouldn't be treated in any other way than files or packages.16:01
m_3joerg_herzinger: then I deploy rails/drupal/wordpress...16:01
joerg_herzingerAnd relations are such a simple thing that I just can't buy that as a big feature...16:02
m_3joerg_herzinger: ensemble handles the events that associate those services16:02
m_3joerg_herzinger: then I just tell ensemble to "add another rails node"16:02
joerg_herzingerJust like I can tell knife (the chef tool) to add another rails node. So thats not a new and big feature...16:03
m_3joerg_herzinger: and the way the ensemble events fire the relation hooks in the formula16:03
m_3joerg_herzinger: makes it happen out to scale16:03
m_3joerg_herzinger: yes, that can accomplish the same thing16:04
m_3joerg_herzinger: but the way you configure the relation is different16:05
joerg_herzingerOk, sorry. Have to run now. I will definitely read more about ensemble, maybe there is something I missed all the time, but for now it really looks like ensemble is not made for me.16:05
m_3joerg_herzinger: I think the relation_changed hook gives great flexibility to how you deploy the service16:05
m_3joerg_herzinger: ok... later man16:06
robbiewm_3: good work man....can't convert them all anyway :P16:10
robbiewm_3: we need for screencasts...maybe something you and kim0 can work on next week16:11
m_3robbiew: yup... setting up blog stuff now16:11
robbiewnice16:12
hazmat  ugly python16:13
hazmatquality feedback16:13
m_3robbiew: I'll talk to kim0 about more screencasts too...16:13
m_3hazmat: yeah... I'll change the rails formula over from bash to ruby too16:13
hazmatm_3, yeah.. for linking to an example formula the wordpress formula in the ensemble source tree is bashified16:14
robbiewhazmat: heh16:14
m_3guys fix anything I said that was wrong please16:14
m_3hazmat: and it'll be prettier once we have more principia tools and templating stuff16:15
m_3it's still young... I gotta remember to mention that too16:15
robbiewdon't sweat it too much16:18
robbiewit's FOSS...so some folks are religiously tied to their tools :P16:18
m_3robbiew: yup16:18
m_3debug-hooks are a huge feature over chef too16:19
hazmatand debug log16:20
hazmatm_3, so going through it, the only additions i'd make.. are relations are more than simple dependency management, they also model a private bi-directional communication channels between services for configuration16:21
m_3hazmat: thanks16:22
hazmatm_3, m_3, imo its sometimes helpful to let people compare the equivalent of doing something in chef vs. ensemble.. if there already inquring about the differences, i think its pretty clear that the ensemble version is significantly simpler..  and the other big thing we do because we have a  higher level way of talking about services, is that we encapsulate real reuse to a much better extent than any of the existing solutions.16:22
hazmatimo16:22
m_3hazmat: what's a good way to say that the relations are dynamic during entire lifecycle? i.e., not just at deployment16:24
m_3hazmat: service mgmt, not just service config?  "live" service config?16:24
hazmatm_3, i think the example of adding another unit of wordpress/mediawiki some app, and then explaining how it is automatically configured with the db, and the memcache, and the front-end proxy is helpful in that regard..16:25
m_3chef search has service discovery, but this is a little different16:25
m_3hazmat: oh, right... that shows it can happen at any point in lifecycle16:25
_mup_txzookeeper/session-event-handling r51 committed by kapil.foss@gmail.com17:14
_mup_unrecoverable is a client property not method.17:14
hazmatbcsaller, if you have a chance, please have a look at the review queue today.. with niemeyer out this week, we're getting a backlog to get to 2 reviews per branch17:23
bcsallerI'll look, yeah17:24
hazmatbcsaller, i think your config-set-lifecycle branched missed the review queue cause it wasn't attached to a bug.. i'm having a look at it now.. where you able to resolve the test failures?17:28
bcsallerhazmat: not specifically 17:29
hazmatbcsaller, is there anything else needed for the service-config stuff to land?17:30
bcsallerhazmat: there are things that can come after, like support in deploy but thats the core feature set17:31
hazmatbcsaller, great17:32
bcsallerI didn't realize you were not seeing it 17:32
bcsallerin the queue17:32
hazmatbcsaller, aha.. i found it!17:33
hazmatbcsaller, a missing yield17:33
bcsallerwow17:33
bcsallerthats awesome17:33
bcsallerI looked at it for a long time...17:33
hazmattest_scheduler setUp needs a yield on add_service_from_formula17:33
hazmatbcsaller, yeah.. a missing yield sucks17:34
SpamapShazmat: so the branch now spawns its own ZK17:34
SpamapShazmat: which means bootstrap/shutdown leaves the system, more or less, as it was (though I leave logs around for analysis)17:35
SpamapShazmat: working on getting it to pass the full test suite now.. I broke some stuff for EC2.. ;)17:35
hazmatSpamapS, how does it resolve the zk it setup? just a state file in ~/.ensemble ?17:36
SpamapSand next week I'll work with hallyn on making spawning a container faster w/ lxc-clone17:36
hazmatSpamapS, awesome!17:36
SpamapShazmat: I'm cheating by assuming 192.168.122.1 is me...17:36
SpamapShazmat: but I intend to make that configurable at some point17:36
hallynSpamapS: lxc-clone is accepted upstream now.  lightning-fast.17:37
hallyn(with lvm)17:37
hazmatthe zk server logs aren't particularly useful.. imo.. they tend to be pretty verbose, i honestly think sending them to dev/null is just as useful ;-)17:37
hazmatgetting stack traces from normal usage in the logs is annoying17:37
SpamapShallyn: so the way I'm thinking I'll do it is bootstrap will be the only thing that calls lxc-create .. it will create and boot the source node, let it do its thing, then shut it down. After that we'll just lxc-clone it.17:38
SpamapShazmat: darn java programmers ;)17:38
hallynSpamapS: sounds good17:38
niemeyerHey all!17:38
hallynSpamapS: note, as it stands, once you lxc-create a natty container, it wont' debootstrap it again17:39
hallynSpamapS: it just rsyncs it from /var/cache/lxc/natty-amd617:39
hallyn417:39
hallynniemeyer: hey17:39
hallynSpamapS: but i'm 2 steps away from having it working with cgroups-bin, stop distracting me :)17:39
niemeyerOn my way to Dublin ATM17:39
SpamapShallyn: I want to do all the cloud-init stuff too, then freeze that17:39
hallynturns out existing code was mistakenly calling the filename 'subsystem'.  so like a trusting idiot i used it as subysstem17:40
hallynSpamapS: yeah, it'd be nice to get that done so smoser would quit his whining17:40
SpamapSniemeyer: I've been hacking away this week.. we have an LXC provider to play with next week.  :-D17:40
hallynniemeyer: have a good flight!17:40
niemeyerhazmat: Was just checking that branch about FORMULA_DIR on the way to the airport, and am curious.. why did you choose to change the idea of running the formula under formula dir and instead providing an explicit variable?17:41
niemeyerSpamapS: Wow!17:41
niemeyerSpamapS: Sweet!17:41
niemeyerSpamapS: Looking forward to checking it out!17:41
SpamapSniemeyer: its still mostly chewing gum and duct tape.. but its functional. :)17:41
niemeyerSpamapS: A spike, that's still awesome for learning and getting an idea of the issues we'll face17:42
robbiewniemeyer: this is your mule. STOP. received 1 of 2 deliveries.STOP.will notify if/when 2nd delivery arrives. STOP17:45
robbiewlol17:45
* SpamapS wonders if robbiew will get the carrot or the stick17:47
hazmatniemeyer, hola, welcome back.. it does both17:49
hazmatniemeyer, it executes the formula in the unit directory and sets the FORMULA_DIR environment variable17:50
niemeyerrobbiew: Haha17:50
hazmats/executes the hook17:50
niemeyerhazmat: Why not executing it in the formula directory?17:50
niemeyerhazmat: As we originally planned?17:50
niemeyerhazmat: This would kill the need for the env variable, and is probably the most comfortable17:51
hazmatniemeyer, its operating on the context of the unit, putting in the root of that context made sense to me.. the formula dir is at a defined subdir and accessible via the env17:51
niemeyerhazmat: Since data files, etc, will always be sitting within that directory17:51
niemeyerrobbiew: The second one is supposed to arrive today, btw :-)17:52
hazmati still like the idea of the env variable it makes things like library path manipulation easier, i'd prefer it over CWD just for having a static reference to it17:52
SpamapSI like CWD .. the env variable can be saved off as the first thing in a hook that needs it..  FORMULA_DIR=$PWD17:52
niemeyerhazmat: My feeling is that the formula directory is a much more important reference point to the author than the unit directory17:53
niemeyerhazmat: "My stuff is here"17:53
robbiewniemeyer: roger that17:54
hazmatniemeyer, its easy enough to change, my thought was  the formula is just a static def, ie. i consider it a read only dir. The hook's purpose is to manipulate the unit context, so it made sense to me have it execute there17:54
hazmati'll switch it out to the formula dir, and drop the env variable17:55
niemeyerhazmat: The formula manipulates / in general, right?17:55
hazmatniemeyer, yeah.. its typically using absolute path to fs locations17:55
hazmatSpamapS, my only concern about $PWD is that its mutable.. so if a formula changes locations and needs to reference its dir, its lost.17:56
hazmats/formula/hook17:56
niemeyerhazmat: Well, that's an issue no matter what17:56
SpamapShazmat: Yeah, I think that its ok to suggest to a formula author that if they need to know where formulas are, they should be saving that off early... but I do see the utility in having it always known..17:58
niemeyerhazmat, SpamapS: Yeah, I guess it'd make utilities that depend on the formula location easier to write17:58
niemeyerhazmat, SpamapS: We can do both! :-)17:59
niemeyerrobbiew: It's Out for Delivery already.. should be there any time17:59
robbiewniemeyer: cool...UPS?18:00
niemeyerrobbiew: Yeah18:00
robbiewokay..they typically come in the late afternoon...just don't want to miss it ;)18:00
hazmatout of curiosity anyway know a way to extract a cd from a new thinkpad drive (i'd like to save it for a windows vm) before installing ubuntu across the entire drive?18:01
niemeyerrobbiew: Yeah, it actually says "Scheduled Delivery: ... By End Of Day"18:01
niemeyerrobbiew: Besides Out For Delivery18:02
niemeyerhazmat: Press that gray button in the middle of the drive18:02
* niemeyer runs18:02
hazmatniemeyer, it looks like a red button18:03
* hazmat likes red buttons18:03
niemeyerhazmat: More seriously, when I actually cared about saving Windows, there was a utility to do this IIRC18:03
niemeyerhazmat: I did that in the previous thinkpad18:03
niemeyerhazmat: It generates the restoration CDs by itself18:03
hazmathmmm.. yeah.. might be a little more problematic on devices lacking a cd writer (x220)..18:05
hazmatniemeyer, cool found it, thanks, not sure its going to work over the network though.. i guess i could extact the drive drop it an enclosure and dd the sectors as an alternative.18:05
niemeyerhazmat: Hmm, yeah, that might be trickier18:06
niemeyerAlright, I better get going18:27
niemeyerHeading to São Paulo soon18:27
SpamapSniemeyer: have nice flights. :)18:29
hazmatniemeyer, have fun18:30
SpamapShazmat: so.. I'm writing tests for the lxc provider.. I notice the order of the tests is alphabetic. Should I be doing a bootstrap/shutdown each time I need to test something that requires a bootstrapped environment?18:32
SpamapShazmat: or does setup/teardown get called before/after each test?18:32
hazmatniemeyer, yes18:32
hazmatSpamapS, the setup/teardown is per test18:33
SpamapShazmat: I suppose it should work fine as long as the machine ids are unique for each test18:33
hazmatSpamapS, the alphabetic ordering is just the default output of the test runner, the actual test definitions are whatever grouping makes sense18:33
SpamapShazmat: but then I need to have a final "ok shut everything down" after all tests..18:33
hazmatSpamapS, hmm. trial doesn't support layers.. i did something sort of akin to it for the zookeeper client session tests where i setup a zk cluster18:34
hazmatSpamapS, basically using the atexit module to register a cleanup handler18:34
SpamapShazmat: yeah I need a per-suite setup/teardown18:35
SpamapSOr, I don't mind just shutting down each test18:35
hazmatthe atexit usage here might be helpful if you want to do the atexit thing... http://bazaar.launchpad.net/~hazmat/txzookeeper/session-event-handling/view/head:/txzookeeper/tests/test_session.py18:36
SpamapShazmat: seems to be working fine.. tho the lag that zookeeper has between opening its listen port and responding does make the tests a little slower18:42
=== daker is now known as daker_
SpamapSalso because we're running 'ensemble-admin' every time it craps all over the output. I should probably redirect that log elsewhere anyway18:43
niemeyerFalse alarm.. flight is delayed :(18:46
SpamapShazmat: realistically, we should have a single test suite that runs through all providers generically18:46
SpamapShmm.. maybe I should make LXC mockable.. that would be more interesting and easier to unit test18:48
hazmatSpamapS, that might work better, since setting up lxc environment with networking does involve some host changes, but you have to be pretty sure of the mock'd response... it would be nice to toss in a few functional tests into ftests for lxc19:03
SpamapSI will definitely do that too19:03
hazmatand the document the host setup for them in the module header19:03
hazmats/the/then19:04
SpamapSYeah, I was planning to do more documenting of "how to play with this" today, but I want tests first.19:04
jimbakerat the denver airport, waiting for my flight to chicago and then on to dublin20:10
SpamapSjimbaker: what time do you arrive tomorrow?20:15
SpamapSmust be early20:15
SpamapSjimbaker: I get in at 10:40pm ... save some jamesons for me. ;)20:16
jimbakerSpamapS, 8:25a on aer lingus 20:16
SpamapSUgh.. I just realized I have a 7 hour layover in Frankfurt. At least I know where the lounge is.20:16
jimbakerSpamapS, don't worry, i'm just interested in the beer :)20:17
jimbakernot a whisky drinker20:17
SpamapSI'm quite fond of Jamesons .. looking forward to maybe trying some of the lesser known brands.20:17
jimbakertime to go to the gate!20:18
SpamapSjimbaker: happy flying!20:19
jimbakerSpamapS, should be good, just trying out my new LTE modem right now. i won't bother with the red carpet club wifi next time in denver, for sure20:38
jimbaker(on plane)20:39
SpamapSjimbaker: hah cool!20:40
SpamapSjimbaker: I use my iphone 4's bluetooth or wifi tether. Works great. :)20:40
jimbakerfwiw, the little samsung 4G LTE wireless router seems to be a good choice20:40
SpamapSLTE would be nicer tho20:40
jimbakerSpamapS, my old droid started malfunctioning in budapest, so i'm now using a prepaid phone + LTE router20:41
jimbakeri figured i rather just get various wifi devices, to be financed in my part by the savings on my monthly telecommunication costs.20:43
jimbakeranyway, the combination of old and new is pretty funny :)20:44
jimbakerdoor is shut. bye!20:45
* SpamapS scratches his head as he tries to figure out mocker20:48
hazmatSpamapS, the mocker docs are worth a gander, and lots of examples in the code base.. i'd avoid looking at the ec2 provider mocks.. their a bit more extensive than what you'll probably be need.20:52
hazmatSpamapS, but basically mocker records unordered events against objects and then when put into replay mode, will patch/replace the original objects and expect the same set of events (in the form of method signatures and return values) to be utilized.20:54
SpamapSYeah I'm figuring it out20:54
dakeris ensemble supporting postgres ?20:55
hazmatdaker, not atm20:55
daker:/20:55
SpamapSdaker: you can write a postgres formula. :)20:55
dakerok20:55
dakeri am going to be crazy with preseed :/20:59
SpamapSHrm..21:26
SpamapShazmat: so I am trying to mock an object that gets called with an unpredictable temporary file name...21:26
hazmatSpamapS, use a MATCH function21:26
SpamapSahh... MATCH . right21:26
SpamapSjust pgdn'd and saw those special bits21:27
hazmatSpamapS, from ensemble.lib.mocker import MATCH.. you pass it a function that takes the parameter in question, if it returns true, its a successful match21:27
hazmatand then you pass the MATCH(function) in the mock call21:27
SpamapSOk I'm getting it.. works well.. cool21:36
SpamapSOk.. my tests don't pass, but I'm on my way to making them pass. :-D21:39
SpamapShazmat: lp:~clint-fewbar/ensemble/lxc-container ... I put in some functional tests for LXC too21:39
hazmatSpamapS, awesome, i had a look yesterday at the branch, looks good. i'm guessing you and the folks in dublin will play around with it next week. 21:45
SpamapShazmat: not coming? :(22:15
hazmatSpamapS, yeah.. sadly not.. my son is in town starting on monday.. the guiness brewery will still be there next time22:16
SpamapShazmat: indeed, well enjoy the week.. you will be missed!22:16
* SpamapS heads off to pack and jump on plane22:16
m_3SpamapS: have a great flight22:24

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!