[00:53] So.. I can deploy and shutdown reliably [00:53] something I've done with the setup.py is borking python-support... [00:55] http://paste.ubuntu.com/631507/ [00:59] hostname=`curl http://169.254.169.254/latest/meta-data/local-hostname` [00:59] THIS is where I'm failing! ;) [01:14] 2011-06-23 17:12:47,535 provision:lxc: ensemble.agents.provision INFO: Starting machine id:2 ... [01:14] ruh roh.. here. we go [01:16] SpamapS, that's a path issue [01:16] SpamapS, the ensemble/bin directory as spec to the hook isn't correct [01:16] I switched to using 'develop' instead of 'install' [01:17] SpamapS, cool, that's much better for a source installation [01:17] develop/symlinks install/copies [01:21] so.. [01:21] it works [01:21] the main problem I have now.. [01:21] wordpress needs the host header to be set.. [01:22] and chrome won't send a non fqdn host header [01:22] but like.. [01:22] they work [01:22] * SpamapS does a dance [01:22] got dnsmasq working too [01:26] ok its just that the wp formula doesn't make its config the default [01:27] So like, I can actually deploy wordpress on LXC [01:27] I do need to figure out the zookeeper shutdown/bootstrap [01:29] SpamapS: cheers... cork popping! [01:29] * SpamapS is tempted now to commence with optimizing it so it doesn't take 3 minutes per machine... [01:30] ironic, huh [01:31] SpamapS: dude, that's awesome. I definitely wanna set it up in Dublin [01:31] Yeah I hope to have it available as a package in a separate PPA by the end of next week [01:31] The refactoring I did needs more testing before its mergable. [02:01] SpamapS, awesome! [02:01] SpamapS, re reset.. this is a simple cli script i used before we had automated zk test setups.. https://pastebin.canonical.com/48970/ [02:01] to reset the tree, it could drop into ensemble/state/initialize [02:01] hazmat: I'm thinking it may make more sense to spawn a bastion ZK [02:02] it doesn't need its own bootstrap node.. [02:02] but it can just listen on a random port and we can feed that port to all the nodes esaily [02:02] easily even [02:02] SpamapS, yeah.. it does.. its one of many reasons that we wanted zk managed separately for tests (ie. don't accidently reset a prod env) [02:02] hmm. more context.. yeah a bastion node sounds nice [02:03] be it just a zk on a random port or.. a container [02:03] isolation is good [02:03] yeah it could be a container with just ZK actually [02:03] anwyay, family time [02:03] SpamapS, machine-0 ;-) enjoy [02:03] I'll try to submit the refactor as a MP tomorrow.. :) [02:03] hazmat: everything is pushed for now === daker_ is now known as daker [12:27] g'morning [13:24] bcsaller, the config-set-lifecycle review isn't in the kanban/associated to a bug report [13:26] fixed [14:50] * hazmat runs out for a quick errand, bbiab [14:55] Hi everybody. I just read the two parts of the ensemble tutorial and I still don't get it. What exactely is ensemble? Is it some kind of configuration management and if so, then what makes it better than chef or puppet? [14:59] Or is ensemble just for deploying machines on amazon?? [15:46] joerg_herzinger: Hi, the channel's pretty quiet right now... big rally next week [15:46] joerg_herzinger: I'm still just learning, but I can try to answer [15:47] joerg_herzinger: the key eye-opener for me was to look at some examples of relation hooks [15:50] Yes, those relations somehow do something like dependencies do for packages. [15:50] But still this is a very very minimal part of what configuration management tools can do... [15:50] joerg_herzinger: http://bazaar.launchpad.net/~ensemble-composers/principia/oneiric/wordpress/trunk/view/head:/hooks/db-relation-joined [15:50] joerg_herzinger: ensemble is more about service configuration than machine configuration [15:51] yuck, but that script does everything manually... and it is python. [15:52] joerg_herzinger: don't have to care about instances as much.. and only concentrate on services [15:52] joerg_herzinger: that's actually a great aspect of ensemble... it's independent of implementation language [15:53] joerg_herzinger: you specify the services you want deployed in your infrastructure [15:53] Ok, so if there is a service for wordpress I can use that one, if there is not I have to write one on my own in whatever language I want? [15:54] joerg_herzinger: and configure the deps [15:54] joerg_herzinger: the idea is similar to apt-get for the cloud, where there will be lots of formulas to use [15:55] joerg_herzinger: that you can also fork and customize yourself if necessary [15:55] joerg_herzinger: (again, I'm no expert yet) [15:55] Ok, so basically it is a reinvention of the wheel, because all of that can already be done in a much nicer way with chef. [15:56] joerg_herzinger: it's fundamentally different than chef though [15:56] And you in chef and puppet your can use ruby instead of ugly python. [15:56] joerg_herzinger: different experience when you sit down and use it [15:56] meaning better of worse? [15:57] joerg_herzinger: really pulls you up away from the cloud and hardware and let's you focus on services within your infrastructure [15:57] joerg_herzinger: it can even work with chef [15:57] joerg_herzinger: different... chef's great at configuring individual machines [15:58] just like chef and puppet can work with ensemble... I still don't get what ensemble really can do better or different. I just see what it can't do... [15:58] joerg_herzinger: but even opscode recognizes that things are moving towards a higher level of abstraction of the infrastructure [15:58] Oh, so ensemble scales up to several machines of the same type (same services...) [15:59] joerg_herzinger: for one it's higher level... "services" [15:59] joerg_herzinger: while at the same time it's tighter to the OS [15:59] joerg_herzinger: ensemble can take advantage of dpkg [16:00] joerg_herzinger: yes [16:00] joerg_herzinger: it can deploy a whole cluster at once [16:00] joerg_herzinger: it's really cool from that perspective [16:00] joerg_herzinger: the key part of ensemble (again my opinion... still learning) [16:01] joerg_herzinger: is the relation hooks [16:01] joerg_herzinger: I deploy mysql service (won't talk about cluster in this example,but it handles that fine) [16:01] Ok, sorry, but I don't take the "higher level" part. It's a tool like many others and those "services" you mention are just services. They shouldn't be treated in any other way than files or packages. [16:01] joerg_herzinger: then I deploy rails/drupal/wordpress... [16:02] And relations are such a simple thing that I just can't buy that as a big feature... [16:02] joerg_herzinger: ensemble handles the events that associate those services [16:02] joerg_herzinger: then I just tell ensemble to "add another rails node" [16:03] Just like I can tell knife (the chef tool) to add another rails node. So thats not a new and big feature... [16:03] joerg_herzinger: and the way the ensemble events fire the relation hooks in the formula [16:03] joerg_herzinger: makes it happen out to scale [16:04] joerg_herzinger: yes, that can accomplish the same thing [16:05] joerg_herzinger: but the way you configure the relation is different [16:05] Ok, sorry. Have to run now. I will definitely read more about ensemble, maybe there is something I missed all the time, but for now it really looks like ensemble is not made for me. [16:05] joerg_herzinger: I think the relation_changed hook gives great flexibility to how you deploy the service [16:06] joerg_herzinger: ok... later man [16:10] m_3: good work man....can't convert them all anyway :P [16:11] m_3: we need for screencasts...maybe something you and kim0 can work on next week [16:11] robbiew: yup... setting up blog stuff now [16:12] nice [16:13] ugly python [16:13] quality feedback [16:13] robbiew: I'll talk to kim0 about more screencasts too... [16:13] hazmat: yeah... I'll change the rails formula over from bash to ruby too [16:14] m_3, yeah.. for linking to an example formula the wordpress formula in the ensemble source tree is bashified [16:14] hazmat: heh [16:14] guys fix anything I said that was wrong please [16:15] hazmat: and it'll be prettier once we have more principia tools and templating stuff [16:15] it's still young... I gotta remember to mention that too [16:18] don't sweat it too much [16:18] it's FOSS...so some folks are religiously tied to their tools :P [16:18] robbiew: yup [16:19] debug-hooks are a huge feature over chef too [16:20] and debug log [16:21] m_3, so going through it, the only additions i'd make.. are relations are more than simple dependency management, they also model a private bi-directional communication channels between services for configuration [16:22] hazmat: thanks [16:22] m_3, m_3, imo its sometimes helpful to let people compare the equivalent of doing something in chef vs. ensemble.. if there already inquring about the differences, i think its pretty clear that the ensemble version is significantly simpler.. and the other big thing we do because we have a higher level way of talking about services, is that we encapsulate real reuse to a much better extent than any of the existing solutions. [16:22] imo [16:24] hazmat: what's a good way to say that the relations are dynamic during entire lifecycle? i.e., not just at deployment [16:24] hazmat: service mgmt, not just service config? "live" service config? [16:25] m_3, i think the example of adding another unit of wordpress/mediawiki some app, and then explaining how it is automatically configured with the db, and the memcache, and the front-end proxy is helpful in that regard.. [16:25] chef search has service discovery, but this is a little different [16:25] hazmat: oh, right... that shows it can happen at any point in lifecycle [17:14] <_mup_> txzookeeper/session-event-handling r51 committed by kapil.foss@gmail.com [17:14] <_mup_> unrecoverable is a client property not method. [17:23] bcsaller, if you have a chance, please have a look at the review queue today.. with niemeyer out this week, we're getting a backlog to get to 2 reviews per branch [17:24] I'll look, yeah [17:28] bcsaller, i think your config-set-lifecycle branched missed the review queue cause it wasn't attached to a bug.. i'm having a look at it now.. where you able to resolve the test failures? [17:29] hazmat: not specifically [17:30] bcsaller, is there anything else needed for the service-config stuff to land? [17:31] hazmat: there are things that can come after, like support in deploy but thats the core feature set [17:32] bcsaller, great [17:32] I didn't realize you were not seeing it [17:32] in the queue [17:33] bcsaller, aha.. i found it! [17:33] bcsaller, a missing yield [17:33] wow [17:33] thats awesome [17:33] I looked at it for a long time... [17:33] test_scheduler setUp needs a yield on add_service_from_formula [17:34] bcsaller, yeah.. a missing yield sucks [17:34] hazmat: so the branch now spawns its own ZK [17:35] hazmat: which means bootstrap/shutdown leaves the system, more or less, as it was (though I leave logs around for analysis) [17:35] hazmat: working on getting it to pass the full test suite now.. I broke some stuff for EC2.. ;) [17:36] SpamapS, how does it resolve the zk it setup? just a state file in ~/.ensemble ? [17:36] and next week I'll work with hallyn on making spawning a container faster w/ lxc-clone [17:36] SpamapS, awesome! [17:36] hazmat: I'm cheating by assuming 192.168.122.1 is me... [17:36] hazmat: but I intend to make that configurable at some point [17:37] SpamapS: lxc-clone is accepted upstream now. lightning-fast. [17:37] (with lvm) [17:37] the zk server logs aren't particularly useful.. imo.. they tend to be pretty verbose, i honestly think sending them to dev/null is just as useful ;-) [17:37] getting stack traces from normal usage in the logs is annoying [17:38] hallyn: so the way I'm thinking I'll do it is bootstrap will be the only thing that calls lxc-create .. it will create and boot the source node, let it do its thing, then shut it down. After that we'll just lxc-clone it. [17:38] hazmat: darn java programmers ;) [17:38] SpamapS: sounds good [17:38] Hey all! [17:39] SpamapS: note, as it stands, once you lxc-create a natty container, it wont' debootstrap it again [17:39] SpamapS: it just rsyncs it from /var/cache/lxc/natty-amd6 [17:39] 4 [17:39] niemeyer: hey [17:39] SpamapS: but i'm 2 steps away from having it working with cgroups-bin, stop distracting me :) [17:39] On my way to Dublin ATM [17:39] hallyn: I want to do all the cloud-init stuff too, then freeze that [17:40] turns out existing code was mistakenly calling the filename 'subsystem'. so like a trusting idiot i used it as subysstem [17:40] SpamapS: yeah, it'd be nice to get that done so smoser would quit his whining [17:40] niemeyer: I've been hacking away this week.. we have an LXC provider to play with next week. :-D [17:40] niemeyer: have a good flight! [17:41] hazmat: Was just checking that branch about FORMULA_DIR on the way to the airport, and am curious.. why did you choose to change the idea of running the formula under formula dir and instead providing an explicit variable? [17:41] SpamapS: Wow! [17:41] SpamapS: Sweet! [17:41] SpamapS: Looking forward to checking it out! [17:41] niemeyer: its still mostly chewing gum and duct tape.. but its functional. :) [17:42] SpamapS: A spike, that's still awesome for learning and getting an idea of the issues we'll face [17:45] niemeyer: this is your mule. STOP. received 1 of 2 deliveries.STOP.will notify if/when 2nd delivery arrives. STOP [17:45] lol [17:47] * SpamapS wonders if robbiew will get the carrot or the stick [17:49] niemeyer, hola, welcome back.. it does both [17:50] niemeyer, it executes the formula in the unit directory and sets the FORMULA_DIR environment variable [17:50] robbiew: Haha [17:50] s/executes the hook [17:50] hazmat: Why not executing it in the formula directory? [17:50] hazmat: As we originally planned? [17:51] hazmat: This would kill the need for the env variable, and is probably the most comfortable [17:51] niemeyer, its operating on the context of the unit, putting in the root of that context made sense to me.. the formula dir is at a defined subdir and accessible via the env [17:51] hazmat: Since data files, etc, will always be sitting within that directory [17:52] robbiew: The second one is supposed to arrive today, btw :-) [17:52] i still like the idea of the env variable it makes things like library path manipulation easier, i'd prefer it over CWD just for having a static reference to it [17:52] I like CWD .. the env variable can be saved off as the first thing in a hook that needs it.. FORMULA_DIR=$PWD [17:53] hazmat: My feeling is that the formula directory is a much more important reference point to the author than the unit directory [17:53] hazmat: "My stuff is here" [17:54] niemeyer: roger that [17:54] niemeyer, its easy enough to change, my thought was the formula is just a static def, ie. i consider it a read only dir. The hook's purpose is to manipulate the unit context, so it made sense to me have it execute there [17:55] i'll switch it out to the formula dir, and drop the env variable [17:55] hazmat: The formula manipulates / in general, right? [17:55] niemeyer, yeah.. its typically using absolute path to fs locations [17:56] SpamapS, my only concern about $PWD is that its mutable.. so if a formula changes locations and needs to reference its dir, its lost. [17:56] s/formula/hook [17:56] hazmat: Well, that's an issue no matter what [17:58] hazmat: Yeah, I think that its ok to suggest to a formula author that if they need to know where formulas are, they should be saving that off early... but I do see the utility in having it always known.. [17:58] hazmat, SpamapS: Yeah, I guess it'd make utilities that depend on the formula location easier to write [17:59] hazmat, SpamapS: We can do both! :-) [17:59] robbiew: It's Out for Delivery already.. should be there any time [18:00] niemeyer: cool...UPS? [18:00] robbiew: Yeah [18:00] okay..they typically come in the late afternoon...just don't want to miss it ;) [18:01] out of curiosity anyway know a way to extract a cd from a new thinkpad drive (i'd like to save it for a windows vm) before installing ubuntu across the entire drive? [18:01] robbiew: Yeah, it actually says "Scheduled Delivery: ... By End Of Day" [18:02] robbiew: Besides Out For Delivery [18:02] hazmat: Press that gray button in the middle of the drive [18:02] * niemeyer runs [18:03] niemeyer, it looks like a red button [18:03] * hazmat likes red buttons [18:03] hazmat: More seriously, when I actually cared about saving Windows, there was a utility to do this IIRC [18:03] hazmat: I did that in the previous thinkpad [18:03] hazmat: It generates the restoration CDs by itself [18:05] hmmm.. yeah.. might be a little more problematic on devices lacking a cd writer (x220).. [18:05] niemeyer, cool found it, thanks, not sure its going to work over the network though.. i guess i could extact the drive drop it an enclosure and dd the sectors as an alternative. [18:06] hazmat: Hmm, yeah, that might be trickier [18:27] Alright, I better get going [18:27] Heading to São Paulo soon [18:29] niemeyer: have nice flights. :) [18:30] niemeyer, have fun [18:32] hazmat: so.. I'm writing tests for the lxc provider.. I notice the order of the tests is alphabetic. Should I be doing a bootstrap/shutdown each time I need to test something that requires a bootstrapped environment? [18:32] hazmat: or does setup/teardown get called before/after each test? [18:32] niemeyer, yes [18:33] SpamapS, the setup/teardown is per test [18:33] hazmat: I suppose it should work fine as long as the machine ids are unique for each test [18:33] SpamapS, the alphabetic ordering is just the default output of the test runner, the actual test definitions are whatever grouping makes sense [18:33] hazmat: but then I need to have a final "ok shut everything down" after all tests.. [18:34] SpamapS, hmm. trial doesn't support layers.. i did something sort of akin to it for the zookeeper client session tests where i setup a zk cluster [18:34] SpamapS, basically using the atexit module to register a cleanup handler [18:35] hazmat: yeah I need a per-suite setup/teardown [18:35] Or, I don't mind just shutting down each test [18:36] the atexit usage here might be helpful if you want to do the atexit thing... http://bazaar.launchpad.net/~hazmat/txzookeeper/session-event-handling/view/head:/txzookeeper/tests/test_session.py [18:42] hazmat: seems to be working fine.. tho the lag that zookeeper has between opening its listen port and responding does make the tests a little slower === daker is now known as daker_ [18:43] also because we're running 'ensemble-admin' every time it craps all over the output. I should probably redirect that log elsewhere anyway [18:46] False alarm.. flight is delayed :( [18:46] hazmat: realistically, we should have a single test suite that runs through all providers generically [18:48] hmm.. maybe I should make LXC mockable.. that would be more interesting and easier to unit test [19:03] SpamapS, that might work better, since setting up lxc environment with networking does involve some host changes, but you have to be pretty sure of the mock'd response... it would be nice to toss in a few functional tests into ftests for lxc [19:03] I will definitely do that too [19:03] and the document the host setup for them in the module header [19:04] s/the/then [19:04] Yeah, I was planning to do more documenting of "how to play with this" today, but I want tests first. [20:10] at the denver airport, waiting for my flight to chicago and then on to dublin [20:15] jimbaker: what time do you arrive tomorrow? [20:15] must be early [20:16] jimbaker: I get in at 10:40pm ... save some jamesons for me. ;) [20:16] SpamapS, 8:25a on aer lingus [20:16] Ugh.. I just realized I have a 7 hour layover in Frankfurt. At least I know where the lounge is. [20:17] SpamapS, don't worry, i'm just interested in the beer :) [20:17] not a whisky drinker [20:17] I'm quite fond of Jamesons .. looking forward to maybe trying some of the lesser known brands. [20:18] time to go to the gate! [20:19] jimbaker: happy flying! [20:38] SpamapS, should be good, just trying out my new LTE modem right now. i won't bother with the red carpet club wifi next time in denver, for sure [20:39] (on plane) [20:40] jimbaker: hah cool! [20:40] jimbaker: I use my iphone 4's bluetooth or wifi tether. Works great. :) [20:40] fwiw, the little samsung 4G LTE wireless router seems to be a good choice [20:40] LTE would be nicer tho [20:41] SpamapS, my old droid started malfunctioning in budapest, so i'm now using a prepaid phone + LTE router [20:43] i figured i rather just get various wifi devices, to be financed in my part by the savings on my monthly telecommunication costs. [20:44] anyway, the combination of old and new is pretty funny :) [20:45] door is shut. bye! [20:48] * SpamapS scratches his head as he tries to figure out mocker [20:52] SpamapS, the mocker docs are worth a gander, and lots of examples in the code base.. i'd avoid looking at the ec2 provider mocks.. their a bit more extensive than what you'll probably be need. [20:54] SpamapS, but basically mocker records unordered events against objects and then when put into replay mode, will patch/replace the original objects and expect the same set of events (in the form of method signatures and return values) to be utilized. [20:54] Yeah I'm figuring it out [20:55] is ensemble supporting postgres ? [20:55] daker, not atm [20:55] :/ [20:55] daker: you can write a postgres formula. :) [20:55] ok [20:59] i am going to be crazy with preseed :/ [21:26] Hrm.. [21:26] hazmat: so I am trying to mock an object that gets called with an unpredictable temporary file name... [21:26] SpamapS, use a MATCH function [21:26] ahh... MATCH . right [21:27] just pgdn'd and saw those special bits [21:27] SpamapS, from ensemble.lib.mocker import MATCH.. you pass it a function that takes the parameter in question, if it returns true, its a successful match [21:27] and then you pass the MATCH(function) in the mock call [21:36] Ok I'm getting it.. works well.. cool [21:39] Ok.. my tests don't pass, but I'm on my way to making them pass. :-D [21:39] hazmat: lp:~clint-fewbar/ensemble/lxc-container ... I put in some functional tests for LXC too [21:45] SpamapS, awesome, i had a look yesterday at the branch, looks good. i'm guessing you and the folks in dublin will play around with it next week. [22:15] hazmat: not coming? :( [22:16] SpamapS, yeah.. sadly not.. my son is in town starting on monday.. the guiness brewery will still be there next time [22:16] hazmat: indeed, well enjoy the week.. you will be missed! [22:16] * SpamapS heads off to pack and jump on plane [22:24] SpamapS: have a great flight