[00:03] <_mup_> ensemble/debug-log-relation-settings-changes r273 committed by jim.baker@canonical.com [00:03] <_mup_> Change items of strings should be cutoff in logging if they exceed some arbitrary, but reasonable limit (for now, that is 100) [00:17] heh.. JFDI has turn into "JFDTFAOP" .. Just F'ing Do This, Forget All Other Priorities [00:26] SpamapS, can you paste the cloud-init data [00:26] hazmat: at this point I've got it connecting to ZK [00:26] SpamapS, are you able to ssh into the bootstrap contaienr? [00:27] SpamapS, awesome [00:27] but there's nothing in ZK [00:27] ensemble.state.errors.MachineStateNotFound: Machine 0 was not found [00:27] SpamapS, right.. there's an initialize zk command that runs before the agent is started [00:28] OH! that got dropped somehow [00:28] SpamapS, its at the bottom of ec2/launch.py.. if you paste your cloud-init config it should be easy to spot [00:29] For me now its in common.get_bootstrap_machine_variables .. [00:29] SpamapS, try running this if your can ssh into the bootstrap container.. /usr/local/bin/ensemble-admin initialize --instance-id=$(curl http://169.254.169.254/1.0/meta-data/instance-id) --admin-identity=abc [00:30] when you say nothing is in zk? are you using the shell to have a look ie. /usr/share/zookeeper/bin/zkCli.sh [00:48] hazmat: ok, the logic I had put in was overwriting those commands [00:48] hazmat: the curl won't work of course.. :) [00:49] SpamapS, cool, if you got zk up and running and clients connecting to it, that should be hopefully most of it. [00:50] hazmat: for the moment, I'm using the ZK running on my box letting them connect to it [00:51] SpamapS, right on.. that's still pretty good progress [00:51] exciting even [00:52] Another twistd server is running, PID 1100 [00:52] hrm [00:53] oh.. now my logic fix reverted my earlier hack [00:55] * dannf waves [00:56] hazmat: way I see it, this should be very easy to morph into libvirt for somebody who understands libvirt better [00:57] SpamapS, indeed, with the first non-ec2 provider impl, the next should go much easier. [00:57] hazmat: this still has more hacks than Wenesday night open mic at the improv.. [00:58] SpamapS, no worries, they can be cleaned up [00:59] Its still likely the top feature requested imo, and having something in-dev/demoable on that is awesome [00:59] hazmat: how does ensemble-admin know what ZK to talk to? In the example it doesn't have ENSEMBLE_ZOOKEEPER set [01:00] SpamapS, it knows its running on the same host as the zk, so it uses localhost [01:00] and the default port... [01:01] ... [01:01] hence my problem ;) [01:01] * SpamapS abandons the "keep it simple" approach and installs zookeeperd [01:01] SpamapS, it should take a cli arg.. [01:01] * hazmat checks [01:01] no its adding logic not making it simpler [01:01] this is better. ;) [01:02] SpamapS, you can set an environment variable for it [01:02] ZOOKEEPER_ADDRESS [01:02] it will use that in preference to the default localhost:2181 address [01:03] thats ok, I'm trying very hard to change as little as possible.. [01:03] as my head wraps around the existing hacks, I let them back in and revert changes to try and fix them the wrong way. ;) [01:03] SpamapS, some change and refactoring is ok in my book. [01:04] we can make the default ec2 command work that way as well [01:04] Yeah there's plenty of that thus far [01:06] UGGGH zookeeperd installs another 25 packages [01:07] SpamapS, indeed. that's pretty much why we have an AMI at all [01:08] [zk: localhost:2181(CONNECTED) 2] ls / [01:08] [services, formulas, relations, zookeeper, initialized, topology, machines, units] [01:08] SpamapS, WOOT! [01:08] still nothing tho [01:08] SpamapS, any agents connected? (ps aux | grep ensemble) [01:09] hazmat: there's a machine-00000000000 [01:09] hazmat: the admin command ran after the agents started tho [01:09] SpamapS, yeah.. that's representing the bootstrap machine (the one with zk), its go a machine agent, but we don't ever reuse it for service unit assignment [01:10] SpamapS, more important is the provisioning agent.. it actually waits till the user tries to deploy a service, and the provider credentials are provided lazily to it (upon deploy).. its that way mainly so we can avoid passing credentials via user-data [01:11] SpamapS, i'd check to see if its running, tail its log, and then try to deploy a service and see what happens [01:11] * hazmat is looking forward to hear about structure 2011 [01:12] looks like their streaming live [01:14] hazmat: so far 'ensemble status' shows empty [01:17] hazmat: I think eventually we can take the AMI thing to a level where each service gets snapshotted whenever you upgrade or deploy.. so add unit is just popping up a snapshot [01:18] hazmat: was thinking the other day that if formulas could call 'ensemble-snapshot-service-unit' or something that would be cool [01:20] clint@clint-MacBookPro:~/src/ensemble/lxc-container$ ZOOKEEPER_ADDRESS=192.168.122.68:2181 PYTHONPATH=`pwd` bin/ensemble status -e lxc [01:20] 2011-06-22 17:20:20,867 ERROR Machine provider information missing: machine 0 [01:20] machines: 0: {instance-id: ensemble-0} [01:20] services: {} [01:20] 2011-06-22 17:20:20,869 INFO 'status' command finished successfully [01:20] clint@clint-MacBookPro:~/src/ensemble/lxc-container$ [01:20] OOOHHHH SNAP [01:26] SpamapS, cool!, that missing info thing, is just implementing the get_machine api of the provider [01:26] I'm having a weird issue.. [01:26] its happening even w/ trunk.. [01:27] http://paste.ubuntu.com/631028/ [01:28] hmmm.. that looks like a speling typo on repository [01:28] SpamapS, ^ [01:29] lol [01:31] ensemble.environment.errors.EnvironmentsConfigError: Environments configuration error: environments.lxc.type: expected 'ec2', got 'lxc' [01:31] Heh.. [01:31] I need to put *my* version of ensemble on there.. duh [01:33] it would be nice to have that doublechecking occur! [01:33] SpamapS, doh.. yeah.. ensemble-branch in environments [01:33] hazmat: I didn't implement that yet. :-P [01:34] SpamapS, ah.. write the switch out is part of cloud-init [01:34] so there are two problems ;) [01:34] hazmat: have to branch it before I can switch it. ;) [01:34] anyway, it will be nice to get bug 719517 in place [01:34] <_mup_> Bug #719517: Ensemble should store the code version in each zookeeper < https://launchpad.net/bugs/719517 > [01:43] crap.. I need to use a mirror.. they just released a firefox update and archive.ubuntu.com is slammed [04:47] <_mup_> Bug #800988 was filed: refactor ec2 provider to extract common functionality for other providers < https://launchpad.net/bugs/800988 > === _mup__ is now known as _mup_ === koolhead11| is now known as koolhead11 [07:19] hi all [07:20] can i assign a bug to myself after reporting it [07:20] i want to work on few formulas and i want to assign it to myself [07:21] ooh yes i can. :) [07:21] koolhead11: have you finished any formulas yet? [07:21] SpamapS, am working on gallery2 [07:21] koolhead11: what happened to phpmyadmin [07:21] ? [07:22] SpamapS, i will try it once i have some confidence in writing simple formulas [07:23] SpamapS, please suggest me if am doing wrong :( [07:24] I just wanted to give you some feedback. :) [07:25] koolhead11: often times its harder to finish one thing than start 10 ;) [07:26] SpamapS, hmm. [08:47] hazmat: So, I think I'm officially in over my head now. lp:~clint-fewbar/ensemble/lxc-container has things the way I think I want them.. [08:48] hazmat: I'm running the provisioning agent locally. This is because we want to run the lxc-* commands locally, not on the container. [08:49] hazmat: it seems to work. Where I'm stuck is in debugging what goes wrong on the provisioning agent (because stdout/stderr are directed to /dev/null) and how to serialize/unserialize the machine info from zookeeper [08:49] hazmat: I would have thought a lot of that would be generic to all providers, so its really confusing that I might have to do it all myself in the lxc provider. [08:49] anyway I'm sure everybody else is asleep [08:49] jimbaker: ^^ [08:49] bcsaller: ^^ [08:49] would love all of your thoughts [08:50] tho I feel we'll probably be more productive in Dublin. :) [08:53] Also for those interested.. the template I used is in debian/lxc-natty-uec .. you'll need to copy that to /usr/lib/lxc/templates === daker_ is now known as daker [15:49] SpamapS: is the latest version at that lp: url you gave me ysterday? [15:50] * hallyn goes to clone [15:50] sbux wifi == majorly laggy [16:01] SpamapS, branch looks great, we should probably refactor it to two branches (the ec2 provider commons extraction) regarding the provisioning agent, the logs are sent to /var/log/ensemble.. the issue might just be that directory is missing on the host machine [16:55] <_mup_> ensemble/expose-ec2-provider r270 committed by jim.baker@canonical.com [16:55] <_mup_> Merged trunk [17:01] hazmat: nope, the logs are sent to ~/.ensemble/var/log/ensemble .. and its created [17:01] hazmat: the stdin/stdout is /dev/null, so any commands run get that as their stdin/stdout as well [17:05] hazmat: agreed that the refactoring into common is a separate change [17:05] hazmat: I'm not really doing this as carefully as one might hope.. I am making it work first. ;) [17:08] kim0: hey! you can change the package on principia bugs?! [17:09] hmm? [17:09] https://bugs.launchpad.net/principia/+source/mysql [17:09] that didn't used to be possible [17:10] SpamapS: yeah I noticed the new functionality .. so you already did it for this bug right ? (it says mysql (Principia Ensemble)) [17:10] Francis did [17:10] maybe he was testing. :) [17:11] hehe :) cool then [17:11] kim0: and I just realized, I read who did the change wrong, as you.. since you were the last commenter before me or francis [17:12] yeah np [17:12] SpamapS: still in mortal kombat with LXC ? :) [17:20] kim0: we agreed to meditate in our corners for a few hours.. but I think LXC is sharpening its sword again.. [17:20] haha :) [17:21] * kim0 hands a lightsaber [17:27] excellent, this will allow me to chop off its hands [17:33] has anyone else seen this error with formulas being retrieved? - http://paste.ubuntu.com/631275/ [17:34] i've tried removing my data bucket etc [17:34] (and at first thought it was a problem in a branch i'm working on) [17:37] jimbaker, odd, no i haven't seen that one [17:37] ok back in the saddle.. [17:38] added logging to the lxc commands.. [17:40] hazmat, i will dig it into further [17:44] so.. I don't have to use twisted, do I? like, I can just let my shutdown or launch commands block until they finish, right? [17:44] I get that its way cooler if I don't.. ;) [17:50] SpamapS, yes you can block in the cli [17:52] we don't really do full shutdown via the provisioning agent yet.. the cli does it, for the ec2 provider via an amazon api is already async, so its not something that cleanly translates to other providers that might be sync. blocking in the cli seems like a best path forward for lxc atm [17:56] Another funny thing is because I'm not ripping down the zookeeper host every time I have to manually clear it out between bootstrap/shutdown [18:03] probably worth it to just move that back to the bootstrap node, as for some reason it seems to insist on spawning a machine 0 anyway :p [18:24] I need to think of a good strategy for getting the hostnames and/or IP's available to the local CLI [18:26] SpamapS, hmm... we could reset zk on initialization for now [18:40] more progress.. I'm now failing because I don't have shared storage for formulas.. which is.. encouraging. :-D [18:41] * SpamapS considers a bind mount === daker is now known as daker_ [19:02] SpamapS, a bind mount would work, the only thing doing write to the storage is the admin cli [19:04] the dummy provider file storage should work for it [19:10] hazmat: the difficulty is figuring out the path to use :P [19:39] hallyn: need your opinion.. would you use a bind mount inside the containers for shared storage, or setup an HTTP server which they can use to fetch data? [19:40] bind mount [19:40] just like with -b option in my template [19:40] hallyn: the bind mount seems simpler.. but I can see it leaving a lot of cruft behind [19:40] which bind mounts in your $home [19:40] i'd definately do bind mount [19:41] hallyn: cool, I'm just going to manage the mount myself.. but maybe in the future the template can do it [19:41] ok. it's been an unproductive day so far (emails won't stop coming in). may not get your template into mine today. But I assume you're happy with yours so far [19:42] i'd like to wait until we figure out the best way to merge stgraber's patches into my template [19:42] hallyn: I made a couple of tweaks so it does less work each time.. but its 99% the same [19:42] SpamapS: still in the same branch? [19:42] hallyn: Eventually we need to talk about this lxc-clone.. :) [19:42] i'll re-fetch whenever i merge it in [19:43] SpamapS: it's in ppa:serge-hallyn/lxc-natty, if you want to work with it [19:43] hallyn: no but I'll push it up sometime [19:43] ok [19:43] Getting very close to actually deploying something on LXC. :-D [19:43] i should get lvm support into my -create so to ease setting up the canonical container, but lxc-clone -s is there, and lightning-fast [19:43] what does lxc-clone do? [19:43] I don't have it on natty [19:44] yes, ppa:serge-hallyn/lxc-natty :) [19:44] if you have your rootfs for continer on lvm, [19:44] you can 'lxc-clone -s -o oldcontainer -n newcontainer', and it does an lvm snapshot for the new contaienr's rootfs [19:44] so you can use it like schroots pretty much [19:44] hallyn: ahhh... thats nice. I'd be happy with just a cp -a at this point [19:44] (especially now that you can do --bindhome) [19:44] if you're not using lvm, it does cp -a :) [19:45] I am going to spec out some additional things to build snapshotting into ensemble providers.. so we don't have to maintain an AMI manually, but we don't have to wait for 50 packages to install over and over. [19:45] * hallyn goes to lok at his todo list to see if he can do lvm support in lxc-create today [19:46] nah, i can't [20:23] omg omg omg .. I just deployed WP in LXC .. [20:23] * SpamapS looks at his glorious tower of chewing gum and duct tape [20:27] hazmat: so it seems like what I'm missing is the mapping between machine-id and instance-id ... is that just because I'm not setting the right properties of the machine object I return during start_machine ? [20:27] SpamapS, you can use any provider specific notion of machine identity [20:27] SpamapS, as long as its unique [20:28] SpamapS, dir names (if precluding reuse of a name) [20:28] machine = LXCMachine(len(self._machines), dns_name=container_name) [20:29] SpamapS, hmmm so for identity i'd use the dirname, for dns name, whatever is resolvable between the host and container [20:29] yeah the host name resolution seems to be a problem [20:30] dnsmasq may not be reliable enough for this. :-/ [20:30] Or I've confused it [20:30] SpamapS, is the host configured to use dnsmasq? or is it just setup for the bridge? [20:30] SpamapS, I'd be happy to beta test the lxc stuff if you have it in a more-or-less usable state :D [20:31] pindonga: lets let the chewing gum dry a bit before we move it [20:31] hazmat: I've tried both ways. [20:31] hazmat: with the host setup to use dnsmasq.. that seems to create more problems than it solves [20:32] SpamapS, can you ssh into one of the containers currently? [20:32] hazmat: I can ssh into all of them [20:32] SpamapS, you can pass an ip address for the dns name [20:32] SpamapS, let me know when it's dry enough , so I can start using it locally for testing ensemble out :) [20:32] I was thinking about just shoving the IP into a file and pulling that out.. [20:32] but hostname needs to work [20:33] hazmat: the issue is, dnsmasq isn't responding to requests for the hosts that it has been told about [20:33] hrm.. and the installation of ensemble is giving python-support fits [20:39] SpamapS, i'm wondering if it might be a dnsmasq misconfigure, looking at http://www.phenona.com/blog/using-lxc-linux-containers-in-amazon-ec2/ the host resolution portion uses a separate domain for the lxc machines [20:42] <_mup_> ensemble/expose-status r261 committed by jim.baker@canonical.com [20:42] <_mup_> Initial commit [20:44] <_mup_> ensemble/expose-status r262 committed by jim.baker@canonical.com [20:44] <_mup_> Removed obsolete comment + PEP8 [23:12] anyone know what version of cobbler, orchestra uses? [23:12] nevermind [23:19] hazmat: did you find it? [23:19] hazmat: https://launchpad.net/ubuntu/+source/cobbler [23:20] hazmat: 2.1.0+git20110602-0ubuntu8 [23:20] hazmat: we're taking git snapshots, at the moment [23:20] done presenting at Structure 2011 :) *whew* [23:20] hazmat: i think zul or roaksoax might know when cobbler releases again [23:20] bcsaller: nice! how did it go? [23:21] bcsaller, awesome.. how was it [23:21] well, it wasn't that big a crowd, but I think our technologies were well received and importantly, understood [23:21] kirkland, i did, i was reviewing their xmlrpc api [23:22] hazmat: ah, cool [23:23] kirkland, in terms of ensemble integration with orchestra, i was thinking just utilizing the cobbler xmlrpc api would be easiest. any suggestions? [23:24] bcsaller, seems to be a nice speaker list.. and the live streams, seem to suggest more strategic/vision than tool oriented per se [23:24] kirkland: it was interesting trying to come up with language that helped people understand that orchestra and ensemble were not competing. I summed it up in a way that I hope you're happy with, "Orchestra is about provisioning infrastructure and ensemble is about provisioning services" [23:25] hazmat: true dat [23:25] but some big players in need of solutions in both these spaces [23:25] <_mup_> ensemble/redirect-zk-log-dev-null r261 committed by jim.baker@canonical.com [23:25] <_mup_> Initial commit [23:25] bcsaller: yeah, that's good! [23:25] bcsaller: i've been saying ... "Ensemble is about how you deploy your applications/services INTO the cloud" [23:25] bcsaller: "Orchestra is how you deploy THE CLOUD" [23:26] yeah, I avoided that word here because it still means different things to different people. Infrastructure seems to have more agreement [23:27] another slightly more involved way of doing ensemble integration into orchestra would be offering an mcollective plugin that would have a limited command set to just activate machines as ensemble nodes [23:27] bcsaller: fair enough [23:28] hazmat: Juan and I were talking about something similar to that today [23:31] bcsaller, nice.. it seems pretty inline with the orchestra message. [23:31] bcsaller: how was juan's part of the talk? [23:32] kirkland: I think he did really well, he also was willing to adapt a little and focus more on the infrastructure side of the story than the puppet side of things though all options were discussed [23:32] bcsaller, where you guys thinking of richer semantics resolved via orchestra than machine activation? [23:34] hazmat: it goes back to the notion of types of formula that run inside other containers. (so not everything has to provide a relation to something like logging, or in this case maybe puppet infrastructure). I suggested that as a model for thinking about it we talk about things like "ensemble deploy wordpress as myblog with rsyslog, puppet-client" [23:34] where those are a new class of in-container formula [23:34] but thats the sort of thing we need to plan out and roadmap more [23:35] bcsaller, indeed, that's perfect [23:35] :) [23:35] its not really inheritance or containment, its association in the same unit same [23:35] s/same/space [23:36] also "ensemble deploy rsyslog in service|unitname" works after the initial deployment [23:36] SpamapS, ^ re policy formulas [23:36] oo hey what? [23:37] then we just need a way those formula's hooks are consistently triggered but we have a good model for it [23:37] with [23:37] I like that *a lot* [23:37] bcsaller, yeah.. colocated units should work fine, its shrinking the granularity of a machine to the unit, its works really well. [23:38] we should paste this into an email, huh? :) [23:38] so then I can say 'ensemble add-relation *:rsyslog rsyslog-central-host [23:39] * SpamapS WINNING [23:40] bcsaller, s/we/you .. definitely captured at least, that's ensemble's way forward on the last remaining conceptual piece for single dc prod deployments. [23:40] SpamapS: yeah, the formula are not concerned with the logging aspect (except that some metadata might be needed) but the contained formula would still relate like that [23:40] s/dc/data center [23:41] I parsed it [23:42] bcsaller, an implementation could also clear the way for server side plugins