/srv/irclogs.ubuntu.com/2011/06/23/#ubuntu-ensemble.txt

_mup_ensemble/debug-log-relation-settings-changes r273 committed by jim.baker@canonical.com00:03
_mup_Change items of strings should be cutoff in logging if they exceed some arbitrary, but reasonable limit (for now, that is 100)00:03
SpamapSheh.. JFDI has turn into "JFDTFAOP" .. Just F'ing Do This, Forget All Other Priorities00:17
hazmatSpamapS, can you paste the cloud-init data00:26
SpamapShazmat: at this point I've got it connecting to ZK00:26
hazmatSpamapS, are you able to ssh into the bootstrap contaienr?00:26
hazmatSpamapS, awesome00:27
SpamapSbut there's nothing in ZK00:27
SpamapSensemble.state.errors.MachineStateNotFound: Machine 0 was not found00:27
hazmatSpamapS, right.. there's an initialize zk command that runs before the agent is started00:27
SpamapSOH! that got dropped somehow00:28
hazmatSpamapS, its at the bottom of ec2/launch.py.. if you paste your cloud-init config it should be easy to spot00:28
SpamapSFor me now its in common.get_bootstrap_machine_variables ..00:29
hazmatSpamapS, try running this if your can ssh into the bootstrap container.. /usr/local/bin/ensemble-admin initialize --instance-id=$(curl http://169.254.169.254/1.0/meta-data/instance-id) --admin-identity=abc00:29
hazmatwhen you say nothing is in zk? are you using the shell to have a look ie. /usr/share/zookeeper/bin/zkCli.sh 00:30
SpamapShazmat: ok, the logic I had put in was overwriting those commands00:48
SpamapShazmat: the curl won't work of course.. :)00:48
hazmatSpamapS, cool, if you got zk up and running and clients connecting to it, that should be hopefully most of it. 00:49
SpamapShazmat: for the moment, I'm using the ZK running on my box letting them connect to it00:50
hazmatSpamapS, right on.. that's still pretty good progress00:51
hazmatexciting even00:51
SpamapSAnother twistd server is running, PID 110000:52
SpamapShrm00:52
SpamapSoh.. now my logic fix reverted my earlier hack00:53
* dannf waves00:55
SpamapShazmat: way I see it, this should be very easy to morph into libvirt for somebody who understands libvirt better00:56
hazmatSpamapS, indeed, with the first non-ec2 provider impl, the next should go much easier.00:57
SpamapShazmat: this still has more hacks than Wenesday night open mic at the improv..00:57
hazmatSpamapS, no worries, they can be cleaned up00:58
hazmatIts still likely the top feature requested imo, and having something in-dev/demoable on that is awesome00:59
SpamapShazmat: how does ensemble-admin know what ZK to talk to? In the example it doesn't have ENSEMBLE_ZOOKEEPER set00:59
hazmatSpamapS, it knows its running on the same host as the zk, so it uses localhost01:00
hazmatand the default port...01:00
SpamapS...01:01
SpamapShence my problem ;)01:01
* SpamapS abandons the "keep it simple" approach and installs zookeeperd01:01
hazmatSpamapS, it should take a cli arg..01:01
* hazmat checks01:01
SpamapSno its adding logic not making it simpler01:01
SpamapSthis is better. ;)01:01
hazmatSpamapS, you can set an environment variable for it01:02
hazmatZOOKEEPER_ADDRESS01:02
hazmatit will use that in preference to the default localhost:2181 address01:02
SpamapSthats ok, I'm trying very hard to change as little as possible..01:03
SpamapSas my head wraps around the existing hacks, I let them back in and revert changes to try and fix them the wrong way. ;)01:03
hazmatSpamapS, some change and refactoring is ok in my book.01:03
hazmatwe can make the default ec2 command work that way as well01:04
SpamapSYeah there's plenty of that thus far01:04
SpamapSUGGGH zookeeperd installs another 25 packages01:06
hazmatSpamapS, indeed. that's pretty much why we have an AMI at all01:07
SpamapS[zk: localhost:2181(CONNECTED) 2] ls /01:08
SpamapS[services, formulas, relations, zookeeper, initialized, topology, machines, units]01:08
hazmatSpamapS, WOOT!01:08
SpamapSstill nothing tho01:08
hazmatSpamapS, any agents connected? (ps aux | grep ensemble)01:08
SpamapShazmat: there's a machine-0000000000001:09
SpamapShazmat: the admin command ran after the agents started tho01:09
hazmatSpamapS, yeah.. that's representing the bootstrap machine (the one with zk), its go a machine agent, but we don't ever reuse it for service unit assignment01:09
hazmatSpamapS, more important is the provisioning agent.. it actually waits till the user tries to deploy a service, and the provider credentials are provided lazily to it (upon deploy).. its that way mainly so we can avoid passing credentials via user-data01:10
hazmatSpamapS, i'd check to see if its running, tail its log, and then try to deploy a service and see what happens01:11
* hazmat is looking forward to hear about structure 201101:11
hazmatlooks like their streaming live01:12
SpamapShazmat: so far 'ensemble status' shows empty01:14
SpamapShazmat: I think eventually we can take the AMI thing to a level where each service gets snapshotted whenever you upgrade or deploy.. so add unit is just popping up a snapshot01:17
SpamapShazmat: was thinking the other day that if formulas could call 'ensemble-snapshot-service-unit' or something that would be cool01:18
SpamapSclint@clint-MacBookPro:~/src/ensemble/lxc-container$ ZOOKEEPER_ADDRESS=192.168.122.68:2181 PYTHONPATH=`pwd` bin/ensemble status -e lxc01:20
SpamapS2011-06-22 17:20:20,867 ERROR Machine provider information missing: machine 001:20
SpamapSmachines: 0: {instance-id: ensemble-0}01:20
SpamapSservices: {}01:20
SpamapS2011-06-22 17:20:20,869 INFO 'status' command finished successfully01:20
SpamapSclint@clint-MacBookPro:~/src/ensemble/lxc-container$ 01:20
SpamapSOOOHHHH SNAP01:20
hazmatSpamapS, cool!, that  missing info thing, is just implementing the get_machine api of the provider01:26
SpamapSI'm having a weird issue..01:26
SpamapSits happening even w/ trunk..01:26
SpamapShttp://paste.ubuntu.com/631028/01:27
hazmathmmm.. that looks like a speling typo on repository01:28
hazmatSpamapS, ^01:28
SpamapSlol01:29
SpamapSensemble.environment.errors.EnvironmentsConfigError: Environments configuration error: environments.lxc.type: expected 'ec2', got 'lxc'01:31
SpamapSHeh..01:31
SpamapSI need to put *my* version of ensemble on there.. duh01:31
jimbakerit would be nice to have that doublechecking occur!01:33
hazmatSpamapS, doh.. yeah.. ensemble-branch in environments01:33
SpamapShazmat: I didn't implement that yet. :-P01:33
hazmatSpamapS, ah.. write the switch out is part of cloud-init01:34
jimbakerso there are two problems ;)01:34
SpamapShazmat: have to branch it before I can switch it. ;)01:34
jimbakeranyway, it will be nice to get bug 719517 in place01:34
_mup_Bug #719517: Ensemble should store the code version in each zookeeper <Ensemble:New> < https://launchpad.net/bugs/719517 >01:34
SpamapScrap.. I need to use a mirror.. they just released a firefox update and archive.ubuntu.com is slammed01:43
_mup_Bug #800988 was filed: refactor ec2 provider to extract common functionality for other providers <Ensemble:New> < https://launchpad.net/bugs/800988 >04:47
=== _mup__ is now known as _mup_
=== koolhead11| is now known as koolhead11
koolhead11hi all07:19
koolhead11can i assign a bug to myself after reporting it07:20
koolhead11i want to work on few formulas and i want to assign it to myself07:20
koolhead11ooh yes i can. :)07:21
SpamapSkoolhead11: have you finished any formulas yet?07:21
koolhead11SpamapS, am working on gallery207:21
SpamapSkoolhead11: what happened to phpmyadmin07:21
SpamapS?07:21
koolhead11SpamapS, i will try it once i have some confidence in writing simple formulas07:22
koolhead11SpamapS, please suggest me if am doing wrong :(07:23
SpamapSI just wanted to give you some feedback. :)07:24
SpamapSkoolhead11: often times its harder to finish one thing than start 10 ;)07:25
koolhead11SpamapS, hmm. 07:26
SpamapShazmat: So, I think I'm officially in over my head now. lp:~clint-fewbar/ensemble/lxc-container has things the way I think I want them..08:47
SpamapShazmat: I'm running the provisioning agent locally. This is because we want to run the lxc-* commands locally, not on the container.08:48
SpamapShazmat: it seems to work. Where I'm stuck is in debugging what goes wrong on the provisioning agent (because stdout/stderr are directed to /dev/null) and how to serialize/unserialize the machine info from zookeeper08:49
SpamapShazmat: I would have thought a lot of that would be generic to all providers, so its really confusing that I might have to do it all myself in the lxc provider.08:49
SpamapSanyway I'm sure everybody else is asleep08:49
SpamapSjimbaker: ^^08:49
SpamapSbcsaller: ^^08:49
SpamapSwould love all of your thoughts08:49
SpamapStho I feel we'll probably be more productive in Dublin. :)08:50
SpamapSAlso for those interested.. the template I used is in debian/lxc-natty-uec .. you'll need to copy that to /usr/lib/lxc/templates08:53
=== daker_ is now known as daker
hallynSpamapS: is the latest version at that lp: url you gave me ysterday?15:49
* hallyn goes to clone15:50
hallynsbux wifi == majorly laggy15:50
hazmatSpamapS, branch looks great, we should probably refactor it to two branches (the ec2 provider commons extraction) regarding the provisioning agent, the logs are sent to /var/log/ensemble.. the issue might just be that directory is missing on the host machine16:01
_mup_ensemble/expose-ec2-provider r270 committed by jim.baker@canonical.com16:55
_mup_Merged trunk16:55
SpamapShazmat: nope, the logs are sent to ~/.ensemble/var/log/ensemble .. and its created17:01
SpamapShazmat: the stdin/stdout is /dev/null, so any commands run get that as their stdin/stdout as well17:01
SpamapShazmat: agreed that the refactoring into common is a separate change17:05
SpamapShazmat: I'm not really doing this as carefully as one might hope.. I am making it work first. ;)17:05
SpamapSkim0: hey! you can change the package on principia bugs?!17:08
kim0hmm?17:09
SpamapShttps://bugs.launchpad.net/principia/+source/mysql17:09
SpamapSthat didn't used to be possible17:09
kim0SpamapS: yeah I noticed the new functionality .. so you already did it for this bug right ? (it says mysql (Principia Ensemble))17:10
SpamapSFrancis did17:10
SpamapSmaybe he was testing. :)17:10
kim0hehe :) cool then17:11
SpamapSkim0: and I just realized, I read who did the change wrong, as you.. since you were the last commenter before me or francis17:11
kim0yeah np17:12
kim0SpamapS: still in mortal kombat with LXC ? :)17:12
SpamapSkim0: we agreed to meditate in our corners for a few hours.. but I think LXC is sharpening its sword again..17:20
kim0haha :)17:20
* kim0 hands a lightsaber17:21
SpamapSexcellent, this will allow me to chop off its hands17:27
jimbakerhas anyone else seen this error with formulas being retrieved? - http://paste.ubuntu.com/631275/17:33
jimbakeri've tried removing my data bucket etc17:34
jimbaker(and at first thought it was a problem in a branch i'm working on)17:34
hazmatjimbaker, odd, no i haven't seen that one17:37
SpamapSok back in the saddle..17:37
SpamapSadded logging to the lxc commands..17:38
jimbakerhazmat, i will dig it into further17:40
SpamapSso.. I don't have to use twisted, do I? like, I can just let my shutdown or launch commands block until they finish, right?17:44
SpamapSI get that its way cooler if I don't.. ;)17:44
hazmatSpamapS, yes you can block in the cli17:50
hazmatwe don't really do full shutdown via the provisioning agent yet.. the cli does it, for the ec2 provider via an amazon api is already async, so its not something that cleanly translates to other providers that might be sync. blocking in the cli seems like a best path forward for lxc atm17:52
SpamapSAnother funny thing is because I'm not ripping down the zookeeper host every time I have to manually clear it out between bootstrap/shutdown17:56
SpamapSprobably worth it to just move that back to the bootstrap node, as for some reason it seems to insist on spawning a machine 0 anyway :p18:03
SpamapSI need to think of a good strategy for getting the hostnames and/or IP's available to the local CLI18:24
hazmatSpamapS, hmm...  we could reset zk on initialization for now18:26
SpamapSmore progress.. I'm now failing because I don't have shared storage for formulas.. which is.. encouraging. :-D18:40
* SpamapS considers a bind mount18:41
=== daker is now known as daker_
hazmatSpamapS, a bind mount would work, the only thing doing write to the storage is the admin cli19:02
hazmatthe dummy provider file storage should work  for it19:04
SpamapShazmat: the difficulty is figuring out the path to use :P19:10
SpamapShallyn: need your opinion.. would you use a bind mount inside the containers for shared storage, or setup an HTTP server which they can use to fetch data?19:39
hallynbind mount19:40
hallynjust like with -b option in my template19:40
SpamapShallyn: the bind mount seems simpler.. but I can see it leaving a lot of cruft behind19:40
hallynwhich bind mounts in your $home19:40
hallyni'd definately do bind mount19:40
SpamapShallyn: cool, I'm just going to manage the mount myself.. but maybe in the future the template can do it19:41
hallynok.  it's been an unproductive day so far (emails won't stop coming in).  may not get your template into mine today.  But I assume you're happy with yours so far19:41
hallyni'd like to wait until we figure out the best way to merge stgraber's patches into my template19:42
SpamapShallyn: I made a couple of tweaks so it does less work each time.. but its 99% the same19:42
hallynSpamapS: still in the same branch?19:42
SpamapShallyn: Eventually we need to talk about this lxc-clone.. :)19:42
hallyni'll re-fetch whenever i merge it in19:42
hallynSpamapS: it's in ppa:serge-hallyn/lxc-natty, if you want to work with it19:43
SpamapShallyn: no but I'll push it up sometime19:43
hallynok19:43
SpamapSGetting very close to actually deploying something on LXC. :-D19:43
hallyni should get lvm support into my -create so to ease setting up the canonical container, but lxc-clone -s is there, and lightning-fast19:43
SpamapSwhat does lxc-clone do?19:43
SpamapSI don't have it on natty19:43
hallynyes, ppa:serge-hallyn/lxc-natty :)19:44
hallynif you have your rootfs for continer on lvm,19:44
hallynyou can 'lxc-clone -s -o oldcontainer -n newcontainer', and it does an lvm snapshot for the new contaienr's rootfs19:44
hallynso you can use it like schroots pretty much19:44
SpamapShallyn: ahhh... thats nice. I'd be happy with just a cp -a at this point19:44
hallyn(especially now that you can do --bindhome)19:44
hallynif you're not using lvm, it does cp -a :)19:44
SpamapSI am going to spec out some additional things to build snapshotting into ensemble providers.. so we don't have to maintain an AMI manually, but we don't have to wait for 50 packages to install over and over.19:45
* hallyn goes to lok at his todo list to see if he can do lvm support in lxc-create today19:45
hallynnah, i can't19:46
SpamapSomg omg omg .. I just deployed WP in LXC .. 20:23
* SpamapS looks at his glorious tower of chewing gum and duct tape20:23
SpamapShazmat: so it seems like what I'm missing is the mapping between machine-id and instance-id ... is that just because I'm not setting the right properties of the machine object I return during start_machine ?20:27
hazmatSpamapS, you can use any provider specific notion of machine identity20:27
hazmatSpamapS, as long as its unique20:27
hazmatSpamapS, dir names (if precluding reuse of a name)20:28
SpamapS        machine = LXCMachine(len(self._machines), dns_name=container_name)20:28
hazmatSpamapS, hmmm so for identity i'd use the dirname, for dns name, whatever is resolvable between the host and container20:29
SpamapSyeah the host name resolution seems to be a problem20:29
SpamapSdnsmasq may not be reliable enough for this. :-/20:30
SpamapSOr I've confused it20:30
hazmatSpamapS, is the host configured to use dnsmasq? or is it just setup for the bridge?20:30
pindongaSpamapS, I'd be happy to beta test the lxc stuff if you have it in a more-or-less usable state :D20:30
SpamapSpindonga: lets let the chewing gum dry a bit before we move it20:31
SpamapShazmat: I've tried both ways.20:31
SpamapShazmat: with the host setup to use dnsmasq.. that seems to create more problems than it solves20:31
hazmatSpamapS, can you ssh into one of the containers currently?20:32
SpamapShazmat: I can ssh into all of them20:32
hazmatSpamapS, you can pass an ip address for the dns name20:32
pindongaSpamapS, let me know when it's dry enough , so I can start using it locally for testing ensemble out :)20:32
SpamapSI was thinking about just shoving the IP into a file and pulling that out..20:32
SpamapSbut hostname needs to work20:32
SpamapShazmat: the issue is, dnsmasq isn't responding to requests for the hosts that it has been told about20:33
SpamapShrm.. and the installation of ensemble is giving python-support fits20:33
hazmatSpamapS, i'm wondering if it might be a dnsmasq misconfigure, looking at http://www.phenona.com/blog/using-lxc-linux-containers-in-amazon-ec2/ the host resolution portion uses a separate domain for the lxc machines20:39
_mup_ensemble/expose-status r261 committed by jim.baker@canonical.com20:42
_mup_Initial commit20:42
_mup_ensemble/expose-status r262 committed by jim.baker@canonical.com20:44
_mup_Removed obsolete comment + PEP820:44
hazmatanyone know what version of cobbler, orchestra uses?23:12
hazmatnevermind23:12
kirklandhazmat: did you find it?23:19
kirklandhazmat: https://launchpad.net/ubuntu/+source/cobbler23:19
kirklandhazmat: 2.1.0+git20110602-0ubuntu8 23:20
kirklandhazmat: we're taking git snapshots, at the moment23:20
bcsallerdone presenting at Structure 2011 :) *whew*23:20
kirklandhazmat: i think zul or roaksoax might know when cobbler releases again23:20
kirklandbcsaller: nice!  how did it go?23:20
hazmatbcsaller, awesome.. how was it23:21
bcsallerwell, it wasn't that big a crowd, but I think our technologies were well received and importantly, understood23:21
hazmatkirkland, i did, i was reviewing their xmlrpc api23:21
kirklandhazmat: ah, cool23:22
hazmatkirkland, in terms of ensemble integration with orchestra, i was thinking just utilizing the cobbler xmlrpc api would be easiest. any suggestions?23:23
hazmatbcsaller, seems to be a nice speaker list.. and the live streams, seem to suggest more strategic/vision than tool oriented per se23:24
bcsallerkirkland: it was interesting trying to come up with language that helped people understand that orchestra and ensemble were not competing. I summed it up in a way that I hope you're happy with, "Orchestra is about provisioning infrastructure and ensemble is about provisioning services"23:24
bcsallerhazmat: true dat23:25
bcsallerbut some big players in need of solutions in both these spaces23:25
_mup_ensemble/redirect-zk-log-dev-null r261 committed by jim.baker@canonical.com23:25
_mup_Initial commit23:25
kirklandbcsaller: yeah, that's good!23:25
kirklandbcsaller: i've been saying ... "Ensemble is about how you deploy your applications/services INTO the cloud"23:25
kirklandbcsaller: "Orchestra is how you deploy THE CLOUD"23:25
bcsalleryeah, I avoided that word here because it still means different things to different people. Infrastructure seems to have more agreement 23:26
hazmatanother slightly more involved way of doing ensemble integration into orchestra would be offering an mcollective plugin that would have a limited command set to just activate machines as ensemble nodes23:27
kirklandbcsaller: fair enough23:27
bcsallerhazmat: Juan and I were talking about something similar to that today23:28
hazmatbcsaller, nice.. it seems pretty inline with the orchestra message.23:31
kirklandbcsaller: how was juan's part of the talk?23:31
bcsallerkirkland: I think he did really well, he also was willing to adapt a little and focus more on the infrastructure side of the story than the puppet side of things though all options were discussed23:32
hazmatbcsaller, where you guys thinking of richer semantics resolved via orchestra  than machine activation?23:32
bcsallerhazmat: it goes back to the notion of types of formula that run inside other containers. (so not everything has to provide a relation to something like logging, or in this case maybe puppet infrastructure). I suggested that as a model for thinking about it we talk about things like "ensemble deploy wordpress as myblog with rsyslog, puppet-client"23:34
bcsallerwhere those are a new class of in-container formula23:34
bcsallerbut thats the sort of thing we need to plan out and roadmap more 23:34
hazmatbcsaller, indeed, that's perfect23:35
bcsaller:)23:35
hazmatits not really inheritance or containment, its association in the same unit same23:35
hazmats/same/space23:35
bcsalleralso "ensemble deploy rsyslog in service|unitname" works after the initial deployment23:36
hazmatSpamapS, ^ re policy formulas23:36
SpamapSoo hey what?23:36
bcsallerthen we just need a way those formula's hooks are consistently triggered but we have a good model for it 23:37
SpamapSwith23:37
SpamapSI like that *a lot*23:37
hazmatbcsaller, yeah.. colocated units should work fine, its shrinking the granularity of a machine to the unit, its works really well. 23:37
bcsallerwe should paste this into an email, huh? :)23:38
SpamapSso then I can say 'ensemble add-relation *:rsyslog rsyslog-central-host23:38
* SpamapS WINNING23:39
hazmatbcsaller, s/we/you .. definitely captured at least, that's ensemble's way forward on the last remaining conceptual piece for single dc prod deployments.23:40
bcsallerSpamapS: yeah, the formula are not concerned with the logging aspect (except that some metadata might be needed) but the contained formula would still relate like that23:40
hazmats/dc/data center23:40
bcsallerI parsed it23:41
hazmatbcsaller, an implementation could also clear the way for server side plugins23:42

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!