/srv/irclogs.ubuntu.com/2011/04/13/#ubuntu-cloud.txt

SpamapSargh!  WARNING: Invalid keys at [https://launchpad.net/~clint-fewbar/+sshkeys]00:16
flaccidlol00:18
flaccidall the above seems about the usual standard for ubuntu00:18
flaccidoh you are a canonical/ubuntu guy?00:19
=== dendro-afk is now known as dendrobates
jairhello everyone00:49
jairit is a requirement to download the server version 10.10 of ubuntu to use landscape and set up the private cloud? or it is the same kernel I have the alternate version 64bit 10.10 should that work?00:51
jairbasically what I am asking do I need to download the version 10.10 server of ubuntu or in reality the version 10.10 desktop 64bit will have the same packages and capabilities?00:56
flaccid3mins, uber patient :)01:15
=== dendrobates is now known as dendro-afk
=== dendro-afk is now known as dendrobates
=== dendrobates is now known as dendro-afk
=== dendro-afk is now known as dendrobates
=== dendrobates is now known as dendro-afk
superxglhi all, i have running an instance : euca-run-instances $AMI_IMAGE  -k $KEY_NAME --addressing private -n 1 -g $CLUSTER_MASTER  -f "$bin"/$USER_DATA_FILE -t $INSTANCE_TYPE $KERNEL_ARG --ramdisk eri-A535178D09:27
superxgland i have installed cloud-init package09:27
superxglbut i don't why "-f "$bin"/$USER_DATA_FILE" do not execute09:27
superxglis there something wrong ?09:28
superxglthe file "$bin"/$USER_DATA_FILE" did *NOT* be executed09:29
flaccidyou might wanna show the stdeer and exit code09:36
TeTeTyeah, is $USER_DATA_FILE a regular shellscript? Can you get it from the instance via wget?09:37
superxglthis is the shell script :   http://pastebin.com/mijcWEmc09:42
flaccidcaptcha, lame09:45
flaccidok so where is the output?09:46
superxglthis shell script starts the hadoop daemon, no output09:47
flaccidyou might wanna run it under -ex so you can see what is occurring with the commands.09:49
superxgli have copy this file to my instance , and it can run correclty09:51
superxglsorry, what do u mean run it under -ex  ?09:52
flaccidthey are bash parameters09:52
flaccidgoing to dinner; bye09:52
superxglflaccid : ok ,tkx.bye09:53
superxglhmm...because it can running in the instance correclty, so i think the shell script is ok10:02
superxglso i think maybe the cloud-init has problems10:02
superxglhttp://pastebin.com/VTneuzuE10:03
superxglTeTeT: how can i get it from the instance via wget ?10:13
superxglwget from the instance meta-data ?10:14
TeTeTsuperxgl: hmm, don't know from the top of my head, something like wget http://169.254.169.254/current/userdata10:18
TeTeTsuperxgl: http://docs.amazonwebservices.com/AmazonEC2/dg/2007-01-03/AESDG-chapter-instancedata.html10:19
superxglTeTeT: thnx. i will have a try12:26
superxglTeTeT:  http://pastebin.com/5vRdxGy112:27
michael__Is there a good opensource project for providing a layer ontop of ec2 or uec for scalable websites or webapps?12:28
michael__something like scalr.net12:28
sorenscalr is open source.12:28
TeTeTsuperxgl: sorry, out of ideas then12:29
superxglTeTeT: tkx. i will try finding it out12:30
michael__soren: yeah but it isn't perfect ;) I just wanted to ask for alternatives :D12:31
sorenmichael__: If you want better answers, ask better questions.12:33
sorenmichael__: Like, say, explain how scalr isn't meeting your needs.12:33
michael__scalr isn't stable yet, the current version has a lot of bugs12:34
michael__no automatic caching (no needed, but would be a great feature)12:34
michael__a Paas layer would be really cool. something like google app engine for uec12:35
nwlCloud Foundry perhaps? :)12:36
superxglwhere is the cloud-init's message output ?12:41
TeTeTsuperxgl: on the console, via euca-get-console-output <instance id>12:43
superxgl[root@CLC bin]# euca-get-console-output i-464C07BC12:45
superxgli-464C07BC12:45
superxgl2011-04-13T11:59:22.263Z12:45
superxglNOT SUPPORTED12:45
superxglhmm...eucalyptus not supported12:46
superxgli found /var/log/cloud-init.log  ,but the file is blank12:46
kim0maybe /var/log/syslog12:53
superxglkim0: i find all the file under /var/log, could not find.13:00
kim0superxgl: cd /var/log ; grep -r cloud-init * ?13:02
superxglhereis the cloud-init file :   http://pastebin.com/LYyrxxJP13:04
superxgl-bash-3.2# cd /var/log ; grep -r cloud-init *13:05
superxglrpmpkgs:cloud-init-0.5.14-23.amzn1.noarch.rpm13:05
superxglmaybe the cloud-init is not running ?13:08
=== dendro-afk is now known as dendrobates
kim0maybe the grepping is just not right too13:14
kim0that's an amazon linux instance right .. never used that before13:14
superxglno, it's not an amazon linux instance, it's an eucalyptus instance13:16
superxglbut the cloud-init package is downloaded from the amazon linux instance13:16
superxglmaybe that's the problem ?13:17
superxglmaybe i have to change something ?13:17
kim0just use an ubuntu instance ? :)13:18
superxglemm..but now i need to use centos instance13:19
kim0you'd have to dig deep then how the package starts with the system? rpm -ql cloud-init13:21
kim0is there a service, is it started, open the script, check where it logs ..etc13:22
superxglkim0 : oh, tkx. it is a really useful command.  http://pastebin.com/rfUS8Zjv13:24
=== chuck_ is now known as zul
superxglhttp://open.eucalyptus.com/forum/mysql-server-eucalyptus-image-how      "This only works currently with Ubuntu guests and CentOS AMI guests."14:37
superxglmaybe not CentOS EMI guests14:37
TREllissmoser: I'm checking the latest cloud-init in natty... I want to use it on a baremetal install, I see the preseed values but I'm missing how I specify meta-data/user-data urls? can I do that as a kernel option?14:39
smoseryes...14:40
TREllisie. If I seed it to 'NoCloud', How'd I seed /var/lib/cloud/seed...?14:40
smoserlet me look.14:40
smosertheres kind of two ways i think14:40
TREllisI'm just poking around the code, but I thought asking you might be quicker :)14:40
smoseri would really like your help on getting this documented...14:41
smoseri think the 2 ways are something like:14:41
smosera.) drop files ('user-data' and 'meta-data') in /var/lib/cloud/seed/nocloud-net14:42
TREllisSure, I'm kinda documenting this as I go anyway but for a different project, should be easy to pull out the relevant bits14:42
smoseruser-data can be empty, meta-data should have a value 'seedfrom: http://tinyurl.com/sm-'14:43
TREllisah-ha :)14:43
smoserwhat will then heppen is it will pull from http://tinyurl.com/sm-meta-data and http://tinyurl.com/sm-user-data14:43
smoserthe second way you can do this is via kernel command line. you can put cloud-config values in14:44
TREllisok, so I'm thinking in a late_command I'll pull the meta-data and user-data files down then... or earlier in the boot grokking /proc/cmdline for urls14:44
TREllisah so there is that already14:44
superxgl curl http://202.115.132.66:8773/latest/user-data14:45
superxgli can get the user-data14:45
TREllissuperxgl: try doing that if you aren't actually on a cloud :-)14:46
TREllisie. on baremetal14:46
superxglTREllis:  sorry, i could not follow u. what do u mean " on baremetal" ?14:49
smoserTREllis, right. that will work.14:50
smoseror should14:50
TREllissmoser: but you say 2nd way, via kernel command line, what's the option 'user-data=url.to/user-data' ? ;-)14:51
smoserTREllis, ./doc/examples/kernel-cmdline.txt has info on how you feed the kernel command line14:51
smoseryou can just set cloud-config values. and one of those values (i think) you can set is 'meta-data'14:52
TREllissuperxgl: ie, you are installing a bunch of physical systems and you have zero meta-data service on your network14:52
smosererr... 'seed-from'14:52
TREllissmoser: ah-ha, found that file thanks, I'll write up a small post on it.14:53
smoserTREllis, please let me know what you find. i really apologize for not having done this myself.14:55
smoserbut i would like to get the doc into cloud-init14:55
TREllissmoser: no worries, it's bleeding edge new stuff :)14:55
smoserand fix anything that doesnt work.14:55
superxglTREllis : sorry for my poor englist. so what should i do to get the cloud-init running ? i am still a little confused..14:56
TREllison a related note, I notice that cobbler doesn't respect --kopts-post for ubuntu14:57
TREllissuperxgl: sorry, I don't follow you, I was talking about a different topic14:57
superxglTREllis : hmm...seems that you were talking about cloud-init , but i could not follow ;-)14:58
TREllissuperxgl: we were discussing ways you can provide the same meta-data and user-data like ec2/eucalyptus for cloud-init but without using ec2 or eucalyptus14:59
superxglTREllis : oic...it's exciting..15:01
smoserTREllis, bummer regading cobbler15:04
TREllissmoser: actually... it might just be me. I notice it only uses them in a post snippit which I'm not using, so as long as our grub2 works with 'grubby' it should be ok15:11
superxglmaybe i can use this way on my instances15:16
superxgli will put the scripts on my instances15:16
superxgland once the instance start , make the scripts running15:17
=== salgado is now known as salgado-lunch
=== dendrobates is now known as dendro-afk
superxgl-bash-3.2# python /usr/bin/cloud-init15:34
superxgl  File "/usr/bin/cloud-init", line 4815:34
superxgl    except IOError as e:15:34
superxgl                    ^15:34
superxglSyntaxError: invalid syntax15:34
superxgllooks like the script has some syntax errors ?15:34
superxgl? looks like cloud-init uses python 2.615:46
=== dendro-afk is now known as dendrobates
superxgli have installed python 2.6, but my default version still is python 2.415:52
superxglbut not this problem15:54
superxgl-bash-3.2# python /usr/bin/cloud-init15:57
superxglTraceback (most recent call last):15:57
superxgl  File "/usr/bin/cloud-init", line 23, in ?15:57
superxgl    import cloudinit15:57
superxglImportError: No module named cloudinit15:57
smoserwell, superxgl, i wonder how you installed, but you're not here.16:27
=== niemeyer is now known as niemeyer_lunch
=== salgado-lunch is now known as salgado
=== niemeyer_lunch is now known as niemeyer
TeTeTsmoser: from what I've seen it seems he copied the scripts to whatever os and hoped it would run17:19
michael__nwl yeah cloud foundry looks cool missing the php support :D18:20
=== daker is now known as daker_
michael__any other opensource layer for using uec as a scalable iaas or as a paas (like scalr.net or google app engine) thanks for suggestions18:21
nwlmichael__: none that are very good or mature18:21
michael__yeah that'S my problem atm18:22
michael__scalr isn'T bad so18:22
michael__but paas would be awesome, but it should support php node.js at least18:22
michael__nwl openfoundry looks great they will add php support, but when :D18:30
=== hallyn is now known as hallyn_afk
kim0Hi everyone19:01
kim0Ensemble community weekly meeting starting19:01
kim0basically this meeting should help keep community members aware of ensemble development19:02
kim0I'll be directing any interested people to attend19:02
kim0also summaries of this meeting will be posted to planet ubuntu ..etc19:02
hazmatThis week and last week in ensemble.. we've had a few things go in and a few things in progress. the overall status board for this milestone development (ends at UDS-O) is at http://people.canonical.com/~niemeyer/budapest.html19:02
kim0ideally, any future community contributions would be discussed as well19:03
kim0Since this is the very first time19:03
kim0I think it might be appropriate19:03
niemeyerWe'll soon be moving that to https://ensemble.canonical.com, btw.. we're just getting the web site organized.19:03
niemeyerWell, s/organized/setup19:03
kim0to introduce what works "today" in ensemble .. and what is being worked on19:04
kim0niemeyer: Great19:04
hazmatLast week the ensemble-team landed some changes to formula authorship, specifically the relation-changed-hook was broken out into three separate hooks the relation-joined, relation-changed, relation-depart.19:04
kim0 hazmat care to explain why that split was needed19:05
hazmatCurrently the team is working on a way for the admin to configure services, formula upgrades and fixes to recipes for the the hook change and bashifying our examples.19:05
hazmatkim0, in the course of three formula authors writing hooks, it became clear that none of them where getting the sequencing write on the exchange between services. Ensemble was calling the relation-changed hook with one three events join, changed, depart.19:06
kim0I think it makes sense to touch a bit deep on what each change would mean to a user or formula developer19:06
jimbakerit should be noted that the example recipes for wordpress and mysql have been updated to the new hook semantics seen with -relation-joined, -relation-changed, and -relation-departed19:07
hazmatjoin denotes that a unit of the related service came online19:07
hazmatchange denotes the unit of a related service changed its settings19:07
hazmatdepart  denotes the unit of a related service went offline19:07
kim0hazmat: perfect19:07
kim0hazmat: can you comment on the stuff you're doing for next week as well19:08
hazmatThere are additional descriptions of what a service/service unit/ and relation in our online docs. http://j.mp/ensemble-docs19:08
kim0Thanks for noting that19:08
kim0hazmat: How is formula upgrades planned to work19:09
hazmatThe problem was that formula authors where coding things to relation-change, but that created a subtle timing issue for the sequence, in a distributed system, the change may have already happened by the time the join event was recieved, such that there would be no changed.. The change event is generated/observed against the last time the unit saw the remote unit settings.19:10
hazmatSince it was clear this was a common and subtle issue, we decided to change the hook invocation, such that 1. the hooks where split out into relation-joined, relation-changed, and relation-broken19:11
hazmat2. That relation-changed would always be invoked after relation-joined19:11
kim0Yeah makes sense19:11
DigitalFluxI think that makes sense from a configuration management point of view19:11
hazmatThere's still some work to be do to upgrade the principia repository with these latest semantics19:11
hazmatSo moving on to formula upgrades19:12
kim0Great way for anyone wanting to wet hands with ensemble .. would be fixing principia to the new hook semantics :)19:12
jimbaker(based on my experience, the upgrade to these new hook semantics is quite painless)19:12
kim0jimbaker: is there some pattern ?19:12
jimbakerkim0, basically what you see is that the handshaking becomes explicit19:13
jimbakerprior to the hook semantic changes, formula authors, including myself, were using the accidental ordering of hooks to do the handshaking19:13
kim0setup -> joined , tear-down -> departed right19:14
jimbakernow what you do in your script is you get your relation settings if you need to consume them. if not there, wait until they become available19:14
hazmatThe scope of upgrades is pretty large when talking about an arbitrary services, for now to we're tackling an initial step of upgrading just the formula itself, after some back and forth, we'll be restricting formula upgrades to only those service units that are currently running, if its a broken state, it won't be upgraded. We're separately adding a notion of resolving workflow errors, with optional hook retry.19:14
jimbakerin your script it is as simple as doing an immediate exit 0 at the point of getting the setting19:14
kim0jimbaker: thanks for exaplining that .. guess reading the built-in examples is also a great way to read an example19:15
jimbaker(end of story on painless changes for new hook semantics, it's that simple)19:15
hazmatcool19:15
hazmatso back to upgrades, end result there will be two new ensemble subcommands, ensemble upgrade-formula --repository=examples mydbservice  to upgrade a service formula.19:16
hazmatand ensemble resolved [--retry] unit_name19:16
kim0hazmat: why isn't upgrading a formula as simple as departing an old service unit .. joining a new one .. or is it ?19:16
hazmatkim0, so we're doing formula upgrades, not service upgrades here19:16
hazmatideally the semantic allows for seamless transition without bringing down units19:17
kim0hazmat: assuming our perfect example of a wordpres SU running, with a backend mysql SU running19:17
hazmatie. no events are lost, all queued hook executions and observed changes would continue with the new formula19:17
kim0could you explain the steps for an upgrade please19:17
kim0how it's planned to work rather19:18
hazmatso say we've got three units of a wordpress service called myblog running19:18
* kim0 nods19:18
hazmati've got a new formula which enables some cool plugin in wordpress19:18
hazmatto upgrade the service, i'd run ensemble upgrade-formula --repository=$REPO_DIR myblog19:19
hazmatThat will look into the repo directory for a formula matching the formula of myblog, verify its a newer revision, upload and it mark the units for upgrade.19:19
hazmatOn the unit side, they'll detect they need upgrades, download the formula, ... and most importantly, execute the upgrade-hook from the new formula before any other hooks from the new formula are executed.19:20
hazmatAs an additional restriction, to ease authorship of a upgrade hooks, we'll only be upgrading service units that are currently running.19:21
kim0aha .. a new  hook19:21
hazmatso if the service unit had failed to install or start correctly, it won't be upgradeable19:21
hazmatotherwise it becomes a very open-ended question of what the upgrade hook needs to consider19:22
hazmatif they can't make any assumptions about the environment their running in19:22
kim0Question: Is there some "good-bye" hook for the old version of the formula?19:22
kim0Question: Also, does "upgrade-hook" know which version it is upgrading from ? For instance some wordpress upgrades can have (or not) DB schema changes based on the version you're coming from19:23
hazmatkim0, there are two good-bye hooks depending on context, in the relation context, if a service is related to other services, if a service unit  goes away, those units will get a -relation-depart hook execution, if the service is removed or the relation between services is removed, they'll get a -relation-broken  hook execution.19:23
hazmatin the context of a service unit, if the unit is removed or shutdown (including if the service is removed), the 'stop' hook will be executed.19:24
kim0hazmat: yes but do any of depart or broken get fired in case of upgrade19:24
hazmatkim0, no.. they don't, the to mysql it looks like the service stays online the whole time19:25
kim0cool19:25
kim0wrt question2 .. what version the upgrade is starting from19:25
kim0is that known to upgrade hook19:26
hazmatthe mysql units don't see any changes to the state of the wordpress unit... optionally we're looking in the future of allowing  an upgrade hook to iterate and  modify the relations of the unit, if a hook uses that api, then the remote side would see changes19:26
hazmatkim0, not yet, but that sounds like a good idea :-)19:26
kim0hehe cool19:26
kim0Ironically for "Wordpress" what version you're coming from matters a lot19:27
kim0Should I file a bug :)19:27
hazmatkim0, i'll add it the formula upgrade bug 75048319:27
uvirtbotLaunchpad bug 750483 in ensemble "unit agent needs to support upgrades" [High,In progress] https://launchpad.net/bugs/75048319:27
kim0awesome19:27
kim0that sounds well covered19:28
kim0Any other points19:28
kim0hazmat: anything else ?19:29
hazmatnothing else comes to mind for me that's landing this week, bcsaller is doing service-config work which is very cool, but is going to be another week, jim's doing some work on exposing/securing services using ec2 security groups.19:30
kim0jimbaker: Wanna talk about anything19:30
DigitalFluxI hope i'm not interrupting, but As far as i remember, i did read somewhere that ensemble can depend on shell scripts or puppet to apply the configuration ..19:30
jimbakercurrently ensemble deploys machines such that the firewall is open19:31
hazmatDigitalFlux, all hook are is effectively just an executable, those executable can be in any language and use any tools they'd like19:31
kim0DigitalFlux: Yeah it's a high level orchestration tool .. the "hooks" can be written in any language19:31
DigitalFluxI don't think that's in the docs, right ? http://people.canonical.com/~niemeyer/ensemble/index.html19:31
kim0DigitalFlux: two answers for the price of one ;)19:31
kim0Docs are very basic right now19:31
kim0DigitalFlux: Help make it better :) bzr branch lp:ensemble19:32
jimbakerwe will be addressing such that the firewall is configurable by the ensemble user (ensemble expose, ensemble unexpose), with corresponding hooks (exposed, unexposed), and hook commands  (open-port, close-port)19:32
niemeyerkim0, DigitalFlux: Well, even suggesting improvements would already be welcome, if you don't have the time to push the changes yourself19:32
kim0jimbaker: wow that's cool stuff19:32
jimbakerDigitalFlux, it's described to a certain extent in the docs. i actually copy edited some text in the formulas that make it more explicit19:33
jimbakerformulas *section*19:33
niemeyerkim0: It is indeed!  Effectively an Ensemble user can just do "ensemble expose myblog" and that works19:33
hazmatniemeyer, do the draft/specs get published on the doc site?19:33
niemeyerhazmat: Yeah19:33
hazmatah. there it is19:33
kim0What is open-port close-port ? why are those needed19:33
hazmatthe expose work is described in further detail at http://people.canonical.com/~niemeyer/ensemble/drafts/expose-services.html19:33
* kim0 clicks19:34
jimbakerkim0, we need to tell ensemble what ports a service needs opened19:34
hazmatand http://people.canonical.com/~niemeyer/ensemble/drafts/expose-services.html#service-exposing-for-formula-authors19:34
jimbakerthere was some debate on whether to describe this in the metadata, but we decided that it was too restrictive to make it static19:34
DigitalFluxThat's great !, i wil just have to read more about the hooks then ..19:34
jimbakeralso, a service can open and close ports for a given service unit when that unit is in fact ready19:34
hazmatkim0, there's a notion that a service isn't nesc. running on a fixed port, that over time we'll need to do some negotiation for service ports, so that we can do multi-tenant on a machine19:35
kim0ah that's cool as well19:35
hazmatie. run three units of different wordpress services on a single machine, each connected to a load balancer for the service.. even with lxc network namespace isolation, we'll be doing a nat against the host machine interface to the container network device.19:36
kim0does {open,close}-port operate on top of ec2 security groups ?19:36
jimbakerDigitalFlux, this copy editing is in a branch right now, see lp:~jimbaker/ensemble/new-hook-semantics-5-docs19:36
hazmatkim0, the mechanism is generic, in future we'll be utilizing machine level firewalls, but for now it will use ec2 security groups19:36
hazmatbut such a change won't effect the formula-author or user workflow19:36
* DigitalFlux pulling ..19:37
jimbakerthis was another important reason - we wanted it provider independent19:37
kim0hazmat: so different backends .. i.e. in the future things like cisco firewall APIs would be useable right19:37
kim0without even formulas needing change! freaking awesome :)19:37
niemeyeralonswartz: How does it feel so far.. looks reasonable?  Anything we can help with?19:37
niemeyerkim0: Exactly!19:38
kim0jimbaker: Great stuff, anything else you want to comment on19:38
DigitalFluxOverwhelming :)19:38
kim0Indeed it is19:38
hazmatto be fair the implementation change won't effect them, but the introduction of port-negotiation/allocation might.19:38
kim0Only because it's the first time I suppose :)19:38
hazmatindeed19:39
jimbakerkim0, i think that pretty much covers it for me. i have some other work to smooth out rough edges and inconsistencies, as might be expected :)19:39
kim0Any special handling to RPC style services .. a la NFS19:39
kim0I guess it's a corner case19:39
kim0but why not bring that up :)19:40
jimbakerkim0, we have only considered a very simple mode of port opening, but potentially we could extend it19:40
kim0definitely makes sense19:40
kim0Great stuff indeed19:40
jimbakerwe do cover tcp and udp, but not how it would interact with stateful firewalls for example19:40
kim0jimbaker: guess ec2 security groups cover those too as well19:40
jimbakerkim0, indeed, it was an easy mapping for now :)19:41
kim0um .. icmp ? ipsec ? future versions right19:41
kim0those not having a notion of port .. might be a bit problematic19:41
jimbakeri do believe the mechanism we have in place should readily support other protocols in the future, it's not restrictive on what open-port means for example19:41
kim0yeah let's leave it till then19:42
kim0cool19:42
alonswartzniemeyer: sorry, I had to handle a quick emergency...19:42
niemeyeralonswartz: Oh, no problem19:42
kim0alonswartz: hey19:42
alonswartzHey kim0 :)19:42
niemeyeralonswartz: Was just wondering if we could help with anything you might be pondering about, since you already had some background on the project previously19:42
kim0o/19:42
alonswartzI didn't get to follow the whole conversation, but I do have a question19:42
alonswartzIs the project ready for outside collaboration yet?19:43
kim0It is!19:43
* kim0 looks around19:43
kim0bzr branch lp:ensemble19:44
niemeyeralonswartz: Absolutely19:44
niemeyeralonswartz: The whole development happens in the open19:44
kim0alonswartz: https://lists.ubuntu.com/mailman/listinfo/Ensemble19:44
niemeyeralonswartz: Open IRC channel, mailing list, source code, etc19:44
alonswartzI have to admit I haven't looked into ensemble since my brush with it at UDS-M19:44
niemeyeralonswartz: We've come a long way.. it actually works now :-)19:45
kim0hehe19:45
alonswartzbut it seems you have made great head way19:45
alonswartzI'll read up the docs, take it for a spin, and then I'll be more clued up as to the design, and hopefully have ideas on how I can help19:45
kim0bcsaller: o/19:46
bcsallerhmm?19:46
alonswartzOT, I saw you chose to use txaws. out of curiosity, why not boto? does ensemble use async?19:47
kim0It's the first time19:47
kim0bcsaller: So you're free to comment on anything you like :)19:47
bcsallerI think I'll have more new stuff to show next week19:47
jimbakeralonswartz, we use twisted completely19:47
jimbaker(this works better with the underlying zookeeper implementation)19:48
kim0bcsaller: sure thing .. we're not limited to last week though for this time ..19:48
alonswartzbefore I ask stupid questions that are answered in the docs, I'll read them first...19:48
kim0Alrighty19:49
hazmatwe debated a while ago about twisted vs. eventlet/gevent.. but its done at this point.19:49
alonswartzbut I can't help myself, does/will (or how does) ensemble handle scaling ?19:49
jimbakershort answer: yes!19:49
kim0as in add service units in runtime19:49
hazmatalonswartz, at the moment, you can scale via adding removing units to any service in the environment19:49
kim0I think it already does huh19:49
alonswartzso the user needs to do it manually, right? there isn't a master ensemble server that handles it transparently...19:50
jimbakeralonswartz, again i'd refer you to our docs, many of the hooks deal with scaling up and down19:50
hazmatthere isn't any autoscaling exposed at the moment, just the ensemble command line to do it.19:50
hazmatalonswartz, but its a bit more automated than manual.. if i have mediawiki connected to mysql, memcached, and haproxy.. adding a unit of mediawiki, auto configures it with all those other services.19:51
alonswartzfair enough, i'll read the docs and come back with better questions :)19:51
kim0Open topic .. any comments, questions19:52
niemeyerThanks all!19:52
jimbakerthe one thing we don't do that's part of the autoscaling is automatically determine how many service units you need. that would seem to depend on your specific stack needs19:52
kim0Can't believe we almost consumed the full hour19:52
kim0this was a blast19:52
niemeyerjimbaker: That's up for debate19:52
jimbakerbut we do handle the orchestration, and that's a differentiating factor compared to say puppet19:53
niemeyerjimbaker: We haven't covered that topic yet, so let's not provide the impression we're not doing it when in fact we really have to think through the use cases.19:53
kim0I would want to hook auto-scaling to some performance-measuring-hook19:53
jimbakerniemeyer, agreed19:53
niemeyerkim0: Exactly19:53
kim0lovely19:53
* kim0 hugs Ensemble19:53
alonswartzThanks folks!19:54
kim0Thanks niemeyer jimbaker bcsaller hazmat alonswartz DigitalFlux19:54
kim0Meeting end19:54
jimbakerkim0, thanks for organizing this19:54
bcsallerkim0: thanks19:54
kim0Thank you folks .. This is really exciting stuff19:54
DigitalFluxYeah, Thanks kim019:55
kim0Lovely Ubuntu communities hehe :)19:55
DigitalFluxautoscaling with upper limits for the resources would be AWESOME !19:56
kim0ec2 limits you to 20 instances any way :P19:56
DigitalFluxkim0: Oh, didn't know that19:56
kim0you have to explictly ask them to raise that limit19:57
DigitalFluxBut still, if that can be done based on some metrics that i specify, and machines will scale up and down ..19:57
DigitalFluxWell, i will have to quit my job then :-)19:57
kim0DigitalFlux: ec2 has some form of that19:58
kim0they can monitor load average and auto-scale up/down19:58
kim0TEH cloud :)19:58
kim0BUT19:58
DigitalFluxSo what form of orchestration do you point to here ?19:58
alonswartzkim0: ec2 has a 20 instance limit?19:58
kim0ec2 would "scale" by launching you an instance and you're on your own19:58
DigitalFluxjimbaker: So what form of orchestration do you point to here ?19:58
DigitalFluxkim0: ofc19:59
kim0alonswartz: yeah afaik .. you can raise it19:59
kim0you'd just have to contact them19:59
kim0or use a different account ..19:59
kim0DigitalFlux: with ensemble .. you're not on your own20:00
kim0an instance is fired, hooks are triggered to configure it and join it to neighbour instances20:00
alonswartzkim0: good to know. I didn't know they had a limit.20:00
DigitalFluxkim0: That's the new slogan for the project ? :-)20:00
kim0alonswartz: you probably wanna check that too :)20:00
kim0dont depend on my leaky memory .. but I found traces of that on google .. so I'm probably not making it up20:01
alonswartzi was planning on spinning up a 100 instance cluster (for a batch job), so I'll need to look into that20:01
kim0hehe perfect timing20:01
alonswartzi'll report back :)20:02
* kim0 is interested what would need 100 instances20:02
alonswartzkim0: batch build of all turnkey appliances, all target formats, all amazon regions, all...20:03
alonswartzthe list goes on and on ..20:04
kim0a ha .. got it20:04
kim0one thing I disliked about ec2 .. is their biggest instance is 64G ram20:04
alonswartzadd 64bit to the equation and it spirals out of control...20:04
kim0a friend of mine doing scientific numerical analysis needs a bigger rig .. any suggestions20:05
=== hallyn_afk is now known as hallyn
=== dendrobates is now known as dendro-afk
jimbakerDigitalFlux, sorry was in another meeting20:22
DigitalFluxjimbaker: no probs, i'm in another one right now :)20:23
jimbakerensemble ensures that all hooks are fired on all service units, in the correct order20:23
DigitalFluxjimbaker: you mentioned orchestration20:24
jimbakerso this is what i would describe as orchestration20:24
DigitalFluxhmm, and puppet doesn't do something similar ?20:24
jimbakera given service unit only needs to take care of itself, responding as its hooks fire20:24
jimbakerpuppet/chef is very different. it's top down. and because there's a need to worry about the layout, it's hard to write generic recipes20:25
DigitalFluxyeah, there have been some efforts done to enhance this, but i guess the model is still different than ensemble20:26
DigitalFluxI need to work on how can i get both of the worlds i guess20:26
* DigitalFlux needs to do some lab20:26
jimbakerDigitalFlux, it's definitely a different model. one thing i was challenged on is that puppet better accommodates a model of sysadmins actually logging into a machine and messing around20:27
jimbakerthis is defended on the basis that critical fixes must be done and now ;)20:28
jimbakeri think the cloud model where we don't care about specific machines is inherently better for this20:28
DigitalFluxhmm, considering puppet+mcollective, wouldn't that be already solved using this combination ?20:29
jimbakeras i recall mcollective, we still have a better foundation in zookeeper, which gives us total ordered events and a consistent, durable view of the world20:29
kim0jimbaker: I would love to capture that in plain English in a faq page :)20:30
jimbakerbut i can really talk best about ensemble20:30
kim0did niemeyer check mcollective as well20:30
kim0and how ensemble would compare20:30
jimbakerkim0, it really comes straight from a zookeeper faq, but there are some ensemble considerations that need to be mixed in20:31
=== dendro-afk is now known as dendrobates
kim0jimbaker: I'm adding a faq page tomorrow .. so would love if you could explain a bit more how ensemble is different from puppet+mcollective20:32
jimbakeri'm probably not the best person on the team, but i'm sure we can help there20:32
niemeyerkim0: There's absolutely nothing like Ensemble out there20:33
kim0niemeyer: I'm writing a faq page tomorrow20:33
kim0some nice explanations are most welcome :)20:33
niemeyerkim0: Not even close.. the perspective taken is completely different20:33
niemeyerkim0: https://j.mp/ensemble-docs  should help20:33
kim0I guess I should read more about mcollective :)20:34
niemeyerkim0: I think this is the wrong perspective..20:34
niemeyerkim0: Documentation on Ensemble should be produced to demonstrate its merits, and why we're developing it20:35
niemeyerkim0: There's no point in *starting* with something like a competition shootdown20:35
kim0niemeyer: yeah but how is ensemble different from puppet/chef and how it relates to augeas, mcollective...etc is FAQ I think20:35
kim0Not that I'd talk down other tools .. each has a purpose in life20:36
niemeyerkim0: Show why we're developing Ensemble, why it rocks, why we love what we're doing20:36
niemeyerkim0: Let people figure by themselves how it compares20:36
jimbakerkim0, fwiw, we recommend using augeas w/ ensemble20:36
kim0jimbaker: awesome20:36
jimbakerit's a nice tool that's perfectly matches w/ what ensemble does20:37
jimbakervs say using puppet templates, which could be done, but just brings in more unnecessary stuff20:37
jimbakerprincipia currently demonstrates the use of augtool, and the bashified examples i've been working on will add it in a future branch20:38
kim0jimbaker: does ensemble discourage templates ..20:41
jimbakerkim0, not at all. they can be the only way to accomplish certain things. but augtool is more robust20:42
jimbakerassuming the lens exist (writing them is more complicated however)20:43
jimbakerhowever, one can readily write templates in nearly any language. the bashified examples for example demonstrate that with bash here docs20:44
jimbakereverything the template needs can simply be obtained from the necessary settings20:44
jimbakerthe local scope that a hook runs in - it only has to work on changing its service unit and coordinating through settings - makes it unnecessary to do more complicated things in the templates20:46
jimbakerensemble really has this wonderful simplicity about it. i think that is what excites us a team20:46
jimbakeras a team20:46
kim0always a sign you're doing things the right way :)20:46
jimbakerkim0, indeed. that's why the apt for the cloud is so appropriate as a way of conceptualizing what we are trying to do20:47
jimbakereven if we are working just on the dpkg part right now20:48
kim0hehe indeed!20:48
Heartsbaneso I am curious with the example listed at https://help.ubuntu.com/community/UEC/BundlingImages20:59
kirklandHeartsbane: hi, trouble bundling?  what's your error?21:00
HeartsbaneI keep getting Invalid certfailed to upload kernel21:00
Heartsbanefailed to bundle kernel lucid-server-uec-amd64-vmlinuz-virtual21:01
Heartsbanefailed: euca-bundle-image --destination /tmp/uec-publish-image.ljX0Sp --arch x86_64 --image /tmp/uec-publish-image.ljX0Sp/.rename.ulD5ZK/lucid-server-uec-amd64-vmlinuz-virtual --kernel true21:01
Heartsbanex86_6421:01
HeartsbaneI am trying to understand what is happening there21:01
HeartsbaneI am assuming there is more reading that someone can suggest or refer me too21:02
=== chuck_ is now known as zul
* Heartsbane kicks himself over and over and over and over.21:36
Heartsbanesorry I figured it out ... time for more caffeine21:37
Heartsbaneit was such a little thing :(21:40
* Heartsbane is a ashamed.21:41
michael__any suggestions for a good layer on top of uec? in the way of scalr.net or google app engine? should be opensource.21:47
SpamapSinteresting presentation about 'Xeround' at the MySQL UC22:23
SpamapSmichael__: just pick your favorite language/framework22:23
michael__SpamapS: and how to make it scalable in the uec ?22:26
michael__I would like to run a few projects in a scalable way with a bit of failover resistance sorta22:26
=== dendrobates is now known as dendro-afk
=== dendro-afk is now known as dendrobates
=== salgado is now known as salgado-afk
=== timwood_ is now known as timwood
=== dendrobates is now known as dendro-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!