/srv/irclogs.ubuntu.com/2012/03/18/#juju.txt

hazmatjcastro, there's one more branch before that works for ec2, i'm going to try and get a review on it today, but i've perused it and looks pretty good already00:00
hazmatyou can define the constraints on the service or unit level00:00
jcastrothat is awesome because we just ran into this problem, so now I can see why we would need it in real practice00:00
jcastrod0od, we severely underestimated how huge OMG is00:07
jcastroI'm going to relaunch it with larges00:07
jcastromarcoceppi, how fast is the ec2->ec2 transfer?00:23
marcoceppijcastro: probably 10-15Mb/s00:23
marcoceppiThe meat of the OMG Site is about 2GB in size (images, plugins, themes, etc)00:24
marcoceppiSorry, not 10-15, it's currently tracking at 60Mbps00:25
marcoceppijcastro: Tear it down, everything is moved off00:27
jcastroon it00:28
marcoceppiIf you want, you can use the patched branch that fixes the haproxy issue00:28
jcastrolink me to it pls?00:28
marcoceppilp:~marcoceppi/charms/oneiric/wordpress/haproxy-patch00:28
jcastrota00:28
jcastrohey so the wife wants ice cream00:29
jcastrowhen I launch these00:29
marcoceppiI'll wait for the team to review, it's a minor change but I'm not sure if there's a *better* way to do it00:29
jcastroand get you in them can you drive for a while?00:29
jcastroI shouldn't be longer than 3000:29
marcoceppiyeah, no problem00:29
marcoceppiLaunch and I can relate, move files, setup, etc00:30
jcastrobootstrapping00:30
marcoceppid0od: I saw that you have cdn.omgubuntu.co.uk setup, but it looks like you're pointing it to the same server as the main site00:31
marcoceppiDo you have a CDN for OMG?00:31
jcastroman, waiting for bootstrap on ec2 vs. lxc is not a fun time00:31
marcoceppiI know, *keeps pinging status*00:32
d0odThe CDN was something our then web admin set up; i really don't know more than that00:32
d0odshocking, I know ;)00:33
marcoceppiha, no problem! just checking00:33
marcoceppiDo you have access to the DNS for the domain?00:34
jcastrook deployed00:35
jcastromarcoceppi, not started yet00:35
d0odmarcoceppi: I do00:35
marcoceppijcastro: it looks like they're finally coming up00:37
jcastromysql is up00:37
jcastrojust waiting on wp00:37
jcastroand ... up00:37
jcastrogo.00:37
jcastrook lmk when you have access to stuff, and then I'll pop out and be back in 30 or so00:38
jcastroNo EC2 Instances selected.00:38
jcastroSelect an instance above00:38
jcastromispaste, sorry00:38
marcoceppijcastro: I'm good! Enjoy00:38
jcastrobbiab!00:38
marcoceppiTransfering the Dabatase dump back over00:39
marcoceppiOkay, database is on the server, importing. WordPress is setup, haproxy in place, and the OMG content is being transferred over00:48
d0od:D00:48
marcoceppiOnce the content is extracted it'll be a quick skip and hop to a living site00:50
d0odI can't thank you enough for doing this :)00:50
marcoceppiNo problem! I hope to have this done before jcastro gets back00:51
marcoceppiDatabase imported, content synced. Moving all the images, plugins, and theme in to place00:53
marcoceppiOkay, d0od, it's not pretty but: http://ec2-23-20-126-44.compute-1.amazonaws.com/ the post links don't work yet either. The images and layout will be fixed when we "redirect" the domain00:56
marcoceppiIf you open your browser in to porno mode, http://ec2-23-20-126-44.compute-1.amazonaws.com/ that should look like OMG Ubuntu!01:01
marcoceppiImages won't all be working yet, because of the domain name01:01
marcoceppiJust need to get the seo friendly URLs setup01:02
jcastromarcoceppi, back, wow that was fast!01:02
marcoceppijcastro: slow poke01:02
marcoceppi:)01:02
marcoceppiIt's always faster the second time around, when you know what to do01:02
jcastroheh01:03
marcoceppitellin' ya, it's like 10 mins away from being a charm01:03
jcastrohey so how's the IO on the larges? Much better?01:03
marcoceppismoooooooooth01:03
jcastronice01:03
marcoceppithe old server was an 8core beast01:03
jcastrook so what do we need to do then ... just the redirect?01:03
marcoceppione more thing01:03
marcoceppipost links aren't working01:03
marcoceppibut I think that's a mis-behaving plugin01:03
jcastroyeah we should have looked at the hw it was on before deciding on smalls01:03
* marcoceppi nods sadly01:04
jcastro1 large for a bootstrap node, heh01:04
marcoceppiwell, Zookeeper should be happy01:05
marcoceppiSo, I'm not sure how to fix the whole post not working, I'm going to try to create myself an account01:07
jcastrod0od, ok get ready01:20
* d0od readies self01:20
jcastrohazmat, any idea how we can move the bootstrap node and haproxy to smalls?01:21
jcastroor is it like "wait a week and redeploy"?01:21
hazmatjcastro, the bootstrap node isn't really resizable without additional work01:25
jcastrook01:26
jcastromarcoceppi, about ready?01:27
hazmatthe whole notion of resizing isn't really supported.. but if you want to try.. i'd first try stopping and restarting a non bootstrap node, check if it comes up okay.. then snapshot, and boot the snapshot as a new instance, hmm.. i don't think its going to work.01:27
marcoceppiyeah, just need to patch up the wordpress install so it functions 100%01:27
hazmatits got to update the zk data with the new instance id or juju will terminate it as an unknown (if it shares the environment group, which it needs to for net security)01:27
jcastrook so I think it's best to wait until the ability to specify a size per node lands01:27
jcastrothen just redeploy?01:27
hazmatjcastro, yeah01:28
marcoceppijcastro: founda  few more things that need to be added to the wp charm, php-mail and sendmail are not installed01:32
jcastro.... adding to the doc01:32
jcastrod0od, ok wordpress problem, I texted SpamapS, he's on his way01:51
jcastrojust this one thing left to fix01:51
jcastromarcoceppi, is there a way to turn off plugins from a config file or something?01:53
SpamapSjcastro: use /mnt and backup/replicate aggressively ;)01:53
SpamapSjcastro: /mnt is the instance storage on an ebs root instance01:53
marcoceppijcastro: I'm in the admin panel01:53
SpamapSits huge, and heavily outperforms EBS01:53
jcastro... and we ran out of space on EBS anyway. :)01:53
jcastroso he moved it01:53
marcoceppiSpamapS: do all units have access to that same /mnt?01:54
SpamapSmarcoceppi: no definitely not01:54
marcoceppidamn, okay01:55
SpamapSthat is local to the instance01:55
jcastrothat would be too easy01:55
jcastromarcoceppi, hey so how many plugins are in there?01:55
jcastromaybe we can shut them off01:55
jcastroand then turn them on one by one?01:55
marcoceppi29 active01:55
jcastrohah, seriously01:55
SpamapSmarcoceppi: whats the trouble?01:56
SpamapSshared image upload?01:56
marcoceppiwell, 29 of 45 :)01:56
SpamapSsimplest thing would be to use the wordpress S3 plugin.01:56
SpamapSwhich I used long before I had my blog in EC2 :)01:56
marcoceppiFIXED01:57
marcoceppiFIRE MISSILES01:57
SpamapSbut I am le tired01:58
jcastrourl please?01:58
marcoceppihttp://www.omgubuntu.co.uk/2012/03/instant-messaging-comes-to-thunderbird-13-speed-dial-to-firefox-13/01:58
jcastrodb error01:59
jcastrothere we go01:59
jcastrothe tbird image is broken though01:59
marcoceppifile isn't on the server02:00
jcastromarcoceppi, your hackergotchi is showing up instead of joey, heh02:01
marcoceppijcastro: yeah, refresh and I should be gone02:01
marcoceppinvm, I'm still there, Page is cached02:02
jcastrostill getting db errors02:03
jcastrois the mysql box getting thrashed?02:03
SpamapSthe tuning on the default mysql is really low scale02:04
SpamapSThere are some big knobs for bumping it up02:04
marcoceppijcastro: connections are maxed out02:04
marcoceppiturning big knobs02:04
SpamapSI believe that is one of them02:04
SpamapSjuju set max-connections=500 should work02:05
jcastromarcoceppi, you comfortable turning them up or want clint to have a look?02:05
marcoceppiNormal circumstances I would say STAND BACK, but Clint I hear you're like a SQL wizard02:06
marcoceppiI can add your key to the db to take a look and work some magic02:06
jcastrocpu looks maxed on the wp one too according to the Aws meter02:06
marcoceppijcastro: I'm going to drop a fix for that in a min02:07
marcoceppisee if we can't keep AWS from getting too pissed02:07
SpamapSmarcoceppi: I do have some mysql scaling experience yes. :)02:07
jcastroyes, let's do this02:07
SpamapSmarcoceppi: probably want to bump dataset-size > 1G unless the database is in fact ~= 1G02:08
marcoceppidb is about 1.2gb02:08
SpamapSnice ok02:08
marcoceppiSpamapS: PM'd you the server address02:08
marcoceppiI haven't done much, but I was planning to move MySQL to a tmpdisk02:08
SpamapSmarcoceppi: tmpdisk? are we replicating or backing up aggressively at all?02:09
marcoceppiright now it's just one instance02:10
SpamapSok02:10
SpamapSWhat I'd recommend is that we spin up another env on t1.micro's in a different region...02:11
SpamapSand manually setup replication to that.02:11
SpamapSAny chance we started with t1.micro's here? Because they have the advantage of being able to be rebooted into any other instance type02:11
jcastrowe started with smalls02:12
SpamapSjcastro: darn.02:14
jcastrook so we need more instances?02:15
SpamapSwow... MyISAM ..02:15
jcastroWe can't tune the existing one?02:15
marcoceppiSpamapS: ah, I converted the first time around, hadn't done it for this time around02:15
SpamapSjcastro: no, but thats a way around the "all larges" problem02:15
marcoceppiall but a few tables can be moved to InnoDB02:15
SpamapSmarcoceppi: was there a mysqldump ?02:15
SpamapSbecause the default table type is innodb02:15
marcoceppiSpamapS:  yeah it's in /usr/src02:16
marcoceppiDump had them as MyISAM02:16
SpamapSBecause MyISAM will have table locks which will make comment traffic suck ass02:16
SpamapSProbably means the original site had MyISAM02:16
SpamapSlets not d*** with that02:16
jcastrocomments are on disqus02:16
marcoceppiOriginally (OMGEC2-v1) I did a quick dump of the table names and update to InnoDB, few tables have fulltext indexes (yuck) and can't be converted02:16
SpamapSobal turned on slow query logging for a bit.. the thing looks healthy02:18
SpamapSjcastro: score, ok02:18
SpamapSjcastro: probably because the old comments were so damn slow.. ;)02:19
jcastromarcoceppi, is the db problem the reason the Continue Reading, etc don't work?02:19
SpamapS33qps .. 1GB db.. could have stayed on small. :)02:19
SpamapStho I suspect it gets a lot busier sometimes. :)02:19
jcastrowe can move it to smalls tomorrow02:19
jcastromarcoceppi, ooh, cpu on wp going down02:20
marcoceppihopefully if this caching works out like it's supposed to, we can move the site back down to a smaller instance and just use add-units when needed02:20
jcastro^^02:20
jcastrothat's why we went with smalls to begin with02:20
jcastromarcoceppi, ... and pegged again02:21
SpamapSYeah seems like without comments this site should be nearly static02:21
marcoceppiSpamapS: should02:21
marcoceppicomments are off-site with disquss02:21
marcoceppijust a lot of plugins02:21
jcastromarcoceppi, hmm so what do you suppose is going on with the wp node?02:22
SpamapSquery cache is handling most of the mysql load...02:23
marcoceppijcastro: php02:23
SpamapSmarcoceppi: apc?02:23
marcoceppiI think that's it02:26
* SpamapS installs mytop and enjoys the show02:26
marcoceppijk, it's back02:26
SpamapSok, I need to run out and get dinner for the family.. you guys need anything else?02:26
jcastrook so mysql is set?02:27
jcastroman02:27
jcastrothat is waaaay faster02:28
SpamapSmysql is doing nearly nothing02:28
jcastroman so all that mess was the ootb mysql being that horrible?02:28
SpamapS Key Efficiency: 97.9%  Bps in/out:   1.0/335.9   Now in/out:   8.3/ 3.2k02:28
SpamapS97.9% means that the indexes are all basically fitting in the anemic buffers that are left over because we tune for InnoDB by default02:29
jcastroI owed you one, <302:29
SpamapSjcastro: this has been so much fun.. I totally miss running a real site. ;)02:29
* SpamapS sighs02:29
SpamapSOk, I have to go get tacos. Will check back in later.02:30
jcastrowell, if you want, you can fix php02:30
jcastrook, cheers!02:30
SpamapSVERY COOL btw02:30
SpamapSinstall APC02:30
SpamapSPHP should fly02:30
SpamapSit seriously helps02:30
SpamapSok, out, bbl02:30
marcoceppiSpamapS: APC is already installed :)02:30
marcoceppiWHAT SAY YOU NOW!02:30
SpamapSSo we're pegging 4 CPU's w/ PHP? time to look at varnish :)02:31
* SpamapS gone02:31
hazmatvarnish rocks02:44
jcastrotoo bad there's no charm02:45
jcastrohah02:45
hazmatalthough i've been using the builtin nginx cache facilities for more trivial caching as of recent02:45
hazmatits hard to charm i think, to allow for flexibility you end up needing to put vcl config down the relation.. for really basic upstream caching its okay02:46
_mup_juju/scheduler-peek-list r474 committed by kapil.thangavelu@canonical.com02:49
_mup_new scheduler v2, much expanded error testing, now for version 302:49
jcastrod0od, ok03:04
jcastroworking03:04
jcastrothe images weren't in the backup03:04
jcastroso marco is looking on the old server03:04
jcastrowe can move dns now if you want03:04
marcoceppid0od: Images recovered03:16
marcoceppieverything's in place03:16
_mup_Bug #958312 was filed: Change zk logging configuration <juju:New> < https://launchpad.net/bugs/958312 >03:56
_mup_Bug #958378 was filed: juju/control should be aware of subordinates <juju:In Progress by bcsaller> < https://launchpad.net/bugs/958378 >07:47
=== Leseb_ is now known as Leseb
=== d0od_ is now known as d0od
jcastromarcoceppi, alright, she's still up and running!13:47
marcoceppijcastro: she lives!13:47
marcoceppiThere's a few quirks with the WordPress charm that I was going over with d0od13:48
jcastroah ok13:48
jcastrodo we have that logged?13:48
marcoceppiit was PM I can put it in the doc13:49
jcastrok, whenevs13:50
jcastrohey I noticed search is wonky, probably like the other links where it needs the real domain I am guessing13:50
marcoceppijcastro: Yeah, on my desktop (with host modification) it works fine. On my laptop without host modification it's hit or miss13:52
jcastroCPU on the wordpress notes is spiking13:52
marcoceppi 13:52:44 up 13:17,  1 user,  load average: 0.44, 0.56, 0.6813:52
marcoceppiwouldn't appear so?13:52
jcastroI am just looking back at the AWS graph13:53
marcoceppiah13:53
marcoceppihttp://i.imgur.com/Aw0FH.png13:53
marcoceppithere were a few spikes over 113:53
jcastromarcoceppi, ok so he can get back in now?13:56
marcoceppijcastro: Yeah, it was a dns issue preventing him from logging in13:57
marcoceppiHe's all setup13:57
marcoceppiStupid Twitter thing doesn't work though14:02
=== koolhead17|away is now known as koolhead17
SpamapSmarcoceppi: good "morning" :)16:10
* hazmat digs himself out of the rabbit hole16:14
marcoceppiSpamapS: o/16:30
_mup_juju/scheduler-peek-list r475 committed by kapil.thangavelu@canonical.com16:32
_mup_rewrite of the relation hook scheduler16:32
SpamapSmarcoceppi: so, did you just accept that PHP was going to get pegged sometimes?16:39
marcoceppiSpamapS: we tried varnish and that sucked. I went back and looked at APC, quick tweak, hard stopped apache, brought it back up16:40
marcoceppithat combined with aggressive caching in WP settled the load to an average of 0.43 for the night on the WP node16:40
SpamapSmarcoceppi: ahh, was APC not actually working?16:40
marcoceppithat's what it looked like16:40
marcoceppiEnable APC, load jumps extra 20 points. Didn't seem right16:40
SpamapSmarcoceppi: yeah varnish is hard to get right16:40
marcoceppiIt seems really complex to charm, but would be great as a proxy/cache drop in for HAProxy16:41
SpamapSmarcoceppi: "quick tweak" ? did it land in the wordpress charm yet? ;)16:41
SpamapSmarcoceppi: yeah m_3 started working on it16:41
marcoceppiSpamapS: there's a bunch of stuff that needs to be tweaked in the WP charm ;)16:41
SpamapSmarcoceppi: yeah I was thinking we should look at switching it to fastcgi16:42
marcoceppimm16:42
marcoceppiI'm making a bunch of changes in the omg fork of the charm16:42
marcoceppithen see what I can slide back in16:42
marcoceppiThere's also some oddities when you throw HAProxy in front16:43
marcoceppiThe silly archive version of WP doesn't handle it very well16:43
SpamapSmarcoceppi: the way the configs are done in the packaged version is kind of confusing16:44
SpamapSmarcoceppi: I think we need to enhance the 'http' relation with some new optional bits.. one of those being 'endpoint hostname'16:45
SpamapSmarcoceppi: I'm not settled on whether haproxy should feed that back to wordpress, or wordpress should feed it to haproxy..16:46
SpamapSmarcoceppi: but either way, I'm sure you had problems with the Host: header16:46
_mup_juju/scheduler-peek-list r476 committed by kapil.thangavelu@canonical.com16:53
_mup_cleanup for review16:53
_mup_juju/scheduler-peek-list r477 committed by kapil.thangavelu@canonical.com16:55
_mup_expand ignores16:55
=== Leseb_ is now known as Leseb
hazmati'm still amazed sometimes how much memory/cpu emacs uses16:58
=== Leseb_ is now known as Leseb
=== almaisan` is now known as al-maisan
=== al-maisan is now known as almaisan-away
_mup_Bug #958662 was filed: Rewrite the scheduler, for simplicity, and better error handling. <juju:New> < https://launchpad.net/bugs/958662 >17:12
_mup_juju/scheduler-peek-list r478 committed by kapil.thangavelu@canonical.com17:16
_mup_log stops17:16
_mup_Bug #958668 was filed: Rewrite the scheduler, for simplicity, and better error handling. <juju:In Progress by hazmat> < https://launchpad.net/bugs/958668 >17:17
* hazmat sighs17:17
_mup_juju/local-provider-container-wait-flags r485 committed by kapil.thangavelu@canonical.com17:20
_mup_wip on lxc containers with bitmask wait flags, something odd in the lxc responses, reporting started units as stopped in lxc-ls, tbd17:20
hazmatbcsaller1, please resolve your branch (subordinate-type) with trunk conflicts17:48
bcsaller1hazmat: ok and I noticed a another issue, I'll push an updated version in a bit18:28
hazmatjust noticed some failures on trunk from my hook-alias branch merge.. argh.18:30
hazmatah.. found it18:31
hazmatbcsaller1, did you end up normalizing log locations as part of subordinates?.. if not i'm going to do a branch against trunk for it18:56
bcsaller1hazmat: I didn't address that, no18:57
hazmatbcsaller1, no worries, the normalized form i'll use should accomodate it, i'm just going to use the local scheme on other providers18:57
hazmatbcsaller1, jimbaker can i get a +1 on this trivial?19:07
hazmathttp://paste.ubuntu.com/889598/19:15
bcsaller1hazmat: in process_reason the first branch returns and the second doesn't now?19:18
hazmatbcsaller1, the second branch and third branch have return at the end of the conditional block, the other two conditional blocks setup the return value, the first block is self-contained19:19
hazmatie. the others define error, and then return deferred.errback(error) at the end of the method.. the first one was failing because..19:21
_mup_juju/trunk r485 committed by kapil.thangavelu@canonical.com19:30
_mup_[trivial] hook exit and test teardown one liner fixes [r=bcsaller]19:30
pindongahi, I'm trying to use juju on precise and I'm getting errors about dns name resolving issues... looks like it's not using --ipv4 when running the equivalent of euca-describe-instances19:44
pindongaany ideas?19:44
hazmatSpamapS, not sure how to  setup --no-recommends cloud-init installs juju19:51
pindongaso, on another angle, what's the recommended way for starting with juju if I don't want to use amazon (due to it's cost basically)?19:52
pindongais it openstack or lxc?19:52
pindongaI have not found a single way to deploy a charm yet in either :(19:52
hazmatpindonga, ugh..20:07
hazmatpindonga, i use lxc all the time, openstack is also viable20:07
hazmatpindonga, do you have a traceback with the dns issues?20:07
hazmater. pastebin/log20:07
pindongalet me get one for you20:07
pindongahazmat, http://pastebin.ubuntu.com/889699/20:09
pindongaalso, when I re-run the bootstrap, now I get an error saying the security group cannot be deleted20:10
pindongahttp://pastebin.ubuntu.com/889701/20:10
pindongais that expected?20:10
pindongaI'm running precise (afaik)20:11
pindongawith lxc i get a log entry saying 'Creating master container...' and nothing after that, so juju will always see the container as pending20:11
pindongaI can share my environments.yaml if that helps20:12
hazmatpindonga, you have to destroy-environment before doing another bootstrap typically20:12
pindongahazmat, I figured, but wanted to check anyway... I think if something fails juju should recover gracefully, but that can be worked on later maybe?20:13
hazmatpindonga, lxc creates the containers in the background, it takes a few minutes, if takes more than 10m, there master-customize.log in data-dir should have some clues if you could paste it, there's all dev tool for it that does the same20:13
pindongalet me check that log file20:14
pindongaone sec20:14
hazmatre openstack, it looks like that setup isn't returning publicly routable dns names20:14
hazmatpindonga, you can either associate a static ip/public ip to the bootstrap node or run juju from a location where the public-addresses returned by the api are routable20:15
pindongahazmat, this is canonistack, so yes, the ips are not public routable by default.... (you can get ips by doing euca-describe-instances --ipv4 normally)20:15
pindonganot sure this is something juju should fix or canonistack should fix20:16
pindongahazmat, so where should I look for the master-customize.log file? I got nothing neither in the juju folder nor in the lxc folder (the lxc folder is completely empty)20:16
pindongajust have a config file and an empty rootfs folder20:16
pindongathough I still see a bunch of lxc processes around... maybe I should keep waiting20:18
marcoceppihey SpamapS why do you want to remove md5 and sha1 checks from ch_get_file ?20:20
pindongahazmat, I appreciate your help btw... while we are on it... I know juju is service-oriented and not machine oriented, but can you tell me if this is too far fetched? in order to learn about juju and charms, I decided to write a charm to deploy my development environment... ie, once deployed, a new host will have my full env ready for development (though it doesn't make sense to have more than one host for this 'service' ,)20:21
hazmatpindonga, thats sound reasonable to me.. effectively a charm for a dev env20:25
hazmatie. its taking advantage of automation not orchestration20:26
pindongahazmat, ok, good news, the master-customize.log showed up in the end... it may have been I was just too anxious :)20:26
hazmatpindonga, try status now out of curiosity?20:26
pindongastill pending20:26
pindongathough the log says Container Customization Complete20:26
pindongaand the juju log says Starting container...20:27
hazmatpindonga, what's lxc-ls show?20:28
pindongabesides the ones I already had it lists ricardo-local-0-template and ricardo-local-devel-120:28
hazmatpindonga, on canonistack, you can allocate and associate a public ip address to the bootstrap node20:28
hazmatpindonga, cool, does it list -devel-1 only once.. it lists them twice if its running20:29
pindongano, it lists everything only once20:29
pindongawhich means no container is running20:29
hazmatyup20:29
hazmathmm.. pindonga  is there an lxc-wait in your ps aux | grep lxc output?20:30
pindongayes20:30
pindongalxc-wait -n ricardo-local-devel-1 -s RUNNING20:30
hazmatpindonga, hmm.. so this is bug 91287920:31
_mup_Bug #912879: Machine agent hangs if lxc container start fails <local> <juju:Confirmed> < https://launchpad.net/bugs/912879 >20:31
=== lifeless_ is now known as lifeless
hazmatpindonga, what's unclear though is why the container fails to start.. there's a console log for the container in data-dir/units/$unit-name20:31
hazmatit might have some useful details20:31
pindongalet me check20:32
pindongahazmat, this doesn't sound right20:35
pindonga     lxc-start 1332102305.341 ERROR    lxc_conf - Permission denied - failed to mount 'proc' on '/usr/lib/lxc/root//proc'20:36
pindonga      lxc-start 1332102305.342 ERROR    lxc_conf - failed to setup the mounts for 'ricardo-local-devel-1'20:36
pindonga      lxc-start 1332102305.342 ERROR    lxc_start - failed to setup the container20:36
hazmathmm.. why's it using /usr/lib/lxc.. that paths look strange20:42
* hazmat tries locally20:43
pindongaI have lxc==0.7.5-3ubuntu4020:44
hazmatoh.. that's in the container20:48
hazmatpindonga, can you pastebin the whole console.log20:48
pindongasure, one sec20:48
pindongahazmat, http://pastebin.ubuntu.com/889766/20:50
pindongathis is container.log , yes?20:50
hazmatpindonga, yup, thanks20:51
hazmathmm.. why would the mount fail..20:51
pindongaone extra bit of info that might be useful20:51
pindongaI don't get the cgroups to mount automatically, so I issues cgroups-mount20:52
hazmatpindonga, there should be an upstart job that takes care of that.. cgroup-lite.conf20:53
pindongaand I get this exact same error when running lxc-start directly20:53
pindongaon this specific container20:53
pindongamhh20:53
pindongaactually on other containers too20:53
pindongak, let me reboot to see if this is a temporary glitch with the cgroups20:54
pindongabe right back20:54
hazmatpindonga, it sounds like something to take up with hallyn on #ubuntu-server.. offhand i don't know what the issue is20:54
pindongahazmat, ok, you've been quite helpful anyway, so thx! :)20:54
pindongafor the canonistack env,you said I can associate a public ip to the bootstrap?20:55
pindongalet's say I get a public ip from openstack first20:55
pindongahow do I invoke juju then?20:55
hazmatpindonga, yeah.. its euca-allocate-address, and then euca-associate-address to the bootstrap instance id20:57
hazmatpindonga, then just use juju normally20:58
pindongakk20:58
pindongathx, I'll try it out and let you know20:58
lifelesshazmat: question for you, when you e.g. start a node, does zookeeper get updated *to trigger* the node being started, juju agent kicking off set, or does zookeeper get updates to record that that stuff *has happened*21:17
lifelesshazmat: e.g. which is cause and which is effect21:17
SpamapSlifeless: zookeeper is changed, the agents react to that change21:24
SpamapSlifeless: states are then updated by the agent to reflect the result of the reactions21:26
hazmatlifeless, there's always a state change to initiate an accompanying action21:27
lifelessgreat, thanks for that21:27
SpamapShazmat: re cloud-init / --no-install-recommends , I think you can tell cloud-init what apt options to use... standby21:29
SpamapShm, actually it would seem rather complex21:32
SpamapShazmat: I believe you'd need to actually drop a file in /etc/apt/apt.conf.d to disable the recommends. But then you'd have to turn back around and re-enable it, otherwise charms will fail. :-/21:35
hazmatSpamapS, i'll just rearrange to use cmds for juju pkg install.. what do we need for proposed upgrade support?21:42
SpamapShazmat: be careful with that. :)21:43
SpamapShazmat: in the past we broke badly because DEBIAN_FRONTEND=noninteractive wasn't set.. and we also should make sure it uses --force-confold when calling dpkg. See cloudinit's code for how it calls apt-get21:44
SpamapShazmat: for proposed we need, IMO, the thing I sent to the mailing list. But at a bare minimum, we need to be able to cause juju to point machines at -proposed at any time..21:45
hazmatSpamapS, yeah.. running cluster upgrade support for juju itself would be nice.. but its out of scope for 12.04 at the moment. but getting initial installation, means what exactly.. just installing juju from a proposed ppa?21:47
hazmati'll check the ml21:47
SpamapShazmat: not ppa21:55
SpamapShazmat: you just need to enable the -proposed repository21:55
hazmatSpamapS, interesting.. so the easiest way to do this.. effectively version-pinning and upgrades.. is to always deploy from bzr or assembled tarball21:56
SpamapShazmat: so add another apt source in cloud-init just like the PPA. The bare minimum would be to have another juju-origin called 'proposed' that would enable proposed in cloud-init21:56
SpamapShazmat: then a set of instructions for how to change the origin in zk so that you can test adding newly -proposed units to a running environment.21:56
hazmatSpamapS, i can do the minimal proposed origin step for testing new installation for today, i think your ml post would be a good starting post for a spec on upgrading/running existing environments, i don't know if it can be done before 12.04 though, it is important though, tbd22:03
pindongahi hazmat , so I have some progress in the lxc case22:27
pindongahazmat, but now I see this when the container starts:22:27
pindonga     lxc-start 1332109039.505 NOTICE   lxc_start - '/sbin/init' started with pid '28727'22:27
pindonga      lxc-start 1332109039.505 WARN     lxc_console - console input disabled22:27
pindonga      lxc-start 1332109039.505 WARN     lxc_start - invalid pid for SIGCHLD22:27
pindonga      lxc-start 1332109147.641 INFO     lxc_af_unix - message denied for '1000/1000'22:27
pindonga      lxc-start 1332109147.673 INFO     lxc_af_unix - message denied for '1000/1000'22:27
pindonga      lxc-start 1332109162.493 DEBUG    lxc_commands - peer has disconnected22:27
pindonga      lxc-start 1332109162.555 DEBUG    lxc_commands - peer has disconnected22:27
pindongaI think I heard about a bug like this, but not sure what the workaround was (the bug was that the default user already existed with the same id (1000))22:27
lifelesshttp://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_strengthsAndLimitations - do we hve something in place to clean up the data snapshot/log files?22:45
=== objectiveous_ is now known as objectiveous
_mup_Bug #958872 was filed: bootstrap Install cron job to cleanup zk logs <juju:New> < https://launchpad.net/bugs/958872 >22:49
lifelesshah22:51
lifelessis it possible to have some code run whenever a new node is setup ? (user supplied code I mean)23:12

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!