| === axw_ is now known as axw | ||
| aisrael | Taking precise/vdbench | 00:00 |
|---|---|---|
| mwhudson | i don't suppose anyone has a juju charm for running a docker registry? :) | 00:02 |
| jose | ok, have to go print some Juju posters, be back in a while! | 00:04 |
| === CyberJacob is now known as CyberJacob|Away | ||
| mbruzek | working on cs:precise/unattended-upgrades | 00:07 |
| aisrael | taking precise/transcode | 00:12 |
| aisrael | tvansteenburgh: http://reports.vapour.ws/charm-tests/charm-bundle-test-641-results/charm/charm-testing-lxc/0 | 00:18 |
| aisrael | taking precise/tracks | 00:18 |
| lazyPower-sprint | cs:precise/qemu-cloud | 00:19 |
| mwhudson | ok, failure mode i haven't seen before | 00:41 |
| mwhudson | a machine (using the manual provider) thinks it is provisioned, but it isn't showing up in juju status | 00:42 |
| mwhudson | ah http://askubuntu.com/questions/433842/how-to-resolve-error-machine-is-already-provisioned-in-manual-provision-set-up | 00:45 |
| cory_fu | This looks really cool: https://www.kickstarter.com/projects/thoughtstem/codespells-express-yourself-with-magic | 00:46 |
| lazyPower-sprint | mwhudson: looking now hang on | 01:01 |
| lazyPower-sprint | mwhudson: did that answer clear you up on gettin the machine added to your manual environment? | 01:02 |
| mwhudson | lazyPower-sprint: no i just gave up and started again :) | 01:02 |
| mwhudson | (this is all mostly automated) | 01:02 |
| lazyPower-sprint | Ok. I've been using digital ocean in phaux manual environment and haven't seen that - but the answer looks solid. | 01:03 |
| lazyPower-sprint | jose: good dev-credit if you've got the bandwidth: https://bugs.launchpad.net/charms/+source/python-moinmoin/+bug/1367532 | 02:21 |
| mup | Bug #1367532: tests need refactoring <audit> <python-moinmoin (Juju Charms Collection):New> <https://launchpad.net/bugs/1367532> | 02:21 |
| lazyPower-sprint | fuzzy: please keep juju help requests isolated to this channel so the community can benefit from the help you recieve, and allow others who may know the answer an opportunity to answer. <3 | 02:29 |
| fuzzy | 2014-09-10 02:38:14 INFO juju.state.api apiclient.go:242 dialing "wss://localhost:17070/" | 02:38 |
| fuzzy | 2014-09-10 02:38:14 INFO juju.state.api apiclient.go:250 error dialing "wss://localhost:17070/": websocket.Dial wss://localhost:17070/: dial tcp 127.0.0.1:17070: connection refused | 02:38 |
| fuzzy | nothing is running on 17070, all I did was reboot it. Restarting mongo doesn't do anything for me. Now juju status just hangs | 02:39 |
| jose | lazyPower-sprint: I may be able to take a look at them later, triaging and shortening the queue atm | 02:41 |
| lazyPower-sprint | fuzzy: this is related to the mongodb server, hang on and let me tap someone in that has a better idea about how to triage the issue | 02:43 |
| lazyPower-sprint | jose: no rush - its not a hot-button charm but would be good to flex your amulet muscle | 02:43 |
| lazyPower-sprint | fuzzy: we need you to pastebin the ~/.juju/environments.yaml for your manual provider sans credentials | 02:45 |
| fuzzy | np one moment | 02:45 |
| fuzzy | I was generating a pastebin for you guys already | 02:45 |
| fuzzy | http://hastebin.com/wefavoqate.avrasm there is the log file example let me get you that yaml file | 02:47 |
| marcoceppi | fuzzy: what is juju@juju, is it the machine you're running juju commands from and what you're trying to bootstrap? | 02:48 |
| marcoceppi | slash did bootstrap? | 02:49 |
| fuzzy | http://hastebin.com/lutenazuwu.cs | 02:50 |
| fuzzy | marcoceppi: juju is my bootstrapping host and juju is the user that does it all | 02:50 |
| fuzzy | I don't run this from a laptop, I run it from a host | 02:50 |
| marcoceppi | fuzzy: so, that's going to be the first likely problem. To confirm. You're running the juju commands from the same host you're trying to bootstrap | 02:51 |
| fuzzy | no | 02:51 |
| fuzzy | i've already bootstrapped like 4 hosts | 02:51 |
| fuzzy | I'm sorry if i'm getting your terminology wrong, but instead of fight linux mint & juju I use a server to springboard from | 02:52 |
| lazyPower-sprint | fuzzy: to be clear, in juju jargon, bootstrapping is with reference to the api-controller node. as in node 0 | 02:52 |
| marcoceppi | fuzzy: right, is that server juju.int.ziphub.com ? | 02:52 |
| fuzzy | yes | 02:52 |
| fuzzy | that is correct | 02:52 |
| marcoceppi | fuzzy: okay that's going to be painful | 02:52 |
| fuzzy | juju.int.ziphub.com is machine 0 and only runs the juju-gui | 02:52 |
| marcoceppi | juju really wasn't meant to be done in such a way. It may be better to simply spin the Juju Vagrant image and drive that from your mint desktop | 02:53 |
| marcoceppi | as it'll give you juju and CLI | 02:53 |
| marcoceppi | but | 02:53 |
| marcoceppi | moving past that | 02:53 |
| marcoceppi | run the following and let us know the output | 02:53 |
| marcoceppi | fuzzy: sudo initctl list | grep juju | 02:53 |
| marcoceppi | so the bootstrap-host is your bootstrap node, it does all your orchestration | 02:53 |
| marcoceppi | it's designed to survive reboots, but ocassionally doesn't for whaatever reason | 02:54 |
| fuzzy | jujud-unit-juju-gui-0 start/running, process 2003 | 02:54 |
| fuzzy | juju-db start/running, process 2019 | 02:54 |
| fuzzy | jujud-machine-0 start/running, process 2029 | 02:54 |
| marcoceppi | so, it has survived reboots | 02:54 |
| marcoceppi | fuzzy: can you run `juju status --debug` ? | 02:56 |
| fuzzy | sure | 02:56 |
| fuzzy | http://hastebin.com/owilogutej.avrasm | 02:57 |
| fuzzy | there is nothing running on 17070 when I ask netstat | 02:57 |
| fuzzy | http://hastebin.com/apezokuqor.hs | 02:58 |
| marcoceppi | fuzzy: try `sudo restart juju-db` then `sudo restart jujud-machine-0` | 02:58 |
| marcoceppi | fuzzy: also, can you include `/var/log/upstart/juju-db.log` | 02:58 |
| marcoceppi | juju-db is lying by saying it's up but it doesn't appear to be up | 02:59 |
| fuzzy | np gimmie one moment | 02:59 |
| marcoceppi | fuzzy: sure, sure | 02:59 |
| fuzzy | http://hastebin.com/luyaqajixa.coffee | 03:01 |
| fuzzy | macroceppi: | 03:01 |
| fuzzy | here is my *juju* process list http://hastebin.com/fiqayoyage.hs | 03:02 |
| lazyPower-sprint | fuzzy: after the restart, still spammig that the state server cannot connect to mongo? | 03:04 |
| fuzzy | http://hastebin.com/hixocukada.avrasm | 03:07 |
| fuzzy | yes | 03:07 |
| lazyPower-sprint | fuzzy: ok the order shouldn't stop, but my thoughts are lets validate that the DB brings up proper, and then lets bring up the api server, and monitor them | 03:17 |
| lazyPower-sprint | tail your syslog in a terminal, and service stop juju-db && juju-machine-0 && juju-unit-juju-gui-0 | 03:18 |
| lazyPower-sprint | confirm once you've got everything halted on the machine | 03:18 |
| lazyPower-sprint | if i were to guess, i'd say this is due to a stale lock causing mongodb to barf, and since the serviceis on a recycle, its not actually up but thinks its up | 03:18 |
| fuzzy | Gimmie one minute guys I gotta eat dinner | 03:29 |
| fuzzy | lazyPower-sprint: juju-machine-0 and juju-unit-juju-gui-0 are not recognized as a service | 03:40 |
| lazyPower-sprint | sorry, jujud-machine-0 | 03:41 |
| lazyPower-sprint | and jujud-unit-juju-gui-0 | 03:41 |
| fuzzy | lazyPower-sprint: ok that worked, syslog is quiet | 03:42 |
| lazyPower-sprint | ok, check and make sure there isn't a stale mongodb.lock file floating around | 03:43 |
| fuzzy | root 1839 1 0 02:36 ? 00:00:00 /usr/bin/python /usr/local/bin/runserver.py --logging=info --guiroot=/var/lib/juju-gui/juju-gui/build-prod --sslpath=/etc/ssl/juju-gui --charmworldurl=https://manage.jujucharms.com/ --apiurl=wss://192.168.201.155:17070 --apiversion=go | 03:43 |
| fuzzy | that is still running if i do ps -aef | grep juju | 03:43 |
| fuzzy | mongo is not running | 03:43 |
| fuzzy | looking for lock file | 03:43 |
| lazyPower-sprint | /var/lib/juju/db | 03:43 |
| fuzzy | find / | grep mongodb.lock returns nothing | 03:44 |
| fuzzy | find / | grep mongo.lock returns nothing | 03:44 |
| fuzzy | oh | 03:44 |
| fuzzy | it's mongod.lock | 03:44 |
| fuzzy | and i found it | 03:44 |
| fuzzy | it's got a stale pid in it | 03:45 |
| fuzzy | lamont: | 03:45 |
| fuzzy | lazyPower-sprint: | 03:45 |
| lazyPower-sprint | so, how familiar are you with removing a mongod.lock and dealing with mongo afterwords? | 03:45 |
| fuzzy | about 0 out of 0 | 03:45 |
| lazyPower-sprint | you can cowboy the removal, and cross fingers that everything goes welll - which it does about 80% of hte time | 03:45 |
| fuzzy | ok so what happens if the shotgun approach doesn't work? | 03:46 |
| lazyPower-sprint | 20% of the time, there are further issues that crop up, and are a byproduct of mongod not shutting down properly and leaving the database in an inconsistent state | 03:46 |
| lazyPower-sprint | hard to say, depends on whats gone wrong. | 03:46 |
| fuzzy | Alight so in the future make sure backups are in place so roll backs can happen | 03:47 |
| fuzzy | I dunno it's still not going | 03:51 |
| fuzzy | i'm just going to rebuild everything and try again | 03:52 |
| fuzzy | It should only take me about an hour | 03:52 |
| fuzzy | lazyPower-sprint: | 03:52 |
| lazyPower-sprint | fuzzy: shouldn't have to do that | 03:53 |
| lazyPower-sprint | did you remove the mongod.llck and attempt to restart? | 03:53 |
| lazyPower-sprint | if it has an issue restarting, it will barf the output to stdout/logs | 03:53 |
| fuzzy | yea | 03:54 |
| fuzzy | I removed mongod.lock | 03:54 |
| fuzzy | and rebooted the machine so the startup process would start everything in the correct order | 03:54 |
| fuzzy | and juju status is still failing trying to connect to 17070 | 03:54 |
| lazyPower-sprint | fuzzy: i wouldn't have rebooted the server - port 17070 is the state server api. and the reason you cannot connnect to that is because it cannot connect to juju-db | 04:00 |
| lazyPower-sprint | restarting the db would have spit back any problems on STDOUT / syslog - which would have given us a next step to start investigation | 04:01 |
| fuzzy | lazyPower-sprint: I know you probably are too busy for this, but if you would like the keys to the castle before I nuke it, just hit me in a more private method | 04:01 |
| fuzzy | I'm also dealing with a head cold | 04:01 |
| fuzzy | brb | 04:02 |
| lazyPower-sprint | if you're adamant on blowing it away and restarting thats acceptable - but the root cause was the reboot, and apparently a database lock thats preventing mongo from coming back up correctly | 04:02 |
| lazyPower-sprint | the fact its not stating that in log output is troublesome to me. MongoDB will fail to start when that mongod.lock file is present. | 04:02 |
| lazyPower-sprint | fuzzy: http://stackoverflow.com/questions/13700261/mongodb-wont-start-after-server-crash take a look at this, about the durability and recovery method | 04:03 |
| === uru_ is now known as urulama | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === urulama-afk is now known as urulama | ||
| === CyberJacob is now known as CyberJacob|Away | ||
| fuzzy | lazyPower-sprint: is MAAS a better solution than manual provisioning? | 08:42 |
| === gsamfira1 is now known as gsamfira | ||
| Odd_Bloke | I'm using the JuJu Vagrant image and can't get either cs:precise/mysql-48 or cs:trusty/mysql-4 to come up. | 11:18 |
| Odd_Bloke | They both report: 'hook failed: "start"' | 11:19 |
| Odd_Bloke | Not really sure how to debug the problem. | 11:19 |
| Odd_Bloke | Ah, have managed to dig in a bit. | 11:22 |
| Odd_Bloke | Looks like it might be a memory issue. | 11:22 |
| Odd_Bloke | Will restart the Vagrant instance with more RAM and see if that helps. | 11:22 |
| === gsamfira1 is now known as gsamfira | ||
| Odd_Bloke | Alright, all happy now. | 12:06 |
| * rick_h_ heads back home from coffee shop | 13:01 | |
| === scuttle|afk is now known as scuttlemonkey | ||
| jcastro | asanjar, https://bugs.launchpad.net/charms/+bug/842202 | 14:52 |
| mup | Bug #842202: Charm needed: Accumulo <Juju Charms Collection:New for fgimenez> <https://launchpad.net/bugs/842202> | 14:52 |
| === hatch__ is now known as hatch | ||
| === tvansteenburgh1 is now known as tvansteenburgh | ||
| corntoegoblin | how do we verify that maas-dns is correctly caching the maas dhcp ips? | 16:11 |
| === julianwa_ is now known as julianwa | ||
| jcastro | jose, around? | 17:28 |
| jose | jcastro: yeah! what's up? | 17:28 |
| jcastro | Are you working on those 2 merges you locked now? | 17:28 |
| jcastro | because if not i can do them right now | 17:28 |
| aisrael | kwmonroe: https://github.com/dergachev/vagrant-vbox-snapshot | 17:29 |
| jose | jcastro: yeah, but I'm waiting for authorization to push | 17:29 |
| jose | remember, I just joined ~charmers | 17:29 |
| jose | if you auth me, I'll merge | 17:29 |
| jcastro | marco says you should post a comment on each MP | 17:30 |
| === roadmr is now known as roadmr_afk | ||
| jcastro | then one of them will post a response with the ok. | 17:31 |
| mbruzek | jose, comment on the review, with a note at the bottom that once another charmer looks at it, it will be merged or not. | 17:31 |
| mbruzek | jose, so basically don't wait to comment on the mp/bug | 17:31 |
| jose | ok | 17:31 |
| mbruzek | comment, and then send marco, chuck, myself a link to the comment and we can take a look. | 17:32 |
| jose | hehe, looks like we've got another jose in here :P | 17:52 |
| jose | marcoceppi, lazyPower-sprint, mbruzek: https://code.launchpad.net/~lazypower/charms/precise/python-moinmoin/fix_proof/+merge/234041, https://code.launchpad.net/~lazypower/charms/precise/qemu-cloud/fix_proof/+merge/234040, https://code.launchpad.net/~jorge/charms/precise/vsftpd/add-icon/+merge/234034 | 17:53 |
| mbruzek | jose, I will take a look | 17:53 |
| jose | thanks | 17:54 |
| mbruzek | jose auth for moinmoin | 17:57 |
| jose | ack, pushed | 17:58 |
| mbruzek | jose auth qemu-cloud | 17:59 |
| jose | ack, pushed | 18:00 |
| mbruzek | jose auth for vsftpd | 18:04 |
| jose | ack, pushed | 18:06 |
| jose | thanks mbruzek! | 18:06 |
| * arosales taking a look at https://code.launchpad.net/~evarlast/charms/trusty/mongodb/no-install-recommends/+merge/230978 | 18:30 | |
| lazyPower-sprint | arosales: make sure that applies cleanly, mongodb hhas been updated since that MP | 18:39 |
| arosales | lazyPower-sprint, ya I was just showing marcoceppi that | 18:40 |
| arosales | the current merge as it stands isn't a clean patch | 18:40 |
| arosales | so I am going to try to clean up and the ntest | 18:40 |
| lazyPower-sprint | asanjar: https://code.launchpad.net/~lazypower/charms/trusty/hdp-hadoop/pass_proof/+merge/234177 | 18:44 |
| mbruzek | wordpress fails on power https://bugs.launchpad.net/charms/+source/wordpress/+bug/1365585 | 18:51 |
| mup | Bug #1365585: wordpress db relation fails in trusty <audit> <ppc64le> <wordpress (Juju Charms Collection):New> <https://launchpad.net/bugs/1365585> | 18:51 |
| arosales | mbruzek, thanks | 18:51 |
| === roadmr_afk is now known as roadmr | ||
| lazyPower-sprint | hey tvansteenburgh1 is bundletester pip installable or still blocked with package deps in the way?L | 19:14 |
| tvansteenburgh1 | lazyPower-sprint: you can pip install it | 19:14 |
| lazyPower-sprint | oooo nvm it looks like its there | 19:14 |
| === tvansteenburgh1 is now known as tvansteenburgh | ||
| lazyPower-sprint | tvansteenburgh: does it expect every project to use venv? | 19:16 |
| lazyPower-sprint | http://paste.ubuntu.com/8312002 | 19:18 |
| aisrael | utlemming: I think I found a bug in the -vagrant-juju cloud images, with the lxc network postrouting. Should I talk to you about that, or someone else? | 19:20 |
| utlemming | aisrael: me is fine....what are you seeing? | 19:21 |
| aisrael | utlemming: if two lxc containers try talking to each other, the source ip is rewritten to 10.0.3.1, instead of the machine's private ip | 19:22 |
| aisrael | Like the NAT POSTROUTING has MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 | 19:22 |
| utlemming | aisrael: hrm, that is ugly. I'm open to a patch :) | 19:24 |
| aisrael | Happy to hunt that down. Is lp:jujuquickimgs/build-trunk the correct place to start? | 19:25 |
| utlemming | aisrael: lp:jujuredirector would be the place to play | 19:25 |
| aisrael | Excellent, thanks! | 19:26 |
| === roadmr is now known as roadmr_afk | ||
| === roadmr_afk is now known as roadmr | ||
| fuzzy | lazyPower-sprint: would maas be a better option than manual provisioning? | 20:06 |
| lazyPower-sprint | fuzzy: it would be better and provide some automation yes - however - MAAS would allow you to orchestrate commodity hardware in *your* racks, or VM's on your hardware. | 20:07 |
| lazyPower-sprint | its not going to provide enlistment/provisioning from linode | 20:07 |
| fuzzy | I've already got two emails into two dc's. I did the math, I basically need 30 instances to cover my deployment | 20:07 |
| fuzzy | For the $ I spend @ linode I can get full boxes anyway I want from datashack | 20:08 |
| fuzzy | for about 3x the resources | 20:08 |
| fuzzy | I have them working up something for me from what primitive knowledge I have of MAAS | 20:08 |
| mbruzek | marcoceppi, https://bugs.launchpad.net/charms/+bug/807784 | 20:09 |
| mup | Bug #807784: Charm needed: etherpad-lite <Juju Charms Collection:Fix Released by james-page> <https://launchpad.net/bugs/807784> | 20:09 |
| fuzzy | Is there any catches to MAAS and juju? | 20:09 |
| mbruzek | marcoceppi, nevermind | 20:10 |
| === CyberJacob|Away is now known as CyberJacob | ||
| lazyPower-sprint | tvansteenburgh: i'm not sure how/when charmworldlib is updated, so i set the bug status to fix-committed. If its an instant release cycle can you update that? https://bugs.launchpad.net/charmworldlib/+bug/1363136 | 21:05 |
| mup | Bug #1363136: Need default request timeout <charmworldlib:Fix Committed by tvansteenburgh> <https://launchpad.net/bugs/1363136> | 21:05 |
| tvansteenburgh | lazyPower-sprint: thanks, i think Fix Committed is correct | 21:06 |
| === scuttlemonkey is now known as scuttle|afk | ||
| lazyPower-sprint | jamespage: is the RabbitMQ merge applicable to trusty as well? the MP is only targeted at precise - without any tests i'm trying to validate this manually. | 21:25 |
| lazyPower-sprint | jjust curious if i need to promote this against the trusty build as well | 21:25 |
| lazyPower-sprint | for reference: https://code.launchpad.net/~cprov/charms/precise/rabbitmq-server/rabbit-admin/+merge/233205 | 21:26 |
| jose | arosales: sorry! I didn't see your lock and n'acked the mongodb review you were doing :( | 21:43 |
| arosales | did /me forget to lock? | 21:44 |
| jose | no, I forgot to lock | 21:44 |
| arosales | jose, ok no worries thanks for reviewing | 21:50 |
| arosales | jose, I am working on an MP that will work, testing now, so I'll have a follow on MP and will comment on Jay's | 21:50 |
| jose | awesome, then! | 21:52 |
| jose | marcoceppi, lazyPower-sprint, mbruzek: auth to push https://code.launchpad.net/~tvansteenburgh/charms/precise/membase/fix-proof/+merge/234214 | 22:20 |
| jose | is anyone around having problems with EC2? | 22:25 |
| === cmagina_ is now known as cmagina | ||
| corntoegoblin | i found the maas dns zone files in /etc/bind/ but the hostnames aren't being added/ | 22:45 |
| corntoegoblin | disregard gentlemen | 22:45 |
| corntoegoblin | time to clock out | 22:45 |
| === CyberJacob is now known as CyberJacob|Away | ||
| === scuttle|afk is now known as scuttlemonkey | ||
| === cory_fu2 is now known as cory_fu | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!