| thomi | Has anyone heard about an issue deploying the postgres charm on trusty where it fails to read /etc/ssl/private/ssl-cert-snakeoil.key (as in http://paste.ubuntu.com/7739744/ ) ? | 01:55 |
|---|---|---|
| thomi | it doesn't happen for me, but alesage is hitting it, and I know others have had it happen as well... | 01:56 |
| === thumper-afk is now known as thumper | ||
| l1fe | quick question - if you bootstrap juju on maas, it will enlist a single node. shouldn't deploying a new service automatically enlist a new node from what's available from maas? | 03:33 |
| jose | l1fe: it should mark another node as used, it's going to use one node per service unless you specify other stuff | 03:40 |
| l1fe | jose: gotcha...and if you do a juju status with no services deployed | 03:41 |
| l1fe | it would still always just show the juju core node | 03:41 |
| l1fe | and nothing else from maas | 03:41 |
| l1fe | i'm having a problem where it tries to deploy a new service...gets the node id from maas, and then gets stuck in pending | 03:42 |
| l1fe | nothing in the machine-0 logs | 03:42 |
| jose | correct | 03:42 |
| jose | hmm | 03:42 |
| l1fe | there isn't even a /var/log/juju | 03:42 |
| l1fe | on the machine-1 or whatever node it's trying to deploy on | 03:42 |
| jose | can you please try destroying the service and re-deploying? | 03:42 |
| l1fe | sure, i've destroyed the environment a few times and tried to redeploy | 03:43 |
| l1fe | (have to manually go in an remove juju-mongodb) | 03:43 |
| l1fe | i'll try again | 03:43 |
| jose | weird thing | 03:43 |
| l1fe | if i manually provision the boxes | 03:44 |
| l1fe | everything works great | 03:44 |
| jose | maybe someone around who has previous experience with maas will be able to help, but I'm no maas expert unfortunately :( | 03:44 |
| l1fe | i'm on juju 1.19.4 | 03:44 |
| l1fe | :( | 03:44 |
| l1fe | not sure if it matters | 03:47 |
| l1fe | but the last line i get in machine-0.log | 03:47 |
| l1fe | is juju.provider.maas environ.go:304 picked arbitrary tools &{1.19.4-trusty-amd64 https://streams.canonical.com/juju/tools/releases/juju-1.19.4-trusty-amd64.tgz 181fac6e269696eb68f1ff1ff819815af5da1beafd2d963bd5165fb7befdee84 8052214} | 03:47 |
| l1fe | and then nothing | 03:48 |
| jose | have you tried using 1.18? maybe it's a bug in 1.19 and we haven't noticed yet | 03:48 |
| jose | I have no maas environment to test in, otherwise I would be already checking | 03:48 |
| l1fe | i'll try 1.18 :) | 03:48 |
| === vladk|offline is now known as vladk | ||
| l1fe | same thing with 1.18 | 03:58 |
| jose | then it's not a bug on juju | 03:59 |
| jose | or, well, it is, but I don't know how to troubleshoot it | 03:59 |
| l1fe | haha, i'm not sure either...unless there are some logs that i don't know about | 03:59 |
| jose | there *is* a log called all-machines.log in /var/log/juju for the bootstrap node | 03:59 |
| jose | maybe check there? | 04:00 |
| l1fe | yeah, nothing | 04:01 |
| === vladk is now known as vladk|offline | ||
| === urulama-away is now known as urulama | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| noodles775 | jamespage: Hey there. Are you able to fix up a ghost revision on the rabbitmq-server branch? If you try `bzr revno lp:~charmers/charms/trusty/rabbitmq-server/trunk` it'll error (at least does for me), the same error which also stops me from pulling that branch in a deploy. | 08:14 |
| noodles775 | or jam1 might have the bzr foo to fix that too? | 08:15 |
| jamespage | noodles775, I'll look | 08:15 |
| noodles775 | Txs | 08:15 |
| jamespage | noodles775, wtf - I don't even know that that means | 08:18 |
| noodles775 | jamespage: afaik, it means that the branch knows about a revision, but doesn't have the details. I don't know much more than that, other than to fix it in the past on our own branches, we've run: `bzr reconfigure --unstacked` on the branch, but I can't say whether that's OK here (I don't know if the charm branches are stacked for a reason etc.) | 08:20 |
| jamespage | noodles775, I'll have to poke someone with more bzr knowledge that I have - mgz? | 08:22 |
| noodles775 | jamespage: k, thanks. fwiw, I verified that I can reproduce and fix the error on my own branch like this: http://paste.ubuntu.com/7740886/ | 08:24 |
| jamespage | noodles775, it might be an side-effect of the fact that I pushed the same branch to both precise and trusty charm branches | 08:29 |
| jamespage | but not 100% sure | 08:29 |
| === vila_ is now known as vila | ||
| === urulama is now known as urulama-away | ||
| jamespage | gnuoy, can you take a look at - http://bazaar.launchpad.net/~james-page/charm-helpers/network-splits/view/head:/charmhelpers/contrib/openstack/ip.py | 10:28 |
| gnuoy | jamespage, sure | 10:28 |
| jamespage | gnuoy, context is for the various combinations of https, clustered-ness and network configuration a charm might be in | 10:29 |
| jamespage | gnuoy, usage example - http://bazaar.launchpad.net/~james-page/charms/trusty/cinder/network-splits/view/head:/hooks/cinder_hooks.py#L181 | 10:31 |
| gnuoy | jamespage, I thought the charms only supported one vip | 10:33 |
| jamespage | gnuoy, right now that is the case; this adds support for 'vip' to be a list | 10:33 |
| jamespage | supporting multiple VIP's | 10:34 |
| gnuoy | jamespage, this make the last one in the last the preferred one is that what you want ? | 10:34 |
| jamespage | gnuoy, the last one within the subnet is fine | 10:35 |
| jamespage | gnuoy, I'm making the assumption that people won't provide multiple VIPs on the same subnet | 10:36 |
| jamespage | which I think is OK | 10:36 |
| gnuoy | jamespage, lgtm | 10:36 |
| urulama-away | i've got a service (rabbitmq-server) with a status "life: dying" due to 'hook failed: "install"' and this service just cant be removed. how can i shoot it in the head? | 11:03 |
| urulama-away | btw, it's been dying for an hour now :D | 11:04 |
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === liam_ is now known as Guest45753 | ||
| === urulama-away is now known as urulama | ||
| === psivaa is now known as psivaa-lunch | ||
| === CyberJacob is now known as CyberJacob|Away | ||
| === psivaa-lunch is now known as psivaa | ||
| lazyPower | urulama: a failed hook will trap all events until you resolve it | 13:19 |
| lazyPower | you can force deletion of the machine, then remove teh service | 13:19 |
| lazyPower | juju destroy-machine # --force | 13:19 |
| lazyPower | juju destroy-service rabbitmq-server | 13:19 |
| urulama | lazyPower: tnx, did that in the end | 13:19 |
| urulama | lazyPower: what about removing containers ... i've got machine 1 (remote) with two containers ... is it possible to remove containers one by one? or is the same "remove-machine --force" the only option? | 13:24 |
| rick_h__ | urulama: you should be able to remove single containers. kadams was working on doing that in the gui and you might be able to test it if you've got the gui there | 13:26 |
| rick_h__ | urulama: though that requires a couple of extra steps | 13:27 |
| l1fe | not sure if anyone with maas+juju background is on now, but anyone have experience where juju bootstrap works, but when trying to deploy a service (juju-gui), it enlists a node, and gets stuck in pending state? | 13:27 |
| l1fe | if i manually provision the node, however, it works fine | 13:28 |
| l1fe | (which kind of defeats the purpose of having maas) | 13:28 |
| urulama | rick_h__: will try it from gui. was playing to much with "oh, let's destroy the service before the container is brought up and redeploy/destroy a few more times" | 13:29 |
| bac | l1fe: sorry i have no maas experience. will try to find someone who might help. | 13:29 |
| urulama | rick_h__: ended up with no services and a lot of containers in pending state | 13:29 |
| rick_h__ | urulama: heh | 13:29 |
| l1fe | bac: thanks | 13:30 |
| rick_h__ | urulama: ok, sounds like you're having fun :) | 13:30 |
| urulama | rick_h__: indeed :) | 13:30 |
| urulama | rick_h__: is quickstart the same as "create manual and then juju deploy juju-gui --to lxc:0"? | 13:32 |
| bac | rick_h__: aren't you on vacation? | 13:32 |
| urulama | ^^^ | 13:32 |
| bac | urulama: it does both of those things | 13:32 |
| bac | urulama: and it deploys a bundle if you give it one | 13:32 |
| bac | urulama: for LXC it does not --to the gui to 0 as it is not allowed | 13:33 |
| urulama | bac: does the last statement mean that it needs lxc for machine 0 because otherwise it is not allowed (as experience shows)? | 13:36 |
| l1fe | quick question with regards to where to install juju-core to - if i'm in a maas environment, if I install juju-core on a node in the maas cluster, will that node theoretically still be able to be enlisted by juju? | 13:39 |
| l1fe | (that's if I ever get juju and maas to work together in the first place...) | 13:39 |
| rick_h__ | bac: urulama yes, heading out now | 13:43 |
| rick_h__ | bac: urulama was finishing hooking things up :) | 13:43 |
| === scuttlemonkey is now known as scuttle|afk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| Tribaal | sinzui: Hi, I've hit https://bugs.launchpad.net/juju-core/+bug/1337340 this morning, and it smells like a race condition to me ("nothing" changed between it happening and a successful bootstrap) :( | 14:24 |
| _mup_ | Bug #1337340: Juju bootstrap fails because mongodb is unreachable <landscape> <juju-core:New> <https://launchpad.net/bugs/1337340> | 14:24 |
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| sinzui | Tribaal, I saw that same error in a test yesterday. I will look into your bug | 14:41 |
| Tribaal | sinzui: ack, thanks! | 14:45 |
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| l1fe | anyone on with maas+juju exp? having a hard time debugging why nodes are stuck in pending state | 15:52 |
| === vladk is now known as vladk|offline | ||
| === roadmr is now known as roadmr_afk | ||
| lazyPower | l1fe: i've run into this before | 16:19 |
| l1fe | lazyPower: were you ever able to resolve this? | 16:20 |
| lazyPower | l1fe: do you have any other enlistments running during the pxe boot phase | 16:20 |
| l1fe | ah, this isn't during the maas pxe boot phase | 16:21 |
| l1fe | this is while deploying charms in juju | 16:22 |
| l1fe | http://askubuntu.com/questions/491306/jujumaas-when-deploying-charms-juju-gui-new-machines-stuck-in-pending-sta | 16:22 |
| lazyPower | l1fe: what you're showing me is the node itself did not finish its enlistment phase when juju requests the machine | 16:23 |
| lazyPower | l1fe: its helpful to know where that's bailing out. These are NUC's right? are you going straight hardware witht he nUCS or are you using VM's on the NUCS? | 16:23 |
| l1fe | straight hardware | 16:24 |
| l1fe | the nodes are properly "ready" in maas, and after running juju deploy, they go, correctly, to allocated to root | 16:24 |
| l1fe | after that, though, not sure what the heck is going on since no logs are created for juju on the new node | 16:24 |
| l1fe | and nothing is logged on the already bootstrapped machine | 16:25 |
| l1fe | if i manually provision the machine, though, everything is fine | 16:25 |
| l1fe | but, that kind of defeats the purpose :) | 16:25 |
| lazyPower | are you assigning the nodes before you run juju deploy? | 16:27 |
| l1fe | no, only thing i did was bootstrap the one machine | 16:28 |
| l1fe | based on the documentation, i kind of assumed juju+maas would handle everything else | 16:28 |
| lazyPower | it should. | 16:28 |
| lazyPower | so, do you see where the machien states pending in your log output in the askubuntu question? | 16:29 |
| l1fe | yup | 16:29 |
| lazyPower | that tells me something is happening during the enlistment / configuration phase thats gone awry. there is a -v flag you can pass to juju. can you run a deployment with the -v flag and pastebin the output? | 16:29 |
| l1fe | not sure how to get it out of "pending" | 16:29 |
| l1fe | ok, will do | 16:29 |
| lazyPower | i want to isolate if its a juju issue, or a maas issue | 16:30 |
| lazyPower | in the instances i've run into this problem - had multiple deployments running and the pxe boot never finishes loading the image | 16:30 |
| lazyPower | destroying the unit and re-requesting usually solves teh problem. | 16:30 |
| lazyPower | it appears to be something related to my TFTP server on my host - i havent dove real deep into it - so i'm crossing fingers this isn't teh same problem :) because its uncharted territory for me if it is. | 16:31 |
| l1fe | http://pastebin.com/BM802wd5 | 16:32 |
| l1fe | it didn't look like the -v changed much in terms of output | 16:32 |
| l1fe | whoops | 16:32 |
| l1fe | http://pastebin.com/YD6i1m3d | 16:33 |
| l1fe | sorry about that | 16:33 |
| lazyPower | when you ssh into dynuc001.maas - is there anything in /var/log/juju? | 16:33 |
| l1fe | nope | 16:34 |
| lazyPower | ok so we aren't connecting then - that should be one of teh first things it does is remote in and setup teh scaffolding | 16:34 |
| l1fe | if i go into auth.log on dynuc001.maas i don't even see that they try to login | 16:34 |
| === hatch__ is now known as hatch | ||
| l1fe | i wonder if it has anything to do with the bug where during the bootstrap process, everything bugs out unless i first go onto the node to manually create a /var/lib/juju/nonce.txt | 16:39 |
| lazyPower | hmm | 16:39 |
| lazyPower | that i don't know. my exposure with maas is limited to my VMAAS setup. | 16:40 |
| lazyPower | i dont have enough hardware to do a proper maas | 16:40 |
| l1fe | not even sure what other logs i can look at | 16:41 |
| l1fe | since at this point, it's like all output just drops into the void | 16:42 |
| jrwren | why does the mongodb for localdb use so much space? 1.2G on one host, 6.5G on another | 16:45 |
| lazyPower | jrwren: depends on how many blocks it allocates to perform the storage. | 16:46 |
| lazyPower | jrwren: typically, mongodb will allocate ~ 2gb of storage space to get started, after that its exponential on growth - since its BSON its creating a blob on disk. | 16:46 |
| lazyPower | and it doesnt take what it needs, it takes more than it needs for rapid growth so it stays snappy. | 16:47 |
| lazyPower | l1fe: it makes me wonder if juju can access the node.. have you tried ssh'ing intot eh nodes ubuntu user using the ~/.juju/ssh/id_rsa key? | 16:48 |
| jrwren | how much does a tiny juju local need? | 16:48 |
| l1fe | yup, juju can ssh in | 16:48 |
| lazyPower | jrwren: that i don't know. I'd ping in #juju-dev as they have the document collection specifics | 16:48 |
| l1fe | lazyPower: i think maybe i have the answer - quick question re: lifecycle | 16:49 |
| l1fe | when i do a juju deploy over maas, does it signal for the node to restart and then go into pxe boot to do its stuff | 16:49 |
| lazyPower | i'm going to start from teh top and try to split apart concerns by starting the line with who's doing the work | 16:54 |
| lazyPower | juju => Signals to mass it needs a machine | 16:54 |
| lazyPower | mass => Powers on the machine and runs enlistment. The node reboots and completes enlistment by loading all of the prefs and ssh keys from teh MAAS server and returns a READY signal to teh juju bootstrap node (or client in terms of a bootstrap) | 16:54 |
| lazyPower | juju => Connects to the fresly provisioned server and loads scaffolding, packages, and the juju-client services | 16:55 |
| lazyPower | deployment begins. | 16:55 |
| lazyPower | i may have missed something in there, but thats my understanding of the complete lifecycle during a requested deployment using juju/maas | 16:55 |
| l1fe | ok, so what happens if in maas, all the nodes are already enlisted and provisioned with something | 16:56 |
| lazyPower | i haven't actually over-allocated - so i would *think* it would return an error since maas is aware of how many nodes it has | 16:56 |
| lazyPower | but it might sit in pending | 16:56 |
| lazyPower | let me try to spin up 20 mongodb nodes hang on | 16:57 |
| l1fe | i guess i'm not sure how the lifecycle works if juju signals to maas to run enlistment, wouldn't the nodes have to already be enlisted for it to signal that it will deploy a service to it? | 16:57 |
| l1fe | so when i do juju status, it comes back and says, i have a machine, it is called dynuc001.maas | 16:58 |
| l1fe | and that it allocated that machine for the service juju-gui | 16:58 |
| lazyPower | l1fe: well its not looking good for juju erroring out on not having enough nodes in its pool | 16:58 |
| lazyPower | its still pending though - there's time for it to fail out | 16:59 |
| l1fe | HA | 17:00 |
| l1fe | got it to work | 17:00 |
| l1fe | son of a... | 17:00 |
| lazyPower | l1fe: thats great news - what was the fix/workaround? and wrt what happens | 17:01 |
| lazyPower | they go forever pending. | 17:01 |
| l1fe | alright, so basically, when you kept on saying that juju will go into pxe boot | 17:01 |
| l1fe | something finally clicked | 17:01 |
| l1fe | in order for my to do juju bootstrap properly on my boxes | 17:02 |
| l1fe | i had to follow: https://bugs.launchpad.net/juju-core/+bug/1314682 | 17:02 |
| _mup_ | Bug #1314682: juju bootstrap fails <bootstrap> <juju> <maas-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1314682> | 17:02 |
| l1fe | and create my own nonce.txt | 17:02 |
| l1fe | i think this is because the power controls for the NUCs in maas doesn't work as expected | 17:02 |
| l1fe | and it never restarts the node properly, to allow it to pxe boot and provision everything properly | 17:03 |
| l1fe | so it never really sunk in, that that was the lifecycle | 17:03 |
| l1fe | i just figured it would ssh into the machine, and run the appropriate commands | 17:03 |
| l1fe | so when i did juju deploy, i figured it would do the same thing | 17:03 |
| l1fe | instead, based on what you said | 17:03 |
| l1fe | it's expecting to go into pxe to have everything configured there | 17:03 |
| l1fe | through maas (which makes sense, since why else would you have maas) | 17:04 |
| l1fe | so i went onto dynuc001.maas and forced a reboot | 17:04 |
| l1fe | let pxe do it's thing | 17:04 |
| l1fe | and the status finally came back as started | 17:04 |
| l1fe | hope that made sense...on the bright side, because of these problems, i come to understand the whole process since i have to manually intervene where normal power management would have worked haha | 17:06 |
| === roadmr_afk is now known as roadmr | ||
| === psivaa is now known as psivaa-afk | ||
| jrwren | it would be cool if local provider had an option to run machine 0 in an lxc | 17:32 |
| lazyPower | l1fe: ah, ok :) | 17:47 |
| lazyPower | jrwren: thats on the books somewhere. to treat teh bootstrap node as its own LXC container instead of hulk-smashing it on the HOST | 17:48 |
| jrwren | lazyPower: excellent! | 17:48 |
| === scuttle|afk is now known as scuttlemonkey | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === vladk|offline is now known as vladk | ||
| === scuttlemonkey is now known as scuttle|afk | ||
| === scuttle|afk is now known as scuttlemonkey | ||
| === vladk is now known as vladk|offline | ||
| === vladk|offline is now known as vladk | ||
| === vladk is now known as vladk|offline | ||
| === alexisb is now known as alexisb_vac | ||
| niedbalski | thumper, you there? | 22:01 |
| thumper | niedbalski: yep | 22:02 |
| niedbalski | thumper, ok, i'll add a mock for that. not sure if this specific package has any mock library imported | 22:03 |
| thumper | niedbalski: I'm just looking in the juju testing library | 22:03 |
| thumper | that may help | 22:03 |
| thumper | niedbalski: if you use "github.com/juju/testing" there is a PatchExecutable there | 22:05 |
| thumper | in cmd.go | 22:06 |
| niedbalski | thumper, cool, i'll use that then. thanks. | 22:09 |
| thumper | niedbalski: thanks for the patch | 22:09 |
| === CyberJacob is now known as CyberJacob|Away | ||
| themonk | lazyPower, hi | 22:59 |
| themonk | how to use juju-gui charm locally? | 23:00 |
| themonk | like if i deploy juju-gui in my lxc container then how to load local charms using juju-gui? | 23:02 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!