| === balloons is now known as Guest6771 | ||
| === liam_ is now known as Guest12049 | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === kadams54 is now known as kadams54-away | ||
| === kadams54 is now known as kadams54-away | ||
| johnmce | Can someone please help with a problem I'm having with Juju. I've asked about this problem here several times in the past 30 hours, but had no response. | 18:20 |
|---|---|---|
| johnmce | There is clearly a bug in Juju that causes it to deploy old versions of charms when deploying Keystone to a MAAS LXC container | 18:21 |
| johnmce | I've used bzr to check the latest Keystone charm to a local repo. This charm clearly supports Juno, as can be seen in the source code. | 18:22 |
| johnmce | However, when it is deployed to a newly instantiated MAAS/LXC container, the charm deployed is an earlier revision that does not support Juno. | 18:24 |
| johnmce | Can anyone please help? I'm rapidly losing faith in Juju, as there is no documentation or community support that gives any clue as to the possible cause. | 18:25 |
| jrwren | jose: how are you deploying it? | 18:27 |
| jrwren | err, johnmce | 18:28 |
| jrwren | sorry jose, misdir | 18:28 |
| jose | no prob :) | 18:28 |
| jrwren | johnmce: can you pastebin the commands you have run or maybe ask on askubuntu to completely describe the problem ? | 18:28 |
| jose | johnmce: probably not getting a response since it's the weekend and most people are hanging out doing some other stuff. but I'll be glad to give you a hand with this! as jrwren mentioned, if you could pastebin the command or the bundle you're using that'd give us a good first clue on what's going on | 18:29 |
| johnmce | jrwren: Thanks for the response. I'll pastebin some details info shortly. | 18:29 |
| jrwren | johnmce: if you have commands which you've run, I can try to reproduce. | 18:33 |
| johnmce | jrwren: This is a complete attempted install I just did: http://pastebin.com/9Tjae302 | 18:40 |
| johnmce | jrwren: As you can see this revision of the charm is called "local:trusty/keystone-243". The same revision number that hits the LXC machine, but with inconsistent content. | 18:43 |
| jrwren | johnmce: whoa, how does that work :) | 18:43 |
| jrwren | johnmce: are any of /opt/charms or /opt/charms/trusty symlinked directories? are they linked to where you expect? | 18:44 |
| jrwren | johnmce: can you paste that bundle? | 18:45 |
| johnmce | jrwren: None are symlinked. Unless of course "bzr branch" does some weird symlinking that I didn't expect | 18:46 |
| jrwren | johnmce: no, bzr branch won't do taht. | 18:46 |
| jrwren | johnmce: what is the value of openstack-origin in config? | 18:46 |
| johnmce | jrwren: Got to go for about 10 minutes. Where will I find the bundle in a suitable format for pasting? | 18:47 |
| jrwren | johnmce: maybe you didn't use a bundle. I see a bundle mentioned on line 112 of that pastebin | 18:48 |
| jose | johnmce: you're using a local charm, which means you're not deploying from the charm store (cc. jrwren ) | 18:48 |
| jose | johnmce: that means, if you have branched and old revision which doesn't have the changes that are on the store yet, then you will not get them | 18:48 |
| jose | johnmce: try going into /opt/charms/trusty/keystone and doing bzr pull | 18:49 |
| jose | otoh, I'm seeing an error about an invalid cloud repo | 18:50 |
| jose | oh, oh, wait! charm helpers! | 18:50 |
| jose | johnmce: may I see what's the output of ls -lh /opt/charms/trusty/keystone, please? | 18:50 |
| jose | also, try deploying with just the following command: | 18:53 |
| jose | juju deploy --to lxc:1 --config=~/openstack.cfg keystone | 18:53 |
| jrwren | seems like config isn't getting set. openstack-origin is defaulting | 18:57 |
| === kadams54 is now known as kadams54-away | ||
| johnmce | jose: http://pastebin.com/8EpmNu9A | 19:00 |
| johnmce | jrwren: I included the relevant part of the config file in my original pastebin. This was the origin line for keystone: openstack-origin: 'cloud:trusty-juno' | 19:02 |
| jrwren | johnmce: right. I think you diagnosis is perfect. I've no idea why the juno definitions aren't in the charm on the deployed lxc. | 19:04 |
| johnmce | jrwren: I've got single quotes around cloud:trusty-juno. Is that the correct syntax? | 19:05 |
| jrwren | johnmce: its yaml. that should be fine with or without | 19:05 |
| johnmce | jrwren: Is there any more information I can provide to help diagnose this? | 19:10 |
| jose | johnmce: did you try deploying with that last command I gave you? | 19:12 |
| johnmce | jose: Sorry, not yet. I'll try it now. | 19:12 |
| jrwren | johnmce: you could run all of the commands with --debug --show-log and paste that. Maybe a log message will give us more clues. | 19:13 |
| jose | johnmce, jrwren: wouldn't recommend pastebinning the full log of debug. | 19:13 |
| jose | there's some sensitive credentials that show there | 19:14 |
| johnmce | jose: Interesting error when trying to deploy: Added charm "cs:trusty/keystone-8" to the environment.\n ERROR unknown option "vip_cidr" | 19:15 |
| johnmce | jose: strange that I didn't get an error with the local charm is that attrib is invalid | 19:16 |
| jose | johnmce: it is, in fact, not a valid config option | 19:17 |
| jose | I'm checking your first paste,s ec | 19:17 |
| jrwren | johnmce: i did get that error wiht local charm deploy using same bzr source that you did. | 19:17 |
| jose | I'm checking https://jujucharms.com/keystone/trusty/8 and I don't see vip_cidr as an option | 19:18 |
| jose | just vip | 19:18 |
| jrwren | yes, its not in config.yaml. unsure why johnmce did not get an error | 19:19 |
| johnmce | jose: Used to be an option: https://jujucharms.com/keystone/trusty/6 | 19:20 |
| jose | hmm, probably deprecated for some reason | 19:20 |
| jose | wait | 19:20 |
| jose | I know what that means | 19:20 |
| jrwren | i think i know too :) | 19:20 |
| johnmce | jose: Presumably vip is just IP/CIDR? | 19:20 |
| jose | (string) Virtual IP(s) to use to front API services in HA configuration. . If multiple networks are being used, a VIP should be provided for each network, separated by spaces. | 19:21 |
| jose | BUT | 19:21 |
| jose | that means you are using an old revision | 19:21 |
| jose | try removing that option from your yaml and deploying again | 19:21 |
| jose | johnmce: ^ | 19:22 |
| johnmce | jose: trying now | 19:23 |
| jrwren | install hook is still running, but I can confirm that it is using the juno cloud archive. So I tend to agree with jose, your local bzr checkout is out of date. | 19:23 |
| johnmce | jose: nope, still failed | 19:23 |
| jose | which error now? | 19:23 |
| johnmce | jrwren: I checked it out with bzr yesterday. They must have changed it in the last 30 hours or so then. | 19:24 |
| johnmce | jose: same error | 19:25 |
| jose | johnmce: have you deleted the value from the .yaml file? | 19:25 |
| johnmce | jose, jrwren: http://pastebin.com/TsDCNgwW | 19:27 |
| johnmce | jose: yes, both vip and vip_cidr are gone. | 19:27 |
| jose | johnmce: deploy using the command I gave you | 19:28 |
| jose | juju deploy --to lxc:1 --config=~/openstack.cfg keystone | 19:28 |
| johnmce | jose: still using my local copy, which I checked out yesterday. I don't think that vip_cidr got pulled in the past 30 hours though. Not mentioned in the config.yaml in my local copy. | 19:29 |
| jose | johnmce: if you use the command i just pasted, you will automatically pull the latest rev from the Charm Store, and the configuration options you will use will be the ones located in ~/openstack.cfg | 19:30 |
| johnmce | jose: Just doing it now | 19:30 |
| jose | johnmce: there's no need to use a local branch and that is, in fact, not recommended because it may cause this kind of issues | 19:30 |
| jose | cool :) | 19:30 |
| johnmce | jose: no installation-blocking error this time, as expected | 19:30 |
| jose | cool! | 19:30 |
| johnmce | jose: Not failing yet either - still pending | 19:31 |
| jose | awesome, let's hope it doesn't | 19:32 |
| johnmce | jose: Side note - bzr pull says I'm up to date I think | 19:32 |
| jose | johnmce: that's weird | 19:32 |
| jose | johnmce: bzr history, rev number? | 19:33 |
| jose | (latest) | 19:33 |
| johnmce | jose: john@maas-cam:/opt/charms/trusty/keystone$ bzr pull\n Using saved parent location: http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/keystone/trunk/\n No revisions or tags to pull. john@maas-cam:/opt/charms/trusty/keystone$ | 19:33 |
| jrwren | johnmce: bzr log and bzr log -p can help you find out when that config option was removed. I can't reproduce what you have experienced. I get juno deployed. | 19:33 |
| johnmce | jrwren, jose: bzr log: http://pastebin.com/LfTTABtR | 19:35 |
| jrwren | johnmce: so unchanged since Oct 9th | 19:36 |
| jose | weird, it is the same revision that's on the store | 19:36 |
| johnmce | jrwren, jose: The charm from the main repo has now successfully installed. So, the problem is that what I'm getting with bzr is broken. | 19:36 |
| jrwren | johnmce: what is out put of juju get keystone? | 19:37 |
| johnmce | Yesterday morning I renamed my old keystone folder and did a fresh bzr branch to get the latest keystone | 19:37 |
| jose | I am really wondering why was that failing, if it's the latest rev. the only thing that comes to mind is a file was changed manually by mistake | 19:39 |
| jose | but if you pulled+deployed then that shouldn't have been the case | 19:39 |
| johnmce | jose: I've made no modifications, and bzr diff confirms this | 19:40 |
| jose | yeah, I know | 19:40 |
| jose | anyways, looks like you were able to successfully deploy the charm, so that's good :) | 19:40 |
| johnmce | jrwren: There's sensitive information in "juju get keystone" output | 19:40 |
| johnmce | jrwren: anything in particular to look for? | 19:41 |
| jrwren | johnmce: just wondering if openstack-origin is set correctly from config | 19:41 |
| jose | johnmce: I'm wondering, is there the need to have the charm locally? if not, you could just go ahead and deploy from the charm store when you need | 19:42 |
| johnmce | jose: It's kind-of nice to be able to deploy this, but I'm trying to follow what I believe to be best-practices, and "bzr branch" each charm so that I know what I've got installed and can modify locally if needs be | 19:43 |
| jrwren | johnmce: i totally agree. | 19:43 |
| jose | same here, that's the benefit of the charm code being open source | 19:43 |
| johnmce | jose: I have previously made my own mods to a couple of charms (not keystone though) | 19:43 |
| jrwren | i wonder if there is a way to remove a charm from an environment. | 19:43 |
| jrwren | its almost as if the state server is deploying the previous one even though you have asked for local. | 19:44 |
| jose | jrwren: juju destroy-service [service-name]; juju destroy-machine # | 19:44 |
| jrwren | jose: that does not remove teh copy of the charm from teh state server AFAIK | 19:44 |
| jose | jrwren: you could upgrade-charm | 19:44 |
| jrwren | oh, there ya go. | 19:44 |
| jrwren | johnmce: did you already have a keystone deployed? is it possible that deploy is really just doing add-unit? | 19:45 |
| johnmce | jrwren: I tried upgrading the charm originally when I still had keystone deployed. That was before I nuked keystone and tried to start again. | 19:45 |
| jose | jrwren: destroy-service does remove the charm code from the state-server, when I debug my charms I don't have to re-bootstrap and I just destroy the service and redeploy | 19:45 |
| jose | johnmce: just a piece of advice, we have a package called charm-tools and it has a tool called `charm get`, so it branches from bzr | 19:46 |
| jrwren | jose: good point. | 19:46 |
| jose | johnmce: it's just an alias to an script that does bzr branch for you but it *should* always get the rev on the charm store, and not any other related branches | 19:47 |
| jose | may be useful for you instead of having to type all the branch name, just 'charm get cs:trusty/keystone' and you're done | 19:47 |
| johnmce | jose: I'll try it now | 19:48 |
| johnmce | jose: I just diffed the keystone I got with "charm get" against the one I got yesterday (excluding the .bzr folder) and they are identical. | 19:50 |
| jose | wat | 19:50 |
| johnmce | jose: Could ownership or permissions be an issue here? | 19:51 |
| jose | johnmce: I don't believe so | 19:51 |
| jose | johnmce: could you try deploying from the branch that you just pulled? | 19:51 |
| johnmce | jose: bzr not being able to set them? But then again, where is this old code coming from? | 19:51 |
| jose | I have no idea | 19:52 |
| johnmce | jose: doing this now: juju deploy --to lxc:1 cs:trusty/keystone --config=~/openstack.cfg | 19:54 |
| jose | johnmce: cs:trusty/keystone is the same as saying just keystone | 19:55 |
| jose | alias | 19:55 |
| johnmce | jose: How did you want me to do a test deployment? | 19:55 |
| jose | johnmce: try removing that folder and branching again | 19:56 |
| jose | I wanna make sure this error doesn't pop up again | 19:56 |
| johnmce | jose: Every time I use the charm store, this succeeds, but local copy always fails | 19:56 |
| johnmce | jose: In theory, I have always been installing the same code from the same source. It's the just the intermediate step of using bzr that breaks it | 19:57 |
| jose | this is so weird | 19:58 |
| johnmce | jose, jrwren: I just tried again from the copy I got with "charm get ...". Same problem. Fails every time. Using charm store works every time - several times now. | 20:03 |
| lazyPower | johnmce: do you have any info about why the deployment failed? | 20:08 |
| johnmce | jose, jrwren: OK, I'm starting to doubt my sanity now. Look at this: http://pastebin.com/RwyfQ43k | 20:08 |
| jose | hmm | 20:09 |
| jose | lazyPower: ^ | 20:09 |
| lazyPower | jose: i see the deployment command returned, however that doesn't necessarily mean it is deploying propper, i'm late to the show admittedely | 20:09 |
| lazyPower | *admittedly | 20:09 |
| johnmce | jose: Is there a typo I'm not seeing? | 20:09 |
| jose | johnmce: has it gotten any error until now? has the install hook failed? | 20:09 |
| johnmce | jose: It's failed as usual. | 20:10 |
| johnmce | jose: it's clearly getting the charm from a folder other than the one I'e specified | 20:10 |
| jrwren | anyone know how to connect to state server's mongod using mongo client? | 20:11 |
| lazyPower | jose: ah - i see what you're running into | 20:11 |
| jose | lazyPower: local fails, cs doesn't, even though they're exactly the same | 20:11 |
| johnmce | jose: how though? I'm being specific about the location | 20:11 |
| lazyPower | johnmce: correct me if i'm wrong, you've deployed keystone before, now you're attempting to deploy a newer revision of the charm and you've run into issues where a cached charm in your state server is continually deploying instead of what you expect to be deployed? | 20:12 |
| jrwren | lazyPower: that is my guess too. | 20:13 |
| johnmce | lazyPower: yes that's right, I always end up with an old charm | 20:13 |
| lazyPower | in some instances, i've run into charm caching on my state server, and have had to run juju upgrade-charm servicename --repository=/path/to/charms while the machine is provisioning. This was happening to me back in the 1.18 days however, not on the newer builds of juju. | 20:13 |
| johnmce | lazyPower: the updated charm is simply not being read | 20:13 |
| lazyPower | I'm spinning up a VM right now to verify if this is still the case - i'm giong to deploy mongodb from the store, switch to local, attempt destroy/re-deploy and see if it registers as local or cs | 20:13 |
| johnmce | lazyPower: to being with I was trying an upgrade, and then switched to destroy and deploy | 20:14 |
| jrwren | lazyPower: he always uses local, so maybe a better test is to bzr branch an older rev, deploy that local, and bzr upgrade and deploy that. | 20:15 |
| lazyPower | ok, can you tail your machine-0.log (the stateserver's machine log) during the deploy phase so we can see what it thinks its doing? | 20:15 |
| lazyPower | jrwren: ack, will do. | 20:15 |
| johnmce | lazyPower: I just tried doing an upgrade, which resulted in no error message, in spite of the keystone folder not existing | 20:18 |
| lazyPower | johnmce: when you juju status keystone - is the charm prefixed with local: and have a revision number that is 1 greater than the previous increment? | 20:18 |
| lazyPower | according to this output i have here... keystone-244, should read local:trusty/keystone-245 | 20:19 |
| johnmce | lazyPower, jose, jrwren: I may have just had a breakthough. | 20:19 |
| lazyPower | \o/ thats news we like to hear | 20:20 |
| johnmce | lazyPower, jose, jrwren: I had old copies of keystone lying around in renamed folders under the local repo | 20:20 |
| lazyPower | doh | 20:20 |
| johnmce | lazyPower, jose, jrwren: I thought that they would be ignored, but they are not | 20:20 |
| lazyPower | johnmce: i've done that as well, i'm gonig to file a bug against that actually | 20:21 |
| johnmce | lazyPower, jose, jrwren: One was called "keystone.old" | 20:21 |
| jrwren | but... what kind of renamed folders? | 20:21 |
| lazyPower | johnmce: I'm glad you discovered that - as i feel juju should warn you its picking something arbitrary | 20:21 |
| jrwren | juju read it from keystone.old ? | 20:21 |
| lazyPower | jrwren: its scanning metadata, so it'll pick whicever it thinks is right | 20:21 |
| jrwren | ugh. | 20:21 |
| lazyPower | jrwren: and thats a blood obnoxious papercut | 20:21 |
| jrwren | that is a stab wound. | 20:22 |
| jrwren | a papercut doesn't take 8 man hours on a sunday :) | 20:22 |
| johnmce | lazyPower, jose, jrwren: It really goes against any behaviour I'm used to seeing in other realms. I thought the folder name was key | 20:22 |
| jrwren | johnmce: yup, it should be, but the trick and key is that it treats the repository as a repository and reads metadata.yaml in each directory instead of just directory names. | 20:23 |
| jrwren | its probably by design, but certainly docs should be better around local deploy in this area. | 20:23 |
| johnmce | lazyPower, jose, jrwren: Thanks for all your help. At least now I know for the future. | 20:24 |
| lazyPower | johnmce: really sorry you ran into that, give me another moment to get you a bug link | 20:24 |
| lazyPower | i'm sure you'd like to tag it for additional traction | 20:24 |
| jose | johnmce: if there are any other questions or things we may help with, feel free to ask here or on juju@lists.ubuntu.com | 20:24 |
| lazyPower | https://bugs.launchpad.net/juju-core/+bug/1390972 | 20:25 |
| mup | Bug #1390972: Juju picks an arbitrary charm when multiple-versions of a charm are present in a local repository <papercut> <juju-core:New> <https://launchpad.net/bugs/1390972> | 20:25 |
| johnmce | lazyPower, jose, jrwren: You know I've probably been tripped up by this before at not known it. I often rename a folder rather than wipe it. I've previously nuked my whole installation to get a clean start. | 20:26 |
| lazyPower | johnmce: I keep multiple charm repositories for my environments and export JUJU_REPOSITORY | 20:26 |
| lazyPower | you can always juju upgrade-charm --switch when you are ready to move from store to devel, and this keeps me straight when im' jumping environments. | 20:27 |
| jrwren | all great details which would be nice in docs. | 20:27 |
| lazyPower | but again, really sorry this tripped you up - its not a great user experience when its doing something on your behalf and not telling you that's what its doing. | 20:27 |
| lazyPower | jrwren: file bug's on github.com/juju/docs - and i'll see about circling back to getting those in there. | 20:28 |
| jrwren | lazyPower: i tend to file pull requests on juju/docs :) | 20:28 |
| lazyPower | jrwren: even better :) | 20:28 |
| johnmce | lazyPower, jose, jrwren: I can't thank you guys enough for your patience. Hopefully others will benefit from this. It's beer o'clock here, so I'll sign off :-) | 20:29 |
| lazyPower | johnmce: best of luck to ya o/ cheers | 20:29 |
| jose | enjoy what's left of your weekend! | 20:29 |
| === scuttlemonkey is now known as scuttle|afk | ||
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| === kadams54 is now known as kadams54-away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!