[18:20] <johnmce> Can someone please help with a problem I'm having with Juju. I've asked about this problem here several times in the past 30 hours, but had no response.
[18:21] <johnmce> There is clearly a bug in Juju that causes it to deploy old versions of charms when deploying Keystone to a MAAS LXC container
[18:22] <johnmce> I've used bzr to check the latest Keystone charm to a local repo. This charm clearly supports Juno, as can be seen in the source code.
[18:24] <johnmce> However, when it is deployed to a newly instantiated MAAS/LXC container, the charm deployed is an earlier revision that does not support Juno.
[18:25] <johnmce> Can anyone please help? I'm rapidly losing faith in Juju, as there is no documentation or community support that gives any clue as to the possible cause.
[18:27] <jrwren> jose: how are you deploying it?
[18:28] <jrwren> err, johnmce
[18:28] <jrwren> sorry jose, misdir
[18:28] <jose> no prob :)
[18:28] <jrwren> johnmce: can you pastebin the commands you have run or maybe ask on askubuntu to completely describe the problem ?
[18:29] <jose> johnmce: probably not getting a response since it's the weekend and most people are hanging out doing some other stuff. but I'll be glad to give you a hand with this! as jrwren mentioned, if you could pastebin the command or the bundle you're using that'd give us a good first clue on what's going on
[18:29] <johnmce> jrwren: Thanks for the response. I'll pastebin some details info shortly.
[18:33] <jrwren> johnmce: if you have commands which you've run, I can try to reproduce.
[18:40] <johnmce> jrwren: This is a complete attempted install I just did: http://pastebin.com/9Tjae302
[18:43] <johnmce> jrwren: As you can see this revision of the charm is called "local:trusty/keystone-243". The same revision number that hits the LXC machine, but with inconsistent content.
[18:43] <jrwren> johnmce: whoa, how does that work :)
[18:44] <jrwren> johnmce: are any of /opt/charms or /opt/charms/trusty  symlinked directories? are they linked to where you expect?
[18:45] <jrwren> johnmce: can you paste that bundle?
[18:46] <johnmce> jrwren: None are symlinked. Unless of course "bzr branch" does some weird symlinking that I didn't expect
[18:46] <jrwren> johnmce: no, bzr branch won't do taht.
[18:46] <jrwren> johnmce: what is the value of openstack-origin in config?
[18:47] <johnmce> jrwren: Got to go for about 10 minutes. Where will I find the bundle in a suitable format for pasting?
[18:48] <jrwren> johnmce: maybe you didn't use a bundle. I see a bundle mentioned on line 112 of that pastebin
[18:48] <jose> johnmce: you're using a local charm, which means you're not deploying from the charm store (cc. jrwren )
[18:48] <jose> johnmce: that means, if you have branched and old revision which doesn't have the changes that are on the store yet, then you will not get them
[18:49] <jose> johnmce: try going into /opt/charms/trusty/keystone and doing bzr pull
[18:50] <jose> otoh, I'm seeing an error about an invalid cloud repo
[18:50] <jose> oh, oh, wait! charm helpers!
[18:50] <jose> johnmce: may I see what's the output of ls -lh /opt/charms/trusty/keystone, please?
[18:53] <jose> also, try deploying with just the following command:
[18:53] <jose> juju deploy --to lxc:1 --config=~/openstack.cfg keystone
[18:57] <jrwren> seems like config isn't getting set. openstack-origin is defaulting
[19:00] <johnmce> jose: http://pastebin.com/8EpmNu9A
[19:02] <johnmce> jrwren: I included the relevant part of the config file in my original pastebin. This was the origin line for keystone: openstack-origin: 'cloud:trusty-juno'
[19:04] <jrwren> johnmce: right. I think you diagnosis is perfect. I've no idea why the juno definitions aren't in the charm on the deployed lxc.
[19:05] <johnmce> jrwren: I've got single quotes around cloud:trusty-juno. Is that the correct syntax?
[19:05] <jrwren> johnmce: its yaml. that should be fine with or without
[19:10] <johnmce> jrwren: Is there any more information I can provide to help diagnose this?
[19:12] <jose> johnmce: did you try deploying with that last command I gave you?
[19:12] <johnmce> jose: Sorry, not yet. I'll try it now.
[19:13] <jrwren> johnmce: you could run all of the commands with --debug --show-log and paste that. Maybe a log message will give us more clues.
[19:13] <jose> johnmce, jrwren: wouldn't recommend pastebinning the full log of debug.
[19:14] <jose> there's some sensitive credentials that show there
[19:15] <johnmce> jose: Interesting error when trying to deploy: Added charm "cs:trusty/keystone-8" to the environment.\n ERROR unknown option "vip_cidr"
[19:16] <johnmce> jose: strange that I didn't get an error with the local charm is that attrib is invalid
[19:17] <jose> johnmce: it is, in fact, not a valid config option
[19:17] <jose> I'm checking your first paste,s ec
[19:17] <jrwren> johnmce: i did get that error wiht local charm deploy using same bzr source that you did.
[19:18] <jose> I'm checking https://jujucharms.com/keystone/trusty/8 and I don't see vip_cidr as an option
[19:18] <jose> just vip
[19:19] <jrwren> yes, its not in config.yaml.  unsure why johnmce did not get an error
[19:20] <johnmce> jose: Used to be an option: https://jujucharms.com/keystone/trusty/6
[19:20] <jose> hmm, probably deprecated for some reason
[19:20] <jose> wait
[19:20] <jose> I know what that means
[19:20] <jrwren> i think i know too :)
[19:20] <johnmce> jose: Presumably vip is just IP/CIDR?
[19:21] <jose> (string) Virtual IP(s) to use to front API services in HA configuration. . If multiple networks are being used, a VIP should be provided for each network, separated by spaces.
[19:21] <jose> BUT
[19:21] <jose> that means you are using an old revision
[19:21] <jose> try removing that option from your yaml and deploying again
[19:22] <jose> johnmce: ^
[19:23] <johnmce> jose: trying now
[19:23] <jrwren> install hook is still running, but I can confirm that it is using the juno cloud archive. So I tend to agree with jose, your local bzr checkout is out of date.
[19:23] <johnmce> jose: nope, still failed
[19:23] <jose> which error now?
[19:24] <johnmce> jrwren: I checked it out with bzr yesterday. They must have changed it in the last 30 hours or so then.
[19:25] <johnmce> jose: same error
[19:25] <jose> johnmce: have you deleted the value from the .yaml file?
[19:27] <johnmce> jose, jrwren: http://pastebin.com/TsDCNgwW
[19:27] <johnmce> jose: yes, both vip and vip_cidr are gone.
[19:28] <jose> johnmce: deploy using the command I gave you
[19:28] <jose> juju deploy --to lxc:1 --config=~/openstack.cfg keystone
[19:29] <johnmce> jose: still using my local copy, which I checked out yesterday. I don't think that vip_cidr got pulled in the past 30 hours though. Not mentioned in the config.yaml in my local copy.
[19:30] <jose> johnmce: if you use the command i just pasted, you will automatically pull the latest rev from the Charm Store, and the configuration options you will use will be the ones located in ~/openstack.cfg
[19:30] <johnmce> jose: Just doing it now
[19:30] <jose> johnmce: there's no need to use a local branch and that is, in fact, not recommended because it may cause this kind of issues
[19:30] <jose> cool :)
[19:30] <johnmce> jose: no installation-blocking error this time, as expected
[19:30] <jose> cool!
[19:31] <johnmce> jose: Not failing yet either - still pending
[19:32] <jose> awesome, let's hope it doesn't
[19:32] <johnmce> jose: Side note - bzr pull says I'm up to date I think
[19:32] <jose> johnmce: that's weird
[19:33] <jose> johnmce: bzr history, rev number?
[19:33] <jose> (latest)
[19:33] <johnmce> jose: john@maas-cam:/opt/charms/trusty/keystone$ bzr pull\n Using saved parent location: http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/keystone/trunk/\n No revisions or tags to pull.                                                                                                               john@maas-cam:/opt/charms/trusty/keystone$
[19:33] <jrwren> johnmce: bzr log and bzr log -p can help you find out when that config option was removed.  I can't reproduce what you have experienced.  I get juno deployed.
[19:35] <johnmce> jrwren, jose: bzr log: http://pastebin.com/LfTTABtR
[19:36] <jrwren> johnmce: so unchanged since Oct 9th
[19:36] <jose> weird, it is the same revision that's on the store
[19:36] <johnmce> jrwren, jose: The charm from the main repo has now successfully installed. So, the problem is that what I'm getting with bzr is broken.
[19:37] <jrwren> johnmce: what is out put of juju get keystone?
[19:37] <johnmce> Yesterday morning I renamed my old keystone folder and did a fresh bzr branch to get the latest keystone
[19:39] <jose> I am really wondering why was that failing, if it's the latest rev. the only thing that comes to mind is a file was changed manually by mistake
[19:39] <jose> but if you pulled+deployed then that shouldn't have been the case
[19:40] <johnmce> jose: I've made no modifications, and bzr diff confirms this
[19:40] <jose> yeah, I know
[19:40] <jose> anyways, looks like you were able to successfully deploy the charm, so that's good :)
[19:40] <johnmce> jrwren: There's sensitive information in "juju get keystone" output
[19:41] <johnmce> jrwren: anything in particular to look for?
[19:41] <jrwren> johnmce: just wondering if openstack-origin is set correctly from config
[19:42] <jose> johnmce: I'm wondering, is there the need to have the charm locally? if not, you could just go ahead and deploy from the charm store when you need
[19:43] <johnmce> jose: It's kind-of nice to be able to deploy this, but I'm trying to follow what I believe to be best-practices, and "bzr branch" each charm so that I know what I've got installed and can modify locally if needs be
[19:43] <jrwren> johnmce: i totally agree.
[19:43] <jose> same here, that's the benefit of the charm code being open source
[19:43] <johnmce> jose: I have previously made my own mods to a couple of charms (not keystone though)
[19:43] <jrwren> i wonder if there is a way to remove a charm from an environment.
[19:44] <jrwren> its almost as if the state server is deploying the previous one even though you have asked for local.
[19:44] <jose> jrwren: juju destroy-service [service-name]; juju destroy-machine #
[19:44] <jrwren> jose: that does not remove teh copy of the charm from teh state server AFAIK
[19:44] <jose> jrwren: you could upgrade-charm
[19:44] <jrwren> oh, there ya go.
[19:45] <jrwren> johnmce: did you already have a keystone deployed? is it possible that deploy is really just doing add-unit?
[19:45] <johnmce> jrwren: I tried upgrading the charm originally when I still had keystone deployed. That was before I nuked keystone and tried to start again.
[19:45] <jose> jrwren: destroy-service does remove the charm code from the state-server, when I debug my charms I don't have to re-bootstrap and I just destroy the service and redeploy
[19:46] <jose> johnmce: just a piece of advice, we have a package called charm-tools and it has a tool called `charm get`, so it branches from bzr
[19:46] <jrwren> jose: good point.
[19:47] <jose> johnmce: it's just an alias to an script that does bzr branch for you but it *should* always get the rev on the charm store, and not any other related branches
[19:47] <jose> may be useful for you instead of having to type all the branch name, just 'charm get cs:trusty/keystone' and you're done
[19:48] <johnmce> jose: I'll try it now
[19:50] <johnmce> jose: I just diffed the keystone I got with "charm get" against the one I got yesterday (excluding the .bzr folder) and they are identical.
[19:50] <jose> wat
[19:51] <johnmce> jose: Could ownership or permissions be an issue here?
[19:51] <jose> johnmce: I don't believe so
[19:51] <jose> johnmce: could you try deploying from the branch that you just pulled?
[19:51] <johnmce> jose: bzr not being able to set them? But then again, where is this old code coming from?
[19:52] <jose> I have no idea
[19:54] <johnmce> jose: doing this now: juju deploy --to lxc:1 cs:trusty/keystone --config=~/openstack.cfg
[19:55] <jose> johnmce: cs:trusty/keystone is the same as saying just keystone
[19:55] <jose> alias
[19:55] <johnmce> jose: How did you want me to do a test deployment?
[19:56] <jose> johnmce: try removing that folder and branching again
[19:56] <jose> I wanna make sure this error doesn't pop up again
[19:56] <johnmce> jose: Every time I use the charm store, this succeeds, but local copy always fails
[19:57] <johnmce> jose: In theory, I have always been installing the same code from the same source. It's the just the intermediate step of using bzr that breaks it
[19:58] <jose> this is so weird
[20:03] <johnmce> jose, jrwren: I just tried again from the copy I got with "charm get ...". Same problem. Fails every time. Using charm store works every time - several times now.
[20:08] <lazyPower> johnmce: do you have any info about why the deployment failed?
[20:08] <johnmce> jose, jrwren: OK, I'm starting to doubt my sanity now. Look at this: http://pastebin.com/RwyfQ43k
[20:09] <jose> hmm
[20:09] <jose> lazyPower: ^
[20:09] <lazyPower> jose: i see the deployment command returned, however that doesn't necessarily mean it is deploying propper, i'm late to the show admittedely
[20:09] <lazyPower> *admittedly
[20:09] <johnmce> jose: Is there a typo I'm not seeing?
[20:09] <jose> johnmce: has it gotten any error until now? has the install hook failed?
[20:10] <johnmce> jose: It's failed as usual.
[20:10] <johnmce> jose: it's clearly getting the charm from a folder other than the one I'e specified
[20:11] <jrwren> anyone know how to connect to state server's mongod using mongo client?
[20:11] <lazyPower> jose: ah - i see what you're running into
[20:11] <jose> lazyPower: local fails, cs doesn't, even though they're exactly the same
[20:11] <johnmce> jose: how though? I'm being specific about the location
[20:12] <lazyPower> johnmce: correct me if i'm wrong, you've deployed keystone before, now you're attempting to deploy a newer revision of the charm and you've run into issues where a cached charm in your state server is continually deploying instead of what you expect to be deployed?
[20:13] <jrwren> lazyPower: that is my guess too.
[20:13] <johnmce> lazyPower: yes that's right, I always end up with an old charm
[20:13] <lazyPower> in some instances, i've run into charm caching on my state server, and have had to run juju upgrade-charm servicename  --repository=/path/to/charms while the machine is provisioning. This was happening to me back in the 1.18 days however, not on the newer builds of juju.
[20:13] <johnmce> lazyPower: the updated charm is simply not being read
[20:13] <lazyPower> I'm spinning up a VM right now to verify if this is still the case - i'm giong to deploy mongodb from the store, switch to local, attempt destroy/re-deploy and see if it registers as local or cs
[20:14] <johnmce> lazyPower: to being with I was trying an upgrade, and then switched to destroy and deploy
[20:15] <jrwren> lazyPower: he always uses local, so maybe a better test is to bzr branch an older rev, deploy that local, and bzr upgrade and deploy that.
[20:15] <lazyPower> ok, can you tail your machine-0.log (the stateserver's machine log) during the deploy phase so we can see what it thinks its doing?
[20:15] <lazyPower> jrwren: ack, will do.
[20:18] <johnmce> lazyPower: I just tried doing an upgrade, which resulted in no error message, in spite of the keystone folder not existing
[20:18] <lazyPower> johnmce: when you juju status keystone - is the charm prefixed with local: and have a revision number that is 1 greater than the previous increment?
[20:19] <lazyPower> according to this output i have here... keystone-244, should read local:trusty/keystone-245
[20:19] <johnmce> lazyPower, jose, jrwren: I may have just had a breakthough.
[20:20] <lazyPower> \o/ thats news we like to hear
[20:20] <johnmce> lazyPower, jose, jrwren: I had old copies of keystone lying around in renamed folders under the local repo
[20:20] <lazyPower> doh
[20:20] <johnmce> lazyPower, jose, jrwren: I thought that they would be ignored, but they are not
[20:21] <lazyPower> johnmce: i've done that as well, i'm gonig to file a bug against that actually
[20:21] <johnmce> lazyPower, jose, jrwren: One was called "keystone.old"
[20:21] <jrwren> but... what kind of renamed folders?
[20:21] <lazyPower> johnmce: I'm glad you discovered that - as i feel juju should warn you its picking something arbitrary
[20:21] <jrwren> juju read it from keystone.old ?
[20:21] <lazyPower> jrwren: its scanning metadata, so it'll pick whicever it thinks is right
[20:21] <jrwren> ugh.
[20:21] <lazyPower> jrwren: and thats a blood obnoxious papercut
[20:22] <jrwren> that is a stab wound.
[20:22] <jrwren> a papercut doesn't take 8 man hours on a sunday :)
[20:22] <johnmce> lazyPower, jose, jrwren: It really goes against any behaviour I'm used to seeing in other realms. I thought the folder name was key
[20:23] <jrwren> johnmce: yup, it should be, but the trick and key is that it treats the repository as a repository and reads metadata.yaml in each directory instead of just directory names.
[20:23] <jrwren> its probably by design, but certainly docs should be better around local deploy in this area.
[20:24] <johnmce> lazyPower, jose, jrwren: Thanks for all your help. At least now I know for the future.
[20:24] <lazyPower> johnmce: really sorry you ran into that, give me another moment to get you a bug link
[20:24] <lazyPower> i'm sure you'd like to tag it for additional traction
[20:24] <jose> johnmce: if there are any other questions or things we may help with, feel free to ask here or on juju@lists.ubuntu.com
[20:25] <lazyPower> https://bugs.launchpad.net/juju-core/+bug/1390972
[20:25] <mup> Bug #1390972: Juju picks an arbitrary charm when multiple-versions of a charm are present in a local repository <papercut> <juju-core:New> <https://launchpad.net/bugs/1390972>
[20:26] <johnmce> lazyPower, jose, jrwren: You know I've probably been tripped up by this before at not known it. I often rename a folder rather than wipe it. I've previously nuked my whole installation to get a clean start.
[20:26] <lazyPower> johnmce: I keep multiple charm repositories for my environments and export JUJU_REPOSITORY
[20:27] <lazyPower> you can always juju upgrade-charm --switch when you are ready to move from store to devel, and this keeps me straight when im' jumping environments.
[20:27] <jrwren> all great details which would be nice in docs.
[20:27] <lazyPower> but again, really sorry this tripped you up - its not a great user experience when its doing something on your behalf and not telling you that's what its doing.
[20:28] <lazyPower> jrwren: file bug's on github.com/juju/docs - and i'll see about circling back to getting those in there.
[20:28] <jrwren> lazyPower: i tend to file pull requests on juju/docs :)
[20:28] <lazyPower> jrwren: even better :)
[20:29] <johnmce> lazyPower, jose, jrwren: I can't thank you guys enough for your patience. Hopefully others will benefit from this. It's beer o'clock here, so I'll sign off :-)
[20:29] <lazyPower> johnmce: best of luck to ya o/  cheers
[20:29] <jose> enjoy what's left of your weekend!