=== freeflying_away is now known as freeflying
[00:38] <zradmin> negronji: odd I cant get it to work with that at all... it never seems to create the database in mysql
[00:39] <zradmin> negronji: I get the following in the debug log when trying to set it up "This charm doesn't know how to handle 'shared-db-relation-joined'
[00:42] <zradmin> negronjl: is there updated documentation anywhere? I'm still following the guide at https://wiki.ubuntu.com/ServerTeam/OpenStackHA
[00:43] <negronjl> zradmin, which charm is giving you the  error ?
[00:45] <zradmin> negronl: quantum-gateway when I'm connecting it to the mysql charm... I finish connecting it to rabbitmq and nova-ccc but the service never seems to come online either, i just get 503 errors
[00:46] <zradmin> negronjl: sorry i keep mispelling your username :)
[00:46] <negronjl> zradmin, no worries ...
[00:46] <negronjl> zradmin: that's interesting ... i don't get those issues at all ...
[00:47] <negronjl> zradmin: I assume that you are trying to deploy this thing in ha correct ?
[00:47] <zradmin> correct
[00:47] <sarnold> zradmin: (most irc clients let you type or two letters of a nickname then hit 'tab' to finish it up :)
[00:47] <negronjl> zradmin, let me check and do some digging
[00:47] <zradmin> sarnold: sweet, thx for the tip
[00:49] <negronjl> zradmin: can you pastebin your relations and your config?  Maybe i can find something odd there.
[00:52] <negronjl> zradmin, how are you relating mysql to quantum ?
[00:52] <zradmin> negronjl: http://pastebin.ubuntu.com/6476662/
[00:52] <negronjl> zradmin: juju add-relation quantum-gateway mysql-hacluster ?
[00:52] <negronjl> zradmin, Are you using juju-deployer ?
[00:53] <negronjl> zradmin: it looks like it
[00:53] <zradmin> negronjl: following the guide, its this command :juju add-relation quantum-gateway mysql
[00:53] <zradmin> negronjl: nope just did a quick export out of juju gui to get the relations and the rest is all my local.yaml file
[00:54] <negronjl> zradmin, ok
[00:54]  * negronjl reads
[00:57] <negronjl> zradmin, what revision of quantum-gateway are you using ?
[00:58] <negronjl> zradmin, the error you are getting tells me that your quantum-gateway charm is either broken or out of date
[01:02] <zradmin> negronjl: i am pullig directly from the charm store... v11 i think?
[01:02] <negronjl> zradmin, Also change your oopenstack-origin to "openstack-origin: cloud:precise-havana/updates"
[01:06] <zradmin> negronjl: ok im making the edits now
[01:08] <negronjl> zradmin, juju remove-relation quantum-gateway mysql and then juju add-relation quantum-gateway mysql
[01:08] <negronjl> zradmin, just to retry the thing ( and hope )
[01:09] <zradmin> negronjl: I'm actually redeploying the quantum charm from scratch just so its clean
[01:10] <negronjl> zradmin: ok
[01:29] <negronjl> zradmin, I can deploy it all on my testing environment
[01:29] <negronjl> zradmin, I'll check back here later but, for now, I have to go.
=== context_ is now known as context
[01:44] <zradmin> negronjl: thanks for the help, I just finished getting it connected to mysql and this was the log file http://pastebin.ubuntu.com/6476811/ - seems to have the same error
[01:46] <davecheney> zradmin: how did you relate to mysql:shared-db
[01:46] <davecheney> there is no relation called that
[01:46] <davecheney> or no implemention for that relation in the charm
[01:47] <zradmin> just running the juju add-relation quantum-gateway mysql command
[01:47] <davecheney> oh dear
[01:47] <davecheney> which version of the mysql charm have you deployed
[01:49] <zradmin> mysql-29
[01:50] <davecheney> zradmin: ok, while I look into this
[01:50] <davecheney> you should remove that relationship
[01:50] <davecheney> and do
[01:50] <davecheney> juju add-relation quantum-gateway mysql:db
[01:52] <zradmin> ok I'll give that a shot
[01:56] <davecheney> zradmin: mysql-29 implements shared-db-relation
[01:56] <davecheney> i think you may have an old versoin of the charm somehow
[01:57] <davecheney> even so, you probvably don't want to relate to that relation endpoint
[01:57] <davecheney> you want mysql:db
[01:57] <zradmin> is there any updated documentation for setting up the test environment?
[01:58] <zradmin> it seems like the publicly available wikis are very out of date
[02:01] <davecheney> zradmin: do you mean the local provider ?
[02:02] <zradmin> davecheney: so when i try and run juju add-relation quantum-gateway mysql:db it tells me "ERROR no relations found"
[02:02] <davecheney> zradmin: my mistake, please trye
[02:03] <zradmin> mysql:shared-db?
[02:03] <zradmin> looking at all the other charm relations that seems to be what they're connecting to
[02:03] <davecheney> yeah, just checked out quantum-gateway
[02:04] <davecheney> ok, i thnk were; back to mysql
[02:04] <davecheney> it looks like mysql-29 implements shared-db properly
[02:04] <davecheney> i cannot explain why you deployment of that charm is acting weirdly
[02:04] <davecheney> how did you deploy the mysql charm ?
[02:05] <zradmin> juju deploy --config local.yaml -n 2 mysql
[02:05] <zradmin> then juju deploy hacluster mysql-hacluster
[02:06] <zradmin> added the relationship to ceph and then to the mysql-hacluster
[02:06] <davecheney> zradmin: i'm sorry, i'm out of ideas
[02:06] <davecheney> if that is mysql-29 then it shold implemnt shared-db
[02:08] <zradmin> yeah i'm fairly confused as well
[02:26] <zradmin> is there a way to manually retrigger that specific hook?
[03:15] <davecheney> zradmin: is the unit in error ?
[03:16] <zradmin> nope.... thats the frustrating part - everything is started
[03:17] <davecheney> zradmin: i'm sorry, there is no way to do this if the unit is not in error
[03:17] <davecheney> you could tru
[03:17] <davecheney> juju resolved --retry $UNIT
[03:17] <davecheney> but I suspect it will tell you thre is nothing to do
[03:18] <zradmin> yeah it tells me the unit is not in an error state so it wont do anything
[03:19] <zradmin> I've tried destroying the relationship a few times and rebuilding it since then... but each time it still says that mysql doesnt understand the shared-db relationship
[03:19] <zradmin> which is pretty strange considering that glance/nova/keystone etc. all have databases and are working fine
[03:20] <davecheney> zradmin: can you do juju status $MYSQL_SERVICE
[03:22] <zradmin> here you go http://pastebin.ubuntu.com/6477065/
[03:29] <davecheney> zradmin: could you get me an ls -al of the charm hooks dir
[03:29] <davecheney> from memory it's somethignlike
[03:29] <davecheney> /var/lib/juju/agents/unit-0/hooks
[03:29] <davecheney> it'll be on machine 4
[03:31] <zradmin> http://pastebin.ubuntu.com/6477095/
[03:33] <davecheney> zradmin: hmm, truncated but looks right
[03:33] <davecheney> on that same machine can you have a look in /var/lib/juju/unit-0.log
[03:34] <davecheney> for the bit about 'charm doesn't implement hook "shared-db-relation-joined" [sic]
[03:39] <zradmin> here it is http://pastebin.ubuntu.com/6477110/
[03:40] <davecheney> zradmin: aah
[03:41] <davecheney> i think that might have been a red herrin
[03:41] <davecheney> the real issue appears to be
[03:41] <davecheney> 2013-11-26 02:35:45 INFO juju juju-log.go:66 mysql/0 shared-db:71: MySQL service is peered, bailing shared-db relation as this service unit is not the leader
[03:41] <davecheney> in fact, i don't think that is the error
[03:41] <davecheney> is an error
[03:41] <davecheney> if you look at the log on mysql/1
[03:41] <davecheney> you'll see (this time) it was the leader
[03:43] <zradmin> hmmm i thought so to at first but on the quantum-gw i cant find anything in neutron.conf relating to the mysql db backend
[03:43] <zradmin> or api-paste
[03:54] <zradmin> so this is what i get when i try and run a neutron command at the moment... http://pastebin.ubuntu.com/6477139/
[03:58] <zradmin> the neutron.conf file on the gateway only has the rabbit host defined... but the neutron.conf on the nova-ccc has a sqllite database specified for some reason.... its the only reference to a database for neutron i can find. http://pastebin.ubuntu.com/6477146/
=== freeflying is now known as freeflying_away
[04:12] <zradmin> davecheney: I've tried standing up this environment 4 or 5 times since the havana release came out and i run into this wall each time.... its driving me crazy :)
[04:13] <davecheney> zradmin: i'm really sorry to hear that
[04:13] <davecheney> this is supposed to be easy
[04:13] <davecheney> did you find anything in the mysql/1 log ?
[04:16] <zradmin> i see it setting up nova.... and it is populating a quantum password field
[04:17] <zradmin> but i also have the 2013-11-26 02:35:42 INFO juju juju-log.go:66 mysql/1 shared-db:71: This charm doesn't know how to handle 'shared-db-relation-joined'.
[04:18] <zradmin> davecheney: yeah i love the power of juju and its much easier setting it up than tooling everything by hand... i just cant seem to figure out why those pices are giving me trouble. I do appreciate the help for sure!
[04:21] <davecheney> zradmin: thanks for the log
[04:21] <davecheney> that error isn't an error
[04:21] <davecheney> see the prevoius discussion
[04:21] <davecheney> *but* does mysql/1 also say that it is bailing out because it isn't the leader ?
[04:22] <zradmin> nope no bailing that i can see
[04:22] <davecheney> zradmin: ok, here is what I thin is happening
[04:22] <davecheney> 1. quantum-gateway is related to mysql:shared-db
[04:22] <davecheney> mysql is supposed to set some relation settings back to qyuantum-gw
[04:23] <davecheney> which in turn will make it switch from sqlite to using that mysql db
[04:23] <davecheney> but mysql isn't sending any relation settings (dunno why yet)
[04:23] <davecheney> and that is hwy quantum is using sqllite
[04:24] <zradmin> and not bringing the api online....
[04:24] <davecheney> it is possible that a change of relation hooks have to run to get things properly configured
[04:24] <zradmin> it should be logged in the /hooks directory right?
[04:24] <davecheney> no, nothing is logged in the hooks directory
[04:25] <davecheney> try removing and readding the relationship
[04:25] <zradmin> by logged... i meant the config its trying to push
[04:26] <davecheney> zradmin: hmm,, not really
[04:26] <davecheney> relation settings are only visible to hook commands via the relation-get hook command
[04:29] <zradmin> is that outside of the normal juju cli?
[04:30] <davecheney> yeah, there is no way to see relation settings unless youre using debug hooks in a hook context
[04:31] <zradmin> same issue with removing and readding the relationship
[04:32] <zradmin> ah so the unit would have to be stuck on a hook in order to see what its trying to do
[04:34] <davecheney> zradmin: i'm not sure stuck is the right thing to say
[04:35] <davecheney> but i'd guess that both mysql units think they are not hte master
[04:35] <davecheney> so *neither* are taking action and setting the relation settings to quantum-gw
[04:35] <davecheney> can you past the log of mysql/1 ?
[04:36] <zradmin> yeah I'll grab the end of it
[04:36] <davecheney> ta
[04:37] <zradmin> http://pastebin.ubuntu.com/6477255/
[04:38] <zradmin> i know mysql is only running on /1 currently so it should be the master
[04:38] <davecheney> ok, so i can see it doing relation set
[04:38] <davecheney> what does the quantum-gw log say ?
[04:41] <zradmin> this looks like the relevant section here.... says it is missing the quantum/nova passwords
[04:41] <zradmin> http://pastebin.ubuntu.com/6477270/
=== freeflying_away is now known as freeflying
[04:46] <zradmin> and the database_host information apparently
[04:47] <davecheney> zradmin: ahh
[04:47] <davecheney> right
[04:47] <davecheney> here is what is going on
[04:47] <davecheney> 2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "quantum_password" "mysql/0"]
[04:47] <davecheney> 2013-11-26 02:35:49 DEBUG worker.uniter.jujuc server.go:109 hook context id "quantum-gateway/1:shared-db-relation-changed:4235752611937240132"; dir "/var/lib/juju/agents/unit-quantum-gateway-1/charm"
[04:47] <davecheney> 2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "nova_password" "mysql/0"]
[04:47] <davecheney> ^ quantum has decided that ONLY mysql/0 has the relation data it wants
[04:48] <davecheney> *but* mysql/1 is the unit which provided the relation data
[04:48] <zradmin> that makes alot of sense
[04:49]  * davecheney disappears down a charm helpers rabbit hole
[04:49] <davecheney> zradmin: short version
[04:49] <zradmin> doesnt it try and get itt from /1 as well though?
[04:49] <davecheney> deploy one mysql unit
[04:49] <zradmin> 2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "db_host" "mysql/1"]
[04:49] <davecheney> and i'll work
[04:49] <davecheney> yeah
[04:49] <davecheney> ok
[04:49] <davecheney> that is weird
[04:49] <davecheney> so what is after line 33
[04:51] <zradmin> next batch http://pastebin.ubuntu.com/6477293/
[04:51] <zradmin> looks like its just writing the config files
[04:52] <zradmin> checked the nova.conf on the host.... and it has its db line sql_connection=mysql://nova:vSk0czUGLt20mXP1@10.10.32.4/nova
[04:54] <davecheney> zradmin: ok, do you want to wait til the unit reaches swtarted state
[04:54] <davecheney> all these logs are truncated
[04:56] <zradmin> is there a way to get more verbosity?
[04:59] <davecheney> zradmin: most people complain they are too verbose :)
[04:59] <davecheney> that's all you get i'm afraid
[04:59] <davecheney> is the unit in started state ?
[04:59] <zradmin> yeah all of them are
[05:00] <davecheney> right
[05:01] <davecheney> does quantum-gw work ?
[05:01] <davecheney> it looks like it's missing a restart action somewhere
[05:03] <zradmin> nope i just get that 503 service not found error everytime i try and query neutron
[05:06] <davecheney> zradmin: i think quantum-gw runs under upstart
[05:06] <davecheney> can you go to the unit and do a service restart
[05:06] <davecheney> on the service
[05:06] <davecheney> maybe that will get it going
[05:08] <zradmin> ok that services doesnt exist on the quantum-gateway nodes
[05:09] <zradmin> looks like its called neutron-server and exists on the nova-ccc
[05:18] <zradmin> davecheney: i restarted the quantum-gw nodes... but still no dice
=== philipballew is now known as philip
[05:27] <freeflying> zradmin, is yours ha deployment?
[05:28] <zradmin> freeflying: yup been following the ha guide posted here: https://wiki.ubuntu.com/ServerTeam/OpenStackHA
[05:29] <zradmin> adjusted for havana of course
[05:29] <freeflying> zradmin, does keystone work properly
[05:31] <freeflying> zradmin, also 10.10.32.4 is the vip of your mysql cluster?
[05:31] <zradmin> freeflying: yup, keystone, glance, nova, and cinder are fine
[05:31] <zradmin> freeflying: correct
[05:32] <zradmin> freeflying: only neutron and horizon are currently not working... horizon is because of neutron though :)
[05:33] <freeflying> zradmin, any log from the node
[05:33] <zradmin> which node
[05:34] <freeflying> on your quantum
[05:34] <zradmin> http://pastebin.ubuntu.com/6477270/, http://pastebin.ubuntu.com/6477293/
[05:37] <freeflying> zradmin, you need login to the node, check it under /var/log/neutron
[05:37] <freeflying> zradmin, also make sure your can connect to mysql from quantum node
[05:38] <zradmin> freeflying: ok workin on it
[05:38] <zradmin> freeflying: in var/log/neutron which agent do you want
[05:39] <freeflying> zradmin, just some clues for your debugging, you need check from there
[05:42] <zradmin> freeflying: ok trying to connect to mysql its telling me mysql is not installed on the quantum node
[05:43] <freeflying> zradmin, by default no mysql client, you need install it
[05:44] <zradmin> freeflying: ok just wanted to make sure I wouldnt mess something up by installing it
[05:44] <freeflying> zradmin, no
[05:45] <zradmin> freeflying: yes, i can connect to the mysql server from the quantum node
[05:49] <zradmin> freeflying: ok... so there is a quantum database/user in mysql, but no neutron database or user as specified in the nova-ccc charm
[05:56] <davecheney> win4
[06:03] <zradmin> davecheney: wasnt there a hook that rewrote all of the quantum references to neutron?
[06:04] <davecheney> zradmin: i'm not sure
[06:04] <davecheney> i'd have to check
[06:10] <zradmin> davecheney: ok its hooks/charmhelpers/contrib/openstack/neutron.py - it looks like most everything from nova-compute/cc and quantum-gw is installing neutron properly... but the database/users created are still named for quantum
[06:13] <davecheney> zradmin: cool
[06:13] <davecheney> so we've eliminated mysql
[06:13] <davecheney> it's down to quantum-gateway charm
[06:16] <freeflying> zradmin, in grizzly, its call neutron already in mysql
[06:17] <zradmin> freeflying: all my grizzly deploys stood up with quantum
[06:19] <freeflying> zradmin, interesting, here it s neutron :)
[06:21] <zradmin> freeflying: what were you using for the source/openstack-origin?
[06:21] <zradmin> freeflying: mine was cloud:precise-grizzly
[06:21] <freeflying> zradmin, then one from cloud archive
[06:25] <zradmin> freeflying: same here
[06:34] <zradmin> davecheney: in the next couple of days I'll stand up a second mysql on a single node and try to attach quantum-gw to that and see what database it creates. thanks for all the help tonight
[06:35] <davecheney> zradmin: you're welcome
[06:35] <davecheney> thanks for your persistance
[06:35] <zradmin> davecheney: hopefully I'll have something to help focus the investigation further :)
=== CyberJacob|Away is now known as CyberJacob
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
=== CyberJacob is now known as CyberJacob|Away
[11:33] <Spacemonkey> Greetings, fellow jujulians. Er, jujumians. Ah, oh whatever.
=== CyberJacob|Away is now known as CyberJacob
[11:33] <Spacemonkey> Anybody here know how to share juju setups for AWS across multiple machines?
[11:34] <Spacemonkey> Looks like just copying the .juju folder is not enough, my developers can create/deploy/destroy but not ssh.
[11:35] <davecheney> Spacemonkey: yes, they all need to share the same ~/.ssh/id_rsa
[11:36] <Spacemonkey> Oh, now that's going to get complicated. LOL
[11:37] <Spacemonkey> Ok thanks dave, glad to hear what I need to do.
[11:38] <davecheney> Spacemonkey: the back story is when you bootstrap the ssh key of the person that bootstrapped it copied to the ubuntu user on all the workstations
[11:38] <davecheney> err, not workstations
[11:38] <Spacemonkey> How about using juju for different aws accounts? Do I just create a second environment, or do I have to swap out .juju folders every time I switch?
[11:38] <davecheney> machines in the environment
[11:38] <davecheney> Spacemonkey: thre are a few optoins
[11:38] <davecheney> one is to set $JUJU_HOME
[11:38] <davecheney> which does the same as switching .juju
[11:38] <davecheney> the second is you can put your AWS creds into your .juju/environments.yaml
[11:39] <davecheney> rather than letting them flow in from your environment vars
[11:40] <Spacemonkey> Ok thanks that makes sense. Perhaps $JUJU_HOME will be the easiest option for me, and just have .juju-personal and .juju-work and .juju-client1 and so on.
[11:41] <Spacemonkey> My ssh key is being used for a bazillion things though - can I just swap out the ssh keys on all running instances for the ubuntu user?
[11:44] <davecheney> Spacemonkey: you can also give the path to the ssh key in your environments.yaml
[11:44]  * davecheney checks that
[11:45] <Spacemonkey> I see access-key and secret-key, but nothing about ssh keys there.
[11:46] <davecheney> actually, scrach that
[11:46] <davecheney> i don't know what i was thinking
[11:47] <Spacemonkey> Dang you got me all excited.
[11:53] <Spacemonkey> Ok thanks for helping me figure this out dave.
=== Guest46324 is now known as med_
=== med_ is now known as medberry
[12:35] <marcoceppi> Spacemonkey: ssh-authorized-keys is what youre looking for
[12:35] <marcoceppi> you can only set it prior to deployment though
[12:36] <marcoceppi> it's not really documented anywhere though
=== gary_poster|away is now known as gary_poster
[14:10] <gnuoy> jamespage, sorry to hassle you but do you have an idea when you might get a chance to look again at https://code.launchpad.net/~gnuoy/charms/precise/quantum-gateway/external-nets/+merge/194153 ? I'm switching projects at then end of the week and it'd be useful to know if it's likely to be re-reviewed before then
[14:18] <jamespage> gnuoy, this week
[14:19] <gnuoy> thanks
=== teknico_ is now known as teknico
[15:30] <Spacemonkey> marcoceppi: aah-authorized-keys? Is that in environments.yaml somewhere or? Thanks for pointing me in this direction though.
=== CyberJacob is now known as CyberJacob|Away
=== CyberJacob|Away is now known as CyberJacob
[16:48] <jcastro> <-- lunch
[19:20] <dpb1> Hi -- how do I import a juju environment onto another machine now?  with pyjuju I could just connect, now I seem to need a CA certificate in the environment configuration?  Do I need to copy the .jenv file out of band?
[19:35] <dpb1> hazmat: do you know ^?
=== thomi_ is now known as thomi
=== Kyle_ is now known as Kyle
[20:25] <hazmat> dpb1, jenv has all you need
[20:25] <hazmat> dpb1, its meant as distribution format for sharing, captures everything into a single file (certs, conn details, etc)
[20:25] <dpb1> hazmat: ok.  That is the "official" way to do it?  or is there a command?
[20:26] <dpb1> ok
[20:26] <hazmat> dpb1, at the moment just share the file afaik, you also need a skeleton in env.yaml, but cli pulls preferentially from jenv over env.yaml config
[20:26] <hazmat> actually not sure about the skeleton haven't tried it..
[20:26]  * hazmat tries
[20:27] <dpb1> hazmat: interesting...  the intersection and discrepencies between these files is confusing, as we already talked about.
[20:28] <hazmat> dpb1, yeah it is.. so environments.yaml has to exist, but that its.. doesn't currently need any reference to the actual env that's in jenv.
[20:29] <dpb1> wow
[20:29] <dpb1> so is the plan to migrate away from environments.yaml?
[20:29] <hazmat> ie mkdir jhome && export JUJU_HOME=jhome && cp /old/home/environments/manual.jenv $JUJU_HOME/environments && juju init && juju status -e manual works.. even though there's no 'manual'  in environments.yaml
[20:30] <hazmat> kinda of funky
[20:30] <hazmat> dpb1, not that i'm aware of re migration
[20:30] <dpb1> ok
[20:30] <hazmat> dpb1, this is more cache and share afaics
[20:30] <dpb1> til: don't rely on environments.yaml so much. :)
[20:31] <hazmat> dpb1, well the issue is when the cache gets stale.. destroy-enviroment clears out the corresponding jenv.. but more importantly
[20:31] <hazmat> environment.yaml is not live data anymore,
[20:31] <hazmat> updates must go through get-env/set-env
[20:32] <dpb1> ya, that is also new to me...  the file is misleading for a pyjuju user for sure.  Maybe it doesn't matter for new users.
[20:33] <InformatiQ> what's up
=== CyberJacob is now known as CyberJacob|Away
[21:43] <cmark> Does juju support moving a bootstrap node from one machine to another?  e.g. for fail-over/high-availability
=== medberry is now known as med_
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
[23:56] <freeflying> cmark, not at this moment