/srv/irclogs.ubuntu.com/2013/11/26/#juju.txt

=== freeflying_away is now known as freeflying
zradminnegronji: odd I cant get it to work with that at all... it never seems to create the database in mysql00:38
zradminnegronji: I get the following in the debug log when trying to set it up "This charm doesn't know how to handle 'shared-db-relation-joined'00:39
zradminnegronjl: is there updated documentation anywhere? I'm still following the guide at https://wiki.ubuntu.com/ServerTeam/OpenStackHA00:42
negronjlzradmin, which charm is giving you the  error ?00:43
zradminnegronl: quantum-gateway when I'm connecting it to the mysql charm... I finish connecting it to rabbitmq and nova-ccc but the service never seems to come online either, i just get 503 errors00:45
zradminnegronjl: sorry i keep mispelling your username :)00:46
negronjlzradmin, no worries ...00:46
negronjlzradmin: that's interesting ... i don't get those issues at all ...00:46
negronjlzradmin: I assume that you are trying to deploy this thing in ha correct ?00:47
zradmincorrect00:47
sarnoldzradmin: (most irc clients let you type or two letters of a nickname then hit 'tab' to finish it up :)00:47
negronjlzradmin, let me check and do some digging00:47
zradminsarnold: sweet, thx for the tip00:47
negronjlzradmin: can you pastebin your relations and your config?  Maybe i can find something odd there.00:49
negronjlzradmin, how are you relating mysql to quantum ?00:52
zradminnegronjl: http://pastebin.ubuntu.com/6476662/00:52
negronjlzradmin: juju add-relation quantum-gateway mysql-hacluster ?00:52
negronjlzradmin, Are you using juju-deployer ?00:52
negronjlzradmin: it looks like it00:53
zradminnegronjl: following the guide, its this command :juju add-relation quantum-gateway mysql00:53
zradminnegronjl: nope just did a quick export out of juju gui to get the relations and the rest is all my local.yaml file00:53
negronjlzradmin, ok00:54
* negronjl reads00:54
negronjlzradmin, what revision of quantum-gateway are you using ?00:57
negronjlzradmin, the error you are getting tells me that your quantum-gateway charm is either broken or out of date00:58
zradminnegronjl: i am pullig directly from the charm store... v11 i think?01:02
negronjlzradmin, Also change your oopenstack-origin to "openstack-origin: cloud:precise-havana/updates"01:02
zradminnegronjl: ok im making the edits now01:06
negronjlzradmin, juju remove-relation quantum-gateway mysql and then juju add-relation quantum-gateway mysql01:08
negronjlzradmin, just to retry the thing ( and hope )01:08
zradminnegronjl: I'm actually redeploying the quantum charm from scratch just so its clean01:09
negronjlzradmin: ok01:10
negronjlzradmin, I can deploy it all on my testing environment01:29
negronjlzradmin, I'll check back here later but, for now, I have to go.01:29
=== context_ is now known as context
zradminnegronjl: thanks for the help, I just finished getting it connected to mysql and this was the log file http://pastebin.ubuntu.com/6476811/ - seems to have the same error01:44
davecheneyzradmin: how did you relate to mysql:shared-db01:46
davecheneythere is no relation called that01:46
davecheneyor no implemention for that relation in the charm01:46
zradminjust running the juju add-relation quantum-gateway mysql command01:47
davecheneyoh dear01:47
davecheneywhich version of the mysql charm have you deployed01:47
zradminmysql-2901:49
davecheneyzradmin: ok, while I look into this01:50
davecheneyyou should remove that relationship01:50
davecheneyand do01:50
davecheneyjuju add-relation quantum-gateway mysql:db01:50
zradminok I'll give that a shot01:52
davecheneyzradmin: mysql-29 implements shared-db-relation01:56
davecheneyi think you may have an old versoin of the charm somehow01:56
davecheneyeven so, you probvably don't want to relate to that relation endpoint01:57
davecheneyyou want mysql:db01:57
zradminis there any updated documentation for setting up the test environment?01:57
zradminit seems like the publicly available wikis are very out of date01:58
davecheneyzradmin: do you mean the local provider ?02:01
zradmindavecheney: so when i try and run juju add-relation quantum-gateway mysql:db it tells me "ERROR no relations found"02:02
davecheneyzradmin: my mistake, please trye02:02
zradminmysql:shared-db?02:03
zradminlooking at all the other charm relations that seems to be what they're connecting to02:03
davecheneyyeah, just checked out quantum-gateway02:03
davecheneyok, i thnk were; back to mysql02:04
davecheneyit looks like mysql-29 implements shared-db properly02:04
davecheneyi cannot explain why you deployment of that charm is acting weirdly02:04
davecheneyhow did you deploy the mysql charm ?02:04
zradminjuju deploy --config local.yaml -n 2 mysql02:05
zradminthen juju deploy hacluster mysql-hacluster02:05
zradminadded the relationship to ceph and then to the mysql-hacluster02:06
davecheneyzradmin: i'm sorry, i'm out of ideas02:06
davecheneyif that is mysql-29 then it shold implemnt shared-db02:06
zradminyeah i'm fairly confused as well02:08
zradminis there a way to manually retrigger that specific hook?02:26
davecheneyzradmin: is the unit in error ?03:15
zradminnope.... thats the frustrating part - everything is started03:16
davecheneyzradmin: i'm sorry, there is no way to do this if the unit is not in error03:17
davecheneyyou could tru03:17
davecheneyjuju resolved --retry $UNIT03:17
davecheneybut I suspect it will tell you thre is nothing to do03:17
zradminyeah it tells me the unit is not in an error state so it wont do anything03:18
zradminI've tried destroying the relationship a few times and rebuilding it since then... but each time it still says that mysql doesnt understand the shared-db relationship03:19
zradminwhich is pretty strange considering that glance/nova/keystone etc. all have databases and are working fine03:19
davecheneyzradmin: can you do juju status $MYSQL_SERVICE03:20
zradminhere you go http://pastebin.ubuntu.com/6477065/03:22
davecheneyzradmin: could you get me an ls -al of the charm hooks dir03:29
davecheneyfrom memory it's somethignlike03:29
davecheney/var/lib/juju/agents/unit-0/hooks03:29
davecheneyit'll be on machine 403:29
zradminhttp://pastebin.ubuntu.com/6477095/03:31
davecheneyzradmin: hmm, truncated but looks right03:33
davecheneyon that same machine can you have a look in /var/lib/juju/unit-0.log03:33
davecheneyfor the bit about 'charm doesn't implement hook "shared-db-relation-joined" [sic]03:34
zradminhere it is http://pastebin.ubuntu.com/6477110/03:39
davecheneyzradmin: aah03:40
davecheneyi think that might have been a red herrin03:41
davecheneythe real issue appears to be03:41
davecheney2013-11-26 02:35:45 INFO juju juju-log.go:66 mysql/0 shared-db:71: MySQL service is peered, bailing shared-db relation as this service unit is not the leader03:41
davecheneyin fact, i don't think that is the error03:41
davecheneyis an error03:41
davecheneyif you look at the log on mysql/103:41
davecheneyyou'll see (this time) it was the leader03:41
zradminhmmm i thought so to at first but on the quantum-gw i cant find anything in neutron.conf relating to the mysql db backend03:43
zradminor api-paste03:43
zradminso this is what i get when i try and run a neutron command at the moment... http://pastebin.ubuntu.com/6477139/03:54
zradminthe neutron.conf file on the gateway only has the rabbit host defined... but the neutron.conf on the nova-ccc has a sqllite database specified for some reason.... its the only reference to a database for neutron i can find. http://pastebin.ubuntu.com/6477146/03:58
=== freeflying is now known as freeflying_away
zradmindavecheney: I've tried standing up this environment 4 or 5 times since the havana release came out and i run into this wall each time.... its driving me crazy :)04:12
davecheneyzradmin: i'm really sorry to hear that04:13
davecheneythis is supposed to be easy04:13
davecheneydid you find anything in the mysql/1 log ?04:13
zradmini see it setting up nova.... and it is populating a quantum password field04:16
zradminbut i also have the 2013-11-26 02:35:42 INFO juju juju-log.go:66 mysql/1 shared-db:71: This charm doesn't know how to handle 'shared-db-relation-joined'.04:17
zradmindavecheney: yeah i love the power of juju and its much easier setting it up than tooling everything by hand... i just cant seem to figure out why those pices are giving me trouble. I do appreciate the help for sure!04:18
davecheneyzradmin: thanks for the log04:21
davecheneythat error isn't an error04:21
davecheneysee the prevoius discussion04:21
davecheney*but* does mysql/1 also say that it is bailing out because it isn't the leader ?04:21
zradminnope no bailing that i can see04:22
davecheneyzradmin: ok, here is what I thin is happening04:22
davecheney1. quantum-gateway is related to mysql:shared-db04:22
davecheneymysql is supposed to set some relation settings back to qyuantum-gw04:22
davecheneywhich in turn will make it switch from sqlite to using that mysql db04:23
davecheneybut mysql isn't sending any relation settings (dunno why yet)04:23
davecheneyand that is hwy quantum is using sqllite04:23
zradminand not bringing the api online....04:24
davecheneyit is possible that a change of relation hooks have to run to get things properly configured04:24
zradminit should be logged in the /hooks directory right?04:24
davecheneyno, nothing is logged in the hooks directory04:24
davecheneytry removing and readding the relationship04:25
zradminby logged... i meant the config its trying to push04:25
davecheneyzradmin: hmm,, not really04:26
davecheneyrelation settings are only visible to hook commands via the relation-get hook command04:26
zradminis that outside of the normal juju cli?04:29
davecheneyyeah, there is no way to see relation settings unless youre using debug hooks in a hook context04:30
zradminsame issue with removing and readding the relationship04:31
zradminah so the unit would have to be stuck on a hook in order to see what its trying to do04:32
davecheneyzradmin: i'm not sure stuck is the right thing to say04:34
davecheneybut i'd guess that both mysql units think they are not hte master04:35
davecheneyso *neither* are taking action and setting the relation settings to quantum-gw04:35
davecheneycan you past the log of mysql/1 ?04:35
zradminyeah I'll grab the end of it04:36
davecheneyta04:36
zradminhttp://pastebin.ubuntu.com/6477255/04:37
zradmini know mysql is only running on /1 currently so it should be the master04:38
davecheneyok, so i can see it doing relation set04:38
davecheneywhat does the quantum-gw log say ?04:38
zradminthis looks like the relevant section here.... says it is missing the quantum/nova passwords04:41
zradminhttp://pastebin.ubuntu.com/6477270/04:41
=== freeflying_away is now known as freeflying
zradminand the database_host information apparently04:46
davecheneyzradmin: ahh04:47
davecheneyright04:47
davecheneyhere is what is going on04:47
davecheney2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "quantum_password" "mysql/0"]04:47
davecheney2013-11-26 02:35:49 DEBUG worker.uniter.jujuc server.go:109 hook context id "quantum-gateway/1:shared-db-relation-changed:4235752611937240132"; dir "/var/lib/juju/agents/unit-quantum-gateway-1/charm"04:47
davecheney2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "nova_password" "mysql/0"]04:47
davecheney^ quantum has decided that ONLY mysql/0 has the relation data it wants04:47
davecheney*but* mysql/1 is the unit which provided the relation data04:48
zradminthat makes alot of sense04:48
* davecheney disappears down a charm helpers rabbit hole04:49
davecheneyzradmin: short version04:49
zradmindoesnt it try and get itt from /1 as well though?04:49
davecheneydeploy one mysql unit04:49
zradmin2013-11-26 02:35:49 INFO worker.uniter.jujuc server.go:108 running hook tool "relation-get" ["--format=json" "-r" "shared-db:71" "db_host" "mysql/1"]04:49
davecheneyand i'll work04:49
davecheneyyeah04:49
davecheneyok04:49
davecheneythat is weird04:49
davecheneyso what is after line 3304:49
zradminnext batch http://pastebin.ubuntu.com/6477293/04:51
zradminlooks like its just writing the config files04:51
zradminchecked the nova.conf on the host.... and it has its db line sql_connection=mysql://nova:vSk0czUGLt20mXP1@10.10.32.4/nova04:52
davecheneyzradmin: ok, do you want to wait til the unit reaches swtarted state04:54
davecheneyall these logs are truncated04:54
zradminis there a way to get more verbosity?04:56
davecheneyzradmin: most people complain they are too verbose :)04:59
davecheneythat's all you get i'm afraid04:59
davecheneyis the unit in started state ?04:59
zradminyeah all of them are04:59
davecheneyright05:00
davecheneydoes quantum-gw work ?05:01
davecheneyit looks like it's missing a restart action somewhere05:01
zradminnope i just get that 503 service not found error everytime i try and query neutron05:03
davecheneyzradmin: i think quantum-gw runs under upstart05:06
davecheneycan you go to the unit and do a service restart05:06
davecheneyon the service05:06
davecheneymaybe that will get it going05:06
zradminok that services doesnt exist on the quantum-gateway nodes05:08
zradminlooks like its called neutron-server and exists on the nova-ccc05:09
zradmindavecheney: i restarted the quantum-gw nodes... but still no dice05:18
=== philipballew is now known as philip
freeflyingzradmin, is yours ha deployment?05:27
zradminfreeflying: yup been following the ha guide posted here: https://wiki.ubuntu.com/ServerTeam/OpenStackHA05:28
zradminadjusted for havana of course05:29
freeflyingzradmin, does keystone work properly05:29
freeflyingzradmin, also 10.10.32.4 is the vip of your mysql cluster?05:31
zradminfreeflying: yup, keystone, glance, nova, and cinder are fine05:31
zradminfreeflying: correct05:31
zradminfreeflying: only neutron and horizon are currently not working... horizon is because of neutron though :)05:32
freeflyingzradmin, any log from the node05:33
zradminwhich node05:33
freeflyingon your quantum05:34
zradminhttp://pastebin.ubuntu.com/6477270/, http://pastebin.ubuntu.com/6477293/05:34
freeflyingzradmin, you need login to the node, check it under /var/log/neutron05:37
freeflyingzradmin, also make sure your can connect to mysql from quantum node05:37
zradminfreeflying: ok workin on it05:38
zradminfreeflying: in var/log/neutron which agent do you want05:38
freeflyingzradmin, just some clues for your debugging, you need check from there05:39
zradminfreeflying: ok trying to connect to mysql its telling me mysql is not installed on the quantum node05:42
freeflyingzradmin, by default no mysql client, you need install it05:43
zradminfreeflying: ok just wanted to make sure I wouldnt mess something up by installing it05:44
freeflyingzradmin, no05:44
zradminfreeflying: yes, i can connect to the mysql server from the quantum node05:45
zradminfreeflying: ok... so there is a quantum database/user in mysql, but no neutron database or user as specified in the nova-ccc charm05:49
davecheneywin405:56
zradmindavecheney: wasnt there a hook that rewrote all of the quantum references to neutron?06:03
davecheneyzradmin: i'm not sure06:04
davecheneyi'd have to check06:04
zradmindavecheney: ok its hooks/charmhelpers/contrib/openstack/neutron.py - it looks like most everything from nova-compute/cc and quantum-gw is installing neutron properly... but the database/users created are still named for quantum06:10
davecheneyzradmin: cool06:13
davecheneyso we've eliminated mysql06:13
davecheneyit's down to quantum-gateway charm06:13
freeflyingzradmin, in grizzly, its call neutron already in mysql06:16
zradminfreeflying: all my grizzly deploys stood up with quantum06:17
freeflyingzradmin, interesting, here it s neutron :)06:19
zradminfreeflying: what were you using for the source/openstack-origin?06:21
zradminfreeflying: mine was cloud:precise-grizzly06:21
freeflyingzradmin, then one from cloud archive06:21
zradminfreeflying: same here06:25
zradmindavecheney: in the next couple of days I'll stand up a second mysql on a single node and try to attach quantum-gw to that and see what database it creates. thanks for all the help tonight06:34
davecheneyzradmin: you're welcome06:35
davecheneythanks for your persistance06:35
zradmindavecheney: hopefully I'll have something to help focus the investigation further :)06:35
=== CyberJacob|Away is now known as CyberJacob
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
=== CyberJacob is now known as CyberJacob|Away
SpacemonkeyGreetings, fellow jujulians. Er, jujumians. Ah, oh whatever.11:33
=== CyberJacob|Away is now known as CyberJacob
SpacemonkeyAnybody here know how to share juju setups for AWS across multiple machines?11:33
SpacemonkeyLooks like just copying the .juju folder is not enough, my developers can create/deploy/destroy but not ssh.11:34
davecheneySpacemonkey: yes, they all need to share the same ~/.ssh/id_rsa11:35
SpacemonkeyOh, now that's going to get complicated. LOL11:36
SpacemonkeyOk thanks dave, glad to hear what I need to do.11:37
davecheneySpacemonkey: the back story is when you bootstrap the ssh key of the person that bootstrapped it copied to the ubuntu user on all the workstations11:38
davecheneyerr, not workstations11:38
SpacemonkeyHow about using juju for different aws accounts? Do I just create a second environment, or do I have to swap out .juju folders every time I switch?11:38
davecheneymachines in the environment11:38
davecheneySpacemonkey: thre are a few optoins11:38
davecheneyone is to set $JUJU_HOME11:38
davecheneywhich does the same as switching .juju11:38
davecheneythe second is you can put your AWS creds into your .juju/environments.yaml11:38
davecheneyrather than letting them flow in from your environment vars11:39
SpacemonkeyOk thanks that makes sense. Perhaps $JUJU_HOME will be the easiest option for me, and just have .juju-personal and .juju-work and .juju-client1 and so on.11:40
SpacemonkeyMy ssh key is being used for a bazillion things though - can I just swap out the ssh keys on all running instances for the ubuntu user?11:41
davecheneySpacemonkey: you can also give the path to the ssh key in your environments.yaml11:44
* davecheney checks that11:44
SpacemonkeyI see access-key and secret-key, but nothing about ssh keys there.11:45
davecheneyactually, scrach that11:46
davecheneyi don't know what i was thinking11:46
SpacemonkeyDang you got me all excited.11:47
SpacemonkeyOk thanks for helping me figure this out dave.11:53
=== Guest46324 is now known as med_
=== med_ is now known as medberry
marcoceppiSpacemonkey: ssh-authorized-keys is what youre looking for12:35
marcoceppiyou can only set it prior to deployment though12:35
marcoceppiit's not really documented anywhere though12:36
=== gary_poster|away is now known as gary_poster
gnuoyjamespage, sorry to hassle you but do you have an idea when you might get a chance to look again at https://code.launchpad.net/~gnuoy/charms/precise/quantum-gateway/external-nets/+merge/194153 ? I'm switching projects at then end of the week and it'd be useful to know if it's likely to be re-reviewed before then14:10
jamespagegnuoy, this week14:18
gnuoythanks14:19
=== teknico_ is now known as teknico
Spacemonkeymarcoceppi: aah-authorized-keys? Is that in environments.yaml somewhere or? Thanks for pointing me in this direction though.15:30
=== CyberJacob is now known as CyberJacob|Away
=== CyberJacob|Away is now known as CyberJacob
jcastro<-- lunch16:48
dpb1Hi -- how do I import a juju environment onto another machine now?  with pyjuju I could just connect, now I seem to need a CA certificate in the environment configuration?  Do I need to copy the .jenv file out of band?19:20
dpb1hazmat: do you know ^?19:35
=== thomi_ is now known as thomi
=== Kyle_ is now known as Kyle
hazmatdpb1, jenv has all you need20:25
hazmatdpb1, its meant as distribution format for sharing, captures everything into a single file (certs, conn details, etc)20:25
dpb1hazmat: ok.  That is the "official" way to do it?  or is there a command?20:25
dpb1ok20:26
hazmatdpb1, at the moment just share the file afaik, you also need a skeleton in env.yaml, but cli pulls preferentially from jenv over env.yaml config20:26
hazmatactually not sure about the skeleton haven't tried it..20:26
* hazmat tries20:26
dpb1hazmat: interesting...  the intersection and discrepencies between these files is confusing, as we already talked about.20:27
hazmatdpb1, yeah it is.. so environments.yaml has to exist, but that its.. doesn't currently need any reference to the actual env that's in jenv.20:28
dpb1wow20:29
dpb1so is the plan to migrate away from environments.yaml?20:29
hazmatie mkdir jhome && export JUJU_HOME=jhome && cp /old/home/environments/manual.jenv $JUJU_HOME/environments && juju init && juju status -e manual works.. even though there's no 'manual'  in environments.yaml20:29
hazmatkinda of funky20:30
hazmatdpb1, not that i'm aware of re migration20:30
dpb1ok20:30
hazmatdpb1, this is more cache and share afaics20:30
dpb1til: don't rely on environments.yaml so much. :)20:30
hazmatdpb1, well the issue is when the cache gets stale.. destroy-enviroment clears out the corresponding jenv.. but more importantly20:31
hazmatenvironment.yaml is not live data anymore,20:31
hazmatupdates must go through get-env/set-env20:31
dpb1ya, that is also new to me...  the file is misleading for a pyjuju user for sure.  Maybe it doesn't matter for new users.20:32
InformatiQwhat's up20:33
=== CyberJacob is now known as CyberJacob|Away
cmarkDoes juju support moving a bootstrap node from one machine to another?  e.g. for fail-over/high-availability21:43
=== medberry is now known as med_
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
freeflyingcmark, not at this moment23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!