/srv/irclogs.ubuntu.com/2016/04/22/#juju.txt

=== redir is now known as redir_eod
=== markthomas_ is now known as markthomas
=== urulama|____ is now known as urulama
=== matthelmke is now known as matthelmke-afk
=== matthelmke is now known as matthelmke-afk
=== rogpeppe2 is now known as rogpeppe
jamespagegnuoy, doing sweepups from yesterdays release - bugs targetted for 16.04 marked fix released; all others bumped to 16.0708:29
gnuoyjamespage, fantastic, ta08:29
ReSamnice job on the recent release! how do I get the new charms with xenial support to show up in my juju-gui search?08:37
jamespageReSam,  I think that should just work...08:37
* jamespage looks08:37
ReSamjamespage: I'm only seeing ceph for trusty - but if I search on the jujucharms.com site, I find the new charm for all supported ubuntu versions08:38
jamespagerogpeppe, hey - can you do a promulgation for me?  rabbitmq-server owned by openstack-charmers - superceeds the ~charms ones08:38
jamespageReSam, might just be a search issue - https://jujucharms.com/ceph/xenial08:38
jamespageits there :-)08:38
jamespagerogpeppe, ~charmers one rather...08:38
ReSamyes - on jujucharms.com --- but not in my local juju-gui search...08:39
ReSamor do I need to update the catalog or something?08:39
ReSamI'm running juju2beta5 from the devel-ppa08:39
jamespageReSam, not sure tbh - not actually used that feature :-)08:40
jamespageReSam, oh thats in the juju-gui itself? urulama might be able to help there...08:41
jamespagemorning urulama :-)08:41
ReSamjamespage: yes - my locally installed juju-gui, which shipped since juju2beta4 I think08:42
jamespageReSam, sorry - long day yesterday - brain not quite functional just yet :-)08:42
ReSamjamespage: np - I'm really thankful for your hard work - makes my life a lot easier -- hopefully :)08:43
jamespageReSam, well let us know if that's not the case :-)08:43
ReSamf I search for "cs:xenial/ceph-0" I can find it - but not if I only search "ceph", then only the trusty charm shows up08:43
jamespageReSam, https://wiki.ubuntu.com/ServerTeam/OpenStackCharms/ReleaseNotes160408:43
jamespagemight be useful for you if you've not already seen them...08:44
ReSamjamespage: thanks - already skimmed through it.08:44
urulamajamespage, ReSam: morning ... OTP, be back in 10min08:46
ReSamjamespage: I'm having problems deploying new charms:08:51
ReSam ERROR juju.worker.dependency engine.go:526 "metric-collect" manifold worker returned unexpected error: failed to read charm from: /var/lib/juju/agents/unit-ceph-21/charm: stat /var/lib/juju/agents/unit-ceph-21/charm: no such file or directory08:51
ReSamthe unit us stuck at: "Waiting for agent initialization to finish"08:51
rogpeppejamespage: sorry, was in a call08:59
rogpeppejamespage: i don't think i have privs to promulgate09:00
jamespagerogpeppe, no problem :-)09:00
jamespagerogpeppe, uh - urulama pointed me at you yesterday...09:00
jamespageincase marcoceppi was not around...09:00
rogpeppejamespage: i can give you a command that you can execute to do it if you have privs09:00
urulamarogpeppe: just to provide the bhttp instructions, jamespage has charmers rights, i believe :)09:01
rogpeppeurulama: will do09:01
urulamaReSam: looking09:01
rogpeppejamespage: first: go get github.com/rogpeppe/bhttp09:02
rogpeppejamespage: (assuming you've got a go env set up)09:02
rogpeppejamespage: then bhttp put -j https://api.jujucharms.com/charmstore/v5/$charmid/promulgate Promulgate:=true09:02
urulamaReSam: indeed, series are missing. that's a regression :-/09:02
urulamaReSam: i suggest in gui you search for "ceph xenial" ... first hit will provide you xenial charm09:03
ReSamurulama: thanks - yes I already found it - still a bit confusing.09:03
urulamaReSam: it is09:04
urulamafiling a bug09:04
ReSamalso xenial is missing in the dropdown menu for "series" in the search view09:04
urulamaall series information is missing09:04
ReSamurulama: any idea about my other problem: "stat /var/lib/juju/agents/unit-ceph-21/charm: no such file or directory" ?09:05
urulamaReSam: that is after deploying the charm?09:06
ReSamyes09:06
urulamafor that i'll have to redirect you to jamespage and the openstack guys09:06
=== dooferlad_ is now known as dooferlad
ReSamjamespage: maybe it is proxy related? my machines do not have direct internet access...09:11
jamespageurulama, I am a charmer yes...09:23
jamespagerogpeppe, how do I resolve the charm id?09:24
jamespagesorry being dumb09:24
jamespageReSam, hmm - that looks familiar09:24
rogpeppejamespage: what charm are you trying to promulgate?09:24
jamespageare there erros in the machine log on the unit09:25
ReSamjamespage: on all my machines in this file: /var/log/juju/unit-ceph-21.log (with different ids of course)09:25
jamespagerogpeppe, https://jujucharms.com/u/openstack-charmers/rabbitmq-server09:26
urulamajamespage: you want to move if from ~charmers to ~openstack-charmers space?09:27
jamespageurulama, yes please09:27
urulamarogpeppe: i think you'll have to explain how to build bhttp as well09:27
rogpeppeurulama: i did, i think09:28
rogpeppejamespage: https://api.jujucharms.com/charmstore/v5/openstack-charmers/rabbitmq-server/promulgate09:28
jamespagerogpeppe, awesome tat09:28
rogpeppejamespage: i think09:28
jamespagerogpeppe, gotach on the build..09:28
jamespagemy go foo is good enough for that these days09:28
rogpeppejamespage: caveat: i haven't actually tried this command for real, 'cos i don't have promulgate privs :)09:28
rogpeppejamespage: but it *should* work :)09:28
jamespagerogpeppe, needed a ~ for the team name09:30
rogpeppejamespage: oops, good point!09:30
jamespagerogpeppe, hmm I might need to unpromulgate the charmers one first...09:30
urulamajamespage: no need09:31
urulamajamespage: charmstore does the switch09:31
jamespageurulama, hmm09:31
* urulama corrects that ... *should do the switch*09:31
jamespageurulama, now I broke it all...09:32
urulama?09:33
jamespageurulama, https://jujucharms.com/rabbitmq-server/09:33
jamespagenot found...09:33
urulamajamespage: what was the bhttp command that you used?09:34
jamespageurulama, bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/rabbitmq-server/promulgate Promulgate:=true09:34
jamespageurulama, I also did a09:34
jamespagebhttp put -j https://api.jujucharms.com/charmstore/v5/~charmers/rabbitmq-server/promulgate Promulgate:=false09:34
jamespagemaybe that was bad of me...09:35
jamespagegnuoy, do we still need todo the hacluster charm -> charm store ?09:36
urulamalet's try bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/xenial/rabbitmq-server/promulgate Promulgate:=true09:36
gnuoyjamespage, looks like its still awaiting a merge https://code.launchpad.net/~gnuoy/charms/trusty/hacluster/1604/+merge/292493 ...09:37
urulamarogpeppe: is'nt it "Promulgate=true" not "Promulgate:=true ?09:37
rogpeppeurulama: nope09:37
urulamaand Promulgated:=True not Promulgate09:37
urulamajamespage: bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/xenial/rabbitmq-server/promulgate Promulgated:=true09:38
simonklbanyone have had any problems with LXD containers not getting the hostname set in /etc/hosts ?09:38
jamespageurulama, ok that worked...09:39
jamespageurulama, still showing as owned by charmers in the UI, but its the right charm...09:39
urulamajamespage: yes, wrong owner resolution. charmstore shows proper one https://api.jujucharms.com/charmstore/v5/rabbitmq-server/meta/owner09:39
urulamajamespage: fix will be deployed after ODS (don't want to touch production during if not critical)09:40
=== urulama is now known as urulama|school
=== urulama|school is now known as urulama|afk
jamespagegnuoy, ok hacluster merged and direct pushed for trusty and xenial to the charm store09:46
gnuoythanks09:46
jamespagegnuoy, https://jujucharms.com/hacluster/xenial09:46
rogpeppejamespage: ah, cool, it worked09:47
jamespagerogpeppe, yes all good now09:47
rogpeppejamespage: grand09:47
ReSamjamespage: any chance you remember from where my problems looks familiar? is there an open issue or maybe even a solution?09:51
jamespageReSam, it was something todo with stale charms on the controller  - did you happen to use multiple models under juju 2.0?09:51
ReSamjamespage: not that I know of09:51
ReSamcan clean up this somehow? deleting a cache or so?09:52
jamespageReSam, can you check the  /var/log/juju/machine-X.log file please09:54
jamespagerogpeppe, does promugation for bundles work in the same way? our current openstack-base and openstack-telemetry bundles are under ~charmers - I'd like to move those over to ~openstack-charmers and switch the pointer to the promugated bundle ...09:56
rogpeppejamespage: yes, it should do09:56
ReSamjamespage: seems to be ok... lots of "block devices change" messages though...09:56
jamespageReSam, I was looking for something related to hash sum mismatches09:57
jamespagethe machine agent downloads the charm that the unit agent then uses09:57
jamespageis there anything in /var/lib/juju/agents/ ?09:57
ReSamjamespage: no hash mismatch lines.09:58
jamespagehmmm09:58
ReSamjamespage: yes, /var/lib/juju/agents contains a folder for the machine and ceph-unit09:58
ReSamjamespage: including agent.conf's for both09:58
jamespageReSam, whats in var/lib/juju/agents/unit-ceph-2109:59
jamespage?09:59
ReSamboth are also running process. so that all works09:59
ReSamls /var/lib/juju/agents/unit-ceph-21/09:59
ReSamagent.conf  metrics-send.socket  run.socket  state/09:59
ReSamseems my state server has 3 ip addresses - and therefore 3 api endpoints. two of which are unreachable though. so I get 2 errors in the logs when it tries to connect to the wrong endpoint10:00
ReSamI can see correct proxy settings in the log - so that should also be fine10:02
ReSamjamespage: seems one of the downloads is misbehaving: I have this file: /var/lib/juju/agents/unit-ceph-21/state/bundles/downloads/inprogress-377015309 and inprogress-00510161210:04
jamespageReSam, this is a juju problem of some description that I've not seen before - lets see if the juju-dev team know about this...10:05
jamespagedimitern, frobware: either of you two seen anything like this before with juju 2.0 betas ? ^^10:06
frobwarejamespage: reading scrollback10:09
jamespagefrobware, it maybe that the ip address that units are getting for the state server is not one they can actually connect over...10:11
frobwarejamespage: are the units in containers?10:12
jamespagefrobware, I'll have to defer that to ReSam10:16
ReSamfrobware: no - directly in the machine root10:57
ReSamjamespage: yes - so my state server is reachable via 3 interfaces - but only 1 is reachable from the machines. I can see the machine trying to connect to the two others - and get "connection timeout" in the logs. but then I guess the third try succeeds.10:58
=== rvba` is now known as rvba
=== urulama|afk is now known as urulama
=== matthelmke-afk is now known as matthelmke
GaryxHey guys, anyone had any luck bootstrapping MAAS 2.0, I've tried Beta5 and Beta4 and get a runtime error11:50
Garyxpanic: runtime error: invalid memory address or nil pointer dereference11:50
ReSamfrobware: yes - looks like it is using the wrong api endpoint for downloading - although before that is uses the correct one to establish a connection: https://paste.ubuntu.com/15980208/12:23
frobwareReSam: are you using a bundle to deploy? Just wanted to repro locally12:36
frobwareReSam: would it be possible to share and collect some logs? Also, the output of `juju status' and `juju show-machines'12:40
GaryxIs the MAAS 2.0 support still a wip?12:52
ReSamfrobware: no bundle - just: juju deploy cs:xenial/ceph-0 -n 512:56
frobwareReSam: ok that helps. can you share the juju status output?12:57
ReSamfrobware: https://paste.ubuntu.com/15980650/12:57
frobwareReSam: and the machine NIC configuration? how many NICs, VLANs, et al?12:58
ReSamfrobware: https://paste.ubuntu.com/15980661/12:58
ReSamfrobware: machines have active 2 interfaces with the same ip subnet12:59
ReSamstate server has 3 interfaces - only one is connected to the machines (172.24.32.2)12:59
frobwareReSam: and can I get access to the machine-0.log and from the machines too?13:02
ReSamsure13:02
ReSamfrobware: you mean the state server or the machine with ceph?13:03
frobwareReSam: all logs really, would be quicker than for me to keep on asking13:03
=== cmars` is now known as cmars
BrunoR'juju-quickstart bundle.yaml' dies with "juju-quickstart: error: unable to connect to the Juju API server on wss://x.y.z/api: 'module' object has no attribute 'default_timeout" (juju 1.25.5)14:48
=== BlackDex_ is now known as BlackDex
BrunoR'juju-quickstart bundle.yaml' dies with "juju-quickstart: error: unable to connect to the Juju API server on wss://x.y.z/api: 'module' object has no attribute 'default_timeout" (juju 1.25.5)15:17
Makyo_Hi BrunoR - do you wind up with a stack trace from that?15:24
=== Makyo_ is now known as Makyo
BrunoRMakyo: http://paste.ubuntu.com/15983523/ you are welcome15:34
MakyoBrunoR: Thanks, I'm digging into the code, and it looks a bit like a problem with websocket-client.  What version do you get when you run `pip show websocket-client`?15:35
BrunoRMakyo: this shows an error, it looks like this modules is installed via deb-package python-websocket (0.18.0-0ubuntu0.14.04~ppa5 from http://ppa.launchpad.net/juju/stable/ubuntu/)15:41
MakyoBrunoR: ah, alright, thank you.15:42
MakyoBrunoR: may I please see the output from `python -c "import websocket;print dir(websocket)"`?  I'm curious as to why default_timeout is missing from the websocket module.15:45
BrunoRMakyo: http://paste.ubuntu.com/15983924/15:48
=== urulama is now known as urulama|afk
MakyoThat's supremely weird.15:50
MakyoBrunoR: I'll file a bug against quickstart and work on it.  In the mean time, you can try `sudo pip install websocket-client` and see if the version you have is overwritten by the one installed by pip.15:50
Makyo(You might need to do `sudo pip install --upgrade websocket-client` due to the versions being the same)15:51
BrunoRMakyo: pip install --upgrade moved websocket-client from 0.35.0 to 0.37.0 ~ trying `juju-quickstart bundle.yaml` again ~ looks like a problem in deb-package python-webclient?15:55
MakyoBrunoR: yeah, that's what I'm seeing now that I play around with it.  Did you install quickstart from the juju/stable PPA, or from the default repo?15:56
BrunoRMakyo: no, same error15:56
BrunoRMakyo: `$ apt-cache madison juju-quickstart` says 2.2.4+bzr147+ppa42~ubuntu14.04.1 from  http://ppa.launchpad.net/juju/stable/ubuntu/15:58
Odd_Blokeaisrael: Are you on Ubuntu and seeing https://bugs.launchpad.net/cloud-images/+bug/1573058?  Or OS X?16:07
mupBug #1573058: Ubuntu 16.04 current not booting in Vagrant (gurumeditation) <cloud-images:Invalid by daniel-thewatkins> <https://launchpad.net/bugs/1573058>16:07
aisraelOdd_Bloke: OS X. I hadn't refreshed the page before I posted that comment.16:08
Odd_Blokeaisrael: No worries; just making sure there's nothing more we can do. :)16:08
BrunoRMakyo: my google-foo found something similar https://bugs.launchpad.net/juju-quickstart/+bug/148415816:09
mupBug #1484158: juju quickstart fails "unable to connect to the Juju API Server" <quickstart> <vagrant> <websocket> <juju-quickstart:Triaged> <Juju Vagrant images:Fix Released> <https://launchpad.net/bugs/1484158>16:09
MakyoBrunoR: aha, good catch. I16:10
MakyoBrunoR: I'll tag on to that, and see if I can provide a more sensible default when things go wrong like this.16:10
MakyoBrunoR: thank you16:10
Gildeployed flannel units with juju, units are stuck in pending state?  Any pointers on where to look for debug and fix information?  I looked in /var/log/juju but the logs don't seem to have any errors, here are the messages (which keep updating)16:24
Gilchecking flannel/6 for flannel leadership16:24
Gilflannel/6 confirmed for flannel leadership until 2016-04-22 16:25:13.080001473 +0000 UTC16:24
Gilflannel/6 will renew flannel leadership at 2016-04-22 16:24:43.080001473 +0000 UTC16:24
Gilthose three lines repeat in the log every few minutes  - which looks all good afaik16:25
Gilso not sure why flannel showing in pending state...16:25
=== redir_eod is now known as redir
=== mup_ is now known as mup
=== chuck__ is now known as zul
LiftedKiltif I ssh to a machine that is running lxc containers with charms installed, how do I view the containers that are running?17:49
LiftedKiltnothing shows up in lxc list / lxc info17:49
jrwrenLiftedKilt: lxc-ls17:52
LiftedKiltjrwren: what's the difference between lxc list and lxc-ls?18:05
jrwrenLiftedKilt: lxc is the lxd client which talks to the lxd sever.  lxc-ls is lower level lxc (no lxd involved) tool.18:06
LiftedKiltjrwren: gotcha - thanks18:06
BrunoRhow does 'charm push', 'charm publish' works for bundles? I just pushed/published/granted but the bundle does only shows 'partly' up at jujucharms.com? urulama|afk?18:09
BrunoRhow does 'charm push', 'charm publish' works for bundles? I just pushed/published/granted but the bundle does only shows 'partly' up at jujucharms.com? urulama|afk? marcoceppi?18:14
lazyPowe_BrunoR what did you publish? i''m happy to take a look18:21
BrunoRlazyPowe_: https://jujucharms.com/u/3-bruno/ or cs:~3-bruno/bundle/demo-018:23
BrunoRlazyPowe_: according to 'charm show' it should be accessible18:23
freak_hi everyone18:24
freak_i need help regarding instance creation in openstack18:24
freak_i have created my first instance18:24
freak_and it is showing error18:25
freak_http://imgur.com/krBt0H918:25
freak_can you please take a look here is the picture of error msg http://imgur.com/krBt0H918:25
lazyPowe_BrunoR ok, taking a look now18:28
lazyPowe_BrunoR i see your bundle up here...18:29
lazyPowe_6 services, 12 units, on 4 machines?18:29
BrunoRlazyPowe_: yes18:29
lazyPowe_https://jujucharms.com/u/3-bruno/demo/18:29
lazyPowe_it totally in the store then :)18:29
lazyPowe_so i guess i'm not understanding what you're asking "how does push and publish work for bundles?"  it works exactly like it does for charms.  if you upload the bundle and dont set public acl's, it'll only be accessible to you (the uploader)18:30
BrunoRlazyPowe_: I see an image of the services in the bundle but can't reach the README.md or deploy it.18:33
lazyPowe_BrunoR - the README does look truncated, however - http://paste.ubuntu.com/15988639/18:42
lazyPowe_BrunoR i was able to deploy just fine using juju 2.0 beta5, i did not test on 1.25.5 howeer18:42
lazyPowe_*however18:42
cholcombethedac, coreycb could you give freak_ a hand?18:48
thedacI'll take a look18:48
freak_hi thedac18:49
freak_i created my first instance in openstack from horizon dashboard18:49
freak_earlier i created network, image, storage volume,,18:49
freak_then i created instance18:49
freak_but upon startup its showing erro18:49
freak_error18:49
BrunoRlazyPowe_: `$ juju deploy cs:~3-bruno/bundle/demo` yields 'ERROR expected a charm URL, got bundle URL "cs:~3-bruno/bundle/demo-0"'18:50
freak_can you please take a look18:50
freak_http://imgur.com/krBt0H918:50
thedacfreak_: looking at it now. This was juju deployed?18:50
thedacfreak_: can I see the bundle used?18:50
freak_thedac ,  i used this bundle https://jujucharms.com/u/charmers/openstack-base/bundle/4018:53
thedacok. So it is unclear from the image what precisely is failing. Can you grab the novarc and try some things from command line?18:54
thedacI would start with checking juju status and make sure nothing is in error or blocking state18:54
freak_yes i will,,18:54
lazyPowe_BrunoR weird...18:54
freak_here is the status  http://paste.ubuntu.com/15988934/18:55
thedacthanks, looking18:56
freak_here is the novarc  http://paste.ubuntu.com/15988999/18:58
thedacfreak_: so that looks good. If you have novaclient installed you might try nova show $INSTANCE_ID and see if we get any more info. And nova console-log $INSTANCE_ID18:58
thedacfreak_: oh, you have that in horizon.19:00
thedacSee the log and console tabs19:00
BrunoRlazyPowe_: thx anyway19:00
lazyPowe_BrunoR yeah sorry i dont know off hand why its doing that19:00
lazyPowe_BrunoR what version of JUJU are you running?19:01
freak_thedac ,  when i click log tab it shows "  Unable to get log for instance "9fc16f89-9d81-457b-9c93-8ac70e6f87ed"."19:02
thedacok, how about the console tab?19:02
freak_and in console tab it shows console is currently unvavailable19:02
kjackalcory_fy: Have you ever seen this error "line 3: /usr/local/bin/charms.reactive: Permission denied"  when calling an action?19:03
kjackalcory_fu ^19:03
coreycbthedac, freak_: the nova-scheduler.log should have some more info on nova-cloud-controller19:03
freak_thedac, is there any set of procedure after openstack bundle installation or we can directly start making instance from horizon19:03
kjackalcory_fu: This action works on trusty and fails on wily19:04
cory_fukjackal: No.  Can you point me to the charm that's failing?19:04
thedacfreak_: after creating images, networks and keystone users you should be able to launch instances19:04
freak_i have created volume, image and network19:04
freak_but didn't modified anything in users section19:05
thedacfreak_: coreycb is correct looking at the nova-scheduler.log on nova-cloud-controller is the next step.19:05
thedacfreak_: you are probalby using admin which is fine19:05
kjackalcory_fu: just a sec to push what I have19:05
freak_thedac, can you specify the location of nova-scheduler.log coz its not in /var/log/juju19:06
thedac/var/log/nova19:06
freak_ok got it19:07
freak_here is the output and it speaks a lot :)    http://paste.ubuntu.com/15989242/19:07
kjackalcory_fu: this import under wily raises an exception https://github.com/juju-solutions/layer-apache-spark/blob/sys-init/actions/stop-spark-job-history-server#L519:08
thedacfreak_: ok, that seems to suggest that mysql is broken19:08
freak_but in juju status there not such thing its in ready state19:09
thedacfreak_: understood. But the communication between nova-cc and mysql "appears" to be broken.19:10
cory_fukjackal: I don't think the error is coming from that19:10
thedacfreak_: I am going to run that bundle locally and see if I can re-create this problem.19:10
thedacfreak_: but that is going to take a while19:10
cory_fukjackal: Do you have a full stack trace handy?19:11
freak_that would be great,,i'll wait no issue19:11
kjackalcory_fu: let me look into this in more. I will get back to you if I hit a wall.19:12
lazyPowe_BrunoR - i'm headed out to travel back home. Feel free to ping lazyPower  and i'll resume this when I get home if you're still around19:12
freak_thedac, i have noticed that it gets the error message when it is doing block device mapping task19:19
freak_and then shows max retries reached19:19
thedacfreak_: ok, good hint19:20
thedacfreak_: are you doing anything special like booting off ceph volumes? You might also look at ceph health on the ceph nodes.19:28
freak_how to check that?19:28
thedacjuju run --unit ceph/0 'ceph health' or 'ceph status'19:29
freak_thedac , here is the output  http://paste.ubuntu.com/15989760/19:31
thedacfreak_: ok, so ntp has not converged yet enough for ceph to be happy. When you boot an instance are you immediately attaching a volume?19:32
thedacicey: cholcombe: any advice on the clock skew issue with ceph ^^19:32
iceythat's not a problem, it will resolve on its own19:33
freak_from horizon when i click the option create instance from there i select boot from volume and then specify the volume there19:33
thedacfreak_: ok. So to rule out ceph can you boot an instance without booting from a volume?19:33
freak_what should i select in instance boot source option?19:34
freak_it doesn't allow to launch instance without any selection19:35
thedacok, yeah, I need to get this stack up to answer that, sorry19:35
freak_i selected the option boot from image and selected the image then it shows this error http://imgur.com/rPY3w1u19:37
thedacok, interesting19:39
thedacthat still seems to be related to booting from a volume19:41
thedacfreak_: I am deploying now. I need to run out for lunch. I'll touch base with you in about an hour. sound good?19:41
=== penguinRaider is now known as hellboy2k8
freak_ok. good19:41
blahdeblahfreak_, thedac: Does your ntp charm have iburst enabled?  That would speed up convergence during deploy.19:49
freak_how to check?19:50
blahdeblahjuju get ntp19:50
blahdeblahpastebin the results if there's nothing sensitive in there19:50
freak_ok20:04
freak_blahdeblah , here is the ntp output http://paste.ubuntu.com/15990466/20:05
blahdeblahfreak_: Looks like it's pretty much unconfigured; what does "juju run --service ntp 'cat /etc/ntp.conf; ntpq -pn'" show?  (Again, be sure to check for sensitive data before pastebinning.)20:08
LiftedKiltis anyone else having problems with percona/mysql on xenial?20:10
LiftedKiltI can't get the charms to install - when I attach to the container, there appears to be problems with the install itself20:10
freak_blahdeblah ,  here is the output http://paste.ubuntu.com/15990619/20:10
LiftedKiltmysql won't start20:10
cory_fuHey, all.  I'd like to get some input on an issue with how config.changed works.  During the install hook, all config options are considered "changed" because they have no previous value.  In the ibm-base layer, this is causing these handlers to be triggered: https://bazaar.launchpad.net/~ibmcharmers/layer-ibm-base/trunk/view/head:/reactive/ibm-base.sh#L9420:12
cory_fuMy question is, should we change the basic layer to not set config.changed.X if the option has no previous value, or should we change the ibm-base layer to account for the fact that they are "changed" from their previous non-existant value?20:13
blahdeblahfreak_: That looks to me like there's a firewall config (or maybe lack of juju expose) on your MAAS boxes stopping them from talking to each other on the ntp port.  Also, you're going to need some extra settings to make NTP do anything useful.20:13
blahdeblahfreak_: (I'm not 100% sure how juju expose works with MAAS)20:13
cory_fumarcoceppi, kjackal, anyone else who wants to chime in20:13
blahdeblahcory_fu: changing from nothing to something seems like a change to me :-)20:14
freak_but after installing bundle i haven't exposed any openstack component manually from juju gui20:14
=== redir is now known as redir_lunch
cory_fublahdeblah: On the one hand, I agree with you.  On the other, the handlers that I linked to seem entirely reasonable, and it would be unfortunate to have to add additional logic in there to see if they "really changed" (from an actual, non-empty previous value)20:15
cory_fuAlso, I'm not sure how useful it is to react to a "change" when you are only just seeing the values for the first time anyway20:16
cory_fubcsaller: Thoughts?20:17
cory_fuI suppose we could fix it in the ibm-base layer fairly easily using config.set.curl_url but I still feel like the current behavior is a little questionable20:17
blahdeblahcory_fu: Charms have always had to expect config-changed to happen pretty early on in their existence.  At least with reactive you'll only have to do it once.20:17
kjackalcory_fu: no preference here. I tend to agree with blahdeblah on "changing from nothing is a change"20:18
cory_fuActually, config.set.curl_url wouldn't help in this case because the user is setting the value before install.  Hrm20:18
cholcombethedac, clock skew?20:20
cory_fuSo that means that the current behavior, while arguably correct, is onerous to work around.  Maybe we could compromise by changing how config.changed.X works and adding a config.new.X?20:20
blahdeblahfreak_: I'll leave the answer about how exposing works when using the GUI, but for NTP to work, you'll need at least to give it some sources, e.g. "juju set ntp source='0.CC.pool.ntp.org 1.CC.pool.ntp.org 2.CC.pool.ntp.org 3.CC.pool.ntp.org'" (where CC is your country code).20:20
cholcombefreak_, thedac yeah ceph freaks out if your clocks are not ntp sync'd to within a certain amount because paxos depends on message ordering20:20
blahdeblahcholcombe: 50ms, by default20:20
cholcombeyeah it's not much20:21
blahdeblahcholcombe: In NTP terms, that's quite a lot ;-)20:21
cholcombe:D20:21
blahdeblahfreak_: and I'd also recommend "juju set ntp auto_peers=false" - that caused us some issues in production, and it's deprecated in recent versions of the charm.20:22
freak_done both things you just said20:23
bcsallercory_fu: doesn't it mean "I have a value you haven't seen before" and thus should fire to give the code a chance to pick it up?20:23
blahdeblahfreak_: You'll also need to make sure that your systems can actually reach the NTP pool servers (i.e. no firewall blocking them).20:24
blahdeblahfreak_: If all that's good, then re-do the "juju run ..." from above and we should see the time starting to sync20:24
cory_fubcsaller: That's certainly what it means now.  I'm wondering if that is the most useful meaning of it, or if this corner case might be better handled some other way (such as the new / changed split I mentioned)20:25
ReSamis there any way I can disable an api endpoint I don't want to be used? (specific IP)20:26
cory_fubcsaller: The problem in the ibm-base layer is that they want to trigger a handler when: it hasn't been run before OR one of the options has changed20:26
blahdeblahcory_fu: So isn't that what it will do right now?20:26
cory_fuThere isn't a clean way of doing that with decorators on a single handler, because we don't have an OR type construct that can do both @when and @when_not20:27
bcsallercory_fu: I am not opposed to the framework being able to generate all the delta detection in the keys, so 'new' is an option.20:27
LiftedKiltyeah the mysql and percona charms are broken20:28
bcsallercory_fu: I do recall that we thought about making the triggers use a better expression language, @when('x OR y AND NOT z'), maybe its something we look at for 2.020:28
cory_fublahdeblah: To get around the OR problem, the ibm-base layer has additional handlers for the other cases, but now they are getting called *as well* as the original handler, if the value is set before the install hook, but *not* if the config is changed after the install hook20:28
cory_fuThat inconsistency seems wrong20:28
cory_fubcsaller: Yeah, that's a possibility, though we wanted to avoid that for clarity's sake.  Perhaps we've made things less clear in the end20:29
bcsallercory_fu: it looks like the real world usage points that way :-/20:30
cory_fuStill, I think the new / changed split can fix this, while keeping an easy way to get the current behavior (@when_any('config.changed.X', 'config.new.X')) and I kind of doubt anyone is actually depending on the current behavior20:31
cory_fuThough, I could be wrong20:31
freak_blahdeblah , is there any command through which i can see ntp sources20:32
blahdeblahfreak_: "juju get ntp" shows you what's configured, and the "juju run ..." command from earlier shows you the running configuration and current status.20:33
blahdeblahfreak_: what you're looking for is the "reach" (reachability) column (line 28 in your last pastebin) to go non-zero, and the "offset" column to be something less than 50.20:36
blahdeblahfreak_: Anyway, it's the weekend for me, and I'm off to get some breakfast; good luck!20:38
freak_blahdeblah , here is the updated ouput http://paste.ubuntu.com/15991217/20:38
freak_ok20:39
blahdeblahfreak_: Seems like you could pick some closer and more reliable servers, but eventually that config should get you to a reasonable point20:39
freak_blahdeblah , actually i'm located in asia so I selected nearest servers20:40
magicaltroutlazyPower: I'm working on doing my demo for the Juju <-> Data management stuff tomorrow20:40
magicaltroutso i'll have some stuff for you to look at next week20:40
blahdeblahfreak_: the delay says that they're not as close as you might think ;-)20:40
blahdeblah> 400 ms delay means they're half-way round the planet20:41
magicaltroutshould probably stand up my fake datamangement stack so i can actually charm it up20:41
blahdeblahfreak_: Keep watching the offsets from "juju run --service ntp 'ntpq -pn'" and when they get under 50, ceph should become more happy.20:42
blahdeblahfreak_: If you're going to rely on this long-term, you should deploy the ntpmaster charm on 4+ bare metal hosts in your cluster in order to insulate your ceph hosts from poor connectivity to the upstream NTP servers.20:42
* blahdeblah bails out for real now20:43
freak_ok.thanks20:43
magicaltroutLiftedKilt: got a log anywhere I'm curious?20:44
LiftedKiltmagicaltrout: my juju debug-log is broken unfortunately20:45
magicaltroutbooo!20:45
LiftedKiltmagicaltrout: it returns ERROR invalid entity name or password20:45
=== redir_lunch is now known as redir
LiftedKiltmagicaltrout: on the container itself, mysql-server wouldn't start, and a dpkg --configure was complaining about errors with mysql-server and mysql-server-5.720:47
LiftedKiltdoing an apt remove/install didn't clear the errors, nor did reconfiguring with dpkg20:48
magicaltroutsorry LiftedKilt its been a while since i last logged onto my dev env and need to bootstrap it, bear with me i'll see what happens20:53
LiftedKiltno worries20:53
LiftedKiltmagicaltrout: I am redeploying with mariadb in a trusty container in the meantime20:53
LiftedKiltwhere I noticed it was with the openstack-lxd bundle's percona-cluster charm, but when I swapped that charm out with the xenial mysql charm and still had the same issues, I thought I would do some poking around and see if anyone else had run into the same issues20:55
magicaltroutno problem, anything to stop me doing my real job is preferable! ;)21:03
LiftedKilthaha21:03
Gili installed etcd and flannel.  both flannel are stuck in pending.  flannel/6 looks ok based on logs in /var/log/juju but flannel/7 shows message : "2016-04-22 15:34:38 INFO juju.utils.fslock fslock.go:146 attempted lock failed "uniter-hook-execution", flannel/7: executing operation: run install hook, currently held: initialise-lxc21:17
Gilany idea how to fix so that flannel units will exit pending state?21:18
ReSamwhy can't I create a backup of my manual controller?21:32
marcoceppicory_fu: it seems we would just want to, on bootstrap, query the config values to see the config database22:01
marcoceppiReSam: manual provider is a second class citizen when it comes to providers22:02
marcoceppithat's probably why?22:02
cory_fumarcoceppi: Not sure what you mean22:02
marcoceppicory_fu: during the bootstrap code, arguably happens during first hook execution but before reactive runs, can't we just config-get and snapshot that so we don't get an erroneous config.changed on first run?22:02
cory_fumarcoceppi: We could, but that's not materially different than my PR.  It's more about the change in behavior and whether people expect one or the other.22:03
marcoceppicory_fu: well yours adds a new state, which will only ever be set once?22:04
cory_fumarcoceppi: Yes.  The new state is so that you can replicate the current behavior if you really need it.  Maybe not necessary, if no one is relying on the current behavior22:05
LiftedKiltjamespage: having some issues with the percona-cluster charm - it fails to deploy with the openstack-lxd bundle as well as by itself in a separate model on Xenial. Have you run into similar issues?22:27
LiftedKiltmagicaltrout: did you happen to get anywhere with it?22:31

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!