=== redir is now known as redir_eod === markthomas_ is now known as markthomas === urulama|____ is now known as urulama === matthelmke is now known as matthelmke-afk === matthelmke is now known as matthelmke-afk === rogpeppe2 is now known as rogpeppe [08:29] gnuoy, doing sweepups from yesterdays release - bugs targetted for 16.04 marked fix released; all others bumped to 16.07 [08:29] jamespage, fantastic, ta [08:37] nice job on the recent release! how do I get the new charms with xenial support to show up in my juju-gui search? [08:37] ReSam, I think that should just work... [08:37] * jamespage looks [08:38] jamespage: I'm only seeing ceph for trusty - but if I search on the jujucharms.com site, I find the new charm for all supported ubuntu versions [08:38] rogpeppe, hey - can you do a promulgation for me? rabbitmq-server owned by openstack-charmers - superceeds the ~charms ones [08:38] ReSam, might just be a search issue - https://jujucharms.com/ceph/xenial [08:38] its there :-) [08:38] rogpeppe, ~charmers one rather... [08:39] yes - on jujucharms.com --- but not in my local juju-gui search... [08:39] or do I need to update the catalog or something? [08:39] I'm running juju2beta5 from the devel-ppa [08:40] ReSam, not sure tbh - not actually used that feature :-) [08:41] ReSam, oh thats in the juju-gui itself? urulama might be able to help there... [08:41] morning urulama :-) [08:42] jamespage: yes - my locally installed juju-gui, which shipped since juju2beta4 I think [08:42] ReSam, sorry - long day yesterday - brain not quite functional just yet :-) [08:43] jamespage: np - I'm really thankful for your hard work - makes my life a lot easier -- hopefully :) [08:43] ReSam, well let us know if that's not the case :-) [08:43] f I search for "cs:xenial/ceph-0" I can find it - but not if I only search "ceph", then only the trusty charm shows up [08:43] ReSam, https://wiki.ubuntu.com/ServerTeam/OpenStackCharms/ReleaseNotes1604 [08:44] might be useful for you if you've not already seen them... [08:44] jamespage: thanks - already skimmed through it. [08:46] jamespage, ReSam: morning ... OTP, be back in 10min [08:51] jamespage: I'm having problems deploying new charms: [08:51] ERROR juju.worker.dependency engine.go:526 "metric-collect" manifold worker returned unexpected error: failed to read charm from: /var/lib/juju/agents/unit-ceph-21/charm: stat /var/lib/juju/agents/unit-ceph-21/charm: no such file or directory [08:51] the unit us stuck at: "Waiting for agent initialization to finish" [08:59] jamespage: sorry, was in a call [09:00] jamespage: i don't think i have privs to promulgate [09:00] rogpeppe, no problem :-) [09:00] rogpeppe, uh - urulama pointed me at you yesterday... [09:00] incase marcoceppi was not around... [09:00] jamespage: i can give you a command that you can execute to do it if you have privs [09:01] rogpeppe: just to provide the bhttp instructions, jamespage has charmers rights, i believe :) [09:01] urulama: will do [09:01] ReSam: looking [09:02] jamespage: first: go get github.com/rogpeppe/bhttp [09:02] jamespage: (assuming you've got a go env set up) [09:02] jamespage: then bhttp put -j https://api.jujucharms.com/charmstore/v5/$charmid/promulgate Promulgate:=true [09:02] ReSam: indeed, series are missing. that's a regression :-/ [09:03] ReSam: i suggest in gui you search for "ceph xenial" ... first hit will provide you xenial charm [09:03] urulama: thanks - yes I already found it - still a bit confusing. [09:04] ReSam: it is [09:04] filing a bug [09:04] also xenial is missing in the dropdown menu for "series" in the search view [09:04] all series information is missing [09:05] urulama: any idea about my other problem: "stat /var/lib/juju/agents/unit-ceph-21/charm: no such file or directory" ? [09:06] ReSam: that is after deploying the charm? [09:06] yes [09:06] for that i'll have to redirect you to jamespage and the openstack guys === dooferlad_ is now known as dooferlad [09:11] jamespage: maybe it is proxy related? my machines do not have direct internet access... [09:23] urulama, I am a charmer yes... [09:24] rogpeppe, how do I resolve the charm id? [09:24] sorry being dumb [09:24] ReSam, hmm - that looks familiar [09:24] jamespage: what charm are you trying to promulgate? [09:25] are there erros in the machine log on the unit [09:25] jamespage: on all my machines in this file: /var/log/juju/unit-ceph-21.log (with different ids of course) [09:26] rogpeppe, https://jujucharms.com/u/openstack-charmers/rabbitmq-server [09:27] jamespage: you want to move if from ~charmers to ~openstack-charmers space? [09:27] urulama, yes please [09:27] rogpeppe: i think you'll have to explain how to build bhttp as well [09:28] urulama: i did, i think [09:28] jamespage: https://api.jujucharms.com/charmstore/v5/openstack-charmers/rabbitmq-server/promulgate [09:28] rogpeppe, awesome tat [09:28] jamespage: i think [09:28] rogpeppe, gotach on the build.. [09:28] my go foo is good enough for that these days [09:28] jamespage: caveat: i haven't actually tried this command for real, 'cos i don't have promulgate privs :) [09:28] jamespage: but it *should* work :) [09:30] rogpeppe, needed a ~ for the team name [09:30] jamespage: oops, good point! [09:30] rogpeppe, hmm I might need to unpromulgate the charmers one first... [09:31] jamespage: no need [09:31] jamespage: charmstore does the switch [09:31] urulama, hmm [09:31] * urulama corrects that ... *should do the switch* [09:32] urulama, now I broke it all... [09:33] ? [09:33] urulama, https://jujucharms.com/rabbitmq-server/ [09:33] not found... [09:34] jamespage: what was the bhttp command that you used? [09:34] urulama, bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/rabbitmq-server/promulgate Promulgate:=true [09:34] urulama, I also did a [09:34] bhttp put -j https://api.jujucharms.com/charmstore/v5/~charmers/rabbitmq-server/promulgate Promulgate:=false [09:35] maybe that was bad of me... [09:36] gnuoy, do we still need todo the hacluster charm -> charm store ? [09:36] let's try bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/xenial/rabbitmq-server/promulgate Promulgate:=true [09:37] jamespage, looks like its still awaiting a merge https://code.launchpad.net/~gnuoy/charms/trusty/hacluster/1604/+merge/292493 ... [09:37] rogpeppe: is'nt it "Promulgate=true" not "Promulgate:=true ? [09:37] urulama: nope [09:37] and Promulgated:=True not Promulgate [09:38] jamespage: bhttp put -j https://api.jujucharms.com/charmstore/v5/~openstack-charmers/xenial/rabbitmq-server/promulgate Promulgated:=true [09:38] anyone have had any problems with LXD containers not getting the hostname set in /etc/hosts ? [09:39] urulama, ok that worked... [09:39] urulama, still showing as owned by charmers in the UI, but its the right charm... [09:39] jamespage: yes, wrong owner resolution. charmstore shows proper one https://api.jujucharms.com/charmstore/v5/rabbitmq-server/meta/owner [09:40] jamespage: fix will be deployed after ODS (don't want to touch production during if not critical) === urulama is now known as urulama|school === urulama|school is now known as urulama|afk [09:46] gnuoy, ok hacluster merged and direct pushed for trusty and xenial to the charm store [09:46] thanks [09:46] gnuoy, https://jujucharms.com/hacluster/xenial [09:47] jamespage: ah, cool, it worked [09:47] rogpeppe, yes all good now [09:47] jamespage: grand [09:51] jamespage: any chance you remember from where my problems looks familiar? is there an open issue or maybe even a solution? [09:51] ReSam, it was something todo with stale charms on the controller - did you happen to use multiple models under juju 2.0? [09:51] jamespage: not that I know of [09:52] can clean up this somehow? deleting a cache or so? [09:54] ReSam, can you check the /var/log/juju/machine-X.log file please [09:56] rogpeppe, does promugation for bundles work in the same way? our current openstack-base and openstack-telemetry bundles are under ~charmers - I'd like to move those over to ~openstack-charmers and switch the pointer to the promugated bundle ... [09:56] jamespage: yes, it should do [09:56] jamespage: seems to be ok... lots of "block devices change" messages though... [09:57] ReSam, I was looking for something related to hash sum mismatches [09:57] the machine agent downloads the charm that the unit agent then uses [09:57] is there anything in /var/lib/juju/agents/ ? [09:58] jamespage: no hash mismatch lines. [09:58] hmmm [09:58] jamespage: yes, /var/lib/juju/agents contains a folder for the machine and ceph-unit [09:58] jamespage: including agent.conf's for both [09:59] ReSam, whats in var/lib/juju/agents/unit-ceph-21 [09:59] ? [09:59] both are also running process. so that all works [09:59] ls /var/lib/juju/agents/unit-ceph-21/ [09:59] agent.conf metrics-send.socket run.socket state/ [10:00] seems my state server has 3 ip addresses - and therefore 3 api endpoints. two of which are unreachable though. so I get 2 errors in the logs when it tries to connect to the wrong endpoint [10:02] I can see correct proxy settings in the log - so that should also be fine [10:04] jamespage: seems one of the downloads is misbehaving: I have this file: /var/lib/juju/agents/unit-ceph-21/state/bundles/downloads/inprogress-377015309 and inprogress-005101612 [10:05] ReSam, this is a juju problem of some description that I've not seen before - lets see if the juju-dev team know about this... [10:06] dimitern, frobware: either of you two seen anything like this before with juju 2.0 betas ? ^^ [10:09] jamespage: reading scrollback [10:11] frobware, it maybe that the ip address that units are getting for the state server is not one they can actually connect over... [10:12] jamespage: are the units in containers? [10:16] frobware, I'll have to defer that to ReSam [10:57] frobware: no - directly in the machine root [10:58] jamespage: yes - so my state server is reachable via 3 interfaces - but only 1 is reachable from the machines. I can see the machine trying to connect to the two others - and get "connection timeout" in the logs. but then I guess the third try succeeds. === rvba` is now known as rvba === urulama|afk is now known as urulama === matthelmke-afk is now known as matthelmke [11:50] Hey guys, anyone had any luck bootstrapping MAAS 2.0, I've tried Beta5 and Beta4 and get a runtime error [11:50] panic: runtime error: invalid memory address or nil pointer dereference [12:23] frobware: yes - looks like it is using the wrong api endpoint for downloading - although before that is uses the correct one to establish a connection: https://paste.ubuntu.com/15980208/ [12:36] ReSam: are you using a bundle to deploy? Just wanted to repro locally [12:40] ReSam: would it be possible to share and collect some logs? Also, the output of `juju status' and `juju show-machines' [12:52] Is the MAAS 2.0 support still a wip? [12:56] frobware: no bundle - just: juju deploy cs:xenial/ceph-0 -n 5 [12:57] ReSam: ok that helps. can you share the juju status output? [12:57] frobware: https://paste.ubuntu.com/15980650/ [12:58] ReSam: and the machine NIC configuration? how many NICs, VLANs, et al? [12:58] frobware: https://paste.ubuntu.com/15980661/ [12:59] frobware: machines have active 2 interfaces with the same ip subnet [12:59] state server has 3 interfaces - only one is connected to the machines (172.24.32.2) [13:02] ReSam: and can I get access to the machine-0.log and from the machines too? [13:02] sure [13:03] frobware: you mean the state server or the machine with ceph? [13:03] ReSam: all logs really, would be quicker than for me to keep on asking === cmars` is now known as cmars [14:48] 'juju-quickstart bundle.yaml' dies with "juju-quickstart: error: unable to connect to the Juju API server on wss://x.y.z/api: 'module' object has no attribute 'default_timeout" (juju 1.25.5) === BlackDex_ is now known as BlackDex [15:17] 'juju-quickstart bundle.yaml' dies with "juju-quickstart: error: unable to connect to the Juju API server on wss://x.y.z/api: 'module' object has no attribute 'default_timeout" (juju 1.25.5) [15:24] Hi BrunoR - do you wind up with a stack trace from that? === Makyo_ is now known as Makyo [15:34] Makyo: http://paste.ubuntu.com/15983523/ you are welcome [15:35] BrunoR: Thanks, I'm digging into the code, and it looks a bit like a problem with websocket-client. What version do you get when you run `pip show websocket-client`? [15:41] Makyo: this shows an error, it looks like this modules is installed via deb-package python-websocket (0.18.0-0ubuntu0.14.04~ppa5 from http://ppa.launchpad.net/juju/stable/ubuntu/) [15:42] BrunoR: ah, alright, thank you. [15:45] BrunoR: may I please see the output from `python -c "import websocket;print dir(websocket)"`? I'm curious as to why default_timeout is missing from the websocket module. [15:48] Makyo: http://paste.ubuntu.com/15983924/ === urulama is now known as urulama|afk [15:50] That's supremely weird. [15:50] BrunoR: I'll file a bug against quickstart and work on it. In the mean time, you can try `sudo pip install websocket-client` and see if the version you have is overwritten by the one installed by pip. [15:51] (You might need to do `sudo pip install --upgrade websocket-client` due to the versions being the same) [15:55] Makyo: pip install --upgrade moved websocket-client from 0.35.0 to 0.37.0 ~ trying `juju-quickstart bundle.yaml` again ~ looks like a problem in deb-package python-webclient? [15:56] BrunoR: yeah, that's what I'm seeing now that I play around with it. Did you install quickstart from the juju/stable PPA, or from the default repo? [15:56] Makyo: no, same error [15:58] Makyo: `$ apt-cache madison juju-quickstart` says 2.2.4+bzr147+ppa42~ubuntu14.04.1 from http://ppa.launchpad.net/juju/stable/ubuntu/ [16:07] aisrael: Are you on Ubuntu and seeing https://bugs.launchpad.net/cloud-images/+bug/1573058? Or OS X? [16:07] Bug #1573058: Ubuntu 16.04 current not booting in Vagrant (gurumeditation) [16:08] Odd_Bloke: OS X. I hadn't refreshed the page before I posted that comment. [16:08] aisrael: No worries; just making sure there's nothing more we can do. :) [16:09] Makyo: my google-foo found something similar https://bugs.launchpad.net/juju-quickstart/+bug/1484158 [16:09] Bug #1484158: juju quickstart fails "unable to connect to the Juju API Server"

[16:10] BrunoR: aha, good catch. I [16:10] BrunoR: I'll tag on to that, and see if I can provide a more sensible default when things go wrong like this. [16:10] BrunoR: thank you [16:24] deployed flannel units with juju, units are stuck in pending state? Any pointers on where to look for debug and fix information? I looked in /var/log/juju but the logs don't seem to have any errors, here are the messages (which keep updating) [16:24] checking flannel/6 for flannel leadership [16:24] flannel/6 confirmed for flannel leadership until 2016-04-22 16:25:13.080001473 +0000 UTC [16:24] flannel/6 will renew flannel leadership at 2016-04-22 16:24:43.080001473 +0000 UTC [16:25] those three lines repeat in the log every few minutes - which looks all good afaik [16:25] so not sure why flannel showing in pending state... === redir_eod is now known as redir === mup_ is now known as mup === chuck__ is now known as zul [17:49] if I ssh to a machine that is running lxc containers with charms installed, how do I view the containers that are running? [17:49] nothing shows up in lxc list / lxc info [17:52] LiftedKilt: lxc-ls [18:05] jrwren: what's the difference between lxc list and lxc-ls? [18:06] LiftedKilt: lxc is the lxd client which talks to the lxd sever. lxc-ls is lower level lxc (no lxd involved) tool. [18:06] jrwren: gotcha - thanks [18:09] how does 'charm push', 'charm publish' works for bundles? I just pushed/published/granted but the bundle does only shows 'partly' up at jujucharms.com? urulama|afk? [18:14] how does 'charm push', 'charm publish' works for bundles? I just pushed/published/granted but the bundle does only shows 'partly' up at jujucharms.com? urulama|afk? marcoceppi? [18:21] BrunoR what did you publish? i''m happy to take a look [18:23] lazyPowe_: https://jujucharms.com/u/3-bruno/ or cs:~3-bruno/bundle/demo-0 [18:23] lazyPowe_: according to 'charm show' it should be accessible [18:24] hi everyone [18:24] i need help regarding instance creation in openstack [18:24] i have created my first instance [18:25] and it is showing error [18:25] http://imgur.com/krBt0H9 [18:25] can you please take a look here is the picture of error msg http://imgur.com/krBt0H9 [18:28] BrunoR ok, taking a look now [18:29] BrunoR i see your bundle up here... [18:29] 6 services, 12 units, on 4 machines? [18:29] lazyPowe_: yes [18:29] https://jujucharms.com/u/3-bruno/demo/ [18:29] it totally in the store then :) [18:30] so i guess i'm not understanding what you're asking "how does push and publish work for bundles?" it works exactly like it does for charms. if you upload the bundle and dont set public acl's, it'll only be accessible to you (the uploader) [18:33] lazyPowe_: I see an image of the services in the bundle but can't reach the README.md or deploy it. [18:42] BrunoR - the README does look truncated, however - http://paste.ubuntu.com/15988639/ [18:42] BrunoR i was able to deploy just fine using juju 2.0 beta5, i did not test on 1.25.5 howeer [18:42] *however [18:48] thedac, coreycb could you give freak_ a hand? [18:48] I'll take a look [18:49] hi thedac [18:49] i created my first instance in openstack from horizon dashboard [18:49] earlier i created network, image, storage volume,, [18:49] then i created instance [18:49] but upon startup its showing erro [18:49] error [18:50] lazyPowe_: `$ juju deploy cs:~3-bruno/bundle/demo` yields 'ERROR expected a charm URL, got bundle URL "cs:~3-bruno/bundle/demo-0"' [18:50] can you please take a look [18:50] http://imgur.com/krBt0H9 [18:50] freak_: looking at it now. This was juju deployed? [18:50] freak_: can I see the bundle used? [18:53] thedac , i used this bundle https://jujucharms.com/u/charmers/openstack-base/bundle/40 [18:54] ok. So it is unclear from the image what precisely is failing. Can you grab the novarc and try some things from command line? [18:54] I would start with checking juju status and make sure nothing is in error or blocking state [18:54] yes i will,, [18:54] BrunoR weird... [18:55] here is the status http://paste.ubuntu.com/15988934/ [18:56] thanks, looking [18:58] here is the novarc http://paste.ubuntu.com/15988999/ [18:58] freak_: so that looks good. If you have novaclient installed you might try nova show $INSTANCE_ID and see if we get any more info. And nova console-log $INSTANCE_ID [19:00] freak_: oh, you have that in horizon. [19:00] See the log and console tabs [19:00] lazyPowe_: thx anyway [19:00] BrunoR yeah sorry i dont know off hand why its doing that [19:01] BrunoR what version of JUJU are you running? [19:02] thedac , when i click log tab it shows " Unable to get log for instance "9fc16f89-9d81-457b-9c93-8ac70e6f87ed"." [19:02] ok, how about the console tab? [19:02] and in console tab it shows console is currently unvavailable [19:03] cory_fy: Have you ever seen this error "line 3: /usr/local/bin/charms.reactive: Permission denied" when calling an action? [19:03] cory_fu ^ [19:03] thedac, freak_: the nova-scheduler.log should have some more info on nova-cloud-controller [19:03] thedac, is there any set of procedure after openstack bundle installation or we can directly start making instance from horizon [19:04] cory_fu: This action works on trusty and fails on wily [19:04] kjackal: No. Can you point me to the charm that's failing? [19:04] freak_: after creating images, networks and keystone users you should be able to launch instances [19:04] i have created volume, image and network [19:05] but didn't modified anything in users section [19:05] freak_: coreycb is correct looking at the nova-scheduler.log on nova-cloud-controller is the next step. [19:05] freak_: you are probalby using admin which is fine [19:05] cory_fu: just a sec to push what I have [19:06] thedac, can you specify the location of nova-scheduler.log coz its not in /var/log/juju [19:06] /var/log/nova [19:07] ok got it [19:07] here is the output and it speaks a lot :) http://paste.ubuntu.com/15989242/ [19:08] cory_fu: this import under wily raises an exception https://github.com/juju-solutions/layer-apache-spark/blob/sys-init/actions/stop-spark-job-history-server#L5 [19:08] freak_: ok, that seems to suggest that mysql is broken [19:09] but in juju status there not such thing its in ready state [19:10] freak_: understood. But the communication between nova-cc and mysql "appears" to be broken. [19:10] kjackal: I don't think the error is coming from that [19:10] freak_: I am going to run that bundle locally and see if I can re-create this problem. [19:10] freak_: but that is going to take a while [19:11] kjackal: Do you have a full stack trace handy? [19:11] that would be great,,i'll wait no issue [19:12] cory_fu: let me look into this in more. I will get back to you if I hit a wall. [19:12] BrunoR - i'm headed out to travel back home. Feel free to ping lazyPower and i'll resume this when I get home if you're still around [19:19] thedac, i have noticed that it gets the error message when it is doing block device mapping task [19:19] and then shows max retries reached [19:20] freak_: ok, good hint [19:28] freak_: are you doing anything special like booting off ceph volumes? You might also look at ceph health on the ceph nodes. [19:28] how to check that? [19:29] juju run --unit ceph/0 'ceph health' or 'ceph status' [19:31] thedac , here is the output http://paste.ubuntu.com/15989760/ [19:32] freak_: ok, so ntp has not converged yet enough for ceph to be happy. When you boot an instance are you immediately attaching a volume? [19:32] icey: cholcombe: any advice on the clock skew issue with ceph ^^ [19:33] that's not a problem, it will resolve on its own [19:33] from horizon when i click the option create instance from there i select boot from volume and then specify the volume there [19:33] freak_: ok. So to rule out ceph can you boot an instance without booting from a volume? [19:34] what should i select in instance boot source option? [19:35] it doesn't allow to launch instance without any selection [19:35] ok, yeah, I need to get this stack up to answer that, sorry [19:37] i selected the option boot from image and selected the image then it shows this error http://imgur.com/rPY3w1u [19:39] ok, interesting [19:41] that still seems to be related to booting from a volume [19:41] freak_: I am deploying now. I need to run out for lunch. I'll touch base with you in about an hour. sound good? === penguinRaider is now known as hellboy2k8 [19:41] ok. good [19:49] freak_, thedac: Does your ntp charm have iburst enabled? That would speed up convergence during deploy. [19:50] how to check? [19:50] juju get ntp [19:50] pastebin the results if there's nothing sensitive in there [20:04] ok [20:05] blahdeblah , here is the ntp output http://paste.ubuntu.com/15990466/ [20:08] freak_: Looks like it's pretty much unconfigured; what does "juju run --service ntp 'cat /etc/ntp.conf; ntpq -pn'" show? (Again, be sure to check for sensitive data before pastebinning.) [20:10] is anyone else having problems with percona/mysql on xenial? [20:10] I can't get the charms to install - when I attach to the container, there appears to be problems with the install itself [20:10] blahdeblah , here is the output http://paste.ubuntu.com/15990619/ [20:10] mysql won't start [20:12] Hey, all. I'd like to get some input on an issue with how config.changed works. During the install hook, all config options are considered "changed" because they have no previous value. In the ibm-base layer, this is causing these handlers to be triggered: https://bazaar.launchpad.net/~ibmcharmers/layer-ibm-base/trunk/view/head:/reactive/ibm-base.sh#L94 [20:13] My question is, should we change the basic layer to not set config.changed.X if the option has no previous value, or should we change the ibm-base layer to account for the fact that they are "changed" from their previous non-existant value? [20:13] freak_: That looks to me like there's a firewall config (or maybe lack of juju expose) on your MAAS boxes stopping them from talking to each other on the ntp port. Also, you're going to need some extra settings to make NTP do anything useful. [20:13] freak_: (I'm not 100% sure how juju expose works with MAAS) [20:13] marcoceppi, kjackal, anyone else who wants to chime in [20:14] cory_fu: changing from nothing to something seems like a change to me :-) [20:14] but after installing bundle i haven't exposed any openstack component manually from juju gui === redir is now known as redir_lunch [20:15] blahdeblah: On the one hand, I agree with you. On the other, the handlers that I linked to seem entirely reasonable, and it would be unfortunate to have to add additional logic in there to see if they "really changed" (from an actual, non-empty previous value) [20:16] Also, I'm not sure how useful it is to react to a "change" when you are only just seeing the values for the first time anyway [20:17] bcsaller: Thoughts? [20:17] I suppose we could fix it in the ibm-base layer fairly easily using config.set.curl_url but I still feel like the current behavior is a little questionable [20:17] cory_fu: Charms have always had to expect config-changed to happen pretty early on in their existence. At least with reactive you'll only have to do it once. [20:18] cory_fu: no preference here. I tend to agree with blahdeblah on "changing from nothing is a change" [20:18] Actually, config.set.curl_url wouldn't help in this case because the user is setting the value before install. Hrm [20:20] thedac, clock skew? [20:20] So that means that the current behavior, while arguably correct, is onerous to work around. Maybe we could compromise by changing how config.changed.X works and adding a config.new.X? [20:20] freak_: I'll leave the answer about how exposing works when using the GUI, but for NTP to work, you'll need at least to give it some sources, e.g. "juju set ntp source='0.CC.pool.ntp.org 1.CC.pool.ntp.org 2.CC.pool.ntp.org 3.CC.pool.ntp.org'" (where CC is your country code). [20:20] freak_, thedac yeah ceph freaks out if your clocks are not ntp sync'd to within a certain amount because paxos depends on message ordering [20:20] cholcombe: 50ms, by default [20:21] yeah it's not much [20:21] cholcombe: In NTP terms, that's quite a lot ;-) [20:21] :D [20:22] freak_: and I'd also recommend "juju set ntp auto_peers=false" - that caused us some issues in production, and it's deprecated in recent versions of the charm. [20:23] done both things you just said [20:23] cory_fu: doesn't it mean "I have a value you haven't seen before" and thus should fire to give the code a chance to pick it up? [20:24] freak_: You'll also need to make sure that your systems can actually reach the NTP pool servers (i.e. no firewall blocking them). [20:24] freak_: If all that's good, then re-do the "juju run ..." from above and we should see the time starting to sync [20:25] bcsaller: That's certainly what it means now. I'm wondering if that is the most useful meaning of it, or if this corner case might be better handled some other way (such as the new / changed split I mentioned) [20:26] is there any way I can disable an api endpoint I don't want to be used? (specific IP) [20:26] bcsaller: The problem in the ibm-base layer is that they want to trigger a handler when: it hasn't been run before OR one of the options has changed [20:26] cory_fu: So isn't that what it will do right now? [20:27] There isn't a clean way of doing that with decorators on a single handler, because we don't have an OR type construct that can do both @when and @when_not [20:27] cory_fu: I am not opposed to the framework being able to generate all the delta detection in the keys, so 'new' is an option. [20:28] yeah the mysql and percona charms are broken [20:28] cory_fu: I do recall that we thought about making the triggers use a better expression language, @when('x OR y AND NOT z'), maybe its something we look at for 2.0 [20:28] blahdeblah: To get around the OR problem, the ibm-base layer has additional handlers for the other cases, but now they are getting called *as well* as the original handler, if the value is set before the install hook, but *not* if the config is changed after the install hook [20:28] That inconsistency seems wrong [20:29] bcsaller: Yeah, that's a possibility, though we wanted to avoid that for clarity's sake. Perhaps we've made things less clear in the end [20:30] cory_fu: it looks like the real world usage points that way :-/ [20:31] Still, I think the new / changed split can fix this, while keeping an easy way to get the current behavior (@when_any('config.changed.X', 'config.new.X')) and I kind of doubt anyone is actually depending on the current behavior [20:31] Though, I could be wrong [20:32] blahdeblah , is there any command through which i can see ntp sources [20:33] freak_: "juju get ntp" shows you what's configured, and the "juju run ..." command from earlier shows you the running configuration and current status. [20:36] freak_: what you're looking for is the "reach" (reachability) column (line 28 in your last pastebin) to go non-zero, and the "offset" column to be something less than 50. [20:38] freak_: Anyway, it's the weekend for me, and I'm off to get some breakfast; good luck! [20:38] blahdeblah , here is the updated ouput http://paste.ubuntu.com/15991217/ [20:39] ok [20:39] freak_: Seems like you could pick some closer and more reliable servers, but eventually that config should get you to a reasonable point [20:40] blahdeblah , actually i'm located in asia so I selected nearest servers [20:40] lazyPower: I'm working on doing my demo for the Juju <-> Data management stuff tomorrow [20:40] so i'll have some stuff for you to look at next week [20:40] freak_: the delay says that they're not as close as you might think ;-) [20:41] > 400 ms delay means they're half-way round the planet [20:41] should probably stand up my fake datamangement stack so i can actually charm it up [20:42] freak_: Keep watching the offsets from "juju run --service ntp 'ntpq -pn'" and when they get under 50, ceph should become more happy. [20:42] freak_: If you're going to rely on this long-term, you should deploy the ntpmaster charm on 4+ bare metal hosts in your cluster in order to insulate your ceph hosts from poor connectivity to the upstream NTP servers. [20:43] * blahdeblah bails out for real now [20:43] ok.thanks [20:44] LiftedKilt: got a log anywhere I'm curious? [20:45] magicaltrout: my juju debug-log is broken unfortunately [20:45] booo! [20:45] magicaltrout: it returns ERROR invalid entity name or password === redir_lunch is now known as redir [20:47] magicaltrout: on the container itself, mysql-server wouldn't start, and a dpkg --configure was complaining about errors with mysql-server and mysql-server-5.7 [20:48] doing an apt remove/install didn't clear the errors, nor did reconfiguring with dpkg [20:53] sorry LiftedKilt its been a while since i last logged onto my dev env and need to bootstrap it, bear with me i'll see what happens [20:53] no worries [20:53] magicaltrout: I am redeploying with mariadb in a trusty container in the meantime [20:55] where I noticed it was with the openstack-lxd bundle's percona-cluster charm, but when I swapped that charm out with the xenial mysql charm and still had the same issues, I thought I would do some poking around and see if anyone else had run into the same issues [21:03] no problem, anything to stop me doing my real job is preferable! ;) [21:03] haha [21:17] i installed etcd and flannel. both flannel are stuck in pending. flannel/6 looks ok based on logs in /var/log/juju but flannel/7 shows message : "2016-04-22 15:34:38 INFO juju.utils.fslock fslock.go:146 attempted lock failed "uniter-hook-execution", flannel/7: executing operation: run install hook, currently held: initialise-lxc [21:18] any idea how to fix so that flannel units will exit pending state? [21:32] why can't I create a backup of my manual controller? [22:01] cory_fu: it seems we would just want to, on bootstrap, query the config values to see the config database [22:02] ReSam: manual provider is a second class citizen when it comes to providers [22:02] that's probably why? [22:02] marcoceppi: Not sure what you mean [22:02] cory_fu: during the bootstrap code, arguably happens during first hook execution but before reactive runs, can't we just config-get and snapshot that so we don't get an erroneous config.changed on first run? [22:03] marcoceppi: We could, but that's not materially different than my PR. It's more about the change in behavior and whether people expect one or the other. [22:04] cory_fu: well yours adds a new state, which will only ever be set once? [22:05] marcoceppi: Yes. The new state is so that you can replicate the current behavior if you really need it. Maybe not necessary, if no one is relying on the current behavior [22:27] jamespage: having some issues with the percona-cluster charm - it fails to deploy with the openstack-lxd bundle as well as by itself in a separate model on Xenial. Have you run into similar issues? [22:31] magicaltrout: did you happen to get anywhere with it?