[00:00] <rick_h_> LiftedKilt: yea, odd. I've not really tried to just add a machine but we've got tests on that that pass.
[00:00] <rick_h_> LiftedKilt: but that's an odd one, will have to try to replicate
[00:00] <arosales> c0s: good points you are uncovering though
[00:00] <c0s> well, too bad am not making much progress because of them ;)
[00:01] <c0s> ah well
[00:01] <arosales> spring XD looks interesting
[00:01] <LiftedKilt> rick_h_: I'll play with it some more tomorrow, but 99% sure all the wily charms fail for me
[00:01] <LiftedKilt> but xenial and trusty work just fine
[00:01] <c0s> arosales: it is ... Pivotal seems to be super-hot about this... Althgough, it is more of a developer tool. ...
[00:01] <arosales> c0s: its part of the dev process
[00:02] <LiftedKilt> rick_h_: leaving work now - thanks for the help
[00:02] <arosales> c0s: the existence of springXD in itself is a proof point of the complexity at this layer
[00:02] <c0s> But supposedly you can do something like
[00:02] <c0s>   xd> stream create kafka-source-test --definition "kafka --zkconnect=localhost:2181 --topic=event-stream | log"--deploy
[00:02] <c0s> and start consuming messages from a kafka topic
[00:02] <c0s> yup, 'tis true sir
[00:04] <arosales> c0s: interesting re springXD. It seems something is going to have to pari with kafka. I am not sure if thats Gobblin, Flume, springXD or something else
[00:04] <arosales> *pair
[00:06] <arosales> rick_h_: would that be an issue with curtain and wiley?
[00:06] <arosales> rick_h_: or was maas putting down a proper wiley that could be ssh'ed into
[00:06] <rick_h_> arosales: I'm not sure. I'd want to see the logs and see where it fell over from the machine and the juju logs
[00:06] <rick_h_> arosales: the not able to ssh I thought was a xenial lxd things, not maas/wily
[00:07] <arosales> well juju aside maas should be able to set down a wiley instance that one can ssh into
[00:07] <arosales> at least from a debug perspective
[00:07] <rick_h_> arosales: yes
[00:07] <c0s> arosales: yes... needs to be defined. Just forcing a user to write their own sinks and consumer code might be too much for most...
[00:07] <rick_h_> arosales: he didn't have any ssh keys setp so not sure what it must have setup there since juju ssh failed.
[00:08] <rick_h_> arosales: should have tried juju ssh with --debug to see if anything jumped out
[00:08] <arosales> c0s: agreed even if for a given set
[00:09] <c0s> yup
[00:09] <c0s> ok, I've created a card.
[00:09] <arosales> c0s: thanks
[00:09] <c0s> Switching off... was a long day
[00:09] <c0s> you're welcome arosales
[00:09] <arosales> c0s: have a good evening
[00:10] <arosales> rick_h_: well I am sure LiftedKilt will be back tomorrow, thanks for the help
[00:10] <c0s> TTL
[02:12] <stub> tvansteenburgh: Some basic tests, but I've been blocked from testing 2.0 (an open bug that breaks the PostgreSQL charm)
[08:16] <A-Kaser> hi$
[08:44] <jamespage> xnox, did you hit this - https://bugs.launchpad.net/juju-core/+bug/1564791 ?
[08:44] <mup> Bug #1564791: 2.0-beta3: LXD provider, jujud architecture mismatch <juju-core:New> <https://launchpad.net/bugs/1564791>
[09:21] <xnox> jamespage, no, because i bootstrap with uploading my own tools.
[09:21] <xnox> jamespage, there are no s390x tools published yet.
[09:22] <jamespage> xnox, hmm
[09:22] <jamespage> ok
[09:23] <jamespage> xnox, ok trying with that now
[09:28] <deanman> is there any way to have a "private" charm store ? Something that the end-user can use with the GUI to deploy custom charms ?
[09:29] <xnox> jamespage, but the bug is valid.... s390x should not try to download and use amd64 tools.
[09:30] <magicaltrout> deanman: no that I am aware of
[09:30] <magicaltrout> +t
[09:35] <jamespage> dosaboy, apologies for not spotting this before but please can we move the context for the internal endpoint config option to charm-helpers please..
[09:44] <dosaboy> jamespage: sure
[09:51] <jamespage> dosaboy, thanks
[10:01] <dosaboy> jamespage: https://code.launchpad.net/~hopem/charm-helpers/lp1456876/+merge/290701
[10:02] <jamespage> dosaboy, cough..
[10:02] <jamespage> unit test ?
[10:02] <dosaboy> jamespage: come on its Friday ;)
[10:02] <dosaboy> yeah sure
[10:16] <dosaboy> jamespage: done.
[13:12] <gnuoy> dosaboy, got anytime for https://code.launchpad.net/~gnuoy/charm-helpers/sysv-service-running/+merge/290729 >
[13:12] <gnuoy> ?
[13:13] <dosaboy> gnuoy: otp but yes
[13:13] <gnuoy> ta
[14:07] <kjackal> Hey cory_fu, kwmonroe, what do you think about changing the default execution mode of spark from yarn-client to standalone?
[14:07] <kjackal> Good morning!
[14:09] <kwmonroe> i'm ok with that kjackal, but let's get c0s' +1
[14:11] <kjackal> Is is reasonable since spark can work out of the box, however there is a catch here, our bundles that use spark should add an option to set the old-default execution mode
[14:14] <cory_fu> That's not a big deal.  +1
[14:18] <mbruzek> Prabakaran: Hello, you mentioned you want to use juju 2.0 on 14.04 ppc64le.
[14:18] <mbruzek> It should work, but I think you would have better results with 16.04.
[14:18] <mbruzek> Prabakaran: have you added the ppa:juju/devel
[14:20] <mbruzek> ?
[14:21] <Prabakaran> yes i have added .. i am trying to take the error what i am getting
[14:21] <mbruzek> OK
[14:25] <kwmonroe> Prabakaran: what environment are you using?  local (containers), maas, or cloud?
[14:32] <thedac> jamespage: rabbitmq-server and percona-cluster do not need default extra-bindings, correct? I'll land those if that is the case.
[14:33] <jamespage> thedac, yeah - relations == network connections in which case extra-bindings are not required!
[14:33] <jamespage> thedac, I was also thinking about horizon
[14:33] <jamespage> i think the same is true there
[14:33] <thedac> ok
[14:34] <jamespage> juju deploy openstack-dashboard --bind "website=public identity-service=internal"
[14:34] <jamespage> thedac, I'll raise a README for that...
[14:34] <thedac> cool
[14:40] <Prabakaran> Hello Matt, I was getting "E: Unable to locate package juju2" on the Linux c277-pkvm-vm54 3.16.0-30-generic #40~14.04.1-Ubuntu SMP ppc64le GNU/Linux machine
[14:40] <Prabakaran> I followed the steps referring https://jujucharms.com/docs/devel/getting-started
[14:41] <mbruzek> Prabakaran: Right but you need to install the ppa:juju/devel instead of the "stable" one.
[14:41] <Prabakaran> k let me try with that
[14:41] <mbruzek> Prabakaran: https://launchpad.net/~juju/+archive/ubuntu/devel/+packages tells me that there is a package for 14.04.2
[14:41] <mbruzek> Prabakaran: a juju2 package
[14:42] <mbruzek> Prabakaran: after you add the devel ppa you will need to do an apt-get update
[14:42] <mbruzek> and apt-get upgrade
[14:42] <Prabakaran> ya it is getting installated..
[14:43] <mbruzek> The error message you gave me listed 14.04.1, perhaps the system has not been updated in a while
[14:43] <mbruzek> ?
[14:44] <Prabakaran> i think i got this error because i have used ppa/stable instead of ppa/deve;
[14:44] <Prabakaran> now it is installaed
[14:45] <mbruzek> Prabakaran: OK great
[14:46] <mbruzek> Prabakaran: Juju 2.0 is *very* different than 1.25. You will need to set up a different directory.  ~/.local/share/juju
[14:46] <Prabakaran> oh k
[14:46] <Prabakaran> can you help me with that?
[14:47] <mbruzek> Follow the devel docs and report a doc issue if you have problems or suggestions: https://github.com/juju/docs/issues/new
[14:47] <Prabakaran> i think i will have to instal lxd for local cloud right?
[14:48] <Prabakaran> This link is asking me credential for GITHUB
[14:48] <mbruzek> Yes that is where we keep our issues for documentation
[14:48] <Prabakaran> i dont have github account
[14:49] <mbruzek> Oh I see
[14:49] <mbruzek> Prabakaran: If you want to set up LXD you can look here: https://jujucharms.com/docs/devel/config-LXD
[14:49] <mbruzek> lxd is pretty new and will work best on 16.04
[14:50] <mbruzek> Prabakaran: actually reading it now, I see 15.10 or 16.04 is a prerequisite
[14:50] <mbruzek> https://jujucharms.com/docs/devel/config-LXD#prerequisites
[14:50] <Prabakaran> k
[14:51] <mbruzek> that is because of the kernel features that are used in LXD
[14:52] <Prabakaran> oh k
[14:53] <Prabakaran> all machine what we are using is of Linux c277-pkvm-vm54 3.16.0-30-generic #40~14.04.1-Ubuntu SMP ppc64le and x86 only.. thats no problem let us swith to ubuntu 16
[14:54] <mbruzek> Prabakaran: It is a good idea to get acquainted with 2.0,
[14:54] <Prabakaran> before that let me check this using vm hypervisors
[14:54] <mbruzek> Prabakaran: you can use your cloud credentials on 2.0 and use AWS or something else, but LXD requires 15.10 or later
[14:55] <Prabakaran> ya k sure.. let me try in ubuntu 15.10 or above
[14:56] <Prabakaran> Thanks for your patience in helping me :)
[14:59] <thedac> jamespage: network spaces meeting?
[14:59] <jamespage> thedac, yes
[15:07] <mbruzek> Prabakaran: not a problem
[15:18] <LiftedKilt> arosales: rick_h_: I really appreciate you guys helping with this. Here's a pastebin of the results http://paste.ubuntu.com/15577480/
[15:19] <LiftedKilt> rick_h_: arosales: juju can't ssh into the machine, but I can manually when using juju's private key and the ubuntu user. No idea what's going on there...
[15:25] <rick_h_> LiftedKilt: can yoilu try with --debug please?
[15:30] <LiftedKilt> rick_h_: I did run it with debug in the paste - you have to put --debug before the ssh command otherwise it get's passed to openssh
[15:41] <stormmore> LiftedKilt: I am looking at that output and it looks like it successfully sshed in
[15:42] <LiftedKilt> stormmore: the bottom part where it is successful is with me manually connecting
[15:43] <LiftedKilt> stormmore: the top portion up until the adding to known hosts is when juju tries, and it just hangs there and never connects
[15:43] <LiftedKilt> line 20 is where juju hangs, line 27 is me manually connecting
[15:45] <stormmore> LiftedKilt: ah ok, I see that, usually when I see problems like that with services it is a network routing issue
[15:46] <stormmore> LiftedKilt: what does your network config look like?
[15:46] <LiftedKilt> stormmore: I don't think it's a network issue because trusty and xenial images work - it's just wily that fails
[15:46] <LiftedKilt> stormmore: maas and juju are on the same box, sitting on a 10.101.0.0/16 network for testing
[15:47] <stormmore> LiftedKilt: log in manually and try and ping the maas server and vice versa
[15:48] <LiftedKilt> I'm logging in manually from the juju machine
[15:50] <LiftedKilt> stormmore: I've narrowed it down to juju not being able to ssh, but on the same machine using the same credentials I am able to manually connect
[15:50] <stormmore> LiftedKilt: one different I do see is juju is using the juju_id_rsa-cert and not juju_id_rsa (which it can’t fine)
[15:51] <stormmore> can you try ssh ubuntu@10.101.103.1 -i /home/toor/.local/share/juju/ssh/juju_id_rsa-cert
[15:51] <LiftedKilt> that file doesn't exist
[15:52] <LiftedKilt> in the juju/ssh folder I only have a juju_id_rsa and a juju_id_rsa.pub
[15:52] <stormmore> which is what I would expect
[15:52] <stormmore> what about the permissions on the files?
[15:52] <stormmore> debug1: identity file /home/toor/.local/share/juju/ssh/juju_id_rsa type 1
[15:52] <stormmore> debug1: key_load_public: No such file or directory
[15:53] <LiftedKilt> stormmore: 600 on both
[15:53] <stormmore> that is what I believe is the problem
[15:53] <LiftedKilt> so why is it only failing on wily?
[15:54] <stormmore> version of OpenSSH maybe?
[16:00] <LiftedKilt> stormmore: openssh client version 1:6.9p1-2
[16:02] <stormmore> do you have a trust/xeniel in the environement that you can get a juju --debug ssh <id> -vvv out of?
[16:03] <LiftedKilt> stormmore: spinning up one of each now
[16:04] <stormmore> LiftedKilt: idea would be to see what difference we are seeing between wily and them using the same commands
[16:04] <LiftedKilt> stormmore: right
[16:09] <LiftedKilt> stormmore: here's trusty http://paste.ubuntu.com/15578169/
[16:16] <stormmore> so doesn’t show anything different. ok how about doing ssh -vvv ubuntu@10.101.103.1 -i /home/toor/.local/share/juju/ssh/juju_id_rsa?
[16:18] <LiftedKilt> stormmore: http://paste.ubuntu.com/15578241/
[16:19] <LiftedKilt> stormmore: also - I rebooted the machine, and now the juju ssh command is returning this: http://paste.ubuntu.com/15578255/
[16:19] <LiftedKilt> rebooted node 0, that is
[16:20] <stormmore> just that looks odd and definitely outside my knowledge
[16:21] <LiftedKilt> stormmore: apparently the macaroons message isn't related to the reboot, because the other wily nodes I didn't reboot are returning the same message as well
[16:32] <stormmore> I am still thinking an openssh version compat issue
[16:33] <stormmore> looks like it is getting stuck trying to determine the remote version
[16:33] <LiftedKilt> stormmore: but if that was the case, then wouldn't it fail when manually initiated outside juju?
[16:34] <stormmore> nope
[16:34] <LiftedKilt> also the maas/juju server is running on wily itself
[16:35] <rick_h_> cmars: ^ any ideas on the "macaroon not found in storage" error?
[16:36] <rick_h_> cmars: is it a rm ~/.go-cookies kind of thing to try ?
[16:36] <LiftedKilt> well this could be a problem
[16:36] <LiftedKilt> the "wily" machine isn't wily, it's xenial
[16:37] <stormmore> that is what I was wondering
[16:37] <stormmore> I was seeing OpenSSH 7 on it
[16:37] <cmars> rick_h_, sorry, i'm looking at the scrollback, what's the context here?
[16:38] <LiftedKilt> juju status stopped working - no macaroon found in storage
[16:39] <cmars> LiftedKilt, ah, ok. try rm ~/.go-cookies
[16:39] <cmars> LiftedKilt, what version of juju client are you using?
[16:39] <LiftedKilt> I'm running locally on the juju server, which is juju2 beta3
[16:39] <LiftedKilt> clearing go-cookies didn't resolve it
[16:41] <cmars> LiftedKilt, you get this error message from the juju client when running `juju status`? or this is in the logs on the server?
[16:41] <LiftedKilt> from the client
[16:43] <LiftedKilt> stormmore: it looks like the problem is that juju ssh 0 is trying to ssh to the wrong machine
[16:44] <LiftedKilt> node 0 is actually 10.101.103.3, which is running wily. the machine it is connecting to (10.101.103.1) is a separate machine running xenial
[16:44] <LiftedKilt> hence the mismatch
[16:44] <LiftedKilt> cmars: I've actually been running into the macaroon problem repeatedly, and I have to blow it all away and re-bootstrap to resolve it
[16:45] <stormmore> yeah that is really odd
[16:45] <cmars> LiftedKilt, what provider is this?
[16:45] <LiftedKilt> maas 1.9
[16:45] <cmars> LiftedKilt, is the controller HA?
[16:46] <LiftedKilt> cmars: No, although when I tried making it HA on a previous deployment, it caused this problem. Had to blow it away and start over
[16:49] <cmars> LiftedKilt, i think this might be an issue with the MAAS provider, I see some bugs open on Juju 2.0-beta3 with MAAS 1.9.1
[16:49] <cmars> LiftedKilt, my suggestion is to open a bug at https://bugs.launchpad.net/juju-core
[16:50] <cmars> LiftedKilt, include terminal output from running the juju status command, as well as machine logs off the controller machines
[16:51] <cmars> LiftedKilt, also the controllers.yaml and models.yaml, with private keys scrubbed if necessary
[16:51] <LiftedKilt> cmars: I wonder if the maas database is getting corrupted, and causing the machine id to stop correlating with the correct node number
[16:52] <cmars> LiftedKilt, it's possible. I know there are some other maas-related bugs right now impacting beta3, this may help provide a clearer picture
[16:53] <LiftedKilt> cmars: ok I'll start gathering some info
[16:53] <cmars> LiftedKilt, thanks!
[16:54] <thedac> jamespage: anyt time for a finaly review? https://code.launchpad.net/~thedac/charm-helpers/apparmor/+merge/290096
[16:54] <LiftedKilt> cmars stormmore rick_h_: thanks for the assistance - I'll keep poking at it
[16:54] <thedac> s/anyt/any
[18:11] <cholcombe> how do you get the JUJU_REMOTE_UNIT in a layers interface?
[18:24] <marcoceppi> cholcombe: the conversation is the JUJU_REMOTE_UNIT typically
[18:24] <marcoceppi> IIRC
[18:24] <marcoceppi> cory_fu: ^?
[18:26] <cholcombe> marcoceppi, i'm basically trying to recreate this with layers: https://github.com/openstack/charm-ceph-mon/blob/master/hooks/ceph_hooks.py#L540
[18:26] <cholcombe> getting the service_name out of the relation in the interface
[18:28] <cory_fu> cholcombe: The conversation instance will have a "units" property that is a list of the units for which that conversation applies.  If the relation scope is UNIT, then it will be a list with only one element.  Otherwise, it may contain more
[18:29] <cholcombe> cory_fu, hmm ok
[18:32] <cory_fu> marcoceppi: Relying on conv.scope to have a particular value was a mistake on my part and I never should have recommended it.  :(
[20:29] <cory_fu> Can I get a review: https://github.com/juju-solutions/layer-basic/pull/51
[20:46] <kwmonroe> +1, merged cory_fu