=== alexlist` is now known as alexlist [05:11] How do bundles handle exposing ports? or do they not? [05:11] I'm about to test this with a simple web and database charm deploy, but thought I'd ask. === frankban|afk is now known as frankban === gnuoy` is now known as gnuoy === spammy is now known as Guest3061 [10:54] marcoceppi and juju at devopsdays Amsterdam in about an hour: http://livestream.com/accounts/20260359/devopsdays-amsterdam-2016 === fginther is now known as fginther` [11:11] woop === bradm_ is now known as bradm === fginther` is now known as fginther [11:54] lp:charms/trusty/ubuntu-repository-cache exists, I prepared a Xenial version at https://code.launchpad.net/~axino/charms/trusty/ubuntu-repository-cache/xenial-ready/+merge/298880, is this MP the right thing to do to make it happen ? [12:22] nice flip flops marcoceppi [13:33] lazyPower, good morning! [13:39] lazyPower, I've created an etcd-local-proxy charm at https://github.com/neiljerram/etcd-local-proxy (by starting from and hacking layer-etcd). I wonder if you might take a look at it and see if it looks sane to you? [13:40] lazyPower, There's also a needed PR for interface-etcd-proxy at https://github.com/juju-solutions/interface-etcd-proxy/pull/6 [13:43] lazyPower, However, when I build and deploy etcd-local-proxy, and also cs:~lazypower/etcd-19, I'm seeing TLS auth issues, so it's possible there's something wrong in how the certificates are generated, or in the etcd config at either end. Have you done successful end-to-end TLS testing with an etcd master and proxy? [13:56] magicaltrout hah, thanks ;) [13:58] good talk though [13:58] looks like you got a decent crowd [14:00] neiljerram - i have not. only verified the data exchange on the wire. If you need server certificates, we can do that but its a different dance over the relation wire. It involves generating a CSR and having it signed by the etcd cluster [14:02] s/cluster/leader/ [14:08] neiljerram - i think i knwo why you're getting ssl errors as well. the fork of etcd there is generating its own certificates [14:09] wait no, i jumped the gun i see you removed that from metadata [14:12] mbruzek - ping [14:12] Hello lazyPower [14:13] hey o/ neiljerram has been working with our latest release of etcd, and had some questions. Woudl be good to get your eyes on https://github.com/neiljerram/etcd-local-proxy [14:13] neiljerram: What TLS issues are you seeing? === narindergupta is now known as narinder [14:41] mbruzek, thanks - first I was seeing (IIRC) 'x509 certificate signed by unknown authority'; then a bit later 'remote error: bad certificate' [14:41] I will start a redeployment now so I can provide more detail... [14:43] neiljerram: OK. Can you describe the model? I have got successful etcd to etcd TLS before (not using proxy) [14:44] mbruzek, the model is etcdctl ----- local etcd proxy ----- remote etcd master [14:45] And you are running etcdctl on your machine or in the cloud? [14:45] mbruzek, there is TLS security for the connection between the proxy and the master; but no TLS security between etcdctl and the local proxy. [14:45] mbruzek, etcdctl invocations are on the same machine where the proxy is running [14:47] mbruzek, so the etcd master is cs:~lazypower/etcd-19, which is Charles's latest unreleased version of layer-etcd [14:47] mbruzek, and the etcd proxy is https://github.com/neiljerram/etcd-local-proxy, which is a charm that basically just uses the 'requires' side of interface-etcd-proxy [14:54] mbruzek - i pushed etcd-19 in my namespace due to the missing etcd-proxy bits. We translated the interface, but missed the reactive code to support that [14:55] everything but the reactive code in layer-etcd has merged in master of their respective repositories though, so we can rev ~containers once that lands [14:55] if its ready for that [14:56] I just diffed etcd-19 and etcd-local-proxy the etcd.py looks quite different. Let me see if I understand correctly, you are connecting the etcd charm with a different etcd-proxy charm and the tls is not working? [14:57] from etcdctl on proxy and the etcd on the master. [15:05] mbruzek, That's all correct. I really just used etcd-19 as a structural starting point, but etcd-local-proxy does a lot less, because it doesn't have to do leader election or registration, or generate its own certificates. In principle - I think - it just needs to do the 'requires' side of the 'proxy' relation, save the credentials once they're available, update /etc/defaults/etcd, and restart the local etcd (proxy) service. [15:07] neiljerram: Yeah that looks right. [15:07] hey everyone, is there any method for pulling the machine ID or INS-ID inside a hook or action? [15:07] My deployment is up now, and here's what I see: http://pastebin.com/Xs4YLDtY [15:09] neiljerram: OK please add to the pastebin: openssl x509 -in /tmp/etcd_cert -text [15:11] neiljerram: That will print out the text of the certificate to make sure the one on your system is valid. [15:11] I suspect it is, but want to start there. [15:11] mbruzek, http://pastebin.com/4Dfa2r1V [15:12] neiljerram: And these addresses are correct ? IP Address:54.164.27.251, IP Address:172.31.51.7 [15:13] mbruzek, 54.164.27.251 is the public IP of the AWS instance that contains etcd-19. [15:14] and 172.31.51.7 is that same machine's private IP [15:14] OK that is good so I believe the certificate is valid and stored correctly on the proxy machine. [15:14] (the machine itself only knows its private IP) [15:14] neiljerram: Yep [15:17] neiljerram: Can you run the following? : etcdctl --debug --cert-file=/tmp/etcd_cert --key-file=/tmp/etcd_key --ca-file=/tmp/etcd_ca and pastebin the result? [15:17] wait add the "ls /" to the end [15:20] mbruzek, that seems to work: http://pastebin.com/3ze47iCz === narinder is now known as naridner === naridner is now known as Narinder [15:21] sweet! [15:21] OK so I think the problem is that etcd-proxy reads the defaults file, but etcdctl does not. [15:23] neiljerram: I am not familiar enough with etcdctl to know how to make those values default, some tools use env variables, other tools have a confiuration file. [15:23] But I didn't think that the hop from etcdctl to the local proxy should use TLS at all... So I'm a bit confused! [15:24] neiljerram: Are you sure that etcd-proxy is properly configured with these keys/certs? [15:25] I was just going to pastebin the proxy's config.... [15:26] ... http://pastebin.com/ewTBy5mJ [15:27] I wonder if "PEER" is not valid in this case, let me do some searching [15:28] Also I know they are in the defaults file, but are we sure that etcd is actually getting the values? Do you see in the etcd logs that the key/cert/ca is read in? [15:28] https://coreos.com/etcd/docs/latest/security.html [15:31] mbruzek, Yes, I think so: http://pastebin.com/jhg1HLve (I just added ETCD_CA_FILE and other non _PEER_ settings to the config - but that didn't make any difference to our observations so far) [15:33] neiljerram: So after doing that, you are still not able to connect using etcdctl without the cert and key flags? [15:33] mbruzek, Perhaps the etcd proxy doesn't really support TLS on one side (towards the cluster) but not on the other (towards the client). So when it gets a client request it requires TLS auth, even if that client request is over http. Is that possible? [15:33] mbruzek, Yes, correct. [15:34] mbruzek, and with those flags, I still _can_ connect [15:34] neiljerram: I don't know etcd proxy well enough to answer that. [15:35] neiljerram: Great. Still that is a pretty bad user experience, let me see if I can find more information about etcdctl to see if we can eliminate those flags [15:35] mbruzek, Do you know who implemented the 'pillowmints' in the etcd charm? I think they support that hypothesis. (They set up env vars for etcdctl.) [15:36] neiljerram: "pillowmints" was used by lazyPower for lack of a better name. I questioned him about that during a code review. [15:37] mbruzek, It is a curious name :-) but nice. [15:38] mbruzek, But it sets up these vars, for local etcdctl usage on the machine where the etcd master node is installed: [15:38] ETCDCTL_CA_FILE=/etc/ssl/etcd/ca.pem [15:38] ETCDCTL_CERT_FILE=/etc/ssl/etcd/server.pem [15:38] ETCDCTL_KEY_FILE=/etc/ssl/etcd/server-key.pem [15:38] Oh there you go! [15:39] export ETCDCTL_CERT_FILE and the others and try etcdctl again [15:42] mbruzek, Yes, that works. But I still don't feel like I understand why. How does TLS auth mix with doing a request over http:// (i.e. _not_ over https://) ? [15:46] neiljerram: According to your pastebin with debug http://pastebin.com/3ze47iCz the http address is getting converted to https and my guess is that the etcdctl is making the direct https connection [15:47] neiljerram: etcdctl must be talking http to the proxy but perhaps it is actually talking https to the server as well. [15:48] mbruzek, Oh! So the proxy isn't really proxying; but rather redirecting! [15:53] neiljerram: That would be my assessment from reading the pastebin you have there. [15:57] mbruzek, It appears this is all a behaviour of etcdctl, rather than of the proxy. 'etcdctl --no-sync ls /' works even without those env vars. [15:58] neiljerram: Interesting [15:58] mbruzek, Thanks so much for helping me to understand all this. I think I have all the pieces and understanding that I need now. [15:58] lazyPower, Thanks also. [15:59] neiljerram: Excellent, glad to help. If you need anything else ping me here. [15:59] mbruzek, Thanks, I'll do that. [15:59] I am the one that wrote the TLS layer so it is my responsibility. What I have found is TLS is hard [16:09] mbruzek i try to help :) === dpm is now known as dpm-afk [17:12] After a reboot a series of my units are stuck with "agent is lost, sorry! See 'juju status-history glance/0'" The status-history is normal until the reboot and nothing since, any way to kickstart a container back to life? (2.0-beta) === frankban is now known as frankban|afk [18:01] xilet: did IP addresses change? [18:03] No [18:10] reactive is doing some odd things i'm having trouble figuring out [18:11] i have a base layer that sets a state after it installs everything. in my layer that uses it i @when() that state and for some reason the base layer never installs what it should but the state is firing [18:11] i r confused :) [18:34] nvm icey and i figured it out [19:44] It seems like --config has been removed from deploying bundles? [19:44] If I create a bundle for a vanillastack application and want to keep the configuration options separate what's the prescribed way in 2.0? === zeus is now known as Guest31961 === Guest31961 is now known as zeus [19:55] ERROR Flags provided but not supported when deploying a bundle: --config [19:55] Excellent. [19:55] Why should a deploy of a bundle be any different than a deploy for a charm? [19:56] They use the same command. === mup_ is now known as mup [20:34] kwmonroe: thank you for the cleanup on hbase. I will study all the changes and aling my work based on your pointers. Many thanks! === urulama|___ is now known as urulama|____ [20:56] np kjackal -- it wasn't too complicated, just doing actions a bit better than we used to. i also like the collapsed report_status() handler vs individual handlers per missing state.