| kwmonroe | +1 R_P_S, i appreciate the lesson! | 00:41 |
|---|---|---|
| siraj | REGISTER sirajsaadi itssiraj@gmail.com | 05:41 |
| siraj | sorry | 05:41 |
| === frankban|afk is now known as frankban | ||
| sirajsadi | is juju charm are stable? | 08:58 |
| EdS | Morning Juju people :) I have just run into ssh WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! when using juju ssh. I'm unsure what's happened here, how do I fix this issue? If it makes any difference I've deployed into a MAAS self hosted setup. | 09:15 |
| EdS | Following up on my issue earlier, I think that this remote host error is benign, but somehow the known hosts from a previous cluster have been retained. Any ideas where Juju keeps ssh known hosts? | 11:07 |
| === petevg is now known as petevg_afk | ||
| jamesbenson | kwmonroe: thanks for the vote of confidence :-) | 15:17 |
| jamesbenson | stokachu : thanks for commenting on the post this morning to help push it along :-) | 15:18 |
| bdx | kwmonroe: one of the fights I always end up fighting seems to be "juju automatically exposes ports to the wan that I dont want exposed on the wan" | 15:24 |
| bdx | whether it be company policy, or red tape for a client | 15:24 |
| bdx | just a bad practice none the less | 15:25 |
| kwmonroe | roger that bdx. but can't we all just get along on the internet? don't hack me bruh, i'm just over here trying to make a living. | 15:25 |
| bdx | aha | 15:26 |
| bdx | yeah | 15:26 |
| bdx | tell that to walmart | 15:26 |
| kwmonroe | lol | 15:26 |
| cory_fu | bdx: I landed a change to the Endpoint PR yesterday evening that might impact you. Specifically, I removed the context collection, because it made it inconsistent to use interfaces with different implementations. Not sure if you were using that or not | 15:27 |
| kwmonroe | bdx: what ports get auto exposed? i get 22.. maybe 17070 for cloud-hosted controllers.. others? or is it the case that 'juju expose' opens things too much? | 15:28 |
| bdx | kwmonroe: http://paste.ubuntu.com/25981632/ | 15:28 |
| bdx | notice whats on the private (nat) subnet and whats on the igw subnet | 15:28 |
| bdx | cory_fu: that definitely will, checking omp | 15:28 |
| kwmonroe | yeah, ack bdx. btw, i see you've got some sense in that deployment. no artful nonsense in there ;) | 15:29 |
| bdx | right, https://github.com/jamesbeedy/redis-snap | 15:30 |
| bdx | like the redis 4.0.2 too huh | 15:30 |
| cory_fu | bdx: It's a little bit more verbose, but you can use endpoint_from_name('endpoint_name') as a drop-in replacement. Also, we're formalizing recommending everyone import things directly from the top level (`from charms.reactive import endpoint_from_name`) rather than depending on the internal organization which we might need to change as we refactor things. | 15:31 |
| bdx | I see, yeah silly me, I missed that srry :P | 15:32 |
| stokachu | jamesbenson: np | 15:39 |
| bdx | cory_fu: that was a needed change | 15:42 |
| cory_fu | :) | 15:42 |
| bdx | cory_fu: really nice work there | 15:42 |
| cory_fu | Thanks | 15:43 |
| cory_fu | I think it's about ready to land as a beta feature, then we'll need to get a new release cut | 15:43 |
| bdx | thats awesome | 15:43 |
| bdx | please do | 15:45 |
| === frankban is now known as frankban|afk | ||
| R_P_S | hey, so I have been continuing to follow these instructions, but the kubernetes-master and kubernetes-worker instances are ending up blocked | 18:33 |
| R_P_S | https://insights.ubuntu.com/2017/02/08/automate-the-deployment-of-kubernetes-in-existing-aws-infrastructure/ | 18:33 |
| R_P_S | kubernetes-master/0* blocked executing 6 34.201.56.143 Missing kubernetes resource | 18:33 |
| R_P_S | I found this in a controller log: ARNING juju.resource.resourceadapters charmstore.go:123 attempt 2/3 to download resource "kubernetes" from charm store [channel (stable), charm (cs:~containers/kubernetes-master-11), resource revision (-1)] failed with error (will retry): revision must be a non-negative integer | 18:36 |
| R_P_S | but I don't understand why the resource revision is negative in the first place | 18:36 |
| bdx | "but I don't understand why the resource revision is negative in the first place" - there is a fix in place for this I think | 18:37 |
| bdx | R_P_S: thats actually a bug I think | 18:37 |
| R_P_S | where exactly is the bug? Do I need to find a different version for something? | 18:39 |
| R_P_S | this affects both the masters and the workers, so I'm guessing this is something underlying? | 18:39 |
| bdx | R_P_S: trying to find it omp | 18:46 |
| bdx | R_P_S: https://bugs.launchpad.net/juju/+bug/1723970 | 18:48 |
| mup | Bug #1723970: unable to attach zero-length resource <papercut> <usability> <juju:Fix Committed by thumper> <https://launchpad.net/bugs/1723970> | 18:48 |
| bdx | looks like it is in the rc1 | 18:48 |
| bdx | R_P_S: `sudo snap install juju --edge` | 18:49 |
| bdx | or `sudo snap refresh juju --edge` if you've already got it, which I assume you do | 18:50 |
| R_P_S | a buig in juju itself? | 18:50 |
| bdx | R_P_S: yea, the fix has already landed though, see ^ | 18:51 |
| R_P_S | I guess I need to wipe the apps and redeploy | 18:52 |
| R_P_S | I find it strange that this didn't affect my conjure-up initial test... I wonder if I installed juju via apt for that one | 18:54 |
| R_P_S | and based on the version, when snap installs version 2.3.x by default, it'll include this fix | 18:57 |
| bdx | R_P_S: not any 2.3, only the one in edge | 19:00 |
| bdx | R_P_S: are you using JAAS? | 19:00 |
| bdx | or you have your own controller(s) deployed? | 19:01 |
| R_P_S | I created my own controller | 19:01 |
| bdx | if you are using jaas it won't matter | 19:01 |
| bdx | ok cool | 19:01 |
| bdx | yeah, from my understanding, you will need to redeploy that controller with your newly install 2.3rc1 snap | 19:01 |
| R_P_S | oh damn, the entire controller needs to go? | 19:02 |
| bdx | well, I think you can upgrade it, but I'm not exactly sure how that works from beta -> rc1 | 19:02 |
| R_P_S | ok, well, I'll get started on that... in the meantime, I have a couple concept questions | 19:03 |
| R_P_S | I'm confused about expsoing of the workers | 19:04 |
| R_P_S | this tutorial doesn't include a load balancer, but I noticed the conjure-up does. If there's a load balancer, wouldn't it just need to be exposed instead of the workers? | 19:05 |
| R_P_S | I would like to be able to create the workers in a private subnet, so they'd have no external IPs | 19:06 |
| R_P_S | exposing the workers just seems strange in that there's no way to know which worker a pod is on, so having an external IP seems counterintuitive | 19:11 |
| bdx | R_P_S: yeah | 19:15 |
| bdx | R_P_S: you totally should do that | 19:16 |
| bdx | you have to setup spaces with your subnets and all | 19:16 |
| R_P_S | srry, can't tell which way your statement is going... do or do not expose workers? | 19:16 |
| R_P_S | yes, I've setup workers, and yesterday worked around the bugs surrounding add-subnets | 19:16 |
| bdx | R_P_S: it just depends how you do your ingress | 19:17 |
| bdx | but yeah, if you are using a proxy infront of the workers for application level ingress | 19:17 |
| R_P_S | now if I were to expose my workers, does that mean I could technically connect to a service running inside kubernetes on the workers via any one of their IPs and it would just magically route to an appropriate worker if that worker doesn't contain a specific pod? | 19:18 |
| bdx | then you can totally put subnets behind a nat gateway (by making a new routing table that routes to the nat gateway instead of the igw and using this routing table with your subnets you want to be private) | 19:18 |
| bdx | then you can (as I do) hide everything in the nat subnets, and just proxy to them from the things that are deployed to igw subnets where the instance gets a wan ip | 19:19 |
| R_P_S | yeah, I built the private subnets in AWS a couple days ago. we just had public prior to that | 19:20 |
| R_P_S | so does this mean that the masters could also technically be in the private subnet? | 19:20 |
| R_P_S | I built the private subnets the instant I saw the 0.0.0.0/0 SSH ACLs. the less instances with publicly accessible IPs, the better | 19:21 |
| bdx | right right | 19:25 |
| bdx | well it all depends on what the ingress story looks like there | 19:25 |
| bdx | if you are operating behind a vpn, or if you want this thing to be publicly accessible, etc etc | 19:26 |
| bdx | if nothing needs to talk to the masters from the wan then .... | 19:27 |
| R_P_S | our security model is fairly basic. admin access via a bastion host (with port forwarding for gui stuff) | 19:27 |
| bdx | R_P_S: what does "admin access" mean? | 19:28 |
| bdx | like a host inside the vpn? | 19:28 |
| bdx | admins get to login to it and use juju from there etc etc? | 19:28 |
| R_P_S | yeah | 19:28 |
| bdx | I see | 19:28 |
| R_P_S | does conjure-up support spaces? When I first used conjure-up, it didn't even ask about VPCs and actually built it in ec2-classic | 19:29 |
| stokachu | R_P_S: not yet | 19:29 |
| R_P_S | before I defined spaces, it was randomly creating instances in both the public and private subnets | 19:29 |
| bdx | stokachu: can't he just clone the spells repo, and put spaces constraints in the k8s bundle? | 19:30 |
| stokachu | bdx: yea | 19:30 |
| bdx | R_P_S: ^^, then use `conjure-up --spells-dir spells/` (I think) | 19:31 |
| stokachu | `conjure-up --spells-dir spells/ --nosync` | 19:31 |
| R_P_S | hrmm... I think I'd like to try and avoid that path for now. I don't even know where the spells repo is... | 19:33 |
| R_P_S | I'll retry without conjure up using the edge juju | 19:34 |
| bdx | R_P_S: https://github.com/conjure-up/spells | 19:34 |
| bdx | git clone https://github.com/conjure-up/spells | 19:34 |
| R_P_S | I do feel like that's a rabbit hole, and at this point I am under the gun to complete an evaluation here at work | 19:36 |
| R_P_S | to quote my manager "everyone uses kubernetes, it can't be that hard, just find the right script to get it up and running" :( | 19:36 |
| tvansteenburgh | HAHAHAHA | 19:37 |
| bdx | lol | 19:37 |
| bdx | right | 19:37 |
| tvansteenburgh | clearly he must be idling in different slack channels than i am | 19:37 |
| R_P_S | i know :( | 19:37 |
| R_P_S | to be honest, O | 19:37 |
| R_P_S | I'm consdering this an RGE, and trying to learn as much myself about kubernetes while still delivering | 19:38 |
| bdx | somehow the whole KOPS thing spread like wildfire | 19:40 |
| R_P_S | that was likely my next option if juju completely failed | 19:40 |
| bdx | glad to see conjure-up is the first item in the list of tools to install production grade k8s on the site though | 19:40 |
| bdx | https://kubernetes.io/docs/getting-started-guides/aws/ | 19:41 |
| R_P_S | but despite all the issues with juju, I still seem to be able to muddle my way through, and the conjure-up did technically work, but had a few anti-patterns that just made me weep | 19:41 |
| bdx | stokachu: am I blind, or are the bundles not in the spells repo anymore? | 19:41 |
| stokachu | bdx: only if you want custom bundles, otherwise it pulls from charmstore | 19:41 |
| R_P_S | which is why I moved away from conjure-up to that tutorial that manually creates the controllers and apps for a kubernetes cluster | 19:41 |
| bdx | stokachu: so a user can manually override the charmstore bundle by using --nosync and supplying their own bundle.yaml in the spell root? | 19:42 |
| R_P_S | anyways, it's lunchtime... I'll be back at this after lunch | 19:42 |
| stokachu | bdx: yep exactly | 19:43 |
| bdx | k, cool, thx | 19:43 |
| bdx | stokachu: can I choose an existing model to deploy to? | 19:53 |
| bdx | stokachu: I got as far as being able to choose my existing controller, but it went straight to review and configure applications | 19:53 |
| bdx | so, the model that I have predefined with the spaces | 19:54 |
| bdx | is moot? | 19:54 |
| bdx | in terms of what conjure-up can do with it | 19:56 |
| bdx | R_P_S: when you get back, https://gist.github.com/jamesbeedy/608a2c819ed852b89de203d7f95cd22e | 19:58 |
| bdx | R_P_S: put ^ in a file called "k8s-bundle-with-spaces.yaml" | 19:58 |
| bdx | then you can | 19:58 |
| bdx | `juju deploy k8s-bundle-with-spaces.yaml` | 19:59 |
| bdx | R_P_S: assuming you have 2 spaces defined: nat, igw | 19:59 |
| bdx | here's what my spaces look like http://paste.ubuntu.com/25983157/ | 20:00 |
| bdx | 3 subnets in each space | 20:00 |
| bdx | with each subnet having an affinity to an az | 20:01 |
| bdx | possibly creating spaces could be a pre-deploy step | 20:03 |
| bdx | so that you can use the model that conjure-up create | 20:03 |
| bdx | and adding spaces could just be a step | 20:03 |
| bdx | R_P_S: it will only take about ~5 minutes after you run `juju deploy k8s-bundle-with-spaces.yaml` | 20:06 |
| R_P_S | that looks pretty straight forward once I have the model and spaces setup | 20:11 |
| bdx | R_P_S: then you will have a beautiful thing (takes about 10mins for it all to settle actually) | 20:12 |
| R_P_S | although for the most part, that just looks like a single file that contains pretty much everything I'd built so far with individual commands | 20:13 |
| R_P_S | looks like I'd just need to add my instance type constraints | 20:14 |
| R_P_S | curious about the master... don't we want that to be redundant? | 20:17 |
| stokachu | bdx: deploy to existing models isn't supported yet either | 20:17 |
| stokachu | it's on our roadmap though | 20:17 |
| bdx | stokachu: cool | 20:18 |
| bdx | R_P_S: this is what it looks like with the spaces constraints when it all settles http://paste.ubuntu.com/25983232/ | 20:18 |
| bdx | R_P_S: yes, add you instance-type constraints where desired | 20:19 |
| bdx | and any other machine constraints | 20:19 |
| bdx | R_P_S: just deploy the bundle I shared with you first | 20:19 |
| R_P_S | yeah, model helps definitely, since I can also put AWS tagging in the models. I have a jenkins job that emails devops daily if it finds any instances not tagged or incorrectly tagged | 20:19 |
| bdx | totally | 20:19 |
| R_P_S | sorry, deploy the bundle shared fisrt? | 20:20 |
| bdx | yea, then add your mods 1 at a time | 20:20 |
| bdx | you will have much better milage when just getting introduced to all this if you just get the base stack up (which it sounds like you have ) then make these mods 1 at a time | 20:21 |
| bdx | instead of seeing the end goal (all the mods and constraints and customizations) and trying to get it all in the first go around | 20:21 |
| R_P_S | yeah, I scewed around with the default conjure-up for a while... but I needed to fix some of those base issues with security | 20:22 |
| bdx | right | 20:22 |
| R_P_S | I hope to have this up and running soon, then I can go back to dev and have them break everything inside kubernetes :P | 20:22 |
| bdx | :) good luck | 20:23 |
| R_P_S | what are gui-x and gui-y... I've never seen those before | 20:25 |
| R_P_S | oh, those are for those map things | 20:25 |
| R_P_S | are those required values? | 20:25 |
| R_P_S | or can I just strip those out? | 20:26 |
| R_P_S | oh, that yaml file is called a bundle... I didn't know that :P | 20:26 |
| R_P_S | is there a reason the subnet constraints are specified twice? | 20:31 |
| R_P_S | each app had constraints, but then each machine at the bottom also had the constraints... is the redundancy necessary? | 20:31 |
| bdx | yeah | 20:32 |
| bdx | I think | 20:32 |
| bdx | idk test it out :) | 20:33 |
| bdx | yeah, you can strip the gui-{x,y} if you don't care about the presentation in the gui | 20:33 |
| bdx | the redundancy is necessary, because you are deploying machines to the desired constraints | 20:34 |
| bdx | so like put this machine in nat space | 20:34 |
| bdx | then at the application level | 20:34 |
| bdx | you are saying "I only go to places where these things are true" | 20:34 |
| R_P_S | ah, so the machines are for the first instance, but then the appluications is for increasing the app later type of thing? | 20:34 |
| bdx | exactly | 20:34 |
| bdx | (I think) | 20:35 |
| bdx | hehe | 20:35 |
| bdx | thats my disclaimer to everything | 20:35 |
| R_P_S | seems a bit odd cause the single juju dieploy command creates the instance and the app with a single set of constraints | 20:35 |
| bdx | right | 20:35 |
| R_P_S | juju deploy --constraints "instance-type=t2.medium root-disk=32G spaces=pre-pub" cs:~containers/kubernetes-master-65 | 20:35 |
| bdx | totally, but in the bundle, you have the "to:" stanza | 20:36 |
| bdx | which to me indicates you want to deploy to the machine specified there | 20:37 |
| R_P_S | so that logically separates creation of instances from creation and deployment of apps? | 20:37 |
| R_P_S | fortunately the lines are identical... vi yy p p p to the rescue :D | 20:38 |
| bdx | R_P_S: you are right | 20:42 |
| bdx | it is redundant | 20:42 |
| bdx | this gives you the exact same thing https://gist.github.com/jamesbeedy/608a2c819ed852b89de203d7f95cd22e | 20:42 |
| R_P_S | oh, the entire machine section is gone! | 20:42 |
| bdx | yea, and the "to:" directives | 20:44 |
| bdx | stanzas | 20:44 |
| bdx | whatever | 20:44 |
| R_P_S | right, just checked the diff, I didn't spot that myself | 20:44 |
| R_P_S | I can't remember if I asked this before... but can I manually modify the security groups after deployment? or will juju go back at some point in the future and overwrite my changes | 20:48 |
| bdx | juju will keep persistence on them, people have workarounds for making mods that persist, but the ports that are open on the instances are open for good reason | 20:49 |
| bdx | R_P_S: either way, for things like that, you have to write the mailing list | 20:49 |
| R_P_S | at least the private subnet will protect most of the machines from those 0.0.0.0/0 ACLs | 20:50 |
| bdx | yeah, but really, how is juju suppose to know where you are ssh'ing from | 20:51 |
| R_P_S | hah, so I had a typo in there with one of the spaces... one of the instances is hung at pending | 20:52 |
| R_P_S | well, in terms off SSH, it feels like it could fit into the model. Using internal IPs with SSH ACLs based on the subnet CIDRs | 20:53 |
| R_P_S | like a model flag for internal. Then that would provide basic support for a bastion host VPC model | 20:54 |
| R_P_S | hey, is there any way to recover this bundle with the space typo? or do I just wipe and rebuild | 20:54 |
| R_P_S | machine 4 (the LB) is never going to spin up... | 20:55 |
| bdx | R_P_S: there are primitives in juju that exist solely to remediate the idea you have of this "bastion host" | 20:58 |
| bdx | R_P_S: I urge you to ditch that whole gang bang | 20:59 |
| bdx | immediately | 20:59 |
| R_P_S | not everything is going to be cerated by juju though... this is just for apps that can be containerized... the bastion host is meant to help secure the rest of our resources including AWS RDS and non-containerized apps | 21:01 |
| R_P_S | I'm curious to hear more about why you recommend ditching bastion hosts, and what kind of setup you'd do in place? | 21:06 |
| R_P_S | heh, I just checked wikipedia... and while I use the terms interchangeably, it appeasr that a jump server is a better definition of what I'm trying to build with | 21:17 |
| R_P_S | now since we don't have a VPN between our office and AWS, it technically still is a bastion host, but I apologize for any confusion between jump vs bastion host | 21:18 |
| R_P_S | hey bdx, I'd like to thank you for all your help. I'm not sure if I'll be quite done with the with the juju layer, but hopefully everything forward is inside kubernetes now | 22:16 |
| bdx | R_P_S: any time | 22:17 |
| bdx | R_P_S: about the bastion host | 22:17 |
| bdx | juju lets you operate it from the juju client | 22:17 |
| bdx | and juju treats "users" as first class citizens so as to give you a supported user management system | 22:17 |
| bdx | controlling acls and user access via the primitives for user management in juju itself | 22:18 |
| bdx | will give you much better milage in the long run then using the "bastion host" thing | 22:18 |
| bdx | when used correctly, anywhere a juju client is run is essentially a "bastion host" or "jump host" | 22:19 |
| bdx | because the security mechanisms in place in juju itself give you identity management (which is basically what you are doing with your bastion host) | 22:20 |
| bdx | you should take advantage of it, instead of doing the thing you do | 22:21 |
| bdx | just an idea | 22:21 |
| bdx | do whatever you please | 22:21 |
| R_P_S | I'd love to be able to ditch the jump host security layer, but experience has taught me to always reduce attack vectors. I'd love to have confidence in the authorization mechanisms of every service and tool, but the reality is different. | 22:23 |
| bdx | right, so juju takes advantage of things like key based access | 22:30 |
| bdx | it doesn't matter if you are key-based access from your laptop or from the bastion host | 22:30 |
| bdx | its the same thing | 22:30 |
| bdx | the bastion host is just an unneeded hop .... possibly I just don't see the extra layer of security you get from it | 22:31 |
| R_P_S | except we have ACLs on our jump host to allow only from our office gateways. our jump host is not accessible publicly | 22:31 |
| bdx | oooo | 22:31 |
| bdx | ok then | 22:31 |
| bdx | :) | 22:31 |
| R_P_S | well, it technically is cause it's got a public IP and we don't have VPN between offices and AWS | 22:32 |
| R_P_S | but you'll find 0 ports open if you try hitting that IP from the wild | 22:32 |
| bdx | nice | 22:32 |
| bdx | https://jujucharms.com/openvpn/ | 22:32 |
| bdx | is what I use | 22:32 |
| bdx | I just deploy ^ to whatever vpc I'm working in | 22:33 |
| R_P_S | haha, the challenge is not the AWS side of the VPN... the challenge would be the office side | 22:33 |
| bdx | I see | 22:33 |
| R_P_S | the offices have... uh... ghetto hardware in places | 22:33 |
| R_P_S | and I know there are subnet clashes between some VPCs and our internal office address space | 22:35 |
| R_P_S | I inherited some... interesting AWS configs when I joined this company. Including a production VPC in a /24 with only a single AZ :P | 22:37 |
| bdx | lol | 22:38 |
| bdx | oh man | 22:38 |
| bdx | #beenthere | 22:38 |
| R_P_S | I don't think we had a single SG that referenced another SG prior to my arrival. everything had an EIP and nothing but IP whitelists everywhere | 22:40 |
| R_P_S | After a single point of failure got cryptolocked, management's ears tuned in a bit more on the security front. | 22:44 |
| bdx | "After a single point of failure got cryptolocked, " - this is exactly why not to use the "bastion host" | 22:53 |
| R_P_S | jump hosts don't need to be SPOF... they can easily be scaled horizontally with minimal state :) | 22:54 |
| R_P_S | this SPOF was particularily bad. it was hosted in godaddy, held a firmware coded IP in customer owned hardware, and we had practically no backups of the thing | 22:56 |
| R_P_S | I really need to write an ansible role for our jump hosts though. Most of it is just making sure the tools are installed and ready to go. Then it's just ndividual configuration of things like .bashrc etc | 23:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!