[00:01] <stormmore> lazyPower, any suggestions on where, if anywhere, I should file a bug about that?
[03:57] <stormmore> Anyone around familiar with K8s services?
[04:56] <stormmore> lP|lappy, just the person
[04:56] <stormmore> I am having problems trying to get a Service of type LoadBalancer to get an external IP from AWS
[04:56] <lP|lappy> stormmore: i think i lost some context
[04:56] <lP|lappy> ah it was incomplete
[04:57] <stormmore> I found an answer that says that I need --cloud-provider=aws as a flag passed to both the apiserver and controller-manager
[04:57] <lP|lappy> stormmore: so, we dont fully support Type: LoadBalancer. is a cloud specific feature and we've modeled with a bare metal approach as its the only consistent interface we can guarantee cross cloud. Is it out of the question to use the ingress loadbalancer?
[04:58] <stormmore> other than the fact that will only work with the host IPs instead of giving the service it's own
[04:58] <stormmore> and that is kind of what I figured
[04:59] <lP|lappy> I'm not sure what you mean it will only work with the host ips... any worker charm that you have deployed that has ingress=true will act as a reverse proxy for your workloads.
[04:59] <lP|lappy> stormmore: you'll need to create both a service def and an ingress definition in order to use this addon, but it ships with CDK out of the box.
[05:00] <stormmore> just for full disclosure, I have tried to add that flag to /etc/default/kube-apiserver and /etc/default/kube-control-manager and now the kube-system pods won't start
[05:00] <stormmore> lP|lappy, the problem with that is that if the node goes down then it's connections and dns for that matter will not function correctly
[05:01] <stormmore> it is why I am looking at doing keepalived in my bare-metal environment
[05:01] <lP|lappy> stormmore: i'm not sure how defining a service with type: loadbalancer is going to solve that.... if your node goes down, any other worker thats still active should be able to tak the workload and ingress works cross host out of the box
[05:02] <stormmore> it solves that by pushing it off to ELB which is suppose to be 100% uptime
[05:02] <lP|lappy> ok so you're concerned that the worker unit itself, being the load balancer, is going to go down
[05:02] <stormmore> exactly
[05:02] <lP|lappy> and hten you're left with no connection vs a 503
[05:03] <lP|lappy> so, i would probably mitigate that with latency based dns response, and a loaded dns record... but you've brought up an interesting point
[05:03] <lP|lappy> stormmore: so, best i can offer today is to file a feature request for it. you're not the first to ask for it.
[05:04] <lP|lappy> but we'll need to prioritize that, and its not on the roadmap today. The downside to just plugging those flags into the configs is you also have work to do cloud side to make that work
[05:04] <lP|lappy> you will need to assign IAM roles to your ec2 instances so they can make api requests on your behalf without key authentication
[05:05] <stormmore> ah that might explain why the kube-system pods aren't starting correctly
[05:05] <lP|lappy> and thats not something we can reasonably automate with juju (that i'm aware of)
[05:05] <lP|lappy> thats a pretty ec2 specific feature
[05:05] <lP|lappy> s/ec2/aws/
[05:05] <stormmore> lP|lappy, well juju does something in the way of security groups when it creates the AWS instances
[05:05] <lP|lappy> this isn't security groups ;)
[05:05] <lP|lappy> its IAM roles
[05:09] <lP|lappy> lovely, they deleted my easy-reference material 8 days ago it looks like
[05:09] <lP|lappy> https://github.com/kubernetes/kubernetes/tree/master/cluster/aws/
[05:09] <lP|lappy> but the gist is, you create an IAM role you can assign the instances, you apply that IAM role to your control plane vm's and they can then request resources
[05:12] <lP|lappy> stormmore: https://github.com/fabric8io/kansible/blob/master/vendor/k8s.io/kubernetes/cluster/aws/templates/iam/kubernetes-master-policy.json
[05:13] <lP|lappy> stormmore: https://github.com/fabric8io/kansible/blob/master/vendor/k8s.io/kubernetes/cluster/aws/templates/iam/kubernetes-minion-policy.json
[05:13] <lP|lappy> these are two such policy json objects that can be used to create your ARN's
[05:15] <lP|lappy> But I'm off to bed. I'll check in on you tomorrow and see how things are going. Cheers stormmore
[07:56] <kjackal> Good morning Juju world!
[10:07] <kklimonda^> is there a documentation on how to configure juju to use an existing controller somewhere?
[10:11] <anrah_> kklimonda^: https://jujucharms.com/docs/2.0/tut-users
[10:12] <kklimonda^> wow, thanks - that was actually easier than expected
[10:18] <anrah_> that is quite nice feature :)
[10:19] <kklimonda^> so add-model complains about credentials missing, even though I've granted the user superuser access - I guess I'm hitting https://bugs.launchpad.net/juju/+bug/1630372
[10:19] <mup> Bug #1630372: "ERROR no credential specified" during add-model as non-admin user <cwr-ci> <matrix> <usability> <v-pil> <juju:Triaged> <https://launchpad.net/bugs/1630372>
[10:20] <kklimonda^> I think I have to add user to MAAS, and then add those credentials to Juju? but that doesn't sound right, I thought juju controller would abstract that part
[10:23] <kjackal> kklimonda^: you do juju add-user myuser; juju grant myuser superuser; and then on the other machine you will need to do juju add-credential with the credentials of the cloud you are expected to use
[10:23] <kjackal> kklimonda^: no wait a sec, let me doublecheck my notes
[10:24] <kklimonda^> yes, but both accounts will be using the same cloud (MAAS)
[10:24] <kjackal> kklimonda^: you need to do a juju register <registration_string_you got from add-user>
[10:25] <kklimonda^> I've done that part
[10:25] <kklimonda^> but it's still missing credentials when I call juju add-model
[10:25] <kklimonda^> and to add credentials, I seem to have to add cloud
[10:25] <kklimonda^> so that's a lot of manual steps
[10:25] <kklimonda^> (I've also granted superuser for that new user)
[10:26] <Hetfield> kjackal: i still have issues on how juju picks ipaddress :(
[10:26] <kklimonda^> I'm just missing the relationship between juju, controller, users and the underlying cloud (MAAS)
[10:26] <kjackal> kklimonda^: there are some manual steps. You need to create a file with the credentails of MAAS
[10:27] <kklimonda^> I've assumed that I can share juju controller between users (that seems to be the case) and that juju controller will abstract an underlying cloud, because authorization is already done in Juju
[10:27] <kklimonda^> but now it seems I should probably have a separate user in MAAS, and add those credentials to Juju
[10:29] <kjackal> kklimonda^: you need to copy the credentials you have in ~/.local/share/juju/credentials.yaml and place them on the side of the other client/user
[10:29] <kjackal> so as to do the juju add-credentials
[10:29] <Hetfield> kjackal: i checked https://jujucharms.com/docs/2.0/network-spaces but it doesn't fit my case: i creaed spaces in maas but constraints just say "pick a machine that has these space"
[10:31] <kjackal> Hetfield: ok, I am not the right person to help you. Can you please send an email to the list juju@lists.ubuntu.com ?
[10:33] <Hetfield> oki
[10:34] <kklimonda^> kjackal: I don't think copying actuall works, I had to generate a new API key in MAAS
[10:34] <kklimonda^> and that finally worked
[10:35] <kjackal> well done kklimonda^
[10:36] <kjackal> thanks Hetfield
[10:36] <Hetfield> thanks to you kjackal
[10:36] <cnf> how do I edit the connection details for a juju controller?
[10:39] <cnf> also, any documentation of how to put it behind a reverse proxy?
[10:44] <kjackal> cnf: is this what you are looking for: http://askubuntu.com/questions/174171/what-are-the-ports-used-by-juju-for-its-orchestration-services
[10:44] <kjackal> ?
[10:45] <cnf> uhm, mybe
[10:45] <cnf> juju isn't very forgiving for non-flat network setups :/
[10:48] <ybaumy> is it possible to make a controller HA?
[11:03] <kjackal> ybaumy: yes let me find the documentation for that
[11:04] <kjackal> ybaumy: https://jujucharms.com/docs/2.1/controllers-ha
[11:04] <cnf> so can you get juju-controller to listen on an IP not in the MAAS admin range?
[11:39] <cnf> hmm
[11:39] <cnf> i need a command reference for juju
[11:39] <cnf> how do i make it retry assigning machines?
[11:42] <cnf> and it seems i still don't have my networking set up right
[11:42] <cnf> o,O
[11:48] <cnf> so how to i edit the proxy that new machines need to use in juju?
[11:49] <cnf> hmz, juju is rather confusing to use :/
[11:57] <Hetfield> cnf: juju retry-provisioning
[11:57] <ybaumy> kjackal: thanks
[11:59] <cnf> Hetfield: thanks
[12:08] <cnf> hmm
[12:08] <cnf> how do i know that my deploy was successful?
[12:09] <cnf> Hetfield: retry-provisioning doesn't try to bring up a machine, again?
[12:11] <cnf> so it seems juju is pushing the http_proxy env vars to the nodes
[12:11] <cnf> but NOT the no-proxy setting
[12:11] <cnf> also, the MAAS proxy doesn't allow non 443 ports for https
[12:12] <cnf> any hints on how to fix this?
[12:17] <cnf> wow, juju is pissing me off
[12:31] <kjackal> cnf: Here is a list of the commands: https://jujucharms.com/docs/2.1/commands however you should also be using juju <command> --help to double check that the docs are uptodate
[12:31] <cnf> yeah...
[12:32] <kjackal> If you want to "free" a machine you can do juju remove-machine  (--force)
[12:32] <kjackal> cnf: ^
[12:32] <cnf> ok
[12:32] <cnf> thanks
[12:32] <cnf> i'm failing to deploy anything on maas with juju atm
[12:33] <kjackal> and for the proxies I am afraid you would better ask on the list as I have no experience in that subject
[12:33] <cnf> been at this for the better part of 5 days, i'm not having fun atm :/
[12:33] <tvansteenburgh> proxy info: https://github.com/juju/docs/issues/206#issuecomment-282773096
[12:33] <tvansteenburgh> (might help, i dunno)
[12:34] <cnf> tvansteenburgh: i'm past bootstrapping
[12:34] <cnf> took me a while to figure that out, as well
[12:36] <cnf> and i did a juju model-config no-proxy as well
[12:36] <cnf> but i don't think that took, either
[12:36] <cnf> takes friggin 15 minutes between tries, as well >,<
[12:37] <tvansteenburgh> does `juju model-config` show the value you set?
[12:38] <cnf> yeah
[12:39] <tvansteenburgh> what about `juju run someunit/0 'env | grep proxy'`
[12:41] <cnf> what is someunit?
[12:41] <tvansteenburgh> some unit in your deployment
[12:43] <tvansteenburgh> like, i unit name from the Unit column printed by `juju status`
[12:43] <tvansteenburgh> s/i/a/
[12:43] <cnf> hmm, doesn't do anything
[12:43] <cnf> now what
[12:44] <disposable2> cnf: your bootstrap timeout tip got me over the problem in the end. thank you
[12:44] <cnf> disposable2: np
[12:44] <cnf> hmm, the hosts can't reach the controller
[12:44] <cnf> wt?
[12:45] <tvansteenburgh> oops
[12:45] <tvansteenburgh> juju run --unit someunit/0 'env |grep proxy'
[12:45] <tvansteenburgh> what about that? ^
[12:46] <cnf> yeah, that doesn't do anything
[12:46] <cnf> just sits there
[12:46] <tvansteenburgh> it hangs?
[12:47] <cnf> yeah
[12:49] <tvansteenburgh> does juju debug-log --replay have any recent errors in it? sounds like you machine can't communicate with the controller. does `juju status` print anything or does that hang too?
[12:51] <cnf> juju status works
[12:51] <cnf> juju debug-log --replay just sits there
[12:57] <tvansteenburgh> cnf: what made you say "the hosts can't reach the controller"?
[12:57] <cnf> tvansteenburgh: i sshed to one of the machines
[12:57] <cnf> and i can't ping the controller
[12:57] <cnf> don't get arp replies, either
[12:58] <cnf> which is weird, because i sshed FROM the controller
[12:58] <cnf> no, i lie
[12:58] <cnf> i sshed from the MAAS controller
[12:58] <cnf> not the juju controller
[12:58] <tvansteenburgh> it's strange that `juju status` doesn't show 'agent lost' errors or something then
[12:59] <cnf> it has a lot of "waiting" and "pending"
[12:59] <tvansteenburgh> ahh
[12:59] <cnf> my juju controller runs in a KVM machine on the MAAS controller
[12:59] <cnf> i guess something is amis there
[13:00] <tvansteenburgh> yeah. i won't be much help sorting out your network probs, but it sounds like that's where the prob is
[13:18] <Hetfield> cnf: it's fine
[14:03] <kklimonda> I've created a new user in juju, using same maas user and a different api key, and deployed bundle to a different model.
[14:04] <kklimonda> now I'm trying to deploy a bundle using original user, and I get an error "cannot deploy bundle: POST [url] Forbidden"
[14:39] <kjackal> kklimonda: could you tell us how you created the new user using the same maas user and a different api key?
[14:40] <kjackal> kklimonda: I was under the impression that you only need to do a juju add-user mynewuser
[16:05] <kklimonda> @kjackal  I did juju add-user (that generated a snippet to be used with juju register)
[16:06] <kklimonda> @kjackal then I got a "credentials missing" when I tried creating a new model, so I went back to MAAS and created a new API key, which I've used with juju add-credentials
[16:06] <kklimonda> my assumption was that given I have different users in juju, and they are working on different models, that would work
[16:08] <kklimonda> and the error returned by Juju is too vague to be useful, I guess I'll have to login to the controller and perhaps its logs will tell me more
[16:09] <lp|Sprinting> mbruzek: ping
[16:10] <mbruzek> lp|Sprinting: pong
[16:12] <lp|Sprinting> mbruzek: hey i have some questions for you re: etcd storage design. (wifi here is terrible, do you mind doing this on irc?)
[16:13] <mbruzek> still at standup, parking lots
[16:13] <mbruzek> but yeah go ahead
[16:14] <lp|Sprinting> ack. - I may have painted myself in a corner. I wanted to model the storage path in layer.yaml to kind of segregate how we traditionally assumed it was in /var/lib/etcd, whereas now its moved to /var/snap/etcd/current. This is default behavior - but what i didn't account for is durable storage mounts which will need to change the path of that storage volume (presumably /media/etcd via the removable-media
[16:14] <lp|Sprinting> interface)
[16:15] <lp|Sprinting> Does it make sense to just nuke that layer.yaml configuration and make a switch in a lib somewhere that has a check "if is_state('storage.attached'): return '/media/etcd'" ?
[16:16] <lp|Sprinting> basically i'm asking what would make sense to you, as a consumer of etcd? I have limited options on where it can be mounted, because of snap confinement
[16:32] <mbruzek> lp|Sprinting: I don't like having to limit the mount points, but I understand that comes with the snap confinement. i am +0 it could go either way.
[16:35] <mbruzek> lp|Sprinting: We just got done with standup so I am more interactive now
[16:36] <lp|Sprinting> mbruzek: i think i answered my question above with the switch. I'm going to set a state and write a getter that determines what path we care about.
[16:36] <kjackal> kklimonda: instead of creating a new API key you should have done "juju credentials --format yaml --show-secrets" and on the side where the new user is you should do a "juju add-credential "yourcloud" --replace -f filewithcredentials.yaml"
[16:36] <lp|Sprinting> i don't immediately see a better way to do this
[16:37] <kklimonda> @kjackal so now I basically have two MAAS API keys competing with each other?
[16:37] <lp|Sprinting> mbruzek: something like the following: https://gist.github.com/chuckbutler/2e8ae364639504250499c838a33a20e1
[16:37] <kklimonda> (because it just started working randomly..)
[16:38] <mbruzek> lp|Sprinting: will /media/etcd-data work with snaps?
[16:39] <lp|Sprinting> mbruzek: interface: removeable-media
[16:39] <mbruzek> lp|Sprinting: What happens to the data in /var/snap/etcd/current if a volume is mounted after install?
[16:39] <kjackal> kklimonda: I am not sure what is going on right now, I have never played with maas
[16:39] <mbruzek> install etcd using it, oh look they support storage!, add storage. Where did my data go?
[16:39] <lp|Sprinting> mbruzek: same thing that happens with the current charms. service is stopped, data is rsync'd to target endpoint, config is updated and the daemon is restarted.
[16:40] <mbruzek> So the storage code would copy that information over, so the storage code would have to know about both directories.
[16:43] <lp|Sprinting> Which it does
[16:44] <lp|Sprinting> well if i change this where i'm thinking of changing it it's going to need some minor refactoring
[16:44] <lp|Sprinting> but yeah, it does/will.
[17:34] <stormmore> o/ juju world
[17:50] <stormmore> so lp|Sprinting the pointers you gave me last night got me most of the way to get it to work. had to make some manual changes to the LB to get it to work but it did create the LB for me
[17:50] <ybaumy> what happens if i add-unit --to a machine where there is already a unit installed .. will there be a lxd container build?
[17:51] <lp|Sprinting> stormmore: cool :) Glad I was able to get you moving
[17:51] <lp|Sprinting> stormmore: but I think having run the exercise manually you see why we're in a state of "we dont officially support this in any capacity out of the box" right?
[17:52] <lp|Sprinting> ybaumy: it will co-locate without lxd. we call this hulk smashing and discourage it
[17:52] <lp|Sprinting> ybaumy: if you want it to colocate in a container, you'll need to specify that in --to, eg:  juju add-unit mysql --to lxd/4
[17:53] <ybaumy> lp|Sprinting: but the container i have to build myself?
[17:53] <lp|Sprinting> ybaumy: nope. Container management is automated with juju. you just have to specify thats where you want it and juju will translate and load the lxd image for you, fire up the unit agent and the rest happens as usual accoding to the charm
[17:53] <lp|Sprinting> *according
[17:55] <ybaumy> lp|Sprinting: is there a piece of documentation for it?
[17:55] <stormmore> actual that is a design decision that I think might be a mistake, in my case this is a temporary measure while I build up our bare metal but no one seems to official support AWS very well
[17:55] <lp|Sprinting> ybaumy: https://jujucharms.com/docs/2.1/charms-deploying
[17:56] <lp|Sprinting> https://jujucharms.com/docs/2.1/charms-deploying#deploying-to-specific-machines-and-containers rather
[17:56] <stormmore> lp|Sprinting, I was always thinking about what you said about knowing if you are on a cloud provider... Juju itself knows what cloud it is on / deploying to, doesn't it provide a variable to the charms to help them determine/
[17:57] <ybaumy> lp|Sprinting: ah i see bootstraping it ok i get it. thanks for the doc. will go from there
[17:59] <lp|Sprinting> stormmore: nope. Charms are cloud agnostic and should strive to be as agnostic as possible. The idea is to make the learnings reusable on every substrate, and things that require special cloud-provider dances, should be encapsulated either as a charm (or i'm not sure what else would go here... gap in my thought) or we should help encourage things to not be tied to cloud specific features.
[18:02] <lp|Sprinting> stormmore: this is why we've been fairly up front about the fact we took a bare metal first approach. If you didn't have an ELB what would you do?  Probably deploy haproxy in front of your workers and scale it independently to act as your webhead yeah?
[18:02] <stormmore> lp|Sprinting, I kinda agree but charms should also take into account the environment they are being deployed to's capabilities
[18:02] <stormmore> lp|Sprinting, I would build it with keepalived and haproxy probably
[18:03] <stormmore> (which is basically what I am planning on doing with bare-metal
[18:07] <stormmore> oh and I plan on running those in the cluster itself instead of on the edge
[18:09] <lp|Sprinting> stormmore: not sure how thats any different than the ingress controller we are shipping ;)
[18:10] <lp|Sprinting> if its running in the cluster, and you're exposing as nodePort, you'll have the same issue if your workers die that you'll have to do some latency based routing w/ health checks at the dns level.
[18:14] <stormmore> lp|Sprinting, does the Ingress controller handle VIPs?
[18:16] <lp|Sprinting> https://github.com/kubernetes/contrib/issues/1140
[18:16] <lp|Sprinting> stormmore: ^ yeah, and there's a good issue thread about this and how its been testd here
[18:16] <stormmore> lp|Sprinting, if so, then I might change my plan. My idea is remove any potential single point of failures based on IPs
[18:16] <lp|Sprinting> stormmore: ingress just relies on dns. Doesn't seem to matter how you get the traffic to it
[18:17] <lp|Sprinting> stormmore: one thing that might be problematic for you, is if you're planning on using socket based workloads. There have been some dragons in there historically if you're wanting to proxy stuff like ssh ports and what not.
[18:17] <lp|Sprinting> thats one thing i'm aware of, and haproxy would be a welcome replacement there instead of asking for NodePort bindings.
[18:18] <lp|Sprinting> stormmore: however, endpoints pretty much abstract all of that, and yes VIP support is in there.
[18:19] <stormmore> lp|Sprinting, the only reason I would want to proxy stuff like ssh is to get to the servers but I was just thinking of exposing the juju controller and using it's proxy capabilities for that
[18:19] <lp|Sprinting> stormmore: well i wasn't referring to ssh'ing to the worker units, i was referring to like say - a gitlab hosted workload in k8s
[18:19] <stormmore> lp|Sprinting, the only other type of socket access I think we will need might be web sockets which I believe NGINX supports
[18:20] <lp|Sprinting> stormmore: you'd have to expose some ssh endpoint there, and in our setup today i could only recommend node-port.
[18:20] <stormmore> lp|Sprinting, well since you mentioned node-port, it is worth noting that the ELB actually uses the node-port to connect to the service
[18:21] <stormmore> lp|Sprinting, as far as git / github / gitlab is concerned, I am planning on running git in the environment too
[18:23] <lp|Sprinting> stormmore: you might find some heartburn there. I've mucked about in this pretty heavily and can lend some help there to getting it setup
[18:23] <stormmore> lp|Sprinting, oh I am sure I will have plenty of heartburn going through this build ;-)
[18:24] <lp|Sprinting> ultimately i created another service def for the ssh access and bound it to type nodePort, and it works pretty well. The downside is if you ever reschedule it, it will rebind the port...
[18:24] <lp|Sprinting> which is frustrating if you've not put anything in front of it to set a consistent access endpoint... so you wind up git remote rm git remote add
[18:24] <lp|Sprinting> minor, but still annoying
[18:26] <stormmore> lp|Sprinting, the git instance in the infrastructure is meant to make it easier and faster for the environment to get the files it needs not necessarily for the devs (although I am sure the company will want me to figure that out too at some point)
[18:27] <lp|Sprinting> stormmore: i'm not as familiar with the architecture you're building, but when we get there and ready to make some design decisions feel free to loop me in
[18:27] <lp|Sprinting> gotta jet for a bit, brb
[18:27] <stormmore> no worries, that is a bit out at the moment and not cleanly defined... OK have fun :)
[19:48] <smgoller> Hi, I'm trying to figure out the root password for the mysql database (percona-cluster charm) for our openstack cluster. We're using the openstack-charm bundle. I asked on #openstack-charms but no response. Anyone have any ideas? I tried looking in /var/lib/charm/mysql but there are no files there.
[20:20] <stormmore> lp|Sprinting, btw I still haven't figure out why the cluster has stopped being to access the container logs via the k8s dashboard, or cli for that matter
[22:05] <petevg> kwmonroe: I merged https://github.com/juju-solutions/matrix/pull/96, so you have your MATRIX_MODEL_PREFIX env variable.
[22:05] <petevg> (thx for approving, cory_fu)
[22:54] <kwmonroe> thx petevg!
[22:54] <petevg> np
[23:32] <lp|Sprinting> stormmore: you're using CDK?
[23:32] <lp|Sprinting> stormmore: or did you deploy kubernetes-core?
[23:33] <lp|Sprinting> stormmore: and what substrate?
[23:33] <kwmonroe> hey axw, bug 1671269 is neat and just for you!  i *think* it's new in 2.1.1 -- i don't remember having to worry about the cred name on remote hosts in 2.1.0.
[23:33] <mup> Bug #1671269: remote lxd credential error if name is the same on lxd/remote hosts (2.1.1) <juju:New> <https://launchpad.net/bugs/1671269>