/srv/irclogs.ubuntu.com/2017/03/08/#juju.txt

stormmorelazyPower, any suggestions on where, if anywhere, I should file a bug about that?00:01
stormmoreAnyone around familiar with K8s services?03:57
stormmorelP|lappy, just the person04:56
stormmoreI am having problems trying to get a Service of type LoadBalancer to get an external IP from AWS04:56
lP|lappystormmore: i think i lost some context04:56
lP|lappyah it was incomplete04:56
stormmoreI found an answer that says that I need --cloud-provider=aws as a flag passed to both the apiserver and controller-manager04:57
lP|lappystormmore: so, we dont fully support Type: LoadBalancer. is a cloud specific feature and we've modeled with a bare metal approach as its the only consistent interface we can guarantee cross cloud. Is it out of the question to use the ingress loadbalancer?04:57
stormmoreother than the fact that will only work with the host IPs instead of giving the service it's own04:58
stormmoreand that is kind of what I figured04:58
lP|lappyI'm not sure what you mean it will only work with the host ips... any worker charm that you have deployed that has ingress=true will act as a reverse proxy for your workloads.04:59
lP|lappystormmore: you'll need to create both a service def and an ingress definition in order to use this addon, but it ships with CDK out of the box.04:59
stormmorejust for full disclosure, I have tried to add that flag to /etc/default/kube-apiserver and /etc/default/kube-control-manager and now the kube-system pods won't start05:00
stormmorelP|lappy, the problem with that is that if the node goes down then it's connections and dns for that matter will not function correctly05:00
stormmoreit is why I am looking at doing keepalived in my bare-metal environment05:01
lP|lappystormmore: i'm not sure how defining a service with type: loadbalancer is going to solve that.... if your node goes down, any other worker thats still active should be able to tak the workload and ingress works cross host out of the box05:01
stormmoreit solves that by pushing it off to ELB which is suppose to be 100% uptime05:02
lP|lappyok so you're concerned that the worker unit itself, being the load balancer, is going to go down05:02
stormmoreexactly05:02
lP|lappyand hten you're left with no connection vs a 50305:02
lP|lappyso, i would probably mitigate that with latency based dns response, and a loaded dns record... but you've brought up an interesting point05:03
lP|lappystormmore: so, best i can offer today is to file a feature request for it. you're not the first to ask for it.05:03
lP|lappybut we'll need to prioritize that, and its not on the roadmap today. The downside to just plugging those flags into the configs is you also have work to do cloud side to make that work05:04
lP|lappyyou will need to assign IAM roles to your ec2 instances so they can make api requests on your behalf without key authentication05:04
stormmoreah that might explain why the kube-system pods aren't starting correctly05:05
lP|lappyand thats not something we can reasonably automate with juju (that i'm aware of)05:05
lP|lappythats a pretty ec2 specific feature05:05
lP|lappys/ec2/aws/05:05
stormmorelP|lappy, well juju does something in the way of security groups when it creates the AWS instances05:05
lP|lappythis isn't security groups ;)05:05
lP|lappyits IAM roles05:05
lP|lappylovely, they deleted my easy-reference material 8 days ago it looks like05:09
lP|lappyhttps://github.com/kubernetes/kubernetes/tree/master/cluster/aws/05:09
lP|lappybut the gist is, you create an IAM role you can assign the instances, you apply that IAM role to your control plane vm's and they can then request resources05:09
lP|lappystormmore: https://github.com/fabric8io/kansible/blob/master/vendor/k8s.io/kubernetes/cluster/aws/templates/iam/kubernetes-master-policy.json05:12
lP|lappystormmore: https://github.com/fabric8io/kansible/blob/master/vendor/k8s.io/kubernetes/cluster/aws/templates/iam/kubernetes-minion-policy.json05:13
lP|lappythese are two such policy json objects that can be used to create your ARN's05:13
lP|lappyBut I'm off to bed. I'll check in on you tomorrow and see how things are going. Cheers stormmore05:15
=== lP|lappy is now known as lp|Sprinting
kjackalGood morning Juju world!07:56
kklimonda^is there a documentation on how to configure juju to use an existing controller somewhere?10:07
anrah_kklimonda^: https://jujucharms.com/docs/2.0/tut-users10:11
kklimonda^wow, thanks - that was actually easier than expected10:12
anrah_that is quite nice feature :)10:18
kklimonda^so add-model complains about credentials missing, even though I've granted the user superuser access - I guess I'm hitting https://bugs.launchpad.net/juju/+bug/163037210:19
mupBug #1630372: "ERROR no credential specified" during add-model as non-admin user <cwr-ci> <matrix> <usability> <v-pil> <juju:Triaged> <https://launchpad.net/bugs/1630372>10:19
kklimonda^I think I have to add user to MAAS, and then add those credentials to Juju? but that doesn't sound right, I thought juju controller would abstract that part10:20
kjackalkklimonda^: you do juju add-user myuser; juju grant myuser superuser; and then on the other machine you will need to do juju add-credential with the credentials of the cloud you are expected to use10:23
kjackalkklimonda^: no wait a sec, let me doublecheck my notes10:23
kklimonda^yes, but both accounts will be using the same cloud (MAAS)10:24
kjackalkklimonda^: you need to do a juju register <registration_string_you got from add-user>10:24
kklimonda^I've done that part10:25
kklimonda^but it's still missing credentials when I call juju add-model10:25
kklimonda^and to add credentials, I seem to have to add cloud10:25
kklimonda^so that's a lot of manual steps10:25
kklimonda^(I've also granted superuser for that new user)10:25
Hetfieldkjackal: i still have issues on how juju picks ipaddress :(10:26
kklimonda^I'm just missing the relationship between juju, controller, users and the underlying cloud (MAAS)10:26
kjackalkklimonda^: there are some manual steps. You need to create a file with the credentails of MAAS10:26
kklimonda^I've assumed that I can share juju controller between users (that seems to be the case) and that juju controller will abstract an underlying cloud, because authorization is already done in Juju10:27
kklimonda^but now it seems I should probably have a separate user in MAAS, and add those credentials to Juju10:27
kjackalkklimonda^: you need to copy the credentials you have in ~/.local/share/juju/credentials.yaml and place them on the side of the other client/user10:29
kjackalso as to do the juju add-credentials10:29
Hetfieldkjackal: i checked https://jujucharms.com/docs/2.0/network-spaces but it doesn't fit my case: i creaed spaces in maas but constraints just say "pick a machine that has these space"10:29
kjackalHetfield: ok, I am not the right person to help you. Can you please send an email to the list juju@lists.ubuntu.com ?10:31
Hetfieldoki10:33
kklimonda^kjackal: I don't think copying actuall works, I had to generate a new API key in MAAS10:34
kklimonda^and that finally worked10:34
kjackalwell done kklimonda^10:35
kjackalthanks Hetfield10:36
Hetfieldthanks to you kjackal10:36
cnfhow do I edit the connection details for a juju controller?10:36
=== kklimonda^ is now known as kklimonda
cnfalso, any documentation of how to put it behind a reverse proxy?10:39
=== kklimonda is now known as Guest71046
=== Guest71046 is now known as kklimonda
kjackalcnf: is this what you are looking for: http://askubuntu.com/questions/174171/what-are-the-ports-used-by-juju-for-its-orchestration-services10:44
kjackal?10:44
cnfuhm, mybe10:45
cnfjuju isn't very forgiving for non-flat network setups :/10:45
ybaumyis it possible to make a controller HA?10:48
kjackalybaumy: yes let me find the documentation for that11:03
kjackalybaumy: https://jujucharms.com/docs/2.1/controllers-ha11:04
cnfso can you get juju-controller to listen on an IP not in the MAAS admin range?11:04
cnfhmm11:39
cnfi need a command reference for juju11:39
cnfhow do i make it retry assigning machines?11:39
cnfand it seems i still don't have my networking set up right11:42
cnfo,O11:42
cnfso how to i edit the proxy that new machines need to use in juju?11:48
cnfhmz, juju is rather confusing to use :/11:49
Hetfieldcnf: juju retry-provisioning11:57
ybaumykjackal: thanks11:57
cnfHetfield: thanks11:59
cnfhmm12:08
cnfhow do i know that my deploy was successful?12:08
cnfHetfield: retry-provisioning doesn't try to bring up a machine, again?12:09
cnfso it seems juju is pushing the http_proxy env vars to the nodes12:11
cnfbut NOT the no-proxy setting12:11
cnfalso, the MAAS proxy doesn't allow non 443 ports for https12:11
cnfany hints on how to fix this?12:12
cnfwow, juju is pissing me off12:17
kjackalcnf: Here is a list of the commands: https://jujucharms.com/docs/2.1/commands however you should also be using juju <command> --help to double check that the docs are uptodate12:31
cnfyeah...12:31
kjackalIf you want to "free" a machine you can do juju remove-machine  (--force)12:32
kjackalcnf: ^12:32
cnfok12:32
cnfthanks12:32
cnfi'm failing to deploy anything on maas with juju atm12:32
kjackaland for the proxies I am afraid you would better ask on the list as I have no experience in that subject12:33
cnfbeen at this for the better part of 5 days, i'm not having fun atm :/12:33
tvansteenburghproxy info: https://github.com/juju/docs/issues/206#issuecomment-28277309612:33
tvansteenburgh(might help, i dunno)12:33
cnftvansteenburgh: i'm past bootstrapping12:34
cnftook me a while to figure that out, as well12:34
cnfand i did a juju model-config no-proxy as well12:36
cnfbut i don't think that took, either12:36
cnftakes friggin 15 minutes between tries, as well >,<12:36
tvansteenburghdoes `juju model-config` show the value you set?12:37
cnfyeah12:38
tvansteenburghwhat about `juju run someunit/0 'env | grep proxy'`12:39
cnfwhat is someunit?12:41
tvansteenburghsome unit in your deployment12:41
tvansteenburghlike, i unit name from the Unit column printed by `juju status`12:43
tvansteenburghs/i/a/12:43
cnfhmm, doesn't do anything12:43
cnfnow what12:43
disposable2cnf: your bootstrap timeout tip got me over the problem in the end. thank you12:44
cnfdisposable2: np12:44
=== tinwood_swap is now known as tinwood
cnfhmm, the hosts can't reach the controller12:44
cnfwt?12:44
tvansteenburghoops12:45
tvansteenburghjuju run --unit someunit/0 'env |grep proxy'12:45
tvansteenburghwhat about that? ^12:45
cnfyeah, that doesn't do anything12:46
cnfjust sits there12:46
tvansteenburghit hangs?12:46
cnfyeah12:47
tvansteenburghdoes juju debug-log --replay have any recent errors in it? sounds like you machine can't communicate with the controller. does `juju status` print anything or does that hang too?12:49
cnfjuju status works12:51
cnfjuju debug-log --replay just sits there12:51
tvansteenburghcnf: what made you say "the hosts can't reach the controller"?12:57
cnftvansteenburgh: i sshed to one of the machines12:57
cnfand i can't ping the controller12:57
cnfdon't get arp replies, either12:57
cnfwhich is weird, because i sshed FROM the controller12:58
cnfno, i lie12:58
cnfi sshed from the MAAS controller12:58
cnfnot the juju controller12:58
tvansteenburghit's strange that `juju status` doesn't show 'agent lost' errors or something then12:58
cnfit has a lot of "waiting" and "pending"12:59
tvansteenburghahh12:59
cnfmy juju controller runs in a KVM machine on the MAAS controller12:59
cnfi guess something is amis there12:59
tvansteenburghyeah. i won't be much help sorting out your network probs, but it sounds like that's where the prob is13:00
Hetfieldcnf: it's fine13:18
kklimondaI've created a new user in juju, using same maas user and a different api key, and deployed bundle to a different model.14:03
kklimondanow I'm trying to deploy a bundle using original user, and I get an error "cannot deploy bundle: POST [url] Forbidden"14:04
kjackalkklimonda: could you tell us how you created the new user using the same maas user and a different api key?14:39
kjackalkklimonda: I was under the impression that you only need to do a juju add-user mynewuser14:40
=== hml_ is now known as hml
kklimonda@kjackal  I did juju add-user (that generated a snippet to be used with juju register)16:05
kklimonda@kjackal then I got a "credentials missing" when I tried creating a new model, so I went back to MAAS and created a new API key, which I've used with juju add-credentials16:06
kklimondamy assumption was that given I have different users in juju, and they are working on different models, that would work16:06
kklimondaand the error returned by Juju is too vague to be useful, I guess I'll have to login to the controller and perhaps its logs will tell me more16:08
lp|Sprintingmbruzek: ping16:09
mbruzeklp|Sprinting: pong16:10
lp|Sprintingmbruzek: hey i have some questions for you re: etcd storage design. (wifi here is terrible, do you mind doing this on irc?)16:12
mbruzekstill at standup, parking lots16:13
mbruzekbut yeah go ahead16:13
lp|Sprintingack. - I may have painted myself in a corner. I wanted to model the storage path in layer.yaml to kind of segregate how we traditionally assumed it was in /var/lib/etcd, whereas now its moved to /var/snap/etcd/current. This is default behavior - but what i didn't account for is durable storage mounts which will need to change the path of that storage volume (presumably /media/etcd via the removable-media16:14
lp|Sprintinginterface)16:14
lp|SprintingDoes it make sense to just nuke that layer.yaml configuration and make a switch in a lib somewhere that has a check "if is_state('storage.attached'): return '/media/etcd'" ?16:15
lp|Sprintingbasically i'm asking what would make sense to you, as a consumer of etcd? I have limited options on where it can be mounted, because of snap confinement16:16
mbruzeklp|Sprinting: I don't like having to limit the mount points, but I understand that comes with the snap confinement. i am +0 it could go either way.16:32
mbruzeklp|Sprinting: We just got done with standup so I am more interactive now16:35
lp|Sprintingmbruzek: i think i answered my question above with the switch. I'm going to set a state and write a getter that determines what path we care about.16:36
kjackalkklimonda: instead of creating a new API key you should have done "juju credentials --format yaml --show-secrets" and on the side where the new user is you should do a "juju add-credential "yourcloud" --replace -f filewithcredentials.yaml"16:36
lp|Sprintingi don't immediately see a better way to do this16:36
kklimonda@kjackal so now I basically have two MAAS API keys competing with each other?16:37
lp|Sprintingmbruzek: something like the following: https://gist.github.com/chuckbutler/2e8ae364639504250499c838a33a20e116:37
kklimonda(because it just started working randomly..)16:37
mbruzeklp|Sprinting: will /media/etcd-data work with snaps?16:38
lp|Sprintingmbruzek: interface: removeable-media16:39
mbruzeklp|Sprinting: What happens to the data in /var/snap/etcd/current if a volume is mounted after install?16:39
kjackalkklimonda: I am not sure what is going on right now, I have never played with maas16:39
mbruzekinstall etcd using it, oh look they support storage!, add storage. Where did my data go?16:39
lp|Sprintingmbruzek: same thing that happens with the current charms. service is stopped, data is rsync'd to target endpoint, config is updated and the daemon is restarted.16:39
mbruzekSo the storage code would copy that information over, so the storage code would have to know about both directories.16:40
lp|SprintingWhich it does16:43
lp|Sprintingwell if i change this where i'm thinking of changing it it's going to need some minor refactoring16:44
lp|Sprintingbut yeah, it does/will.16:44
stormmoreo/ juju world17:34
stormmoreso lp|Sprinting the pointers you gave me last night got me most of the way to get it to work. had to make some manual changes to the LB to get it to work but it did create the LB for me17:50
ybaumywhat happens if i add-unit --to a machine where there is already a unit installed .. will there be a lxd container build?17:50
lp|Sprintingstormmore: cool :) Glad I was able to get you moving17:51
lp|Sprintingstormmore: but I think having run the exercise manually you see why we're in a state of "we dont officially support this in any capacity out of the box" right?17:51
lp|Sprintingybaumy: it will co-locate without lxd. we call this hulk smashing and discourage it17:52
lp|Sprintingybaumy: if you want it to colocate in a container, you'll need to specify that in --to, eg:  juju add-unit mysql --to lxd/417:52
ybaumylp|Sprinting: but the container i have to build myself?17:53
lp|Sprintingybaumy: nope. Container management is automated with juju. you just have to specify thats where you want it and juju will translate and load the lxd image for you, fire up the unit agent and the rest happens as usual accoding to the charm17:53
lp|Sprinting*according17:53
ybaumylp|Sprinting: is there a piece of documentation for it?17:55
stormmoreactual that is a design decision that I think might be a mistake, in my case this is a temporary measure while I build up our bare metal but no one seems to official support AWS very well17:55
lp|Sprintingybaumy: https://jujucharms.com/docs/2.1/charms-deploying17:55
lp|Sprintinghttps://jujucharms.com/docs/2.1/charms-deploying#deploying-to-specific-machines-and-containers rather17:56
stormmorelp|Sprinting, I was always thinking about what you said about knowing if you are on a cloud provider... Juju itself knows what cloud it is on / deploying to, doesn't it provide a variable to the charms to help them determine/17:56
ybaumylp|Sprinting: ah i see bootstraping it ok i get it. thanks for the doc. will go from there17:57
lp|Sprintingstormmore: nope. Charms are cloud agnostic and should strive to be as agnostic as possible. The idea is to make the learnings reusable on every substrate, and things that require special cloud-provider dances, should be encapsulated either as a charm (or i'm not sure what else would go here... gap in my thought) or we should help encourage things to not be tied to cloud specific features.17:59
lp|Sprintingstormmore: this is why we've been fairly up front about the fact we took a bare metal first approach. If you didn't have an ELB what would you do?  Probably deploy haproxy in front of your workers and scale it independently to act as your webhead yeah?18:02
stormmorelp|Sprinting, I kinda agree but charms should also take into account the environment they are being deployed to's capabilities18:02
stormmorelp|Sprinting, I would build it with keepalived and haproxy probably18:02
stormmore(which is basically what I am planning on doing with bare-metal18:03
stormmoreoh and I plan on running those in the cluster itself instead of on the edge18:07
lp|Sprintingstormmore: not sure how thats any different than the ingress controller we are shipping ;)18:09
lp|Sprintingif its running in the cluster, and you're exposing as nodePort, you'll have the same issue if your workers die that you'll have to do some latency based routing w/ health checks at the dns level.18:10
stormmorelp|Sprinting, does the Ingress controller handle VIPs?18:14
lp|Sprintinghttps://github.com/kubernetes/contrib/issues/114018:16
lp|Sprintingstormmore: ^ yeah, and there's a good issue thread about this and how its been testd here18:16
stormmorelp|Sprinting, if so, then I might change my plan. My idea is remove any potential single point of failures based on IPs18:16
lp|Sprintingstormmore: ingress just relies on dns. Doesn't seem to matter how you get the traffic to it18:16
lp|Sprintingstormmore: one thing that might be problematic for you, is if you're planning on using socket based workloads. There have been some dragons in there historically if you're wanting to proxy stuff like ssh ports and what not.18:17
lp|Sprintingthats one thing i'm aware of, and haproxy would be a welcome replacement there instead of asking for NodePort bindings.18:17
lp|Sprintingstormmore: however, endpoints pretty much abstract all of that, and yes VIP support is in there.18:18
stormmorelp|Sprinting, the only reason I would want to proxy stuff like ssh is to get to the servers but I was just thinking of exposing the juju controller and using it's proxy capabilities for that18:19
lp|Sprintingstormmore: well i wasn't referring to ssh'ing to the worker units, i was referring to like say - a gitlab hosted workload in k8s18:19
stormmorelp|Sprinting, the only other type of socket access I think we will need might be web sockets which I believe NGINX supports18:19
lp|Sprintingstormmore: you'd have to expose some ssh endpoint there, and in our setup today i could only recommend node-port.18:20
stormmorelp|Sprinting, well since you mentioned node-port, it is worth noting that the ELB actually uses the node-port to connect to the service18:20
stormmorelp|Sprinting, as far as git / github / gitlab is concerned, I am planning on running git in the environment too18:21
lp|Sprintingstormmore: you might find some heartburn there. I've mucked about in this pretty heavily and can lend some help there to getting it setup18:23
stormmorelp|Sprinting, oh I am sure I will have plenty of heartburn going through this build ;-)18:23
lp|Sprintingultimately i created another service def for the ssh access and bound it to type nodePort, and it works pretty well. The downside is if you ever reschedule it, it will rebind the port...18:24
lp|Sprintingwhich is frustrating if you've not put anything in front of it to set a consistent access endpoint... so you wind up git remote rm git remote add18:24
lp|Sprintingminor, but still annoying18:24
stormmorelp|Sprinting, the git instance in the infrastructure is meant to make it easier and faster for the environment to get the files it needs not necessarily for the devs (although I am sure the company will want me to figure that out too at some point)18:26
lp|Sprintingstormmore: i'm not as familiar with the architecture you're building, but when we get there and ready to make some design decisions feel free to loop me in18:27
lp|Sprintinggotta jet for a bit, brb18:27
stormmoreno worries, that is a bit out at the moment and not cleanly defined... OK have fun :)18:27
smgollerHi, I'm trying to figure out the root password for the mysql database (percona-cluster charm) for our openstack cluster. We're using the openstack-charm bundle. I asked on #openstack-charms but no response. Anyone have any ideas? I tried looking in /var/lib/charm/mysql but there are no files there.19:48
stormmorelp|Sprinting, btw I still haven't figure out why the cluster has stopped being to access the container logs via the k8s dashboard, or cli for that matter20:20
petevgkwmonroe: I merged https://github.com/juju-solutions/matrix/pull/96, so you have your MATRIX_MODEL_PREFIX env variable.22:05
petevg(thx for approving, cory_fu)22:05
kwmonroethx petevg!22:54
petevgnp22:54
lp|Sprintingstormmore: you're using CDK?23:32
lp|Sprintingstormmore: or did you deploy kubernetes-core?23:32
lp|Sprintingstormmore: and what substrate?23:33
kwmonroehey axw, bug 1671269 is neat and just for you!  i *think* it's new in 2.1.1 -- i don't remember having to worry about the cred name on remote hosts in 2.1.0.23:33
mupBug #1671269: remote lxd credential error if name is the same on lxd/remote hosts (2.1.1) <juju:New> <https://launchpad.net/bugs/1671269>23:33

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!