| jose-phillips | question | 00:08 |
|---|---|---|
| jose-phillips | i have a charm locally and i need to redeploy with the changes i do locally | 00:08 |
| jose-phillips | but upgrade-charm --path ./path and resolved dont upgrade the charm on the destination | 00:09 |
| hloeung | jose-phillips: maybe upgrade-charm --switch ? | 02:02 |
| === Guest34686 is now known as manadart | ||
| manadart | window | 11:33 |
| manadart | Oops :) | 11:33 |
| === rogpeppe1 is now known as rogpeppe | ||
| === rogpeppe1 is now known as rogpeppe | ||
| === frankban is now known as frankban|afk | ||
| === frankban|afk is now known as frankban | ||
| rick_h | kwmonroe: you feeling up to the Juju Show today? | 15:56 |
| kwmonroe | sure thing rick_h. i've got some conjure-up/k8s news to share. | 15:58 |
| rick_h | kwmonroe: woot | 15:58 |
| tychicus | Hi, I have an issue where my controller is stuck in an error loop, I attempted to destroy kibana, elasticsearch, filebeat, packetbeat, and top beat from the juju-gui | 16:03 |
| tychicus | the machine-0.log shows an endless stream of the following type of messages | 16:04 |
| tychicus | 2018-01-24 16:01:30 ERROR juju.state unit.go:339 cannot delete history for unit "u#filebeat/16#charm": <nil> | 16:04 |
| tychicus | 2018-01-24 16:01:30 ERROR juju.state unit.go:339 cannot delete history for unit "u#topbeat/7#charm": <nil> | 16:04 |
| tychicus | 2018-01-24 16:01:30 ERROR juju.state unit.go:339 cannot delete history for unit "u#filebeat/13#charm": <nil> | 16:04 |
| tychicus | 2018-01-24 16:01:35 ERROR juju.state unit.go:339 cannot delete history for unit "u#packetbeat/13#charm": <ni | 16:04 |
| tychicus | followed by | 16:05 |
| tychicus | 2018-01-24 16:04:59 ERROR juju.rpc server.go:510 error writing response: write tcp 10.110.0.111:17070->10.110.0.117:34072: write: broken pipe | 16:05 |
| tychicus | 2018-01-24 16:04:59 ERROR juju.rpc server.go:510 error writing response: write tcp 10.110.0.111:17070->10.110.0.117:34072: write: broken pipe | 16:05 |
| tychicus | 2018-01-24 16:04:59 ERROR juju.rpc server.go:510 error writing response: write tcp 10.110.0.111:17070->10.110.0.117:34072: write: broken pipe | 16:05 |
| tychicus | mongodb is maxing out all available processors, is there anything I can do to reset my controller to a healthy state | 16:07 |
| === mlb is now known as mlbiam | ||
| mlbiam | hello, i'm trinyg to configure a canonical k8s distro with oidc. I found this medium blog (https://medium.com/@tvansteenburgh/the-canonical-distribution-of-kubernetes-weekly-development-summary-49274b78b5c2) that says its there but we can't find where we configure the api server flags | 16:32 |
| kwmonroe | mlbiam: i can't find a reference to that oidc interface in the k8s charms or bundles, so i'm guessing that was more of a foundation that is meant to be exploited later. i do, however, see an config option for api server flags. you'd set them like: juju config kubernetes-master api-extra-args="foo=bar ham=burger" | 16:51 |
| mlbiam | kwmonroe - awesome, that confirms what we were thinking | 16:52 |
| mlbiam | here's my next question, whats the right way to import our OIDC ca cert? | 16:53 |
| kwmonroe | mlbiam: i'm not really sure because i typically let the easyrsa charm handle all my CA/cert needs. Cynerva or ryebot, do you guys know if/how/where you might inject your own ca cert? | 16:58 |
| mlbiam | this is only the CA for our oidc provider, we're not changing any of kubernetes internal CA information | 16:59 |
| mlbiam | (we're trying to figure out how to set the --oidc-ca-file parameter on the API server) | 17:00 |
| ryebot | ah | 17:00 |
| mlbiam | or even better get juju to install our CA cert into the trust store on the api server | 17:01 |
| ryebot | mlbiam if that's all you need, you can set it with the api-extra-args config option on kubernetes-master | 17:01 |
| mlbiam | right, but where do i put the cert so its accessible by the API server? | 17:01 |
| ryebot | of course, you'd need to maintain it on each master node yourself, unfortunately | 17:01 |
| ryebot | mlbiam: anywhere accessible by root should do; apiserver is run by root | 17:02 |
| mlbiam | ok, so juju won't over write it? i see that the easyrsa files are in /root/cdk. would that be as good a place as any? | 17:03 |
| Cynerva | it needs to be in a non-hidden folder in /root 'cause it's a confined snap with the home interface | 17:03 |
| mlbiam | got it | 17:04 |
| mlbiam | does juju have any way of updating the os cert store? | 17:08 |
| mlbiam | (ie if i want to distribute a cert to be trusted by multiple servers) | 17:10 |
| mlbiam | that way i could skip that paraeter entirely | 17:11 |
| ybaumy | hi. is there some good documentation on how to use juju and kubernetes persistent storage somewhere | 17:21 |
| ybaumy | i want to setup nfs shares | 17:22 |
| ybaumy | to use | 17:22 |
| rick_h | kwmonroe: do you have something for ybaumy ? ^ | 17:42 |
| ybaumy | rick_h: im still trying to convince ppl that redhat openshift platform well lets put that way falls short of expectations | 17:46 |
| ybaumy | rick_h: but its a long and hard way | 17:46 |
| rick_h | ybaumy: heh, always is more work to change minds than to do the thing | 17:47 |
| ybaumy | rick_h: you wont believe .. i did already tech demos and stuff and presentations ... compared both ways to roll out clusters across multicloud providers | 17:48 |
| ybaumy | rick_h: they still think because its redhat it cant be that bad | 17:48 |
| ybaumy | ;) | 17:48 |
| * rick_h whistles to the wind | 17:51 | |
| ybaumy | same goes for SuSe CaaS platform | 17:51 |
| ybaumy | if you guys have a piece of documnetation for me i can use it would be nice if you put as PM. i have to eat now and drink beer | 17:53 |
| knobby | ybaumy: Once you get cdk installed you can just use nfs persistant volumes and persistent volume claims. Are you trying to get juju to manage your nfs or something? | 17:53 |
| ybaumy | knobby: yes thats what im trying to accomplish | 17:54 |
| ybaumy | knobby: juju should handle the nfs shares if that is even possible | 17:54 |
| ybaumy | else show me a way around | 17:54 |
| ybaumy | to scale out | 17:55 |
| ybaumy | for workers | 17:55 |
| ybaumy | im talking kubernetes | 17:55 |
| knobby | I'm not sure a follow. Once you setup a pv and pvc you just run pods. There isn't any work outside of getting the nfs server up and reachable by the cluster. | 17:56 |
| ybaumy | but how do i do that initialy when setting up the cluster | 17:56 |
| knobby | are you trying to put the nfs server on the cluster? | 17:56 |
| ybaumy | maybe i missed the options | 17:56 |
| knobby | I just set the pvc in the deployment description and specify the mount point inside the pod and magic happens. There isn't anything to do. | 17:57 |
| knobby | it's all in k8s land | 17:57 |
| ybaumy | ok maybe im thinking to complicated | 17:58 |
| ybaumy | let me try that tonight | 17:59 |
| ybaumy | i have an application that needs a nfs volume accross all pods so i thought | 18:00 |
| ybaumy | i could setup the cluster with that nfs volume from the start | 18:00 |
| ybaumy | but then again i dont have the pods when adding a unit | 18:01 |
| ybaumy | though im stupid | 18:01 |
| knobby | once you get kubectl you just add the pv with some yaml and then make a pvc for it and then specify the pvc in the deployment. I can try to find a tutorial for that if you want. | 18:01 |
| knobby | it's not related to juju at all. Juju doesn't know about your nfs or the workloads on k8s | 18:01 |
| ybaumy | knobby: thats what i understand now | 18:02 |
| ybaumy | knobby: sorry for asking stupid questions | 18:02 |
| knobby | if you have trouble with it, ybaumy, I have a bare metal k8s with nfs beside me and I'm happy to answer questions. | 18:03 |
| knobby | ybaumy: not at all. This stuff is complex and it is so easy to get lost in the trees. | 18:03 |
| ybaumy | knobby: if i cant get it to work i would be happy to get back to you | 18:03 |
| ybaumy | let me ask you guys one last question about scaling pods accross vsphere. one colleague of mine says we should put one pod on one worker on one VM | 18:04 |
| ybaumy | for performance reasons in vsphere | 18:05 |
| ybaumy | what do you think of that | 18:05 |
| ybaumy | small VM's with only one pod ? | 18:05 |
| ybaumy | why do i need kubernetes then i dont get it | 18:05 |
| ybaumy | he says vmware guys recommand it that way | 18:06 |
| knobby | I agree with you. The point of k8s is to help utilize those machines without having to pick your VM size to match your workload. K8s will move things around and make sure things fit while giving guaranteed resources to a pod. | 18:06 |
| ybaumy | knobby: thats what my picture is | 18:07 |
| ybaumy | im trying to understand .. and im falling short | 18:07 |
| ybaumy | on friday some vmware guy will be with us in a meeting i hope he can clarify this. i would like to understand it before but ... | 18:08 |
| knobby | I can't help you there. You could certainly put a single pod per node, but it seems like an odd choice. You'd spend your life adjusting nodes to scale out. | 18:09 |
| ybaumy | and down... its like a neverending stream of create/delete VM's | 18:10 |
| ybaumy | well i hope he will shed light on this. | 18:10 |
| ybaumy | let me check my beer status and eat something ... then i try the nfs stuff | 18:12 |
| ybaumy | bbl | 18:12 |
| rick_h | 34min until the Juju Show!!!!! get ready. | 18:26 |
| rick_h | kwmonroe: hml agprado magicaltrout bdx and anyone else that's been holding their breath ^ | 18:27 |
| hml | w00t! | 18:27 |
| bdx | oh yeaaa | 18:28 |
| * rick_h needs to refill this glass pre-show | 18:28 | |
| agprado | rick_h: yep! holding my breath! for my first jujushow! | 18:41 |
| agprado | rick_h, I can't see the hangout link in the calendar for the juju show | 18:51 |
| rick_h | https://hangouts.google.com/hangouts/_/ygz6alasjvg4lf3fuurxu4hprye for joining the call and https://www.youtube.com/watch?v=K9x3Xhlnl1s for watching the stream | 18:53 |
| rick_h | bdx: you coming this week? | 18:59 |
| kwmonroe | the cloudinit bug rick_h is referring to is https://bugs.launchpad.net/juju/+bug/1535891 | 19:06 |
| mup | Bug #1535891: Feature request: Custom/user definable cloud-init user-data <cpe-onsite> <juju:Fix Released by hmlanigan> <https://launchpad.net/bugs/1535891> | 19:06 |
| EdS | hi Juju people :) | 19:12 |
| EdS | I have just run into a failing update charm operation | 19:12 |
| EdS | I have connected to the affected unit and tailing the log of the juju unit shows some python stuff about "INFO upgrade-charm TypeError: 'NoneType' object is not subscriptable" | 19:13 |
| EdS | I'm trying to update a production k8s cluster and this is a bit of a surprise | 19:14 |
| EdS | the units affected seem to be sitting in a loop trying to update and repeatedly failing at the same step. | 19:15 |
| EdS | any ideas what I should do? | 19:15 |
| bdx | `juju model-config | grep proxy` - why are these proxy configs not applied to the containers? | 19:16 |
| bdx | per conversation in juju show | 19:16 |
| bdx | shouldn't a container and a machine get the same proxy config if defined at the model level? | 19:17 |
| rick_h | bdx: so they should be on containers but apt caches and such aren't in there I believe | 19:17 |
| kwmonroe | monitoring a k8s cluster: https://medium.com/@kwmonroe/monitor-your-kubernetes-cluster-a856d2603ec3 | 19:17 |
| rick_h | bdx: http/https proxies are | 19:17 |
| bdx | the apt proxies aren't then? | 19:17 |
| rick_h | bdx: so I think it's less proxy but more cache | 19:18 |
| rick_h | bdx: or custom repos | 19:18 |
| bdx | ahh | 19:18 |
| bdx | I see | 19:18 |
| EdS | any ideas how I kick it to try a new version and forget about failing, please? | 19:19 |
| kwmonroe | conjure-up spell addon repo: https://github.com/conjure-up/spells/tree/master/canonical-kubernetes/addons | 19:25 |
| rick_h | EdS: this is for a charm you're updating? | 19:37 |
| rick_h | EdS: just mark it "juju resolved xxxx --no-retry" so that it will go green but not rerun anything | 19:38 |
| EdS | not as such - I'm updating our canonical kubernetes setup | 19:38 |
| rick_h | EdS: and then run your upgrade-charm command (if it's a charm you mean) | 19:38 |
| EdS | all I did was (some time ago) deploy https://api.jujucharms.com/charmstore/v5/canonical-kubernetes-117/archive/bundle.yaml | 19:39 |
| EdS | after editing the charm versions mentioned in my local copy, I have updated the deplyment | 19:39 |
| EdS | I did several minor versions no trouble | 19:39 |
| EdS | now all my kubernetes worker and master nodes are stuck in a loop with the message: hook failed: "upgrade-charm" | 19:40 |
| rick_h | hml: do you have that bug link again please for the cloud-init stuff for the show notes? | 19:41 |
| * rick_h hangs head in shame he didn't copy it from the hangout chat | 19:41 | |
| * hml looking | 19:42 | |
| hml | rick_h: https://bugs.launchpad.net/bugs/1535891 | 19:43 |
| mup | Bug #1535891: Feature request: Custom/user definable cloud-init user-data <cpe-onsite> <juju:Fix Released by hmlanigan> <https://launchpad.net/bugs/1535891> | 19:43 |
| EdS | ok thanks... | 19:43 |
| rick_h | hml: ty! | 19:43 |
| hml | rick_h: it only defines the first set of work, that’s already out there | 19:44 |
| rick_h | hml: right, bdx was wondering about that so I wanted to have that link in the notes | 19:44 |
| hml | not the replication from machine to container piece | 19:44 |
| rick_h | EdS: sorry, I'm not sure to be honest. You upgrade the charms by doing an upgrade-charm command and editing the yaml file doesn't do anything. So I'm assuming you did an upgrade-charm on each application name and it errored in some way? | 19:45 |
| rick_h | EdS: if so, the folks that work on those charms would want to see what the juju debug-log looks like for those errors and if you're wanting to try again you'd run the "juju resolved xxxx --no-retry" and then the "juju upgrade-charm xxx" as I noted | 19:45 |
| EdS | ok thanks very much. I'm trying to work out what happened and agree I'll try and gather some info. | 19:46 |
| knobby | I'd think the first step is to look at the debug-log and figure out why it doesn't want to upgrade | 21:04 |
| knobby | EdS: `juju debug-log --replay | pastebinit` | 21:04 |
| EdS | knobby: ok :) | 21:05 |
| EdS | I've carried on working on it. I'm assuming log will be massive by now | 21:06 |
| knobby | no doubt | 21:06 |
| knobby | if it is too big for pastebinit, we might have to get to it another way | 21:06 |
| EdS | *Blammo* https://api.jujucharms.com/charmstore/v5/canonical-kubernetes-117/archive/bundle.yaml | 21:11 |
| EdS | ah crud | 21:11 |
| EdS | Failed to contact the server: [Errno socket error] [Errno socket error] The read operation timed out | 21:11 |
| EdS | please ignore link | 21:11 |
| EdS | ;) | 21:11 |
| EdS | yeah pastebinit died | 21:11 |
| knobby | timeout smells like too large of a file to me | 21:11 |
| EdS | yes | 21:12 |
| knobby | maybe like `juju debug-log --replay|tail -n 10000|pastebinit` | 21:12 |
| kwmonroe | EdS: you can limit the log to just one failing instance with "juju debug-log --include kubernetes-worker/0 --replay" or "--lines 1000" as an alt to --replay. | 21:12 |
| knobby | or that is even better | 21:13 |
| kwmonroe | knobby: you rascal.. --lines (-n) makes the tail unnecessary. | 21:13 |
| EdS | haha just dumped to a file | 21:13 |
| EdS | 36MB | 21:13 |
| EdS | :/ | 21:13 |
| knobby | just do the --lines 10000 and pastebinit that | 21:14 |
| knobby | or tail that file to it or something | 21:14 |
| EdS | http://paste.ubuntu.com/26454287/ | 21:15 |
| EdS | line 127-142 | 21:16 |
| EdS | I was seeing a lot of that sort of "nonetype is not subscriptable" errors. | 21:16 |
| EdS | does "watch juju status" being left in a terminal make a load of log noise anywhere? | 21:17 |
| kwmonroe | nah EdS, i'm not aware of juju status increasing log noise anywhere. | 21:23 |
| EdS | phew. | 21:23 |
| knobby | so there's supposed to be a dictionary in the unit with information like the cluster credentials. Your credentials don't exist. | 21:26 |
| knobby | investigating | 21:27 |
| EdS | interesting. that may be one stage of the brokenness but I think that was all fine at several points (was a running cluster, was working fine until update failed) | 21:31 |
| EdS | \o/ happy cluster again | 21:34 |
| EdS | I ended up killing the workers but keeping headroom. | 21:34 |
| EdS | now I have "updated" workers ;) | 21:34 |
| ryebot | \o/ | 21:35 |
| EdS | not pretty but it's late and that was weirdness I would like to be without | 21:35 |
| EdS | thanks for taking a look :) please buzz if you want any more info (but I'm going away soon and will be back tomorrow) | 21:51 |
| kwmonroe | EdS: i opened this to make sure we have required creds in place before doing things like "start_worker": https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/474 | 21:59 |
| EdS | thanks :) | 22:00 |
| kwmonroe | np EdS - thanks for breaking stuff to make it better ;) | 22:00 |
| EdS | I'm still here trying to work out how upgrade nginx too ;) | 22:00 |
| EdS | I want new ingress controller, but cannot figure that -_- | 22:00 |
| EdS | haha no worries. any time. | 22:01 |
| EdS | still trying to get this bit straight and it's juju related, I'm sure - I wonder if you could help shed some light on it? | 22:09 |
| EdS | using juju and maas, I set up canonical kubernetes | 22:10 |
| EdS | I have nginx as the ingress controller in kubernetes | 22:10 |
| EdS | it's on the workers | 22:10 |
| EdS | it all got updated | 22:10 |
| EdS | but nginx version is the same | 22:10 |
| EdS | I don't quite know where it comes from, when setting it up with juju since I'm so far removed from the setup | 22:10 |
| knobby | is it beta.13? | 22:20 |
| knobby | EdS: I just added something to the charm to allow changing that. It was hard-coded in the charm to a specific version and we weren't tracking releases well. I updated the version and made it a config option to specify the image to use, but that isn't in stable yet. | 22:21 |
| EdS | :) yep it is NGINX 0.9.0-beta.13 git-949b83e | 22:22 |
| EdS | this has been driving me nuts, I just didn't know where it was from ;) | 22:23 |
| knobby | I bumped it to beta.15. Unfortunately it is hard-coded in the charm, so unless you want to fork the charm to update it you'll just have to live with it for a bit longer. | 22:23 |
| knobby | EdS: the code is around here: https://github.com/kubernetes/kubernetes/blob/cab439b20fbb02cc086bf63b6dd7d51f1908067c/cluster/juju/layers/kubernetes-worker/reactive/kubernetes_worker.py#L658 | 22:24 |
| EdS | not to worry - I now know I'm not going crazy... | 22:28 |
| EdS | when do you expect that to be available? | 22:29 |
| EdS | thank you | 22:29 |
| EdS | thanks all :) I gotta get some sleep | 22:37 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!