[07:15] <kjackal> Good morning Juju world!
[14:20] <Zic> hi here
[14:21] <Zic> lazyPower: hmm, I have a problem with a juju upgrade-charm kubernetes-worker : http://paste.ubuntu.com/24737881/
[14:21] <Zic> running apt-get update manually on the unit machine does not return any error, I don't understand
[14:23] <lazyPower> Zic: thats fun. if you run apt-get upgrade does it complain about the 404's now that you've run the upgrade?
[14:23] <lazyPower> typically when i see that, its because i have a stale cache lying around
[14:23] <lazyPower> *update
[14:24] <Zic> lazyPower: nop, if I run apt-get update directly on the unit machine all is fine :(
[14:25] <lazyPower> Zic: thats funky. if you re-execute upgrade-charm does it work?
[14:25] <lazyPower> is this in aws or your on-prem solution?
[14:25] <Zic> this is the fully-AWS cluster
[14:26] <Zic> lazyPower: I see in juju debug-log that the upgrade-charm still retrying the apt-get update, but with the same error 404
[14:28] <lazyPower> Zic: hang on, so you ran the apt-get update && apt-get upgade on that host thats currently locked trying to do the same operation in upgrade-charm?
[14:30] <Zic> running it after it give-up in juju debug-log yup
[14:30] <Zic> hmmmm
[14:30] <Zic> solved \o/
[14:30] <Zic> lazyPower: I just did what it advices : apt-get update --fix-missing
[14:30] <Zic> upgrade-charm is now unlocked and proceed
[14:30] <lazyPower> good deal
[14:31] <Zic> did an "apt update --fix-missing" earlier before alerting you, but as the charm use apt-get and not apt, it seems to not be the same cache
[14:33] <Zic> I have just one error left, but don't know it is important: unit-kubernetes-master-0: 14:33:22 ERROR juju.worker.metrics.sender could not remove batch "6009d068-6c17-4403-8c4e-bf186e66b8ea" from spool: remove /var/lib/juju/metricspool/6009d068-6c17-4403-8c4e-bf186e66b8ea: no such file or directory
[14:44] <Zic> lazyPower: hmm, upgrade-charm kubeapi-load-balancer literally broke Juju, don't know what happened : http://paste.ubuntu.com/24738034/
[14:44] <Zic> the juju controller machine suddenly froze then came back with this
[14:45] <lazyPower> Zic: that smells like the juju db stopped.
[14:45] <pathcl> hey anybody using vsphere with distributed virtual switch having this bug ? 1619812
[14:45] <Zic> lazyPower: happily it seems to recover automatically
[14:46] <Zic> all machine agent was in "unknown" but they just came back
[14:46] <Zic> and all is green, even the kube-apilb
[14:46] <Zic> -> don't know what happens but it works
[14:58] <mattyw> lazyPower, ping?
[14:58] <lazyPower> mattyw: pong
[14:59] <mattyw> lazyPower, hey there, the problem Zic saw about with juju.worker.metrics - that was following a charm upgrade on k8s right?
[14:59] <lazyPower> yeah, i think so
[14:59] <lazyPower> Zic: i'm going to talk about you like you're not here ;)
[15:00] <Zic> :D
[15:00] <Zic> yup
[15:00] <Zic> was after a juju upgrade-charm on kubernetes-master
[15:01] <Zic> (or maybe just after juju config channel=1.6/stable)
[15:01] <mattyw> any idea what the before and after versions were?
[15:02] <mattyw> and if metrics had been added
[15:02] <mattyw> because if so I believe that's a known bug - just want to make sure there's not a new bug I should know about
[15:06] <Zic> mattyw: hmm, if I can find the old revision number of charm somewhere, say it to me but if it does not exist: this was a CDK cluster last fully-upgraded a month ago
[15:06] <Zic> was already in 1.6.2, just upgrade Juju operationnal code
[15:06] <mattyw> lazyPower, latest version should be recording metrics right?
[15:07] <lazyPower> mattyw: yeah, we've had metric support in the k8s charms for the last 3 releases that i'm aware of
[15:07] <Zic> unit-kubernetes-master-0: 15:07:09 INFO unit.unit-kubernetes-master-0.collect-metrics error: persistentVolume is not namespaced
[15:07] <Zic> it's not in INFO instead of ERROR
[15:07] <Zic> s/not/now/
[15:08] <mattyw> Zic, what happens if you run juju collect-metrics kubernetes-master/0 ?
[15:10] <Zic> ERROR "kubernetes-master/0" is not a local charm
[15:10] <Zic> (it is... in fact)
[15:22] <Zic> lazyPower: multi-master is still considered "not for production usage"?
[15:22] <Zic> I'm asking myself if I switch this production cluster to multimaster again now it is upgraded to the latest charms
[16:21] <lazyPower> Zic: its supported
[16:22] <lazyPower> well, supported in our solution as you're using the load balancer
[16:22] <lazyPower> Zic: sorry about the delay was out getting my walk in before the day turned up the heat :)
[16:23] <Zic> np :)
[16:28] <tvansteenburgh> cmars: i think you were right about zetcd
[16:28] <lazyPower> what'd i miss about zetcd?
[16:29] <cmars> tvansteenburgh, ok :) yeah, i was trying to get it to work with the etcd charm. ended up with https://github.com/cmars/charm-zetcd but couldn't get the tls stuff working
[16:29] <tvansteenburgh> lazyPower: i snapped it, cmars has been playing with it
[16:30] <cmars> lazyPower, just some hacking around. i was thinking it'd be neat to use kafka with etcd instead of zookeeper
[16:45] <lazyPower> interesting
[16:45] <lazyPower> ok :) i thought maybe this was something i should be tracking
[21:40] <bdx> new use case popping up
[21:41] <bdx> I'm just having my users add their local vms to JAAS models
[21:42] <bdx> this eliminates each user needing to bootstrap and know how to operate a controller
[21:42] <bdx> and alleviates the load of a controller on their on their dev vms
[21:44] <bdx> this has allowed users to hop right in and deploy things to a local vm from their osx hosts
[21:44] <bdx> I have a lxd-proxy charm that I'm having users deploy to the the local vm
[21:45] <bdx> then they just access the lxd services on the vm via the lxd proxy
[21:45] <bdx> super streamlines the osx dev use case
[22:20] <lazyPower> :o you can do that?
[22:20] <lazyPower> #TIL
[22:22] <tvansteenburgh> bdx: blog post please :P
[22:22] <bdx> I am so backed up as far as blog posts go, oh man
[22:24] <bdx> creating some quality blog content is going to be my summer project