=== axw_ is now known as axw [07:15] Good morning Juju world! === salmankhan1 is now known as salmankhan [14:20] hi here [14:21] lazyPower: hmm, I have a problem with a juju upgrade-charm kubernetes-worker : http://paste.ubuntu.com/24737881/ [14:21] running apt-get update manually on the unit machine does not return any error, I don't understand [14:23] Zic: thats fun. if you run apt-get upgrade does it complain about the 404's now that you've run the upgrade? [14:23] typically when i see that, its because i have a stale cache lying around [14:23] *update [14:24] lazyPower: nop, if I run apt-get update directly on the unit machine all is fine :( [14:25] Zic: thats funky. if you re-execute upgrade-charm does it work? [14:25] is this in aws or your on-prem solution? [14:25] this is the fully-AWS cluster [14:26] lazyPower: I see in juju debug-log that the upgrade-charm still retrying the apt-get update, but with the same error 404 [14:28] Zic: hang on, so you ran the apt-get update && apt-get upgade on that host thats currently locked trying to do the same operation in upgrade-charm? [14:30] running it after it give-up in juju debug-log yup [14:30] hmmmm [14:30] solved \o/ [14:30] lazyPower: I just did what it advices : apt-get update --fix-missing [14:30] upgrade-charm is now unlocked and proceed [14:30] good deal [14:31] did an "apt update --fix-missing" earlier before alerting you, but as the charm use apt-get and not apt, it seems to not be the same cache [14:33] I have just one error left, but don't know it is important: unit-kubernetes-master-0: 14:33:22 ERROR juju.worker.metrics.sender could not remove batch "6009d068-6c17-4403-8c4e-bf186e66b8ea" from spool: remove /var/lib/juju/metricspool/6009d068-6c17-4403-8c4e-bf186e66b8ea: no such file or directory [14:44] lazyPower: hmm, upgrade-charm kubeapi-load-balancer literally broke Juju, don't know what happened : http://paste.ubuntu.com/24738034/ [14:44] the juju controller machine suddenly froze then came back with this [14:45] Zic: that smells like the juju db stopped. [14:45] hey anybody using vsphere with distributed virtual switch having this bug ? 1619812 [14:45] lazyPower: happily it seems to recover automatically [14:46] all machine agent was in "unknown" but they just came back [14:46] and all is green, even the kube-apilb === pathcl is now known as path [14:46] -> don't know what happens but it works === path is now known as pathcl [14:58] lazyPower, ping? [14:58] mattyw: pong [14:59] lazyPower, hey there, the problem Zic saw about with juju.worker.metrics - that was following a charm upgrade on k8s right? [14:59] yeah, i think so [14:59] Zic: i'm going to talk about you like you're not here ;) [15:00] :D [15:00] yup [15:00] was after a juju upgrade-charm on kubernetes-master [15:01] (or maybe just after juju config channel=1.6/stable) [15:01] any idea what the before and after versions were? [15:02] and if metrics had been added [15:02] because if so I believe that's a known bug - just want to make sure there's not a new bug I should know about [15:06] mattyw: hmm, if I can find the old revision number of charm somewhere, say it to me but if it does not exist: this was a CDK cluster last fully-upgraded a month ago [15:06] was already in 1.6.2, just upgrade Juju operationnal code [15:06] lazyPower, latest version should be recording metrics right? [15:07] mattyw: yeah, we've had metric support in the k8s charms for the last 3 releases that i'm aware of [15:07] unit-kubernetes-master-0: 15:07:09 INFO unit.unit-kubernetes-master-0.collect-metrics error: persistentVolume is not namespaced [15:07] it's not in INFO instead of ERROR [15:07] s/not/now/ [15:08] Zic, what happens if you run juju collect-metrics kubernetes-master/0 ? [15:10] ERROR "kubernetes-master/0" is not a local charm [15:10] (it is... in fact) [15:22] lazyPower: multi-master is still considered "not for production usage"? [15:22] I'm asking myself if I switch this production cluster to multimaster again now it is upgraded to the latest charms === scuttlemonkey is now known as scuttle|afk [16:21] Zic: its supported [16:22] well, supported in our solution as you're using the load balancer [16:22] Zic: sorry about the delay was out getting my walk in before the day turned up the heat :) [16:23] np :) [16:28] cmars: i think you were right about zetcd [16:28] what'd i miss about zetcd? [16:29] tvansteenburgh, ok :) yeah, i was trying to get it to work with the etcd charm. ended up with https://github.com/cmars/charm-zetcd but couldn't get the tls stuff working [16:29] lazyPower: i snapped it, cmars has been playing with it [16:30] lazyPower, just some hacking around. i was thinking it'd be neat to use kafka with etcd instead of zookeeper === jcamino100080 is now known as jcamino [16:45] interesting [16:45] ok :) i thought maybe this was something i should be tracking === scuttle|afk is now known as scuttlemonkey [21:40] new use case popping up [21:41] I'm just having my users add their local vms to JAAS models [21:42] this eliminates each user needing to bootstrap and know how to operate a controller [21:42] and alleviates the load of a controller on their on their dev vms [21:44] this has allowed users to hop right in and deploy things to a local vm from their osx hosts [21:44] I have a lxd-proxy charm that I'm having users deploy to the the local vm [21:45] then they just access the lxd services on the vm via the lxd proxy [21:45] super streamlines the osx dev use case [22:20] :o you can do that? [22:20] #TIL [22:22] bdx: blog post please :P [22:22] I am so backed up as far as blog posts go, oh man [22:24] creating some quality blog content is going to be my summer project