| === axw_ is now known as axw | ||
| kjackal | Good morning Juju world! | 07:15 |
|---|---|---|
| === salmankhan1 is now known as salmankhan | ||
| Zic | hi here | 14:20 |
| Zic | lazyPower: hmm, I have a problem with a juju upgrade-charm kubernetes-worker : http://paste.ubuntu.com/24737881/ | 14:21 |
| Zic | running apt-get update manually on the unit machine does not return any error, I don't understand | 14:21 |
| lazyPower | Zic: thats fun. if you run apt-get upgrade does it complain about the 404's now that you've run the upgrade? | 14:23 |
| lazyPower | typically when i see that, its because i have a stale cache lying around | 14:23 |
| lazyPower | *update | 14:23 |
| Zic | lazyPower: nop, if I run apt-get update directly on the unit machine all is fine :( | 14:24 |
| lazyPower | Zic: thats funky. if you re-execute upgrade-charm does it work? | 14:25 |
| lazyPower | is this in aws or your on-prem solution? | 14:25 |
| Zic | this is the fully-AWS cluster | 14:25 |
| Zic | lazyPower: I see in juju debug-log that the upgrade-charm still retrying the apt-get update, but with the same error 404 | 14:26 |
| lazyPower | Zic: hang on, so you ran the apt-get update && apt-get upgade on that host thats currently locked trying to do the same operation in upgrade-charm? | 14:28 |
| Zic | running it after it give-up in juju debug-log yup | 14:30 |
| Zic | hmmmm | 14:30 |
| Zic | solved \o/ | 14:30 |
| Zic | lazyPower: I just did what it advices : apt-get update --fix-missing | 14:30 |
| Zic | upgrade-charm is now unlocked and proceed | 14:30 |
| lazyPower | good deal | 14:30 |
| Zic | did an "apt update --fix-missing" earlier before alerting you, but as the charm use apt-get and not apt, it seems to not be the same cache | 14:31 |
| Zic | I have just one error left, but don't know it is important: unit-kubernetes-master-0: 14:33:22 ERROR juju.worker.metrics.sender could not remove batch "6009d068-6c17-4403-8c4e-bf186e66b8ea" from spool: remove /var/lib/juju/metricspool/6009d068-6c17-4403-8c4e-bf186e66b8ea: no such file or directory | 14:33 |
| Zic | lazyPower: hmm, upgrade-charm kubeapi-load-balancer literally broke Juju, don't know what happened : http://paste.ubuntu.com/24738034/ | 14:44 |
| Zic | the juju controller machine suddenly froze then came back with this | 14:44 |
| lazyPower | Zic: that smells like the juju db stopped. | 14:45 |
| pathcl | hey anybody using vsphere with distributed virtual switch having this bug ? 1619812 | 14:45 |
| Zic | lazyPower: happily it seems to recover automatically | 14:45 |
| Zic | all machine agent was in "unknown" but they just came back | 14:46 |
| Zic | and all is green, even the kube-apilb | 14:46 |
| === pathcl is now known as path | ||
| Zic | -> don't know what happens but it works | 14:46 |
| === path is now known as pathcl | ||
| mattyw | lazyPower, ping? | 14:58 |
| lazyPower | mattyw: pong | 14:58 |
| mattyw | lazyPower, hey there, the problem Zic saw about with juju.worker.metrics - that was following a charm upgrade on k8s right? | 14:59 |
| lazyPower | yeah, i think so | 14:59 |
| lazyPower | Zic: i'm going to talk about you like you're not here ;) | 14:59 |
| Zic | :D | 15:00 |
| Zic | yup | 15:00 |
| Zic | was after a juju upgrade-charm on kubernetes-master | 15:00 |
| Zic | (or maybe just after juju config channel=1.6/stable) | 15:01 |
| mattyw | any idea what the before and after versions were? | 15:01 |
| mattyw | and if metrics had been added | 15:02 |
| mattyw | because if so I believe that's a known bug - just want to make sure there's not a new bug I should know about | 15:02 |
| Zic | mattyw: hmm, if I can find the old revision number of charm somewhere, say it to me but if it does not exist: this was a CDK cluster last fully-upgraded a month ago | 15:06 |
| Zic | was already in 1.6.2, just upgrade Juju operationnal code | 15:06 |
| mattyw | lazyPower, latest version should be recording metrics right? | 15:06 |
| lazyPower | mattyw: yeah, we've had metric support in the k8s charms for the last 3 releases that i'm aware of | 15:07 |
| Zic | unit-kubernetes-master-0: 15:07:09 INFO unit.unit-kubernetes-master-0.collect-metrics error: persistentVolume is not namespaced | 15:07 |
| Zic | it's not in INFO instead of ERROR | 15:07 |
| Zic | s/not/now/ | 15:07 |
| mattyw | Zic, what happens if you run juju collect-metrics kubernetes-master/0 ? | 15:08 |
| Zic | ERROR "kubernetes-master/0" is not a local charm | 15:10 |
| Zic | (it is... in fact) | 15:10 |
| Zic | lazyPower: multi-master is still considered "not for production usage"? | 15:22 |
| Zic | I'm asking myself if I switch this production cluster to multimaster again now it is upgraded to the latest charms | 15:22 |
| === scuttlemonkey is now known as scuttle|afk | ||
| lazyPower | Zic: its supported | 16:21 |
| lazyPower | well, supported in our solution as you're using the load balancer | 16:22 |
| lazyPower | Zic: sorry about the delay was out getting my walk in before the day turned up the heat :) | 16:22 |
| Zic | np :) | 16:23 |
| tvansteenburgh | cmars: i think you were right about zetcd | 16:28 |
| lazyPower | what'd i miss about zetcd? | 16:28 |
| cmars | tvansteenburgh, ok :) yeah, i was trying to get it to work with the etcd charm. ended up with https://github.com/cmars/charm-zetcd but couldn't get the tls stuff working | 16:29 |
| tvansteenburgh | lazyPower: i snapped it, cmars has been playing with it | 16:29 |
| cmars | lazyPower, just some hacking around. i was thinking it'd be neat to use kafka with etcd instead of zookeeper | 16:30 |
| === jcamino100080 is now known as jcamino | ||
| lazyPower | interesting | 16:45 |
| lazyPower | ok :) i thought maybe this was something i should be tracking | 16:45 |
| === scuttle|afk is now known as scuttlemonkey | ||
| bdx | new use case popping up | 21:40 |
| bdx | I'm just having my users add their local vms to JAAS models | 21:41 |
| bdx | this eliminates each user needing to bootstrap and know how to operate a controller | 21:42 |
| bdx | and alleviates the load of a controller on their on their dev vms | 21:42 |
| bdx | this has allowed users to hop right in and deploy things to a local vm from their osx hosts | 21:44 |
| bdx | I have a lxd-proxy charm that I'm having users deploy to the the local vm | 21:44 |
| bdx | then they just access the lxd services on the vm via the lxd proxy | 21:45 |
| bdx | super streamlines the osx dev use case | 21:45 |
| lazyPower | :o you can do that? | 22:20 |
| lazyPower | #TIL | 22:20 |
| tvansteenburgh | bdx: blog post please :P | 22:22 |
| bdx | I am so backed up as far as blog posts go, oh man | 22:22 |
| bdx | creating some quality blog content is going to be my summer project | 22:24 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!