[04:32] <mup> Issue operator#429 closed: Test harness set_leader erases relation data <Created by stub42> <Closed by stub42> <https://github.com/canonical/operator/issues/429>
[12:46]  * Chipaca brb
[13:31]  * facubatista didn't say hello today, shame on me
[13:32] <Chipaca> facubatista: i've been grumpy for days now it seems
[13:32] <Chipaca> facubatista: ｇｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｄ  ｍｏｒｎｉｎｇ！
[13:32] <facubatista> Chipaca, :)
[13:55]  * bthomas had IRC disruption due to 2.4ghz interference 
[14:16]  * Chipaca had coffee, instead
[14:29] <bthomas> :)
[14:38]  * justinclark has a question
[14:38] <justinclark> Is there a way to create an event that is triggered when the app.status is Active? There's a configuration update that needs to take place after all k8s pods have settled and I'm struggling to find a good way to do that.
[14:39] <justinclark> I'm not tied to the app.status thing - really any way to have a piece of code run in a charm after all k8s pods are ready
[14:40] <justinclark> I've tried deferring events until that condition is true - and it kind of works. However, there could be several minutes before the deferred event re-fires.
[14:41] <bthomas> justinclark: How about tweaking liveness probe or adding a readyness probe ?
[14:42] <bthomas> justinclark: If Juju is forwarding requests to an application container that is not ready then either we are not doing the lifecyle management correctly or Juju has a bug or some behaviour has not been documented properly.
[14:45] <justinclark> I will try tweaking the probes. Though it looks like Juju is sending requests to pods that haven't completely started up yet.
[14:49] <Chipaca> justinclark: i think there is the intention of having juju events for that kind of thing but i don't know if it's planned yet
[14:49] <Chipaca> ^ jam?
[14:50] <Chipaca> also, aside, i'm going to make myself a lot of tea and have some paracetamol and lie down for a bit
[14:50] <bthomas> hope you get well soon
[14:53] <jam> @Chipaca, it is work on this cycle, though AIUI Mark is under the impression that Operator Framework could probably prototype it with the K8s integration already available inside the pod (by forking a process that would call back into juju-run to execute charm code)
[14:53] <jam> justinclark, bthomas ^
[14:53] <jam> The roadmap item is: https://docs.google.com/document/d/12aAYfhrKybaoWMV6HgTnJHP-sByqSurOj6PsNkdenRo/edit
[14:54] <jam> "lifecycle events as charm events"
[15:01] <bthomas> jam : With the current implementation can we assume Juju will set a application container's status as "active" only if both liveness and readiness probes pass ?
[15:02] <jam> bthomas, "set application container's status" what concrete field are you thinking about? (iow, what data are you trying to read)
[15:02] <jam> the value in "juju status"
[15:02] <jam> the contents of "self.model.app.status" ?
[15:02] <jam> some other field in K8s API?
[15:08] <bthomas> jam : following up on justinclark comment I was looking for a way we can ensure the load balanced ingress address does not forward an HTTP API request to an application container that was not yet ready to service it. I am thinking if we could do this by adding a readiness probe or tweaking the existing liveness probe. justinclark is sending an API request in response to a new peer relation and seems to be finding that it is bei
[15:08] <bthomas> forwarded to one of the new peers that are not yet ready to respond to it. I do not think the charm is doing any self.model.app.status check, and perhaps it does not need to since it is sending the request to the ingress address. justinclark : please correct me if I have misunderstood.
[15:09] <jam> bthomas, I am not sure how Juju is configuring the Service abstraction in front of the pods.
[15:09] <jam> https://kubernetes.io/docs/concepts/services-networking/service/ says that you should be using "readiness probes"
[15:09] <jam> so that K8s doesn't try to use a backend that isn't actually ready
[15:09] <jam> IIRC, *juju* can't know how to determine if an application is actually ready
[15:10] <bthomas> jam : agreed this is certainly one of the current shortcomings of our charm.
[15:10] <jam> so that is the responsibility of the charm to define the readiness of the application
[15:10] <bthomas> yes indeed
[15:11] <justinclark> bthomas - that is correct. And it does seem as though reducing the timeoutSeconds/initialDelaySeconds in the readiness probe helped things. The error will pop up once, but it always seems to resolve itself.
[15:12] <justinclark> This is using the ES Python module btw.
[15:12] <justinclark> This will all be unnecessary in ES 7.x :)
[15:13] <bthomas> justinclark: awesome! if readyness probe seems to work it saves us lot of clunky code logic. IMHO it may be a good idea to choose a timeout that does not lead to unpleasent behavior.
[15:13] <bthomas> justinclark: what will be unnecessary in ES 7.x ?
[15:14] <justinclark> Setting "minimum_master_nodes" is no longer necessary. We will probably still set some dynamic settings, but this one in particular is the issue.
[15:15] <bthomas> got it
[15:15] <justinclark> bthomas, we get this error: "illegal value can't update [discovery.zen.minimum_master_nodes] from [1] to [2]" because we're trying to set the value to be higher than the total number of nodes in the cluster.
[15:16] <justinclark> Anyways, I can continue this and give a more complete description in the Github issue.
[15:16] <bthomas> justinclark: will read . Can catch up after standup.
[15:16] <justinclark> Thanks bthomas, Chipaca, jam for helping out
[16:12] <justinclark> bthomas, holding off on the PR for a bit. Investigating some options to eliminate the error completely.
[16:28] <bthomas> justinclark: ack. enjoy the deep think :-)
[18:59] <mup> PR operator#430 opened: Docstrings for storage, log and model modules <Created by facundobatista> <https://github.com/canonical/operator/pull/430>
[22:12]  * facubatista eods