Chipacamo'in all08:08
Chipaca<jam> Chipaca or facubatista  in case bthomas comes back, it feels like he has it backwards. leader_settings_changed notifies *non* leaders that the settings have changed, leader_elected is telling the unit that they are now the leader08:19
bthomasChipaca: Does leader_elected always follow leader_settings_changed or vice versa ?08:22
Chipacabthomas: what does 'before' and 'after' mean when it's different units?08:23
Chipaca(nothing, that's what)08:23
Chipacaunless you have atomic clocks on the units08:23
Chipacathen we can talk08:23
bthomasChipaca: wouldn't all unit receive the same events ? I do appreciate the problem with distributed clocks. However isn't there a single source of the firing of events - Juju .08:24
Chipacabthomas: yes, but my point is there can be no ordering guarantee, so why do you care to ask?08:33
bthomasChipaca: If there is a single source of events then the source gaurantees ordering . Suppose there is no ordering guarantee but there is the weaker gaurantee that a leader_elected is always assoicated with a leader_setting_changed. Then even with this weaker gaurantee it becomes hard to make a case on differentiating the use case for these two events. An ordering gaurantee actually helps differentiate the use case. Do I make se08:36
oscar51I have a question regarding juju's Operator framework09:15
oscar51does anyone know what exactly triggers "on.config_changed"?09:18
Chipacaoscar51: running 'juju config <app> <config>=<something>'09:24
oscar51I see09:25
Chipacaoscar51: why?09:25
oscar51I'm having trouble with one of our charms, on.config_changed keeps being called, causing downstream trouble. I'm trying to troubleshoot it09:26
oscar51but it helps to know that something is calling juju config09:27
Chipacaoscar51: https://discourse.juju.is/t/charm-hooks/1040#heading--config-changed09:27
Chipacaoscar51: there are 3 other situations in which it's triggered09:27
Chipacaoscar51: from your description, assuming you don't have something rogue doing 'juju config', your agent might be dieing a lot?09:28
Chipacaoscar51: what does debug-log show?09:29
oscar51thank you for those helpful pointers09:30
Chipacaoscar51: what is the charm in question?09:32
oscar51I'm using debug-log a lot, it's useful but does not reveal what is going on09:32
oscar51we have multiple custom charms. our stack is complex09:32
Chipacaoscar51: have you done 'juju model-config logging-config="<root>=DEBUG"'?09:32
Chipaca(to up the log level of most everything)09:32
Chipacaoscar51: is this on k8s?09:33
oscar51model-config is already set to debug:09:34
oscar51logging-config                model       <root>=DEBUG;unit=DEBUG09:34
oscar51no, it's not on k8s.09:34
Chipacaoscar51: if it were the hook itself having issues i'd say try 'juju debug-code', but iiuc the issue is with whatever is triggering config-changed09:36
Chipacaoscar51: one other thing09:40
oscar51I see in debug-log that before entering "on.config_changed" juju prints: "Using local storage: /var/lib/juju/agents/unit-*****/charm/.unit-state.db already exists", do you think this might be a cause?09:40
Chipacaoscar51: your charm itself could be doing .on.config_changed.emit()09:40
oscar51good point09:41
oscar51something I'm going to check09:41
Chipacaoscar51: nah, that's correct for non-k8s09:41
oscar51I checked the charm code, and there are no emit() at all09:42
Chipacaoscar51: (on k8s you don't necessarily have non-ephemeral storage on the operator pod so state gets shipped back to the controller)09:42
Chipacaoscar51: good hunting, let me know what you find09:43
Chipacaif it turns out there's something we could log to make things easier we can do that09:44
oscar51I'm continuing to hunt this.. we need to resolve it :)09:44
oscar51Do you think this might trigger a config event in the operator framework:10:27
oscar51DEBUG juju.worker.uniter.remotestate update status timer triggered10:27
oscar51There is also this one:10:36
oscar51juju.worker.uniter.operation running operation run update-status hook10:36
facubatista¡Muy buenos días a todos!10:51
bthomasनमस्ते facubatista10:54
Chipacaoscar51: no, that's the update-status event that happens periodically, https://discourse.juju.is/t/charm-hooks/1040#heading--update-status10:55
facubatistahola bthomas10:55
Chipacaoscar51: (you can change the periodicity with e.g. «juju model-config update-status-hook-interval=1m»)10:56
Chipacafacubatista: 👋❗10:56
facubatistaChipaca, hola10:57
oscar51Chipaca: I see10:57
Chipacaoscar51: now, your charm could connect update_status to your config-changed handler10:57
Chipacaoscar51: i.e., e.g., self.framework.observe(self.on.update_status, self._config_changed)10:58
Chipacaoscar51: but that's on you :)10:58
oscar51Chipaca: I will look at the code if that happens10:58
Chipacaoscar51: but presumably you've observed something like 'juju.worker.uniter.remotestate got config change' in the logs also11:00
oscar51Chipaca: I appreciate your input. I'm still looking into code and logs11:06
oscar51I was hoping for something simple, like assigning the on.update to the config function ;)11:07
facubatistajam, 'juju show-status-log' is not really what we want... it just shows the past changes (no "follow"), and you need to run it in a separate terminal (no mixed lines with the rest of the logs)11:21
facubatistajam, what would be more like the users are doing would be something like "juju debug-log --show-status-changes", for example11:22
facubatistajam, do you think an option like this would be something feasible for juju to implement?11:22
oscar51I see these in debug-log, does it mean that the charm is restarted?: DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-****-***/agent.socket <nil>}12:06
oscar51And after that line comes this one: Operator Framework 0.9.0 up and running.12:09
Chipacaoscar51: the charm isn't a long-lived thing, it runs once per event12:26
Chipacaoscar51: so you should se ethe "Operator Framework X.Y.Z up and running" once per event12:27
Chipacaoscar51: the 'starting jujuc server', I don't know, I'd have to ask somebody with more juju juju12:27
oscar51Chipaca: okay, thanks12:27
Chipacamaybe jam is around?12:27
oscar51is there some way to know what config changed when on.config_changed is called?12:31
Chipacaoscar51: maybe setting unit=TRACE ?12:36
Chipaca(i don't know but i'd try that)12:36
oscar51for debug-log?12:36
Chipacaoscar51: i mean, juju model-config logging-config="<root>=DEBUG;unit=TRACE"12:37
Chipacaoscar51: and then debug-log yes12:37
oscar51running that command required admin access to the controller, so I did it using what was described here: https://juju.is/docs/logs12:45
oscar51it didn't reveal anything. what is the purpose of the 'event' argument in the events (talking about the Python package used to write charms)12:52
oscar51it has the type ops.charm.ConfigChangedEvent, can it be used to figure out what config has changed?12:53
Chipacaoscar51: not in the event itself, but you can grab the whole config and print it out12:56
jamChipaca, oscar51 : when running a hook, we spin up a worker to respond to the events from the charm, it is expected, and is DEBUG for a reason, since you shouldn't typically care12:56
Chipacajam: oscar51 is seeing config-changed again and again12:57
jamoscar51, Chipaca my guess is something in the framework is calling event.defer() so the framework emits it again before the next hook12:58
oscar51jam: what does event.defer() do?12:58
oscar51I saw a defer in the code12:58
Chipacajam: queues the event for processing later12:58
Chipacaoscar51: ^12:59
Chipacaoscar51: so the next time you get e.g. update-status, first you'd handle the defered event12:59
Chipacai should've known to ask this already :-/ thanks jam :)12:59
oscar51hmm. not sure defer is the problem but I will check the code13:01
jamits pretty easy to end up with an infinite defer() loop if the precondition to why you might want to defer never changes. but defer() in config-changed is almost never what you want13:04
oscar51there is a event.defer() at the end of on_config_changed, do you think it might be the cause?13:04
jambecause you aren't waiting for the same config again, you are waiting for the *next* config-changed with new config13:04
jamoscar51, that will tell the framework that it should emit on.config_changed before the next hook, and since it defers at the end of the function, it will be queued before the next event as well13:05
oscar51sounds like it might be the cause. I really wonder why it was typed there in the first place13:05
Chipacaoscar51: in particular for config-changed, it's almost certainly a bug13:09
Chipacaand if it's unconditional at the end of a handler, it's also a bug13:10
oscar51I'm testing out the updated code right now13:10
oscar51it looks like it was solved, but13:13
oscar51thanks chipaca and jam for your help13:14
oscar51very much appreciated13:14
Chipacamethinks we shold log "defering this event" and "running this event because it was defered"13:14
jamChipaca, if we don't, we definitely should, but I thought there were messages around that13:14
Chipacai don't think there are13:15
Chipacai think we talked about doing it but didn't13:15
Chipacai'll file a bug13:15
mupIssue #404: log event deferral and re-emission <Created by chipaca> <https://github.com/canonical/operator/issues/404>13:17
mupIssue operator#404 opened: log event deferral and re-emission <Created by chipaca> <https://github.com/canonical/operator/issues/404>13:17
jamChipaca, ah, it is not trivial because *new* events are "record this event, reemit()" vs running deferred events are reemit()13:18
Chipacashould still be doable :)13:18
jamChipaca, I think we could watch for "single_event_path is not None"13:18
Chipacajam: looks like that will work yes13:20
facubatistaChipaca, it looks your timing comparations are flaky13:27
facubatista>               assert p1.stat().st_mtime == p2.stat().st_mtime13:27
facubatistaE               assert 1600257598.0 == 1600257598.679217313:27
facubatistamy branch failed only for py 3.5 and py 3.7 in arm (amd64 both ok)13:27
Chipacalooks like it use utime instead of utimensat13:28
Chipacasilly copy213:28
Chipacafacubatista: i can fix in a bit13:28
facubatistaChipaca, thanks13:28
Chipacafacubatista: but you should be able to retry for it to pass for now13:28
facubatistaChipaca, I still miss a review there, so no rush13:29
facubatista(actually, I'm two approves behind, so...)13:29
Chipacafacubatista: I'd change that test to be |p1-p2| < 113:29
facubatistaChipaca, you have a pytest.approx()13:30
* Chipaca checks his pockets13:30
Chipacano, i probably left it on the piano13:30
Chipacafacubatista: what would be a good __str__ for EventBase?13:48
Chipacafeels like a generic <EventFoo at 0xDEADBEEF> works13:49
Chipacaor maybe <EventFoo via eventhandle>13:49
facubatistathe type of event for sure, don't know if there's some internal info that could surface13:56
facubatistathe id() of it doesn't make sense, right? as we would not have N of those, and if we defer() it later it will have another id13:56
Chipacaeventhandle probably is the best shot13:57
Chipacadeploying some charms now to get a handle on what that looks like irl13:57
* facubatista -> early lunch before afternoon's meetings storm14:57
axinois there an easy way to test running a hook with some data in StoredState() ?15:06
bthomasaxino: Not sure what you are asking. But you can always assign a key with an integer counter to your charms instance of StoredState. Then on each update_status hook set ActiveStatus with the message being the value of the counter from stored state and incrementing the counter in stored state. This way you will see that counter increasing in jujus show-status . I have not tried this.15:12
Chipacaaxino: harness.charm._stored.foo = 42 should dtrt?15:16
deejChipaca: I suspect he's looking for something like what i was asking for a couple weeks back, a good way to bulk set state15:17
deejAla what you can do with config15:17
bthomasChipaca, jam facubatista : Q4) Why does one need relation_depated hook/event ? Current docs say that relation departed is "to give the charm an opportunity to clean up any optimistically-generated configuration." But it is not clear what optimistically generated configuration is.15:27
jambthomas, several reasons. Wordpress is using Postgresql as its storage, wordpress is going away, Postgresql needs to know that it can delete those dbs.15:28
jambthomas, Cassandra cluster was scaled up to 7 nodes, it is being scaled down to 515:28
jamcassandra 0-5 need to know that 6/7 are going away, and should no longer be used as part of the cluster.15:29
Chipacacassandra going away parties are *wild*15:29
bthomasjam : but there is relation depated and relation broken. Sorry I pasted the explanation for relation broken as relation departed. Never the less I am unclear as to why both are required. BTW standup now15:29
jambthomas, I have a different meeting now, so won't be there.15:30
bthomasno worries will catcup with you later15:30
jamdeparted is per unit15:30
jambroken is the overall relation to the application15:30
bthomasok thank you15:30
bthomaswill think a bit more about this.15:31
* facubatista is back15:31
mupIssue operator#405 opened: Harness begin_with_initial_hooks() fails when no relations added <Created by zzehring> <https://github.com/canonical/operator/issues/405>15:46
mupPR operator#406 opened: Add empty list as default return if no relation ids in map <Created by zzehring> <https://github.com/canonical/operator/pull/406>17:19
mupIssue operator#405 closed: Harness begin_with_initial_hooks() fails when no relations added <Created by zzehring> <Closed by chipaca> <https://github.com/canonical/operator/issues/405>18:36
mupPR operator#406 closed: Add empty list as default return if no relation ids in map <Created by zzehring> <Merged by chipaca> <https://github.com/canonical/operator/pull/406>18:36
mupIssue operator#407 opened: Have a quick "what you should look for" as packaging docs <Created by jameinel> <https://github.com/canonical/operator/issues/407>20:12
=== ec0[m] is now known as ec0

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!