/srv/irclogs.ubuntu.com/2020/04/03/#smooth-operator.txt

Dmitrii-Sho/05:46
jamDmitrii-Sh, morning. (I'm not really here, but there were comments left by someone on https://github.com/canonical/operator/pull/212)06:16
Dmitrii-Shjam: thanks, just responded there06:17
Dmitrii-ShI think extending add_relation by allowing it to accept initial data would be a good way to move some checks away from update_relation_data and make it less overloaded than it is right now06:18
jamDmitrii-Sh, maybe we have both options written out and can compare the tests that result and what we feel is easier to understand06:19
Dmitrii-Shjam: ok, I'll make a branch locally so that I have the current implementation preserved06:20
Dmitrii-Shand see what I can come up with06:20
jamI had initially done the initial_app_data form, and then was trying to streamline and saw that you could use update for self. but I wasn't convinced which way would be better.06:21
Dmitrii-Shjam: also, if I look at this signature `def add_relation(self, app, *, initial_unit_data=None, initial_app_data=None, remote_app_data=None):`, it gets confusing on which argument is for which purpose for peer relations06:24
jamapp is redundant for peer but the rest isn't06:24
jamI would say for peers we just make one of them illegal to set06:25
Dmitrii-Shhmm, there is a case where our unit comes app and there's already something in peer app relation data06:26
Dmitrii-Shjust thinking about a later addition of `relation-created` events - add_relation should trigger "relation-created" for our unit but during that there may already be some app relation data present for that relation-id and a "remote app" (which may be local on a peer rel)06:29
Dmitrii-ShI think it should be illegal to use the `remote_app_data` argument for peer relations - I am going to raise an exception if that happens and see how it looks like06:31
t0mb0Hi I've been struggling with some issues with actions. It seems every time I tried to retrieve some data from StorageState with my action I keep getting the initial value of my variable, however this variable gets populated with some data as part of the configuration process and I can confirm the variable is set. It seems that when I call my action it is referencing a different state object?? https://pastebin.canonical.com/p/jWT6bDcRVR/06:39
t0mb0am I missing something ^^ or should I raise a bug06:39
jamt0mb0, can you link the code?06:39
t0mb0jam, https://pastebin.canonical.com/p/7BnhqRxDfG/06:40
jamt0mb0, so #1 i would suggest moving from an on_start setting all the attributes to a 'set_default()' call in __init__06:41
jamt0mb0, the on_start doing apt-get stuff is fine06:42
t0mb0jam, I had done a set_default for in __init__ for the data i'm trying to retrieve06:42
jamt0mb0, that isn't the code you linked06:43
jamhard to debug code that isn't what I'm looking at :)06:43
t0mb0jam, yeah I know, I'm just saying it's something I've already tested06:43
t0mb0I've been trying all number of things to try and isolate the issue :P06:43
t0mb0I'll move it back to __init__ though06:43
jamt0mb0, ah, you're on k8s, right? Charms fire hooks in the Operator pod, but fire actions in the Workload pod.06:44
jam(currently)06:44
jamI believe there is a flag06:44
t0mb0:O06:44
t0mb0I am in k8s yup06:44
jamt0mb0, sigh, the flag is only in dev mode. So if you set "export JUJU_DEV_FEATURE_FLAG=juju-v3" then "juju run-action" is renamed to "juju run" and has a "juju run --operator" to run the action in the operator pod.06:47
jamt0mb0, I'd bring that up in #juju about not being able to supply --operator to 'juju run-action'06:47
t0mb0jam, does this work with microk8s juju?06:49
t0mb0I don't seem to have that flag when I export the variable there06:49
jamt0mb0, it might be a 2.8 feature, not sure.06:49
jamt0mb0, with any k8s06:50
t0mb0hmm06:50
jamt0mb0, submitted https://bugs.launchpad.net/juju/+bug/1870487 for you.06:50
t0mb0thanks!06:52
t0mb0jam, I do have juju run --operator06:52
t0mb0but I'm not trying to run a shell cmd, I'm trying to run a charm action06:52
t0mb0ah - on another read of that bug report you're saying there is no reason not to also include --operator to "run-action"06:53
jamt0mb0, ah, it was juju run that grew --operator, not run-action06:54
jamt0mb0, bug update06:55
jamupdated06:55
t0mb0jam, shouldn't state be the same whether it's running in the operator pod or on the workload?06:56
jamt0mb0, state is read from the local disk atm, no way to read it from the other pod06:56
t0mb0jam, so if the operator pod was to die then you'd lose that state?06:57
jamt0mb0, operator pods are currently stateful stets06:57
jamsets06:57
jamt0mb0, so they ask k8s for a disk mount that will float with a restart of the pod06:58
jamt0mb0, in 2.8 there are some hook tools that will let us save state back in the controller, and work that would allow operator pods to become stateless06:58
jamthe operator framework doesn't support that yet. its on our 'todo next' list06:58
t0mb0jam, I also noticed that if I initialised admin_password in on_start, then my action was able to read it, it was just when that value was overwritten in the configure hook06:58
jamt0mb0, once you declare a pod spec, the Juju code copies the charm in an init container to the application pod06:59
jamt0mb0, currently the charm state is stored in the state dir, causing it to get copied06:59
jambut that is 'state as it exists at one point in time'06:59
jamwhich is naughty of how it is all done and definitely shouldn't be relied upon07:00
Dmitrii-Shthat also has an effect of not being able to update StoredState from actions such that regular hooks can see those updates07:07
Dmitrii-Shas jam says, it should work when we have controller-side state storage used in the framework with juju 2.807:08
t0mb0am I correct in understanding that by emitting an event like this https://github.com/dshcherb/cockroachdb-operator/blob/master/src/charm.py#L192 emit() will block or should it be asynchronous?08:14
Chipacamo'in08:16
niemeyert0mb0: That call dispatches the event to handlers08:16
niemeyerMorning08:16
t0mb0niemeyer, so if I was to do something like this https://pastebin.canonical.com/p/hpzGSrTSGh/ it shouldn't cause a stack overflow or hit maximum recursion limits?08:18
niemeyerIf you emit an event inside the handler of that event, yes bad things can happen if that takes place forever08:20
niemeyerIt's also not super nice to read and maintain despite anything else08:21
niemeyerIt's awkward to say "Hey, it's now initialized!" inside the thing that handles "Oh, is it initialized? Cool, let me handle that.".. makes sense?08:22
t0mb0niemeyer, it shouldn't take place forever in that eventually the pod would have loaded (this is an interim solution until juju notifies us when the pod is alive).08:22
niemeyerIt's better that it's known to be going away, but it doesn't make they nice.. I'd still look for a different solution08:23
niemeyer*that08:24
niemeyert0mb0: You might just defer the event, for example08:24
t0mb0niemeyer, yeah that's my preferred solution atm however it's kind of annoying having to wait around for an update-status hook to fire for that event to be processed again08:25
niemeyert0mb0: Okay, let me think about this and I will try to suggest something08:27
t0mb0niemeyer, it's not super critical, I think we're happy to just wait for an update-status for the time being until we have more k8s hooks to work off08:28
t0mb0I was just wondering if the emit() loop was viable08:29
niemeyert0mb0: Viable it is.. it's just not a very nice way to organize things.. it's a very complex way to run an unbounded loop08:45
Dmitrii-Sht0mb0: .emit() is not asynchronous but handling of that event may be deferred by whoever receives it08:46
Dmitrii-Sht0mb0: from what I understand you would like to update the unit status based on whether a workload pod is actually ready or not08:48
Dmitrii-Shthis is a little odd to do from the operator pod because (in my view) its status is determined by whether it has enough data to produce a pod spec08:50
Dmitrii-ShI can see the use case where you need to, for example, expose a service URL only when your services are actually up because otherwise a remote unit might fail trying to connect using that URL immediately after receiving it08:51
Dmitrii-Shideally a client would retry on connection failures because we cannot guarantee that a service is up and will stay up when we expose a URL over a relation08:52
niemeyerDmitrii-Sh: That seems to be well understood. It's exactly the trying that is being discussed here.08:53
Chipacahow do we define when a charm is 'done'?10:12
Chipacain my mind that's a case-by-case thing, with a process for ekeing out what done is for each charm10:13
niemeyerWhat would it mean for a charm to be done?10:13
Chipacaniemeyer: that the person assigned with doing it can tick it off a list :)10:15
niemeyerChipaca: This is moving the question to what was the person aiming to do with that list..10:16
Chipacawe can't say "we need you to do this thing" without saying what that thing being "done" actually is10:16
niemeyerChipaca: The charm manages the entire lifecycle.. it's never done10:16
niemeyerIt's an event consumer.. it's always waiting for the next reason to do something10:16
Chipacaheh10:17
niemeyerChipaca: Exactly.. and we can't say what "done" means without us knowing what "we need you to do this thing" means10:17
niemeyerChipaca: The charm can be done handling a hook10:17
niemeyerChipaca: "heh" is pretty much never the right answer, though10:18
ChipacaI thought you made a joke10:18
Chipacaniemeyer: we _have_ told people "we need you to do this charm". So I'm askig, how do they, and how do we, know when they're done?10:20
niemeyerChipaca: So the actual question is "When are people done writing a charm?".. okay, that's something else altogether10:21
niemeyerChipaca: The charm is supposed to manage the lifecycle of the applications involved. If we can start, stop, handle the applications relationships so it can actually work, communicate the necessary information for the underlying software to be useful.. it seems like we've got something tangible in hands10:23
Chipacaniemeyer: so questions of vert/horiz scalability, managing secrets, upgrades, metrics and logs, are all beyond the scope of a basic "charm is done"?10:58
Dmitrii-Shjam: (for when you're around). Updated https://github.com/canonical/operator/pull/212 per our discussion and will have a look at debug-code.10:59
niemeyerChipaca: I'm assuming you're talking specifically about the conversation we had yesterday, which was in the context of getting the Charmcraft developers to come up with working examples so that they can benefit from having actual experience on how to write a charm in the framework, and so that we can develop experience to help other people11:01
Chipacaniemeyer: yes but also for teams beyond just ours, that are also being tasked with charming things11:02
niemeyerChipaca: If that's the case, then we don't need something fancy and comprehensive to start documenting these charms in our catalog of examples11:02
niemeyerChipaca: If it's something else, then I cannot possibly answer that question without knowing what that something else is..11:03
niemeyerChipaca: Yes, sometimes managing secrets, scalability, logs, metrics, are fundamental to consider basic work on a charm donw11:03
niemeyer*done11:03
facubatistaMuy buenos días a todos!11:04
Chipacafacubatista: buen día!11:04
Chipacafacubatista: buen viernes, even :)11:04
facubatistahola Chipaca! :)11:04
niemeyerfacubatista: Que tal?11:09
facubatistaniemeyer, hola! all fine, I already did some gym, have my mate, sun enters through the window...11:13
facubatistayou?11:13
niemeyerThat sounds nice11:13
niemeyerI just finished my morning bottle of mate11:13
facubatistagood11:14
facubatistaniemeyer, Chipaca, in my mind, we should have a comprehensive set of charms that shows how to do each of those things (vert/horiz scalability, managing secrets, upgrades, metrics, logs, etc) but we should NOT do all those things on each example11:23
facubatista(of course, we need a couple of "big fully done" examples)11:24
facubatistabut the small ones are very useful for other people writing charms... when a developer writing a charm says "I'm doing my charm, have much done, need to add 'secrets' here", we can point to docs, and an example where is easy to find *that*11:25
facubatista(note that even they should be very small and focused examples, they will be fully functional in the sense that can be deployed, we run functional tests on them, etc)11:26
niemeyerfacubatista: The medium term task for the charmcraft team in that regard should be to identify what patterns need to be enabled.. it's not a goal to be maintaining comprehensive charm that can e.g. do everything one might possibly want to do in PostgresSQL in production11:33
facubatistayeap11:34
niemeyerfacubatista: For this example, stub would be much better positioned to do that.. but what we must do is ensuring that he can indeed do that and isn't running short in support from the underlying framework11:34
facubatistaperfect11:35
Chipacafacubatista: could you look at #196 again?12:04
facubatistaChipaca, yes, I'm on that right now12:05
Chipacafacubatista: 👍12:05
ChipacaI went to look at https://github.com/juju/juju/pull/11389 and at the bottom of all the discussion, it's merged :-D12:23
facubatistaChipaca, done with #19612:28
facubatista(not sure what you were expecting from that)12:28
Chipacafacubatista: what you've done seems fine12:29
Chipacafacubatista: i'm just allergic to "changes requested" and then nothing12:30
facubatistasure12:30
facubatistawe probably need to interact more here, regarding PRs... like "answered your comments, pushed a commit, something still open to discussion"12:31
facubatistaotherwise we (I) get to those changes after reading other 12496 issues12:31
Dmitrii-Shrelation-created in a to-be-published 2.8-beta1:12:40
Dmitrii-Shjuju show-status-log cockroachdb/012:40
Dmitrii-ShTime                        Type       Status       Message12:40
Dmitrii-Sh03 Apr 2020 15:25:20+03:00  juju-unit  executing    running proxy-listen-tcp-relation-created hook12:40
Dmitrii-Sh03 Apr 2020 15:25:21+03:00  juju-unit  executing    running cluster-relation-created hook12:40
Dmitrii-Sh03 Apr 2020 15:25:22+03:00  juju-unit  executing    running leader-elected hook12:40
Dmitrii-Sh03 Apr 2020 15:25:23+03:00  juju-unit  executing    running config-changed hook12:40
Dmitrii-Sh03 Apr 2020 15:25:24+03:00  juju-unit  executing    running start hook12:40
ChipacaDmitrii-Sh: nice12:49
ChipacaDmitrii-Sh, niemeyer, did I understand right yesterday that juju changed the command we need to run to set a pod spec?12:51
Dmitrii-ShChipaca: https://github.com/juju/juju/pull/1132312:53
Dmitrii-Shyes, that's correct12:53
Chipacawhat does the deprecation look like? are we ok to carry on using the old one, or do we need to use jujuversion to run one or the other?12:54
crodriguezGood morning everyone! Just wanted to say thank you for yesterday, I was able to convert my pod_spec to v3 and incorporate the secret successfully :)13:08
crodriguezso is the whole charm-tech team EMEA?13:09
facubatistagood morning crodriguez!13:10
crodriguezhappy friday too !13:10
Chipacacrodriguez: part EMEA, part americas13:14
Dmitrii-Shcrodriguez: glad to hear it!14:14
Dmitrii-Shfacubatista: https://paste.ubuntu.com/p/kvNvF45Jm4/ - seems to work fine14:40
Dmitrii-Sh(first impression)14:40
facubatistaDmitrii-Sh, great!14:47
facubatistaDmitrii-Sh, just checking the value of the envvar is enough, for the different cases14:47
facubatista(I'm actually writing the Python code to react on that)14:49
Dmitrii-Shfacubatista: need to split the string if multiple breakpoints are specified https://paste.ubuntu.com/p/QfTSwnnjXg/14:54
facubatistaDmitrii-Sh, yeap14:54
crodriguezdesign question: My charm deploys a mssql container and its operator container. If I want to deploy (optionally) another container that contains tools to interact with the main container, should I create a new class for it? I see that if I simply add the container info in the same pod_spec, it includes it in the main mssql pod, which is not what I want, I would like a separate mssql-tools pod15:05
crodriguezor should it be a separate charm altogether and relate them in a bundle? (I hope not)15:09
niemeyercrodriguez: It's a pretty good question.. I don't have a good answer myself because I don't yet understand the design of the k8s integration as well as I want to15:12
niemeyercrodriguez: What I'd like to enable is a situation where the main charm container lives right next to the managed container, and you might have any number of workload containers next to it, but that's not there yet15:12
niemeyercrodriguez: Would be nice to have the view of someone that is comfortable with the ways currently work and might provide some advice on that15:13
niemeyercrodriguez: From the charm team, the best person would be jam.. tvansteenburgh might also have a good view on this from the k8s perspective15:13
niemeyercrodriguez: jam is off today15:14
crodriguezniemeyer: yes, maybe they could help. This is my first k8s charm so I'm not always sure of best practices . I was thinking that I'd try to create a 2nd charm class that is triggered optionally, and see how that does15:15
Dmitrii-Shcrodriguez: just trying to understand the use-case a little bit more: is there anything specific that you would like to do from a separate pod with mssql-tools?15:16
crodriguezDmitrii-Sh: mostly just make it visible. Otherwise it's hidden, the pod/mssql-0 gets 2 containers instead of 1, like this (ignore the crashloopbackoff lol) https://usercontent.irccloud-cdn.com/file/be7sDsYu/image.png15:18
crodriguezfor the use case, I am also going to add other sql components in this charm, such as a Machine Learning component, and I will have to decide if I want to extend the base mssql container or if I would prefer to deploy these components in separate containers15:20
Dmitrii-Shcrodriguez: hmm, the first impression is that a machine component would be a separate application15:21
Dmitrii-Shand tools to access mysql would be, for example, a part of its container image15:22
Dmitrii-Shwhen Juju deploys an application to k8s it creates a Deployment in k8s with a pod template specified15:23
Dmitrii-Shthat pod template may have multiple containers15:24
Dmitrii-Shbut, as far as Juju is concerned, a combination of a workload pod and an operator pod is considered to be a unit15:24
Dmitrii-Shso individual containers in that unit are not visible15:24
crodriguezOkay, so there is not really a way to display more than one pod with one k8s charm right now, is that correct?15:26
Dmitrii-Shcrodriguez:  1 app can have multiple units which would correspond to multiple pods15:29
Dmitrii-Sheach pod can have multiple containers - that part is within the scope of 1 unit15:29
Dmitrii-Shso, yes, that's correct15:29
Dmitrii-Shcrodriguez: containers within a pod share networking and storage, for example15:30
crodriguezDmitrii-Sh: ok! so to make multiple units in this framework, would it be to create multiple charm classes in charm.py ?15:30
Dmitrii-ShSimilar to non-k8s charms, all units share the same code and have one Charm class. Except for actions, that code gets executed in operator pods, not workload pods (where multiple containers would be)15:32
Dmitrii-Shso you'd still have one class providing the same code to run multiple units15:32
Dmitrii-Sha leader unit would be able to use pod-spec-set, other units would mostly react to lifecycle events and, if a leader is gone, one of the units would take leadership over and be able to use pod-spec-set15:34
Dmitrii-Shcrodriguez: what are the component-specific things that you need to do in the charm code?15:35
Dmitrii-Shis it about feeding some parameters to the container spec?15:35
Dmitrii-Shcrodriguez: if so, I think you could have multiple classes responsible for different parts of the final podspec15:36
crodriguezDmitrii-Sh:  I'm not sure yet. I was working with mssql-tools image first as it helps me see how the charm structure works with different containers in one charm15:44
crodriguezyou said one app can have multiple units, which would correspond to multiple pods, and that this would be only one class in the code. I do not see that. With using the same class (so the same pod spec), it deploys only one pod.15:46
Dmitrii-Shcrodriguez: just to demonstrate what I mean https://paste.ubuntu.com/p/rgFzBtpSWC/15:52
Dmitrii-Shlet me get some sample output from k8s as well to demonstrate multiple units15:53
crodriguezok thanks Dmitrii-Sh15:57
=== facubatista is now known as Teclado
=== Teclado is now known as facubatista
Dmitrii-Shcrodriguez: https://paste.ubuntu.com/p/2P93kyFxd9/16:13
Dmitrii-Shso in this example there is one charm that gets deployed, the app is called "wp" with 2 units. As a result you get 1 operator pod and 2 workload pods16:14
Dmitrii-Shcrodriguez: if you exec into the operator pod, you will see that there are multiple directories under /var/lib/juju16:17
Dmitrii-Shmicrok8s.kubectl exec -n controller-microk8s-localhost wp-operator-0 -it -- bash16:17
Dmitrii-Shroot@wp-operator-0:/var/lib/juju# ls /var/lib/juju/agents/16:17
Dmitrii-Shapplication-wp  unit-wp-0  unit-wp-116:17
Dmitrii-Shso the charm code for different units gets executed in the same operator pod, however, it runs with a different context and in different charm directories16:18
Dmitrii-Shthe operator pod is the same, the code is the same, the stored state is different, leadership status is different16:18
facubatistaI need help for a error message17:13
facubatistathis looks too long: "breakpoint names must start and end with lowercase alphanumeric characters, and only contain lowercase alphanumeric characters, the hyphen '-' or full stop '.'"17:13
niemeyerfacubatista: 'breakpoint names must look like "foo" or "foo-bar"'17:14
niemeyerfacubatista: I'd drop the dot..17:14
niemeyerfacubatista: Reserve that for namespacing if we have to17:14
facubatistaniemeyer, I just used the same rule than for actions17:14
facubatistabut totally agree17:14
* facubatista fixes the spec17:14
niemeyerfacubatista: These are not actions.. if we have to add namespacing, we should have a good plan in mind17:15
facubatistaniemeyer, and I like the error with the examples; developer can go for documentation if we want the "full rules" anyway17:16
niemeyerfacubatista: Agreed17:16
crodriguezDmitrii-Sh: Sorry if it's late for you, we can continue this conversation Monday morning. So I think we misunderstand each other, maybe it's because we do not reference the same thing when we say "unit". The example with wp still shows the same wordpress pod deployed multiple times. It's the same container image, etc. What I am trying to achieve is to have something like:19:09
crodriguez- 1x mssql-server pod19:09
crodriguez- 1x mssql-tools container (which depends on a different image that mssql-server)19:09
crodriguez- 1x mssql-machine-learning-components19:09
crodriguez- 1 operator pod19:09
crodriguezSince they are dependant on a different image, it's a different unit to deploy. I'm not looking to just scale my mssql-server to 2 containers.19:09
crodriguezBut then, maybe my approach is wrong, and that I should combine all these components in a unique container image.19:09
Dmitrii-Shcrodriguez: hmm, I think I need to read on https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-machine-learning-docker?toc=%2Fsql%2Fadvanced-analytics%2Ftoc.json&view=sql-server-ver1519:20
Dmitrii-Shskimming through it I think I understand it better. mssql-machine-learning-components is just an add-on service to MSSQL to be able to execute python and R scripts in the DB itself.19:20
Dmitrii-Shit feels like ML services should be a separate container in the mssql-server pod19:21
* facubatista loves `watch -n 5 -d juju status`19:21
Dmitrii-Shmssql-tools could be as well just for the case where you want to exec into a container and access it through the CLI client19:21
crodriguezyeah I'm reading more about it and it might actually be a bigger image that includes mssql-server itself. From https://github.com/Microsoft/mssql-docker/tree/master/linux/preview/examples/mssql-mlservices19:22
crodriguezMy point remains that if I wanted to deploy separate pods from one charm, there's no clear answer on how to do that19:23
crodriguezand mssql-tools was just a practice to figure out how to do this. because it's actually included by default in the mssql-server, I do not need to deploy it separately. But I would like to understand how to do it if I wanted to go that route, you know?19:24
Dmitrii-Shcrodriguez: multiple pods, each with different sets of container specs in one charm - no. It's not possible to do that with deployment objects in k8s IIRC (https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment) and Juju apps result in deployment objects being created19:29
Dmitrii-ShI think we need to work out the target yaml for K8s that we would like to see19:30
Dmitrii-Shand then map it to a podspec19:30
Dmitrii-Shcrodriguez: "would like to understand how to do it if I wanted to go that route", - understood, I sent out an invite for Monday to go over this19:31
Dmitrii-Shlet me know if the time is OK for you19:31
crodriguezok thank you! yes that time works. Thanks for the additional info19:33
Dmitrii-Shgreat, np19:35
Dmitrii-ShHave a good weekend!19:36
crodriguezyou too Dmitrii-Sh !19:38
facubatistareviews appreciated! Added breakpoint manual call to the framework - https://github.com/canonical/operator/pull/21320:29
niemeyerThanks, will look on Monday20:30
niemeyerFor now, have a great weekend20:30
* facubatista eods20:53
facubatistaBye all! have a nice weekend20:53
Chipacahave a good weekend, all20:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!