/srv/irclogs.ubuntu.com/2020/06/03/#smooth-operator.txt

jammorning all04:36
mupPR operator#319 opened: ops: get version from git <Created by chipaca> <https://github.com/canonical/operator/pull/319>08:14
mupIssue operator#313 closed: If ops is installed, the test run picks up the wrong one <Created by chipaca> <Closed by chipaca> <https://github.com/canonical/operator/issues/313>08:17
mupPR operator#314 closed: test_main: set up PYTHONPATH so the sub-process picks up this ops <Created by chipaca> <Merged by chipaca> <https://github.com/canonical/operator/pull/314>08:17
jammorning Chipaca08:18
Chipacajam: moin moin :)08:18
jamChipaca, I'd like to chat a bit about serialization of StoredState, we can start on IRC and move from there if you feel it is too involved08:22
Chipacajam: sure08:22
jamChipaca, so I'm pretty sure you're aware that we currently use a pickle into a sqlite database for all StoredState content08:23
Chipacayep08:23
jamJuju will accept key=value pairs for 'state-set/get' and can take that 'state-set' as 'key=value' on the CLI or as YAML key:value from --file  (which can be '-' for stdin)08:24
jamThe current spike was done by putting the pickle bytes into base64 and then writing that as the value08:24
Chipacamhmm08:24
jamit is a pretty obtuse way to understand what your state is, if you ever wanted to do "juju run --unit u/0 state-get"08:25
jamfoo: gANLCi4=08:25
jamI have some concerns about pickle support, though AIUI since we force it to be simple types with a pass by Marshal08:26
jamthere shouldn't be too much trouble if you wrote a pickle with python 3.7 and read it with python 3.808:26
jam(say you did a do-release-upgrade on the machine that was running the charm, and it is now running a different version of python)08:26
Chipacapickle's fine with that, yes08:27
jamChipaca, it doesn't *feel* like a 'stable format to keep your state in over time"08:27
Chipacaeh, it's stable enough (you can use 3.8 to load a pickle from 2.2 for example)08:27
Chipacabut08:27
Chipacait's not very approachable08:27
Chipacaand that might be a bigger issue :)08:27
Chipacajam: given that they're simple types, wouldn't yaml also work for us?08:28
jamChipaca, so the one caveat is that yaml or json as the actual payload loses the type for set vs list08:28
jamAlso, JSON keys are strictly strings IIRC, so you can't have {0: 'foo'}08:29
jamI think YAML supports non-string keys08:29
Chipacaare yaml keys also -- ah08:29
Chipacahmm08:29
Chipacajam: and python?08:30
jamPython has the nice behavior that json.dumps({0: 'foo'}) => '{"0": "foo"}'08:30
jamiow, it silently casts your numeric keys into string keys08:30
Chipacano i mean, make it a python literal08:30
Chipacaand use ast.literal_eval to bring it back08:31
jam$ python3 -c "import json; print(json.dumps({'0': 'foo', 0: 'bar'}))"08:31
jam{"0": "foo", "0": "bar"}08:31
jamChipaca, what is the cleanest way to turn a nested dict into a literal? just repr() ?08:32
Chipacayep08:32
Chipacabah, dicts you can str because repr is str there :) but yes, repr08:32
jamChipaca, and... there isn't much in the way of versioning, so we have a way to change our mind about how we want to store the content.08:32
Chipacas/have/don't have/ ?08:33
Chipacawe could add that ourselves, make the first byte of the value be the version?08:33
Chipacadoes juju care at all what we dump in there? what further processing does it do on it?08:34
jam>>> repr({'0': [1,2,3], 0: set(['a', 'b', 'c'])})08:34
jam"{'0': [1, 2, 3], 0: {'b', 'c', 'a'}}"08:34
jam>>> ast.literal_eval("{'0': [1, 2, 3], 0: {'b', 'c', 'a'}}")08:34
jam{'0': [1, 2, 3], 0: {'b', 'c', 'a'}}08:34
jamChipaca, so Juju *should* just round trip the content we give it.08:34
Chipacayeah, sets have literals too ;)08:34
jamChipaca, yeah, I always use set() because the empty set doesn't have another literal08:35
Chipacaah, yeah08:35
jamsince {} would be ambiguous. I'm also not a huge fan of "did you notice there were no :"08:36
Chipacaand it might fail here :-/08:36
Chipacabecause 'set()' isn't a literal08:36
jam>>> ast.literal_eval(repr({'0': [1,2,3], 0: set()}))08:36
jamTraceback (most recent call last):08:36
jam  File "<stdin>", line 1, in <module>08:36
jam  File "/usr/lib/python3.6/ast.py", line 85, in literal_eval08:36
jam    return _convert(node_or_string)08:36
jam  File "/usr/lib/python3.6/ast.py", line 66, in _convert08:36
jam    in zip(node.keys, node.values))08:36
jam  File "/usr/lib/python3.6/ast.py", line 65, in <genexpr>08:36
jam    return dict((_convert(k), _convert(v)) for k, v08:36
jam  File "/usr/lib/python3.6/ast.py", line 84, in _convert08:36
jam    raise ValueError('malformed node or string: ' + repr(node))08:36
jamValueError: malformed node or string: <_ast.Call object at 0x7f2b34d35160>08:36
jamChipaca, indeed, it does fail :908:36
jam:(08:36
Chipacaso close08:37
jamit is also true that we are running pickle.loads() on unknown data08:37
jamyes we write it from things that are 'safe', but we are still running the engine on data that we're reading back from 'outside'08:38
jamChipaca, code layering wise, it is also quite hard to not use pickle, because Framework is the part that decides pickle, while Storage decides to write it to the DB08:38
Chipacayeah08:39
jam(Framework.save_snapshot is where the marshal.dumps check and pickle.dumps is done)08:39
Chipacai think that could use a refactor, but separately08:39
jamChipaca, though again, we can just move that down into the SQLiteSnapshot layer08:39
jamSQLiteStorage08:39
jamChipaca, indeed. snapshot() from the actual objects that people implement is still defined as a dict so we're ok there08:40
Chipacajam: do we care about the spike being forwards-compatible?08:40
Chipacai guess we do but just checking :)08:40
jamChipaca, as in that the current version of the code can read the content written by future versions?08:40
Chipacaas in the state written by the spike be readable by future versions08:41
jamor if it can't read it, fail gracefully08:41
jamChipaca, we don't care about the current spike, because it was just to show proof-of-concept. no charms that I'm aware of are using it.08:41
Chipacajam: ah so the spike isn't what we'd put in 0.708:41
jamChipaca, correct. I'm using it as a starting point, but taking the time to carefully structure things and make sure it is the serialization that we're happy with.08:42
Chipacaok08:42
jamI'd really rather use something like json/yaml and have it be inspectable by humans.08:42
Chipacaso for the initial spike, pickle is fine then :)08:42
jamI think you missed a comment by Stub that people are writing sqlite3 hacks to unpickle the snapshot fields to try to figure out what is going on.08:42
Chipacai've seen that in bugs and etc, yes08:43
Chipacawhich seems obvious to me -- if i were trying to figure out what was going wrong, i'd want to look at anything i could too08:43
jamChipaca, and we can write a common helper to make it easier to inspect that state08:43
jamChipaca, but ideally it would be less opaque to start with.08:44
jamCould we make the statement "sets aren't allowed" and just go with JSON ?08:44
Chipacajson also mangles numbers08:44
jamah right08:44
jamyeah, I hate that part.08:44
Chipacaso it's my least favourite for this :)08:44
jamif it *failed* I might be ok with it.08:44
jamthe fact that it silently corrupts makes it out08:44
jamsee my above comment where you can json.dumps() and get the integer and string keys colliding with no error.08:45
Chipacawe can come up with special handling for empty sets and work with that, also08:45
stubI'd actually assumed sets were not allowed by 'simple types only', like json and yaml and toml and everything else people are used to now days.08:45
davigar15Hello!08:45
davigar15ERROR juju.worker.uniter.operation hook "start" (via explicit, bespoke hook script) failed: exit status 108:45
stubRecommend feeding the CLI tool using the --yaml argument to avoid CLI command length limitations and encoding and locale issues08:45
davigar15I'm getting that error in k8s charms in 2.8.108:45
davigar15Chipaca: any ideas?08:46
Chipacadavigar15: what operator version?08:46
davigar15latest08:46
Chipacadavigar15: can we see the debug logs?08:46
davigar15yes08:46
jamstub, yeah, --file is currently the way that I'm interacting with state-set.08:46
jamdavi08:46
jamdavigar15, I'd start with "juju model-config logging-config=DEBUG" and "juju debug-log" to get more information about what is going wrong.08:47
stubjam: Maybe this time we can use = and . in our keys :)08:47
Chipacawe _could_ restrict what you can put in state even further, for this feature08:47
Chipacaor, we could special-handle empty sets :)08:47
Chipaca(or sets in general!)08:48
davigar15oooh, good trick. Need to debug it :-)08:48
Chipacajam: yaml.safe_load(yaml.dump(set()))08:49
Chipacajam: → set()08:49
Chipacayaml.safe_load(yaml.dump({'0': [1,2,3], 0: set(['a', 'b', 'c'])})) → {0: {'a', 'b', 'c'}, '0': [1, 2, 3]}08:50
Chipacawe'd have to check if the !!set annotations require a particular pyyaml version08:51
davigar15application-zookeeper-k8s: 10:52:25 DEBUG unit.zookeeper-k8s/2.start     raise TypeError(08:52
davigar15application-zookeeper-k8s: 10:52:25 DEBUG unit.zookeeper-k8s/2.start TypeError: observer methods must now be explicitly provided; please replace observe(self.on.config_changed, self) with e.g. observe(self.on.config_changed, self._on_config_changed)08:52
davigar15There is it08:52
davigar15It has to be explicit. I have self instead08:53
davigar15:-)08:53
davigar15thanks08:53
Chipacajam: but it works in 3.11 (xenial) and 3.12 (bionic) at least08:53
jamChipaca, ^^ that looks like the "catch exceptions and log at higher than DEBUG" :)08:53
Chipacadavigar15: :) sorry we broke your charm, glad we were trying to tell you about it at least :)08:53
Chipacajam: yep08:54
Chipacadid we have an issue for that? i should do that one next08:54
jamwe do08:54
davigar15It's okay! nothing in production yet hahaha08:54
jamChipaca, #28908:54
mupIssue #289: hook exception logged at DEBUG <Created by jetpackdanger> <https://github.com/canonical/operator/issues/289>08:54
Chipacaperfect08:54
Chipacajam: any reason not to go the yaml route (after the refactor to move sqlite decision out of framework) ?08:54
jamChipaca, so I've been told historically that YAML is good if a human is going to edit it, but it has enough edge cases it doesn't make a great machine<=>machine format.08:55
stubI think you will need to keep sqlite if you want Juju 2.7 compatibility08:55
jam(things like base 60 numbers)08:55
jamstub, yes, it isn't removing sqlite08:55
jamit is adding an alternative when it is available08:56
Chipacayeah i bungled and meant 'pickle' there08:56
jambut what do we want *that* to look like08:56
Chipacaframework is pickoing, storage is sqliting08:56
Chipacapickling08:56
* Chipaca used to be able to type08:56
jamChipaca, I think I'll want to keep the marshal() check, but move the pickling into Storage and out of JujuStorage and try using YAML there.08:56
jamChipaca, it kind of has to happen at the same time, since the interface to the Storage changes from taking a bytes to taking a dict08:57
jambut it shouldn't be very invasive.08:57
Chipacajam: so what _is_ a good machine ⇔ machine format?08:58
stubWith the marshal check, the yaml should always be bog simple except for sets (and maybe other builtin Python types I can't think of right now), and seems fine to me as machine <-> machine08:58
jamwell, the recommendation was JSON, but for obvious reasons that is out. I don't think pickle/python ast fall into that category. XML is overbroad08:58
jamstub, Chipaca certainly YAML is my top pick right now08:59
Chipacabase64 of a pickled xml ast dump \o/08:59
* Chipaca hides08:59
stubeval(repr(blob))08:59
Chipacaexec*08:59
Chipaca]:-)08:59
Chipacawait is exec eval now08:59
* Chipaca checks09:00
Chipacanope exec is still the right one for maximum pawnage09:00
Chipacaanyway, sorry for the silliness :)09:01
* Chipaca is only a little bit sorry09:01
Chipacahush you09:01
jamChipaca, if only there was some way to get "sudo" in there09:01
Chipacajam: charms run as root ¯\_(ツ)_/¯09:02
jamChipaca, so one nice property is that refactoring is actually quite localized09:06
jamChipaca, what is the type annotation for "snapshot". Currently we only really ever use 'dict' or None, but it has so far been anything that python can pickle09:07
jamis that just "object" ?09:08
Chipacajam: Any ?09:09
Chipacait's not strictly Any, but I don't think there's something better09:09
ChipacaOTOH maybe we should enforce it being dict-or-None09:10
Chipacaanything else isn't going to be extensible enough for long-term09:10
davigar15Chipaca: Why could my charm be in wating for container, without giving any errors?09:10
Chipaca'waiting for container' sounds like a juju thing to me :)09:11
Chipacaie not yet the charm09:11
Chipacajam: OTOH you *could* spell it all out09:13
Chipacaas in, Union[int, float, … ]09:14
jamChipaca, so we do have tests that snapshot() -> str and 'def snapshot(self): return self.value' where value was passed in to .emit(value)09:15
jam(where in the test value is only ever a string)09:15
Chipacaand it's probably overly opinionated for a framework :)09:18
Chipacai mean the 'always use a dict' thing09:18
jamChipaca, well, dict or None would be acceptable IMO, and we are little if not opinionated :)09:21
jamChipaca, if our review of people's code would always be "you really should use a dict" then we should push that a bit harder, I think.09:21
mupPR operator#320 opened: ops/framework.py: Move pickle into Storage <Created by jameinel> <https://github.com/canonical/operator/pull/320>09:22
davigar15Chipaca: happening with juju stable version... I'm going to check my charm.09:24
davigar15Can't see anything wrong... https://pastebin.canonical.com/p/BMqcdkT6fN/09:26
Chipacadavigar15: what does the debug log say09:29
davigar15juju model-config logging-config=DEBUG09:29
davigar15ERROR config value expected '=', found "DEBUG"09:29
davigar15haha09:29
jamthat exact line worked here09:30
jamyou could also do: juju model-config "logging-config=<root>=DEBUG"09:31
jamwhich is the more explicit version09:31
jam(that allows you to specify each module separately)09:31
jamdavigar15, though your particular statement "waiting for container", I believe, is created by Juju09:31
jam(if a charm sets a pod spec, and otherwise goes into Active, Juju sets the state to 'waiting for container' until the pod has started)09:32
jamdavigar15, https://discourse.juju.is/t/how-to-avoid-the-waiting-for-container-message/1369/3 might help?09:32
jamspecifically https://discourse.juju.is/t/how-to-avoid-the-waiting-for-container-message/1369/609:34
davigar15https://pastebin.canonical.com/p/CxPhPygZb4/09:35
davigar15$ microk8s.kubectl -n zk get pods09:35
davigar15NAME                             READY   STATUS    RESTARTS   AGE09:35
davigar15modeloperator-658686f765-cfsr6   1/1     Running   0          2m59s09:35
davigar15zookeeper-k8s-operator-0         1/1     Running   0          2m11s09:35
davigar15But the application pods are not even created09:35
jamdavigar15, so offhand your charm seems to be missing a self.state.set_default(spec=None) as on_config_changed introspects self.state.spec directly09:37
davigar15create Pod zookeeper-k8s-0 in StatefulSet zookeeper-k8s failed error: Pod "zookeeper-k8s-0" is invalid: spec.containers[0].ports[2].name: Invalid value: "leader-election-port": must be no more than 15 characters09:38
davigar15found09:38
davigar15haha09:38
jam(I don't think this the problem here, as it means hitting config_changed without start, which will happen with 2.8)09:38
jamdavigar15, where did you find that? I don't see it in your pastebin09:39
Chipacait's one of those no-rest-for-the-wicked days09:55
Chipacataking a breka now and getting the lunch sorted because then i'll be eating in meetings09:56
=== facundo__ is now known as facubatista
facubatistaMuy buenos días a todos!11:01
davigar15Chipaca: how can I get the number of ensured units with the op framework11:04
jamdavigar15, is that len(self.get_relation(PEER_RELATION).units)11:08
jamor do you need the count before they are actually started?11:08
davigar15good question11:09
davigar15I think I need it before11:09
jamhttps://github.com/canonical/operator/issues/165 is about possibly having the framework expose the count (reading it from goal-state), but that hasn't been implemented, so understanding your use case/adding it to that issue is useful11:16
jamdavigar15, ^^11:16
Chipacafacubatista: https://github.com/canonical/operator/releases/tag/0.6.012:05
facubatistaChipaca, wonderful12:05
Chipacafacundo__: note #294 is gtg13:12
mupPR #294: Log when a served breakpoint is not really activated because of name mismatch <Created by facundobatista> <https://github.com/canonical/operator/pull/294>13:12
facundo__Chipaca, ack13:12
* facundo__ will restart the router to see if all these disconnections stop13:13
jamChipaca, and for issue #21, if it is a script, and we set JUJU_DISPATCH_PATH it will try to reinvoke hooks/install but that will notice the OPERATOR_DISPATCH env and exit early, right?13:13
mupPR #21: Persist stored state before framework commit <Created by dshcherb> <Merged by niemeyer> <https://github.com/canonical/operator/pull/21>13:13
Chipacajam: indeed13:14
Chipacacharmcraft#21, mup m'dear13:14
mupIssue charmcraft#21: Event charm.py not defined <Created by knkski> <https://github.com/canonical/charmcraft/issues/21>13:14
Chipacafacundo__: want me to fix this ^? i can do that right now13:14
facundo__Chipaca, be my guest13:15
jamChipaca, btw, if you want easier ways to get to source code from a git submodule you can open the '.gitmodules' file which contains the URL that git will use to find the revision13:19
jamhttps://git.launchpad.net/charm-k8s-mattermost/tree/.gitmodules13:19
Chipacajam: 👍13:19
jamgithub does that for you, eg if you look at this: https://github.com/jameinel/ubuntu-lite/pull/2/files#diff-b1d90847e9ac9d08a6205ad8f09fd5a9L14 it has a direct link to the Operator code13:20
mupPR jameinel/ubuntu-lite#2: Operator <Created by jameinel> <https://github.com/jameinel/ubuntu-lite/pull/2>13:20
jamor https://github.com/jameinel/ubuntu-lite/tree/operator/mod if you are browsing the Tree rather than the PR13:21
jamChipaca, while sending the mattermost email, I realized we also really need to review https://github.com/johnsca/resource-oci-image/blob/master/oci_image.py13:23
jamsince that is as common to be reused as k8s.py13:23
Chipacaah13:24
Chipacajam: ok13:24
=== facundo__ is now known as facubatista
Chipacaclosed charmcraft#18, charmcraft#19 and charmcraft#20. I hope I was able to explain why clearly enough for the reporter not to hate us too much :-|14:32
mupIssue charmcraft#18: Poetry integration <Created by knkski> <Closed by chipaca> <https://github.com/canonical/charmcraft/issues/18>14:32
mupIssue charmcraft#19: Pep 518 support for charm metadata <Created by knkski> <Closed by chipaca> <https://github.com/canonical/charmcraft/issues/19>14:32
mupIssue charmcraft#20: Support pep 517 for building charms <Created by knkski> <Closed by chipaca> <https://github.com/canonical/charmcraft/issues/20>14:32
mupPR operator#321 opened: have a Code of Conduct <Created by chipaca> <https://github.com/canonical/operator/pull/321>14:51
Chipacafacubatista, jam, reviews on #23 would mean we could ship it today14:54
mupPR #23: Add leadership to model <Created by johnsca> <Closed by johnsca> <https://github.com/canonical/operator/pull/23>14:55
Chipacacharmcraft#2314:55
mupPR charmcraft#23: build: set JUJU_DISPATCH_PATH so we get the right hook <Created by chipaca> <https://github.com/canonical/charmcraft/pull/23>14:55
facubatistaChipaca, ack14:55
mupIssue operator#220 closed: Log in WARNING when a breakpoint is not served in debug mode <Created by facundobatista> <Closed by chipaca> <https://github.com/canonical/operator/issues/220>14:56
mupPR operator#294 closed: Log when a served breakpoint is not really activated because of name mismatch <Created by facundobatista> <Merged by chipaca> <https://github.com/canonical/operator/pull/294>14:56
mupIssue operator#262 closed: Harness should support resource-get <Created by jameinel> <Closed by chipaca> <https://github.com/canonical/operator/issues/262>14:58
mupPR operator#296 closed: Oci resources <Created by chris-sanders> <Merged by chipaca> <https://github.com/canonical/operator/pull/296>14:58
facubatistaChipaca, would you mind to explain what magic are you doing in charmcraft#23?15:08
mupPR charmcraft#23: build: set JUJU_DISPATCH_PATH so we get the right hook <Created by chipaca> <https://github.com/canonical/charmcraft/pull/23>15:08
Chipacafacubatista: i see no magic :-/15:09
Chipacafacubatista: the {{}} is because I need actual {s in the output15:09
Chipacafacubatista: WRT ${parameter:-word}    Use Default Values.  If parameter is unset or null, the expansion of word is substituted; otherwise, the value of parameter is substituted.15:10
Chipacaso ${JUJU_DISPATCH_PATH:-$0} is "either JUJU_DISPATCH_PATH or, if that is unset or null, $0"15:11
facubatistaChipaca, fantastic, thanks!!!15:13
facubatistaChipaca, mind to put a comment saying something like "if Juju don't support the dispatch mechanism, it will execute the hook, and we need sys.argv[0] to be the name of the hook, so we fake JUJU_DISPATCH_PATH to be that value"?15:14
facubatistaChipaca, thanks!15:14
Chipacawill do15:15
facubatistaChipaca, approved15:16
* facubatista -> lunch15:16
Chipacahmmmmmm15:58
* facubatista is back15:58
* Chipaca hmms15:58
Chipacadunno if my lxd is stuck, or just my juju16:01
Chipacafacubatista: do you have a working thing you could juju ssh into to try something with juju-log?16:01
facubatistaChipaca, yes, but on a meeting right now16:02
Chipacaah16:02
Chipacak16:02
* facubatista brb16:18
Chipacaok i think i'm off17:02
* Chipaca EODs17:02
* facubatista is back17:03
* facubatista eods19:44

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!