mthaddonjam: you may be interested in https://code.launchpad.net/~pjdc/charm-k8s-mattermost/+git/charm-k8s-mattermost/+merge/385953 - just looking over it now myself07:00
jammthaddon, thanks for the heads up07:22
mthaddonjam: have just asked for some updates - I noticed the test pod spec structure is a bit different from what we're actually defining in the charm07:25
jamso it actually was something different than I thought09:17
jamChipaca, https://paste.ubuntu.com/p/4RKzm4PW5h/09:17
jamso I noticed that if I did: foo: "bar: 42" | state-set --file -09:17
jamthen when I did 'state-get'09:17
jamIt would say:09:17
jamfoo: "bar: 42"09:17
jambut if I did09:17
jamfoo: |\n  bar: 42 | state-set -file -09:18
jamthen the output would be09:18
jamfoo: |09:18
jam  bar: 4209:18
jamHowever, that is because those 2 strings are subtly different, can you tell why?09:18
Chipacajam: newline?09:19
jamChipaca, yep '|' includes a trailing newline '|-' and the one-line form do not09:19
Chipacaif it wasn't that, it was cursed :-)09:20
jamChipaca, you can see the output of 'state-get foo' was 'bar: 42\n\n'09:20
jamI thought it was somehow remembering how I set the value and preserving it, which didn't really make sense09:20
Chipacabar: 12\n\n i guess?09:20
Chipacai believe preserving layout was one of the goals of gustavo's yaml.v3, but i'm not 100% positive on that (nor on whether he achieved it)09:21
Chipacai never got to play with that, actually09:21
Chipacastill on my ToDo09:21
jamChipaca, that is as go not python, though, right?09:23
jamand not a YAML spec but just a parser quirk?09:23
jamChipaca, do we think it is a problem if we treat foo='' as NoSnapshot ?09:24
Chipacaand maybe it wasn't layout but comments?09:24
jamIt is fine as long as your snapshot isn't just a string that can be empty09:24
jam(we could enforce snapshots are dicts again :)09:24
jamChipaca, or I could do the "if we get '' read the whole state-get and see whether the key exists"09:24
Chipacaso this is only a problem for a snapshot that serialises as a string, and only if it cares about the difference between a string and not being there09:25
Chipacasnapshot returns a simple serialisable object and the ones writing it out into yaml is us, right?09:26
Chipacaso what we could do, if the above is right, is !!str strings09:26
jamChipaca, actually, it isn't a problem, because we yaml.dump('') => "''\n"09:26
Chipacaand then we know because we get '' vs '!!str ""'09:26
jamyaml.load('') => None, but yaml.lod("''\n") => ''09:26
Chipacaalso aaaaah *and* ahahaha09:27
Chipacajam: cool cool, then :)09:27
jamChipaca, I'll make sure to add '' to my permutation tests09:27
jamhm. seems I also need to handle None09:31
jam>>> print(yaml.dump(None))09:31
jamyaml.safe_load('') => None, yaml.safe_load('null\n...\n') => None09:32
jamChipaca, and, of course, I'm simulating some of this via my own scripts, which is definitely stuff that I'd want to test against actual Juju09:35
Chipacajam: do we have a list of 'things we want to try in real jujus tee emm'? (should we?)09:43
jamChipaca, well, I consider that writing a 'juju test suite' as part of test-main was one of my goals of this PR09:44
Chipacafacundo__: 'charmcraft --verbose build' not being exactly the same as 'charmcraft build --verbose' is surprising and painful09:45
jambut we can split it out if we prefer09:45
Chipacajam: i hadn't realised that, is there any of this already up? (how were you approaching it?)09:46
jamChipaca, that was part of my trying to make test-main be a RealCharm09:48
jambut it isn't something I've actively started09:48
jamChipaca, my intent was to add an action to test_main, such that you could run "juju deploy test_main; juju run-action ops-tests"09:48
Chipacajam: given you can make travis install snaps, and in particular lxd snaps, i was hoping we'd be able to use that to run tests09:48
jamChipaca, snap install lxd; snap install --classic juju; juju bootstrap lxd; juju deploy; juju run-action09:49
jamChipaca, I don't expect it to be very fast (bootstrap lxd here is a minute or so)09:49
Chipacajam: i was thinking of looking at having it as a cron test rather than unit tests unless we got them to be faster than i expect them to be, though09:49
Chipacai mean, up to ~5 minutes is probably fine for PR tests09:50
Chipacalonger than that and i'd punt them to cron / manual09:50
Chipacaheh, https://github.com/juju-solutions/charms.reactive/blob/master/.travis.yml09:54
Chipacaand a job run: https://travis-ci.org/github/juju-solutions/charms.reactive/jobs/67578970009:55
Chipaca(somebody needs to tweak that travis, but it essentially seems to work?)09:55
jamChipaca, "ran for 2min43s" isn't bad09:58
jamapt install snapd pwgen is 80s of it09:59
Chipacajam: "didn't actually *run* run" kinda is tho =)09:59
jamyeah, I see that now09:59
Chipacai mean, juju bootstrap fell over09:59
jamIt is doing the usermod, but not doing the newgrp so the active process knows it is in the group09:59
Chipacahopefully it's that and not some weird apparmor-inside-apparmor-inside-apparmor thing10:00
Chipaca(travis already runs inside lxd in some situations)10:00
Chipaca(i think it's when you're doing weird arches, and then it's in vms and not containers, but still)10:00
jamChipaca, ah right, they recently switched so amd64 can be run in a container10:05
jamChipaca, the actual failure is that it can't talk to the lxd socket, but that might be because lxd failed to start?10:06
jamChipaca, do you have any quick magic for timing what is taking long in a test run?10:06
jamI've gotten some stuff to work with cProfile, etc, but it is always a bit clumsier than I would expect10:07
Chipacajam: i've monkeypatched the test constructors to track time10:07
Chipacajam: otoh pytest i think has things for that?10:07
Chipacalet me see if i still have the monkey patching thing somewhere10:08
Chipacajam: pytest --durations=10 will print the 10 slowest tests10:09
jamChipaca, yeah, pytest-profiling sems to get me where I'd like to be10:09
jam py.test test/test_storage.py::TestJujuStorage::test_emit_event --profile10:12
jamgives me a cumulative time spent10:12
Chipacajam: --durations gives you something like10:13
Chipaca2.24s call     test/test_main.py::TestMainWithNoDispatchButJujuIsDispatchAware::test_multiple_events_handled10:13
Chipaca2.21s call     test/test_main.py::TestMainWithNoDispatch::test_setup_event_links10:13
Chipacajam: also, pip install pytest-xdist  and then  pytest -n $(nproc)10:13
jamChipaca, sure. The issue is that I know one test is running slow, and I'm trying to figure out why10:14
Chipacayou want timings _inside_ the test?10:15
Chipacathat i don't have10:15
jamChipaca, yeah. Which --profiling gives me. the issue is that one of my tests with SQLite backend is about 3ms, and with Juju backend it is about 600ms10:15
jamI'd like to understand why :)10:15
Chipacajam: did you set SSH_TO_THUMPERS_BOX=0 ?10:16
Chipaca600ms is enough time to do that and more … :)10:17
Chipacajam: what happens if you set PYTHONVERBOSE in the environ?10:17
jam376ms spent in subprocess.communicate, 376ms spent in wait()10:19
jamstupid process barriers10:19
jamChipaca, PYTHONVERBOSE is a bit... verbose? :)10:20
jamChipaca, stripping it down to a single emit() shows it as 200ms, https://paste.ubuntu.com/p/XY7dmtXNxv/10:25
Chipacajam: is something reading stdin/stdout with it being buffered on a timeout?10:26
jamChipaca, it is about 20m-30ms per subprocess invocation, we just do a lot of them for each emit()10:26
jamChipaca, $ time python3 -c ''10:27
jamreal    0m0.015s10:27
Chipacajam: try -Sc10:27
jamgoes down to 7ms10:28
Chipaca(not practical irl but good to know how low we could take it with some environ handholding)10:28
jamChipaca, gives "No module named 'yaml'" if I try it in the script10:29
Chipacayeah, as i say, not practical :)10:29
Chipacait comes up with a very empty pythonpath, for one10:29
jamChipaca, it works for the ones that don't need yaml, though10:30
jambut that only shaves ~10ms off the test time10:31
Chipacajam: what are all the calls the reemit does?10:32
jamso emit() calls _emit(), which does a 'save_snapshot()' for the event, then 'save_notice()' for each observer.10:33
Chipacaand each of those is a save-set10:33
jamthen _reemit calls 'read all notices', and for each one it will then call load_snapshot and possibly drop_snapshot/drop_notice10:34
Chipacajam: right10:34
Chipacajam: sounds to me like we want to consider refactoring things to at least save in a batch10:35
Chipacabut later10:35
Chipacai mean, let's not block save-set on this :)10:35
jamso I think in practice it won't be as bad, 'juju run 'time state-get' is 5ms vs 20ms10:35
Chipacabut let's not call save-set *done* without looking at it10:35
jambut it is still a lot slower than interacting with an in-process SQLite file10:36
* Chipaca reaches for his 'shocked pikachu' face again10:36
jamChipaca, 😲10:37
Chipaca:-) that's the on10:37
Chipacanah that's the dog :-p10:37
jamChipaca, I do think given how much the Operator framework is based around events, we'll want to be cautious about its performance10:38
Chipaca(somebody pointed it 'the scream' kinda looks like the guy tried to draw a dog and then somebody said 'oh wow a person screaming' and they went with it)10:38
jamhttps://emojipedia.org/face-screaming-in-fear/ I'm not seeing the dog10:38
jamnor here: https://www.bookdepository.com/Edvard-Munch-Masterpieces-of-Art-Candice-Russell/9781783613564?redirected=true&utm_medium=Google&utm_campaign=Base3&utm_source=AE&utm_content=Edvard-Munch-Masterpieces-of-Art&selectCurrency=AED&w=AFCFAU968L892MA8VCPZ&gclid=Cj0KCQjwoaz3BRDnARIsAF1RfLeKR4-ljbzhpZgajGSi2f4h-DLQp_FaX-dDHKqVeiWmSi23iue6HVAaAlf2EALw_wcB10:38
jamChipaca, ah, the hands are the ears10:39
Chipacathere you go :)10:39
Chipacaonce you see it it's easy to flip-flop10:39
jamif you get rid of the nose and mouth, then I can see the dog nose, but I don't see how they fit on the dog face10:40
jamnot as good a flip/flop as https://www.independent.co.uk/news/science/duck-or-rabbit-the-100-year-old-optical-illusion-that-tells-you-how-creative-you-are-a6873106.html10:40
jamduck bunny10:40
Chipacathe duck/bunny one is very good (i first saw it on a joke)10:41
Chipacaanyway. pop.10:42
Chipacafacundo__: so, wrt charmcraft and secrets, i think a reasonable approach is to use keyring and have the snap ask for the secrets plug; it won't auto-connect, but we can detect the failure in code and fall back to plaintext (or notify the user to manually connect or pass --plaintext?)10:44
Chipacathe token isn't a general-purpose do-anything token fwiw, which is why storing it plaintext isn't _that_ bad an idea10:45
jam'time python3 -c import yaml' is 30ms here.. :(10:49
Chipaca0.05user 0.00system 0:00.06elapsed 98%CPU (0avgtext+0avgdata 11216maxresident)k10:50
jam50-60ms ?10:51
Chipacaand 11MB10:52
facundo__Muy buenos días a todos!10:54
=== facundo__ is now known as facubatista
Chipacajam: of course that's slower in great part because my cpu aggressively freqs down; if i pin it hot it drops to 30ms10:55
Chipacaand if i were to enable 'turbo' mode it'd probably drop another ~5ms10:55
Chipacabut i don't like my fans being on loud all the time =)10:55
Chipacafacubatista: muyy buen día su señoría mantantirulirulá10:56
facubatistaChipaca, supercalifragilísticoespialidosas mañanas!10:57
jamChipaca, thinking about Ajduk... It came to light that Operators can trigger an action to be run, which means the Operator charm *should* be able to cause a script to run in the application container11:26
jamChipaca, https://discourse.juju.is/t/coordinating-actions-for-a-k8s-operators/3161/211:27
jamChipaca, I added a comment to the doc11:30
facubatistaChipaca, https://github.com/canonical/charmcraft/issues/4211:31
jamChipaca, facubatista : all wired up! turns out we aren't very good at cleaning up after ourselves:11:41
jam$ juju run --unit uo/2 'state-get'11:41
jam'#notices#': |11:41
jam  []11:41
jamStoredStateData[_stored]: |11:41
jam  {event_count: 12}11:41
jamUbuntu/on/config_changed[7]: |11:41
jam  null11:41
jam  ...11:41
jamUbuntu/on/install[1]: |11:41
jam  null11:41
jam  ...11:41
jamUbuntu/on/leader_elected[4]: |11:41
jam  null11:41
jam  ...11:41
jamon/commit[3]: |11:41
jam  null11:41
jam  ...11:41
jamon/commit[6]: |11:41
jam  null11:41
jam  ...11:41
jamon/commit[9]: |11:41
jam  null11:41
jam  ...11:41
jamon/commit[12]: |11:41
jam  null11:41
jam  ...11:42
jamon/pre_commit[2]: |11:42
jam  null11:42
jam  ...11:42
jamon/pre_commit[5]: |11:42
jam  null11:42
jam  ...11:42
jamon/pre_commit[8]: |11:42
jam  null11:42
jam  ...11:42
jamon/pre_commit[11]: |11:42
jam  null11:42
jam(didn't mean to paste the content here)11:42
Chipacai'll be 10 minutes late for the revue11:51
Chipaca(small lunch chaos here)11:51
jamChipaca, facubatista so the bug is that events that have no handlers are always treated as deferred11:51
* facubatista plans the meeting for 10' past normal time11:55
Chipacajam: that's a nasty bug :-|12:12
facubatistaChipaca, https://discourse.juju.is/t/how-to-build-a-charm-using-modern-tools/324612:13
mupIssue operator#333 opened: Events with no observers never get deleted <Created by jameinel> <https://github.com/canonical/operator/issues/333>12:23
Chipacawhat's the generic name for what 'juju deploy' deploys to?13:07
facubatistaChipaca, the "backing cloud"?13:13
Chipacafacubatista: hmm13:14
Chipacafacubatista: in the end i rewrote the sentence to not need it, fwiw13:14
Chipacafacubatista: https://discourse.juju.is/t/what-files-would-you-expect-charmcraft-build-copies-into-the-charm/324713:14
Chipacastub: mthaddon: ^ if you have opinions on it please speak up :)13:14
mthaddonack, will try and take a look soon13:38
mupPR operator#334 opened: Unobserved events <Created by jameinel> <https://github.com/canonical/operator/pull/334>14:14
jamChipaca, PRs should be up for review14:15
jamfacubatista, review of charmcraft is up14:15
facubatistajam, thanks14:15
Chipacajam: which are the python implementations of state-set that are slow? where can i play with that?14:15
Chipacas/set/[sg]et/ fwiw14:17
facubatistajam, what do you mean with "make it contextual"?14:18
Chipacafacubatista: pass it around either explicitly or implicity, i think14:19
facubatistait could be done14:23
facubatistaI like it how it's now, though, as it should be an omnipresent object, the only interaction with the module, etc14:24
jamChipaca, https://github.com/canonical/operator/pull/323/files#diff-54d96dff01765335f1f08f6254aa0de6R197-R250 is the python backend15:08
mupPR #323: 317 state get <Created by jameinel> <https://github.com/canonical/operator/pull/323>15:08
jamfacubatista, yeah, the idea was 'is it worth passing one around' rather than being used as a process global15:08
Chipacaheh, silly me was looking for #! :)15:14
jamChipaca, yeah, the #! is fixed to be bash by fake_script15:40
mupPR operator#335 opened: make tests 15% faster <Created by chipaca> <https://github.com/canonical/operator/pull/335>16:03
Chipacatomorrow is RTD DAY \o/16:33
Chipacaok, i think i'm at eod, mostly16:57
Chipacafacubatista: i'm still hoping to finish the review on charmcraft#33 tonight though16:58
mupPR charmcraft#33: Refactored all the message sending to the user, including storing everything in a file <Created by facundobatista> <https://github.com/canonical/charmcraft/pull/33>16:58
Chipacabut that'll depend on external factors :)16:58
facubatistaChipaca, tomorrow is fine, too17:00
mupPR operator#335 closed: make tests 15% faster <Created by chipaca> <Merged by chipaca> <https://github.com/canonical/operator/pull/335>20:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!