[00:42] <mup> PR snapd#6618 opened: tests: validates snapd from ppa <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6618>
[05:58] <zyga> Good morning
[06:33] <mborzecki> morning
[06:47] <mborzecki> mvo: morning
[06:47] <mvo> mborzecki: hey, good morning
[07:43] <zyga> hey mborzecki, mvo
[07:43] <mborzecki> zyga: hey
[07:43] <zyga> kids off to school, time to work
[07:45] <zyga> I took sime time to clean my office yesterday
[07:45] <zyga> I must say, it feels great to work in a tidy place
[07:45] <zyga> I will try to preserve this condition better than last time
[07:46] <zyga> I have a few small PRs that could land quickly, if anyone wants to see
[07:47] <mborzecki> zyga: which ones?
[07:47] <zyga> mborzecki: typedef struct mountinfo is trivial
[07:48] <zyga> mborzecki: UMOUNT_NOFOLLOW is super trivial
[07:48] <zyga> those two can just land
[07:48] <zyga> mborzecki: there are some short but not trivial branches like fixed private /tmp
[07:48] <zyga> or fixes to mountinfo parsing
[07:58] <mvo> hey zyga
[07:58] <zyga> good morning :)
[07:58] <zyga> how is your daughter today?
[07:59] <mvo> zyga: all well
[07:59] <mvo> zyga: and yours :) ?
[08:00] <zyga> she was very hungry today, eating a few times in a row
[08:00] <zyga> the older one went to school recently :)
[08:00] <zyga> (need to remember about both)
[08:03] <pstolowski> mornings!
[08:04] <mborzecki> pstolowski: hey
[08:04] <mvo> good morning pstolowski
[08:14] <mborzecki> zyga: do you need a review from pedronis on https://github.com/snapcore/snapd/pull/6608  and the mountinfo one?
[08:14] <mup> PR #6608: cmd/snap-confine: umount scratch dir using UMOUNT_NOFOLLOW <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6608>
[08:14] <zyga> I will wait for a pass, yes
[08:17] <dot-tobias> good morning
[08:30] <zyga> hey dot-tobias
[08:32] <dot-tobias> zyga: Morning 😊 BTW, thank you again for the layouts feature, made both of my snaps possible in the first place! Huge improvement for snapping otherwise “unconfinable” software IMHO
[08:52] <pedronis> 6608 can land
[09:05] <Chipaca> man i wish each log line said what machine it was from :-(
[09:05] <Chipaca> morning all
[09:08] <mborzecki> Chipaca: wouldn't be suprised everyone has a patch for spread to do just that :P i had one
[09:08] <zyga> pedronis: hello
[09:08] <zyga> pedronis: I updated https://github.com/snapcore/snapd/pull/6502
[09:08] <mup> PR #6502: dirs,overlord/snapstate: add Soft and Hard refresh checks <Created by zyga> <https://github.com/snapcore/snapd/pull/6502>
[09:08] <zyga> dot-tobias: I'm very glad to hear that :-)
[09:09] <zyga> thank you :)
[09:09] <zyga> Chipaca: good morning
[09:09] <zyga> pedronis: thank you, merged now
[09:09] <mup> PR snapd#6608 closed: cmd/snap-confine: umount scratch dir using UMOUNT_NOFOLLOW <Reviewed> <Simple 😃> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6608>
[09:15] <mvo> zyga: hey, when you have a spare slot, could you please re-review 6491 - aiui it "just" needs a re-review at this point
[09:15] <mvo> pstolowski: (or is anything missing in 6491)
[09:18] <pstolowski> mvo: no, it needs 2nd review
[09:24] <pstolowski> Chipaca: hey, can you cast your eye on https://github.com/snapcore/snapd/pull/6617 ?
[09:25] <mup> PR #6617: cmd/snap: fix regression of snap saved command <⚠ Critical> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6617>
[09:26] <zyga> mvo: sure, doing so now
[09:26] <Chipaca> ah, yes
[09:27] <mvo> zyga: thank you!
[09:32] <Chipaca> kjackal: ping
[09:32] <kjackal> hey Chipaca
[09:32] <Chipaca> kjackal: hey
[09:32] <Chipaca> kjackal: which 16.04 image from ovh doesn't support squashfs?
[09:33] <kjackal> Let me see what that person said, give me a sec
[09:34] <kjackal> Chipaca: unfortunately he did not came back to answer your questions: https://github.com/ubuntu/microk8s/issues/362
[09:35] <kjackal> I did point him to the discourse topic
[09:37] <Chipaca> kjackal: thanks
[09:37] <Chipaca> kjackal: I've raised the issue internally as well
[09:37] <Chipaca> if it doesn't support squashfs it's running some random kernel, meaning it's not Ubuntu
[09:37] <kjackal> does the ovh get certified ubuntu images from us?
[09:38] <Chipaca> they've been in trouble with canonical before for selling something as Ubuntu and it not being Ubuntu
[09:38] <Chipaca> but I don't know how that got resolved
[09:38] <Chipaca> if it did, even
[09:39] <pstolowski> grrr the testing situation is really annoying
[09:40] <zyga> can we all stop the line and fix tests
[09:41] <zyga> including perhaps spending a moment to ensure that the situation is really better, not just not broken again
[09:42] <Chipaca> zyga: what's broken now?
[09:42] <zyga> Chipaca: is the situation with /var/snap fixed now?
[09:42] <Chipaca> zyga: no
[09:42] <zyga> so that's enough ;)
[09:42] <Chipaca> i'm still working on it
[09:42] <zyga> tests cannot be red for many days
[09:42] <Chipaca> i'm still working on it
[09:42] <zyga> I understand, I'm saying we should all jump at fixing the fundamental issue
[09:43] <Chipaca> which is the fundamental issue?
[09:43] <zyga> that is the one-to-all influence of every test
[09:43] <zyga> any test can break any other test
[09:43] <zyga> we do insufficinent work in making sure the environment is pristine
[09:43] <zyga> and the definition of pristine is not strict
[09:43] <zyga> it's all reactionary
[09:43] <Chipaca> quick fix: set workers to the number of tests
[09:45] <zyga> I'm partially not joking, it's just impractical to develop snapd for several years and constantly be in a situation where nearly every PR needs to be restarted
[09:45] <zyga> github is down, fine, happens once a year
[09:45] <zyga> tests don't clean up: *not* fine
[09:45] <zyga> we should really fix this
[09:46] <Chipaca> i'm not disagreeing, but
[09:46] <Chipaca> this issue is something else
[09:47] <Chipaca> 'snap changes' indicates that snapd _did_ remove the snap
[09:47] <Chipaca> so there's an actual bug somewhere
[09:47] <zyga> that's okay, but the first test that experiences this should stop and fail,
[09:48] <zyga> and not influence something entirely different later
[09:48] <zyga> once you fix the problem with current we will still have the very same unreliability problem with tests
[09:49] <Chipaca> what do you mean by "experiences"
[09:50] <zyga> a test that, by the end of the execution, due to the bug or otherwise leaves the test environment (system) in a state different from the state it was in at the beginning of the test
[09:51] <zyga> my impression is that right now our tests feel like DOS software
[09:51] <zyga> one program runs and corrupts something
[09:51] <zyga> them sometime later another program dies because of that corruption
[09:51] <zyga> it's very time-consuming to debug
[09:51] <zyga> we should spend the time in a smarter way
[09:52] <mborzecki> can i get some eyes on https://github.com/snapcore/snapd/pull/6609 ?
[09:53] <mup> PR #6609: snap/gadget: introduce volume update info <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6609>
[09:53] <Chipaca> so, we have the snapd-state code
[09:53] <zyga> mborzecki: enqueued
[09:53] <mborzecki> zyga: thanks
[09:53] <Chipaca> this problem would go away, i expect, if /var/snap were also part of the things that got nuked every iteration
[09:54] <Chipaca> OTOH, maybe the restore code should fail if there's things that shouldn't be there
[09:54] <Chipaca> so then the tests _need_ to clean up after themselves
[09:54] <zyga> perhaps the assumption that there's a magic cleanup system was flawed
[09:54] <Chipaca> so then we are testing that things are clean after cleaning up
[09:54] <mborzecki> Chipaca: wonder if that may be the package post/pre rm cleanup code
[09:54] <zyga> would tests be really that much longer if install_local enqueued a restore action that removes the test
[09:54] <zyga> and the restore code would just check that nothing is left behind (and fail if there was)
[09:54] <mborzecki> though it's only happening on ubuntu-core, hmm
[09:55] <Chipaca> mborzecki: what's only happening on ubuntu-core? these failures you mean?
[09:55] <zyga> mborzecki: interesting
[09:55] <zyga> on core we get extra snaps for testing
[09:55] <mborzecki> Chipaca: yeah, i recall seeing only ubuntu-core there
[09:55] <zyga> anyway, I'm grumpy because I see us happily igoring the ephant in the room
[09:55] <Chipaca> hmmm!
[09:55] <zyga> our testing infrastructure needs love
[09:55] <Chipaca> mborzecki: i hadn't noticed :-/
[09:55] <zyga> it's not healthy
[09:55] <Chipaca> if it's that, maybe it's silly
[09:56] <Chipaca> because is_snapd_state_saved behaves differently on core
[09:56] <mborzecki> heh
[09:56] <Chipaca> save_snapd_state behaves differently on core too
[09:56] <mborzecki> Chipaca: yup, looking at 3 logs atm, all ubuntu-core
[09:57] <Chipaca> i thought i'd seen a non-core one, but can't find it now
[09:57] <Chipaca> so it might've been unrelated
[09:57] <Chipaca> the whole 'snap state file' thing does not exist on core
[09:57] <mborzecki> and we dont uninstall the package on ubuntu-core, because how we'd do it?
[09:57] <Chipaca> afaict
[09:58] <Chipaca> mborzecki: we don't do that either, in the save state thing -- are you saying we do that each loop?
[09:59] <mborzecki> Chipaca: my bad, we don't
[10:00] <Chipaca> mborzecki: might just be coincidence tho
[10:01] <Chipaca> hm
[10:01] <Chipaca> mborzecki: in save_snapd_state, when it's core, we copy /var/lib/snapd
[10:02] <Chipaca> ah and then restore_snapd_lib
[10:02] <Chipaca> ok fair
[10:04] <Chipaca> eep! finally got a shell in a failed one!
[10:04]  * Chipaca runs
[10:05] <mborzecki> hahah
[10:09] <mborzecki> Chipaca: hm, but we dont restore snapd state for each test
[10:09] <Chipaca> mborzecki: in core we do
[10:09] <Chipaca> mborzecki: reset_all_snap does that
[10:09] <Chipaca> and this might be the problem?
[10:10] <Chipaca> or at least i think we do
[10:10] <Chipaca> i mean, that's the code that fails
[10:11] <Chipaca> reset.sh calls reset_all_snap on load
[10:11] <Chipaca> so everything that sources it will do that
[10:11] <Chipaca> (on core)
[10:11] <Chipaca> (on classic, reset_classic)
[10:12] <Chipaca> so prepare_suite_each and restore_suite both do that
[10:16] <pedronis> we should restore the state for each test,  lots of things assume that
[10:18] <mborzecki> Chipaca: ok, then reset_classic does rm -rvf /var/snap, reset_all_snap doesnot
[10:19] <mborzecki> it calls snap remove which sould do the trick
[10:19] <Chipaca> pedronis: the -each versions do run for each test
[10:19] <Chipaca> mborzecki: indeed
[10:19] <mborzecki> maybe we should have a check there to verify that /var/snap does not have any extra data?
[10:21] <Chipaca> mborzecki: we kinda do :-)
[10:21]  * zyga wonders if we will ever return to the spread measurements idea that he proposed about a year ago
[10:21] <Chipaca> i mean, that's why it's dying
[10:21] <zyga> well, more than a year now
[10:24] <Chipaca> zyga: ?
[10:25] <zyga> Chipaca: I proposed a spread extension where we can add a "measure: " section; anything printed there is stored compared with before/after for each level of nesting
[10:25] <zyga> Chipaca: so if you print the set of installed packages
[10:25] <zyga> you will notice leaked packages
[10:25] <zyga> if you print the state of systemd services
[10:25] <zyga> you will see systemd services changing state across tests
[10:25] <zyga> we can easily measure anything we want
[10:26] <zyga> I iterated on this idea
[10:26] <zyga> and while doing so found a bunch of bugs in our test setup then
[10:34] <Chipaca> zyga: and what happened?
[10:36] <zyga> I think not enough eyes looking or agreeing with the gravity of the problem
[10:38] <zyga> Chipaca: I think that we need something that would alert us of a test doing unexpected system change
[10:38] <zyga> not just trying to fix them
[10:38] <zyga> fixing is costly (we untar / remove all the time)
[10:38] <zyga> checking is presumably cheaper
[10:39] <zyga> so this might also speed up things in the end
[10:41] <pedronis> it was also mixed with a change to spread which come with friction
[10:41] <zyga> pedronis: we could replicate that without spread but arguably the issue is that spread is too open and doesn't give us the ability to tame that
[10:41] <zyga> pedronis: perhaps that's the way forward?
[10:42] <pedronis> anyway "flaky tests" and "test invariant" are still part of my open problems list
[10:43] <zyga> I would argue we should solve that with priorty because it hinders any other development done by the team
[10:51] <pedronis> let's look at this in the standup (cc mvo)
[10:59] <zyga> gladly!
[11:04] <zyga> brb
[11:06] <mup> PR snapd#6619 opened: partition,bootloader: rename 'partition' package to 'bootloader' <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6619>
[11:29] <zyga> pstolowski: https://github.com/snapcore/snapd/pull/6491 reviewed
[11:29] <mup> PR #6491: interfaces: hotplug nested vm test, updated serial-port interface <Hotplug 🔌> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6491>
[11:30] <zyga> mborzecki: looking at https://github.com/snapcore/snapd/pull/6609
[11:30] <mup> PR #6609: snap/gadget: introduce volume update info <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6609>
[11:30] <pstolowski> zyga: thanks!
[11:32] <zyga> mborzecki: https://github.com/snapcore/snapd/pull/6609 done
[11:32] <mup> PR #6609: snap/gadget: introduce volume update info <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6609>
[11:32] <zyga> pstolowski: I think that some of my questions got answers by the time I read to the bottom but I chose not to remove them.
[11:33] <zyga> pstolowski: it's safer and easier to ask and just get a quick answer
[11:33] <pstolowski> zyga: sure, makes sense
[11:36]  * zyga goes for a break
[11:48] <mborzecki> Chipaca: have you found anything interesting in the debug shell?
[11:49] <Chipaca> mborzecki: nothing as yet, but i'm retrying with more debug
[12:03] <Chipaca> hm
[12:03] <Chipaca> mvo: 'snap remodel' doesn't print a last \n
[12:04] <Chipaca> HAH! found the bugger
[12:04]  * Chipaca notes that is "the thing that causes the bug", not any other meaning of the word
[12:05] <Chipaca> mborzecki: the 'snap remodel' spread test leaves things in a confused state
[12:06] <mborzecki> 'confused'
[12:06] <pedronis> well, it's a remodel
[12:06] <pedronis> :/
[12:06] <mborzecki> Chipaca: installed snap disappeared from the state?
[12:07] <Chipaca> well, it's hard to call it 'disappear' when the restore of the test overwrites state.json
[12:09] <Chipaca> pedronis: it isn't the remodel per se; the restore of the test just was untidy
[12:09] <Chipaca> i now need to test the fix a few times
[12:10] <Chipaca> pedronis: we can remove required-snaps. Are we supposed to be able to remove required-snaps?
[12:10] <pedronis> Chipaca: no, it's a bug in remodel, I actually discussed that mvo this morning
[12:10] <pedronis> *with mvo
[12:10] <pedronis> Chipaca: it's not setting the flag properly
[12:11] <Chipaca> grr
[12:11] <Chipaca> then this fix gets a lot harder
[12:18] <Chipaca> pedronis: can you remodel 'back'?
[12:18] <mborzecki> do we have a helper to parse size information that has format like 100M ?
[12:18] <pedronis> Chipaca: yes
[12:18] <Chipaca> mborzecki: yes, strutil
[12:18] <mborzecki> Chipaca: ta!
[12:18] <pedronis> it doesn't remove things
[12:18] <Chipaca> mborzecki: strutil.ParseByteSize
[12:18] <pedronis> but under correct behavior
[12:18] <Chipaca> mborzecki: it's ISO tho
[12:18] <pedronis> it would unset the required flag
[12:19] <sergiusens> Chipaca: hey, do you know in what snapd version will the """download""" local snaps feature show up in?
[12:19] <Chipaca> pedronis: so I can remodel back, and snap remove, and it'd be happy
[12:19] <pedronis> Chipaca: yes
[12:19] <pedronis> assuming no bugs :)
[12:19] <Chipaca> sergiusens: 2.38 afaik
[12:19] <pedronis> but we are interesetd to know if we have some
[12:19] <Chipaca> sergiusens: but let me confirm
[12:19] <Chipaca> sergiusens: this reminds me we still need to talk about the commandline options (not this week tho)
[12:20] <Chipaca> sergiusens: maybe 2.39 :-(
[12:20] <sergiusens> Chipaca: also, can we consider having a `snap wait-for-ready` command for easier scripting, and until that is done, what is the most bullet proof way to write my own? (e.g.; for multipass we run `multipass version` and wait for the `multipassd` version to show up)
[12:20] <Chipaca> sergiusens: what is wait-for-ready
[12:21] <sergiusens> Chipaca: block until snapd is ready
[12:21] <pedronis> ready in which sense?
[12:21] <Chipaca> sergiusens: what pedronis said
[12:21] <Chipaca> sergiusens: snap wait system seed.loaded <- that's one version of 'wait for ready'
[12:22] <sergiusens> ready to be able to carry over "snap install" commands through the socket (no need to consider networking)
[12:23] <pedronis> then what John showed is mostly that
[12:23] <sergiusens> to the minimum, the socket existing and taking connections
[12:23] <sergiusens> great
[12:35] <Chipaca> sergiusens: I should probably ask mvo every time somebody asks me about versions-to-features
[12:35] <Chipaca> sergiusens: 2.38 isn't cut yet, so 2.38 will have the download api
[12:36] <sergiusens> thanks Chipaca, that will be a pleasure 🙂
[12:36] <Chipaca> sergiusens: you saw what its final form was?
[12:36]  * pstolowski lunch
[12:37] <sergiusens> Chipaca: is that to counter my "pleasure" comment? Should I be afraid?
[12:37] <Chipaca> sergiusens: GET ..../v2/snaps/<snapname>/file
[12:37] <sergiusens> Chipaca: I did not get to see it though, no... did it change much from what we discussed
[12:37] <sergiusens> ah, that sounds usable
[12:37] <Chipaca> :-)
[12:38] <Chipaca> sergiusens: when you saw it it was the same, but .../blob
[12:38]  * Chipaca <- really good at bad names
[12:38] <sergiusens> I remember the tusks and husks and all that 😉
[12:39] <Chipaca> it's all been downhill since candirú
[12:39] <Chipaca> cerati/candirú/curucú
[12:45] <Chipaca> popey: BTW, Feb 11 17:07:15 hal snapd[14673]: desktop.go:129: cannot use line "Exec=/usr/share/code-insiders/code-insiders --new-window %F" for desktop file "/var/lib/snapd/desktop/applications/code-insiders_code-insiders.desktop" (snap code-insiders)
[12:45] <Chipaca> maybe something to raise with code-insiders
[12:47] <Chipaca> popey: and, as feared, Mar 19 12:31:02 hal snapd[3271]: main.go:141: DEBUG: activation done in 21.615s
[12:47] <Chipaca> that's one slow activation
[12:49] <mup> PR snapcraft#2505 closed: Test store <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2505>
[12:53] <zyga> Chipaca: question to you sir
[12:53] <zyga> Chipaca: when is a refresh done
[12:53] <Chipaca> zyga: it was like that when I got there!
[12:53] <zyga> conceptually
[12:53] <zyga> hah :)
[12:54] <zyga> I would like to reset a new time stored in the state when a refresh is ready
[12:54] <Chipaca> zyga: I did not understand that last statement
[12:54] <Chipaca> zyga: but
[12:54] <Chipaca> zyga: please explain
[12:54] <zyga> Chipaca: when refresh is done, when a part of a change that refreshes a particular snap is deemed complete, I'd like to perfrom some operations
[12:55] <zyga> Chipaca: to account for that in the state
[12:55] <zyga> Chipaca: should that be link-snap/
[12:55] <zyga> Chipaca: or start services?
[12:55] <zyga> or a new task?
[12:55] <zyga> this may also be relevant for health checks
[12:55] <Chipaca> pedronis: you _can't_ set a revision 'back' :-/
[12:55] <zyga> Chipaca: do you know CLOS?
[12:56] <pedronis> Chipaca: no, you can make a new one though, that is like the first
[12:56] <Chipaca> pedronis: but i don't have the key for that
[12:56] <Chipaca> pedronis: dunno who does
[12:56] <Chipaca> pedronis: do you?
[12:56] <pedronis> Chipaca: it's the test key
[12:56] <Chipaca> zyga: no i do not know clos
[12:56] <pedronis> it's in the code
[12:56] <pedronis> but let's talk at the standup about this at this point
[12:57] <zyga> (defmethod refresh :after ((s snap)) (setf (postponed-refresh-time (state-of-snap s)) 0))
[12:57] <Chipaca> :-( ok
[12:57] <zyga> something along those lines
[12:57]  * Chipaca kickbans zyga
[12:57] <zyga> hehe
[12:58] <Chipaca> zyga: CLOs are collateralized loan obligations, the internet tells me
[12:58] <zyga> https://lispcookbook.github.io/cl-cookbook/clos.html#method-qualifiers-before-after-around
[12:58]  * pedronis eyes his copy of "The Art of The Metaobject Protocol"
[12:58]  * Chipaca writes back to the kind person from google
[12:58] <Chipaca> well, recruiter, 'person' might be a stretch
[12:59] <Chipaca> I think the answer to "thinks are rather complex and starting to fail in ways that escape us" is not "let's make it more complex"
[12:59] <zyga> all things evolve till the contain an implementation of lisp, that sort of thing ;)
[12:59] <Chipaca> things*
[13:02] <Chipaca> pedronis: I mean, I could push a fix that just does 'snap remove', which will then fail when it gets fixed
[13:03] <pedronis> Chipaca: that's fine I think
[13:03] <zyga> Chipaca: so.... where would I put something like that?
[13:03] <pedronis> it will fail in a clear way
[13:03] <pedronis> (at least)
[13:03] <Chipaca> zyga: nowhere
[13:04] <pedronis> zyga: it depends what you need to know?  link-snap is the easiest place if it's good enough
[13:05] <zyga> pedronis: yeah, I think link-snap is OK for now
[13:05] <zyga> pedronis: conceptually where post-refresh hooks run probably
[13:05] <zyga> s/where/when/
[13:05] <pedronis> we have rerefresh now , so you have options if you need something else
[13:05] <pedronis> but link-snap would be easier
[13:05] <pedronis> if it's enough
[13:05] <mborzecki> off to pick up the kids
[13:06] <zyga> I think it's enough for now
[13:06] <zyga> thanks!
[13:06] <zyga> it will all be a PR soon
[13:07]  * Chipaca honestly didn't understand
[13:07]  * Chipaca will look at the pr
[13:08] <mvo> Chipaca: reading backlog - verison-to-feature - for some we have this in the forum. for all the "important" ones. remodel test - I miss some details, how can it leave things in state when we restore state.json from a backup?
[13:08] <Chipaca> mvo: remodel test, fixing it
[13:08] <Chipaca> mvo: with comments about how to fix it again when you fix remodel and thus break the test fixer
[13:09] <Chipaca> mvo: i'll push as soon as it's green here
[13:11] <mvo> Chipaca: woah
[13:11] <mvo> Chipaca: looking forward to that
[13:11] <Chipaca> ¯\_(ツ)_/¯ it's not lisp
[13:11] <mvo> Chipaca: really curious what I'm missing
[13:11] <Chipaca> =)
[13:13] <roadmr> lots of incredibly silly parentheses
[13:13] <mvo> Chipaca: at least this part I understand
[13:13]  * mvo is very proud about this 
[13:25]  * zyga is unsure about the LISP comments
[13:33] <Chipaca> mvo: vvv
[13:33] <mup> PR snapd#6620 opened: tests/main/remodel: clean up before reverting the state <Created by chipaca> <https://github.com/snapcore/snapd/pull/6620>
[13:33] <Chipaca> mvo: ^^^
[13:34] <mup> PR snapd#6621 opened: overlord/snapstate: track time of postponed refreshes <Created by zyga> <https://github.com/snapcore/snapd/pull/6621>
[13:36] <mvo> Chipaca: looking
[13:37] <mvo> Chipaca: thanks! do we actually keep state.json between when we do a restroe for the next test?
[13:37] <mvo> Chipaca: what I'm missing is why this makes the state untidy
[13:37] <Chipaca> mvo: this test itself does
[13:37] <Chipaca> this test is rather piggy
[13:38] <Chipaca> on classic, where we restore the state by hand, it doesn't break (because that restoring cleans things up also)
[13:38] <mvo> Chipaca: oh, so this restore runs *after* the full restore?
[13:38] <Chipaca> on core we use snapd to remove stuff, and that fails because of this
[13:38] <mvo> Chipaca: aha
[13:38] <Chipaca> so
[13:38] <Chipaca> it's like this:
[13:38] <Chipaca> this test restores the state, but does not clean up /var/snap
[13:39] <mvo> Chipaca: it seems on core we also want to restore the previous state.json. it sounds dangerous what we do right now
[13:39] <Chipaca> the next test that installs test-snapd-tools, fails on cleanup
[13:39] <mvo> Chipaca: ohhhhh
[13:39] <mvo> Chipaca: now I get it
[13:39] <Chipaca> so it might not be the immediate next test
[13:39] <Chipaca> it would have to install test-snapd-tools first :-)
[13:39] <Chipaca> anyway, that
[13:40] <Chipaca> mvo: on the contrary i'd be happier if restoring state.json wasn't needed (unless we were snapshotting the whole machine)
[13:40] <Chipaca> because we _should_ be able to clean back to a known-good state
[13:40] <Chipaca> but, alas, sometimes we don't
[13:40] <Chipaca> so i dunno
[13:40] <mvo> Chipaca: right, except that some tests use "jq" to do things
[13:40] <mvo> Chipaca: which might make a mess
[13:40] <Chipaca> yeah
[13:40] <mvo> Chipaca: but yeah, I guess even then we should write good "jq"
[13:40] <Chipaca> mvo: those do restore it tho
[13:41] <Chipaca> at least, i grepped and it looked like they did
[13:41] <Chipaca> i wasn't too thorough because i'm tired of red tests
[13:41] <Chipaca> and the jq-using tests are all old :)
[13:42] <mvo> Chipaca: thanks for finding this one, you get a hero medal for this one (https://www.google.com/imgres?imgurl=https://vignette.wikia.nocookie.net/wreckitralph/images/1/1d/RalphHeroesDutyMedal.png/revision/latest?cb%3D20130516213303&imgrefurl=https://wreckitralph.fandom.com/wiki/Medal_of_Heroes&docid=OoNm4g5X82ygWM&tbnid=4S4dCW2qcULhzM:&vet=1&w=783&h=446&source=sh/x/im)
[13:42] <zyga> mvo: quick trivial for your consideration https://github.com/snapcore/snapd/pull/6622
[13:42] <mup> PR #6622: cmd/libsnap: rename C enum for feature flag <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6622>
[13:42] <mvo> zyga: thanks
[13:42] <mup> PR snapd#6622 opened: cmd/libsnap: rename C enum for feature flag <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6622>
[13:47] <zyga> hmm, comparing time is apparently harder than I hoped
[13:52] <zyga> brb, see you at standup
[13:54] <popey> Chipaca: sorry, i don't know what that means, whether it's good or bad (the DEBUG thing)
[13:55] <Chipaca> popey: I'll be getting you a snapd with moar debug, if you don't mind
[13:55] <popey> hah
[13:55] <Chipaca> popey: because your snapd is taking 21 seconds to do … nothing? :-)
[13:55] <popey> sweet
[13:55] <Chipaca> not nothing, but all very quick things
[13:55] <Chipaca> so we're missing something :-)
[13:56] <Chipaca> popey: i've got a few things on my plate before that tho
[13:56] <Chipaca> so maybe tomorrow
[13:56] <popey> kk
[13:56] <popey> np
[13:56] <Chipaca> any more things on my plate and the carrots will start touching the fish
[13:57] <popey> Create a breakwater with some potatoes
[13:57] <Chipaca> the potatos are at gravy carrying capacity
[13:57] <Chipaca> as is law
[13:58] <pedronis> Chipaca: we initialize the interface manager, that might have slow bits
[14:03] <zyga> pedronis: perhaps we should inject a change (even a fake one) that accumulates the time of the system-key regeneration cost
[14:03] <zyga> pedronis: it would help with accountability
[14:03] <pedronis> well, the new infra can measure
[14:03] <pedronis> out of changes too
[14:04] <zyga> does the new infra measure non-task cost like this? if so that's great
[14:18] <sergiusens> Chipaca: if I do not wait by other means, I get `error: cannot communicate with server: Get http://localhost/v2/snaps/system/conf?keys=seed.loaded: dial unix /run/snapd.socket: connect: no such file or directory`
[14:20] <sergiusens> should I just loop until that call works? I want to avoid using anything internal that is subject to change
[14:34] <Chipaca> sergiusens: that's strange, because multi-user.target and cloud-final.target both wait for that
[14:34] <Chipaca> sergiusens: how are you getting 'in'?
[14:35] <sergiusens> Chipaca: lxc exec
[14:35] <ogra> HRM
[14:35] <Chipaca> sergiusens: does that ssh?
[14:36] <ogra> shouldnt "apt remove snapd" also remove all installed snaps ??
[14:36] <sergiusens> Chipaca: no, it just runs whatever I tell it to run in that container
[14:36] <Chipaca> ogra: purge does
[14:36] <ogra> bah
[14:36] <Chipaca> ogra: dpkg -P snapd
[14:36] <sergiusens> Chipaca: if you say that waiting for cloud-init is enough, then I will do that
[14:36] <ogra> apt purge does it too ... thanks ... i thought a simple remove suffices
[14:36] <Chipaca> sergiusens: how do you wait for cloud-init?
[14:36] <sergiusens> `apt remove --purge snapd`
[14:37] <sergiusens> Chipaca: I am not, but I can
[14:37] <Chipaca> sergiusens: if you can wait for cloud-init, can't you wait for snapd?
[14:38] <sergiusens> Chipaca: I think we just agreed on what to do
[14:38] <Chipaca> sergiusens: wait for snapd :)
[14:53] <zyga> pstolowski: there are new imacs today :) https://www.apple.com/pl/imac/
[14:54] <pstolowski> zyga: i'm not looking. lalalla.
[14:54] <pstolowski> zyga: ok i lied; i looked
[14:58] <Chipaca> i'm going for a walk to wake up a bit, bbiab
[15:02]  * zyga lunch & walk
[15:02] <zyga> jdstrand: 3 second PR https://github.com/snapcore/snapd/pull/6607
[15:02]  * zyga really gone now
[15:02] <mup> PR #6607: cmd: typedef mountinfo structures <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6607>
[15:31]  * cachio lunch
[15:44] <jdstrand> zyga: I'm aware of the PR :) I glanced at it, it isn't 3 seconds, but it is fast. I have several things I'm trying to get to, and that is towards the top
[15:47] <zyga> jdstrand: how can it not be 3 seconds, it's a typedef :)
[15:47] <zyga> jdstrand: but I'm happy to see the final review
[15:47] <zyga> jdstrand: don't forget it's the 360 day
[15:47] <jdstrand> zyga: cause it is a bunch of them :)
[15:47] <zyga> that's always a priority
[15:48] <jdstrand> zyga: indeed, that is one of the things that was before it (I just completed that, fwiw)
[15:48] <zyga> I'm doing that now
[15:48] <zyga> just NaN people left :)
[15:48] <jdstrand> heh, yeah
[15:50] <Chipaca> brb need to reroute my booter
[16:14] <mup> PR # closed: core-build#11, core-build#22, core-build#26, core-build#37
[16:15] <mup> PR # opened: core-build#11, core-build#22, core-build#26, core-build#37
[16:27] <zyga> almost done with 360s
[16:27] <zyga> (and then the walk I promised myself)
[16:34] <zyga> 360 done, I'll go now because it's getting dark
[16:34] <zyga> ttyl
[16:39] <mup> PR snapd#6623 opened: interfaces/builtin/opengl: allow access to Tegra X1 <Created by alfonsosanchezbeato> <https://github.com/snapcore/snapd/pull/6623>
[16:54] <pedronis> Chipaca: mborzecki: we can close 6615 at this point, right?
[16:54] <Chipaca> pedronis: i'm still running that
[16:55] <Chipaca> pedronis: to confirm my hypothesis
[16:55] <Chipaca> (so far it's failed because opensuse is having a day)
[16:56] <pedronis> ah
[16:57] <ackk> hi, does snapd do something special with /snap and snap dirs if the underlying fs is btrfs?
[16:58] <Chipaca> ackk: no
[16:58] <Chipaca> ackk: any weirdness you see is entirely of btrfs's making :-)
[17:00] <Chipaca> pedronis: … and again opensuse :-(
[17:04] <zyga> ackk: can you tell me more please?
[17:06] <pedronis> Chipaca: push to make it manual if it's having a day
[17:07] <pedronis> Chipaca: that's another area that we'll need to think about in our tests
[17:08] <ackk> zyga, so it's a bit complicated dev setup I have here. basically my disk is all btrfs, and I'm using "snap try" in a lxd container (which uses btrfs backend as well)
[17:09] <ackk> zyga, the issue I'm having which I suspect is related is that sometimes I rsync some files to the prime/ dir then "snap restart mysnap" and I get "no such file or directory" when trying to execute snap commands
[17:09] <ackk> but the files are there
[17:09] <zyga> Can you provide a small reproducer
[17:09] <zyga> I wonder if rsync replaces the directory
[17:10] <zyga> Making the bind mount die
[17:10] <ackk> zyga, it doesn't happen all the times. what I noticed is that the /snap dir in the snap shows as a btrfs subvolume mounted, but that's not true
[17:10] <zyga> Can you provide a copy of proc self moujtinfo please
[17:10] <ackk> zyga, specifically: /dev/mapper/ubuntu-root on /snap type btrfs (rw,noatime,ssd,space_cache,subvolid=1337,subvol=/lxd/storage-pools/default/containers/maas/rootfs/snap)
[17:11] <ackk> sure one sec
[17:11] <zyga> Tying from phone sorry
[17:11] <zyga> I will be home in 10 minutes
[17:12] <ackk> zyga, http://paste.ubuntu.com/p/SXmVh3XYM9/
[17:12] <ackk> zyga, if you're trying to reproduce in a lxd container on your phone, that's impressive :)
[17:12] <zyga> It is an Ubuntu phone ;-)
[17:12] <zyga> Kidding though
[17:12] <zyga> I will try at home
[17:12] <ackk> thanks
[17:13] <zyga> Do you have the broken setup still
[17:13] <zyga> Can you jump into lxd
[17:13] <zyga> And mointinfo there
[17:14] <zyga> Lastly jump into broken mount namespace (use nsenter -m/run/snapd/ns/snapname.mnt)
[17:14] <zyga> And provide that third mountinfo please
[17:14] <ackk> zyga, I don't have it anmore at the moment
[17:14] <zyga> Please wrap that into a bug report
[17:14] <zyga> Ok
[17:14] <zyga> Once you do please :-)
[17:15] <ackk> zyga, fwiw the snap-try'd dir also was showing as a btrfs mount
[17:15] <ackk> with a subvol= pointing at the dir (which is not really a subvol)
[17:15] <ackk> zyga, that paste is from within the lxd fwiw
[17:16] <ackk> zyga, ok I will, thanks
[17:16] <ackk> I've seen this happening a lot of times
[17:16] <ackk> zyga, if it matters, the snap is --devmode
[17:16] <zyga> Snap try is just a bind mount
[17:17] <zyga> Snaps sets that up based on the path given
[17:17] <zyga> So it would likely be whatever is powering the lxd container
[17:17] <zyga> My personal setup is not using btrfs
[17:18] <zyga> ETOOMUCHHASSLE
[17:18] <zyga> but perhaps I should dog food more
[17:18] <zyga> Is this on openSUSE?
[17:20] <ackk> no, ubuntu
[17:20] <ackk> I've been using btrfs since xenial here
[17:21] <zyga> Aha
[17:22] <ackk> zyga, ah yeah indeed if you bind-mount a dir it shows in subvol
[17:30] <zyga> ackk: so the bind mount subvol thing is a non-issue?
[17:30] <zyga> ackk: I wonder about the breakage
[17:30] <zyga> ackk: I suspect rsync is replacing the prime directory
[17:30] <zyga> ackk: if this happens, look if mountinfo says "deleted"
[17:30] <zyga> ackk: that would be a good bug report
[17:32] <mup> PR # closed: snapcraft#1649, snapcraft#1875, snapcraft#2020, snapcraft#2135, snapcraft#2176, snapcraft#2229, snapcraft#2239, snapcraft#2398, snapcraft#2413, snapcraft#2433, snapcraft#2444, snapcraft#2445, snapcraft#2463, snapcraft#2470, snapcraft#2493, snapcraft#2495, snapcraft#2500, snapcraft#2504
[17:32] <ackk> zyga, yeah it seems the bind mount just shows the dir as subvol, so I guess it's not an issue
[17:32] <zyga> ackk: but you said it was broken too
[17:33] <ackk> zyga, yeah sorry I mean, it's not something different that snapd is doing wrt the mountpoint
[17:33] <zyga> ok
[17:33] <ackk> zyga, yeah it seems binaries are not found althoguh they're there
[17:34] <zyga> can you run "snap run --strace=--raw snap.app"
[17:34] <zyga> I wonder what really happens
[17:35] <mup> PR # opened: snapcraft#1649, snapcraft#1875, snapcraft#2020, snapcraft#2135, snapcraft#2176, snapcraft#2229, snapcraft#2239, snapcraft#2398, snapcraft#2413, snapcraft#2433, snapcraft#2444, snapcraft#2445, snapcraft#2463, snapcraft#2470, snapcraft#2493, snapcraft#2495, snapcraft#2500, snapcraft#2504
[17:47]  * zyga EODs
[17:47] <zyga> tty
[17:47] <zyga> ackk: please follow up tomorrow, I'd love to see that strace
[17:51] <Chipaca> #6620 could use a second review
[17:51] <mup> PR #6620: tests/main/remodel: clean up before reverting the state <Created by chipaca> <https://github.com/snapcore/snapd/pull/6620>
[17:51] <mup> PR snapd#6624 opened: overlord/snapstate, store: retry less for auto-stuff <Created by chipaca> <https://github.com/snapcore/snapd/pull/6624>
[17:52] <Chipaca> and, pedronis if you're around, 6624 ^ is nasty but nice
[17:52] <Chipaca> my wifi goes AWOL in 8 minutes, so my EOD is then
[17:52]  * Chipaca trying a new thing to see if he can get his sleep schedule sorted
[17:54]  * Chipaca thinks better about it, and adds two minutes to the scheduled wifi shutoff
[18:00] <Chipaca> ok, EOD, ttyl, hand, etc etc
[18:31] <cachio> niemeyer, hey
[18:32] <cachio> niemeyer, when you have a minute, could you pleas take a look to https://github.com/snapcore/spread/pull/75
[18:32] <mup> PR spread#75: Make spread tests for spread project run on google backend <Created by sergiocazzolato> <https://github.com/snapcore/spread/pull/75>
[18:32] <cachio> ?
[18:46] <pedronis> mvo: I don't think the whole of 6624 is viable for 2.38
[18:56] <cachio> pedronis, about the card to add tests for service.ssh.disable
[18:56] <pedronis> cachio: yes?
[18:57] <cachio> pedronis, how should be the gadget scenario?
[19:00] <cachio> jdstrand, hey, is there any lp issue for the sru validation?
[19:02] <pedronis> cachio: we need something like  main/ubuntu-core-gadget-config-defaults/task.yaml at least
[19:02] <pedronis> cachio: then we should explore if we can do something where we boot for real (not just snapd restart) with something made with ubuntu-image
[19:03] <jdstrand> cachio: see privmsg
[19:04] <cachio> pedronis, ok
[19:05] <cachio> pedronis, starting with this, thanks
[19:05] <mvo> pedronis: looking at 6624
[19:09] <mvo> pedronis: yeah, it looks risky, lets discuss tomorrow, my brain is a bit fried
[20:26] <mup> Issue # closed: core18#56, core18#86, core18#89, core18#117
[20:26] <mup> PR # closed: core18#43, core18#63, core18#72, core18#90, core18#98, core18#120
[20:27] <mup> Issue # opened: core18#56, core18#86, core18#89, core18#117
[20:27] <mup> PR # opened: core18#43, core18#63, core18#72, core18#90, core18#98, core18#120