[02:10] PR snapcraft#2418 closed: Release changelog for 3.0.1 [06:11] morning [06:27] Hi [06:57] zyga: hey [07:07] Hej [07:07] Breakfasting [07:07] I sent a PR last night [07:08] Fixing leap issues [07:08] I will make a small variant for 2.36 [07:46] good morning! it looks like the tests in 6243 got restarted, did we see any trace of the protocol error in the last run? [07:48] looks like the current run was good, I do another one now [07:50] mvo: i restarted them twice already, first time failed in prepare of juju observe test (unrelated), next failure was centos mirrors being flaky [07:51] mborzecki: heh [07:51] mborzecki: ok, it had one green now and I just triggered it again (via rename of GIL->GML) [07:57] super simple review #6241 [07:57] PR #6241: tests/main/parallel-install-store: verify installation of more than one instance at a time === pstolowski|afk is now known as pstolowski [07:58] mornings [08:01] Back from walk [08:01] Brrrr [08:01] Though I really like it when winter makes everything dry [08:02] Hey mvo [08:02] I will be around shortly [08:03] zyga: hey, no worries [08:03] zyga: all is looking great, I'm in a very good mood, PRs look super interessting [08:03] I posted the leap fix last night [08:03] Just the master version [08:04] 2.36 will be microscopic [08:04] Master has a cleanup as well [08:04] zyga: aha, even better [08:28] mborzecki: I put a question in #6079 [08:28] PR #6079: [RFC] `snap connections` command [08:29] PR snapd#6234 closed: overlord: don't write system key if security setup fails (2.36) [08:33] pedronis: thanks, finishing up with snapd update for arch now, will lack at your review in a bit [08:41] PR snapd#6221 closed: interfaces: return security setup errors (2.36) <⛔ Blocked> [08:46] mvo: https://github.com/snapcore/snapd/pull/6245 [08:46] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [08:46] it won't pass entirely [08:47] after that there will be only one small patch that makes apparmor-parser errors really reported and 2.36 is green [08:47] similar to the one you closed above in 6221 [08:47] PR snapd#6245 opened: interfaces/backends: detect too old apparmor_parser (2.36) [08:47] if you want I can add it here [08:50] mvo: that case you found with brave install & removal sequence is amazing; it really has to involve all the three ops, just installing the instance and removing it (2 ops) is not enough [08:55] pstolowski: yeah, I found this too, its interessting [08:56] zyga: it sounds like like it can be added to 6245 if its small [08:56] mvo: yes, and one will not pass without the other [08:56] I'll do just that [08:56] mvo: thank you for merging the 2.36 fix even though it was not passing [08:57] pstolowski, mvo: what's the special case with brave? [08:57] zyga: sudo snap install brave && sudo snap install brave_test && sudo snap remove brave brave_test [08:57] zyga: (with instances feature on) [08:57] mhm [08:57] pstolowski: hi, do we have developer docs for interface hooks? [08:58] zyga: results in a bunch of conflicts and retries on the last step, take a good while to finish [08:58] pstolowski: conflicts on what? [08:59] pedronis: yes, they were written a couple of weeks ago [08:59] pedronis: conflicts on disconnects on removal (last op) [08:59] I see [09:00] pstolowski: I suppose at this point they are expected, no? we'll need to step back to improve that? or is there a low hanging fruit? [09:01] pedronis: yes, we expect it, but mvo observed a noticable slowdown with edge (which i can confirm) [09:02] ok [09:02] pedronis: it was noticed ~last week [09:02] pedronis: and again, it completes and work, is just slow [09:03] pstolowski: any candidate on why? feel free to dig a bit when you have a bit of time [09:05] pedronis: yeah i'm on, investigating, don't know yet [09:10] hmm no smart way of reloading snap-confine aa profile on arch when installing/updating snapd, i expect people to run into problems with snap-confine [09:10] oh [09:10] can we do that ourselves? [09:10] in the package file? [09:11] zyga: i can, but it's unclear whether i should, you know, the unwritten policy of doing nothing :) [09:11] hehe [09:11] does that also apply to libc breaking people on upgrades? [09:11] is there any limit? [09:11] hahah, it's /r/archlinux, or bbs.archlinux.org or just archlinux.org main page with 'manual steps needed' memo [09:12] /o\ [09:12] though i don't recall any major screwups for years now [09:12] * zyga is happy with opensuse :) [09:13] i mean, surprisingly, it just works [09:19] does it make sense to request snapd restart on a classic system where reexec is not supported? [09:19] mmm [09:20] probably not [09:20] that's what happens on arch, fedora and prooably opensuse too [09:21] morning Chipaca [09:21] morning [09:21] this basically https://paste.ubuntu.com/p/xRnrQMNTH8/ [09:21] Chipaca: morning, sir [09:21] Chipaca: hi [09:22] 'sup [09:24] mborzecki: hi, you have various things under review, but no current task that isn't blocked on feedback? is that true? [09:26] pedronis: i'm addressing feedback on the PRs and chopping selinux branch up to pull out the bits that aren't strictly related (mostly tests) [09:27] mborzecki: ah ok, so you are not blocked atm? [09:27] Chipaca: did you see https://github.com/snapcore/snapd/pull/6236#discussion_r237451827 ? [09:27] PR #6236: Staticcheck fixes [09:27] pedronis: no [09:27] pedronis: I did [09:27] I'm assuming it's technically correct [09:28] in practice, not sure how important it is :-) [09:30] Chipaca: ? [09:30] * pedronis is probably missing something [09:30] pedronis: the current regep means we won't run staticcheck on ".0" releases [09:31] because go doesn't do .0, instead just does "" [09:31] apparently [09:31] Chipaca: not sure why you say is not important? having a test that doesn't run for a while [09:31] and then again [09:32] is usually source of confusion [09:32] of which if have enough I think [09:32] s/if/we/ [09:33] Chipaca: or you thinking that .0 never makes it into Ubuntu anyway? [09:33] I'll fix it (not that way though) [09:33] anyway changing it is likely quicker than this silly discussion :) [09:33] but, i'm worried that already 1.11 is breaking things for us [09:34] wonder what 1.12 will bring [09:34] and then there's go 2 [09:34] one thing at a time [09:35] pedronis: did you see"go 2 considered harmful"? [09:35] https://homepages.cwi.nl/~storm/teaching/reader/Dijkstra68.pdf [09:35] * Chipaca grins [09:36] no [09:36] lol [09:37] changing subjects, I pushed a little "make snapd start faster" pr after talking with cachio yesterday [09:37] and mvo fixed the spread tests (thanks mvo!) [09:37] and now it's so fast, systemd freaks out :-/ [09:38] I saw it [09:38] ah [09:38] :-) [09:38] no good deed goes unpunished [09:38] bah, it might be something else [09:38] i need to dig [09:38] but [09:38] Job for snapd.service failed because start of the service was attempted too often. See "systemctl status snapd.service" and "journalctl -xe" for details. [09:38] that thing :) [09:38] Chipaca: let's try to merge 6192 first [09:38] ok [09:39] * Chipaca looks at what's missing there [09:39] couple of minors there, and it needs a 2nd review (I cannot review twice) [09:44] pedronis: looking at the revision of the local snap would split error messages even more though [09:44] i mean, either snap could be local [09:47] * Chipaca gives it a stab [09:47] Chipaca: ? [09:47] pedronis: your suggestion to, in the local case, say "current local" [09:47] yes [09:47] pedronis: is about the current snap [09:48] pedronis: the revision we talk about in the error messages so far is about the new snap [09:48] cachio: oh seems it worked this time: https://travis-ci.org/MirServer/mir/jobs/461685223 [09:48] Chipaca: let me look [09:48] I might be very confused [09:49] pedronis: unless you meant to always say "current local", as in that-which-is-locally-current [09:49] and not as in that-which-is-locally-current-and-has-a-revision-that-is-local [09:50] Chipaca: I see, I'm confused [09:50] but something is off either way [09:50] hello confused i'm dad [09:51] Chipaca: so we are looking at snapInfo revision [09:51] great [09:51] except that calling a local installed snap new revision in the 2nd message [09:51] i could've caled them newInfo and curInfo instead of snapInfo and curInfo i guess [09:51] is confusing and what threw me off [09:52] * Chipaca looks [09:52] Chipaca: if I do snap install --dangerous foo.snap [09:52] calling the incoming thing new revision [09:52] is confusing [09:52] aha [09:52] (tough technically correct) [09:52] i'll happily drop in any other noun :-) [09:53] I might even drop in a verb if i'm in a hurry [09:54] pedronis: I mean: «cannot refresh "foo" because [this new thing] has epoch 123 that can't read [the old thing]'s epoch of 0» [09:55] Chipaca: "new epoch %s being locally installed" would be the verbose thing [09:57] «cannot refresh "foo" because new epoch 123 being locally installed can't read [...]»? [09:57] yes [09:57] that's confusing to read [09:57] I'm happy to suggestion, the current error confused me though, so we need to do better [09:58] so, it can be a try or a --dangerous [09:58] yes [09:59] cannot refresh "foo" with local snap because its epoch ...? [10:00] «cannot refresh "foo" because the one you're trying to install has epoch [...]» [10:00] ? [10:00] hm [10:01] cannot refresh "foo" with local snap with epoch %s ... ? [10:01] cannto refresh "foo" to local snap with epoch...? [10:02] that also work [10:02] s [10:02] good, let me write it out in full just in case [10:03] cannot refresh %q to local snap with epoch %s, because that can't read current revision's epoch of %s [10:03] or maybe change the that to an it [10:03] cannot refresh %q to local snap with epoch %s, because it can't read current revision's epoch of %s [10:03] bah, that "revision's" there is noise [10:03] noise][1: no offence [10:04] can be dropped I suppose [10:04] cannot refresh %q to local snap with epoch %s, because it can't read the current epoch of %s [10:04] snap install hello-world does-not-exist, giving 'error: store.SnapNotFound with 2 snaps' is not super user friendly [10:04] mborzecki: my PR will fix it [10:04] pedronis: ack [10:04] mborzecki: i disagree, it's super friendly. You're just not its friend. [10:04] I'm actually hoping to get to it today [10:06] Chipaca: hehe :P didn't bother me until yesterday i didn't notice a typo in snap name i tried to install and ended up going through snapd code to figure it out [10:07] mborzecki: tbf that error means there's a bug, it's meant as an error for us and not for users [10:08] we're lucky then that users install snaps one by one [10:08] but it did give me a bit of panic yesterday when 'snap install hello-world_foo hell-world_bar' didn't work :/ [10:09] Chipaca: there's two part of the code that make different assumptions, anyway #5712 should stop that afaiu [10:09] PR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates <⛔ Blocked> [10:09] pedronis: 🖒 [10:14] Chipaca: we probably should drop revision's alos in the other one, current epoch alone seems ok, no? [10:15] pedronis: heh [10:15] given also that it might be local or not in its own [10:15] return fmt.Errorf("cannot refresh %q to %s with epoch %s, because it can't read the current epoch of %s", snapInfo.InstanceName(), desc, snapInfo.Epoch, curInfo.Epoch) [10:15] ^ current code [10:15] where desc is "local snap" or "new revision %s" [10:15] sounds good [10:16] Chipaca: we get the {} notation right if the epoch is complicated ? [10:16] yes [10:17] that's fine for now [10:17] mvo, pedronis ok, i understood the problem with slowness on brave+brave_test install+remove. perhaps HO would be fastest to explain if you have a few minutes [10:17] for now it's the json one [10:17] although from using it, just {[],[]} might be enough [10:18] (as opposed to {"read":[],"write":[]} ) [10:18] pstolowski: I'm available in a couple of minutes [10:19] pedronis: ok, ping me then (mvo feel free to skip it) [10:22] silly Go, not telling that when I write lcoal I mean local [10:23] pstolowski: ready now, in the standup [10:23] pedronis: ok coming [10:24] another thing we maybe could improve, when installing a snap that pulls in a content snap, the prompt will stay on 'Automatically connect eligible ..' for a long time, while the content snap is downloading [10:45] mvo: https://github.com/snapcore/snapd/pull/6245 needs a review [10:45] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [11:03] pstolowski: reading https://github.com/snapcore/snapd/pull/6180 [11:03] PR #6180: snap/info: bind global plugs/slots to implicit hooks [11:04] zyga: thanks! [11:08] mvo: zyga: Chipaca: I'll be late for standup, I posted a forum message to the team about this [11:08] thank you [11:09] pedronis: thank you [11:09] pstolowski: sorry, was away but I take it you clarified this? [11:10] mvo: yep, no worries [11:10] zyga: I do a new round of review now [11:10] thank you [11:10] the local authority has finally (nearly 12 months later!) come to a decision about the boys' school \o/ [11:10] I'm not feeling well, sorry about being very slow today [11:10] … so I might miss the standup entirely today [11:10] zyga: no worries [11:10] * Chipaca hugs zyga [11:10] Chipaca: good news! (about the boys - not that you miss the standup ;) [11:10] but … from far away just in case it's cooties [11:11] * mvo hugs zyga as well [11:11] just cold, nothing fancy :) [11:11] I'll be allright [11:11] Chipaca: ok [11:13] pedronis: retry mechanism seems to work *most of the time*, but there is something fishy there, *sometimes* it doesn't pick any other task until retry time of previous one is reached - see the proof with some extra debug https://pastebin.ubuntu.com/p/TYMffsdTQX/ [11:15] Chipaca, nice, I'll try it [11:15] thanks [11:16] Saviq, nice [11:16] please let me know if we need any change to that image [11:17] pstolowski: that seems strange, we are missing something, I suppose some prints in the taskrunner might help [11:18] Chipaca, which is the PR number? [11:18] #6192 needs a 2nd review [11:18] PR #6192: overlord/snapstate: on refresh, check new rev can read current [11:18] pedronis: yep [11:19] cachio: #6242 [11:19] PR #6242: overlord/snapstate: use file timestamp to initialize timer [11:19] degville: might you could look quickly at the new errors in #6192, if you have some feedback on them [11:20] notice that in most sitation the store should avoid us hitting them tough [11:20] they are more relevant for local development using try or local installs [11:20] zyga: did you mean to open #6245 with target set to release/2.36? it's targeting master now [11:20] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [11:20] ooh [11:20] shit :) [11:20] yes [11:21] fixed [11:21] this version is for the stable release [11:21] I was surprised it passed alone [11:21] heh, same here [11:21] (also surprised) [11:22] degville: thanks for https://forum.snapcraft.io/t/the-docs-roadmap/8763 [11:22] cachio: the problem is nfs-support restarts snapd 11 times :-) [11:22] cachio: and it's fast enough for that to be a problem now [11:22] cachio: I'm fixing [11:23] Chipaca, nice, thanks for that !!! [11:24] 5845 is ready for a second review [11:24] no problem. [11:24] * pedronis can't type today [11:26] PR snapd#6241 closed: tests/main/parallel-install-store: verify installation of more than one instance at a time [11:28] mup: kicked off another spread run in #6243 [11:28] mborzecki: In-com-pre-hen-si-ble-ness. [11:28] mvp: kicked off another spread run in #6243 [11:28] mvo: kicked off another spread run in #6243 [11:28] PR #6243: systemd: allow only a single daemon-reload at the same time <⛔ Blocked> [11:29] mborzecki: thanks, I run this now also locally against google:ubuntu-18.04-64, so far no issues, lets see [11:31] actually, i wonder if systemd seeing the restarts as problematic are because we don't exit from the main loop properly [11:31] * Chipaca digs [11:32] could I get a second review of #6236? it's got a minor fix pending (in a Suggestion already) but I'd rather not re-trigger a travis just for that unless it's the only thing left [11:32] PR #6236: Staticcheck fixes [11:32] jamesh: I am keeping your user session pr in mind, have not finished the review yet though [11:32] Chipaca: I have it on my list, hope to get to it before lunch [11:33] mvo: excellent thanks [11:41] mvo: https://github.com/snapcore/snapd/pull/6245 is green! [11:41] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [11:41] that's lucky [11:52] mvo: looks like we dropped the ball on https://forum.snapcraft.io/t/anything-changed-around-snapcraft-yaml-gpio-interface-for-2-36/7852/4 (unless there was more communication out of band) [11:54] moar fire [11:55] toasty [11:56] pretty please review on 6245 [11:56] sir yes sir [11:57] thank you, that's almost all needed to make 2.36 green again [11:57] Chipaca: I think we fixed this but we did not communicate this well :(/ [11:57] Chipaca: I need to look what PR fixed it, don't quite remember offhand [11:57] mvo: is is in 2.36.x ? [11:57] Chipaca: thanks. [11:57] maybe we need a .3 [11:57] zyga: I think so [11:57] zyga: oh, why .3? what is missing? [11:57] I mean, if this is not in stable [11:57] but in 2.37 / beta only [11:58] but no need otherwise [11:58] don't propose a new .X on the eve of a sprint dude, that's just mean [11:58] you need to do it the monday of the sprint and write a CVE, that's the sporting way [11:58] Chipaca: *wahhhhh* [11:58] * mvo runs away screaming [11:58] HAHHAHA [11:59] * Chipaca 's job is done [11:59] Chipaca: make sure to msg mvo on his cell at 23:00 on sunday ;) [11:59] PLANES FALLING OFF THE SKY, NEED .3 [12:01] zyga: you guys are so mean! [12:01] mvo: we will attach patches :) that's foss spirit, right? [12:01] * zyga hugs mvo [12:01] lol [12:01] * mvo hugs zyga [12:01] no worries, we will wait until after breakfast [12:05] mborzecki: updated https://github.com/snapcore/snapd/pull/6244 [12:05] PR #6244: release: detect too old apparmor_parser [12:13] PR snapd#6246 opened: spread: show AVC audits when debugging, start auditd on Fedora [12:15] PR snapd#6247 opened: tests/lib: do not pick up xatts when repacking core snap [12:16] mborzecki: ^ don't we have xattrs that we care about in core tough? [12:17] pedronis: afaik not yet [12:17] pedronis: not that I know of, ubuntu will enable xattrs but it's not done yet [12:17] still is that change just exchaing a bug for another later? [12:17] heh, in some way yes [12:18] but xattrs will not show up by accident [12:18] and I think we will have some changes around snapd too [12:18] we are not the one adding them, though? are we [12:18] ? [12:18] sorry? [12:18] zyga: I mean, it depends on what you mean by accident, [12:19] other teams doing things without telling can be considered an "accident" [12:19] mborzecki: can we add some code to detect if there's xattrs that are not selinux related around? [12:19] I meant that there will be coordination and I expect things will fail if we rip them out in some tests. programs that were setuid root will require on capabilities instead [12:20] zyga: you are an optimistic [12:20] pedronis: i'll think about it, with the context applied via mount that change may not be needed at all [12:20] thx [12:21] otherwise a way of doing this without adding selinux context would be nice, but not sure if that's entirely possible when we repack the snap [12:24] mborzecki: is there no utility to run something controlling the context? [12:24] anyway I let you look into this further [12:31] PR snapd#6248 opened: tests/lib/reset: restore context of removed snapd directories [12:34] PR snapd#6237 closed: client, store: don't use store from client (use client from store) [12:53] zyga: hey, curious, does leap run on arm? [12:53] jdstrand: hey [12:53] yes, I think there were some pi3 announcements by suse [12:54] https://en.opensuse.org/HCL:Raspberry_Pi3 [12:54] I haven't tried myself [12:54] why? :) [12:54] jdstrand: FYI, I sent some patches that disable apparmor on 43.3 because apparmor_parser 2.10.3 doesn't parse our profiles [12:55] zyga: it is because of your fyi :) [12:55] haha [12:55] I see [12:55] :) [12:55] can you expand, I don't yet see a correlation between leap and arm [12:56] Chipaca: https://github.com/Juniper/libxo [12:56] remember the topic about snap foo --json [12:56] and friends? [12:56] zyga: you surely recollected that we added conditional support for unsafe in classic and there is https://github.com/snapcore/snapd/pull/5982 [12:56] PR #5982: interfaces/many: conditionally use 'unsafe' with docker-support change_profile rules [12:56] yes [12:56] but to interfaces [12:57] yes [12:57] snap-confine's own profile goes down in flames [12:57] we just never noticed [12:57] I know [12:57] until all hell broke loose [12:57] again, I read the pr [12:57] so I was thinking "should we even consider conditionally adding unsafe to the snap-confine profile?" [12:58] then I reread why we added it: https://github.com/snapcore/snap-confine/pull/133 [12:58] PR snap-confine#133: Adjust apparmor policy for compatibility with snap-exec [12:58] 42.3 will be EOL in june next year [12:58] we added it for arm [12:59] so if arm is on leap, and we conditionally added it, the on arm leap would break [12:59] or rather, if leap is on arm [13:00] hmm, I don't recall the whole arm / leap part of the branch [13:00] so the answer is "no, we need to unconditionally add it" [13:00] ah [13:00] zyga: that branch wasn't done for leap [13:00] the branch was done to fix snapd on arm at all [13:01] ok, I read the patch again, I see what that was about [13:01] now back to the current patches [13:01] point is, if we decided to conditionally add unsafe to the snap-confine profile, then wherever we didn't add it, arm would be broken. that's silly [13:01] off to pick up the kids [13:02] jdstrand: where are we doing this conditionally? I must be missing something [13:03] zyga: huh? the interfaces [13:03] jdstrand: are you talking about the current set of patches? [13:03] jdstrand: ah. sorry, too many patches [13:03] so [13:03] there are two cases of broken [13:03] one was louder than another [13:03] zyga: I think you may be reading this irc conversation too quickly [13:03] we ignored some errors so the snap-confine profile being broken was below the radar for a long time [13:03] 06:57 < jdstrand> again, I read the pr [13:03] 06:57 < jdstrand> so I was thinking "should we even consider conditionally [13:03] adding unsafe to the snap-confine profile?" [13:03] but interfaces were [13:03] aha [13:03] I see [13:04] zyga: I asked you about arm on leap to answer my question that was in my head [13:04] thank you for explaining [13:04] so, we can do that in 2.37, I don't mind doing that [13:04] zyga: the pr is doing the right thing [13:04] we'd just have to check [13:04] (on a real board) [13:04] (in principle) [13:05] * jdstrand hasn't reviewed the code [13:06] jdstrand: note that there are two versions [13:06] zyga: are you planning a followup pr to clean up the classic template's conditional use of unsafe? [13:06] https://github.com/snapcore/snapd/pull/6245/files is super short for 2.36 [13:06] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [13:06] zyga: I saw [13:06] I can do that, yeah [13:06] I didn't connect that dot last night [13:06] zyga: yeah, well, I didn't connect the dots with the snap-confine profile when doing the classic template pr [13:07] in hindsight logging errors and carrying on was a costly thing :) [13:07] zyga: when these land, I'll update that open docker-support pr [13:07] jdstrand: did you see https://github.com/snapcore/snapd/pull/6233 [13:07] PR #6233: overlord: don't write system key if security setup fails [13:08] no [13:08] that took some time to track down [13:08] it was breaking 2.36 for all last week and this week [13:09] I need to read that carefully-- my first question is 'why is snapd restarting when snap-seccomp and apparmor_parser are running? [13:09] ' [13:09] jdstrand: it is done in test prep [13:09] that's actually the only thing we don't have a 100% firm answer [13:09] as in [13:10] to be precise [13:10] why snapd is being restarted while those things are in flight [13:10] exactly [13:10] since we use sd_notify to tell systemd we are ready only after the managers have been constructed [13:10] one possible explanation [13:11] that sounds... weird and probably worth knowing cause there might be other bugs that are caused by this [13:11] is that overlord is indeed done but this is a part of some ensure cycle [13:11] (so normal activity) [13:11] I pasted traces [13:11] and there's a PR that can show you all the data in one spot [13:11] (closed now, I can revive it) [13:11] that 'weird' comment was for your 'why snapd is being restarted while those things are in flight' comment, not your sd_notify idea [13:11] yes [13:12] so we have journald/strace/forkstat and snap changes [13:12] I would love if someone had a deeper look at this [13:12] I feel tired after fighting 2.36 issues this week [13:12] jdstrand: https://github.com/snapcore/snapd/pull/6230 [13:12] PR #6230: spread: detect "signal: terminated" in journal logs <⛔ Blocked> [13:12] the PR name is stale, it became a box of experiments [13:13] I can find the seed that reproduces the problem [13:13] jdstrand: to be clear, this is in tests, there is quite a bit of manual restarting of snapd going on [13:13] though honestly any run on that branch will fail quickly [13:14] zyga: did you consider trapping the signal and waiting to stop til it was safe? [13:14] jdstrand: it's not our side that can trap [13:14] I mean, we'd have to change apparmor parser [13:14] jdstrand: this is killing the helpers [13:14] systemd looks at the whole cgroup [13:14] not snapd [13:14] and kills one by one [13:14] anyway we had a different solution as well [13:14] yes, mvo sent a PR with KillMode=process [13:14] though it was not merged [13:15] it's not useful for non reexcing [13:15] zyga: well... snapd is calling the parser. if systemd tells us to stop and parser didn't complete... [13:15] sorry, it's not useful for distros that don't update [13:15] jdstrand: systemd is killing both us and the parser and snap-confine [13:15] but it would be cleaner [13:15] oh, I see [13:15] is that configurable in the systemd unit? [13:15] every process is [13:15] yes [13:15] mvo: why did we close the KillMode change ? [13:16] I mean, sure, this may happen regularly in the tests, but maybe it happens legitimately somewhere [13:16] pedronis: I believe we could do both actually, it seems saner for kill mode to be process for snapd [13:16] zyga: yes, my point [13:16] I don't know when that would be-- an admin restarting it? a deb upgrade? [13:16] but it doesn't fix reexecing cases [13:16] I know [13:16] but it gives us a saner future [13:17] yes [13:17] (theoretically) [13:17] it seems rather harsh for systemd to just kill everything in the cgroup without given the parent a chance to cleanup [13:18] jdstrand: that's the default :) [13:18] but in hindsight I agree [13:18] jdstrand: well, indeed there are ways to control that, it's the default tough as zyga said [13:19] there is no timeout associated with that? [13:19] anyway, I won't question the default of systemd here cause I don't have the context for that decision [13:20] but, if that is configurable, certainly for sanity we would want to at least try to have snapd shutdown cleanly [13:20] no? [13:20] yes [13:20] sorry, but I made a note, but we need mvo around but he is lunching [13:21] jdstrand: no, that's just SIGTERM [13:21] jdstrand: after that there is a wave of SIGKILL [13:22] * pstolowski lunch [13:26] zyga: that is my question. how long before the wave comes? [13:26] jdstrand: the wave of sigkill? [13:26] 07:21 < zyga> jdstrand: after that there is a wave of SIGKILL [13:27] zyga: you said that SIGTERM is sent. then a wave of SIGKILL [13:27] yes [13:27] DefaultTimeoutStopSec [13:27] zyga: I'm saying, trap SIGTERM, try to shutdown before the SIGKILLs [13:27] 90 seconds [13:28] jdstrand: assuming we are using kill mode process yes [13:28] but that would need some serious work [13:28] it's not easy to do now [13:28] e.g. manager startup is synchronous [13:28] so is overlord startup [13:28] we don't notify systemd that some startups need more time either [13:28] zyga: ok. so, certainly we can trap SIGTERM and signal the backend to finish the current parser and snap-seccomp calls, then exit without moving to the next batch, no? [13:29] jdstrand: we trap sigterm but the interaction between this and the rest of the daemon is not simple [13:29] I agree we can do this [13:29] hmm [13:30] well, I was only asking a question. it sounds like we have a stability/robustness concern here that the 6233 papered over [13:30] yes, I agree [13:30] not saying that 6233 is invalid for moving forward [13:30] we didn't notice that daemon was incorrectly stopping either [13:31] john sent some patches that caught that with static analysis [13:31] well, first we need to stop systemd to send the SIGTERM to apparmor_parser as well, etc [13:31] ok, well, I don't mean to distract. just rasing the point that perhaps a followup for cleaning up cleanly when receiving SIGTERM is a good idea [13:31] raising* [13:31] it also makes testing more complex, since right now all systemd started snapd's behave the same [13:31] with changing of kill mode some will run under the old snapd.service unit [13:31] some will run under the new one [13:32] jdstrand: all fair points jamie, [13:32] I think it would be nice if we could fix this with reexec magic and systemd services [13:34] ok, reading the description of 6233, that seems like a good improvement regardless of sigterm [13:38] pedronis: hi, KillMode=process got closed mostly because the other changes made it less urgent, it is probably still a good idea [13:39] mvo: with kill mode = process, we should make sure that it also applies to snapd run from snapd.snap via that special service [13:39] zyga: yeah, we can install a snapd.service.d file [13:39] zyga: I mean, write it automatically [13:39] oh [13:39] super nice idea [13:39] +1001 [13:39] zyga: :) [13:40] zyga: I can reopen my PR and add code to add the .d file [13:40] mvo: it's not urgent but yeah, for master +1 [13:41] zyga: +1 [13:41] mvo: though we need to think about revert consequences or just managing those .d files over time [13:41] zyga: indeed [13:46] PR snapd#6249 opened: cmd/libsnap: introduce and use sc_strdup [13:47] mborzecki: could you try to look at 6149 today? [13:48] PR snapd#6250 opened: data: set KillMode=process for snapd [14:08] PR snapd#6247 closed: tests/lib: do not pick up xatts when repacking core snap <⛔ Blocked> [14:49] degville: "keeping it the same" is that a +1 :-D [14:52] Chipaca: yes! [14:52] :) [14:52] wooowoeoeoo [14:58] pedronis: you're right, it's about AddBlocked predicate [15:07] mborzecki: hey [15:07] remember when you asked for a cleanup of how tools are invoked from snap-confine? [15:16] mborzecki: so you have your wish [15:16] pstolowski: good, but yes we'll need to think a bit what to do about that [15:16] mborzecki: found a branch that does that :) [15:16] zyga: hah :) [15:17] mborzecki: i think you did half a review of 6912 already, could you finish it? [15:17] #6912 [15:17] * Chipaca kicks mup [15:17] * Chipaca jumps up and down on mup [15:18] Chipaca: wrong PR #? [15:18] mborzecki: https://github.com/snapcore/snapd/pull/6192 [15:18] PR #6192: overlord/snapstate: on refresh, check new rev can read current [15:18] #6192 [15:19] 6192, 6912, what EVAR [15:19] aah epoch read thing [15:19] yeh [15:19] tempted to run with graham's +1 but thought i'd follow process for once :-p [15:20] Chipaca: friday called and wants to +1 your PRs [15:20] well there's one less to +1 now [15:21] PR snapd#6192 closed: overlord/snapstate: on refresh, check new rev can read current [15:21] and I guess I need to address the comments on the other ones [15:21] ugh, that's too much like work [15:23] Chipaca: the media one has a question by me, it probably needs discussion [15:23] pedronis: yeah [15:23] brb [15:23] pedronis: do you have the time to discuss? [15:24] Chipaca: not really today, that one we'll have to wait when we are both back [15:24] pedronis: ok [15:42] re [15:42] mvo: hey [15:42] mvo: do you have a sec [15:42] https://github.com/snapcore/snapd/pull/6235 is green [15:42] PR #6235: overlord,apparmor: new syskey behaviour + non-ignored snap-confine profile errors <⛔ Blocked> [15:42] and should make 2.36 green [15:42] would you mind finishing your review there [15:42] I'm happy to backport / cherry pick tests [15:43] or other stuff [15:43] from the corresponding master branches [15:55] * zyga considers EODing [15:55] unless someone wants to look at 2.36 PRs [15:56] pedronis: I pushed the discussed reorg to #6104. If you won't have time to re-review, can I discard your block? [15:56] PR #6104: snapctl: add "services" [15:59] Chipaca: yes, the direction looks what we discussed [16:00] Chipaca: done myself [16:08] pedronis: i included your rationale in the commit message, for posterity, fwiw [16:08] pedronis: thank you [16:08] zyga: #6104 is now needing a second review, if you're still itching to do one [16:08] PR #6104: snapctl: add "services" [16:09] yep [16:09] * Chipaca goes to get tea and some ibuprofen [16:09] Chipaca: can you trade for trivial https://github.com/snapcore/snapd/pull/6249 [16:09] I wanna land it :) [16:09] PR #6249: cmd/libsnap: introduce and use sc_strdup [16:10] 2.36.2 looks good on suse :) [16:10] and startup time for snaps is nice [16:10] though I wonder if we need to refresh each snap to see the effect [16:11] I disable/enable things to see that [16:11] mvo: ^ [16:11] Is that expected? [16:11] zyga: one refresh or enable/disable should be enough to fix the fontconfig cache [16:11] zyga: (also in a meeting) [16:12] ta [16:12] PR snapd#6236 closed: Staticcheck fixes [16:14] kenvandine: ping [16:14] is this expected? https://www.irccloud.com/pastebin/G8zDQBdJ/ [16:16] zyga: yes... jamesh submitted a PR to Yaru to fix that [16:16] cool [16:16] thanks [16:16] i'll check on the status [16:16] it's been weeks [16:17] * cachio lunch [16:23] PR snapd#6213 closed: interfaces: let NM access ifindex/ifupdown files [16:28] Chipaca: +1 [16:29] mvo: suse updated :) [16:30] error: snap "test-snapd-policy-app-consumer" has "remove-snap" change in progress [16:30] pstolowski: ^ is that something you were describing during the standup? [16:31] zyga: \o/ [16:31] zyga: hmm not really, my issue doesn't cause errors [16:33] zyga: where do you see it? [16:33] random failure on strdup [16:33] (on that PR) [16:33] I restarted as it looked like noise [16:34] uhm [16:36] mvo: vscode startup on leap 42.3 and TW is very good [16:37] Chipaca: did you see my ping about http://juniper.github.io/libxo/libxo-manual.html xo? [16:37] zyga: i did [16:38] cool [16:38] zyga: I'm not adding anything to my stack right now though :-) [16:38] sure sure [16:38] it's a C api [16:38] but the idea is neat [16:40] zyga: cool [16:41] zyga: thanks for checking/confirming! [16:48] * pedronis calls it a week [16:48] have a good weekend! [16:51] mvo, 2.36.2 in candidate [16:53] cachio: nice! [16:55] Now with spread results on customer hw :D [16:57] Chipaca: I managed to update my PR: https://github.com/snapcore/snapd/pull/5712 [16:57] PR #5712: overlord: make InstallMany work like UpdateMany, issuing a single request to get candidates [16:57] pedronis: i saw! excellent [16:57] pedronis: is that enough to unblock it? [16:58] yes [16:58] cwayne: that sounds like a good thing. Is that a good thing? [16:58] pedronis: neat [16:58] Chipaca: no it's a great thing :) [16:58] :) [16:58] Chipaca: if it gets green, there is really the last commit than needs a review and then it could go in [16:58] I mean real commit, not the master merges [16:58] since we've now got my team's tests plus yours running on every core update in beta :) [16:58] a'ight [16:58] pedronis: nice [16:59] cwayne: woah, nice! [16:59] what, only beta? [16:59] :-D [16:59] beta's my jam [17:21] PR snapd#6249 closed: cmd/libsnap: introduce and use sc_strdup [17:23] PR snapd#6251 opened: cmd/snap-confine: refactor calling snapd tools into helper module [17:25] mvo: not sure if you EOW but I'd love an ack on https://github.com/snapcore/snapd/pull/6245 [17:25] PR #6245: interfaces/backends: detect too old apparmor_parser (2.36) [17:27] PR snapd#6242 closed: overlord/snapstate: use file timestamp to initialize timer [17:32] Bug #1781906 changed: Content provider interfaces introduced to snaps aren't correctly set up [17:40] PR snapd#6104 closed: snapctl: add "services" [17:43] PR snapd#6252 opened: userd: handle help urls which requires prepending XDG_DATA_DIRS === pstolowski is now known as pstolowski|afk [17:45] jdstrand: hey, not sure if you want or have time on Friday evening but... https://github.com/snapcore/snapd/pull/6244 :D [17:45] PR #6244: release: detect too old apparmor_parser [17:45] you looked at it [17:54] EOW'ing here [17:54] o/ [17:54] when you see me next, december will be into two (decimal) digits! [17:55] :-D [17:55] maaan [17:55] envy * [17:55] well, unless i'm bored or something [17:55] :-) [17:55] you can always do reviews *D* [17:55] NOT :) [17:55] enjoy the time off [17:56] if I'm tempted, instead I'll work on werm [17:56] ttfn :-) [17:56] werm? [17:56] well, I'll find out in Dec [18:01] zyga: I'll try. have a few things I must get done today [18:01] thanks cwayne [18:01] thank you jdstrand :) [18:01] zyga: <3 [18:17] zyga: the CLA check shouldn't fail if I'm a member of canonical, right? [18:17] kenvandine: if it does it's a bug :) [18:17] are you asking about https://github.com/snapcore/snapd/pull/6252 ? [18:17] PR #6252: userd: handle help urls which requires prepending XDG_DATA_DIRS [18:17] zyga: yes [18:17] review sent :) [18:19] I should EOW [18:19] I may be back in the office but not actively paying attention unles pinged [18:19] *unless [18:58] PR snapd#6225 closed: tests: fix for failover test on how logs are checked [19:00] * cachio afk [19:08] PR snapd#6253 opened: Members of canonical LP group should pass CLA check [19:11] PR snapd#6224 closed: interfaces: return security setup errors [19:14] kenvandine: https://github.com/snapcore/snapd/pull/6252/files#r237971362 [19:14] PR #6252: userd: handle help urls which requires prepending XDG_DATA_DIRS [20:37] PR snapd#6254 opened: tests: improve how the log is checked to see if the system is waiting for a reboot [23:43] Hi! Can anyone help me to setup snapcraft dev environment? I'm trying to fix a bug related to Go modules support, but I can't get the local version of snapcraft working.