[07:02] morning [07:02] home schooling day 2 :/ [07:32] good morning [07:35] good morning zyga-mbp ! [07:35] hey mvo [07:35] hopefully today will be more productive, yesterday I was a bit sleepy after very busy weekend [07:35] I'm looking at the GPIO issue reported by cert [07:37] zyga-mbp: thank you [07:40] replied there [07:42] mvo: zyga-mbp: hey [07:42] hey mborzecki [07:42] hey mborzecki [07:53] hey pedronis, good morning [07:57] hello [07:59] good morning pedronis :) [08:02] morning [08:02] good morning pstolowski [08:04] good morning pawel [08:05] either everything is loading very very slowly or eschooling is saturating my home uplink [08:14] okay, finally in the office [08:14] should be more manageable now [08:17] zyga: a small update in the status of 7700 would be great, it lists next steps and step 2 (desktop notifications) is certainly done now :) [08:18] zyga: but no rush, just noticed while going over open pts [08:18] zyga: open prs [08:21] mvo, yeah that's certainly something to pick up soon [08:21] I know time is short, I'll try to do a bit more today [08:21] zyga: tiny comment is fine for now :) [08:21] zyga: just curious where in the schema we stand [08:22] mvo, let me look quickly [08:23] ta! [08:41] pedronis: if my style comments in 9549 make sense I can push them myself but I think it does not make sense if I touch this before you did a review(?) [08:45] mvo: I have the PR checked out, I can apply the changes, I think the new file name is right but indeed it will break old installs [08:45] pedronis: thanks [08:52] pedronis: mvo: either of you working on tweaks to 9549? [08:52] mborzecki: I am [08:53] pedronis: ah, ok [09:04] PR snapd#9551 opened: dirs, boot: add ubuntu-save directories and related locations [09:05] pedronis: mvo: ^^ extracted some trivial bits from #9528 [09:05] PR #9528: cmd/snap-bootstrap: mount ubuntu-save during boot if present <⛔ Blocked> [09:33] mborzecki: in a meeting, will get to it right after that :) [09:33] mborzecki: and thank you! [09:42] we need reviews for https://github.com/snapcore/snapd/pull/9542 [09:42] PR #9542: interfaces: deny connected x11 plugs access to ICE <⚠ Critical> [09:45] PR snapd#9552 opened: usersession: fix typo in test name [09:45] mvo ^ plz merge typo fix [09:47] zyga-mbp: sure [09:47] thank you [09:56] mhm nested stuff is fun... [09:57] pstolowski are you using cachio's new nested tool? [09:58] zyga-mbp: actually not! but this is just syntactic sugar.. [09:59] just having an issue i don't understand yet [09:59] I see [10:04] mvo: mborzecki: I pushed to #9549, it needs 2nd reviews [10:04] PR #9549: many: update to secboot v1 (part 1) [10:04] pedronis: thanks, let me see [10:05] pedronis: #9551 is trivial and needs a pass from you or mvo [10:05] PR #9551: dirs, boot: add ubuntu-save directories and related locations [10:06] mborzecki: sorry, I had a typo in my new code, repushed [10:06] (well in a comment) [10:10] PR snapd#9552 closed: usersession: fix typo in test name [10:18] brb [10:25] PR snapd#9551 closed: dirs, boot: add ubuntu-save directories and related locations [11:20] mvo: mborzecki: could either of you look at my own last 3 commits here (they are small tweaks or error formatting changes): https://github.com/snapcore/secboot/pull/12 [11:20] PR secboot#12: Make policyPCRData serializable [11:20] oops [11:20] mvo: mborzecki: could either of you look at my own last 3 commits here (they are small tweaks or error formatting changes): https://github.com/snapcore/secboot/pull/125 [11:20] PR secboot#125: Add ActivateVolumeWithMultipleTPMSealedKeys [11:21] pedronis: sure, once I'm done with 9418 I can look [11:21] pedronis: sure [11:40] PR snapd#9553 opened: tests: add spread test for refreshing from an old snapd and core18 [12:00] PR snapd#9418 closed: many: implement snap routine console-conf-start for synchronizing auto-refreshes [12:02] mvo: mborzecki: (nested) kernel reseal failed on #9549 [12:02] PR #9549: many: update to secboot v1 (part 1) [12:03] pedronis: hm, also have some trouble unsealing data (if logs are to be believed) in recover mode with 9549 merged to my branch, but that may well be my fault [12:13] hm false alarm, worked on the second run, a base update kicked in between [12:19] that kernel-reseal log is a bit hard to follow [12:20] pedronis: hmm `2020-10-27T11:59:39.1587365Z [0;1;39mPlease enter the recovery key for disk /dev/disk/by-partuuid/2709461b-d8cd-2244-85cb-a924abd353a6:` [12:39] pstolowski, interesting bug in the snapshot code https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1901578 [12:39] Bug #1901578: Cannot remove snap package ubuntu 20.10 [12:39] pstolowski, we should probably skip users without a name [12:39] or warn, but not fail [12:41] zyga: indeed [12:41] mborzecki: are we using the wrong initramfs somehow? gadget-reseal worked unless for some reason is not testing what we think it tests [12:43] zyga: i'll assign to myself [12:43] thanks [13:03] mborzecki, any advice on type=AVC msg=audit(10/27/20 12:31:35.999:23751) : avc: denied { rmdir } for pid=12800 comm=snap-update-ns name=.X11-unix dev="sda2" ino=17778788 scontext=system_u:system_r:snappy_mount_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir permissive=1 [13:03] * zyga will be back in a moment, going to fetch tea [13:04] pedronis: trying to make it work locally, so far no luck [13:05] Error: 2020-10-27 10:04:21 Error executing google:debian-sid-64:tests/main/interfaces-timeserver-control (oct270953-348560) : [13:05] This system cannot use NTP, test precondition failed [13:05] cachio, ^ new image? perhaps the test needs to be adjusted [13:08] pedronis: hm i can't reproduce this, i do the same modifications as the test does, resign the kernel.efi and it boots with the new kernel [13:17] re [13:31] brb, main network link at home has failed [13:33] mborzecki, ping [13:33] mborzecki, we have fs_manage_tmpfs_dirs(snappy_mount_t) [13:33] mborzecki, yet we deny rmdir on scontext=system_u:system_r:snappy_mount_t:s0 [13:33] tcontext=system_u:object_r:tmp_t:s0 [13:34] does this make any sense to you? [13:34] zyga: maybe something more is needed for tmp_t [13:36] zyga: try looking for something that allow managing tmp_t [13:36] zyga: why are we even removing this directory? [13:51] PR snapd#9554 opened: tests/nested/core20/gadget,kernel-reseal: add sanity checks to the reseal tests [14:01] test [14:01] ok [14:26] PR snapcraft#3340 opened: include SNAPCRAFT_STAGE in XDG_DATA_DIRS, needed to find Gir files built by other parts [14:37] cmatsuoka: any ideas what i should look for? [14:38] well that was a first for me [14:38] maybe my headset is dying [14:40] mborzecki: hmm, if the problem is not the in the pcr profile I don't know what it could be [14:40] mborzecki: and I don't think it's in the pcr profile, but I think we should check the measurements just in case [14:42] mborzecki: I'm trying to retrieve the measurement data from the nested image [14:48] cmatsuoka: i got something from the vm i use locally, passed it through this parser: https://github.com/ValdikSS/binary_bios_measurements_parser [14:49] mborzecki: that's the one I'm using too, is it booting correctly on that machine after a reseal? [14:49] mborzecki: if yes the profiles must be correct [14:50] cmatsuoka: yes [14:51] we must get the measurements from the failing system which is the nested vm, perhaps we could try to reproduce the problem on a local nested vm? [14:52] cmatsuoka: go a diff between dumps of state before and after a kernel refresh https://paste.ubuntu.com/p/VjJKcYmrNk/ there are differnces but not quite sure what i should be looking for there [14:53] mborzecki: the PR logs show the PCR profile dump before and after the reseal in the gce nested vm, but we need the measurement data on that same machine to be able to compare [14:54] cmatsuoka: the differences show up after the following entry `chainloader (hd0,gpt3)/EFI/ubuntu/try-kernel.efi ...` and then diferent bytes appear, so i guess it's the old and new kernel measureemnt [14:55] the profile dump in the logs shows what the system is expecting and the measurements show the actual data [14:56] if they match the problem is elsewhere [14:56] cmatsuoka: ok, so the list of final PCRs is the same for both boots here [14:59] mborzecki: I have an 1:1 now, quick chat after that? [15:00] sure [15:01] cachio: zyga-mbp: so i have this project: https://github.com/bboozzoo/spread-mini i've pushed a test which clones snapd and then runs govendor sync, and it's not failing 100% of time on our tumbleweed images [15:02] mborzecki, I tested yesterday and the master fails 8/12 aprox doing govendor sync [15:03] cachio: and then if i keep on cloning and syncing in a loop it works too [15:03] (on a host) [15:04] you are using gce image right? [15:04] cachio: yes, the one we have [15:04] yesterday I published a last image with all the latest updated [15:04] updates [15:04] perhaps it works a bit better [15:12] * cachio lucnch [15:15] * zyga-mbp lunch soon [15:16] not a good weather for walk [15:16] jdstrand do you think you could review a +3 security change https://github.com/snapcore/snapd/pull/9542 [15:16] PR #9542: interfaces: deny connected x11 plugs access to ICE <⚠ Critical> [15:16] er [15:16] +9 -0 [15:16] sorry [15:16] technically three denials [15:17] * zyga-mbp goes to plug the laptop [15:22] https://github.com/snapcore/snapd/pull/9554 <-- trivial sanity checks [15:22] PR #9554: tests/nested/core20/gadget,kernel-reseal: add sanity checks to the reseal tests [15:25] mborzecki: I'm looking at the test, afaik it can't pass for simple reasons [15:26] mborzecki: it's using the kernel from edge and just tweaking it [15:26] is not repacking it with a recent initramfs [15:26] unless I'm reading it wrong [15:27] cmatsuoka: ^ [15:28] pedronis: heh. yes it's not repacking the kernel with new s-b [15:29] * cmatsuoka checks [15:30] another reason to add a prompter interface to secboot [15:30] mborzecki, how did it pass for you earlier? [15:30] it would allow us to maybe log things before asking for a recovery key [15:31] zyga: it was landed later before the last beta, we haven't really changed relevant bits since [15:31] so edge was good until the switch to v1 [15:31] that the PR does [15:31] zyga: and locally it works because i have a signle repacked kernel tree which i reused [15:31] PR snapd#9554 closed: tests/nested/core20/gadget,kernel-reseal: add sanity checks to the reseal tests [15:31] well that explains a lot [15:31] haha [15:32] if we could log the issue before getting stuck with the recovery key asking we would have seen an invalid key file error or something like that [15:33] anyway [15:34] edge has v0-only code, so it can't use v1 keys [15:35] mborzecki: cmatsuoka: we need to add some real repacking I suppose [15:36] or unpack and repack the installed kernel, not the one from edge [15:38] yes, doing it with the installed kernel should be easy [15:39] cmatsuoka: i think it should be around somewhere in /tmp [15:40] mborzecki: could you have a look at it? otherwise I'll check after lunch, I need to finish some cooking right now [15:44] cmatsuoka: yes, trying now [15:46] cmatsuoka: hopefull this is the whole diff ps://paste.ubuntu.com/p/SKvdFKKscj/ [15:46] cmatsuoka: https://paste.ubuntu.com/p/SKvdFKKscj/ [15:51] PR snapd#9550 closed: osutil/disks/mockdisk: panic if same mountpoint shows up again with diff opts [15:55] zyga: I saw one of the mountns inherit spread tests fail on 18.04 like this: [15:56] -+0:+1 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:+1 - cgroup cgroup rw,cpuset [15:56] ++0:+1 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:+1 - cgroup cgroup rw,cpuset,clone_children [15:56] and [15:56] -+0:+1 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:+1 - cgroup2 cgroup rw,nsdelegate [15:56] ++0:+1 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:+1 - cgroup2 cgroup rw [15:56] any idea why the cgroup mounts would be different like that? [16:02] hmm [16:02] that's lxd for sure [16:02] perhaps something is not doing lxc cleanup [16:02] didn't pawel add a lxd test recently? [16:06] eh, damn quoting [16:25] zyga: I did a pass on #9546 , special.go defeated me a bit though [16:25] PR #9546: overlord: add inert export manager [16:29] pedronis, thank you, looking [16:29] pedronis, was special.go confusing? [16:31] maybe it's because it's the end but I find it hard to read [16:31] lots of small helpers [16:35] cmatsuoka: pushed a patch, i think it should work now [16:35] need to run some errands, i'll check back later [16:37] PR snapd#9555 opened: asserts: implement "storage-safety" in uc20 model assertion [16:37] zyga: it was never the intention to decentralize all directory definition, also managers are a prolematic place to define things like that [16:37] mborzeck1: woah, nice! [16:37] mborzeck1: you rock [16:37] pedronis, I see, that's an easy change [16:38] pedronis, let's sync in the morning about remaining topics [16:38] pedronis, some are easy [16:38] pedronis, some I need to understand your point better [16:38] thank you for going through this, it's the hardest part of this PR [16:38] * zyga manged to tame selinux errors [16:38] quick recheck on systems other than f32 [16:39] cachio, f33 released today, perhaps we should swap f31 for f33 [16:47] zyga: I added a couple more questions [16:48] zyga: we can chat at 10am tomorrow if that works for you? [16:50] yeah, that's fine [16:51] grr, me sees a denial on centos-7 that doesn't show up elsewhere [16:55] zyga, yes, I'll create the image [16:57] thanks [16:57] nothing urgent [17:00] pstolowski: I looked again at #9522, we need to chat a bit about it, I'm slightly worried that it improves some things and not others [17:00] PR #9522: o/snapstate: ignore remove errors in clear-snap handler, only log them <⛔ Blocked> [17:01] pedronis: hmm ok, sure [17:01] pstolowski: I explained in the PR [17:02] pstolowski: maybe you can also try to add more motivation to the description, it says what it does but not why [17:02] zyga, f33 has a gcp image now [17:02] I'll try that one [17:02] that's nice [17:02] pedronis: sure, will do [17:02] thx [17:04] mborzeck1: thanks! [17:06] pstolowski: I approved #9535, thanks for the last commits there [17:06] PR #9535: o/snapstate: generate snapd snap wrappers again after restart on refresh [17:07] pedronis: great, thank you [17:16] * zyga stabs his eyes with selinux [17:16] ijohnson: hey, if you could re-review #8395 that would be great (and it's green) [17:16] PR #8395: o/ifacestate: handle interface hooks when preseeding [17:17] pstolowski: yes will add it to my queue, thanks for the reminder [17:24] I really, really dislike selinux tooling [17:24] it's both baroque and complex _and_ imprecise [17:25] applying the policy is more of a whack-a-mole than anything [17:25] so /tmp and /tmp/X11-unix have different types [17:25] but only on centos 7 for some reason [17:26] and I have no idea to connect the dots between the type I see in the FS [17:26] and the abstraction I'm forced to use to write the policy [17:26] I really give up on that now [17:26] I can try tomorrow [17:26] I've wasted half of my day on this already :/ [17:28] I'll talk to maciek tomorrow [17:49] cachio: i've requested your review on https://github.com/snapcore/snapd/pull/9553 [17:49] PR #9553: tests: add spread test for refreshing from an old snapd and core18 [17:49] pstolowski, sure, I'll take a look today [17:49] cachio: i wasn't sure if it should be migrated to nested tool already [17:50] cachio: i think the tool doesn't have execute command yet? [17:53] pstolowski, don't use the nested tool [17:54] I'll migrate that later this week [17:54] cachio: ok, great [18:04] pedronis: added rationale to #9522 [18:04] PR #9522: o/snapstate: ignore remove errors in clear-snap handler, only log them <⛔ Blocked> [18:06] PR snapcraft#3318 closed: plugin handler: set -x for scriptlets [18:16] re === mborzeck1 is now known as mborzecki [18:17] cmatsuoka: i see that the nested uc20 tests are green in #9549 yay [18:17] PR #9549: many: update to secboot v1 (part 1) [18:17] mborzecki: excellent [18:17] cmatsuoka: 14.04 did not build though, may need looking into [18:18] yeah. I'm checking it [18:18] src/github.com/snapcore/snapd/vendor/github.com/snapcore/secboot/keydata.go:43:2: cannot find package "maze.io/x/crypto/afis" [18:18] cmatsuoka: ok, cool, so i'm wrapping it up for today :) [18:52] cachio: we have a problem in a 14.04 test apparently caused by libcurl being unable to connect to certain servers, how do you usually fix this kind of problem? [18:53] cachio: using curl/libcurl from xenial should be enough to make it work again [18:53] cmatsuoka: they are green but also say they took 4s, so something is weird there [18:54] the PR has run nested set [18:54] pedronis: the low times are probably the result of a test re-run [18:56] pedronis: and the trusty test is failing to connect to a certain https server because apparently libcurl is using some older crypto that's no longer supported [18:57] that is a bit of a problem though, 14.04 is still required [18:58] yes, I don't know if we had this sort of problem before or how cachio usually handles them [18:58] using a slightly newer curl/libcurl should solve it but it's not available for trusty [19:02] cmatsuoka: that repo is a bit in an usual location [19:02] maybe we have to mirror it anyway [19:02] yes, maze.io. We could perhaps clone it to github? [19:03] that would be fine I suppose [19:03] the license is MIT [19:08] cmatsuoka, hey [19:08] I'll checkthat [19:08] cachio: thanks. I don't know if we ran into this sort of thing before but I thought perhaps you already have a solution [19:14] cmatsuoka, I think it is the first time with this [19:15] cmatsuoka, it is happeninnig 100% of the time ? [19:15] or it is just something that happens sometimes [19:15] ? [19:16] cachio: it's deterministic [19:16] cmatsuoka, on which pr? [19:16] cachio: it seems that the version of libcurl used in trusty lacks some new crypto used by maze.io [19:17] cachio: it's PR #9549 [19:17] PR #9549: many: update to secboot v1 (part 1) [19:23] cmatsuoka, checking [19:28] cmatsuoka: I made a mirror here (at least temporarely) [19:28] https://github.com/pedronis/maze.io-x-crypto/tree/master/afis [19:29] pedronis: can you also change the secboot vendor.json to use the mirror? [19:29] erm [19:29] no, not needed, we change ours [19:29] ok [19:29]