[00:19] thank [00:19] thanks Eighth_Doctor [02:09] how do i go about turning that ./spec file into an RPM? [06:35] morning [06:36] starting around 9, need to run an errand and do groceries before the panicked crowds show up [07:39] mvo: hi, maybe it was already noticed, but it seems we have a real issue with seccomp unit tests on 20.04: https://api.travis-ci.org/v3/job/662738603/log.txt [07:41] pedronis: yeah, I'm checking that right now [07:41] pedronis: also the postrm test is still flaky, a bit annoying [07:43] PR snapd#8262 closed: interfaces/udisks2: also allow Introspection on /org/freedesktop/UDisks/** [07:43] PR snapd#8263 closed: interfaces/udisks2: also allow Introspection on /org/freedesktop/UDisks2/** - 2.44 [07:43] mvo: this test seems also flaky sometimes snap-session-agent-socket-activation [07:43] seems a timing issue [07:44] pedronis: on all OSes or just specific ones? I noticed it only on ubuntu-core-16 [07:44] PR snapd#8261 closed: interfaces: work around apparmor_parser slowness affecting uio (2.44) [07:44] I seen it also on non-core [07:45] pedronis: ok, I see what I can find [07:46] mvo: maybe jamesh can also help with that one [07:52] good morning [07:53] pedronis: this is known, I reported it on tumbleweed about a week ago [07:53] zyga: well we have it on 20.04 now [07:53] https://bugs.launchpad.net/snapd/+bug/1866183 [07:53] Bug #1866183: seccomp tests fail in tumbleweed on 2020-03-05 [07:53] fun :| [07:53] ok [07:54] I had a terrible night, I'm barely awake [07:54] how can I help? [07:54] should I jump at seccomp, at reproducible maas content bug, at something existing like app tracking? [07:54] zyga: I'm looking at this right now [07:54] zyga: maas content bug sounds worthwhile [07:55] zyga: I can reproduce the seccomp one [07:55] mvo: is it segfaulting ? [07:55] I'll get to it [07:55] pedronis: not sure yet, seccomp got upgraded from 2.4.2 to 2.4.3 [07:56] pedronis: this is triggering it [07:56] ok, seccomp upgrade seemed the most likely candidate [07:56] mvo: if you do please check if the list in the bug I referenced is the same as in ubuntu now [07:57] I hope it's the same bug [07:57] zyga: yeah, output looks exactly the same [07:57] zyga: I bet the same lib update happend there too [07:57] yeah, that's a good sign [07:57] whatever it is we will need just one solution [07:58] this is fixed now: https://bugs.launchpad.net/snapd/+bug/1863807 ? [07:59] Bug #1863807: core snap rev 8716 breaks keyboard input in chromium and inkscape [07:59] pedronis: I think it was [07:59] pedronis: it was a change in input handling in ibus or something alike [08:00] pedronis: jamie fixed it a few weeks ago [08:00] mvo: they asked you a question here: https://bugs.launchpad.net/snapd/+bug/1863816 [08:00] Bug #1863816: trusty: snapd deb package does not start systemd [08:03] https://bugs.launchpad.net/snapd/+bug/1867415 is interesting :| [08:03] morning [08:03] Bug #1867415: Hardlinking snaps wastes 400 MB tmpfs RAM in live CDs [08:03] hey pawel [08:03] pstolowski: it's a morning but not particularly good one ;) [08:04] zyga: oh, bugs? [08:04] pstolowski: bugs and spiders [08:05] oh [08:05] public service announcement [08:05] if you are running focal [08:05] be careful not to break your system [08:05] zyga: I assigned this to you: https://bugs.launchpad.net/snapd/+bug/1866855 not sure what's it's overall priority [08:05] Bug #1866855: nvidia driver integration is incompatible with Debian [08:05] pedronis: ack, I can check on my nvidia box [08:05] thank you [08:05] I'll look at it to evaluate after maas bug [08:06] I suspect it might go away if we update snapd in debian though [08:06] probably a worthy goal anyway [08:06] mvo: debian question, is debian in some sort of freeze? new queue is full of things that are many weeks old [08:06] or is that just shortage of ftp masters? [08:06] re about focal: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1866844 [08:07] Bug #1866844: package libc6:amd64 2.31-0ubuntu3 failed to install/upgrade: installed libc6:amd64 package post-installation script subprocess returned error exit status [08:07] 127

[08:07] I got affected, need to spend some time to fix my laptop so that it boots normally again [08:07] I see it's got fixed now, that's really good [08:09] mvo: one you have a moment: https://bugs.launchpad.net/snapd/+bug/1867415 is this medium or high? [08:09] Bug #1867415: Hardlinking snaps wastes 400 MB tmpfs RAM in live CDs [08:09] pedronis: note, we can use statfs to detect overlayfs and switch to symlinks [08:09] it sucks we only learned this the hard way so close to release date :/ [08:20] what was the conclusion on this at the sprint: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1865519 ? [08:20] Bug #1865519: apparmor depends on python3 [08:20] pedronis: I think jamie said that it's not bad that python3 is a dependency but I don't remember if the people in the room had another consistent opinion [08:21] anyway is not a snapd bug strictly [08:24] why is this bug is saying install/update, when it looks like a purge op: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1866932 ? [08:24] Bug #1866932: package snapd 2.44~pre1+20.04 failed to install/upgrade: installed snapd package post-removal script subprocess returned error exit status 1

[08:24] morning [08:25] mvo: got logs from the postrm test? [08:28] mborzecki: yes, one sec [08:28] mborzecki: I was initially suspecting a race, i.e. the output not quite in sync [08:28] mborzecki: but my naive (late night) adding a sleep did not help at all so it's probably something else [08:30] mborzecki: https://api.travis-ci.org/v3/job/662738604/log.txt [08:30] mborzecki: + grep -E 'snap\..*\.(service|timer|socket)' [08:30] â— snap.test-snapd-service-stop-timeout.svc.service not-found failed failed snap.test-snapd-service-stop-timeout.svc.service [08:30] + echo 'found unexpected leftovers' [08:30] found unexpected leftovers [08:30] + exit 1 [08:30] ha a different service this time [08:31] mvo: should be a simple fix [08:31] mborzecki: oh, ok [08:33] zyga: this is not fixed yet https://bugs.launchpad.net/snapd/+bug/1821302 right? I'm marking it just high [08:33] Bug #1821302: /usr/lib/snapd unmounted in app + core18 + core (tooling) use case; possibly on core18 refresh [08:34] pedronis: correct [08:34] zyga: did suse upgrade to libc6 2.31 too? I suspect it's actually the libc6 update that breaks the tests (and I think I have a fix, just making sure I understand it) [08:34] mvo: well, maybe not that simple though, it's the same thing as i investigated last week, there's systemd service state leaking between the tests, even though we call systemctl reset-failed in restore :/ [08:34] mvo: I don't know, let me check [08:34] mborzecki: yeah, that part confused me [08:34] mborzecki: it looks like we explicitly reset it :( [08:35] mborzecki: hence suspecting the race [08:35] mvo: and it's on sid, before the problem was seen on arch [08:35] mvo: but I would expect so, tumbleweed rolls weekly [08:36] booting now [08:37] mvo: systemd 245 here on arch, same in sid [08:37] zyga: what's the current version of systemd in tw? [08:37] mborzecki: still updating, give me a moment [08:38] mvo: again, once you have a moment: should this be won't-fixed also on the package: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1673247 [08:38] Bug #1673247: package snapd 2.23.1 failed to install/upgrade: trying to overwrite '/etc/apparmor.d/usr.lib.snapd.snap-confine', which is also in package snap-confine 2.23.1

Progress by mvo> [08:38] [08:41] mborzecki: focal has systemd 244, so that's the most recent we have [08:42] pstolowski: I opened it now and once I'm done with the test issue will look in detail [08:42] pedronis: ^-- (and thanks for the pointer) [08:42] pstolowski: ups, sorry [08:44] zyga: according to their website also glibc 2.31, I think that's the one that breaks things for us, fixing now [08:50] pstolowski: hi, maybe you can review #8264 when you have a moment [08:50] PR #8264: many: introduce snapdenv to present common snapd env options [08:50] pedronis: sure [08:54] mborzecki: 244 [08:54] zyga@yantra:~> systemctl --version [08:54] systemd 244 (+suse.138.gf8adabc2b1) [08:54] +PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN +PCRE2 default-hierarchy=hybrid [08:54] mvo: glibc version 2.31-2.1 [08:56] zyga: ta [08:57] zyga: thx [09:08] PR snapd#8265 opened: tests: add regression test for MAAS refresh bug [09:08] so some good news [09:08] if mborzecki can help me with selinux denials [09:08] hm? [09:08] I can fix the bug affecting maas by just enabling robust mount ns updates [09:09] i shuld hijack my wife's fedora laptop for a couple of days and try to weed out all the denials if there's any left [09:09] https://github.com/snapcore/snapd/pull/8089 is critical [09:10] PR #8089: features: enable robust mount ns updates [09:10] as in, fixing those denials will really improve snap experience for people [09:11] i investigated one on thursday evening (because there's better to do on a day off), and i think that desktop-helpers were causing a problem by doing cp -a on user-dirs.locale [09:11] mborzecki: not all denials are equal [09:11] if we just get over the hump that allows 8089 to land [09:11] it will mean a lot [09:13] omg, reproduced the problem with postrm-purge in a spread run where i forgot to add -debug /o\ [09:13] mborzecki: ouch :| [09:13] btw, it failed for me today [09:14] zyga: btw, I did work on 8242 end of last week [09:14] yeah, I noticed, I didn't review the changes yet but I plan to soon [09:15] thank you :) [09:15] I looked at snapdenv just now and it looks good [09:15] it's a much needed cleanup [09:17] yea, though it was an indirect reminder that store (and daemon) are still our least loved code [09:19] fwiw, I will work more on 8266 but wanted to unblock you guys [09:19] PR snapd#8266 opened: snap-seccomp: allow mprotect() to unblock the tests [09:28] heh, so with new systemd, there's a unicode dot printed in the sssytemctl line [09:28] ● snap.test-snapd-service-stop-timeout.svc.service not-found failed failed snap.test-snapd-service-stop-timeout.svc.service [09:28] ofc it doesn't match the awk pattern we search [09:29] fun, otoh we are culpable of those kind of changes to info or list, so can't blame [09:30] heh [09:33] mborzecki: mvo: on the UC20 uboot front, I worked a bit on it and #8228 needs 2nd reviews [09:33] PR #8228: boot,image: ARM kernel extract prepare image === ricab_ is now known as ricab [09:35] pedronis: nice, thank you! [09:35] pedronis: we got ourselves a deadlock in the reviews of that PR, looks like each of us pushed a bit there :) [09:38] * mvo will look at this [09:39] mborzecki: well, I think if both mvo and you look at it again, we are ok [09:44] mborzecki: mvo: it's missing a unit test though, but that can come in a follow up, anyway there's the todo and rename as well [09:49] * mvo is in a meeting but will look at the PR after that [09:49] hmm reset-failed does not clear the status though, the unit is still in not-found/failed state, wtf? [09:49] maybe someone in #systemd has a clue [09:59] mborzecki: sergio said he needs to reset twice [10:00] reset what? the service? [10:00] reset failed [10:00] just do it twice [10:00] google:debian-sid-64 .../tests/main/postrm-purge# systemctl reset-failed --force snap.test-snapd-service.test-snapd-service-refuses-to-stop.service [10:00] Failed to reset failed state of unit snap.test-snapd-service.test-snapd-service-refuses-to-stop.service: Unit snap.test-snapd-service.test-snapd-service-refuses-to-stop.service not loaded. [10:00] google:debian-sid-64 .../tests/main/postrm-purge# systemctl reset-failed --force snap.test-snapd-service.test-snapd-service-refuses-to-stop.service [10:01] Failed to reset failed state of unit snap.test-snapd-service.test-snapd-service-refuses-to-stop.service: Unit snap.test-snapd-service.test-snapd-service-refuses-to-stop.service not loaded. [10:01] zyga: ^^ :) [10:01] not loaded [10:01] but does it list as failed? [10:01] weird [10:01] yup [10:01] maybe deamon reload and then? [10:01] however, i can do `systemctl reset-failed` (no params, resets all units) and the status is cleared [10:01] huh [10:01] magic [10:01] or [10:01] maybe the unit is really gone [10:02] and it cannot be referred to by name anymore [10:02] and reset-failed can clear "unreachable" units somehow [10:02] it is, but we were able to reset-failed a unit that was not found before [10:07] ackk: hey [10:07] ackk: some good news :) [10:08] hi zyga, I saw :) [10:08] ackk: please try that and let me know if you run into any issues having that enabled [10:09] zyga, how do I enable? [10:09] ackk: like this https://github.com/snapcore/snapd/pull/8265/files#diff-09aaa35036aab4cfb8a7a8ae90869822R14 [10:09] PR #8265: tests: add regression test for MAAS refresh bug [10:09] oh I see, thanks [10:10] zyga, I'll let you know if I hit the issue again, thanks [10:10] ackk: I just wanted to say I'm very grateful for the bug report with information on how to reproduce [10:10] zyga, is the plan to enable the feature by default in the next release? [10:10] ackk: I know this was plaguing you for many weeks now [10:11] ackk: yes, we just need to adjust the selinux policy because it currently blocks the branch that enables this from landing [10:11] zyga, no problem, happy that it was helpful :) [10:11] zyga, thank you for jumping on it [10:11] my pleasure :) I really like making things robust [10:13] heh [10:14] PR snapd#8267 opened: tests/lib/reset: workaround unicode dot in systemctl output [10:16] now i can't reproduce the reset-failed problem on sid, wth? [10:16] anyways, mvo ^^ 8267 shoudl fix the postrm-purge and snap-mgmt tests [10:16] mborzecki: nice, thank you [10:25] mborzecki: then maybe more than it alone needs to reset-failed [10:39] zyga: suse is using /var/lib/snapd/snap too? [10:39] nope [11:06] mvo, hi WRT https://bugs.launchpad.net/maas/+bug/1867571 (ISTR we talked about the issue in the past) looking at snapd core it seems there is an apparmor rule allowing read access to /run/uuidd/request, is the denial because of the "w" ? [11:07] Bug #1867571: apparmor denied warning for /run/uuidd/request [11:14] ackk: in a meeting right now, will get back to you [11:15] mvo, np, thanks [11:15] ackk: looks like it [11:15] ackk: let me check snapd to be ure [11:15] *sure [11:16] zyga, fwiw it seems you need write access too, as libuuid does write to the socket to get the uuid [11:16] I'm not sure of the protocol [11:16] ackk: you get write access with openvswitch-support [11:16] ackk: everyone else get s read access by default [11:16] zyga, right, but read access is probably useless? [11:17] I don't know [11:17] I suspect not [11:17] it is a socket [11:17] perhaps just reading is enough? [11:18] zyga, https://paste.ubuntu.com/p/kfNs4pkMyS/ is what "uuidgen -t" does [11:18] ackk: as I said, I don't know [11:18] one would have to read the source of uuidd [11:18] and check what's the protocol === valentind_ is now known as valentind [11:25] the mprotect() fix failed aparently in non-ubuntu, does anyone have details? if it's something silly like the postrm-purge failure I can merge and override the failure [11:25] but it looks like I can't see the reason for the restart of the testrun unfortunately [11:26] wasn't me [11:26] appreciate y'alls thoughts on https://forum.snapcraft.io/t/uuidd-apparmor-denial/16013 [11:27] zyga, looking at https://github.com/karelzak/util-linux/blob/master/libuuid/src/uuidd.h#L43-L49 you have to write the type of operation you want, so you need write access [11:28] zyga, I suspect the current rule isn't really working (although libuuid will still work as it falls back to local getrandom() if the socket fails) [11:28] ackk: I see [11:28] * ackk adds comment to sparkiegeek's post [11:28] pedronis: I reviewed 8264 [11:28] ackk, sparkiegeek: thank you [11:28] I suspect it's a jamie topic [11:28] I can change stuff but he needs to ultimately review that anyway [11:31] zyga, can we test locally by hacking apparmor rules for the maas snap? (adding "w" support) [11:31] yes [11:31] always :) [11:31] zyga, how do I do that? [11:31] ackk: you need to edit the appropriate text file [11:31] they are stored in /var/lib/snapd/apparmor/profiles [11:31] and are named after the snap and application name [11:32] the correct one should be easy to locate [11:32] then you need to locate the line that mentions uuidd [11:32] and change "r," to "rw," - mind the comma, they are relevant [11:32] and issue a command to compile and insert the profile into the kernel [11:32] sudo apparmor_parser -r /path/to/that/profile/file [11:32] mind you that snapd will regenerate profiles for many operations [11:32] so your changes may get undone [11:33] let me know if you need ehlp [11:33] *help [11:35] zyga, that seems to fix the issue [11:35] sparkiegeek, ^ [11:35] ackk: ack [11:40] PR snapd#8266 closed: snap-seccomp: allow mprotect() to unblock the tests [11:40] PR snapd#8267 closed: tests/lib/reset: workaround unicode dot in systemctl output === alan_g is now known as alan_g_ [11:45] zyga: thanks, I tried to answer your questions [11:46] looking [11:49] zyga: also notice that the description has a list of TODOs for follow-ups already, this PR is already quite big given what is doing [11:49] ok [11:49] ah [11:49] I missedthe PR message somehow [11:50] the main motivation for this is actually PreseedMode, but yes ideally all SNAPD_/SNAPPY_ vars would be handled like this, buit is quite a bit of work that is hard to justify atm [11:51] at least we have a place now [11:51] pedronis: LGTM, happy to see progress in this area :) [12:02] sparkiegeek: note that 8537ba5b2816ac1a2f77dc03ba709947b11a2531 added it to snap-update-ns profile. it was only in 2049f2c8245da1d97c96fcc76bab871eb0491d23#diff-a34e166c5b3016c122430c5884f41e9b from last week where it was added to the default template. [12:03] mvo, hey [12:03] it still only had 'r' though [12:03] jdstrand: thanks [12:03] has* [12:03] PR snapd#8268 opened: snap-seccomp: robustness improvements [12:04] mvo, snapd,failure is failing also on uc18 na uc20, on PR #8252 [12:04] PR #8252: tests: update test to make snapd snap fixed twice [12:05] guten morgen from the quarantine :-) [12:06] jdstrand: ninja-edit'd post to reference the correct commit [12:06] pedronis: added StoreAcount in #8246 [12:07] PR #8246: client, daemon, overlord/devicestate: structures and stubs for systems API [12:17] Saviq, hey [12:17] cachio: oi! [12:18] I plan to delete fedora 28/9 images from gce, are you using them ? [12:18] mborzecki: reviewed [12:18] I need to clean up a bit [12:18] pedronis: thanks! [12:18] cachio: fire away https://github.com/MirServer/mir/blob/master/.travis.yml#L47 [12:19] nice, thanks [12:19] or more importantly https://github.com/MirServer/mir/blob/master/spread.yaml#L5 [12:20] ijohnson: hello, I pushed a bit forward your PRs on Friday [12:20] hey pedronis [12:20] Great, I see some of them were merged [12:21] * zyga breaks for coffee [12:21] or actually [12:22] maybe I'll grab lunch and then coffee [12:22] ahead of the standup [12:26] PR snapd#8269 opened: apparmor: add rw for uuidd request to default and remove from snap-update-ns [12:27] sparkiegeek, ackk: fyi, ^ [12:28] jdstrand, thanks! [12:28] jdstrand: thank you [12:28] jdstrand, fwiw the openvswitch interface also grants that, so it could maybe be dropped from there? [12:29]