[00:09] PR snapd#9495 opened: logger: use KernelCommandLineSplit to parse debug flag [03:05] PR snapd#9495 closed: logger: use KernelCommandLineSplit to parse debug flag [05:35] o/ [05:45] PR snapd#9494 closed: logger: use strutil.KernelCommandLineSplit in debugEnabledOnKernelCmdline [05:51] morning [05:51] mborzecki: o/ [05:51] zyga-x240: hey [05:51] mborzecki: I think we should revert bits of the speedup changes [05:51] it's been hanging yesterday [05:51] or investigate and fix [05:51] it may be a python-version-specific bug or just a general bug [05:52] hmmm [05:53] zyga-x240: did the change where we respect workers count land? [05:53] I think so [05:54] afk for some time, lucy just woke up [06:31] PR snapd#9480 closed: snap: support different exit-code in the snap command [06:36] mvo: we need to fix spread-shellcheck [06:38] zyga-x240: tests/unit/go hangs on 20.04 too right? [06:39] zyga-x240, mborzecki yeah, something is wrong, I was just startng a spread google run to see what is going on. do you have an idea already? [06:42] mborzecki: not sure [06:42] mborzecki: I just don't know [06:42] mvo: not immediately, either bug in older python (related to recursive executor submit) or something else [06:43] tryign with --max-procs=2 might be useful [06:46] zyga-x240: otoh, the unit tests job as run by gh actions does not seem to fail or hang for that matter [06:46] yeah, that's what makes me think it may be python version-specific behavior [06:48] I'm finishing my breakfast now, I'll start in a moment [06:49] zyga: looking at one of my PRs, it failed on 20.04, failed on 18.04 (though tests/unit/spread-shellcheck test) hit kill-timeout in both cases, 16.04 passed [06:49] heh [06:49] it's all over the place [06:49] shall we revert and fix this async? [06:50] I'd rewrite the code so that it returns todo units like what jamesh suggested [06:50] then it's one loop and one executor only [06:50] zyga: sounds like more work though ;) [06:50] yes [06:50] zyga: anyways, i think we should revert that last change [06:50] that's why separate actions 1) revert 2) fix [06:51] mvo how does that sound? [06:51] also tests/unit/spread-shellcheck duplicates work right now, run-checks is run in tests/unit/go [06:51] so either we can disable spread-shellcheck in tests/unit/go or drop the other test [06:51] mborzecki: I think we can kill tests/unit/spread-shellcheck now [06:52] mvo: sgtm [06:54] running manually now, those spread nodes have 1 cpu [06:55] zyga: heh, so some deadlock, a number of jobs submitted, nothing happening, cpu usage 0% [06:55] backtrace! [06:55] that's quick [06:57] good morning! [06:58] pstolowski: hello [06:58] zyga-x240: https://paste.ubuntu.com/p/ZqDTCcFvnb/ heh (cc mvo) [06:58] pstolowski: hey [06:58] zyga-x240: only 2 threads and both are waiting [06:59] interesting [06:59] and good idea to use gdb! [07:00] so one is running checkpaths [07:00] going through each location [07:01] while the other is in checkfile [07:01] waiting for the result [07:01] yeah [07:01] mborzecki perhaps just ensuring we have 3 workers miniumum :P [07:01] a bit lame but ... [07:01] (as in the minimum N) [07:01] mborzecki what do you think? [07:05] zyga: --max-procs 3 seems to work [07:06] In the end, you really want the code waiting for futures moved outside of the thread pool [07:07] yeah, I think that's the proper solution [07:11] next step: overengineer it with asyncio [07:14] jamesh no no ;) [07:14] haha [07:15] zyga: but just think of all the threads you'd save! [07:16] mborzecki: nice find! [07:16] jamesh: save all the threads! [07:21] hmm so maybe we just need +1 worker really [07:22] mborzecki I think that's the easy fix [07:22] and we should rewrite that slightly as jamesh mentioned [07:22] zyga: i'd land an easy fix with a comment, and then maybe work on the larger fix/refactor [07:23] +1 [07:23] +1 [07:24] zyga: fwiw i can reproduce this locally with --max-procs=1 [07:24] yeah, my fault for testing on my beefiest system [07:41] zyga: are you opening a quick pr with the workaround? [07:41] mborzecki nope, I thought you want to do that [07:41] I can though [07:41] zyga: no worries, i can push it [07:41] thanks! [07:42] mvo: hi! #8960 got +1 from Samuele and has 3 reviews; would be great to land at the most convienient moment after cutting a new release branch. perhaps it would make sense to squash-merge it in case of anything unexpected (and a need for a revert)? [07:42] PR #8960: o/snapstate,servicestate: use service-control task for service actions (9/9) <⛔ Blocked> [07:47] I think I need to solve the blockers on https://github.com/snapcore/snapd/pull/9204 [07:47] PR #9204: sandbox: track applications unconditionally [07:47] as that's really required to enable r-a-a [07:50] mborzecki: IIRC fedora will disable getenforce/setenforce soon [07:50] how will that affect our test suite? [07:50] zyga-x240: hm, got more info? [07:50] mborzecki: one sec [07:51] https://lwn.net/Articles/831748/ [07:51] PR snapd#9496 opened: spread-shellcheck: temporary workaround for deadlock, drop unnecessary test [07:52] zyga-x240: it's ok for us, we only switch between permissive/enforcing [07:57] zyga-mbp: mvo: ^^ 9496 [08:00] mborzecki: looking [08:01] PR snapd#9497 opened: Have session agent connect to the D-Bus session bus [08:03] zyga-x240: ^^ this PR might help out a bit with your notifications work. [08:04] jamesh: interesting [08:04] I was using the other socket for broadcast but this may be useful as well [08:05] other socket? [08:05] jamesh: the snapd-user-agent socket [08:05] not dbus :) [08:06] why do we have to be launched by systemd? [08:06] ah [08:06] zyga-x240: we're already launched by systemd [08:06] that's a dbus service [08:06] got confused for a sec [08:06] we want to make sure that if we get activated via D-Bus first, we still get our file descriptor [08:06] for REST [08:07] mmm [08:07] I'm not suggesting replacing the snapd <-> agent communication with D-Bus [08:07] this would be for a return path of the agent <-> desktop shell communication [08:07] right [08:10] jamesh: I've sent a quick review just now [08:22] something weird on centos 7 [08:30] hmm [08:30] type=SYSCALL msg=audit(10/13/20 08:11:54.336:587) : arch=x86_64 syscall=kill success=yes exit=0 a0=0xffffffffffffa7e8 a1=SIGKILL a2=0x0 a3=0xf91e50 items=0 ppid=1 pid=22326 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=snapd exe=/usr/libexec/snapd/snapd subj=system_u:system_r:snappy_t:s0 key=(null) [08:30] type=AVC msg=audit(10/13/20 08:11:54.336:587) : avc: denied { sigkill } for pid=22326 comm=snapd scontext=system_u:system_r:snappy_t:s0 tcontext=system_u:system_r:snappy_cli_t:s0 tclass=process permissive=1 [08:31] type=SYSCALL msg=audit(10/13/20 08:01:54.377:396) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x8 a1=0xc000320f10 a2=0x22 a3=0x4 items=0 ppid=22326 pid=22552 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=snap exe=/usr/bin/snap subj=system_u:system_r:snappy_cli_t:s0 key=(null) [08:33] mborzecki: do you know how to recover from: Corrupted checkpoint file. Inode match, but newer complete event (1602577959.287:791) found before loaded checkpoint 1602577772.146:790 [08:33] that's from ausearch [08:34] nvm I got it [08:34] so [08:34] type=AVC msg=audit(10/13/20 08:32:39.287:791) : avc: denied { connectto } for pid=23299 comm=snap path=/run/dbus/system_bus_socket scontext=system_u:system_r:snappy_cli_t:s0 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=1 [08:34] type=AVC msg=audit(10/13/20 08:32:39.287:791) : avc: denied { search } for pid=23299 comm=snap name=dbus dev="tmpfs" ino=13502 scontext=system_u:system_r:snappy_cli_t:s0 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=dir permissive=1 [08:34] those are the first two denials to fix [08:45] * zyga-x240 tries to adjust the policy [08:57] zyga-x240: did you manage to fix it? [08:57] (the policy i mean) [08:58] mborzecki: I'm slow at iterating at this [08:58] not yet [08:59] mborzecki: should we bump the snapd policy version perhaps? :) [08:59] maybe it should match snapd version [09:00] zyga-x240: try with dbus_stream_connect_system_dbusd(snappy_cli_t) and dbus_chat_system_bus(snappy_cli_t) [09:00] zyga-x240: perhaps we should, though we never did [09:01] zyga-x240: setting it to version of snapd sounds ok [09:01] thanks, trying [09:02] zyga-x240: the modules in core policy ahve a version bump on each change (in theory) [09:02] mm [09:02] I suspect nothing just cares about this number [09:02] just an observation, it's not important [09:03] zyga-x240: there maybe some automation under the hood, in our development we replace the module by hand, so it automatically gets a higher priority than the currently loaded one [09:09] mborzecki: I'll read https://github.com/SELinuxProject/refpolicy/blob/master/policy/modules/services/dbus.te and friends to see if there's something we should use [09:12] haha [09:12] zyga-x240: though you really want to read this: https://github.com/SELinuxProject/refpolicy/blob/master/policy/modules/services/dbus.if [09:13] (I meant all three) [09:13] zyga-x240: the *.te (type enforcement?) file is for the actual system dbus daemon, *.fc (file context?) is the files/sockets/dirs [09:13] zyga-x240: *.if is the interfaces for use from other modules [09:13] mmm [09:22] mborzecki: is all the comment XML in those .if files processed by anything? [09:22] is there a "compiled" version anywhere? [09:23] zyga-x240: yes, there should be documentation in your system, though i don't think is avaialble anywhere online [09:23] ah, ok [09:23] I'll check on F32 [09:23] zyga-x240: latest tech :P [09:28] mborzecki: more denials: https://pastebin.ubuntu.com/p/5K5TTpKwkT/ [09:28] I think the blocker is type=AVC msg=audit(10/13/20 09:16:35.588:270) : avc: denied { search } for pid=22502 comm=snap name=dbus dev="tmpfs" ino=13389 scontext=system_u:system_r:snappy_cli_t:s0 tcontext=system_u:object_r:system_dbusd_var_run_t:s0 tclass=dir permissive=1 [09:29] but the rest is also interesting, it seems snapd cannot stop the hook [09:29] (after a timeout) [09:29] yeah, sigkill and all [09:30] zyga-x240: dbus_system_bus_client(snappy_cli_t) should do it [09:30] yep [09:31] PR snapd#9293 closed: snap: auto-import will not try to auto-create users on managed devices [09:31] PR snapd#9498 opened: client,daemon,snap: auto-import does not error on managed devices [09:32] zyga-x240: and you can probably drop dbus_connect_system_bus(snappy_cli_t) looks like it duplicates some of the things from *_bus_client [09:32] ok [09:38] #9463 needs 2nd reviews (it's small) [09:38] PR #9463: seed/seedwriter/writer.go: check DevModeConfinement for dangerous features [09:40] ack [09:44] https://github.com/snapcore/snapd/pull/9463#pullrequestreview-507255123 [09:44] PR #9463: seed/seedwriter/writer.go: check DevModeConfinement for dangerous features [09:44] made one remark, so perhaps something to adjust before landing [09:45] mborzecki: nice, passing [09:45] mborzecki: we need session equivalent though [09:46] I don't see any in the refpolicy gh repo [09:46] mborzecki: rebased and pushed back to https://github.com/snapcore/snapd/pull/9204 [09:46] PR #9204: sandbox: track applications unconditionally [09:55] pedronis: should I push a trivial patch for https://github.com/snapcore/snapd/pull/9463 [09:55] PR #9463: seed/seedwriter/writer.go: check DevModeConfinement for dangerous features [09:55] er [09:55] for https://github.com/snapcore/snapd/pull/9463#discussion_r503813935 [09:55] zyga-x240: if you have time yes [09:56] on it [09:56] mvo: should we close #8845 and #8929 until we have time to discuss/adjust them and reprose? [09:56] PR #8845: [RFC] many: add "system.service.snapd-autoimport.disable" setting <⛔ Blocked> [09:56] PR #8929: [RFC] many: add new "daemon-startup: inhibit" option [09:57] *re-propose [09:57] pedronis: sure [09:58] thx [09:58] closed [10:01] pushed [10:01] https://github.com/snapcore/snapd/pull/9463/commits/26f6b6680027ea9b1262a03b27a28ac6f3d60a9e if anyone wants to cross-check [10:01] PR #9463: seed/seedwriter/writer.go: check DevModeConfinement for dangerous features [10:02] PR snapd#8845 closed: [RFC] many: add "system.service.snapd-autoimport.disable" setting <⛔ Blocked> [10:14] mborzecki: any idea on how to provide sigkill and other permissions [10:14] I think we're missing a test [10:14] that we can kill a hook that is running for too long [10:14] (or miss a test that runs selinux checks) [10:14] mborzecki: do you think we could move the "no denials" check to an invariant? [10:19] zyga-x240: hm this should do it `allow snappy_t snappy_cli_t:process { sigkill };` [10:19] zyga-x240: wondering though, why the hook process was still snappy_cli_t [10:19] can you reproduce that and grab ps -Z ? [10:19] oh [10:19] I will try [10:19] sure [10:21] ah I know why [10:21] mborzecki: because that was still the "snap run" phase [10:21] we got a denialy on dbus method call [10:21] and got stuck waiting for a response [10:22] mborzecki: I wonder if we should setup a timeout / guard of some sort [10:22] but that explains the label [10:22] we switch that in snap-exec [10:22] er [10:22] snap-confine === JanC_ is now known as JanC [10:25] mborzecki: what is the label we transition to when we run as a snap app? [10:25] I added this [10:25] # snapd tries to kill hooks that run for over 10 minutes. [10:25] allow snappy_t snappy_cli_t:process { sigkill }; [10:25] zyga-x240: if it's under systemd the final label is unconfined_service_t [10:25] but I think we need more than that [10:26] not under systemd [10:26] those are specifically hooks [10:26] systemd just tracks them [10:26] not spawns them [10:26] zyga-x240: right, but the transitions take the same route iirc [10:26] zyga-x240: so the hook ends up as unconfined service too [10:28] oh [10:28] ok, that's good then [10:28] I'll add two lines [10:29] * zyga-x240 tests killing w [10:30] with disabled dbus perms [10:38] pedronis: hi, i've spent quite a bit investigation undo on remove, found a couple of issues and filed https://bugs.launchpad.net/snapd/+bug/1899614 (and also worked on addressing point #2 there) [10:38] Bug #1899614: multiple problems with undo for 'snap remove' [10:40] *investigating [11:01] pstolowski: rogpeppe is around and could provide extra information about the other bug we've discussed [11:02] zyga-x240: thanks, but i don't have anything specific atm, but mvo had requested some info there before, would be good to add it if possible [11:02] rogpeppe ^ [11:03] pstolowski: ok, i've added that command output [11:03] thank you! [11:04] thanks! [11:04] i don't seem to get email notifications from launchpad, so please ping me here if you want any further interaction from me in that issue, thanks! [11:05] ah, will do [11:06] ok, that log is very interesting.. we saw " there was a rollback across reboot" before, didn't we? [11:07] aha [11:08] this at least explains the refresh and undo on Oct 5 that I also saw in the state timings; but what happened after next 2 reboots and why did it work till Oct 7th is a mistery [11:08] pstolowski: I don't recall how this layer works, what would happen if the boot partition is corrputed (vfat) and we cannot really set the next boot to anything different [11:08] like it's always stuck at one thing [11:08] would that explain anything? [11:08] i've no idea about that code [11:08] rogpeppe: I think you could try unmounting the boot partition (vfat) and fscking it [11:08] rogpeppe: also if you want to recover the system, we should be able to help [11:08] just not sure what's the best way to do that [11:09] I was also talking to mborzecki about this issue [11:09] and we don't remove snapd.service from disk [11:09] rogpeppe: could you check if snapd.service is in /etc/systemd/system? [11:09] afaiu the system works again, no? logs from oct12 [11:09] rogpeppe: oh, is snapd running now? [11:10] maybe i'm making assumptions.. but afaiu it stopped working on 7th? [11:10] one mo, let me check [11:10] the system isn't working currently [11:11] snapd does seem to be running: [11:11] rogpeppe@localhost:~$ ps alxw | grep snapd [11:11] 4 0 725 1 20 0 928048 18712 - Ssl ? 2:57 /snap/snapd/9169/usr/lib/snapd/snapd [11:11] ! [11:11] rogpeppe: ha, that's weird! [11:11] can you run /snap/snapd/9169/usr/bin/snap? [11:11] rogpeppe: and no /snap/snapd/current symlink right? [11:11] /snap/snapd/current is still not there, right [11:12] i can run /snap/snapd/9169/usr/bin/snap ok [11:12] pstolowski: ^ and ideas on what to explore? [11:12] rogpeppe: maybe snap list --all [11:12] for a first sanity check [11:12] i could give you a login to the system if you want [11:12] then maybe snap install snapd? [11:12] pstolowski: do you want to debug this? [11:14] this is the output of snap list --all: https://paste.ubuntu.com/p/Rwvr2dPD9M/ [11:14] rogpeppe: yes, sure, that would be great [11:15] core 16-2.46.1 9995 latest/stable canonical* core [11:15] core is linked [11:15] do we do anything magical where snapd would start without current symlink after rebooot? (don't think so, but...) [11:15] but the boot base is core18, right? [11:15] rogpeppe: what's /meta/snap.yaml snap name? [11:15] * rogpeppe tries to remember how to grant ssh access. so rusty! [11:15] pstolowski: I don't think we do [11:15] rogpeppe: you can try ssh-import-id [11:15] not sure if it's preinstalled [11:15] that's the command i was trying to remember! [11:16] cool :) [11:16] mvo: ^ very interesting bug [11:16] lots for us to learn on robustness [11:16] pstolowski: is your launchpad username pstolowski ? [11:17] ok, no ssh-import-id command [11:17] rogpeppe: 1 sec, i will import ssh keys from my current box [11:17] :thumbsup: [11:19] zyga-x240: is there more news? [11:19] mvo: yes, we have access to the device [11:19] snapd is disabled! [11:20] woah, how did that happen :( ? [11:20] the boot partition is still corrputed I suspect [11:20] rogpeppe: my lp user is stolowski [11:20] we can get all the logs [11:20] \o/ [11:20] mvo: missing undo in unlink snap, I bet [11:20] but could be something much more complex [11:20] i'm not sure it's this zyga-x240 [11:20] but cannot exclude it of course [11:21] thank you so much rogpeppe ! [11:21] pstolowski: ack [11:21] mvo, zyga-x240 any suggestions what to collect? anything regarding boot? [11:21] * mvo hugs pstolowski and zyga-x240 for their tireless debugging also [11:21] rogpeppe@localhost:~/.ssh$ ed [11:21] -bash: ed: command not found [11:21] dammit! [11:21] pstolowski: maybe to be safe collect all journal logs [11:21] +1 [11:21] rogpeppe: vi is there [11:21] rogpeppe: you can also echo >> [11:21] yeah, i'll use cat [11:21] i don't use an ANSI terminal so vi isn't good for me [11:23] zyga-x240: is tar /var/log.. good enough, or is there a better way? [11:23] thanks rogpeppe, checking [11:23] I think that's good [11:23] pstolowski: you can use journalctl with a standlone directory to examine machine logs without journald itself [11:24] zyga-x240: yeah i did it once.. slightly inconvinent but works [11:25] pstolowski: journalctl -D /path/to/var/log/journal [11:25] and then it works IIRC [11:31] mborzecki: is spread-shellcheck fixed? [11:31] ah [11:31] I see the PR [11:31] thanks! [11:32] approved [11:36] zyga-x240: hey so you mentioned re docker-support/multipass-support being broken with aa3 on groovy that you would prefer a snap-update-ns approach - can you elaborate more on what you are thinking here? I am not sure I understand exactly what you have in mind. [11:36] sure [11:36] I was thinking that the special interfaces they rely on could provide a mount profile that puts the base snap's apparmor config in /etc [11:37] mvo: anything re boot env that can be useful? [11:37] something like mount --bind /snap/core18/current/etc/apparmor.d /etc/apparmor.d [11:37] pstolowski: if you can, try fscking the boot partition [11:37] or [11:37]