[00:01] I am trying to get to dashboard from another computer since Ubuntu core doesn’t have GUI === Eleventh_Doctor is now known as Pharaoh_Atem [05:30] good morning [05:30] wow, I'm early [06:07] morning [06:08] mborzecki: hey [06:08] mborzecki: I think today is get-master-to-work day [06:08] I left some notes on https://github.com/snapcore/snapd/pull/9121 [06:08] PR #9121: github: remove Ubuntu 19.10 from actions workflow <⚠ Critical> [06:08] zyga: hey [06:08] but there are other bugs that essentially make master red - I restarted that branch all the time over weekend [06:08] mborzecki: something is killing/stopping systemd --user for root [06:09] which takes down dbus for root [06:09] and makes snapd fail on installing anything with user services [06:09] so those tests almost always fail [06:09] zyga: which systems are affected? [06:09] this is in addition to the regular set of random failures [06:09] I'm running fedora-32 as that's what I was debugging on Friday [06:09] but it affects ubuntu 20.04 as well [06:10] I think it's more of a luck factor (some systems managed to pass) or a specific-test factor (test only on some system that breaks the world for the rest) [06:10] unsure [06:10] I set workers to one and will have some more info [06:10] from Friday: [06:10]

2020-08-07T15:54:53.3788099Z - google:fedora-32-64:tests/main/snap-user-service [06:10]

2020-08-07T15:54:53.3788764Z - google:fedora-32-64:tests/main/snap-user-service-socket-activation [06:10]

2020-08-07T15:54:53.3797466Z - google:fedora-32-64:tests/main/snap-user-service-start-on-install [06:10]

2020-08-07T15:54:53.3798599Z - google:fedora-32-64:tests/main/snap-user-service-upgrade-failure [06:11] those failed but pass in isolation [06:11] mhm [06:11]

google:fedora-32-64 .../tests/main/snap-mgmt# ls -ld /run/user/0/snapd-session-agent.socket [06:11]

srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/snapd-session-agent.socket [06:11]

google:fedora-32-64 .../tests/main/snap-mgmt# systemctl --user status snapd-session-agent.socket [06:11]

Failed to connect to bus: Connection refused [06:11]

wat? [06:11]

google:fedora-32-64 .../tests/main/snap-mgmt# ls /run/user/0/bus -l [06:11]

srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/bus [06:11] in addition: [06:11]

root 45646 0.0 0.2 27040 9888 ? Ss 19:47 0:00 /usr/bin/python3 /snap/test-snapd-service/x1/bin/start-stop-mode sighup [06:11] there's a test that installs malfunctioning services [06:11] and those clearly leak and just stay behind [06:11] we should fix that as well though it seems harmless [06:12] some more debugging: [06:12]

connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/bus"}, 18) = -1 ECONNREFUSED (Connection refused) [06:12] strace from systemctl --user status ... [06:12] lastly: [06:12]

this is the test sequence: google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/login google:fedora-32-64:tests/main/snapshot-cross-revno google:fedora-32-64:tests/main/interfaces-content-mimic google:fedora-32-64:tests/main/refresh-undo google:fedora-32-64:tests/main/media-sharing google:fedora-32-64:tests/main/snap-switch google:fedora-32-64:tests/main/try-snap-g [06:12] oes-awa [06:12]

y:test_snapd_service google:fedora-32-64:tests/main/security-seccomp google:fedora-32-64:tests/main/core18-with-hooks google:fedora-32-64:tests/main/security-dev-input-event-denied google:fedora-32-64:tests/main/snap-run google:fedora-32-64:tests/main/interfaces-content-mkdir-writable:snap google:fedora-32-64:tests/main/retryable-error google:fedora-32-64:tests/main/snap-handle-link google:fedora-32-64:tests/main/interfaces-location-control [06:12]

google:fedora-32-64:tests/main/interfaces-netlink-audit google:fedora-32-64:tests/main/snap-mgmt google:fedora-32-64 .../tests/runtime-state# [06:12] that's all I know [06:12] I forgot the seed value :/ [06:12] sorry [06:13] oh, and master is still broken due to 19.10 removal [06:13] so perhaps fixes should go to https://github.com/snapcore/snapd/pull/9121 [06:13] PR #9121: github: remove Ubuntu 19.10 from actions workflow <⚠ Critical> [06:13] or mvo could merge that and we could target fixes to individual branches [06:16] mborzecki: I wonder if my fixes didn't break master: I added this to some tests: systemctl --user stop dbus.service [06:16] perhaps that's wrong and after getting dbus activated over the socket, it's impossible to reliably stop it [06:16] note that this was specifically used to stop root's dbus [06:17] zyga: as if once it becomes part of the session, it goes down with a session? [06:17] perhaps [06:17] I really don't know [06:17] we normally have systemd --user [06:17] but not dbus.service (--user for root) [06:17] at least that's what I think, [06:17] need to find the reproducing seed first [06:37] mvo: good morning [06:37] mvo: we need your help [06:38] zyga: good morning [06:38] zyga: how can I help? [06:38] mvo: please merge https://github.com/snapcore/snapd/pull/9121 [06:38] PR #9121: github: remove Ubuntu 19.10 from actions workflow <⚠ Critical> [06:38] mvo: we have several problems breaking master [06:38] that branch was restarted many times over weekend [06:38] mvo: I updated mborzecki on the issue as much as I know [06:38] mvo: hey [06:39] mvo: the short summary is that something is causing systemd --user for root to shut down, breaking all tests that install snaps with user services [06:39] mvo: I'm working on debugging that now [06:40] zyga: ok [06:40] mborzecki: good morning [06:41] zyga: merged [06:43] zyga: is 9119 ready? it looks ok to me and has 2 +1 but is in draft so can't be merged [06:44] PR snapd#9121 closed: github: remove Ubuntu 19.10 from actions workflow <⚠ Critical> [06:44] PR snapd#9123 closed: vendor: update secboot to support dual signed EFI binaries [06:55] re, back from breakfast [06:56] mvo: yeah, it was ready [06:56] thank you for merging, I hope we can fix the remaining issues quickly [06:57] mvo: merging https://github.com/snapcore/snapd/pull/9112 would help as well, somehow I keep hitting that in my debug runs [06:57] PR #9112: tests: run as hightest via tests.session [06:57] oh [06:57] haha, nice [06:58] zyga: I mark 9119 as ready then [06:59] thanks [06:59] I wanted to get a clean pass but I think that's less practical now [06:59] PR snapd#9109 closed: github: run CLA checks on self-hosted workers [06:59] PR snapd#9112 closed: tests: run as hightest via tests.session [06:59] PR snapd#9113 closed: tests: port regression-home-snap-root-owned to tests.session [07:01] I started early today as I have physio at 18;00 [07:01] morning [07:01] good morning Pawel [07:04] pstolowski: hey [07:04] good morning pstolowski [07:04] PR snapd#9104 closed: tests: fix for timing issues on journal-state test [07:04] PR snapd#9119 closed: many: remove usage and creation of hijacked pid cgroup [07:30] brb, [07:30] rebased on master, no luck with fixes yet [07:30] back in 20 [07:44] reproduced [07:45] master [07:45] spread -seed=1597043051 -debug -v google:fedora-32-64:tests/main/ [07:45] this fails on - Make snap "test-snapd-user-service" (unset) available to the system (Post "http://0/v1/service-control": dial unix /run/user/0/snapd-session-agent.socket: connect: connection refused) [07:45] inside 2020-08-10 09:32:24 Error executing google:fedora-32-64:tests/main/snap-user-service (aug100704-587445) : [07:46] I'll try to understand what causes this now [07:47] prior tests on this machine: [07:47] google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/try-snap-goes-away:test_snapd_tools google:fedora-32-64:tests/main/services-stop-timeout google:fedora-32-64:tests/main/interfaces-fuse-support:parallel google:fedora-32-64:tests/main/install-refresh-remove-hooks:regular google:fedora-32-64:tests/main/interfaces-home google:fedora-32-64:tests/main/snap-interface [07:47] google:fedora-32-64:tests/main/debug-sandbox google:fedora-32-64:tests/main/auto-refresh-retry google:fedora-32-64:tests/main/change-errors google:fedora-32-64:tests/main/umask google:fedora-32-64:tests/main/user-session-env google:fedora-32-64:tests/main/security-apparmor google:fedora-32-64:tests/main/parallel-install-services google:fedora-32-64:tests/main/abort google:fedora-32-64:tests/main/interfaces-appstream-metadata google:fedora-32-64:tests/main/r [07:47] etry-network google:fedora-32-64:tests/main/interfaces-network-control-ip-netns google:fedora-32-64:tests/main/snapctl-is-connected google:fedora-32-64:tests/main/cgroup-tracking:root google:fedora-32-64:tests/main/services-disabled-kept-unhappy google:fedora-32-64:tests/main/op-remove-retry google:fedora-32-64:tests/main/revert-devmode:remote google:fedora-32-64:tests/main/base-snaps google:fedora-32-64:tests/main/interfaces-shutdown-introspection [07:47] google:fedora-32-64:tests/main/snapd-notify google:fedora-32-64:tests/main/install-sideload-epochs:reexec1 google:fedora-32-64:tests/main/auto-aliases google:fedora-32-64:tests/main/snap-remove-not-mounted google:fedora-32-64:tests/main/interfaces-netlink-connector google:fedora-32-64:tests/main/debug-confinement google:fedora-32-64:tests/main/services-stop-mode google:fedora-32-64:tests/main/known google:fedora-32-64:tests/main/user-data-handling [07:48] google:fedora-32-64:tests/main/interfaces-framebuffer google:fedora-32-64:tests/main/base-policy google:fedora-32-64:tests/main/chattr google:fedora-32-64:tests/main/interfaces-system-observe google:fedora-32-64:tests/main/searching google:fedora-32-64:tests/main/interfaces-desktop-host-fonts google:fedora-32-64:tests/main/interfaces-hardware-random-observe google:fedora-32-64:tests/main/snap-user-service + echo '# free space' [07:48] (sorry for pasting most of the log directly) [07:49] is pedronis around today? [07:49] I don't recall [07:49] heh, cats fighting outside [07:49] funny noises [07:59] interesting that user@0.service is stopped normally [07:59] Aug 10 07:25:30 aug100704-587445 systemd[912]: Reached target Exit the Session. [07:59] 7:25:30 [08:00] zyga:linger? [08:00] we should log tests [08:00] as in log the start / stop of a test to journal [08:00] this will make it evident what's going on [08:00] possibly [08:01] we had some changes in how linger is enabled [08:01] probably like ~month ago [08:02] worth a try [08:02] heh, 9 files changed, 577 insertions(+), 137 deletions(-), can't really make the content update observer smaller [08:02] tests inflate it a lot [08:03] mborzecki: yeah [08:03] tests.session -u root stops [08:03] stops user@0.service [08:04] hmm [08:04] maybe restore for root [08:04] should remember if we had really started that [08:04] and re-start it [08:04] that would give us working dbus [08:08] ok, I have an idea, let's try to write a patch [08:08] mborzecki: I also wonder what amplified this problem [08:08] was it something we did or was it the dependencies changing [08:09] zyga: afaiu it's observable on stable distros now? [08:09] so i'd bet on something in the tests [08:09] mborzecki: on 20.04 [08:09] so presumably that's us [08:11] mborzecki: but this is so weird [08:11] we disable root linger [08:11] but not stop anything [08:11] maybe the recent changes I did for systemd-logind caused this [08:12] pedronis: hi, i've opened #9124, it's mounted fs updater & gadget.Update(), still the diffstat is +579 −137 [08:12] PR #9124: gadget: introduce content update observer [08:14] PR snapd#9124 opened: gadget: introduce content update observer [08:14] mborzecki: thx [08:20] hmm An error occurred with the instance when trying to launch with 'LXD': Unable to fetch https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz: 404 Not Found. [08:20] at least it does return 404 [08:23] the date is very precise, do we use a fixed image URL? [08:23] perhaps we need to "lxd something" for lxd to find the proper images [08:25] https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz does not 404 [08:25] mborzecki zyga: this is a known issue - https://forum.snapcraft.io/t/snapcraft-with-lxd-pulling-an-outdated-lxc-image/19324/3 - well not sure if the right cpc folks who could fix it know about it but is due to out of date simplestreams metadata [08:26] ah, thanks for sharing that [08:50] I found the problem [08:50] eh [08:50] systemd bugs :/ [08:50] restarting logind is just not a good idea [08:50] I think we need to do two things [08:50] move the workaround for an older systemd bug into the system global prepare [08:50] then combine that with the upgrade of systemd [08:50] and REBOOT [08:50] mborzecki: the problem seems to be that logind doesn't remember sessions [08:51] and when we disable linger [08:51] it forgets it had a root session that was made by spread [08:52] so the recent fix for sid has amplified the problem [08:55] lennart replied to our bug: https://github.com/systemd/systemd/issues/16685#issuecomment-670166505 [08:56] systemd maintainers are sure responsive, that's really nice [08:56] eh [08:57] yeah, confirmed [08:57] restarting logind does lose state [08:57] at least in 245 [08:58] mborzecki: https://github.com/systemd/systemd/issues/16685#issuecomment-671239737 [08:59] mvo: I will try to fix master, at least make it better, in the next hour, sorry for the mess [08:59] it's the fallout from the sid fix indeed [09:07] * zyga tests a fix now [09:26] mborzecki: yet-untested draft of the fix: https://github.com/snapcore/snapd/pull/9125 [09:26] PR #9125: tests: fix issues related to restarting systemd-logind.service <⚠ Critical> [09:27] locally tests are in flight, I'll report back when I get some results [09:30] PR snapd#9125 opened: tests: fix issues related to restarting systemd-logind.service <⚠ Critical> [09:31] can we do something about the issue with lxd images? [09:41] mborzecki: eh, can we REBOOT from project prepare in general? [09:41] I recall arch does [09:41] but it has modern bash [09:41] are we still affected by the bash bug with exit code propagation? [09:41] zyga: iirc REBOOT works in prepare, there were problems in restore [09:42] ah, I see [09:43] oh man [09:43] catch 22 [09:43] core workaround [09:43] and linger workaround [09:44] core16 is a big problem now :/ [09:44] but first, let me go downstairs, I really don't know where to work today [09:44] bedroom is an oven, office a frying pan [10:09] mborzecki: mvo: #9095 needs a 2nd review [10:09] PR #9095: boot/bootstate20: unify commit method impls, rm bootState20MarkSuccessful [10:09] pedronis: it's in my queue [10:25] mborzecki: hmm, it seems we cannot REBOOT from project prepare though [10:25] https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/10735/signedlogcontent/75?urlExpires=2020-08-10T10%3A26%3A15.6985947Z&urlSigningMethod=HMACV1&urlSignature=KBzWVhJfo58j12Er5gL0cjRFu7LXUhaRJ52TCxy8R%2B4%3D [10:25] 2020-08-10T09:39:43.8002986Z [10:26] or am I reading that wrong [10:26] after REBOOT the test fails [10:26] hm [10:26] maybe it's not producing the same exit status as REBOOT? [10:26] but this is weird [10:27] we only have one REBOOT [10:27] the one provided by spread is unset immediately after cloning it [10:27] and this is long after [10:27] but this is still weird, I mean, I'm testing this locally [10:27] zyga: can you try with this https://github.com/snapcore/spread/pull/85 ? [10:27] PR spread#85: spread/client: workaround bash 4.3 subshell errexit issues [10:27] and I surely don't get failures (I was only testing Fedora) [10:28] oh [10:28] so this didn't land!? [10:28] mborzecki: I can but it works for me (I'm using the same spread as our workers do) [10:28] something doesn't make sense here [10:29] hm, fedora 32 failed after 24 minutes [10:29] so perhaps it does work on fedora [10:29] but it fails in a different way [10:29] 2020-08-10T10:01:49.7316727Z ##[error]2020-08-10 10:01:49 Error preparing google:fedora-32-64 (aug100938-989118) : rebooted on aug100938-989118 (google:fedora-32-64) more than 10 times [10:29] like a reboot loop [10:29] yet my local run is at 3/4 of completion without an issue [10:30] I think my code is naive [10:30] I need to have a condition for REBOOT [10:30] PR snapd#9126 opened: boot: introduce current_boot_assets and current_recovery_boot_assets to modeenv [10:30] pedronis: hopefully this is a trvial one ^^ [10:30] how do I check the running version of systemd-logind [10:33] ok I think I have an idea [10:36] mborzecki: how does https://paste.ubuntu.com/p/vj7B5DNf3d/ [10:36] look like [10:37] ouch [10:37] sorry [10:37] back pain [10:37] I tried to work sitting [10:38] mborzecki: we can detect if systemd was updated across 245-246 line [10:38] uh back to bed for now [10:40] ouch [10:41] easier said [10:45] zyga: so when we're running < 246 and update pulls in 246+ we restart? [10:45] zyga: does it happen anywhere outside of arch/tw/sid? [10:46] mborzecki: yeah, that's the idea [10:46] we also restart if we had to reconfigure systemd [10:46] er, logind [10:46] so the actual upgrade happens on sid for now [10:46] but the other condition also happens on many systems [10:46] it's just less frequent [10:47] the bottom line is that restarting logind is just unsafe [10:47] I have no idea how to solve this on 16.04 [10:47] where we need to make mount changes to make logind fixable [10:47] fix it [10:47] restart it to apply the fix [10:47] (that's already wrong then) [10:55] zyga: I think I've addressed all the feedback on #9043 either with code changes or responses to comments. [10:55] PR #9043: daemon: replace access control flags on commands with access checkers [10:55] ack [10:55] I'll look after current master woes [10:56] zyga: as far as naming of the access checkers, I think there is value in keeping the naming consistent with the documentation [11:01] jamesh: perhaps, but only if the name is accurate [11:01] zyga: sure. I'm not saying the docs are immutable [11:32] brb [11:32] got a clean run on fedora 32 [11:32] thinking about core16 options [11:49] mborzecki: yeah, so REBOOT is busted [11:49] https://github.com/snapcore/snapd/pull/9125/checks?check_run_id=966349834 [11:49] PR #9125: tests: fix issues related to restarting systemd-logind.service <⚠ Critical> [11:49] I'm certain we need some kind of workaround for that bash issue [11:49] zyga: have you tried with the spread branch? [11:49] I'll try merging it locally to see if that helps [11:49] not yet [11:50] I will [11:50] if it helps we need to get that reviewed [11:50] I wonder what to do now [11:50] should we revert the restart "fix" for sid [11:50] disable sid for now [11:50] do something else [11:50] mvo: ^ advice welcome [11:50] PR snapd#9127 opened: bootloader: introduce TrustedAssetsBootloader, implement for grub [11:50] pedronis: trusted assets bootloader thingy ^^ [11:51] time to do some reviews [11:53] hm we haven't landed any bits for debug that call uname -a or somesuch? [11:54] mborzecki: we have a PR from cachio [11:54] mhm, just +1'ed it [11:55] zyga: https://github.com/snapcore/snapd/pull/9116/checks?check_run_id=965583423 [11:55] PR #9116: tests: add system information and image information when debug info is displayed [11:55] 2020-08-10T07:31:53.5421376Z Linux aug100702-249153 4.15.0-1080-gcp #90~16.04.1-Ubuntu SMP Fri Jul 10 19:11:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [11:55] PR #90: Default to SocketMode=0660 if no socket mode option is given [11:55] and we have cgroup2fs under /sys/fs/cgroup [11:55] on xenial [11:56] mborzecki: that's ... just wrong? [11:56] how can we get that?! [11:56] is it overmounted [11:56] as in shadowing the tmpfs normally there [11:58] zyga: hm i misread, it's tmpfs, we check whether /sys/fs/cgroup/unified exists, but it does not? [11:58] mborzecki: hmmmm [11:58] that's possible [11:58] though I don't remember what the conditions are [11:58] maybe our check is wrong? [11:59] mborzecki: I reviewed #9124, looks good but couple of questions [11:59] PR #9124: gadget: introduce content update observer