[00:01] <javatexan> I am trying to get to dashboard from another computer since Ubuntu core doesn’t  have GUI
[05:30] <zyga> good morning
[05:30] <zyga> wow, I'm early
[06:07] <mborzecki> morning
[06:08] <zyga> mborzecki: hey
[06:08] <zyga> mborzecki: I think today is get-master-to-work day
[06:08] <zyga> I left some notes on https://github.com/snapcore/snapd/pull/9121
[06:08] <mup> PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121>
[06:08] <mborzecki> zyga: hey
[06:08] <zyga> but there are other bugs that essentially make master red - I restarted that branch all the time over weekend
[06:08] <zyga> mborzecki: something is killing/stopping systemd --user for root
[06:09] <zyga> which takes down dbus for root
[06:09] <zyga> and makes snapd fail on installing anything with user services
[06:09] <zyga> so those tests almost always fail
[06:09] <mborzecki> zyga: which systems are affected?
[06:09] <zyga> this is in addition to the regular set of random failures
[06:09] <zyga> I'm running fedora-32 as that's what I was debugging on Friday
[06:09] <zyga> but it affects ubuntu 20.04 as well
[06:10] <zyga> I think it's more of a luck factor (some systems managed to pass) or a specific-test factor (test only on some system that breaks the world for the rest)
[06:10] <zyga> unsure
[06:10] <zyga> I set workers to one and will have some more info
[06:10] <zyga> from Friday:
 2020-08-07T15:54:53.3788099Z     - google:fedora-32-64:tests/main/snap-user-service
 2020-08-07T15:54:53.3788764Z     - google:fedora-32-64:tests/main/snap-user-service-socket-activation
 2020-08-07T15:54:53.3797466Z     - google:fedora-32-64:tests/main/snap-user-service-start-on-install
 2020-08-07T15:54:53.3798599Z     - google:fedora-32-64:tests/main/snap-user-service-upgrade-failure
[06:11] <zyga> those failed but pass in isolation
[06:11] <mborzecki> mhm
 google:fedora-32-64 .../tests/main/snap-mgmt# ls -ld /run/user/0/snapd-session-agent.socket
 srw-rw-rw-. 1 root root 0 Aug  7 19:46 /run/user/0/snapd-session-agent.socket
 google:fedora-32-64 .../tests/main/snap-mgmt# systemctl --user status snapd-session-agent.socket
 Failed to connect to bus: Connection refused
 wat?
 google:fedora-32-64 .../tests/main/snap-mgmt# ls /run/user/0/bus -l
 srw-rw-rw-. 1 root root 0 Aug  7 19:46 /run/user/0/bus
[06:11] <zyga> in addition:
 root       45646  0.0  0.2  27040  9888 ?        Ss   19:47   0:00 /usr/bin/python3 /snap/test-snapd-service/x1/bin/start-stop-mode sighup
[06:11] <zyga> there's a test that installs malfunctioning services
[06:11] <zyga> and those clearly leak and just stay behind
[06:11] <zyga> we should fix that as well though it seems harmless
[06:12] <zyga> some more debugging:
 connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/bus"}, 18) = -1 ECONNREFUSED (Connection refused)
[06:12] <zyga> strace from systemctl --user status ...
[06:12] <zyga> lastly:
 this is the test sequence: google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/login google:fedora-32-64:tests/main/snapshot-cross-revno google:fedora-32-64:tests/main/interfaces-content-mimic google:fedora-32-64:tests/main/refresh-undo google:fedora-32-64:tests/main/media-sharing google:fedora-32-64:tests/main/snap-switch google:fedora-32-64:tests/main/try-snap-g
[06:12] <zyga> oes-awa
 y:test_snapd_service google:fedora-32-64:tests/main/security-seccomp google:fedora-32-64:tests/main/core18-with-hooks google:fedora-32-64:tests/main/security-dev-input-event-denied google:fedora-32-64:tests/main/snap-run google:fedora-32-64:tests/main/interfaces-content-mkdir-writable:snap google:fedora-32-64:tests/main/retryable-error google:fedora-32-64:tests/main/snap-handle-link google:fedora-32-64:tests/main/interfaces-location-control
 google:fedora-32-64:tests/main/interfaces-netlink-audit google:fedora-32-64:tests/main/snap-mgmt google:fedora-32-64 .../tests/runtime-state#
[06:12] <zyga> that's all I know
[06:12] <zyga> I forgot the seed value :/
[06:12] <zyga> sorry
[06:13] <zyga> oh, and master is still broken due to 19.10 removal
[06:13] <zyga> so perhaps fixes should go to https://github.com/snapcore/snapd/pull/9121
[06:13] <mup> PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121>
[06:13] <zyga> or mvo could merge that and we could target fixes to individual branches
[06:16] <zyga> mborzecki: I wonder if my fixes didn't break master: I added this to some tests: systemctl --user stop dbus.service
[06:16] <zyga> perhaps that's wrong and after getting dbus activated over the socket, it's impossible to reliably stop it
[06:16] <zyga> note that this was specifically used to stop root's dbus
[06:17] <mborzecki> zyga: as if once it becomes part of the session, it goes down with a session?
[06:17] <zyga> perhaps
[06:17] <zyga> I really don't know
[06:17] <zyga> we normally have systemd --user
[06:17] <zyga> but not dbus.service (--user for root)
[06:17] <zyga> at least that's what I think,
[06:17] <zyga> need to find the reproducing seed first
[06:37] <zyga> mvo: good morning
[06:37] <zyga> mvo: we need your help
[06:38] <mvo> zyga: good morning
[06:38] <mvo> zyga: how can I help?
[06:38] <zyga> mvo: please merge https://github.com/snapcore/snapd/pull/9121
[06:38] <mup> PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121>
[06:38] <zyga> mvo: we have several problems breaking master
[06:38] <zyga> that branch was restarted many times over weekend
[06:38] <zyga> mvo: I updated mborzecki on the issue as much as I know
[06:38] <mborzecki> mvo: hey
[06:39] <zyga> mvo: the short summary is that something is causing systemd --user for root to shut down, breaking all tests that install snaps with user services
[06:39] <zyga> mvo: I'm working on debugging that now
[06:40] <mvo> zyga: ok
[06:40] <mvo> mborzecki: good morning
[06:41] <mvo> zyga: merged
[06:43] <mvo> zyga: is 9119 ready? it looks ok to me and has 2 +1 but is in draft so can't be merged
[06:44] <mup> PR snapd#9121 closed: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9121>
[06:44] <mup> PR snapd#9123 closed: vendor: update secboot to support dual signed EFI binaries <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9123>
[06:55] <zyga> re, back from breakfast
[06:56] <zyga> mvo: yeah, it was ready
[06:56] <zyga> thank you for merging, I hope we can fix the remaining issues quickly
[06:57] <zyga> mvo: merging https://github.com/snapcore/snapd/pull/9112 would help as well, somehow I keep hitting that in my debug runs
[06:57] <mup> PR #9112: tests: run as hightest via tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9112>
[06:57] <zyga> oh
[06:57] <zyga> haha, nice
[06:58] <mvo> zyga: I mark 9119 as ready then
[06:59] <zyga> thanks
[06:59] <zyga> I wanted to get a clean pass but I think that's less practical now
[06:59] <mup> PR snapd#9109 closed: github: run CLA checks on self-hosted workers <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9109>
[06:59] <mup> PR snapd#9112 closed: tests: run as hightest via tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9112>
[06:59] <mup> PR snapd#9113 closed: tests: port regression-home-snap-root-owned to tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9113>
[07:01] <zyga> I started early today as I have physio at 18;00
[07:01] <pstolowski> morning
[07:01] <zyga> good morning Pawel
[07:04] <mborzecki> pstolowski: hey
[07:04] <mvo> good morning pstolowski
[07:04] <mup> PR snapd#9104 closed: tests: fix for timing issues on journal-state test <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9104>
[07:04] <mup> PR snapd#9119 closed: many: remove usage and creation of hijacked pid cgroup <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9119>
[07:30] <zyga> brb,
[07:30] <zyga> rebased on master, no luck with fixes yet
[07:30] <zyga> back in 20
[07:44] <zyga> reproduced
[07:45] <zyga> master
[07:45] <zyga> spread -seed=1597043051 -debug -v google:fedora-32-64:tests/main/
[07:45] <zyga> this fails on - Make snap "test-snapd-user-service" (unset) available to the system (Post "http://0/v1/service-control": dial unix /run/user/0/snapd-session-agent.socket: connect: connection refused)
[07:45] <zyga> inside 2020-08-10 09:32:24 Error executing google:fedora-32-64:tests/main/snap-user-service (aug100704-587445) :
[07:46] <zyga> I'll try to understand what causes this now
[07:47] <zyga> prior tests on this machine:
[07:47] <zyga> google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/try-snap-goes-away:test_snapd_tools google:fedora-32-64:tests/main/services-stop-timeout google:fedora-32-64:tests/main/interfaces-fuse-support:parallel google:fedora-32-64:tests/main/install-refresh-remove-hooks:regular google:fedora-32-64:tests/main/interfaces-home google:fedora-32-64:tests/main/snap-interface
[07:47] <zyga> google:fedora-32-64:tests/main/debug-sandbox google:fedora-32-64:tests/main/auto-refresh-retry google:fedora-32-64:tests/main/change-errors google:fedora-32-64:tests/main/umask google:fedora-32-64:tests/main/user-session-env google:fedora-32-64:tests/main/security-apparmor google:fedora-32-64:tests/main/parallel-install-services google:fedora-32-64:tests/main/abort google:fedora-32-64:tests/main/interfaces-appstream-metadata google:fedora-32-64:tests/main/r
[07:47] <zyga> etry-network google:fedora-32-64:tests/main/interfaces-network-control-ip-netns google:fedora-32-64:tests/main/snapctl-is-connected google:fedora-32-64:tests/main/cgroup-tracking:root google:fedora-32-64:tests/main/services-disabled-kept-unhappy google:fedora-32-64:tests/main/op-remove-retry google:fedora-32-64:tests/main/revert-devmode:remote google:fedora-32-64:tests/main/base-snaps google:fedora-32-64:tests/main/interfaces-shutdown-introspection
[07:47] <zyga> google:fedora-32-64:tests/main/snapd-notify google:fedora-32-64:tests/main/install-sideload-epochs:reexec1 google:fedora-32-64:tests/main/auto-aliases google:fedora-32-64:tests/main/snap-remove-not-mounted google:fedora-32-64:tests/main/interfaces-netlink-connector google:fedora-32-64:tests/main/debug-confinement google:fedora-32-64:tests/main/services-stop-mode google:fedora-32-64:tests/main/known google:fedora-32-64:tests/main/user-data-handling
[07:48] <zyga> google:fedora-32-64:tests/main/interfaces-framebuffer google:fedora-32-64:tests/main/base-policy google:fedora-32-64:tests/main/chattr google:fedora-32-64:tests/main/interfaces-system-observe google:fedora-32-64:tests/main/searching google:fedora-32-64:tests/main/interfaces-desktop-host-fonts google:fedora-32-64:tests/main/interfaces-hardware-random-observe google:fedora-32-64:tests/main/snap-user-service + echo '# free space'
[07:48] <zyga> (sorry for pasting most of the log directly)
[07:49] <mborzecki> is pedronis around today?
[07:49] <zyga> I don't recall
[07:49] <zyga> heh, cats fighting outside
[07:49] <zyga> funny noises
[07:59] <zyga> interesting that user@0.service is stopped normally
[07:59] <zyga> Aug 10 07:25:30 aug100704-587445 systemd[912]: Reached target Exit the Session.
[07:59] <zyga> 7:25:30
[08:00] <mborzecki> zyga:linger?
[08:00] <zyga> we should log tests
[08:00] <zyga> as in log the start / stop of a test to journal
[08:00] <zyga> this will make it evident what's going on
[08:00] <zyga> possibly
[08:01] <mborzecki> we had some changes in how linger is enabled
[08:01] <mborzecki> probably like ~month ago
[08:02] <zyga> worth a try
[08:02] <mborzecki> heh, 9 files changed, 577 insertions(+), 137 deletions(-), can't really make the content update observer smaller
[08:02] <mborzecki> tests inflate it a lot
[08:03] <zyga> mborzecki: yeah
[08:03] <zyga> tests.session -u root stops
[08:03] <zyga> stops user@0.service
[08:04] <zyga> hmm
[08:04] <zyga> maybe restore for root
[08:04] <zyga> should remember if we had really started that
[08:04] <zyga> and re-start it
[08:04] <zyga> that would give us working dbus
[08:08] <zyga> ok, I have an idea, let's try to write a patch
[08:08] <zyga> mborzecki: I also wonder what amplified this problem
[08:08] <zyga> was it something we did or was it the dependencies changing
[08:09] <mborzecki> zyga: afaiu it's observable on stable distros now?
[08:09] <mborzecki> so i'd bet on something in the tests
[08:09] <zyga> mborzecki: on 20.04
[08:09] <zyga> so presumably that's us
[08:11] <zyga> mborzecki: but this is so weird
[08:11] <zyga> we disable root linger
[08:11] <zyga> but not stop anything
[08:11] <zyga> maybe the recent changes I did for systemd-logind caused this
[08:12] <mborzecki> pedronis: hi, i've opened #9124, it's mounted fs updater & gadget.Update(), still the diffstat is +579 −137
[08:12] <mup> PR #9124: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124>
[08:14] <mup> PR snapd#9124 opened: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124>
[08:14] <pedronis> mborzecki: thx
[08:20] <mborzecki> hmm An error occurred with the instance when trying to launch with 'LXD': Unable to fetch https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz: 404 Not Found.
[08:20] <mborzecki> at least it does return 404
[08:23] <zyga> the date is very precise, do we use a fixed image URL?
[08:23] <zyga> perhaps we need to "lxd something" for lxd to find the proper images
[08:25] <mborzecki> https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz does not 404
[08:25] <amurray> mborzecki zyga: this is a known issue - https://forum.snapcraft.io/t/snapcraft-with-lxd-pulling-an-outdated-lxc-image/19324/3 - well not sure if the right cpc folks who could fix it know about it but is due to out of date simplestreams metadata
[08:26] <zyga> ah, thanks for sharing that
[08:50] <zyga> I found the problem
[08:50] <zyga> eh
[08:50] <zyga> systemd bugs :/
[08:50] <zyga> restarting logind is just not a good idea
[08:50] <zyga> I think we need to do two things
[08:50] <zyga> move the workaround for an older systemd bug into the system global prepare
[08:50] <zyga> then combine that with the upgrade of systemd
[08:50] <zyga> and REBOOT
[08:50] <zyga> mborzecki: the problem seems to be that logind doesn't remember sessions
[08:51] <zyga> and when we disable linger
[08:51] <zyga> it forgets it had a root session that was made by spread
[08:52] <zyga> so the recent fix for sid has amplified the problem
[08:55] <zyga> lennart replied to our bug: https://github.com/systemd/systemd/issues/16685#issuecomment-670166505
[08:56] <zyga> systemd maintainers are sure responsive, that's really nice
[08:56] <zyga> eh
[08:57] <zyga> yeah, confirmed
[08:57] <zyga> restarting logind does lose state
[08:57] <zyga> at least in 245
[08:58] <zyga> mborzecki: https://github.com/systemd/systemd/issues/16685#issuecomment-671239737
[08:59] <zyga> mvo: I will try to fix master, at least make it better, in the next hour, sorry for the mess
[08:59] <zyga> it's the fallout from the sid fix indeed
[09:07]  * zyga tests a fix now
[09:26] <zyga> mborzecki: yet-untested draft of the fix: https://github.com/snapcore/snapd/pull/9125
[09:26] <mup> PR #9125: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125>
[09:27] <zyga> locally tests are in flight, I'll report back when I get some results
[09:30] <mup> PR snapd#9125 opened: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125>
[09:31] <zyga> can we do something about the issue with lxd images?
[09:41] <zyga> mborzecki: eh, can we REBOOT from project prepare in general?
[09:41] <zyga> I recall arch does
[09:41] <zyga> but it has modern bash
[09:41] <zyga> are we still affected by the bash bug with exit code propagation?
[09:41] <mborzecki> zyga: iirc REBOOT works in prepare, there were problems in restore
[09:42] <zyga> ah, I see
[09:43] <zyga> oh man
[09:43] <zyga> catch 22
[09:43] <zyga> core workaround
[09:43] <zyga> and linger workaround
[09:44] <zyga> core16 is a big problem now :/
[09:44] <zyga> but first, let me go downstairs, I really don't know where to work today
[09:44] <zyga> bedroom is an oven, office a frying pan
[10:09] <pedronis> mborzecki: mvo: #9095 needs a 2nd review
[10:09] <mup> PR #9095: boot/bootstate20: unify commit method impls, rm bootState20MarkSuccessful <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9095>
[10:09] <mborzecki> pedronis: it's in my queue
[10:25] <zyga> mborzecki: hmm, it seems we cannot REBOOT from project prepare though
[10:25] <zyga> https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/10735/signedlogcontent/75?urlExpires=2020-08-10T10%3A26%3A15.6985947Z&urlSigningMethod=HMACV1&urlSignature=KBzWVhJfo58j12Er5gL0cjRFu7LXUhaRJ52TCxy8R%2B4%3D
[10:25] <zyga> 2020-08-10T09:39:43.8002986Z <REBOOT>
[10:26] <zyga> or am I reading that wrong
[10:26] <zyga> after REBOOT the test fails
[10:26] <mborzecki> hm
[10:26] <mborzecki> maybe it's not producing the same exit status as REBOOT?
[10:26] <zyga> but this is weird
[10:27] <zyga> we only have one REBOOT
[10:27] <zyga> the one provided by spread is unset immediately after cloning it
[10:27] <zyga> and this is long after
[10:27] <zyga> but this is still weird, I mean, I'm testing this locally
[10:27] <mborzecki> zyga: can you try with this https://github.com/snapcore/spread/pull/85 ?
[10:27] <mup> PR spread#85: spread/client: workaround bash 4.3 subshell errexit issues <Created by bboozzoo> <https://github.com/snapcore/spread/pull/85>
[10:27] <zyga> and I surely don't get failures (I was only testing Fedora)
[10:28] <zyga> oh
[10:28] <zyga> so this didn't land!?
[10:28] <zyga> mborzecki: I can but it works for me (I'm using the same spread as our workers do)
[10:28] <zyga> something doesn't make sense here
[10:29] <zyga> hm, fedora 32 failed after 24 minutes
[10:29] <zyga> so perhaps it does work on fedora
[10:29] <zyga> but it fails in a different way
[10:29] <zyga> 2020-08-10T10:01:49.7316727Z ##[error]2020-08-10 10:01:49 Error preparing google:fedora-32-64 (aug100938-989118) : rebooted on aug100938-989118 (google:fedora-32-64) more than 10 times
[10:29] <zyga> like a reboot loop
[10:29] <zyga> yet my local run is at 3/4 of completion without an issue
[10:30] <zyga> I think my code is naive
[10:30] <zyga> I need to have a condition for REBOOT
[10:30] <mup> PR snapd#9126 opened: boot: introduce current_boot_assets and current_recovery_boot_assets to modeenv <Simple 😃> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9126>
[10:30] <mborzecki> pedronis: hopefully this is a trvial one ^^
[10:30] <zyga> how do I check the running version of systemd-logind
[10:33] <zyga> ok I think I have an idea
[10:36] <zyga> mborzecki: how does https://paste.ubuntu.com/p/vj7B5DNf3d/
[10:36] <zyga>  look like
[10:37] <zyga> ouch
[10:37] <zyga> sorry
[10:37] <zyga> back pain
[10:37] <zyga> I tried to work sitting
[10:38] <zyga> mborzecki: we can detect if systemd was updated across 245-246 line
[10:38] <zyga> uh back to bed for now
[10:40] <zyga> ouch
[10:41] <zyga> easier said
[10:45] <mborzecki> zyga: so when we're running < 246 and update pulls in 246+ we restart?
[10:45] <mborzecki> zyga: does it happen anywhere outside of arch/tw/sid?
[10:46] <zyga> mborzecki: yeah, that's the idea
[10:46] <zyga> we also restart if we had to reconfigure systemd
[10:46] <zyga> er, logind
[10:46] <zyga> so the actual upgrade happens on sid for now
[10:46] <zyga> but the other condition also happens on many systems
[10:46] <zyga> it's just less frequent
[10:47] <zyga> the bottom line is that restarting logind is just unsafe
[10:47] <zyga> I have no idea how to solve this on 16.04
[10:47] <zyga> where we need to make mount changes to make logind fixable
[10:47] <zyga> fix it
[10:47] <zyga> restart it to apply the fix
[10:47] <zyga> (that's already wrong then)
[10:55] <jamesh> zyga: I think I've addressed all the feedback on #9043 either with code changes or responses to comments.
[10:55] <mup> PR #9043: daemon: replace access control flags on commands with access checkers <Needs security review> <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9043>
[10:55] <zyga> ack
[10:55] <zyga> I'll look after current master woes
[10:56] <jamesh> zyga: as far as naming of the access checkers, I think there is value in keeping the naming consistent with the documentation
[11:01] <zyga> jamesh: perhaps, but only if the name is accurate
[11:01] <jamesh> zyga: sure.  I'm not saying the docs are immutable
[11:32] <zyga> brb
[11:32] <zyga> got a clean run on fedora 32
[11:32] <zyga> thinking about core16 options
[11:49] <zyga> mborzecki: yeah, so REBOOT is busted
[11:49] <zyga> https://github.com/snapcore/snapd/pull/9125/checks?check_run_id=966349834
[11:49] <mup> PR #9125: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125>
[11:49] <zyga> I'm certain we need some kind of workaround for that bash issue
[11:49] <mborzecki> zyga: have you tried with the spread branch?
[11:49] <zyga> I'll try merging it locally to see if that helps
[11:49] <zyga> not yet
[11:50] <zyga> I will
[11:50] <zyga> if it helps we need to get that reviewed
[11:50] <zyga> I wonder what to do now
[11:50] <zyga> should we revert the restart "fix" for sid
[11:50] <zyga> disable sid for now
[11:50] <zyga> do something else
[11:50] <zyga> mvo: ^ advice welcome
[11:50] <mup> PR snapd#9127 opened: bootloader: introduce TrustedAssetsBootloader, implement for grub <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9127>
[11:50] <mborzecki> pedronis: trusted assets bootloader thingy ^^
[11:51] <mborzecki> time to do some reviews
[11:53] <mborzecki> hm we haven't landed any bits for debug that call uname -a or somesuch?
[11:54] <zyga> mborzecki: we have a PR from cachio
[11:54] <mborzecki> mhm, just +1'ed it
[11:55] <mborzecki> zyga: https://github.com/snapcore/snapd/pull/9116/checks?check_run_id=965583423
[11:55] <mup> PR #9116: tests: add system information and image information when debug info is displayed <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9116>
[11:55] <mborzecki> 2020-08-10T07:31:53.5421376Z Linux aug100702-249153 4.15.0-1080-gcp #90~16.04.1-Ubuntu SMP Fri Jul 10 19:11:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[11:55] <mup> PR #90: Default to SocketMode=0660 if no socket mode option is given <Created by mvo5> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/90>
[11:55] <mborzecki> and we have cgroup2fs under /sys/fs/cgroup
[11:55] <mborzecki> on xenial
[11:56] <zyga> mborzecki: that's ... just wrong?
[11:56] <zyga> how can we get that?!
[11:56] <zyga> is it overmounted
[11:56] <zyga> as in shadowing the tmpfs normally there
[11:58] <mborzecki> zyga: hm i misread, it's tmpfs, we check whether /sys/fs/cgroup/unified exists, but it does not?
[11:58] <zyga> mborzecki: hmmmm
[11:58] <zyga> that's possible
[11:58] <zyga> though I don't remember what the conditions are
[11:58] <zyga> maybe our check is wrong?
[11:59] <pedronis> mborzecki: I reviewed #9124, looks good but couple of questions
[11:59] <mup> PR #9124: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124>
[11:59] <mborzecki> pedronis: thanks
[12:05] <mup> PR snapd#9128 opened: tests/main/cgroup-tracking: try to collect some information about cgroups <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9128>
[12:05] <mborzecki> zyga: ^^ some more info
[12:10] <zyga> nice
[12:10] <zyga> let's see what that shows
[12:10] <zyga> today is not a very good day, is it?
[12:10] <zyga> I'm making progress, will try a core 16 idea nonw
[12:10] <zyga> *now
[12:11] <pedronis> mborzecki: question(s) in #9127
[12:11] <mup> PR #9127: bootloader: introduce TrustedAssetsBootloader, implement for grub <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9127>
[12:19] <mborzecki> pedronis: right, that's for the case if one gets a bootloader without NoSlashBoot in options, it should probably be internal error as you indicated, it's not a reasonable use case
[12:34] <mborzecki> pedronis: current_trusted_recovery_boot_assets?
[12:34] <pedronis> mborzecki: ok, let's have the error and see how it goes
[12:35] <mborzecki> instead of current_recovery_trusted_boot_assets
[12:35] <pedronis> that probably reads better
[12:43] <zyga> amazing
[12:43] <zyga> what am I missing
[12:43]  * zyga checks spread source
[12:43] <zyga> nah, I am running master
[12:43] <zyga> ok, lets see if main passes
[12:43] <zyga> core16 did not crash and burn for me locally
[12:44] <ijohnson> morning folks
[12:47] <pedronis> ijohnson: hello
[12:47] <ijohnson> hi pedronis
[12:50] <ijohnson> pedronis: thoughts on moving envRefExtractedKernelBootloaderKernelState and extractedRunKernelImageBootloaderKernelState to a different file, maybe boot/bootstate20_kernelstate.go ?
[12:50] <ijohnson> that would make the bootstate20 file a little bit easier to navigate I think
[12:50] <ijohnson> of course not for my current PR, but a followup
[12:50] <zyga> hey ijohnson :)
[12:51] <ijohnson> hey zyga
[12:54] <pedronis> ijohnson: sounds ok
[12:58] <ijohnson> pedronis: cool I'll try to do that after we land the current PR
[13:15] <zyga> mvo: I've added https://github.com/snapcore/snapd/pull/9129 for 2.46
[13:15] <mup> PR #9129: sandbox/cgroup: remove temporary workaround for multiple cgroup writers <Created by zyga> <https://github.com/snapcore/snapd/pull/9129>
[13:16] <mup> PR snapd#9129 opened: sandbox/cgroup: remove temporary workaround for multiple cgroup writers <Created by zyga> <https://github.com/snapcore/snapd/pull/9129>
[13:25] <zyga> mvo: what would it take to make /var/lib/systemd/ writable on core16 (it is writable on 18 and 20 already) - I'm specifically after /var/lib/systemd/linger so we could have a subset of the directory (just linger) available to help
[13:32] <ijohnson> zyga: just for our tests or IRL too ?
[13:37] <zyga> ijohnson: it's just for out tests but we seem to have done this on core18+ so I guess that's for a reason
[13:37] <zyga> and IRL is a weird way to describe that :)
[13:38] <zyga> brb, small break
[13:43] <pedronis> in general it's easier to have the 3 consistent when there's no strong reason for not
[13:45] <ijohnson> zyga: yeah in that case we should just do it IRL for core snap
[13:47] <zyga> I agree
[13:47] <zyga> consistency
[13:47] <zyga> and less special cases
[13:47] <ijohnson> indeed
[13:55] <zyga> I will break for lunch and an errand and then go for physio
[13:55] <zyga> I will do my best to work when I'm back to prepare the fixes for logind
[14:10] <cachio> pstolowski, ijohnson thanks for the review fo #8942, I already answer the question about the variable
[14:10] <mup> PR #8942: tests: support different images on nested execution <Run nested> <Squash-merge> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8942>
[14:13] <ijohnson> thanks cachio
[14:14] <cachio> ijohnson, sorry for the delay, didn't see that on friday
[14:14] <ijohnson> no worries
[14:17] <pstolowski> cachio: thanks
[14:27] <pedronis> mvo: cachio: are we releasing .3.1 today?
[14:27] <cachio> pedronis, I am waiting confirmation from store team
[14:27] <pedronis> thx
[14:27] <cachio> pedronis, they asked me to wait
[14:47] <cachio> pedronis, progressive release for core snap started
[14:47] <cachio> mvo, ~
[14:47] <mvo> cachio: nice, thanks
[14:47] <pedronis> thx
[14:48] <pedronis> mvo: it would probably good to land #9086 but it needs a 2nd review (it's a cleanup)
[14:48] <mup> PR #9086: many: reorg cmd/snapinfo.go into snap and new client/clientutil <Cleanup :broom:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/9086>
[14:48] <mvo> pedronis: looking
[14:50] <pedronis> mvo: I moved my assertions PR to 2.47
[14:50] <mvo> pedronis: thank you
[15:16] <zyga> made some more progress on core16, should have a green pass soon
[15:16] <zyga> but now physio
[15:16] <zyga> see you in a few hours
[15:18]  * cachio lunch
[15:19] <ijohnson> mborzecki: have you seen the triggerwatch unit test  triggerwatchSuite.TestNoDevsWaitKey fail before?
[15:19] <ijohnson> FAIL: triggerwatch_test.go:80: triggerwatchSuite.TestNoDevsWaitKey
[15:19] <ijohnson> triggerwatch_test.go:87:
[15:19] <ijohnson>     c.Assert(err, IsNil)
[15:19] <ijohnson> ... value *errors.errorString = &errors.errorString{s:"trigger not detected"} ("trigger not detected")
[15:19] <ijohnson> OOPS: 4 passed, 1 FAILED
[15:25] <pedronis> mvo: I forgot to mention last week, did you see this: https://github.com/snapcore/core18/pull/155 ?  (same for the parallel PRs), the name of the dir is wrong
[15:25] <mup> PR core18#155: hooks: fix broken symlink /etc/sysctl.conf.d/99-sysctl.conf <Created by mvo5> <https://github.com/snapcore/core18/pull/155>
[15:26] <mvo> pedronis: oh, missed that, sorry
[15:27] <ijohnson> interestingly a model assertion unit test panic'd on riscv64, seems to be timing related somehow https://pastebin.ubuntu.com/p/7ppt8pTbdz/
[15:28] <ijohnson> cachio: do you need to delete some images on gce again? I see `2020-08-10 14:33:27 Cannot allocate google:opensuse-tumbleweed-64: cannot allocate new Google server google:opensuse-tumbleweed-64 (aug101430-988733): cannot find ready marker in console output for google:opensuse-tumbleweed-64 (aug101430-988733)` in spread runs now
[15:35] <mborzecki> ijohnson: heh, probably a timing issue
[15:35] <mborzecki> was outside for a bit, +35C
[15:39] <mvo> pedronis: I reviewed 9086 now
[16:03] <kenvandine> mvo: what's the status of lzo support?  Is that ready to be used?  What snapd version first supported it?
[16:04] <mvo> kenvandine: for snapd on install it's actually transparent, so any snapd version will work
[16:04] <kenvandine> but not at runtime.  We're talking about injecting an "assumes" in there
[16:05] <mvo> kenvandine: not at runtime? what issue/error do you see with older snapd versions?
[16:05] <mvo> kenvandine: "snap pack --compression=lzo" is more recent
[16:06] <kenvandine> wait... when you launch the snap on an old version of snapd
[16:06] <mvo> kenvandine: but the ability to run lzo snaps should be transparent to snapd
[16:06] <kenvandine> that would work?
[16:06] <mvo> kenvandine: yes
[16:06] <mvo> kenvandine: I don't see why it would not :) maybe I'm missing something
[16:07] <kenvandine> ok, i thought there was talk about needing specific versions
[16:07] <kenvandine> i guess i misunderstood :)
[16:07] <kenvandine> mvo: and no known issues on other distros?  centos, fedora, etc?
[16:08] <mvo> kenvandine: same, as long as they have a kernel that supports lzo for squashfs (which they all do) then this will be fine
[16:08] <kenvandine> ok
[16:09] <kenvandine> mvo: thanks, we're going to start testing it :)
[16:09] <kenvandine> with real snaps
[16:09] <mvo> kenvandine: \o/ nice!
[16:09] <mvo> kenvandine: AIUI chromium is already whitelisted in the reviewtools to allow lzo
[16:09] <kenvandine> ok
[16:10] <kenvandine> Wimpress was suggesting we try a few snapcrafter snaps first, so we'll work that out
[16:10] <kenvandine> and maybe experiment with the gnome platform snap too
[16:30] <cachio> ijohnson,  done
[16:30] <Aavar> How would I go about building a snap of deluge 1.3 for 20.04? Or better yet, would anyone be interested in creating such a snap?
[16:31] <ijohnson> thanks cachio
[16:43] <oerheks> Aavar, there is 2.0.3 already
[16:43] <oerheks> https://snapcraft.io/deluge-lukewh
[17:36]  * zyga-mbp is back from physio but not working ye
[17:36] <zyga-mbp> *yet
[17:37] <mup> PR snapd#9130 opened: [RFC] Support for gadget.yaml "$kernel:" style references <Created by mvo5> <https://github.com/snapcore/snapd/pull/9130>
[17:45] <ijohnson> cachio: I
[17:46] <ijohnson> I'm still seeing that ready marker error in spread with tumbleweed
[17:46] <ijohnson> `2020-08-10 17:21:32 Cannot allocate google:opensuse-tumbleweed-64: cannot allocate new Google server google:opensuse-tumbleweed-64 (aug101718-511919): cannot find ready marker in console output for google:opensuse-tumbleweed-64 (aug101718-511919)`
[17:46] <ijohnson> does it take a while for that to deploy ?
[17:46] <cachio> ijohnson, ah, ok, in that case this is another error
[17:46] <cachio> let me cehck again
[17:46] <ijohnson> ok
[17:46] <cachio> ijohnson, fixed
[17:47] <ijohnson> thanks I'll try again
[17:47] <cachio> I'll need to re create a new image for tumbleweed
[18:33] <zyga> back from physio
[18:33] <zyga> how are things?
[18:34] <zyga> I will get back to 16.04 part of the problem in a moment, let me review public tests first
[18:35] <cachio> zyga, I could reproduce the error in the cgroup-tracking
[18:35] <zyga> cachio: tell me more!
[18:35] <cachio> but still couldn't find which test is producing it
[18:35] <zyga> cachio: I reproduced it a few times
[18:36] <zyga> cachio: but I didn't focus on debugging it yet, I'll try tomorrow - assuming the current issues are fixed
[18:36] <zyga> cachio: we added some detectors to invariant checker and it's not the kernel - it is clearly some other factor
[18:36] <cachio> zyga, what I found is that there is another test which if it executed before the  cgroup-tracking test fail
[18:36] <zyga> cachio: when it happened it was running the 4.15 gce kernel
[18:36] <zyga> yeah, I think that must be the case
[18:37] <zyga> we could find an intersection of sets after a few runs
[18:37] <zyga> I'm sure we'll find it
[18:37] <cachio> zyga, yes
[18:37] <cachio> let me find that
[18:37] <cachio> I'll send you what I find
[18:38] <cachio> zyga, I think in hours I can have that info
[18:39] <zyga> cool, drop me a line or go and fix it if you find the broken test
[18:39] <zyga> I'm no longer always on IRC so mattermost is best for a message box style
[18:39] <cachio> zyga, sure
[18:39] <cachio> I'll let you know
[18:39] <cachio> enjoy your evening
[18:41] <zyga> thank you :)
[20:22] <mup> PR core#116 opened: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116>
[20:23] <cachio> ijohnson, hey
[20:23] <cachio> I see this is uc20 https://paste.ubuntu.com/p/rhzB3sRQyV/
[20:23] <ijohnson> hey cachio
[20:24] <ijohnson> cachio: ah yes that was fixed recently
[20:24] <ijohnson> --purge only works on a single snap
[20:24] <ijohnson> you need to do `snap remove --purge jq` `snap remove --purge jq-core18` etc.
[20:24] <cachio> ijohnson, ah, ok, I'll fix my scripts in that case
[20:24] <cachio> thanks
[22:36] <mup> PR snapcraft#3242 opened: tests: remove expected failure case from legacy integration test <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3242>