| javatexan | I am trying to get to dashboard from another computer since Ubuntu core doesn’t have GUI | 00:01 |
|---|---|---|
| === Eleventh_Doctor is now known as Pharaoh_Atem | ||
| zyga | good morning | 05:30 |
| zyga | wow, I'm early | 05:30 |
| mborzecki | morning | 06:07 |
| zyga | mborzecki: hey | 06:08 |
| zyga | mborzecki: I think today is get-master-to-work day | 06:08 |
| zyga | I left some notes on https://github.com/snapcore/snapd/pull/9121 | 06:08 |
| mup | PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121> | 06:08 |
| mborzecki | zyga: hey | 06:08 |
| zyga | but there are other bugs that essentially make master red - I restarted that branch all the time over weekend | 06:08 |
| zyga | mborzecki: something is killing/stopping systemd --user for root | 06:08 |
| zyga | which takes down dbus for root | 06:09 |
| zyga | and makes snapd fail on installing anything with user services | 06:09 |
| zyga | so those tests almost always fail | 06:09 |
| mborzecki | zyga: which systems are affected? | 06:09 |
| zyga | this is in addition to the regular set of random failures | 06:09 |
| zyga | I'm running fedora-32 as that's what I was debugging on Friday | 06:09 |
| zyga | but it affects ubuntu 20.04 as well | 06:09 |
| zyga | I think it's more of a luck factor (some systems managed to pass) or a specific-test factor (test only on some system that breaks the world for the rest) | 06:10 |
| zyga | unsure | 06:10 |
| zyga | I set workers to one and will have some more info | 06:10 |
| zyga | from Friday: | 06:10 |
| zyga | <zyga-x240> 2020-08-07T15:54:53.3788099Z - google:fedora-32-64:tests/main/snap-user-service | 06:10 |
| zyga | <zyga-x240> 2020-08-07T15:54:53.3788764Z - google:fedora-32-64:tests/main/snap-user-service-socket-activation | 06:10 |
| zyga | <zyga-x240> 2020-08-07T15:54:53.3797466Z - google:fedora-32-64:tests/main/snap-user-service-start-on-install | 06:10 |
| zyga | <zyga-x240> 2020-08-07T15:54:53.3798599Z - google:fedora-32-64:tests/main/snap-user-service-upgrade-failure | 06:10 |
| zyga | those failed but pass in isolation | 06:11 |
| mborzecki | mhm | 06:11 |
| zyga | <zyga-x240> google:fedora-32-64 .../tests/main/snap-mgmt# ls -ld /run/user/0/snapd-session-agent.socket | 06:11 |
| zyga | <zyga-x240> srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/snapd-session-agent.socket | 06:11 |
| zyga | <zyga-x240> google:fedora-32-64 .../tests/main/snap-mgmt# systemctl --user status snapd-session-agent.socket | 06:11 |
| zyga | <zyga-x240> Failed to connect to bus: Connection refused | 06:11 |
| zyga | <zyga-x240> wat? | 06:11 |
| zyga | <zyga-x240> google:fedora-32-64 .../tests/main/snap-mgmt# ls /run/user/0/bus -l | 06:11 |
| zyga | <zyga-x240> srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/bus | 06:11 |
| zyga | in addition: | 06:11 |
| zyga | <zyga-x240> root 45646 0.0 0.2 27040 9888 ? Ss 19:47 0:00 /usr/bin/python3 /snap/test-snapd-service/x1/bin/start-stop-mode sighup | 06:11 |
| zyga | there's a test that installs malfunctioning services | 06:11 |
| zyga | and those clearly leak and just stay behind | 06:11 |
| zyga | we should fix that as well though it seems harmless | 06:11 |
| zyga | some more debugging: | 06:12 |
| zyga | <zyga-x240> connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/bus"}, 18) = -1 ECONNREFUSED (Connection refused) | 06:12 |
| zyga | strace from systemctl --user status ... | 06:12 |
| zyga | lastly: | 06:12 |
| zyga | <zyga-x240> this is the test sequence: google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/login google:fedora-32-64:tests/main/snapshot-cross-revno google:fedora-32-64:tests/main/interfaces-content-mimic google:fedora-32-64:tests/main/refresh-undo google:fedora-32-64:tests/main/media-sharing google:fedora-32-64:tests/main/snap-switch google:fedora-32-64:tests/main/try-snap-g | 06:12 |
| zyga | oes-awa | 06:12 |
| zyga | <zyga-x240> y:test_snapd_service google:fedora-32-64:tests/main/security-seccomp google:fedora-32-64:tests/main/core18-with-hooks google:fedora-32-64:tests/main/security-dev-input-event-denied google:fedora-32-64:tests/main/snap-run google:fedora-32-64:tests/main/interfaces-content-mkdir-writable:snap google:fedora-32-64:tests/main/retryable-error google:fedora-32-64:tests/main/snap-handle-link google:fedora-32-64:tests/main/interfaces-location-control | 06:12 |
| zyga | <zyga-x240> google:fedora-32-64:tests/main/interfaces-netlink-audit google:fedora-32-64:tests/main/snap-mgmt google:fedora-32-64 .../tests/runtime-state# | 06:12 |
| zyga | that's all I know | 06:12 |
| zyga | I forgot the seed value :/ | 06:12 |
| zyga | sorry | 06:12 |
| zyga | oh, and master is still broken due to 19.10 removal | 06:13 |
| zyga | so perhaps fixes should go to https://github.com/snapcore/snapd/pull/9121 | 06:13 |
| mup | PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121> | 06:13 |
| zyga | or mvo could merge that and we could target fixes to individual branches | 06:13 |
| zyga | mborzecki: I wonder if my fixes didn't break master: I added this to some tests: systemctl --user stop dbus.service | 06:16 |
| zyga | perhaps that's wrong and after getting dbus activated over the socket, it's impossible to reliably stop it | 06:16 |
| zyga | note that this was specifically used to stop root's dbus | 06:16 |
| mborzecki | zyga: as if once it becomes part of the session, it goes down with a session? | 06:17 |
| zyga | perhaps | 06:17 |
| zyga | I really don't know | 06:17 |
| zyga | we normally have systemd --user | 06:17 |
| zyga | but not dbus.service (--user for root) | 06:17 |
| zyga | at least that's what I think, | 06:17 |
| zyga | need to find the reproducing seed first | 06:17 |
| zyga | mvo: good morning | 06:37 |
| zyga | mvo: we need your help | 06:37 |
| mvo | zyga: good morning | 06:38 |
| mvo | zyga: how can I help? | 06:38 |
| zyga | mvo: please merge https://github.com/snapcore/snapd/pull/9121 | 06:38 |
| mup | PR #9121: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9121> | 06:38 |
| zyga | mvo: we have several problems breaking master | 06:38 |
| zyga | that branch was restarted many times over weekend | 06:38 |
| zyga | mvo: I updated mborzecki on the issue as much as I know | 06:38 |
| mborzecki | mvo: hey | 06:38 |
| zyga | mvo: the short summary is that something is causing systemd --user for root to shut down, breaking all tests that install snaps with user services | 06:39 |
| zyga | mvo: I'm working on debugging that now | 06:39 |
| mvo | zyga: ok | 06:40 |
| mvo | mborzecki: good morning | 06:40 |
| mvo | zyga: merged | 06:41 |
| mvo | zyga: is 9119 ready? it looks ok to me and has 2 +1 but is in draft so can't be merged | 06:43 |
| mup | PR snapd#9121 closed: github: remove Ubuntu 19.10 from actions workflow <Simple 😃> <⚠ Critical> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9121> | 06:44 |
| mup | PR snapd#9123 closed: vendor: update secboot to support dual signed EFI binaries <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9123> | 06:44 |
| zyga | re, back from breakfast | 06:55 |
| zyga | mvo: yeah, it was ready | 06:56 |
| zyga | thank you for merging, I hope we can fix the remaining issues quickly | 06:56 |
| zyga | mvo: merging https://github.com/snapcore/snapd/pull/9112 would help as well, somehow I keep hitting that in my debug runs | 06:57 |
| mup | PR #9112: tests: run as hightest via tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9112> | 06:57 |
| zyga | oh | 06:57 |
| zyga | haha, nice | 06:57 |
| mvo | zyga: I mark 9119 as ready then | 06:58 |
| zyga | thanks | 06:59 |
| zyga | I wanted to get a clean pass but I think that's less practical now | 06:59 |
| mup | PR snapd#9109 closed: github: run CLA checks on self-hosted workers <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9109> | 06:59 |
| mup | PR snapd#9112 closed: tests: run as hightest via tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9112> | 06:59 |
| mup | PR snapd#9113 closed: tests: port regression-home-snap-root-owned to tests.session <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9113> | 06:59 |
| zyga | I started early today as I have physio at 18;00 | 07:01 |
| pstolowski | morning | 07:01 |
| zyga | good morning Pawel | 07:01 |
| mborzecki | pstolowski: hey | 07:04 |
| mvo | good morning pstolowski | 07:04 |
| mup | PR snapd#9104 closed: tests: fix for timing issues on journal-state test <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9104> | 07:04 |
| mup | PR snapd#9119 closed: many: remove usage and creation of hijacked pid cgroup <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9119> | 07:04 |
| zyga | brb, | 07:30 |
| zyga | rebased on master, no luck with fixes yet | 07:30 |
| zyga | back in 20 | 07:30 |
| zyga | reproduced | 07:44 |
| zyga | master | 07:45 |
| zyga | spread -seed=1597043051 -debug -v google:fedora-32-64:tests/main/ | 07:45 |
| zyga | this fails on - Make snap "test-snapd-user-service" (unset) available to the system (Post "http://0/v1/service-control": dial unix /run/user/0/snapd-session-agent.socket: connect: connection refused) | 07:45 |
| zyga | inside 2020-08-10 09:32:24 Error executing google:fedora-32-64:tests/main/snap-user-service (aug100704-587445) : | 07:45 |
| zyga | I'll try to understand what causes this now | 07:46 |
| zyga | prior tests on this machine: | 07:47 |
| zyga | google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/try-snap-goes-away:test_snapd_tools google:fedora-32-64:tests/main/services-stop-timeout google:fedora-32-64:tests/main/interfaces-fuse-support:parallel google:fedora-32-64:tests/main/install-refresh-remove-hooks:regular google:fedora-32-64:tests/main/interfaces-home google:fedora-32-64:tests/main/snap-interface | 07:47 |
| zyga | google:fedora-32-64:tests/main/debug-sandbox google:fedora-32-64:tests/main/auto-refresh-retry google:fedora-32-64:tests/main/change-errors google:fedora-32-64:tests/main/umask google:fedora-32-64:tests/main/user-session-env google:fedora-32-64:tests/main/security-apparmor google:fedora-32-64:tests/main/parallel-install-services google:fedora-32-64:tests/main/abort google:fedora-32-64:tests/main/interfaces-appstream-metadata google:fedora-32-64:tests/main/r | 07:47 |
| zyga | etry-network google:fedora-32-64:tests/main/interfaces-network-control-ip-netns google:fedora-32-64:tests/main/snapctl-is-connected google:fedora-32-64:tests/main/cgroup-tracking:root google:fedora-32-64:tests/main/services-disabled-kept-unhappy google:fedora-32-64:tests/main/op-remove-retry google:fedora-32-64:tests/main/revert-devmode:remote google:fedora-32-64:tests/main/base-snaps google:fedora-32-64:tests/main/interfaces-shutdown-introspection | 07:47 |
| zyga | google:fedora-32-64:tests/main/snapd-notify google:fedora-32-64:tests/main/install-sideload-epochs:reexec1 google:fedora-32-64:tests/main/auto-aliases google:fedora-32-64:tests/main/snap-remove-not-mounted google:fedora-32-64:tests/main/interfaces-netlink-connector google:fedora-32-64:tests/main/debug-confinement google:fedora-32-64:tests/main/services-stop-mode google:fedora-32-64:tests/main/known google:fedora-32-64:tests/main/user-data-handling | 07:47 |
| zyga | google:fedora-32-64:tests/main/interfaces-framebuffer google:fedora-32-64:tests/main/base-policy google:fedora-32-64:tests/main/chattr google:fedora-32-64:tests/main/interfaces-system-observe google:fedora-32-64:tests/main/searching google:fedora-32-64:tests/main/interfaces-desktop-host-fonts google:fedora-32-64:tests/main/interfaces-hardware-random-observe google:fedora-32-64:tests/main/snap-user-service + echo '# free space' | 07:48 |
| zyga | (sorry for pasting most of the log directly) | 07:48 |
| mborzecki | is pedronis around today? | 07:49 |
| zyga | I don't recall | 07:49 |
| zyga | heh, cats fighting outside | 07:49 |
| zyga | funny noises | 07:49 |
| zyga | interesting that user@0.service is stopped normally | 07:59 |
| zyga | Aug 10 07:25:30 aug100704-587445 systemd[912]: Reached target Exit the Session. | 07:59 |
| zyga | 7:25:30 | 07:59 |
| mborzecki | zyga:linger? | 08:00 |
| zyga | we should log tests | 08:00 |
| zyga | as in log the start / stop of a test to journal | 08:00 |
| zyga | this will make it evident what's going on | 08:00 |
| zyga | possibly | 08:00 |
| mborzecki | we had some changes in how linger is enabled | 08:01 |
| mborzecki | probably like ~month ago | 08:01 |
| zyga | worth a try | 08:02 |
| mborzecki | heh, 9 files changed, 577 insertions(+), 137 deletions(-), can't really make the content update observer smaller | 08:02 |
| mborzecki | tests inflate it a lot | 08:02 |
| zyga | mborzecki: yeah | 08:03 |
| zyga | tests.session -u root stops | 08:03 |
| zyga | stops user@0.service | 08:03 |
| zyga | hmm | 08:04 |
| zyga | maybe restore for root | 08:04 |
| zyga | should remember if we had really started that | 08:04 |
| zyga | and re-start it | 08:04 |
| zyga | that would give us working dbus | 08:04 |
| zyga | ok, I have an idea, let's try to write a patch | 08:08 |
| zyga | mborzecki: I also wonder what amplified this problem | 08:08 |
| zyga | was it something we did or was it the dependencies changing | 08:08 |
| mborzecki | zyga: afaiu it's observable on stable distros now? | 08:09 |
| mborzecki | so i'd bet on something in the tests | 08:09 |
| zyga | mborzecki: on 20.04 | 08:09 |
| zyga | so presumably that's us | 08:09 |
| zyga | mborzecki: but this is so weird | 08:11 |
| zyga | we disable root linger | 08:11 |
| zyga | but not stop anything | 08:11 |
| zyga | maybe the recent changes I did for systemd-logind caused this | 08:11 |
| mborzecki | pedronis: hi, i've opened #9124, it's mounted fs updater & gadget.Update(), still the diffstat is +579 −137 | 08:12 |
| mup | PR #9124: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124> | 08:12 |
| mup | PR snapd#9124 opened: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124> | 08:14 |
| pedronis | mborzecki: thx | 08:14 |
| mborzecki | hmm An error occurred with the instance when trying to launch with 'LXD': Unable to fetch https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz: 404 Not Found. | 08:20 |
| mborzecki | at least it does return 404 | 08:20 |
| zyga | the date is very precise, do we use a fixed image URL? | 08:23 |
| zyga | perhaps we need to "lxd something" for lxd to find the proper images | 08:23 |
| mborzecki | https://cloud-images.ubuntu.com/buildd/daily/xenial/20200728/xenial-server-cloudimg-amd64-lxd_combined.tar.gz does not 404 | 08:25 |
| amurray | mborzecki zyga: this is a known issue - https://forum.snapcraft.io/t/snapcraft-with-lxd-pulling-an-outdated-lxc-image/19324/3 - well not sure if the right cpc folks who could fix it know about it but is due to out of date simplestreams metadata | 08:25 |
| zyga | ah, thanks for sharing that | 08:26 |
| zyga | I found the problem | 08:50 |
| zyga | eh | 08:50 |
| zyga | systemd bugs :/ | 08:50 |
| zyga | restarting logind is just not a good idea | 08:50 |
| zyga | I think we need to do two things | 08:50 |
| zyga | move the workaround for an older systemd bug into the system global prepare | 08:50 |
| zyga | then combine that with the upgrade of systemd | 08:50 |
| zyga | and REBOOT | 08:50 |
| zyga | mborzecki: the problem seems to be that logind doesn't remember sessions | 08:50 |
| zyga | and when we disable linger | 08:51 |
| zyga | it forgets it had a root session that was made by spread | 08:51 |
| zyga | so the recent fix for sid has amplified the problem | 08:52 |
| zyga | lennart replied to our bug: https://github.com/systemd/systemd/issues/16685#issuecomment-670166505 | 08:55 |
| zyga | systemd maintainers are sure responsive, that's really nice | 08:56 |
| zyga | eh | 08:56 |
| zyga | yeah, confirmed | 08:57 |
| zyga | restarting logind does lose state | 08:57 |
| zyga | at least in 245 | 08:57 |
| zyga | mborzecki: https://github.com/systemd/systemd/issues/16685#issuecomment-671239737 | 08:58 |
| zyga | mvo: I will try to fix master, at least make it better, in the next hour, sorry for the mess | 08:59 |
| zyga | it's the fallout from the sid fix indeed | 08:59 |
| * zyga tests a fix now | 09:07 | |
| zyga | mborzecki: yet-untested draft of the fix: https://github.com/snapcore/snapd/pull/9125 | 09:26 |
| mup | PR #9125: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125> | 09:26 |
| zyga | locally tests are in flight, I'll report back when I get some results | 09:27 |
| mup | PR snapd#9125 opened: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125> | 09:30 |
| zyga | can we do something about the issue with lxd images? | 09:31 |
| zyga | mborzecki: eh, can we REBOOT from project prepare in general? | 09:41 |
| zyga | I recall arch does | 09:41 |
| zyga | but it has modern bash | 09:41 |
| zyga | are we still affected by the bash bug with exit code propagation? | 09:41 |
| mborzecki | zyga: iirc REBOOT works in prepare, there were problems in restore | 09:41 |
| zyga | ah, I see | 09:42 |
| zyga | oh man | 09:43 |
| zyga | catch 22 | 09:43 |
| zyga | core workaround | 09:43 |
| zyga | and linger workaround | 09:43 |
| zyga | core16 is a big problem now :/ | 09:44 |
| zyga | but first, let me go downstairs, I really don't know where to work today | 09:44 |
| zyga | bedroom is an oven, office a frying pan | 09:44 |
| pedronis | mborzecki: mvo: #9095 needs a 2nd review | 10:09 |
| mup | PR #9095: boot/bootstate20: unify commit method impls, rm bootState20MarkSuccessful <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9095> | 10:09 |
| mborzecki | pedronis: it's in my queue | 10:09 |
| zyga | mborzecki: hmm, it seems we cannot REBOOT from project prepare though | 10:25 |
| zyga | https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/10735/signedlogcontent/75?urlExpires=2020-08-10T10%3A26%3A15.6985947Z&urlSigningMethod=HMACV1&urlSignature=KBzWVhJfo58j12Er5gL0cjRFu7LXUhaRJ52TCxy8R%2B4%3D | 10:25 |
| zyga | 2020-08-10T09:39:43.8002986Z <REBOOT> | 10:25 |
| zyga | or am I reading that wrong | 10:26 |
| zyga | after REBOOT the test fails | 10:26 |
| mborzecki | hm | 10:26 |
| mborzecki | maybe it's not producing the same exit status as REBOOT? | 10:26 |
| zyga | but this is weird | 10:26 |
| zyga | we only have one REBOOT | 10:27 |
| zyga | the one provided by spread is unset immediately after cloning it | 10:27 |
| zyga | and this is long after | 10:27 |
| zyga | but this is still weird, I mean, I'm testing this locally | 10:27 |
| mborzecki | zyga: can you try with this https://github.com/snapcore/spread/pull/85 ? | 10:27 |
| mup | PR spread#85: spread/client: workaround bash 4.3 subshell errexit issues <Created by bboozzoo> <https://github.com/snapcore/spread/pull/85> | 10:27 |
| zyga | and I surely don't get failures (I was only testing Fedora) | 10:27 |
| zyga | oh | 10:28 |
| zyga | so this didn't land!? | 10:28 |
| zyga | mborzecki: I can but it works for me (I'm using the same spread as our workers do) | 10:28 |
| zyga | something doesn't make sense here | 10:28 |
| zyga | hm, fedora 32 failed after 24 minutes | 10:29 |
| zyga | so perhaps it does work on fedora | 10:29 |
| zyga | but it fails in a different way | 10:29 |
| zyga | 2020-08-10T10:01:49.7316727Z ##[error]2020-08-10 10:01:49 Error preparing google:fedora-32-64 (aug100938-989118) : rebooted on aug100938-989118 (google:fedora-32-64) more than 10 times | 10:29 |
| zyga | like a reboot loop | 10:29 |
| zyga | yet my local run is at 3/4 of completion without an issue | 10:29 |
| zyga | I think my code is naive | 10:30 |
| zyga | I need to have a condition for REBOOT | 10:30 |
| mup | PR snapd#9126 opened: boot: introduce current_boot_assets and current_recovery_boot_assets to modeenv <Simple 😃> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9126> | 10:30 |
| mborzecki | pedronis: hopefully this is a trvial one ^^ | 10:30 |
| zyga | how do I check the running version of systemd-logind | 10:30 |
| zyga | ok I think I have an idea | 10:33 |
| zyga | mborzecki: how does https://paste.ubuntu.com/p/vj7B5DNf3d/ | 10:36 |
| zyga | look like | 10:36 |
| zyga | ouch | 10:37 |
| zyga | sorry | 10:37 |
| zyga | back pain | 10:37 |
| zyga | I tried to work sitting | 10:37 |
| zyga | mborzecki: we can detect if systemd was updated across 245-246 line | 10:38 |
| zyga | uh back to bed for now | 10:38 |
| zyga | ouch | 10:40 |
| zyga | easier said | 10:41 |
| mborzecki | zyga: so when we're running < 246 and update pulls in 246+ we restart? | 10:45 |
| mborzecki | zyga: does it happen anywhere outside of arch/tw/sid? | 10:45 |
| zyga | mborzecki: yeah, that's the idea | 10:46 |
| zyga | we also restart if we had to reconfigure systemd | 10:46 |
| zyga | er, logind | 10:46 |
| zyga | so the actual upgrade happens on sid for now | 10:46 |
| zyga | but the other condition also happens on many systems | 10:46 |
| zyga | it's just less frequent | 10:46 |
| zyga | the bottom line is that restarting logind is just unsafe | 10:47 |
| zyga | I have no idea how to solve this on 16.04 | 10:47 |
| zyga | where we need to make mount changes to make logind fixable | 10:47 |
| zyga | fix it | 10:47 |
| zyga | restart it to apply the fix | 10:47 |
| zyga | (that's already wrong then) | 10:47 |
| jamesh | zyga: I think I've addressed all the feedback on #9043 either with code changes or responses to comments. | 10:55 |
| mup | PR #9043: daemon: replace access control flags on commands with access checkers <Needs security review> <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9043> | 10:55 |
| zyga | ack | 10:55 |
| zyga | I'll look after current master woes | 10:55 |
| jamesh | zyga: as far as naming of the access checkers, I think there is value in keeping the naming consistent with the documentation | 10:56 |
| zyga | jamesh: perhaps, but only if the name is accurate | 11:01 |
| jamesh | zyga: sure. I'm not saying the docs are immutable | 11:01 |
| zyga | brb | 11:32 |
| zyga | got a clean run on fedora 32 | 11:32 |
| zyga | thinking about core16 options | 11:32 |
| zyga | mborzecki: yeah, so REBOOT is busted | 11:49 |
| zyga | https://github.com/snapcore/snapd/pull/9125/checks?check_run_id=966349834 | 11:49 |
| mup | PR #9125: tests: fix issues related to restarting systemd-logind.service <Test Robustness> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9125> | 11:49 |
| zyga | I'm certain we need some kind of workaround for that bash issue | 11:49 |
| mborzecki | zyga: have you tried with the spread branch? | 11:49 |
| zyga | I'll try merging it locally to see if that helps | 11:49 |
| zyga | not yet | 11:49 |
| zyga | I will | 11:50 |
| zyga | if it helps we need to get that reviewed | 11:50 |
| zyga | I wonder what to do now | 11:50 |
| zyga | should we revert the restart "fix" for sid | 11:50 |
| zyga | disable sid for now | 11:50 |
| zyga | do something else | 11:50 |
| zyga | mvo: ^ advice welcome | 11:50 |
| mup | PR snapd#9127 opened: bootloader: introduce TrustedAssetsBootloader, implement for grub <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9127> | 11:50 |
| mborzecki | pedronis: trusted assets bootloader thingy ^^ | 11:50 |
| mborzecki | time to do some reviews | 11:51 |
| mborzecki | hm we haven't landed any bits for debug that call uname -a or somesuch? | 11:53 |
| zyga | mborzecki: we have a PR from cachio | 11:54 |
| mborzecki | mhm, just +1'ed it | 11:54 |
| mborzecki | zyga: https://github.com/snapcore/snapd/pull/9116/checks?check_run_id=965583423 | 11:55 |
| mup | PR #9116: tests: add system information and image information when debug info is displayed <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9116> | 11:55 |
| mborzecki | 2020-08-10T07:31:53.5421376Z Linux aug100702-249153 4.15.0-1080-gcp #90~16.04.1-Ubuntu SMP Fri Jul 10 19:11:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 11:55 |
| mup | PR #90: Default to SocketMode=0660 if no socket mode option is given <Created by mvo5> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/90> | 11:55 |
| mborzecki | and we have cgroup2fs under /sys/fs/cgroup | 11:55 |
| mborzecki | on xenial | 11:55 |
| zyga | mborzecki: that's ... just wrong? | 11:56 |
| zyga | how can we get that?! | 11:56 |
| zyga | is it overmounted | 11:56 |
| zyga | as in shadowing the tmpfs normally there | 11:56 |
| mborzecki | zyga: hm i misread, it's tmpfs, we check whether /sys/fs/cgroup/unified exists, but it does not? | 11:58 |
| zyga | mborzecki: hmmmm | 11:58 |
| zyga | that's possible | 11:58 |
| zyga | though I don't remember what the conditions are | 11:58 |
| zyga | maybe our check is wrong? | 11:58 |
| pedronis | mborzecki: I reviewed #9124, looks good but couple of questions | 11:59 |
| mup | PR #9124: gadget: introduce content update observer <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9124> | 11:59 |
| mborzecki | pedronis: thanks | 11:59 |
| mup | PR snapd#9128 opened: tests/main/cgroup-tracking: try to collect some information about cgroups <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9128> | 12:05 |
| mborzecki | zyga: ^^ some more info | 12:05 |
| zyga | nice | 12:10 |
| zyga | let's see what that shows | 12:10 |
| zyga | today is not a very good day, is it? | 12:10 |
| zyga | I'm making progress, will try a core 16 idea nonw | 12:10 |
| zyga | *now | 12:10 |
| pedronis | mborzecki: question(s) in #9127 | 12:11 |
| mup | PR #9127: bootloader: introduce TrustedAssetsBootloader, implement for grub <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9127> | 12:11 |
| mborzecki | pedronis: right, that's for the case if one gets a bootloader without NoSlashBoot in options, it should probably be internal error as you indicated, it's not a reasonable use case | 12:19 |
| mborzecki | pedronis: current_trusted_recovery_boot_assets? | 12:34 |
| pedronis | mborzecki: ok, let's have the error and see how it goes | 12:34 |
| mborzecki | instead of current_recovery_trusted_boot_assets | 12:35 |
| pedronis | that probably reads better | 12:35 |
| zyga | amazing | 12:43 |
| zyga | what am I missing | 12:43 |
| * zyga checks spread source | 12:43 | |
| zyga | nah, I am running master | 12:43 |
| zyga | ok, lets see if main passes | 12:43 |
| zyga | core16 did not crash and burn for me locally | 12:43 |
| ijohnson | morning folks | 12:44 |
| pedronis | ijohnson: hello | 12:47 |
| ijohnson | hi pedronis | 12:47 |
| ijohnson | pedronis: thoughts on moving envRefExtractedKernelBootloaderKernelState and extractedRunKernelImageBootloaderKernelState to a different file, maybe boot/bootstate20_kernelstate.go ? | 12:50 |
| ijohnson | that would make the bootstate20 file a little bit easier to navigate I think | 12:50 |
| ijohnson | of course not for my current PR, but a followup | 12:50 |
| zyga | hey ijohnson :) | 12:50 |
| ijohnson | hey zyga | 12:51 |
| pedronis | ijohnson: sounds ok | 12:54 |
| ijohnson | pedronis: cool I'll try to do that after we land the current PR | 12:58 |
| zyga | mvo: I've added https://github.com/snapcore/snapd/pull/9129 for 2.46 | 13:15 |
| mup | PR #9129: sandbox/cgroup: remove temporary workaround for multiple cgroup writers <Created by zyga> <https://github.com/snapcore/snapd/pull/9129> | 13:15 |
| mup | PR snapd#9129 opened: sandbox/cgroup: remove temporary workaround for multiple cgroup writers <Created by zyga> <https://github.com/snapcore/snapd/pull/9129> | 13:16 |
| zyga | mvo: what would it take to make /var/lib/systemd/ writable on core16 (it is writable on 18 and 20 already) - I'm specifically after /var/lib/systemd/linger so we could have a subset of the directory (just linger) available to help | 13:25 |
| ijohnson | zyga: just for our tests or IRL too ? | 13:32 |
| zyga | ijohnson: it's just for out tests but we seem to have done this on core18+ so I guess that's for a reason | 13:37 |
| zyga | and IRL is a weird way to describe that :) | 13:37 |
| zyga | brb, small break | 13:38 |
| pedronis | in general it's easier to have the 3 consistent when there's no strong reason for not | 13:43 |
| ijohnson | zyga: yeah in that case we should just do it IRL for core snap | 13:45 |
| zyga | I agree | 13:47 |
| zyga | consistency | 13:47 |
| zyga | and less special cases | 13:47 |
| ijohnson | indeed | 13:47 |
| zyga | I will break for lunch and an errand and then go for physio | 13:55 |
| zyga | I will do my best to work when I'm back to prepare the fixes for logind | 13:55 |
| cachio | pstolowski, ijohnson thanks for the review fo #8942, I already answer the question about the variable | 14:10 |
| mup | PR #8942: tests: support different images on nested execution <Run nested> <Squash-merge> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8942> | 14:10 |
| ijohnson | thanks cachio | 14:13 |
| cachio | ijohnson, sorry for the delay, didn't see that on friday | 14:14 |
| ijohnson | no worries | 14:14 |
| pstolowski | cachio: thanks | 14:17 |
| pedronis | mvo: cachio: are we releasing .3.1 today? | 14:27 |
| cachio | pedronis, I am waiting confirmation from store team | 14:27 |
| pedronis | thx | 14:27 |
| cachio | pedronis, they asked me to wait | 14:27 |
| cachio | pedronis, progressive release for core snap started | 14:47 |
| cachio | mvo, ~ | 14:47 |
| mvo | cachio: nice, thanks | 14:47 |
| pedronis | thx | 14:47 |
| pedronis | mvo: it would probably good to land #9086 but it needs a 2nd review (it's a cleanup) | 14:48 |
| mup | PR #9086: many: reorg cmd/snapinfo.go into snap and new client/clientutil <Cleanup :broom:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/9086> | 14:48 |
| mvo | pedronis: looking | 14:48 |
| pedronis | mvo: I moved my assertions PR to 2.47 | 14:50 |
| mvo | pedronis: thank you | 14:50 |
| zyga | made some more progress on core16, should have a green pass soon | 15:16 |
| zyga | but now physio | 15:16 |
| zyga | see you in a few hours | 15:16 |
| * cachio lunch | 15:18 | |
| ijohnson | mborzecki: have you seen the triggerwatch unit test triggerwatchSuite.TestNoDevsWaitKey fail before? | 15:19 |
| ijohnson | FAIL: triggerwatch_test.go:80: triggerwatchSuite.TestNoDevsWaitKey | 15:19 |
| ijohnson | triggerwatch_test.go:87: | 15:19 |
| ijohnson | c.Assert(err, IsNil) | 15:19 |
| ijohnson | ... value *errors.errorString = &errors.errorString{s:"trigger not detected"} ("trigger not detected") | 15:19 |
| ijohnson | OOPS: 4 passed, 1 FAILED | 15:19 |
| pedronis | mvo: I forgot to mention last week, did you see this: https://github.com/snapcore/core18/pull/155 ? (same for the parallel PRs), the name of the dir is wrong | 15:25 |
| mup | PR core18#155: hooks: fix broken symlink /etc/sysctl.conf.d/99-sysctl.conf <Created by mvo5> <https://github.com/snapcore/core18/pull/155> | 15:25 |
| mvo | pedronis: oh, missed that, sorry | 15:26 |
| ijohnson | interestingly a model assertion unit test panic'd on riscv64, seems to be timing related somehow https://pastebin.ubuntu.com/p/7ppt8pTbdz/ | 15:27 |
| ijohnson | cachio: do you need to delete some images on gce again? I see `2020-08-10 14:33:27 Cannot allocate google:opensuse-tumbleweed-64: cannot allocate new Google server google:opensuse-tumbleweed-64 (aug101430-988733): cannot find ready marker in console output for google:opensuse-tumbleweed-64 (aug101430-988733)` in spread runs now | 15:28 |
| mborzecki | ijohnson: heh, probably a timing issue | 15:35 |
| mborzecki | was outside for a bit, +35C | 15:35 |
| mvo | pedronis: I reviewed 9086 now | 15:39 |
| kenvandine | mvo: what's the status of lzo support? Is that ready to be used? What snapd version first supported it? | 16:03 |
| mvo | kenvandine: for snapd on install it's actually transparent, so any snapd version will work | 16:04 |
| kenvandine | but not at runtime. We're talking about injecting an "assumes" in there | 16:04 |
| mvo | kenvandine: not at runtime? what issue/error do you see with older snapd versions? | 16:05 |
| mvo | kenvandine: "snap pack --compression=lzo" is more recent | 16:05 |
| kenvandine | wait... when you launch the snap on an old version of snapd | 16:06 |
| mvo | kenvandine: but the ability to run lzo snaps should be transparent to snapd | 16:06 |
| kenvandine | that would work? | 16:06 |
| mvo | kenvandine: yes | 16:06 |
| mvo | kenvandine: I don't see why it would not :) maybe I'm missing something | 16:06 |
| kenvandine | ok, i thought there was talk about needing specific versions | 16:07 |
| kenvandine | i guess i misunderstood :) | 16:07 |
| kenvandine | mvo: and no known issues on other distros? centos, fedora, etc? | 16:07 |
| mvo | kenvandine: same, as long as they have a kernel that supports lzo for squashfs (which they all do) then this will be fine | 16:08 |
| kenvandine | ok | 16:08 |
| kenvandine | mvo: thanks, we're going to start testing it :) | 16:09 |
| kenvandine | with real snaps | 16:09 |
| mvo | kenvandine: \o/ nice! | 16:09 |
| mvo | kenvandine: AIUI chromium is already whitelisted in the reviewtools to allow lzo | 16:09 |
| kenvandine | ok | 16:09 |
| kenvandine | Wimpress was suggesting we try a few snapcrafter snaps first, so we'll work that out | 16:10 |
| kenvandine | and maybe experiment with the gnome platform snap too | 16:10 |
| cachio | ijohnson, done | 16:30 |
| Aavar | How would I go about building a snap of deluge 1.3 for 20.04? Or better yet, would anyone be interested in creating such a snap? | 16:30 |
| ijohnson | thanks cachio | 16:31 |
| oerheks | Aavar, there is 2.0.3 already | 16:43 |
| oerheks | https://snapcraft.io/deluge-lukewh | 16:43 |
| * zyga-mbp is back from physio but not working ye | 17:36 | |
| zyga-mbp | *yet | 17:36 |
| mup | PR snapd#9130 opened: [RFC] Support for gadget.yaml "$kernel:" style references <Created by mvo5> <https://github.com/snapcore/snapd/pull/9130> | 17:37 |
| ijohnson | cachio: I | 17:45 |
| ijohnson | I'm still seeing that ready marker error in spread with tumbleweed | 17:46 |
| ijohnson | `2020-08-10 17:21:32 Cannot allocate google:opensuse-tumbleweed-64: cannot allocate new Google server google:opensuse-tumbleweed-64 (aug101718-511919): cannot find ready marker in console output for google:opensuse-tumbleweed-64 (aug101718-511919)` | 17:46 |
| ijohnson | does it take a while for that to deploy ? | 17:46 |
| cachio | ijohnson, ah, ok, in that case this is another error | 17:46 |
| cachio | let me cehck again | 17:46 |
| ijohnson | ok | 17:46 |
| cachio | ijohnson, fixed | 17:46 |
| ijohnson | thanks I'll try again | 17:47 |
| cachio | I'll need to re create a new image for tumbleweed | 17:47 |
| zyga | back from physio | 18:33 |
| zyga | how are things? | 18:33 |
| zyga | I will get back to 16.04 part of the problem in a moment, let me review public tests first | 18:34 |
| cachio | zyga, I could reproduce the error in the cgroup-tracking | 18:35 |
| zyga | cachio: tell me more! | 18:35 |
| cachio | but still couldn't find which test is producing it | 18:35 |
| zyga | cachio: I reproduced it a few times | 18:35 |
| zyga | cachio: but I didn't focus on debugging it yet, I'll try tomorrow - assuming the current issues are fixed | 18:36 |
| zyga | cachio: we added some detectors to invariant checker and it's not the kernel - it is clearly some other factor | 18:36 |
| cachio | zyga, what I found is that there is another test which if it executed before the cgroup-tracking test fail | 18:36 |
| zyga | cachio: when it happened it was running the 4.15 gce kernel | 18:36 |
| zyga | yeah, I think that must be the case | 18:36 |
| zyga | we could find an intersection of sets after a few runs | 18:37 |
| zyga | I'm sure we'll find it | 18:37 |
| cachio | zyga, yes | 18:37 |
| cachio | let me find that | 18:37 |
| cachio | I'll send you what I find | 18:37 |
| cachio | zyga, I think in hours I can have that info | 18:38 |
| zyga | cool, drop me a line or go and fix it if you find the broken test | 18:39 |
| zyga | I'm no longer always on IRC so mattermost is best for a message box style | 18:39 |
| cachio | zyga, sure | 18:39 |
| cachio | I'll let you know | 18:39 |
| cachio | enjoy your evening | 18:39 |
| zyga | thank you :) | 18:41 |
| mup | PR core#116 opened: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116> | 20:22 |
| cachio | ijohnson, hey | 20:23 |
| cachio | I see this is uc20 https://paste.ubuntu.com/p/rhzB3sRQyV/ | 20:23 |
| ijohnson | hey cachio | 20:23 |
| ijohnson | cachio: ah yes that was fixed recently | 20:24 |
| ijohnson | --purge only works on a single snap | 20:24 |
| ijohnson | you need to do `snap remove --purge jq` `snap remove --purge jq-core18` etc. | 20:24 |
| cachio | ijohnson, ah, ok, I'll fix my scripts in that case | 20:24 |
| cachio | thanks | 20:24 |
| mup | PR snapcraft#3242 opened: tests: remove expected failure case from legacy integration test <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3242> | 22:36 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!