[00:09] <dstathis> n 20.04 many services (in particular kubelet) are packaged as snaps
[00:09] <dstathis> Does anyone know how to correctly interact with services running as a snap? There doesn't seem to be a systemd unit
[00:10] <dstathis> for example 'systemctl status kubelet' (or docker or whatever) fails with a not found error
[01:29] <jamesh> dstathis: systemd units owned by snaps will have names like "snap.$snapname.$app.service".  You should see them in the "systemctl list-units" output.
[03:05] <ish> I've installed a few Snaps on Fedora 32, and those with tray icons have garbled menus...  Anything obvious I'm missing?
[03:09] <oerheks> For font issue, libpango can be a reason
[03:09] <oerheks> or libfreetype
[05:15] <mup> PR snapd#8301 closed: interfaces/many: deny arbitrary desktop files and misc from /usr/share <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8301>
[05:15] <mup> PR snapd#9183 closed: tests: use "set -ex" in prep-snapd-in-lxd.sh <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9183>
[05:54] <mborzecki> morning
[05:57] <mvo> good morning mborzecki
[05:57] <mborzecki> mvo: hey
[06:38] <zyga> good morning
[06:38] <zyga> I will be around shortly
[06:38] <zyga> just need to send some paperwork for the insurance
[06:41] <zyga> mvo if you can, please merge https://github.com/snapcore/snapd/pull/9175
[06:41] <mup> PR #9175: tests: find -ignore_readdir_race when scanning cgroups <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9175>
[06:43] <mborzecki> zyga: hey
[06:45] <zyga> hi
[06:48] <mvo> zyga: looking
[06:49] <zyga> just random test failure fix
[06:51] <mvo> zyga: thanks, what did you change in the force push, unfortunately GH does not show me a diff for that :/
[06:51] <zyga> mvo I moved the -ignore_readdir_race after the path
[06:51] <zyga> originally it was the first argument but old find didn't like that
[06:52] <zyga> compare https://github.com/snapcore/snapd/commit/cd5b94776066bbe76359e912c960c9d4258abc9c and https://github.com/snapcore/snapd/commit/8c10ddfc32cf8f909ba73bfcfe691f174917c2e4
[06:53] <mvo> zyga: ta
[06:55] <mup> PR snapd#9175 closed: tests: find -ignore_readdir_race when scanning cgroups <Simple 😃> <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9175>
[06:56] <zyga> thank you
[06:56] <zyga> uh, it's a meeting day
[06:56] <zyga> just meetings and meetings
[07:03] <pstolowski> morning!
[07:05] <mborzecki> pstolowski: hey
[07:06] <pstolowski> hey mborzecki, welcome back!
[07:14] <mvo> good morning pstolowski
[07:43] <mvo> fwiw, I'm working on the lxd test failures currently
[07:43] <zyga> mvo thank you
[07:44] <mvo> yw - it's very painful as each iteration takes forever
[07:44] <mvo> but I hope I have a handle on it now (but I thought this the two previous runs too :(
[07:44] <zyga> mvo 300+ MB download is not free
[07:44] <pstolowski> mvo: thank you, i'm trying something as well on top of your existing PR but i'm not clear what root problem is and yes it is super slow
[07:45] <mvo> pstolowski: http://paste.ubuntu.com/p/p6pFKnRxrb/ is my best attempt so far, test is running
[07:46]  * mvo also wonders why wait_for_service seems to be not using the retr-tool
[07:59] <jamesh> mvo: I noticed the desktop code review meeting isn't on my calendar for this week.  Did you cancel it, or is that a glitch?
[08:00] <mvo> jamesh: I canceled it because we lack people but if you want to do it I can be available for you
[08:00] <mvo> jamesh: you also should have goten a mail about this by the calendar :/
[08:01] <jamesh> mvo: I don't see any email about the cancellation, which is why I asked.  That's fine.
[08:02] <mborzecki> mvo: https://github.com/snapcore/snapd/pull/9150 can land too
[08:02] <mup> PR #9150: gadget,kernel: add new kernel.{Info,Asset} struct and helpers (1/N) <Created by mvo5> <https://github.com/snapcore/snapd/pull/9150>
[08:02] <mborzecki> mup: and https://github.com/snapcore/snapd/pull/9156
[08:02] <mup> mborzecki: In-com-pre-hen-si-ble-ness.
[08:03] <mborzecki> mvo: and https://github.com/snapcore/snapd/pull/9156
[08:03] <mup> PR #9156: boot: copy boot assets cache to new root <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9156>
[08:06] <jamesh> mvo: I think https://github.com/snapcore/snapd/pull/9168 is good to merge, but is failing on ubuntu-20.04-64 for some tests/main/lxd:snapd_cgroup_* tests
[08:06] <mup> PR #9168: o/hookstate/ctlcmd: make is-connected check whether the plug or slot exists <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9168>
[08:07] <pstolowski> jamesh: yes these tests have been investigated since yesterday
[08:07] <jamesh> pstolowski: is it best just to wait til they get fixed then?
[08:08] <mvo> jamesh: if the failure is just on lxd I can force-merge
[08:09] <mvo> jamesh: merged
[08:09] <jamesh> mvo: cheers!
[08:09] <mvo> mborzecki: landed 9150 now too
[08:10] <mvo> mborzecki: and 9156
[08:10] <mborzecki> mvo: thanks!
[08:10] <mup> PR snapd#9150 closed: gadget,kernel: add new kernel.{Info,Asset} struct and helpers (1/N) <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9150>
[08:10] <mup> PR snapd#9156 closed: boot: copy boot assets cache to new root <UC20> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9156>
[08:10] <mup> PR snapd#9168 closed: o/hookstate/ctlcmd: make is-connected check whether the plug or slot exists <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9168>
[08:11] <mvo> fwiw, 9184 passed locally now for me on 20.04, let's see if it survies a full spread run
[08:11]  * mvo takes a short break while waiting for this
[08:18] <zyga-x240> nice work
[09:02] <pstolowski> \o/
[09:23] <mvo> so looks like 20.04 is now working but 16.04 is still failing :( *oh well* but with a very different error
[09:28] <zyga-x240> mvo: what does 16.04 say?
[09:28] <mvo> zyga-x240: "Failed to execute operation: Connection timed out"
[09:29] <mvo> zyga-x240: in a meeting right now, I can paste the full error late but it's also in the CI i think
[09:30] <zyga-x240> hmmm
[09:41] <zyga-x240> ijohnson: https://github.com/snapcore/snapd/pull/9187#discussion_r473820780
[09:41] <mup> PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187>
[09:52] <zyga-x240> hmmm
[09:55] <mborzecki> zyga-x240: hmm trouble getting self-hosted runners? https://github.com/snapcore/snapd/runs/1006992307
[09:57] <zyga-x240> hmm checking
[09:57] <zyga-x240> the host is up
[09:57] <zyga-x240> workers are up
[09:58] <zyga-x240> I think I know what's going on
[09:59] <zyga-x240> it seems that cla checks are hitting a time-out to grab a worker
[09:59] <zyga-x240> I don't recall seeing that before, we have queueing for a reason after all
[09:59] <zyga-x240> I restarted that and it passed instantly
[09:59] <zyga-x240> so... no idea/
[09:59] <mborzecki> hahah
[10:00] <mborzecki> well, maybe it's a one off occurrence
[10:05] <zyga-x240> I hope so but I fear we will learn in time
[10:05] <zyga-x240> time for coffee, I'm falling asleep here
[10:06] <zyga-x240> maybe pressure is low or something
[10:06] <zyga-x240> mvo: I looked and it looks like systemd is not responding, maybe the socket is gone somehow?
[10:06] <zyga-x240> but no idea why
[10:14] <mvo> zyga-x240: so the nested lxd shows me a gazillion "permission denied", e.g. /bin/mount for / exited 1, mount: permission denied etc
[10:14] <mvo> on 16.04
[10:14] <zyga-x240> mvo: seems like lxd apparmor
[10:15] <zyga-x240> mount is really disallowed
[10:15] <zyga-x240> only bind may be allowed
[10:15] <mvo> yeah, strange that it's inside the nested though
[10:15] <zyga-x240> maybe nesting is broken somehow
[10:15] <zyga-x240> I wonder if we can stop doing something and get nested working without snapd
[10:15] <zyga-x240> and then see what part breaks it
[10:16] <mborzecki> zyga-x240: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1889318 is it because when run by lxc there's no apparmor namespace setup like lxd does?
[10:16] <mup> Bug #1889318: install chromium in lxc container for 20.04 fails <amd64> <apport-bug> <focal> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1889318>
[10:16] <mvo> zyga-x240: it fails even before snpad it seems
[10:16] <mvo> zyga-x240: hm, well, maybe or maybe not
[10:16] <zyga-x240> mborzecki: looking
[10:17] <zyga-x240> I ... don't know
[10:17] <mvo> hm, "interessting" - it fails in systemctl daemon-reload but everyting else seems to work
[10:17] <zyga-x240> well
[10:17] <zyga-x240> when systemd is not responding
[10:17] <zyga-x240> it's not really a place where things work
[10:18] <mvo> the funny thing is - systemctl restart snapd works
[10:18] <mvo> systemctl --list works
[10:18] <mvo> afaict all the things I tried work
[10:18] <zyga-x240> hmm
[10:18] <mvo> just not the daemon-reload
[10:18] <mvo> it's a bit strange
[10:18] <zyga-x240> what does daemon-reload say?
[10:19] <zyga-x240> I mean, there is journal
[10:19] <mvo> and I think this is broken since forever
[10:19] <mvo> we never saw it because that script did not have set -e
[10:19] <zyga-x240> ohhhh
[10:19] <zyga-x240> that's interesting
[10:19] <zyga-x240> so it would fail on stable releases as well
[10:19] <zyga-x240> mvo: do you have 5 minutes for a quick call?
[10:24] <jdstrand> mvo: hi! thanks for committing PR 8301! have you decided to marge master into release/2.46? if you aren't, I need to prepare a PR for 8301 (that would include 9167) for 2.46 (which is fine, just need to know that I should do it :)
[10:24] <mup> PR #8301: interfaces/many: deny arbitrary desktop files and misc from /usr/share <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8301>
[10:26] <jdstrand> mvo: all that is left for new PRs that small k8s-support one that I need to investigate. then I'll be doing 'needs security review' reviews
[10:29] <mvo> jdstrand: thank you
[10:34] <jdstrand> mvo: (also, PR 8920 needs final reviews)
[10:34] <mup> PR #8920: interfaces: update cups-control and add cups for providing snaps <Needs Samuele review> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8920>
[10:36] <jdstrand> mvo: do note that I intentially did not milestore PR 9186 for 2.46. that needs discussion. if there happened to be a 2.46 point release after that is merged, we could consider it, but it floating into 2.47 would be fine too
[10:36] <mup> PR #9186: interfaces: add vcio interface <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9186>
[10:38] <mvo> ok
[10:42] <jdstrand> mvo: I'm looking at https://github.com/snapcore/snapd/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+security+review%22. nothing is milestoned for 2.46. is there anything in there that should be that you would like me to prioritize?
[10:43] <jdstrand> if not, I have a good idea of the priority
[10:49] <zyga> I keep getting vendor.json changes
[10:49] <zyga> I purged cache
[10:49] <zyga> purged the tree
[10:49] <zyga> it keeps changing
[10:52] <jdstrand> zyga: me too! I'd love to see that resolved. (my dev container is on bionic still. wonder if it is a focal vs bionic thing)
[10:53] <pstolowski> mvo: why is #9171 blocked?
[10:53] <mup> PR #9171: [RFC] config: "virtual" configuration for timezone <Needs Samuele review> <Run nested> <⛔ Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9171>
[10:54] <mborzecki> zyga: jdstrand: perhapos a different version of govendor was used when vendor.json was last updated in the tree
[10:54] <zyga> hmm
[10:54] <mborzecki> the difference is a checksum only
[10:54] <mborzecki> (at least here)
[10:54] <zyga> perhaps but what's the version and who has it is interesting
[10:55] <mborzecki> i have 1.0.9
[10:56] <zyga> I have 1.0.9 in /usr/bin and 1.0.8 in GOPATH
[10:56] <mborzecki> btw. i guess we all should be using the latest master govendor, since we're go getting it in run-checks
[10:56] <zyga> and get-deps uses that
[11:01] <zyga> mborzecki: totally offtopic: https://github.com/snapcore/snapd/pull/9189
[11:01] <mup> PR #9189: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <https://github.com/snapcore/snapd/pull/9189>
[11:01] <mup> PR snapd#9189 opened: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <https://github.com/snapcore/snapd/pull/9189>
[11:02] <zyga> mborzecki: running 1.0.9 modifies my vendor.json
[11:04] <mborzecki> zyga: there's a couple of recent commits to vendor.json, mine was on 3.08 (and i'm using 1.0.9), the later commits were by claudio, samuele and jamesh
[11:04] <zyga> hmmmm
[11:05] <zyga> the change comes from 7a9cb154a0c (Claudio Matsuoka     2020-08-13 09:08:15 -0300 115)                        "revision": "68200eea7bdcb97e27fe8e5ff443776383908637",
[11:05] <zyga> so maybe claudio has older version
[11:06] <mborzecki> let's ping him when he's online
[11:06] <zyga> yeah
[11:10] <mvo> pstolowski: I think 9171 really needs samuele approval, I think it's okay otherwise, if the design gets approval I would like to tweak it a bit more with some helpers
[11:22] <pstolowski> mvo: i made a few comments, not sure what was already discussed and agreed, so maybe my comments make no sense
[11:22] <mvo> pstolowski: cool, happy for any feedback at this point
[11:24] <mvo> pstolowski: hm, great point about snap get -d
[11:24] <mvo> pstolowski: I think you are right, we should probably not bypass this for that
[11:24]  * mvo scratches head
[11:25] <pstolowski> mvo: yes i think we are breaking some high level assumptions here. and i fear it's going to be annoying to handle :/
[11:25] <mvo> pstolowski: yeah, this needs more discussion for sure
[11:26] <mvo> pstolowski: your comment about the transations is also interessting, maybe we need to hook into commit() here instead
[11:27] <mvo> lxd tests passed locally in 9184
[11:27]  * mvo vanishes for lunch
[11:27] <zyga> enjoy
[11:27] <pstolowski> mvo: what if we do store in state but synchronize config state with system setup? or is it too terrible?
[11:29]  * pstolowski lunch as well
[11:29] <zyga> pstolowski: who wins?
[11:29] <zyga> pstolowski: if system was modified when snapd was down?
[11:30] <pstolowski> zyga: system always wins. we update system on snap set.
[11:30] <pstolowski> but maybe it's naive
[11:30] <pstolowski> just throwing ideas
[11:31] <zyga> pstolowski: so when would we use the value from state?
[11:33] <pstolowski> zyga: we would always update state from system before reading. that could simplify transaction logic without special-casing. but just brainstorming at this point
[11:33] <pstolowski> anyway, time for lunch
[11:33] <zyga> pstolowski: I see
[11:34] <zyga> I don't know either, just trying to understand your point better
[11:41] <cachio> zyga, hi
[11:42] <cachio> zyga, do you have any idea about what could be causing that when I test beta image and do "systemctl --user daemon-reload" as root I get "Failed to connect to bus: No such file or directory
[11:42] <cachio> "
[11:42] <cachio> If I do that as ubuntu user I dont see any error, but as root I see that error
[11:42] <cachio> and it makes fail the snapd-failover test
[11:43] <zyga> cachio: hi
[11:43] <zyga> cachio: yes, I explained that a few days ago
[11:43] <zyga> cachio: when we reload systemd-logind.service on older versions of systemd
[11:43] <zyga> cachio: we cause it to forget about the session of the root user
[11:43] <zyga> cachio: then subsequent test that prepares a session for the root user
[11:44] <zyga> cachio: enables linger, which starts services and so on
[11:44] <zyga> cachio: but then the restore disables linger
[11:44] <zyga> cachio: because systemd logind is no longer tracking the incoming ssh session of the root user
[11:44] <zyga> cachio: it shuts down systemd --user for root
[11:46] <cachio> zyga, do you know which is the difference between the test we run in github and the one I run in beta validation to explain why it works in github and fails with the beta image?
[11:46] <zyga> cachio: I don't know what version beta was forked form
[11:47] <zyga> cachio: please check if it contains changes to prepare-restore.sh
[11:47] <zyga> talking exactly about this issue in the commit message
[11:51] <cachio> zyga, I see the change
[11:51] <cachio> I need to see how to make it for external backend
[11:51] <cachio> thanks for the explanation
[11:54] <zyga> cachio: the fix I made is generic
[11:54] <zyga> cachio: if you have it and the bug persists then there's something more broken
[11:54] <cachio> zyga, yes
[11:54] <cachio> but in case of external backend we exit before that code
[11:54] <zyga> cachio: output from loginctl list-sessions would help
[12:00] <zyga> cachio: when do we exit then?
[12:00] <cachio> zyga, most of the prepare_project is not done for external backend
[12:01] <cachio> I am trying to move the linger configuration to see if it works
[12:02] <zyga> cachio: I see
[12:02] <zyga> cachio: yeah, you need to apply the fixes to systemd-logind
[12:02] <zyga> cachio: those are one-off
[12:02] <zyga> cachio: do you perform package upgrades? what's the target?
[12:03] <zyga> is that core16?
[12:03] <cachio> yes
[12:04] <zyga> core16 needs a special workaround
[12:04] <zyga> in essence, I repackage core
[12:04] <zyga> though recently ijohnson applied a fix to core so maybe that is no longer required
[12:05] <zyga> following that /var/lib/systemd/linger is bind-mounted to writable using a mount unit
[12:05] <zyga> look at the patches I apply to replicate that
[12:05] <zyga> I wasn't aware the external target skips all of that
[12:05] <cachio> zyga, no problem, I'll try to extend that to external
[12:05] <zyga> you may only need the 1) mount unit 2) change to logind configuration followed up by REBOOT
[12:37] <ijohnson> zyga: the core fix has not landed yet, PR is still open
[12:37] <ijohnson> also morning folks
[12:37] <ijohnson> hey mborzecki welcome back
[12:45] <cachio> pstolowski, hey, any idea about thie error https://paste.ubuntu.com/p/SRTGRMHx45/
[12:45] <cachio> it is happening in Core20-early-config test
[12:46] <cachio> I see this gadget.yaml parse error: Duplicate key: system
[12:46] <cachio> not sure if it could be the raeson
[12:48] <pstolowski> looking
[12:49] <zyga> ijohnson: I see
[12:49]  * zyga was having pierogis for lunch
[12:49] <ijohnson> cachio: that was what I fixed in my PR
[12:49] <pstolowski> cachio: yeah, definately
[12:49] <zyga> cachio: there's a duplicate key :)
[12:49] <ijohnson> cachio: that is fixed by #9187
[12:49] <mup> PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187>
[12:50] <pstolowski> right, i was just going to say that :), thanks ijohnson
[12:50] <cachio> ijohnson, ah, thanks!!!
[12:58] <mvo> ijohnson: should I force merge 9187? there are still failures in nested
[12:59] <cachio> mvo, the errors in nested are fixed in other pr
[12:59] <ijohnson> mvo: let me look quickly
[12:59] <cachio> in mine
[12:59] <ijohnson> mvo: the tests are still failing
[12:59] <ijohnson> also SU time now
[12:59] <cachio> pr: #9098
[12:59] <mup> PR #9098: tests: new organization for nested tests <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9098>
[13:00] <ijohnson> mvo: sorry I meant the tests still seem to be running?
[13:00] <cachio> but mine fails sometimes because the erorr which is fixed on 9187
[13:00] <cachio> ijohnson, I retriggered the tests
[13:16] <mup> PR snapd#9081 closed:  secboot,cmd/snap-bootstrap: cross-check partitions before unlocking, mounting <Run nested> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9081>
[13:37] <zyga-mbp> oh, and I'm making progress on unit testing bash, https://listed.zygoon.pl/ has the details
[13:40] <cachio> zyga-mbp, so I see we do https://github.com/snapcore/snapd/blob/master/tests/lib/prepare.sh#L662
[13:40] <zyga-mbp> right
[13:40] <cachio> we need that for external as well?
[13:40] <zyga-mbp> you want that and the configuration change immediately below that
[13:40] <zyga-mbp> yes
[13:41] <cachio> ok, that plus the change https://github.com/snapcore/snapd/blob/master/tests/lib/prepare-restore.sh#L446
[13:41] <zyga-mbp> cachio note that this is the static part, the dynamic part is where we decide systemd-logind needs reloading and REBOOT
[13:42] <zyga-mbp> yes
[13:42] <cachio> perfect
[13:42] <zyga-mbp> we try to enable linger for the test user, if that fails we know we need to restart
[13:42] <zyga-mbp> note that this does essentially the same change (configuration file edit) so make sure to just reboot in that case as the config file is already in place, just inactive
[13:43] <cachio> ok
[13:43] <zyga-mbp> I hope this works :)
[13:43] <zyga-mbp> cachio I left a small comment on v
[13:43] <zyga-mbp> https://github.com/snapcore/snapd/pull/8986/files
[13:43] <mup> PR #8986: tests: new snaps-state command - part1 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8986>
[13:44] <cachio> zyga-mbp, thanks!!
[14:26] <zyga> cachio: any luck?
[14:26] <cachio> zyga, no yes, trying in 5 minutes
[14:26] <zyga> ok
[14:26] <cachio> still making some changes
[14:42] <mup> PR snapd#9188 closed: interfaces: misc policy updates xlvi <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9188>
[14:46] <ijohnson> mvo: 9184 is super super green :-)
[14:47] <mup> PR snapd#9190 opened: [RFC] cmd/s-b/initramfs-mounts: make recover -> run mode transition automatic <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9190>
[14:48] <cachio> zyga-mbp, Could not enable linger: No such file or directory
[14:48] <cachio> I  see that
[14:49] <zyga-mbp> cachio, ok do you have a shell
[14:49] <cachio> when I do loginctl enable-linger test
[14:49] <cachio> yes
[14:49] <zyga-mbp> is the logind.conf file modified?
[14:49] <cachio> yes
[14:49] <cachio> StateDirectory=systemd/linger
[14:49] <zyga-mbp> is /var/lib/systemd/linger a mount point to the corresponding directory in writable?
[14:49] <cachio> with this config
[14:49] <cachio> I created that in writable
[14:50] <zyga-mbp> that's not all, is a mount in place?
[14:50] <cachio> I cant write to /var/lib/systemd/linger
[14:50] <zyga-mbp> is there a mount unit that makes /var/lib/systemd/linger a bind mount to /writable/system-data/....
[14:50] <cachio> it is protected
[14:51] <zyga-mbp> you need the linger directory to exist
[14:51] <zyga-mbp> and it must be a mount point as I've explained
[14:51] <zyga-mbp> no way around that
[14:51] <zyga-mbp> THEN you can reboot to reconfigure logind
[14:51] <zyga-mbp> (via REBOOT)
[14:51] <cachio> I cant create /var/lib/systemd/linger
[14:51] <zyga-mbp> and then it will work
[14:51] <zyga-mbp> cachio so merge the core change and rebuild the snap
[14:52] <cachio> which is the change to merge?
[14:52] <zyga-mbp> ian proposed a PR for core
[14:53] <cachio> zyga-mbp, ok, so no workaround until the chagne in core is merged
[14:53] <zyga-mbp> correct
[14:53] <zyga-mbp> unless you can repackage core
[14:54] <cachio> zyga-mbp, agree but on beta validation we don't repack core
[14:54] <cachio> so I'll push the change done by ian
[14:55] <cachio> ijohnson, is this the change? https://github.com/snapcore/core/pull/116
[14:55] <mup> PR core#116: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116>
[14:55] <ijohnson> yes
[14:56] <cachio> waiting for a second review?
[14:56] <cachio> ijohnson,
[14:56] <zyga-mbp> ijohnson what happens when we remove entries?
[14:57] <ijohnson> cachio: I can actually merge it, not sure if I should wait though
[14:57] <ijohnson> zyga-mbp: what do you mean ?
[14:57] <zyga-mbp> ijohnson this looks good but may need follow ups for hacks in prepare-restore
[14:57] <zyga-mbp> ijohnson the removal of /var/lib/systemd/rfkill and random-seed lines
[14:57] <ijohnson> hmm
[14:58] <zyga-mbp> I'd +1 without those
[14:58] <zyga-mbp> but now that I see them
[14:58] <zyga-mbp> I would not keep them
[14:58] <zyga-mbp> I just want to understand what happens in practice
[14:58] <ijohnson> I need to think about this, I don't remember how refreshes work, but I think it's just the initramfs that modifies the etc/fstab according to the writable-paths
[14:58] <zyga-mbp> on existing systems that shipped with those lines
[14:58] <zyga-mbp> cachio can you test that upgrade
[14:58] <ijohnson> so after a refresh to the revision here it should work fine
[14:58] <zyga-mbp> I think it's very important for correctness
[14:59] <ijohnson> there have been other situations where we drop things there though that have landed without controversy however I think
[14:59] <ijohnson> one moment
[14:59] <zyga-mbp> ijohnson we are trigger happy sometimes
[15:00] <pstolowski> mvo: +1 for #9184
[15:00] <mup> PR #9184: packaging: umount /snap on purge in containers <⚠ Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184>
[15:02] <ijohnson> zyga-mbp: see https://github.com/snapcore/core18/pull/127
[15:02] <mup> PR core18#127: Make /var/cache writable and disable dynamic motd <Created by woodrow-shen> <Merged by vorlonofportland> <https://github.com/snapcore/core18/pull/127>
[15:02] <zyga-mbp> ijohnson in another call
[15:05] <cachio> zyga-mbp, ijohnson sorry, which upgrade?
[15:05] <ijohnson> cachio: nvm I can test the upgrade, it is about the writable-paths changes
[15:06] <cachio> ijohnson, ah, ok, thanks
[15:13] <zyga-mbp> re
[15:13] <zyga-mbp> cachio check if we can upgrade core from what it looks like now to what is proposed in that PR 127
[15:13] <mup> PR #127: Use cap instead of cap1 where only one capability is being manipulated <Created by zyga> <Merged by elopio> <https://github.com/snapcore/snapd/pull/127>
[15:13] <zyga-mbp> cachio to see if anything explodes
[15:13] <zyga-mbp> I mean
[15:13] <zyga-mbp> we'll know if we land it
[15:13] <zyga-mbp> and run tests
[15:14] <zyga-mbp> but a quick run might be helpful as well
[15:14] <zyga-mbp> mvo ^ do you know what happens if we remove existing entries from writable paths?
[15:16] <ijohnson> ugh building the core snap is so annoying
[15:16] <ijohnson> I wish we could update the core snap to build with modern snapcraft
[15:16] <cachio> ijohnson, yes
[15:17] <cachio> tomorrow I can test that we new core is in edge
[15:17] <ijohnson> cachio: but the PR hasn't been merged though is the thing
[15:17] <cachio> I really don't know how to build core
[15:17] <ijohnson> I just built it locally though
[15:17] <ijohnson> cachio: do you use wormhole? I can send it to you in a few moments
[15:17] <cachio> I can test that if you built core?
[15:17] <cachio> ijohnson, yes
[15:18] <cachio> please send me it
[15:18] <mvo> zyga-mbp: if we remove writable-path they stop being writable which may break things - what in particular are you asking?
[15:18] <zyga-mbp> mvo please look at https://github.com/snapcore/core/pull/116/files
[15:18] <mup> PR core#116: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116>
[15:18] <zyga-mbp> are those changes safe in your eyes
[15:19] <ijohnson> zyga-mbp: fwiw he already approved it
[15:19] <zyga-mbp> I know but not sure if he really looked ;D (sorry mvo)
[15:19] <mvo> zyga-mbp: oh, this should be fine
[15:19] <zyga-mbp> mvo in that case let's land it
[15:19] <zyga-mbp> and rebuild core
[15:19] <mvo> zyga-mbp: removing subpath but making more available is fine
[15:19] <zyga-mbp> we can always revert
[15:19] <zyga-mbp> and this will help cachio
[15:20] <mvo> zyga-mbp: please approve first
[15:20] <ijohnson> well I have it built now, testing what happens on refresh of a core16 vm
[15:20] <ijohnson> cachio: I don't think you need to test what I built here anymore
[15:21] <cachio> ok
[15:23] <mup> PR snapcraft#3250 closed: cli: implement list-tracks <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3250>
[15:23] <mup> PR snapcraft#3252 closed: snapcraft: use system certificates by default for https requests <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3252>
[15:23] <ijohnson> jdstrand: is this a typo or intentional? https://github.com/snapcore/snapd/pull/9188/files#r474066839
[15:23] <mup> PR #9188: interfaces: misc policy updates xlvi <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9188>
[15:24]  * zyga-mbp needs to break soon
[15:28] <mup> PR snapcraft#3255 closed: remote-build: use requests.get() instead of urlopen() <bug> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3255>
[15:32] <mvo> ijohnson: if you test is successful please let me know and I will merge
[15:32] <ijohnson> mvo: yes sorry got distracted it is rebooting now
[15:32] <mvo> ijohnson: no worries
[15:38] <ijohnson> mvo: mmm the VM didn't come back after the refresh
[15:38] <ijohnson> mvo: need to debug, perhaps there is a problem
[15:39] <mvo> ta
[15:43]  * cachio lunch
[15:44] <mvo> ijohnson: uh, I accidently merged 116 :/ let's hope it's not a  problem with that change or I have to revert
[15:45] <ijohnson> mvo: my hope is that this is a multipass problem, I am recreating a uc16 VM with plain qemu :-/
[15:46] <mup> PR core#116 closed: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/core/pull/116>
[15:46]  * zyga runs away from meetings and works for a while in quiet
[15:47] <zyga> mvo: shall we merge https://github.com/snapcore/snapd/pull/9184
[15:47] <mup> PR #9184: packaging: umount /snap on purge in containers <⚠ Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184>
[15:48] <zyga> just to make some progress and see how it behaves over time
[15:48] <ijohnson> I'd say so
[15:49] <zyga> yeah
[15:49] <zyga> me too
[15:51] <dstathis> Has anyone here used kubelet installed via snap? If so do you know how to get it to use the systemd cgroup driver?
[15:52] <ijohnson> joedborg: can you answer dstathis question ?
[15:56] <ijohnson> mmm mvo, better revert that core PR, I can reproduce the problem in qemu, I need to debug this, but it somehow broke ssh
[15:56] <ijohnson> so sorry :-/
[15:56] <joedborg> Hey dstathis.  We’re currently working on this under strict confinement for both microk8s and a fully working kubernetes node (I.e. kubelet, proxy, containerd etc).  This isn’t complete yet though as there are lots of working parts that need working through
[15:58] <dstathis> joeborg: The snap of kubelet is not complete yet? I'm asking because kubelet is no longer available via apt. Only snap
[15:59] <dstathis> (in Ubuntu 20.04)
[16:01] <mvo> ijohnson: my bad, I should not have merged prematurely
[16:02] <ijohnson> no, it's my bad I didn't test an actual refresh of it
[16:02] <zyga> mvo: https://github.com/snapcore/snapd/pull/9184#pullrequestreview-471764868 can I merge?
[16:02] <mup> PR #9184: packaging: umount /snap on purge in containers <⚠ Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184>
[16:03] <mvo> zyga: yes, I'm still trying to create a reproudce for the lxd team about the systemctl daemon-reload
[16:03] <joedborg> dstathis: that is usually used by our kubernetes distribution Charmed Kubernetes.  It is a classic snap, so shouldn’t need any interfaces connected.  However it’s not really documented outside of CK context
[16:03] <mvo> zyga: but merging will make things work again
[16:03] <zyga> mvo: ok
[16:03] <zyga> merging
[16:03] <zyga> it's ok to revert if it explodes in new ways
[16:04] <dstathis> joedborg: is there another recommended way to install kubelet on 20.04? Or do you know how to configure the cgroup diver using the snap?
[16:07] <mup> PR snapd#9184 closed: packaging: umount /snap on purge in containers <⚠ Critical> <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9184>
[16:09] <joedborg> dstathis: I don’t know for sure, but I’m reasonably confident you do `snap set kubelet [kubelet arg]=[value]
[16:10] <joedborg> You need to swap hyphens for underscores i.e `api-server` becomes `api_server`
[16:10] <ijohnson> how does a change to /var/lib/systemd/private cause `sshd[1151]: Missing privilege separation directory: /var/run/sshd` :-(
[16:12] <dstathis> joedborg: is there a way to ask the snap which configuration options it will recognize? I know there is a command line argument that can be passed in this case
[16:12] <joedborg> dstathis: sadly not as snap config is key value without any firm validation
[16:29] <mvo> ijohnson: pushed the revert now
[16:29]  * mvo gets dinner
[16:29] <ijohnson> thanks, I am very puzzled how this is broken
[16:31] <mvo> yeah
[16:31] <zyga> what is broken?
[16:31] <zyga> I heard qemu
[16:32] <ijohnson> zyga: my writable-paths PR somehow breaks ssh
[16:32] <zyga> ijohnson: aha
[16:32] <zyga> maybe it only breaks ssh in testing
[16:32] <ijohnson> so you were right :-O
[16:32] <zyga> what are the symptoms?
[16:32] <zyga> ijohnson: ha
[16:32] <dstathis> joedborg: 'sudo snap set kubelet cgroup-driver=systemd' worked. Thanks
[16:32] <zyga> ijohnson: heh, I didn't mean to
[16:33] <ijohnson> zyga: it makes the ssh server service fail to start like this:
[16:33] <ijohnson> https://www.irccloud.com/pastebin/NbSCym9q/
[16:35] <zyga> interesting
[16:35] <zyga> but how is that related to what you did?
[16:35] <ijohnson> I have no idea
[16:35] <zyga> maybe we neutered the whole directory
[16:35] <zyga> there used to be more state in that systemd tree
[16:35] <zyga> now it's all an empty directory
[16:35] <zyga> maybe that's why?
[16:36] <ijohnson> zyga: no the other files are still there
[16:36] <zyga> oh
[16:36] <zyga> I'll look at ssh source
[16:36] <ijohnson> zyga: the files are /var/lib/systemd/... and those are still there in the core snap after I made this change
[16:36] <ijohnson> unless systemd is doing something weird with /var/lib/private
[16:36] <zyga> hah
[16:37] <zyga> do we have /var/empty?
[16:37] <zyga> this is what this is complaining about
[16:37] <zyga> ah, no
[16:37] <zyga> sorry
[16:37] <zyga> confflags += --with-privsep-path=/run/sshd
[16:38] <ijohnson> zyga: /var is not empty
[16:38] <zyga> if /run/sshd doesn't exist I'd love to log in interactively
[16:38] <zyga> and see the status
[16:38] <zyga> of all the services
[16:38] <zyga> I bet something else failed
[16:38] <ijohnson> mmm actually there is a tmpfiles failure earlier
[16:38] <zyga> ijohnson: can we retry with just added linger
[16:38] <zyga> and then circle back with more oomph
[16:39] <ijohnson> https://www.irccloud.com/pastebin/4GceHjg4/
[16:39] <zyga> unsafe symlinks? never heard that one before
[16:39] <zyga> checking
[16:40] <ijohnson> yeah very odd
[16:42] <zyga> I cannot find that in current systemd
[16:42] <zyga> trying older
[16:44] <zyga> ijohnson: I think this explains it
[16:44] <zyga> https://github.com/systemd/systemd/issues/14226#issuecomment-560467107
[16:45] <ijohnson> mmmm interesting let me see
[16:45] <zyga> ijohnson: I'm too tired to build core and boot it in qemu
[16:45] <zyga> I wish there was a fire-and-forget for that
[16:46] <ijohnson> zyga: no worries thank you for looking!
[16:46] <ijohnson> I am actively debugging this
[16:48] <ijohnson> ha actually I'm a fool I didn't even build the right branch, so this core snap that I built is just master
[16:48] <zyga> ijohnson: it would be amazing if we could test core against snapd test suite
[16:48] <zyga> just build it there and clone snapd in an action
[16:48] <zyga> and set some env in a spread run
[16:48] <zyga> so it gets that core
[16:49] <zyga> I think it's doable now
[16:49] <ijohnson> it would be amazing if I didn't need all this chroot and vm nonsense with a legacy snapcraft to build the core snap
[16:49] <ijohnson> and I could just do `snapcraft --use-lxd`
[16:49] <zyga> ijohnson: we could use one action to build it and store an artefact
[16:49] <zyga> ijohnson: and then another action to test it
[16:50] <ijohnson> zyga: sorry what do you mean? for the snapd repo ?
[16:51] <zyga> run all of snapd tests against a core built from a branch of the core repo
[16:51] <zyga> right?
[16:51] <zyga> so we get way better coverage
[16:52] <ijohnson> mmm that would be nice
[16:53] <zyga> the checkout action can checkout any repo
[16:53] <zyga> so we have 90% of what we want in the snapd repo already
[16:54] <zyga> we just need to teach snapd test suite to use a core that's wgettable from somewhere
[16:54] <zyga> it can still rebuild / repack snapd
[16:54] <zyga> but the base would be from the PR changing the core snap
[16:54] <zyga> we could even write a test that runs only in this mode
[16:55] <zyga> where a vanilla core can refresh to the new core
[16:55] <zyga> and boot
[16:55] <zyga> anyway
[16:55] <zyga> you get the point
[16:55] <ijohnson> also :-( I think the whole reason this failed is because I forgot to do this:
[16:55] <ijohnson>   - sudo sed -i '/^deb/s/$/ universe/' $CHROOT/etc/apt/sources.list
[16:55] <ijohnson> so the snap I built built from the wrong packages
[16:56] <ijohnson> *sighes
[16:58] <zyga> oh?
[16:58] <zyga> oh well
[16:58]  * zyga suspends and takes a break
[16:58] <ijohnson|lunch> https://github.com/snapcore/core/blob/master/.travis.yml is how I build all my core snaps
[16:58] <ijohnson|lunch> but just doing that locally
[16:58] <ijohnson|lunch> and I missed a step
[17:29] <cachio> zyga-mbp, this test is failing as well
[17:29] <cachio> https://paste.ubuntu.com/p/GXdqXc4YWg/
[17:30] <cachio> could be related to the revious issue ?
[17:30] <zyga-mbp> it's the same failure
[17:30] <zyga-mbp> yeah
[17:30] <zyga-mbp> it's expected
[17:30] <cachio> or you think it is something different
[17:30] <cachio> zyga-mbp, nice, thanks, I thought that but just wanted to make sure
[17:30] <cachio> thanks
[17:31] <zyga-mbp> cool
[17:57]  * cachio afk
[18:02] <mup> PR snapd#9189 closed: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9189>
[18:40] <joedborg> dstathis: ah thanks for letting me know
[19:28] <cachio> cmatsuoka, https://github.com/snapcore/snapd/pull/9098/checks?check_run_id=1008878037#step:5:7412
[19:28] <mup> PR #9098: tests: new organization for nested tests <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9098>
[19:28] <cmatsuoka> cachio: let's see...
[19:29] <cachio> now without any error during reboot
[19:29] <cachio> during boot
[19:29] <cachio> no reboots involved
[19:29] <cmatsuoka> ah nice!
[19:29] <cmatsuoka> excellent
[19:29] <cmatsuoka> ah wait
[19:29] <cmatsuoka> but it failed?
[19:30] <cmatsuoka> ugh
[19:30] <cachio> couldnt login after that
[19:30] <cmatsuoka> and this is qemu without kvm?
[19:30] <cachio> yes
[19:31] <cmatsuoka> let me see this log in detail
[19:32] <cachio> ijohnson, I merged master and still see
[19:32] <cachio> 2020-08-20T18:18:54.2486726Z + mount /dev/mapper/ /tmp/tmp.wFbPmXutiu
[19:32] <cachio> 2020-08-20T18:18:54.2486835Z mount: /tmp/tmp.wFbPmXutiu: /dev/mapper is not a block device.
[19:33] <cachio> sorry, thought that #9187 was merged
[19:33] <mup> PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187>
[19:34] <ijohnson> cachio yeah it's not merged, needs mvo to merge it though maybe now If I merge master to that PR it will be green
[19:34] <cachio> ijohnson, I see lxd tests failing there https://github.com/snapcore/snapd/pull/9187/checks?check_run_id=1007716427#step:5:7003
[19:34] <mup> PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187>
[19:40] <ijohnson> cachio yeah let me merge master to this pr so we have some chance of being all green
[19:42] <cachio> ijohnson, nice, thanks
[19:43] <ijohnson> cachio: running now, let's see how it does
[19:43] <cachio> cool
[19:43] <cmatsuoka> cachio: I think I found the problem
[19:44] <cachio> cmatsuoka, awesome
[19:44] <cachio> which one?
[19:44] <cachio> which is?
[19:47] <cmatsuoka> I think there's a call missing there, I'm checking the change history to see if this is was lost at some refactoring, or it was never there
[19:47] <cachio> cmatsuoka, nice
[20:11] <cmatsuoka> cachio: I'm checking a possible fix with chris
[20:16] <cachio> cmatsuoka, nice, thanks
[20:16] <cachio> please ping me if you need to test it
[20:17] <cmatsuoka> cachio: thanks, I think we'll need to test it a lot since it seems that it happens infrequently
[20:20] <cachio> cmatsuoka, yes
[20:23] <mup> PR snapd#9191 opened: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9191>
[20:28] <mup> PR snapd#9192 opened: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9192>
[21:03] <mup> PR snapcraft#3251 closed: build providers: honour http proxy settings for snapd <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3251>
[21:08] <mup> PR snapd#9193 opened: interfaces/many: deny arbitrary desktop files and misc from /usr/share - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9193>
[21:13] <mup> PR snapd#9194 opened: Interface cups slot 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9194>
[21:23] <mup> PR snapd#9195 opened: interfaces: misc policy updates xlvi - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9195>
[21:28] <mup> PR snapd#9159 closed: cmd/snap-update-ns: detach all bind-mounted file <Bug> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9159>
[21:33] <mup> PR snapd#9196 opened: osutil: add OpenExistingLockForReading <Created by zyga> <https://github.com/snapcore/snapd/pull/9196>
[21:53] <mup> PR snapd#9187 closed: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <⚠ Critical> <Created by anonymouse64> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9187>