dstathis | n 20.04 many services (in particular kubelet) are packaged as snaps | 00:09 |
---|---|---|
dstathis | Does anyone know how to correctly interact with services running as a snap? There doesn't seem to be a systemd unit | 00:09 |
dstathis | for example 'systemctl status kubelet' (or docker or whatever) fails with a not found error | 00:10 |
jamesh | dstathis: systemd units owned by snaps will have names like "snap.$snapname.$app.service". You should see them in the "systemctl list-units" output. | 01:29 |
ish | I've installed a few Snaps on Fedora 32, and those with tray icons have garbled menus... Anything obvious I'm missing? | 03:05 |
oerheks | For font issue, libpango can be a reason | 03:09 |
oerheks | or libfreetype | 03:09 |
mup | PR snapd#8301 closed: interfaces/many: deny arbitrary desktop files and misc from /usr/share <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8301> | 05:15 |
mup | PR snapd#9183 closed: tests: use "set -ex" in prep-snapd-in-lxd.sh <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9183> | 05:15 |
mborzecki | morning | 05:54 |
mvo | good morning mborzecki | 05:57 |
mborzecki | mvo: hey | 05:57 |
zyga | good morning | 06:38 |
zyga | I will be around shortly | 06:38 |
zyga | just need to send some paperwork for the insurance | 06:38 |
zyga | mvo if you can, please merge https://github.com/snapcore/snapd/pull/9175 | 06:41 |
mup | PR #9175: tests: find -ignore_readdir_race when scanning cgroups <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9175> | 06:41 |
mborzecki | zyga: hey | 06:43 |
zyga | hi | 06:45 |
mvo | zyga: looking | 06:48 |
zyga | just random test failure fix | 06:49 |
mvo | zyga: thanks, what did you change in the force push, unfortunately GH does not show me a diff for that :/ | 06:51 |
zyga | mvo I moved the -ignore_readdir_race after the path | 06:51 |
zyga | originally it was the first argument but old find didn't like that | 06:51 |
zyga | compare https://github.com/snapcore/snapd/commit/cd5b94776066bbe76359e912c960c9d4258abc9c and https://github.com/snapcore/snapd/commit/8c10ddfc32cf8f909ba73bfcfe691f174917c2e4 | 06:52 |
mvo | zyga: ta | 06:53 |
mup | PR snapd#9175 closed: tests: find -ignore_readdir_race when scanning cgroups <Simple 😃> <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9175> | 06:55 |
zyga | thank you | 06:56 |
zyga | uh, it's a meeting day | 06:56 |
zyga | just meetings and meetings | 06:56 |
pstolowski | morning! | 07:03 |
mborzecki | pstolowski: hey | 07:05 |
pstolowski | hey mborzecki, welcome back! | 07:06 |
mvo | good morning pstolowski | 07:14 |
mvo | fwiw, I'm working on the lxd test failures currently | 07:43 |
zyga | mvo thank you | 07:43 |
mvo | yw - it's very painful as each iteration takes forever | 07:44 |
mvo | but I hope I have a handle on it now (but I thought this the two previous runs too :( | 07:44 |
zyga | mvo 300+ MB download is not free | 07:44 |
pstolowski | mvo: thank you, i'm trying something as well on top of your existing PR but i'm not clear what root problem is and yes it is super slow | 07:44 |
mvo | pstolowski: http://paste.ubuntu.com/p/p6pFKnRxrb/ is my best attempt so far, test is running | 07:45 |
* mvo also wonders why wait_for_service seems to be not using the retr-tool | 07:46 | |
jamesh | mvo: I noticed the desktop code review meeting isn't on my calendar for this week. Did you cancel it, or is that a glitch? | 07:59 |
mvo | jamesh: I canceled it because we lack people but if you want to do it I can be available for you | 08:00 |
mvo | jamesh: you also should have goten a mail about this by the calendar :/ | 08:00 |
jamesh | mvo: I don't see any email about the cancellation, which is why I asked. That's fine. | 08:01 |
mborzecki | mvo: https://github.com/snapcore/snapd/pull/9150 can land too | 08:02 |
mup | PR #9150: gadget,kernel: add new kernel.{Info,Asset} struct and helpers (1/N) <Created by mvo5> <https://github.com/snapcore/snapd/pull/9150> | 08:02 |
mborzecki | mup: and https://github.com/snapcore/snapd/pull/9156 | 08:02 |
mup | mborzecki: In-com-pre-hen-si-ble-ness. | 08:02 |
mborzecki | mvo: and https://github.com/snapcore/snapd/pull/9156 | 08:03 |
mup | PR #9156: boot: copy boot assets cache to new root <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9156> | 08:03 |
jamesh | mvo: I think https://github.com/snapcore/snapd/pull/9168 is good to merge, but is failing on ubuntu-20.04-64 for some tests/main/lxd:snapd_cgroup_* tests | 08:06 |
mup | PR #9168: o/hookstate/ctlcmd: make is-connected check whether the plug or slot exists <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9168> | 08:06 |
pstolowski | jamesh: yes these tests have been investigated since yesterday | 08:07 |
jamesh | pstolowski: is it best just to wait til they get fixed then? | 08:07 |
mvo | jamesh: if the failure is just on lxd I can force-merge | 08:08 |
mvo | jamesh: merged | 08:09 |
jamesh | mvo: cheers! | 08:09 |
mvo | mborzecki: landed 9150 now too | 08:09 |
mvo | mborzecki: and 9156 | 08:10 |
mborzecki | mvo: thanks! | 08:10 |
mup | PR snapd#9150 closed: gadget,kernel: add new kernel.{Info,Asset} struct and helpers (1/N) <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9150> | 08:10 |
mup | PR snapd#9156 closed: boot: copy boot assets cache to new root <UC20> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9156> | 08:10 |
mup | PR snapd#9168 closed: o/hookstate/ctlcmd: make is-connected check whether the plug or slot exists <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9168> | 08:10 |
mvo | fwiw, 9184 passed locally now for me on 20.04, let's see if it survies a full spread run | 08:11 |
* mvo takes a short break while waiting for this | 08:11 | |
zyga-x240 | nice work | 08:18 |
pstolowski | \o/ | 09:02 |
mvo | so looks like 20.04 is now working but 16.04 is still failing :( *oh well* but with a very different error | 09:23 |
zyga-x240 | mvo: what does 16.04 say? | 09:28 |
mvo | zyga-x240: "Failed to execute operation: Connection timed out" | 09:28 |
mvo | zyga-x240: in a meeting right now, I can paste the full error late but it's also in the CI i think | 09:29 |
zyga-x240 | hmmm | 09:30 |
zyga-x240 | ijohnson: https://github.com/snapcore/snapd/pull/9187#discussion_r473820780 | 09:41 |
mup | PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <âš Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187> | 09:41 |
zyga-x240 | hmmm | 09:52 |
mborzecki | zyga-x240: hmm trouble getting self-hosted runners? https://github.com/snapcore/snapd/runs/1006992307 | 09:55 |
zyga-x240 | hmm checking | 09:57 |
zyga-x240 | the host is up | 09:57 |
zyga-x240 | workers are up | 09:57 |
zyga-x240 | I think I know what's going on | 09:58 |
zyga-x240 | it seems that cla checks are hitting a time-out to grab a worker | 09:59 |
zyga-x240 | I don't recall seeing that before, we have queueing for a reason after all | 09:59 |
zyga-x240 | I restarted that and it passed instantly | 09:59 |
zyga-x240 | so... no idea/ | 09:59 |
mborzecki | hahah | 09:59 |
mborzecki | well, maybe it's a one off occurrence | 10:00 |
zyga-x240 | I hope so but I fear we will learn in time | 10:05 |
zyga-x240 | time for coffee, I'm falling asleep here | 10:05 |
zyga-x240 | maybe pressure is low or something | 10:06 |
zyga-x240 | mvo: I looked and it looks like systemd is not responding, maybe the socket is gone somehow? | 10:06 |
zyga-x240 | but no idea why | 10:06 |
mvo | zyga-x240: so the nested lxd shows me a gazillion "permission denied", e.g. /bin/mount for / exited 1, mount: permission denied etc | 10:14 |
mvo | on 16.04 | 10:14 |
zyga-x240 | mvo: seems like lxd apparmor | 10:14 |
zyga-x240 | mount is really disallowed | 10:15 |
zyga-x240 | only bind may be allowed | 10:15 |
mvo | yeah, strange that it's inside the nested though | 10:15 |
zyga-x240 | maybe nesting is broken somehow | 10:15 |
zyga-x240 | I wonder if we can stop doing something and get nested working without snapd | 10:15 |
zyga-x240 | and then see what part breaks it | 10:15 |
mborzecki | zyga-x240: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1889318 is it because when run by lxc there's no apparmor namespace setup like lxd does? | 10:16 |
mup | Bug #1889318: install chromium in lxc container for 20.04 fails <amd64> <apport-bug> <focal> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1889318> | 10:16 |
mvo | zyga-x240: it fails even before snpad it seems | 10:16 |
mvo | zyga-x240: hm, well, maybe or maybe not | 10:16 |
zyga-x240 | mborzecki: looking | 10:16 |
zyga-x240 | I ... don't know | 10:17 |
mvo | hm, "interessting" - it fails in systemctl daemon-reload but everyting else seems to work | 10:17 |
zyga-x240 | well | 10:17 |
zyga-x240 | when systemd is not responding | 10:17 |
zyga-x240 | it's not really a place where things work | 10:17 |
mvo | the funny thing is - systemctl restart snapd works | 10:18 |
mvo | systemctl --list works | 10:18 |
mvo | afaict all the things I tried work | 10:18 |
zyga-x240 | hmm | 10:18 |
mvo | just not the daemon-reload | 10:18 |
mvo | it's a bit strange | 10:18 |
zyga-x240 | what does daemon-reload say? | 10:18 |
zyga-x240 | I mean, there is journal | 10:19 |
mvo | and I think this is broken since forever | 10:19 |
mvo | we never saw it because that script did not have set -e | 10:19 |
zyga-x240 | ohhhh | 10:19 |
zyga-x240 | that's interesting | 10:19 |
zyga-x240 | so it would fail on stable releases as well | 10:19 |
zyga-x240 | mvo: do you have 5 minutes for a quick call? | 10:19 |
jdstrand | mvo: hi! thanks for committing PR 8301! have you decided to marge master into release/2.46? if you aren't, I need to prepare a PR for 8301 (that would include 9167) for 2.46 (which is fine, just need to know that I should do it :) | 10:24 |
mup | PR #8301: interfaces/many: deny arbitrary desktop files and misc from /usr/share <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8301> | 10:24 |
jdstrand | mvo: all that is left for new PRs that small k8s-support one that I need to investigate. then I'll be doing 'needs security review' reviews | 10:26 |
mvo | jdstrand: thank you | 10:29 |
jdstrand | mvo: (also, PR 8920 needs final reviews) | 10:34 |
mup | PR #8920: interfaces: update cups-control and add cups for providing snaps <Needs Samuele review> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8920> | 10:34 |
jdstrand | mvo: do note that I intentially did not milestore PR 9186 for 2.46. that needs discussion. if there happened to be a 2.46 point release after that is merged, we could consider it, but it floating into 2.47 would be fine too | 10:36 |
mup | PR #9186: interfaces: add vcio interface <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9186> | 10:36 |
mvo | ok | 10:38 |
jdstrand | mvo: I'm looking at https://github.com/snapcore/snapd/pulls?q=is%3Aopen+is%3Apr+label%3A%22Needs+security+review%22. nothing is milestoned for 2.46. is there anything in there that should be that you would like me to prioritize? | 10:42 |
jdstrand | if not, I have a good idea of the priority | 10:43 |
zyga | I keep getting vendor.json changes | 10:49 |
zyga | I purged cache | 10:49 |
zyga | purged the tree | 10:49 |
zyga | it keeps changing | 10:49 |
jdstrand | zyga: me too! I'd love to see that resolved. (my dev container is on bionic still. wonder if it is a focal vs bionic thing) | 10:52 |
pstolowski | mvo: why is #9171 blocked? | 10:53 |
mup | PR #9171: [RFC] config: "virtual" configuration for timezone <Needs Samuele review> <Run nested> <â›” Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9171> | 10:53 |
mborzecki | zyga: jdstrand: perhapos a different version of govendor was used when vendor.json was last updated in the tree | 10:54 |
zyga | hmm | 10:54 |
mborzecki | the difference is a checksum only | 10:54 |
mborzecki | (at least here) | 10:54 |
zyga | perhaps but what's the version and who has it is interesting | 10:54 |
mborzecki | i have 1.0.9 | 10:55 |
zyga | I have 1.0.9 in /usr/bin and 1.0.8 in GOPATH | 10:56 |
mborzecki | btw. i guess we all should be using the latest master govendor, since we're go getting it in run-checks | 10:56 |
zyga | and get-deps uses that | 10:56 |
zyga | mborzecki: totally offtopic: https://github.com/snapcore/snapd/pull/9189 | 11:01 |
mup | PR #9189: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <https://github.com/snapcore/snapd/pull/9189> | 11:01 |
mup | PR snapd#9189 opened: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <https://github.com/snapcore/snapd/pull/9189> | 11:01 |
zyga | mborzecki: running 1.0.9 modifies my vendor.json | 11:02 |
mborzecki | zyga: there's a couple of recent commits to vendor.json, mine was on 3.08 (and i'm using 1.0.9), the later commits were by claudio, samuele and jamesh | 11:04 |
zyga | hmmmm | 11:04 |
zyga | the change comes from 7a9cb154a0c (Claudio Matsuoka 2020-08-13 09:08:15 -0300 115) "revision": "68200eea7bdcb97e27fe8e5ff443776383908637", | 11:05 |
zyga | so maybe claudio has older version | 11:05 |
mborzecki | let's ping him when he's online | 11:06 |
zyga | yeah | 11:06 |
mvo | pstolowski: I think 9171 really needs samuele approval, I think it's okay otherwise, if the design gets approval I would like to tweak it a bit more with some helpers | 11:10 |
pstolowski | mvo: i made a few comments, not sure what was already discussed and agreed, so maybe my comments make no sense | 11:22 |
mvo | pstolowski: cool, happy for any feedback at this point | 11:22 |
mvo | pstolowski: hm, great point about snap get -d | 11:24 |
mvo | pstolowski: I think you are right, we should probably not bypass this for that | 11:24 |
* mvo scratches head | 11:24 | |
pstolowski | mvo: yes i think we are breaking some high level assumptions here. and i fear it's going to be annoying to handle :/ | 11:25 |
mvo | pstolowski: yeah, this needs more discussion for sure | 11:25 |
mvo | pstolowski: your comment about the transations is also interessting, maybe we need to hook into commit() here instead | 11:26 |
mvo | lxd tests passed locally in 9184 | 11:27 |
* mvo vanishes for lunch | 11:27 | |
zyga | enjoy | 11:27 |
pstolowski | mvo: what if we do store in state but synchronize config state with system setup? or is it too terrible? | 11:27 |
* pstolowski lunch as well | 11:29 | |
zyga | pstolowski: who wins? | 11:29 |
zyga | pstolowski: if system was modified when snapd was down? | 11:29 |
pstolowski | zyga: system always wins. we update system on snap set. | 11:30 |
pstolowski | but maybe it's naive | 11:30 |
pstolowski | just throwing ideas | 11:30 |
zyga | pstolowski: so when would we use the value from state? | 11:31 |
pstolowski | zyga: we would always update state from system before reading. that could simplify transaction logic without special-casing. but just brainstorming at this point | 11:33 |
pstolowski | anyway, time for lunch | 11:33 |
zyga | pstolowski: I see | 11:33 |
zyga | I don't know either, just trying to understand your point better | 11:34 |
cachio | zyga, hi | 11:41 |
cachio | zyga, do you have any idea about what could be causing that when I test beta image and do "systemctl --user daemon-reload" as root I get "Failed to connect to bus: No such file or directory | 11:42 |
cachio | " | 11:42 |
cachio | If I do that as ubuntu user I dont see any error, but as root I see that error | 11:42 |
cachio | and it makes fail the snapd-failover test | 11:42 |
zyga | cachio: hi | 11:43 |
zyga | cachio: yes, I explained that a few days ago | 11:43 |
zyga | cachio: when we reload systemd-logind.service on older versions of systemd | 11:43 |
zyga | cachio: we cause it to forget about the session of the root user | 11:43 |
zyga | cachio: then subsequent test that prepares a session for the root user | 11:43 |
zyga | cachio: enables linger, which starts services and so on | 11:44 |
zyga | cachio: but then the restore disables linger | 11:44 |
zyga | cachio: because systemd logind is no longer tracking the incoming ssh session of the root user | 11:44 |
zyga | cachio: it shuts down systemd --user for root | 11:44 |
cachio | zyga, do you know which is the difference between the test we run in github and the one I run in beta validation to explain why it works in github and fails with the beta image? | 11:46 |
zyga | cachio: I don't know what version beta was forked form | 11:46 |
zyga | cachio: please check if it contains changes to prepare-restore.sh | 11:47 |
zyga | talking exactly about this issue in the commit message | 11:47 |
cachio | zyga, I see the change | 11:51 |
cachio | I need to see how to make it for external backend | 11:51 |
cachio | thanks for the explanation | 11:51 |
zyga | cachio: the fix I made is generic | 11:54 |
zyga | cachio: if you have it and the bug persists then there's something more broken | 11:54 |
cachio | zyga, yes | 11:54 |
cachio | but in case of external backend we exit before that code | 11:54 |
zyga | cachio: output from loginctl list-sessions would help | 11:54 |
zyga | cachio: when do we exit then? | 12:00 |
cachio | zyga, most of the prepare_project is not done for external backend | 12:00 |
cachio | I am trying to move the linger configuration to see if it works | 12:01 |
zyga | cachio: I see | 12:02 |
zyga | cachio: yeah, you need to apply the fixes to systemd-logind | 12:02 |
zyga | cachio: those are one-off | 12:02 |
zyga | cachio: do you perform package upgrades? what's the target? | 12:02 |
zyga | is that core16? | 12:03 |
cachio | yes | 12:03 |
zyga | core16 needs a special workaround | 12:04 |
zyga | in essence, I repackage core | 12:04 |
zyga | though recently ijohnson applied a fix to core so maybe that is no longer required | 12:04 |
zyga | following that /var/lib/systemd/linger is bind-mounted to writable using a mount unit | 12:05 |
zyga | look at the patches I apply to replicate that | 12:05 |
zyga | I wasn't aware the external target skips all of that | 12:05 |
cachio | zyga, no problem, I'll try to extend that to external | 12:05 |
zyga | you may only need the 1) mount unit 2) change to logind configuration followed up by REBOOT | 12:05 |
ijohnson | zyga: the core fix has not landed yet, PR is still open | 12:37 |
ijohnson | also morning folks | 12:37 |
ijohnson | hey mborzecki welcome back | 12:37 |
cachio | pstolowski, hey, any idea about thie error https://paste.ubuntu.com/p/SRTGRMHx45/ | 12:45 |
cachio | it is happening in Core20-early-config test | 12:45 |
cachio | I see this gadget.yaml parse error: Duplicate key: system | 12:46 |
cachio | not sure if it could be the raeson | 12:46 |
pstolowski | looking | 12:48 |
zyga | ijohnson: I see | 12:49 |
* zyga was having pierogis for lunch | 12:49 | |
ijohnson | cachio: that was what I fixed in my PR | 12:49 |
pstolowski | cachio: yeah, definately | 12:49 |
zyga | cachio: there's a duplicate key :) | 12:49 |
ijohnson | cachio: that is fixed by #9187 | 12:49 |
mup | PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <âš Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187> | 12:49 |
pstolowski | right, i was just going to say that :), thanks ijohnson | 12:50 |
cachio | ijohnson, ah, thanks!!! | 12:50 |
mvo | ijohnson: should I force merge 9187? there are still failures in nested | 12:58 |
cachio | mvo, the errors in nested are fixed in other pr | 12:59 |
ijohnson | mvo: let me look quickly | 12:59 |
cachio | in mine | 12:59 |
ijohnson | mvo: the tests are still failing | 12:59 |
ijohnson | also SU time now | 12:59 |
cachio | pr: #9098 | 12:59 |
mup | PR #9098: tests: new organization for nested tests <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9098> | 12:59 |
ijohnson | mvo: sorry I meant the tests still seem to be running? | 13:00 |
cachio | but mine fails sometimes because the erorr which is fixed on 9187 | 13:00 |
cachio | ijohnson, I retriggered the tests | 13:00 |
mup | PR snapd#9081 closed: secboot,cmd/snap-bootstrap: cross-check partitions before unlocking, mounting <Run nested> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9081> | 13:16 |
zyga-mbp | oh, and I'm making progress on unit testing bash, https://listed.zygoon.pl/ has the details | 13:37 |
cachio | zyga-mbp, so I see we do https://github.com/snapcore/snapd/blob/master/tests/lib/prepare.sh#L662 | 13:40 |
zyga-mbp | right | 13:40 |
cachio | we need that for external as well? | 13:40 |
zyga-mbp | you want that and the configuration change immediately below that | 13:40 |
zyga-mbp | yes | 13:40 |
cachio | ok, that plus the change https://github.com/snapcore/snapd/blob/master/tests/lib/prepare-restore.sh#L446 | 13:41 |
zyga-mbp | cachio note that this is the static part, the dynamic part is where we decide systemd-logind needs reloading and REBOOT | 13:41 |
zyga-mbp | yes | 13:42 |
cachio | perfect | 13:42 |
zyga-mbp | we try to enable linger for the test user, if that fails we know we need to restart | 13:42 |
zyga-mbp | note that this does essentially the same change (configuration file edit) so make sure to just reboot in that case as the config file is already in place, just inactive | 13:42 |
cachio | ok | 13:43 |
zyga-mbp | I hope this works :) | 13:43 |
zyga-mbp | cachio I left a small comment on v | 13:43 |
zyga-mbp | https://github.com/snapcore/snapd/pull/8986/files | 13:43 |
mup | PR #8986: tests: new snaps-state command - part1 <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/8986> | 13:43 |
cachio | zyga-mbp, thanks!! | 13:44 |
zyga | cachio: any luck? | 14:26 |
cachio | zyga, no yes, trying in 5 minutes | 14:26 |
zyga | ok | 14:26 |
cachio | still making some changes | 14:26 |
mup | PR snapd#9188 closed: interfaces: misc policy updates xlvi <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9188> | 14:42 |
ijohnson | mvo: 9184 is super super green :-) | 14:46 |
mup | PR snapd#9190 opened: [RFC] cmd/s-b/initramfs-mounts: make recover -> run mode transition automatic <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9190> | 14:47 |
cachio | zyga-mbp, Could not enable linger: No such file or directory | 14:48 |
cachio | I see that | 14:48 |
zyga-mbp | cachio, ok do you have a shell | 14:49 |
cachio | when I do loginctl enable-linger test | 14:49 |
cachio | yes | 14:49 |
zyga-mbp | is the logind.conf file modified? | 14:49 |
cachio | yes | 14:49 |
cachio | StateDirectory=systemd/linger | 14:49 |
zyga-mbp | is /var/lib/systemd/linger a mount point to the corresponding directory in writable? | 14:49 |
cachio | with this config | 14:49 |
cachio | I created that in writable | 14:49 |
zyga-mbp | that's not all, is a mount in place? | 14:50 |
cachio | I cant write to /var/lib/systemd/linger | 14:50 |
zyga-mbp | is there a mount unit that makes /var/lib/systemd/linger a bind mount to /writable/system-data/.... | 14:50 |
cachio | it is protected | 14:50 |
zyga-mbp | you need the linger directory to exist | 14:51 |
zyga-mbp | and it must be a mount point as I've explained | 14:51 |
zyga-mbp | no way around that | 14:51 |
zyga-mbp | THEN you can reboot to reconfigure logind | 14:51 |
zyga-mbp | (via REBOOT) | 14:51 |
cachio | I cant create /var/lib/systemd/linger | 14:51 |
zyga-mbp | and then it will work | 14:51 |
zyga-mbp | cachio so merge the core change and rebuild the snap | 14:51 |
cachio | which is the change to merge? | 14:52 |
zyga-mbp | ian proposed a PR for core | 14:52 |
cachio | zyga-mbp, ok, so no workaround until the chagne in core is merged | 14:53 |
zyga-mbp | correct | 14:53 |
zyga-mbp | unless you can repackage core | 14:53 |
cachio | zyga-mbp, agree but on beta validation we don't repack core | 14:54 |
cachio | so I'll push the change done by ian | 14:54 |
cachio | ijohnson, is this the change? https://github.com/snapcore/core/pull/116 | 14:55 |
mup | PR core#116: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116> | 14:55 |
ijohnson | yes | 14:55 |
cachio | waiting for a second review? | 14:56 |
cachio | ijohnson, | 14:56 |
zyga-mbp | ijohnson what happens when we remove entries? | 14:56 |
ijohnson | cachio: I can actually merge it, not sure if I should wait though | 14:57 |
ijohnson | zyga-mbp: what do you mean ? | 14:57 |
zyga-mbp | ijohnson this looks good but may need follow ups for hacks in prepare-restore | 14:57 |
zyga-mbp | ijohnson the removal of /var/lib/systemd/rfkill and random-seed lines | 14:57 |
ijohnson | hmm | 14:57 |
zyga-mbp | I'd +1 without those | 14:58 |
zyga-mbp | but now that I see them | 14:58 |
zyga-mbp | I would not keep them | 14:58 |
zyga-mbp | I just want to understand what happens in practice | 14:58 |
ijohnson | I need to think about this, I don't remember how refreshes work, but I think it's just the initramfs that modifies the etc/fstab according to the writable-paths | 14:58 |
zyga-mbp | on existing systems that shipped with those lines | 14:58 |
zyga-mbp | cachio can you test that upgrade | 14:58 |
ijohnson | so after a refresh to the revision here it should work fine | 14:58 |
zyga-mbp | I think it's very important for correctness | 14:58 |
ijohnson | there have been other situations where we drop things there though that have landed without controversy however I think | 14:59 |
ijohnson | one moment | 14:59 |
zyga-mbp | ijohnson we are trigger happy sometimes | 14:59 |
pstolowski | mvo: +1 for #9184 | 15:00 |
mup | PR #9184: packaging: umount /snap on purge in containers <âš Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184> | 15:00 |
ijohnson | zyga-mbp: see https://github.com/snapcore/core18/pull/127 | 15:02 |
mup | PR core18#127: Make /var/cache writable and disable dynamic motd <Created by woodrow-shen> <Merged by vorlonofportland> <https://github.com/snapcore/core18/pull/127> | 15:02 |
zyga-mbp | ijohnson in another call | 15:02 |
cachio | zyga-mbp, ijohnson sorry, which upgrade? | 15:05 |
ijohnson | cachio: nvm I can test the upgrade, it is about the writable-paths changes | 15:05 |
cachio | ijohnson, ah, ok, thanks | 15:06 |
zyga-mbp | re | 15:13 |
zyga-mbp | cachio check if we can upgrade core from what it looks like now to what is proposed in that PR 127 | 15:13 |
mup | PR #127: Use cap instead of cap1 where only one capability is being manipulated <Created by zyga> <Merged by elopio> <https://github.com/snapcore/snapd/pull/127> | 15:13 |
zyga-mbp | cachio to see if anything explodes | 15:13 |
zyga-mbp | I mean | 15:13 |
zyga-mbp | we'll know if we land it | 15:13 |
zyga-mbp | and run tests | 15:13 |
zyga-mbp | but a quick run might be helpful as well | 15:14 |
zyga-mbp | mvo ^ do you know what happens if we remove existing entries from writable paths? | 15:14 |
ijohnson | ugh building the core snap is so annoying | 15:16 |
ijohnson | I wish we could update the core snap to build with modern snapcraft | 15:16 |
cachio | ijohnson, yes | 15:16 |
cachio | tomorrow I can test that we new core is in edge | 15:17 |
ijohnson | cachio: but the PR hasn't been merged though is the thing | 15:17 |
cachio | I really don't know how to build core | 15:17 |
ijohnson | I just built it locally though | 15:17 |
ijohnson | cachio: do you use wormhole? I can send it to you in a few moments | 15:17 |
cachio | I can test that if you built core? | 15:17 |
cachio | ijohnson, yes | 15:17 |
cachio | please send me it | 15:18 |
mvo | zyga-mbp: if we remove writable-path they stop being writable which may break things - what in particular are you asking? | 15:18 |
zyga-mbp | mvo please look at https://github.com/snapcore/core/pull/116/files | 15:18 |
mup | PR core#116: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <https://github.com/snapcore/core/pull/116> | 15:18 |
zyga-mbp | are those changes safe in your eyes | 15:18 |
ijohnson | zyga-mbp: fwiw he already approved it | 15:19 |
zyga-mbp | I know but not sure if he really looked ;D (sorry mvo) | 15:19 |
mvo | zyga-mbp: oh, this should be fine | 15:19 |
zyga-mbp | mvo in that case let's land it | 15:19 |
zyga-mbp | and rebuild core | 15:19 |
mvo | zyga-mbp: removing subpath but making more available is fine | 15:19 |
zyga-mbp | we can always revert | 15:19 |
zyga-mbp | and this will help cachio | 15:19 |
mvo | zyga-mbp: please approve first | 15:20 |
ijohnson | well I have it built now, testing what happens on refresh of a core16 vm | 15:20 |
ijohnson | cachio: I don't think you need to test what I built here anymore | 15:20 |
cachio | ok | 15:21 |
mup | PR snapcraft#3250 closed: cli: implement list-tracks <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3250> | 15:23 |
mup | PR snapcraft#3252 closed: snapcraft: use system certificates by default for https requests <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3252> | 15:23 |
ijohnson | jdstrand: is this a typo or intentional? https://github.com/snapcore/snapd/pull/9188/files#r474066839 | 15:23 |
mup | PR #9188: interfaces: misc policy updates xlvi <Created by jdstrand> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9188> | 15:23 |
* zyga-mbp needs to break soon | 15:24 | |
mup | PR snapcraft#3255 closed: remote-build: use requests.get() instead of urlopen() <bug> <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3255> | 15:28 |
mvo | ijohnson: if you test is successful please let me know and I will merge | 15:32 |
ijohnson | mvo: yes sorry got distracted it is rebooting now | 15:32 |
mvo | ijohnson: no worries | 15:32 |
ijohnson | mvo: mmm the VM didn't come back after the refresh | 15:38 |
ijohnson | mvo: need to debug, perhaps there is a problem | 15:38 |
mvo | ta | 15:39 |
* cachio lunch | 15:43 | |
mvo | ijohnson: uh, I accidently merged 116 :/ let's hope it's not a problem with that change or I have to revert | 15:44 |
ijohnson | mvo: my hope is that this is a multipass problem, I am recreating a uc16 VM with plain qemu :-/ | 15:45 |
mup | PR core#116 closed: extra-files/writable-paths: make all /var/lib/systemd writable <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/core/pull/116> | 15:46 |
* zyga runs away from meetings and works for a while in quiet | 15:46 | |
zyga | mvo: shall we merge https://github.com/snapcore/snapd/pull/9184 | 15:47 |
mup | PR #9184: packaging: umount /snap on purge in containers <âš Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184> | 15:47 |
zyga | just to make some progress and see how it behaves over time | 15:48 |
ijohnson | I'd say so | 15:48 |
zyga | yeah | 15:49 |
zyga | me too | 15:49 |
dstathis | Has anyone here used kubelet installed via snap? If so do you know how to get it to use the systemd cgroup driver? | 15:51 |
ijohnson | joedborg: can you answer dstathis question ? | 15:52 |
ijohnson | mmm mvo, better revert that core PR, I can reproduce the problem in qemu, I need to debug this, but it somehow broke ssh | 15:56 |
ijohnson | so sorry :-/ | 15:56 |
joedborg | Hey dstathis. We’re currently working on this under strict confinement for both microk8s and a fully working kubernetes node (I.e. kubelet, proxy, containerd etc). This isn’t complete yet though as there are lots of working parts that need working through | 15:56 |
dstathis | joeborg: The snap of kubelet is not complete yet? I'm asking because kubelet is no longer available via apt. Only snap | 15:58 |
dstathis | (in Ubuntu 20.04) | 15:59 |
mvo | ijohnson: my bad, I should not have merged prematurely | 16:01 |
ijohnson | no, it's my bad I didn't test an actual refresh of it | 16:02 |
zyga | mvo: https://github.com/snapcore/snapd/pull/9184#pullrequestreview-471764868 can I merge? | 16:02 |
mup | PR #9184: packaging: umount /snap on purge in containers <âš Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9184> | 16:02 |
mvo | zyga: yes, I'm still trying to create a reproudce for the lxd team about the systemctl daemon-reload | 16:03 |
joedborg | dstathis: that is usually used by our kubernetes distribution Charmed Kubernetes. It is a classic snap, so shouldn’t need any interfaces connected. However it’s not really documented outside of CK context | 16:03 |
mvo | zyga: but merging will make things work again | 16:03 |
zyga | mvo: ok | 16:03 |
zyga | merging | 16:03 |
zyga | it's ok to revert if it explodes in new ways | 16:03 |
dstathis | joedborg: is there another recommended way to install kubelet on 20.04? Or do you know how to configure the cgroup diver using the snap? | 16:04 |
mup | PR snapd#9184 closed: packaging: umount /snap on purge in containers <âš Critical> <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9184> | 16:07 |
joedborg | dstathis: I don’t know for sure, but I’m reasonably confident you do `snap set kubelet [kubelet arg]=[value] | 16:09 |
joedborg | You need to swap hyphens for underscores i.e `api-server` becomes `api_server` | 16:10 |
ijohnson | how does a change to /var/lib/systemd/private cause `sshd[1151]: Missing privilege separation directory: /var/run/sshd` :-( | 16:10 |
dstathis | joedborg: is there a way to ask the snap which configuration options it will recognize? I know there is a command line argument that can be passed in this case | 16:12 |
joedborg | dstathis: sadly not as snap config is key value without any firm validation | 16:12 |
mvo | ijohnson: pushed the revert now | 16:29 |
* mvo gets dinner | 16:29 | |
ijohnson | thanks, I am very puzzled how this is broken | 16:29 |
mvo | yeah | 16:31 |
zyga | what is broken? | 16:31 |
zyga | I heard qemu | 16:31 |
ijohnson | zyga: my writable-paths PR somehow breaks ssh | 16:32 |
zyga | ijohnson: aha | 16:32 |
zyga | maybe it only breaks ssh in testing | 16:32 |
ijohnson | so you were right :-O | 16:32 |
zyga | what are the symptoms? | 16:32 |
zyga | ijohnson: ha | 16:32 |
dstathis | joedborg: 'sudo snap set kubelet cgroup-driver=systemd' worked. Thanks | 16:32 |
zyga | ijohnson: heh, I didn't mean to | 16:32 |
ijohnson | zyga: it makes the ssh server service fail to start like this: | 16:33 |
ijohnson | https://www.irccloud.com/pastebin/NbSCym9q/ | 16:33 |
zyga | interesting | 16:35 |
zyga | but how is that related to what you did? | 16:35 |
ijohnson | I have no idea | 16:35 |
zyga | maybe we neutered the whole directory | 16:35 |
zyga | there used to be more state in that systemd tree | 16:35 |
zyga | now it's all an empty directory | 16:35 |
zyga | maybe that's why? | 16:35 |
ijohnson | zyga: no the other files are still there | 16:36 |
zyga | oh | 16:36 |
zyga | I'll look at ssh source | 16:36 |
ijohnson | zyga: the files are /var/lib/systemd/... and those are still there in the core snap after I made this change | 16:36 |
ijohnson | unless systemd is doing something weird with /var/lib/private | 16:36 |
zyga | hah | 16:36 |
zyga | do we have /var/empty? | 16:37 |
zyga | this is what this is complaining about | 16:37 |
zyga | ah, no | 16:37 |
zyga | sorry | 16:37 |
zyga | confflags += --with-privsep-path=/run/sshd | 16:37 |
ijohnson | zyga: /var is not empty | 16:38 |
zyga | if /run/sshd doesn't exist I'd love to log in interactively | 16:38 |
zyga | and see the status | 16:38 |
zyga | of all the services | 16:38 |
zyga | I bet something else failed | 16:38 |
ijohnson | mmm actually there is a tmpfiles failure earlier | 16:38 |
zyga | ijohnson: can we retry with just added linger | 16:38 |
zyga | and then circle back with more oomph | 16:38 |
ijohnson | https://www.irccloud.com/pastebin/4GceHjg4/ | 16:39 |
zyga | unsafe symlinks? never heard that one before | 16:39 |
zyga | checking | 16:39 |
ijohnson | yeah very odd | 16:40 |
zyga | I cannot find that in current systemd | 16:42 |
zyga | trying older | 16:42 |
zyga | ijohnson: I think this explains it | 16:44 |
zyga | https://github.com/systemd/systemd/issues/14226#issuecomment-560467107 | 16:44 |
ijohnson | mmmm interesting let me see | 16:45 |
zyga | ijohnson: I'm too tired to build core and boot it in qemu | 16:45 |
zyga | I wish there was a fire-and-forget for that | 16:45 |
ijohnson | zyga: no worries thank you for looking! | 16:46 |
ijohnson | I am actively debugging this | 16:46 |
ijohnson | ha actually I'm a fool I didn't even build the right branch, so this core snap that I built is just master | 16:48 |
zyga | ijohnson: it would be amazing if we could test core against snapd test suite | 16:48 |
zyga | just build it there and clone snapd in an action | 16:48 |
zyga | and set some env in a spread run | 16:48 |
zyga | so it gets that core | 16:48 |
zyga | I think it's doable now | 16:49 |
ijohnson | it would be amazing if I didn't need all this chroot and vm nonsense with a legacy snapcraft to build the core snap | 16:49 |
ijohnson | and I could just do `snapcraft --use-lxd` | 16:49 |
zyga | ijohnson: we could use one action to build it and store an artefact | 16:49 |
zyga | ijohnson: and then another action to test it | 16:49 |
ijohnson | zyga: sorry what do you mean? for the snapd repo ? | 16:50 |
zyga | run all of snapd tests against a core built from a branch of the core repo | 16:51 |
zyga | right? | 16:51 |
zyga | so we get way better coverage | 16:51 |
ijohnson | mmm that would be nice | 16:52 |
zyga | the checkout action can checkout any repo | 16:53 |
zyga | so we have 90% of what we want in the snapd repo already | 16:53 |
zyga | we just need to teach snapd test suite to use a core that's wgettable from somewhere | 16:54 |
zyga | it can still rebuild / repack snapd | 16:54 |
zyga | but the base would be from the PR changing the core snap | 16:54 |
zyga | we could even write a test that runs only in this mode | 16:54 |
zyga | where a vanilla core can refresh to the new core | 16:55 |
zyga | and boot | 16:55 |
zyga | anyway | 16:55 |
zyga | you get the point | 16:55 |
ijohnson | also :-( I think the whole reason this failed is because I forgot to do this: | 16:55 |
ijohnson | - sudo sed -i '/^deb/s/$/ universe/' $CHROOT/etc/apt/sources.list | 16:55 |
ijohnson | so the snap I built built from the wrong packages | 16:55 |
ijohnson | *sighes | 16:56 |
=== ijohnson is now known as ijohnson|lunch | ||
zyga | oh? | 16:58 |
zyga | oh well | 16:58 |
* zyga suspends and takes a break | 16:58 | |
ijohnson|lunch | https://github.com/snapcore/core/blob/master/.travis.yml is how I build all my core snaps | 16:58 |
ijohnson|lunch | but just doing that locally | 16:58 |
ijohnson|lunch | and I missed a step | 16:58 |
cachio | zyga-mbp, this test is failing as well | 17:29 |
cachio | https://paste.ubuntu.com/p/GXdqXc4YWg/ | 17:29 |
cachio | could be related to the revious issue ? | 17:30 |
zyga-mbp | it's the same failure | 17:30 |
zyga-mbp | yeah | 17:30 |
zyga-mbp | it's expected | 17:30 |
cachio | or you think it is something different | 17:30 |
cachio | zyga-mbp, nice, thanks, I thought that but just wanted to make sure | 17:30 |
cachio | thanks | 17:30 |
zyga-mbp | cool | 17:31 |
=== King_InuYasha is now known as Conan_Kudo | ||
=== Conan_Kudo is now known as King_InuYasha | ||
=== King_InuYasha is now known as Conan_Kudo | ||
=== Conan_Kudo is now known as King_InuYasha | ||
=== ijohnson|lunch is now known as ijohnson | ||
* cachio afk | 17:57 | |
mup | PR snapd#9189 closed: snap/snapenv: set SNAP_REAL_HOME <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9189> | 18:02 |
joedborg | dstathis: ah thanks for letting me know | 18:40 |
cachio | cmatsuoka, https://github.com/snapcore/snapd/pull/9098/checks?check_run_id=1008878037#step:5:7412 | 19:28 |
mup | PR #9098: tests: new organization for nested tests <Run nested> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9098> | 19:28 |
cmatsuoka | cachio: let's see... | 19:28 |
cachio | now without any error during reboot | 19:29 |
cachio | during boot | 19:29 |
cachio | no reboots involved | 19:29 |
cmatsuoka | ah nice! | 19:29 |
cmatsuoka | excellent | 19:29 |
cmatsuoka | ah wait | 19:29 |
cmatsuoka | but it failed? | 19:29 |
cmatsuoka | ugh | 19:30 |
cachio | couldnt login after that | 19:30 |
cmatsuoka | and this is qemu without kvm? | 19:30 |
cachio | yes | 19:30 |
cmatsuoka | let me see this log in detail | 19:31 |
cachio | ijohnson, I merged master and still see | 19:32 |
cachio | 2020-08-20T18:18:54.2486726Z + mount /dev/mapper/ /tmp/tmp.wFbPmXutiu | 19:32 |
cachio | 2020-08-20T18:18:54.2486835Z mount: /tmp/tmp.wFbPmXutiu: /dev/mapper is not a block device. | 19:32 |
cachio | sorry, thought that #9187 was merged | 19:33 |
mup | PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <âš Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187> | 19:33 |
ijohnson | cachio yeah it's not merged, needs mvo to merge it though maybe now If I merge master to that PR it will be green | 19:34 |
cachio | ijohnson, I see lxd tests failing there https://github.com/snapcore/snapd/pull/9187/checks?check_run_id=1007716427#step:5:7003 | 19:34 |
mup | PR #9187: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <âš Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9187> | 19:34 |
ijohnson | cachio yeah let me merge master to this pr so we have some chance of being all green | 19:40 |
cachio | ijohnson, nice, thanks | 19:42 |
ijohnson | cachio: running now, let's see how it does | 19:43 |
cachio | cool | 19:43 |
cmatsuoka | cachio: I think I found the problem | 19:43 |
cachio | cmatsuoka, awesome | 19:44 |
cachio | which one? | 19:44 |
cachio | which is? | 19:44 |
cmatsuoka | I think there's a call missing there, I'm checking the change history to see if this is was lost at some refactoring, or it was never there | 19:47 |
cachio | cmatsuoka, nice | 19:47 |
cmatsuoka | cachio: I'm checking a possible fix with chris | 20:11 |
cachio | cmatsuoka, nice, thanks | 20:16 |
cachio | please ping me if you need to test it | 20:16 |
cmatsuoka | cachio: thanks, I think we'll need to test it a lot since it seems that it happens infrequently | 20:17 |
cachio | cmatsuoka, yes | 20:20 |
mup | PR snapd#9191 opened: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9191> | 20:23 |
mup | PR snapd#9192 opened: interfaces/{docker,kubernetes}-support: load overlay and support systemd cgroup driver - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9192> | 20:28 |
mup | PR snapcraft#3251 closed: build providers: honour http proxy settings for snapd <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3251> | 21:03 |
mup | PR snapd#9193 opened: interfaces/many: deny arbitrary desktop files and misc from /usr/share - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9193> | 21:08 |
mup | PR snapd#9194 opened: Interface cups slot 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9194> | 21:13 |
mup | PR snapd#9195 opened: interfaces: misc policy updates xlvi - 2.46 <Created by jdstrand> <https://github.com/snapcore/snapd/pull/9195> | 21:23 |
mup | PR snapd#9159 closed: cmd/snap-update-ns: detach all bind-mounted file <Bug> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9159> | 21:28 |
mup | PR snapd#9196 opened: osutil: add OpenExistingLockForReading <Created by zyga> <https://github.com/snapcore/snapd/pull/9196> | 21:33 |
mup | PR snapd#9187 closed: tests/lib/nested.sh: use more robust code for finding what loop dev we mounted <Run nested> <Test Robustness> <âš Critical> <Created by anonymouse64> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9187> | 21:53 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!