[03:39] PR snapd#8429 opened: github: port CLA check to Github Actions [05:40] morning [06:20] mvo: hey [06:21] Hey [06:21] Please merge my last PR into master and merge master into your branches. [06:21] zyga: hey [06:22] zyga: here's a simple one porting another job off Travis: https://github.com/snapcore/snapd/pull/8429 [06:22] PR #8429: github: port CLA check to Github Actions [06:22] zyga: which one is that? this one https://github.com/snapcore/snapd/pull/8428 [06:22] PR #8428: tests/session-tool: stop cron/anacron from meddling [06:22] jamesh: thanks! [06:23] zyga: also, I'm not convinced that the use of actions/cache@v1 for the apt cache actually works [06:23] (the way it is written, I don't think it will ever get a cache hit) [06:23] That one [06:24] jamesh: I noticed [06:24] Something is off [06:24] Do you know what the issue is? [06:24] hey mborzecki, good morning [06:25] jamesh: quick question about gtk2, we have gtk2-common-themes snap, but one would still need the right theme engine to render those, isn't that right? [06:26] jamesh: asking in the context of https://forum.snapcraft.io/t/inkscape-snap-uses-a-windows-98-era-window-scheme/16432/3 which looks like the snap plugs gtk2-common-themes, but when starting there's a log about missing pixbuf engine to render the adwaita-like theme with gtk2 [06:28] zyga: ah. you're doing the chown on /var/cache/apt after calling actions/cache, so the action does not have permission to write to that location [06:29] Mmmmm [06:30] So we need two chowns [06:30] zyga: It's not obvious that it is a good use of caching anyway: the cache key you're using doesn't specify exact revisions of packages, so each user could be testing against a different set of packages [06:30] Or update the action to a version that fixed this and uses sudo [06:30] We rarely rev our dependencies [06:31] mborzecki: gtk2-common-themes only contains theme engines. We left the actual theme data files in gtk-common-themes, since most GTK 2 themes are distributed beside their GTK 3 counterpart these days [06:31] So it is not a perfect set but a sufficient set [06:31] jamesh: so gtk2-common-themes is effectively dead? [06:32] mborzecki: no. You need theme data files plus any engines the theme uses [06:32] mborzecki: this is also a split between platform independent data files (gtk-common-themes), and platform dependent engines (gtk2-common-themes) [06:33] maybe we should have named the second package gtk2-common-engines [06:39] zyga: It might also be a good idea to combine some of the spread tests back into fewer jobs: it's nice not having mingled output, but we're obviously hitting Github's runner limits now [06:40] jamesh: I will surely consider that but over weekend we also hit limits on our account in GCE [06:40] now that we actually run spread when we push stuff [06:40] we run out of capacity quickly [06:40] we need to measure and count and decide [06:41] jamesh: hmm so the gtk2-common-themes engines part content is connected, the snap sets GTK_PATH in meta/snap.yaml, but once the process runs GTK_PATH is set to a different path :/ looks like it's coming from desktop-launch [06:45] * zyga breakfast [06:48] jamesh: looks like GTK_PATH ends up without the location where the engines from gtk2-common-themes snap are, i added a note under https://forum.snapcraft.io/t/inkscape-snap-uses-a-windows-98-era-window-scheme/16432/3 [06:56] zyga: when you have a moment, a look/merge of https://salsa.debian.org/debian/snapd/-/merge_requests/5 would be great [06:58] ehh 1080p is such a downgrade from 4k [06:58] o/ [06:58] at my desk [06:58] mvo: looking [06:59] mborzecki: display change? [06:59] morning [06:59] zyga: yeah, had to rma the 4k one, so i'm using my old 1080p display now [07:00] zyga: not ugent but it will bring the packaging git trees in sync again [07:00] pstolowski: hey [07:00] ta [07:00] good morning pawel [07:00] zyga: which is ironic, since it's also lg, 10+ years and not a single bad pixel [07:00] hmm, salsa doesn't load for me, feels like none of the JS bits are getting loaded [07:03] mvo: commits 8482? is this bundled with development history? [07:04] (this explains why js is slow to load) [07:05] zyga: yes, I think that's how we setup our branch [07:05] zyga: it contains all the history [07:05] ta [07:06] zyga: TBH i have no idea why this style is popular, I think it's kinda useless [07:07] maybe easier for cherry picking patches [07:07] libzt uses separate history [07:07] anyway [07:07] approved [07:07] PR snapd#8430 opened: packaging: use debian/not-installed to ignore snap-preseed [07:08] mvo: I see you have two more extra PRs there [07:08] but I guess they are obsolete now [07:09] mvo: closed both, please shout at me if I was meant to land them instead [07:10] zyga: probably fine, I forgot those [07:10] zyga: that was when we were not directly involved I think [07:10] aha [07:10] looking at 8430 [07:10] do you know what's broken in our sid tests? [07:11] mvo: can you review and land https://github.com/snapcore/snapd/pull/8428 please [07:11] PR #8428: tests/session-tool: stop cron/anacron from meddling [07:11] it will prevent random failures on centos 7 [07:11] looking at this branch I also wonder if we should do a pass and review the timers and background services in our images [07:11] who knows what else is randomly running [07:12] zyga: yeah, I have a look soon, just need to finish my current task [07:12] thanks [07:13] zyga: is there anything like "skip-spread" in the new gh-actions world? just an idle thought [07:13] mvo: not yet, we can certainly code it [07:13] actions has enough logic for it [07:14] https://github.community/t5/GitHub-Actions/Labeled-action-on-pull-request/m-p/37839#M3093 has some ideas [07:14] we can just get the context as a json object, select the labels and skip the whole spread part if a particular one is set [07:14] mvo: if it's not urgent I'll do it in the evening [07:15] actually [07:15] zyga: nice, not urgent at all [07:15] https://stackoverflow.com/questions/59588605/how-to-check-for-a-label-in-a-github-action-condition [07:15] is even easier [07:15] if: contains( github.event.pull_request.labels.*.name, 'My Label') [07:15] we could just do [07:15] if: !contains( github.event.pull_request.labels.*.name, 'Skip Spread') [07:16] PR snapd#8428 closed: tests/session-tool: stop cron/anacron from meddling [07:16] thanks for that mvo ^ ! [07:17] pstolowski, jamesh, mborzecki, mvo: gentle reminder to merge master into any branches you push or iterate on [07:17] mainly to get the fail-fast: false part [07:17] ack [07:17] I will share details at the standup [07:22] mvo: pstolowski: hi, I would appreciate reviews for https://github.com/snapcore/snapd/pull/8427 it's huge but is splitting files up basically, except for reorganizing things such that we can have mutiple store* test suites [07:22] PR #8427: store: start splitting store.go and store_test.go into subtopic files [07:23] over weekend I noticed the ubuntu archive was failing again, I reported this to IS but they asked for more details. If this happens frequently we should cook a patch that prints the IP address of the node in case "apt-get *" fails [07:23] pedronis: sure, will do [07:38] pedronis: ok, thanks, I have a look. so this is no code change, just reorg? [07:38] mvo: yes [07:40] mvo: description and commit summary should hopefully be understandable [07:40] ta [07:42] mvo: can you take another look at https://github.com/snapcore/snapd/pull/8415 ? [07:42] PR #8415: cmd/snap-recovery-chooser: tweaks [07:43] mborzecki: sure thing! [07:43] mvo: thanks! [07:48] hehe firefox chokes on the full diff in #8427 [07:48] PR #8427: store: start splitting store.go and store_test.go into subtopic files [07:51] brb [07:55] fwiw, I updated the branch protection rules for master so that all stable gh actions need to be successful [08:31] The GH Actions workflow should probably also be updated to run on pushes to master and release branches. At present, only the unit tests are being run for pushes to master [08:48] re [08:48] jamesh: we had something similar but I agree, we wanted not to test a push and a PR at the same time [08:48] jamesh: but I think we should do tests on PRs for both master and release branches [08:53] PR snapd#8431 opened: github: give jobs shorter names [09:04] PR snapd#8427 closed: store: start splitting store.go and store_test.go into subtopic files [09:08] mvo: mborzecki: thanks [09:14] PR snapd#8218 closed: [RFC] cmd/snapd, daemon/snapd: use multi call binary <⛔ Blocked> [09:16] Bug #1871058 opened: Tell how many bytes "install" will take [09:19] Bug #1871060 opened: Don't hide each step after it completes [09:28] PR snapd#8432 opened: travis.yml: disable unit tests on travis [09:32] #8411 needs 2nd reviews [09:32] PR #8411: boot: cleanup more things, simplify code [09:33] jamesh: https://github.com/snapcore/snapd/pull/8429#issuecomment-609683448 [09:33] PR #8429: github: port CLA check to Github Actions [09:34] mvo: https://github.com/snapcore/snapd/pull/8432#pullrequestreview-388082166 [09:34] PR #8432: travis.yml: disable unit tests on travis [09:36] hmm seems like ijohnson's standup notes were cut short or he didn't manage to publish the snapd branch of grub.cfg on friday [09:38] zyga: thanks, replied [09:39] mvo: I replied twice there, weird stuff [09:42] I think we are leaking session from tests/main/snap-session-agent-service-control [09:43] they break tests/main/snap-confine-undesired-mode-group [09:47] pedronis: mvo: looking into adding 'recover' and 'run' modes into system actions, and actually i'm wondering a bit about run, it doesn't make sense to have run for other system than current, does it? [09:49] mborzecki: no, it's only for current [09:50] we discussed that in Frankfurt [09:50] pedronis: ok, let me check the notes [09:50] mborzecki: +1 - nly relevent for current [10:05] mvo: pedronis: doesn't look like we put that down in the notes, nevertheless i take that recover applies to the current system only too [10:11] zyga: that seems like a good idea, especially since we can't use the job as a prereq for jobs that should run on pushes [10:11] +1 [10:12] zyga: the macos test should be fairly easy to move over to GH Actions too. At that point, the main thing the Travis job is doing is coverage [10:12] yeah [10:13] the test coverage provider we use has a dedicated action IIRC - we could adopt it [10:19] and codecov's first party action also downloads and executes a shell script... [10:19] of course it does :) [10:26] brb [10:55] re [11:17] zyga: hm that centos7 session-tool restore is still failing [11:17] and travis is green fwiw [11:18] mborzecki: same was as you showed me? [11:18] yeah [11:18] session that was running before the test closes during the test [11:18] ok, I'll switch gears to that in a moment [11:18] ta [11:18] tbh making gh actions spread jobs as require feels a bit premature [11:19] why? [11:19] note that travis doesn't run spread anymore [11:20] zyga: bc we cannot restart just one job [11:20] I see [11:20] I disagree but I wanted to understand [11:20] note that we can collapse all spread runs into one big run [11:20] and have that instea [11:20] *instead [11:23] zyga: that'd make it the same as it is now [11:24] we've traded ability to restart half the workers for being able to get tests started faster [11:24] (and having control over how many workers we can run) [11:25] mvo: hey! do you guys saw issues with a not existing /run/snapd-snap.socket on configure hook execution recently? Running quite frequently into https://paste.ubuntu.com/p/qQGSQjRjM7/ with snapd 2.44.1 [11:27] to be precise, snapd isn't restart and was active all time. [11:27] morphis: what is the status of the socket? [11:27] I suspect it's in CI but perhaps you know [11:28] I have a shell active so happy to debug [11:28] please start with systemctl status snapd.socket [11:28] zyga: socket is active and happy for >1h [11:28] morphis: both? [11:29] Listen: /run/snapd.socket (Stream) [11:29] /run/snapd-snap.socket (Stream) [11:29] ^ that's what I get on my box [11:29] https://paste.ubuntu.com/p/phZ4H7SGXb/ [11:29] jepp, same here [11:29] ok [11:29] and is /run/snapd-snap.socket really there on disk? [11:29] yes, touched an hour ago [11:29] ok [11:29] can you jump into the mount namespace of lxd [11:29] and then check if the socket is there [11:30] sudo nsenter -m/run/snapd/ns/lxd.mnt [11:30] this will give you a shell in the mount namespace of lxd [11:30] just check if the sockets are there as well [11:31] yes, exists there too [11:31] can you CURL to it? [11:31] or maybe instead [11:31] can you snapctl to it? [11:33] ah, I think I know what happens here [11:34] it's a race between `snap install lxd` and `snap set ..` [11:34] something we should prevent from the charm side [11:35] that matches with the dates the socket within the lxd ns is created and the snap got installed and the last time the error occurred in the juju output [11:37] morphis: cool [11:37] morphis: are you saying lxd doesn't really work after snap install lxd [11:37] in the sense that snap set is invoked too early? [11:37] PR snapd#8433 opened: overlord/devicestate: support for recover and run modes [11:38] it looks like we invoke `snap set` while performing `snap install` [11:38] as in during? [11:38] yeah that would not work [11:38] jepp [11:38] makes sense that it doesn't work [11:40] zyga: is there a simple `snap wait-until-install lxd` thing? [11:40] I don't think there is [11:41] ok [11:42] well, the configure hook runs after install [11:42] perhaps move your stuff over to it ? [11:42] morphis: hmm snap watch --last=install? could work [11:42] mborzecki: interesting [11:44] morphis: though it's still racy, since you'd expect the install to be started already to watch it [11:44] morphis: may I ask how do you run the two operations concurrenctly? [11:48] zyga: it's a problem in our charms I can fix on that level, was just looking for a hotfix [11:48] I see [11:48] ok [11:54] mborzecki: testing a fix for what you found now [11:54] zyga: cool thanks [11:55] cachio: hey [11:55] cachio: are you around? [11:55] zyga, yes [11:55] hi [11:55] hey [11:57] hey mborzecki sorry I didn't finish putting the snapd branch for grub.cfg things in the SU doc [11:57] I just put it in there if you wanted to take a look, it's quite simple [12:05] ijohnson: no biggie, i worked on adding the recovery/run modes to devicemgr [12:08] PR snapd#8431 closed: github: give jobs shorter names [12:10] * zyga breaks for a moment, [12:11] mborzecki: I've started reviewing 8433 [12:15] zyga: thanks [12:16] hi pedronis did you have a chance to look at how I organized things in #8424 at all ? [12:16] PR #8424: cmd/snap-bootstrap/initramfs-mounts: cross-check partitions when mounting [12:17] PR snapd#8434 opened: systemd: move the doc comments to the interface so their are visible [12:17] pstolowski_: ^ [12:20] ijohnson: I skimmed it, I didn't see anything unreasonable... but there's some comments about details from others [12:21] pedronis: ok, sure I will continue on with it [12:21] pedronis: I just didn't know if you would have opinions on having stuff in the boot package vs in the partition package of snap-bootstrap, etc. [12:21] pedronis: I assume you will have opinions on the names of things and that's fine to rename later :-) [12:28] * zyga breaks for lunch, see you soon [12:29] PR snapcraft#3013 closed: plugins: use v1 import path for all plugins [12:38] PR snapcraft#3014 opened: meta: migrate get_build_base to Snap [12:50] hey ogra, is your zoom-client snap supposed to work with nvidia drivers? I tried it with 440 drivers on Sunday and it crashed on me [12:51] PR snapd#8430 closed: packaging: use debian/not-installed to ignore snap-preseed [12:51] PR snapd#8432 closed: travis.yml: disable unit tests on travis [13:34] ijohnson, works gret with everything but nvidia-440 [13:34] *great even [13:35] ijohnson, https://github.com/ogra1/zoom-snap/issues/2 [13:36] is this an issue with the 440 drivers in general with snaps? [13:37] due to the awful nature of the zoom binary (forcefully setting LD_LIBRARY_PATH to ./zoom (and ./zoom only) on app startup) i actually need to LD_PRELOAD the drivers ... and -440 somehow refuses to load libOpenGL.so for no apparent reason [13:37] diddledan, there used to be one but zyga has fixed that ... i asked people to try snapd from edge but that didnt help [13:37] ogra: ack I can try with 390 or 435 then instead [13:37] -400 has a new directory structure and seemingly the linkage between the libs is differet [13:38] err [13:38] -440 indeed [13:39] aah, yeah, I see that message now on https://forum.snapcraft.io/t/nvidia-beta-drivers-completely-break-snaps/12392/15 - I must have overlooked it [13:39] right ... but that is supposed to be fixed [13:39] gotcha :-) [13:39] * diddledan goes snooping to see what the fix was [13:39] and according to the logs users actually have the proper content in /var/lib/snapd/lib/gl ... [13:40] so it is something with the LD_PRELOAD load order that i need to research to get t going (but for that i need a system that runs -440 ... my nvidia equipped desktop is still on 16.04 for which that driver doesnt exist) [13:41] I just took my nvidia card out to move it into my plex server [13:41] I now have reasonable transcoding on the fly for plex though, so WINNING [13:42] but you cant use zoom-client on it ! [13:42] :D [13:42] I had to install the deb variant of plex though. something in the snap config is unable to do hardware transcoding [13:43] mvo: so I guess this means you approve of my incoming valve index :D [13:43] and on the upside - everyone can then test on their nvidia GPUs [13:43] zyga: haha [13:43] man, we will catch all the bugs this way [13:43] the quality of nvidia support will skyrocket after this happens [13:43] snapd: add half life alyx support [13:43] snapd: fix level 3, achievement #42 [13:44] I vote for leaving a few gotchas to keep the users on their toes [13:44] snapd: fix 2nd playthrough ... [13:44] ;-) [13:44] PR #42: rework filelock to make it less surprising wrt flock(2) semantics [13:53] PR core20#35 opened: run-snapd-from-snap: log first snap watch better [13:55] Issue core20#36 opened: SSH login shell says the system was minimized and to run "unminimize", but should not [13:56] PR pc-amd64-gadget#38 closed: grub.cfg-recovery: fix typo, filter vars when loading them [14:00] jdstrand: where's the trello card for the appstream-metadata inteface fixes? I want to link it to mine card, but must be looking at the wrong board [14:24] Issue pc-amd64-gadget#41 opened: Hide the menu by default [14:30] mborzecki: https://github.com/snapcore/snapd/pull/8433#pullrequestreview-388194361 [14:30] PR #8433: overlord/devicestate: support for recover and run modes [14:39] mvo: I wonder if we should add some of my hosted workers to the pool [14:39] to spread the load [14:40] spread the spread ? [14:40] exactly [14:50] kenvandine: hey, I privmsg'd you [14:57] jdstrand: hello :) [14:57] jdstrand: thank you for the review from Friday [14:57] sure, I need to finish and followup [14:57] jdstrand: the tracking PR will be ready soon, there's a core20 issue still, where some packages related to dbus are missing, but it should be ready soon [14:57] and hi [14:57] ok [14:57] jdstrand: not end of the story but looking like something we could look at merging soon [14:58] jdstrand: I'll spend some more time on your feedback tomorrow, today I'm trying to wrap up that core20 problem first [14:59] great! :) [15:02] * cachio lunch [15:06] PR snapd#8435 opened: configcore,tests: fix setting watchdog options on UC18/20 [15:13] mvo: hey, so it sounds like you will roll another 2.44 point release for some policy updates? [15:14] jdstrand: if it's just safe policy updates I can do that [15:14] jdstrand: at this point I want to keep the risk minimal [15:14] mvo: is later today ok? [15:14] jdstrand: yeah [15:15] ok, I'll do a couple low risk PRs then [15:15] I'll target 2.44, 2.45 and master [15:15] jdstrand: or in my morning tomorrow [15:15] jdstrand: ok [15:15] jdstrand: 2.44 and master is fine, 2.45 is not branched yet [15:22] mvo: ah, ok, thanks [15:35] * zyga runs an errand in the office [15:36] running in place, zyga? 🤔 [15:36] roadmr: mainly cleaning and that one box that's open because a cable went loose and I need to solder the power switch back on [15:37] just rebooted.. getting: `cannot change profile for the next exec call: No such file or directory \n snap-update-ns failed with code 1: File exists` from running _any_ snaps [15:38] diddledan: I think we debugged that recently [15:38] diddledan: can you please collect your journal and pastebin it [15:38] diddledan: to fix your machine restart apparmor.service [15:38] diddledan: it's *not* fixed though [15:38] popey: ^ can you recall if you reported that on launchpad? [15:43] is this enough journal, or more needed? https://paste.ubuntu.com/p/RBc2r4xqJY/ [15:44] PR snapd#8436 opened: configcore,tests: use daemon-reexec to apply watchdog config [15:44] this is everything for this boot: https://paste.ubuntu.com/p/Jyx6gfFc3q/ [15:45] diddledan: looking [15:46] diddledan: when you said "just rebooted", roughly what time was there for you? [15:46] is that the start of your log? [15:46] I used journalctl --no-pager -b | pastebinit [15:46] ok, so 16:32~~ [15:47] yup [15:47] /quit/quit [15:47] so I see [15:47] Apr 06 16:32:58 defiant lxd.activate[3329]: cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_Iq23MU//dev: No such file or directory [15:47] 16:32:58 - we use snaps already [15:47] and then Apr 06 16:35:02 defiant audit[12031]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.daemon" pid=12031 comm="apparmor_parser" [15:48] at 16:35 we load the releated profile [15:48] FYI: stgraber ^ (lxd runs before apparmor profiles of snapd are loaded) [15:48] diddledan: please report a bug "services start before apparmor profiles are loaded" [15:48] we have a race [15:48] ok [15:48] include this journal if you don't mind, please highligh the two lines I mentioned above [15:48] popey: ^ if you reported a bug this this is a dupe, if not please track the one diddledan will open [15:48] mvo: ^ FYI [15:49] mvo: system startup races startup of services and apparmor.service loading profiles [15:49] mvo: observed for LXD here and for a lot of snaps (half of stuff IIRC) on popey's laptop [15:49] FYI: ^ jdstrand [15:49] though not a security issue, more a "it doesn't work" issue [15:51] diddledan: can you confirm that restarting apparmor.service fixed the issue? [15:53] yes, restarting apprmor fixes it - I've put that info in the bug: #1871148 [15:53] Bug #1871148: services start before apparmor profiles are loaded [15:53] thank you [16:04] #8390 and #8411 need 2nd reviews [16:05] PR #8390: state: add state.CopyState() helper [16:05] PR #8411: boot: cleanup more things, simplify code [16:10] zyga: oh, interesting, good that it's not something I have to fix for once ;) [16:11] stgraber: yeah, in case someone comes along and says this is broken on their fast server, it's on ous [16:11] *us [16:16] #blamepopey [16:18] * zyga hugs popey [16:18] sorry, been in meetings [16:19] * popey confirmificates the bug [16:20] * diddledan watching terminiser black inevitibility [16:26] PR core20#35 closed: run-snapd-from-snap: log first snap watch better [16:28] PR snapd#8437 opened: tests/session-tool: collect information about services on startup [16:29] PR snapd#8438 opened: tests/session-tool: stop anacron.service in prepare [16:31] zyga: uh, is this a new thing? [16:32] mvo: no [16:32] mvo, zyga: https://bugs.launchpad.net/snapd/+bug/1871148/comments/6 [16:32] Bug #1871148: services start before apparmor profiles are loaded [16:32] jdstrand: isn't that the message that is printed from /etc/apparmor.d [16:32] there's a separate one for /var/... [16:32] IIRC [16:32] * zyga checks [16:33] zyga: what are you talking about? [16:33] zyga: if it's not new I wonder why it's happening now? [16:33] zyga: 'Finished loading...'? [16:33] jdstrand: should we have an "After=apparmor" in all our snaps? [16:33] please forgive me for being unclear, let me check and show [16:33] mvo: please see my comment [16:34] mvo: apparmor was already finished. something else seems to be going on [16:34] oh, ok [16:35] mvo will there be a 2.44.3 ? [16:35] zyga: it can't be snapd rebuilding the profiles, right? we stop/disable services before we do that iirc [16:35] $CUSTOMER would like https://github.com/snapcore/snapd/pull/8426 asap, but not like super asap [16:35] PR #8426: interfaces/docker-support: add overlayfs file access [16:35] ijohnson: yes, jdstrand needs some interface updates [16:35] mvo: this is during system boot [16:35] mvo: ack cool this is just an interface update too [16:36] ijohnson: yeah [16:36] thanks mvo! [16:36] oh, that was in my list for today. thanks :) [16:37] ijohnson: note, for the other policy updates, mvo asked me to target 2.44 too. can you do the same? [16:37] jdstrand: mvo already milestonned 8426 to 2.44 [16:37] jdstrand: or did you mean to open a release/2.44 PR for 8426 ? [16:38] ijohnson: the latter [16:38] * ijohnson didn't open a 2.44 PR, but can if desired [16:38] ok, I will open a 2.44 PR then [16:38] ijohnson: if it's just a cherry pick I can do that, no need for a PR then [16:38] zyga: we have to restart all github actions jobs, right? we can't just restart one ? [16:38] correct [16:38] mvo: ok I think a cherry pick should be fine it's literally 1 line change [16:39] ok, thanks zyga [16:39] ijohnson: ta, just let me know when i landed and I cherry pick (or feel free to just cherry-pick yourself) [16:39] ijohnson: 8426? [16:39] ijohnson: I think that is green, I can merge now [16:39] mvo: oh [16:40] mvo: it wasn't green there was a spread failure for nfs-support, which is unrelated [16:40] ijohnson: yeah, I will merge anyway [16:40] mvo: I just restarted them, but I suppose you could use your magic powers to merge anyways [16:40] ijohnson: sorry, I meant "green" as in "green-enough" [16:40] :-D [16:40] diddledan: can you ls -ld /var/lib/snapd/apparmor/profiles/snap-update-ns.telegram-desktop [16:41] ijohnson: and merged and cherry-picked, thank you! [16:41] thanks mvo! [16:41] PR snapd#8426 closed: interfaces/docker-support: add overlayfs file access [16:41] zyga: -rw-r--r-- 1 root root 53984 Apr 5 19:53 /var/lib/snapd/apparmor/profiles/snap-update-ns.telegram-desktop [16:41] april 5 19:53 [16:41] thanks [16:42] diddledan: do you have /var/lib/snapd mounted on external disk? [16:42] or something of this kind [16:42] nopep [16:42] k [16:43] it's on a fairly normal setup except 20.04.. I do have root-on-zfs as installed by ubiquity's new feature [16:43] diddledan: what is the ls -l output for /var/lib/snapd/apparmor/profiles/snap.multipass.* [16:43] popey: do you have root on zfs? [16:43] diddledan: you you remember when you ran the command that restarted apparmor.service? [16:43] i do [16:43] https://paste.ubuntu.com/p/cJ8bxBjRjq/ [16:43] zyga, not precisely, no [16:44] * ijohnson remembers zyga asked us to setup tests to use zfs when 19.10 was released [16:44] indeed [16:44] popey: are you using zfs on your machine that exhibited this problem? [16:45] yes [16:45] (restarting apparmor.service) [16:45] aha [16:45] I feel a trend [16:45] rpool/ROOT/ubuntu_t60y5t/var/snap on /var/snap type zfs (rw,relatime,xattr,posixacl) [16:45] diddledan: can you provide your /etc/fstab [16:45] /var/snap is a separate mount from the zfs [16:45] https://paste.ubuntu.com/p/Z7TJpV5TmT/ [16:46] diddledan: /var/snap is a separate thing [16:46] diddledan: /var/lib/snapd is the thing [16:46] oops, sorry, misread [16:46] diddledan: can you attach to the bug: sudo systemd-analyze plot > ./1871148.svg [16:46] rpool/ROOT/ubuntu_t60y5t/var/lib on /var/lib type zfs (rw,relatime,xattr,posixacl) [16:46] jdstrand: great idea [16:46] hmm [16:47] * zyga also uses zfs on his 20.04 desktop but that machine sees very little use [16:49] diddledan: can you also attach: sudo systemd-analyze dot snap.multipass.multipassd.service > ./1871148-multipass.dot [16:50] both added [16:50] hmm [16:51] no apparmor.service at all?? [16:51] WAT? [16:51] zyga: which are you looking at? [16:51] jdstrand: the .svg [16:51] full system stuff [16:52] diddledan: can you journalctl -u apparmor.service [16:52] I wonder if it ran once or twice [16:52] (during this boot) [16:53] uh [16:53] maybe I should boot that desktop [16:53] that is real weird [16:53] jdstrand: if you don't have ZFS I can try that [16:53] looks like it only ran once until I restarted it manually: https://paste.ubuntu.com/p/W8bzwDpcR3/ [16:53] I do not [16:53] !!! [16:54] diddledan: can you reboot [16:54] diddledan: and see if it starts then [16:54] diddledan: if no, do not start it please [16:54] roger. gimme a sec [16:54] thank you [16:54] jdstrand: I guess it didn't run and why is the whole mystery [16:54] zyga: it did run [16:54] Apr 06 16:32:56 defiant systemd[1]: Finished Load AppArmor profiles. [16:55] jdstrand: yeah but it was invoked manually [16:55] or could have been [16:55] why is it not in the .svg file? [16:55] zyga: the restarting apparmor was the manual one [16:55] ok. back. snaps are broken :-) [16:56] zyga: the one I pointed at is from the paste in the bug that I mentioned [16:56] but why isn't it in the svg file, indeed [16:56] diddledan: ok [16:56] please don't restart apparmor.service [16:56] diddledan: systemctl status apparmor.service [16:56] diddledan: I bet there's a reason it was not started [16:56] and that's the big mystery [16:57] yeah [16:57] https://paste.ubuntu.com/p/h69jvBV6Kd/ [16:57] huh [16:57] jdstrand: is there some debug switch to get a list of things loaded? [16:58] diddledan: can you edit /lib/apparmor/apparmor.systemd and add set -x up top [16:58] and restart apparmor.service [16:58] that'll make everything start working, remember.. [16:59] yes but we will see that it works and the log of what happened [16:59] if that log is useful [16:59] then let's reboot again [16:59] and see the early -boot log [16:59] I still suspect the profiles weren't there at the time the service was start [16:59] wait!! [16:59] then we might know what to do [16:59] don't reboot yet [16:59] ok [16:59] diddledan: can you update /lib/apparmor/rc.apparmor.functions to comment out: [16:59] if [ "${QUIET:-no}" = yes ] || [ "${quiet:-n}" = y ]; then [16:59] PARSER_OPTS="$PARSER_OPTS --quiet" [16:59] fi [17:00] diddledan: then reboot [17:01] restarting apparmor with those changes BEFORE rebooting: https://paste.ubuntu.com/p/HHQsFPZpYP/ [17:01] diddledan: actually [17:01] * diddledan holds [17:01] diddledan: let's change that to be: [17:01] #if [ "${QUIET:-no}" = yes ] || [ "${quiet:-n}" = y ]; then [17:01] # PARSER_OPTS="$PARSER_OPTS --quiet" [17:01] #fi [17:01] PARSER_OPTS="$PARSER_OPTS -k" [17:02] that will show cache hits and misses [17:02] mmm [17:02] ok. made the change. going for reboob [17:03] jdstrand: like a good mystery novel [17:03] the murderer is about to be revealed [17:04] here we go: https://paste.ubuntu.com/p/dMMdZRZBbN/ [17:04] Apr 06 18:03:26 defiant apparmor.systemd[2717]: + [ -d /var/lib/snapd/apparmor/profiles ] [17:05] it didn't find snapd profiles [17:05] PROFILE_DIRS is only assigned to once [17:05] huh [17:05] wow [17:05] right/ [17:05] diddledan: can you pastebin /proc/self/mountinfo please [17:05] maybe there's some zfs magic we didn't anticipate [17:05] why isn't /var/lib/snapd/apparmor/profiles being looked at [17:05] jdstrand: ^ because it's not there at the time [17:05] https://paste.ubuntu.com/p/7783bjqXNF/ [17:06] line 13 of the paste [17:06] oh, yes [17:06] ok, that part of the mystery is solved [17:06] 143 31 0:53 / /var/lib rw,relatime shared:79 - zfs rpool/ROOT/ubuntu_t60y5t/var/lib rw,xattr,posixacl [17:06] it's a separate volume [17:06] we don't have requires mounts for [17:06] so ... bummer [17:06] we need to rev apparmor.service [17:06] to have that [17:06] that is probably not satisfied by local-fs [17:07] diddledan: can you please create [17:07] diddledan: /etc/systemd/system/apparmor.service.d [17:07] diddledan: there create a file fixes.conf [17:07] with contents: [17:07] [Unit] [17:07] RequiresMountsFor=/var/lib/snapd/apparmor/profiles [17:07] then reboot [17:07] that should be it [17:08] I wonder what local fs thing is [17:08] ok. rebooting [17:08] or what makes that zoo of pools [17:08] do we know someone in foundations that works on zfs? [17:09] I wonder why is everything in /var/* a separate pool [17:09] and how does zfs interact with local-fs.target [17:09] it feels like a bug that's deeper than RequiresMountsFor [17:10] dingdingding! we have a weiner! [17:10] xnox: (sorry for pinging you, I just recall you as a foundations member that's very well connected), do you know who works on zfs on ubuntu? [17:10] diddledan: cool! [17:10] zyga: we definitely need foundations to advise [17:10] jdstrand: agreed [17:10] mvo: ^ it's a bug introduced by zfs [17:10] popey: ^ a workaround for you is spelled out above [17:10] fyi, I summarized in https://bugs.launchpad.net/snapd/+bug/1871148/comments/10 [17:10] Bug #1871148: services start before apparmor profiles are loaded [17:10] super, thanks! [17:11] I did not put in the workaround [17:11] I read [17:11] ok, I need to take the dog out [17:11] it's been hours now [17:11] but we should definitely ask foundations to help [17:11] it's a RC blocker IMO [17:11] RequiresMountsFor=/var/lib/snapd/apparmor/profiles is probably fine [17:11] god knows what else is broken [17:11] yeah, I think we should have that [17:11] I can do a quick upload [17:12] jdstrand: can you upload new apparmor package to add that? [17:12] super ^ 2 [17:12] :D [17:12] zyga, jdstrand: didrocks / jibel are your zfs-on-root / zsys experts [17:12] zyga: we don't do zfs on ubuntu-core today, nor in cloud. the only place it's available is on ubuntu desktop => which is desktop team stuff, i.e. jibel and friends. Not foundations thing =) [17:12] jibel: ^ [17:12] thanks guys! [17:12] xnox: do you agree that RequiresMountsFor=/var/lib/snapd/apparmor/profiles is reasonable to upload? [17:12] jibel: we may have a problem with zfs-on-root and local-fs.target [17:13] let's ping jibel again so he doesn't get a heart attack at all :D [17:13] the details are in https://bugs.launchpad.net/snapd/+bug/1871148 [17:13] Bug #1871148: services start before apparmor profiles are loaded [17:13] jdstrand: complete context switch =) que? =) [17:13] should we ping jibel? [17:13] diddledan: I did :) ^ [17:13] :-p [17:13] super [17:13] I'll leave you guys to it [17:13] yeah I was trolly [17:13] my dog is looking at me with those puppy eyes [17:13] jdstrand: you probably want rbalint [17:14] what idiot gave puppies puppyeyes?! [17:14] xnox: so, zfs-on-root does different pools for /var such that local-fs.target is satisfied but /var/lib/snapd/apparmor/profiles does not exist at the time the service starts [17:14] diddledan: puppy conspiracy :) [17:14] jdstrand: I wonder if zfs mounts are done by systemd [17:14] it's the poopscoop industrial complex! [17:14] I guess it must be [17:14] because the requirement was enough to fix it [17:15] * zyga afk [17:15] o/ [17:15] xnox: ack [17:15] * jdstrand -> ubuntu-devel [17:15] jdstrand: and AppArmorProfile= key doesn't synthesize such a dep already? it should? [17:15] &> #ubuntu-devel [17:16] zyga: https://github.com/openzfs/zfs/tree/master/etc/systemd/system [17:16] PR snapd#8439 opened: [RFC] secboot: import secboot on ubuntu, provide dummy on !ubuntu [17:17] pedronis: I started looking into the tpm PR finally and the above is a first RFC to help moving it into master (cc cmatsuoka). hope this is vaguely the right direction, please let me know if you would like to see tags used differently or some totally different approach [17:17] https://github.com/openzfs/zfs/blob/5a42ef04fd390dc96fbbf31bc9f3d05695998211/etc/systemd/system/zfs-mount.service.in#L8 [17:19] cmatsuoka: feel free to pickup/improve 8439, I have dinner now [17:19] xnox: it might be able to synthesize it, but that is in the snapd units. by that time, the apparmor unit already ran and finished, ignoring the dir [17:20] xnox: but, we aren't using AppArmorProfile in the units anyway (snap-confine handles it) [17:25] mvo: ack [17:29] mvo: that looks painful once types enter the picture [17:31] but I'm probably missing something [17:34] jdstrand: hm, apparmor.service (or whatever it is) really should require mounts for dirs it uses, i.e /var/lib/foo/bar/baz/ which systemd will correctly resolve as to which ones are needed (i.e. /var, or /var/lib, or /varlib/foo, etc.) [17:35] jdstrand: and units that use ApparmorProfile should like synthese dep+after apparmor.service (whatever it is called) [17:35] cmatsuoka: I made some comments there [17:35] pedronis: thanks, will check asap [17:36] PR snapd#8415 closed: cmd/snap-recovery-chooser: tweaks [17:36] xnox: ok, so it sounds like you answered my question about RequiresMountsFor=/var/lib/snapd/apparmor/profiles [17:50] * zyga EODs and goes to set up some DNS at home === Trevinho_ is now known as Trevinho [18:11] jdstrand: so this needs both an apparmor package fix and a snapd snap services fix? [18:23] pedronis: no. snapd does not need a change. I am going to make a change to apparmor since it is the correct thing for it. it should address this issue. it sounds like jibel is going to perhaps adjust zsys in some manner [18:23] pedronis: re 8439> it should be enough to get the current tpm PR in progress working everywhere but yes, will not scale well. I guess it all depends on how much of the secboot code will be used outside of cmd/snap-bootstrap (which is skipped entirely on !ubuntu) [18:23] pedronis: anyway, totally happy about idea, just wanted to help unblock things [18:23] mvo: ok, working on that. I refactored it a little bit to have CheckEncryptionAvailability() in different versions [18:24] PR pc-amd64-gadget#40 closed: gadget.yaml: reduce data partition size to 1G [18:25] jdstrand: don't we need this: jdstrand: and units that use ApparmorProfile should like synthese dep+after apparmor.service (whatever it is called) [18:25] cmatsuoka: sounds great [18:25] cmatsuoka: fwiw, I'm not attached to this at all, just wanted to draft something to help us moving forward which can land in isolation to unblock the rest of the tpm work [18:26] mvo: shall I push my version to the same PR so we can discuss it better? [18:26] cmatsuoka: it looks also like my govendor changes are busted, sorry for that (https://travis-ci.org/github/snapcore/snapd/jobs/671744586?utm_medium=notification&utm_source=github_status) - I can look at it now unless you are already on this [18:26] cmatsuoka: yeah, go for it! [18:27] cmatsuoka: just push to the open PR or open a new one, like I said, no attachment to htis partical version :) [18:30] mvo: pushed it, we can always revert it if necessary [18:31] cmatsuoka: thanks! appreciated [18:32] oops, I didn't read the last comment from pedronis before pushing it [18:32] cmatsuoka: I ran "govendor remove +unused" just now [18:33] cmatsuoka: it seems like you adressed his comment, no? [18:33] I think it's in the same line [18:33] I renamed the function but ok, it's not important now [18:34] cmatsuoka: aha, I see [18:35] mvo I see that google:ubuntu-core-20-64:tests/core/writablepaths failed because /etc/machine-id was not writable, do we need to port that patch to release/2.44 ? [18:36] i.e. https://github.com/snapcore/snapd/pull/8422 might not have been applied to 2.44 iiuc [18:36] PR #8422: tests: skip "/etc/machine-id" in "writablepaths" test <⚠ Critical> [18:38] ijohnson: in 2.44? yeah, I think there is a cherry pick missing [18:38] mvo: yes [18:38] I don't see that commit on 2.44 [18:39] and the 2.44 travis run failed on that [18:39] ijohnson: just picked it [18:39] ijohnson: *hopefully* that's enough [18:39] cool :-) [18:39] ijohnson: thanks for noticing! [18:39] * ijohnson crosses fingers [18:39] PR snapd#8440 opened: github: move spread to self-hosted workers [20:11] ijohnson: did you work with the microk8s snap recently? [20:11] ijohnson: you may want to have a look at https://bugs.launchpad.net/snapd/+bug/1871189 [20:11] Bug #1871189: Snapd `cannot update snap namespace` when connecting / disconnecting interfaces [20:13] cmatsuoka: yes I will add a reproducer there today, it is what was reported in #snappy-internal the other day [20:13] ijohnson: ah ok, should I assign that to you then? [20:14] sure [20:14] thanks [20:38] zyga: have you seen trap invalid opcode error messages in xdg-open invoked from a snap? [20:56] PR snapd#8441 opened: interfaces: add case for rootWritableOverlay + NFS [21:03] PR snapd#8442 opened: tests/many: stop importing osutil from dirs, mock ProcSelfMountInfo more [21:06] PR snapcraft#3015 opened: [WIP] ros2 (colcon) extension preview [21:46] ijohnson: I marked #8442 blocked, *util packages should really depend only on stdlib or other util packages (there might be counterexamples but we should fix them), so importing dirs into osutil is not a solution as is to your conundrum [21:46] PR #8442: tests/many: stop importing osutil from dirs, mock ProcSelfMountInfo more <⛔ Blocked> [21:51] PR snapd#8434 closed: systemd: move the doc comments to the interface so they are visible [22:19] cmatsuoka: not that I recall [22:43] PR snapd#8443 opened: interfaces/many: miscellaneous policy updates xliv [22:48] PR snapd#8444 opened: interfaces/many: miscellaneous policy updates xliv - 2.44 [23:03] zyga: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1870861 shows some instances of that happening [23:03] Bug #1870861: chromium-browser is unable to open downloads