[04:20] Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly [04:29] Bug #1909412 opened: LazyUnmount in .mount units prevents root fs to umount cleanly [04:32] Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly [04:35] Bug #1909412 opened: LazyUnmount in .mount units prevents root fs to umount cleanly [04:38] Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly [06:09] morning [06:43] mvo good morning :) [06:43] good morning zyga ! did you had a nice easter weekend too? [06:43] mvo yes, very relaxing [06:44] the weather was not much fun so we stayed mostly indoors [06:44] spring gardening has started though [06:44] I've planted some seeds and I have first sprouts growing [06:44] how was yours? [06:45] mvo: zyga: hey [06:45] o/ [06:46] I need to tidy my desk, bbl [06:53] zyga: very nice, we had a bit of good weather so I spend time in the sun. but then the weather changed and it got very cold, we even have a bit of snow this morning :) [06:53] mborzecki: hey! how are you? [06:54] mvo: hey, still a bit sleepy 😉 time to get some coffee [06:54] mvo: i've pinged you in a couple of PRs that can land but are blocked on failing, though unrelated, spread tests [07:07] mborzecki: sure, I'm going over my mail right now but feel free to ping me out-of-band for force-merges [07:14] morning [07:26] pstolowski: hey [07:27] mborzecki: hey! [07:59] PR snapd#10109 closed: interfaces/builtin/gpio_test.go: actually test the generated gpio apparmor [08:03] mvo: I haven't had any luck getting enough green CI check marks to allow https://github.com/snapcore/snapd/pull/9963 to merge. I don't believe any of the problems are related to the branch contents, so could you hit the merge button? [08:03] PR #9963: wrappers: install D-Bus service activation files for snapd session tools on core [08:03] jamesh: sure [08:04] PR snapd#10104 closed: many: add x-gvfs-hide option to mount units [08:04] mborzecki: arch looks unhappy in https://github.com/snapcore/snapd/pull/9963/checks?check_run_id=2257669181 - is that a known issue? [08:04] PR #9963: wrappers: install D-Bus service activation files for snapd session tools on core [08:04] mborzecki: "curl: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by curl)" [08:05] mvo: pstolowski fixed that on friday i think [08:05] mvo: yes [08:05] cool, thanks pstolowski and mborzecki [08:05] pstolowski: and good morning :) [08:05] i'm going to push a trivial followup with --ignore [08:05] pstolowski: btw. cgroupv2 change was introduced by systemd 248 [08:05] hey :) [08:07] mvo: thanks [08:09] PR snapd#9963 closed: wrappers: install D-Bus service activation files for snapd session tools on core [08:15] mborzecki: https://github.com/snapcore/snapd/pull/10112 [08:15] PR #10112: spread: ignore linux kernel upgrade in early stages for arch preparation [08:19] PR snapd#10112 opened: spread: ignore linux kernel upgrade in early stages for arch preparation [08:39] PR # closed: snapd#10077, snapd#10079, snapd#10080, snapd#10095, snapd#10097 [08:44] PR snapd#8883 closed: packaging: stop snapd early on purge [08:44] PR snapd#10050 closed: tests: use snaps-state commands and remove them from the snaps helper [10:07] pedronis is off today, isn't he? [10:08] mborzecki: correct [10:08] that's ok, i'm wondering a bit about extending modeenv to track 'current' gadgets [10:09] otherwise it'd be a pain to find out the gadget snap ourselves when we mark the boot as successful [11:05] xnox: hi, maybe you would know if what i suggested in https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1922293 wrt waiting of zfs makes sense? [11:05] Bug #1922293: Snaps appear broken on 21.04 Beta with ZFS

[11:21] snap downloads seem particularly slow today [11:22] pstolowski: yeah, we can also have zfs-mount.service have Before=snapd.service if you want it that way? or is that of no help, cause we start individual snaps .mount units without snapd.service on subsequent boots? [11:23] xnox: i'm afraid we need it on each individual mount unit [11:23] does Before= even work if i uninstall snapd ? [11:23] pstolowski: i wonder if var.mount After=zfs-mount.service will suffice? [11:24] (which i suppose still many people do) [11:24] xnox: i guess the question is also if zfs-mount.service is the *right* one to wait for, i tried other zfs-related units (such as volumes-ready) but fwtw only this one seemed to work [11:24] pstolowski: mount units should allow ordering like that. as in before mount of /var/lib/any/parts/of/path are executed, the ordering of things of all the parent components should be good. [11:25] xnox: ah i didn't know we have var.mount, let me try that [11:25] ogra: before/after are pure ordering-only hints. which can be ignored to break deadlocks. and have no affect on adding/removing dependencies and don't care if things are present or not. [11:26] yep exactly [11:26] wants/wantedby/conflicts/requires => are actuall dependencies, which can be soft or hard. [11:26] ah, good then (i sismply didnt know, thanks for educating me 🙂 ) [11:26] i.e wants/wantedby are optional, whereas conflicts/requires will break things =) [11:27] good to know ! [11:28] pstolowski: there is zfs.target too.... and well zfs-mount tries to run as early as possible beofre local-fs.target.... [11:28] pstolowski: i wonder if snap*.mount units should have After=local-fs.target [11:29] such that all the things from /etc/fstab (and zfs things) are all mounted, then we go and try to mount snaps. [11:29] pstolowski: cause you do want /var & /home and so on to all be mounted before you attempt mounting snaps right? [11:29] xnox: this sounds reasonable too; nb i fail to see var.mount, where is it? [11:29] pstolowski: create it =) [11:29] ah ok :) [11:30] pstolowski: when ever there is What=/path/to/anything/blah.snap => it will try to ensure that [11:30] got it [11:30] / is satisfied, and /path, and /path/to, and /path/to/anything.... [11:30] hopefully =0 [11:30] or things are broken =) [11:34] xnox: var.mount with After=... broke it :/ [11:34] xnox: "!!! Failed to isolate default target" and stuck [11:34] pstolowski: yes [11:34] because there is /var in /etc/fstab [11:34] pstolowski: and it gets generated into .mount unit in /run/ [11:35] ah [11:35] pstolowski: you can do drop-in i.e. /etc/systemd/system/var.mount.d/foo.conf [Service]After= [11:35] pstolowski: or i think you can edit fstab too [11:36] pstolowski: in fstab, where you have /var specify after zfs-mount.service [11:36] sorry [11:36] x-systemd.after=zfs-mount.service i think [11:36] yeap, as an option. [11:36] as per $ man systemd.mount [11:37] pstolowski: sorry that i misled you [11:40] xnox: no worries, trying [11:54] xnox: so, /etc/systemd/system/var.mount.d/foo.conf didn't help; After=local-fs.target didn't work either; only After=zfs-mount.service works [11:55] xnox: something in a single central place would be ideal obviously if we can figure it out, rather than patching all snapd units [11:55] but maybe it's more subtle [11:55] pstolowski: after=local-fs.target on the snap units? and after=zfs-mount.service on the snap*.mount units? [11:56] also kind of odd that snap*.mounts units already don't have implicit after local-fs.target to be fair =/ [11:56] xnox: yes, both tried on snap.*mount units [11:59] xnox: mhm, it's shocking we got away with it for so many years ;) [12:02] I think we debugged this at some point and there should be implicit rules for after=local-fs.target already, I can try t ofind the PR were we looked into this [12:06] then maybe it's not enough for zfs [12:07] yeah, not against fixing it, just saying there might be more to it and it may well break more people [12:12] yep, absolutely [12:12] lunch, bbl [12:12] * pstolowski lunch [12:20] PR snapd#10112 closed: spread: ignore linux kernel upgrade in early stages for arch preparation [13:28] mvo: this could land https://github.com/snapcore/snapd/pull/10052 [13:28] PR #10052: features: add gate-auto-refresh-hook feature flag [13:29] pstolowski: sure [13:29] thanks [13:30] PR snapd#10052 closed: features: add gate-auto-refresh-hook feature flag [13:42] mvo: pstolowski: i would err on the side of zfs*.service units not doing something right. [13:42] rather than snapd*.mount units being incomplete. We must have a tonne of people who have /var on ext4 xfs etc. and it all works for them. [13:43] xnox: do you know who is maintaining this in ubuntu? [13:44] pstolowski: dpkg -S /lib/systemd/system/zfs-mount.service should give you the package and apt chagnelog the last uploader, what do you get there? [13:44] in any case i agree it should probably be fixed elsewhere (and not by snapd) [13:47] mvo: it's Colin Ian King [13:47] pstolowski: thanks, I have the changelog now too, let me poke a bit [14:00] mvo: sure, thanks [14:06] mvo: do you want me to force push the final solution I ended up at to #10107 ? I think adding that kind of code to the the debian maintainer scripts should probably be for another PR [14:06] Bug #10107: pppoeconf: new changes from Debian require merging [14:06] PR #10107: packaging: stop snapd.{socket,service} before purging data [14:25] pstolowski: do you have that system with zfs still around? do you have sometihng when you run "find /run/systemd -name "var*.mount" ? [14:32] mvo I have a system with zfs but not on rootfs [14:37] mvo: yes i've ,let me check [14:40] mvo: nothing [14:42] pstolowski: hm, and "mount" will show you that there is a mount for /var ? [14:42] pstolowski: there is a generator that should create a var.mount file it seems, no wonder if systemd is confused if non of those are generated :) [14:44] mvo: nope.. https://pastebin.ubuntu.com/p/QTfknGYY3r/ [14:45] mvo: (note snaps are only mounted because of my workaround) [14:47] pstolowski: can you pastebin "ls /run/systemd/generator" as well please [14:48] mvo: https://pastebin.ubuntu.com/p/tXxcyNhzz2/ [14:56] mvo maybe just look at all the mount units [14:56] at runtime [14:58] zyga, pstolowski good idea. what does "sudo systemctl status var-lib.mount [14:58] " print? [14:58] * zyga goes to boot up his zfs box [14:59] mvo: it's active, mounted [15:02] pstolowski: where does it come from? there should be a line that says from where it is [15:02] mvo: https://pastebin.ubuntu.com/p/pd8N9Qg7F8/ [15:05] pstolowski: that is interessting [15:09] * cachio lunch [15:14] mvo so perhaps I can explain [15:14] mvo zfs pool has some configured volumes [15:14] mvo they just get mounted by the kerenl [15:14] *kernel [15:15] mvo system noticed and creates the (virtual) mount unit [15:15] so they don't have to be spelled out in any place other than zfs pool configuration [15:15] mvo similarly to how you can mount a thing from command line, by hand [15:15] and immediately ask systemd about the .mount unit it corresponds to [15:17] zyga: right, I summarized this in https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1922293/comments/14 I hope, I wonder what the right fix is - probably a geneator? [15:17] Bug #1922293: Snaps appear broken on 21.04 Beta with ZFS

[15:19] mvo I think this is a good explanation [15:19] what I wonder if the systemd special automatic dependency is working or not [15:20] those described by systemd.mount(5) "AUTOMATIC DEPENDENCIES" [15:20] to quote: [15:20] • If a mount unit is beneath another mount unit in the file system [15:20] hierarchy, both a requirement dependency and an ordering dependency [15:20] between both units are created automatically. [15:28] zyga: yeah, was wondering the same if this is a systemd bug maybe [15:29] mvo I doubt that but let's see [15:33] mvo: thanks for this! [16:05] pstolowski: no worries, thank *you* for all the hard work finding the details [16:26] mvo: i saw the comment from didrocks; opened https://github.com/snapcore/snapd/pull/10113 [16:26] PR #10113: systemd: wait for zfs mounts [16:28] eod [16:30] PR snapd#10113 opened: systemd: wait for zfs mounts [16:31] mvo uh, changing all the mount units will be painful [16:32] can a generator drop an override to all the snapd mount units that injects the new dependency? [16:32] I think generators can do that [16:32] I think we do that for lxd containers to fix some bug recently [16:33] maybe that was slightly different though [16:33] but also we will need to rewrite service units *soon* anyways for other features so probably not a big deal if we need to implement that feature too [16:34] I think so as well, but I don't recall details [16:47] zyga, ijohnson yeah, I think it's okay if this is not perfect, this is new installs anyway [16:47] that's fair === ijohnson is now known as ijohnson|lunch [17:35] PR snapd#10111 closed: tests: update "spread: disable Go modules support in environment" [19:07] * cachio afk === ijohnson|lunch is now known as ijohnson [19:56] PR snapd#10114 opened: tests/lib/prepare-restore.sh: clean out snapd changes and snaps before purging [22:26] PR snapd#10115 opened: tests: force a proxy to use cache action