[04:20] <mup> Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly <Snappy:Expired> <https://launchpad.net/bugs/1909412>
[04:29] <mup> Bug #1909412 opened: LazyUnmount in .mount units prevents root fs to umount cleanly <Snappy:Expired> <https://launchpad.net/bugs/1909412>
[04:32] <mup> Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly <Snappy:Expired> <https://launchpad.net/bugs/1909412>
[04:35] <mup> Bug #1909412 opened: LazyUnmount in .mount units prevents root fs to umount cleanly <Snappy:Expired> <https://launchpad.net/bugs/1909412>
[04:38] <mup> Bug #1909412 changed: LazyUnmount in .mount units prevents root fs to umount cleanly <Snappy:Expired> <https://launchpad.net/bugs/1909412>
[06:09] <mborzecki> morning
[06:43] <zyga> mvo good morning :)
[06:43] <mvo> good morning zyga ! did you had a nice easter weekend too?
[06:43] <zyga> mvo yes, very relaxing
[06:44] <zyga> the weather was not much fun so we stayed mostly indoors
[06:44] <zyga> spring gardening has started though
[06:44] <zyga> I've planted some seeds and I have first sprouts growing
[06:44] <zyga> how was yours?
[06:45] <mborzecki> mvo: zyga: hey
[06:45] <zyga> o/
[06:46] <zyga> I need to tidy my desk, bbl
[06:53] <mvo> zyga: very nice, we had a bit of good weather so I spend time in the sun. but then the weather changed and it got very cold, we even have a bit of snow this morning :)
[06:53] <mvo> mborzecki: hey! how are you?
[06:54] <mborzecki> mvo: hey, still a bit sleepy 😉 time to get some coffee
[06:54] <mborzecki> mvo: i've pinged you in a couple of PRs that can land but are blocked on failing, though unrelated, spread tests
[07:07] <mvo> mborzecki: sure, I'm going over my mail right now but feel free to ping me out-of-band for force-merges
[07:14] <pstolowski> morning
[07:26] <mborzecki> pstolowski: hey
[07:27] <pstolowski> mborzecki: hey!
[07:59] <mup> PR snapd#10109 closed: interfaces/builtin/gpio_test.go: actually test the generated gpio apparmor <Simple 😃> <Skip spread> <Test Robustness> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10109>
[08:03] <jamesh> mvo: I haven't had any luck getting enough green CI check marks to allow https://github.com/snapcore/snapd/pull/9963 to merge.  I don't believe any of the problems are related to the branch contents, so could you hit the merge button?
[08:03] <mup> PR #9963: wrappers: install D-Bus service activation files for snapd session tools on core <Needs Samuele review> <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9963>
[08:03] <mvo> jamesh: sure
[08:04] <mup> PR snapd#10104 closed: many: add x-gvfs-hide option to mount units <Created by lhotari> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10104>
[08:04] <mvo> mborzecki: arch looks unhappy in https://github.com/snapcore/snapd/pull/9963/checks?check_run_id=2257669181 - is that a known issue?
[08:04] <mup> PR #9963: wrappers: install D-Bus service activation files for snapd session tools on core <Needs Samuele review> <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/9963>
[08:04] <mvo> mborzecki: "curl: /usr/lib/libc.so.6: version `GLIBC_2.33' not found (required by curl)"
[08:05] <mborzecki> mvo: pstolowski fixed that on friday i think
[08:05] <pstolowski> mvo: yes
[08:05] <mvo> cool, thanks pstolowski and mborzecki
[08:05] <mvo> pstolowski: and good morning :)
[08:05] <pstolowski> i'm going to push a trivial followup with --ignore
[08:05] <mborzecki> pstolowski: btw. cgroupv2 change was introduced by systemd 248
[08:05] <pstolowski> hey :)
[08:07] <jamesh> mvo: thanks
[08:09] <mup> PR snapd#9963 closed: wrappers: install D-Bus service activation files for snapd session tools on core <Needs Samuele review> <Created by jhenstridge> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9963>
[08:15] <pstolowski> mborzecki: https://github.com/snapcore/snapd/pull/10112
[08:15] <mup> PR #10112: spread: ignore linux kernel upgrade in early stages for arch preparation <Simple 😃> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10112>
[08:19] <mup> PR snapd#10112 opened: spread: ignore linux kernel upgrade in early stages for arch preparation <Simple 😃> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10112>
[08:39] <mup> PR # closed: snapd#10077, snapd#10079, snapd#10080, snapd#10095, snapd#10097
[08:44] <mup> PR snapd#8883 closed: packaging: stop snapd early on purge <Test Robustness> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8883>
[08:44] <mup> PR snapd#10050 closed: tests: use snaps-state commands and remove them from the snaps helper <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10050>
[10:07] <mborzecki> pedronis is off today, isn't he?
[10:08] <mvo> mborzecki: correct
[10:08] <mborzecki> that's ok, i'm wondering a bit about extending modeenv to track 'current' gadgets
[10:09] <mborzecki> otherwise it'd be a pain to find out the gadget snap ourselves when we mark the boot as successful
[11:05] <pstolowski> xnox: hi, maybe you would know if what i suggested in https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1922293 wrt waiting of zfs makes sense?
[11:05] <mup> Bug #1922293: Snaps appear broken on 21.04 Beta with ZFS <hirsute> <iso-testing> <snapd (Ubuntu):Confirmed> <https://launchpad.net/bugs/1922293>
[11:21] <jamesh> snap downloads seem particularly slow today
[11:22] <xnox> pstolowski:  yeah, we can also have zfs-mount.service have Before=snapd.service if you want it that way? or is that of no help, cause we start individual snaps .mount units without snapd.service on subsequent boots?
[11:23] <pstolowski> xnox: i'm afraid we need it on each individual mount unit
[11:23] <ogra> does Before= even work if i uninstall snapd ?
[11:23] <xnox> pstolowski:  i wonder if var.mount After=zfs-mount.service will suffice?
[11:24] <ogra> (which i suppose still many people do)
[11:24] <pstolowski> xnox: i guess the question is also if zfs-mount.service is the *right* one to wait for, i tried other zfs-related units (such as volumes-ready) but fwtw only this one seemed to work
[11:24] <xnox> pstolowski:  mount units should allow ordering like that. as in before mount of /var/lib/any/parts/of/path are executed, the ordering of things of all the parent components should be good.
[11:25] <pstolowski> xnox: ah i didn't know we have var.mount, let me try that
[11:25] <xnox> ogra:  before/after are pure ordering-only hints. which can be ignored to break deadlocks. and have no affect on adding/removing dependencies and don't care if things are present or not.
[11:26] <pstolowski> yep exactly
[11:26] <xnox> wants/wantedby/conflicts/requires => are actuall dependencies, which can be soft or hard.
[11:26] <ogra> ah, good then (i sismply didnt know, thanks for educating me 🙂 )
[11:26] <xnox> i.e wants/wantedby are optional, whereas conflicts/requires will break things =)
[11:27] <ogra> good to know !
[11:28] <xnox> pstolowski:  there is zfs.target too.... and well zfs-mount tries to run as early as possible beofre local-fs.target....
[11:28] <xnox> pstolowski:  i wonder if snap*.mount units should have After=local-fs.target
[11:29] <xnox> such that all the things from /etc/fstab (and zfs things) are all mounted, then we go and try to mount snaps.
[11:29] <xnox> pstolowski:  cause you do want /var & /home and so on to all be mounted before you attempt mounting snaps right?
[11:29] <pstolowski> xnox: this sounds reasonable too; nb i fail to see var.mount, where is it?
[11:29] <xnox> pstolowski:  create it =)
[11:29] <pstolowski> ah ok :)
[11:30] <xnox> pstolowski: when ever there is What=/path/to/anything/blah.snap => it will try to ensure that
[11:30] <pstolowski> got it
[11:30] <xnox> / is satisfied, and /path, and /path/to, and /path/to/anything....
[11:30] <xnox> hopefully =0
[11:30] <xnox> or things are broken =)
[11:34] <pstolowski> xnox: var.mount with After=... broke it :/
[11:34] <pstolowski> xnox: "!!! Failed to isolate default target" and stuck
[11:34] <xnox> pstolowski:  yes
[11:34] <xnox> because there is /var in /etc/fstab
[11:34] <xnox> pstolowski:  and it gets generated into .mount unit in /run/
[11:35] <pstolowski> ah
[11:35] <xnox> pstolowski:  you can do drop-in i.e. /etc/systemd/system/var.mount.d/foo.conf [Service]After=
[11:35] <xnox> pstolowski:  or i think you can edit fstab too
[11:36] <xnox> pstolowski:  in fstab, where you have /var specify after zfs-mount.service
[11:36] <xnox> sorry
[11:36] <xnox> x-systemd.after=zfs-mount.service i think
[11:36] <xnox> yeap, as an option.
[11:36] <xnox> as per $ man systemd.mount
[11:37] <xnox> pstolowski:  sorry that i misled you
[11:40] <pstolowski> xnox: no worries, trying
[11:54] <pstolowski> xnox: so, /etc/systemd/system/var.mount.d/foo.conf didn't help; After=local-fs.target didn't work either; only After=zfs-mount.service works
[11:55] <pstolowski> xnox: something in a single central place would be ideal obviously if we can figure it out, rather than patching all snapd units
[11:55] <pstolowski> but maybe it's more subtle
[11:55] <xnox> pstolowski:  after=local-fs.target on the snap units? and after=zfs-mount.service on the snap*.mount units?
[11:56] <xnox> also kind of odd that snap*.mounts units already don't have implicit after local-fs.target to be fair =/
[11:56] <pstolowski> xnox: yes, both tried on snap.*mount units
[11:59] <pstolowski> xnox: mhm, it's shocking we got away with it for so many years ;)
[12:02] <mvo> I think we debugged this at some point and there should be implicit rules for after=local-fs.target already, I can try t ofind the PR were we looked into this
[12:06] <pstolowski> then maybe it's not enough for zfs
[12:07] <mvo> yeah, not against fixing it, just saying there might be more to it and it may well break more people
[12:12] <pstolowski> yep, absolutely
[12:12] <pstolowski> lunch, bbl
[12:12]  * pstolowski lunch
[12:20] <mup> PR snapd#10112 closed: spread: ignore linux kernel upgrade in early stages for arch preparation <Simple 😃> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10112>
[13:28] <pstolowski> mvo: this could land https://github.com/snapcore/snapd/pull/10052
[13:28] <mup> PR #10052: features: add gate-auto-refresh-hook feature flag <Refresh control> <Simple 😃> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10052>
[13:29] <mvo> pstolowski: sure
[13:29] <pstolowski> thanks
[13:30] <mup> PR snapd#10052 closed: features: add gate-auto-refresh-hook feature flag <Refresh control> <Simple 😃> <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10052>
[13:42] <xnox> mvo:  pstolowski: i would err on the side of zfs*.service units not doing something right.
[13:42] <xnox> rather than snapd*.mount units being incomplete. We must have a tonne of people who have /var on ext4 xfs etc. and it all works for them.
[13:43] <pstolowski> xnox: do you know who is maintaining this in ubuntu?
[13:44] <mvo> pstolowski: dpkg -S /lib/systemd/system/zfs-mount.service should give you the package and apt chagnelog <pkgname> the last uploader, what do you get there?
[13:44] <pstolowski> in any case i agree it should probably be fixed elsewhere (and not by snapd)
[13:47] <pstolowski> mvo: it's Colin Ian King
[13:47] <mvo> pstolowski: thanks, I have the changelog now too, let me poke a bit
[14:00] <pstolowski> mvo: sure, thanks
[14:06] <ijohnson> mvo: do you want me to force push the final solution I ended up at to #10107 ? I think adding that kind of code to the the debian maintainer scripts should probably be for another PR
[14:06] <mup> Bug #10107: pppoeconf: new changes from Debian require merging <pppoeconf (Ubuntu):Fix Released by doko> <https://launchpad.net/bugs/10107>
[14:06] <mup> PR #10107: packaging: stop snapd.{socket,service} before purging data <Precious Logs> <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/10107>
[14:25] <mvo> pstolowski: do you have that system with zfs still around? do you have sometihng when you run "find /run/systemd -name "var*.mount" ?
[14:32] <zyga> mvo I have a system with zfs but not on rootfs
[14:37] <pstolowski> mvo: yes i've ,let me check
[14:40] <pstolowski> mvo: nothing
[14:42] <mvo> pstolowski: hm, and "mount" will show you that there is a mount for /var ?
[14:42] <mvo> pstolowski: there is a generator that should create a var.mount file it seems, no wonder if systemd is confused if non of those are generated :)
[14:44] <pstolowski> mvo: nope.. https://pastebin.ubuntu.com/p/QTfknGYY3r/
[14:45] <pstolowski> mvo: (note snaps are only mounted because of my workaround)
[14:47] <mvo> pstolowski: can you pastebin "ls /run/systemd/generator" as well please
[14:48] <pstolowski> mvo: https://pastebin.ubuntu.com/p/tXxcyNhzz2/
[14:56] <zyga> mvo maybe just look at all the mount units
[14:56] <zyga> at runtime
[14:58] <mvo> zyga, pstolowski good idea. what does "sudo systemctl status var-lib.mount
[14:58] <mvo> " print?
[14:58]  * zyga goes to boot up his zfs box
[14:59] <pstolowski> mvo: it's active, mounted
[15:02] <mvo> pstolowski: where does it come from? there should be a line that says from where it is
[15:02] <pstolowski> mvo: https://pastebin.ubuntu.com/p/pd8N9Qg7F8/
[15:05] <mvo> pstolowski: that is interessting
[15:09]  * cachio lunch
[15:14] <zyga> mvo so perhaps I can explain
[15:14] <zyga> mvo zfs pool has some configured volumes
[15:14] <zyga> mvo they just get mounted by the kerenl
[15:14] <zyga> *kernel
[15:15] <zyga> mvo system noticed and creates the (virtual) mount unit
[15:15] <zyga> so they don't have to be spelled out in any place other than zfs pool configuration
[15:15] <zyga> mvo similarly to how you can mount a thing from command line, by hand
[15:15] <zyga> and immediately ask systemd about the .mount unit it corresponds to
[15:17] <mvo> zyga: right, I summarized this in https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1922293/comments/14 I hope, I wonder what the right fix is - probably a geneator?
[15:17] <mup> Bug #1922293: Snaps appear broken on 21.04 Beta with ZFS <hirsute> <iso-testing> <snapd (Ubuntu):Confirmed> <zfs-linux (Ubuntu):Confirmed> <https://launchpad.net/bugs/1922293>
[15:19] <zyga> mvo I think this is a good explanation
[15:19] <zyga> what I wonder if the systemd special automatic dependency is working or not
[15:20] <zyga> those described by systemd.mount(5) "AUTOMATIC DEPENDENCIES"
[15:20] <zyga> to quote:
[15:20] <zyga>        •   If a mount unit is beneath another mount unit in the file system
[15:20] <zyga>            hierarchy, both a requirement dependency and an ordering dependency
[15:20] <zyga>            between both units are created automatically.
[15:28] <mvo> zyga: yeah, was wondering the same if this is a systemd bug maybe
[15:29] <zyga> mvo I doubt that but let's see
[15:33] <pstolowski> mvo: thanks for this!
[16:05] <mvo> pstolowski: no worries, thank *you* for all the hard work finding the details
[16:26] <pstolowski> mvo: i saw the comment from didrocks; opened https://github.com/snapcore/snapd/pull/10113
[16:26] <mup> PR #10113: systemd: wait for zfs mounts <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10113>
[16:28] <pstolowski> eod
[16:30] <mup> PR snapd#10113 opened: systemd: wait for zfs mounts <Bug> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10113>
[16:31] <zyga> mvo uh, changing all the mount units will be painful
[16:32] <zyga> can a generator drop an override to all the snapd mount units that injects the new dependency?
[16:32] <ijohnson> I think generators can do that
[16:32] <ijohnson> I think we do that for lxd containers to fix some bug recently
[16:33] <ijohnson> maybe that was slightly different though
[16:33] <ijohnson> but also we will need to rewrite service units *soon* anyways for other features so probably not a big deal if we need to implement that feature too
[16:34] <zyga> I think so as well, but I don't recall details
[16:47] <mvo> zyga, ijohnson yeah, I think it's okay if this is not perfect, this is new installs anyway
[16:47] <ijohnson> that's fair
[17:35] <mup> PR snapd#10111 closed: tests: update "spread: disable Go modules support in environment" <Squash-merge> <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10111>
[19:07]  * cachio afk
[19:56] <mup> PR snapd#10114 opened: tests/lib/prepare-restore.sh: clean out snapd changes and snaps before purging <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/10114>
[22:26] <mup> PR snapd#10115 opened: tests: force a proxy to use cache action <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/10115>