[06:39] morning [07:51] mvo: morning [08:08] morning [08:28] pstolowski: pedronis: morning guys [08:35] good morning pstolowski and mborzecki [08:42] o/ [08:55] pedronis: i'm looking at #9889 and at the handling of successful boot assets in snapd, for command lines we already check that mode is "run" before picking the one that we successfuly booted with, but we don't have a similar check in the boot assets code, but probably there should be one? [08:55] PR #9889: cmd/snap-bootstrap/initramfs-mounts: write realistic modeenv for recover+install [08:56] mborzecki: yes, that sounds like a bug? [08:58] mborzecki: it's probably hidden by the fact that ensureBootOk itself does nothing if mode != run [08:58] ? [08:58] pedronis: yeah, could be, so many layers [08:59] yup, it's fine then [09:03] mborzecki: fine in which sense? we should be consistent in boot I suppose [09:05] pedronis: i mean, it's a bug, but not the omg-everything-is-broken one type afaict [09:06] i'll prep a PR, should be an easy fix [09:06] ok, yes [09:28] PR snapd#9891 opened: boot: do not observe successful boot assets if not in run mode [09:30] pedronis: ^^ [10:45] mborzecki: I re-reviewed #9867, I have a question about the managers_test tests, a bit unsure what they test [10:45] PR #9867: overlord/devicestate: task for updating boot configs, spread test [10:45] pedronis: thanks, let me take a look [11:29] PR snapd#9892 opened: asserts: introduce AtSequence

[11:59] PR snapd#9868 closed: tests: fix umount for snapd snap on fsck-on-boot test [11:59] PR snapd#9890 closed: misc: little tweaks [11:59] PR snapd#9893 opened: store: support validation sets with fetch-assertions action

[12:06] pedronis: thanks, let me take a look [12:06] duh, wrong window ;) [12:07] pedronis: anyways, good point with that test, the scenario did not account for restart after link-snap, so the test was finishing too early [12:16] pedronis: and pushed now [12:36] pedronis: i updated remote validation set PR and proposd 2 new PRs [12:37] pstolowski: thx, I saw that, going over some older things in my queue atm [12:40] sure [12:40] * pstolowski lunch [13:18] pstolowski: I commented on #9892 [13:18] PR #9892: asserts: introduce AtSequence

[13:49] pedronis: on #9889, what do you think about adding the current kernel snap to CurrentKernels in the modeenv for recover/install mode? I originally had it there but took it out when you mentioned the other day we want a more minimal modeenv so I just included the bare minimum necessary to get snap-repair working [13:49] PR #9889: cmd/snap-bootstrap/initramfs-mounts: write realistic modeenv for recover+install [13:50] ijohnson: sorry, was in a meeting [13:51] no worries [13:51] we can talk in SU too [14:25] re [14:26] mvo I've sent a review for https://github.com/snapcore/snapd/pull/9772#pullrequestreview-582385202 [14:26] PR #9772: desktop/notification: test against a real session bus and notification server implementation [14:27] zyga: \o/ [14:27] I'm trying to catch up with my github notifications [14:27] I should be somewhat better over time [14:27] please @me explicitly [14:27] that helps [14:29] zyga: it's okay, no worries, thanks again for caring so much [14:31] I'm sorry for not being on IRC [14:31] i finished a small arc that was started on the H corpo laptop [14:31] and now I'm back to my regular gear [15:00] PR snapd#9891 closed: boot: do not observe successful boot assets if not in run mode [15:05] PR snapd#9888 closed: data/env/snapd: use quoting in case PATH contains spaces [15:10] PR snapd#9894 opened: snap/info.go: add doc-comment for SortServices [15:20] mvo: https://github.com/zyga/zmk/releases/tag/v0.5 :-) === King_InuYasha is now known as Conan_Kudo === Conan_Kudo is now known as King_InuYasha === King_InuYasha is now known as Conan_Kudo === Conan_Kudo is now known as King_InuYasha [16:01] xnox, to get modules included in the Pi initrd for UC20, whom would i have to nag, is that foundations or kernel team ? [16:02] * ogra would really like to see the screen working during boot and that needs some vc4 framebuffer modules [16:12] xnox, this looks a little overzealous https://github.com/snapcore/core-initrd/commit/3e9bf1fce8f0aed86f473d5bf0428acd543c63e4 (removing hid_generic and usbhid means no kbd input, so no way to reach the recovery chooser etc) [16:14] ogra: UC20 pi kernel initrd comes without modules. Ask kernel team to make those modules built-into the kernel config. [16:15] PR snapd#9132 closed: o/hookstate/ctlcmd: add optional --pid and --apparmor-label arguments to "snapctl is-connected" [16:15] ogra: there are issues around loading modules, without matching firmware. [16:15] xnox, that wont work [16:15] there are bits that *need* to be modular [16:15] ogra: do vc4 & hid require firmware too? [16:16] nope [16:16] ogra: what do you mean that "wont work"? built-in modules do work. or are you implying you have conflicting ones? [16:17] ogra: it would be nice, if you attached /proc/modules with teh screen/keyboard that you want to use from booted pi image => and then ask for them to be avaiable. then kerenl/initrd will work out how to provide them. [16:17] xnox, no, but there are cases where not having some modules load on boot is desired and where you dont wont certain modules loaded at all ... [16:18] if you hard-build-in the wprld thats not a solution [16:18] *world [16:18] ogra: please open a bug report, against linux-raspi ubuntu package, with attached /proc/modules of things you want to be available at initrd time (and not just boot time), and we will look into it. [16:19] i don't do bug reports over irc, busy fixing .2 release. [16:19] xnox, yeah, indeed, my prob is that nobody seems to feel responsible ... kernel team points to foundationy and your answer was "just build everythng in, ask kernel" [16:20] i didnt plan to make a bug report over IRC but to find who the hell is responsible for which part of the system now [16:20] i'll file proper LP bugs once i know what to file them against [16:21] but having two teams just bounce me back and forth between them, telling me the other is in charge is not helpful at all [16:23] ogra: please open bug report against pad.lv/u/linux-raspi => that's where we grack bugs about pi-kernel snap [16:24] xnox, thanks [17:15] PR snapd#9895 opened: gadget/many: rm, delay sector size + structure size checks to runtime === ijohnson is now known as ijohnson|lunch [18:00] PR snapd#9896 opened: osutil: skip TestReadBuildGo inside sbuild === ijohnson|lunch is now known as ijohnson [19:22] hey ogra [19:22] ogra any plan for ubuntu microcore for pico pi :D [19:23] I'm joking, wondering when will my pico order arrive [19:43] hey King_InuYasha! [19:43] thanks for taking the review for zmk [19:44] let me know if I should update the package to 0.5-1 before the initial review === mpontillo_ is now known as mpontillo [20:37] https://github.com/coreos/go-systemd/issues/331 might be interesting to someone in here who understands the low level details of it / AppArmor better than I do O:) [20:38] (I've tracked down a minor regression in my testing of Docker 20.10.3 to that issue, which was caused by https://github.com/coreos/go-systemd/commit/728309f70581336d9cf61afb335a815e2a5db1bf) [20:42] hey tianon let me take a look [20:42] <3 [20:42] I mean, your suggestion of getting log-observe auto-connected on docker would "fix" it, but it still feels odd [20:43] and means any other snap using that Go package to detect whether journald is supported for writing will be broken [20:43] hmm [20:43] (as noted by the reporter of that issue) [20:43] even though snaps do not require log-observe to write to journald [20:44] well I will point out that log-observe is just an -observe interface, so we don't consider it privileged in the way that we consider process-control or even docker-support privileged [20:44] but this change means this library thinks journald is disabled, and the implementation details mean the program has to be restarted after connecting log-observe for it to even pick up the change [20:44] so most program authors that come to the forum and ask to have log-observe auto-connected will almost always get it [20:44] but I see your point about it being a confusing thing [20:45] yeah, I mean it makes sense for programs that want to write to journald to also possibly want to read from it, but in many cases I'd imagine it's entirely write-only (since the logs written there are typically for users, not the program itself to read back) [20:45] journald as a database is cute :) [20:45] tianon: the other option of course is that we could just allow this specific write to journald's unix socket in the default template, not sure about that though I need to dig a bit more [20:46] yeah [20:46] I figured it's probably relevant to more than just Docker, and more relevant as time goes on and more things update to the newer go-systemd package :) [20:47] I wonder if AppArmor would have a way to allow the bind, but then disallow reads from it without log-observe? honestly out of my depth here so I'm reaching :) [20:47] (because it seems the bind being denied is what's causing the failure) [20:48] actually I'm looking at the log-observe and I don't see where we allow bind [20:48] oh, maybe I should test whether that actually does even fix this [20:48] (I just assumed, which you know, makes me bad /o\) [20:49] tianon: in terms of writing to files at least, apparmor's write permission implies read afair, so there isn't a way to allow only "writes" and not "reads" [20:49] but maybe jdstrand remembers if he's still around :-) [20:51] tianon: ah-ha actually k8s already pulled in the new go-systemd and ran into the reported problem and on top of that there is a bug with how `unix (bind) type=dgram addr=non` works, so onyly `unix (bind) type=dgram` works [20:51] ouch, but also :D [20:52] so we have this interesting variant of the kubernetes-support interface which _just_ allows auto-binding unix dgram sockets [20:53] https://github.com/snapcore/snapd/blob/master/interfaces/builtin/kubernetes_support.go#L214-L225 [20:53] ah but that was also eventually fixed in upstream apparmor [20:54] https://bugs.launchpad.net/apparmor/+bug/1867216 [20:54] Bug #1867216: unix syntax does not easily accommodate unix autobind sockets [20:55] tianon: so instead of using log-observe you could have the docker snap do the confusing thing and instead specify something like: [20:55] ```yaml [20:55] plugs: [20:55] autobind-unix: [20:55] interface: kubernetes-support [20:55] flavor: autobind-unix [20:55] apps: [20:55] dockerd: [20:55] command: ... [20:55] plugs: [20:55] - autobind-unix [20:55] ... [20:55] ``` [20:55] if that makes sense [21:04] I mean, it seems kind of strange for Docker to use the kubernetes-support interface to work around what's arguably a bug in the combination of the default profile + go-systemd changes O:) [21:04] and even that we'd need auto-connected, right? [21:05] so that would let us use the journald log driver in write-only mode like in the updated 19.03 builds (fixing the regression) but log-observe would still be necessary for "docker logs" to work on a journald-using container [21:05] so we might as well go for log-observe since that's what actually enables all the functionality of this feature IMO [21:05] (assuming it allows this bind) [21:05] (but it sounds like maybe it doesn't) [21:12] tianon: yeah I still think that auto-connecting log-observe is the most "clearly correct" thing to do, but it's not clear to me anymore that log-observe will fix your go-systemd issue with detecting journald [21:13] current test build has about 5-6 more minutes before I'll know for sure, but might also need a daemon restart (which will be 8-10 more minutes for another build) [21:13] :-) those build times are really reasonable compared to how long it takes for snapd CI to run [21:13] 😩 [21:14] I'm sure it doesn't help that we're building git from source -- every time I look at that it feels like there's a better way for us to do that / define a relationship there somehow [21:16] I don't remember the issue with staging git from the debian package, perhaps that just works now [21:16] it could be that the version of git with xenial is too old, so another reason why switching to a newer base snap may be useful [21:16] now that I've got some CI around the bits that invoke that, I'm planning to play with that a little [21:16] O:) [21:17] (I plan to play with that Soon too) [21:20] using newer snapcraft with lxd means I could go all the way to an Ubuntu 20.04 dev environment regardless of my "base:" value, right? (I think I've understood correctly that that's really the whole point of why it needs multipass/lxd to begin with?) [21:23] tianon: yes [21:24] tianon: if you were super ambitious you could even use mac os as your dev environment to build the snap (but not install it obviously) with snapcraft from brew and multipass from brew [21:24] maybe even windows now too [21:24] * ijohnson never knows what the state of snapcraft on windows is [21:24] lol, I would use Windows long before macOS [21:25] haha fair enough [21:25] but I know snappy on Windows is Complicated (thanks to WSL not using systemd) [21:26] WSL2 makes it easier and we are working towards full support of snaps on WSL2 [21:26] that's good to hear :D [21:26] there are some forum posts and such that explain how make snaps work ootb on WSL2 but it's a bit involved to set that up afaik [21:26] (although I really wish it were easier to run proper systemd in WSL) [21:26] yeah [21:26] systemd is basically the pain point there [21:26] yep [21:26] I've read those posts :) [21:26] it's pretty hacky [21:26] :-) [21:35] is there a config file somewhere I can add --use-lxd permanently? [21:35] (https://snapcraft.io/docs/build-on-lxd doesn't seem to mention anything, unless I've missed it) [21:37] tianon: you can set an env var, `SNAPCRAFT_BUILD_ENVIRONMENT=lxd` [21:39] that seems to work! thanks :) [21:39] tianon: also unrelated, any idea why dockerd would be responding this way? seems really odd that it says "unauthorized" https://pastebin.ubuntu.com/p/sf93dBHgfq/ [21:40] sounds like https://snapcraft.io/docs/t/the-snapcraft-build-environment-environment-variable/9110 is probably outdated in a few ways [21:40] tianon: that log is all from root, so not sure how it could be unauthorized [21:40] https://github.com/docker-library/official-images/issues/9562 :) [21:40] tianon: ahhh I see thanks [21:41] (TLDR, there was a Hub outage that was giving "authentication denied" errors even for anonymous images, but while it was happening it broke official images pushing leading to bad manifest lists) [21:41] also regarding that doc, I think actually that doc is right and you should use `host` instead of `lxd` [21:41] * ijohnson hasn't updated his bash profile in a long while [21:41] but "host" will try to apt-get install on my host, right? [21:41] like old snapcraft [21:41] oh wait sorry I have gone and confused myself [21:42] lxd does appear to be supported [21:42] yes use lxd, that doc is indeed very out of date [21:42] although it then fails to find python3-pip inside the container, so I guess I have more debugging of this VM to do :) [21:43] the build environment from the snapcraft managed lxd container is rather minimal [21:43] but the sources.list in it should be wide enough to include python3-pip right? [21:43] Could not find a required package in 'build-packages': python3-pip [21:43] yes it should be [21:43] is that core20 or core18 base ? [21:44] that's with a million warnings that I don't have base: set at all yet :) [21:44] (so I would assume it should've defaulted to xenial like it claims) [21:44] ah yes, so if you want to keep building on xenial but upgrade to newer snapcraft syntactical things you can do `base: core` [21:44] then snapcraft will let you build with a LXD container but the LXD container will be xenial [21:45] ah, the output implies it did that, but I guess there's probably more it's missing; I figured I should start with like-for-like to make sure the dev environment is good before I get too deep in any actual changes O:) [21:45] (especially since I assume this is probably similar to how snapcraft.io is building it?) [21:46] I don't actually remember how snapcraft.io/build builds snaps w/o `base:` [21:51] the output is already cleaner with "base: core" in my yaml [21:51] and appears to be working more successfully :) [21:57] aha, there's the infamous pip error which is now harder to work around since snapcraft is a snap so I can't trivially just patch the plugin file myself O:) [22:04] to confirm, log-observe was not sufficient, even with a daemon restart [22:04] on to kubernetes-support hacks :) [22:34] the kubernetes-support hackery appears to have been enough to make it work (with log-observe and a daemon restart) [22:34] nice [22:35] any chance that gets moved somewhere more common? [22:35] anything wanting to use go-systemd to detect journald having to connect this k8s interface is interesting :) [22:35] yeah as I said I think that's fine for docker to do to get the new version out the door, and I updated one of the related bugs to see about getting the new, more specific version of that access into either the default template or maybe something like network [22:35] :D [22:36] tianon: you could mark yourself as affected by https://bugs.launchpad.net/snapd/+bug/1867216 :-) [22:36] Bug #1867216: unix syntax does not easily accommodate unix autobind sockets [22:36] done! [23:07] PR snapd#9897 opened: usersession/autostart: change ~/snap perms to 0700 on startup [23:57] PR snapd#9896 closed: osutil: skip TestReadBuildGo inside sbuild