=== chihchun_afk is now known as chihchun [03:44] would be good to get snaps listed on this page https://github.com/google/sandboxed-api/blob/master/sandboxed_api/docs/sandbox-overview.md === ubott2 is now known as ubottu === chihchun is now known as chihchun_afk [06:09] morning [06:09] Good morning [06:10] just been outside, really feels like spring [06:10] amurray: interesting [06:10] mborzecki: I’m just stepping out [06:10] My back is killing me today, way too many hours coding yesterday [06:11] I will need a longer walk today [06:11] zyga: buy a standing desk [06:11] Eventually :-) [06:12] I need to buy new NAS disks next month [06:12] zyga: every couple of months tehre are some nice discouts at ikea === chihchun_afk is now known as chihchun [06:15] Does one go with mdraid or zfs nowadays? [06:17] I plan to get 3x(~5) TB in raid-5 and a 10+ TB drive for weekly snapshot off-case in case the main nas melts [06:46] looking at gadget validation again, obviously forgot to check that size must be set at the structure level :/ [07:02] back in the office [07:13] mborzecki: could you please look at https://github.com/snapcore/snapd/pull/6636 [07:13] PR #6636: cmd: prevent umask from breaking snap-run chain [07:13] it's not big but I want a second pair of UNIXy eyes :) [07:16] zyga: LOOKING [07:20] damn caps [07:21] mvo: morning [07:21] hey mborzecki [07:21] mborzecki: good morning [07:29] zyga: lgtm, left a comment about trying out umask 000 in the spread test too [07:30] after all if it's supposed to be restored fully [07:38] mborzecki: thank you, I will add more tests [08:02] linux is funny [08:02] mborzecki: you can have a process that lives in a current working directory that doesn't exist in its own mount namespace at all [08:02] funny things happen [08:05] morning [08:07] pstolowski: hey [08:08] hey pawel :) [08:08] hey mvo [08:14] hey zyga and pstolowski [08:23] zyga, hi, I just hit that issue with the snap (under try) reporting not found on executables inside the snap [08:23] zyga, how do I cat mountinfo in the right ns again? [08:24] hey! thank you for asking: use sudo nsenter -m/run/snapd/ns/foo.mnt [08:24] then inside the new shell run cat /proc/self/mountinfo [08:24] this information will be very useful [08:24] a bit of usabilty warning: there cannot be a space between -m and the path [08:25] ah that's what was hitting me [08:26] zyga, btw strace doesn't seem to give much info: https://paste.ubuntu.com/p/xf3WQRZWyw/ [08:28] zyga, what should I look for in the mountifo? [08:28] deleted [08:28] but I would love to see the whole listing [08:28] zyga, http://paste.ubuntu.com/p/zBq4X3TMXb/ [08:28] (this is in the ns) [08:28] no deleted [08:29] interesting [08:29] hmm hmm [08:29] can you run snap run --shell maas? [08:29] same errorr (execv failed) [08:30] hmmm [08:30] mborzecki: ^ any ideas [08:30] * zyga reads the mountinfo data [08:31] zyga, ftr I've run the rsync like hundred times yesterday and no issue [08:31] it's always like that, it randomly happens once in a while [08:31] ackk: what I find curious is that you can clearly nsenter [08:31] ackk: what's the base snap? [08:31] zyga, core18 [08:31] ackk: what are you tracking (which channel of the base snap) [08:32] ackk: did it refresh recently? [08:32] oh, I'm on core18 edge [08:32] hm that strace doesn't have much content [08:32] ah yes esterday [08:32] is that a classic snap? [08:32] refresh-date: yesterday at 13:38 UTC [08:32] ackk: did it break at around the same time it refreshed? [08:33] ackk: ahhh [08:33] I know :D [08:33] man [08:33] hold on please [08:33] * ackk curious [08:34] ah, no :/ [08:34] man, I thought this is it [08:34] we don't support try mode base snaps [08:34] but that's clearly not it [08:34] zyga: looks like it faiedl executing snap-confine, unless i'm missing something [08:34] ackk: when you rsync, what do you do exactly? [08:35] Permission denied is curious [08:35] zyga, https://github.com/maas/maas/blob/master/Makefile#L781 [08:35] ackk: do you have any apparmor denials? [08:36] zyga, it doesn't seem so [08:36] I mean I have stuff in dmesg but I don't see new errors if I try to run it [08:36] ackk: can you run /snap/core/6531/usr/lib/snapd/snap-confine --base core18 snap.maas.maas /usr/lib/snapd/snap-exec maas at all? [08:36] mborzecki: don't forget SNAP_INSTANCE_NAME=maas [08:37] no, same error [08:37] permission? [08:37] check dmesg again please [08:37] $ SNAP_INSTANCE_NAME=maas /snap/core/6531/usr/lib/snapd/snap-confine --base core18 snap.maas.maas /usr/lib/snapd/snap-exec maas [08:37] execv failed: No such file or directory [08:37] wait [08:37] before it was a permission denied error [08:37] no new error [08:37] ackk: since this a core18 snap [08:37] snapd (and snap-exec) come from another snap [08:37] can you check one more thing [08:38] run that same SNAP_INSTANCE_NAME=... line but run /bin/sh at the end [08:38] instead of snap-exec that is [08:38] once on the inside [08:38] run ls -ld /usr/lib/snapd/snap-exec [08:38] * zyga looks at mountinfo list again [08:38] er [08:38] zyga: s-c does some work, if it ran i'd expect to see taht in strace log [08:38] $ SNAP_INSTANCE_NAME=maas /snap/core/6531/usr/lib/snapd/snap-confine --base core18 snap.maas.maas /bin/ls -ld /usr/lib/snapd/snap-exec [08:38] /bin/ls: cannot access '/usr/lib/snapd/snap-exec': No such file or directory [08:39] there's no /usr/lib/snapd! [08:39] there is but it's empty [08:39] SNAP_INSTANCE_NAME=maas /snap/core/6531/usr/lib/snapd/snap-confine --base core18 snap.maas.maas /bin/sh [08:39] then cd /usr/lib [08:39] ls -ld snapd [08:39] ls -l snapd [08:40] * zyga checks stuff [08:40] $ ls -lad snapd [08:40] drwxr-xr-x 2 root root 0 Mar 21 11:41 snapd [08:40] $ ls -l snapd [08:40] total 0 [08:40] the dir is there, but empty [08:40] is that a mountpoint? [08:40] I mean, does snapd bind-mount stuff on top? [08:41] it should be a mountpoint [08:41] zyga, so you think it could be related to the core refresh? [08:41] that explainswhat you are seeing [08:41] perhaps [08:42] can you collect snap changes and add this to the bug report [08:42] it might well be that's been broken since yesterday,but I haven't restarted the service (nor called the cli) since then, although I think I did [08:42] ackk: do you have snapd.snap? [08:42] zyga, where? [08:42] is snapd the snap installed on your system [08:43] no [08:43] snap list | grep snapd :) [08:43] ok [08:43] thank you [08:43] ackk: I think we have enough to explore this now [08:43] the key question is what happens when core18 + core + app combo has a refresh of core18 [08:43] or a refresh of core [08:43] when core refreshes it's different [08:43] actually [08:43] that's a separate bug [08:43] we will never update the tooling [08:43] mvo: core18/core bug [08:44] are core18 apps still somewhat dependent on core? [08:44] I think I had filed a bug a long time ago about it, and I've still seen the issue recently [08:44] mvo: when /usr/lib/snapd comes from core on a core18 base snap app, the core refresh is not reflected in the mount namespace of the app [08:44] that is [08:44] zyga: they shouldn't be - hm, hm [08:44] mvo: future snapd is using old snap-exec [08:44] mvo: it's like a stale base snap prolem [08:44] zyga: oh, snap-exec is not coming from snap? [08:44] mvo: but a stale core snap sub-problem [08:44] zyga: not from the snapd snap? [08:45] mvo: when it comes from core [08:45] there is snapd snap aroung yet [08:45] mvo: core18 + core (as source of snap tools) + app [08:45] there is *no* [08:45] mvo: core refreshes [08:45] mvo: now you have stale tools in the app snap [08:45] ok [08:45] mvo: or perhaps that is exactly what ackk observed, an tools just disappear somehow [08:45] zyga, fwiw "core" refreshed 10 days ago [08:45] zyga: basically all the snaps that compose the mount namespace can have a staleness prolem no? [08:45] also content snaps [08:45] mvo: thank you [08:46] pedronis: yes [08:46] zyga: what do we need to do to unstable the mount namespace? [08:46] zyga: we check only base though? [08:46] pedronis: content snaps are better [08:46] pedronis: because those are new paths that change in the mount profile [08:46] pedronis: so they are correctly updated [08:46] pedronis: it's just the core tooling for now [08:46] pedronis: nothing else exists outside of the snap-update-ns system [08:47] pedronis: yes, we check for base only explicitly because that's the do-I-rebuild-from-scratch decision [08:47] mvo: can you rephrase the question please? [08:47] zyga, should I try refreshing core18 (to stable perhaps)? [08:47] ackk: to aid in debugging? no, to lower the inconvenience for your development: yes [08:47] zyga: what do we need to do to fix this .) ? [08:47] ackk: you can discard the mount namespace to fix it [08:48] mvo: I dont know yet, I don't fully undestand the reason /usr/lib/snapd was empty [08:48] but I think we should be able to reproduce it now [08:48] zyga, the var/lib/snapd in the snap? [08:48] ackk: sorry? [08:48] mvo: perhaps it's one issue after all [08:48] zyga, oh sorry, discard the namespace I see [08:49] zyga, I thought you meant unmounting something [08:49] mvo: sudo /usr/lib/snapd/snap-discard-ns mass [08:49] er [08:49] ackk: ^ :) [08:49] zyga, thanks [08:49] zyga, oh refreshing to stable also worked [08:50] yes [08:50] because then you have new base [08:50] and you get a fresh mount ns [08:50] I see [08:50] ackk: thank you very much for this [08:50] please do report the bug [08:51] we may split it if this turns out to be a pair of issues [08:51] zyga, thank you for digging it [08:51] I will work on it next week for sure [08:51] fyi, I need to do some non-PR work before getting back to reviewing [08:52] zyga, is "snap stops working after base refresh" accurate as a title? [08:52] ackk: /usr/lib/snapd unmounted in app + core18 + core (tooling) use case; possibly on core refresh [08:52] that is the summary in my head [08:53] zyga, but it was a core18 refresh, not core [08:53] ah, sorry [08:53] I will check all combinations though [08:54] there are clearly missing bits there [08:55] brb, coffee ran out [08:59] zyga, https://bugs.launchpad.net/snapd/+bug/1821302 [08:59] Bug #1821302: /usr/lib/snapd unmounted in app + core18 + core (tooling) use case; possibly on core18 refresh [09:06] ack :) [09:07] this is the week of snap-confine fixes and changes [09:22] zyga, IDK if it could be related, but I still see the issue where if I install a core18-based snap on a clean system (no core snap) it fails to install (the configure hook fails) [09:22] zyga, e.g. snap install --edge maas [09:22] ackk: yes, this is related [09:22] ackk: we track that bug separately now [09:22] zyga, do you have the link? [09:22] ackk: I have a draft solution on my system but it will not be ready yet [09:24] ackk: there's a forum thread aboug this, i have not upgraded it to a bug report yet: https://forum.snapcraft.io/t/problems-installing-a-base-core18-snap-on-bionic/10084/ [09:25] zyga, ah cool thanks [09:28] mborzecki: https://www.omgubuntu.co.uk/2019/03/linux-data-cap-survey [09:29] aah survey [09:29] nice [09:31] mvo: hey, added some extra tests you suggested for #6591 [09:31] PR #6591: overlord/ifacemgr: basic measurements [09:33] zyga: haha 'RSS feeds' in question 'How would you prioritise these data to download if you had limited internet access?' [09:33] yeah [09:33] who cares about that [09:33] who ordered RSS? [09:34] pstolowski: thanks, looking [09:35] completed the survey [09:35] as someone who is on capped links 24/7 it's important that people care about this [09:36] PR snapd#6637 opened: interfaces: deal with the snapd snap correctly for apparmor 2.13 [09:38] mvo: there's a card for that https://trello.com/c/1h4hrY3W/180-implement-snapd-snap-support-in-apparmor-backend-dbus-backend-setup-etc ^ [09:39] also I think there were todo left in other backends too [09:40] pedronis: indeed, added the PR and will see what else needs to be done there [10:00] mvo: should also be moved to doing :) === ricab is now known as ricab|bbiab [10:31] mborzecki: hi, do you know why we would get a line like this: [10:31] content gnome-calculator:gtk3-themes gtk-common-themes:gtk3-themes [10:31] in connections, there is not content discriminant there === ricab|bbiab is now known as ricab [10:31] pedronis: that with 2.38? [10:31] with edge [10:32] but anyway I get the other ones with [] [10:32] but not that one [10:32] I thought we always set one internally [10:32] mborzecki: I'm doing snap connections gnome-logs [10:32] pedronis: mhm, installing the snap right now [10:34] hmmm pinner with 'Automatically connect eligible plugs and slots of snap "gnome-calculator"', didn'g we have a fix that would switch to showing progress of the download if a dependncy is being installed? [10:34] s/pinner/spinner/ [10:36] pedronis: .38 has // [10:36] content[gnome-3-28-1804] gnome-calculator:gnome-3-28-1804 gnome-3-28-1804:gnome-3-28-1804 - [10:37] PR snapd#6638 opened: interfaces: add support for the snapd snap in the dbus backend [10:37] mborzecki: I'm talking about gtk3-themes [10:37] but I don't see that in the snaps [10:37] mmh === chihchun is now known as chihchun_afk [10:37] content[gtk-3-themes] gnome-logs:gtk-3-themes gtk-common-themes:gtk-3-themes - [10:37] content[icon-themes] gnome-logs:icon-themes gtk-common-themes:icon-themes - [10:38] content[sound-themes] gnome-logs:sound-themes gtk-common-themes:sound-themes - [10:38] mborzecki: it's the issue that we don't remove non-existent connections I suppose [10:38] mborzecki: it's not a connections problem, it's other issues [10:38] mborzecki: if you just installed it you won't see that [10:39] pedronis: buf if i disconect, remove and reinstall? [10:39] yes, a different case [10:47] mvo: fun bug, command-not-found is still in use while in snap run- --hell [10:47] (I meant --shell) [10:47] but the binary is not there anymore [10:52] failed to prepare google:opensuse-15.0-64:project, do we have an issue with opensuse in gcloud again? [10:54] PR core18#122 opened: hooks: add apt/apt-get/apt-cache warning [11:03] pstolowski: what was the failure? [11:04] pstolowski: ah, I see it now [11:04] cachio: ^ we need to change how we install packages [11:04] mborzecki: ^ any ideas, is that a stale cache? [11:05] openSUSE Leap 15.0 package installation issue in spread https://www.irccloud.com/pastebin/Z5UvBD4B/ [11:06] hmm [11:07] zyga: interesting, i didn't see any error output for google:opensuse-15.0-64:project, just failed to prepare [11:07] I saw this in one of my own runs [11:07] * zyga needs to take a break [11:08] to avoid the chronic back pain [11:08] mborzecki: there will be an interesting patch later today [11:08] mborzecki: another unixy thing :) [11:09] no clue about that opensuse thing, perhaps unfinished mirror sync? [11:09] I think it's like dnf upgrade vs dnf distrosync [11:10] pstolowski: is there anything left for 6491 that can't be follow ups about improving the test? [11:10] we are probably doing something wrong and should be using "dnf dup" instead of another operation [11:10] but it's leap, zypper up should be enough [11:10] mborzecki: up != dup [11:10] mborzecki: but not sure :/ [11:10] anyway, I spawned some larger testing for an important snap-confine change, I will break for an hour to move around a bit [11:10] zyga: i mean zypper up should be enough for leap, tw has it's own weirdnness :) [11:10] I'm on telegram if anyone needs me urgently [11:11] \\\\\\! [11:11] Sorry, words of wisdom from my daughter there [11:12] haha :) [11:16] zyga, just to confirm, I hit the issue again (no core refresh this time) and calling snap-discard-ns fixed it [11:19] zyga, actually not so much, now I get a "read only filesystem" error [11:21] cannot create user data directory: /root/snap/maas/x1: Read-only file system [11:25] ackk: x1 got unmounted somehow? [11:26] zyga, well I suspect it's because of the discard ns ? [11:26] ackk: is your snap "broken" (as in snap list shows broken next to it) [11:26] no, that's not doing that [11:26] no it wasn't [11:26] discard will not change /snap/anything [11:26] (now I actually removed it) [11:26] ah, wait [11:26] I misread that [11:26] what is /root/snap/maax/x1 [11:26] is that a bind mount? [11:27] it would only be a ROFS in case it's a bind mount or if /root inside the mount namespace was unmounted [11:27] but if _that_ is the case then I would really like to know what's going on [11:28] * zyga is AFK now [11:29] zyga, it's the user dir [11:39] the fun start when offset 0 is a valid value [11:40] looks like some of the fields in our gadget structures should be pointers [11:46] off to the kids' school for a bit [12:12] pedronis: sorry, missed your earlier message; no, nothing left to do in #6491 [12:12] PR #6491: interfaces: hotplug nested vm test, updated serial-port interface [12:14] nb it failed on 14.04 with 'system does not fully support snapd: cannot mount squashfs image' ?! === jdahlin_ is now known as jdahlin === lfaraone_ is now known as lfaraone [12:14] among 2 other unrelated test failures.. interfaces-daemon-notify seems to be flaky [12:17] People : hi ! I've come to this issue with the snap version of lxd : https://github.com/lxc/lxd/issues/5590 is that related to snap or to lxd and is there a way to get things work as they are expected to ? To be more specific, I guess that snap don"t like user's home directory to be out of /home [12:19] geodb27: correct, user homes need to be in /home currently, no way around that [12:21] So, my guess was right, thanks for you answer pstolowski. However, to my knowledge, snap is the only one I encounter that relies on this. Is there any technical reason for that (I'm only curious) ? There is an env var set at user login which is filled in with the home directory. Why not use it ? [12:21] pstolowski: did it have a recent master merge? [12:21] pedronis: hmm possibly not, i merged yesterday. doing [12:26] geodb27: it's more complicated due to confinement (apparmor profiles etc) [12:28] oki, I see, thanks again for your kind explanation pstolowski. This will lead me to to many changes on my servers... I'll see how I can go with the less damages. [12:28] geodb27: you're welcome! [12:35] zyga, taking a look to opensuse now [12:36] geodb27: I can refer you to the technical reasons but I am AFK now [12:36] Thanks cachio [12:53] re [12:58] PR snapd#6639 opened: snap: tweak parsing errors of gadget updates === ComradeCrocodill is now known as Crocodillian [13:19] zyga, why would snapd try to create that directory (/root/snap/maas/x1 when starting a service? [13:30] ackk: we create SNAP USER DATA [13:30] because snaps themselves cannot [13:31] yeah but that dir is there [13:31] why does the snap suddenly thinks it's readonly === ricab is now known as ricab|lunch === ShibaInu is now known as Shibe [14:41] PR snapd#6640 opened: spread: refresh metadata on openSUSE [14:47] zypper tell me the latest version of libudev is 234-lp150.20.15.1, not 234-lp150.19.1.x86_64 [14:49] cachio: was opensuse-15.0 image already updated? [14:57] did the forum fall over? https://usercontent.irccloud-cdn.com/file/kZeiOkOL/image.png [15:02] seems opening a different tab rectified it [15:02] ignore me [15:06] mborzecki, no yet [15:07] mborzecki, found more problems [15:07] mborzecki, some package digests are different === ricab|lunch is now known as ricab [15:20] mborzecki, the image is ready and uploaded [15:21] uploading [15:28] cachio: cause before asking i tried running zypper in that were failing in the test and it worked, so maybe just a mirror sync problem [15:28] cachio: still, zypper up show a number of updates avaialble at that point, will try with an updated image now [15:33] mborzecki, yes, the mirrors are giving us problems [15:33] mborzecki, I already updated all the packages to make the update process faster [15:33] I is almost uploaded [15:36] mborzecki, image ready [15:38] cachio: it probably was something with mirror sync, #6640 started before an updated image was uploaded and it's green [15:38] PR #6640: spread: refresh metadata on openSUSE [15:41] * cachio lunch [15:42] The next Snapcraft Summit has been announced - https://snapcraft.io/blog/snapcraft-summit-montreal [15:46] PR snapd#6641 opened: snap-gdb-shim: switch to the SUDO_UID when available [15:46] zyga: we're continuing to see reports on the forum of users not able to run the gnome snaps because the content interface isn't mounted [15:47] zyga: it's something i've only seen a couple times and snap-discard-ns seems to resolve it [15:48] zyga: latest reference https://forum.snapcraft.io/t/gnome-calculator-problem/4579 [15:48] PR snapd#6640 closed: spread: refresh metadata on openSUSE === cpaelzer__ is now known as cpaelzer [15:50] zyga: if i can't find an existing bug report, i'll file a new one [15:51] pedronis: you've requested John's review on #6591; is it required or can it land now that it has mvo's review? [15:51] PR #6591: overlord/ifacemgr: basic measurements [15:51] pstolowski: it can land, I asked him in case mvo didn't have time [15:51] ack [15:52] thx [15:52] PR snapd#6591 closed: overlord/ifacemgr: basic measurements [15:54] zyga: bug 1785410 [15:55] Bug #1785410: Can not open Gnome snaps as they are not connected to «gnome platform snap»