[00:10] <mup> PR # closed: snapd#5644, snapd#5915, snapd#5962, snapd#6079, snapd#6098, snapd#6108, snapd#6111, snapd#6162, snapd#6177, snapd#6238, snapd#6258, snapd#6270, snapd#6281, snapd#6313, snapd#6322, snapd#6324, snapd#6325, snapd#6327, snapd#6329, snapd#6341, snapd#6347, snapd#6356, snapd#6360,
[00:10] <mup> snapd#6367, snapd#6376, snapd#6381, snapd#6401, snapd#6404, snapd#6410, snapd#6418, snapd#6436, snapd#6472, snapd#6482, snapd#6485, snapd#6487, snapd#6488, snapd#6491, snapd#6492, snapd#6494, snapd#6496, snapd#6497, snapd#6498, snapd#6499, snapd#6501, snapd#6502, snapd#6503, snapd#6504
[00:11] <mup> PR # opened: snapd#5644, snapd#5915, snapd#5962, snapd#6079, snapd#6098, snapd#6108, snapd#6111, snapd#6162, snapd#6177, snapd#6238, snapd#6258, snapd#6270, snapd#6281, snapd#6313, snapd#6322, snapd#6324, snapd#6325, snapd#6327, snapd#6329, snapd#6341, snapd#6347, snapd#6356, snapd#6360,
[00:11] <mup> snapd#6367, snapd#6376, snapd#6381, snapd#6401, snapd#6404, snapd#6410, snapd#6418, snapd#6436, snapd#6472, snapd#6482, snapd#6485, snapd#6487, snapd#6488, snapd#6491, snapd#6492, snapd#6494, snapd#6496, snapd#6497, snapd#6498, snapd#6499, snapd#6501, snapd#6502, snapd#6503, snapd#6504
[03:11] <NickZ> man, the arm build farm on launchpad is backed up something fierce
[03:11] <NickZ> i have a 3 hour wait time on my snap packages
[06:51] <zyga> Good morning
[07:14] <zyga> Hey mvo
[07:27]  * zyga updated his aquaris E4.5 to "16"
[07:27] <zyga> guess my phone runs xenial now
[07:30] <mvo> zyga: good morning
[07:31] <zyga> wow, the phone says in the future it will support snaps!
[07:31] <zyga> that's neat
[07:31] <zyga> (though they said that it is not supported now because of upstart)
[07:31] <mvo> nice
[07:45] <MattJ> Hmm, is there a way to reinstall a snap with --classic without losing its data?
[07:45] <mvo> zyga: btw, did you find out more from morphis about the jenkins issue he has with 2.37?
[07:45] <zyga> mvo: no, we need to get more feedback about how to reproduce the issue
[07:46] <mvo> zyga: thanks! so he didn't reply yet?
[07:46] <zyga> I failed to reproduce it on 18.04 with lxd and classic snaps in /var/lib/test
[07:46] <zyga> correct
[07:46] <mvo> zyga: thanks for looking into this!
[07:46] <zyga> morphis: ^ please ping us when you are up
[07:54] <mup> PR snapd#6501 closed:  tests/main/auto-refresh-private: make sure to actually download with the expired macaroon <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6501>
[07:54] <mvo> 6503 is an easy win
[07:57] <morphis> zyga, mvo: hey!
[07:57] <zyga> good morning :)
[07:57] <zyga> mvo: (reviewed)
[07:57] <morphis> zyga: didn't you say you can easily reproduce it?
[07:58] <zyga> morphis: that was dry run mode, it is easy to reproduce conceptually but it doesn't actually happen for me
[07:58] <morphis> ok
[07:58] <zyga> morphis: so I started digging deeper to the point where I had lxd and the same version and it still wasn't happening
[07:58] <morphis> zyga: so what do you need from me?
[07:58] <zyga> morphis: in your setup, how was jenkins installed?
[07:58] <morphis> good question, plars did it long time ago
[07:58] <morphis> I guess via debs
[07:59] <zyga> can you drop us a bug report with: os version, container installation system on host, guest container os version, snap version on both sides, jenkins installation method in guest and the name of the snap used in the CI job
[07:59] <zyga> I will use that to reproduce and fix it
[08:00] <zyga> I mean, we kind of feel the fix is a one liner
[08:00] <zyga> but without a regression test
[08:00] <zyga> it's a hand-wave-y one
[08:03] <morphis> zyga: where do you guys file bugs these days? github?
[08:03] <pstolowski> morning
[08:03] <zyga> morphis: still on lp
[08:03] <morphis> zyga: ok
[08:03] <zyga> launchpad.net/snapd
[08:05] <morphis> zyga: the snap isn't from the store but I will drop it by mail
[08:05] <zyga> morphis: it is probably sufficient to classify it (strict or classic)
[08:05] <zyga> the denial was not related to the snap but to snap-confine
[08:07] <morphis> it's a classic snap
[08:08] <zyga> perfect
[08:08] <zyga> ok
[08:11] <pedronis> zyga: does this rings any bell:   cannot perform readlinkat() on the mount namespace file descriptor of the init process: Permission denied
[08:11] <zyga> yes
[08:13] <zyga> phone
[08:14] <morphis> zyga: https://bugs.launchpad.net/snapd/+bug/1815869
[08:14] <mup> Bug #1815869: Command of classic snap fails with denial when output is redirected <snapd:New> <https://launchpad.net/bugs/1815869>
[08:17] <mvo> pedronis: do we have dmesg output with the appamror denials for this one?
[08:17] <pedronis> mvo: I don't know, it was on a server
[08:18] <pedronis> mvo: I'm asking
[08:22] <zyga> One sec, home stuff
[08:23] <zyga> re
[08:24] <zyga> pedronis: it is an issue that surfaces once in a while, typically on older kernels
[08:24] <zyga> pedronis: it doesn't affect anything, that I'm aware of, is in production
[08:24] <zyga> morphis: thank you
[08:24] <pedronis> zyga: a service died on it
[08:24] <pedronis> but there were other denials before as well
[08:25] <pedronis> then it restarted
[08:25] <pedronis> I mean snap-confine denials
[08:25] <zyga> do you have the log?
[08:26] <pedronis> yes
[08:34] <zyga> morphis: I asked one more question on the bug
[08:35] <morphis> zyga: answered
[08:37] <zyga> morphis: I'm not familiar with lxd that much, how do I set up nesting + privileged
[08:37] <zyga> morphis: is lxd on the host a snap?
[08:38] <morphis> yes
[08:39] <morphis> lxc launch ubuntu:x test -c security.nesting=true -c security.privileged=true
[08:39] <zyga> thank you!
[08:39] <morphis> yw
[08:39] <zyga> launching now
[08:39] <zyga> I suspect it is an unsupported configuration in some form, I will confirm soon :/
[08:41] <zyga> https://www.irccloud.com/pastebin/3G9BKJuL/
[08:42] <zyga> morphis: ^ this is from installing core in that system
[08:42] <zyga> Feb 14 08:41:18 xenial snapd[378]: 2019/02/14 08:41:18.178394 backend.go:303: cannot create host snap-confine apparmor configuration: cannot synchronize snap-confine apparmor profile: open /var/lib/snapd/apparmor/profiles/snap-confine.core.6350.Ll210MNsYGb3~: no such file or directory
[08:44] <zyga> I'm installing core now
[08:44] <zyga> mvo: will xenial get SRU for newer snapd?
[08:45] <morphis> zyga: that one is new, I am running snapd in various LXD containers for a long time now
[08:46] <mvo> zyga: for 2.37.2 ? yes
[08:46] <zyga> mvo: perfect, thank you
[08:46] <mvo> zyga: I think its even in the unapproved queue, I need to check
[08:46] <morphis> zyga: hm, that doesn't happen in a unprivileged container
[08:46] <mvo> zyga: is that the fix?
[08:46] <mvo> zyga: if so I will push a bit harder for that
[08:47] <zyga> morphis: specifically apparmor stacking but please hold on, I don't know the key part yet
[08:47] <zyga> mvo: no, just curious
[08:47] <morphis> zyga: unprivileged does not mean there is no apparmor profile for the LXD container
[08:48] <morphis> s/unprivileged/privileged/
[08:48] <zyga> installing strictly confined snap in xenial container with privileged/nesting enabled https://www.irccloud.com/pastebin/ghbLQFrn/
[08:50] <zyga> morphis: ^ I cannot even install a snap in this container
[08:51] <zyga> morphis: but I managed to reproduce the issue
[08:51] <zyga> lut 14 09:50:40 crusty kernel: audit: type=1400 audit(1550134240.752:161): apparmor="DENIED" operation="file_inherit" namespace="root//lxd-xenial_<var-snap-lxd-common-lxd>" profile="/snap/core/6405/usr/lib/snapd/snap-confine" name="/home/ubuntu/foo" pid=15267 comm="snap-confine" requested_mask="w" denied_mask="w" fsuid=1000 ouid=1000
[08:51] <zyga> at least that much is good
[08:51] <zyga> I will investigate
[08:51] <morphis> zyga: then the initial snapd setup broke at some point. This container is running for ~1.5y now
[08:52] <morphis> zyga: great!
[08:55] <zyga> I updated the bug report again
[08:56] <zyga> morphis: unsupported may not be broken for some time but may still be unsupported, I'm trying to get to the bottom of what that container setup does
[08:56] <morphis> zyga: security.privileged=true means the container is not running in a UI namespace
[08:57] <morphis> security.nesting=true allows processes inside the container to use a stacked AppArmor profile which is otherwise  forbidden by the default LXD AppArmor policy
[08:57] <zyga> UI namespace?
[08:57] <morphis> s/UI namespace/UID namespace/
[08:57] <zyga> ah
[08:58] <zyga> isn't security.nesting the default? otherwise snaps would just be permanently broken inside containers
[08:58] <morphis> no
[08:58] <zyga> that's unexpected, are you sure?
[08:58]  * zyga tries another container
[08:58] <morphis> security.nesting=false is the default
[08:58] <morphis> check https://lxd.readthedocs.io/en/latest/containers/
[08:59] <morphis> that is why you need security.nesting=true if you want to run snaps inside LXD containers
[08:59] <zyga> we never set that in our lxc tests
[08:59] <morphis> lxc or LXD?
[09:01] <morphis> zyga: looks like snapd works without security.nesting=true too
[09:01] <zyga> snapd works correctly then
[09:01] <zyga> the whole container has nesting
[09:01] <zyga> perhaps that option controls the ability to use nested apparmor inside the container
[09:01] <zyga> so more nesting
[09:01] <zyga> otherwise we do get nesting
[09:02] <morphis> no, you still can't run nested LXD or docker inside a container without security.nesting=true
[09:02] <zyga> the whole profile is namespaced
[09:02] <zyga> (ie: nested)
[09:02] <zyga> that's different
[09:02] <zyga> lxd does nesting
[09:02] <zyga> so you need security.nesting to run lxd inside lxd
[09:02] <zyga> snapd doesn't do nesting
[09:02] <morphis> yes
[09:02] <zyga> it just requires a clean slate (nested setup as lxd does automatically)
[09:02] <morphis> see https://github.com/lxc/lxd/blob/a6b780703350faff8328f3d565f6bac7b6dcf59f/lxd/apparmor.go#L418
[09:03] <zyga> so a non-privleged container without extra nesting support works OK
[09:04] <zyga> I ran a busybox snap inside a default container
[09:04] <zyga> zyga@crusty:/proc/19399$ cat attr/apparmor/current
[09:04] <zyga> lxd-xenial-default_</var/snap/lxd/common/lxd>//&:lxd-xenial-default_<var-snap-lxd-common-lxd>:snap.snapd-hacker-toolbelt.busybox (enforce)
[09:04] <zyga> it has a nested profile applied
[09:04] <zyga> ok, so that much is good
[09:05] <zyga> lxd nowadays also uses zfs
[09:05] <zyga> man, all of those are not tested by us :/
[09:07] <popey> degville: when you have a moment, I have some PRs for the tutorial :)
[09:08] <degville> popey: awesome, thanks!
[09:08] <zyga> pstolowski: hey
[09:08] <zyga> sudo snap remove gnome-3-26-1604 gnome-calculator gnome-characters gnome-logs gnome-system-monitor gtk-common-themes
[09:09] <zyga> this takes forever on 18.04
[09:09] <zyga> conflicts
[09:09] <morphis> zyga: it uses ZFS for quite a long time, if I remember right even 2.0 used it
[09:09] <zyga> pstolowski: conflicts https://www.irccloud.com/pastebin/fgPcJpHn/
[09:10] <zyga> pstolowski: is this expected?
[09:10] <pedronis> zyga: yes, known bug
[09:10] <pedronis> there's actually a fix up
[09:10] <zyga> aha, good
[09:10] <pedronis> but needs to land early in 2.39 cycle
[09:10] <pedronis> because is delicate
[09:11] <pstolowski> yep
[09:12] <pstolowski> zyga: it should succeed after w (long) while
[09:13] <pstolowski> pedronis: found intersting bug around hotplug and disconnect thanks to my spread test
[09:15] <zyga> morphis: a container using lxc launch ubuntu:x xenial -c security.nesting=true works OK
[09:15] <zyga> morphis: I can install more snaps but I still get the denial
[09:16] <zyga> morphis: I should be able to fix that this way
[09:16] <Chipaca> o/
[09:16] <zyga> morphis: but using privileged containers is (for whatever reason) not supported now as things break on installation
[09:16] <zyga> hey Chipaca :)
[09:17] <Chipaca> zyga: dunno if you've seen https://forum.snapcraft.io/t/9927/5
[09:17] <mup> PR snapd#6505 opened: packaging: avoid race in snapd.postinst <Created by mvo5> <https://github.com/snapcore/snapd/pull/6505>
[09:17] <morphis> zyga: it was working until 2.37.*
[09:18] <zyga> morphis: that's unrelated
[09:18] <mvo> hey Chipaca - good morning
[09:18] <zyga> I'm saying _supported_
[09:18] <zyga> it may work by accident
[09:18] <Chipaca> mvo: suuuup
[09:18] <zyga> Chipaca: I haven't
[09:18] <pstolowski> pedronis: almost at the end of the test after a series of succesful plugin/replugging i re-plug the device, snap interfaces is happy, connection is restored, but "snap disconnect .." creates an empty change with no tasks (status=Hold). adding an artificial sleep in the test just before disconnect makes it happy. the disconnect bit in api.go looks suspicious, we query repo a few times but there is no locking, so in theory
[09:18] <pstolowski> we may get inconsistent view of things. ionterestingly it happens all the time on 16.04, but not on 18.04 (in nested vm)
[09:18] <morphis> zyga: is it documented anywhere which configuration is supported and which not?
[09:18] <Chipaca> zyga: any snap app gets "cannot create lock directory /run/snapd/lock: Permission denied"
[09:19] <zyga> morphis: only what we test
[09:19] <morphis> that is pretty hard to dig out
[09:19] <zyga> morphis: regular and cloud versions of ubuntu + several other systems
[09:19] <zyga> morphis: lxd on ubuntu with default config
[09:19] <mvo> Chipaca: more fun with .37
[09:19] <zyga> morphis: that's that, anything more may work or not but is not something we test for
[09:20] <zyga> morphis: but I'm not saying it's hopeless :)
[09:20] <zyga> morphis: just clarifying why we have not caught that
[09:20] <morphis> sure
[09:20] <zyga> morphis: do you need to run a privileged container in your case or was that just a convenience/accident?
[09:24] <Chipaca> mvo: in what shape, this time?
[09:24] <pedronis> pstolowski: ah, interesting
[09:25] <zyga> hmm, I opened a forum topic for unsupported features
[09:25] <zyga> and I cannot find it
[09:25] <Chipaca> zyga: finding forum topics for unsupported features is an unsupported feature
[09:28] <pedronis> pstolowski: ok, I'm not surprised we found issues around our use of the repo, it has an internal lock, doesn't mean it be consulted many times without an external one
[09:28] <pstolowski> pedronis: yes
[09:28] <zyga> pedronis: yes, that's true
[09:29] <zyga> pedronis: in all the cases that we need consistency we should craft a method that uses private unlocked versions while holding the lock for the repo
[09:29] <pstolowski> pedronis: although what's puzzling me is the fact that snap interfaces check done before disconnect reports the connection, so in theory things should be settled at this point
[09:30] <pstolowski> but clearly the need for short "sleep" before disconnect contradicts it
[09:33] <mvo> Chipaca: apparently the catalog update is a bit too non-random and causes store issues
[09:33] <pedronis> mvo: I'm discussing that with Chipaca
[09:33] <mvo> ta
[09:33] <mvo> I was about to say - pedronis knows the details better :)
[09:34] <pstolowski> zyga: re the waiting.. issue you asked about earlier - https://github.com/snapcore/snapd/pull/6472
[09:34] <mup> PR #6472: overlord: fix ensure before slowness on Retry <⛔ Blocked> <Created by stolowski> <https://github.com/snapcore/snapd/pull/6472>
[09:35] <zyga> pedronis: is that overlord change so risky that we cannot just land it for .38?
[09:35] <Chipaca> mvo: I was hoping it was the same thing =)
[09:36] <pedronis> zyga: we can reevaluate, remember in theory we should haved cut 2.38 already
[09:36] <pedronis> but here we are
[09:37] <zyga> yeah
[09:37] <zyga> I would vote on landing it to master and reverting if we need to release and feel uneasy about it
[09:39] <pstolowski> zyga: it looks innocent but is of high risk. if there is anything wrong around this area you may end up with snapd getting stuck and not picking tasks (been there while working on this fix)
[09:39] <zyga> pstolowski: hence edge
[09:41] <pedronis> pstolowski: we could probably mitigate the risk of getting totally stuck doing something in the Loop
[09:47] <pstolowski> pedronis: i can look at it ~next week if time on sprint permits as this week i'd like to address hotplug PRs. i wouldn't rush this fix though given that we don't hear many complaints about it (do we?)
[09:47] <pedronis> zyga: pstolowski: I don't think we risk getting totally stuck actually, because of the PruneTicker
[09:47] <morphis> zyga: I currently trying to figure out what the reason was for that
[09:49] <pstolowski> pedronis: getting stuck was most likely specifically related to my attempts to read from timer channel, so you may be right
[09:50] <pedronis> pstolowski: yes, that we don't do
[09:51] <pedronis> we have one place doing that
[09:51] <zyga> morphis: so I applied a simple fix
[09:51] <zyga> but ... it has no effect?
[09:51] <zyga> digging
[09:52] <zyga> ah, sorry
[09:52] <zyga> all good
[09:52] <zyga> morphis: I will have a fix and a regression test
[09:52] <zyga> mvo: ^ CC, low risk patch re-introducing /var/lib/** rw,
[09:53] <mvo> zyga: that sounds good
[09:53] <mvo> zyga: so re-adding this fixed morphis bug?
[09:53] <zyga> mvo: yes, as expected
[09:53] <zyga> it is interesting that lxd is a factor
[09:54] <pedronis> do we have a +1 from jdstrand?
[09:54] <zyga> because of how profile transitions work
[09:54] <pedronis> is the setup supported anyway?
[09:54] <zyga> pedronis: he said as much yesterday in snappy-internal
[09:54] <zyga> pedronis: lxd is supported, home in /var/lib is not but this was working before so we can keep it working until apparmor grows the delegation features jdstrand described
[09:55] <zyga> pedronis: the specific case of lxd with stock profile, snapd with classic snaps and home in /var/lib/* will work
[09:58] <pedronis> zyga: can we make a test?
[09:58] <zyga> pedronis: certainly, I'm on it now :)
[09:58] <zyga> pedronis: that's why I ask for bug reports
[09:58] <zyga> then I can make a regression test and have a matching bug number with more details
[10:12] <pedronis> pstolowski: mvo: I removed the blocked on #6472 with a comment
[10:12] <mup> PR #6472: overlord: fix ensure before slowness on Retry <Created by stolowski> <https://github.com/snapcore/snapd/pull/6472>
[10:12] <mvo> thanks
[10:12] <pstolowski> pedronis: k, thanks
[10:18] <pstolowski> it needs 2nd review still
[10:18] <pstolowski> pedronis: i've addressed/replied to your comments on https://github.com/snapcore/snapd/pull/5962
[10:18] <mup> PR #5962: ifacestate/hotplug: hotplug handlers <Complex> <Hotplug 🔌> <Created by stolowski> <https://github.com/snapcore/snapd/pull/5962>
[10:30] <pedronis> pstolowski: do we know already an example that would produce different interfaces at the same devpath?
[10:31] <pstolowski> pedronis: now, but i was thinking about an "foo-observer" and "foo-controller" type of interfaces
[10:31] <pstolowski> *no
[10:31] <pedronis> ok, I see
[10:31] <pstolowski> so, read-only or full control
[10:34] <pedronis> pstolowski: did you not push some changes?
[10:35] <pstolowski> pedronis: ah, of course
[10:35] <pedronis> or is github confusing me
[10:38] <pstolowski> pedronis: sorry, now
[10:42] <pstolowski> pedronis: aaa, we have err shadowed in api.go disconnect - this is the reason of earlier disconnect issue i talked about: error: snap "serial-port-hotplug" has "hotplug-add-slot-serial-port" change in progress
[10:43] <pstolowski> repo access wrt locks is a separate potential problem, but not in this case i think
[10:45] <mup> PR snapd#6506 opened: overlord/snapstate: add some randomness to the catalog refresh <Created by chipaca> <https://github.com/snapcore/snapd/pull/6506>
[10:46] <Chipaca> pedronis: ^ à votre santé
[10:58] <zyga> ok, that should do it :)
[11:01] <mup> PR snapcraft#2472 closed: project loader: do not leak a part's build-environment <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2472>
[11:04] <pedronis> pstolowski: +1 on 5962 but with some requests of tweaks
[11:06] <pstolowski> pedronis: yay! thank you, will do
[11:07]  * pstolowski runs malta-related errand
[11:07]  * pstolowski runs malta-related errand (not a haircut)
[11:11]  * marcustomlinson loves pstolowski's beautiful long locks
[11:12] <pstolowski> marcustomlinson: lol
[11:15] <mup> PR snapd#6507 opened: cmd/snap-confine: allow writes to /var/lib/** <Created by zyga> <https://github.com/snapcore/snapd/pull/6507>
[11:16] <zyga> morphis: https://github.com/snapcore/snapd/pull/6507
[11:16] <mup> PR #6507: cmd/snap-confine: allow writes to /var/lib/** <Created by zyga> <https://github.com/snapcore/snapd/pull/6507>
[11:17] <zyga> popey: nice talk at fosdem!
[11:17] <zyga> popey: I asked one question https://www.youtube.com/watch?v=UAEdSjou3c4&lc=UgwCtVt0IXrkcLD1jLB4AaABAg
[11:17] <popey> thanks
[11:18] <zyga> mvo, pedronis ^ (jenkins regression fix)
[11:21] <popey> zyga: answered there, but the question came from an endless developer who wanted to see how multiple versions of the same app was presented to the desktop (desktop files) compared to flatpak
[11:21] <zyga> ah
[11:21] <zyga> neat
[11:21] <zyga> I think making desktop icons smarter would be great
[11:22] <zyga> to add some disambigution in gnome shell
[11:22] <zyga> (classic, snap, flatpak, etc)
[11:22] <zyga> or version informationi
[11:22] <zyga> my wonky "i" make me type like some fake italian
[11:32] <mvo> zyga: thank you
[11:45] <mup> PR snapd#6505 closed: packaging: avoid race in snapd.postinst <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/6505>
[11:47] <pedronis> Chipaca: spread tests for catalog are now failing on that PR :/
[11:47] <pedronis> not too unexpectedly
[11:47] <Chipaca> pedronis: :-)
[11:49] <pedronis> Chipaca: one test doing: test -s /var/cache/snapd/commands.db
[11:54] <ijohnson> hey zyga, mind pointing me to the spot in snap-confine where nvidia specific udev stuff is run? I only see udev setup from https://github.com/snapcore/snapd/blob/master/cmd/snap-confine/snap-confine.c#L354 AFAICT which won't run if we turn off udev tagging from snapd
[11:54] <zyga> ijohnson: hey
[11:54] <zyga> one sec
[11:54] <ijohnson> thanks
[11:55] <zyga> ijohnson: https://github.com/snapcore/snapd/blob/master/cmd/snap-confine/udev-support.c#L237
[11:56] <zyga> but looking at this now I think you are good
[11:56] <ijohnson> okay, cool. I tested this with my machine with nvidia on it, and still didn't see any udev setup with my PR so it looks good in empirical testing too
[11:56] <ijohnson> I'll add a comment to the PR
[12:11] <mup> PR snapd#6508 opened: packaging: avoid race in snapd.postinst <Created by mvo5> <https://github.com/snapcore/snapd/pull/6508>
[12:22] <zyga> pstolowski: hey
[12:22] <zyga> do you have a sec for a quick discussion
[12:22] <zyga> pstolowski: the context is this bug https://bugs.launchpad.net/snapd/+bug/1815722
[12:22] <mup> Bug #1815722: Namespaces for failed snap installations are not discarded <snapd:In Progress by zyga> <https://launchpad.net/bugs/1815722>
[12:22] <zyga> pstolowski: I'm trying to figure out where the code for this should live
[12:22] <zyga> pedronis, Chipaca: ^ CC
[12:22] <zyga> when the install hook fails we will undo the whole change
[12:23] <zyga> but it's unclear which task should be responsible for discarding the mount namespace
[12:23] <zyga> should we encode that as an undo task for the install hook
[12:23] <zyga> should we encode that as special logic inside the implementation of that specific task (in the do path, when it fails just discard)
[12:23] <zyga> or should we do something entirely different
[12:24] <pedronis> zyga: can this happen in some form on a refresh as well?
[12:25] <zyga> pedronis: install hooks don't run then, if any other hook fails we just restore the old revision and that's handled correctly
[12:25] <zyga> this feels like a special case
[12:25] <pedronis> is it handled correctly also if layout related stuff fails?
[12:27] <zyga> pedronis: it depends on what you mean by fine, if layout application fails we fail loudly (commands and hooks cannot run), then proceed to undo; any changes made to the mount namespace remain intact and are accounted for
[12:28] <zyga> the subsequent call to snap-update-ns will attempt to change the mount namespace in accordance to what is the new desired format
[12:28] <zyga> but in either case we never discard the namespace in such situation
[12:28] <pstolowski> re
[12:39] <morphis> zyga: thanks
[12:41] <zyga> any advice?
[12:45] <pedronis> zyga: we have the same problem on install if starting services fails, no?
[12:46] <zyga> pedronis: anything that causes a freshly installed snap to fail to install
[12:46] <pedronis> and runs something potentially
[12:46] <zyga> yeah,
[12:46] <pedronis> just saying that I don't think concetrating on install-hook
[12:46] <pedronis> makes sense
[12:48] <pedronis> zyga: anyway it seems something to do in the undo of link snap if the situation is a first install attempt?
[12:48] <zyga> yes, I was thinking about anything that is special to first-install
[12:48] <zyga> I agree
[12:48] <zyga> perhaps link is the right place
[12:48] <zyga> it's also annoying because this is cross manager
[12:48] <zyga> but I think we have handlers for that
[12:49] <pedronis> zyga: well , we need to merge bits of tasks of ifacestate and snapstate anyway
[12:51] <zyga> yes but hopefully not for this fix :)
[12:51] <pedronis> zyga: where do we discard the namespace for remove ?
[12:51] <zyga> there's a dedicated task for that IIRC
[12:52] <zyga> let me look
[12:54] <pedronis> zyga: it's done in discard-snap
[12:54] <pedronis> so that's already snapstate afaict
[12:55] <pedronis> zyga: pstolowski: my impression is basically there might be more bits done in discard-snap
[12:56] <pedronis> that we also need to do in undo link-snap
[12:56] <pedronis> if it's a first install
[12:57] <pstolowski> reading the backlog
[12:57] <pedronis> pstolowski: do we need to remove the config as well for example?
[12:59] <pedronis> pstolowski: zyga: basically compare  the code under https://github.com/snapcore/snapd/blob/master/overlord/snapstate/handlers.go#L1387 vs https://github.com/snapcore/snapd/blob/master/overlord/snapstate/handlers.go#L1084
[12:59] <pedronis> what does make sense for both
[13:00] <pedronis> do the 2nd seems to have more len(seq) == 1 sections
[13:04] <pstolowski> pedronis: indeed, we have a problem with config too, we should remove it in undo of link-snap
[13:04] <pstolowski> pedronis, zyga and i agree, undo of link snap is a good place for the namespace cleanup
[13:05] <pedronis> pstolowski: there is a call to config.RestoreRevisionConfig with an XXX later there
[13:06] <pedronis> pstolowski: not sure what it does? are we missing a test?
[13:06] <zyga> pedronis: indeed, thank you, I'm looking at this now
[13:09] <pstolowski> cachio: i pushed fixes to hotplug-nested-vm, it's now passing for me on 16.04. old qemu is not a problem.
[13:12] <cachio> pstolowski, nice, I'll give a run again
[13:12] <cachio> thanks for fixing it
[13:15] <pstolowski> pedronis: RestoreRevisionConfig would restore configuration of previous revision of the snap, but that's not going to help on undo of new install, we will leave config behind. yes i think undo is missing a test for this
[13:16] <pedronis> pstolowski: yea, that code looks broken, like the condition should be the reverse?
[13:16] <pedronis> != 1
[13:21] <Chipaca> pstolowski: pedronis: that's one of the places I added an XXX i think
[13:21] <Chipaca> as the logic didn't seem to make much sense
[13:21] <pedronis> Chipaca: yea, seems there is a bug lurking there and missing tests
[13:22]  * zyga breaks for a walk with the dog
[13:22] <pstolowski> pedronis: yes i think you're right
[13:23] <Chipaca> https://github.com/snapcore/snapd/blob/master/overlord/snapstate/handlers.go#L1125 <- that's the one i meant
[13:23] <pstolowski> pedronis: i can add it to my todo list and address soon
[13:23] <pedronis> pstolowski: thx
[13:24] <Chipaca> pedronis: it's a different one -- with possibly the same logic
[13:24]  * Chipaca ⇝ lunch
[13:25] <pedronis> Chipaca: heh
[13:40] <popey> degville: fyi, I edited "Installing snap on kde neon" and replaced your dark theme screenshots with default theme ones :)
[13:41] <popey> (at the kind and friendly request of upstream) :)
[13:41] <degville> popey: thanks! you're absolutely right... I've used it so long I forget it's not default.
[13:42] <popey> I think the words were "Which nerd uses the dark theme?" :)
[13:42] <popey> (I do too)
[13:42] <popey> I <3 KDE Neon
[13:43] <degville> ahaha... yeah. Arc Dark and Papirus ftw.
[13:44] <mup> PR snapd#6509 opened: overlord/snapstate: discard mount namespace when undoing 1st link snap <Created by zyga> <https://github.com/snapcore/snapd/pull/6509>
[13:45] <zyga> pedronis: ^ thank you for the idea again
[13:56] <Chipaca> hmmm
[13:56]  * Chipaca hmms
[13:57] <t1mp> How do I downgrade to snapcraft2 with snap? On https://snapcraft.io/snapcraft I can only find 3.x versions
[13:57] <pstolowski> zyga a question there
[14:06] <Chipaca> t1mp: snapcraft3 ships 2 as well
[14:06] <Chipaca> t1mp: if your snapcraft.yaml doesn't have a base, you get 2
[14:07] <t1mp> I use core18 as base. But snapcraft3 broke some things (probably I can fix them, but it takes more time which I don't have right now)
[14:09] <Chipaca> t1mp: which things? (maybe the fix is easy)
[14:09] <Chipaca> sergiusens: ^
[14:18] <t1mp> some things were easy, like the version should be "2" instead of 2 (int).
[14:18] <popey> version is indeed a string
[14:18] <t1mp> I worked around the cloud parts that I couldn't use any more
[14:18] <t1mp> and now, I have this:
[14:18] <t1mp> version-script: python3 ../qmenta_gui/python/qmenta/gui/version.py
[14:18] <popey> it was a bug that it allowed int
[14:18] <t1mp> so when I build the package now, that cannot be executed. I don't have those files in the vm that multipass created. I guess with snapcraft2 it was not executed in a vm.
[14:20] <popey> one thing you might want to try is "snapcraft --shell-after" which drops you into the vm post-build, which allows you to debug these things
[14:20] <popey> I have started using adopt-info: partname, and using "snapcraftctl set-version '$(something)'" to set the version in override-build: | section in the part
[14:20] <popey> (there's some examples of this in the snapcrafters repo)
[14:23] <mvo> zyga: https://github.com/snapcore/snapd/blob/master/sanity/apparmor_lxd.go#L33 <- this is what we have currently for testing inside containers
[14:49] <t1mp> debugging is a bit of a hassle, because every time I rebuild the snap it takes 10 minutes or so
[14:50] <popey> the handy thing about debugging with --shell-after, is you get dropped in the VM, and can cd project/ and then run snapcraft *in* the VM
[14:50] <popey> which will save time
[14:50] <t1mp> I'll try that in the next re-run :)
[14:55] <zyga> mvo: right, we would have to have some more logic to check if we are running without pid namespace
[14:55] <zyga> mvo: er, uid namespace
[14:55] <zyga> mvo: this is not sufficient (separate issue)
[14:55] <t1mp> hmm.. The snap I create locally works as expected. But the snap created from the same source, on jenkins, has a bug.
[15:00] <t1mp> I can try to install snapcraft 3 there. But then I have Google VM -> Docker Ubuntu 18.04, with snapcraft 3 snap package, creating a qemu (?) vm to build the snap package..?
[15:02] <t1mp> if I am already in an ubuntu 18.04 docker image, do I really need a VM for creating snap packages?
[15:02] <Chipaca> t1mp: no, if you're in docker, you can tell snapcraft to chill about the vm
[15:02] <popey> there's a way you can do that, yes.
[15:02] <Chipaca> if you're already in an ad-hoc vm that is
[15:02] <t1mp> yeah, I'm in a container
[15:03]  * Chipaca nods
[15:03] <Chipaca> it wasn't --chill-about-the-vm-already
[15:03] <popey> I can't for the life of me remember what it is
[15:03] <t1mp> the container is only created to run tests and build the package
[15:03] <popey> sergiusens: ^ what's the "let me do dangerous things" parameter for snapcraft? (to build on a host)
[15:05] <Chipaca> t1mp: popey: --destructive-mode
[15:05]  * Chipaca hugs grep
[15:05] <popey> 💣
[15:06] <Chipaca> fwiw: grep -hor --include '2019*' 'snapcraft.*--[a-z_-]*' ~/.config/hexchat/logs/
[15:11] <t1mp> Chipaca: I guess that is not printed on purpose when I do --help?
[15:11] <t1mp> Chipaca: anyway, thanks :)
[15:11] <Chipaca> t1mp: your guess is as good as mine
[15:12] <Chipaca> t1mp: it's probably supposed to only be internal
[15:12] <popey> bugs.launchpad.net/snapcraft :)
[15:12] <Chipaca> t1mp: as it's what snapcraft uses to launch snapcraft inside the vm, afaik
[15:12] <t1mp> ah. snapcraft help build shows it.
[15:27] <mup> PR snapd#6510 opened: tests: stop catalog-update test for now <Created by mvo5> <https://github.com/snapcore/snapd/pull/6510>
[15:29] <sergiusens> Chipaca: popey t1mp it is printed with `snapcraft <pull|build|stage|prime|snap> --help`
[15:30] <sergiusens> snap is the default param for snapcraft, I wish we did not have a default, life would be simpler
[15:32] <sergiusens> popey: when inside the VM, no need to `cd project`, you can run snapcraft from whatever PWD
[15:32] <popey> TIL
[15:46] <jdstrand> mvo: I commented on https://github.com/snapcore/snapd/pull/6508. I think the access() is unneeded...
[15:46] <mup> PR #6508: packaging: avoid race in snapd.postinst <⚠ Critical> <Created by mvo5> <https://github.com/snapcore/snapd/pull/6508>
[15:47] <t1mp> is snapcraft using qemu instead of lxd to be compatible with windows/macos?
[15:51] <sparkieg`> t1mp: essentially, yes
[15:54] <zyga> for all the meetings https://twitter.com/garabatokid/status/1016240967702204416 ;-)
[15:55]  * cachio lunch
[15:58] <jdstrand> mvo: oh, I misread. sorry, nm
[15:59] <mvo> jdstrand: thank you
[15:59] <mvo> jdstrand: maybe I should have added a comment there or rewrite the code, its definitely not obivous (which is why this is buggy :(
[16:00] <jdstrand> mvo: maybe an added comment "If after the rewrite we are still not ok, die" or something
[16:01] <jdstrand> mvo: but pr approved as is
[16:07] <Chipaca> jdstrand: very low priority ping about icdiff
[16:09] <mvo> jdstrand: \o/ thanks, I will add the chnage
[16:15] <pedronis> mvo: why the mount-support.c change there ?
[16:16] <pedronis> I mean I mentioned that issue in another PR
[16:16] <pedronis> but why mixing it with that?
[16:19] <mvo> pedronis: the upgrade test from 2.15.2 fails without it
[16:19] <mvo> pedronis: it will die because it claims it can't find ubuntu-core
[16:19] <mvo> pedronis: 2.15.2ubuntu1 is still using ubuntu-core as base
[16:20] <jdstrand> Chipaca: you wouldn't know it because I haven't been transparent, but I've been working hard on a review-tools update to support that
[16:21] <jdstrand> they had some old janky code that was brittle and I rewrote it. should be committing today, at which point I'll add the aliases
[16:21] <jdstrand> (and ask for a store pull, etc)
[16:21] <pedronis> mvo: can put it in a comment somewhere in it
[16:23] <mvo> pedronis: sure, adding it now
[16:25] <pstolowski> zyga: i see that repo.ResolveDisconnect is only used by api - disconnect; i'm going to change it to return []*Connection instead of []*ConnRef to remove the need for second call to repo. do you see any problem with that?
[16:25] <mvo> pedronis: updated the comment
[16:26] <zyga> pstolowski: I honestly don't know, is it tricky?
[16:26] <zyga> looking at the code now
[16:26] <pstolowski> zyga: no, not really
[16:27] <pstolowski> zyga: and it's easy to get ConnRef from Connection, if needs be
[16:27] <pstolowski> zyga: this will just let avoid ping-pong with the repo for disconnect case
[16:27] <zyga> yeah, looks ok
[16:27] <zyga> I wonder if connref should grow something
[16:28] <zyga> to detect unexpected changes colliding
[16:28] <pstolowski> zyga: connref? can you elabortae
[16:28] <pstolowski> *elaborate
[16:29] <zyga> a cookie derrived from the number of changes to the repo
[16:29] <zyga> so you can get a conn ref in one place
[16:29] <zyga> pass it elsewhere
[16:29] <zyga> and use it if it is still "fresh'
[16:29] <zyga> your desire to change the return type, is it to avoid a race?
[16:29] <pstolowski> ah, a "smart pointer" ;)
[16:32] <pstolowski> zyga: yes, to avoid repo.Connection() on a connref returned by repo.ResolveDisconnect(), that may no longer be there
[16:33] <pstolowski> on the other hand, i'm not sure it's worth fixing
[16:34] <pstolowski> it's basically for the cases where you don't fully specify plugs or slots, and let resolve compute the list
[16:39] <pekkari> hI #snappy, I hit an error with juju snap, cannot create user data directory, Permission denied, can anybody help?
[16:39] <zyga> pekkari: can you be more specific please
[16:40] <pekkari> I see -> cannot create user data directory: /path_to_my_home_folder/snap/juju/6362: Permission denied when I execute anything in juju
[16:40] <zyga> what is your home folder path?
[16:42] <pekkari> for some reason, it's decided to be under /var/lib/home/my_user, I cannot tell who was having that great idea :$
[16:42] <zyga> pekkari: that location is not supported by snapd
[16:43] <pekkari> nevertheless the folder ~/snap/juju/6362 exists
[16:43] <pekkari> hummm... unfortunately I cannot decide on the folder path zyga, customer stuff
[16:43] <mup> PR snapd#6511 opened: daemon/api: fix error case for disconnect conflict <Created by stolowski> <https://github.com/snapcore/snapd/pull/6511>
[16:43] <zyga> pekkari: we cannot support arbitrary folder names either
[16:44] <pekkari> I'll need to try to bind mount the folder on /home/my_user
[16:48] <zyga> pekkari: I'm curious what was the motivation to store the home folder in /var/lib/ rather than in /home, is it a service (e.g. jenkins) or just some specific customer convention?
[16:48] <pekkari> zyga: customer convention, no service, but shared account for a team
[16:48] <Chipaca> jdstrand: thanks for the update! no worries about the rest of it
[16:49] <pekkari> have you any experience using mount --move zyga? seems to be the way to go as long as bind still resolves the /var/lib/home/blah folder
[16:49] <pekkari> I just never happen to use it
[16:50] <zyga> pekkari: using it in which sense? yeah it works and snapd used to use it internally for one specific case
[16:50] <zyga> pekkari: but I'm not using the feature myself much
[16:51] <pekkari> well, use it to solve the path problem without screwing the home folder where I have the sensitive info
[16:51] <zyga> pekkari: I don't understand, can you explain what you mean?
[16:52] <pekkari> I just want to know it's a safe operation, ta
[16:52] <zyga> perhaps? I don't know for sure
[16:52] <zyga> on a live production system?
[16:53] <pekkari> no, testing env, but still I have an fce workspace there that I'm taking backups just in case
[17:01] <pekkari> grrrr. shared mount...
[17:12] <Pharaoh_Atem> zyga: does that mean that home directories on Silverblue don't work either?
[17:12] <Pharaoh_Atem> they're in /var/home rather than /home
[17:15]  * cachio afk
[17:55] <zyga> Pharaoh_Atem: correct
[17:55] <zyga> Pharaoh_Atem: we cannot support all the random locations people come up with
[17:55] <Pharaoh_Atem> well, it's symlinked to /home
[17:55] <zyga> Pharaoh_Atem: we can add support for /var/home perhaps but not for every single random combination :)
[17:55] <zyga> Pharaoh_Atem: with mount namespaces, symlinks don't matter sadly
[17:56] <Pharaoh_Atem> well /var/home is used by Atomic Fedora and SUSE MicroOS
[17:56] <zyga> well, they should have just used home
[17:56] <zyga> it's just a location after all
[17:56] <zyga> they can mount there
[17:56] <Pharaoh_Atem> it's when there isn't separate partitions
[17:56] <zyga> we can support /var/home eventually
[17:56] <zyga> just weird
[17:56] <zyga> I don't see how this is related to partitions
[17:57] <Pharaoh_Atem> the standard layout has two partitions, one managed by OSTree and another umanaged
[17:57] <Pharaoh_Atem> *unmanaged
[17:57] <Pharaoh_Atem> they've moved all the written stuff to there by default
[17:57]  * Pharaoh_Atem shrugs
[17:57] <zyga> if your ostree has /home you can mount there, right?
[17:57] <Pharaoh_Atem> yes
[17:57] <Pharaoh_Atem> and people do if they have a separate /home partition
[17:57] <zyga> so what's the need for /var/home?
[17:58] <Pharaoh_Atem> if when they don't have it
[17:58] <zyga> you don't need a separate home partition
[17:58] <Pharaoh_Atem> usually it'd be / if you don't have a separate /home
[17:58] <zyga> you can bind mount /home or even each /home/whatever
[17:58] <zyga> you can mount subtrees
[17:58] <Pharaoh_Atem> I know
[17:58] <Pharaoh_Atem> but they do it by symlink right now
[17:59] <zyga> so /home/foo is a symlink to /var/home/foo?
[18:00] <zyga> I guess snapd could learn a trick to make /var/home/$LOGNAME a way to access user's (whatever) home directory
[18:00] <zyga> but dragons apply :)
[18:02] <zyga> anyway
[18:02] <zyga> I meant to thank you for the release coordination the other day
[18:02] <zyga> I'm sleepy as hell now
[18:03] <zyga> I was meant to say I'll EOD now
[18:10] <mup> PR snapd#6510 closed: tests: stop catalog-update test for now <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6510>
[18:35] <Chipaca> niemeyer: could spread error if you told it to run on a system that doesn't exist?
[18:40] <niemeyer> Chipaca: Hmm.. doesn't it?
[18:40] <Chipaca> niemeyer: not with globs
[18:41] <Chipaca> niemeyer: just spotted a test that had systems: [ubuntu-16-*, ubuntu-18-*]
[18:41] <Chipaca> (neither of those matches anything)
[18:41] <niemeyer> Chipaca: Yeah, it's definitely plausible since we have the full real set in memory.. we ought to do that.
[18:41] <niemeyer> Chipaca: ah, wait
[18:42] <niemeyer> Chipaca: I misunderstood.. I thought you were asking for it to run on the cmdline and didn't match anything
[18:42]  * Chipaca feels misunderstood
[18:42] <niemeyer> Chipaca: Maybe it should warn in those cases you mention
[18:43] <niemeyer> Chipaca: Not sure if error as it could make a refactoring pretty boring
[18:43] <niemeyer> Chipaca: Or maybe let's just do it and see if people get mad at us
[18:44] <niemeyer> Erroring on the safe side can't go too wrong
[18:44] <Chipaca> niemeyer: a warning would've helped indeed
[18:45] <Chipaca> niemeyer: and yeah there're probably people depending on the behaviour :-)
[19:03] <pedronis> Chipaca: see mvo find on the PR, we need to use int64 and corresponding rand helper
[19:15]  * Chipaca looks
[19:15] <Chipaca> pesky puny computers
[19:16] <Chipaca> I'm going to EOD now before I break something important
[19:16] <Chipaca> o/
[20:42] <mup> PR snapd#6507 closed: cmd/snap-confine: allow writes to /var/lib/** <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6507>
[20:43] <mup> PR snapd#6509 closed: overlord/snapstate: discard mount namespace when undoing 1st link snap <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6509>
[20:45] <mup> PR snapcraft#2471 closed: test: test-beta <do-not-merge-yet> <Created by sergiusens> <Closed by sergiusens> <https://github.com/snapcore/snapcraft/pull/2471>
[20:52] <pedronis> Chipaca: surprisingly butn not, catalog-update passes also with /names down
[20:52] <pedronis> so we shouldn't turn it off
[20:52] <Chipaca> pedronis: yes I noticed that locally, assumed mvo saw it fail
[20:53] <pedronis> Chipaca: we should push to turn it on
[20:53] <pedronis> again
[20:54] <Chipaca> pedronis: push who?
[20:54] <pedronis> Chipaca: you were pushing stuff recently? can you otherwise I can try to do it
[20:55] <mvo> how come this works when /names is down? just curoious haven't looked at the details of the test
[20:55] <pedronis> mvo: it doesn't really check that we got date, it checks that we tried
[20:55] <Chipaca> mvo: because all it does is check that catalog refreshes was attempted
[20:55] <Chipaca> yeah
[20:55] <pedronis> data
[20:55] <Chipaca> "at least you tried" golden star
[20:56] <pedronis> the other two do check data
[20:56] <pedronis> apt-hooks
[20:56] <pedronis> and snap-advise-command (except that it was not run)
[20:59] <pedronis> Chipaca: are you working on turning it back on? otherwise I'm close myself
[21:09] <pedronis> Chipaca: mvo: pushed to turn catalog-refresh back on
[21:09] <pedronis> *catalog-update
[21:11] <mvo> pedronis: ta
[21:12] <mup> PR snapd#6508 closed: packaging: avoid race in snapd.postinst <⚠ Critical> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6508>
[21:15] <Chipaca> pedronis: sorry, was afk for a bit
[21:16] <pedronis> Chipaca: it's ok
[21:16] <pedronis> we should all be eoded really
[21:18] <mvo> pedronis: yes, I need to go
[21:18] <mvo> 5606 should be squashed
[21:18] <mvo> I will merge tomorrow into 2.37
[21:18]  * mvo waves