=== zigford_ is now known as zigford [06:31] morning [06:31] wow, forum got updated? [06:31] hey mborzecki [06:31] mvo: hey [06:32] PR snapd#7752 closed: tests: enable degraded test on arch linux after latest image updates [06:34] mvo: https://github.com/snapcore/snapd/pull/7751 simple one, could land easily [06:34] PR #7751: cmd: fix the get command help message [06:36] mborzecki: sure [06:39] PR snapd#7751 closed: cmd: fix the get command help message [07:03] PR snapd#7753 opened: po: sync translations from launchpad [07:30] PR snapd#7754 opened: release: 2.42.2 === pstolowski|afk is now known as pstolowski [08:03] morning [08:50] pstolowski: hey [08:50] o/ [08:50] PR snapd#7755 opened: cmd/snap-failure: fallback to snap from core, extend tests [08:50] mvo: ^^ [08:52] mborzecki: nice [09:22] can someone please sanity check 7754, should be super simple [09:23] single review for this is enough [09:36] PR snapd#7713 closed: seed: Core 20 seeds channel overrides support for grade dangerous [09:42] PR snapd#7754 closed: release: 2.42.2 [10:11] zyga: when you're around https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1852635 looks to me like the default mount options changed for the virtual filesystems, maybe some systemd updated triggered that? [10:11] Bug #1852635: mount-ns test failing with latest bionic updates [10:15] mborzecki, zyga : i'm on it [10:16] 2 hrs in, almost finished [10:16] a couple of changes affecting it [10:18] zyga: btw, i'd drop the debug section (for fname in ./*.deterministic.txt...) from this test.. it produces walls of text which is not helping. actual diff that failed is most useful to understand what changes [10:18] *changed [11:39] good morning [11:39] sorry for starting late, I still cannot sleep at night [11:39] mborzecki: looking [11:40] pstolowski: that debug section is useful when there are larger changes and the diff is not something you want to work with [11:41] pstolowski: perhaps put it behind an environment variable test? [11:50] zyga: good idea [12:02] pstolowski: I released 2.42.1 to tw yesterday, did you update to it? [12:05] zyga: yes, i'm on it [12:05] why? [12:08] good, just wanted to know if you see anything wrong [12:08] it worked ok for me [12:15] zyga: silly question, where can i see the apparmor profile that is actually loaded? e.g. the profile in /var says 'include abstractions/foo', where can i see the pre-processed(?) one? [12:16] Chipaca: those are in /etc/apparmor.d [12:17] zyga: I know, which /etc tho [12:17] zyga: :) [12:17] i have four of 'em [12:17] outer /etc, core /etc (twice), and core18 /etc [12:17] and they're different in important ways :) [12:17] Chipaca: in the one seen when you log in [12:17] Chipaca: if you are talking about core that's the /etc from the boot base snap [12:18] Chipaca: on classic that's the classic /etc [12:18] Chipaca: sorry, I didn't understand your question initially [12:18] mvo: do you sleep normally now? [12:18] so, say, a core18 snap will use abstractions/python from the host? [12:18] yes [12:18] Chipaca: that's correct [12:18] that's bad and wrong [12:18] Chipaca: as in, that's what happens, your understanding is correct [12:18] indeed, it can cause syntax from apparmor 3 to interfere, for example [12:18] or, we need to retro-tweak it :) [12:19] Chipaca: what are you seeing? [12:19] it means a core18-based python snap on 1604 might not work [12:19] zyga: i'm not, but somebody else is seeing an issue [12:19] where it doesn't load a .so [12:19] https://forum.snapcraft.io/t/ubuntu-16-04-host-denies-core18-snap-python-3-6-library-import/14247 [12:20] now, i'm on 1604 and i can load that .so without issue, but my system might be "special" [12:20] so i dunno [12:20] mvo, hey [12:20] mvo, sru almost ready but 1 test failing on bionic [12:20] https://paste.ubuntu.com/p/dKrJRZb7Jb/ [12:20] Chipaca: one more reason why sharing /etc was a mistake [12:20] and we need to work towards undoing that [12:21] mvo, the test is snap-seccomp-syscalls [12:22] mvo, I remmeber we fixed it weeks ago but not sure if it is something we need to address [12:23] cachio: yeah, this is fixed in 2.42.2 I think its fine to ignore this failure [12:23] mvo, perfect [12:23] cachio: 2.42.2 will not need a sru I think, the re-exec is fine, this is only important for lxd-support [12:23] PR snapd#7749 closed: tests: restart the snapd service in the snapd-failover test [12:23] mvo, ok [12:23] makes sense [12:44] PR snapd#7756 opened: tests: update mount-ns test for bionic image changes [12:44] cachio, zyga ^ [12:44] ack [12:44] pstolowski, nice, I'll take a look [12:45] pstolowski: what is the test-bionic image? is that the image we are going to upgrade to? [12:46] pstolowski: do we know which packages are responsible for those changes? [12:47] reading the diff I would love for the .expected.txt files to use delta encoding, so all numbers would be like +1 instead of 12 (even after renumbering) [12:47] it would remove most of the diff [12:47] pstolowski: I agree with mborzecki that we should know what's the origin of the change [12:47] but other than that +1 [12:48] zyga. mborzecki i don't know what package is the origin of them [12:48] zyga: the image name will probably change, i will update once cachio has it [12:48] cachio: is there a manifest of changes to the bionic image? [12:49] zyga, I attached that to the bug [12:49] let me see [12:49] thanks, let me find it [12:49] Chipaca: hey, fyi, apparmor_parser -p /path/to/profile will show the profile with all the includes in it === jdstrand_ is now known as jdstrand [12:50] 237-3ubuntu10.31 -> 237-3ubuntu10.29 ? [12:50] cachio: can you attach an unified diff before/after [12:50] it looks like the diff is after -> before now [12:50] zyga, https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1852635/+attachment/5305454/+files/bionic-diff.txt [12:51] Bug #1852635: mount-ns test failing with latest bionic updates [12:51] does https://packages.ubuntu.com/ 500 for you too? [12:52] zyga, sure, on sec [12:52] thanks [12:52] mborzecki: no [12:53] zyga, https://launchpadlibrarian.net/452319755/bionic-diff2.txt [12:54] unified :) [12:54] but ok [12:54] unified diffs are easier to read === ricab is now known as ricab|lunch [12:54] zyga, ahh, ok ,one sec [12:55] zyga, https://launchpadlibrarian.net/452319957/bionic-diff-unified.txt [12:56] pstolowski: ^ given this can you check if the systemd changelog explains what we are seeing please? [12:57] pstolowski: in addition, can you please add a "Fixes: " line to the commit message, [12:57] it helps with tracking things [12:57] zyga: sure [12:58] reviewed now [12:59] jdstrand: dunno if you saw that forum topic, do we need to tweak /etc/apparmor.d/abstractions/python so people can use core18-based python snaps? [12:59] jdstrand: using the python from the base i mean [12:59] mvo: I left some initial comments and a question in 7746 [13:02] Chipaca: I haven't yet, but hope to start my misc policy updates today and can add that to the list of investigations [13:02] jdstrand: k [13:03] Chipaca: thanks! [13:16] zyga: systemd changes looks completely innocent and unrelated, must be something else [13:17] i looked at debian changelog and patches listed there [13:17] pstolowski: would you mind looking at what the cause may be for some more time? I would like to at least know that it's not a mistake that the changes exist [13:17] yeah i will [13:19] pstolowski: perhaps ask foundations for ideas [13:19] pstolowski: xnox may know something about this [13:24] can i please have context summary? also possibly you want rbalint [13:25] xnox: sure. we have a pedantic test that checks mount tables and namespaces. it picked changes after latest bionic upgrade, and we'd like to understand if the changes are legit [13:25] xnox: https://github.com/snapcore/snapd/pull/7756 - see PR description for summary of the changes [13:25] PR #7756: tests: update mount-ns test for bionic image changes [13:30] it looks harmless, yet it odd [13:30] it looks harmless, yet it is odd [13:30] there might be CPC image changes too (as in the base image itself, rather than an sru of something) [13:32] aha [13:33] cachio: do you know what versions of images we should be comparing? [13:34] brb [13:37] degville: hey, your CentOS 8 instructions for installing EPEL are can be simplified to just `dnf install epel-release` [13:37] for this page, I mean: https://snapcraft.io/docs/installing-snap-on-centos [13:37] `epel-release` is in the default repos for CentOS 8 [13:38] Eighth_Doctor: ah, thank you! I'll update them! [13:38] pstolowski, those images are using the same base image [13:38] uhm [13:39] fir older one was created by updating and upgrading one month ago [13:39] the newer it was created following the same procedure 1 week ago [13:39] cachio: pstolowski: zyga: nothing obvious in https://changelogs.ubuntu.com/changelogs/pool/main/s/systemd/systemd_237-3ubuntu10.31/changelog [13:40] xnox: we also have the list of packages that got updated - https://launchpadlibrarian.net/452319957/bionic-diff-unified.txt - anything worth checking? [13:40] mborzecki: yes, i checked that too, plus patches in the source package [13:42] mborzecki: as long as changelos is thruthful about what patches were applied, they look innocent [13:47] if it is the cloud images [13:47] could it be something that is not a package that is changed there [13:52] cachio says the base image is the same. that means changes can only come from updated packages afaiu? [13:53] that's weird [13:53] I don't mean to block you, perhaps ask mvo for advice [13:54] let's check on standup. if anything, it's blocking bionic update for our tests [13:57] zyga: what is this about? [13:57] mvo: standup [13:57] let's chat there? [13:58] zyga: 1min, need to prepare a cup of tea === ricab|lunch is now known as ricab [14:36] zyga: ijohnson: irc or meet? [14:36] ijohnson: so I have an idea where we could optimize away snap-update-ns for _some_ snaps [14:36] let's do it here [14:36] my network is busy today [14:36] sure works for me [14:36] part of the work will involve snap-update-ns working on slightly different basis [14:37] and if we can statically detect from snapd that the fstab profile to apply doesn't need sophistication (more on what that means in a moment) [14:37] we could implement that in snap-confine like before [14:37] sophistication is specifically needing to create mimics [14:37] without mimics input profile is the output profile [14:37] and we can just execute mounts in order and be done with it [14:38] we could detect if mimics are needed with a simulation approach, that plans what it would take to apply a mount profile we generate for a snap [14:38] right now we don't know until runtime [14:38] last week I was experimenting a little, extending the Assumptions struct with knowledge about the base snap [14:38] so that we could simulate application and know if mimic is needed or not [14:39] it's a bit hand wave-y at the moment *but* we could optimize away snap-update-ns in many cases, I believe [14:39] related question: when we preserve a mount ns, does that persist across boots, or is it just valid for the current boot? I think it's not persistent since it's in /run, correct? [14:39] ijohnson: no, it's just a memory construct [14:39] right [14:39] it's only in kernel memory [14:39] ijohnson: even outside of /run [14:40] oh right cause it's just a fd got it [14:40] it's a nsfs filesystem object we are saving [14:41] do we create mimics for content interfaces as well as layouts? I think yes? [14:41] ijohnson: yes [14:41] ijohnson: there's no difference between them [14:42] hmm I guess I'm not quite following which information we're missing when snap-confine runs that currently we have to wait until runtime for [14:42] ijohnson: let's have a call :") [14:42] it could be faster than me typing [14:43] haha ok [14:43] mborzecki did you want to join as well? [14:43] https://meet.google.com/nbn-avuw-zwh?authuser=1 [15:01] * zyga breaks for lunch [15:03] * cachio lunch [15:10] ijohnson, FYI ... https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/mangling.py#L28 [15:10] ... of course [15:10] not sure that is still used though ... but it used to re-write all python shebangs in all python scripts during snap build [15:11] (that is why i suggested the change in the forum) [15:12] well I suggest snapcraft not do that in the name of performance, but perhaps there's a good reason why snapcraft used to do that [15:12] yes, because you cant be sure core/core18 has the same interpreter as the snap [15:12] (someone might pull python 3.9 from a PPA r whatnot) [15:14] but yeah, the performance impact is quite significant ... i noticed that with my little robot arm when i tried to snap some movement scripts (shell wrappers that then call a python script ... the moves got seriously slower) [15:15] less than half the speed in the end ... [15:15] ogra: but even if you pull python 3.9 from a PPA into your snap, you would know that you want your python script to use python 3.9, so you would set that up? [15:15] i.e. $SNAP/usr/bin/python3.9 or some such [15:16] sure, if i only have one .py script and have full control over the upstream source ... [15:16] do any of your python scripts need to load C libraries? [15:16] if you pull in multiple scripts from external parts/sources it gets more tricky [15:16] err just dynamic libraries in general ? [15:17] it might make sense to keep this shebang rewrite but make it configurable ... [15:17] sure I could see making it configurable [15:17] so you can actually give the target shebang in a snapcraft.yaml option [15:17] and let it default to /usr/bin/env ... [15:18] but thats for the snapcraft team [15:19] why is the forum so slow now, I know it was updated but it feels like it takes much longer to load just small pages [15:20] i use an electron app for the frum ... and noticed wiping all the app cache gets it back to being normally responsive [15:20] perhaps try wiping the cache for snapcraft.io in your browser ? [15:23] hmm good idea [15:23] ijohnson: missed that, went for lunch [15:24] mborzecki: np main conclusion was that I was going to do some profiling of snap-update-ns and zyga had some ideas on how to make it quicker [15:24] indeed [15:25] mborzecki: you didn't miss anything new [15:25] mborzecki: we can have another chat about the details that relate to robustness later next week [15:25] ijohnson: cool, there's a bunch of operations triggered by the themes, wonder if skipping themes snap would show a significant change [15:25] yeah there is a lot from the themes snap, so that's an interesting thing to try as well [15:25] though with themes that's only gonna get worse as we get greedy plugs :-( [15:26] ijohnson: discarding the mount ns and disconnecting the contnt interface should be enough [15:27] well it's a bit more tricky since the theme interface auto-connects and I'm testing first launch, so there's a bit of caching that the desktop-launch helper does as well that's a bit tedious to cleanup [15:27] but just removing the plug from snap.yaml is probably sufficient [15:28] ijohnson: rm -rf ~/snap/gnome-calculator make it pretty much like the first run [15:28] * pstolowski lunch [15:28] mborzecki: for the purpose of helpers, yes [15:28] yes but I still think it's a more accurate representation to remove the whole snap and try [15:28] ijohnson: you can rm snap user data and discard [15:29] ijohnson: it's really a good performance reset method, especially if you drop disk caches [15:29] ijohnson: otherwise installation may mask some of that [15:29] yes I know about all of this, my scripts will delete all of $SNAP_USER_DATA, remove the snap, discard the ns, drop all caches then reinstall and measure performance [15:30] I just think it's easier to just remove the snap to measure performance data [15:30] ijohnson: that's okay but keep the fact that installation absorbs some of first run cost [15:30] ijohnson: so you may masking part of the measurement [15:30] that's my whole point [15:30] ijohnson: yes but it's not representative [15:30] ijohnson: it depends of what the snap is looking [15:30] it is representative of a user who runs the software [15:30] er [15:30] doing [15:31] ijohnson: not quite, it's only representative of that snap alone [15:31] ijohnson: e.g. presence of install or configure hook will skew results [15:31] this is why I am doing these tests across multiple different kinds of snaps [15:31] ijohnson: this is why I emphasize it matters what happens during installation [15:31] ijohnson: to get realistic results neuter install and configure hooks (remove them entirely) [15:32] yes I understand all of what you're saying and it's all being taken into account, I am taking these things into account [15:32] great [15:33] ijohnson: looking forward to what you find, let's talk about it tomorrow [15:35] zyga: ijohnson: looking at both snaps, connecting gnome-platform shouldn't generate many 'actions' in snap-update-ns, so it's just themes really [15:36] mborzecki: one thing worth of note [15:36] and wonder if that's a dynamic cost incurred by actual layout ops, or a static cost due to just forking & running snap-update-ns [15:36] mborzecki: is the mimic cost [15:36] mborzecki: I think it's both [15:37] mborzecki: static cost is fixed and should be good to compare huge layout/content vs empty one that is there so that we fork anyway [15:37] PR snapd#7757 opened: devicestate: rename ensureSeedYaml -> ensureSeeded [15:37] mborzecki: dynamic cost largely depends on where we mount stuff, as mimic cost varies hugely [15:37] yeah, we might need to rebind lots of stuff if it hits at the wrong place :/ [15:38] yes [15:44] zyga, mborzecki: just with the measurements I already have, if I run `sudo snap run --shell gnome-calculator -c "echo"` (note the sudo so we are just setting up the per-snap ns and not the per-user ns), then when I run `snap run gnome-calculator`, the time for the first launch of snap-update-ns is 406ms vs 9ms so I really don't think that just forking/execing a go binary is adding that much [15:45] ijohnson: hmm? [15:45] ijohnson: I could be misunderstanding things [15:45] but the first invocation, as root, cached everything [15:45] ijohnson: and the second one reused that [15:45] well the second one had to still do the per-user ns [15:45] it still ran snap-update-ns [15:46] ijohnson: yes but it's running a binary that was just invoked [15:46] ijohnson: not saying it's wrong [15:46] ijohnson: it would be good to get the SNAP_CONFINE_DEBUG=yes timings [15:46] I drop all caches between these two runs [15:46] yes will provide that later, still catching up on some things this morning [15:46] ijohnson: that will show us how long it takes to mount everything [15:46] ok [15:47] btw. thinking we could strace it ;) just the mounts [15:48] PR snapd#7758 opened: devicestate: add reading of modeenv to uc20 firstboot code [15:50] ijohnson: zyga: totally unscientific: https://paste.ubuntu.com/p/txq2Z3Zj59/ [15:51] that's realistic, cost of crafting a large mimic [16:01] zyga: ijohnson: https://paste.ubuntu.com/p/STzxjc9htm/ wait4 is really high, is that the deskotp helpers? [16:01] dunno [16:01] could that be snap-confine waiting for snap-update-ns? [16:02] or desktop helpers and their shelly nature [16:03] zyga: ijohnson: /snap/gnome-calculator/544/snap/command-chain/desktop-launch is at the top when running with --trace-exec, so meh [16:03] so that's that [16:04] not doing a gazillion mounts would be a good optimization [16:04] Unclear, the go runtime does do lots of wait4 for it's threads and such [16:04] kernel work upstream to get apparmor and overlayfs to talk to each other would remove the need for virtually all of those [16:04] But also the shell desktop helpers are really slow too [16:05] * ijohnson has no unread emails, will commence investigations after some toast === heather is now known as hellsworth [16:07] * ogra sets up his spam filter to re-direct some of the mails to ijohnson so he doesnt run out of mails to read [16:10] PR snapd#7758 closed: devicestate: add reading of modeenv to uc20 firstboot code [16:24] PR snapd#7759 opened: devicestate: add reading of modeenv to uc20 firstboot code [16:29] PR snapcraft#2814 opened: Remove gsettings from comment in kde extension [16:47] PR snapd#7760 opened: snap-bootstrap: write /run/mnt/ubuntu-data/var/lib/snapd/modeenv [16:57] zyga: well, 'yes' but we also need the gagillion mounts cause overlayfs in the form needed to play well with LSMs is not in supported kernel versions either [16:58] (assuming it and apparmor did play well together, which they eventually will) [16:58] jdstrand: totally, just as an optimization [16:58] (eventually) [16:59] it will be nice to have that, yes [17:16] PR snapd#7761 opened: devicestate: make /var/lib/snapd/seed available in install mode [17:18] xnox: I would love to talk about pr #7744 tomorrow, essentially just double checking if the approach works for you etc. maybe we can chat tomorrow morning about this? [17:18] PR #7744: snap-bootstrap: set expected filesystem labels [17:19] ok [17:23] I'll grab some drinks and be back soon [17:37] mvo, zyga: i asked cachio to run updated bionic against old 4.15 kernel, and the mount-ns test passed. so it means it's 5.0 kernel indeed [17:37] xnox: ^ also fyi [17:39] pstolowski: thank you! [17:39] mvo: credit goes to Sergio! [17:40] mvo: and thanks for hinting at the kernel! === pstolowski is now known as pstolowski|afk [17:50] pstolowski|afk: thank you [17:53] mvo: what would be a good place to add a kernel cmdline parser? osutil maybe? [18:34] cmatsuoka: osutil is becoming dense, perhaps a sub-package? though I believe pedronis has an opinion on that too [18:46] cmatsuoka: I think we may not need one actually - please check https://github.com/snapcore/snapd/pull/7760 [18:46] PR #7760: snap-bootstrap: write /run/mnt/ubuntu-data/var/lib/snapd/modeenv [18:47] cmatsuoka: this will add the "mode=install" to /var/lib/snapd/modenv [18:47] cmatsuoka: AIUI you use the cmdline to get the snapd_recovery_mode? [18:47] cmatsuoka: or for something else? if for snapd_recovery_mode modeenv is probably the easiest way [18:53] mvo: yes, it was for snapd_recovery_mode [18:53] mvo: I'll check #7760 [18:53] PR #7760: snap-bootstrap: write /run/mnt/ubuntu-data/var/lib/snapd/modeenv [18:56] cmatsuoka: https://github.com/snapcore/snapd/pull/7759 is probably slightly simpler [18:56] cmatsuoka: it contains just the reading part [18:56] PR #7759: devicestate: add reading of modeenv to uc20 firstboot code [18:56] cmatsuoka: not yet pedronis approved so be warned :) [18:59] cmatsuoka: I added a question in the https://docs.google.com/document/d/1UZm-Eis2p9owBpskuwxPjMqKYHq9zm2hUGP_VD0w9Rw/edit# [19:00] cmatsuoka: hopefully this does not block you? [19:03] mvo: don't worry, it doesn't [19:05] excellent [19:05] mvo: is there a more direct way to get a gadget MountDir() than getting the snap name and then getting the info from snapstate.CurrentInfo()? [19:07] cmatsuoka: I think that's the quickest way currently [19:07] mvo: ok, thanks [19:07] * cmatsuoka still a bit confused with the data structures [19:18] I've implemented the new systemd thing and it seems to be okay [19:18] I'll try to propose it tomorrow, there are a few details I want to get nicer before I open it so nothing tonight [19:19] I'll call it a day [19:19] see you all tomorrow [19:31] have a good evening zyga! [20:02] PR snapd#7702 closed: tests: adding fedora 31 [20:12] PR snapcraft#2815 opened: minor developer env/doc updates