[00:05] PR snapd#6176 closed: snap: check max description length in validate [06:17] zyga: hey, 5845 is ready for a re-review (you commented some days ago there) [06:19] morning [06:21] hey mborzecki [06:21] mvo: how are the tests looking today? still mostly red? [06:22] mborzecki: I saw a green one [06:22] mborzecki: 5845 which also needs a second review [06:22] mborzecki: but also some reds :( [06:22] mborzecki: I just retriggered 6156, that should tell us about the testing weather today [06:22] heh :) [06:56] o/ [06:56] mvo: looking [06:57] 1K wow [07:00] zyga: 1k? [07:00] about 1K of diff [07:01] but no worries, reading [07:11] PR snapd#6178 opened: snap,snap-exec: add SNAP_DEBUG_START_TIME environment [07:24] zyga: heh, things look pretty good with snap_debug_start_time set, it seems we take only ~0.1sec for brave to setup our stuff [07:24] nice [07:24] so what is slow? [07:25] zyga: I think the wrappers, I'm on it, not sure conclusively yet [07:25] super [07:25] zyga: slowly working my way up :) [07:25] I will finish this review, garden my branches and work more on perf monitoring [07:26] while not immediately helpful to your cause it will allow us to do this more or less automatically down the line [07:31] mvo: wonder if something like snap run --strace='--raw -e execve -r' would work [07:32] mborzecki: hm, good idea, maybe no need for a special tool. [07:32] looks a bit noisy for gnome-calculator [07:32] mborzecki: heh, its even worse with brave :) [07:32] mborzecki: forkstat is what I'm currently using [07:34] mvo: well, with gnome calc, there's clearly lots of stuff happening [07:34] mborzecki: yeah, tons [07:34] mborzecki: including update-mime-database which is ~2s here [07:36] mvo: gnome-calcualtor https://paste.fedoraproject.org/paste/rCbAYtyL02YLJ9xm~-pTZg/raw [07:37] heh, funny how not using absolute path makes multiple execve() calls [07:39] mvo: reviewed [07:39] hm xdg-user-dirs update takes less time than head -c0 calls [07:40] zyga: ta [07:40] mvo: I'm sorry that it is not a +1 yet [07:41] let me know if you want to talk about any of this [07:47] zyga: no worries, thanks for the review, the comments look really useful [07:50] mvo: on my machine, each call to mkdir/ln/ls/head/date takes 15ms+ (~90 calls), glib-compile-schemas ~100ms, gdk-query-pixbus-loaders ~700ms, update-icon-caches is called twice ~30ms total [07:50] mvo: that's for gnome-calculator [07:51] mborzecki: that's all in shell, right? [07:52] mvo: yes [07:52] a shell program making those calls? [07:52] mborzecki: whats the total overhead for you? [07:54] mvo: ~2s [07:54] mvo: but that's ssd, multi core and so on :P [07:55] mvo: i'll try with dropped caches === pstolowski|afk is now known as pstolowski [07:58] mornings [08:00] (mac chime) [08:00] hey pawel [08:00] :) [08:04] can anyone make a quick test and confirm if this trivial change https://pastebin.ubuntu.com/p/zSbp4VKkgq/ causes massive mem leak and oom with go test inside snap/ dir (make sure you're ready to stop it before your desktop gets unresponsive) [08:04] mborzecki: ta, I see ~2s here as well (also multi-core, ssd etc) [08:05] pstolowski: good morning! [08:05] yep, checking [08:14] mvo: hm dropped caches, removed $SNAP_USER_DATA and so on, g-c takes visibly longer to start, but it's not reflected in desktop helpers, the change there is minimal [08:15] So what is the slow part? [08:15] mvo: https://paste.fedoraproject.org/paste/w-X6E2U7OKjM9n0210BGTA see line 644, there's a bunch of processes started, but they don't exec [08:15] Apart from fonts [08:15] mvo: some glib/gtk thing? [08:17] To compute 2+2 you need to fork 100 times [08:19] g_async_* [08:19] maybe is spawning gjs under the hood :) [08:21] mborzecki: looking [08:21] mborzecki: why does a pastebin require 7 script to work - oh well [08:21] mborzecki: hard to say but I would not be surprised if that was some sort of gjs [08:34] zyga: any findings? [08:41] mborzecki: you landed some improvements for diffing on failing go tests some time ago; what's the approximate location of this diffing code? [08:41] pstolowski: that'd be in check.v1 [08:42] pstolowski: checkers.go, there's formatUnequal() [08:43] mborzecki: got it, thanks! [08:44] mborzecki: heh, it seems it's related to the leak i'm seeing [08:44] mborzecki: i removed that entire logic [08:44] mborzecki: getting just a few test failures now (expected) [08:44] pstolowski: is that in interfaces? [08:44] i recall getting oom when something went awry there [08:45] it in snap/ , yaml parsing tests [08:45] mborzecki: it eats mem pretty quickly with the diff i pasted earlier [08:45] pstolowski: iirc it didn't like when structs were pointing to each other [08:46] mborzecki: yep, i saw something suspicious yesterday with more complex changes (it would output "CYCLIC REFERENCE"). but in this simple case i created it's actually less references (i'm not creating slot <-> hook references) [08:47] mborzecki: in other words i deliberately broke code not to create references, and it causes oom :/ [08:47] zyga: mvo: i think g-c is actually pulling some exchange rate data from the network [08:48] mborzecki: I wrote a small script to gather the strace datahttps://paste.ubuntu.com/p/vD2rsyXHJr/ [08:48] [08:48] mborzecki: https://paste.ubuntu.com/p/vD2rsyXHJr/ [08:48] mborzecki: this is from brave, I will post the script in a sec [08:48] mborzecki: i suppose the problem is actually in pretty? [08:48] mvo: nice [08:49] pstolowski: yes, that's quite possible [08:57] pstolowski: re, sorry for the lag - some house stuff [08:58] pstolowski: looking again [08:58] np [08:58] pstolowski: ah so check.v1 bug? [08:58] zyga: it seems so [08:59] zyga: or rather - it's dependency that's doing the diffing [09:02] pstolowski: keep in mind that the print only happens if the values are not equal in the tests [09:03] mborzecki: sure [09:03] mborzecki: only it's impossible to find out what's failing ;) [09:04] mborzecki: for now i just disabled this pretty printing [09:04] pstolowski: i can relate :) this was super annoying when working on interfaces [09:05] PR snapd#6172 closed: Send epochs followup [09:17] mvo: my breakdown for brave, 7.6s from s-c to exec'ing brave, top 3 are update-mime-database (2.9s), gtk-update-icon-cache (1.1s), snap-update-ns (~700ms apparently) [09:17] suprising how far one can go solely with emacs :) [09:17] mborzecki: I think I should measure snap-update-ns [09:17] can you please tell me what is the mount profile there [09:17] both system and user [09:18] mborzecki: aha, ok [09:19] mborzecki: http://paste.ubuntu.com/p/Gn95x59tqT/ <- this is what I used [09:19] zyga: http://paste.ubuntu.com/p/tg5VcSyJw8/ [09:19] mborzecki: I will include it in my bench script to make reproducing this easier [09:20] mvo: great [09:25] thanks [09:26] mborzecki: is that all? [09:26] that's the system profile, is there a user profile too? [09:27] maybe there's a bug but I don't see why this should take 700ms [09:27] well [09:27] something to check [09:27] zyga: https://paste.ubuntu.com/p/wG63T4jQjd/ [09:27] ah [09:27] so yes [09:27] portals [09:27] I bet a beer it is portal startup check [09:27] though [09:27] * zyga thinks [09:27] perhaps not [09:28] that was added to snap-run IIRC [09:28] mborzecki: when you measured, 700 was for which of the two executions of snap-update-ns? [09:29] zyga: regular one, let me grab that part of strace [09:30] with roughly 30 mount calls there that's ~~ 20 ms per entry [09:31] (aka looooooots) [09:33] zyga: https://paste.ubuntu.com/p/mR7vpYFRgS/ [09:34] zyga: hm, it'd be easier if snap-update-ns printed it's pid :) [09:35] is that [pid 20700] 0.674152 [????????????????] +++ exited with 0 +++ [09:35] 0.67 is the duration? [09:36] zyga: yes, that's my guess [09:39] mborzecki: I started using "-ttt" to get more reliable timing [09:39] PR snapd#6179 opened: snap: epoch lists must contain no duplicate entries [09:47] mvo: yeah, seems to give quite different results when tracking many syscalls [09:49] zyga: with -ttt s-u-n seems to take ~100+ ms for snap fstab and ~10ms for user mounts https://paste.ubuntu.com/p/cp7cs6jvFC/ [09:49] what is -ttt? [09:50] given that this x7 difference is the remaining allocation accurate? [09:50] (to other programs in the chain) [09:51] Hi snappy people, is there a way to clean a channel so that it shows nothing is published there? I want to have nothing released for microk8s 1.10/edge [09:51] zyga: -t is wall clock, -r is relative time between syscalls (supposedly using montonic clock) [09:52] zyga: also, from the point where sc_fork_helper is called to where sigchld is received it's ~650ms [09:53] that's ok [09:53] sc_fork_helper lives for the duration of both s-u-n calls [09:53] in between we do our business and call sun twice [09:55] hm actually intersting, between calling sc_fork_helper and sc_populate_mount_ns there's 500ms difference, not that it matters much [09:55] mmmmmmm [09:55] that's interesting [09:55] note [09:55] which version is this [09:55] that code changed recently [09:55] is this master? [09:56] i've built s-u-n from master btw [09:56] ah sorry [09:56] master is "old" [09:56] I was thinking about my recent changes [09:56] so [09:56] hmmm [09:57] maybe the slow part is aa change hat? [09:57] zyga: lines 8 and 15 in that last paste [09:57] can you do an experiment [09:57] comment that out [09:57] and see what happens [09:57] you may need to splice the helper hat profile into the main profile to avoid denials [09:57] it'd be fun if that small operation took this long [09:58] * zyga is happy snap-confine is in C an can be timed reliably with symbols [10:05] PR snapcraft#2409 closed: python plugin: process deps before and separately from setup.py === alan_g is now known as alan_g_ [10:33] brb [10:34] PR snapd#6156 closed: wrappers: remove all desktop files from a snap on removal [11:09] PR snapd#6180 opened: snap/info: bind global plugs/slots to implicit hooks [11:21] * pstolowski lunch [11:47] mvo: updated https://gist.github.com/bboozzoo/d4b142229b1915ef7cc0cf8593599ad9 added content verification and variants with systemd units loaded before start/stop, with systemd-mount and just simple mount [11:48] mvo: systemd-mount is flaky, so it fails early when stopping, mounts are so so, updating the state of loopback devices seem to take a while; loading mount units before reload makes the problem disappear [11:49] mvo: so the only path where it fails reliably is when reload interleaves with start (and maybe stop) [11:52] Chipaca: have you seen https://forum.snapcraft.io/t/display-non-autoconnect-interfaces-at-install/8609 ? sounds like a nice ux improvement [11:57] mborzecki: nice findings"! [11:57] mborzecki: this also means we can serialize daemon reload and the problem hopefully goes away? [11:57] GRL (GLOBAL RELOAD LOCK) [12:14] PR snapcraft#2412 opened: schema: add support for title [12:21] zyga, https://paste.ubuntu.com/p/j7BdJ6jhJJ/ [12:21] any idea what could cause that? [12:21] it is using core from edge [12:21] mvo, ~ [12:21] yes [12:21] but no idea why [12:22] what distribution and kernel is this? [12:22] bionic [12:22] 4.15.0-36-generic [12:22] mborzecki: ^ [12:22] is the calc open now? [12:23] hah, intersting [12:23] zyga: does edge have the fixes you landed recently? [12:23] mborzecki: yes [12:23] no [12:23] well [12:23] I suspect so [12:23] yes/no/maybe :) [12:24] but in either case those are not related [12:24] none affect discarding [12:24] maybe gnome-calculator is still working? [12:25] it is not working now [12:25] no process running at least [12:26] can you look at [12:26] cd /sys/fs/cgroup/freezer [12:26] I tried another snap [12:26] supercalc-snap 1 stable keshavnrj disabled,broken [12:26] and also failed [12:26] then cd to snap.gnome-calculator [12:26] and look at cgroup.procs [12:26] check what processes are there [12:28] zyga, I dont see calculator [12:28] what do you see [12:28] zyga, other gnome snapd [12:28] snaps [12:29] can you run [12:29] logs, characters [12:29] sudo SNAPD_DEBUG=1 /usr/lib/snapd/snap-discard-ns gnome-calculator [12:30] zyga, what I see [12:30] is that if I remove from command line a snap [12:30] it works [12:30] if I remove from ubuntu-software the snap fails to be removed [12:31] zyga, https://paste.ubuntu.com/p/cNHpvdCqMM/ [12:32] that's the one from the distro, can you run the one from core [12:32] that will be representative of what snapd runs [12:32] it has more stuff going on [12:33] sorry, I didn't think about this [12:33] zyga, I already tried removing from core and it works [12:33] ? [12:33] I mean [12:33] can you try running snap-discard-ns like you did but use /snap/core/current/usr/lib/snapd/snap-discard-ns instead [12:34] yes [12:34] and capturing the log if that [12:34] when it fails [12:35] https://paste.ubuntu.com/p/9Ng8pfysj9/ [12:35] fails [12:36] aha [12:36] I suspect I know what the problem is [12:36] this is fixed (I suspect) by ... [12:36] https://github.com/snapcore/snapd/pull/6159 [12:37] PR #6159: cmd/snap-confine: handle mounted shared /run/snapd/ns [12:37] cachio: can you post your host's mountinfo table [12:37] what I don't know is how that can happen [12:37] seems like another bug [12:37] mborzecki: yes i saw that [12:37] mborzecki: and yes i agree [12:37] Chipaca: we should have a papercuts lane in trello [12:37] mborzecki: my plate is full but i've got my eyes on it in case nobody picks it up before i finish the smallest pea [12:38] mborzecki: +23 [12:38] note how we never unmounted the file [12:38] * Chipaca accidentally burned through all his +1 reserves [12:38] just unliked it [12:38] that is because it's a tmpfs [12:38] because we re-mounted /run/snapd/ns over itself [12:38] hiding mounts inside [12:38] not sure why [12:38] zyga, https://paste.ubuntu.com/p/H5sCQkVWkm/ [12:38] can you get the mountinfo table instead [12:39] mounts shows partial data [12:40] but yeah, I suspect that's the case [12:40] tmpfs /run/snapd/ns tmpfs rw,nosuid,noexec,relatime,size=1222540k,mode=755 0 0 [12:40] tmpfs /run/snapd/ns tmpfs rw,nosuid,noexec,relatime,size=1222540k,mode=755 0 0 [12:44] zyga, sorry, this is what you need https://paste.ubuntu.com/p/YrDVdcNyQV/? [12:44] zyga, sorry, this is what you need https://paste.ubuntu.com/p/YrDVdcNyQV/ ? [12:45] no [12:45] please cat /proc/self/mountinfo [12:45] brb, lunch [12:45] (not urgent) [12:45] cachio: I think the bug is fixed in that PR I linked to [12:45] zyga, https://paste.ubuntu.com/p/Fhb8GrpJnM/ [12:46] thanks, I'll check soon] [12:46] zyga, ok, but why if I do snap remove foo [12:46] works [12:46] and if I remove from ubuntu software it fails? [12:49] mborzecki: trello says it isn't for papercut kind of thing [12:49] mborzecki: so, https://forum.snapcraft.io/tags/papercut [12:50] mborzecki: (in "about the snapd team board", no backlog and no quick tasks) [12:54] Chipaca: ah, fair enough [12:55] zyga: https://github.com/systemd/systemd/issues/10872 [13:07] mborzecki: nice [13:07] mborzecki: I'm not 100% sure it's systemd but let's see [13:07] zyga: couldn't reproduce it any other way [13:07] cachio: it depends on the order in the info table [13:07] mborzecki: I will look again later [13:07] mborzecki: need to wrap up and wrap up and do more tasks [13:09] zyga, ahh, ok [13:29] mvo: I now have collection of performance events from snap-update-ns [13:29] detailed actions as well [13:29] now looking at C [13:30] PR # closed: core-build#11, core-build#22, core-build#26, core-build#37 [13:31] PR # opened: core-build#11, core-build#22, core-build#26, core-build#37 [13:32] * Chipaca hugs mup [13:32] some days are like that [13:44] mborzecki: super nice [13:44] zyga: cool, thanks. what times are we talking about? [13:44] zyga: I mean, how much time does snap-update-ns take on a typical snap? [13:47] mvo: zyga: it'd be nice to have a baseline reference, eg. hello-world and then the snap with all the interfaces [13:47] nice idea [13:47] mvo: hold on [13:48] mborzecki: indeed [13:49] mvo: note that I'm measuring inside the process [13:49] but now I see 0ms on trivial apps [13:49] let me try brave [13:49] 0ms rounded to ms [13:49] in snap-confine I will measure the whole chunk [14:01] ho issues [14:01] I seem to be stuck at the joining screen [14:02] 2fa [14:37] mvo: perf trace https://paste.ubuntu.com/p/kpNMpxj5jm/ [14:38] mborzecki: thanks! [14:47] mvo: mksquashfs options -noI -noD -noF [14:48] mborzecki: thank you [14:56] mvo: mount thing reproduces on 18.04 too [14:57] * cachio lunch [14:58] mvo: what do you think about cherry-picking centos support to 2.36? [14:59] mborzecki: +1 [14:59] mborzecki: how big is the diff? [14:59] PR snapd#6173 closed: sanity: extend the kernel version check to cover CentOS/RHEL kernels [15:00] mvo: most of it are tests, a little of packaging, i'd pick the sanity check too [15:00] mvo: i can open a PR [15:00] mborzecki: please do [15:08] mvo: running under spread now [15:24] mvo, mborzecki: I need to land https://github.com/snapcore/snapd/pull/6170 [15:24] PR #6170: overlord,tests: expose enabled features to /var/lib/snapd/features [15:25] mvo: I'm preparing to send feature PRs [15:27] pedronis: I guess 6170 is uncontroversial, right? a look at the description is enough, we can do the in-depth review [15:48] mvo: then I need this https://github.com/snapcore/snapd/pull/6181 [15:48] PR snapd#6181 opened: features: add feature test methods [15:48] PR #6181: features: add feature test methods [15:51] PR snapd#6182 opened: Revert "cmd/snap-confine: don't allow mapping lib{uuid,blkid}" [15:55] PR snapd#6183 opened: sanity, spread, tests: add CentOS (2.36) [15:55] mvo: ^^ [15:59] mborzecki: ta [16:06] mvo: prepping remaining branches [16:08] zyga: thank you! [16:09] adding missing tests etc [16:14] mvo: yes, about 6170 seems fine, we could bike shed about the name features, but I know other cases where it has been used fairly generically [16:18] PR snapcraft#2413 opened: kernel plugin: introduce a _get_kernel_version() helper [16:18] pedronis: great, thanks. yeah, lets not bike shed too much :) [16:21] mvo: what is #6178 about, startup performance of snaps? [16:22] PR #6178: snap,snap-exec: add SNAP_DEBUG_START_TIME environment [16:24] pedronis: yeah, the brave discussion on the mailinglist [16:24] * roadmr prefers cowardly discussion :P [16:24] pedronis: happy to give you more context if you want, we can have a quick HO or something (or tomorrow or later, whatever works for you) [16:26] mvo: https://github.com/snapcore/snapd/pull/6184 [16:26] PR #6184: perf: add performance helpers [16:26] PR snapd#6184 opened: perf: add performance helpers [16:27] mvo: the next part is the perf manager but I need to make tea now [16:27] I'm starting to feel sick [16:27] zyga: yeah, lets get pedronis a chance to look at this too [16:27]