[05:09] morning [05:09] morning [05:12] Hey hey [05:13] today's gonna be interesting [05:22] Why? :-) === cpaelzer__ is now known as cpaelzer [05:36] zyga: alleged memleak, or rather trying to find out if we are the memory hogs [06:07] pulled down some random ubuntu core 16 image, came with snapd 2.32.5, can't refresh core because no updates available, wth? [06:17] mborzecki: core updates have been disabled for the last 14h due to the memory hog situation [06:17] But I'm minutes away from reenabling them [06:17] wgrant: thanks for the info [06:28] Interesting [06:29] Certainly more likely than pushing a deb to a repo [06:29] this is actually quite interesting 2.32.5 vs 2.38 vs 2.37.4/2.38 classic https://paste.ubuntu.com/p/Vs65Wyg8XW/ [06:29] mborzecki: core should be refreshable now. [06:29] s/likely/fun/ [06:30] I’m with the dog outside [06:30] I’ll check back in the office [06:30] wgrant: thanks, updating :) [06:31] mborzecki: I have an idea [06:31] Is it hotplug? [06:34] zyga: idk, we can't turn off the feature anyway [06:34] zyga: fwiw, this doesn't look so bad === ricekrispie2 is now known as ricekrispie [06:35] mborzecki: to be clear, we more looking at kernel memory, than process memory at this point [06:36] Did we change compiler version? [06:36] zyga: of course, remember we went from go 1.6 to go 1.10/1.11 [06:36] pedronis: yeah, what i mean is that there doesn't seem to be an easily visible regression [06:37] pedronis: 2.32.5 was surely go 1.6 [06:37] yes [06:37] think so [06:38] Mmmmm [06:39] Go changed the memory model recently [06:39] Well it maps much larger structures into memory without compression [06:39] But the garbage collector also changed significantly [06:39] zyga: 6621 has two reviews, and is now used, could just land, no? [06:40] zyga: the question is, does it make the kernel use more space [06:41] anyway I made a suggestion in email to test that to mborzecki [06:41] zyga: also memory is given back gradually in background, so rss will decrease but only after a while, we could call https://golang.org/pkg/runtime/debug/#FreeOSMemory though if the memory is lost elsewhere all this is in vain [06:41] pedronis: I think Chipaca wanted it to have an approved dependency first [06:41] But I will look [06:41] Still afk [06:43] pedronis: to the best of my knowledge it does use more kernel memory but it is a static, compile time decided cost [06:44] we need to find out how much it is on arm 32 [06:45] It is pretty huge AFAIK, dominates binary size [06:45] But [06:45] It is not a growing number [06:46] we are probably talking past each other [06:47] pmap on 2.35.2 and 2.38 https://paste.ubuntu.com/p/xdm4tFhDwz/ [06:48] hm .text grew, idk what's the r---- section of snapd, but it changed a bit, rw--- snapd is probably .bss, there's a change there, but rather insignificant [06:52] mborzecki: you have email [06:53] mborzecki: as I said, I don't think there is something obvious happening that relates only to user space mem [06:54] no mvo yet to sync [06:56] pedronis: btw. rebuilding with 1.6 might be some effort, we made a transition to "context" and updated some http request context related things in the process [06:56] pedronis: 1.7 is probably worth a try though [06:56] mborzecki: we can do the reverse [06:56] buld 2.37 with 1.10 [06:57] if the issue is really only go runtime [06:57] back in the office now [06:57] it might be apparent just that way [06:57] pedronis: right, that too [06:57] (it might be a combination of a lot of things, then it's harder) [06:57] mborzecki: good idea on 2.37 with 1.10 === pstolowski|afk is now known as pstolowski [07:07] mornings [07:08] ok, got 1.11 set up, and it's quite funny, https://paste.ubuntu.com/p/YFTYBSTmPJ/ [07:08] going to try 1.10 next [07:15] mborzecki: ha [07:15] fun indeed [07:16] mborzecki: interesenstingly though vsz is somewhat larger for 2.38 (not a problem, unless it really cuts into kernel resources though). anyway reminder the devices are arm 32 [07:17] mvo, morning! [07:18] mvo, which is the current status of the systemd daemon-reload race? is the fix already in stable? [07:19] HOOOOLY MOLY [07:19] pidfd!!!!! [07:19] hahahaha [07:19] * zyga is super happy [07:21] abeato: the systemd fix went to the distro yesterday, it will be part of the next stable core [07:21] abeato: if its import for the customer we can do a .1 with just these changes [07:22] mvo, hm, when is the next stable core snap expected? [07:23] abeato: 4-6 weeks if nothing else comes up [07:23] abeato: because 19.04 is released soonish we may do a .1 if we find any bigger issues this is why its a bit up in the air right now [07:24] abeato: we are debugging a customer issue that may require a .1 but its a bit unclear where we stand on this right now (i.e. if its a snapd, kernel, $unknown issue) [07:24] mvo, ok, I do not think it is extremely urgent, will let you know if that changes - it depends mostly on when we release the GA image [07:48] 2.37 & 2.38 with go 1.10 https://paste.ubuntu.com/p/Snt9mqK3m5/ [07:54] ~0.5G more vmem [08:00] PR snapd#6629 closed: overlord/snapshotstate: helpers for snapshot expirations (automatic snapshots 2/4) [09:08] * zyga spends some time to manage paperwork and the endless list of open browser tabs [10:35] PR snapd#6595 closed: tests: add regression test for systemctl race fix [10:44] mvo: what shall we do about core18? [10:44] zyga: aha, sorry, I forgot about this, let me look at this [10:44] mvo: do you want to HO? [10:44] might be easier [10:47] zyga: let me quickly poke at this for some minutes to familiarize myself [10:49] PR snapcraft#2522 closed: build providers: add API for friendly instance type names === cpaelzer__ is now known as cpaelzer [10:59] zyga: mvo: I need to be in that HO [10:59] if/when we have one [11:03] * pstolowski lunch [11:07] ask me if my parents' wifi goes away when they use the microwave [11:12] zyga: lets talk after the standup, I have lunch now and then I have a meeting, maybe if this meeting is shorter we can also talk before [11:12] mvo: ping me when you are ready to talk [11:12] zyga: but yeah, we need to decide, I think I hvae a slightly better idea now what is going on. [11:12] zyga: thanks again for raising this! [11:13] mvo: zyga: ping me when you have it [11:13] ack [11:23] * Chipaca ⇝ lunch [11:31] PR snapd#6691 opened: Move extractkernelassets [11:32] * zyga break [11:33] ondra: reworded that PR a little [11:34] zyga yeah more fitting [11:34] zyga thanks :) [11:40] * zyga really break now [11:52] re [12:07] Son_Goku: hey, on F30 I cannot get snapd update because of some gnome-software dependency mismatch, [12:07] Son_Goku: is that expected? [12:08] no? [12:08] what are you seeing? [12:09] I don't have that machine with me but it was saying something that gnome-software depends on snapd-auth-provider or something to that vein [12:09] and that was it, no 2.38 [12:09] fuck [12:09] snapd-login-service dependency is still in gnome-software [12:09] damn it [12:10] I have to put it back now [12:11] :D [12:11] Son_Goku: so what just happened? [12:11] well, I decided to rip out an old Provides [12:11] when snapd-login-service was retired, I obsoleted it by snapd [12:11] but gnome-software-snap requires it still :( [12:12] sorry I didn't vote on the update before, I noticed others did [12:12] but those must have been headless machines [12:12] my laptop complained [12:22] zyga: yup i tried the update in a headless vm [12:24] mborzecki: https://github.com/snapcore/snapd/pull/6661 has 3 +1s, can land? [12:24] PR #6661: data/selinux, tests/main/selinux-clean: fine tune the policy, make sure that no denials are raised [12:24] pstolowski: i'm pushing the fix for the test zyga suggested [12:24] ack [12:26] oSoMoN: hey, I'm still slogging through the chromium approvals. it is a large snap so it takes a while for the next revision to be reviewed after I approve one [12:28] jdstrand, thanks! [12:36] * Chipaca shakes his fist at doLinkSnap [12:43] zyga, pedronis I am free now if you want we can chat about /var/lib/snapd/void and friends [12:44] mvo: I can join standup [12:45] pedronis: I'm there now, ready when you are [12:45] zyga: -^ [12:47] mvo: same [12:48] joining === ricab is now known as ricab|lunch [13:28] PR snapcraft#2520 closed: snap: set core as a base [13:30] PR snapd#6616 closed: boot: add flag file "meta/force-kernel-extraction" [13:51] Bug #1823343 opened: Getting Started Intro out of date [14:08] PR snapd#6692 opened: interfaces: cleanup internal tool lookup in system-key [14:09] PR snapd#6693 opened: cmd: tweak internal tool lookup to accept more possible locations <⛔ Blocked> === ricab|lunch is now known as ricab [15:16] PR snapcraft#2523 opened: ci: improve travis integration conditionals [17:04] pedronis: you can't have an alias for a snap name, right? [17:07] Chipaca: correct, no such thing [17:07] noise][: I meant, an alias in one snap that was the name of a different snap [17:07] noise][: like, aliasing tree-classic to 'tree', which is the name of a snap [17:08] nope [17:10] Chipaca: you can but is very much discouraged because you cannot then install conflicting snaps around it in any way at the same time [17:11] we don't let you override the commands of another snap [17:11] so this tree-strict and nano-strict thing is not gonna fly [17:11] oh, i think i misunderstood [17:12] Chipaca: it depends, it means you cannot install nano and nano-strict at the same time [17:12] pedronis: right, but what happens if you try? [17:12] but you wouldn't [17:12] you get an error [17:13] about namespace conflicts [17:13] i mean i'd like to see that ux [17:13] when you install the 2nd [17:13] … but i have enough on my plate already [17:13] but the existence of `tree`, not installed, does not preclude a snap tree-strict from aliasing 'tree' , right? [17:13] no [17:14] it's up to the alias reviewer in the store [17:14] we didn't implement strict checks there about this [17:14] I should clarify in the forum then [17:14] pedronis: what do you mean by 'strongly discouraged' btw [17:15] because the people reviewing and approving aliases didn't even bring it up [17:15] that if the snaps are unrelated [17:15] you will just bring confusion into the world [17:15] a snap owns its name [17:15] you are basically giving it away [17:16] pedronis: that doesn't sound like 'strongly discouraged' beyond the abstract sense. I mean, what steps have been taken to discourage it? [17:16] it seems perfectly reasonable to me to allow for 'foo' and 'foo-classic' snaps that are the same thing but one with classic confinement. [17:17] noise][: there yare related though, which is actually not a typical case at all [17:17] for the case outlined with tree or jq or whatever where there's a desire to be able to use them on Core systems [17:17] at it wasn't a typical case when aliases were designed [17:17] both from the same publisher [17:17] or i mean that would be the preferred scenario [17:19] Chipaca: the guideline around aliases as always been to avoid clashes [17:19] popey suggested using a separate `classic` track for the alternative classic confined versions, which may mitigate the problem [17:20] Chipaca: that included with other snap names, maybe is not splelt out but was in our mind [17:21] For the alias problem if there's a popular fork the same name software, like `blender-some-popular-edition`, people might want to make alias of the original name [17:21] brlin: o/ [17:22] now I just want to have a snap with an alias for another snap just to see how it explodes [17:23] Chipaca: I can point you at the errors [17:23] classic confinement is at the snap level, can't be per-track [17:23] noise][: classic confinement just means you can have classic confinement in some revisions [17:24] whether we enforce more I don't know [17:24] Chipaca: to be clear we didn't design thinking that this would be a common case that we would need to guide people around [17:25] right, i mean we don't have a way to selectively enforce [17:25] brlin: updated both topics with a summary of the above. I don't think [revoked] is correct … even if I were correct, i'm not a reviewer :-) [17:25] noise][: Then I guess the separate name space and overriding alias is the only way to go :-/ [17:25] to be clear, _allowing_ classic is not per-track, but a snap being classic is per revision [17:25] what Chipaca said :) [17:26] so you _could_ have a channel be classic and a channel be strict, but you could ship a classic one in the strict track by accident [17:26] this certainly could bear more discussion, but it's Friday afternoon [17:26] yeah, so it's possible we could allow that route for some snaps and just have to trust the publisher not to break people on the strict track [17:26] also bears don't like discussion [17:27] and then at some point add guardrails, but i'd like to explore this more before recommending such an approach [17:27] Chipaca: errors are here https://github.com/snapcore/snapd/blob/master/overlord/snapstate/aliasesv2.go#L339 and here https://github.com/snapcore/snapd/blob/master/overlord/snapstate/aliasesv2.go#L368 [17:28] The snap edit page on LaunchPad seemed to support this fashion AFAICT https://usercontent.irccloud-cdn.com/file/prUHOKkw/image.png [17:31] FYI the users of the `nano` snap are currently blocked from auto-updates due to the confinement switch [17:32] yeah, confinement switches are nasty [17:32] they'll be getting warnings at some point [17:32] that's why, e.g. with tree, that i suggest not messing with confinement of the existing snap [17:32] that's #1811063 FWIW [17:33] Bug #1811063: "snap refresh" does not report failure to update because snap switched to classic confinement [17:33] anyway, i need to run, happy weekend! [17:33] same here [17:33] brlin: ttfn, have a good one [17:33] also to be clear you can install both snaps, but you need to manouver around, and the one with the actual name (vs the alias) needs to win [17:33] (you'll need to use snap unalias and/or install --unaliased) [17:34] Chipaca: Bye! [17:55] PR snapcraft#2523 closed: ci: improve travis integration conditionals [18:12] mvo, hey [18:13] mvo, I know which is the problem with the install --dangerous in core18 [18:31] cachio: tell us! [18:33] zyga, so, the problem is in the tests [18:34] basically when it is installing --dangerous is downloading the core snap as part of that process [18:34] even when we have the core snap in /var/lib/snapd/snaps [18:34] oh [18:34] that's very curious [18:34] cachio: thank you [18:36] zyga, do you know if snapd is searching in the state to define if a snap could be in the cache? [18:36] cachio: I don't know yet, we will check it out [18:36] snapd uses the cache for sure [18:36] but repackaged core has different cache [18:36] so perhaps that's one issue [18:36] s/cache/hash/ [18:38] zyga, https://paste.ubuntu.com/p/J3w8bxrqfm/ [18:38] in the first install it downloads the core snap, even when it is already in /var/lib/snapd/snaps/ [18:38] but second time it is not downloading it [18:39] zyga, I don't know why [18:39] cachio: what is in /var/cache/snapd? [18:39] the one in /var/lib/snapd/snaps is, I think, irrelevant now [18:39] the one in /var/lib/cache/snapd matters [18:39] I mean, irrelevant only if not installed [18:40] zyga, https://paste.ubuntu.com/p/qvj5QCgkwK/ [18:40] that's odd [18:40] so some back story [18:40] last week, for local testing in qemu [18:41] I wrote some patches that let me seed /var/cache/snapd with heavy snaps [18:41] then tests only fetch assertions online, speeding up testing tremendously [18:41] the way I did this was by keeping /var/cache/snapd.tar.gz in the qemu image and unpacking it early in the test prepare [18:41] it works for that purpose [18:42] the files there are not .snap files but instead long file names based on some hash of the content [18:42] snapd uses that cache as hardlink source AFAIK [18:42] so I'm surprised to see none of that in the cache directory here? [18:42] ah [18:42] sorry [18:42] I'm mistaken [18:42] the path is different [18:43] the correct path is /var/lib/snapd/cache [18:43] that's the one [18:43] I was arguing with chipaca to change it because it's odd that we have two cache directories [18:43] anyway [18:43] please look at /var/lib/snapd/cache [18:43] before / after snap install --dangerous [18:44] https://paste.ubuntu.com/p/RdbGtyc79c/ [18:44] zyga, seems to be there [18:44] that's more like it [18:44] what happens when you remove and install a snap? [18:44] it should reuse that cache and do things "instantly" [18:46] zyga, the core is not installed in my machine but it is in cache dir [18:46] so, I think I know what I need to change to cache snaps in the tests [18:46] I will check later (eating dinner now) but perhaps --dangerous implies some of the hash based logic doesn't get used [18:46] what we are doing currently doesn't work [18:46] cachio: cool [18:47] cachio: looking forward to that patch :D [18:47] zyga, --dangerous works well [18:47] but basically we need to install/remove the snap to cache it before we save the state [18:47] that should be enough [18:47] yep [18:48] currently we are copying to /var/lib/snapd/snaps the .snap file but it is not enough [18:48] cachio: as a hint [18:48] cachio: perhaps to speed things up more [18:48] cachio: instead of creating a tarball of /var/lib/snapd/cache [18:49] zyga, yes, could be, I'll make then change [18:49] cachio: make a directory, say /var/lib/snapd-cache [18:49] zyga, thanks for the help [18:49] cachio: wait :) [18:49] cachio: and hardlink the cache items there [18:49] cachio: then instead of unpacking that tarball [18:49] cachio: hardlink them back [18:49] cachio: this will be instant [18:49] cachio: and will work just like a tarball would [18:49] zyga, perfect, I'll do that [18:50] thanks for the info [18:50] cachio: cp has some options, perhaps there are some that do hardlink copies [18:50] cachio: good luck! [18:51] zyga, thnanks [18:51] enjoy your weekend [18:51] likewise :) [18:54] --dangerous doesn't use the cache [18:54] (the cache is for downloads) [18:54] pedronis: oh? [18:54] why would it [18:54] you are giving it a snap here [18:54] pedronis: for dependencies [18:54] pedronis: it downloads core over and over [18:54] that is stranger [19:22] pedronis, I think install --dangerous is using the cache [19:22] I just see that the core is not being downloaded it is correctly cached [19:24] pedronis, the problem we have is that we are caching the snaps in the wrong way on our test suite [19:25] so when I run on the boards and vms 5/6 in parallel in my home all the installs we do are downloading hte core [19:25] and this is the bottleneck which is causing the install --dangerous is getting stuck [19:45] 6687 needs a second review, should be simple [19:45] PR snapd#6689 closed: tests: run create-user on core devices [20:38] PR snapd#6694 opened: tests: improve how snaps are cached [20:43] * cachio afk [21:32] PR snapcraft#2524 opened: Snapcraft try [21:45] pedronis: ok, I got through all the store review stuff and knocked out 5 PR reviews (well, one was asking for a different approach). I will do PR 6502 and PR 6605 on monday [21:45] PR #6502: dirs,overlord/snapstate: add Soft and Hard refresh checks [21:45] PR #6605: cmd/libsnap,osutil: fix parsing of mountinfo [21:45] zyga: fyi ^ [21:46] there is a small chance I'll look at those over the weekend, but let's say 'by my eod Monday' [21:46] pedronis: I also responded to https://forum.snapcraft.io/t/phase-1-of-opt-in-per-snap-users-groups-aka-the-daemon-user/10624/15 [21:52] jdstrand: hey! the dockerd daemon now needs to use data inside $XDG_RUNTIME_DIR, which ends up being /run/user/0 for the daemon, but this dir isn't created... what's the eventual resolution to lp #1656340 do you think? [21:52] Bug #1656340: XDG_RUNTIME_DIR is not created on app startup

[21:52] for now I will just overwrite that to be $SNAP_COMMON/run and it works, but just curious as I seem to recall discussion around that bug recently on this channel === Katnip- is now known as Guest17428 === ondra_ is now known as ondra