=== chihchun_afk is now known as chihchun [04:40] o/ [05:10] morning [05:13] mborzecki: hey [05:13] https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954 [05:13] yup reading [05:16] zyga: any clue if mvo got access to a vm where the refresh-mode issue was reproducible? [05:16] no [05:16] let's make a spread test that has similar characteristics [05:16] zyga: do we know what disto tha was? [05:16] and strace the hell out of it [05:17] not really [05:17] note that we have not reproduced it yet [05:17] but I will let mvo discuss tihs [05:17] he knows the most [05:21] hm that memory usage, we could play with GOGC, but there's another possibility that we're just keeping that much stuff in memory [05:22] (i'm taking about RES, VIRT is high but it should not be a problem) [05:47] mborzecki: yes, RES is key, though VIRT is weirdly huge [05:47] but maybe that's the nature of golang gc [05:47] ok, kids off to school [05:47] and I can have my coffee and focus [06:32] gdm is just so useless, it stats a whole gnome session, does nothing else than shows a login screen and has a 50% chance of locking up the system when switching back to it after logout [06:33] at least autostart works as expected with 2.32.4 [06:33] Heh, the modern desktop stack [06:34] btw. clicking on quit label in tray menu of discord: https://paste.ubuntu.com/p/CtG3Kx7Kxt/ === katnip- is now known as Guest67740 [06:39] Well [06:39] On my system gnome shell logs errors every second [06:40] Finalization issue with some object [06:40] I frankly miss unity quality [06:40] Later on unity was way more solid === Guest67740 is now known as katnip [06:51] ok, bumped arch package to 2.32.4 [06:52] good morning [06:52] mvo: morning! [06:53] hey mvo [06:55] hey mborzecki and zyga ! how is life this morning? [06:55] great, it's a bit chilly in the morning though [06:55] I'm looking into the memory leak [06:56] but if I get stuck or it goes nowhere I will look at the mount issue [06:56] hello [06:57] hey good morning [06:58] so I discovered that init is 2M and snapd nowadays 21M , just the executable (of course systemd has other daemons) [06:58] I really wonder what is inside golang executables [06:58] small instance of google Borg? [06:58] well, as I said, systemd has many daemons [06:58] so it's not a fair comparison [06:59] but it doesn't put in a good spot [06:59] we have not to run at all, not to use memory [07:00] that ~21M would mostly count to virt though [07:00] mborzecki: looking at smaps it doesn't seem so [07:00] but it still needs to be loaded into memory and so on [07:01] anyway on a constrained system probably even running and stopping is bad [07:03] mvo: let me know if we should have a HO in 1h or so to talk about that issue at least and options [07:04] pedronis: yeah, sounds good a HO in the morning sounds good, I am currently baby-sitting autopkgtest but hopefully that does not take too long [07:06] morning [07:06] hey pawel [07:06] http://paste.ubuntu.com/p/GtVYbzfgC8/ [07:06] pstolowski: FIY https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954 [07:07] zyga: yes reading [07:13] with GOGC=10 http://paste.ubuntu.com/p/Gbnz675BT3/ [07:13] playing with GOGC, off -> 33MB, 5 -> 19MB, 50 -> 20MB, 100 (default) -> 22MB [07:17] zyga: what tool is that? [07:17] pmap [07:17] * mvo nods [07:19] zyga: comparison of GOGC=1 and GOGC=100 https://paste.ubuntu.com/p/74zgkbZBWj/ [07:19] (disabled ASLR so that we get stable addresses) [07:19] nice [07:19] how did you disable ASLR? [07:20] /proc/sys/kernel/randomize_va_something [07:20] but the bottom line is that rss shrinks just a little and virt doesn't [07:20] you can also disble aslr for one invocation with setarch [07:21] setarch `uname -m` -R whatever [07:21] nice [07:26] wonder why there's 7 * 64MB and 6*8MB blocks, 64MB ones are ---p, 8MB are rw-p [07:28] PR snapd#5035 closed: release: snapd 2.32.4 [07:29] good news, spread on xenial is happy again and we have some tests passing already, the other ones are retriggered [07:29] (missing build of test-snapd-account-services on s390x was one failure which is fixed now) [07:31] mborzecki: noite that the 64MB ones are empty [07:31] they take almost no memory [07:32] zyga: cow? [07:34] they feel like a/b gc [07:35] hm bottom line, we're not going to get lower than whatever is the size of snapd binary, i.e. ~16MB [07:38] zyga: installing core snap https://paste.ubuntu.com/p/dQDYNJz2Vf/ [07:41] also the binary in core built with go 1.7 right? [07:41] zyga: before and after installing core snap and reexec https://paste.ubuntu.com/p/xPpCQjV4j8/ [07:56] zyga, mvo, mborzecki would it make sense to add profiling code to snapd and land it in edge, eg something like https://flaviocopes.com/golang-profiling/ in the daemon? [07:58] pstolowski: there's pprof which is super simple [07:59] pstolowski: I don't have experience with that, no idea [08:00] i can try building it locally though [08:01] yeah i'm going to try local build too [08:11] man we are leaking that memory quickly [08:13] I'm in a loo where test-snapd-poicy-app-consumer:bluez is constantly discon reconnected [08:14] this also stresses udev [08:15] mborzecki: the binary in core is build with go1.6 (xenial) [08:15] mvo: I'm around [08:15] zyga: kernel mem? [08:15] pedronis: ok, I join in 2min then [08:15] mvo: perhaps, I'm not sure yet [08:17] interestingly doing this puts lots of load on X as well [08:17] (perhaps due to udev) [08:17] to the point where session is unresponsive (not stuck but very slow) [08:17] I see constantly raising buffer_head allocations [08:20] I see several dozen udev processes at a time [08:20] the leak may be unrelated to apparmor if udevadm trigger / settle leaks something into the kernel / netlink area [08:22] BjornT_: hi, do you have a ppa with a snapcraft that has https://github.com/snapcore/snapcraft/pull/2058 applied by any chance? [08:22] PR snapcraft#2058: python plugin: install python-distutils when run on bionic [08:24] mwhudson: no. but it's in the snap edge channel now [08:25] BjornT_: hm but you can't build a snap in launchpad from a snapcraft from a snap can you? [08:26] (that sentence is quite a tongue twister) [08:29] mwhudson: right, you can't. but i'm guessing you will have problems building a bionic snap in launchpad anyway, won't you? [08:32] mvo: I saw you put #1616629 into 'in progress', does that mean you have a branch for it? [08:33] Bug #1616629: could not unmarshal state entry "snap-type" [08:46] BjornT_: nope, been doing that for months :) [08:58] Chipaca: it means "we are looking at it", pedronis is working on improving the error at least to fail way earlier [08:58] mvo: ah, ok [08:59] Chipaca: one theory is that it relates to removing and reinstalling (maybe we the fs of the snap in use) and lazy unmounting [08:59] pedronis: was that what pindonga did yesterday? [09:00] yes, he said he had removed the snap [09:00] Chipaca: anyway current plan is 1) improve error, 2) possibly add belt and suspenders 3) somebody might want to play with loops of remove and install of the same thing [09:01] i can start on 3 right no [09:01] now* [09:01] thx [09:02] I will work now on 1) [09:06] zyga: here is something strange. this is the LowMem value after running each of the tests, the tests are in order http://paste.ubuntu.com/p/k8wvw689B9/ [09:07] zyga: the fluctuation is also strange [09:07] I suspect what is going on is that as memory pressure raises kernel is dropping more and more caches [09:08] releasing some memory in that zone [09:08] zyga: aha, yeah, that makes sense [09:09] pstolowski: hooked up pprof finally, https://paste.ubuntu.com/p/XXBv8qFFym/ :6060 is the regular http one, the one over /run/snapd.socket you need to wrap around with socat [09:11] pstolowski: something like `socat -v tcp-listen:6070,reuseaddr,fork "unix-connect:/run/snapd.socket"` should work [09:12] mborzecki: pstolowski: we discussed with mvo some ideas about not starting at all , it seems hard to halve or /3 memory usage in one week [09:13] pedronis: especially as the binary is 16MB already ;) [09:13] pedronis: so what are the current ideas? [09:14] mborzecki: mvo is working on that I think, but basically only play at the level of the systemd units [09:15] mborzecki: thanks, checking; i was pursuing something else in the meantime - run 2.32.4 snapd under gcvis but didn't find anything conclusive [09:15] is it still the case that a snap can't call a binary in another snap? [09:16] mborzecki: the idea is to use a generator that looks if there is a snap or a seed and if none of this is true generates a service file that is socket activated only [09:16] mborzecki: this way we do not run at all on cloud instances until first use [09:16] mborzecki: the downside is that we don't get the c-n-f data so we need to put that into an external timer [09:17] but I think its an acceptable trade-off given the time contrains [09:18] sounds fair [09:18] mvo: do you need help with that? [09:19] mborzecki: sure, I also have the sigterm issue that needs attention, if you want to add the generator that would be great. [09:20] mborzecki: we have two options: a) generator b) snapd is also socket activated and we add a "wakeup" service that has conditions on /snap/* and /var/lib/snapd/seed/* [09:20] mborzecki: and this wakeup would just run snap version to ensure the daemon runs [09:21] mborzecki: plus one idea from zyga was to use the "idle" option of systemd so that snapd starts a bit later (when it starts) when the system has settled [09:21] mborzecki: sounds sensible? happy to have a quick HO if you want [09:21] mborzecki: alternatively the SIGTERM issue is something I am willing to trade too, whatever you find more interessting :) [09:22] to be clear: /snap/* OR /var/lib/snapd/seed/seed.yaml [09:22] not AND [09:22] mvo: let's do HO [09:23] pedronis: /snap/*/current i guess [09:23] something [09:23] it's really /snap/* ignoring bin [09:24] and also /snap is not always snap [09:26] the problem with doing install/remove in a loop is that i quickly remember how badly that grows :-( [09:28] loop #1: ~1s; loop #200: ~6s [09:28] PR #200: Improve error handling in client package [09:28] mup: er... thanks [09:28] Chipaca: I apologize, but I'm pretty strict about only responding to known commands. [09:36] mup: #1 [09:36] *pfff* [09:41] mvo: istr #1 being an issue, which are now gone [09:44] * cjwatson is lost in a twisty maze [09:44] Is there some way to force snapd to forget about its existing store device macaroon? [09:44] It must be serialised to disk somewhere to persist between snapd startups, but I can't find it [09:45] Chipaca: yes, gone, let me update the post [09:46] * zyga builds 4.17 [09:46] well, master [09:46] cjwatson: it's in state.json under data.auth.device [09:46] kissiel: /me hugs his computer :) [09:46] cjwatson: session-macaroon [09:47] mvo: which post? [09:47] cjwatson: /var/lib/snapd/state.json. stop and restart snapd for editing [09:47] Aha, thanks [09:50] mvo: np on go 1.6/stable, i'm just waiting for the coinstallable tracks feature :) [10:00] zyga: would you be interested in my "use x/net/context instead of context" sed of doom? :) [10:00] find -type f -name \*.go \! -path '*/.pc/*' -print0 | xargs -0 sed -i -e 's%^$\s\+$"context"$%\1"golang.org/x/net/context"%' [10:00] zyga: it works on docker.io :) [10:01] mwhudson: across the whole tree? [10:01] mwhudson: well, I don't mind that, though I don't see how that helps us this week (fire squad) [10:01] zyga: i saw something about spread not building with go 1.6? [10:01] ah [10:02] mvo fixed that earlier today [10:02] ok [10:02] never mind me then :) [10:02] mwhudson: but thank you for the offer [10:02] we are working through this list: https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954/1 [10:04] PR snapd#5036 opened: overlord/snapstate: allow to get an error from readInfo instead of a broken stub, use it in doMountSnap [10:04] mvo: Chipaca: ^ [10:06] pedronis: thank you! [10:07] hmmm /snap/README, there goes ConditionExistsGlob [10:07] mborzecki: /snap/*/current [10:08] Chipaca: oh, that's good [10:09] pedronis: just to be sure, that's an /snap/.. OR /var/lib/../seed.yaml ? [10:09] mborzecki: yes, either of those should trigger a wakeup [10:09] Apr 12 11:09:21 fleet snapd[7512]: 2018/04/12 11:09:21.252174 services.go:226: DEBUG: StopServices called for [%!q(*snap.AppInfo=&{0xc820397600 webserver [] command-webserver.wrapper simple 0 map[network:0xc8203f9270 network-bind:0xc8203f92c0] map[] map[] {[] map[]} [] [] })], reason: remove [10:09] PR #12: Bugfix/lp1480248 test reenable [10:09] hmmm [10:09] mvo: ok [10:10] something awry in a printf :-) [10:11] Chipaca: woah, where did you get this? [10:13] mvo: snap remove a snap with a service with debug on [10:14] i looped installing/removing a snap without a service and it couldn't repro 1616629, so now trying with a service [10:16] Chipaca: doInstall also has int flags, yes but things that are exported or go into state I agree we shouldn't use int flags [10:16] pedronis: we still have a lot of int flags, and i don't think it's a blocker [10:28] so the wakeup service works [10:28] spread test for this will be tricky [10:31] jdstrand: hey [10:31] jdstrand: is apparmor@lists.ubuntu.com still the place to send patches? [10:32] sergiusens: fwiw 'd bet patchelf is doing something differently horrible to java [10:33] +i [10:34] Issue snapcraft#1676 closed: Support for set-version [10:34] PR snapcraft#2063 closed: many: add snapcraftctl set-version [10:38] Chipaca: btw seems we don't have undo on error code in doMountSnap, not sure it's good time to add that tough [10:39] pedronis: where? [10:39] PR snapd#5037 opened: snap: snap.AppInfo is now a fmt.Stringer [10:39] pedronis: undo as in in-task undo? [10:40] mvo ^ 5037 fixes that %!q thing [10:40] Chipaca: nice [10:41] Chipaca: yes, in-task undo [10:41] pedronis: around readInfo? [10:41] pedronis: what are we missing? [10:42] pedronis: removing the mount unit? [10:42] * Chipaca looks [10:42] removing the mount unit [10:42] I suppose [10:42] in fairness SetupSnap is an opaque name [10:42] no, we remove the unit [10:42] one rror [10:42] RemoveSnapFiles removes the unit [10:42] stops and removes, even [10:43] yes, but now ReadInfo itself can fail [10:44] (well it always could but not materially) [10:44] pedronis: ah! now i gotcha [10:45] pedronis: sounds like you want a m.backend.RemoveSnapFiles there then [10:45] yes, seems so [10:45] or maybe UndoSnapSetup? [10:45] UndoSetupSnap* [10:45] which, er, just calls RemoveSnapFIles :-) but might make the intent clearer from the calling side [10:46] yes [10:49] woah? [10:50] WOAH [10:50] mvo: wanna hear a funny story? [10:50] my test [10:50] Chipaca: sadly that thing needs a type [10:50] where I ate 8GB of ram pretty quickly [10:50] this was my command [10:50] sudo sh -c 'for i in $(seq 10000); do snap connect test-snapd-policy-app-consumer:bluez && snap disconnect test-snapd-policy-app-consumer:bluez; done' [10:51] I took a snapshot of all the profiles before/after this [10:51] before disconnect and after disconnect [10:51] it was connected initially [10:51] and the profiles are identical [10:51] all the apparmor profiles on my system [10:51] so we were leaking all that memory _despite_ not having _any_ changes [10:52] pedronis: which thing? [10:52] UndoSnapSetup [10:52] sorry, UndoSetupSnap [10:52] now, that I look at the code, there's a lot of silliness going on [10:52] so, a bit of a WTF moment [10:52] pedronis: :-) [10:52] I will write a loop that recompiles that one profile in a loop and see if that is eating any memory [10:52] Chipaca: SnapSetup read the type from the snap, and the its caller tries to read the type from the mount [10:52] zyga: wtf yes [10:53] iff it does that's also very interesting [10:53] because apparmor parser supposedly does nothing when you are not really changing the profile [10:53] zyga: the 8G is kernel, not snapd? [10:53] so perhaps simply reading one of the apparmorfs files leaks [10:53] Chipaca: yes [10:53] noice [10:53] almost all in kmalloc-1024 [10:53] I'm on mainline master [10:53] so that should give us all the fixes [10:54] I'll start by confirming the loop I used before still leaks [10:54] Chipaca: so apart details of how is easy it is to untangle this, the readInfo is not stricly needed, it's a nice fence though [10:54] pedronis: only if we check the error [10:55] but then we need the type to undo [10:55] I wonder how painful it is to add returning the type to SetupSnap [10:55] and how tisky [10:55] risky even [10:56] jdstrand: updating the machine (kernel) fixed that issue, thanks. [10:57] mvo: I have a feeling this issue, whatever it is, is fixed in mainline [10:57] so it _might_ be in ubuntu sauce [10:57] Chipaca: I'll try to see what I can do [10:58] or just not present in mainline [10:58] I'm on a mainline kernel now [10:58] pedronis: you could even return the whole info [10:58] pedronis: and avoid that readinfo entirely [10:58] I'll leave it running for 15 minutes [10:58] ah, but we only use the type [10:58] what would be the idiomatic way to disable just one service in debian/rules? [10:58] pedronis: now i gotcha [10:58] but even if it is raising, it's doing so at a far slower pace [10:59] the readInfo is realy a check, it this thing mounted at all [10:59] mborzecki: carg-cult copying of a 2MB bash file, probably [10:59] but yes we need only the type [10:59] jdstrand: ^ FYI [11:00] zyga: the 8GB was also on mainline, and the difference now is just reloading in a loop? [11:00] Chipaca: AFAIK we never used mainline kernel before [11:00] on the ubuntu kernel we quickly eat memory [11:00] zyga: so the 8GB was on ubuntu kernel? [11:00] yes [11:03] *cough* is the forum inaccessible to everyone, or just me? [11:03] sparkiegeek: oh? [11:03] I _just now_ posted a message [11:03] https://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101/11?u=zyga [11:03] zyga: The certificate expired on 12 April 2018, 11:17. The current time is 12 April 2018, 12:01 [11:03] zyga: with HSTS enabled [11:03] oops :) [11:03] perhaps my browser is more lousy [11:03] i also don't see that fwiw [11:04] niemeyer: ^ FYI, it looks like forum cert is has expired [11:04] https://www.ssllabs.com/ssltest/analyze.html?d=forum.snapcraft.io&latest [11:04] confirms that it's expired [11:04] after running for ~10 minutes my memory usage is below 2GB [11:07] speaking of expiry, i need to go make lunch for the tribe [11:07] ttfn [11:07] Chipaca: bon appetit [11:08] zyga: Yeah, I was going to make sure it was renewed yesterday and missed it... [11:08] I'm on it [11:09] woot, thanks [11:11] are you aware of expired cert for https://forum.snapcraft.io ? [11:11] vidal72[m]: has just been fixed [11:12] sparkiegeek: I can confirm, thx [11:19] PR snapd#5038 opened: data/systemd: helper service for waking up the main snapd service [11:19] popey: nice! [11:20] zyga: yes, posting your findings to the mailing list would make sense, or adding to https://bugs.launchpad.net/apparmor/+bug/1750594 [11:20] Bug #1750594: Eventual OOM with profile reloads [11:20] jdstrand: nice, thanks [11:22] jdstrand: I found some interesting things already: https://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101/11?u=zyga [11:22] and sent some silly patches to apparmor tree [11:28] zyga: I see that. that should help jjohansen. I also summarized the email I sent yesterday in that topic [11:28] I see that, thanks [11:29] mwhudson: yeah, it seems to be the case as it is not an isolated incident [11:30] sergiusens: reading the patchelf code makes it seem a bit ... optimistic, maybe? [11:33] mwhudson: eventually I will have to do what I wanted to avoid doing when tasked to work on classic confinement and just master that code base and elf [11:34] sergiusens: i learnt waaaay too much about all that stuff doing go shared libraries [11:34] it's fun, mostly :) [11:34] PR snapd#5039 opened: overlord/snapstate: use the readInfo in doMountSnap as a check only, undo if it errors [11:34] Chipaca: mvo: digging deeper ^ [11:35] sergiusens: have you read ian lance taylor's blog? https://www.airs.com/blog/archives/38 through https://www.airs.com/blog/archives/57 [11:36] I have not; adding to the reading list. Thanks! [11:42] cjwatson: hey, sorry if the question was probably already asked (it was probably already done), but is there any way to get daily build of a snap (even if the branch didn't change, as a snapcraft.yaml can refer to multiple repos) on https://build.snapcraft.io/ or launchpad? [11:43] I guess the workaround can be to use launchpad API to start a build, but I would prefer something more integrated [11:43] PR snapcraft#2065 opened: many: update the yaml loading logic [11:44] didrocks: i personally use lp-build-snap (a snap) to poke the launchpad API daily via a cron job to do that [11:44] build does do daily builds now AIUI. [11:45] popey: oh? (well, I guess it's still building on xenial, which is an issue for me), but I didn't find any option [11:45] it doesnt have to be building on xenial [11:45] you can pick in your repo what it builds on [11:46] on build.snapcraft.io? [11:46] no, launchpad [11:46] I dont use build, because i need the flexibility launchpad offers [11:46] ah yeah, for this, sure :) [11:46] build isn't supposed to cover every use case. [11:46] "build does do daily builds now AIUI" -> you meant launchpad or build.snapcraft.io? [11:47] * cachio afk [11:47] build [11:48] ok ;) so yeah, I have to use launchpad and having a ping script (like lp-build-snap) seems necessary [11:48] thanks popey ;) [11:48] np [11:50] PR snapd#5037 closed: snap: snap.AppInfo is now a fmt.Stringer [11:58] finishing up with spread test [11:59] didrocks: the above all seems correct. [12:00] thx for confirming cjwatson [12:04] off to pick up the kids, coming back for standup [12:15] jdstrand: some more leak findings on the forum [12:15] PR snapd#5040 opened: overlord/snapstate: poll for up to 10s if a snap is unexpectedly not mounted in doMountSnap [12:19]