zygaHey hey05:12
mborzeckitoday's gonna be interesting05:13
zygaWhy? :-)05:22
=== cpaelzer__ is now known as cpaelzer
mborzeckizyga: alleged memleak, or rather trying to find out if we are the memory hogs05:36
mborzeckipulled down some random ubuntu core 16 image, came with snapd 2.32.5, can't refresh core because no updates available, wth?06:07
wgrantmborzecki: core updates have been disabled for the last 14h due to the memory hog situation06:17
wgrantBut I'm minutes away from reenabling them06:17
mborzeckiwgrant: thanks for the info06:17
zygaCertainly more likely than pushing a deb to a repo06:29
mborzeckithis is actually quite interesting 2.32.5 vs 2.38 vs 2.37.4/2.38 classic https://paste.ubuntu.com/p/Vs65Wyg8XW/06:29
wgrantmborzecki: core should be refreshable now.06:29
zygaI’m with the dog outside06:30
zygaI’ll check back in the office06:30
mborzeckiwgrant: thanks, updating :)06:30
zygamborzecki: I have an idea06:31
zygaIs it hotplug?06:31
mborzeckizyga: idk, we can't turn off the feature anyway06:34
mborzeckizyga: fwiw, this doesn't look so bad06:34
=== ricekrispie2 is now known as ricekrispie
pedronismborzecki: to be clear, we more looking at kernel memory, than process memory at this point06:35
zygaDid we change compiler version?06:36
pedroniszyga: of course, remember we went from go 1.6 to go 1.10/1.1106:36
mborzeckipedronis: yeah, what i mean is that there doesn't seem to be an easily visible regression06:36
mborzeckipedronis: 2.32.5 was surely go 1.606:37
pedronisthink so06:37
zygaGo changed the memory model recently06:39
zygaWell it maps much larger structures into memory without compression06:39
zygaBut the garbage collector also changed significantly06:39
pedroniszyga: 6621 has two reviews,  and is now used, could just land, no?06:39
pedroniszyga: the question is, does it make the kernel use more space06:40
pedronisanyway I made a suggestion in email to test that to mborzecki06:41
mborzeckizyga: also memory is given back gradually in background, so rss will decrease but only after a while, we could call https://golang.org/pkg/runtime/debug/#FreeOSMemory though if the memory is lost elsewhere all this is in vain06:41
zygapedronis: I think Chipaca wanted it to have an approved dependency first06:41
zygaBut I will look06:41
zygaStill afk06:41
zygapedronis: to the best of my knowledge it does use more kernel memory but it is a static, compile time decided cost06:43
pedroniswe need to find out how much it is on arm 3206:44
zygaIt is pretty huge AFAIK, dominates binary size06:45
zygaIt is not a growing number06:45
pedroniswe are probably talking past each other06:46
mborzeckipmap on 2.35.2 and 2.38 https://paste.ubuntu.com/p/xdm4tFhDwz/06:47
mborzeckihm .text grew, idk what's the r---- section of snapd, but it changed a bit, rw--- snapd is probably .bss, there's a change there, but rather insignificant06:48
pedronismborzecki: you have email06:52
pedronismborzecki: as I said, I don't think there is something obvious happening that relates only to user space mem06:53
pedronisno mvo yet to sync06:54
mborzeckipedronis: btw. rebuilding with 1.6 might be some effort, we made a transition to "context" and updated some http request context related things in the process06:56
mborzeckipedronis: 1.7 is probably worth a try though06:56
pedronismborzecki: we can do the reverse06:56
pedronisbuld 2.37 with 1.1006:56
pedronisif the issue is really only go runtime06:57
zygaback in the office now06:57
pedronisit might be apparent just that way06:57
mborzeckipedronis: right, that too06:57
pedronis(it might be a combination of a lot of things, then it's harder)06:57
zygamborzecki: good idea on 2.37 with 1.1006:57
=== pstolowski|afk is now known as pstolowski
mborzeckiok, got 1.11 set up, and it's quite funny, https://paste.ubuntu.com/p/YFTYBSTmPJ/07:08
mborzeckigoing to try 1.10 next07:08
zygamborzecki: ha07:15
zygafun indeed07:15
pedronismborzecki: interesenstingly though vsz is somewhat larger for 2.38 (not a problem, unless it really cuts into kernel resources though). anyway reminder the devices are arm 3207:16
abeatomvo, morning!07:17
abeatomvo, which is the current status of the systemd daemon-reload race? is the fix already in stable?07:18
* zyga is super happy07:19
mvoabeato: the systemd fix went to the distro yesterday, it will be part of the next stable core07:21
mvoabeato: if its import for the customer we can do a .1 with just these changes07:21
abeatomvo, hm, when is the next stable core snap expected?07:22
mvoabeato: 4-6 weeks if nothing else comes up07:23
mvoabeato: because 19.04 is released soonish we may do a .1 if we find any bigger issues this is why its a bit up in the air right now07:23
mvoabeato: we are debugging a customer issue that may require a .1 but its a bit unclear where we stand on this right now (i.e. if its a snapd, kernel, $unknown issue)07:24
abeatomvo, ok, I do not think it is extremely urgent, will let you know if that changes - it depends mostly on when we release the GA image07:24
mborzecki2.37 & 2.38 with go 1.10 https://paste.ubuntu.com/p/Snt9mqK3m5/07:48
pedronis~0.5G more vmem07:54
mupPR snapd#6629 closed: overlord/snapshotstate: helpers for snapshot expirations (automatic snapshots 2/4) <Created by stolowski> <Merged by stolowski> <https://github.com/snapcore/snapd/pull/6629>08:00
* zyga spends some time to manage paperwork and the endless list of open browser tabs09:08
mupPR snapd#6595 closed: tests: add regression test for systemctl race fix <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6595>10:35
zygamvo: what shall we do about core18?10:44
mvozyga: aha, sorry, I forgot about this, let me look at this10:44
zygamvo: do you want to HO?10:44
zygamight be easier10:44
mvozyga: let me quickly poke at this for some minutes to familiarize myself10:47
mupPR snapcraft#2522 closed: build providers: add API for friendly instance type names <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2522>10:49
=== cpaelzer__ is now known as cpaelzer
pedroniszyga: mvo: I need to be in that HO10:59
pedronisif/when we have one10:59
* pstolowski lunch11:03
Chipacaask me if my parents' wifi goes away when they use the microwave11:07
mvozyga: lets talk after the standup, I have lunch now and then I have a meeting, maybe if this meeting is shorter we can also talk before11:12
zygamvo: ping me when you are ready to talk11:12
mvozyga: but yeah, we need to decide, I think I hvae a slightly better idea now what is going on.11:12
mvozyga: thanks again for raising this!11:12
pedronismvo: zyga: ping me when you have it11:13
* Chipaca ⇝ lunch11:23
mupPR snapd#6691 opened: Move extractkernelassets <Created by kubiko> <https://github.com/snapcore/snapd/pull/6691>11:31
* zyga break11:32
zygaondra: reworded that PR a little11:33
ondrazyga yeah more fitting11:34
ondrazyga thanks :)11:34
* zyga really break now11:40
zygaSon_Goku: hey, on F30 I cannot get snapd update because of some gnome-software dependency mismatch,12:07
zygaSon_Goku: is that expected?12:07
Son_Gokuwhat are you seeing?12:08
zygaI don't have that machine with me but it was saying something that gnome-software depends on snapd-auth-provider or something to that vein12:09
zygaand that was it, no 2.3812:09
Son_Gokusnapd-login-service dependency is still in gnome-software12:09
Son_Gokudamn it12:09
Son_GokuI have to put it back now12:10
zygaSon_Goku: so what just happened?12:11
Son_Gokuwell, I decided to rip out an old Provides12:11
Son_Gokuwhen snapd-login-service was retired, I obsoleted it by snapd12:11
Son_Gokubut gnome-software-snap requires it still :(12:11
zygasorry I didn't vote on the update before, I noticed others did12:12
zygabut those must have been headless machines12:12
zygamy laptop complained12:12
mborzeckizyga: yup i tried the update in a headless vm12:22
pstolowskimborzecki: https://github.com/snapcore/snapd/pull/6661 has 3 +1s, can land?12:24
mupPR #6661: data/selinux, tests/main/selinux-clean: fine tune the policy, make sure that no denials are raised <SELinux> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6661>12:24
mborzeckipstolowski: i'm pushing the fix for the test zyga suggested12:24
jdstrandoSoMoN: hey, I'm still slogging through the chromium approvals. it is a large snap so it takes a while for the next revision to be reviewed after I approve one12:26
oSoMoNjdstrand, thanks!12:28
* Chipaca shakes his fist at doLinkSnap12:36
mvozyga, pedronis I am free now if you want we can chat about /var/lib/snapd/void and friends12:43
pedronismvo: I can join standup12:44
mvopedronis: I'm there now, ready when you are12:45
mvozyga: -^12:45
zygamvo: same12:47
=== ricab is now known as ricab|lunch
mupPR snapcraft#2520 closed: snap: set core as a base <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2520>13:28
mupPR snapd#6616 closed: boot: add flag file "meta/force-kernel-extraction" <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6616>13:30
mupBug #1823343 opened: Getting Started Intro out of date <Snappy:New> <https://launchpad.net/bugs/1823343>13:51
mupPR snapd#6692 opened: interfaces: cleanup internal tool lookup in system-key <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6692>14:08
mupPR snapd#6693 opened: cmd: tweak internal tool lookup to accept more possible locations <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/6693>14:09
=== ricab|lunch is now known as ricab
mupPR snapcraft#2523 opened: ci: improve travis integration conditionals <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2523>15:16
Chipacapedronis: you can't have an alias for a snap name, right?17:04
noise][Chipaca: correct, no such thing17:07
Chipacanoise][: I meant, an alias in one snap that was the name of a different snap17:07
Chipacanoise][: like, aliasing tree-classic to 'tree', which is the name of a snap17:07
pedronisChipaca: you can but is very much discouraged because you cannot then install conflicting snaps around it in any way at the same time17:10
pedroniswe don't let you override the commands of another snap17:11
Chipacaso this tree-strict and nano-strict thing is not gonna fly17:11
noise][oh, i think i misunderstood17:11
pedronisChipaca: it depends, it means you cannot install nano and nano-strict at the same time17:12
Chipacapedronis: right, but what happens if you try?17:12
noise][but you wouldn't17:12
pedronisyou get an error17:12
pedronisabout namespace conflicts17:13
Chipacai mean i'd like to see that ux17:13
pedroniswhen you install the 2nd17:13
Chipaca… but i have enough on my plate already17:13
noise][but the existence of `tree`, not installed, does not preclude a snap tree-strict from aliasing 'tree' , right?17:13
pedronisit's up to the alias reviewer in the store17:14
pedroniswe didn't implement strict checks there about this17:14
ChipacaI should clarify in the forum then17:14
Chipacapedronis: what do you mean by 'strongly discouraged' btw17:14
Chipacabecause the people reviewing and approving aliases didn't even bring it up17:15
pedronisthat if the snaps  are unrelated17:15
pedronisyou will just bring confusion into the world17:15
pedronisa snap owns its name17:15
pedronisyou are basically giving it away17:15
Chipacapedronis: that doesn't sound like 'strongly discouraged' beyond the abstract sense. I mean, what steps have been taken to discourage it?17:16
noise][it seems perfectly reasonable to me to allow for 'foo' and 'foo-classic' snaps that are the same thing but one with classic confinement.17:16
pedronisnoise][: there yare related though, which is actually not a typical case at all17:17
noise][for the case outlined with tree or jq or whatever where there's a desire to be able to use them on Core systems17:17
pedronisat it wasn't a typical case when aliases were designed17:17
noise][both from the same publisher17:17
noise][or i mean that would be the preferred scenario17:17
pedronisChipaca: the guideline around aliases as always been to avoid clashes17:19
brlinpopey suggested using a separate `classic` track for the alternative classic confined versions, which may mitigate the problem17:19
pedronisChipaca: that included with other snap names, maybe is not splelt out but was in our mind17:20
brlinFor the alias problem if there's a popular fork the same name software, like `blender-some-popular-edition`, people might want to make alias of the original name17:21
Chipacabrlin: o/17:21
Chipacanow I just want to have a snap with an alias for another snap just to see how it explodes17:22
pedronisChipaca: I can point you at the errors17:23
noise][classic confinement is at the snap level, can't be per-track17:23
pedronisnoise][: classic confinement just means you can have classic confinement in some revisions17:23
pedroniswhether we enforce more I don't know17:24
pedronisChipaca: to be clear we didn't design thinking that this would be a common case that we would need to guide people around17:24
noise][right, i mean we don't have a way to selectively enforce17:25
Chipacabrlin: updated both topics with a summary of the above. I don't think [revoked] is correct … even if I were correct, i'm not a reviewer :-)17:25
brlinnoise][: Then I guess the separate name space and overriding alias is the only way to go :-/17:25
Chipacato be clear, _allowing_ classic is not per-track, but a snap being classic is per revision17:25
noise][what Chipaca said :)17:25
Chipacaso you _could_ have a channel be classic and a channel be strict, but you could ship a classic one in the strict track by accident17:26
noise][this certainly could bear more discussion, but it's Friday afternoon17:26
noise][yeah, so it's possible we could allow that route for some snaps and just have to trust the publisher not to break people on the strict track17:26
Chipacaalso bears don't like discussion17:26
noise][and then at some point add guardrails, but i'd like to explore this more before recommending such an approach17:27
pedronisChipaca: errors are here  https://github.com/snapcore/snapd/blob/master/overlord/snapstate/aliasesv2.go#L339 and here https://github.com/snapcore/snapd/blob/master/overlord/snapstate/aliasesv2.go#L36817:27
brlinThe snap edit page on LaunchPad seemed to support this fashion AFAICT https://usercontent.irccloud-cdn.com/file/prUHOKkw/image.png17:28
brlinFYI the users of the `nano` snap are currently blocked from auto-updates due to the confinement switch17:31
Chipacayeah, confinement switches are nasty17:32
Chipacathey'll be getting warnings at some point17:32
noise][that's why, e.g. with tree, that i suggest not messing with confinement of the existing snap17:32
Chipacathat's #1811063 FWIW17:32
mupBug #1811063: "snap refresh" does not report failure to update because snap switched to classic confinement <snapd:In Progress by chipaca> <https://launchpad.net/bugs/1811063>17:33
noise][anyway, i need to run, happy weekend!17:33
Chipacasame here17:33
Chipacabrlin: ttfn, have a good one17:33
pedronisalso to be clear you can install both snaps, but you need to manouver around, and the one with the actual name (vs the alias) needs to win17:33
pedronis(you'll need to use snap unalias and/or install --unaliased)17:33
brlinChipaca: Bye!17:34
mupPR snapcraft#2523 closed: ci: improve travis integration conditionals <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2523>17:55
cachiomvo, hey18:12
cachiomvo, I know which is the problem with the install --dangerous in core1818:13
zygacachio: tell us!18:31
cachiozyga, so, the problem is in the tests18:33
cachiobasically when it is installing --dangerous is downloading the core snap as part of that process18:34
cachioeven when we have the core snap in /var/lib/snapd/snaps18:34
zygathat's very curious18:34
zygacachio: thank you18:34
cachiozyga, do you know if snapd is searching in the state to define if a snap could be in the cache?18:36
zygacachio: I don't know yet, we will check it out18:36
zygasnapd uses the cache for sure18:36
zygabut repackaged core has different cache18:36
zygaso perhaps that's one issue18:36
cachiozyga, https://paste.ubuntu.com/p/J3w8bxrqfm/18:38
cachioin the first install it downloads the core snap, even when it is already in /var/lib/snapd/snaps/18:38
cachiobut second time it is not downloading it18:38
cachiozyga, I don't know why18:39
zygacachio: what is in /var/cache/snapd?18:39
zygathe one in /var/lib/snapd/snaps is, I think, irrelevant now18:39
zygathe one in /var/lib/cache/snapd matters18:39
zygaI mean, irrelevant only if not installed18:39
cachiozyga, https://paste.ubuntu.com/p/qvj5QCgkwK/18:40
zygathat's odd18:40
zygaso some back story18:40
zygalast week, for local testing in qemu18:40
zygaI wrote some patches that let me seed /var/cache/snapd with heavy snaps18:41
zygathen tests only fetch assertions online, speeding up testing tremendously18:41
zygathe way I did this was by keeping /var/cache/snapd.tar.gz in the qemu image and unpacking it early in the test prepare18:41
zygait works for that purpose18:41
zygathe files there are not .snap files but instead long file names based on some hash of the content18:42
zygasnapd uses that cache as hardlink source AFAIK18:42
zygaso I'm surprised to see none of that in the cache directory here?18:42
zygaI'm mistaken18:42
zygathe path is different18:42
zygathe correct path is /var/lib/snapd/cache18:43
zygathat's the one18:43
zygaI was arguing with chipaca to change it because it's odd that we have two cache directories18:43
zygaplease look at /var/lib/snapd/cache18:43
zygabefore / after snap install --dangerous18:43
cachiozyga, seems to be there18:44
zygathat's more like it18:44
zygawhat happens when you remove and install a snap?18:44
zygait should reuse that cache and do things "instantly"18:44
cachiozyga, the core is not installed in my machine but it is in cache dir18:46
cachioso, I think I know what I need to change to cache snaps in the tests18:46
zygaI will check later (eating dinner now) but perhaps --dangerous implies some of the hash based logic doesn't get used18:46
cachiowhat we are doing currently doesn't work18:46
zygacachio: cool18:46
zygacachio: looking forward to that patch :D18:47
cachiozyga, --dangerous works well18:47
cachiobut basically we need to install/remove the snap to cache it before we save the state18:47
cachiothat should be enough18:47
cachiocurrently we are copying to /var/lib/snapd/snaps the .snap file but it is not enough18:48
zygacachio: as a hint18:48
zygacachio: perhaps to speed things up more18:48
zygacachio: instead of creating a tarball of /var/lib/snapd/cache18:48
cachiozyga, yes, could be, I'll make then change18:49
zygacachio: make a directory, say /var/lib/snapd-cache18:49
cachiozyga, thanks for the help18:49
zygacachio: wait :)18:49
zygacachio: and hardlink the cache items there18:49
zygacachio: then instead of unpacking that tarball18:49
zygacachio: hardlink them back18:49
zygacachio: this will be instant18:49
zygacachio: and will work just like a tarball would18:49
cachiozyga, perfect, I'll do that18:49
cachiothanks for the info18:50
zygacachio: cp has some options, perhaps there are some that do hardlink copies18:50
zygacachio: good luck!18:50
cachiozyga, thnanks18:51
cachioenjoy your weekend18:51
zygalikewise :)18:51
pedronis--dangerous doesn't use the cache18:54
pedronis(the cache is for downloads)18:54
zygapedronis: oh?18:54
pedroniswhy would it18:54
pedronisyou are giving it a snap here18:54
zygapedronis: for dependencies18:54
zygapedronis: it downloads core over and over18:54
pedronisthat is stranger18:54
cachiopedronis, I think install --dangerous is using the cache19:22
cachioI just see that the core is not being downloaded it is correctly cached19:22
cachiopedronis, the problem we have is that we are caching the snaps in the wrong way on our test suite19:24
cachioso when I run on the boards and vms 5/6 in parallel in my home all the installs we do are downloading hte core19:25
cachioand this is the bottleneck which is causing the install --dangerous is getting stuck19:25
mvo6687 needs a second review, should be simple19:45
mupPR snapd#6689 closed: tests: run create-user on core devices <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6689>19:45
mupPR snapd#6694 opened: tests: improve how snaps are cached <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6694>20:38
* cachio afk20:43
mupPR snapcraft#2524 opened: Snapcraft try <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2524>21:32
jdstrandpedronis: ok, I got through all the store review stuff and knocked out 5 PR reviews (well, one was asking for a different approach). I will do PR 6502 and PR 6605 on monday21:45
mupPR #6502: dirs,overlord/snapstate: add Soft and Hard refresh checks <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6502>21:45
mupPR #6605: cmd/libsnap,osutil: fix parsing of mountinfo <Created by zyga> <https://github.com/snapcore/snapd/pull/6605>21:45
jdstrandzyga: fyi ^21:45
jdstrandthere is a small chance I'll look at those over the weekend, but let's say 'by my eod Monday'21:46
jdstrandpedronis: I also responded to https://forum.snapcraft.io/t/phase-1-of-opt-in-per-snap-users-groups-aka-the-daemon-user/10624/1521:46
ijohnsonjdstrand: hey! the dockerd daemon now needs to use data inside $XDG_RUNTIME_DIR, which ends up being /run/user/0 for the daemon, but this dir isn't created... what's the eventual resolution to lp #1656340 do you think?21:52
mupBug #1656340: XDG_RUNTIME_DIR is not created on app startup <bionic> <cosmic> <eco-team> <xenial> <Snappy:Triaged by zyga> <snapd (Ubuntu):Confirmed> <https://launchpad.net/bugs/1656340>21:52
ijohnsonfor now I will just overwrite that to be $SNAP_COMMON/run and it works, but just curious as I seem to recall discussion around that bug recently on this channel21:52
=== Katnip- is now known as Guest17428
=== ondra_ is now known as ondra

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!