/srv/irclogs.ubuntu.com/2018/04/12/#snappy.txt

=== chihchun_afk is now known as chihchun
zygao/04:40
mborzeckimorning05:10
zygamborzecki: hey05:13
zygahttps://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/495405:13
mborzeckiyup reading05:13
mborzeckizyga: any clue if mvo got access to a vm where the refresh-mode issue was reproducible?05:16
zygano05:16
zygalet's make a spread test that has similar characteristics05:16
mborzeckizyga: do we know what disto tha was?05:16
zygaand strace the hell out of it05:16
zyganot really05:17
zyganote that we have not reproduced it yet05:17
zygabut I will let mvo discuss tihs05:17
zygahe knows the most05:17
mborzeckihm that memory usage, we could play with GOGC, but there's another possibility that we're just keeping that much stuff in memory05:21
mborzecki(i'm taking about RES, VIRT is high but it should not be a problem)05:22
zygamborzecki: yes, RES is key, though VIRT is weirdly huge05:47
zygabut maybe that's the nature of golang gc05:47
zygaok, kids off to school05:47
zygaand I can have my coffee and focus05:47
mborzeckigdm is just so useless, it stats a whole gnome session, does nothing else than shows a login screen and has a 50% chance of locking up the system when switching back to it after logout06:32
mborzeckiat least autostart works as expected with 2.32.406:33
zygaHeh, the modern desktop stack06:33
mborzeckibtw. clicking on quit label in tray menu of discord: https://paste.ubuntu.com/p/CtG3Kx7Kxt/06:34
=== katnip- is now known as Guest67740
zygaWell06:39
zygaOn my system gnome shell logs errors every second06:39
zygaFinalization issue with some object06:40
zygaI frankly miss unity quality06:40
zygaLater on unity was way more solid06:40
=== Guest67740 is now known as katnip
mborzeckiok, bumped arch package to 2.32.406:51
mvogood morning06:52
mborzeckimvo: morning!06:52
zygahey mvo06:53
mvohey mborzecki and zyga ! how is life this morning?06:55
zygagreat, it's a bit chilly in the morning though06:55
zygaI'm looking into the memory leak06:55
zygabut if I get stuck or it goes nowhere I will look at the mount issue06:56
pedronishello06:56
zygahey good morning06:57
pedronisso I discovered that init is 2M and snapd nowadays 21M , just the executable  (of course systemd has other daemons)06:58
zygaI really wonder what is inside golang executables06:58
zygasmall instance of google Borg?06:58
pedroniswell, as I said,  systemd has many daemons06:58
pedronisso it's not a fair comparison06:58
pedronisbut it doesn't put in a good spot06:59
pedroniswe have not to run at all, not to use memory06:59
mborzeckithat ~21M would mostly count to virt though07:00
pedronismborzecki: looking at smaps it doesn't seem so07:00
mborzeckibut it still needs to be loaded into memory and so on07:00
pedronisanyway on a constrained system probably even running and stopping is bad07:01
pedronismvo: let me know if we should have a HO in 1h or so to talk about that issue at least and options07:03
mvopedronis: yeah, sounds good a HO in the morning sounds good, I am currently baby-sitting autopkgtest but hopefully that does not take too long07:04
pstolowskimorning07:06
zygahey pawel07:06
zygahttp://paste.ubuntu.com/p/GtVYbzfgC8/07:06
zygapstolowski: FIY https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/495407:06
pstolowskizyga: yes reading07:07
zygawith GOGC=10 http://paste.ubuntu.com/p/Gbnz675BT3/07:13
mborzeckiplaying with GOGC, off -> 33MB, 5 -> 19MB, 50 -> 20MB, 100 (default) -> 22MB07:13
mvozyga: what tool is that?07:17
zygapmap07:17
* mvo nods07:17
mborzeckizyga: comparison of GOGC=1 and GOGC=100 https://paste.ubuntu.com/p/74zgkbZBWj/07:19
mborzecki(disabled ASLR so that we get stable addresses)07:19
zyganice07:19
zygahow did you disable ASLR?07:19
mborzecki/proc/sys/kernel/randomize_va_something07:20
zygabut the bottom line is that rss shrinks just a little and virt doesn't07:20
mwhudsonyou can also disble aslr for one invocation with setarch07:20
mwhudsonsetarch `uname -m` -R whatever07:21
mborzeckinice07:21
mborzeckiwonder why there's 7 * 64MB and 6*8MB blocks, 64MB ones are ---p, 8MB are rw-p07:26
mupPR snapd#5035 closed: release: snapd 2.32.4 <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/5035>07:28
mvogood news, spread on xenial is happy again and we have some tests passing already, the other ones are retriggered07:29
mvo(missing build of test-snapd-account-services on s390x was one failure which is fixed now)07:29
zygamborzecki: noite that the 64MB ones are empty07:31
zygathey take almost no memory07:31
mborzeckizyga: cow?07:32
zygathey feel like a/b gc07:34
mborzeckihm bottom line, we're not going to get lower than whatever is the size of snapd binary, i.e. ~16MB07:35
mborzeckizyga: installing core snap https://paste.ubuntu.com/p/dQDYNJz2Vf/07:38
mborzeckialso the binary in core built with go 1.7 right?07:41
mborzeckizyga: before and after installing core snap and reexec https://paste.ubuntu.com/p/xPpCQjV4j8/07:41
pstolowskizyga, mvo, mborzecki would it make sense to add profiling code to snapd and land it in edge, eg something like https://flaviocopes.com/golang-profiling/ in the daemon?07:56
mborzeckipstolowski: there's pprof which is super simple07:58
zygapstolowski: I don't have experience with that, no idea07:59
mborzeckii can try building it locally though08:00
pstolowskiyeah i'm going to try local build too08:01
zygaman we are leaking that memory quickly08:11
zygaI'm in a loo where test-snapd-poicy-app-consumer:bluez is constantly discon reconnected08:13
zygathis also stresses udev08:14
mvomborzecki: the binary in core is build with go1.6 (xenial)08:15
pedronismvo: I'm around08:15
mvozyga: kernel mem?08:15
mvopedronis: ok, I join in 2min then08:15
zygamvo: perhaps, I'm not sure yet08:15
zygainterestingly doing this puts lots of load on X as well08:17
zyga(perhaps due to udev)08:17
zygato the point where session is unresponsive (not stuck but very slow)08:17
zygaI see constantly raising buffer_head allocations08:17
zygaI see several dozen udev processes at a time08:20
zygathe leak may be unrelated to apparmor if udevadm trigger / settle leaks something into the kernel / netlink area08:20
mwhudsonBjornT_: hi, do you have a ppa with a snapcraft that has https://github.com/snapcore/snapcraft/pull/2058 applied by any chance?08:22
mupPR snapcraft#2058: python plugin: install python-distutils when run on bionic <bug> <Created by bjornt> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2058>08:22
BjornT_mwhudson: no. but it's in the snap edge channel now08:24
mwhudsonBjornT_: hm but you can't build a snap in launchpad from a snapcraft from a snap can you?08:25
mwhudson(that sentence is quite a tongue twister)08:26
BjornT_mwhudson: right, you can't. but i'm guessing you will have problems building a bionic snap in launchpad anyway, won't you?08:29
Chipacamvo: I saw you put #1616629 into 'in progress', does that mean you have a branch for it?08:32
mupBug #1616629: could not unmarshal state entry "snap-type" <Snappy:In Progress> <https://launchpad.net/bugs/1616629>08:33
mwhudsonBjornT_: nope, been doing that for months :)08:46
mvoChipaca: it means "we are looking at it", pedronis  is working on improving the error at least to fail way earlier08:58
Chipacamvo: ah, ok08:58
pedronisChipaca: one theory is that it relates to removing and reinstalling   (maybe we the fs of the snap in use) and lazy unmounting08:59
Chipacapedronis: was that what pindonga did yesterday?08:59
pedronisyes, he said he had removed the snap09:00
pedronisChipaca: anyway current plan is 1) improve error, 2) possibly add belt and suspenders 3) somebody might want to play with loops of remove and install of the same thing09:00
Chipacai can start on 3 right no09:01
Chipacanow*09:01
pedronisthx09:01
pedronisI will work now on 1)09:02
mvozyga: here is something strange. this is the LowMem value after running each of the tests, the tests are in order http://paste.ubuntu.com/p/k8wvw689B9/09:06
mvozyga: the fluctuation is also strange09:07
zygaI suspect what is going on is that as memory pressure raises kernel is dropping more and more caches09:07
zygareleasing some memory in that zone09:08
mvozyga: aha, yeah, that makes sense09:08
mborzeckipstolowski: hooked up pprof finally, https://paste.ubuntu.com/p/XXBv8qFFym/ :6060 is the regular http one, the one over /run/snapd.socket you need to wrap around with socat09:09
mborzeckipstolowski: something like `socat -v tcp-listen:6070,reuseaddr,fork "unix-connect:/run/snapd.socket"` should work09:11
pedronismborzecki: pstolowski:  we discussed with mvo some ideas about not starting at all  , it seems hard to  halve or /3 memory usage in one week09:12
mborzeckipedronis: especially as the binary is 16MB already ;)09:13
mborzeckipedronis: so what are the current ideas?09:13
pedronismborzecki: mvo is working on that I think,   but basically only play at the level of the systemd units09:14
pstolowskimborzecki: thanks, checking; i was pursuing something else in the meantime - run 2.32.4 snapd under gcvis but didn't find anything conclusive09:15
Chipacais it still the case that a snap can't call a binary in another snap?09:15
mvomborzecki: the idea is to use a generator that looks if there is a snap or a seed and if none of this is true generates a service file that is socket activated only09:16
mvomborzecki: this way we do not run at all on cloud instances until first use09:16
mvomborzecki: the downside is that we don't get the c-n-f data so we need to put that into an external timer09:16
mvobut I think its an acceptable trade-off given the time contrains09:17
mborzeckisounds fair09:18
mborzeckimvo: do you need help with that?09:18
mvomborzecki: sure, I also have the sigterm issue that needs attention, if you want to add the generator that would be great.09:19
mvomborzecki: we have two options: a) generator b) snapd is also socket activated and we add a "wakeup" service that has conditions on /snap/* and /var/lib/snapd/seed/*09:20
mvomborzecki: and this wakeup would just run snap version to ensure the daemon runs09:20
mvomborzecki: plus one idea from zyga was to use the "idle" option of systemd so that snapd starts a bit later (when it starts) when the system has settled09:21
mvomborzecki: sounds sensible? happy to have a quick HO if you want09:21
mvomborzecki: alternatively the SIGTERM issue is something I am willing to trade too, whatever you find more interessting :)09:21
pedronisto be clear: /snap/*   OR  /var/lib/snapd/seed/seed.yaml09:22
pedronisnot AND09:22
mborzeckimvo: let's do HO09:22
Chipacapedronis: /snap/*/current i guess09:23
pedronissomething09:23
pedronisit's really /snap/* ignoring bin09:23
pedronisand also /snap is not always snap09:24
Chipacathe problem with doing install/remove in a loop is that i quickly remember how badly that grows :-(09:26
Chipacaloop #1: ~1s; loop #200: ~6s09:28
mupPR #200: Improve error handling in client package <Created by zyga> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/200>09:28
Chipacamup: er... thanks09:28
mupChipaca: I apologize, but I'm pretty strict about only responding to known commands.09:28
mvomup: #109:36
mvo*pfff*09:36
Chipacamvo: istr #1 being an issue, which are now gone09:41
* cjwatson is lost in a twisty maze09:44
cjwatsonIs there some way to force snapd to forget about its existing store device macaroon?09:44
cjwatsonIt must be serialised to disk somewhere to persist between snapd startups, but I can't find it09:44
mvoChipaca: yes, gone, let me update the post09:45
* zyga builds 4.1709:46
zygawell, master09:46
pedroniscjwatson: it's in   state.json  under  data.auth.device09:46
zygakissiel: /me hugs his computer :)09:46
pedroniscjwatson: session-macaroon09:46
Chipacamvo: which post?09:47
pedroniscjwatson: /var/lib/snapd/state.json.  stop and restart snapd for editing09:47
cjwatsonAha, thanks09:47
mwhudsonmvo: np on go 1.6/stable, i'm just waiting for the coinstallable tracks feature :)09:50
mwhudsonzyga: would you be interested in my "use x/net/context instead of context" sed of doom? :)10:00
mwhudsonfind -type f -name \*.go \! -path '*/.pc/*' -print0 | xargs -0 sed -i -e 's%^\(\s\+\)"context"$%\1"golang.org/x/net/context"%'10:00
mwhudsonzyga: it works on docker.io :)10:00
zygamwhudson: across the whole tree?10:01
zygamwhudson: well, I don't mind that, though I don't see how that helps us this week (fire squad)10:01
mwhudsonzyga: i saw something about spread not building with go 1.6?10:01
zygaah10:01
zygamvo fixed that earlier today10:02
mwhudsonok10:02
mwhudsonnever mind me then :)10:02
zygamwhudson: but thank you for the offer10:02
zygawe are working through this list: https://forum.snapcraft.io/t/issues-to-address-ahead-of-ubuntu-18-04-release/4954/110:02
mupPR snapd#5036 opened: overlord/snapstate: allow to get an error from readInfo instead of a broken stub, use it in doMountSnap <Created by pedronis> <https://github.com/snapcore/snapd/pull/5036>10:04
pedronismvo: Chipaca: ^10:04
mvopedronis: thank you!10:06
mborzeckihmmm /snap/README, there goes ConditionExistsGlob10:07
Chipacamborzecki: /snap/*/current10:07
mborzeckiChipaca: oh, that's good10:08
mborzeckipedronis: just to be sure, that's an /snap/.. OR /var/lib/../seed.yaml ?10:09
mvomborzecki: yes, either of those should trigger a wakeup10:09
ChipacaApr 12 11:09:21 fleet snapd[7512]: 2018/04/12 11:09:21.252174 services.go:226: DEBUG: StopServices called for [%!q(*snap.AppInfo=&{0xc820397600 webserver [] command-webserver.wrapper simple 0        map[network:0xc8203f9270 network-bind:0xc8203f92c0] map[] map[] {[] map[]} [] [] <nil>})], reason: remove10:09
mupPR #12: Bugfix/lp1480248 test reenable <Created by mvo5> <Merged by niemeyer> <https://github.com/snapcore/snapd/pull/12>10:09
Chipacahmmm10:09
mborzeckimvo: ok10:09
Chipacasomething awry in a printf :-)10:10
mvoChipaca: woah, where did you get this?10:11
Chipacamvo: snap remove a snap with a service with debug on10:13
Chipacai looped installing/removing a snap without a service and it couldn't repro 1616629, so now trying with a service10:14
pedronisChipaca: doInstall also has int flags, yes but things that are exported or go into state I agree we shouldn't use int flags10:16
Chipacapedronis: we still have a lot of int flags, and i don't think it's a blocker10:16
mborzeckiso the wakeup service works10:28
mborzeckispread test for this will be tricky10:28
zygajdstrand: hey10:31
zygajdstrand: is apparmor@lists.ubuntu.com still the place to send patches?10:31
mwhudsonsergiusens: fwiw 'd bet patchelf is doing something differently horrible to java10:32
mwhudson+i10:33
mupIssue snapcraft#1676 closed: Support for set-version <Created by sergiusens> <Closed by sergiusens> <https://github.com/snapcore/snapcraft/issue/1676>10:34
mupPR snapcraft#2063 closed: many: add snapcraftctl set-version <Created by kyrofa> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2063>10:34
pedronisChipaca: btw seems we don't have undo on error code in doMountSnap, not sure it's good time to add that tough10:38
mborzeckipedronis: where?10:39
mupPR snapd#5037 opened: snap: snap.AppInfo is now a fmt.Stringer <Created by chipaca> <https://github.com/snapcore/snapd/pull/5037>10:39
Chipacapedronis: undo as in in-task undo?10:39
Chipacamvo ^ 5037 fixes that %!q thing10:40
mvoChipaca: nice10:40
pedronisChipaca: yes, in-task undo10:41
mborzeckipedronis: around readInfo?10:41
Chipacapedronis: what are we missing?10:41
Chipacapedronis: removing the mount unit?10:42
* Chipaca looks10:42
pedronisremoving the mount unit10:42
pedronisI suppose10:42
pedronisin fairness SetupSnap is an opaque name10:42
Chipacano, we remove the unit10:42
Chipacaone rror10:42
ChipacaRemoveSnapFiles removes the unit10:42
Chipacastops and removes, even10:42
pedronisyes, but now ReadInfo itself can fail10:43
pedronis(well it always could but not materially)10:44
Chipacapedronis: ah! now i gotcha10:44
Chipacapedronis: sounds like you want a m.backend.RemoveSnapFiles there then10:45
pedronisyes, seems so10:45
Chipacaor maybe UndoSnapSetup?10:45
ChipacaUndoSetupSnap*10:45
Chipacawhich, er, just calls RemoveSnapFIles :-) but might make the intent clearer from the calling side10:45
pedronisyes10:46
zygawoah?10:49
zygaWOAH10:50
zygamvo: wanna hear a funny story?10:50
zygamy test10:50
pedronisChipaca: sadly that thing needs a type10:50
zygawhere I ate 8GB of ram pretty quickly10:50
zygathis was my command10:50
zygasudo sh -c 'for i in $(seq 10000); do snap connect test-snapd-policy-app-consumer:bluez && snap disconnect test-snapd-policy-app-consumer:bluez; done'10:50
zygaI took a snapshot of all the profiles before/after this10:51
zygabefore disconnect and after disconnect10:51
zygait was connected initially10:51
zygaand the profiles are identical10:51
zygaall the apparmor profiles on my system10:51
zygaso we were leaking all that memory _despite_ not having _any_ changes10:51
Chipacapedronis: which thing?10:52
pedronisUndoSnapSetup10:52
pedronissorry, UndoSetupSnap10:52
pedronisnow, that I look at the code, there's a lot of silliness going on10:52
zygaso, a bit of a WTF moment10:52
Chipacapedronis: :-)10:52
zygaI will write a loop that recompiles that one profile in a loop and see if that is eating any memory10:52
pedronisChipaca: SnapSetup read the type from the snap,   and the its caller tries to read the type from the mount10:52
Chipacazyga: wtf yes10:52
zygaiff it does that's also very interesting10:53
zygabecause apparmor parser supposedly does nothing when you are not really changing the profile10:53
Chipacazyga: the 8G is kernel, not snapd?10:53
zygaso perhaps simply reading one of the apparmorfs files leaks10:53
zygaChipaca: yes10:53
Chipacanoice10:53
zygaalmost all in kmalloc-102410:53
zygaI'm on mainline master10:53
zygaso that should give us all the fixes10:53
zygaI'll start by confirming the loop I used before still leaks10:54
pedronisChipaca: so apart details of how is easy it is to untangle this,   the readInfo is not stricly needed, it's a nice fence though10:54
Chipacapedronis: only if we check the error10:54
pedronisbut then we need the type to undo10:55
pedronisI wonder how painful it is to add returning the type to SetupSnap10:55
pedronisand how tisky10:55
pedronisrisky even10:55
popeyjdstrand: updating the machine (kernel) fixed that issue, thanks.10:56
zygamvo: I have a feeling this issue, whatever it is, is fixed in mainline10:57
zygaso it _might_ be in ubuntu sauce10:57
pedronisChipaca: I'll try to see what I can do10:57
zygaor just not present in mainline10:58
zygaI'm on a mainline kernel now10:58
Chipacapedronis: you could even return the whole info10:58
Chipacapedronis: and avoid that readinfo entirely10:58
zygaI'll leave it running for 15 minutes10:58
Chipacaah, but we only use the type10:58
mborzeckiwhat would be the idiomatic way to disable just one service in debian/rules?10:58
Chipacapedronis: now i gotcha10:58
zygabut even if it is raising, it's doing so at a far slower pace10:58
pedronisthe readInfo is realy a check, it this thing mounted at all10:59
Chipacamborzecki: carg-cult copying of a 2MB bash file, probably10:59
pedronisbut yes we need only the type10:59
zygajdstrand: ^ FYI10:59
Chipacazyga: the 8GB was also on mainline, and the difference now is just reloading in a loop?11:00
zygaChipaca: AFAIK we never used mainline kernel before11:00
zygaon the ubuntu kernel we quickly eat memory11:00
Chipacazyga: so the 8GB was on ubuntu kernel?11:00
zygayes11:00
sparkiegeek*cough* is the forum inaccessible to everyone, or just me?11:03
zygasparkiegeek: oh?11:03
zygaI _just now_ posted a message11:03
zygahttps://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101/11?u=zyga11:03
sparkiegeekzyga: The certificate expired on 12 April 2018, 11:17. The current time is 12 April 2018, 12:0111:03
sparkiegeekzyga: with HSTS enabled11:03
zygaoops :)11:03
zygaperhaps my browser is more lousy11:03
Chipacai also don't see that fwiw11:03
zyganiemeyer: ^ FYI, it looks like forum cert is has expired11:04
sparkiegeekhttps://www.ssllabs.com/ssltest/analyze.html?d=forum.snapcraft.io&latest11:04
sparkiegeekconfirms that it's expired11:04
zygaafter running for ~10 minutes my memory usage is below 2GB11:04
Chipacaspeaking of expiry, i need to go make lunch for the tribe11:07
Chipacattfn11:07
sparkiegeekChipaca: bon appetit11:07
niemeyerzyga: Yeah, I was going to make sure it was renewed yesterday and missed it...11:08
niemeyerI'm on it11:08
zygawoot, thanks11:09
vidal72[m]are you aware of expired cert for https://forum.snapcraft.io ?11:11
sparkiegeekvidal72[m]: has just been fixed11:11
vidal72[m]sparkiegeek: I can confirm, thx11:12
mupPR snapd#5038 opened: data/systemd: helper service for waking up the main snapd service <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/5038>11:19
jdstrandpopey: nice!11:19
jdstrandzyga: yes, posting your findings to the mailing list would make sense, or adding to https://bugs.launchpad.net/apparmor/+bug/175059411:20
mupBug #1750594: Eventual OOM with profile reloads <AppArmor:New> <https://launchpad.net/bugs/1750594>11:20
zygajdstrand: nice, thanks11:20
zygajdstrand: I found some interesting things already: https://forum.snapcraft.io/t/oom-for-interfaces-many-on-bionic-i386/4101/11?u=zyga11:22
zygaand sent some silly patches to apparmor tree11:22
jdstrandzyga: I see that. that should help jjohansen. I also summarized the email I sent yesterday in that topic11:28
zygaI see that, thanks11:28
sergiusensmwhudson: yeah, it seems to be the case as it is not an isolated incident11:29
mwhudsonsergiusens: reading the patchelf code makes it seem a bit ... optimistic, maybe?11:30
sergiusensmwhudson: eventually I will have to do what I wanted to avoid doing when tasked to work on classic confinement and just master that code base and elf11:33
mwhudsonsergiusens: i learnt waaaay too much about all that stuff doing go shared libraries11:34
mwhudsonit's fun, mostly :)11:34
mupPR snapd#5039 opened: overlord/snapstate: use the readInfo in doMountSnap as a check only, undo if it errors <Created by pedronis> <https://github.com/snapcore/snapd/pull/5039>11:34
pedronisChipaca: mvo: digging deeper ^11:34
mwhudsonsergiusens: have you read ian lance taylor's blog? https://www.airs.com/blog/archives/38 through https://www.airs.com/blog/archives/5711:35
sergiusensI have not; adding to the reading list. Thanks!11:36
didrockscjwatson: hey, sorry if the question was probably already asked (it was probably already done), but is there any way to get daily build of a snap (even if the branch didn't change, as a snapcraft.yaml can refer to multiple repos) on https://build.snapcraft.io/ or launchpad?11:42
didrocksI guess the workaround can be to use launchpad API to start a build, but I would prefer something more integrated11:43
mupPR snapcraft#2065 opened: many: update the yaml loading logic <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2065>11:43
popeydidrocks: i personally use lp-build-snap (a snap) to poke the launchpad API daily via a cron job to do that11:44
popeybuild does do daily builds now AIUI.11:44
didrockspopey: oh? (well, I guess it's still building on xenial, which is an issue for me), but I didn't find any option11:45
popeyit doesnt have to be building on xenial11:45
popeyyou can pick in your repo what it builds on11:45
didrockson build.snapcraft.io?11:46
popeyno, launchpad11:46
popeyI dont use build, because i need the flexibility launchpad offers11:46
didrocksah yeah, for this, sure :)11:46
popeybuild isn't supposed to cover every use case.11:46
didrocks"build does do daily builds now AIUI" -> you meant launchpad or build.snapcraft.io?11:46
* cachio afk11:47
popeybuild11:47
didrocksok ;) so yeah, I have to use launchpad and having a ping script (like lp-build-snap) seems necessary11:48
didrocksthanks popey ;)11:48
popeynp11:48
mupPR snapd#5037 closed: snap: snap.AppInfo is now a fmt.Stringer <Created by chipaca> <Merged by chipaca> <https://github.com/snapcore/snapd/pull/5037>11:50
mborzeckifinishing up with spread test11:58
cjwatsondidrocks: the above all seems correct.11:59
didrocksthx for confirming cjwatson12:00
mborzeckioff to pick up the kids, coming back for standup12:04
zygajdstrand: some more leak findings on the forum12:15
mupPR snapd#5040 opened: overlord/snapstate: poll for up to 10s if a snap is unexpectedly not mounted in doMountSnap <Created by pedronis> <https://github.com/snapcore/snapd/pull/5040>12:15
zygamvo: I have to break now and go to the vet but I can propose a simple tweak to apparmor backend to limit memory usage12:19
=== pstolowski is now known as pstolowski|lunch
mupPR snapd#5041 opened: interfaces/apparmor: don't reload unchanged profiles <Created by zyga> <https://github.com/snapcore/snapd/pull/5041>12:26
zygamvo: hey12:26
zygamvo: can you run adt tests against that PR somehow?12:26
zygaI need to break now and go to the vet or I will miss my time12:26
zygaI will resume my analysis later today,12:27
mvozyga: yeah, I can12:30
zygathanks12:30
zygaI'll go now, I'll be on telegram/irc on the phone though12:30
* kalikiana lunch12:38
diddledanpopey: for atom, in snapcrafters, there's a new release upstream (a point release: x.y.1) do I just need to trigger a manual build in the build service to get it to update?12:42
diddledanalternatively, Wimpress ^^^^12:42
WimpressYep12:42
diddledanaww, it says I don't have admin perms on the github repo so I can't trigger it :-(12:44
zygamvo: standup postponed by one hour, right?12:51
pedronismvo: I proposed a chain of PR,   John reviewed them, we need to decide a bit what to do with them (whether we care more to understand the bug vs preventing it)12:52
pedronisit's also rare afawk12:52
mvopedronis: I have a look, thank you and Chipaca12:54
pedronismvo: I also tried to play with remove/install while keeping the old mount in use, but seems to just work12:55
pedronisat least here12:55
mvozyga: standup is in 3min12:57
mvomborzecki: I think I figured the refresh-mode issue out12:57
mborzeckimvo: what was it?12:57
zygamvo: Gustavo asked to postpone12:57
mvozyga: oh, ok, sorry, I miseed that12:58
mvozyga: in this case, sure12:58
pedronisby 1h12:58
mvomborzecki: we need KillMode=process in the service file for the non -all signal because it looks like system cleans up the whole cgroup if the main pid dies otherwise12:58
mvomborzecki: which makes sense12:59
mvomborzecki: I changed the test a bit and added some code to do this and *hopefully* this fixes this12:59
mborzeckimvo: a maze of manpages13:00
mvomborzecki: I looked at the source :)13:00
mvomborzecki: but yeah, its a bit difficult to get the picture13:00
mvomborzecki: I may still miss pieces but it all makes some sense now13:00
* mvo is a happy camper again13:00
pedronisit's a maze of non-orthogonal settings13:00
pedronisbtw it does make sense13:01
pedronis:/13:01
mborzeckimvo: does ssytemctl stop work correctly still?13:01
pedronisprobably not13:02
pedroniswe need to specify --kill-who=all then?13:04
pedronisfor when is not just a refresh13:04
pedronisbut that's the default13:05
pedronismborzecki: my reading is that stop will kill all processes one by one, instead of killing the cgroup13:06
pedronis(I haven't tried)13:06
mvomborzecki: looking now, sorry for the delay13:09
mvopedronis: yeah13:09
* pedronis needs a break13:13
mborzeckithere's a problem with snapd.wakeup.service on fedora, it's disabled by systemd presets, explicitly enabling it will probably make Son_Goku unhappy13:15
mborzeckiand fedora has presets13:15
Son_Gokuit needs to be requested to be enabled via central presets13:15
mborzeckiopensuse has none?13:15
mborzeckiSon_Goku: bz ticket then?13:16
Son_Gokuyep13:16
Son_GokuopenSUSE has central presets too13:16
popeydiddledan: thanks for the ping - I triggered a build13:25
pedronismborzecki: can't we do it only on ubuntu, and keep the other distros as is for now13:25
diddledanthankyou popey13:25
mborzeckipedronis: yeah, i'm afraid that's what we'll have to do13:26
mvomborzecki, pedronis you are right about the "stop", with KillMode=process the refresh mode works like we want it to work (yay) but the side-effect of course is that "systemctl stop service-that-uses-refresh-mode-sig{term,hup,..}" also only stops the main process13:28
mborzeckiomg13:28
mborzeckimvo: mixed?13:28
=== pstolowski|lunch is now known as pstolowski
mvomborzecki: oh13:30
mvomborzecki: let me try that13:30
pedronismvo: btw, how did the autopkgtests in xenial go?13:36
mvopedronis: it died in interfaces-many but I try again without this one, this test is especially problematic13:36
pedronisok, but at least unblocked13:37
pedronisthx13:37
mvopedronis: well, not sure yet if unblocked or not, I had to restart13:37
pedroniswell, it's no worse than bionic13:37
pedronissomething runs :)13:38
* pedronis is about the small victories these days13:38
mvopedronis: heh, yeah13:38
mvomborzecki: test with mixed running now :)13:39
greybackkalikiana: hey, snapcraft stuck at "Preparing to build desktop-gtk3" for me, using 100% of a cpu. How could I track down what's wrong?13:40
mvomborzecki: the way I read it I don't have high hopes *but* I was wrong before!13:40
mborzeckimvo: aiui if the main process ignores sigterm kill is tried again, but this time everyone gets sigkill13:41
mvocachio: how is the sru validation going? any fires there?13:42
mvomborzecki: oh, that would be what we want I think. if the main process dies the rest is ignored if it does not die/restart the entire group is killed13:42
cachiomvo, hey13:43
cachioI finished it13:43
mvocachio: yay, any issues?13:43
mvomborzecki: the answer is just a spread run (and some patience) away :-D13:43
cachiomvo, but I saw the oom on autopackgtests for i38613:43
mborzeckimvo: patience is key today13:44
cachiofor artful and bionic13:44
zygamvo: hey, just checking about that auto package test in my PR13:44
zygaIs it triggered, I cannot see that on the pull request page13:44
cachiomvo, that's the only relevant issue13:44
cachiomvo, I think it is because of the environment13:44
jbichahi, somehow my gnome-characters snap got disconnected from the gnome-3-26-1604 snap. I'm on bionic and it used to work. I'm using stable channels for this stuff13:46
jbichahow can I figure out how this happened?13:46
mvocachio: yeah, the oom is what we also see everywhere its a sad and known issue13:47
cachiothe rest is13:47
cachiook13:47
cachiomvo, well known issues on autopckgtests13:48
mvocachio: great, I think we can document that in the bug and then move on and set it to validated13:48
mvocachio: are there more known issue that need addressing?13:48
cachioyes13:48
cachiomvo, just that the tests on xenial are not being executed because spread cannot be built13:48
mvomborzecki: mixed did not work the way we want it to work, it did kill the children13:48
mvocachio: spread is fixed now13:49
mvocachio: since last night :)13:49
mvocachio: gustavo applied my fix (yay)13:49
cachiomvo, great13:49
cachiomvo, ok, but it failed for the sru :(13:49
mvozyga: tests are running, I had to restart because it died in interfaces-many?13:49
mvocachio: http://autopkgtest.ubuntu.com/browse.cgi/packages/s/snapd <- xenial is mostly green again13:50
jbichahmm, gnome-characters worked after restarting my computer 😦13:50
mvocachio: I manually restarted some13:50
cachiomvo, nice13:50
cachiothankis13:50
mvozyga: interfaces-many is a bit of a beast, I disabled that (and it is already disabled for autopkgtest)13:51
mvozyga: so if the run survives the remaining tests, that would be huge13:51
mvozyga: its at 17/224 right now13:51
zyga Thanks13:51
greybackkalikiana: ignore me, seems it was doing legitimate work13:52
mupPR snapd#5042 opened: many: fix "refresh-mode: sig{term,hup,usr[12]}" via KillMode=process <Created by mvo5> <https://github.com/snapcore/snapd/pull/5042>13:55
cachiomvo, sru in lp updated14:00
* kalikiana totally ignores greyback and walks away again14:08
greybackkalikiana: I'm trying to use cleanbuild, where my part has its source in a directory elsewhere in the system. It builds ok with dirty build, but with cleanbuild it complains it cannot determine the source type - and I see no source type specifier for a directory14:12
greybackany ideas?14:12
kenvandinejbicha, so it resolved itself on reboot?14:18
jbichakenvandine: yes14:18
kenvandine:/14:18
kalikianagreyback: elsewhere meaning outside of the source tree? is it a folder?14:21
greybackkalikiana: correct, and yes14:21
kalikianagreyback: so how do you get the folder into the container?14:21
greybackkalikiana: I'm not, hence the problem.14:22
kalikianagreyback: Can you manually copy it to the tree? Well, or if that's not an option, you could... use a persistent container that you've prepared beforehand.14:29
greybackkalikiana: I'm doing option 2.14:29
greybackkalikiana: I'll log a bug tho14:30
kalikianagreyback: Okay.14:34
greybackkalikiana: https://bugs.launchpad.net/snapcraft/+bug/176339414:34
mupBug #1763394: cleanbuild gets confused with directory source type <Snapcraft:New> <https://launchpad.net/bugs/1763394>14:34
kalikianaThanks!14:39
Chipacahah, browser crashed14:54
Chipacaand now y'all're gone14:54
pedronismvo: about my PRs, the easiest is probably to squash merge #5040 with everything (so it's easier to backport for the SRU)  but I can wait for that, as you prefer14:54
mupPR #5040: overlord/snapstate: poll for up to 10s if a snap is unexpectedly not mounted in doMountSnap <Created by pedronis> <https://github.com/snapcore/snapd/pull/5040>14:54
mvopedronis: yeah, otoh, its just three commits so we can just backport those via three cherry-picks (hopefully not too many conflicts)14:55
pedronismvo: as your prefer14:56
pedronisyou I merge them today? or wait a bit?14:56
pedronissorry14:56
pedronismvo: should I merge them today or wait a bit?14:57
mupPR snapd#5042 closed: many: fix "refresh-mode: sig{term,hup,usr[12]}" via KillMode=process <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/5042>14:57
pedronismvo: I marked them blocked for now, let me know14:59
mvozyga: adt run died with oom in 153/224, sorry for that14:59
mvopedronis: either is fine, its edge so feel free to merge and we can backport at our leisure15:00
pedronismvo: I think I'll merge them tomorrow,  I need to EOD in a bit to go to some event15:02
pedronisfor some of my evening15:02
pstolowskizyga: one question to 540115:03
mvopedronis: sure, enjoy15:05
mvopedronis: its not urgent15:05
pedronismvo: I don't think we have fires not under control at this point,  I can be pinged on TG if needed15:05
jdstrandzyga: thanks for those patches! nothing silly about typo fixes :)15:08
mvopedronis: yeah, thanks for all the fires you extinguished! have a great evening15:16
zygapstolowski: ack, looking15:22
zygapstolowski: replied15:26
zygamvo: do you have an URL to the test results?15:26
pstolowskizyga: thanks!15:27
mvozyga: I ran it locally15:27
zygaah15:27
zygaand? :-)15:27
mvozyga: its easy to reproduce, just use the ubuntu-18.04-32 image15:27
mvozyga: it died at test 153 or so :(15:27
zyga:/15:27
mvozyga: what kind of data do you want?15:27
zygacrap, same way as before, right?15:28
zygamvo: I wanted a pass/fail status, I got that15:28
mvozyga: yeah, *maybe* it survived longer15:28
mvozyga: but its hard to say as the pattern is not consistent, I would have to run with the same -seed= setting once with the branch and once without15:28
zygaI will do more checks at home, I need to write a small automatic program that triggers the bug and checks that memory is leaking15:28
zygathen try all the kernels gustavo suggested15:29
mvozyga: cool, thanks15:29
popeydiddledan: atom now in stable, thanks for the nudge15:32
=== dpb1_ is now known as dpb1
* zyga is starving, having feed the dog I need to eat something myself15:46
=== chihchun is now known as chihchun_afk
=== pstolowski is now known as pstolowski|afk
mupPR snapd#5043 opened: many: fix "refresh-mode: sig{term,hup,usr[12]}" via KillMode=process <Created by mvo5> <https://github.com/snapcore/snapd/pull/5043>16:17
mupPR snapcraft#2061 closed: go: only use Go build package if not using the snap <Created by kalikiana> <Closed by kalikiana> <https://github.com/snapcore/snapcraft/pull/2061>16:18
mvoChipaca: 5043 might also be interessting for you, I think it is really as simple as this, when systemd sees the main-pid die it goes into stop-state which triggers the cleanups. but double checking is certainly appreciated16:23
* kalikiana eod, o/16:24
suebtzyga: hey there :]16:50
zygasuebt: hi16:56
Chipacamvo: what happens in 5043 if start-refresh-mode-sigterm tries a little harder?16:56
zygaI’m busy now, sorry, no change yet. I’ll tell you what I know back home16:56
suebtzyga: Okay, no problem, will hang around for a while, thank you16:57
zygaSorry for keeping you waiting. It is a busy day16:59
Chipacamvo: i mean, something like 'setsid nohup sleep' instead of just sleep :-)17:00
mupPR snapcraft#2066 opened: errors: feature flag error reports <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2066>17:24
naccis there possibly a bug in snapd that let's older snaps get unmounted when they are in use?17:29
nacc:)17:29
nacc*older snap revisions17:29
naccspecifically, i've got a long-running app from a snap in the edge channel, which is building from master17:29
naccit was at r409, now at r41817:29
naccthe r409 run ended up dying suddenly because it could not find executables in it's squashfs17:30
nacci'm guessing because it got unmounted on some update17:30
Chipacanacc: we have a bug around long-running processes and refreshes17:42
Chipacanacc: basically if an app is running, and the snap gets refreshed, the file descriptors it has are now useless17:42
naccChipaca: yeah17:43
naccthat's ... seriously bad for us17:43
Chipacanacc: it's on the 'fix soon' list, i think it's one of the first things zyga will work on post 18.0417:43
Chipacanacc: how so?17:43
naccChipaca: because we are importing Ubuntu source packages using a snap and it crashes17:44
naccfor 'official' Ubuntu use :)17:44
Chipacanacc: you are importing Ubuntu source packages using a snap, and the snap refreshes while the import is happening? Every time?17:44
naccChipaca: the snap *can* refresh while it is happening yes17:45
naccnot every time, but when it does, this clearly would break things17:45
naccso we have an importer, which is short-lived (sort of)17:45
naccand control scripts which are long-lived (watching the publisher itself)17:45
naccthe latter is quite long-lived (one never dies)17:45
Chipacanacc: long lived but not a daemon?17:45
naccChipaca: right, not yet17:45
naccwould a daemon avoid this?17:46
Chipacanacc: the daemon would be restarted on refresh (unless you ask not to be, but then presumably you're only using _COMMON17:46
naccChipaca: restarting the daemon is 'bad' for us, sort of too17:47
naccChipaca: hence why it's not a daemon yet :)17:47
Chipacanacc: as a short term workaround you can ask snapd to hold off refreshes until you're done17:47
naccChipaca: yeah, i might need to do that, quick syntax/cli?17:47
Chipacanacc: ach, i always get that one wrong17:48
nacc:)17:48
Chipacait's 'snap set <something something>'17:48
Chipaca:-)17:48
Chipacamvo knows -- and it should be documented17:48
Chipacaand I've got to go make dinner17:49
naccChipaca: np, i'll look it up17:49
nacchttps://forum.snapcraft.io/t/disabling-automatic-refresh-for-snap-from-store/707/6 i think17:50
Chipacanacc: snap set core refresh.hold=<when>17:51
Chipacanacc: some work to be done around the UX of it though17:51
mupIssue snapcraft#2033 closed: LP: #1752344 <bug> <Created by sergiusens> <Closed by sergiusens> <https://github.com/snapcore/snapcraft/issue/2033>17:52
mupPR snapcraft#2065 closed: many: update the yaml loading logic <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2065>17:52
naccChipaca: thanks17:52
Chipacanacc: it requires a timestamp (the error will show you the format)17:52
naccChipaca: ok :)17:52
Chipacanacc: but the expectation is that a script or something would say "give me 5 minutes" every 5 minutes until it was done17:52
Chipacahence, ux needs more work (or maybe an interface that isn't 'snap set')17:53
Chipacaanyhoo. dinner.17:53
zyganacc: refresh sanity is on my plate post 18.0417:53
zygaIs the bug related to an apparmor denial?17:53
nacczyga: Chipaca: thanks, i was able to set it to be at least a week away17:54
nacczyga: is that to me?17:54
zygaYes17:54
nacczyga: no apparmor denials in dmesg17:54
zygaAbout refresh breaking some source import17:54
nacczyga: it just failed to find the `git` binary that was in the snap17:54
zygaCan you let me know what is the problem?17:54
naccbecause, i'm guessing, the fd was closed underneath it17:54
zygaHmmm17:54
Chipacao/17:55
zygaIt would not be closed17:55
zygaBut can be revalidated17:55
zygaBut that would log a denial17:55
zygaCan you give me a link to a case that reproduces the issue17:55
nacczyga: i don't have the logs handy anymore, but i can try and scrape them (about to leave though) -- basically our snap was running (git-ubuntu.source-package-walker) ... and it ran for some time17:55
naccat an older review17:55
nacc*revision17:55
zygaIncluding a refresh to another channel in a side terminal is ok if that helps17:56
naccwe then refreshed (auto) edge from which we're running17:56
zygaIs that snap classic or strict?17:56
naccat some point, we must have gotten past the 'keep 2 versions' or whatever snapd does17:56
naccclassic17:56
naccand our snap crashes becasue it's unable to find an executable that is in the snap17:56
zygaHmmm17:56
nacci believe with a 'no such file or directory'17:56
zygaClassic snaps have no constraints17:56
zygaSo it is probably another part17:57
zygaThanks17:57
zygaI know what I needed now17:57
zygaRight, I understand why this happens17:57
naccconfirmed i got a no such file or directory for 'git' in our case (which we build from source and is in our snap's path)17:57
nacczyga: ok :)17:58
zygaThank you17:58
nacczyga: we admittedly are a weird classic snap -- we are basically confined (by munging the environment) but we need to be able to write anywhere on the fs17:58
zygare18:06
zyganacc: classic snaps are confined but have wide open permissions and are not sealed in a mount namespace18:06
nacczyga: yeah i know18:07
naccwell the former part is why :)18:07
nacc*why we use a classic snap18:07
zygato have access to /var/lib/snapd/hostfs easily?18:08
zyga(as in, at /)18:08
nacczyga: yeah basically (and network)18:08
naccbut network obviously we could use an interface for18:08
nacclast i checked there was not a 'hostfs' interface18:08
zyganetwork?18:08
zyganetwork should not be a problem18:08
zygawe could add an interface for hostfs and hand it out just like classic18:09
zygaif that helps18:09
nacczyga: that would probably help us, yes18:09
naccnot sure if we have other needs, but that woudl be my guess right now18:09
nacczyga: gotta go afk for a bit18:09
zygathanks, no worries :)18:10
* zyga is still at school, waiting to talk to the teacher18:10
* zyga is very sleepy somehow18:20
mupPR snapcraft#2067 opened: many: add snapcraftctl set-grade <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2067>18:46
=== Guest26743 is now known as ^arcade_droid
* zyga returns for evening hacking18:49
zygajdstrand: hey, can you please look at https://github.com/snapcore/snapd/pull/5041 and specifically at the question from pawel as well as my response there18:56
mupPR #5041: interfaces/apparmor: don't reload unchanged profiles <Created by zyga> <https://github.com/snapcore/snapd/pull/5041>18:56
zygajdstrand: I'm mainly trying to raise any red flags that you may notice18:59
=== zarcade_droid is now known as Guest76636
* cachio afk19:02
=== Guest76636 is now known as ^arcade_droid
mupPR snapcraft#2068 opened: states: track override scriptlets <Created by kyrofa> <https://github.com/snapcore/snapcraft/pull/2068>19:40
mupPR core#38 closed: Add another pi-config option <Created by sergey-borovkov> <https://github.com/snapcore/core/pull/38>21:02
mupPR core#38 opened: Add another pi-config option <Created by sergey-borovkov> <https://github.com/snapcore/core/pull/38>21:03
mupPR core#83 opened: move most of the ubuntu-core config deb into the snap snap build <Created by mvo5> <https://github.com/snapcore/core/pull/83>21:03
jjohansenzyga, mvo: with your testing you are only seeing this with the bionic kernel?21:05
mupIssue snapcraft#1681 closed: Design/document support for architecture selection during builds <Created by sergiusens> <Closed by kyrofa> <https://github.com/snapcore/snapcraft/issue/1681>21:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!