[00:00] PR snapcraft#2080 opened: project_loader: support architectures for CI [02:09] Issue snapcraft#1680 closed: Documentation overhaul for scriptlets and setters [02:21] bah the snapcraft autopkgtests failed in bionic [02:21] snapcraft.internal.errors.StagePackageDownloadError: Failed to fetch stage packages: Error downloading packages for part 'python2': The package 'python-distutils' was not found.. [02:21] i guess that patch wasn't quite right === mup_ is now known as mup === mup_ is now known as mup === mup_ is now known as mup === mup_ is now known as mup === mup_ is now known as mup [05:06] morning [05:15] Cześć [05:17] zyga: hej [05:19] https://forum.snapcraft.io/t/unexpected-reboot-during-snapd-tests-execution-on-ubuntu-core-16/5009 [05:21] PR snapd#5024 closed: systemd: add helper for opening stream file descriptors to the journal [05:21] There is a new kernel there [05:21] And one of the logs looks like OOM [05:21] zyga: this is with the new kernel? [05:21] But others did not [05:21] I didn’t check cachio’s post in detail but I assume so [05:22] Just woke up :-) [05:25] hey zyga and mborzecki [05:25] mvo: hey there [05:26] PR snapd#4989 closed: tests: add arch to CI [05:27] mvo: #5059 is something you want to merge or just wanted to run it through the CI? [05:27] PR #5059: tests: add pending shutdown detection [05:29] mborzecki: both, its fine to keep it open for now I will also want to backport it to 2.32. it hopefully helps us to get clues about the reboot issue that cachio is reporting [05:29] ok [05:30] mvo: about the changelog, is this some tool that picks up the commit messages? (git-bp?) [05:33] mborzecki: yeah, its a hand written one that is not very good in lp:~mvo/+junk/snappy-dch [05:41] good morning [05:42] mborzecki: now you know about mvo's secret stash of secrets :) [05:46] is there any script that creates an image, prepares it for testing and runs tests against it [05:46] I always have to hand-craft things and that feels weak [05:46] yesterday I installed elementary to test a collection of snaps there [05:47] and I noticed that elementary ships HWE kernels by default [05:49] zyga: maybe they like running recent-ish kernels? [05:49] yeah, just more leaky kernels [05:49] but that's fine, the fix will come out soon enough [05:49] mvo: do you suspect the error is just OOM again? [05:50] after the last log from cachio late last night I think that's just it [06:02] zyga: maybe it is that. I just had one run that was completely good and now I got a kernel oops with a _dma_segment_allow failure [06:04] zyga: hm, /proc/meminfo looks happy, I don't think its just mem [06:08] zyga: I wonder if we should add a more generic check for kernel oopses during the test runs [06:08] zyga: just like the oom test [06:10] mmm, good idea [06:10] so do you think the new kernel is just busted? [06:12] zyga: not sure, I refresh to stable now and re-run the tests [06:12] zyga: its strange no reboot so far for me [06:12] how are you running tests? [06:12] spread + external backend + ... [06:13] zyga: yeah, tests/external-backends.md [06:13] zyga: so 1. build image using ubuntu-image (from the deb in my case) 2. create user on image 3. slavishly follow steps in tests/external-backends.md [06:13] graargh [06:14] yeah [06:14] so handheld test [06:14] zyga: I think sergio has better code for this [06:14] zyga: I'm just a bit set in my ways (and I almost never do that kind of tests) [06:20] casual testing on elementary shows everything is ok with 2.32.5 [06:31] om26er: installing sublime-text now, thank you! [06:32] great to know [06:35] PR snapd#5060 opened: tests: detect kernel oops during tests and abort tests in this case [06:37] om26er: is the snap owned by upstream developers? [06:37] om26er: I would love if they could se the percentage of snap vs native package userss [06:38] zyga: not yet, there is an ongoing discussion with them, will se where that goes [06:39] currently its under snapcrafters [06:39] ok [06:39] it works just great so thank you :) [06:39] om26er: is there a chance to put dev builds in edge? [06:41] @zyga: that should be possible, I think they have public apis with which we can check the latest dev build version [06:41] the dev build is just in a different archive [06:41] I don't know how this snap is built but if you can use a different archive for edge vs stable then it should be painless [06:41] this has to be done externally though as their project is proprietary and we dont have access to its vcs [06:42] zyga: https://github.com/snapcrafters/sublime-text/blob/master/snap/snapcraft.yaml [06:43] I am working on a service that notifies us when a new release is available upstream [06:43] where is SNAPCRAFT_PROJECT_VERSION coming from [06:43] and that's some unusual variable replacement [06:43] sublime, android studio and probably others could benefit [06:44] not shell compatible (cc sergiusens) [06:44] om26er: ack, thank you [06:44] ah, I see version is just placed there [06:58] PR snapd#5061 opened: tests: ensure interfaces-network-bind daemon is really stopped [07:04] mvo: does this mean we remove snaps incorrectly on cleanup [07:04] That is, keep processes up and running [07:05] zyga: maybe - or maybe my analysis was too shallow [07:06] zyga: it was a drive-by while searching for the real bug [07:06] zyga: but there is definitely a runaway process that caused the journalctl --sync hangs [07:07] PR snapd#5062 opened: tests: skip interfaces-content test on core devices [07:07] zyga: ^- is the reason for the reboot (or one reason, not sure yet, need to keep this running) [07:07] zyga: but it all makes sense [07:12] mvo: ah :( [07:12] pedronis: no :( more :) because we found the issue and its just test related and the fix is easy [07:14] Pho [07:14] Oho [07:14] But why is that rebooting the system [07:15] Is that test special? It just checks content connection [07:15] it's removing state.json [07:15] zyga: it purges the state which triggers a download of core and for core core wants to remove [07:16] to reboot [07:16] sorry [07:16] Aha [07:16] I didn’t remember that test is doing such things [07:16] mvo: well, on core it shouldn't download core [07:16] but it might trigger a reboot if strange things happen [07:17] or maybe nowadays it download core even on core? [07:17] pedronis: indeed you are right, it is not downloading core, it is seeding again it seems [07:18] I'm not sure, given the new prereq code [07:18] it might be even trying to do both [07:18] pedronis: the snap change output here indicates its seeding that is happening [07:18] ok [07:18] pedronis: and in the content snap I don't see it tries to fetch core [07:21] mvo: there is a small change that it could try to do both tough [07:21] pedronis: aha, its interessting. its is not core that triggers the reboot (because we are smart and know that we already have that). it is the kernel seeding [07:22] pedronis: interessting, there might be a race here? [07:23] mvo: we start Loop and we start accepting api calls, the first ensure pass will create the tasks for seeding, but is not super clear that an install api couldn't come in before [07:23] rarely [07:23] * mvo nods [07:24] * zyga read the test just now [07:24] (back from the walk) [07:25] mvo: don't think is something to worry about now, but we should think about it some more [07:25] I see this test was re-purposed a ltitle [07:26] "ensure that prereq installs work even with an empty state" [07:27] yes [07:28] mvo: runaway process, may this be caused by the changes that affected `systemctl stop` and postrm doing exactly that to stop the services? [07:28] mborzecki: possible but unlikely unless special syntax is used nothing changes (i.e. you need to set stop-mode in snap.yaml for this to trigger) [07:29] mvo: we also leak some other things but probably started by individual tests [07:29] mborzecki: also we had the journalctl --sync hangs before but never figured out what triggered it [07:29] mvo: most notably dbus-daemon [07:29] zyga: good point [07:29] there's enough of them to use up all ram on intense testing loops [07:29] (I ran into it a few times but I didn't find the responsible test) [07:29] zyga, mborzecki hrm, hrm, it looks like we already stop so my PR is probably too simplisitc [07:30] zyga: prepare-restore.sh has a couple of checks now, we could add one more for runaway dbus [07:30] yes, more whack-a-mole [07:30] but yes [07:31] zyga: well, yes whack-a-mole but this time each whack also closes the hole for the moles so hopefully instead of always having to whack N holes with this approach on each step its just N-1 holes :) [07:32] mvo: we could also check that there are no new processes since test startup [07:32] mvo: I mainly oppose ad-hoc solutions to system problems [07:33] mornings [07:33] pstolowski: o/ [07:33] zyga: +1 [07:33] hey pstolowski [07:34] hey pawel, good morning [07:50] maybe we should run the tests through systemd-run [07:52] but this would probably have to go up to spread, and the benefit is only the processes started inside the test would get cleaned up automatically [07:52] PR snapd#5058 closed: packaging: fix incorrectly auto-generated changelog entry for 2.32.5 [08:00] pstolowski: hi, yes, I can look at the interface hooks PR again today [08:00] pedronis: great, thank you! [08:00] moin moin [08:02] mvo: does it all means we need a .6 ? [08:04] hey chipaca [08:07] zyga: how many ways can an X app set its icon [08:07] zyga: :-) [08:07] Chipaca: ETOOMANY [08:07] hm if we're to do .6, #5024 and #5034 would be nice, this would fix the user autostart logging this [08:07] PR #5024: systemd: add helper for opening stream file descriptors to the journal [08:07] PR #5034: userd: set up journal logging streams for autostarted apps [08:07] zyga: sorry, your answer was wrong. The right answer is: -1. [08:08] .6 is happening? [08:08] i was joking about it yesterday [08:08] Chipaca: I think .7 will be it [08:08] it's also a lucky number [08:08] as many as we need ;) [08:11] what's broken in .5? [08:14] Chipaca, zyga what are you guys talking about? .5 is perfect. PERFECT [08:14] mvo: it's perfect in vacuum in flat space [08:14] mvo: mvo: does it all means we need a .6 ? [08:15] mvo: that's all i know [08:15] pedronis: I don't think we need .6, its all in tests so far [08:15] Chipaca: ok :) [08:15] Chipaca: I though you were pulling my leg [08:15] pedronis: also tests that affect only core itself which we don't run via autopkgtest [08:54] mborzecki: updated https://github.com/snapcore/snapd/pull/5033 [08:54] PR #5033: cmd: generalize locking to global, snap and per-user locks [08:54] zyga: thanks, looking [08:55] niemeyer: good news and bad news: when using bolt directly, both in rw and ro mode, RSS is ~5MB after the call to debug.FreeOSMemory() [08:56] so, we're probably being too clever with caching, interfaces, something else, or all of the above [08:56] I might dig, or I might refactor the whole thing to be less loopy [09:04] Chipaca: are you saying something just leaks memory or that we keep something deliberately open for extended period [09:07] zyga: yes [09:07] Chipaca: are those numbers for the small example? or for snapd? [09:08] pedronis: it's a small example [09:08] pedronis: yesterday this same small example with rw was using 14MB [09:09] pedronis: the difference between that and this is that that one used snapd/store's decodeCatalog, calling advisor's AddSnap, and this one uses bolt directly for that bit [09:09] it still queries using advisor [09:10] 14MB that dropped to (IIRC) 12MB after FreeOSMemory; this one uses 8MB before, 5MB after [09:10] so, we're caching the store, and its http client [09:11] (today's loads a local json file directly) [09:11] and we're probably caching other stuff as well, i'd have to dump or sth to know in full [09:12] and we're using interfaces a lot, which puts pressure on the heap [09:12] and runtime [09:12] yeah, i'm just looking at memory, as this is mostly io-bound [09:12] Chipaca: I see, but most of those changes aren't applicable inside snapd itself [09:12] is there a chance we cache some responsese from the store? [09:12] pedronis: well, maybe. I'll be adding things back now to see :-) [09:13] first up, using an http client [09:19] holy crap [09:20] changing the os.Open to do a net/http Get went from 5/8/5 rss to 8/13/11 [09:24] Chipaca: this is go 1.6 ? or something else [09:25] slightly curious if newer go uses less or more memory for this [09:25] checking [09:26] pedronis: go 1.10: 6/11/9 [09:27] that's before writing, after writing, after debug.FreeOSMemory [09:42] PR snapd#5063 opened: tests: run interfaces-boradcom-asic-control early [09:48] Chipaca: so 1.10 does a bit better [09:49] ish, yes [09:50] (1.10 uses more memory in the boring case) [09:50] (5/4 vs 6/5) [09:50] but that's the case we don't care about :-) [09:57] remember when there were revision control systems that used inodes for stuff so you couldn't cp things around? [09:58] * Chipaca just copied a git tree and was greatful [10:01] mborzecki: this isn't true anymore since 4.14: https://github.com/snapcore/snapd/blob/master/tests/regression/lp-1618683/task.yaml#L3 , linux,linux-lts and linux-hardened have user_ns enabled but restricted to root by default. They use same patch as debian but I don't see debian mentioned there so probably Arch shouldn't be there also: https://wiki.archlinux.org/index.php/Security#Sandboxing_applications [10:04] vidal72[m]: thanks, i'll update the test [10:12] mborzecki: also you use vanillia "go" package in test: https://github.com/snapcore/snapd/blob/master/tests/lib/pkgdb.sh#L561 , while "go-pie" is used in package: https://github.com/bboozzoo/aur-snapd/blob/master/PKGBUILD#L15 . Maybe switch to "go-pie" in test as well. Anyway thx for supporting Arch :) [10:13] vidal72[m]: hah, thanks for noticing ;) [10:33] mmm, pie [10:45] PR snapd#5064 opened: tests: smaller fixes for Arch tests [11:24] * Chipaca -> lunch === pstolowski is now known as pstolowski|lunch [11:46] is https://forum.snapcraft.io/t/request-automatic-alias-for-ubuntu-make/4958/5 missing something to be treated like an alias request? [11:47] Chipaca: the original request didn't sound like one [11:50] Chipaca: I updated #4889 based on Gustavo's feedback [11:50] PR #4889: cmd/snap-update-ns: don't trespass on host filesystem [11:50] could you please look at https://github.com/snapcore/snapd/pull/4889/commits/f371bf27d2189c12f9ce5f29b0ed220aaa06101a and give me some quick feedback if that makes sense to you? [11:51] zyga: how quick exactly :) [11:51] zyga: e.g. can i do it after lunch [11:51] sure, that's fine [11:51] * Chipaca said lunch but then didn't [11:51] k [11:51] * Chipaca afk [11:52] Ha. I should go too [12:06] mvo: how are we looking wrt release [12:06] all green, no smoke on the horizon? [12:11] off to pick up my daughter from school [12:11] popey: ERROR: ld.so: object '/snap/shattered-pixel-dungeon/5/usr/lib/x86_64-linux-gnu/pulseaudio/libpulsedsp.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. [12:11] Chipaca: which revision? [12:12] popey: based on the /5/, 5 [12:12] lulz [12:12] :-) [12:13] popey: it's /snap/shattered-pixel-dungeon/5/lib/libpulsedsp.so [12:13] wtf, this used to work [12:14] works fine here [12:14] ogra_: sound? [12:14] (16.04, stable chanel) [12:14] yep [12:15] i played 1-2h per day since sat.... sound and all works fine [12:15] 16.04, stable channel (even for core! need to fix that), and no sound [12:15] and get that error on startup [12:15] ogra_: unity? [12:15] bah, it says error, but the game works (just without sound) [12:15] yes, standard 16.04 [12:16] oi hold on [12:16] core also stable [12:16] other things also don't have sound [12:16] susie, sitting next to me has it running on her laptop as well ... sounds works there too [12:18] pstolowski|lunch: lgtm, couple of comments, I think jdstrand might want to look a bit at the policy checks changes [12:18] popey: um [12:18] popey: sound does work [12:18] popey: my bad [12:18] \o/ [12:18] wot u do? [12:18] popey: firmware update left me with no sound out of my speakers [12:19] oof [12:19] popey: so that's _probably_ not your snap [12:19] it also no longer detects headphone inserts [12:19] sounds pretty low-level then [12:20] double yew tea eff [12:20] Chipaca: oops :/ [12:20] how did you apply the update? [12:20] with a funnel ? [12:21] zyga: machine rebooted, it spent a bunch of time doing 'updating bios' and 'updating the stuff intel uses to hack your machine' and such [12:21] maybe I need to re-dkms or sth [12:22] popey: but, that ERROR thing is still there fwiw (because as stated the file is not there but in lib) [12:22] ok, will take a look after my call [12:22] good ol' java falling back gracefully and making debugging harder [12:23] zyga: all green so far [12:24] popey, Chipaca https://paste.ubuntu.com/p/MXBYZKhr4w/ ... everything works though ... (joystick interface not connected since i have none on my laptop) [12:25] looks like it would like "process-control" too [12:25] most games do [12:25] they used to crash, but since 2.32 they carry on even though they dont have it [12:34] PR snapd#5053 closed: release-tools: handle the snapd-x.y.z version [12:34] thanks mvo [12:35] zyga: thank *you* [12:35] zyga: changes in 4889 sgtm [12:36] thansk! [12:38] mvo, hey [12:38] I ran with -m 2000 and the new kernel and I could not reproduce the reboot error [12:38] I wonder if we run with -m 512 [12:39] but, it failed in the refresh scenario [12:39] ogra_: how much does smallest product you are aware of has? [12:39] *memory* [12:39] how much memory it has [12:39] I am now rexecuting the refresh scenario with the systemctl-redirector snap [12:39] zyga: I think ondra had a 256MB one somewhere [12:39] but I'm not sure [12:39] zyga, we always said our lowest end is the beaglebone ... so typically the low end products are also in that area, 256M ... [12:40] thanks guys [12:40] while running our test suite is not the same as running a single purpose product [12:40] I wonder how we fare in such world === pstolowski|lunch is now known as pstolowski [12:43] cachio: hey, I think we are good, I added some infrormation to the forum post and did some digging [12:44] cachio: some open PRs based on that but with those applied I am at run 4 now and still no issues [12:49] mvo, yes, I saw your comments [12:49] thanks for adding that info [12:49] I am rexecuting the refresh scenario now beacuse it failed [12:50] and I added the systemctl-redirector in case it fails again [12:50] the regular beta execution is working properly [12:58] * Chipaca goes to grab a cuppa [12:59] zyga, well even in the sigle purpose low-level products customers ask for wifi-ap and network-manager to be preinstalled etc ... [13:05] oSoMoN: I just saw it (was off yesterday). I'll look into it [13:09] oSoMoN: where is the snapcraft.yaml for chromium? [13:13] sergiusens, what's the status of snapcraft being able to build on a bionic base? [13:20] jdstrand, https://git.launchpad.net/~chromium-team/chromium-browser/+git/snappy-packaging/tree/snap/snapcraft.yaml [13:21] ratliff: interesting: https://forum.snapcraft.io/t/automated-reviews-and-snapcraft-2-38/4982/8 [13:21] oSoMoN: thanks [13:23] jdstrand: is that causing all electron apps to fail automated review currently? [13:23] ratliff: I suspect so [13:24] chromium is failing because the store has r1021 and not r1022 [13:24] ratliff (cc roadmr): ^ [13:25] hi :) [13:25] and cc oSoMoN ^ [13:25] * roadmr checks [13:25] jdstrand, what's in r1022 ? [13:25] + don't enforce 'app' snaps that have setuid/setgid overrides since they can't resquash properly without fakeroot or similar [13:26] jdstrand: I never added 1022 to the store :( 1021 was added/enabled last Thursday and the next one I have in the queue is r1025 [13:27] roadmr: yes, 1022+ is all that is needed [13:29] roadmr: but it seems the electron-builder is unsquashing then running its own mksquashfs with extra arguments that would affect the resquash. Let's turn the resquash off for now. I will investigate [13:29] popey: please see https://forum.snapcraft.io/t/automated-reviews-and-snapcraft-2-38/4982/8 [13:29] popey: what is signal using, electron-builder? [13:30] is there a way to gat old versions of snap packets? the update to lxd 3.0 broke all of our containers, we have to revert to 2.0 but someone in our team deleted snap cache. [13:31] s/gat/get [13:31] stgraber: fyi, ^ [13:32] oSoMoN: I'm manually approving chromium [13:32] jdstrand, thanks! [13:32] oSoMoN: (it passes with the review-tools snap, which as r1022) [13:32] has* [13:33] yeah, I ran that locally and didn't get any errors, which is why I was puzzled by the store rejection [13:33] glad that you sorted this out already [13:33] the harder one will be what popey is seeing [13:33] jdstrand: i believe it's electron-builder, yes [13:34] popey: I need to *know* since I will be filing a bug against the project [13:34] popey: :) [13:34] (also, one of our ISVs has sent a mail telling me he is using 2.35 so just hit this problem, because he can't use newer than 2.35 because patchelf breaks it) [13:35] jdstrand: ok, sorry I missed that (5 minutes only!). Turning off resquashfs enforcement now. [13:35] popey: that ISV needs to work with sergiusens so they aren't stuff forever. they could work around this by unsquashing and calling mksquash with the appropriate args [13:35] they could, but this is yet another issue they've had in the last 2 weeks. [13:35] I have offered them turning off patchelf in 2.40 [13:35] popey: but, see what roadmr just said. this is only temporary until electron-builder issue is worked out [13:35] I am loathe to give them a bunch of commands to work around this. [13:36] the ISV isn't using electron-builder (I am though) [13:36] they don't need to :) I've turned off enforcement, popey, so their snaps will work as they did last week before Thursday [13:36] popey: that doesn't matter. I asked roadmr to turn off resquash inforcement for everything [13:36] roadmr: but they need to get on 2.40 [13:36] err [13:36] jdstrand: (I'm quite happy we went the extra centimeter to add the feature flag, super easy to flip this behavior) [13:36] popey: but they need to get on 2.40 [13:37] roadmr: yes [13:37] Sure. But we can't _force_ people to upgrade if our tools are broken [13:37] Yes, I will speak to sergio :) [13:37] is there a way to build snap packages from source? when yes where do i find them? [13:38] popey: did they turn off patchelf? [13:38] popey: and upgrade? if so, they will be fine when we turn this back on [13:39] I asked them to [13:39] Not replied yet [13:48] popey: can you tell me definitively if signal-desktop is using electron-builder? I see that you referenced the snap-template.sh (great!) but if they are using something else, that also needs to be fixed [13:49] jdstrand: it is using electron builder [13:52] kenvandine: that's on 2.40 already, 2.41 to take into account some breaking changes that got into bionic; there is no container build support though and if doing classic you need to install core18 manually (as it is not on stable) [13:52] a build-snaps entry should solve that ;-) [13:52] niemeyer: for reference this is the topic about gadget connections: https://forum.snapcraft.io/t/interface-connection-from-gadget-in-firstboot/1431 [13:52] mvo, I managed to reproduce the reboot error running the refresh scenario [13:53] same log than what we saw yesterday [13:53] sergiusens, so it'll build confined snaps for core18? [13:53] it is stable image using a core snap from beta [13:53] we want to rebuild our snaps for core18/bionic [13:54] kenvandine: yes, BjornT_ is using it for maas I believe [13:55] kenvandine: you need to specify `base: core18` in snapcraft.yaml, also `snap info core18` [13:55]