[01:28] PR snapcraft#2017 closed: many: optimize retrieval of the linker version [01:31] PR snapcraft#2018 opened: tests: minor fixes to autopkgtests [01:34] PR snapcraft#2018 closed: tests: minor fixes to autopkgtests [01:49] PR snapd#4578 closed: Expose payment and account URLs in /v2/system-info === chihchun_afk is now known as chihchun [06:24] morning [06:37] good morning [06:37] zyga: hey [06:37] I lost my phone [06:37] ayy [06:37] I put it somewhere and now I cannot find it [06:37] and it's on mute [06:37] na [06:37] man :) [06:37] it's somewhere at home [06:37] or the dog ate it very very politely ;-) [06:37] well, that might still be a problem [06:38] I need to fix https://github.com/snapcore/snapd/pull/4889 [06:38] PR #4889: cmd/snap-update-ns: don't trespass on host filesystem [06:38] and then I'm blissfully happy and can help mvo with the release [06:38] jokes aside, i'll be looking into glvnd today, didn't manage to do anything yday, my son has flu-like thing again, so fever and the rest of the package [06:39] mborzecki: uhh, not fun [06:39] this winter is not going anywhere it seems [06:40] but, it's spring time (officially) [06:44] I need to wake kids up [06:52] mborzecki: I was thinking about making sense of prepare restore [06:52] mborzecki: and I came up with a very simple mechanism for making it modular and understandable [06:52] mborzecki: so that we can put some code affecting a given concept in a single file [06:53] mborzecki: and that file can affect any phase of the lifecycle (prepare project, suite, test, restore, etc) [06:53] mborzecki: I will propose the mechanism first and then chop prepare-restore into parts and move it there piece-by-piece [06:53] mborzecki: I did this while running tests all day, looking for the bug that was blocking master yesteday [06:53] mborzecki: I did not reproduce the bug ever [06:53] mborzecki: but I did find that we leave snap userd processes around [06:54] mborzecki: and that leak dbus-daemon processes (probably in the same tests) [06:54] mborzecki: and that my 1.5GB VM runs out of memory after a few hours just because there are so many of those [06:54] mborzecki: anyway, I'll get some coffee and start proposing this [06:54] zyga: hm yeah, we should kill whatever userd is left after the test completes [06:55] yes, that's obvious [06:55] just our way of doing prepare restore is making it hard to ensure and reason about [06:55] zyga: that might be the reason for one of the failures that pstolowski saw [06:55] although that was in unt tests [06:55] the scripts are too large now [06:55] and not maintainable much [06:57] heh running into build problems with glvnd on xenial, i'm probably better of pulling the source package from bionic [06:57] good luck with that! [06:57] I need to manage kids a little bit [06:57] then I'll be back [07:02] hm given the problems, we should look into having at least one spread test that tries to run glxinfo on a host with nvidia/radeon/intel, perferrably also something with PRIME [07:03] mborzecki: yeah, this is not supe easy due to infra [07:03] mborzecki: I wanted to have a rule where each release is tested on a modern nvidia machine running 16.04 and now 18.04 [07:03] zyga: maybe at least during validation [07:03] so that we have reasonable coverage of the most popular release [07:03] yes [07:04] zyga: and radeon would be intersting too, amdgpu probably works since it's mesa, but i'd guess the pro variant is broken [07:05] zyga: you have r9 right? :) [07:09] mborzecki: yes [07:09] mborzecki: I can boot and test things periodically for release [07:09] I was thinking about selling this t470 and getting some variant with the nvidia 150 GPU [07:09] or was it MX150 [07:09] something like that [07:09] low end modern core GPU [07:10] oh, it's just 1030 relabelled [07:10] wow, that's impressive then [07:12] It seems I want T480s [07:12] eh, expensive [07:12] * zyga wonders when the next laptop refresh cycle is [07:13] extgpu? [07:13] or that :) [07:14] but I'd have to check that it 1) works 2) doesn't need external display [07:14] I want only one display, the one in my laptop [07:14] i still need to buy one of rx580/vegas, 1030 is to be only the backup card, the more powerful one will go for passthrough [07:14] do you want one r9 280x? [07:15] I have two [07:15] vega :P but I need to wait for btc/xmr/eth to drop [07:15] haha [07:15] yes, that's sad :) [07:16] the prices are dropping though, slowly but still, just enough to keep the hopes up [07:17] I think at this rate we'll skip a gen or two [07:17] zyga: i have libglvnd rebuilt for xenial, what would be the best way of getting that into the namespace of a snap? [07:18] so, dragons ahead but let's try this [07:18] mount bind it somewhere and nsenter? [07:18] open a shell with nsenter -m/run/snapd/ns/foo.mnt [07:18] haha that's what i thought :P [07:18] then inside that shell go to /var/lib/snapd/lib/gl [07:18] then it should be a swarm of symlinks, remove them (or move them aside) all [07:19] the place should be a tmpfs, unpack your deb there [07:19] use ldd to see if things resolve one by one [07:19] and let's see what we have [07:22] mvo is not around, I hope he's okay [07:24] good morning mvo [07:26] hey zyga [07:27] mvo: morning [07:33] zyga: bad news, still segfaulting, although i have a nicer backtrace now [07:34] that's better now [07:34] can you list all the files in that directory? [07:34] as seen from inside the nsenter world [07:35] zyga: https://paste.ubuntu.com/p/jBSVD8bG84/ [07:35] is lrwxrwxrwx 1 root maciek 53 Mar 20 17:52 libEGL_nvidia.so.390.42 -> /var/lib/snapd/hostfs/usr/lib/libEGL_nvidia.so.390.42 this okay? [07:35] can you triple check that we only ever open things that are in the nvidia download for linux [07:35] nothing distros build [07:35] zyga: yes, that's the glvnd part on nvidia's side [07:35] zyga: bt https://paste.ubuntu.com/p/G9SdS24tTJ/ [07:35] actually, just download that an unpack it somewhere [07:36] yes, this does look better [07:36] hey mborzecki - good morning! [07:37] mborzecki: another idea, move them out of hostfs [07:37] in case some logic then looks at files in the same directory [07:37] little by little well' get to the bottom of this :) [07:52] moin moin [07:58] pedronis: er how can i make a date string that snap set core refresh.hold= accepts? [07:58] date --rfc-3339=seconds doesn't do it [07:59] ahh --iso-8601=seconds [07:59] yes [08:01] * pedronis breakfast [08:01] mborzecki, mvo: https://github.com/snapcore/snapd/compare/master...zyga:feature/test-modules-and-phases?expand=1 [08:02] I will open it as soon as I get a pass on my machine [08:08] moin moin [08:09] * Chipaca not really here yet [08:09] hey Chipaca [08:09] zyga: did the themes thing land at any point? [08:10] mvo: how's things on this lovely spring day? [08:10] zyga: interessting idea [08:10] Chipaca: things are in progress, we almost landed a fixed 2.32 in bionic, one autopkgtest issue though [08:10] asking about themes because of this 'papirus theme crashes snapped evince' thing [08:11] zyga: is all your 2.32 stuff in PRs now? threspass was the last one? [08:11] mvo: nice [08:12] zyga: hm this doesn't make sense, the segfault is in a mt_mutex_lock (a wrapper around pthread_mutex_lock), it is called a couple of times before the place where it segfaults and seems to work ok, i've confimed that it's trying to load "libGLX_nvidia.so.0", does that successfuly and then goes on to allocate new vendor id and this is where the crash happens [08:12] zyga: the wrappers are initialized into a larger struct, and the symbols are picked up one by one using RTLD_DEFAULT at the very beginning [08:12] zyga: unless loading libGLX_nvidia.so.0 break all this [08:12] zyga: https://api.travis-ci.org/v3/job/355950075/log.txt is interessting, I added code to reset.sh that checks for the snap-confine profiles and it explodes in what looks like total random places [08:12] mborzecki: it must be doint something when it loads libGLX_nvidia [08:13] maybe some constructors fire and they clobber things [08:13] mvo: I added the same thing I suspect [08:13] zyga: same result? [08:13] i have a trace with LD_DEBUG=all, gonna look at it now [08:13] https://www.irccloud.com/pastebin/hyOQrz3C/ [08:13] mvo: ^ I never managed to reproduce it [08:13] not once, it ran for 12 hours [08:14] oh, or coffee first [08:14] mvo: I ran into two bugs where my test VM ran out of memory though [08:14] mvo: I will fix those separately [08:15] zyga: interessting, it seems to happen in google everytime, I wonder if something strange is going on in the google vm [08:15] it'd be really nice to have a 'debug' image of the core snap (tar.[gx]z), that you could just set your gdb sysroot to [08:15] oh [08:15] mvo: we reuse machines in google [08:15] oh we don't sorry [08:15] mvo: I need to take the dog out, I opened 4891 for the phase/module system [08:16] mvo: and I'll porpose that anomaly detector I pasted above next [08:16] PR snapd#4891 opened: tests: add support for phased prepare-restore logic [08:16] mvo: I'll be back in 20 minutes === pstolowski|afk is now known as pstolowski [08:17] Morning! [08:18] pstolowski: Chipaca: kalikiana: morning guys [08:22] zyga: is all your 2.32 stuff in PRs now? threspass was the last one? [08:27] It is the last one [08:27] zyga: I just looked at the threspass PR and the errors in the spread run look real (at least some) [08:27] Yes [08:27] See my comment there [08:27] I will fix after the walk [08:28] zyga: aha, sure, thank [08:28] s === chihchun is now known as chihchun_afk [08:50] PR snapd#4884 closed: debian: run snap.mount upgrade fixup *before* debhelper [08:51] PR snapd#4879 closed: tests: revert "tests: disable 18.04 spread tests in google for now" [08:51] PR snapd#4881 closed: tests: make tests run with specific bionic release avoiding take the last one [08:54] re [08:54] mvo: Chipaca: have you seen this https://forum.snapcraft.io/t/var-cache-snapd-commands-db-permission-denied/4590 seems CNF stuff breaks inside classic snap [08:55] hmm, I just saw /etc error on google [08:55] man, I need to run a google loop next [08:55] pedronis: looking [08:55] pedronis: nice [08:55] zyga: as I said my PRs atm get it all the time [08:55] jdstrand: afaict #4509 is unblocked now, do you want to have a final look (the interface itself is trivial) ? [08:55] PR #4509: interfaces/builtin: add support for software-watchdog interface [09:02] pedronis: I'm testing this against google now [09:16] PR snapd#4885 closed: Specify charset in po/snappy.pot === ikey|zzz is now known as ikey === chihchun_afk is now known as chihchun [09:19] PR snapd#4892 opened: tests: update tests to deal with s390x quirks [09:20] PR snapd#4893 opened: po: specify charset in po/snappy.pot [09:49] zyga: 4765 has a conflict, otherwise its good to go [09:49] Thanks, resolving now [09:57] hi, I'm on bionic and I cannot start spotify. [09:57] it fails with [09:57] $ /snap/bin/spotify [09:57] execl failed: No such file or directory [09:57] child exited with status 1 [09:58] anyone seeing this? [10:04] jibel: hey this is a known issue [10:04] https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1756793 [10:04] Bug #1756793: Can't run snaps on Ubuntu 18.04

[10:05] zyga, ah it's the same. Thanks! [10:07] jibel: the snapd in -proposed fixes it, also sudo snap refresh --beta core should work too === chihchun is now known as chihchun_afk [10:08] mvo, yes, I saw the workaround but I'll stay on candidate and wait for the fix. [10:10] is there a way to set configuration options when snapd starts like in a config file instead of running "snap set ..." ? === chihchun_afk is now known as chihchun [10:17] jibel: thanks. autopkgtest is green now so the new snapd should make it to bionic soon (we need to override s390x, autopkgtest is a bit confused, it thinks there is a regression but in reality it was skipped because it was container virt until very recently) [10:33] 4890 needs a second review [10:44] mvo: doing that [10:44] ta [11:05] PR snapd#4894 opened: snap: Call SanitizePlugsSlots from InfoFromSnapYaml (2.32) [11:07] PR snapd#4895 opened: cmd/snap-confine: fix ptrace rule with snap-confine peer (2.32) [11:08] hey zyga mvo could someone take a quick look to #4778, we already fixed spread and it it passing [11:08] PR #4778: tests: moving debian 9 from linode to google backend === chihchun is now known as chihchun_afk [11:09] mvo, about sru, did you reduce the amount of tests to run for autopkgtests? [11:09] because I see that all of them were executed [11:09] zyga: i suspect the problem may be TLS related [11:09] tls? [11:09] thread local storage? [11:09] yes [11:10] i've reordered some calls and got stack smashing detection to abort the binary [11:11] though outside of libglvnd, so i rebuilt libglvnd with -fstack-protector -fstack-protector-all and went on debugging [11:11] stepped through the prologue, pick up the canary, figure out an address and try cathing with watchpoints, nothing turns up [11:12] and the thing is that the value at the address does not change [11:12] cachio: for 2.32 the number of tests in autopkgtest will be reduced, I hope we can get 2.32-final today [11:12] cachio: a bunch of the expensive ones (core transiton) are disabled [11:12] mvo, ok [11:13] but at the end of the function, the canary is calculated again, there's: [11:13] │0x7ffff721365e <__glXLookupVendorByName+7862> mov -0x18(%rbp),%rbx │ [11:13] │0x7ffff7213662 <__glXLookupVendorByName+7866> xor %fs:0x28,%rbx [11:13] cachio: some stuff is still flaky [11:13] zyga: and $fs is different this time [11:13] cachio: like it looks like the upstream autopkgtest env is (somtimes) super slow [11:13] mvo, is it gona be a new snapd for sru? [11:13] cachio: and that kills tests [11:13] zyga: thought this should not change [11:13] mvo, or I should continue with 2.31.2 [11:13] cachio: I think so, I think we can go with 2.32 [11:13] mborzecki: does it happen if you run the same app outside? [11:14] mborzecki: I wonder what is at fault now [11:14] outside of snap stuff works [11:14] i'd guess libc/pthread or something close by [11:14] let me debug this a bit more [11:15] mvo, you can propose "force-badtest snapd/s390x/" into hints-ubuntu branch of britney..... documenting the override..... [11:15] mvo: snap run --gdb is very nice, although `run` in gdb does not work :/ [11:18] mvo, ok [11:29] xnox: I hope the next upload fixes the two remaiing tests [11:30] mborzecki: indeed, just "continue" [11:38] Laney, mvo (repeating from #u-desktop) On a live session, it seems that setting refresh.hold triggers a refresh https://paste.ubuntu.com/p/CKmgFZ8vHV/ [11:38] mwhudson, ^ [11:39] mvo, I repeated the experiment twice and there is a refresh event immediately after setting the config option. [11:39] on a live session we don't want any update at all [11:39] jibel: this is core 16-2.31.2 4206 canonical core [11:39] jibel: we need 2.32 for this to work [11:40] ah crap [11:40] jibel: oh, hold on [11:40] jibel: what does `snap version` print? [11:40] jibel: the core snap is 2.31.2 but snapd on bionic may be newer [11:40] 2.31.2 [11:41] jibel: ok, curious, why? is there a delay in the updates? [11:41] jibel: could you try to manually install the new snapd in the live-cd? [11:41] mvo, I'll try with a newer version [11:41] pstolowski: https://github.com/snapcore/snapd/pull/4890#pullrequestreview-105677607 [11:41] PR #4890: snap: Call SanitizePlugsSlots from InfoFromSnapYaml [11:42] https://www.irccloud.com/pastebin/mD9VRH2i/ [11:42] jibel: thank you [11:42] pedronis: ^ does this make sense? [11:42] I have a debug shell now [11:43] mborzecki: can you have a quick look at 4891 please [11:43] zyga: thanks for the review! [11:48] mvo, I' am going to the doctor now, perhaps I'll be few minutes late for standup [11:48] cachio: ok [11:48] cachio: good luck [11:49] tx [11:49] * cachio afk [11:53] zyga: thanks [11:54] PR snapd#4892 closed: tests: update tests to deal with s390x quirks [11:55] PR snapd#4890 closed: snap: Call SanitizePlugsSlots from InfoFromSnapYaml [11:57] jibel: I think you can `snap set' arbitrary keys before they're known, so this should work with After=snapd.service if you get that problem fixed [12:02] zyga: https://forum.snapcraft.io/t/nvidia-proprietary-driver-no-h-w-acceleration-in-chromium-and-firefox-stack-mashing-problem-also/4532/12 would appreciate any advice [12:02] ack [12:04] mborzecki: quick question, can you link to the code of glXLookupVendorByName pelase [12:04] *please [12:04] zyga: it's there [12:05] ah, I see now [12:06] did you try playing with __GLX_VENDOR_LIBRARY_NAME [12:06] or with __GLX_FORCE_VENDOR_LIBRARY_%d (%d is screen number) [12:06] yeah, but I only have nvidia there [12:07] and it seemed to ignore indirect [12:10] zyga: I have a todo to improve on that [12:10] but it can happen [12:10] zyga: it's related to the fact that our retry mechanism and nonces don't go well together [12:10] ok [12:15] mborzecki: I think I'm lost === chihchun_afk is now known as chihchun [12:19] zyga: that makes the two of us [12:22] Morning all [12:24] sergiusens: Can we have a quick snapcraft release supporting the new refresh-mode option? [12:24] We really need a way to not be blocked by snapcraft on these new features [12:26] Perhaps some sort of --force-passthrough option [12:27] mvo: ack to merge https://github.com/snapcore/snapd/pull/4765/ [12:27] PR #4765: interfaces: harden snap-update-ns profile [12:27] PR snapd#4896 opened: store: support macaroon refreshes in store.InstallRefresh [12:27] I think you said so already but you didn't vote on it [12:32] zyga: ack [12:33] * zyga prepares for some backport time now [12:33] PR snapd#4765 closed: interfaces: harden snap-update-ns profile [12:34] niemeyer: I do not even know what that feature is about [12:35] sergiusens: That's part of the issue :) [12:35] sergiusens: https://forum.snapcraft.io/t/process-lifecycle-on-snap-refresh/140/21 [12:36] yeah, would be nice to be included in the conversations; I do not keep track of every single forum post (and these are easy to flag by your team as needing support from ours) [12:36] sergiusens: Sergio, this is an extremely important part of that job [12:36] sergiusens: If you are not reading the forum, something is not right [12:37] sergiusens: Let's catch up elsewhere about this [12:38] niemeyer: this is 10 days older than your post and not a single comment from the snapd team https://forum.snapcraft.io/t/architectures-keyword-for-build-snapcraft-io/4272 something is certainly wrong [12:42] sergiusens: 1. I've just read it again, and still feel no need to respond; 2. That's completely irrelevant for the point I just made [12:42] mvo: PR 4509 was already on my list. hope to get to it, 4868 and 4851 today (as you may have noticed, I had a lot of forum backlog to tend to) [12:42] PR #4509: interfaces/builtin: add support for software-watchdog interface [12:42] niemeyer: ok, you win, like always [12:43] * sergiusens is not in the mood today [12:43] sergiusens: I only win if we fix the problem, so not yet [12:45] sergiusens: Getting back to the original point, I think we need a way to tell snapcraft to override unknown options and pass them through to snap.yaml on request [12:47] mborzecki: By comparing the config.log between the build environments, the problem turns out to be "-fno-plt" in CFLAGS and "-z,now" in LDFLAGS, which Arch added recently. Wine built with these two flags enabled will have this issue. [12:47] did you see that? [12:48] also https://bugs.winehq.org/show_bug.cgi?id=43530#c35 [12:48] > Someone from NVIDIA confirmed it's a bug in the driver. Hopefully this will be resolved in future releases. [12:48] pffff [12:50] but yeah, i saw that before when it was posted in the forum [12:51] anyways, not sure we would be affected, snaps are built against xenial, so we could have picked up something with libglvnd delivering libGLX*, but I've rebuilt libglvnd on xenial and replaced those pieces [12:52] hmmm [12:52] yes [12:52] i think we are missing something [12:52] there's an ABI conflict that causes GL explosions [12:53] * Son_Goku increasingly wishes he could spend time on making the Fedora core snap [12:54] kyrofa, you've not submitted https://bodhi.fedoraproject.org/updates/FEDORA-2018-81ef149b8b [12:55] it looks like you probably need to deal with this issue: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NZY3P352COQVUWMW6VRNN7CJR7TKKCAQ/ [12:56] i.e.: https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_from_automated_tests [12:56] PR snapd#4897 opened: Specify charset in po/snappy.pot [12:56] err: https://fedoraproject.org/wiki/Package_update_HOWTO#Waive_the_absence_of_a_result [12:56] kyrofa ^ [12:57] niemeyer: ok [12:59] Son_Goku: can you tell us more about the abi issue [13:00] zyga, what it comes down to is that the GL symbols don't bind well between Fedora GL and Ubuntu GL [13:00] the Ubuntu Core is so far in the past, things get broken when the underlying infra changes [13:01] to some extent, we're lucky that the Mesa stuff still works because of how glvnd works [13:02] pstolowski: hey [13:02] pstolowski: standup time [13:25] Son_Goku, zyga: After we're done with the current changes, we should look into implementing a better mechanism to packing such libraries [13:25] yes, yes yes [13:25] YES, please [13:26] any further debugging advice for https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1757427 ? [13:26] Bug #1757427: Unable to find snap at revision, broken snaps