[01:28] <mup> PR snapcraft#2017 closed: many: optimize retrieval of the linker version <bug> <enhancement> <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2017>
[01:31] <mup> PR snapcraft#2018 opened: tests: minor fixes to autopkgtests <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2018>
[01:34] <mup> PR snapcraft#2018 closed: tests: minor fixes to autopkgtests <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2018>
[01:49] <mup> PR snapd#4578 closed: Expose payment and account URLs in /v2/system-info <Created by robert-ancell> <Closed by niemeyer> <https://github.com/snapcore/snapd/pull/4578>
[06:24] <mborzecki> morning
[06:37] <zyga> good morning
[06:37] <mborzecki> zyga: hey
[06:37] <zyga> I lost my phone
[06:37] <mborzecki> ayy
[06:37] <zyga> I put it somewhere and now I cannot find it
[06:37] <zyga> and it's on mute
[06:37] <zyga> na
[06:37] <zyga> man :)
[06:37] <zyga> it's somewhere at home
[06:37] <zyga> or the dog ate it very very politely ;-)
[06:37] <mborzecki> well, that might still be a problem
[06:38] <zyga> I need to fix https://github.com/snapcore/snapd/pull/4889
[06:38] <mup> PR #4889: cmd/snap-update-ns: don't trespass on host filesystem <Created by zyga> <https://github.com/snapcore/snapd/pull/4889>
[06:38] <zyga> and then I'm blissfully happy and can help mvo with the release
[06:38] <mborzecki> jokes aside, i'll be looking into glvnd today, didn't manage to do anything yday, my son has flu-like thing again, so fever and the rest of the package
[06:39] <zyga> mborzecki: uhh, not fun
[06:39] <zyga> this winter is not going anywhere it seems
[06:40] <zyga> but, it's spring time (officially)
[06:44] <zyga> I need to wake kids up
[06:52] <zyga> mborzecki: I was thinking about making sense of prepare restore
[06:52] <zyga> mborzecki: and I came up with a very simple mechanism for making it modular and understandable
[06:52] <zyga> mborzecki: so that we can put some code affecting a given concept in a single file
[06:53] <zyga> mborzecki: and that file can affect any phase of the lifecycle (prepare project, suite, test, restore, etc)
[06:53] <zyga> mborzecki: I will propose the mechanism first and then chop prepare-restore into parts and move it there piece-by-piece
[06:53] <zyga> mborzecki: I did this while running tests all day, looking for the bug that was blocking master yesteday
[06:53] <zyga> mborzecki: I did not reproduce the bug ever
[06:53] <zyga> mborzecki: but I did find that we leave snap userd processes around
[06:54] <zyga> mborzecki: and that leak dbus-daemon processes (probably in the same tests)
[06:54] <zyga> mborzecki: and that my 1.5GB VM runs out of memory after a few hours just because there are so many of those
[06:54] <zyga> mborzecki: anyway, I'll get some coffee and start proposing this
[06:54] <mborzecki> zyga: hm yeah, we should kill whatever userd is left after the test completes
[06:55] <zyga> yes, that's obvious
[06:55] <zyga> just our way of doing prepare restore is making it hard to ensure and reason about
[06:55] <mborzecki> zyga: that might be the reason for one of the failures that pstolowski saw
[06:55] <mborzecki> although that was in unt tests
[06:55] <zyga> the scripts are too large now
[06:55] <zyga> and not maintainable much
[06:57] <mborzecki> heh running into build problems with glvnd on xenial, i'm probably better of pulling the source package from bionic
[06:57] <zyga> good luck with that!
[06:57] <zyga> I need to manage kids a little bit
[06:57] <zyga> then I'll be back
[07:02] <mborzecki> hm given the problems, we should look into having at least one spread test that tries to run glxinfo on a host with nvidia/radeon/intel, perferrably also something with PRIME
[07:03] <zyga> mborzecki: yeah, this is not supe easy due to infra
[07:03] <zyga> mborzecki: I wanted to have a rule where each release is tested on a modern nvidia machine running 16.04 and now 18.04
[07:03] <mborzecki> zyga: maybe at least during validation
[07:03] <zyga> so that we have reasonable coverage of the most popular release
[07:03] <zyga> yes
[07:04] <mborzecki> zyga: and radeon would be intersting too, amdgpu probably works since it's mesa, but i'd guess the pro variant is broken
[07:05] <mborzecki> zyga: you have r9 right? :)
[07:09] <zyga> mborzecki: yes
[07:09] <zyga> mborzecki: I can boot and test things periodically for release
[07:09] <zyga> I was thinking about selling this t470 and getting some variant with the nvidia 150 GPU
[07:09] <zyga> or was it MX150
[07:09] <zyga> something like that
[07:09] <zyga> low end modern core GPU
[07:10] <zyga> oh, it's just 1030 relabelled
[07:10] <zyga> wow, that's impressive then
[07:12] <zyga> It seems I want T480s
[07:12] <zyga> eh, expensive
[07:12]  * zyga wonders when the next laptop refresh cycle is 
[07:13] <mborzecki> extgpu?
[07:13] <zyga> or that :)
[07:14] <zyga> but I'd have to check that it 1) works 2) doesn't need external display
[07:14] <zyga> I want only one display, the one in my laptop
[07:14] <mborzecki> i still need to buy one of rx580/vegas, 1030 is to be only the backup card, the more powerful one will go for passthrough
[07:14] <zyga> do you want one r9 280x?
[07:15] <zyga> I have two
[07:15] <mborzecki> vega :P but I need to wait for btc/xmr/eth to drop
[07:15] <zyga> haha
[07:15] <zyga> yes, that's sad :)
[07:16] <mborzecki> the prices are dropping though, slowly but still, just enough to keep the hopes up
[07:17] <zyga> I think at this rate we'll skip a gen or two
[07:17] <mborzecki> zyga: i have libglvnd rebuilt for xenial, what would be the best way of getting that into the namespace of a snap?
[07:18] <zyga> so, dragons ahead but let's try this
[07:18] <mborzecki> mount bind it somewhere and nsenter?
[07:18] <zyga> open a shell with nsenter -m/run/snapd/ns/foo.mnt
[07:18] <mborzecki> haha that's what i thought :P
[07:18] <zyga> then inside that shell go to /var/lib/snapd/lib/gl
[07:18] <zyga> then it should be a swarm of symlinks, remove them (or move them aside) all
[07:19] <zyga> the place should be a tmpfs, unpack your deb there
[07:19] <zyga> use ldd to see if things resolve one by one
[07:19] <zyga> and let's see what we have
[07:22] <zyga> mvo is not around, I hope he's okay
[07:24] <zyga> good morning mvo
[07:26] <mvo> hey zyga
[07:27] <mborzecki> mvo: morning
[07:33] <mborzecki> zyga: bad news, still segfaulting, although i have a nicer backtrace now
[07:34] <zyga> that's better now
[07:34] <zyga> can you list all the files in that directory?
[07:34] <zyga> as seen from inside the nsenter world
[07:35] <mborzecki> zyga: https://paste.ubuntu.com/p/jBSVD8bG84/
[07:35] <zyga> is lrwxrwxrwx 1 root   maciek     53 Mar 20 17:52 libEGL_nvidia.so.390.42 -> /var/lib/snapd/hostfs/usr/lib/libEGL_nvidia.so.390.42 this okay?
[07:35] <zyga> can you triple check that we only ever open things that are in the nvidia download for linux
[07:35] <zyga> nothing distros build
[07:35] <mborzecki> zyga: yes, that's the glvnd part on nvidia's side
[07:35] <mborzecki> zyga: bt https://paste.ubuntu.com/p/G9SdS24tTJ/
[07:35] <zyga> actually, just download that an unpack it somewhere
[07:36] <zyga> yes, this does look better
[07:36] <mvo> hey mborzecki - good morning!
[07:37] <zyga> mborzecki: another idea, move them out of hostfs
[07:37] <zyga> in case some logic then looks at files in the same directory
[07:37] <zyga> little by little well' get to the bottom of this :)
[07:52] <kalikiana> moin moin
[07:58] <mwhudson> pedronis: er how can i make a date string that snap set core refresh.hold= accepts?
[07:58] <mwhudson> date --rfc-3339=seconds doesn't do it
[07:59] <mwhudson> ahh --iso-8601=seconds
[07:59] <pedronis> yes
[08:01]  * pedronis breakfast
[08:01] <zyga> mborzecki, mvo: https://github.com/snapcore/snapd/compare/master...zyga:feature/test-modules-and-phases?expand=1
[08:02] <zyga> I will open it as soon as I get a pass on my machine
[08:08] <Chipaca> moin moin
[08:09]  * Chipaca not really here yet
[08:09] <mvo> hey Chipaca
[08:09] <Chipaca> zyga: did the themes thing land at any point?
[08:10] <Chipaca> mvo: how's things on this lovely spring day?
[08:10] <mvo> zyga: interessting idea
[08:10] <mvo> Chipaca: things are in progress, we almost landed a fixed 2.32 in bionic, one autopkgtest issue though
[08:10] <Chipaca> asking about themes because of this 'papirus theme crashes snapped evince' thing
[08:11] <mvo> zyga: is all your 2.32 stuff in PRs now? threspass was the last one?
[08:11] <Chipaca> mvo: nice
[08:12] <mborzecki> zyga: hm this doesn't make sense, the segfault is in a mt_mutex_lock (a wrapper around pthread_mutex_lock), it is called a couple of times before the place where it segfaults and seems to work ok, i've confimed that it's trying to load "libGLX_nvidia.so.0", does that successfuly and then goes on to allocate new vendor id and this is where the crash happens
[08:12] <mborzecki> zyga: the wrappers are initialized into a larger struct, and the symbols are picked up one by one using RTLD_DEFAULT at the very beginning
[08:12] <mborzecki> zyga: unless loading libGLX_nvidia.so.0 break all this
[08:12] <mvo> zyga: https://api.travis-ci.org/v3/job/355950075/log.txt is interessting, I added code to reset.sh that checks for the snap-confine profiles and it explodes in what looks like total random places
[08:12] <zyga> mborzecki: it must be doint something when it loads libGLX_nvidia
[08:13] <zyga> maybe some constructors fire and they clobber things
[08:13] <zyga> mvo: I added the same thing I suspect
[08:13] <mvo> zyga: same result?
[08:13] <mborzecki> i have a trace with LD_DEBUG=all, gonna look at it now
[08:13] <zyga> https://www.irccloud.com/pastebin/hyOQrz3C/
[08:13] <zyga> mvo: ^ I never managed to reproduce it
[08:13] <zyga> not once, it ran for 12 hours
[08:14] <mborzecki> oh, or coffee first
[08:14] <zyga> mvo: I ran into two bugs where my test VM ran out of memory though
[08:14] <zyga> mvo: I will fix those separately
[08:15] <mvo> zyga: interessting, it seems to happen in google everytime, I wonder if something strange is going on in the google vm
[08:15] <mborzecki> it'd be really nice to have a 'debug' image of the core snap (tar.[gx]z), that you could just set your gdb sysroot to
[08:15] <zyga> oh
[08:15] <zyga> mvo: we reuse machines in google
[08:15] <zyga> oh we don't sorry
[08:15] <zyga> mvo: I need to take the dog out, I opened 4891 for the phase/module system
[08:16] <zyga> mvo: and I'll porpose that anomaly detector I pasted above next
[08:16] <mup> PR snapd#4891 opened: tests: add support for phased prepare-restore logic <Created by zyga> <https://github.com/snapcore/snapd/pull/4891>
[08:16] <zyga> mvo: I'll be back in 20 minutes
[08:17] <pstolowski> Morning!
[08:18] <mborzecki> pstolowski: Chipaca: kalikiana: morning guys
[08:22] <mvo> zyga: is all your 2.32 stuff in PRs now? threspass was the last one?
[08:27] <zyga>  It is the last one
[08:27] <mvo> zyga: I just looked at the threspass PR and the errors in the spread run look real (at least some)
[08:27] <zyga> Yes
[08:27] <zyga> See my comment there
[08:27] <zyga> I will fix after the walk
[08:28] <mvo> zyga: aha, sure, thank
[08:28] <mvo> s
[08:50] <mup> PR snapd#4884 closed: debian: run snap.mount upgrade fixup *before* debhelper <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4884>
[08:51] <mup> PR snapd#4879 closed: tests: revert "tests: disable 18.04 spread tests in google for now" <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/4879>
[08:51] <mup> PR snapd#4881 closed: tests: make tests run with specific bionic release avoiding take the last one <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4881>
[08:54] <zyga> re
[08:54] <pedronis> mvo: Chipaca: have you seen this  https://forum.snapcraft.io/t/var-cache-snapd-commands-db-permission-denied/4590  seems CNF stuff breaks inside classic snap
[08:55] <zyga> hmm, I just saw /etc error on google
[08:55] <zyga> man, I need to run a google loop next
[08:55] <mvo> pedronis: looking
[08:55] <Chipaca> pedronis: nice
[08:55] <pedronis> zyga: as I said my PRs atm get it all the time
[08:55] <mvo> jdstrand: afaict #4509 is unblocked now, do you want to have a final look (the interface itself is trivial) ?
[08:55] <mup> PR #4509: interfaces/builtin: add support for software-watchdog interface <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4509>
[09:02] <zyga> pedronis: I'm testing this against google now
[09:16] <mup> PR snapd#4885 closed: Specify charset in po/snappy.pot <Created by gunnarhj> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4885>
[09:19] <mup> PR snapd#4892 opened: tests: update tests to deal with s390x quirks <Created by mvo5> <https://github.com/snapcore/snapd/pull/4892>
[09:20] <mup> PR snapd#4893 opened: po: specify charset in po/snappy.pot <Created by mvo5> <https://github.com/snapcore/snapd/pull/4893>
[09:49] <mvo> zyga: 4765 has a conflict, otherwise its good to go
[09:49] <zyga> Thanks, resolving now
[09:57] <jibel> hi, I'm on bionic and I cannot start spotify.
[09:57] <jibel> it fails with
[09:57] <jibel> $ /snap/bin/spotify
[09:57] <jibel> execl failed: No such file or directory
[09:57] <jibel> child exited with status 1
[09:58] <jibel> anyone seeing this?
[10:04] <zyga> jibel: hey this is a known issue
[10:04] <zyga> https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1756793
[10:04] <mup> Bug #1756793: Can't run snaps on Ubuntu 18.04 <amd64> <apport-bug> <bionic> <package-from-proposed> <snapd (Ubuntu):Fix Committed> <https://launchpad.net/bugs/1756793>
[10:05] <jibel> zyga, ah it's the same. Thanks!
[10:07] <mvo> jibel: the snapd in -proposed fixes it, also sudo snap refresh --beta core should work too
[10:08] <jibel> mvo, yes, I saw the workaround but I'll stay on candidate and wait for the fix.
[10:10] <jibel> is there a way to set configuration options when snapd starts like in a config file instead of running "snap set ..." ?
[10:17] <mvo> jibel: thanks. autopkgtest is green now so the new snapd should make it to bionic soon (we need to override s390x, autopkgtest is a bit confused, it thinks there is a regression but in reality it was skipped because it was container virt until very recently)
[10:33] <mvo> 4890 needs a second review
[10:44] <zyga> mvo: doing that
[10:44] <mvo> ta
[11:05] <mup> PR snapd#4894 opened:  snap: Call SanitizePlugsSlots from InfoFromSnapYaml (2.32) <Created by mvo5> <https://github.com/snapcore/snapd/pull/4894>
[11:07] <mup> PR snapd#4895 opened: cmd/snap-confine: fix ptrace rule with snap-confine peer (2.32) <Created by mvo5> <https://github.com/snapcore/snapd/pull/4895>
[11:08] <cachio> hey zyga mvo could someone take a quick look to #4778, we already fixed spread and it it passing
[11:08] <mup> PR #4778: tests: moving debian 9 from linode to google backend <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/4778>
[11:09] <cachio> mvo, about sru, did you reduce the amount of tests to run for autopkgtests?
[11:09] <cachio> because I see that all of them were executed
[11:09] <mborzecki> zyga: i suspect the problem may be TLS related
[11:09] <zyga> tls?
[11:09] <zyga> thread local storage?
[11:09] <mborzecki> yes
[11:10] <mborzecki> i've reordered some calls and got stack smashing detection to abort the binary
[11:11] <mborzecki> though outside of libglvnd, so i rebuilt libglvnd with -fstack-protector -fstack-protector-all and went on debugging
[11:11] <mborzecki> stepped through the prologue, pick up the canary, figure out an address and try cathing with watchpoints, nothing turns up
[11:12] <mborzecki> and the thing is that the value at the address does not change
[11:12] <mvo> cachio: for 2.32 the number of tests in autopkgtest will be reduced, I hope we can get 2.32-final today
[11:12] <mvo> cachio: a bunch of the expensive ones (core transiton) are disabled
[11:12] <cachio> mvo, ok
[11:13] <mborzecki> but at the end of the function, the canary is calculated again, there's:
[11:13] <mborzecki>    │0x7ffff721365e <__glXLookupVendorByName+7862>   mov    -0x18(%rbp),%rbx                                                                                                                                                                   │
[11:13] <mborzecki>    │0x7ffff7213662 <__glXLookupVendorByName+7866>   xor    %fs:0x28,%rbx
[11:13] <mvo> cachio: some stuff is still flaky
[11:13] <mborzecki> zyga: and $fs is different this time
[11:13] <mvo> cachio: like it looks like the upstream autopkgtest env is (somtimes) super slow
[11:13] <cachio> mvo, is it gona be a new snapd for sru?
[11:13] <mvo> cachio: and that kills tests
[11:13] <mborzecki> zyga: thought this should not change
[11:13] <cachio> mvo, or I should continue with 2.31.2
[11:13] <mvo> cachio: I think so, I think we can go with 2.32
[11:13] <zyga> mborzecki: does it happen if you run the same app outside?
[11:14] <zyga> mborzecki: I wonder what is at fault now
[11:14] <mborzecki> outside of snap stuff works
[11:14] <mborzecki> i'd guess libc/pthread or something close by
[11:14] <mborzecki> let me debug this a bit more
[11:15] <xnox> mvo, you can propose "force-badtest snapd/s390x/<the right version>" into hints-ubuntu branch of britney..... documenting the override.....
[11:15] <mborzecki> mvo: snap run --gdb is very nice, although `run` in gdb does not work :/
[11:18] <cachio> mvo, ok
[11:29] <mvo> xnox: I hope the next upload fixes the two remaiing tests
[11:30] <mvo> mborzecki: indeed, just "continue"
[11:38] <jibel> Laney, mvo (repeating from #u-desktop) On a live session, it seems that setting refresh.hold triggers a refresh https://paste.ubuntu.com/p/CKmgFZ8vHV/
[11:38] <jibel> mwhudson, ^
[11:39] <jibel> mvo, I repeated the experiment twice and there is a refresh event immediately after setting the config option.
[11:39] <jibel> on a live session we don't want any update at all
[11:39] <mvo> jibel: this is core                  16-2.31.2  4206  canonical  core
[11:39] <mvo> jibel: we need 2.32 for this to work
[11:40] <jibel> ah crap
[11:40] <mvo> jibel: oh, hold on
[11:40] <mvo> jibel: what does `snap version` print?
[11:40] <mvo> jibel: the core snap is 2.31.2 but snapd on bionic may be newer
[11:40] <jibel> 2.31.2
[11:41] <mvo> jibel: ok, curious, why? is there a delay in the updates?
[11:41] <mvo> jibel: could you try to manually install the new snapd in the live-cd?
[11:41] <jibel> mvo, I'll try with a newer version
[11:41] <zyga> pstolowski: https://github.com/snapcore/snapd/pull/4890#pullrequestreview-105677607
[11:41] <mup> PR #4890: snap: Call SanitizePlugsSlots from InfoFromSnapYaml <Created by stolowski> <https://github.com/snapcore/snapd/pull/4890>
[11:42] <zyga> https://www.irccloud.com/pastebin/mD9VRH2i/
[11:42] <mvo> jibel: thank you
[11:42] <zyga> pedronis: ^ does this make sense?
[11:42] <zyga> I have a debug shell now
[11:43] <zyga> mborzecki: can you have a quick look at 4891 please
[11:43] <mvo> zyga: thanks for the review!
[11:48] <cachio> mvo, I' am going to the doctor now, perhaps I'll be few minutes late for standup
[11:48] <mvo> cachio: ok
[11:48] <mvo> cachio: good luck
[11:49] <cachio> tx
[11:49]  * cachio afk
[11:53] <pstolowski> zyga: thanks
[11:54] <mup> PR snapd#4892 closed: tests: update tests to deal with s390x quirks <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4892>
[11:55] <mup> PR snapd#4890 closed: snap: Call SanitizePlugsSlots from InfoFromSnapYaml <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/4890>
[11:57] <Laney> jibel: I think you can `snap set' arbitrary keys before they're known, so this should work with After=snapd.service if you get that problem fixed
[12:02] <mborzecki> zyga: https://forum.snapcraft.io/t/nvidia-proprietary-driver-no-h-w-acceleration-in-chromium-and-firefox-stack-mashing-problem-also/4532/12 would appreciate any advice
[12:02] <zyga> ack
[12:04] <zyga> mborzecki: quick question, can you link to the code of glXLookupVendorByName pelase
[12:04] <zyga> *please
[12:04] <mborzecki> zyga: it's there
[12:05] <zyga> ah, I see now
[12:06] <zyga> did you try playing with __GLX_VENDOR_LIBRARY_NAME
[12:06] <zyga> or with __GLX_FORCE_VENDOR_LIBRARY_%d (%d is screen number)
[12:06] <mborzecki> yeah, but I only have nvidia there
[12:07] <mborzecki> and it seemed to ignore indirect
[12:10] <pedronis> zyga: I have a todo to improve on that
[12:10] <pedronis> but it can happen
[12:10] <pedronis> zyga: it's related to the fact that our retry mechanism and nonces don't go well together
[12:10] <zyga> ok
[12:15] <zyga> mborzecki: I think I'm lost
[12:19] <mborzecki> zyga: that makes the two of us
[12:22] <niemeyer> Morning all
[12:24] <niemeyer> sergiusens: Can we have a quick snapcraft release supporting the new refresh-mode option?
[12:24] <niemeyer> We really need a way to not be blocked by snapcraft on these new features
[12:26] <niemeyer> Perhaps some sort of --force-passthrough option
[12:27] <zyga> mvo: ack to merge https://github.com/snapcore/snapd/pull/4765/
[12:27] <mup> PR #4765: interfaces: harden snap-update-ns profile <Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/4765>
[12:27] <mup> PR snapd#4896 opened: store: support macaroon refreshes in store.InstallRefresh <Created by pedronis> <https://github.com/snapcore/snapd/pull/4896>
[12:27] <zyga> I think you said so already but you didn't vote on it
[12:32] <mvo> zyga: ack
[12:33]  * zyga prepares for some backport time now
[12:33] <mup> PR snapd#4765 closed: interfaces: harden snap-update-ns profile <Critical> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4765>
[12:34] <sergiusens> niemeyer: I do not even know what that feature is about
[12:35] <niemeyer> sergiusens: That's part of the issue :)
[12:35] <niemeyer> sergiusens: https://forum.snapcraft.io/t/process-lifecycle-on-snap-refresh/140/21
[12:36] <sergiusens> yeah, would be nice to be included in the conversations; I do not keep track of every single forum post (and these are easy to flag by your team as needing support from ours)
[12:36] <niemeyer> sergiusens: Sergio, this is an extremely important part of that job
[12:36] <niemeyer> sergiusens: If you are not reading the forum, something is not right
[12:37] <niemeyer> sergiusens: Let's catch up elsewhere about this
[12:38] <sergiusens> niemeyer: this is 10 days older than your post and not a single comment from the snapd team https://forum.snapcraft.io/t/architectures-keyword-for-build-snapcraft-io/4272 something is certainly wrong
[12:42] <niemeyer> sergiusens: 1. I've just read it again, and still feel no need to respond; 2. That's completely irrelevant for the point I just made
[12:42] <jdstrand> mvo: PR 4509 was already on my list. hope to get to it, 4868 and 4851 today (as you may have noticed, I had a lot of forum backlog to tend to)
[12:42] <mup> PR #4509: interfaces/builtin: add support for software-watchdog interface <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/4509>
[12:42] <sergiusens> niemeyer: ok, you win, like always
[12:43]  * sergiusens is not in the mood today
[12:43] <niemeyer> sergiusens: I only win if we fix the problem, so not yet
[12:45] <niemeyer> sergiusens: Getting back to the original point, I think we need a way to tell snapcraft to override unknown options and pass them through to snap.yaml on request
[12:47] <zyga> mborzecki: By comparing the config.log between the build environments, the problem turns out to be "-fno-plt" in CFLAGS and "-z,now" in LDFLAGS, which Arch added recently. Wine built with these two flags enabled will have this issue.
[12:47] <zyga> did you see that?
[12:48] <zyga> also https://bugs.winehq.org/show_bug.cgi?id=43530#c35
[12:48] <mborzecki> > Someone from NVIDIA confirmed it's a bug in the driver. Hopefully this will be resolved in future releases.
[12:48] <mborzecki> pffff
[12:50] <mborzecki> but yeah, i saw that before when it was posted in the forum
[12:51] <mborzecki> anyways, not sure we would be affected, snaps are built against xenial, so we could have picked up something with libglvnd delivering libGLX*, but I've rebuilt libglvnd on xenial and replaced those pieces
[12:52] <zyga> hmmm
[12:52] <zyga> yes
[12:52] <zyga> i think we are missing something
[12:52] <Son_Goku> there's an ABI conflict that causes GL explosions
[12:53]  * Son_Goku increasingly wishes he could spend time on making the Fedora core snap
[12:54] <Son_Goku> kyrofa, you've not submitted https://bodhi.fedoraproject.org/updates/FEDORA-2018-81ef149b8b
[12:55] <Son_Goku> it looks like you probably need to deal with this issue: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NZY3P352COQVUWMW6VRNN7CJR7TKKCAQ/
[12:56] <Son_Goku> i.e.: https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_from_automated_tests
[12:56] <mup> PR snapd#4897 opened: Specify charset in po/snappy.pot <Created by gunnarhj> <https://github.com/snapcore/snapd/pull/4897>
[12:56] <Son_Goku> err: https://fedoraproject.org/wiki/Package_update_HOWTO#Waive_the_absence_of_a_result
[12:56] <Son_Goku> kyrofa ^
[12:57] <sergiusens> niemeyer: ok
[12:59] <zyga> Son_Goku: can you tell us more about the abi issue
[13:00] <Son_Goku> zyga, what it comes down to is that the GL symbols don't bind well between Fedora GL and Ubuntu GL
[13:00] <Son_Goku> the Ubuntu Core is so far in the past, things get broken when the underlying infra changes
[13:01] <Son_Goku> to some extent, we're lucky that the Mesa stuff still works because of how glvnd works
[13:02] <zyga> pstolowski: hey
[13:02] <zyga> pstolowski: standup time
[13:25] <niemeyer> Son_Goku, zyga: After we're done with the current changes, we should look into implementing a better mechanism to packing such libraries
[13:25] <zyga> yes, yes yes
[13:25] <Son_Goku> YES, please
[13:26] <sparkiegeek> any further debugging advice for https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1757427 ?
[13:26] <mup> Bug #1757427: Unable to find snap at revision, broken snaps <amd64> <apport-bug> <bionic> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1757427>
[13:27] <sparkiegeek> my systemd/journald-fu is weak, so if anyone has tips on how I can figure out *why* the mount units are AWOL I'm all ears :)
[13:27] <mup> PR snapd#4893 closed: po: specify charset in po/snappy.pot <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4893>
[13:35] <Chipaca> #1757427 seems to be a dupe but I'm not sure of which one
[13:35] <mup> Bug #1757427: Unable to find snap at revision, broken snaps <amd64> <apport-bug> <bionic> <snapd (Ubuntu):New> <https://launchpad.net/bugs/1757427>
[13:37] <sparkiegeek> Chipaca: hmm, I am aware of that bug, but it didn't seem to be the same? I didn't see the execl issue, and the workaround that zyga posted didn't work
[13:39] <sparkiegeek> Chipaca: or am I missing that the underlying cause is the same?
[13:39] <Chipaca> sparkiegeek: it's not the 'snap.mount makes everything broken' afaik
[13:39] <Chipaca> sparkiegeek: but the other one that i don't know the bug# of
[13:39] <Chipaca> sparkiegeek: the 'snap.mount makes everything broken' one is #1756793
[13:39] <mup> Bug #1756793: Can't run snaps on Ubuntu 18.04 <amd64> <apport-bug> <bionic> <package-from-proposed> <snapd (Ubuntu):Fix Committed> <https://launchpad.net/bugs/1756793>
[13:39] <sparkiegeek> so right now my bug is duped to #1756793
[13:40] <Chipaca> ah :-)
[13:40] <Chipaca> afaik it's a separate issue, but has also been addressed if I understood zyga in the standup just now
[13:41] <zyga> two bugs, yes
[13:41] <zyga> to fix the other one just reboot
[13:41] <zyga> (the mount is broken one)
[13:49] <zyga> cachio: can you give me a reference to the failure you saw that you just mentioned
[14:01] <zyga> I have some follow-up for that makes everything modular and easier to understand
[14:11] <kyrofa> Son_Goku, yes, thank you, I meant to reach out to you yesterday about that
[14:12] <Son_Goku> kyrofa, no problem
[14:13] <kyrofa> Son_Goku, FEDORA-EPEL-2018-0c373ee2d6 has issues as well, but I can't figure out what they are :P
[14:13] <Son_Goku> it has one more day left before you can push it
[14:13] <Son_Goku> look at the "days to stable" on the bottom right ;)
[14:13] <kyrofa> Oh! I thought they were all a week
[14:13] <Son_Goku> the counter starts from when it makes it into the testing repo
[14:13] <kyrofa> Ah
[14:13] <Son_Goku> nope, EPEL is 14 days from when it hits testing
[14:14] <Son_Goku> Fedora is 7 days from when it hits testing
[14:14] <kyrofa> Okay, that makes sense then
[14:14] <Son_Goku> ;)
[14:21] <zyga> I will break for lunch/walk now
[14:23] <mup> PR snapd#4895 closed: cmd/snap-confine: fix ptrace rule with snap-confine peer (2.32) <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4895>
[14:24] <mup> PR snapd#4894 closed:  snap: Call SanitizePlugsSlots from InfoFromSnapYaml (2.32) <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4894>
[14:30] <jdstrand> roadmr: hi! can you sync r1013 of the review tools? this will cut the review time by half
[14:31] <roadmr> jdstrand: oh how does it do it? sounds magical! yes, I'll put it in the queue in a few mins (otp)
[14:31] <jdstrand> roadmr: I noticed that the tools were running expensive python magic calls on files twice! now it is only once :)
[14:31] <roadmr> oh hahah :) nice catch
[14:32] <jdstrand> and the review time is dominated by python-magic. at some point I'll see if I can optimize further, but I thought this speed improvement quite nice
[14:34] <sparkiegeek> roadmr: wait, when you said it sounds magical, did you know that it was python-magic that was fixed?!
[14:34] <cachio> zyga, https://travis-ci.org/snapcore/snapd/builds/356291630#L6914
[14:35] <cachio> zyga, this is quite new
[14:35] <cachio> zyga, the branch should have the latest changes from master
[14:36] <zyga> Ack
[14:41] <mborzecki> off to pick u pthe kids
[15:08] <sparkiegeek> zyga: thanks, reboot seems to have fixed that issue
[15:08] <sparkiegeek> I suggest that the bug be de-duped though, since IIUC you ack'd that they are separate issues
[15:16] <pedronis> zyga: I just got the snap-confine error running only google:ubuntu-16.04-64:tests/main/...
[15:16] <pedronis> (using staging store but shouldn't be different)
[15:25] <zyga> pedronis: the one with extra profile with garbage filename?
[15:25] <pedronis> yes
[15:25]  * zyga is almost back, will check stuff now
[15:25] <pedronis> running from here
[15:25] <pedronis> to google
[15:25] <pedronis> not from travies
[15:25] <pedronis> fwiw
[15:28]  * cachio afk
[15:32] <mvo> niemeyer: sorry for being a pain, your input on 4882 would be great, this and 4889 are the only things left for 2.32-final
[15:32] <zyga> mvo: I'm about to backport things I did to 2.32
[15:32] <zyga> mvo: did you do any of my branches by any chance?
[15:32] <mvo> zyga: I didn't
[15:32] <zyga> ok
[15:33] <zyga> https://github.com/snapcore/snapd/pull/4889 is green!
[15:33] <mup> PR #4889: cmd/snap-update-ns: don't trespass on host filesystem <Created by zyga> <https://github.com/snapcore/snapd/pull/4889>
[15:33] <mvo> zyga: I'm fine with blacklisting the layouts test for now in autopkgtest
[15:33] <zyga> ok
[15:33] <mvo> zyga: unless the fixes for the test are easy and quick to backport, that is of course perferable
[15:34]  * mvo needs to step out for some minutes first
[15:35] <zyga> well, given that we still see odd things happening I suspect there's something broken still
[15:38] <niemeyer> mvo: Will look right away
[15:38] <niemeyer> zyga, mvo: Do we really want https://github.com/snapcore/snapd/pull/4765 now?  Similar changes have taken a while to stabilize.. unless this is fixing something, it doesn't seem like the right time to touch it
[15:38] <mup> PR #4765: interfaces: harden snap-update-ns profile <Critical> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4765>
[15:38] <niemeyer> Is that actually fixing something, or required by something else?  It doesn't look like that from the description
[15:52] <zyga> niemeyer: this is fixing the open profiles per request from jdstrand
[15:52] <zyga> niemeyer: when we did layouts we opened the profile for snap-confine/snap-update-ns significantly
[15:53] <zyga> niemeyer: this is the last bit that undoes that
[15:53] <zyga> niemeyer: jamie requested that it is present in the release
[15:54] <niemeyer> zyga: Which again makes me quite concerned
[15:55] <niemeyer> zyga: Previous times we played with that took a while to stabilize, and we don't have a lot of time anymore.
[15:55] <zyga> right, I see
[15:55] <zyga> if jdstrand acks this we can keep the open (permissive profiles)
[15:55] <zyga> and land the hardening for 2.33
[15:55] <niemeyer> We have plenty of other issues to worry about
[15:57] <popey> https://forum.snapcraft.io/t/cant-launch-skype-snap-relocation-error/4536
[15:57] <niemeyer> zyga, mvo: What's the plan for 2.32 to land in stable/
[15:57] <niemeyer> ?
[15:57] <popey> Any of you on 17.10 or 18.04 can reproduce this issue with skyp?
[15:57] <zyga> niemeyer: I don't know more beyond prepare the release PRs and land them
[15:57] <popey> Skype works on 16.04 for me, but not on 18.04. Did we break it or did they?
[15:59] <zyga> gnome-shell just crashed, eh
[16:00] <zyga> popey: actually, running skype on my bionic system crashes gnome shell
[16:00] <zyga> I get a segvfault in xwayland
[16:00] <popey> Maybe don't use wayland?
[16:01] <mvo> niemeyer: I had hoped early next week, but that may be too ambitious, I want to get 2.32-final out today, two PRs left:  4889 and 4882
[16:01] <zyga> popey: x11 corrputs my display
[16:01] <zyga> popey: wayland doesn't
[16:01] <niemeyer> mvo: Those testing periods are getting too short for the kind of change being merged I'm afraid
[16:01] <zyga> (I get to pick my kind of pain, that's all)
[16:02] <mvo> niemeyer: agreed, we need to be careful at this point
[16:02] <mvo> niemeyer: we can extend the testing, we could add an extra week
[16:02] <niemeyer> mvo: For example, we just merged that major change in the profile of snap-confine, which is fundamental for everything
[16:02] <mvo> niemeyer: that might be a good idea in any case, lots of churn in 2.32
[16:02] <mvo> niemeyer: that merge only hit master, no?
[16:03] <niemeyer> mvo: Yeah, and not just that.. besides the timing between releases, there's the time from merge to stable
[16:03] <niemeyer> mvo: IOW, merging now and branching off means pretty much only two days of testing
[16:04] <niemeyer> mvo: Right
[16:04] <zyga> mvo: note: I will have some backports still for 2.32
[16:04] <mvo> niemeyer: release/2.32 is branched since some days alrerady, we only "cherry-pick". but we did some large cherry-picks, I guess this is your concern (and I agree)
[16:04] <zyga> those that merged into master with 2.32 label
[16:05] <mvo> zyga: for what exactly?
[16:05] <zyga> 4898 is one
[16:05] <zyga> this fixes layouts on refreshes
[16:05] <zyga> (and content interface to some extent)
[16:05] <mup> PR snapd#4898 opened: cmd/snap-update-ns: use x-snapd.{synthetic,needed-by} in practice <Created by zyga> <https://github.com/snapcore/snapd/pull/4898>
[16:06] <mvo> zyga: hm, that is not even merged into master yet :/
[16:06] <zyga> what?
[16:06] <zyga> it is merged, I linked to the 2.32 PR
[16:06] <mvo> zyga: oh, sorry
[16:07] <mvo> zyga: this one is fine, is there more to come? (and if so, how much?)
[16:07] <zyga> I think two more, one moment
[16:07] <zyga> the rest is smaller
[16:07] <zyga> and all are backports from master
[16:08] <niemeyer> mvo: Sure, but as I understand it these changes are/were going into it?
[16:08] <niemeyer> mvo: I mean, snap-confine changes, snap-update-ns, ...
[16:08] <zyga> s-u-n
[16:09] <niemeyer> zyga: 4889 reviewed.. I'm worried about progress in that corner
[16:09] <niemeyer> zyga: Feels too loose
[16:10] <mvo> niemeyer: yes, a lot of changes went into 2.32
[16:11] <niemeyer> mvo: I get it, but I'm raising a more specific issue:
[16:11] <niemeyer> mvo: There's a big difference between landing something in master early in the cycle, and committing something in master right before we bake the beta, even more if the idea is to have a fast release that goes to stable too quickly
[16:12] <mvo> niemeyer: yes, agreed. if you want we can have a quick HO to discuss quicker (but here is fine as well of course)
[16:13] <niemeyer> mvo: Yeah, let's get to the standup
[16:13] <niemeyer> zyga: Can you come?
[16:13] <zyga> yes
[16:13] <niemeyer> jdstrand: And you too, if we're lucky enough to have you around
[16:39] <abeato> tyhicks, do you know how the rng_core.default_quality kernel setting works?
[16:48] <tyhicks> abeato: hey - I used to have all of those details in my head
[16:48] <tyhicks> abeato: if you have specific questions, it might jog my memory
[16:49] <abeato> tyhicks, if it is set to, say, 700, how is that related to /proc/sys/kernel/random/entropy_avail?
[16:51] <tyhicks> abeato: give me a sec to scan the code
[17:02] <tyhicks> abeato: it is starting to come back to me now... default_quality is a way to express the "Estimation of true entropy in RNG's bitstream (per mill)" for hardware random number generators that don't have a pre-defined quality value
[17:03] <tyhicks> abeato: some hwrng devices have a pre-defined quality value and default_quality is ignored in those cases
[17:04] <tyhicks> abeato: for devices that don't have a pre-defined quality value, default_quality is used when determining how much entropy is available after reading a certain number of bytes from the hwrng device
[17:04] <tyhicks> abeato: that would then help determine the value of entropy_avail
[17:05] <abeato> tyhicks, oh, ok, so it is a constant tied to the hw to help to estimate entropy
[17:05] <abeato> (available entropy)
[17:06] <tyhicks> abeato: that's correct
[17:06] <abeato> tyhicks, got it then, thanks a lot
[17:06] <tyhicks> abeato: the idea is that some hwrng devices have a known quality constant that's hard coded into the kernel sources
[17:07] <tyhicks> abeato: other devices, for example, may not have enough public documentation published about them and it is impossible for the driver developer to place a pre-defined quality constant in the source code
[17:08] <abeato> right, makes sense
[17:08] <tyhicks> abeato: in those situations, the driver developer defines the hwrng quality as "0" and lets the system integrator/administrator define the quality constant that they are comfortable with
[17:11] <tyhicks> abeato: one last tip - the range is from 0 (no quality at all) to 1024
[17:12] <abeato> tyhicks, got it. Another question, is the randomness exposed through /dev/random or via /dev/hwrng only?
[17:14] <mup> PR snapd#4899 opened: many: backported fixes for layouts and symlinks (2.32) <Created by zyga> <https://github.com/snapcore/snapd/pull/4899>
[17:14] <zyga> mvo: ^ that's all of it
[17:14] <tyhicks> abeato: it is injected into the /dev/random pool
[17:14] <zyga> mvo: and this will also explain why some 2.32 branches were failing, the fixes are only in master
[17:15] <zyga> I will take a short break and then look at adressing some feedback from the call
[17:15] <abeato> tyhicks, ok, thanks
[17:17] <mvo> zyga: thanks, I have a look in a little bit
[17:17] <zyga> all of those were clean cherry picks
[17:17] <zyga> no conflicts
[17:18] <zyga> hmmm, I don't see travis running against 2.32
[17:20] <mborzecki> zyga: got back just now and tried xenial chroot, glxinfo is segfaulting at exactly the same spot
[17:20] <zyga> mborzecki: cool
[17:20] <zyga> can you boot an unbuntu kernel
[17:20] <zyga> ubuntu kernel
[17:20] <zyga> and see if that makes it go away in your xenial chroot first
[17:21] <zyga> and then in your snap env
[17:21] <zyga> I smell a kernel option that breaks  this in arch, assuming all code is compiled with arch toolchain with some specific tweaking
[17:21] <zyga> mvo: offtopic, on pre6 none of my classic snaps run
[17:21] <zyga> oh, sorry, atom ran, it was just slow
[17:22] <zyga> it was just skype that doesn't run
[17:22] <mborzecki> ok, let's baseline, libglvnd built with xenial toolchain, nvidia 390 as it comes from arch packages, xenial chroot (from current cloud image)
[17:23] <mborzecki> i'm leaving playing with ubuntu kernel (and getting the matching drivers) for tomorrow, need to prep dinner for the kids
[17:24] <zyga> ok
[17:26] <niemeyer> mvo: Quick thought: I think it's safer to compare the in-memory value of unmarshaling the json than the json blob itself
[17:26] <niemeyer> mvo: With reflect.DeepEqual
[17:27] <niemeyer> mvo: I'm considering ordering issues, etc
[17:56] <jdstrand> niemeyer: sorry, I was out when I got pinged. I'll happily follow whatever outcome you, mvo and others came up with
[17:57] <niemeyer> jdstrand: No worries, it's all good
[17:57] <niemeyer> jdstrand: We'll just move forward, but work harder on testing this time
[17:57] <niemeyer> jdstrand: So we get a similar benefit of having more resting time
[17:59] <jdstrand> niemeyer: reading backscroll, it sounds like this is a 'general problem' rather than a 'specific incident' to avoid. it makes sense to be careful with what gets committed right before cutting a beta, sure
[17:59] <niemeyer> jdstrand: Right, that's the key idea
[17:59] <niemeyer> jdstrand: That's extra true if we need to fast track the release due to requirements
[17:59] <jdstrand> it's always difficult to balance that with outside pressures, but I don't think there is anything wrong with saying "no, you need to wait" more often
[18:00]  * jdstrand nods
[18:00] <niemeyer> jdstrand: Yeah, definitely.. it's a careful balance
[18:00]  * jdstrand nods
[18:02] <mup> PR snapd#4897 closed: Specify charset in po/snappy.pot <Created by gunnarhj> <Closed by gunnarhj> <https://github.com/snapcore/snapd/pull/4897>
[18:04] <mup> PR snapd#4898 closed: cmd/snap-update-ns: use x-snapd.{synthetic,needed-by} in practice (2.32) <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/4898>
[18:12] <jdstrand> niemeyer: perhaps this is worth discussing: it sounds like this is prompted by zyga's snap-update-ns hardening branch. this is part of layouts and because everything was broken up in many PRs, the last major piece of layouts I acked some time ago, but I ack'd it conditional on (at the time) a future hardening PR such that 2.32 would have the hardening in place
[18:13] <jdstrand> niemeyer: this was probably a mistake since other PRs built on top of the layouts feature and rework, and the hardening came quite a bit later and wanting to be jammed in at the last minute
[18:14] <jdstrand> niemeyer: because of my conditional ack, it put zyga and everyone in a hard place since it made backing out the conditionally acked pr difficult. in the future I won't do that
[18:15]  * zyga recalls the moment when we realised this was a * rw, permission
[18:15] <jdstrand> niemeyer: if a similar situation comes up, I'll instead leave as 'Request changes' until the conditional followup PR is approved and ready, then they could both go at the same time
[18:17] <Chipaca> hmm
[18:17] <Chipaca> what's "grade" doing in snap.yaml in the wild?
[18:18] <zyga> Chipaca: hiding
[18:19] <jdstrand> Chipaca: personally, I use grade to make sure nothing ever goes to stable. I do this with my test snaps, for example
[18:19] <Chipaca> jdstrand: but that's in snapcraft.yaml, right?
[18:19] <jdstrand> I'm not sure if that was your question, but that's my answer
[18:21] <jdstrand> Chipaca: grade was added to the review tools ages ago. I think it needs to be in the snap.yaml because the store looks at it to enforce things (it doesn't have snapcraft.yaml). I could be wrong. cprov should know since he added initial grade support to the review tools
[18:25] <niemeyer> jdstrand: Thanks for the details, and I definitely don't blame you for that one. I think it was a reasonable decision in the context we had at the time.
[18:25] <niemeyer> Chipaca: No, grade is for snap.yaml
[18:26] <cprov> jdstrand, Chipaca: grade is used in the store to prevent release of `devel` revisions in stable risks (stable, candidate). What is the problem in the snap.yaml hierarchy you are referring to ?
[18:26] <niemeyer> Chipaca: It's supposed to prevent people from releasing snaps into candidate and stable if they are marked as unstable
[18:26] <Chipaca> I was just surprised because I didn't know all this, and snapd ignores its existence
[18:26] <niemeyer> Chipaca: We should at least validate it, but we don't need to do anything with it I think
[18:26] <niemeyer> Chipaca: It's a snap.yaml setting that is more critical for the store
[18:27] <Chipaca> niemeyer: we don't even parse it, today :-)
[18:27] <jdstrand> niemeyer: ok, thanks. I suspect attempting my suggested future workflow is a good idea. some PRs end up being quite special though.... let's hope we don't have too many more of those ;)
[18:27] <niemeyer> Chipaca: I get it.. we should probably do it for validation, but it's not crticial.. it's okay for snapd to accept it. If it gets there it's too late for the setting to make sense. The store is the one that should reject it.
[18:28] <jdstrand> cprov: no problem, Chipaca was wondering if it should be in snapcraft.yaml and I thought I remembered why it needed to be in snap.yaml, and mentioned you since you could back up my claim (which you did) or correct me
[18:28] <niemeyer> No, it's really meant for snap.yaml.. none of snapcraft.yaml is defined in the output format
[18:28] <Chipaca> jdstrand: cprov: niemeyer: it was just me being curious :-)
[18:28] <cprov> :-)
[18:29] <niemeyer> Chipaca: I knoooowww.. :)
[18:29] <niemeyer> Chipaca: Was simply explaining it
[18:29] <jdstrand> niemeyer: re snap.yaml> yep
[18:29] <niemeyer> We also agreed to rename it to unstable instead of devel, but that never went through for lack of time
[18:37] <jdstrand> mvo: is 'core18' the final name for the core 18 base snap? I'm adjusting the review tools for it and see it was base-18 before. core18, not core-18, right?
[18:39] <Chipaca> niemeyer: I have a problem with an aspect of 'broken' snapshots
[18:40] <Chipaca> niemeyer: basically one of the snapshots in a set could be broken, which means listing it in 'snap saved' is tricky
[18:41] <niemeyer> Hmm
[18:41] <Chipaca> niemeyer: I played with adding a '!' after the name of the snap snapshot that was broken, but (a) the information of _how_ it was broken is lost, and (b) if you have 200 snaps installed and the broken one is zzt, you won't see that '!'
[18:41] <niemeyer> Chipaca: I wonder if we should expand it out
[18:42] <niemeyer> Chipaca: I thought about this earlier for different reasons, but I think we never discussed it
[18:42] <Chipaca> so one option would be to have a saved --long
[18:42] <Chipaca> that would list more info about each set
[18:42] <jdstrand> kalikiana: hi! I see the 'vendorize' snap is classic. what does it do?
[18:43] <Chipaca> niemeyer: then the short listing could have a single marker to say "check engine"
[18:43] <niemeyer> Chipaca: It's also pretty obscure to just have a !
[18:43] <niemeyer> Chipaca: Too subtle, too rarely.. probably won't work
[18:43] <Chipaca> niemeyer: you're right, I'll make it a ‼
[18:43] <niemeyer> Chipaca: It worked for yaml.. :)
[18:44] <Chipaca> FWIW it was: have ! in the list, and if you have any !, add a "! means broken, yadda yadda"
[18:44] <niemeyer> Which probably means it's a counter example.. :P
[18:44] <Chipaca> which with --long would be "! means broken, see --long"
[18:44] <niemeyer> Chipaca: The current design, although not nearly as bad, reminds me a bit of snap interfaces in terms of the grouping
[18:45] <Chipaca> niemeyer: the current design of snapshotses?
[18:45] <niemeyer> Chipaca: No, the output format..
[18:45] <niemeyer> Chipaca: I mean, we also have that sort of grouping per line on "snap interfaces"
[18:45] <niemeyer> Chipaca: But we list it all, and it sucks for other unrelated reasons..
[18:46] <niemeyer> Chipaca: Still, the grouping issue is similar.. we lose the ability to talk about the individual entries
[18:46] <kyrofa> Son_Goku, this "waive absence of result" thing is soaring over my head. I seriously have to send a raw POST to greenwave to unblock things? What is an NVR?
[18:46] <niemeyer> Chipaca: We can play with an analogy with "snap list", though
[18:46] <niemeyer> Chipaca: e.g. snap list --all
[18:47] <Son_Goku> kyrofa, name-version-release
[18:47] <kyrofa> Ah ha
[18:47] <Son_Goku> and yeah, there's a button coming to bodhi in the coming weeks
[18:47] <Son_Goku> greenwave is... not fully integrated yet
[18:52] <niemeyer> Chipaca: So, strawman for brainstorm: "snap saved" keeps the grouping, shows "broken" in notes similar to how it works in "snap list"  , and "snap saved --all" expands the listing to show a single snap per line (so repeated set ids, potentially), and identifies which specific snap is broken
[18:53] <niemeyer> Chipaca: "snap list" can take an optional [<snap name>] as usual, in which case the output would be effectively the same as "snap list --all" filtered for that particular snap
[18:53] <niemeyer> Chipaca: Waddathink?
[18:54] <Chipaca> niemeyer: basic idea seems a'ight, although there's more info we could show than is conveyed by --all
[18:54] <Chipaca> niemeyer: give me a minute or 5
[18:54] <mup> PR snapd#4900 opened: many: use the new install/refresh API by switching snapstate to use store.InstallRefresh <Created by pedronis> <https://github.com/snapcore/snapd/pull/4900>
[18:56] <Saviq> cachio: hey, we're still ENOSPC'ing on fedora... https://travis-ci.org/MirServer/mir/jobs/356462120 am I using the right images and such https://github.com/MirServer/mir/pull/267/files? (-64 suffix doesn't work yet)
[18:56] <mup> PR MirServer/mir#267: [travis] move Fedora over to GCE, too <Created by Saviq> <https://github.com/MirServer/mir/pull/267>
[18:59] <cachio> Saviq, try using this image --> image: fedora-cloud-base-27-1-6-x86-64
[19:00] <cachio> Saviq, I'll update the name soon to make it match automatically
[19:00] <Saviq> tx
[19:00] <cachio> Saviq, which sprad are you using?
[19:01] <cachio> are you downloading spread for each test execution?
[19:01] <Saviq> cachio: Gustavo's https://github.com/Saviq/mir/blob/9df8e34b514608503f88a723ade8df4bf2446be6/.travis.yml#L57
[19:02] <cachio> Saviq, this one should be updated, could you please debug the execution from your localhost and check if the host has 10GB of disk?
[19:02] <kyrofa> Son_Goku, so I got the test failure from greenwave, but waiverdb-cli doesn't have the same interface as documented there, at least on f26
[19:03]  * Son_Goku sighs
[19:03] <cachio> just to make sure that spread is the correct one
[19:03] <kyrofa> It requires an ID, and the curl command it's giving me gies me nothing :P
[19:03] <Son_Goku> kyrofa, I suggest checking in #fedora-releng
[19:03] <cachio> otherwise we can try adding more space
[19:03] <Son_Goku> one of those folks can help
[19:03] <Son_Goku> I have no clue :P
[19:03] <kyrofa> Son_Goku, haha, okay glad I'm not the only one
[19:03] <Son_Goku> I've been lucky enough that greenwave hasn't failed my updates yet
[19:03] <Son_Goku> though I think that's going to happen when I make the Mir update for Fedora 28
[19:04] <niemeyer> cachio, Saviq: Question might be how much free space that specific image has, instead of how much space was allocated for it
[19:05] <niemeyer> cachio: How much space is the image itself taking out of the 10GB?
[19:05] <Chipaca> niemeyer: https://pastebin.ubuntu.com/p/kH8Wj5vZsS/
[19:06] <cachio> niemeyer, well, first I want to know is the image has 10GB, in that case, it they are going out of space they need to define the storage for that specific image
[19:06] <niemeyer> Chipaca: That reminds me a *lot* of the output of snap interfaces
[19:07] <niemeyer> Chipaca: That help message at the end also feels a bit awkward to have there all the time
[19:07] <Chipaca> niemeyer: it'd only be there if a ! is shown
[19:07] <cachio> niemeyer, sadly, the don't have any free space
[19:07] <niemeyer> Chipaca: If the UX is right we shouldn't need it
[19:07] <Chipaca> niemeyer: the output with --long should also remind you of the output of snap list :-)
[19:07] <niemeyer> cachio: That doesn't make much sense
[19:08] <niemeyer> cachio: The image wouldn't take 10GB for data by itself
[19:09] <niemeyer> Chipaca: The second output looks so much better that it should really be the default one
[19:09] <Chipaca> niemeyer: oh wait, the _first_ one is the one you were objecting to?
[19:09] <Chipaca> heh
[19:09] <niemeyer> Chipaca: Yeah, totally :)
[19:09] <Chipaca> hmmmm
[19:10] <Chipaca> niemeyer: it's not really suited to show more than one set at a time though
[19:10] <Chipaca> but, let me play with it a little bit
[19:10] <niemeyer> Chipaca: I'd take a long list in that format over a short list in the first one any day
[19:10] <Chipaca> niemeyer: fair enough
[19:10] <niemeyer> Chipaca: People can constrain by providing the snap name they are interested on
[19:11] <Chipaca> niemeyer: which reminds me, what do you say to start paging by default?
[19:11] <Chipaca> niemeyer: that is, snap feeding its output to less instead of straight to stdout
[19:11] <niemeyer> Chipaca: The thing that second output is missing is a way to identify the particular snapshot
[19:11] <niemeyer> So it can be restored, etc
[19:11] <cachio> niemeyer, we are requesting by default diskSizeGb=10
[19:12] <Chipaca> niemeyer: yep, but fixable with indentation (although that'd make grepping harder)
[19:12] <niemeyer> Chipaca: I'm not a huge fan to be honest.. it feels like getting in the way of the terminal flow
[19:12] <niemeyer> Chipaca: | less is so easy
[19:12] <Chipaca> niemeyer: do you find git getting in the way?
[19:13] <Chipaca> or journalctl
[19:13] <niemeyer> Chipaca: Much of git doesn't go to a pager
[19:14] <niemeyer> Chipaca: Things like git log etc, I pipe to vi instead of using the default pager
[19:15] <niemeyer> Chipaca: It'd also be easy to make the same argument towards every other tool that does not go to a pager
[19:15] <niemeyer> Chipaca: "Do you find ls broken?"
[19:15] <Chipaca> niemeyer: yes :-)
[19:15] <Chipaca> but that's another conversation
[19:15] <niemeyer> :)
[19:16] <Chipaca> niemeyer: changing subjects, I might be looking at working on an initial (warning-less) implementation of the "keep some space for refreshing core" thing; would you have any concerns over that?
[19:17] <Chipaca> waiting for the customer to get back about whether it is snapd filling the disk, or if it's them :-)
[19:17] <niemeyer> Chipaca: I just wonder if we should do warnings first
[19:18] <Chipaca> that was the plan
[19:18] <niemeyer> Chipaca: Every other day we stumble upon an issue that would benefit from having that mechanism we discussed in London.. would be a UX win to have that, and then do disk space/etc
[19:18] <pedronis> Chipaca: niemeyer: for what is worth lots of recent work would have been improved UX wise by warnings
[19:19] <niemeyer> Yea
[19:19] <niemeyer> h
[19:19] <Chipaca> awwww
[19:19] <pedronis> I had to use logging and leave TODOs around
[19:19] <niemeyer> It's one of those things that it's not a deal breaker and we keep pushing forward
[19:19] <Chipaca> I've got half of warnings done since that sprint and look at it every so often
[19:19] <niemeyer> Chipaca: Go go go :)
[19:19] <Chipaca> but … snapshots!
[19:20] <Chipaca> augh
[19:20] <Chipaca> anyway maybe it's syslog filling this customer's disc
[19:20] <Chipaca> we'll see :-)
[19:21] <niemeyer> Chipaca: Well, to be clear that's not a suggestion not to do the disk check, but rather to do it right after warns
[19:23] <Chipaca> https://forum.snapcraft.io/t/out-of-disk-space-protection/1632/13 fwiw
[19:23] <Chipaca> ooh look it's beer o'clock
[19:23] <niemeyer> :)
[19:28] <Chipaca> how are people monitoring big deploys with core? because warnings should tie into that as well :-)
[19:28] <Chipaca> i wonder who i need to talk with about that
[19:29]  * Chipaca has a sudden surge of compassion for the person on pagerduty for 100k toasters
[19:29]  * Chipaca 's feeling passes
[19:38] <mup> PR snapcraft#2019 opened: elf: avoid duplicating rpath entries <bug> <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2019>
[19:39] <niemeyer> cachio: Going back to the image conversation, how much free space is left in the 10GB for Fedora?  That was the original question
[20:04] <cachio> niemeyer, I don't know, let me check that
[20:05] <niemeyer> cachio: Ideally we should try to keep those images low profile, and make sure to collect garbage as we did in Linode
[20:05] <niemeyer> cachio: I won't be bothering you anymore about the individual images, so please make sure to account for those details
[20:06] <cachio> niemeyer, sure
[20:13] <Chipaca> niemeyer: https://pastebin.ubuntu.com/p/dM6wMqJc37/ ?
[20:14] <niemeyer> Chipaca: <3
[20:15] <niemeyer> I need to step out quickly or will miss the window in which I can exercise.. but will be back later
[20:15]  * Chipaca won't :-)
[20:15] <Chipaca> niemeyer: enjoy
[20:26] <cachio> niemeyer, taking a look to fedora image
[20:26] <cachio> niemeyer, https://paste.ubuntu.com/p/9m9p8pdCnD/
[20:27] <cachio> the resize is not as useful as I thought
[20:27] <cachio> just 2.5 GB free on /
[20:27]  * zyga found https://packagecontrol.io/packages/Colorcoder interesting
[20:30] <cachio> niemeyer, I'll resize the filesistems to get a better distribution once the image is resized
[20:34] <mvo> jdstrand: yeah, its core18
[20:35] <mvo> jdstrand: base-18 will be phased out
[21:11] <jdstrand> roadmr: hey, can you queue up r1014? small change for core18 (see above). not urgent in the least
[21:12] <roadmr> jdstrand: sure! I'd just QAd r1013 :) no problem, this'll make it in to the queue and I'll aim to roll out tomorrow
[21:12] <cachio> Saviq, I am working to generate a new fedora image
[21:12] <jdstrand> roadmr: thanks. sorry it was post QA
[21:13] <cachio> Saviq, it won't be needed to specify the image anymore once it is uploaded
[21:13] <roadmr> jdstrand: hehe no worries, QA is really fast: I just approve the merge, wait 2 hours or so, then push a snap and ensure it was processed by the correct tools version :)
[21:13] <cachio> just to call the system "fedora-27-64"
[21:13] <jdstrand> heh, cool :)
[21:14] <jdstrand> I like that we have the black box functional tests for 50+ snaps
[21:14] <jdstrand> (beyond the extensive unit tests)
[21:18] <popey> When will we be able to build snaps against an 18 core?
[21:23] <cachio> niemeyer, if you have time, could you please take a look to this one? #4778
[21:23] <mup> PR #4778: tests: moving debian 9 from linode to google backend <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/4778>
[21:23] <cachio> niemeyer, it is passing after the last change we did to spread
[21:33] <mup> PR snapcraft#2019 closed: elf: avoid duplicating rpath entries <bug> <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/2019>
[21:35] <Saviq> cachio: it's still just 4GB though: https://travis-ci.org/MirServer/mir/jobs/356561944#L3492
[21:36] <mup> PR snapcraft#2020 opened: elf: set no-default-lib for all elf files if patching <bug> <enhancement> <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2020>
[21:36] <cachio> Saviq, yes, working on that, I could reproduce the same with our machines
[21:37] <Saviq> ack
[21:37] <cachio> Saviq, it is failing just on fedora the resize
[21:37] <cachio> I need to see what is wrong with that image
[21:37] <Saviq> maybe missing resize2fs or something, /me grabs cloud-init log
[21:40] <cachio> Saviq, apparently also missing growpart
[21:40] <cachio> I'll create a new image with those deps
[21:41] <cachio> Saviq, thanks for the info
[21:43]  * cachio afk
[22:15] <Chipaca> niemeyer: it just struck me that possibly the most important bit of info is missing from that 'saved' proposal, and adding it looks relaly nasty: https://pastebin.ubuntu.com/p/Kb8mc7nb3b/
[22:18] <niemeyer> Chipaca: I think we can fix two problems at once there
[22:19] <niemeyer> One issue in that output is it had spaces, breaking the column-based parsing
[22:20] <niemeyer> Chipaca: We discussed before on another context the use of a shorter format.. If we use something like that we can remove the repetition and avoid the spaces
[22:20] <niemeyer> E.g.:
[22:20] <niemeyer> 18:20
[22:20] <niemeyer> 3days
[22:20] <niemeyer> 2018-01-15
[22:21] <Chipaca> for parsing you'd use --abs-time which doesn't have spaces though
[22:21] <niemeyer> Well, ideally we'd make both of them without spaces
[22:21] <niemeyer> We've managed to do that in every other table so far, I think
[22:22] <niemeyer> Or at least it's a nice property to preserve
[22:24] <Chipaca> not sure it'll solve the thing i dislike about it being two columns all the same, but I'll take a look
[22:24] <niemeyer> Two columns?
[22:25]  * niemeyer looks again
[22:26] <niemeyer> Chipaca: Yeah, still don't see the two columns
[22:26] <Chipaca> niemeyer: https://pastebin.ubuntu.com/p/75BdH8g9rc/
[22:26] <niemeyer> Chipaca: What I suggested above would be a single column, but ... Looking
[22:26] <Chipaca> niemeyer: the repetition of the set id and the time (now age) column
[22:26] <Chipaca> niemeyer: creates two columns that look nasty
[22:27] <niemeyer> I quite like that output now
[22:27] <Chipaca> ok
[22:27] <niemeyer> Duplication is not by itself all bad
[22:27] <niemeyer> It also makes us realize the associtation
[22:28] <Chipaca> niemeyer: would you have the columns ordered like this, or age before snap?
[22:28] <niemeyer> As long as it's not a lot of data duplicated, but the shorter values solve some of that
[22:28] <Chipaca> I'd rename the column to Time when --abs-time was given
[22:28] <Chipaca> i guess
[22:28] <niemeyer> That 22.6h looks a bit strange, but the rest looks quite nice
[22:29] <niemeyer> The 2h10m form in particular looks nicer
[22:29] <Chipaca> heh, yes, decimal hours are strange -- quantity's thing is that it's fixed width, which might not be wanted here
[22:29] <niemeyer> Yeah.. Just rounding to 22h would be fine
[22:30] <Chipaca> niemeyer: while you're at it, any other that look weird? https://github.com/snapcore/snapd/blob/master/strutil/quantity/example_test.go#L92
[22:31] <SuperJonotron> how do you find dhcp lease information in ubuntu core?
[22:31] <Chipaca> FWIW i wanted to let it do things like "2¼h" but that's Hard, again
[22:31] <Chipaca> (especially in the context of tabwriter)
[22:32]  * niemeyer looks
[22:33]  * Chipaca just realised that µs will suffer the same issue
[22:33]  * Chipaca is sad now
[22:33] <Chipaca> SuperJonotron: what is handling dhcp for you on core?
[22:33] <SuperJonotron> NetworkManager/netplan
[22:34] <niemeyer> Chipaca: Yeah, I'd drop all of these decimals
[22:34] <niemeyer> Just accepting the rounded integer form, or presenting the fractional part with a proper unit if it's relevant
[22:35] <niemeyer> Also E is weird :)
[22:36] <niemeyer> Ah, and the leading zeros can probably be dropped for a nicer output too..
[22:36] <niemeyer> That is 4h9m instead of 4h09m
[22:36] <Chipaca> SuperJonotron: leading zeros?
[22:36] <Chipaca> um
[22:36] <Chipaca> niemeyer: leading zeros?
[22:36] <Chipaca> niemeyer: ah
[22:36] <niemeyer> Hmm.  Although that last  one is more  unclear
[22:37] <niemeyer> (Now that I write it out)
[22:37] <SuperJonotron> Chipaca: leading zeros?
[22:37] <Chipaca> SuperJonotron: trying to carry two conversations at once when I should be carrying none
[22:37] <SuperJonotron> gotchya
[22:38] <Chipaca> SuperJonotron: there's nothing core-specific about dhcp leases, the info'll be where it is without core
[22:38] <Chipaca> mostly
[22:38] <Chipaca> :-)
[22:38] <Chipaca> SuperJonotron: i mean, how would you get that info without core?
[22:40] <SuperJonotron> Chipaca: according to forums, /var/lib/dhcp/dhcp.leases but that folder is empty
[22:41] <SuperJonotron> Chipaca: that folder is empty though
[22:41] <SuperJonotron> Chipaca: there's also mention to looking at /var/logNetworkManager* which also doesn't exist in core
[22:42] <Chipaca> SuperJonotron: I don't think that's the location of the leases file in reasonably new ubuntus though
[22:42] <SuperJonotron> Chipaca: where did they move to in core?
[22:42] <Chipaca> SuperJonotron: as NetworkManager is a snap, I'd expect it to be under /var/snap/
[22:42] <Chipaca> SuperJonotron: but you can figure it out with 'find':
[22:43] <Chipaca> SuperJonotron: sudo find / -name \*.lease 2>/dev/null
[22:43] <SuperJonotron> ls
[22:43] <SuperJonotron> whoops, wrong window
[22:44] <SuperJonotron> i did find the lease files starting in /var/snap
[22:44] <SuperJonotron> but that find command will likely be needed since it is not a pretty file name
[22:44] <Chipaca> heh
[22:44] <Chipaca> SuperJonotron: if it's looking there like it looks here, the names have a reason
[22:45] <Chipaca> SuperJonotron: (but that reason might not make sense to you :-) )
[22:46] <SuperJonotron> Chipaca: looks like a naming pattern with a UUID as part of it both pointing to the same interface so that is a bit confusing about which one to read
[22:46] <Chipaca> SuperJonotron: as i understand it it's a UUID which is the id of the network connection in network manager, then "-", then the device name, then ".lease"
[22:46] <Chipaca> SuperJonotron: nmcli con
[22:46] <Chipaca> SuperJonotron: ^ list the connections, including the uuids
[22:46] <SuperJonotron> perfect
[22:48] <SuperJonotron> now to figure out how to read this file...
[22:48] <Chipaca> SuperJonotron: what are you trying to do?
[22:49] <SuperJonotron> Chipaca: working with an application that needs to report DHCP Server, Lease Granted Time and Least Expiration time
[22:49] <SuperJonotron> for all interfaces using dhcp that is
[22:50] <Chipaca> hmmm
[22:51] <Chipaca> SuperJonotron: so,  nmcli can give you that info
[22:52] <Chipaca> SuperJonotron: if $CON is your connection name or uuid, then 'nmcli connection show $FOO' will give you all that, i think
[22:52] <Chipaca> um
[22:52] <Chipaca> CON and FOO got jumbled there
[22:52] <Chipaca> see: me needing to go to bed
[22:53] <Chipaca> ah, maybe not granted time
[22:53] <Chipaca> dunno
[22:53] <SuperJonotron> was inspecting that now
[22:54] <Chipaca> SuperJonotron: good luck
[22:54] <Chipaca> and good night
[22:54] <SuperJonotron> thanks