[02:48] <mup> PR snapd#6217 opened: tests: reset snapd state on tests restore <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6217>
[04:18] <mup> Bug #1753760 changed: Adding public RSA Key doesn't make login on device possible <Snappy:Expired> <https://launchpad.net/bugs/1753760>
[06:46] <zyga> o/
[07:34] <geodb27> People : hi !
[07:34] <zyga> hello
[07:48] <zyga> brb, need to take the dog out
[07:57] <pstolowski> mornings
[07:58] <pstolowski> back to yesterday's test panic issue, the debug log from zyga shed some light on it
[07:59] <mvo> pstolowski: good morning and good luck wit hthat
[08:00] <zyga> Hey Paweł, Michael
[08:00] <zyga> mvo: I’m off the 2.36 bug
[08:00] <zyga> No progress made
[08:00] <zyga> Need to finish my stuff now
[08:04] <mvo> zyga: ok
[08:06] <pstolowski> zyga: may i ask you to try one more patch>
[08:06] <pstolowski> ?
[08:18] <zyga> Sure
[08:19] <pstolowski> zyga: https://pastebin.ubuntu.com/p/Kgs7tXTfRH/
[08:26] <mvo> pedronis: re 6195 - shall we (after this got a second review) merge and do 2.36.2 and then look for a better place for this than link_snap.go? or would you rather like to look for this place now before it goes to master?
[08:31] <mup> PR snapd#6218 opened:  snapstate: update fontconfig caches on install (2.36) <Created by mvo5> <https://github.com/snapcore/snapd/pull/6218>
[08:59] <pstolowski> zyga: actually that patch won't help; i'll have another one soon
[09:00] <pstolowski> i wish i understand what the delta is between our setups and why it doesn't fail for me on travis, pretty weird
[09:17] <zyga> pstolowski: re, sorry, had a delay and network woes
[09:18] <zyga> pstolowski: yeah
[09:18] <zyga> pstolowski: if you want I can give you ssh access
[09:18] <zyga> not sure if that would help
[09:18] <zyga> to my box
[09:18] <zyga> where this happens
[09:18] <zyga> I can work on another
[09:23] <pstolowski> zyga: yeah, that could help, thanks. also question, did we change anything related to finding if apparmor should be enabled or not recenlty? it looks like setup-profiles in this test now wants to read vanilla snap-confine profile of core; this happens if we find that release.AppApparmorLevel != NoApparmor. we're not mocking vanilla profile in the mock core snap obviously
[09:24] <pstolowski> mvo: you might know as well^
[09:24] <zyga> pstolowski: not sure when is recently
[09:24] <zyga> when apparmor is not entirely disabled we operate
[09:24] <zyga> including making a special profile for snap-confine itself
[09:24] <zyga> if you think mocking is missing by all means, add some
[09:25] <pstolowski> yep. it's just curious it's not happening consistently
[09:26] <Chipaca> moin moin
[09:26] <zyga> hey Chipaca
[09:26] <Chipaca> have you seen https://forum.snapcraft.io/t/disable-enable-not-longer-working-when-snap-have-gpio-pins-as-connect/6473 ?
[09:26] <zyga> oh noes
[09:26] <zyga> well
[09:26] <zyga> this is a bug we fixed
[09:27] <Chipaca> zyga: "withthe edge channel" of core presumably?
[09:27] <zyga> yes
[09:27] <pstolowski> morning Chipaca
[09:28] <zyga> Chipaca: your questions got me curious
[09:28] <zyga> so I did some more experiments on https://github.com/snapcore/snapd/pull/6147
[09:28] <mup> PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147>
[09:29] <zyga> and I believe we understand why additional permissions are required
[09:29] <zyga> my only question now
[09:29] <zyga> should I split the permissions so that snap-update-ns is called with the existing profile transition
[09:29] <zyga> and the only permission we have remains "r"
[09:29] <zyga> adding a new block for snap-discard-ns which runs with the same profile
[09:29] <zyga> or is the patch as-is good enough?
[09:30] <Chipaca> zyga: WWJ(S)D?
[09:30] <zyga> split it
[09:30] <Chipaca> :-)
[09:30] <zyga> for maximum confinement
[09:30] <zyga> I'll do that
[09:30] <zyga> btw, love the acronym :D
[09:39] <zyga> Chipaca: https://github.com/snapcore/snapd/pull/6147 is ready now
[09:39] <mup> PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147>
[09:40] <zyga> eh
[09:40] <zyga> https://gitlab.gnome.org/GNOME/gnome-control-center/issues/230#note_373259
[09:50] <mup> PR snapd#6219 opened: overlord/tests: [WIP] fix panic in managers test <Created by stolowski> <https://github.com/snapcore/snapd/pull/6219>
[10:00] <zyga> I need a 2nd review for https://github.com/snapcore/snapd/pull/6147
[10:00] <mup> PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147>
[10:01] <zyga> Chipaca: thank you!
[10:21] <zyga> I also need a 2nd review on https://github.com/snapcore/snapd/pull/6159
[10:21] <mup> PR #6159: cmd/snap-confine: handle mounted shared /run/snapd/ns <Created by zyga> <https://github.com/snapcore/snapd/pull/6159>
[10:31] <pstolowski> zyga: +1 with one question
[10:33] <mup> PR snapd#6159 closed: cmd/snap-confine: handle mounted shared /run/snapd/ns <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6159>
[10:33] <zyga> answered
[10:33] <zyga> thanks! :)
[10:33] <zyga> mvo: what's the 2.36 weather like?
[10:37] <mvo> zyga: medium, still see failures but can't reproduce them
[10:38] <zyga> mvo: magic
[10:38] <zyga> mvo: pstolowski is working on a fix for the panic I can reproduce trivially
[10:38] <zyga> I'll just run tests for his incremental patches
[10:38] <pstolowski> zyga:you can try https://github.com/snapcore/snapd/pull/6219
[10:38] <mup> PR #6219: overlord/tests: [WIP] fix panic in managers test <Created by stolowski> <https://github.com/snapcore/snapd/pull/6219>
[10:38] <zyga> sure thing
[10:39] <pstolowski> zyga: btw, it's green, does it mean anything?
[10:39] <zyga> test in progress
[10:39] <pstolowski> cause i can't repro here, pretty annoying
[10:40] <mvo> pstolowski: I can try to run this after my current run, I had some luck reproducing the panic
[10:45] <pstolowski> mvo: thanks. i tried running it with spread -debug -repeat 10 google:ubuntu-16.04-64:tests/unit/go
[10:51] <pedronis> mvo: I hopefully answered in the PR
[11:02] <mvo> pedronis: ta
[11:03] <pedronis> np
[11:03]  * pedronis goes back to swap daying
[11:04] <mvo> pstolowski: I am running it now again, lets see
[11:13] <zyga> Brb, coffee or something warm
[11:26] <mvo> pstolowski: https://paste.ubuntu.com/p/9rj8ydFg5p/ <- I cherry picked  1c5b166643a5623b0a534fd1a8acb7d03d50d856 from 6219 earlier
[11:29] <pstolowski> mvo: aha, interesting, let me mock core rev 1 as well
[11:30] <mvo> pstolowski: shall I run it again?
[11:30] <mvo> pstolowski: I mean, after you upated the pr?
[11:35] <pstolowski> mvo: i've just pushed a change
[11:35] <pstolowski> mvo: thanks!
[11:35] <cachio> pedronis, I am working again with the #5714
[11:36] <mup> PR #5714: tests: new test for cifs-mount interface <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/5714>
[11:50] <mvo> 6195 needs a second review
[11:55] <zyga> mvo: I reproduced 2.36 bug again
[11:55] <zyga> pstolowski: no panic yet
[11:56] <pstolowski> zyga: are you running with the update i pushed  a few minutes ago ^ ?
[11:56] <zyga> pstolowski: no, the one from earlier today
[11:56] <zyga> mock vanilla profile for snap-confine
[11:57] <zyga> mvo: same thing, apparmor parser killed by signal
[11:57] <pstolowski> zyga: ok; yeah, i think one more revision of core in the tests needs it too, i added it
[11:57] <zyga> ok
[11:59] <mvo> zyga: anything interessting why its killed?
[11:59] <zyga> mvo: I cannot find any information about why
[11:59] <mvo> zyga: or what signal? sigter
[11:59] <zyga> hits appreciated
[11:59] <mvo> zyga: meh :(
[12:00] <zyga> I suspect sigterm because it says "terminated"
[12:01] <zyga> ah
[12:01] <zyga> something interesting
[12:01] <zyga> mvo: look at this
[12:01] <zyga> lots of terminated errors https://www.irccloud.com/pastebin/FVHUi8t8/
[12:01] <zyga> I thought it is just one time
[12:02] <zyga> mvo: it's not just once
[12:02] <zyga> it seems to happen all the time
[12:02] <zyga> snap changes https://www.irccloud.com/pastebin/2UoYFGwX/
[12:04] <zyga> *hmm*
[12:04] <zyga> mvo: note that this branch doesn't have the tweak I used before
[12:04] <zyga> so the error was not fatal
[12:04] <zyga> I was testing pawel's patches
[12:04] <zyga> I will record the seed and continue unless anyone has some better idea
[12:07] <zyga> ok, seed is -seed=1543315161
[12:07] <zyga> I will pull more of pawel's patches
[12:08] <zyga> as well as my stashed error handling
[12:08] <zyga> and let's see
[12:08] <mvo> zyga: thank you
[12:08] <mvo> zyga: I run it also with your diff to make the error fatal
[12:09] <mvo> zyga: but so far no luck in reproducing
[12:09] <zyga> restarted
[12:09] <zyga> back to working on feature
[12:13] <Chipaca> what's holding back snapd on debian?
[12:14] <zyga> Chipaca: AFAIK there's a test failure that happens there
[12:14] <zyga> but no investigated
[12:14] <Chipaca> I mean packaging-wise
[12:14] <Chipaca> it's on 2.30
[12:14] <Chipaca> and no reexec
[12:15] <zyga> I believe you are wrong, there is reexec
[12:15] <Chipaca> hmm
[12:15] <zyga> as for the package, nobody released updates
[12:17] <Chipaca> I wonder if deepin mangles os-release or sth, then
[12:17]  * Chipaca installs deepin
[12:17] <zyga> note that we have a blacklist, not a whitelist
[12:17] <Chipaca> no we don't
[12:17] <Chipaca> I mean
[12:17] <Chipaca> 	if !release.DistroLike("debian", "ubuntu") {
[12:18] <Chipaca> that's a whitelist, even if it has a "not" in front of it
[12:19] <zyga> oh
[12:19] <zyga> indeed
[12:19] <zyga> that's new to me
[12:19] <Chipaca> zyga: reading a bit of above, "terminated" is the %s of signal 15
[12:19] <zyga> aha
[12:19] <zyga> sigkill
[12:19] <zyga> that's odd
[12:19] <zyga> wonder why we do that
[12:19] <Chipaca> zyga: /usr/share/go-1.6/src/syscall/zerrors_linux_amd64.go
[12:19] <zyga> (not really us probably)
[12:19] <zyga> ze errorz zey keep koming!
[12:20] <Chipaca> I think it's just shorthand for "zee autogeneraed files"
[12:20] <Chipaca> zyga: 15 is TERM, so what else would you call it?
[12:21] <zyga> ah, sorry
[12:21] <zyga> 9 is kill
[12:21] <Chipaca> :-)
[12:21] <zyga> I did make tea, not coffee
[12:21] <zyga> so terminated
[12:21] <Chipaca> and 9 would say "killed"
[12:21] <zyga> maybe test setup?
[12:22] <zyga> there is one silly way to fix it :)
[12:22] <zyga> we could wrap aa-parser with an apparmor profile
[12:22] <zyga> that rejects SIGTERM :D
[12:22] <zyga> (I'm half serious)
[12:22] <zyga> might help us to pinpoint the sender
[12:22] <zyga> because we would get a denial
[12:25]  * pstolowski lunch
[12:33] <zyga> is maciej around today?
[12:33] <zyga> ah, he called in sick
[12:33] <zyga> ok
[12:43] <Chipaca> https://forum.snapcraft.io/t/cannot-run-any-program-trace-breakpoint-triggered-errors/8707?u=chipaca if anybody has clues about apparently x11 being broken on deepin
[12:44]  * zyga thinks
[12:45] <zyga> I know
[12:45] <Chipaca> zyga: i'm going to get lunch, but i'll read when i return
[12:45] <zyga> ls /tmp/.X11-unix
[12:45] <zyga> on ubuntu we use abstract socket to talk to x
[12:45] <zyga> on some systems we don't
[12:46] <zyga> without special support we won't have x socket
[12:51] <Wimpress> If snap A connects to snap B via a content interface, can snap A execute binaries within snap B?
[12:51] <zyga> Wimpress: yes, but must use the internal interface, not via "snap run ..." or /snap/bin/..."
[12:51] <zyga> and said binaries run with confinement of snap A
[12:52] <Wimpress> snap A is confined.
[12:52] <Wimpress> snap B might be classic ;-)
[12:52] <zyga> by internal interface I mean by knowing how to execute the binary directly from where the content sharing is establieshed
[12:52] <zyga> Wimpress: content sharing shares bits and bytes
[12:52] <zyga> not changing anything else
[12:52] <zyga> does't grant more permissions to run
[12:52] <zyga> or not run
[12:53] <Wimpress> OK, so by virtue of having the snaps connected, snap A can not execute binaries from snap B. Correct?
[12:53] <zyga> no, that's not correct
[12:53] <zyga> if I have a snap that contains a shell script
[12:53] <zyga> and I expose that via a content interface
[12:53] <zyga> and that interface is connected to some other sna
[12:54] <zyga> that other snap can invoke that shell script
[12:54] <Wimpress> Excellent.
[12:54] <zyga> that shell script may not be an exposed application
[12:54] <zyga> and in fact if it is an exposed application
[12:54] <zyga> I cannot use that exposed application from any snap
[12:54] <zyga> note that this doesn't imply any more permissions
[12:55] <zyga> if you content-connect to LXD, for example, you don't get to run LXD with all the powers attached
[12:55] <Wimpress> So snap A would have to invoke the binary in snap B via a full path?
[12:55] <zyga> it's just a way to share bytes
[12:55] <zyga> Wimpress: with path, environment and any other quirks required
[12:55] <zyga> it's as easy or hard as the associated binary implementation details are
[12:57] <Wimpress> Would binaries snap A want to execute from snap B be referenced via /snap/snapA/current/bin/foo or via /snap/snapB/current/bin/foo? Or will $SNAP be fine?
[12:57] <zyga> neither, you need to use the real path where they are shared
[12:57] <zyga> so if you share them to, say
[12:57] <zyga> $SNAP/plugins/
[12:57] <zyga> and find them there
[12:58] <zyga> you need to know to call $SNAP/plugins/foo/bin/plugin.bin
[12:58] <zyga> $SNAP or any other variables are not aware of that
[12:58] <zyga> they will not know anything about any connected content interfaces (perhaps many)
[12:58] <zyga> they will always point to the snap for which the main process was started
[12:58] <zyga> (snap A in this case)
[12:58] <Wimpress> OK. Sounds promising.
[12:59] <Wimpress> So, if snap A (confined) is connecting to snap B (classic), snap A can execute binaries exposed from snap B but within the confinement defined by snap A. Correct?
[12:59] <zyga> yes
[13:00] <Wimpress> Cool.
[13:00] <zyga> it's really just the same
[13:00] <zyga> as if they were copied into snap A
[13:00] <zyga> it's just a way to share bytes
[13:00] <Wimpress> Just wanted to be clear.
[13:00] <zyga> sure
[13:00] <Chipaca> zyga: am back. Indeed /tmp/.X11-unix/X0 is where it's at
[13:01] <zyga> yay
[13:01] <Chipaca> zyga: so what do I do?
[13:01] <zyga> we can easily fix that too
[13:02] <zyga> hold on
[13:03] <zyga> looking for something
[13:05] <zyga> Chipaca: in cmd_run.go we handle XAUTHORITY
[13:05] <zyga> we could handle X11 socket
[13:06] <Chipaca> zyga: handle it how?
[13:06] <zyga> ha, that's tricky
[13:07] <zyga> we can see the socket
[13:07] <zyga> but not access it
[13:07] <zyga> perhaps we should preserve /tmp/.X11* in snap-conifne
[13:07] <zyga> which is not terribly hard but not super easy either
[13:07] <zyga> Chipaca: can you please file a bug
[13:07] <zyga> or find one
[13:07] <zyga> I bet people reported it
[13:07] <zyga> there are also mentions of this on the forum
[13:07] <cachio> zyga, hey
[13:08] <Chipaca> zyga: I mean, this is from the forum :-)
[13:08] <zyga> Chipaca: let's cross reference them then
[13:08] <zyga> cachio: hello
[13:08] <cachio> zyga, I could make the cifs test work but ...
[13:08] <cachio> zyga, just adding this to the interface "/run/mount/utab rw,"
[13:08] <Chipaca> zyga: I wonder if there's a way to tell X to use abstract sockets
[13:08] <zyga> I think we discussed that
[13:09] <zyga> cachio: we should not do that
[13:09] <zyga> Chipaca: some systems don't have one
[13:09] <cachio> zyga, based on the comment we shouldn't
[13:09] <zyga> cachio: we cannot add that permission because that would allow snaps to confuse the host's libmount about what certain attributes are
[13:11] <Chipaca> zyga: it seems to also have an abstract socket, @/tmp/.X11-unix/X0
[13:11] <zyga> Chipaca: is there anything in the environment
[13:11] <zyga> any X... variable set?
[13:11] <zyga> maybe we can do a low cost fix
[13:12] <zyga> "unset XSTH"
[13:13] <Chipaca> nothing that I can see
[13:14] <zyga> dunno then
[13:18] <Chipaca> zyga: and if I install with devmode it workss
[13:18] <zyga> oh
[13:18] <zyga> that's odd
[13:18] <Chipaca> zyga: so it's not the /tmp/.X11 socket
[13:18] <zyga> what's the denial?
[13:18] <Chipaca> zyga: it's access to the socket itself
[13:18] <pstolowski> mvo: any news on the 2nd run of https://github.com/snapcore/snapd/pull/6219 ?
[13:18] <Chipaca> 1 sec because I accidentally killed the vm :-)
[13:18] <mup> PR #6219: overlord/tests: [WIP] fix panic in managers test <Created by stolowski> <https://github.com/snapcore/snapd/pull/6219>
[13:19] <zyga> pstolowski: I ran it again, 0 failures
[13:19] <zyga> restarted (thanks for reminidng me)
[13:19] <pstolowski> nice, thanks
[13:19] <pstolowski> fwtw travis pass was green again
[13:22] <Chipaca> [  157.486384] audit: type=1400 audit(1543324906.843:348): apparmor="DENIED" operation="create" profile="snap.xbill-xaw.xbill-xaw" pid=3187 comm="xbill" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none
[13:22] <Chipaca> had to purge the charon thing to get to see that
[13:22] <zyga> hold on
[13:22]  * Chipaca goes to check on the soup
[13:22] <zyga> weird
[13:23] <Chipaca> soup for lunch isn't that weird
[13:23] <Chipaca> i've got a bad cold
[13:23] <Chipaca> ¯\_(ツ)_/¯
[13:23] <Son_Goku> niemeyer, zyga: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-ba8845b83b
[13:23] <Son_Goku> this might make you guys happy :)
[13:23] <zyga> indeed
[13:23] <zyga> Son_Goku: mborzecki is sick, he's off today
[13:23] <zyga> but I'll make sure he knows
[13:23] <Son_Goku> ah, that's too bad
[13:23] <Son_Goku> hope he feels better soon :)
[13:24] <zyga> Chipaca: which interfaces do you have connected
[13:24] <zyga> Chipaca: x11 plug grants socket AF_NETLINK, not unix
[13:24] <zyga> also allows some x abstractions, reading on those now
[13:25] <zyga> Chipaca: but nothing grants access to that socket
[13:25] <zyga> Chipaca: could it be the case that apparmor simply doesn't handle abstract sockets
[13:25] <zyga> and we never noticed?
[13:26] <Chipaca> sounds like a jdstrand question :-)
[13:26] <Chipaca> or, maybe a jjohansen ? if i remember people right
[13:26]  * Chipaca often doesn't
[13:30] <Chipaca> mmm, tasty tasty schadenfreude http://forum.asrock.com/forum_posts.asp?TID=10174
[13:32] <zyga> Chipaca: both would know
[13:32] <Chipaca> ok, i'm off to have actual lunch now
[13:32] <Chipaca> ttfn
[13:32] <zyga> good idea
[13:33] <zyga> I'll wait till after standup
[13:33] <zyga> I'd love to land https://github.com/snapcore/snapd/pull/6147
[13:33] <mup> PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147>
[13:33] <zyga> mvo: can you do a quick review, in mborzecki's absence
[14:22] <Chipaca> zyga: https://pastebin.ubuntu.com/p/B5d5n36p2h/ might this be relevant to the x11 thing?
[14:22] <zyga> maybe
[14:27] <zyga> mvo: the bug is https://bugs.launchpad.net/snapd/+bug/1805182
[14:27] <mup> Bug #1805182: /tmp is sometime cleaned up after snaps are launched <snapd:Triaged> <https://launchpad.net/bugs/1805182>
[14:39] <Chipaca> socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) vs socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0)
[14:53] <mup> PR snapd#6220 opened: cmd: drop cruft from snap-discard-ns build rules <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6220>
[14:53] <zyga> mvo: can you do a quick review of https://github.com/snapcore/snapd/pull/6147 please
[14:53] <mup> PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6147>
[14:53] <zyga> it's green and if it lands I can continue
[14:53] <zyga> and has one and a half review
[14:54] <mvo> zyga: I try, have a lot of meeting now :/
[14:54] <zyga> ok
[14:54] <zyga> pstolowski: perhaps you ^
[14:54] <pstolowski> 6147?
[14:54] <pstolowski> zyga: ^
[14:55] <zyga> yes
[14:55] <pstolowski> ok
[14:58] <zyga> meanwhile, 2.36 branch for those errors
[15:02] <zyga> mvo: I just realized something
[15:02] <zyga> the terminated error is not new
[15:02] <zyga> we probably had it since forever
[15:02] <zyga> but we would not notice it until the special situation in 2.36 branch
[15:02] <zyga> this doesn't change why it is broken
[15:03] <zyga> but may help us form new perspective
[15:03] <zyga> because we had ignored those errors all the time
[15:03] <zyga> we would just happily carry on
[15:03] <mup> PR snapcraft#2417 opened: Revert "lifecycle: make snapcraft init template use > not | (#2393)" <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/2417>
[15:09] <mup> PR snapd#6221 opened: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6221>
[15:10] <zyga> mvo: this one is for you https://github.com/snapcore/snapd/pull/6221
[15:10] <mup> PR #6221: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6221>
[15:29] <mup> PR snapd#6222 opened: cmd/snap: handle DNS error gracefully <Created by stolowski> <https://github.com/snapcore/snapd/pull/6222>
[15:33] <zyga> pstolowski: nice
[15:33] <zyga> pstolowski: 0 errors
[15:34] <pstolowski> zyga: very nice, thanks, i'm cleaning that PR atm, will push an update
[15:34] <zyga> pstolowski: I woudn't mind a follow up
[15:34] <zyga> this looks sane as-is
[15:38] <pstolowski> well it outputs debug
[15:38] <pstolowski> pushed
[15:38] <pstolowski> i'll cherry pick into master
[15:38]  * cachio lunch
[15:40] <zyga> cachio: offtopic, can we start instances not in us-east
[15:40] <zyga> it would be awesome if we could do europe too
[15:40] <zyga> cachio: latency to us-east sucks :/
[15:41] <zyga> not sure if this is something you or gustavo needs to handle
[15:41] <zyga> spread on travis is ok
[15:41] <zyga> but interactive spread for debugging is not
[15:41] <zyga> I'm fine with editing my spread.yaml if I can achieve that
[15:49] <mup> PR snapd#6147 closed: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces <Per-user mount ns  🐎> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/6147>
[15:49] <zyga> thank you mvo!
[15:51] <zyga> mvo: https://github.com/snapcore/snapd/pull/6221 no errors yet!
[15:51] <mvo> zyga: meeting ended early :)
[15:51] <mvo> zyga: oh no
[15:51] <mup> PR #6221: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6221>
[15:51] <zyga> well, there are unit test panics that pstolowski is fixing
[15:51] <mvo> zyga: its so annoying
[15:51] <zyga> but none of the other errors
[15:52] <zyga> so while I don't get it, it's interesting output
[15:53] <mvo> zyga: it seems to be happening all the time in the 6128 PR
[15:55] <zyga> mvo: 6128 is merged, I assume you mean a 2.36 backport of that
[15:55] <zyga> well, let's see
[15:55] <zyga> I need to break for dinner now
[15:55] <zyga> I will be back
[15:55] <zyga> mvo: https://github.com/snapcore/snapd/pull/6220 looks like an easy win
[15:55] <mup> PR #6220: cmd: drop cruft from snap-discard-ns build rules <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6220>
[15:56] <zyga> and helps me move forward too
[15:56] <mvo> zyga: 6218 - sorry
[15:57] <mvo> zyga: as a general rule I always approve PRs that have only reds
[16:14] <zyga> re
[16:14] <zyga> looking
[16:17] <zyga> mvo: https://github.com/snapcore/snapd/pull/6221 didn't fail on that apparmor issue
[16:17] <mup> PR #6221: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6221>
[16:17] <zyga> mvo: do you have any theory as to why the patch may make it better?
[16:19] <mvo> zyga: in a meeting, can you just retrigger?
[16:21] <cachio> zyga, I think we can
[16:21] <zyga> yep
[16:21] <zyga> did
[16:21] <cachio> nice
[16:21] <cachio> there is a problem
[16:21] <zyga> cachio: if you tell me how I'd love to try
[16:21] <zyga> but I need to take the dog out first :)
[16:22] <cachio> the spread cleaner is configured to clean machines just on us-east1-b
[16:22] <zyga> oh
[16:22] <cachio> it is basically because you query instances by zone
[16:30] <kenvandine> mvo: were you saying desktop-launch in brave was taking ~2s on the 2nd launch on 18.10?
[16:30] <zyga> kenvandine: AFAIK it was update-mime-cache or something like that
[16:30] <kenvandine> that should only be on the first run
[16:32] <zyga> pstolowski: I just got a notification that ISS is rising
[16:32] <zyga> coool
[16:32] <zyga> the app is not even open
[16:33] <pstolowski> zyga: :)
[16:34] <zyga> kenvandine: yeah but from what mvo researched it seemed to be a constant cost
[16:34] <kenvandine> interesting
[16:34] <zyga> I didn't examine that, I think it's best to wait for mvo to leave the meeting frenzy
[16:35] <kenvandine> i just tried it on my 18.10 laptop and i'm getting good results
[16:35] <kenvandine> desktop-launch elapsed time:  0.116752997
[16:35] <zyga> mmm
[16:36] <mvo> kenvandine: interessting - here is my second run:http://paste.ubuntu.com/p/hFbMF8CQKT/
[16:36] <zyga> pstolowski: https://github.com/snapcore/snapd/pull/6222#issuecomment-442126406
[16:36] <mup> PR #6222: cmd/snap: handle DNS error gracefully <Created by stolowski> <https://github.com/snapcore/snapd/pull/6222>
[16:36] <mvo> kenvandine: maybe specific to my machine, its a pretty beaten setup :)
[16:37] <kenvandine> mvo update-mime-database and update-icon-cache should only run on the 1st run
[16:37] <mvo> kenvandine: ok, should I set -x the script to see what is going on?
[16:37] <mvo> kenvandine: let me do that
[16:37] <kenvandine> yes
[16:38] <kenvandine> mvo:  that would be useful
[16:39] <kenvandine> mvo: that should only run if needs_update is true
[16:39] <pstolowski> zyga: damn, i think it's not deteministtic... saw that error once but then stopped getting it as the test was tweaked
[16:40] <kenvandine> mvo: https://pastebin.ubuntu.com/p/C2M7Y6XQpf/
[16:41] <mvo> kenvandine: it looks like the user-dirs.dirs check is setting needs_update=true
[16:41] <mvo> kenvandine: I'm trying to figure out now why
[16:41] <kenvandine> oh
[16:42] <kenvandine> mvo: that's not setting it for me
[16:42] <kenvandine> so maybe that's the trigger
[16:42] <kenvandine> perhaps we shouldn't overload needs_update there
[16:43] <mvo> kenvandine: I have no .md5sum file in my dir it seems
[16:45] <kenvandine> mvo: what's your actual $HOME/.config/user-dirs.{dirs,locale} look like
[16:46] <mvo> kenvandine: I don't have a user-dirs.locale
[16:46] <kenvandine> sigh...
[16:46] <kenvandine> i wonder why not
[16:46] <mvo> kenvandine: I have no idea :/
[16:46] <mvo> kenvandine: but its there on my other box - what creates it?
[16:47] <kenvandine> not sure
[16:47] <kenvandine> mvo: so for you it's doing a bunch of things on every run that it shouldn't
[16:47] <mvo> kenvandine: indeed
[16:47] <kenvandine> seb128: ^^ what creates user-dirs.locale?
[16:48] <roadmr> hey folks, a question! two old revisions of each snap are kept for rollback purposes. But these are squashfs-mounted. Why are they mounted if they're not really in use?
[16:49] <zyga> roadmr: two reasons
[16:49] <zyga> roadmr: we didn't do it better
[16:49] <zyga> roadmr: we don't cache meta/snap.yaml
[16:49] <zyga> so we really go and look each time you ask
[16:49] <zyga> we also rely on the filesystem being there to discover things like hooks
[16:49] <roadmr> ok :)
[16:51] <mup> PR snapd#6220 closed: cmd: drop cruft from snap-discard-ns build rules <Simple 😃> <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/6220>
[16:51] <zyga> roadmr: given that they consume memory it would be nice to not have to mount those
[16:52] <zyga> roadmr: we might get to a point where snapd caches enough stuff about them to stop needing that
[16:52] <roadmr> zyga: indeed, it did seem weird and wasteful but doesn't seem to be the end of the world :)
[16:52] <zyga> pstolowski: I keep getting panics
[16:52] <zyga> those that your PR fixes
[16:53] <zyga> cannot wait for 6219
[16:53] <pstolowski> zyga: noooo..
[16:53] <pstolowski> zyga: uh oh, you getting them without my PR?
[16:53] <zyga> yes
[16:53] <zyga> btw
[16:53] <pstolowski> ufff
[16:53] <zyga> inside your pr
[16:53] <zyga> what were the calls to the "snap" command like?
[16:54] <zyga> can you check that in the test quickly
[16:54] <zyga> even locally
[16:55] <pstolowski> zyga: probably none, i added just in case run-hook is exectued, as this was a case in many other tests before. i can probably remove it here, but there is no harm in leving it
[16:55] <zyga> can you just double check before this goes in
[16:56] <pstolowski> ok
[17:01] <cachio>  zyga already tried mounting on /run/mount but still get the same denial
[17:01] <cachio> apparmor="DENIED" operation="open" profile="snap.test-snapd-sh.with-cifs-mount" name="/run/mount/utab" pid=24510 comm="mount" requested_mask="wrc" denied_mask="wrc" fsuid=0 ouid=0
[17:01] <zyga> cachio: you'd have to change the interface
[17:01] <zyga> cachio: to do two more things:
[17:01] <zyga> allow that path to be written
[17:01] <zyga> and create a symlink in /run/mount/utab
[17:01] <zyga> the problem is that it's not easy to do 2)
[17:01] <zyga> for the moment I have no solution
[17:02] <pstolowski> zyga: no calls to snap cmd, verified with fmt.Printf("%q\n", ms.snapCmd.Calls()) in teardown
[17:02] <zyga> cool
[17:04] <cachio> zyga, ok, I'll try that
[17:04] <zyga> cachio: no, please don't
[17:05] <zyga> cachio: there is no fix now
[17:05] <cachio> zyga, ah, ok
[17:05] <cachio> :(
[17:05] <zyga> sorry, no easy win
[17:05] <zyga> mvo: one more trivial https://github.com/snapcore/snapd/pull/6223
[17:05] <mup> PR #6223: cmd/libsnap: move apparmor-support to libsnap <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6223>
[17:06] <zyga> this all setting the stage for the big tool cleanup that mborzecki requested
[17:06] <mup> PR snapd#6223 opened: cmd/libsnap: move apparmor-support to libsnap <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6223>
[17:22] <pstolowski> #6219 failed on unrelated stuff, for the 2nd time
[17:22] <pstolowski> google:ubuntu-16.04-64:tests/main/interfaces-content-mkdir-writable
[17:22] <mup> PR #6219: overlord/tests: fix panic in managers test <Created by stolowski> <https://github.com/snapcore/snapd/pull/6219>
[17:23] <pstolowski> cannot create temporary directory for /var/lib/snapd mount point: Permission denied
[17:23] <pstolowski> (previously it failed on the .mnt unit bug)
[17:24] <zyga> cachio: some quick comments on 6217
[17:25] <zyga> pstolowski: eh
[17:25] <zyga> 2.36 of doom
[17:25] <zyga> pstolowski: perhaps we should merge the two 2.36 fixes into one branch
[17:26] <zyga> WOAH
[17:26] <zyga> we foudn some useful stuff
[17:26] <cachio> zyga, tx
[17:26] <zyga> error from apparmor-parser https://www.irccloud.com/pastebin/4APIpXG2/
[17:28] <zyga> jdstrand: hey, I recall there were some parser incompatiblity with older LEAP releases
[17:28] <zyga> this looks similar ^
[17:28] <zyga> do I recall if you had a PR that fixed this in master
[17:29] <zyga> this comes from a PR that makes errors actually reported: https://github.com/snapcore/snapd/pull/6221
[17:29] <mup> PR #6221: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6221>
[17:29] <zyga> it is against 2.36, not master
[17:30] <pstolowski> zyga: !
[17:31] <mup> PR snapd#6224 opened: interfaces: return security setup errors <Created by zyga> <https://github.com/snapcore/snapd/pull/6224>
[17:31] <zyga> I opened one against master
[17:31] <zyga> let's see what happens
[17:31] <zyga> I need to take Bit out for that walk!
[17:31] <zyga> I'm terrible at tracking time sometimes
[17:31] <zyga> ttyl
[18:03] <zyga> mvo: I'll break to get some groceries with wife
[19:05]  * cachio afk
[19:14] <mvo> zyga|afk: hm - so "AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap-confine.core.6016 in /var/lib/snapd/apparmor/profiles/snap-confine.core.6016 at line 103: Exec condition must begin with '/'." - in my profile line 103 is: "    change_profile unsafe /** -> [^u/]**,
[19:14] <mvo> "
[19:53] <zyga|afk> Hey
[19:53] <zyga|afk> mvo: I know what that is but I m afk now
[20:28] <zyga|afk> mvo: still here?
[20:36] <mvo> zyga|afk: back now
[20:36] <zyga> hey
[20:36] <zyga> so long story, we use a incompatible format in master
[20:36] <mvo> zyga: you know what it is?
[20:36] <mvo> zyga: meh
[20:36] <zyga> yes, we knew about it
[20:36] <zyga> I think jdstrand assumed that leap is not needed anymore
[20:36] <zyga> and because we didn't return the error all the way
[20:36] <zyga> it was non-fatal
[20:37] <zyga> I actually think there's some correlation between this and the unit test failure
[20:37] <zyga> see
[20:37] <zyga> if some things weren't mocked
[20:37] <zyga> and really ran
[20:37] <zyga> they may really fail
[20:37] <zyga> and then we'd start undoing
[20:37] <zyga> look at master version of the PR that uncovers this
[20:37] <mvo> zyga: hm, so why is it flaky then and does not fail all the time?
[20:37] <zyga> the error in master is exactly the panic that pawel was chasing
[20:37] <mvo> zyga: and its incompatible with 16.04?
[20:37] <zyga> it depends on what ran before
[20:37] <zyga> I don't think it is incompatible with xenial
[20:38] <zyga> but I don't know yet
[20:38] <zyga> I just realised what it was while outside
[20:38] <zyga> I will see for myself on leap 42.3 - i have all those VMs around
[20:38] <mvo> zyga: ok, thats great that we finally have a lead
[20:38] <zyga> I didn't check
[20:38] <zyga> or I don't remember
[20:38] <zyga> if the unit test failure was specific to one system
[20:38] <mvo> zyga: I'm still puzzled that we see these failures on ubuntu
[20:38] <zyga> or was it all over
[20:39] <zyga> yeah
[20:39] <zyga> what's curious is that here (now) we got a message from apparmor parser
[20:39] <zyga> we got the output saying "stuff is bad here and here"
[20:39] <zyga> previously it was just "terminated"
[20:39] <zyga> that's still hinting at another problem
[20:39] <zyga> that's all the insight I have so far
[20:39] <mvo> 2018-11-27 17:11:14 Error executing google:ubuntu-16.04-64:tests/main/parallel-install-interfaces-content:common :
[20:39] <mvo> cannot create temporary directory for /var/lib/snapd mount point: Permission denied
[20:39] <zyga> yeah
[20:39] <zyga> that means you run with edge profile
[20:39] <mvo> zyga: aha, ok
[20:39] <zyga> because you were not able to load the real profile
[20:40] <zyga> maybe 16.04 _is_ incompatible
[20:40] <mvo> zyga: yeah, indeed
[20:40] <zyga> I'll check
[20:40] <zyga> fun stuff
[20:40] <zyga> well
[20:40] <mvo> zyga: that would be nice, thanks a lot, again, great to hear the progress
[20:40] <zyga> "fun"
[20:40] <zyga> logging a problem vs returning it
[20:40] <zyga> because reliability
[20:41] <zyga> while I have you
[20:41] <zyga> https://github.com/snapcore/snapd/pull/6223
[20:41] <mup> PR #6223: cmd/libsnap: move apparmor-support to libsnap <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/6223>
[20:41] <zyga> :D
[20:41] <zyga> pretty plesae
[20:41] <zyga> *please
[20:42] <mvo> zyga: heh, ok
[20:42] <zyga> \o/
[20:42] <zyga> thanks!
[20:43] <zyga> I'm moving this because the next PR will add a tool.c tool.h combo that is all about "I want to run snapd tool from another snapd tool written in C"
[20:43] <zyga> this is the cleanup that maciej requested
[20:43] <zyga> so I'm shaving some yak there
[20:43] <zyga> mvo: https://github.com/snapcore/snapd/pull/6149 this is the next critical in the pipe, but that's for tomorrow
[20:43] <zyga> mvo: I'll get to that apparmor issue now
[20:43] <mup> PR #6149: cmd/snap-confine: capture initialized per-user mount ns <Per-user mount ns  🐎> <Created by zyga> <https://github.com/snapcore/snapd/pull/6149>
[20:43] <zyga> and let's see what we get
[20:44] <mvo> zyga: ok, my critical pipe is 2.36, can we somehow test the theory that its incompatible apparmor? and if incompatible, what exactly is incompatible and is there something we could revert back to?
[20:44] <zyga> yes
[20:44] <zyga> one part of the syntax is
[20:44] <zyga> I have it on screen now
[20:44] <zyga> I don't think we can revert it easily
[20:44] <zyga> it was intertwined with other work jdstrand did for docker and some other snaps
[20:45] <zyga> what we have now is rich enough to know we have old apparmor parser
[20:45] <zyga> so we might be able to offer alternative syntax
[20:45] <zyga> stay tuned
[20:45] <zyga> or
[20:45] <zyga> go get some rest
[20:45] <zyga> go get github.com/mvo5/rest
[20:46] <zyga> leap 42.3 has apparmor parser 2.10.3
[20:46] <zyga> checking 14.04
[20:47] <zyga> 14.04 has 2.10.95
[20:47] <zyga> feels like the .95 is a "backport of lots of stuff"
[20:47] <zyga> I'll check compatibility now
[20:48] <zyga> TW has 2.13
[20:49] <mvo> zyga: thank you
[20:52] <pedronis> mvo: zyga: do we have issues in 2.36 that affects only SUSE ?
[20:52] <pedronis> do we turn on apparmor by default there?
[20:53] <zyga> pedronis: it affects one suse release but it seems that there's some overlap with other distributions because the error was ignored
[20:53] <zyga> no
[20:53] <zyga> but we don't special case that in apparmor backend when  setting up core/snapd
[20:53] <zyga> we could also fix it there somewhat
[20:53] <pedronis> is this specific to snap-confine own profile?
[20:54] <zyga> yes
[20:54] <pedronis> zyga: so we did something there that breaks on some distros != ubuntu where we do use apparmor?
[20:54] <pedronis> but didn't notice because we were ignoring apparmor parsing errors?
[20:54] <zyga> yes
[20:54] <pedronis> mmh
[20:54] <pedronis> fun, not
[20:55] <zyga> not really parsing errors, we ignored all setup errors
[20:55] <zyga> with a comment saying "better to carry on"
[20:55] <zyga> I guess this goes back to the age when we had mysterious errors when snap revision was not in the state
[20:55] <zyga> and we just ignored it
[20:55] <zyga> without knowing what that ultimate cause was
[20:55] <zyga> but as I told mvo, I think that's not the full story
[20:55] <zyga> and that we are missing something
[20:56] <pedronis> zyga: in which sense?
[20:57] <zyga> I think there are two issues related to apparmor parser
[20:57] <zyga> one has a clear message that says we use unknown syntax and happens on leap 42.3
[20:57] <zyga> another has no message at all
[20:57] <zyga> and happens all over the place
[20:57] <pedronis> mmh
[20:57] <zyga> (certainly on xenial)
[20:57] <pedronis> this is related to recent changes to the snap-confine profile?
[20:57] <zyga> no
[20:58] <pedronis> since when is this broken?
[20:58] <zyga> I think ~2 months
[20:58] <pedronis> then
[20:58] <zyga> let me check
[20:58] <pedronis> also on xenial?
[20:58] <pedronis> you are saying 2.35 ... ?
[20:58] <pedronis> or just that 2.36 has been around so long?
[20:58] <zyga> I don't believe xenial parser has issues with this
[20:58] <zyga> one sec, git blame'ing
[20:59] <zyga> hmmm, no that's not right
[20:59] <zyga> it must be something else
[20:59] <zyga> the patch introducing this syntax is from 2016
[20:59] <pedronis> well you said we have two problems
[20:59] <zyga> back to the idea board
[21:00] <pedronis> zyga: anyway this sounds more and more a needs fresh brain issue, that solve it now
[21:00] <zyga> I will check the backend code now
[21:00] <zyga> yeah
[21:00] <zyga> I'm just happy we have something to look at now
[21:00] <pedronis> I would recommend to sleep on it and we talk in the morning
[21:00] <zyga> all yesterday was chasing ghosts
[21:00] <zyga> good idea
[21:00] <zyga> let's EOD
[21:00] <mvo> ok
[21:01] <zyga> see you tomorrow
[21:01] <pedronis> mvo: if it's not a regression we can decide to do 2.36 anyway
[21:01] <pedronis> mvo: sounds like too many moving parts atm
[21:01] <pedronis> mvo: we should discuss in the morning
[21:01] <zyga> pedronis: he's left now
[21:01] <pedronis> ok
[21:01] <pedronis> good either way
[21:02] <pedronis> as stand by my "needs fresh brain" "proclamation"
[21:02] <pedronis> let's chat in the morning again
[21:02] <zyga> yep
[21:02] <pedronis> s/as/I/
[21:02] <roadmr> 🧠
[21:04] <pedronis> roadmr: brain emoji?
[21:04] <roadmr> pedronis: you wanted fresh brain :)
[21:05] <pedronis> now you make me sound like a zombie, "brains" "brains"  :)
[21:05] <roadmr> there's unsurprisingly also an emoji for that :P
[21:06]  * pedronis is unsurprisingly unsurprised :)
[21:07] <roadmr> hehe
[21:10] <kyrofa> It's so tiny here, I thought it was ice cream
[21:11] <roadmr> kyrofa: oh, you mean, like: https://bit.ly/2AsL6KS ?
[21:11] <kyrofa> Haha
[21:34] <mup> PR snapd#6225 opened: tests: fix for failover test on how logs are checked <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/6225>