[02:48] PR snapd#6217 opened: tests: reset snapd state on tests restore [04:18] Bug #1753760 changed: Adding public RSA Key doesn't make login on device possible === chihchun_afk is now known as chihchun === zyga|afk is now known as zyga [06:46] o/ [07:34] People : hi ! [07:34] hello [07:48] brb, need to take the dog out === pstolowski|afk is now known as pstolowski [07:57] mornings [07:58] back to yesterday's test panic issue, the debug log from zyga shed some light on it [07:59] pstolowski: good morning and good luck wit hthat [08:00] Hey Paweł, Michael [08:00] mvo: I’m off the 2.36 bug [08:00] No progress made [08:00] Need to finish my stuff now [08:04] zyga: ok [08:06] zyga: may i ask you to try one more patch> [08:06] ? [08:18] Sure [08:19] zyga: https://pastebin.ubuntu.com/p/Kgs7tXTfRH/ [08:26] pedronis: re 6195 - shall we (after this got a second review) merge and do 2.36.2 and then look for a better place for this than link_snap.go? or would you rather like to look for this place now before it goes to master? [08:31] PR snapd#6218 opened: snapstate: update fontconfig caches on install (2.36) [08:59] zyga: actually that patch won't help; i'll have another one soon [09:00] i wish i understand what the delta is between our setups and why it doesn't fail for me on travis, pretty weird [09:17] pstolowski: re, sorry, had a delay and network woes [09:18] pstolowski: yeah [09:18] pstolowski: if you want I can give you ssh access [09:18] not sure if that would help [09:18] to my box [09:18] where this happens [09:18] I can work on another [09:23] zyga: yeah, that could help, thanks. also question, did we change anything related to finding if apparmor should be enabled or not recenlty? it looks like setup-profiles in this test now wants to read vanilla snap-confine profile of core; this happens if we find that release.AppApparmorLevel != NoApparmor. we're not mocking vanilla profile in the mock core snap obviously [09:24] mvo: you might know as well^ [09:24] pstolowski: not sure when is recently [09:24] when apparmor is not entirely disabled we operate [09:24] including making a special profile for snap-confine itself [09:24] if you think mocking is missing by all means, add some [09:25] yep. it's just curious it's not happening consistently [09:26] moin moin [09:26] hey Chipaca [09:26] have you seen https://forum.snapcraft.io/t/disable-enable-not-longer-working-when-snap-have-gpio-pins-as-connect/6473 ? [09:26] oh noes [09:26] well [09:26] this is a bug we fixed [09:27] zyga: "withthe edge channel" of core presumably? [09:27] yes [09:27] morning Chipaca [09:28] Chipaca: your questions got me curious [09:28] so I did some more experiments on https://github.com/snapcore/snapd/pull/6147 [09:28] PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [09:29] and I believe we understand why additional permissions are required [09:29] my only question now [09:29] should I split the permissions so that snap-update-ns is called with the existing profile transition [09:29] and the only permission we have remains "r" [09:29] adding a new block for snap-discard-ns which runs with the same profile [09:29] or is the patch as-is good enough? [09:30] zyga: WWJ(S)D? [09:30] split it [09:30] :-) [09:30] for maximum confinement [09:30] I'll do that [09:30] btw, love the acronym :D [09:39] Chipaca: https://github.com/snapcore/snapd/pull/6147 is ready now [09:39] PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [09:40] eh [09:40] https://gitlab.gnome.org/GNOME/gnome-control-center/issues/230#note_373259 [09:50] PR snapd#6219 opened: overlord/tests: [WIP] fix panic in managers test [10:00] I need a 2nd review for https://github.com/snapcore/snapd/pull/6147 [10:00] PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [10:01] Chipaca: thank you! [10:21] I also need a 2nd review on https://github.com/snapcore/snapd/pull/6159 [10:21] PR #6159: cmd/snap-confine: handle mounted shared /run/snapd/ns [10:31] zyga: +1 with one question [10:33] PR snapd#6159 closed: cmd/snap-confine: handle mounted shared /run/snapd/ns [10:33] answered [10:33] thanks! :) [10:33] mvo: what's the 2.36 weather like? [10:37] zyga: medium, still see failures but can't reproduce them [10:38] mvo: magic [10:38] mvo: pstolowski is working on a fix for the panic I can reproduce trivially [10:38] I'll just run tests for his incremental patches [10:38] zyga:you can try https://github.com/snapcore/snapd/pull/6219 [10:38] PR #6219: overlord/tests: [WIP] fix panic in managers test [10:38] sure thing [10:39] zyga: btw, it's green, does it mean anything? [10:39] test in progress [10:39] cause i can't repro here, pretty annoying [10:40] pstolowski: I can try to run this after my current run, I had some luck reproducing the panic [10:45] mvo: thanks. i tried running it with spread -debug -repeat 10 google:ubuntu-16.04-64:tests/unit/go [10:51] mvo: I hopefully answered in the PR [11:02] pedronis: ta [11:03] np [11:03] * pedronis goes back to swap daying [11:04] pstolowski: I am running it now again, lets see [11:13] Brb, coffee or something warm [11:26] pstolowski: https://paste.ubuntu.com/p/9rj8ydFg5p/ <- I cherry picked 1c5b166643a5623b0a534fd1a8acb7d03d50d856 from 6219 earlier [11:29] mvo: aha, interesting, let me mock core rev 1 as well [11:30] pstolowski: shall I run it again? [11:30] pstolowski: I mean, after you upated the pr? [11:35] mvo: i've just pushed a change [11:35] mvo: thanks! [11:35] pedronis, I am working again with the #5714 [11:36] PR #5714: tests: new test for cifs-mount interface [11:50] 6195 needs a second review [11:55] mvo: I reproduced 2.36 bug again [11:55] pstolowski: no panic yet [11:56] zyga: are you running with the update i pushed a few minutes ago ^ ? [11:56] pstolowski: no, the one from earlier today [11:56] mock vanilla profile for snap-confine [11:57] mvo: same thing, apparmor parser killed by signal [11:57] zyga: ok; yeah, i think one more revision of core in the tests needs it too, i added it [11:57] ok [11:59] zyga: anything interessting why its killed? [11:59] mvo: I cannot find any information about why [11:59] zyga: or what signal? sigter [11:59] hits appreciated [11:59] zyga: meh :( [12:00] I suspect sigterm because it says "terminated" [12:01] ah [12:01] something interesting [12:01] mvo: look at this [12:01] lots of terminated errors https://www.irccloud.com/pastebin/FVHUi8t8/ [12:01] I thought it is just one time [12:02] mvo: it's not just once [12:02] it seems to happen all the time [12:02] snap changes https://www.irccloud.com/pastebin/2UoYFGwX/ [12:04] *hmm* [12:04] mvo: note that this branch doesn't have the tweak I used before [12:04] so the error was not fatal [12:04] I was testing pawel's patches [12:04] I will record the seed and continue unless anyone has some better idea [12:07] ok, seed is -seed=1543315161 [12:07] I will pull more of pawel's patches [12:08] as well as my stashed error handling [12:08] and let's see [12:08] zyga: thank you [12:08] zyga: I run it also with your diff to make the error fatal [12:09] zyga: but so far no luck in reproducing [12:09] restarted [12:09] back to working on feature [12:13] what's holding back snapd on debian? [12:14] Chipaca: AFAIK there's a test failure that happens there [12:14] but no investigated [12:14] I mean packaging-wise [12:14] it's on 2.30 [12:14] and no reexec [12:15] I believe you are wrong, there is reexec [12:15] hmm [12:15] as for the package, nobody released updates [12:17] I wonder if deepin mangles os-release or sth, then [12:17] * Chipaca installs deepin [12:17] note that we have a blacklist, not a whitelist [12:17] no we don't [12:17] I mean [12:17] if !release.DistroLike("debian", "ubuntu") { [12:18] that's a whitelist, even if it has a "not" in front of it [12:19] oh [12:19] indeed [12:19] that's new to me [12:19] zyga: reading a bit of above, "terminated" is the %s of signal 15 [12:19] aha [12:19] sigkill [12:19] that's odd [12:19] wonder why we do that [12:19] zyga: /usr/share/go-1.6/src/syscall/zerrors_linux_amd64.go [12:19] (not really us probably) [12:19] ze errorz zey keep koming! [12:20] I think it's just shorthand for "zee autogeneraed files" [12:20] zyga: 15 is TERM, so what else would you call it? [12:21] ah, sorry [12:21] 9 is kill [12:21] :-) [12:21] I did make tea, not coffee [12:21] so terminated [12:21] and 9 would say "killed" [12:21] maybe test setup? [12:22] there is one silly way to fix it :) [12:22] we could wrap aa-parser with an apparmor profile [12:22] that rejects SIGTERM :D [12:22] (I'm half serious) [12:22] might help us to pinpoint the sender [12:22] because we would get a denial [12:25] * pstolowski lunch [12:33] is maciej around today? [12:33] ah, he called in sick [12:33] ok [12:43] https://forum.snapcraft.io/t/cannot-run-any-program-trace-breakpoint-triggered-errors/8707?u=chipaca if anybody has clues about apparently x11 being broken on deepin [12:44] * zyga thinks [12:45] I know [12:45] zyga: i'm going to get lunch, but i'll read when i return [12:45] ls /tmp/.X11-unix [12:45] on ubuntu we use abstract socket to talk to x [12:45] on some systems we don't [12:46] without special support we won't have x socket [12:51] If snap A connects to snap B via a content interface, can snap A execute binaries within snap B? [12:51] Wimpress: yes, but must use the internal interface, not via "snap run ..." or /snap/bin/..." [12:51] and said binaries run with confinement of snap A [12:52] snap A is confined. [12:52] snap B might be classic ;-) [12:52] by internal interface I mean by knowing how to execute the binary directly from where the content sharing is establieshed [12:52] Wimpress: content sharing shares bits and bytes [12:52] not changing anything else [12:52] does't grant more permissions to run [12:52] or not run [12:53] OK, so by virtue of having the snaps connected, snap A can not execute binaries from snap B. Correct? [12:53] no, that's not correct [12:53] if I have a snap that contains a shell script [12:53] and I expose that via a content interface [12:53] and that interface is connected to some other sna [12:54] that other snap can invoke that shell script [12:54] Excellent. [12:54] that shell script may not be an exposed application [12:54] and in fact if it is an exposed application [12:54] I cannot use that exposed application from any snap [12:54] note that this doesn't imply any more permissions [12:55] if you content-connect to LXD, for example, you don't get to run LXD with all the powers attached [12:55] So snap A would have to invoke the binary in snap B via a full path? [12:55] it's just a way to share bytes [12:55] Wimpress: with path, environment and any other quirks required [12:55] it's as easy or hard as the associated binary implementation details are [12:57] Would binaries snap A want to execute from snap B be referenced via /snap/snapA/current/bin/foo or via /snap/snapB/current/bin/foo? Or will $SNAP be fine? [12:57] neither, you need to use the real path where they are shared [12:57] so if you share them to, say [12:57] $SNAP/plugins/ [12:57] and find them there [12:58] you need to know to call $SNAP/plugins/foo/bin/plugin.bin [12:58] $SNAP or any other variables are not aware of that [12:58] they will not know anything about any connected content interfaces (perhaps many) [12:58] they will always point to the snap for which the main process was started [12:58] (snap A in this case) [12:58] OK. Sounds promising. [12:59] So, if snap A (confined) is connecting to snap B (classic), snap A can execute binaries exposed from snap B but within the confinement defined by snap A. Correct? [12:59] yes [13:00] Cool. [13:00] it's really just the same [13:00] as if they were copied into snap A [13:00] it's just a way to share bytes [13:00] Just wanted to be clear. [13:00] sure [13:00] zyga: am back. Indeed /tmp/.X11-unix/X0 is where it's at [13:01] yay [13:01] zyga: so what do I do? [13:01] we can easily fix that too [13:02] hold on [13:03] looking for something [13:05] Chipaca: in cmd_run.go we handle XAUTHORITY [13:05] we could handle X11 socket [13:06] zyga: handle it how? [13:06] ha, that's tricky [13:07] we can see the socket [13:07] but not access it [13:07] perhaps we should preserve /tmp/.X11* in snap-conifne [13:07] which is not terribly hard but not super easy either [13:07] Chipaca: can you please file a bug [13:07] or find one [13:07] I bet people reported it [13:07] there are also mentions of this on the forum [13:07] zyga, hey [13:08] zyga: I mean, this is from the forum :-) [13:08] Chipaca: let's cross reference them then [13:08] cachio: hello [13:08] zyga, I could make the cifs test work but ... [13:08] zyga, just adding this to the interface "/run/mount/utab rw," [13:08] zyga: I wonder if there's a way to tell X to use abstract sockets [13:08] I think we discussed that [13:09] cachio: we should not do that [13:09] Chipaca: some systems don't have one [13:09] zyga, based on the comment we shouldn't [13:09] cachio: we cannot add that permission because that would allow snaps to confuse the host's libmount about what certain attributes are [13:11] zyga: it seems to also have an abstract socket, @/tmp/.X11-unix/X0 [13:11] Chipaca: is there anything in the environment [13:11] any X... variable set? [13:11] maybe we can do a low cost fix [13:12] "unset XSTH" [13:13] nothing that I can see [13:14] dunno then [13:18] zyga: and if I install with devmode it workss [13:18] oh [13:18] that's odd [13:18] zyga: so it's not the /tmp/.X11 socket [13:18] what's the denial? [13:18] zyga: it's access to the socket itself [13:18] mvo: any news on the 2nd run of https://github.com/snapcore/snapd/pull/6219 ? [13:18] 1 sec because I accidentally killed the vm :-) [13:18] PR #6219: overlord/tests: [WIP] fix panic in managers test [13:19] pstolowski: I ran it again, 0 failures [13:19] restarted (thanks for reminidng me) [13:19] nice, thanks [13:19] fwtw travis pass was green again [13:22] [ 157.486384] audit: type=1400 audit(1543324906.843:348): apparmor="DENIED" operation="create" profile="snap.xbill-xaw.xbill-xaw" pid=3187 comm="xbill" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none [13:22] had to purge the charon thing to get to see that [13:22] hold on [13:22] * Chipaca goes to check on the soup [13:22] weird [13:23] soup for lunch isn't that weird [13:23] i've got a bad cold [13:23] ¯\_(ツ)_/¯ [13:23] niemeyer, zyga: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-ba8845b83b [13:23] this might make you guys happy :) [13:23] indeed [13:23] Son_Goku: mborzecki is sick, he's off today [13:23] but I'll make sure he knows [13:23] ah, that's too bad [13:23] hope he feels better soon :) [13:24] Chipaca: which interfaces do you have connected [13:24] Chipaca: x11 plug grants socket AF_NETLINK, not unix [13:24] also allows some x abstractions, reading on those now [13:25] Chipaca: but nothing grants access to that socket [13:25] Chipaca: could it be the case that apparmor simply doesn't handle abstract sockets [13:25] and we never noticed? [13:26] sounds like a jdstrand question :-) [13:26] or, maybe a jjohansen ? if i remember people right [13:26] * Chipaca often doesn't [13:30] mmm, tasty tasty schadenfreude http://forum.asrock.com/forum_posts.asp?TID=10174 [13:32] Chipaca: both would know [13:32] ok, i'm off to have actual lunch now [13:32] ttfn [13:32] good idea [13:33] I'll wait till after standup [13:33] I'd love to land https://github.com/snapcore/snapd/pull/6147 [13:33] PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [13:33] mvo: can you do a quick review, in mborzecki's absence [14:22] zyga: https://pastebin.ubuntu.com/p/B5d5n36p2h/ might this be relevant to the x11 thing? [14:22] maybe [14:27] mvo: the bug is https://bugs.launchpad.net/snapd/+bug/1805182 [14:27] Bug #1805182: /tmp is sometime cleaned up after snaps are launched [14:39] socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) vs socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) [14:53] PR snapd#6220 opened: cmd: drop cruft from snap-discard-ns build rules [14:53] mvo: can you do a quick review of https://github.com/snapcore/snapd/pull/6147 please [14:53] PR #6147: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [14:53] it's green and if it lands I can continue [14:53] and has one and a half review [14:54] zyga: I try, have a lot of meeting now :/ [14:54] ok [14:54] pstolowski: perhaps you ^ [14:54] 6147? [14:54] zyga: ^ [14:55] yes [14:55] ok [14:58] meanwhile, 2.36 branch for those errors [15:02] mvo: I just realized something [15:02] the terminated error is not new [15:02] we probably had it since forever [15:02] but we would not notice it until the special situation in 2.36 branch [15:02] this doesn't change why it is broken [15:03] but may help us form new perspective [15:03] because we had ignored those errors all the time [15:03] we would just happily carry on [15:03] PR snapcraft#2417 opened: Revert "lifecycle: make snapcraft init template use > not | (#2393)" [15:09] PR snapd#6221 opened: interfaces: return security setup errors [15:10] mvo: this one is for you https://github.com/snapcore/snapd/pull/6221 [15:10] PR #6221: interfaces: return security setup errors [15:29] PR snapd#6222 opened: cmd/snap: handle DNS error gracefully [15:33] pstolowski: nice [15:33] pstolowski: 0 errors [15:34] zyga: very nice, thanks, i'm cleaning that PR atm, will push an update [15:34] pstolowski: I woudn't mind a follow up [15:34] this looks sane as-is [15:38] well it outputs debug [15:38] pushed [15:38] i'll cherry pick into master [15:38] * cachio lunch [15:40] cachio: offtopic, can we start instances not in us-east [15:40] it would be awesome if we could do europe too [15:40] cachio: latency to us-east sucks :/ [15:41] not sure if this is something you or gustavo needs to handle [15:41] spread on travis is ok [15:41] but interactive spread for debugging is not [15:41] I'm fine with editing my spread.yaml if I can achieve that [15:49] PR snapd#6147 closed: cmd/snap-confine: use snap-discard-ns ns to discard stale namespaces [15:49] thank you mvo! [15:51] mvo: https://github.com/snapcore/snapd/pull/6221 no errors yet! [15:51] zyga: meeting ended early :) [15:51] zyga: oh no [15:51] PR #6221: interfaces: return security setup errors [15:51] well, there are unit test panics that pstolowski is fixing [15:51] zyga: its so annoying [15:51] but none of the other errors [15:52] so while I don't get it, it's interesting output [15:53] zyga: it seems to be happening all the time in the 6128 PR [15:55] mvo: 6128 is merged, I assume you mean a 2.36 backport of that [15:55] well, let's see [15:55] I need to break for dinner now [15:55] I will be back [15:55] mvo: https://github.com/snapcore/snapd/pull/6220 looks like an easy win [15:55] PR #6220: cmd: drop cruft from snap-discard-ns build rules [15:56] and helps me move forward too [15:56] zyga: 6218 - sorry [15:57] zyga: as a general rule I always approve PRs that have only reds [16:14] re [16:14] looking [16:17] mvo: https://github.com/snapcore/snapd/pull/6221 didn't fail on that apparmor issue [16:17] PR #6221: interfaces: return security setup errors [16:17] mvo: do you have any theory as to why the patch may make it better? [16:19] zyga: in a meeting, can you just retrigger? [16:21] zyga, I think we can [16:21] yep [16:21] did [16:21] nice [16:21] there is a problem [16:21] cachio: if you tell me how I'd love to try [16:21] but I need to take the dog out first :) [16:22] the spread cleaner is configured to clean machines just on us-east1-b [16:22] oh [16:22] it is basically because you query instances by zone [16:30] mvo: were you saying desktop-launch in brave was taking ~2s on the 2nd launch on 18.10? [16:30] kenvandine: AFAIK it was update-mime-cache or something like that [16:30] that should only be on the first run [16:32] pstolowski: I just got a notification that ISS is rising [16:32] coool [16:32] the app is not even open [16:33] zyga: :) [16:34] kenvandine: yeah but from what mvo researched it seemed to be a constant cost [16:34] interesting [16:34] I didn't examine that, I think it's best to wait for mvo to leave the meeting frenzy [16:35] i just tried it on my 18.10 laptop and i'm getting good results [16:35] desktop-launch elapsed time: 0.116752997 [16:35] mmm [16:36] kenvandine: interessting - here is my second run:http://paste.ubuntu.com/p/hFbMF8CQKT/ [16:36] pstolowski: https://github.com/snapcore/snapd/pull/6222#issuecomment-442126406 [16:36] PR #6222: cmd/snap: handle DNS error gracefully [16:36] kenvandine: maybe specific to my machine, its a pretty beaten setup :) [16:37] mvo update-mime-database and update-icon-cache should only run on the 1st run [16:37] kenvandine: ok, should I set -x the script to see what is going on? [16:37] kenvandine: let me do that [16:37] yes [16:38] mvo: that would be useful [16:39] mvo: that should only run if needs_update is true [16:39] zyga: damn, i think it's not deteministtic... saw that error once but then stopped getting it as the test was tweaked [16:40] mvo: https://pastebin.ubuntu.com/p/C2M7Y6XQpf/ [16:41] kenvandine: it looks like the user-dirs.dirs check is setting needs_update=true [16:41] kenvandine: I'm trying to figure out now why [16:41] oh [16:42] mvo: that's not setting it for me [16:42] so maybe that's the trigger [16:42] perhaps we shouldn't overload needs_update there [16:43] kenvandine: I have no .md5sum file in my dir it seems [16:45] mvo: what's your actual $HOME/.config/user-dirs.{dirs,locale} look like [16:46] kenvandine: I don't have a user-dirs.locale [16:46] sigh... [16:46] i wonder why not [16:46] kenvandine: I have no idea :/ [16:46] kenvandine: but its there on my other box - what creates it? [16:47] not sure [16:47] mvo: so for you it's doing a bunch of things on every run that it shouldn't [16:47] kenvandine: indeed [16:47] seb128: ^^ what creates user-dirs.locale? [16:48] hey folks, a question! two old revisions of each snap are kept for rollback purposes. But these are squashfs-mounted. Why are they mounted if they're not really in use? [16:49] roadmr: two reasons [16:49] roadmr: we didn't do it better [16:49] roadmr: we don't cache meta/snap.yaml [16:49] so we really go and look each time you ask [16:49] we also rely on the filesystem being there to discover things like hooks [16:49] ok :) [16:51] PR snapd#6220 closed: cmd: drop cruft from snap-discard-ns build rules [16:51] roadmr: given that they consume memory it would be nice to not have to mount those [16:52] roadmr: we might get to a point where snapd caches enough stuff about them to stop needing that [16:52] zyga: indeed, it did seem weird and wasteful but doesn't seem to be the end of the world :) [16:52] pstolowski: I keep getting panics [16:52] those that your PR fixes [16:53] cannot wait for 6219 [16:53] zyga: noooo.. [16:53] zyga: uh oh, you getting them without my PR? [16:53] yes [16:53] btw [16:53] ufff [16:53] inside your pr [16:53] what were the calls to the "snap" command like? [16:54] can you check that in the test quickly [16:54] even locally [16:55] zyga: probably none, i added just in case run-hook is exectued, as this was a case in many other tests before. i can probably remove it here, but there is no harm in leving it [16:55] can you just double check before this goes in [16:56] ok [17:01] zyga already tried mounting on /run/mount but still get the same denial [17:01] apparmor="DENIED" operation="open" profile="snap.test-snapd-sh.with-cifs-mount" name="/run/mount/utab" pid=24510 comm="mount" requested_mask="wrc" denied_mask="wrc" fsuid=0 ouid=0 [17:01] cachio: you'd have to change the interface [17:01] cachio: to do two more things: [17:01] allow that path to be written [17:01] and create a symlink in /run/mount/utab [17:01] the problem is that it's not easy to do 2) [17:01] for the moment I have no solution [17:02] zyga: no calls to snap cmd, verified with fmt.Printf("%q\n", ms.snapCmd.Calls()) in teardown [17:02] cool [17:04] zyga, ok, I'll try that [17:04] cachio: no, please don't [17:05] cachio: there is no fix now [17:05] zyga, ah, ok [17:05] :( [17:05] sorry, no easy win [17:05] mvo: one more trivial https://github.com/snapcore/snapd/pull/6223 [17:05] PR #6223: cmd/libsnap: move apparmor-support to libsnap [17:06] this all setting the stage for the big tool cleanup that mborzecki requested [17:06] PR snapd#6223 opened: cmd/libsnap: move apparmor-support to libsnap [17:22] #6219 failed on unrelated stuff, for the 2nd time [17:22] google:ubuntu-16.04-64:tests/main/interfaces-content-mkdir-writable [17:22] PR #6219: overlord/tests: fix panic in managers test [17:23] cannot create temporary directory for /var/lib/snapd mount point: Permission denied [17:23] (previously it failed on the .mnt unit bug) [17:24] cachio: some quick comments on 6217 [17:25] pstolowski: eh [17:25] 2.36 of doom [17:25] pstolowski: perhaps we should merge the two 2.36 fixes into one branch [17:26] WOAH [17:26] we foudn some useful stuff [17:26] zyga, tx [17:26] error from apparmor-parser https://www.irccloud.com/pastebin/4APIpXG2/ [17:28] jdstrand: hey, I recall there were some parser incompatiblity with older LEAP releases [17:28] this looks similar ^ [17:28] do I recall if you had a PR that fixed this in master [17:29] this comes from a PR that makes errors actually reported: https://github.com/snapcore/snapd/pull/6221 [17:29] PR #6221: interfaces: return security setup errors [17:29] it is against 2.36, not master [17:30] zyga: ! [17:31] PR snapd#6224 opened: interfaces: return security setup errors [17:31] I opened one against master [17:31] let's see what happens [17:31] I need to take Bit out for that walk! [17:31] I'm terrible at tracking time sometimes [17:31] ttyl === pstolowski is now known as pstolowski|afk [18:03] mvo: I'll break to get some groceries with wife === zyga is now known as zyga|afk [19:05] * cachio afk [19:14] zyga|afk: hm - so "AppArmor parser error for /var/lib/snapd/apparmor/profiles/snap-confine.core.6016 in /var/lib/snapd/apparmor/profiles/snap-confine.core.6016 at line 103: Exec condition must begin with '/'." - in my profile line 103 is: " change_profile unsafe /** -> [^u/]**, [19:14] " [19:53] Hey [19:53] mvo: I know what that is but I m afk now [20:28] mvo: still here? [20:36] zyga|afk: back now === zyga|afk is now known as zyga [20:36] hey [20:36] so long story, we use a incompatible format in master [20:36] zyga: you know what it is? [20:36] zyga: meh [20:36] yes, we knew about it [20:36] I think jdstrand assumed that leap is not needed anymore [20:36] and because we didn't return the error all the way [20:36] it was non-fatal [20:37] I actually think there's some correlation between this and the unit test failure [20:37] see [20:37] if some things weren't mocked [20:37] and really ran [20:37] they may really fail [20:37] and then we'd start undoing [20:37] look at master version of the PR that uncovers this [20:37] zyga: hm, so why is it flaky then and does not fail all the time? [20:37] the error in master is exactly the panic that pawel was chasing [20:37] zyga: and its incompatible with 16.04? [20:37] it depends on what ran before [20:37] I don't think it is incompatible with xenial [20:38] but I don't know yet [20:38] I just realised what it was while outside [20:38] I will see for myself on leap 42.3 - i have all those VMs around [20:38] zyga: ok, thats great that we finally have a lead [20:38] I didn't check [20:38] or I don't remember [20:38] if the unit test failure was specific to one system [20:38] zyga: I'm still puzzled that we see these failures on ubuntu [20:38] or was it all over [20:39] yeah [20:39] what's curious is that here (now) we got a message from apparmor parser [20:39] we got the output saying "stuff is bad here and here" [20:39] previously it was just "terminated" [20:39] that's still hinting at another problem [20:39] that's all the insight I have so far [20:39] 2018-11-27 17:11:14 Error executing google:ubuntu-16.04-64:tests/main/parallel-install-interfaces-content:common : [20:39] cannot create temporary directory for /var/lib/snapd mount point: Permission denied [20:39] yeah [20:39] that means you run with edge profile [20:39] zyga: aha, ok [20:39] because you were not able to load the real profile [20:40] maybe 16.04 _is_ incompatible [20:40] zyga: yeah, indeed [20:40] I'll check [20:40] fun stuff [20:40] well [20:40] zyga: that would be nice, thanks a lot, again, great to hear the progress [20:40] "fun" [20:40] logging a problem vs returning it [20:40] because reliability [20:41] while I have you [20:41] https://github.com/snapcore/snapd/pull/6223 [20:41] PR #6223: cmd/libsnap: move apparmor-support to libsnap [20:41] :D [20:41] pretty plesae [20:41] *please [20:42] zyga: heh, ok [20:42] \o/ [20:42] thanks! [20:43] I'm moving this because the next PR will add a tool.c tool.h combo that is all about "I want to run snapd tool from another snapd tool written in C" [20:43] this is the cleanup that maciej requested [20:43] so I'm shaving some yak there [20:43] mvo: https://github.com/snapcore/snapd/pull/6149 this is the next critical in the pipe, but that's for tomorrow [20:43] mvo: I'll get to that apparmor issue now [20:43] PR #6149: cmd/snap-confine: capture initialized per-user mount ns [20:43] and let's see what we get [20:44] zyga: ok, my critical pipe is 2.36, can we somehow test the theory that its incompatible apparmor? and if incompatible, what exactly is incompatible and is there something we could revert back to? [20:44] yes [20:44] one part of the syntax is [20:44] I have it on screen now [20:44] I don't think we can revert it easily [20:44] it was intertwined with other work jdstrand did for docker and some other snaps [20:45] what we have now is rich enough to know we have old apparmor parser [20:45] so we might be able to offer alternative syntax [20:45] stay tuned [20:45] or [20:45] go get some rest [20:45] go get github.com/mvo5/rest [20:46] leap 42.3 has apparmor parser 2.10.3 [20:46] checking 14.04 [20:47] 14.04 has 2.10.95 [20:47] feels like the .95 is a "backport of lots of stuff" [20:47] I'll check compatibility now [20:48] TW has 2.13 [20:49] zyga: thank you [20:52] mvo: zyga: do we have issues in 2.36 that affects only SUSE ? [20:52] do we turn on apparmor by default there? [20:53] pedronis: it affects one suse release but it seems that there's some overlap with other distributions because the error was ignored [20:53] no [20:53] but we don't special case that in apparmor backend when setting up core/snapd [20:53] we could also fix it there somewhat [20:53] is this specific to snap-confine own profile? [20:54] yes [20:54] zyga: so we did something there that breaks on some distros != ubuntu where we do use apparmor? [20:54] but didn't notice because we were ignoring apparmor parsing errors? [20:54] yes [20:54] mmh [20:54] fun, not [20:55] not really parsing errors, we ignored all setup errors [20:55] with a comment saying "better to carry on" [20:55] I guess this goes back to the age when we had mysterious errors when snap revision was not in the state [20:55] and we just ignored it [20:55] without knowing what that ultimate cause was [20:55] but as I told mvo, I think that's not the full story [20:55] and that we are missing something [20:56] zyga: in which sense? [20:57] I think there are two issues related to apparmor parser [20:57] one has a clear message that says we use unknown syntax and happens on leap 42.3 [20:57] another has no message at all [20:57] and happens all over the place [20:57] mmh [20:57] (certainly on xenial) [20:57] this is related to recent changes to the snap-confine profile? [20:57] no [20:58] since when is this broken? [20:58] I think ~2 months [20:58] then [20:58] let me check [20:58] also on xenial? [20:58] you are saying 2.35 ... ? [20:58] or just that 2.36 has been around so long? [20:58] I don't believe xenial parser has issues with this [20:58] one sec, git blame'ing [20:59] hmmm, no that's not right [20:59] it must be something else [20:59] the patch introducing this syntax is from 2016 [20:59] well you said we have two problems [20:59] back to the idea board [21:00] zyga: anyway this sounds more and more a needs fresh brain issue, that solve it now [21:00] I will check the backend code now [21:00] yeah [21:00] I'm just happy we have something to look at now [21:00] I would recommend to sleep on it and we talk in the morning [21:00] all yesterday was chasing ghosts [21:00] good idea [21:00] let's EOD [21:00] ok [21:01] see you tomorrow [21:01] mvo: if it's not a regression we can decide to do 2.36 anyway [21:01] mvo: sounds like too many moving parts atm [21:01] mvo: we should discuss in the morning [21:01] pedronis: he's left now [21:01] ok [21:01] good either way [21:02] as stand by my "needs fresh brain" "proclamation" [21:02] let's chat in the morning again [21:02] yep === zyga is now known as zyga|afk [21:02] s/as/I/ [21:02] 🧠 [21:04] roadmr: brain emoji? [21:04] pedronis: you wanted fresh brain :) [21:05] now you make me sound like a zombie, "brains" "brains" :) [21:05] there's unsurprisingly also an emoji for that :P [21:06] * pedronis is unsurprisingly unsurprised :) [21:07] hehe [21:10] It's so tiny here, I thought it was ice cream [21:11] kyrofa: oh, you mean, like: https://bit.ly/2AsL6KS ? [21:11] Haha [21:34] PR snapd#6225 opened: tests: fix for failover test on how logs are checked === phoenix_firebrd is now known as murthy