[01:52] https://lists.debian.org/debian-devel/2017/08/msg00090.html <- thats a lot of words for "we kinda want snapd working." === ikey is now known as ikey|zzz [03:27] PR snapcraft#1600 closed: options: fix core-dynamic-linker on ppc64el/s390x [03:30] PR snapcraft#1595 closed: tests: fix the skip of snapd integration tests in armhf [03:38] hello, where can i find an actual list of snap packages [03:38] i am struggling to that end === JoshStrobl is now known as JoshStrobl|zzz [06:02] good morning [06:07] hey zyga-ubuntu [06:14] zyga-ubuntu: hrm, hrm, looks like we need 2.28.2 and pull in #4004 (nice number). I didn't see this pr and I'm not sure I would have spotted the significance. did we improve the validation in 2.28 or why did it start breaking suddenly? [06:14] PR #4004: interfaces/lxd: lxd slot implementation can also be an app snap [06:16] mvo: hello, I think this is lxd on core (not classic) and we never touched this [06:16] mvo: so lxd-as-a-snap vs lxd slot provided by core [06:16] mvo: I think it was never working but perhaps my assumptions are wrong [06:17] mvo: and it looks untested, maybe we test it in the nightly "important snaps work" thing but I didn't check yet [06:18] mvo: although, the wording "stopped working" in the lxd bug report seems to imply improved validation [06:19] zyga-ubuntu: yeah, stgraber indicated it is a regression - anyways, I am fixing now but would love to understand better why we see it now, maybe pawel knows [06:21] mvo: as far as I understand it interface validation never prevented snaps from refreshing (we juts logged it) and while this has changed rencently it's not 2.28 material [06:22] zyga-ubuntu: yeah, its not the refresh, its the connection that is no longer working for him [06:51] mvo: hmm, on Ubuntu 14.04 we only have 2.27.5 available, why is that? [06:51] mvo: I was trying to understand a failure in 2.27.6 when installed on 14.04 [06:52] mvo: it would fail with Failed to enable unit: File snapd.refresh.timer: No such file or directory [06:54] zyga-ubuntu: 2.27.6 should be in proposed, also via core re-exec [06:54] zyga-ubuntu: what failure on trusty is that, is there a forum post? [07:03] mvo: it's this bug https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1722075 [07:03] Bug #1722075: package snapd 2.27.6~14.04 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1

[07:03] mvo: but I think it's weirder than I initially thought === JanC is now known as Guest86327 === JanC_ is now known as JanC [07:05] PR snapd#4019 closed: release: snapd 2.28.2 [07:07] zyga-ubuntu: is it 2.28.3 material? i.e. should I hold back this release? [07:09] mvo: no, I think it's not meant to work [07:09] mvo: but I'm still looking [07:10] mvo: 17.04 system (upgrade from 16.10) with 14.04 package [07:13] zyga-ubuntu: thanks for investigating [07:15] PR snapd#4020 opened: tests: add test for lxd interface [07:17] __chip__: my snapcraft pr is only about half way down the list, they must be slacker than you guys :) [07:17] zyga-ubuntu: so um i guess we should package 2.28.2 for debian? [07:20] mwhudson: yes, I think so [07:20] mwhudson: btw, interesting observation [07:20] mwhudson: recent 9.2 stable release of debian had brand new flatpak [07:20] mwhudson: perhaps we could do the same [07:20] zyga-ubuntu: good thing i don't have anything at all to do between now and artful release, eh? [07:20] zyga-ubuntu: hmm [07:21] zyga-ubuntu: although 2.28 should work ok via reexec thanks to the graceful apparmor stuff you did [07:21] mwhudson: I need to fix my VM box (reinstall the OS on new drive and then copy data over) [07:21] mwhudson: yep but we could try to get it into 9.3, if there is one [07:21] mwhudson: we could pick the right stable release and go along with it [07:22] yeah that'd be good [07:24] hmm, has the new snap pack landed? [07:24] * zyga-ubuntu looks [07:35] o/ [07:38] PR snapd#4015 closed: run-checks: use nakedret to check for naked returns on long functions [07:40] Issue snapcraft#1604 opened: Homebrew still on snapcraft 2.33 [07:41] homebrew? [07:41] wow [07:41] and mup is still using wrong URLs for github issues [07:42] <__chip__> zyga-ubuntu: snap pack is on master, if that's your q [07:45] yep, that's what I wanted, thank you [07:55] Issue snapcraft#1605 opened: lib* files in $SNAP/usr/lib/ links to /usr/lib/ [07:58] cachio: good morning. we have a new 2.28.3 that fixes an issue with the lxd interface in the beta channel [08:00] PR snapd#4021 opened: release: snapd 2.28.3 [08:14] * zyga-ubuntu -> tea [08:14] spread test for content looks good; I fixed some issues and found one missing case that is now handled (source bind mount) [08:17] oh, chipaca is gone? [08:43] jdstrand: hello [08:44] jdstrand: not sure if you are up yet but I'd like to review and land https://github.com/snapcore/snapd/pull/3965 [08:44] PR #3965: interfaces/mount: add support for parsing x-snapd.{mode,uid,gid}= [08:44] jdstrand: it has one +1 already and I could use a review from you next [09:11] pedronis: are you really around? [09:12] i guess it's early even if he were [09:22] Chipaca: o/ [09:22] thank you for the response on the froum [09:22] how are you feeling [09:23] zyga-ubuntu: better today, thank you [09:24] i'll be off to physio in a bit, that also helps [09:28] PR snapcraft#1606 opened: kbuild plugin: if the parts build dir already contains a .config file, use it as the base config [09:42] ok, off to physio [10:16] Issue snapcraft#1604 closed: Homebrew still on snapcraft 2.33 [10:19] it seems my snap is not able to resolve hostnames, but it has the 'network' interface (and it's connected). how can I debug what's going on? [10:19] Issue snapcraft#1605 closed: lib* files in $SNAP/usr/lib/ links to /usr/lib/ [10:31] ackk: I find running it in devmode helpful in case something's being blocked by apparmor because you'll get log messages [10:31] kalikiana, so, it seems it's trying to access stuff under hostfs, like resolv.conf or os-release [10:31] but it shouldn't really need that [10:32] I mean, having the "network" plug it should have connectivity? [10:32] As a user, I have a snap which I think broke in 2.28. To confirm, how do I easily go back to core 2.27? [10:35] popey: `snap revert` gets you back to the previous revision, you can also do `snap revert core --revision=2925` for a specific one, but afair it must be one of the last three you had on your system [10:35] popey: can you share more information [10:35] hiri is broken on my nvidia laptop [10:35] works on my intel laptop [10:35] it _used_ to work [10:35] I am trying to figure out if hiri broke their snap or we did [10:36] ackk: Yes. Probably something's trying to be smart. Although /etc/os-release should be accessible just fine, other things may be blocked [10:36] popey: can you show any apparmor denials [10:37] popey: and confirm that it works on 2.27.x [10:37] uh [10:37] https://forum.snapcraft.io/t/apparmor-denials-for-hiri-after-updates/2432 [10:37] popey: also can you run hiri from command line with SNAP_CONFINE_DEBUG=yes [10:37] I asked how I can go back to 2.27 :D [10:37] kalikiana, I see stuff like Oct 11 10:36:48 snap kernel: [2414545.321369] audit: type=1400 audit(1507718208.501:1586): apparmor="DENIED" operation="open" profile="snap.lxd.lxc" name="/var/lib/snapd/hostfs/etc/nsswitch.conf" pid=5797 comm="lxc" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0 [10:37] ah, wait [10:37] Tell me that and I will :) [10:37] kalikiana, I doubt lxc is opening that file explicitely [10:37] popey: that's all I need [10:37] thank you [10:39] PR snapd#4021 closed: release: snapd 2.28.3 [10:40] ackk: does lxd have no network? if that's the case you might want to `snap disable lxd; snap enable lxd`. I've seen it some time ago and that was my work-around [10:41] popey: can you please help me out [10:41] popey: I need you to do this: [10:41] ackk: and, put those logs in a bug report ideally [10:41] popey: sudo /usr/lib/snapd snap-discard-ns hiri # (correct the snap name if needed) [10:42] kalikiana, disable/enable didn't work [10:42] kalikiana, the snap has the network plug connected [10:42] popey: SNAP_CONFINE_DEBUG=yes snap run --shell hiri [10:43] kalikiana, lxd does have network, lxc doesn't seem to [10:43] alan@hal:~$ sudo /usr/lib/snapd snap-discard-ns hiri [10:43] sudo: /usr/lib/snapd: command not found [10:43] popey: sudo SNAP_NAME=hiri strace /snap/core/current/usr/lib/snapd/snap-confine snap.hiri.hiri -o snap-confine.strace [10:43] kalikiana, but lxd runs unconfined [10:43] popey: popey sorry, /usr/lib/snapd/snap-discard-ns hiri [10:43] (missing /) [10:43] kk [10:43] ackk: I'm not following... `lxc` is just the command line tool of lxd. What exactly is not working? [10:43] kalikiana, if I run lxc image list ubuntu: [10:44] popey: snap-confine should never access /sys/module/nvidia_modeset/uevent so I'm curious what is going on [10:44] kalikiana, that fetches the simplestreams index from the image server. that fails [10:44] popey: also paste snap version here (I only care about the distrO) [10:44] kalikiana, a curl of the same url from cmdline of course works [10:44] zyga-ubuntu: hang on [10:45] ackk: Can you tr `lxc image list --debug --verbose ubuntu:` and see if it gives some diagnostics? [10:45] http://paste.ubuntu.com/25719311/ [10:46] alan@hal:~$ snap version [10:46] snap 2.28.1 [10:46] snapd 2.28.1 [10:46] series 16 [10:46] ubuntu 16.04 [10:46] kernel 4.10.0-35-generic [10:49] zyga-ubuntu: ^ What next? [10:49] kalikiana, just the same error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: lookup cloud-images.ubuntu.com: no such host [10:51] snappy-m-o autopkgtest 1603 xenial:amd64 xenial:armhf xenial:arm64 artful:amd64 zesty:arm64 [10:51] elopio: I've just triggered your test. [10:52] snappy-m-o autopkgtest 1602 xenial:amd64 xenial:armhf zesty:amd64 [10:52] elopio: I've just triggered your test. [10:56] popey: please exit the shell [10:56] popey: and run strace command again outside [10:56] ok [10:56] sorry for the confusing input [10:56] * zyga-solus fixed his T430 just now [10:57] zyga-solus: are you expecting an strace file, because I have none, just spat to terminal [10:57] zyga-solus: http://paste.ubuntu.com/25719391/ [10:57] popey: perhaps put -o just after strace [10:58] but let me read this [10:59] ackk: Maybe ask in #lxc-dev I'm running out of ideas [10:59] kalikiana, ok, thanks [11:01] popey: sudo SNAP_NAME=hiri strace -f -o snap-confine.strace /snap/core/current/usr/lib/snapd/snap-confine snap.hiri.hiri /snap/core/current/usr/lib/snapd/snap-exec hiri.hiri [11:01] * zyga-solus is walking in the dark here but let's keep our minds open [11:05] zyga-solus: http://paste.ubuntu.com/25719424/ [11:06] oooh [11:06] * zyga-ubuntu found a bug [11:06] (elsewhere) [11:06] *man* [11:08] popey: when you ran this, did you see any denial? [11:11] zyga-ubuntu: [Wed Oct 11 12:11:09 2017] audit: type=1400 audit(1507720265.562:197783): apparmor="DENIED" operation="file_mmap" profile="snap.hiri.hiri" name="/snap/core/3017/usr/lib/snapd/snap-exec" pid=26893 comm="snap-exec" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0 [11:11] thats the only denial when i re-ran your strace above [11:11] hmmm, that's weird [11:12] it says that the profile disallows running snap-exec [11:12] ah [11:12] sorry [11:12] ok [11:12] re-run that with snap-exec path of just /usr/lib/snapd/snap-exec [11:13] (so everything the same but with different command at the end [11:13] cannot snap-exec: cannot parse revision "": invalid snap revision: "" [11:13] alan@hal:~$ sudo SNAP_NAME=hiri strace -f -o snap-confine.strace /snap/core/current/usr/lib/snapd/snap-confine snap.hiri.hiri /usr/lib/snapd/snap-exec hiri.hiri [11:13] ^ was the command [11:13] right [11:14] after sudo add SNAP_REVISION=... where ... is the real revision of hiri [11:14] (little by little) [11:14] ok [11:15] alan@hal:~$ sudo SNAP_REVISION=15 SNAP_NAME=hiri strace -f -o snap-confine.strace /snap/core/current/usr/lib/snapd/snap-confine snap.hiri.hiri /usr/lib/snapd/snap-exec hiri.hiri [11:15] seems to be just sat there [11:15] look at denials please [11:15] nothing [11:16] zyga-ubuntu: what does snap-confine.strace do? [11:16] it's just a file name [11:17] d'oh [11:17] so, jd.strand shared a magic strace of magics [11:17] strace -o /tmp/trace -D -f -vv -e '!_newselect,select,clock_gettime' -- /usr/bin/snap run [11:18] the filtered-out thing is the important bit imho [11:18] popey: ls -ld /sys/module/nvidia/version [11:18] without that strace gets bamboozled by go [11:18] (or by our unusual usage of go) [11:18] -r--r--r-- 1 root root 4096 Oct 2 20:44 /sys/module/nvidia/version [11:18] popey: ok [11:18] I'm lost [11:18] snap list --all [11:19] popey: and please revert to one of the older core revisions you have on your system (the previous one) [11:19] popey: then discard the namespace and try again [11:19] https://pastebin.canonical.com/200343/ is my snap list --all [11:20] (sorry, contains private snaps so not publicly sharing) [11:20] kk [11:20] so core 2898 which is 16-2.27.6 i guess? [11:20] popey: don't forget to stand on the _left_ side of your _right_ foot while reverting, and the order is: adopt stance, start whistling tune, execute commands, leave stance, stop whistling [11:21] <3 [11:21] popey: core 16-2.27.6 2844 canonical core,disabled [11:21] popey: try tihs one [11:21] popey: actually [11:21] before you do that [11:22] popey: try Chipaca's advice on how to strace [11:22] need i mention that the tune is rms's free software tune, backwards and in a minor key? [11:22] zyga-ubuntu: popey is reverting to 2898. [11:23] His irc client is a snap and froze. Which is why he is not replying right now. [11:23] phew! [11:23] * Chipaca ROTBL [11:23] * flexiondotorg end popey-proxy mode [11:24] ok, interestingly my irc clent (irccloud-desktop) locked up during 'phase 2' whatever that is, but carried on after the rollback which is neat. well done team :) [11:24] installed: 16-2.27.6 (2898) 85MB core [11:25] flexiondotorg: interesting! [11:25] Ok, hiri now works fine atfer rolling back to 2.27 [11:25] popey: try hhir now [11:25] popey: ok [11:25] popey: please do this: [11:25] popey: send me all the apparmor profiles from /etc/apparmor.d/snap.core.*.usr.lib.snapd.snap-confine [11:27] zyga-ubuntu: sent [11:28] (email) [11:28] thanks [11:28] let me look [11:28] * zyga-solus walks between kitchen and office [11:30] popey: ok so the working one is 2898 [11:30] * zyga-ubuntu looks [11:33] popey: it makes no sense :/ it looks like the new profile is more forgiving actually [11:36] zyga-ubuntu: we have confirmed that this is also the case with a completely different snap, on another nvidia machine which is 17.10 [11:36] reverting to core 2898 made the snap work there too. [11:36] popey: it looks like nvidia is affected in general [11:36] also an opengl snap [11:36] not specific to hiri [11:36] yes [11:36] now would be a good time for me to have easy way to use nvidia :/ [11:36] (which I don't) [11:36] This is not the first time this problem has bitten us. [11:37] I understand, we don't have any testing of nvidia AFAIK [11:37] Indeed, wasn't pointed at you. It's common across the company. [11:37] I meant me for debugging, I think we need to fix our release process to include simple "works / not works" opengl testing on intel, nvidia and amd GPUs [11:38] I happen to be lucky enough to have two laptops side by side, nvidia and intel, so pick this stuff up. Not everyone has that luxury [11:38] or just nvidia for a start [11:38] popey: I have 3 but all intel [11:38] I understand you can spin up a machine in AWS with a GPU [11:38] There's your problem :) [11:38] popey: with ubuntu kernel? [11:38] Hm, good question. I am not sure. [11:38] probably not [11:38] * zyga-solus looks on ebay [11:39] separately, will I be stuck on rev 2898 now I reverted to it? [11:39] or will the next update push me forward again? [11:40] popey: not sure, I think refresh vs revert but not sure if this is implemented [11:41] Shall I file a bug in snapd for this? [11:41] Feels somewhat critical [11:41] popey: no, the forum thread is enough [11:41] ok [11:45] zyga-solus: do you need remote access to an nvidia machine (I have a desktop here I can slap 16.04 on)? [11:45] popey: that wold help [11:46] lemme dig it out, one mo [11:46] popey: thank you [11:47] mvo: what is the status of your nvidia box? [11:47] mvo: I have a laptop but unused, I could try to boot it from live media or maybe swap HDD and install that way (it's not used by me and has other os) [11:47] zyga-ubuntu: I have a nvidia card somewhere I don't use it usually, but I can plug it in [11:48] mvo: if that fails I'll try to install ubuntu on mine [11:49] zyga-ubuntu: let me try to find it, any opengl game will do? [11:49] mvo: yes [11:49] mvo: or just try hiri [11:50] zyga-ubuntu: what did change in this area recently? could it be snap-confine? [11:50] mvo: I was looking at the diff between the snap-confine profiles and we did change various things for how the lib directory is called in solus and elsewhere [11:50] mvo: but we never ever touch the file we get the denial on [11:50] mvo: so I think something else may be broken [11:51] mvo: and we just see the weird side effect [11:51] zyga-solus: I see - I wonder how we can test this in an integration test [11:51] mvo: modprobe nvidia (maybe) if it sticks [11:51] zyga-ubuntu: what file gets the denials [11:52] [ 2879.277628] audit: type=1400 audit(1507710260.432:1680): apparmor=“DENIED” operation=“open” profile="/snap/core/3017/usr/lib/snapd/snap-confine" name="/sys/bus/pci/drivers/nvidia/uevent" pid=13789 comm=“snap-confine” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0 [11:53] not sure how this happens [11:53] hold on, I think I know [11:54] jdstrand said that nvidia is not in sysfs the regular way and added a hack for supporting udev tagging, remember? [11:54] one sec [11:55] b17d5ba8d46c269dbaba08c0484a13e1e1c9e9e7 [11:56] mvo: it is this patch [11:56] zyga-solus: strange that denial, I add the card now, bbiab (need to open the machine etc) [11:56] * Chipaca needs to reboot, brb [11:56] k [11:57] mvo: I'm making a patch [11:59] popey: still there? [11:59] popey: refresh to stable again [11:59] ok [11:59] popey: and edit the profile in /etc/apparmor.d/ [12:00] popey: the one for snap-confine on revision 3017 (if I recall correctly) [12:00] anyway, the latest one [12:01] popey: ls /sys/bus/pci/drivers/nvidia/ [12:02] popey: in the file you are editing go to line 6 (so inside the { } spanning the whole file) [12:02] popey: and add this line(s) (waiting for your ls output) [12:03] popey: /sys/bus/pci/drivers/nvidia/uevent r, [12:03] popey: then save the file and use "apparmor_parser -r /etc/apparmor.d/snap.core.3017.usr.lib.snapd.snap-confine" to re-load [12:03] popey: then start hiri again [12:03] ok. took a while to get to refresh core. updated now [12:03] popey: thank you [12:05] alan@hal:~$ ls /sys/bus/pci/drivers/nvidia/ [12:05] 0000:01:00.0 bind module new_id remove_id uevent unbind [12:05] ok [12:06] add the uevent line as I said, reload the profile and start hiri [12:06] we'll iterate till it works [12:06] ok [12:06] so added that one uevent r line [12:07] hiri still core dumps [12:08] denials? [12:09] [Wed Oct 11 13:09:07 2017] audit: type=1400 audit(1507723743.250:198339): apparmor="DENIED" operation="open" profile="snap.hiri.hiri" name="/proc/17939/mounts" pid=17939 comm="hirimain" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 [12:09] [Wed Oct 11 13:09:09 2017] audit: type=1400 audit(1507723744.825:198340): apparmor="DENIED" operation="open" profile="snap.hiri.hiri" name="/etc/" pid=17939 comm="hirimain" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0 [12:09] [Wed Oct 11 13:09:09 2017] audit: type=1400 audit(1507723744.833:198341): apparmor="DENIED" operation="open" profile="snap.hiri.hiri" name="/proc/17953/mounts" pid=17953 comm="hirimain" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 [12:09] [Wed Oct 11 13:09:09 2017] audit: type=1400 audit(1507723744.837:198342): apparmor="DENIED" operation="open" profile="snap.hiri.hiri" name="/proc/17954/mounts" pid=17954 comm="hirimain" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 [12:09] nothing else? can you connect the mount interfaces [12:09] * ogra_ sets +q on popeybot [12:09] :P [12:09] or try a some simple opengl snap/ [12:10] i have another gl one that broke [12:10] https://www.irccloud.com/pastebin/saHHCGu4/ [12:10] (for ogra) [12:10] heh [12:11] https://www.irccloud.com/pastebin/1NEBw33c/ [12:11] and another [12:11] you can ignoe inet6 [12:11] popey: works? [12:11] no [12:11] all broken [12:12] https://www.irccloud.com/pastebin/FZfuNsJx/ [12:12] (and we shoudl really quieten it, they should not be printed if there is no v6 network configured) [12:12] ok [12:12] zyga-solus: \o/ for making a patch, let me know once I can test something [12:12] mvo: still at the /o\ stage, I need to wrap my head around this, I assumed a simple tweak would do but I think we tag things incorrectly somewhere and there are devices missing in the namespace [12:13] mvo: so no patch yet [12:18] popey: can you try a richer patch please [12:18] + # See https://github.com/snapcore/snapd/commit/b17d5ba8d46c269dbaba08c0484a13e1e1c9e9e7 [12:18] + /sys/bus/pci/drivers/nvidia/uevent r, [12:18] + /dev/nvidia[0-9]* r, [12:18] + /dev/nvidiactl r, [12:18] + /dev/nvidia-uvm r, [12:19] popey: reload the profile and watch journalctl -f [12:19] popey: then restart hiri [12:19] what was the generic name, on the phone, of things that mediated access to things? like the file manager [12:19] Chipaca: media hub [12:19] Chipaca, content-hub [12:19] ok [12:19] ah [12:19] sorry :) [12:20] but media-hub was clöose enough ;) (same thing. just for media files) [12:20] right, the content hub was one of them, wasn't there a name for all of them? [12:20] trusted helper? [12:20] trusted helper was the equivalent to snap intefaces [12:20] well, not really [12:21] we will use trusted helpers in snapd (we already use some) [12:21] still broken.. [12:21] (to allow direct accesds to bits ... like location or so) [12:21] anyway, distraction [12:21] popey: any denials? [12:21] did you reload the profile? [12:21] popey: try discarding the namespace (just to be sure, I don't think we need to) [12:21] http://paste.ubuntu.com/25719780/ [12:21] i did reload the profile [12:22] ^ me running hiri, goxel and ohmygiraffe [12:23] tried discarding namespace too, also fails [12:25] popey: go to /sys/fs/cgroup/devices [12:26] popey: what do you have there [12:27] lots of files, anything in particular you're interested in? [12:27] popey: snap.* [12:27] http://paste.ubuntu.com/25719806/ [12:28] great [12:28] go to hiri [12:28] then show devices.allow please [12:30]