[01:47] PR snapcraft#800 closed: LP: #1623509 nodejs: fix to install dev-deps in build phase === chihchun_afk is now known as chihchun === drizztbsd is now known as timothy [05:54] ogra_: I just wanted to report that the snap updates for QOwnNotes now work automatically after Launchpad published them to the Ubuntu Store, thanks a lot. I released 16.09.8 this morning... [06:21] Is there a way to automatically create a snap from a launchpad build recipie ... At the moment it's asking me to build the yaml ...I guess the information is there in the control file and make lists... [07:01] PR snapd#1931 opened: patch: fix patch4 transition [07:09] PR snapd#1932 opened: add HACKING.md and move content from README.md over [07:18] Bug #1571497 changed: snapd crashes on slot disconnection [07:42] Bug #1571497 opened: snapd crashes on slot disconnection [08:00] PR snapd#1931 closed: patch: fix patch4 transition === test is now known as Guest38976 === john-mca` is now known as john-mcaleely [08:27] Bug #1624265 opened: Ubuntu Core pi3 image booting up hang at 4 icons [09:02] mvo, so what about that release ? want me to do it ? we cant do a new ubuntu-core promotion anyway until console-conf behaves, so i guess waiting for the ARU wont get us much further anyway [09:02] s/ARU/SRU/ [09:04] ogra_: sru happend this morning [09:04] mvo, yes, but how would that help ? [09:04] ogra_: new snapd 2.15 is in the ppa, so this part is ready [09:04] ogra_: what is the issue with console-conf? [09:05] it doesnt generate a network config at all on the pi3 currently [09:05] (well, it does, but after reboot it is gone) [09:05] ogra_: is that a regression? [09:05] well, it is there since wifi support was added [09:05] so yes and no [09:05] hm, tricky [09:06] (regression in new feature) [09:06] so with the old console-conf it was possible to get working wired networking on the pi3? [09:06] you get all the wifi stuff in the UI and it behaves just fine, but after reboot the config is gone [09:06] if so and the new one does not manage that then I think its a regression and thats bad :/ [09:06] yes [09:07] so even wired network does not work on the pi3? or will that part work? [09:07] ogra_: do you know what the status of that is? is mwhudson working on it? [09:07] (or someone else?) [09:07] i have to check with todays edge, but i think its bogus all the way [09:07] hm hm [09:08] the non-persistence should be fixed with new snapd [09:08] he is looking into various wifi issues (last screen doesnt show teh IP for ssh) [09:08] mwhudson: aha, cool [09:08] (some issue with SSIDs) [09:08] https://github.com/CanonicalLtd/subiquity/pulls <- some fixes here [09:08] (and the persistence) [09:08] i wanted cyphermox to review and land and do a release [09:08] but if there's some urgency i can merge them anyway... [09:09] ogra_: I would love to make a new beta image based on snapd 2.15 and with a new snapweb (that actually starts at boot), lool made some really nice fixes here [09:09] so the new snapd in the PPA should fix the persistence ? i'll do a quick image test ... [09:09] ogra_: uploaded only some minutes ago, not sure if its published in the ppa yet [09:09] ogra_: if it has https://github.com/snapcore/snapd/pull/1901 in it [09:09] PR snapd#1901: firstboot: do not overwrite any existing netplan config [09:09] ogra_: but please do and let me know, I go and work on snapweb now for a bit to see if I can make it releasable [09:10] ok [09:10] mwhudson: that is in [09:10] ok [09:10] mwhudson, what abotu the last screen not showing the login data [09:10] ogra_: i think that's fixed by accident by https://github.com/CanonicalLtd/subiquity/pull/162 [09:10] PR CanonicalLtd/subiquity#162: Indicate if a wired interface is connected or not [09:11] ogra_: my theory is that console-conf doesn't re-probe to get the new addresses properly [09:11] hmm, ok [09:11] i guess thast not in the PPA package yet ? [09:13] heh, i didnt get tzo read bugmail yet ... i see you answered on all my bugs :) [09:13] yeah, not in ppa yet [09:20] PR snapd#1933 opened: store: add "ready to buy" method [09:24] mvo, snapd just published in the PPA ... i triggered a new ubuntu-core ... [09:25] Hi! where does pip and npm fall in the snappy world, can I install pip packages today on a Ubuntu Core system ? [09:26] both are used by snapcraft when building a snap AFAIK [09:27] stgraber: broke lxc again... error: not found (not a fingerprint, partial fingerprint (first 12 chars) or valid alias) [09:31] you really need to stop beating it with a club all the time :) [09:32] :( [09:52] hi, I've got 3 snaps been waiting for review for ~1 week, eventstat, forkstat and powerstat, can they be looked at for me? thanks [09:53] if they went to review something is surely wrong ? [09:53] ogra_, like they exist in the main repo, so I have a name clash [09:53] s/repo/archive/ [09:54] you mean you didnt claim the names before uploading ? [09:54] ogra_, I've claimed the names and it's now marked "pending review" [09:55] you mean the name request ? [09:55] weird [09:55] i'm even the upstream developer, so it should be easy to claim stuff I own and package ;-) [09:56] * cking just keen to get the latest crack out [09:58] mvo, mwhudson, very weird, i dont get wlan offered at all on the pi3 with the latest edge image [10:01] PR snapd#1934 opened: many: refresh all snap decls [10:03] hmm, and no credentials even for wired now ... [10:03] thats bad [10:03] * ogra_ tries again [10:08] same on the dragonboard ... no credentials on last screen [10:09] pi3 shows them on second try [10:09] ~/wg 54 [10:09] sorry [10:24] ogra_, so I assume I just wait until somebody eventually allows me to upload my snaps oneday?! [10:25] well, i havent seen such issues with name registration ... and i dont knwo who our contact for EU timezone in the store team is [10:25] ah, OK :-/ [10:26] i guess you need a store person ... are you actualyl sure the name isnt actually registered for you already ? if you try to register twice that might cause an issue [10:28] pretty sure I've not registered twice, well, it's not a big rush, just a nice to snap kinda thing [10:29] well, if you did, it would show that on the store page for your upload at the bottom i think [10:29] (teh store UI is really good hiding things at the bottom ... scrolled off the screen) [10:29] * cking has a look [10:30] nah, it's not got a link to it, it's black, no link and "Pending review", so kinda stuck at that [10:30] cking: approved [10:30] popey, woot, my here [10:30] *hero [10:31] \o/ [10:31] mvo, so there seems to be a race with console-conf on the pi3 now ... it doesnt show wlan0 at all ... another thing is, if you reset the board hard before console-conf did write anything, firestboot goes all mad and you can throw away the image [10:31] (wlan shows up in 7proc/net/dev after i configured wired network and rebooted) [10:49] Bug #1624322 opened: console-conf wlan race on pi3 [10:50] mvo, so bug 1623119 seems fixed at least ... [10:50] Bug #1623119: wlan setup done by console-conf not persistent [10:51] mvo, but bug 1623120 hits hard on the dragonboard [10:51] Bug #1623120: console-conf does not show ssh credentials on last page when wlan is involved [10:51] mvo, and bug 1624322 on the pi3 ... but here we can resort to recommend to only use wired networking [10:51] Bug #1624322: console-conf wlan race on pi3 [10:58] Bug #1624329 opened: snapd firstboot setup fails constantly when rebooting before console-conf has finished [11:07] mvo, so i'd say we can release but with a big fat hint that wifi config is still pretty much broken ... and with pointers to all these bugs above [11:08] OH ! [11:09] thats a new one [11:09] ogra@pi3:~$ sudo snap install classic --devmode --edge [11:09] error: cannot install "classic": cannot get nonce from store: Post https://myapps.developer.ubuntu.com/identity/api/v1/nonces: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers [11:09] eeek ... seems there is no resolver set up :/ [11:10] woah ... and i have two default routes ... thats gross [11:20] * ogra_ takes a break [11:51] PR snapcraft#805 opened: Refresh discharge macaroon if necessary [11:53] ogra_: if core is build, I would like to promote it to "beta" [11:53] ogra_: any concerns aobut this? [11:53] mvo, well, see my babbling in the last hour above :P [11:54] there are three serious bugs ... that we should mention in the announcement ... beyond that i think we are fine [11:54] and i think none of them affect an already installed system, so updates shoould be fine too [11:55] ogra_: aha, so all wireless broken but the rest is ok(ish)? [11:55] ogra_: I think thats ok, its still a beta (or even pre-beta) but it looks encouraging [11:55] well, wireless is semi broken ... on the dragonboard it works just fine but console-conf doesnt show the IP .... [11:55] on the pi3 there is some weird race between console-conf and the wlan device coming up [11:56] which makes everything else weird [11:56] ogra_: ok [11:56] but yeah, i think we are good, but should mention the issues [11:58] ogra_: if you have a running system, could you please "sudo snap refresh --edge snapweb" and reboot and check if snapweb:4200 comes up? [11:58] ogra_: ++ [11:58] mvo, btw, bug 1624329 is a really bad one ... it actually forces you to do a fresh dd if console-conf doesnt finish [11:58] Bug #1624329: snapd firstboot setup fails constantly when rebooting before console-conf has finished [11:58] ogra_: oh, fun, I can only promote ubuntu-core to beta now, not stable, is there something with confinement: devmode somewhere? [11:58] sadly hard to debug without being able to log in [11:59] i dont have a running system from the last image, but i can snap install snapweb quickly ... [11:59] ogra_: ta, I will also test [12:00] comes up right after install [12:00] * ogra_ reboots to see if it comes up on boot too [12:01] uuh ... snapweb UI still allows me to remove the kernel [12:02] ogra_: do we set "grade: devel" somewhere? [12:02] mvo, looks good ... [12:02] ogra_: \o/ [12:02] i have to check what we set in ubuntu-core ... [12:03] yeah [12:03] ubuntu-core has garde:devel atm [12:03] ogra_: I think we want to not set this, we want ubuntu-core to be able to get to stable [12:03] ogra_: not important right now, but probably soon(ish) [12:03] well, i can change it right now ... next build will ppick it up [12:04] we dont set grade at all for the kernels ... so whatever snapcraft puts in place is used there [12:05] mvo, long term we need to stop using the PPA built ubunt-core for anything else but the stable channel anyway ... we'll have multiple builds once i killed the PPA [12:05] err [12:05] for anything else but the edge channel ... indeed [12:05] the build coming only from the archvive will then be grade: stable [12:07] ogra_: yeah [12:11] I kept SSH running to my RPi3 last night so I could discover why it's dark and dead after many hours. I thought it was crashing somehow. But it's not. Last night I got a new "ubuntu-core" (634) from edge, and it set a 10-minute reboot timer, and then shut down. It never recovered. Network and console are dead until I unplug and re-plug. [12:12] mvo, snapweb on arm64 looks good too [12:13] qengho, file a bug (worked here) [12:13] ogra_: file a bug on what? [12:13] ubuntu-core? [12:13] qengho, dunno, snappy in general for a start [12:14] Okay. [12:14] i.e. see topic :) [12:14] I guess we don't have a bug tracker for snaps themselves. I wonder if we will. [12:15] well, having one for ubuntu-core would be nonsense ... [12:15] and others can define their bugtracker in the store [12:16] there is a "support url" you can set up [12:16] uappexplorer actually exposes it as a clickable link ... i hope one day snapweb will too [12:18] PR snapcraft#806 opened: Remove snapcraft.storeapi.common [12:20] qengho, btw, an rpi without serial cable is useless ... get one and you can actually debug such issues ... ssh wont help [12:21] (it really needs to become company policy that people getting *any* devboard also get a matching serial cable) [12:21] Canonical didn't pay for this. I'm just nice. [12:21] ah [12:22] I have a keyboard and HDMI display directly attached. [12:22] yeah, wont help wither [12:22] *either [12:22] Even after setting boot params? [12:23] embedded boards need serial if you want to do proper debugging ... they are ... well ... embedded ... and usually used headless ... even if people start abusing them in other ways (like running desktops) [12:24] I guess so. I hadn't considered my retail Pi3 to be a development board. [12:27] on the pi the prob is that the console suspends at some point ... regardless ... so the above reboot prob can only really be debugged with serial attached ... unless you are sitting next to the board while it does the auto-reboot === chihchun is now known as chihchun_afk [12:38] Bug #1624371 opened: pi3 never recovers from ubuntu-core refresh [12:39] qengho, oooh ... you used an ancient all-snaps image ... yeah, thats not expected to work at all [12:40] Dang. [12:41] Okay, getting the ~mvo 10-day-old one. [12:43] qengho, no, wait a bit [12:43] we're in the middle of doing a release [12:44] (especially since we had to pull back the last pi3 one due to xz corruption) [12:44] Bug #1624265 changed: Ubuntu Core pi3 image booting up hang at 4 icons [12:44] Bug #1624371 changed: pi3 never recovers from ubuntu-core refresh [12:45] PR snapcraft#807 opened: Fix parts integration tests [12:58] I am having trouble with content sharing. I am trying to get my content interface under /kf5 in the consuming snap but have a permission error: cannot mount /snap/kde-frameworks-5/x8 at /snap/yomo/x13/kf5 with options bind,ro. errmsg: Permission denied [12:59] https://paste.kde.org/pd8l5excq content interface https://paste.kde.org/puxumufie app === matteo` is now known as matteo [13:24] PR snapcraft#808 opened: Don't litter /tmp with test artifacts [13:27] jdstrand: hey [13:27] jdstrand: around by any chance? :) [13:27] do I need to set XDG_DATA_DIRS manually to e.g. $SNAP/usr/share or does this happen automaticall? [13:27] zyga: I am [13:27] jdstrand: I'm chasing that kernel bug, I feel I get an idea what this may be about [13:27] jdstrand: I'll know after the next reboot [13:27] oh [13:28] jdstrand: the tracking bug is https://bugs.launchpad.net/snap-confine/+bug/1624300 [13:28] Bug #1624300: Cannot open /proc/self/ns/net [13:28] that's curious [13:28] jdstrand: I'll update it with my findings [13:28] yes, I saw your bug [13:28] (I had a crash course in kernel development last night) [13:28] nice :) [13:29] jdstrand: the thing I know for sure is that this is *not* related to nsfs [13:29] but to fs permissions actually [13:29] well, more data after next reboot [13:29] I got into the devel feedback cycle now [13:31] if it is the kernel, I'm going to immediately wonder why it ever worked and why downgrading didn't help, but I'm definitely all ears [13:32] jdstrand: that I cannot answer yet [13:46] sitter, hey, have you tried sharing sub-directories rather than the entire snap? Do you get the same error? [13:51] hm [13:52] kyrofa: no error if I use expose read: [usr] rather than / [13:52] kyrofa: there was a bug in older snap-confine [13:52] kyrofa: use a subdirectory please [13:53] zyga: can I wildcard instead? [13:53] sitter: no [13:53] meh [13:53] dpb, jdstrand, mvo: https://github.com/snapcore/snapd/pull/1935 [13:53] PR snapd#1935: interfaces/apparmor: allow reading /etc/environment [13:53] mhall119: http://mhall119.com/2016/09/sharing-is-caring-with-snaps/ you may want to update that given exposing / doesn't work anymore :) [13:54] sitter: I personally fixed that bug [13:54] PR snapd#1935 opened: interfaces/apparmor: allow reading /etc/environment [13:54] sitter: I don't know if the fix got SRUd [13:54] sitter: but it is fixed in master for sure [13:54] sitter: or maybe it is another thing [13:54] sitter: do you have a denial/error message or anything I can look at? [13:55] all I get is permission error: cannot mount /snap/kde-frameworks-5/x8 at /snap/yomo/x13/kf5 with options bind,ro. errmsg: Permission denied [13:55] I can get you more output if you tell me how ;) [13:56] sitter: can you link to your snapcraft.yaml for both? [13:56] sitter: which version of snap-confine do you have? [13:56] sitter: it was fixed for the way I was using it [13:56] zyga: Version: 1.0.38-0ubuntu0.16.04.8 [13:56] sitter: and look at dmesg/journalctl for the interesting apparmor denial [13:57] mhall119, see above, he pastebinned it already [13:57] * mhall119 scrolls [13:58] jdstrand: can you please review and merge https://github.com/snapcore/snapd/pull/1935 [13:58] PR snapd#1935: interfaces/apparmor: allow reading /etc/environment [13:58] sitter: can you pastebin /etc/apparmor.d/usr.lib.snapd.snap-confine [13:59] zyga: Sep 16 15:59:01 smith kernel: audit: type=1400 audit(1474034341.962:76): apparmor="DENIED" operation="mount" info="failed srcname match" error=-13 profile="/usr/lib/snapd/snap-confine" name="/snap/yomo/x20/kf5/" pid=27361 comm="ubuntu-core-lau" srcname="/snap/kde-frameworks-5/x10/" flags="rw, bind" [14:00] mhall119: https://paste.kde.org/pgou4aa9l [14:01] sitter: ah, you don't have that fix [14:01] * sitter shakes fist [14:01] zyga: https://paste.kde.org/pgou4aa9l#line-120 he still has the older mount permissions that didn't work with a snap's root [14:02] mhall119: correct [14:03] sitter: you can fix this locally with a simple patch and one command run [14:03] * zyga gets back to debugging the kernel [14:05] zyga: don't be a tease, tell him the patch and command :-P [14:05] * mhall119 remembers the patch, but not the commant [14:05] well. I'd rather have a fix :P [14:05] I can't really sell a patch to my fellow lazydevs ^^ [14:06] sitter: use a subdirectory (even dummy) for now, I'll ensure the update is released [14:06] true, zyga do you know if there's an SRU in the works? [14:06] mhall119: no, because I'm debugging kernel to make it releaseble [14:06] I have another question though... http://snapcraft.io/docs/reference/interfaces suggests that the read property of the slot would be an array. the target property of the plug is a string though? [14:07] sitter: they are both strings for now [14:07] well [14:07] each one is [14:07] well, let me just look [14:07] :) [14:07] they are both arrays [14:08] sorry for the confusion [14:08] so the documentation probably needs a fix to indicate that :) [14:08] PR snapd#1936 opened: asserts: revert change that made the account-key's name mandatory [14:08] I am going to roll with subdirs for now. thanks [14:28] PR snapd#1937 opened: image: ensure local snaps are put last in seed.yaml === chihchun_afk is now known as chihchun [14:30] PR snapd#1935 closed: interfaces/apparmor: allow reading /etc/environment [14:51] can anyone point me at a place to track support for snaps on trusty? [14:51] * ogra_ hands mthaddon a cable with a plug to tvoss' brain interface [14:52] haha - I was meaning is there a bug or feature request somewhere on it? [14:53] mthaddon: not really, we have a trello card but I do not know if that's publicly available [14:54] ok thanks - is do you have a rough idea of when it might happen? Would it be worth me filing a bug about it? [14:54] mthaddon: I'm proposing required changes to snapd as we speak [14:54] sweet [14:55] mthaddon: the ppa I'm using for testing purposes is here: https://launchpad.net/~thomas-voss/+archive/ubuntu/trusty [14:55] thx muchly [14:55] ogra_: ohh, brain interface [14:55] I must connect that one next time I'm debugging things [14:57] zyga: bit painful in the beginning, but hey [14:58] mthaddon: http://pastebin.ubuntu.com/23186546/ should get you off the ground [14:58] :D [14:58] tvoss: that looks like something snap-confine could do to test in soread [14:58] spread* [14:59] thx [14:59] zyga: wow, snap-confine can test brain interfaces? ;) [14:59] tvoss: the test may fail on Friday though [15:03] $ sudo snap install appx [15:03] sudo: unable to resolve host repackappx [15:03] 74.93 MB / 74.93 MB [===============================================================] 100.00 % 4.42 MB/s [15:03] error: cannot perform the following tasks: [15:03] - Mount snap "ubuntu-core" (423) ([start snap-ubuntu\x2dcore-423.mount] failed with exit status 1: Job for snap-ubuntu\x2dcore-423.mount failed. See "systemctl status "snap-ubuntu\\x2dcore-423.mount"" and "journalctl -xe" for details. [15:03] ) [15:03] .. [15:03] i can't install snaps inside a lxd container? [15:03] jdstrand: has there been anymore developments behind https://bugs.launchpad.net/snapcraft/+bug/1577514 ? [15:04] Bug #1577514: Support preloading to make snaps feel at home [15:04] xnox it certainly fails in chroot, systemd refuses to play along nicely [15:04] zyga: would certainly fail over here [15:05] SamYaple: not to my knowledge. I think people have been focusing on GA (but I'm not assigned to it or doing the assigning) [15:06] jdstrand: I traced the error we're seeing to apparmor [15:06] jdstrand: thats about to be a real blocker to me in packaging openstack, do you know someone I could reach out to to have a chat about it? [15:06] sforshee: o/ [15:06] zyga: o/ [15:06] jdstrand, sforshee: so I think I wrapped my head around the bug now, I guess I'll need you and tyhicks to guide me further [15:07] the root cause is apparmor and this ALLOWED (but really not) message: [15:07] Sep 16 17:01:45 xenial-server audit[5944]: AVC apparmor="ALLOWED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="snap.snapd-hacker-toolbelt.busybox//null-/home/zyga2/snap-confine/spread-tests/xfail/lp-1623400/bug" name="" pid=5944 comm="bug" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0 [15:07] this is allowed but we really see -13, aka EACCES [15:07] PR snapd#1934 closed: many: refresh all snap decls [15:08] to confirm this you can use devmode snapd-hacker-toolbelt, run it with sudo and switch the profile to unconfined with aa-exec [15:08] so I see two things here that are odd: [15:08] 1) apparmor is in complain mode and says ALLOWED but we still get -13 [15:08] I wonder if it has something to do with nnp [15:08] 2) how do we not get the disconnected path, I suspect I messed up pivot_root somehow [15:08] nnp? [15:08] NO_NEW_PRIVS [15:09] mmm [15:09] how can I quickly check that? is that in snap-confine's aa profile [15:09] * zyga looks [15:09] fyi, on 15.04 snappy vm I am able to open the file [15:10] jdstrand: I'll try with snap-confine master [15:10] to see if this is a new issue [15:10] it shouldn't be new [15:10] jdstrand: I'd love to get some advice as to where to look in apparmor kernel sources to see why issue 1) may happen (if my theory is right) [15:10] ie, not introduced in the recent pr, but sure [15:11] zyga: would needs tyhicks or jjohansen [15:12] jdstrand: master is affected the same way [15:12] jdstrand: I'll check back before we did pivot_root [15:13] ok [15:13] if I add 'file,' to the complain mode profile, it works [15:14] hmm? [15:14] I don't understand what that means, can you explain please [15:14] it means there's probably a file access rule that needs to be added to the profile [15:15] zyga: install hello-world snap in devmode. add to /var/lib/snapd/apparmor/profiles/snap.hello-world.sh 'file,'. apparmor_parser -r ... ; hello-world.sh, then ip netns list [15:15] tyhicks: well, it is complain mode [15:16] jdstrand: the attach-disconnected flag is already present on the profile? [15:16] yes [15:16] profile "snap.devmode.sh" (attach_disconnected,complain) { [15:16] jdstrand: what is the meaning of the 'file,' rule? [15:16] if I remove 'file,' and reload, ir fails. if I add it and load, it works [15:17] zyga: allow all file access [15:17] hmmm [15:17] well, that's a big cannon [15:17] well, sure [15:17] the point is, I just narrowed it down to file rules in complain mode [15:17] well, we did [15:17] ok, I see now [15:18] so what do you think we should do next, [15:18] I'd like to understand why apparmor runs into this error, why is /proc disconnected [15:18] PR snapd#1936 closed: asserts: revert change that made the account-key's name mandatory [15:18] (and is it /proc/self/ns/net or everything in proc/self/ns) [15:18] how do you know that it is /proc? [15:19] tyhicks: nothing else seems to fail, it just affects open() on that file [15:20] zyga: ah, I misunderstood - what's the full path? /proc/self/ns/net? [15:21] tyhicks: https://bugs.launchpad.net/snap-confine/+bug/1624300 [15:21] Bug #1624300: Cannot open /proc/self/ns/net [15:21] /proc/** does not allow it [15:22] zyga: ok, so /proc/ is probably not disconnected but the target of the ns symlink is [15:23] zyga: which is some virtual "ns:[NNN]" reference [15:23] tyhicks: snap-confine doesn't touch that [15:23] tyhicks: and it happens even without the recent work towards namespace sharing [15:23] we aren't getting good logging [15:24] jdstrand: there's not much to log, unfortunately [15:24] jdstrand: I printk'd my way around procfs, it is nothing there [15:24] jdstrand: apparmor is asking the vfs to look up this path and the vfs is saying that it can't [15:24] jdstrand: I'd like to look at apparmor next [15:25] oh weird [15:25] tyhicks: in what sense? The kernel can resolve the path to an inode in nsfs [15:25] /** ix, allows it [15:25] mthaddon: let me know how it goes, I will grab dinner now but I'll be around later [15:26] sforshee: I can't claim to know all the technical details of apparmor's disconnected path issues - we'll have to have jj here for that :/ [15:26] sforshee: it happens in aa_path_name() -> d_namespace_path() [15:26] jjohansen: ^^ [15:26] tvoss: I won't have time to test just yet, but will let you know if I find time next week - thx [15:26] jjohansen: if you are around we could use a hand [15:26] zyga: he pulled an all nighter - he won't be around for a while [15:27] tyhicks: thx, I'll take a look [15:27] I know the feeling [15:27] /{bin,sbin}/ip ix, [15:27] that allows it ^ [15:27] oh? [15:27] wut [15:27] jdstrand: so might it be that [15:27] it is not the namespace [15:27] it is the executable? [15:27] wat? [15:27] that also answers why the reporter had it working-- he was connecting the network-control interface even though it was in complain mode [15:27] ok, that part of the mystery is solved [15:28] but how come it fails for a simple reproducing C program? [15:28] jdstrand: I still don't get it can you tell me how allowing ip fixes it? [15:28] it has to do with the exec transition [15:29] if we don't have the ix rule for the binary that is accessing /proc/self/ns/net, then it fails even in complain mode [15:29] jdstrand: (I just checked that it doesn't fix it if another program, say "bug", compiled from bug.c somewhere in home tries to access it [15:29] I have to step away for a little bit [15:29] tyhicks: o/ [15:29] zyga: sure. add an ix rule for your bug program [15:29] I'll do so now. I bet it will work [15:29] jdstrand: sure but why does that fix it? is it becasue aa cannot find /home somehow? [15:29] * zyga tries [15:30] well, again, that is the mystery and the bug [15:30] I'm just trying to narrow this down [15:30] jdstrand: yes, confirmed! [15:30] ah, ok, I thought you cracked that one :) [15:30] ok [15:30] so I think there are two consequences: [15:30] @{HOME}/Desktop/open-proc-self-ns-net ix, [15:31] jdstrand: No such command! [15:31] 1) we can merge ns sharing without thinking it breaks something [15:31] if I add that ^, it works. if I don't, it doesn't [15:31] 2) we have to look at apparmor deeper to figure out what is broken [15:31] zyga: do one '1' [15:31] err [15:31] do '1' [15:32] :-) [15:32] yep, that will be my pleasure [15:32] we can say in the LP bug on namespace sharing that there is an apparmor bug which requires people to use and connect network-control even in --devmode [15:32] jdstrand: btw, I managed to crash the stock xenial kernel with a few bind mounts yesterday, ran out of memory looping on something [15:32] cool :) [15:32] jdstrand: I'll look at /media sharing next so I'll try to reproduce and report it [15:33] jdstrand: if you give me your blessing on https://github.com/snapcore/snap-confine/pull/146 I'll do a release [15:33] PR snap-confine#146: Use sanity timeouts around blocking operations [15:33] zyga, tyhicks: I'll file a bug on the exec transition issue, with a reduced test case [15:34] jdstrand: thank you, please dupe the bug I reported [15:35] ok, I'm back [15:36] zyga: thank you for looking into this so much. that was a really weird bug. I'm glad that I thought to try on 15.04. cause 15.04 doesn't have devmode, I just added a bunch of rules (like 'file,') and saw it worked. then I was like-- "huh-- I wonder if it is just one of those..." :) [15:37] jdstrand: there's only one thing left worrying me: [15:37] jdstrand: I'm glad we have a workaround but, boy, I'm confused about it [15:37] https://github.com/snapcore/snap-confine/pull/133 [15:37] PR snap-confine#133: Adjust apparmor policy for compatibility with snap-exec [15:37] this doesn't fail for me in qemu (for sanity I just started another run) [15:37] but fails all the time in linode [15:38] and we cannot release without it as snapd now uses snap-run/snap-exec [15:38] tyhicks: hopefully with the reduced test case it will get people to zero in quickly on the problematic code [15:39] zyga: are we sure that the linode environment has an up-to-date Ubuntu xenial kernel and apparmor package? [15:40] zyga: you are saying that on linode, if you unapply 133 that the tests pass with no problem, but when you apply it the fail undeterministically? [15:40] tyhicks: great question, I bet it doesn't [15:40] jdstrand: yes [15:40] I bet some of the nodes are up to date and some aren't [15:40] jdstrand: though test still "pass" because the snapd there is not using snap-run/exec I suspect [15:40] jdstrand: they all share one image [15:40] jdstrand: and my qemu image is 3 days old [15:41] jdstrand: I'll see if I can get gustavo to refresh that next week [15:41] if it works in qemu again I'll merge it with a note describing this [15:41] the unsafe stuff requires apparmor 2.10.95-0ubuntu2.2 and it'll also behave differently from kernels prior to xenial to xenial kernels [15:42] zyga: I can't explain why if what you are saying is correct (every node has the same kernel and apparmor) why it would sometimes pass [15:42] jdstrand: no, it never passes [15:42] jdstrand: it always fails in linode [15:42] ah [15:42] ok [15:42] PR snapcraft#807 closed: Fix parts integration tests [15:42] jdstrand: and always works in qemu [15:42] yes, make sure those images are up to date [15:42] that makes a lot more sense then [15:42] tyhicks: I'll update dependency in the packaging [15:42] something doesn't sound right about the linode env [15:43] zyga: ok, let me double check that version before you do so [15:43] thank you [15:44] zyga: you should depend on 2.10.95-0ubuntu2.2 or newer [15:45] jdstrand: refresh to get GPUness. [15:45] \o/ [15:46] jdstrand: quick question, do we need to add anything to snapd's interfaces or base profile to unblock "ip netns" to work/ [15:46] mvo: did you do the 2nd release today yet? [15:47] jdstrand: You know much about the "personality" syscall? I have a C-compiled regular snap that uses it (and thus gets an mightly slaying) on ARMHF only, and I don't know why, and I'm trying to decide if there's something wrong or if compiling differently would avoid it. [15:47] zyga: I don't think so. network-control already use the ix rule for ip [15:47] I'me going to double check that though [15:47] zyga: no, still waiting for final tests [15:48] mvo: ok, I think we're good :) [15:51] jdstrand: I just confirmed too, woot [15:52] what a fantastic way to finish the week friends :) [15:52] yeah [15:52] what a weird bug [15:52] I'm really glad we figured out why it worked on the mailing list too. that was a very puzzling mystery [15:53] btw, this is still a bug in apparmor, regardless of the disconnected path thing [15:53] I literally couldn't stop thinking about how to get the original test case to work again [15:53] oh yes [15:53] totally [15:53] as EACCES bleeds through apparmor complain mode [15:53] and we can fix it [15:54] I'll have a stab at this over weekend, maybe my printk-based deduction will lead me to it ;-) [15:54] but much of the mystery is gone and we have a workaround that is fairly natural [15:54] indeed, it's surprising it didn't hit us earlier though [15:54] well, not really [15:54] and I still don't understand how the location of the executable influences the problem [15:54] no one is using namespaces within snaps much [15:54] well, lxd [15:54] hmm [15:54] for instance it worked in python3 because that had the workaround you found naturally in the base template [15:54] oh, they have a bare file rule [15:55] and docker I think has the ip command [15:55] but is it *just* for nsfs or is it a general issue? [15:55] seems just nsfs [15:55] I mean, you can open regular files all day long in complain mode [15:55] that we would have definitely heard about [15:55] well, still some mystery as to what is going on there :) [15:55] well, perhaps it is a proc thing [15:56] next week I'll work on media sharing and, perhaps, on live namespace updates [15:56] no, not just proc. it does seem nsfs [15:56] anyhoo [15:56] feels great to move on :) [15:56] a little bit, but jj will be able to see the problem I bet pretty quickly [15:56] zyga: indeed! :) [15:57] zyga: wrt 146, I was surprised about the alarm surronding the flock()s (not saying it is wrong). can you explain your thinking there> [15:57] ? [15:57] Bug #1611444 changed: Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP [15:58] yes, if the child process survives the parent for any reason [15:58] it may lurk around forever holding the flock alive [15:58] this just ensures that, for *whaever* reason, if there are bugs an the child doesn't die [15:59] we don't hang forever but die [15:59] I suspect we could just keep the timers in the child [15:59] with the same effect [16:00] I looked at 146 only a little bit the other day and had that question and thought I'd ask. I'll take a closer look [16:00] I asked on #juju but also asking here as I'm not sure if this is a juju or snapd problem, but I'm getting stuck deploying an instance on xenial. It appears it's stuck in the upgrade of snapd because it hasn't finished running 'snap booted' after a very long time. [16:00] hmm, interesting, the unsafe transitions don't fail as before [16:00] but now I see this: [16:00] Is there some known problem with this? [16:00] cannot change apparmor hat of the support process for mount namespace capture. errmsg: Operation not permitted [16:00] support process for mount namespace capture exited abnormally [16:01] plars: I think mvo is the person you want [16:01] plars: I think it might be [16:01] zyga: there should be a denial for that [16:01] jdstrand: I'm still mid way my run in qemu [16:01] it hasn't reached that point yet [16:02] jdstrand: do I need any unsafe transition markers for the child and the hat? [16:02] plars: I think this is https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1621336 [16:02] Bug #1621336: snapd.boot-ok.service hangs eternally on cloud image upgrades [16:02] have to see the log entry [16:03] jdstrand: spread run in lindoe didn't have it, I hope qemu will [16:04] mvo: that looks like exactly what I'm seeing :( [16:05] plars: looks like xenial-proposed has a fixed cloud-init [16:10] ok, key test running now [16:10] jdstrand: it's good in qemu, I'll ignore it and work on upgrading the linode kernels === JanC is now known as Guest68752 === JanC_ is now known as JanC [16:13] zyga: ok, cool [16:14] ok [16:14] I've merged everything but the timer [16:14] I *need* a break now [16:15] I will work on packaging and downstream testing to see if we're missing anything [16:15] and if it all goes green I'll release it upstream tomorrow [16:15] and start pushing updates to debian, ubuntu, arch, fedora and gentoo [16:15] * zyga EODs (for once, on time) [16:16] sforshee, jdstrand, tyhicks: thank you for your help! [16:16] zyga: enjoy your weekend and go see your family! :) [16:16] :-( [16:16] I won't see them till november [16:16] oh gosh [16:17] I thought they were in the next roomn [16:17] they are in spain and I stayed to sprint almost all october [16:17] * jdstrand hugs zyga [16:17] go video chat with them then :) [16:17] I'll go out and see some friends though [16:17] I need to see real people, faces, not walls of my office [16:25] zyga: have some fun! you deserve it :) [17:24] PR snapcraft#806 closed: Remove snapcraft.storeapi.common [17:46] Bug #1624490 opened: Snap access to /dev/uinput [17:50] hi. I just upgraded snapd to 2.15, and installed a snap and got: cmd_run.go:179: WARNING: cannot create user data directory: cannot create "/home/ubuntu/snap [17:51] after this I tried upgrading ubuntu-core-launcher and then things worked correctly. So is this a known issue? Should snapd depend on a certain version of ubuntu-core-launcher? [17:57] PR snapcraft#804 closed: Adjust login to conform to UX specification === drizztbsd is now known as timothy [18:08] tyhicks (cc zyga and jjohansen): https://bugs.launchpad.net/apparmor/+bug/1624497 [18:08] Bug #1624497: complain mode blocks access to nsfs (/proc/self/ns/*) without exec rule [18:18] jdstrand: thanks for the nice writeup [18:19] jdstrand: jjohansen has some urgent work on his plate today so this is something that he'll probably have to wait until next week to triage [18:27] tyhicks: thanks. totally understood. I tried to make it clear in the bug that there is a workaround available. also, the new snap-confine that fixes bug #1611444 (ie, the ns sharing pr) won't land in xenial til next week or so anyway [18:27] Bug #1611444: Cannot share a namespaces created with 'ip netns' between apps in a devmode SNAP [20:17] Hi. What plug do I give a snap for sound? I have tried "audio" and "sound". Would it be "alsa"? [20:19] That doesn't seem to work either. Is there a list of interfaces somewhere? [20:20] you can get a list of available interfaces that are on your system by doing snap interfaces [20:21] lpotter: thanks! I'll try "pulseaudio" :) [20:35] Bug #1624533 opened: app can't reach USB smart-card device [20:35] PR snapd#1938 opened: asserts: check that validation assertions are signed by the publisher of the gating snap [20:38] ogra_: you wouldn't happen to have a recent image for rpi2? O:-) [20:43] lool, gimme a bit ... [20:43] mvo wanted to release a set today, not sure why he didnt [20:45] * mcphail has just packaged tombraider as a snap! If only it could be released to the store... [20:46] ogra_: last minute snapd bug preventing it [20:46] ogra_: fix is submitted [20:46] ogra_: IIUC, interfaces are not well connected on first boot [20:46] ah [20:46] so bad user experinece [20:46] I wouldn't care about it though [20:46] so ont image related [20:47] *not [20:48] i'm pushing to http://people.canonical.com/~ogra/snappy/all-snaps/ ... give it 5min [20:49] awesome [20:51] done ... enjoy [20:51] * ogra_ is afk again ... [20:54] mwhudson: FYI https://drive.google.com/a/canonical.com/file/d/0B5RxJIi-SaMdTVB3emo4SjNGUHM/view?usp=sharing when waiting for too long before entering email :-) [20:55] ogra_: thanks a ton man! [20:55] ogra_: you rock :) [21:24] is there a snappy ubuntu core image for the raspberry pi 3? I know the image for the rpi2 will work, but I'd like to get one that will get the wifi and bluetooth working ootb [21:31] actually, can anbody confirm that the rpi2 image works on the 3? [22:08] mhall119: yes, there's a pi3 image [22:09] mhall119: no the pi2 image shouldnt work there [22:09] mhall119: http://people.canonical.com/~ogra/snappy/all-snaps/ is best you can get until monday where beta images will be rerolled [22:11] thanks lool