[06:03] morning guys [06:12] good morning [06:12] it's snowing!!! [06:12] man, I haven't seen snow in years [06:14] zyga-ubuntu: hey [06:15] yeah, splendid weather [06:19] mborzecki: just like I remember it my whole childhood :) [06:19] mborzecki: it will be white by the time my kids go to school today [06:21] zyga-ubuntu: https://forum.snapcraft.io/t/core-snap-fails-auto-review-in-the-store-since-friday/2897 [06:21] mborzecki: yep, I know :/ [06:21] though, let me read about new developments [06:22] mborzecki: the store review code diff was one character [06:28] mborzecki, ogra_: commented [06:28] man it's whiter every second [06:28] the roof is now fully white [06:28] and my wife's car too (poor her!) [06:28] I only wish it was colder so the snow would not melt away so quickly [06:29] so, today, I have a few branches to iterate on [06:29] but I'd like to fix the other LXD issue once and for all [06:29] to have that boulder off my back [06:30] well, it's raining right here, there's still some spots of snow that apparently fell over the night, but it's ugly as hell (i.e. typical november) [06:35] mvo: hello [06:36] mvo: SNOW! :D [06:36] mvo: some unhappy store news [06:36] mvo: new review tools not deployed yet (it seems), jdstrand sent us email over weekend [06:36] mvo: not sure if there are things he said we could do [06:40] hey zyga-ubuntu ! good morning [06:40] zyga-ubuntu: looking at my mail now. any other unhappy news? [06:40] zyga-ubuntu: snow is nice [06:42] mvo: no, nothing other than that [06:42] mvo: but I haven't checked much yet [06:42] mvo: waking kids up now [06:42] mvo: morning [06:43] hey mborzecki - good morning [06:43] (I don't like when they don't go to school at 8AM) [07:02] zyga-ubuntu: FYI, https://bugs.launchpad.net/snapd/+bug/1733211 [07:02] Bug #1733211: lxd snap fails to install w/apparmor "permission denied" error [07:03] zyga-ubuntu: going to bed now, but I expect we'll have more people run into this one as we've pushed a deprecation warning to all our PPA users on Friday telling them to move to the snap or backports [07:03] stgraber: thank you for the note, reading [07:04] stgraber: that looks like a new bug right? [07:04] stgraber: or a regression technically [07:05] but this is weird, we have the rule to read that: [07:05] @{PROC}/@{pid}/cmdline r, [07:06] stgraber: if this is about privileged containers that don't stack apparmor then this is ECANNOTFIX [07:06] stgraber: I sincerely hope it's not, let me read the bug carefully [07:06] zyga-ubuntu: no, I hit this outside of a container [07:06] zyga-ubuntu: usually on machines that are already running other snaps [07:06] stgraber: aha! [07:07] stgraber: thank you, I'll investigate [07:07] since you said that rebooting helped it smells like apparmor cache bug we ran into before [07:07] zyga-ubuntu: the fact that rebooting the system fixes the issue is somewhat puzzling [07:07] I'll try to reproduce now [07:07] oh, yeah, could be the apparmor cache [07:08] not sure why I didn't think of that, I could have dumped the content of the cache and profile before rebooting the system (though to be fair, they were broken production systems, so didn't have much time for debugging :)) [07:08] which makes me wonder, I have that backup host that may actually have the same issue and would let me check this quickly without risk [07:09] and nope, no such luck, that system installs the lxd snap without any problem... [07:09] * stgraber tries a few other random systems in his basement real quick [07:09] stgraber: no worries, enjoy your evening and let us investigate [07:19] good morning [07:19] there is a recent increase of this problem on errors.u.c https://errors.ubuntu.com/problem/cf07c2fa3f39a98f0b52aaf20ff82faab7969129 [07:19] jibel: good morning [07:19] it started around Nov. 14th [07:19] jibel: thank you for the notice [07:21] that sounds surprisingly similar to that issue we were just discussing, also permission denied, also for a path that should be allowed and also in the configure hook [07:22] jibel: oh, interesting, this is a yet another issue [07:22] stgraber: yes, we ran into this one a while ago ~1 month [07:22] stgraber: it was a bug in apparmor then [07:22] stgraber: and we included a workaround for that [07:22] something we need to look into, apparently [07:22] zyga-ubuntu, yeah other crash are with specific snaps, but this one is with the core snap [07:22] on 17.10 and 18.04 [07:23] mvo: so we have more "news" apparently [07:23] 2.29.3 is very worrying, it should have been fixed before [07:28] zyga-ubuntu: the configure script error is probably not 2.29.3 itself, just the fact that we have an update, no? [07:28] zyga-ubuntu: I mean, the mass update of core triggered this I presume? triggered the apparmor bug [07:29] zyga-ubuntu: do you remember the details for the previous workaround? that was part of 2.28.X, right? [07:31] zyga-ubuntu: I don't see a mail from jamie about the script in my inbox, maybe it just went to you? [07:33] mvo: let me check [07:34] PR snapd#4249 opened: release: snapd 2.29.4 [07:34] mvo: the previous bug we did something (AFAIR you did after talking to tyhicks) that casuses cache to be purged on core updates on core [07:35] mvo: ok, I should eat something before jumping into the fray [07:36] mvo: I see that we have three issues: LXD + remove bug (close to being fixed, oldest), bug reported by stgraber above (mysteriously fixes itself on reboot) and the bug form errors.u.c from 14th Nov [07:36] mvo: the other two may be related but not sure if they are yet [07:36] mvo: let's split work in 20 minutes, I need to eat something [07:37] zyga-ubuntu: sure, eat [07:37] zyga-ubuntu: ssee you then [08:07] zyga-ubuntu mvo anything I can help with? [08:12] mvo: ok properly back now [08:12] * zyga-ubuntu made _warm_ breakfast for a change [08:17] mborzecki: definitely! [08:19] zyga-ubuntu, mborzecki I created http://pad.ubuntu.com/Wkp2bPRAnI so that we have a scratchpad, please add all you know there for now and once we have a bit more we can make that three forum topics [08:19] zyga-ubuntu: I'm especially interessted in "lxd-+remove bug", is that the one you were chasing for some time ? [08:20] zyga-ubuntu: i.e. the one where the rshared is not set correctly on /snap ? [08:20] mvo: yes [08:20] mvo: yes [08:20] zyga-ubuntu: and if it is this one, why is it on the list? its not something new in 2.29.4, or is it? [08:21] mvo: can you paste the bug report from errors.ubuntu.com in the pad? I've only requested access there [08:21] mvo: no, nothin new [08:22] zyga-ubuntu: ok, if its nothing new, lets move it down the priorites just for now to figure out if 2 and 3 are regressions and what we can do about them (?) [08:22] mvo: +1 [08:23] done [08:23] mborzecki: I added some more data, unfortunately its not much, I guess we need a reproducer, might be as simple as going from an old core to the new core on 17.10, but then I do that all the time and have not seen it :/ [08:23] I added some ideas on what might happen [08:24] zyga-ubuntu: ta [08:30] mvo: is issue 2 (renumbered) affecting those as LXD guest or host? [08:30] mvo: (those releases) [08:32] zyga-ubuntu: yeah, I wonder the same, I just tried it on a normal system in 17.10 and did not see the error [08:33] mvo: I don't see 2.29.3 in the archive [08:33] on 17.10 [08:33] it's 2.28.5 iirc [08:33] yes, that's what I see [08:33] zyga-ubuntu: not even in -proposed? [08:33] mvo: ah, I didn't look there (not a default setup) [08:34] do you think people affected by 2nd bug used proposed? [08:34] zyga-ubuntu: maybe but its strange [08:35] zyga-ubuntu: because I can't reproduce it on a regular system [08:35] zyga-ubuntu: with 2.29.3 from the debs on 17.10 [08:35] zyga-ubuntu: but maybe I need to try harder [08:36] mvo: same result with stable debs + core [08:36] thanks jibel btw for raising this issue! [08:39] zyga-ubuntu: worth exploring if its happening inside lxd only, but then - the amount of users(17.10) & users(lxd) & users(snapd) are producing a relative high number of errror reports [08:47] mvo, yw [08:49] o/ [08:54] PR snapd#4250 opened: packaging: fix typo [09:18] brb [09:38] re [09:39] mvo: do we retry core configure failures/ [09:41] zyga-ubuntu: no, I we just ignore failures on them [09:41] mvo: that's curious [09:41] mvo: look at those: [09:41] https://errors.ubuntu.com/oops/c8ec99fe-cdd2-11e7-919f-fa163ef911dc [09:41] https://errors.ubuntu.com/oops/7fecbe46-cdd2-11e7-9435-fa163ebeb28a [09:42] mvo: that's one system [09:42] right? [09:43] zyga-ubuntu: yeah, thats strange indeed [09:43] zyga-ubuntu: it looks almost like the install fails and it is retrying [09:43] mvo: could it be a fresh install? [09:43] mvo: after restarting into new snapd [09:44] mvo: maybe another LXD cluster with privileged containers for that cloud certification? [09:44] zyga-ubuntu: it has DidRexec set to 0 [09:44] mvo: ah, I see [09:44] puzzlin [09:44] zyga-ubuntu: yeah [09:44] mvo: so ... proposed testing? [09:44] mvo: maybe this is some adt machine? [09:44] mvo: running off proposed? [09:45] zyga-ubuntu: that sounds like a good theory [09:45] zyga-ubuntu: and its a 16.04 one [09:47] did you have any luck in reproducing this? [09:48] PR snapd#4251 opened: timeutil: include test input in error message in TestParseSchedule() === mup_ is now known as mup [10:16] guys is there any nightly for snappy? [10:16] for ubuntu [10:17] Shibe: we have the edge channel which is just like that [10:17] zyga-ubuntu: how do I enable it on ubuntu? [10:17] Shibe: install snapd, install core snap (if you don't have one already) and snap refresh core --edge [10:31] mvo: no idea how to reproduce this, I tried in various containers with and without proposed [10:34] zyga-ubuntu: yeah, I am also clueless so far, I think we need to move it to the forum and see if we can find someone who actually hits this or can give us other clues [10:35] hmm [10:35] http://pad.ubuntu.com/ep/pad/reconnect - forbidden [10:36] how i "love" etherpad [10:39] zyga-ubuntu: http://pad.ubuntu.com/Wkp2bPRAnI ? [10:39] zyga-ubuntu: and yeah, its very annoying [10:39] mvo: connected back, thanks [10:40] yw [10:40] pull requests are kind of red [10:41] + snap refresh [10:41] error: cannot refresh []: cannot query the store for updates: got unexpected [10:41] HTTP status code 500 via POST to [10:41] "http://localhost:11028/api/v1/snaps/metadata" [10:41] zyga-ubuntu: yeah, I have a look in a tiny bit. but this sounds much like a problem for our friends in #snapstore :) [10:41] that's interesting, why are we refreshing a list of no snaps? [10:43] man [10:43] I *HATE* travis moronic log output behavior [10:44] why does clicking on any line closes the fold? [10:44] Nov 20 09:09:04 li157-129 snapd[31385]: 2017/11/20 09:09:04.824015 logger.go:76: DEBUG: < "HTTP/1.1 500 Internal Server Error\r\nContent-Length: 111\r\nContent-Type: text/plain; charset=utf-8\r\nDate: Mon, 20 Nov 2017 09:09:04 GMT\r\nX-Content-Type-Options: nosniff\r\n\r\ninternal error collecting snaps: open /tmp/read-file456639681/unpack/meta/snap.yaml: no such file or directory\n" [10:44] mvo: is that us or the store is sending that? [10:45] ah, that's fakestore [10:45] zyga-ubuntu: that's our fakestore [10:45] zyga-ubuntu: uhhh, which PR is thisß [10:45] ? [10:46] PR snapd#4252 opened: store: fix download caching and add integration test [10:46] this was 4242 but I just hit restart [10:46] it looks like timing / race in fakestore [10:46] zyga-ubuntu: ok, no worries, I try to reproduce [10:47] PR snapd#4251 closed: timeutil: include test input in error message in TestParseSchedule() [10:47] zyga-ubuntu: hm, not all is terrible, 4251 was green for example. there is a silver lining :) [10:48] PR snapd#4250 closed: packaging: fix typo [10:50] PR snapd#4247 closed: interfaces: allow /bin/chown and fchownat to root:root [10:50] if someone could do a second review on 4242 that would be great, looks like an easy win [10:52] mvo: that 4247 one had a question [10:53] zyga-ubuntu: yes, I'm 99% sure the answer is "its fine", let me write that in the PR [10:53] mvo: done [10:53] zyga-ubuntu: ha! even better, thank you :) [10:53] mvo: I mean 4242 :) [10:54] PR snapd#4235 closed: cmd: pretend we're running on Ubuntu in TestExecInCoreSnapUnsetsDidRe… [10:54] zyga-ubuntu: let me check [10:54] zyga-ubuntu: aha, sorry, async communication is hard ;) [10:57] zyga-ubuntu: comment and I also looked at the source and my confidence is at 99.9 now [10:58] mvo: thank you, I shared that but wanted to check just in case [10:58] mvo: the BPF code just ingores those arguments [10:58] zyga-ubuntu: yeah [11:05] zyga-ubuntu: some PRs in flight, once there tests are green we can merge and will be below 25 again :) [11:09] PR snapd#4253 opened: task tunner: extended task runner test [11:11] mvo: yeah I'll merge them as soon as we can, I restarted a few random failures today [11:11] though way too more for my liking [11:12] zyga-ubuntu: yeah, its too fragile currently [11:27] zyga-ubuntu: the did-rexec in the error reports is a red-herring :/ the problem is that we unset SNAP_DID_REEXEC to not confuse classic confinement snaps. and that is the env that the err tracker picks up [11:28] mvo: are you saying that we should set a new flag SNAPD_WE_ACTUALLY_DID=reexec [11:31] pstolowski: hey, do you have a moment for 2nd look on https://github.com/snapcore/snapd/pull/4224 [11:31] PR #4224: cmd/snap-update-ns: teach update logic to handle synthetic changes [11:31] pstolowski: I changed the approach slightly after your comment [11:31] zyga-ubuntu: maybe, I look into this right now [11:32] mvo: I kind of feel that the set of constraints is lost [11:32] mvo: we have some super ancient versions of snapd that put them [11:32] mvo: but they have now escaped me [11:32] mvo: maybe you could write this down in some reegex.go extented comment? [11:32] ha, and I said "regeexec" instead of reexec :) [11:33] PR snapd#4242 closed: asserts/sysdb: panic early if pointed to staging but staging keys are not compiled-in [11:39] zyga-ubuntu, will do in a few moments [11:46] Hello hello [11:47] niemeyer: hello [11:52] niemeyer: hi there [11:52] o/ [11:54] hey niemeyer ! welcome back [11:54] is there a way to have check do a pretty print of unequal values like spew does? [11:55] mvo: Thanks! [12:04] mborzecki: Not a pretty print, yet at least [12:15] jdstrand: ok got ubports-installer to work pretty nice in devmode now :) now the confined part, how do i handle the usb-raw part? post-install connect thing? [12:15] PR snapd#4254 opened: cmd, errtracker: get rid of SNAP_DID_REEXEC environment [12:17] mvo: ^ you have a conflict there [12:46] PR snapd#4245 closed: interfaces/screen-inhibit-control: handle ScreenSaver/Screensaver in DBus API [13:00] fg [13:00] full grin ? [13:00] :P [13:04] PR snapd#4255 opened: interfaces/screen-inhibit-control: fix case in screen inhibit control [13:11] yikes, my /var/lib/snapd/hostfs is empty!? [13:11] no wonder I'm getting strange failures [13:11] you mean from inside a "snap run --shell" ? [13:12] no, from outside as well [13:12] kalikiana: just passing by, but note it is only populated inside the snap [13:12] it is supposed to be empty on the host [13:12] kalikiana, thats normal, [13:12] only snap-confine mounts sumething there at runtime [13:12] hmmm so it's "only" an issue in the snap [13:12] I got a file not found error [13:14] I can't "--shell" since this is a classic snap... (snapcraft) [13:14] I'm pretty sure the relevant code hasn't changed [13:15] Is it me or hangouts just broke? [13:15] niemeyer: you [13:15] "There is a problem connecting to this video call. Try again in a few minutes." [13:15] niemeyer: you froze after "guys" [13:16] oh, actually --shell does something, but it looks like my real shell [13:16] niemeyer: some more investigation (quite raw) http://pad.ubuntu.com/Wkp2bPRAnI [13:17] jdstrand: ogra_ : so hostfs is empty within `snapcraft run --shell snapcraft` as well [13:17] feels like snapd changed something here [13:17] kalikiana: is snapcraft classic? [13:18] jdstrand: yes [13:18] iirc, is is only populated for non-classic [13:19] jdstrand: correct [13:19] hostfs on classic is / [13:19] so we don't do anything [13:19] perhaps we could use a trick [13:19] so in classic distros hostfs is a symlink to / [13:20] and for confined apps it is a directory and we pivot root there as we do [13:20] jdstrand, zyga-ubuntu: the code in snapcraft hasn't changed, though... [13:22] hrm, lemme debug this a bit [13:24] it's weird actually the fact that hosts even shows up as a path here. the code isn't doing that.. [13:33] hrmpf, now it works again, no change... [13:42] https://forum.snapcraft.io/t/corebird-1-7-3-release-with-added-emoji/2906 [13:42] \o/ [13:51] PR snapd#4256 opened: snap-confine: add workaround for snap-confine on 4.13/upstream [13:52] mvo: +1 [13:52] jdstrand: ^^ [13:54] zyga-ubuntu: I'm running the test in my manual created sid spread image now, will take a little bit [13:56] mvo: mvo thank you [13:56] mvo: my fix didn't work, I probably broke something just a moment ago as it was in a better shape last fix, digging [13:56] zyga-ubuntu: thank *you*, you came up with the right apparmor line [13:57] mvo: no, that was jdstrand :) [13:57] zyga-ubuntu: oh? the fix for the lxd remove? or a different fix that does not work? [13:57] mvo: for the remove [13:57] zyga-ubuntu: aha, ok. still, I think you put it in the forum, I'm sure there is plenty of reason to thank you :) [13:57] :D [13:57] * jdstrand reviewed 4256 [13:57] * zyga-ubuntu tries -shell-before to see what's wrong [14:19] zyga-ubuntu: testing on debian-sid shows some other issues, I will expand this pr a little bit, just fyi [14:21] mvo: ack [14:26] roadmr: hi! hope you had a nice weekend. not sure you read email yet, but can you update the review tools for r945? [14:27] popey: hey, do you have a moment to talk about snapcrafters? [14:33] jdstrand: hey! [14:33] jdstrand: I hadn't, but I'll get the ball rolling [14:34] sergiusens: just to double-check, we have our weekly tomorrow/ Tuesday again? [14:34] roadmr: thanks! [14:34] jdstrand: I was aware of the situation on Friday but there was not much we could do then :( [14:34] sergiusens: I see it in the calendar, just at the back of my head someone said it would be Monday again [14:34] roadmr: yeah. r944 and r945 came in at an unfortunate time [14:36] need to finish for today, enjoy the rest of the day guys, cu [14:39] mvo: hey, I _think_ found the issue now (removed one line of apparmor by vim mistake probably) [14:39] mvo: running again to see [14:39] mvo: how are your stuff? [14:39] (debian) [14:40] * kalikiana going out for lunch shortly, will be back in ~hour or so [14:41] zyga-ubuntu: debian is running, a bit slow but running. it was net-tools missing that broke it earlier, I added it to the PR [14:41] zyga-ubuntu: looking forward for the removal fix PR :) [14:42] zyga-ubuntu: hrm, seems I'm still seeing the weird hostfs issue, it only went away since I tried straight from git, not from the snap... it seems the lxd snap/ lxc command is receiving a path starting with /var/lib/snapd/hostfs/ and that's what I see in the error [14:42] mvo: once this run goes green I'll drop the unrelated changes and test again [14:42] mvo: and publush [14:42] *publish [14:42] kalikiana: can you please refresh my memory? [14:52] zyga-ubuntu: what I just described above [14:53] zyga-ubuntu: error: open /var/lib/snapd/hostfs/home/cris/snap/lxd/common/snapcrafts3h6q44p/core_3440.assert: no such file or directory from subprocess.CalledProcessError: Command '['lxc', 'file', 'push', '/home/cris/snap/lxd/common/snapcrafts3h6q44p/core_3440.assert', 'helga:snapcraft-oman/run/core_3440.assert']' returned non-zero exit status 1. [14:53] when the file exists in the folder [14:53] on the host [14:54] zyga-ubuntu: Wondering what could cause this... the path from snapcraft is the real one [14:59] PR snapd#4257 opened: interfaces/opengl: also allow read on 'revision' in /sys/devices/pci [15:03] PR snapd#4249 closed: release: snapd 2.29.4 [15:06] re [15:07] kalikiana: sorry, no (immediate) idea, trying to wrap up one issue, then will switch context to this if you need help [15:08] zyga-ubuntu: aye, will have my lunch break first, no rush [15:08] well, late lunch haha [15:09] jdstrand: silly question, I get an apparmor denial in snap-confine/debina-sid http://paste.ubuntu.com/26005784/ but the rule looks valid to me, how can I check how @multiarch is expanded? [15:10] * jdstrand looks [15:10] jdstrand: I have the VM in front of me if you need more info, its slightly puzzling [15:12] mvo: the rule looks right [15:12] mvo: is this a reexec situation? [15:12] (or lack thereof) [15:13] jdstrand: no, re-exec is disabled on debian-sid right now [15:13] jdstrand: its also happening when I directly run snap-confine in the shell [15:13] mvo: so /etc/apparmor.d/usr.lib.snapd.snap-confine.real has the rule you pasted? [15:14] * jdstrand isn't sure .real is used in Debian... [15:14] jdstrand: *cough* the file is 0 bytes long [15:14] ah [15:14] jdstrand: so something is fishy here [15:14] well, that might have a consequence or two [15:14] sounds like the cache file is being used [15:15] jdstrand: indeed, looks like the debian packaging is broken :/ the deb is also zero bytes [15:15] yikes [15:15] deb is zero byts?! [15:16] hmmm [15:16] something fishy here: [15:16] umount: /snap/core/3495 unmounted [15:16] + unsquashfs '/var/lib/snapd/snaps/core_3495.snap [15:16] /var/lib/snapd/snaps/core_3495.snap' [15:16] Could not open /var/lib/snapd/snaps/core_3495.snap [15:16] /var/lib/snapd/snaps/core_3495.snap, because No such file or directory [15:17] there's a newline in that filename [15:17] zyga-ubuntu: the snap-confine.real in the deb is zero byte [15:21] mvo: I dropped junk from my branch, trying again [15:30] mvo: it passed :D [15:30] mvo: man, I just had to throw the garbage out [15:30] zyga-ubuntu: yay [15:31] mvo: I'll run a bit more of main to see one more test that broke on me earlier [15:31] mvo: any luck with that @multiarch thing? [15:33] zyga-ubuntu: zero size snap-confine.real apparmor in the package in debina [15:33] zyga-ubuntu: thats the culprit, I still try to understand why [15:33] mvo: kk [15:35] zyga-ubuntu: did you find the source of that permission denied thing? [15:35] stgraber: yes, it was regular permission in the end [15:35] stgraber: that issue is fixed now snap-confine is now setgid root [15:35] stgraber: thank you :) [15:35] stgraber: also chasing (and almost fixed) the bug where snaps don't get removed in LXD [15:35] zyga-ubuntu: oh, any idea why rebooting would solve it? [15:36] stgraber: ah, no, not that one [15:36] stgraber: sorry, no idea what caused that [15:36] stgraber: we tried to reproduce it in the morning without much luck, there's an etherpad with some details: http://pad.ubuntu.com/Wkp2bPRAnI [15:37]