=== kalikiana_ is now known as kalikiana === zyga_ is now known as zyga === cmagina_ is now known as cmagina === tai271828_ is now known as tai271828 === TinoGuest_ is now known as TinoGuest === victorbjelkholm_ is now known as victorbjelkholm === pstolowski|afk_ is now known as pstolowski|afk === mwhudson is now known as Guest96057 === Guest96057 is now known as mwhudson === erio|away is now known as erio === prime is now known as Guest2033 [04:58] good morning [05:14] morning [05:20] hey [05:21] zyga: anything on fire today? [05:25] nothing apart from my joints [05:25] everything hurts as if I had a flu [05:25] there's one bug about apparmor but it looks like partial removal of snapd [05:25] your joints hmm? :) [05:27] zyga: #177351 ? [05:27] Bug #177351: evince crashed with SIGSEGV in CairoOutputDev::setCairo() [05:27] damn, not that one, this one #1773515? [05:28] Bug #1773515: apparmor fails after removal of snapd [05:43] mborzecki: yeah, that one [05:44] drat [05:44] I made an appointment today to replace my phone battery at a service centre [05:44] sigh [05:44] what a day [05:44] well, ok, it's at 10:00 AM, close by, I can just go and work from some place nearby [05:45] I will just login home and work this way [05:45] I updated to bionic yesterday to have more recent autopkgtest [05:45] I find the whole image construction toolkit super fragile [05:45] it times out, doesn't say what's going on, etc [06:13] I’m going to the service center now [06:13] I will install myself there and just work [06:13] ETA 30 minutes [06:24] I’m close now but man, committing at this time is not fun [06:46] re :) [06:47] I'm at a starbucks exactly in front of the service center, time to work [06:47] mborzecki: mvo will be around later today [06:47] ok [06:49] mborzecki: can you please look at #5231 [06:49] PR #5231: interfaces/joystick: force use of the device cgroup with joystick interface [06:49] we need to cherry pick it into the release branch as well [06:50] zyga: to recap, this is to have the device cgroup created always, using a device (/dev/full) that we're sure is present at all times, right? [06:50] yes, almost, [06:51] we don't make device cgroups unless there's a device tagged with an udev tag that matches the app we're running [06:51] btw. reading systemd-environment-generators manpages, it's a nice feature of systemd [06:51] and if you give broad apparmor permissions (like all of input devices here) [06:51] and not tag any dveices [06:51] yup [06:51] then there's no cgroup and you can do anything you want [06:52] mborzecki: I know, I talked to zbyszek about it about a year ago when I was trying to figure out how to inject /snap/bin into PATH in a nicer way than we did before [06:52] he suggested environment generators but they were not merged back then :) [06:52] zyga: any idea how far back in systemd versions this feature is available? [06:52] it's very recent [06:53] this is why I made a remark about it on the PR [06:53] I don't know how we plan to use it across the released versions of systemd people are using [07:01] zyga: why not just always create the device cgroup? i see in the code that we're adding a bunch of static devices iff there are any (other) devices tagged for particular snap [07:01] that's my feedback as well [07:01] oh and i see nvidia ... [07:01] nvidia is a special snowflake because they cannot use GPL symbols and udev [07:04] zyga: there's a chance that device group by default may break some snaps === pstolowski|afk is now known as pstolowski [07:04] morning [07:05] or at least that's my guess why jdstrand didn't go down this path [07:05] pstolowski: hey [07:05] hey pawel [07:05] mborzecki: note that nvidia works with udev tagged devices today [07:05] we have special code for taht [07:06] in my review I gave a +1 but indicated that we should probably not make mistakes again by simply always doing a cgroup anyway [07:07] PR snapd#5227 closed: interfaces/hardware-observe: allow access to /etc/sensors* for libsensors [07:07] PR snapd#5228 closed: interfaces/hardware-observe: allow access to /etc/sensors* for libsensors (2.33) [07:10] pstolowski: hi, should we chat again? [07:12] zyga: tagging short udev related PRs with "simple" is a bit misleading I think [07:12] because it's udev? [07:14] because reviewing quickly to means I should not need to be on alert for subtle issues [07:14] by definition a PR that solves a subtle issue cannot be quick to review [07:15] zyga: maybe i'll play a bit with that change to snap-confine later today [07:15] zyga: same for the new interface PR [07:15] but the issue is very clearly stated there [07:15] pedronis: the ones about sensors? [07:16] any of them [07:16] again I don't think they can be simple [07:16] pedronis: probably, but give me a few moments to collect thoughts [07:16] zyga: you invented the label, but if I need to read the description carefully [07:16] it is also not simple [07:16] zyga: simple is something that can be reviewed before coffee [07:17] otherwise is just a trap [07:17] of mindless +1 [07:17] mmm, [07:17] and we should remove the label [07:17] I'll apply more restraint [07:17] there's certainly the case that everyone is familiar with a specific part of the code more closely [07:17] and indeed then the label can be deceptive [07:17] ok, then again it cannot be simple [07:18] (because everybody sees the label) [07:21] pedronis: some quick feedback on #5221 [07:21] PR #5221: [RFC] snap: parse connect instructions in gadget.yaml [07:22] pedronis: WDYT of adding a "mounted-from" key to snaps' json, and have that be the EvalSymlinks of the MountFile of all snaps? [07:24] Chipaca: seems ok to me [07:24] k, i'll push a pr for that soonish [07:27] zyga: I'm sort of tempted to kill the label completely tbh, the fact we don't agree how it should be used it's probaly a sign is a nuisance [07:28] pedronis: I'd like to keep the label because it's an encouragement to look at PRs [07:28] we should just agree how to use it [07:30] pedronis: ok, i'm ready when you're [07:30] PR snapd#5232 opened: interfaces/builltin/tpm: Allow access to the kernel resource manager [07:30] * zyga notices a ~2K lines changed PR and wonders if 30 minutes is enough time to even look [07:31] which one? [07:32] pstolowski: joining [07:34] spineau: hey, do you have any links about what the TPM resource manager is for? [07:35] zyga: I'm collecting them right now [07:35] thanks [07:35] please add them to the PR [07:35] PR snapd#5231 closed: interfaces/joystick: force use of the device cgroup with joystick interface [07:51] popey_: https://forum.snapcraft.io/t/is-the-author-of-a-package-verified/5671 [07:51] something that well deserves a good answer [07:54] ok [07:54] Son_Goku: hey [07:55] * zyga needs to go to the service center [07:55] Son_Goku: please review your PRs, there's feedback and patches but we cannot push to your branch [07:55] * Son_Goku waves at zyga [07:55] I pushed the patch requested [07:55] * zyga goes to replace his phone battery now [07:55] ttyl [07:55] thanks! I'll check soon [07:55] Son_Goku: hey, left you a link to a patch that fixes media-sharing in opensuse pr [07:55] mborzecki: done already [07:55] now we wait forever for spread :D [07:56] hehe :) [07:56] forever as in ~30 minutes [08:10] PR snapd#5233 opened: client, daemon: add a "mounted-from" entry to local snaps' JSON [08:24] re [08:24] my phone should have a brand new battery by noon [08:25] mvo: hey, welcome back [08:26] zyga: hello [08:27] mvo: you're not under water are you? [08:27] Chipaca: no, all good [08:28] mvo: throw one bubble for yes, two bubbles for no [08:29] * Son_Goku throws three bubbles at Chipaca [08:29] * Chipaca starts selling a dragon-based arcade game remake reusing Son_Goku's bubbles [08:29] :D [08:30] ahh Spyro :) [08:30] bubble bobble, you uncultured yob [08:31] * Chipaca removes his monacle [08:31] that thing's evil [08:33] * zyga wonders what is going on [08:34] zyga: an area in the west of germany (but a few km from mvo) is under water [08:34] spineau: feedback on your PR [08:35] zyga: also bubble bobble is a game with dragons that throw bubbles and that makes no sense [08:35] Chipaca: because dragons make sense ;-) ? [08:36] zyga: about as much sense as unicorns [08:36] :D [08:36] * zyga sings "narwhals" [08:36] zyga: thx, I was wondering which one caused the error [08:36] Chipaca: yeah, some heavy rain here in the area but where I am its mostly fine [08:36] * zyga just moved further away from the window becase OMG SO HOT today [08:37] it's weird not to have one's phone around [08:37] pedronis: ok, i think i nailed the conflict check.. at least the tests seem happy [08:37] it's the most important thing to bring in many ways [08:37] 29C in the shade here [08:37] zyga: do you want one last look at #5219 ? [08:37] PR #5219: packaging/opensuse: Refactor packaging to support all openSUSE targets [08:38] yep, looking [08:38] fun fact, the man page of mount uses both "file system" and "filesystem" at the same time [08:39] that's... annoying [08:39] well to be fair, the former only a few times [08:39] mborzecki: one question on this https://github.com/snapcore/snapd/pull/5219#discussion_r191685550 [08:39] PR #5219: packaging/opensuse: Refactor packaging to support all openSUSE targets [08:41] mborzecki, Pharaoh_Atem do you know what's the tumbleweed version in the RPM macro? [08:41] I'm AFK from my home machine to check [08:41] Son_Goku: there's some fine print note about tumbleweed version right here: https://en.opensuse.org/openSUSE:Packaging_for_Leap#RPM_Distro_Version_Macros [08:41] I mean, it's 1555 [08:41] zyga: ^^ [08:41] but what will happen later [08:41] I know, I read that :) [08:41] sorry, my question on the PR was worder better than my question here [08:42] you can't not rely on it in some fashion [08:42] right but I'm trying to guess if the next leap will have all the features we target for tumbleweed today [08:42] if so that's even better [08:42] but basically if you use it as a checkpoint (that is, don't do 0%{?suse_version} == 1550), you're fine [08:42] zyga, well, Leap is branched from Factory [08:43] err, SLE is branched from Factory [08:43] and Leap is cloned from SLE [08:43] and tumble? [08:43] and Tumbleweed is regular snapshots of Factory [08:43] we should be good with >= check then [08:44] yes [08:44] SUSE has made version checking harder than it should be :/ [08:44] +1 then, let's mere it [08:44] aaand merge [08:44] (d) [08:45] ugh [08:45] I hate touching openSUSE packaging sometimes [08:45] mvo: any chance you could approve the two new revisions of the test-snapd-eds snap? (this is just changing interface names based on the review) [08:45] zyga: do you plan to sync the spec in obs? [08:45] PR snapd#5219 closed: packaging/opensuse: Refactor packaging to support all openSUSE targets [08:45] mborzecki: yes but only for the next release [08:45] zyga: ack [08:45] jamesh: sure, let me do that right away [08:45] mvo: thank you! [08:45] zyga, don't forget to merge my changes entry into the OBS changes file [08:45] .33 is branched, once we are good for release we should do it in OBS as well [08:46] Son_Goku: ack [08:46] PR snapd#5234 opened: [RFC] snap: add `snap list --format=...` option [08:46] I guess since I'm up already and looking at this [08:47] might as well add the %bcond to create the /snap symlink in the spec for amazon linux in the fedora spec [08:47] May 30 08:17:56 arch snapd[10146]: 2018/05/30 08:17:56.829601 helpers.go:521: cannot retrieve info for snap "test-snapd-content-slot": cannot find installed snap "test-snapd-content-slot" at revision 2: missing file /var/lib/snapd/snap/test-snapd-content-slot/2/meta/snap.yaml [08:48] who was looking into the disappearing meta/snap.yaml bug? [08:48] zyga: was that you? [08:48] Chipaca: that was perdronis and to a small extent me [08:48] is that on your system or in spread? [08:48] https://api.travis-ci.org/v3/job/385570236/log.txt [08:49] was wondering whether I should keep that, or just blow it away and retry [08:49] hmmm [08:49] I'd be much happier from a user report, our tests do so much magic I don't trust some of those failures [08:50] - Mount snap "test-snapd-content-slot" (2) ([start var-lib-snapd-snap-test\x2dsnapd\x2dcontent\x2dslot-2.mount] failed with exit status 1: Job for var-lib-snapd-snap-test\x2dsnapd\x2dcontent\x2dslot-2.mount failed. [08:50] mount failed [08:50] * zyga looks for the journal error [08:51] Son_Goku: why the dislike for opensuse packaging? [08:51] it timed out https://www.irccloud.com/pastebin/5xyF1sNO/ [08:51] Chipaca: interesting, I wonder why it can ever do that [08:51] mborzecki, I don't generally have a problem with openSUSE packaging as a whole [08:51] * zyga scans the log for more clues [08:51] I actually do quite a lot of it these days [08:51] but one thing that has annoyed me a lot is that figuring out how to detect what release I'm targeting for a build has become a ton more difficult [08:51] woah [08:52] protocol error again :) https://www.irccloud.com/pastebin/DNFrcteJ/ [08:52] zyga: whassat [08:52] Chipaca: so, at this stage I'm tempted to get to the bottom of the stack to see what the hell is a protocol error [08:52] is that kernel doing barf [08:52] systemd doing barf [08:52] or something else [08:52] zyga: if it were the kernel wouldn't something in the journal say so? [08:52] Son_Goku: heh, that version macros matrix is a bit confusing [08:52] Chipaca: because the kernel is known for its impeccable error reporting [08:53] ;-) [08:53] Chipaca: I wonder if there's some logging going on that we don't show [08:53] e.g. stuff in dmesg that doesn't end up in the journal [08:53] May 30 08:18:01 arch snapd[10146]: May 30 08:17:59 arch kernel: print_req_error: I/O error, dev loop2, sector 0 [08:53] there shouldn't be, but ¯\_(ツ)_/¯ [08:53] ooh [08:53] is our snap corrupt? [08:54] that is building and installing the snap in devmode [08:54] so we send the snap over a socket [08:54] maybe there's a race and we get garbage [08:54] Chipaca: how would you feel if we added CRC to side-loaded snaps [08:54] for transfer [08:55] client and server both compute on the fly [08:55] and then the client sends at the end [08:55] mm? [08:55] more IO errors https://www.irccloud.com/pastebin/lx013vGf/ [08:55] zyga: if we were looking to do work in that area, I'd look instead at passing a file descriptor [08:55] but look, loop1 and loop2 [08:55] I wonder what's going on there [08:55] I wonder if this is "I cannot read the loop device" [08:56] mvo: hey! I have a hacky branch that fixes running travis CI on core18 (and non-bionic systems) [08:56] or "I read the loop device but man this is not a valid squashfs" [08:56] mvo: (the CI is failing for it as there are actual failures in the tests now) [08:56] zyga: what happens if you grab a squashfs, truncate it, and mount it [08:56] mvo: it's ugly, but besides being ugly currently it shouldn't have any real side-effects [08:56] zyga: truncate it by a half or sth i mean, not empty [08:56] hold on [08:57] Chipaca: this is not a side-load [08:57] it's from the store [08:57] and we apply a delta [08:57] it's a delta https://www.irccloud.com/pastebin/6kcddRIr/ [08:57] ah, no [08:57] I'm blind sorry [08:57] it's not a delta [08:57] it's a store pull [08:57] we check store assertions that they match the downloaded blob, right? [08:59] zyga: yes [08:59] hmm hmm hmm [08:59] yes, that's what validate-snap does [08:59] so we have a valid snap [08:59] yet the kernel chokes on mounting it [08:59] but one out of 100s we tested [08:59] and randomly [08:59] zyga: and if it were a delta, we hashsum the chunks, and then hashsum the resulting squash after applying the delta [09:00] because of the validate I don't think the snap is corrput [09:00] more like a rare bug in loop devices [09:00] the store doesn't try to mount it though [09:00] but this is just guessing [09:00] it should resquash it [09:00] and compare [09:00] pedronis: yes but here it's one of the snaps we test dozens of time a day [09:01] I bet if the blob arrived safely it is not the content that is at fault [09:01] we don't change that snap often [09:01] and it mounted and worked fine on all the other tests [09:01] (in the same run) [09:01] zyga: i'm about to hit restart on the test unless you shout [09:01] zyga: it might be that the network data is fine but it got corrupted while writing to disk [09:01] go [09:01] restart [09:01] mvo: mmmm, yes, perhaps [09:02] mvo: that's interesting [09:02] maybe google SSDs are just not what they used to be [09:02] mvo: well we check it again in validate-snap [09:02] and once in blue moon we will hit this one bit flip [09:02] but the pastebin doesn't have that bit [09:02] zyga: there's your failed with result protocol: https://github.com/systemd/systemd/blob/5300857701ff5e87169f3c90c6b570c750379dfb/src/core/mount.c#L1286 [09:02] pedronis: right if we read it again from disk, then my theory is bogus [09:03] mborzecki: interesting, thank you [09:03] mvo, pedronis: unless for whatever reason the kernel gives us some page cache but the loop device tries to read it from somewhere else [09:03] I don't know how the caching is layered in the kernel [09:03] that is, from userspace the file is ok [09:04] /* Note that due to the io event priority logic, we can be sure the new mountinfo is loaded [09:04] but from the loop device's point of view the blocks don't use the same page cache that userspace reading the file is [09:04] * before we process the SIGCHLD for the mount command. */ [09:04] heh, nice comment ^^ [09:04] mborzecki: yes :) [09:04] we also have our own code that goes and check _iff_ the mount was succcesful [09:05] but here it would not have a chance to run [09:05] if that systemd code is broken [09:05] right [09:05] and the mount really worked [09:05] but systemd thinks it did not [09:05] note that arch has a quite recent kernel [09:05] we could add some code to check for that condition [09:05] I wonder if we have enough data to see if this happens on a specific kernel more often [09:05] or if it's just all over the place but infrequent [09:06] I'll write down what I know so far on the forum [09:06] zyga: was it seen on ubuntu and fedora? [09:06] and let's keep watching this [09:06] mborzecki: I don't remember [09:06] pstolowski: cool, let me know when I should re-review [09:06] I think it was [09:07] fedora should be running recent kernels too, so if it's related to recent(ish) kernels it should pop up there too [09:11] * Chipaca toddles off to do physio [09:13] Fedora shipped 4.16.x to all stable releases a few weeks ago [09:13] mborzecki, btw, here's my proposed addition to the fedora spec for amazon linux 2: https://github.com/Conan-Kudo/snapd/commit/cb60c53c4c005de8f67dd3095faccde4daf2f518 [09:14] I'd actually probably prefer not to support classic snaps for amazon linux 2, because then once I bring snapd into epel, the experience would be consistent across the board [09:14] and people can just install snapd from EPEL once I'm happy with the experience and we push it there [09:15] mborzecki: https://forum.snapcraft.io/t/unexplained-mount-failure-protocol-error-what-we-know-so-far/5682 [09:15] Son_Goku: thanks! i'll look into it, but have to figure out what's the current plan for amzn2 first [09:15] that's what I know [09:15] please extend that thread [09:15] mborzecki: that's why it's not proposed as a PR ;) [09:15] mborzecki, but that gives you an idea of how easy it is to extend for more distros [09:16] Chipaca: ^ [09:31] PR snapd#5223 closed: image: set model.DisplayName() in bootenv as "snap_menuentry" [09:32] pedronis: pushed [09:33] pstolowski: will look in a litte bit, finishing the forum topic I promised [09:35] * zyga goes to pick up his phone [09:45] zyga: thanks for the ping earlier === popey_ is now known as popey [09:56] popey: pleasure [09:56] * zyga is picking up his phone and heads home [10:05] * thresh just pushed a new stable vlc [10:05] many thanks everyone involved in fixing bugs :-) [10:06] thresh: thank you for the new vlc! [10:16] pstolowski: did another pass [10:17] pedronis: thanks, looking === chihchun_afk is now known as chihchun [10:33] pedronis: is the cumbersome/fragile bit in 5217 that it does not use the model assertion in SetNextBoot? or is there more to it? [10:34] re [10:35] phone with new battery, I can take a shower and get back to work [10:36] mvo: it should check if the snap is the device base (using some helper) [10:37] pedronis: ok, this is what I read from your review. so snapstate will grow a way to get the model and pass it to the backend [10:37] mvo: I will be picking up making sure systemctl --failed is clean now if anything o/ I see there's one service failed there still so I'll get to it today [10:38] sil2100: thank you [10:38] sil2100: I will look at your UID pr after lunch [10:38] mvo: if you have a moment there are 3 PRs for review, but no haste with that ones [10:38] s/ones// [10:39] The travis one is ugly, but it does its job at least [10:48] PR snapd#5233 closed: client, daemon: add a "mounted-from" entry to local snaps' JSON [11:10] mvo, hey [11:15] cachio_: hey [11:15] mvo, about the reboot issue [11:16] I think it is ok the reboot [11:16] the test is forcing a auto refresh [11:16] and there is a new kernel snap in stable [11:16] so, the reboot happens after the kernel snap is installed [11:17] the same happens when the revert test is executed [11:17] mvo, I still don't understand why it was not happening before [11:18] mvo, any ideas? [11:21] cachio_: maybe pure luck, if before the stable kernel in the image and the store was the same? [11:21] mvo, this is my guess too [11:21] I am not sure, but probably there was not any new kernel snap [11:22] in stable channel [11:22] * mvo nods [11:24] mvo, I am testing a change to skip refreshing the kernel snap [11:24] mvo I'll create a PR soon === niemeyer_ is now known as niemeyer [11:43] zyga, mborzecki (cc mvo): I think we should add the device cgroup by default now. it *shouldn't* break snaps since we are applying it in many places, *but* I think that is too risky to jam into 2.33 right before release. the code change would be small, but I'd like a full cycle or so to make sure [11:43] jdstrand: ack [11:44] zyga, mborzecki (cc mvo): so I did it this way. I'll prepare another PR for master that does that after I do the PR for the spread test [11:45] jdstrand: good idea, aiming for 2.34 then? [11:45] mborzecki: yes [11:45] ok [11:45]