[03:07] zyga: when you get in can you take a look at https://bugs.launchpad.net/snapd/+bug/1850720? thanks! [03:07] Bug #1850720: installing snap with layout on /etc/ld.so.cache results in deleted mount [06:28] morning [07:01] ijohnson: done [07:01] hey mborzecki :) [07:01] zyga: hey hey [07:01] mborzecki: I think I restored my sleep budget to normal state [07:02] a lot of red and a funch of stuck travis jobs? [07:02] mborzecki: must be Thursday ;) [07:08] mborzecki: I'll review the other PR today [07:52] pedronis: mvo: hey [07:52] mvo: fixed the typo in #7701 but the unit tests fail, want me to look into that? [07:52] PR #7701: overlord: add kernel rollback accross reboots manager test and fixes [07:53] hey mborzecki [07:53] mborzecki: oh, hm, hm [07:53] mborzecki: you are welcome to look [07:53] mvo: this is what's failing there: https://paste.ubuntu.com/p/Hdzh8kbqF9/ [07:54] mborzecki: oh, fun [07:54] mborzecki: /o\ yeah, I worked on the manager tests [07:54] mborzecki: but did not run the whole thing [07:55] mvo: haven't done much around reboots, but i can dive in and check what's happening there [07:56] mborzecki: is the parallel install and gadget all blocked? if so, yes, a look at this would be great [07:57] mvo: parallel installs is waiting for a review from zyga, jdstrand already +1'ed it [07:57] mborzecki: yep, on my plate today [07:57] mborzecki: I think it will land today [07:58] mvo: and gadget remodel needs reviews ;) you're welcome to do a pass there [07:58] zyga: yeah, had to restart the tests [07:58] mborzecki: aha, nice. I will do have a look [07:58] mborzecki: cool, more eyes on the boot code are always good [07:58] and wanted to take a look into #7702, see hwther i can move some stuff over from #7193 [07:58] mborzecki: I suspect its missing fixtures now [07:58] PR #7702: tests: adding fedora 31 to google-unstable backend [07:58] PR #7193: [WIP] many: cgroupsv2 spread run === pstolowski|afk is now known as pstolowski [08:00] morning [08:12] mborzecki: yeah, it looks like the mocks are now incomplete in the booted_test. I suspect its just a matter of adding Type: snap.TypeOS etc [08:15] wow, Tyler got mentioned on https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-speculative-behavior-swapgs-and-segment-registers [08:17] btw https://medium.com/nttlabs/cgroup-v2-596d035be4d7 [08:18] mborzecki: yeah, its just a missing Type: snap.TypeBase for the snapsetup fixture [08:18] mborzecki: I push the (trivial) fix [08:18] mvo: ah ok :) [08:19] mborzecki: thank you still! [08:19] mborzecki: lol [08:19] runc, the reference implementation of OCI Runtime Spec, gained the initial support for cgroup v2 just last month (PR: #2113). This is not ready for production, especially because it lacks the implementation for eBPF device controller [08:19] PR #2113: interfaces/builtin: add i2c interface [08:19] mborzecki: at least they have a PR open [08:19] zyga: yeah, heh [08:19] anyway, I think we are "state of the art" in the sense that the state is shit anyway [08:20] zyga: feels like switching the defaults in systemd is a bit premature, but it's fine that fedora did it, otherwise nobody would care to transition [08:20] yeah [08:20] I think it's just like that [08:21] you have to toggle the experimental feature [08:21] or it will never really work [08:21] free systems are the beta testers [08:21] (in fedora world) [08:21] haha, that sounds familiar tbh :) [08:22] TBH I find the whole runc/crun/podman/docker/moby stuff a terrible mess [08:22] as in, all the $$$ behind this wants to land grab [08:22] probably inevitable but feels like mud with leeches [08:24] zyga: you left out skopeo [08:24] I'm sure I did [08:24] it's a zoo of new things that do part of what docker did [08:24] all with quirky geeky names [08:24] oh well [08:24] xD [08:25] brb, cold today, need tea === pedronis_ is now known as pedronis [08:30] mvo: hi, I made a pass on 7701 (covering also the one is based on I think) [08:30] s/made/did/ [08:30] hey pedronis :) [08:30] * zyga is back with hot tea [08:34] mvo: maybe close 7649 and keep only 7701? I find reasoning about both situations together easier [08:34] 7649 is very small [08:37] pedronis: sure, happy to do that [08:38] pedronis: thank you! will look at the feedback now [08:38] fwiw 7651 needs a second review, it will unblock 7652 [08:40] PR snapd#7649 closed: overlord: fix TestRemodelSwitchToDifferentKernel for bootvars [09:02] mborzecki: did another pass on #7665, looking good [09:02] PR #7665: devicestate: add support for gadget->gadget remodel [09:02] pedronis: thanks! [09:05] mvo: I looked what that was, but it's my slots-per-plug stuff, I cannot do the 2nd review :) [09:05] mvo: thx for the review [09:07] pedronis: haha, yes [09:07] pedronis: would love to unblock this today [09:27] pedronis: I updated the PR based on your feedback (7701) [09:28] mvo: thx, will look in a bit [09:55] Chipaca: I updated 7624 a bit more based on your suggestions, would love to add some more tests, a second look would be great (just to doulbe check) and then your opinion if I should merge and or do some PRs with test improvements first [09:55] mvo: omw [09:56] Chipaca: no rush [10:01] mborzecki: you got some feedback on the remodel PR, more to come :) [10:01] mvo: ah ok, i'll wait with the fixes then [10:02] mborzecki: do the fixes, thats fine, I will not be able to look for at least 1h [10:03] * zyga -> walk [10:05] mvo: any particular reason you went with syscall.SIGINT instead of os.Interrupt? [10:05] Chipaca: no reason, we use that elsewhere too [10:06] fair [10:07] drat, there's a chunk of functionality we should abstract somewhere neat [10:07] :) [10:11] mvo: looks good! i'll be building and testing it locally in a bit to see if i catch any other issues [10:15] Chipaca: this is why I said I'm not super happy yet :/ it feels ok but not great (yet) [10:16] mvo: maybe it's all that 'snap donwload' [10:18] siiigh [10:20] mvo: interrupting the download prints a spurious error message, at least in part because Download() isn't receiving a context form the command so it doesn't know it's been canceled [10:20] mvo: otoh the command could know it's canceled and ignore the error at that point [10:20] mvo: oto²h it's third-order nits at this point [10:20] mvo: oto³h it's the sort of polish we like [10:21] and they're ux regressions [10:21] the --direct version doesn't suffer from this [10:22] :-/ [10:22] mvo: so [10:22] mvo: how about this [10:22] mvo: switch the defaults, land as is, work on the nits until they're where we want them, _then_ switch the defaults [10:22] Chipaca: nice one! [10:22] mvo: maybe even make the options hidden so we can play with them without breaking stuff [10:24] so don't add --direct, but add a hidden --indirect or something [10:24] mvo: it feels like it's a couple of PRs of refactorings to get the UX where we want it [10:25] that's my gut feeling about it, but my gut is full of shit [10:25] wait [10:25] ¯\_(ツ)_/¯ [10:26] mvo: disadvantage is that, because people are needing this functionality, they'll start using it even if it's hidden so we'd have to carry it forever [10:26] mvo: but OTOH having a hidden --indirect forever doesn't seem too onerous [10:30] Bug #1574487 changed: possibly unclean shutdown

[10:31] Bug #1606574 changed: SSH Interface is missing [10:33] Unexpected errand, 1.5 hrs [10:34] Bug # changed: 1593450, 1613971, 1624829, 1637611 [10:37] Bug #1641631 changed: Raspberry Pi images do not support boot from USB [10:37] Bug #1642082 changed: Timestamp error when we try to sign a model assertion [10:40] Bug #1646144 changed: ACLs to devices need to be supported in core

[10:40] Bug #1646333 changed: bind mounts related to content interface plugs remain stale on snap disconnect/connect or snap updates [10:43] Bug # changed: 1647169, 1654588, 1655711, 1656820, 1657751, 1659149 [10:46] Bug # changed: 1659724, 1659744, 1660865, 1673757, 1675054 [10:47] somebody's having fun on their triage day [10:49] Bug # changed: 1663177, 1671266, 1676244, 1680088, 1683368, 1705486, 1737427, 1743301 [10:52] Bug #1747794 changed: cannot resolve host name with avahi interface [10:52] Bug #1758849 changed: Snap not able to enable ssh after core upgrade [11:01] sorry for the spam [11:05] Errand almost done, grabbing some food now [11:05] mvo: I’ll make it up tomorrow [11:05] mborzecki: what is Ubuntu-core-meta? [11:06] mborzecki: don't be sorry! good stuff [11:10] zyga: no clue, better ask ogra [11:12] zyga: it's a meta package [11:12] I wonder if we could close some weird projects [11:12] Or disable them from bug reports at least [11:14] zyga: in particular it has nothing to do with us [11:14] ubuntu-core-meta is the source of ubuntu-core-libs [11:15] zyga: where are we getting dragged into it? [11:15] hmmmm [11:15] * Chipaca reads what he wrote [11:15] ok, in the above, change 'us' or 'we' to 'snappy' [11:15] * Chipaca needs to work on that some more [11:16] ah, i just saw the bug [11:17] so snappy isn't involved at the project level, just that bug that affected it [11:17] ok [11:31] PR snapd#7699 closed: release: 2.42.1 [11:55] PR snapd#7704 opened: snap: extract printInstallHint in cmd_download.go [11:58] PR snapd#7705 opened: o/devicestate: Handle preseed in firstboot [WIP] [12:00] pedronis: hey, i marked this ^ WIP because i had to push changes to parts that i intend to propose separately (and that miss tests right now), but i was impossible to avoid these bits [12:01] Chipaca: :D [12:01] * zyga is back from errand and lunch [12:01] and back to coding [12:08] mborzecki: btw, about those locks, we create locks 0600 [12:08] so no catastrophy :) [12:11] zyga: https://i.imgur.com/HCe563J.jpg [12:11] * Chipaca goes for lunch [12:11] hmm [12:11] Chipaca: meouch! [12:12] PR snapd#7193 closed: [WIP] many: cgroupsv2 spread run [12:20] Chipaca: thanks for the review, I tried to answer your questions, maybe you have further feedback on the first and last [12:20] pedronis: the .More() was to error if there was garbage after the json, fwiw [12:21] Chipaca: we could, it's a bit more interesting if it was a network connection [12:21] joeubuntu: wait so you and joedborg _aren't_ the same person? [12:21] :) [12:21] true [12:22] Not that I know of 😂 [12:22] joeubuntu: i learned of joeubuntu leading the robotics team at the same time i saw your nick [12:23] In the singularity all joes are unified in to one... [12:29] Chipaca: would it be better if that label was inside an Options struct ? [12:29] (I would do that change in one of the follow ups though) [12:30] pedronis: not until we know what we're doing :) [12:31] ok [12:32] pedronis: found the bug in the completion pr (i'm having lunch so figured it was a good time to poke at it) [12:32] ah, good [12:33] Chipaca: the fun with labels happens here: https://github.com/snapcore/snapd/pull/7658/files#diff-eb9825aa18d9bbbcc41ca31728af8157R75 in the later PR [12:33] PR #7658: cmd/snap-preseed: add snap-preseed executable [12:33] there's just a TODO for now there, it relates to your work (modes and modeenv etc) [12:37] pedronis: i think that's the link to the nil db pr, no the labels one :) [12:39] Chipaca: oops [12:40] https://github.com/snapcore/snapd/pull/7695/files#diff-0b649265c21137e8fd367d4a16607a82R409 [12:40] PR #7695: o/devicestate: the basics of Core 20 firstboot support with test [12:40] Chipaca: ^ [12:42] pedronis: ta [12:42] i'll get to that eventually (hoping for today!) [13:06] Chipaca: any I have a chain with another 2 or 3 PRs on top of the ones there, so fill free to do a rename to OpenWithLabel next week [13:06] s/any/anyway/ [13:24] brb [13:33] pedronis: with the completion tweak we no longer offer 'interfaces'; is that ok? [13:33] Chipaca: it's been deprecated since a while [13:33] but no strong opinion either way [13:34] pedronis: i think it's ok, i do need to fix a test though :) [13:34] very easy fix [13:34] use connections instead [13:34] nah, the test is checking the completion of 'snap in' [13:35] ah ok [13:35] :) [13:35] we still do have interface [13:35] singular [13:36] Chipaca: whom to thank for the nice pre-baking label icon ;) [13:36] yep, that's the fix [13:36] ? [13:36] pstolowski: not me! [13:37] not this time :-D [13:37] hmm [13:38] but whoever did it, thank you for using the actual 🍞 and not :bread: [13:44] mborzecki: good suggestion in the systemd prebaking PR [13:54] PR snapcraft#2786 opened: cli: add support for 'http-proxy' and 'https-proxy' parameters [13:56] pstolowski: will you have time to stay after the standup? [13:57] pedronis: yes, but i may need to drop for a while at around 4pm to pick my daughter from school [13:57] pstolowski: ok, let's see what we can, I would like to go over the firstboot stuff with you live [13:59] Chipaca: Q to your fix [14:01] popey: `snapd` is now in EPEL 8, so the website needs instructions for CentOS 8 [14:02] the instructions should basically be the same as for CentOS 7 [14:03] so I think only the notes about lack of availablity on RHEL 8 and CentOS 8 need to be removed === ricab is now known as ricab|lunch [14:06] kyrofa: you might want to give me co-maintainership of squashfuse [14:07] you can do that by going to https://src.fedoraproject.org/rpms/squashfuse, logging in, going to the project settings, go to users & groups, and add "ngompa" as an admin [14:14] pedronis: running spread locally so can't push a fix for the typo, but made a suggestion there if you want to commit it [14:15] Eighth_Doctor: I've been tracking the EPEL updates - I'll update the install docs. [14:16] degville: thanks [14:16] `snapd` just synced out to EPEL 8 this morning [14:16] brilliant, thanks! [14:59] Chipaca: I committed [15:07] nice OOM running spread locally [15:19] PR snapd#7707 opened: snap: add TestDownloadDirectStoreHappy test === ricab|lunch is now known as ricab [15:29] re [15:29] family home now [15:29] back to work [15:33] \o/ [15:33] mvo, pstolowski, mborzecki: we will likely visit graves tomorrow as it's bound to rain all weekend here [15:34] * cachio lunch [15:36] has anyone figured out why the tests are unhappy? [15:37] mvo: because they hate us [15:38] mvo: i've just seen an error in prepare on centos, for example [15:38] + systemctl daemon-reload [15:38] Failed to execute operation: Connection reset by peer [15:39] and on restore, also on centos, [15:39] + systemctl daemon-reload [15:39] Failed to execute operation: Connection timed out [15:39] it's a fractal of WAT [15:39] a WATdelbrot curve [15:44] hrm, hrm, if its all centos I'm inclined to move it to unstable-systems until its fixed [15:44] kind of annoying that it blocks landing stuff [15:44] mvo: ah it is in unstable already [15:44] mvo: sorry [15:44] mvo: at least this particular centos is :) [15:45] * mvo nods [15:45] the task that says CentOS contains no CentOS [15:47] Eighth_Doctor, are you just looking to get squashfuse updated? Or is there a larger issue that I've missed? [15:50] error: Get https://fastly.cdn.snapcraft.io/download-origin/fastly/b8X2psL1ryVrPt5WEmpYiqfr5emixTd7_1797.snap?token=1572548400_b95c66d9e68375e958b164a800f1b618a0e69653: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "GlobalSign CloudSSL CA - SHA256 - G3") [15:50] mvo: ^ [15:51] mvo: that might have something to do with the red [15:56] Chipaca: *grumpf* not cool [15:59] wahaat? [15:59] SSL is such a shitshow [16:00] ijohnson: did my response to the bug related to ldcache make sense? [16:01] zyga: yes sorry for not responding in the bug, but I was able to work around it by having ldconfig work on some other file, then I just copy that file to what's bind-mounted so it never calls unlink on the file [16:02] it is unfortunate that there's not much better we can do about that though [16:02] ijohnson: I suggested a workaround [16:02] ijohnson: a symlink would make it easier [16:02] ehh not quite [16:02] I tried the symlink but it still has problems [16:02] no? is the code sensitive to symlinks? [16:02] I see, well worth a try [16:02] I think it will be nicer [16:03] once we have /etc tmpfs [16:03] I don't remember the issue with symlink, but I tried it and it didn't work for some reason [16:03] or not tmpfs [16:03] just a /var/lib/snapd/.../etc [16:03] perhaps it was not an issue with the symlink directly, but anyways I have a workaround [16:03] kyrofa: 1. squashfuse needs to be updated, it needs to be introduced to epel8, and squashfuse binary needs to be replaced with a symlink to squashfuse_ll binary [16:08] mborzecki: I have some ideas on how to solve the lock issue [16:08] (1) is a known issue, but squashfuse_ll is news to me. Can you give me some background? [16:09] mborzecki: the inhibit files can be locks [16:09] mborzecki: held by snapd [16:09] mborzecki: and unlocked by snapd [16:10] mborzecki: and snap run, if the files are present, try to grab a shared lock [16:10] mborzecki: while snapd holds an exclusive lock [16:10] mborzecki: I need to think how to create an inhibition file in a way that is race free [16:11] mborzecki: but I think it can be done by allowing snapd to check for ability to grab the exclusive lock [16:11] mborzecki: so if the lock is held by snap run already [16:11] PR snapd#7708 opened: parts/plugins: don't xz-compress a deb we're going to discard [16:11] mborzecki: we will know in snapd [16:11] mborzecki: anyway, enjoy your evening [16:11] zyga: we probably need to draw/write this down [16:11] mborzecki: yeah, I'll write it down in the doc [16:21] zyga: you still here? [16:22] zyga: i think something's change such that google:ubuntu-18.04-64:tests/main/snap-seccomp-syscalls fails [16:23] zyga: https://paste.ubuntu.com/p/FGp43JqGGJ/ [16:23] or maybe that's an mvo ^ not sure [16:23] looks like we got a bunch of new syscalls [16:23] Chipaca: meh, ok [16:23] most around time64 [16:24] mvo: so that's probably why a lot of things are red [16:24] there's also a fsconfig/fsmount/fsopen/fspick thing we might be interested in :) [16:25] read more about 'em here: https://lwn.net/Articles/759499/ [16:25] Chipaca: hm, thats on 18.04? [16:26] they look like a change in the right direction, aiui, given some of our woes [16:26] mvo: ye [16:28] so i suspect this is a blocker, righ tnow [16:29] Chipaca: I can fix this probably [16:29] Chipaca: yes I'm here [16:29] Chipaca: looking now [16:29] p > 0.9? [16:29] oh [16:29] lots new [16:29] Chipaca: yeah [16:30] we have some problems [16:30] clone3 :/ [16:30] it may bite us [16:30] as soon as glibc uses it [16:30] and we respond with EPERM [16:30] and glibc doesn't use older clone [16:31] Chipaca: 180.04 though is very surprising [16:31] Chipaca: is this new seccomp in 18.04 [16:31] ? [16:31] jdstrand: ^ do you know if we have a backported libseccomp in 18.04 [16:31] zyga: this is in 18.04, as above [16:31] jdstrand: or a new kernel and libseccomp now reports new syscalls it knew about [16:31] zyga: ie this is failing google:ubuntu-18.04-64:tests/main/snap-seccomp-syscalls [16:32] Chipaca: I'll park my work and look [16:45] pedronis: +1 on #7694 [16:45] PR #7694: many: load/consume Core 20 seeds (aka recovery systems) [16:48] Chipaca: testing locally [16:48] I'll send a PR soon [16:48] brb [16:50] zyga: FWIW 4.15.0-47-generic landed in 18.04 recently [16:51] zyga: no seccomp update since august afaik [16:51] (that's according to the dpkg.log in this kvm 1804, fwiw) [16:52] pstolowski: thanks, tried to answer your question [16:53] PR snapd#7694 closed: many: load/consume Core 20 seeds (aka recovery systems) [16:55] Chipaca: pstolowski: I'm going now to rebase the two follow ups [16:55] pedronis: thank you [16:55] great [16:59]