mupPR snapd#9514 opened: Enforce the confined fwupd to align Ubuntu Core ESP layout <Created by woodrow-shen> <https://github.com/snapcore/snapd/pull/9514>02:48
=== jamesh_ is now known as jamesh
mborzeckizyga: mvo: hey06:31
mvogood morning mbo06:32
mvogood morning mborzecki06:32
amurrayzyga: so I have a couple small fixes for PR #9509 but I noticed you said not to push anything more there - also I am pretty sure all the commits already use my @canonical.com email address... so I just wanted to see where things were at from your point-of-view06:36
mupPR #9509: multipass/docker-support: Mount /etc/apparmor* from the base snap <Squash-merge> <⚠ Critical> <Created by alexmurray> <https://github.com/snapcore/snapd/pull/9509>06:36
zygaamurray go ahead and push them06:49
zygaamurray I'll discuss one bigger issue with mvo shortly06:49
zygawe may need to have a different implementation altogether, we'll see06:50
zygahey mborzecki, mvo06:50
zygasorry for starting late, not feeling good, I think I'm going to be sick this week :/06:50
zygamvo: can we briefly discuss what I tried and found on Friday06:50
zygato keep amurray in the loop?06:50
mvozyga: no worries, I would love to have samuele in this too06:51
zygamaybe I can state what I found now while amurray is here06:52
zygaand we can discuss more with Samuele later06:52
zygaI don't have a solution yet, I think, only ideas06:52
zygaamurray, mvo: the problem with the current implementation is that by the time we compute security profiles, the base snap is unlinked and (masked by the absent error handling on readlink) we get a non-sense profile definition06:53
zygaamurray, mvo: I tried to fix this by using connected plug/slot version of the interface, after all, that seems natural06:53
mvozyga: +106:53
zygaamurray, mvo: and we know exactly the revision we are after, so that's even better06:53
zygaamurray, mvo: but then realized this doesn't work because those interfaces can be provided by _either_ core or snapd, in neither case that is the base snap we are after06:54
zygaamurray, mvo: from here on I was exploring ideas, each with their own flaws:06:54
zygaamurray, mvo: 1) leave the symlink in the mount profile and teach snap-update-ns to resolve the current symlink automatically06:54
zygathis works on connect06:55
zygait works on disconnect06:55
zygawhen the base revision changes and we discard and re-generate, it also works just fine06:55
zygait's a bit of special case logic in snap-update-ns06:55
zyga2) have new mount point for the base snap explicitly, like /base and work from there (I like this because we should do /app later as well, this would fix so so so many issues)06:56
zygathe /base mount is useful as it is the naked base snap, without anything covered06:56
zygaall the ideas on exposing fixed mount points for key snaps are interesting to consider, because IMO they would fix some of the snaps-are-hard-and-require-loads-of-env-variables-to-work-around over time06:57
zyganot as many as runtime + /app model in flatpak but I think that would be a step in the right direction06:57
zyga3) have a new variable in snap-update-ns that explicitly talks about base snap name and revision06:57
zygathis is like 1) but not "magic" that some symlinks work while others are ignored06:57
zygathe problem in 3) is, IIRC, snap-update-ns does not have the right information to provide those other than by reading YAML by itself and readlinking current files06:58
zygait's a variation of 1 that it might be regarded as 1b) perhaps06:58
zygathe reason I find 2) interesting is that, over time, we could have non-shared, curated /etc06:59
zygaand /etc/apparmor.d -> /base/etc/apparmor.d is nice06:59
zygaor perhaps /snap/.base/06:59
zyga(we could come up with a way to inject .base there easily)06:59
zygathose are my ideas07:00
zygaamurray: does this make sense? I'm not sure if I'm making myself comprehensibly clear today07:00
zygamvo: same question07:00
mvozyga: it does make sense07:02
mvopstolowski: good morning07:02
mborzeckipstolowski: hey07:03
mvozyga: I like (3) as it seems relatively easy? I like (2) better but that seems more work?07:03
mvozyga: or at least conceptually more change?07:03
zygamvo I agree,07:03
mvozyga: in (3) you say snap-update-ns does not have the information, can we just pass it somehow?07:03
zygamvo: we'd have to teach snap-update-ns about $SNAP_BASE_NAME and $SNAP_BASE_REVISION07:04
zygamvo: I need to check07:04
zygaI mean, base is easy07:04
zygabase is in the yaml of the snap we are operating on07:05
zygaand we should be able to read that07:05
zygaI hope07:05
* mvo nods07:06
zygamy throat hurts today so I really prefer to type rather than talk07:07
zygabut we can have a call about this later today07:07
zygaI'll start fixing it07:07
mvozyga: sounds good, like I said, I would love to have Samueles opinion because I'm worried that I may give you the "wrong" direction07:11
amurrayzyga: so is this only an issue on refresh of the base snap? I prefer 2) as well since either 1 or 3 needs to pass more info to snap-update-ns which I feel is a bit like the wrong place to be doing this - I don't love the idea of expanding variables in snap-update-ns either07:13
zygaI think 2, since it involves new FS elements, may take longer to agree on07:14
zygawhile 1 and 3 we could do "locally" without introducing new elements that need design07:14
zyga(the new paths in the filesystem that is)07:15
amurraytrue... what if we did the snap-confine approach, would that also suffer problems on refresh of the base? perhaps that is a better temporary fix that we can back out later once /base etc is bike-shedded07:15
zygaamurray snap-confine based fix has the problem of not being able to handle interfaces at all, which is not too good in my opinion, we also would really like to shrink that list ot fixed mount points it does07:16
zygaamurray: technically it might work but I think we should not do it07:16
amurrayzyga: sure but I wonder (given ideally we want this fixed in a few days) whether it is the more pragmatic approach to take07:17
zygaI think 1-3 are doable today07:17
zygait's not a big change07:17
amurrayif 1, how does snap-update-ns recognise that it is a symlink? currently we seem to try very hard to purposefully not resolve symlinks in snap-update-ns (and so fail if a mount point contains them)07:19
zygaamurray we could really whitelist /snap/*/current07:20
zygaor perhaps /snap/$base_snap_name/current only07:20
amurrayzyga: for #base_snap_name we would then still need to teach snap-update-ns what the base snap is (or how to find it) - so that is more like option 3) from what I can see.. at the end of the day I am not too fussed what option we go with, I just want to take an approach which is achievable given the very tight deadline we have07:22
zyga1-3 only differ in the presentation layer07:23
zygain both cases we need to know base snap name and revision in snap-update-ns07:23
* zyga nods 07:23
amurrayhence my preference for snap-confine since it is the most minimal touch approach07:23
zygaamurray: but then we are stuck with two apparmor-specific mount points in snap-confine and no clear way to undo that on snapd upgrade07:24
zygathat's my main worry07:24
zygaif we ever want to stop sharing /etc, handling that is a nightmare07:25
amurrayzyga: but we have that already for /etc/ssl.conf etc - so we would have to solve that then anyway right? so adding these apparmor ones doesn't appear to increase the complexity to me07:25
zygaamurray: yes, but also less nice IMO07:26
zygaamurray: I didn't check what happens in fedora when those directories are gone07:26
zygas-u-n code handles that07:27
zygas-c will fail07:27
zygawould require more hacking to get that to work07:27
* zyga breaks for breakfast07:27
mupPR snapd#9515 opened: cmd/snap-confine: update path to snap-device-helper in AppArmor profile <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9515>07:29
zygaThere is an entirely new user-space API for GPIO lines. Naturally, this API is rigorously undocumented, but some information can be gleaned from this commit.07:34
zygafrom https://lwn.net/Articles/834157/07:34
mborzeckizyga: only 5+ years from becomign available in every major vendor's BSPs07:35
amurrayzyga: got disconnected from my bouncer so not sure if my last msg got through - ok, cool, you know this better than me - I'll push those couple fixes - let me know if there is anything else you need me to look at07:43
zygaamurray, your last message I got was about /etc/ssl.conf07:46
zygaamurray: please push them, I'll rebase my changes on top07:46
zygaamurray: and I see your point of view, I just want to try doing the less problematic over time thing instead07:46
amurrayzyga: thanks, I understand where you are coming from and appreciate your focus on avoiding more technical debt - I just want to make sure we get it fixed in time for release - I'll stop worrying now though, I've pushed my fixes (fwiw I managed to create a local arch qemu VM for testing... so that was a minor win 🏆)07:54
mborzeckiehh, power outage08:05
zygaamurray your mode wins08:39
zygawe cannot use snap-update-ns08:39
mupPR #9516: cmd/snap-confine: mask host's apparmor config <Created by zyga> <https://github.com/snapcore/snapd/pull/9516>08:40
mupPR snapd#9516 opened: cmd/snap-confine: mask host's apparmor config <Created by zyga> <https://github.com/snapcore/snapd/pull/9516>08:40
zygaI'll take your test and adjust08:40
mvopedronis: could you please check 9516? if it looks good we should land it into groovy today to unblock the release team08:52
pedronisshould it have a spread test?  on top of docker one?09:00
zygayes, I will09:00
zygait's just something we drafted during the 1:109:00
zygaI'll adjust the mount ns test first to make it not red09:01
zygathen convert the test Alex wrote09:01
pedronisI did a pass09:03
mupPR snapd#9515 closed: cmd/snap-confine: update path to snap-device-helper in AppArmor profile <Simple 😃> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9515>09:05
zygaerror: invalid tests/main/snap-env environment: invalid variable name: INSTANCE_KEY/regular09:05
zygadid master spread regress?09:05
mvozyga: uh, where do you see that?09:07
zygamay have been a fluke, I copied ~/go from x240 over to a pristine system and ran spread from there09:08
zygait goes away if I give the entry an empty string value: INSTANCE_KEY/regular: ""09:08
mborzeckizyga: something strange on opensuse, we seem to be reloading s-c apparmor profile from the core snap there09:14
zygamborzecki note, that is possible09:15
zygathose are separate profiles09:15
mborzeckizyga: https://paste.ubuntu.com/p/fQFXkdN8tQ/09:15
zygais that within one test?09:16
zygaperhaps snapd reload?09:16
mborzeckizyga: this one type=AVC msg=audit(1603098317.172:596): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/snap/core/10251/usr/lib/snapd/snap-confine" pid=19468 comm="apparmor_parser"09:17
zygaagain, that is normal09:17
zygawhat's  the concern?09:17
mborzeckizyga: i think it messes up the profiles, and then s-c complains about elevated permissions09:24
zygamborzecki how can it mess up the profiles?09:24
mborzeckizyga: https://paste.ubuntu.com/p/C3Xcrx82y2/09:24
zygawhich snap-confine is complaining (what is the path that is executed)09:24
zygacan you go to /proc/pid/attr09:25
zygaI think this could be a different bug09:25
zygawhat do you see there?09:25
zygamborzecki can you compare09:25
zyga /proc/PID/attr/current and /proc/PID/attr/apparmor/current09:26
zygaspecifically, if you could gdb snap-confine invoked from that path09:26
zygabreak it somewhere09:26
zygaand compare09:26
zyganot that the absolute path of invoked snap-confine matters09:26
zygabecause it is that path alone that determines the confinement profile09:26
zygamborzecki tl;dr; version is that IIRC kernel has changed things09:26
zygaand apparmor bits are in apparmor/current09:26
zygaand not in current anymore09:26
mborzeckizyga: it's opensuse, we are invoking /usr/libexec/snapd/snap-confine09:26
zygabecause of reasons09:27
zygathat's easier to debug then09:27
zygaand the profile you see there is not used09:27
zygaas it's compiled for an absolute path /snap/core/nubmer/usr/lib/snapd/snap-confine09:27
mborzeckiit's not, s-c reports it's unconfined (and proceeds to run as a user)09:27
zygaI think snap-confine is broken :)09:27
zygaas in, it's doing this check incorrectly09:27
mborzeckipff, proceeds to run as root (complains when running as user)09:27
zygaplease look for what I mentioned09:28
zygaand then we know09:28
zygaif the profile is loaded then it's applied based on path09:28
zygasnap-confine reads one of the two files09:28
zygaperhaps it should read the other now09:28
zygait's done via libapparmor09:28
zygabut perhaps it's also buggy09:28
zygamborzecki I remember reading there was some motivation for this change09:28
zygajust not sure if it's some default or when it was merged09:29
zygaprobably very few apps check their own confinement09:29
mborzeckizyga: fwiw, there' no /proc/pid/attr/apparmor/ at all09:29
zygamborzecki what's the kernel version?09:29
mborzeckizyga: 5.3.1809:30
mborzeckiwhy so old?09:30
mborzeckii have 5.8.14 in a tumbleweed vm09:30
zygamborzecki: hah09:31
zygasomething is fiiishy09:31
zygamaybe desync with kernel and libapparmor?09:31
zygakernel from GCE09:31
zygaand library from actual suse09:31
zygamborzecki what's the full uname?09:31
mborzeckizyga: looks like a leap 15.2 kernel but tumbleweed?09:31
zygamaybe cachio invention09:32
zygadis-upgraded or something09:32
* zyga hands maciek the pieces back09:32
zyganot a09:32
zyganot a good bug09:32
* zyga runs tests and picks up tea09:36
mborzeckihah :P don't run away09:36
zygano no, just a moment09:36
zygamborzecki can we upgrade the kernel in-place?09:36
zygamborzecki maybe we do need a sanity test for kernel version09:36
mborzeckizyga: hm, really feels like the updates should be done via tumbleweed-cli, i don't understand why it's not done like this09:37
zygamborzecki: I think it starts with the image09:37
zygathey are almost all custom made09:37
zygain ways that nobody reviewed09:37
mvozyga: want a hand with 9516 or are you on it (the busy-work like pulling the tests from the other PR, adding comments/bug references)09:37
mborzeckizyga: i'll poke cachie when he's around09:37
zygamvo sure: if you pick up the spread test from the other location that would be good09:38
zygamvo  note that the only test we should run now is a 20.1009:38
zygawe never ran it on 20.10 so we didn't notice the original fix was failing on the current symlink09:38
* zyga really goes to get tea09:38
mupPR snapd#9517 opened: spread, tests: tweaks for openSUSE <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9517>09:45
mvozyga: hm, the test we have in the other pr is a bit indirect, can we test more directly?09:47
zygamvo: I think the test should be09:47
zygaon 20.10 we can compile a profile from within a snap using {multipass,docker}-support09:48
zygamvo: what did you had on your mind?09:48
mvozyga: I cherry picked all the bits now from the other PR and run it on 20.1009:48
mvozyga: I had in mind actually installing the problematic snaps09:48
mvozyga: well, we already have docker-smoke09:49
mvozyga: so that one should work once we have the fix09:49
zygamvo: real snaps are poor regression tests09:49
zygaas they shift around09:49
mvozyga: was wondering if we should have a multipass-smoke09:49
zygawe might have both09:49
mvozyga: yeah, sorry, you are right09:49
zygasmoke tests + simplified regression test09:49
zygamborzecki can we update the kernel to tw kernel and REBOOT?09:50
mborzeckizyga: not sure it's worth meddling with the kernel at this point, i'd rather have cachio look into building proper images starting from some recent tw snapshow09:51
zygaI think nobody but him knows exactly what is in that image09:51
zygamvo: I'll push some more fixes there, (apparmor and mount ns changes)09:53
zygajust doing one more iteration to make sure it's good09:53
mborzeckizyga: pushed one more patch to #9517, we were not reloading the right apparmor profile file on install09:54
mupPR #9517: spread, tests: tweaks for openSUSE <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9517>09:54
zygasee it09:54
zygathat's a fallout from the recent change, right?09:55
mborzeckiyes, lib -> libexec09:55
mvozyga, amurray I cherry picked all the test fixes into pr#9516 please double check. also tweaked comments. test is running now09:55
mborzeckiheh, i'm sure we would have caught that earlier if the testing worked :/09:56
zygamvo: yeah09:56
zygamborzecki: yeah09:56
zygamvo: thanks, I'll check shortyl09:56
mvozyga: no rush09:56
clmsycurious to know if there is anyone else having trouble while booting into core20. with the gadget.yaml file from here: https://github.com/snapcore/pc-amd64-gadget/blob/20/gadget.yaml09:58
clmsyi always get ubuntu-boot not found message, and device falls back to grub emergency09:58
* zyga defers clmsy's question to other team members09:59
clmsycould it be possible because my usecase is a bit different10:00
mvoclmsy: this is the gadget we use for all our image tests so it *should* work (not saying it does :) what can I do to reproduce this?10:00
mvoclmsy: i.e. how do you build the image etc?10:00
clmsyi build the image through ubuntu-image with a signed model assertion, here i posted the full thing with some parts obv redacted: https://pastebin.com/HvaENzMN10:03
clmsyi use the exact same gadget.yaml file10:03
clmsyimage build is a success but when i boot, i get ubuntu-boot not found message than it drops to emergency grub section, then the-tool service has a fatal error where it says one of the following is not found [kernel core snapd]10:04
clmsythe thing that makes me question is setting both core20 and core18 as type: base10:04
clmsybut if i don't do that the ubuntu-image fails on me, saying that one of the snaps need core18 as base10:04
mborzeckiclmsy: what's the version of snapd on your build host?10:06
clmsyi do want to ask something related to this, i used to work on core16 for the same device, and when i did snapcraft build on my gadget it used to create staging folder etc, when i do it now with core20 as base:core20 in snapcraft file. there is no stage folder at all but still build succeeds10:06
clmsythis is the output for snap --version for my build host: snap    2.4710:07
clmsysnapd   2.4710:07
clmsyseries  1610:07
clmsyubuntu  20.0410:07
clmsykernel  5.4.0-48-generic10:07
clmsyah shit sorry for multiple messages :/10:07
mborzeckiclmsy: does it build successfuly if you keep type: base for core20 only?10:08
zygano worries10:08
mvoclmsy: when you boot, can you see it creating partitions and rebooting and then failing? or does it not even make it through the "install" state of the boot?10:08
mvo(and is my question comprehensible?)10:09
mborzeckiclmsy: a screenshot/serial log would be useful if can get that10:12
clmsymborzecki: if i change core18 from base to core and keep base for core20 only this is what happens during ubuntu-image: error: core snap has unexpected type: base10:12
clmsyyes let me get more information on the boot what exactly is happening10:13
mborzeckiclmsy: try completely dropping the `type: ...` line from the core18 entry in the gadget10:13
zygaIIRC some people bumped into this before10:14
mborzeckizyga: hm thought it was fixed in 2.4710:14
clmsybeforehand i was told it was related to this bug:10:16
mupBug #1883973: cannot boot uc20 model with multiple base snaps <snapd:Fix Released by pedronis> <https://launchpad.net/bugs/1883973>10:16
clmsymborzecki: this is what happens than: error: snap "core18" has unexpected type: base10:19
clmsysomehow defaults to it ?10:19
zygamaybe just drop core18 entirely?10:20
clmsybtw a quick capture of what is happening at boot: https://streamable.com/hoqzfz10:21
ijohnsonmorning folks10:23
zygaijohnson hello10:24
zygawelcome back, how was your week off?10:24
ijohnsono/ zyga10:24
ijohnsonit was nice, it's gotten a bit colder now but feeling well rested :-)10:24
mvoijohnson: hey, welcome back!10:26
ijohnsonhi mvo10:27
zygamvo, mborzecki: https://github.com/snapcore/snapd/pull/951810:35
mupPR #9518: tests: add value to INSTANCE_KEY/regular <Created by zyga> <https://github.com/snapcore/snapd/pull/9518>10:35
zygaweird, would love if you guys could confirm master fails on your machines, with current spread10:35
mupPR snapd#9518 opened: tests: add value to INSTANCE_KEY/regular <Created by zyga> <https://github.com/snapcore/snapd/pull/9518>10:35
mborzeckizyga: what fails?10:38
zygamborzecki read the commit message please10:38
zygaspread refuses to run10:38
zygamaster spread, master snapd10:38
mborzeckizyga: https://build.opensuse.org/request/show/84252710:47
zygapedronis: can I merge https://github.com/snapcore/snapd/pull/9512?10:50
mupPR #9512: o/snapstate: move setting updated SnapState after error paths <Created by pedronis> <https://github.com/snapcore/snapd/pull/9512>10:50
zygamvo: pushed trivial fixes to apparmor PR11:01
* zyga afk to make more tea11:02
mborzeckizyga: haha, so tw is green now :P https://github.com/snapcore/snapd/pull/9517/checks?check_run_id=127450867811:05
mupPR #9517: spread, tests: tweaks for openSUSE <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9517>11:06
pedroniszyga: we should chat with mvo about it and #942211:10
mupPR #9422: overlord: add link participant for linkage transitions <Needs Samuele review> <Created by zyga> <https://github.com/snapcore/snapd/pull/9422>11:10
mvopedronis: now?11:16
mvopedronis: or later today?11:17
pedronismvo: I have a meeting in ~10,  anyway when it works11:17
mvopedronis: if you could give me a quick status of your thoughts in the standup HO that would be great(unless you need the 10min to prepare or for a break or something else)11:19
mvozyga: did alex approve the approach btw? ideally we need a +1 from him too but I guess it's late11:25
zygamvo: it was his preference11:26
zygamvo: as it was simpler conceptually11:26
zygawe discussed it here in #snappy11:26
zygaboth this morning and last week11:27
mvozyga: \o/ thank you11:29
mvozyga: just wanted to confirm11:29
zygaI think this was worth exploring, only if to learn what the limitations of the mount systems are11:29
zygaor perhaps more of the interface system11:29
cjwatsonIs there any way I can figure out why snapd on my laptop appears to be in a fairly tight loop using ~10% of a CPU?  "snap changes" says "1566  Doing   6 days ago, at 18:52 BST  -      Auto-refresh snaps "lxd", "skype""11:36
zygacjwatson I think this is something that pstolowski was debugging11:37
zygacjwatson it's stuck trying to refresh lxd but fails on a mount namespace, perhaps11:37
zygapstolowski ^^11:37
mvomborzecki: do you think you could do a second review for 9516 ?11:38
zygamvo: what should I do about 949911:39
zygaI can adjust it to pass and add comments that it fails on core16 and core2011:40
zygaI could dig and try to fix the problem11:40
pstolowskizyga. cjwatson not really, but maybe related. can you get a list of all tasks (snap change <changeid>) and journal log?11:40
cjwatsonpstolowski: https://paste.ubuntu.com/p/WBkysQcyRr/, and roughly what journalctl parameters?11:41
zygaDoing   6 days ago, at 18:52 BST  -                         Download snap "skype" (156) from channel "latest/stable" (40.33%)11:41
zygait's the other bug11:41
zygastuck download11:41
zygaalso Doing   6 days ago, at 18:52 BST  -                         Download snap "lxd" (17738) from channel "latest/stable" (65.26%)11:41
zygacjwatson are the percentages moving?11:42
zygaor are they stuck at 011:42
zygaat zero change I mean11:42
pstolowskicjwatson: hit the elusive download issue11:42
cjwatsonzyga: not changing as far as I can see11:42
pstolowskicjwatson: journalctl -u snapd11:42
cjwatsonAlso "snap change 1566" takes 16 seconds ...11:42
zygacjwatson that's because we hold the lock and save all the time apparently11:43
zygaand there are only write locks, no read locks11:43
zygathat's a known issue (CC mborzecki )11:43
cjwatsonpstolowski: adding -b to just get information from the current boot: https://paste.ubuntu.com/p/MYdjW8d6NJ/11:44
pstolowskicjwatson: thanks, actually the changes are from 6 days ago,  might be useful to have log from back then11:49
cjwatsonpstolowski: the log I gave you covers that time though?11:50
cjwatsonmy last reboot was nearly 8 days ago11:50
pstolowskicjwatson: ah yes, you're right, sorry11:50
pstolowskicjwatson: could you please send me your /var/lib/snapd/state.json (you may want to obfuscate your private macaroon there)?11:53
zygaijohnson thanks for the review on fsck11:53
pstolowskicjwatson: after that you can restart snapd service to recover from this11:54
ijohnsonzyga: yeah thanks for eventually catching my mistake!11:54
zygaijohnson: the means of landing this are unclear so if you can push the fixes for fsck that would be great11:55
ijohnsonzyga: ah ok, yeah the fix should be real simple, I just prefered to be able to demonstrate that the pr fixes the issue with that spread test already in master but sure I can open the presumptive fix to snap-bootstrap and manually verify with your spread test as it exists11:55
zygaijohnson if the fix is easy and in snapd itself, just go for it here11:56
ijohnsonzyga: sure that should be easy actually11:56
zygathat will leave only core16 left11:56
zygaand core20 in a much better state!11:56
zygawoot, thanks for that ijohnson11:56
ijohnsonzyga: I will push in a bit still going through all the emails, etc. from last week11:56
cjwatsonpstolowski: Thanks, emailed to you and restarted11:59
pstolowskicjwatson: thanks! got it11:59
cjwatsonand indeed it finished the refresh now12:01
pstolowskicjwatson: peculiar... it's an issue we got reported by one of the customers. not reproducible and we had no logs and very little info so far... maybe the info from you will help progress on that, thank you!12:03
cjwatsonpstolowski: good luck12:03
cjwatsonI guess laptops are good at reproducing things that might happen in not very reliable network environments12:04
pstolowskiindeed. may be something related to suspending & resuming12:05
pstolowskior actually network going down12:05
mupPR snapd#9519 opened: tests: moving the lib section from systems.sh helper to os.query tool <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9519>12:06
cjwatsonboth of which are reasonably frequent occurrences for me; this network is not 100% reliable, and I'm moving between two houses at the moment so I suspend and resume a lot12:07
pedroniszyga: should we close #9509 ?12:08
mupPR #9509: multipass/docker-support: Mount /etc/apparmor* from the base snap <Squash-merge> <⚠ Critical> <Created by alexmurray> <https://github.com/snapcore/snapd/pull/9509>12:08
zygayes, we should12:08
zygait was an interesting exploration but we cannot proceed that way12:09
mupPR snapd#9509 closed: multipass/docker-support: Mount /etc/apparmor* from the base snap <Squash-merge> <⚠ Critical> <Created by alexmurray> <Closed by zyga> <https://github.com/snapcore/snapd/pull/9509>12:11
zygaijohnson: i've iterated on fsck test a bit, I'll add mkfs test as well but I'm going to have a quick peek if I can fix core16 fsck story12:22
zygamvo: the spread test needs some help12:34
zyga+ test-snapd-docker-support-app.test-snapd-docker-support12:34
zygaWarning: unable to find a suitable fs in /proc/mounts, is it mounted?12:35
zygait is trying to load the apparmor profile12:35
zygabut without apparmor, that won't work12:35
zygamaybe we should merely attempt to compile the profile?12:35
cmatsuokazyga: did you see this bug report about livepatch problems? does it make sense to you? https://bugs.launchpad.net/snapd/+bug/178447412:35
mupBug #1784474: canonical-livepatch failing to enable <Canonical Live Patch Client:Incomplete by danneboom.dieter> <snapd:New> <https://launchpad.net/bugs/1784474>12:35
zygamvo with: apparmor_parser --skip-kernel-load12:35
zygacmatsuoka: I did not, let me look12:35
zyga18.04 but no freezer?12:36
zygacmatsuoka thanks again, I asked some questions that should clarify what's going on12:37
zygamvo: I can push that change if you don't mind12:38
cmatsuokazyga: thank you12:38
mvozyga: please do12:39
zygathank you12:39
mupPR snapd#9520 opened: tests: moving the core test suite from systems.sh to os.query tool <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9520>12:41
zygamvo changed locally, will test in private before pushing to avoid spread churn12:42
mvozyga: this is 9516 ?12:43
mvozyga: it's also failing on !ubuntu12:43
mvozyga: or is this what you are talking about?12:43
zygamvo, oh, anything with !apparmor?12:44
cmatsuokazyga: and this one is also related to livepatch, but it was diagnosed as snapd being unable to launch the process in a sandbox: https://bugs.launchpad.net/snapd/+bug/189847312:45
mupBug #1898473: canonical-livepatch snap not working <bot-comment> <Canonical Live Patch Client:Incomplete> <snapd:New> <https://launchpad.net/bugs/1898473>12:45
zygacmatsuoka looking12:46
mborzeckiquick restart12:48
mvozyga: yes, it's all very red, I have not yet looked at every one of them but on centos it's something like "Kernel needs AppArmor compatiblity patch)12:48
mvozyga: i.e. the regression test is failing12:48
zygamvo: I bet that's exactly what this is12:48
zygamvo: my change should fix this, I'll make sure it passes12:48
mvozyga: cool, did you push it yet?12:49
zygano, just waiting for this iteration to see that it fixes fedora3212:49
zygaI will push it if so12:49
mvozyga: ok12:49
zygaijohnson: have a look at the bottom of https://manpages.debian.org/stretch/systemd/systemd-fsck@.service.8.en.html12:52
zygaijohnson: the defaults are safe12:52
zygaijohnson but I was wondering if recovery mode should have different set?12:53
zygaijohnson e.g. fsck.repair=yes12:53
zygacmatsuoka replied as well12:54
cmatsuokazyga: thanks!12:55
zygathank you for bringing those to my attention :)12:55
zygamvo: oh well, it passes but then fails on AppArmor parser error for /snap/test-snapd-docker-support-app/x1/test-snapd-docker-support.profile in /snap/test-snapd-docker-support-app/x1/test-snapd-docker-support.profile at line 1: Could not open 'tunables/global'12:56
zygaI wonder what's missing there12:56
zygait looks like a real problem12:56
zygamvo: it fails because /etc/apparmor.d is missing on fedora, obviously :|12:57
zygahmm hmm12:57
mvozyga: meh12:59
zygamvo: https://pastebin.ubuntu.com/p/6dfFSMjPyp/12:59
zygamvo so now we have a choice12:59
zygamvo: accept that multipass and docker do not work on fedora13:00
zyga(unless they handle this error internally)13:00
zygaor create those directories13:00
zygaor ... not sure13:00
zygago back to snap-update-ns13:00
zygaand create more ideas13:00
zygastandup time anyway13:00
mupPR snapd#9508 closed: tests: remove ausearch which fails during prepare <Created by sergiocazzolato> <Closed by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9508>13:11
zygamvo: pushed13:12
mvozyga: ta13:12
zygapedronis: to get a non-inherited keyring you need to13:20
zyga        keyring = keyctl(KEYCTL_JOIN_SESSION_KEYRING, 0, 0, 0, 0);13:21
zygathen for shared keyring:13:21
zyga                if (keyctl(KEYCTL_LINK,13:21
zyga                           KEY_SPEC_USER_KEYRING,13:21
zyga                           KEY_SPEC_SESSION_KEYRING, 0, 0) < 0) {13:21
zygathat's all13:21
zygakeyctl is a syscall13:21
pstolowskizyga: does it look sensible https://github.com/lxc/lxd-pkg-snap/blob/latest-edge/snapcraft/hooks/remove ?13:38
zygapstolowski: it's unmounted inside the mount ns :)13:41
zygait's not leaving the ns, is it?13:41
zygaso it's not unmounted on the host13:41
pstolowskizyga: but umount on the host works; is that expected?13:42
zygapstolowski: one sec13:43
zygapstolowski: well some more time please13:46
zygapstolowski but the point is13:46
zygapstolowski: the propagation is not letting that unmount event reach the host mount namespace13:46
zygapstolowski so it is really unmounted, from the hook point of view13:46
zygapstolowski: but it stays mounted from ours13:46
zygapstolowski: presumably because the mount is created by something that escapes the mount namespace13:46
zygapstolowski: but removed by something that does not13:47
zygapstolowski the fix is really simple13:47
zygapstolowski: prefix the unmount with nsenter -m/proc/pid/1/ns/mnt13:47
zygathat should be enough13:47
zygabrb  though13:47
zygapstolowski if you want I can explain in a HO13:51
pstolowskizyga: yes please13:51
zygain standup13:51
mupPR snapd#9517 closed: spread, tests: tweaks for openSUSE <Simple 😃> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9517>13:56
mborzeckiuh, didn't notice there's only 1 review there, but it seems to work fine13:57
mupPR pc-amd64-gadget#51 opened: gadget: add ubuntu-save <Created by bboozzoo> <https://github.com/snapcore/pc-amd64-gadget/pull/51>14:03
zygaI'll go and debug fsck but first LUNCH as I'm starving14:04
zygamvo: pushed to https://github.com/snapcore/snapd/pull/951614:27
mupPR #9516: cmd/snap-confine: mask host's apparmor config <Squash-merge> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9516>14:27
mvozyga: thank you14:28
mvozyga: 9516 has some strange unit test error that looks like a fluke - okay if I restart?14:50
zygayes please14:50
zygaI tried to cancel things14:50
zygabecause of gpg error14:50
zygabut I think cancel is not reliable with spread14:50
zygaI just looked at it a moment ago14:50
zygamaybe cancel does work but takes a while to finish?14:50
mvozyga: yeah, but spread should only run if the unit tests were ok14:50
zygaoh, indeed14:50
zygamaybe it's not spread but all of canceling :)14:51
mvozyga: sure, no worries - I looked at 9499 and it looks like we can almost merge it?14:59
zygayes, except it doesn't work :)15:00
zygaI mean core20 and core16 fail15:00
zygawe could mask those failures but yeah15:00
zygaapart from that it's good to go15:00
zygashall I mask them?15:00
zygaI'm looking at 16 failure now15:00
zygaas it, in theory at least, should work15:00
zygaI can mask 16 and 20 quickly and push that15:01
zyga20 will be fixed with ijohnson's patch15:01
ijohnsonyes sorry will push later, need to go afk for a bit now actually15:01
zygaijohnson that's totally fine15:01
mupPR snapd#9518 closed: tests: add value to INSTANCE_KEY/regular <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9518>15:01
zygathanks mvo!15:01
zygamvo: just let me know what we should do in your opinion, I'll make coffee and return here to push this forward15:02
zygaI must be missing something as, at least on paper 16 should be good, it hast to be something silly in the integration path15:02
* zyga -> coffee15:02
mvozyga, ijohnson no worries, let's not mask and simply wait for fixes15:02
mvoijohnson: and no worries, no real rush at this point15:03
ijohnsonwill definitely get done today15:03
mvozyga: 16 is surprising15:03
mupPR snapcraft#3321 opened: meta: add error check for "command not found" <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3321>15:07
zygamade tea instead15:11
zygabut still good :)15:11
zygamvo: yeah, I guess, "gulp" but yeah15:11
* zyga digs15:11
zygamvo: do you think we could have a security review of https://github.com/snapcore/snapd/pull/951615:12
mupPR #9516: cmd/snap-confine: mask host's apparmor config <Squash-merge> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9516>15:12
zygaeven a brief one?15:12
zygajdstrand: I know you are extremely busy lately, do you think you could look at the C changes in ^ and comment15:13
mvozyga: sure, if you feel it's important. time is of the essence though, the release team is waiting for this15:13
zygaalex +1 the design already15:13
zygamvo: I understand, I hope we can merge this today15:13
zygait should be green on this pass15:13
pstolowskifun fuct, all our retry strategies share the same problem as download (maybe except for concurrent access)15:16
mupPR snapd#9521 opened: tests: moving main suite from systems.sh to os.query tool <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9521>15:16
zygapstolowski that's a fantastic find ahead of core2015:16
* cachio lunch15:19
mvopstolowski: !!!15:35
mvopstolowski: thanks for finding this issue15:35
pstolowskimvo: probably my bug, and 3-4 years old!15:36
mvozyga: are you looking into the fsck failure on core16 ? if not I'm trying to look at this now15:36
mvopstolowski: yeah15:36
pstolowskimaybe except for download15:36
zygayes I am15:36
zygaI can share what I have15:36
pstolowskithat one may be new15:36
zygaI've enabled persistent journal15:36
zygaran the test with debug15:36
zygaI'll have the improved log soon15:36
pstolowskistgraber: hey15:36
mupPR snapd#9317 closed: [RFC] devicestate: keep log from install-mode on installed system <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/9317>15:37
zygamvo found an embarrassing typo, iterating15:44
mvozyga: fun, I just had the "NESTED_TYPE" error you had15:45
zygaI haven't seen that in ages15:46
zygamore info on fsck in 15 minutes15:46
zygamvo good news15:55
zygait may be okay15:55
zygaI'll know in 30 minutes15:55
zygaafk for now15:55
stgraberpstolowski: hi16:05
pstolowskistgraber: hey, see my message on mattermost16:06
zygacore16 passes16:07
mvozyga: nice! what was it?16:07
zygamvo: it was mounted twice16:08
zygaI really didn't look deeper16:08
zygabut now core20 is the only failure16:08
zygaijohnson if you fix core20, we're good16:09
mupPR snapd#9512 closed: o/snapstate: move setting updated SnapState after error paths <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9512>16:12
zygamvo: I pushed one, last I hope, thing to https://github.com/snapcore/snapd/pull/951616:17
mupPR #9516: cmd/snap-confine: mask host's apparmor config <Squash-merge> <⚠ Critical> <Created by zyga> <https://github.com/snapcore/snapd/pull/9516>16:17
ijohnsonzyga: mvo: I've got the fix for uc20, just waiting on spread locally before pushing it up17:05
ijohnson(assuming there are no other gotchas that pop up but seems to be straight forward I think)17:06
=== ijohnson is now known as ijohnson|lunch
=== ijohnson|lunch is now known as ijohnson
mvoijohnson: \o/ thank so much17:51
ijohnsonone more try on the spread test, first one had an issue unmounting the snapd snap17:54
mupPR snapcraft#3312 closed: spread tests: introduce electron-builder test <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3312>18:13
mupPR snapcraft#3322 opened: package repositories: drop $SNAPCRAFT_APT_HOST_ARCH variable <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3322>18:13
mupPR snapcraft#3315 closed: build(deps-dev): bump junit from 3.8.1 to 4.13.1 in /tests/spread/plugins/v1/maven/snaps/maven-hello/my-app <dependencies> <java> <Created by dependabot[bot]> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3315>18:23
mupPR snapcraft#3316 closed: build(deps-dev): bump junit from 3.8.1 to 4.13.1 in /tests/spread/plugins/v1/maven/snaps/legacy-maven-hello/my-app <dependencies> <java> <Created by dependabot[bot]> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3316>18:23
mupPR snapd#9516 closed: cmd/snap-confine: mask host's apparmor config <Squash-merge> <⚠ Critical> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9516>18:27
mvoI'm off but please telegram me if anything comes up about the snapd groovy fix that I just uploaded there18:29
* ijohnson coffees quickly19:01
mupPR snapd#9520 closed: tests: moving the core test suite from systems.sh to os.query tool <Simple 😃> <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9520>19:48
mupPR snapd#9521 closed: tests: moving main suite from systems.sh to os.query tool <Simple 😃> <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9521>19:48
mupPR snapcraft#3307 closed: setup.py: assert with helpful error when unable to determine version <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3307>20:03
* ijohnson EODs23:09

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!