=== WhatsGoingOn is now known as iMadper [05:08] morning [05:47] Hi [05:47] mborzecki: voting is over [05:48] zyga: yeah, it is [05:48] Could be better, could be worse [05:48] zyga: though, we have domestic round in october right? :) [05:48] Too bad 70% of young people did not vote [05:49] zyga: tbh, wasn't much to choose from [05:49] Yes, that will be critical [05:49] zyga: same faces each time [05:49] Eh [05:49] Yes, that is true [05:50] One more kid to send to school, ttyl [05:52] zyga: at least konfederacja is outside, don't think they need more lunatics in brussels [06:13] Iā€™m happy to see wiosna, it means we are not all crazy yet [06:26] back in the office now [06:26] ok, time to set everything else aside [06:26] and look at initramfs [06:26] mborzecki: ping me for reviews [06:26] mborzecki: if you ever want a puzzle to solve https://github.com/snapcore/snapd/pull/6891 is critical for .1 [06:26] PR #6891: many: make per-snap mount namespace MS_SHARED [06:26] and has exactly one apparmor denial in one test on one system!!! [06:26] and I'm out of ideas why [06:36] zyga: on 14.04? [06:39] correct [06:40] on 14.04 only [06:40] 4.4 kernel [06:40] hey mborzecki and zyga - good morning! [06:40] and, to my looks, the denial is bogus because we have that rule [06:40] mvo: good morning! [06:40] mborzecki: I didn't look, at the time, about environmental differences, like /tmp tmpfs vs ext4 [06:40] mborzecki: I know that a bare "mount," rule fixes it [06:40] and the denial was on flags [06:40] perhaps there's a bug on 14.04 parser [06:41] the bad thing is that apparmor blob format is opaque, I wrote some tools to disassemble it a while ago but I didn't manage to crack the essential part [06:41] mvo: hey [06:41] the encoding of the state engine transition tables [06:41] those are highly compressed and optimized [06:41] and I just didn't understand the kernel code that walks over them [06:41] there's no documentation that helps that I could find [06:41] mvo: hey [06:41] mvo: some bad news [06:42] mvo: the fix for the bug is blocked [06:42] I'm happy to HO to discuss this quickly [06:42] zyga: have you reached out to jdstrand_ or jjohansen maybe? [06:42] mborzecki: jj no, jdstrand yes [06:42] maybe it's like a known issue or sth :) [06:42] we talked about this on friday, no effect [06:42] nope [06:42] zyga: hm, ok - is there a tl;dr summary? [06:44] mvo: a single test fails, only on 14.04, it makes no sense: https://github.com/snapcore/snapd/pull/6891#issuecomment-495643768 [06:44] PR #6891: many: make per-snap mount namespace MS_SHARED [06:44] mvo: we get a single apparmor denial for a rule we definitely hold [06:44] mvo: requires jumps to kernel to debug [06:46] zyga: given that 14.04 is EOL I'm not sure we should block things. how bad is the denial? [06:46] mvo: snap-confine doesn't work [06:46] all snaps fail [06:46] it's not great [06:46] zyga: :( [06:46] see [06:46] zyga: it does not work *at all* ? [06:46] yes, it stops early on a mount permission and dies [06:47] mvo: I added "mount," rule and that fixes it [06:47] PR snapd#6915 opened: spread: enable Fedora 30 (2.39) [06:47] but as soon as I try to express the arguments used by the call it is failing again [06:48] mvo: perhaps I missed something, it was late on friday [06:48] mvo: fresh pair of eyes (or even one) appreciated [06:48] zyga: what PR is it? [06:48] the one linked above, 6891 [06:49] mvo: AFAIR we fail on line https://github.com/snapcore/snapd/pull/6891/files#diff-af477950316a096b57d91c74478bc4d2R252 which is handled by this rule https://github.com/snapcore/snapd/pull/6891/files#diff-798ce6f0668878eda67847b4ab492745R150 [06:49] PR #6891: many: make per-snap mount namespace MS_SHARED [06:49] but again, perhaps I missed something [06:50] but suspicious that it is only 14.04 [06:50] other systems pass this test [06:50] zyga: looking [06:50] thank you! [06:51] mvo: note: failed flags match error says that apparmor found the rule for the mount path, but not for the flags [06:51] PR snapd#6914 closed: tests: change strace parameters on snap-run test to avoid the test gets stuck [06:51] that's very suspect IMO [06:51] flags are "rw, rshared" in the denial [06:52] anyway, back to initramfs [06:52] please ping me if you find anything [06:56] we should also look at "settle is not converging" bug [06:56] it is 100% reproducible in packaging builds [06:56] PR snapd#6916 opened: cmd/snap-confine, tests: tweak comments, reenable symlink check in RHBZ 1584461 regression [06:56] something fishy [06:57] PR snapd#6895 closed: cmd/snap-confine, data/selinux: cherry pick Fedora 30 fixes to 2.39 [06:57] mborzecki: do you have the comments in 6874 on your radar? the post-merge ones from jamie? [06:59] mvo: yup, opened #6916 [06:59] PR #6916: cmd/snap-confine, tests: tweak comments, reenable symlink check in RHBZ 1584461 regression [06:59] mborzecki: \o/ thank you [06:59] it's even reviewed already :') [07:00] gnome shell bug where background doesn't render drives me crazy [07:00] quality of the linux desktop has never been thix mised [07:00] *mixed [07:00] woah [07:00] * mvo hugs zyga for already reviewing it [07:00] on one hand side it's really the golden age where hardware works great and there's lots of polish [07:01] on the other hand we're building a desktop shell in javascript and running it ends with stream of crap javascript errors [07:01] and this is cross dirstro: suse, ubuntu - all broken === pstolowski|Afk is now known as pstolowski [07:01] morning! [07:01] I'm afraid to update fedora ( [07:01] hey pawel, good morning, welcome to our new right-wing world [07:03] huh, suse update resulted in EFI mok enroll? [07:03] (with an opensuse key) [07:03] pstolowski: hey [07:03] zyga: background doesn't render? [07:03] yep [07:04] zyga: how so? [07:04] https://www.irccloud.com/pastebin/Sjl5oiPM/ [07:04] like this [07:04] if you google for the "tweener" and some other messages it's a pretty widespread problem [07:05] doesn't *for whatever reason* happen on wayland [07:05] happens 100% on X11 on all my up-to-date distros [07:05] must be the new shell [07:05] 18.04 is ok [07:05] the key is May 27 09:04:22 fyke gnome-shell[3767]: Object Meta.Background (0x5584c4024190), has been already deallocated ā€” impossible to access it. This might be caused by the object having been destroyed from C code using something such as destroy(), dispose(), or remove() vfuncs. [07:05] nothing like working on bright white background in the morning [07:06] oh, suse update just fixed it [07:07] zyga: hm, that's been fixed afaik [07:07] right [07:07] QA [07:08] zyga: i think you also need to have some specific extensions to trigger that [07:08] 100% vanilla [07:08] but anyway, even if that is true [07:08] do you recall something this silly in any old desktops? [07:08] I mean, ever? [07:09] hmmm, let me think, gnome panel going crazy was rather common [07:09] kde crashed a lot too [07:09] so now we traded crashes to javascript errors on mouse motion [07:09] PR snapd#6835 closed: snapstate: allow removal of non-model kernels [07:09] guess that's just inevitabale ;) [07:09] zyga: it's called progress :P [07:09] zyga: at least it's not an electron app [07:10] yet! [07:10] Download snap "snapd-hacker-toolbelt" (26) from channel "stable" (received an unexpected http response code (408) when trying to download https://api.snapcraft.io/api/v1/snaps/download/FMONi3pH7TfSv15FusziadAGCjQ6t4EG_26.snap) [07:11] hm, do we retry on 408? it seems we should [07:25] mvo: hi, I made a comment after it was merged on #6835 [07:25] PR #6835: snapstate: allow removal of non-model kernels [07:26] pedronis: thank you! I will do a followup with your comments and refactor this code a bit [07:30] zyga: there is some funny stuff happening in the VM with the fix for 6891 - the test failed on 14.04, I ran it manually and it failed. I ran it again and now its not failing anymore [07:30] !??! [07:30] whaat [07:30] zyga: yeah, quite puzzling [07:30] can you discard and re-run [07:30] does it fail? [07:30] zyga: I'm looking at the profiles not etc [07:30] I mean, it seems to fail on just construction [07:30] zyga: sure, one sec [07:30] so discarding and running that hello command should be enough [07:30] got to go to school to pick up my son, he's not feeling too well, back in a bit [07:31] zyga: just looking at the timestamp of the apparmor profile to double check nothing has changed [07:31] mvo: remember about reexec, are you editing the right profile? [07:31] mborzecki: o/ [07:32] zyga: I did not edit anything so far and tried "SNAP_REXEC=0|1" without any difference this is why I'm puzzled :) [07:33] zyga: aha, now its consistently failing again, but I need to set "SNAP_REEXEC=1 ..." [07:33] indeed, that's a good find though [07:33] we repackage, right? [07:34] so reexec vs not should not matter [07:34] zyga: let me compare the profiles [07:36] zyga: hm, so /var/lib/snapd/apparmor/snap-confine.snapd.x1 seems to miss bits, i.e. the rshared bits that got added [07:37] zyga: it looks like the profile is outdated [07:37] hmmmmm [07:37] that's weird [07:37] repackaging is broken? [07:37] zyga: which of course raises the question - why on 14.04 only? [07:37] exactly! [07:38] zyga: oh, maybe because we have some strange if 14.04 in the prepare code :( [07:38] zyga: let me look [07:38] some tabs-vs-spaces in prepare-restore.sh [07:40] mvo: I don't see any smoking guns, looking at delta in packaging/ [07:41] zyga: let me poke at this, I have an idea [07:41] mvo: there's a difference wrt .real vs non profile [07:41] maybe what we are hitting is a bug in snapd + packaging [07:42] 14.04 doesn't have the .real suffix [07:42] mvo: we should drop the .real suffix in 19.10 [07:42] mvo: I'll keep you to it, thank you for looking and for the insight [07:43] I'll resume initramfs poking [07:43] zyga: thank you! [07:43] * zyga hugs mvo! :) [07:43] zyga: yeah, let me poke for 5min and hopefully I get an idea [07:43] zyga: no sense in duplicating the effort [08:00] PR pc-amd64-gadget#10 closed: Add mmx64.efi (MokManager) to support mokutil [08:00] PR pc-amd64-gadget#11 closed: Add mmx64.efi (MokManager) to support mokutil [08:00] PR pc-amd64-gadget#14 closed: gadget.yaml: add system-recovery partition [08:01] PR pc-amd64-gadget#10 opened: Add mmx64.efi (MokManager) to support mokutil [08:01] PR pc-amd64-gadget#11 opened: Add mmx64.efi (MokManager) to support mokutil [08:01] PR pc-amd64-gadget#14 opened: gadget.yaml: add system-recovery partition [08:02] re [08:27] PR snapd#6917 opened: Add endpoint for snap download in the daemon [08:30] zyga: i think i can split #6890 [08:30] PR #6890: gadget: mounted filesystem writer & updater [08:49] mborzecki: in a call [08:58] re [08:58] mborzecki: that's neat, thank you, I will be looking at gadget reviews all week; please let me know which one to start with [08:59] * as long as it's not the 2K one [08:59] zyga: haha :) [09:01] zyga: I am running a final test now on 6891 now, if its green I push a 2 line fix in the test setup to it (if you don't mind) [09:01] mvo: how could I mind :D [09:01] mvo: thank you so much [09:14] zyga: worked locally, pushed now [09:15] \o/ [09:15] thank you! [09:18] brb, need to run an errand at school, 30min [09:23] * pstolowski needs to run an errand, bb in ~1h [09:45] back now [09:49] PR snapd#6918 opened: snaptest: add helper for mocking snap with contents [09:49] a really simple one ^^ [10:08] mborzecki: don't we have something like that already? [10:08] mborzecki: +1 on the patch but perhaps look at the tree, I'm pretty sure we have ad-hoc implementations [10:08] that could be reduced [10:08] zyga: afaik no, we have something that packs a *.snap, but that's not what i'm looking for [10:09] I mean there are bits that drop files on disk, along with a meta/snap.yaml [10:09] then parse the yaml and return that [10:09] we have way too many helpers like that [10:09] it'd be great if all such helpers *had to* use snaptest [10:11] we did a pass of reducing that afaik, that's were the helper are coming in the first place, might still be some [10:11] zyga: to be precise, i don't see anything similar in snaptest, there's bits in random tests that do ioutil.WriteFile [10:11] *where [10:11] yeah [10:11] that's what I mean [10:12] zyga: tbh, this is pulled from #6750 which.. introduces such helper locally :) [10:12] PR #6750: overlord/devicestate: update-gadget task handler with stubbed gadget callbacks [10:14] zyga: mborzecki: I don't think it should be a blocker either way, the test grew organically, also if all it's inolved is a couple WriteFile, it's unclear if the helper are a big win or not [10:14] +1 [10:24] mvo: https://github.com/snapcore/snapd/pull/6891 is green! [10:24] PR #6891: many: make per-snap mount namespace MS_SHARED [10:26] zyga: yay! once its in we need to make sure we have a 2.39 PR too [10:26] zyga: but we can discuss in the standup [10:26] zyga: maybe we add this in .40 only [10:27] mvo: yeah, let's review it first! [10:27] pedronis: about https://github.com/snapcore/snapd/pull/6750/files/34e6a2ba202c127efa934e72d2cd6f57d429a8d1#r281945201 you're thinking some operation log for later auditing? [10:27] PR #6750: overlord/devicestate: update-gadget task handler with stubbed gadget callbacks [10:52] brr [10:52] brb :) [10:58] re [11:09] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/6918/ [11:09] PR #6918: snaptest: add helper for mocking snap with contents [11:09] k [11:09] pstolowski: thanks! [11:20] mborzecki: yes, and also to help debugging [11:28] PR snapd#6918 closed: snaptest: add helper for mocking snap with contents === ricab is now known as ricab|lunch [11:50] no space left on device keeps breaking the builds from time to time: https://paste.ubuntu.com/p/TjtdxMmKzB/ [12:10] mvo: https://github.com/snapcore/snapd/pull/6899 needs de-conflicting [12:10] PR #6899: image: make prepare-image recovery-system aware [12:12] pedronis: also, remodelling PRs have conflicts [12:13] zyga: hey, what's the status of https://github.com/snapcore/snapd/pull/6347 ? [12:13] PR #6347: many: allow snap-update-ns to write user mount profile [12:15] mvo: pstolowski: I applied some of the feedback to #6838 [12:15] PR #6838: overlord/devicestate: introduce remodel kinds and contexts [12:18] thanks, i'll finish this review [12:19] pstolowski: hey, a little bit on hold this week, I mergef fmaster into them on Friday but I need a moment to iterate towards something reviewable [12:21] zyga: k [12:46] pedronis: 2 small questions to the PR [12:50] PR snapd#6916 closed: cmd/snap-confine, tests: tweak comments, reenable symlink check in RHBZ 1584461 regression === ricab|lunch is now known as ricab [13:35] mvo: what was the 19.10 low-hanging fruit you mentioned in the standup? [13:37] šŸŽ šŸŒ šŸ‡ [13:41] pstolowski: --explain but it needs some design, so I don't think it's that low hanging [13:42] * zyga goes for lunch [13:47] pedronis: i see [13:52] pstolowski: what is the status of the slot-snap-type changes? and of fixing the content bug in a more general way? [14:04] pedronis: still needs work, i need to return to that branch. as mentioned before some i found a few interfaces problematic [14:04] pstolowski: I probably need to understand the problem to help [14:06] pedronis: i'll check that branch and summarize then issue(s), then get back to you [14:19] * zyga finished lunch, thinking about either taking a short break and walk or getting coffeee [14:19] after that, grub.cfg hacking :) [14:19] and some more fun in initramfs [14:25] * cachio afk [14:32] pstolowski: thx [15:02] sil2100: do you think you can look at https://github.com/snapcore/pc-amd64-gadget/pull/14 ? note that it will only be in the new "20" branch (which is only used for experimental UC20 images) so very low risk [15:02] PR pc-amd64-gadget#14: gadget.yaml: add system-recovery partition [15:07] mvo: hey! Sure thing - I'll have a quick fix for core18 PR'ed soon, can I poke you for a review of that one in return? ;) [15:08] sil2100: sure [15:14] PR core18#130 opened: gpg (dirmngr actually) panics when there's no random/urandom [15:17] mvo: ^ that's the PR I was talking about, just pushed the latest version. Let me look at your PR now o/ [15:17] (I'll have to look why it suddenly stopped working though, out of curiosity) [15:20] sil2100: yeah, I would love to figure out why it stops working, maybe a snapcraft change? [15:25] sil2100: so you have a link to the failure without the pr 130? [15:25] PR #130: Basic kernel/os handling [15:26] mvo: you can reproduce it locally even, so I just checked and the reason is that now in our ubuntu-base tarballs our /dev directory is empty [15:26] mvo: previously we had all the /dev/random and /dev/urandom shipped in the tarball [15:26] Need to check if that's intentional that they're gone [15:27] mvo: (as for failures, you can see them on LP as well: https://launchpad.net/~ubuntu-core-service/+snap/core18) [15:27] But as said, it's just that the base tarball stopped shipping those - investigating why now [15:33] sil2100: aha, nice. thanks for digging into the root cause [15:39] mvo: ok, I guess this is due to livecd-rootfs 2.525.23 ;/ Apparently this change was made for docker, will have to ask mwhudson a bit about this one then [15:39] "Backport two minimizations for the docker images: remove apt lists that are removed downstream anyway, and remove device nodes from the image. (LP: #1828118)" [15:39] Bug #1828118: docker tarballs contain /dev/null Disco):Fix Released> [15:41] mvo: I guess this might be an architectual question what we should actually expect having in the ubuntu-base tarball [15:42] Maybe I should add some conditionals checking for the existance of these files and only then create/delete them [15:42] Actually, wonder what happening in the end snap [15:56] * cachio lunch [16:02] Ok, images with the snap work - but still, let me bring that up with Steve [16:06] sil2100: ta [16:07] sil2100: yeah, the change itself looks fine but I'm a bit worried it might have unintended side-effects === pstolowski is now known as pstolowski|afk [16:37] PR core18#130 closed: gpg (dirmngr actually) panics when there's no random/urandom [17:08] * zyga was going through some core20 ideas during the walk [17:08] now shower and more work :) [19:31] * cachio afk [19:44] PR snapd#6915 closed: spread: enable Fedora 30 (2.39) [20:00] PR snapd#6838 closed: overlord/devicestate: introduce remodel kinds and contexts [21:07] PR snapd#6919 opened: cmd/okay: Remove err message when warning file not exist [21:10] wait, ubuntu-base:minimized builds are used for core18??