[05:50] morning [06:19] mborzecki: good morning [06:21] zyga: hey [06:39] deconflicted https://github.com/snapcore/snapd/pull/8464 [06:39] PR #8464: cmd/snap-boostrap, boot: use /run/mnt/data instead of ubuntu-data <⛔ Blocked> [06:39] otherwise out doing errands, i'll be back for sprint meetings [06:41] oh and install/re(install)/recover worked with pc_98.snapm, pc-kernel_473.snap and snapd from edge [06:41] Good luck and see you later mborzecki [07:02] morning [07:02] hey pawel [07:03] this will be a quiet morning [07:09] yep [07:16] PR snapd#8436 closed: configcore,tests: use daemon-reexec to apply watchdog config <⛔ Blocked> [07:17] PR snapd#8436 opened: configcore,tests: use daemon-reexec to apply watchdog config <⛔ Blocked> [07:32] pedronis: good morning [07:32] pedronis: I will write the details of the inhibit lock idea and share the link with you [07:34] ok [07:36] PR snapd#8562 opened: store: implement DownloadAssertions [08:09] pedronis: https://forum.snapcraft.io/t/wip-refresh-app-awareness/10736/33 [08:09] * zyga AFK for some time [08:13] actually no [08:16] store is still responding with 429 - too many requests - sometimes [08:26] PR snapd#8563 opened: asserts/internal: introduce Grouping and Groupings [08:29] PR snapd#8564 opened: asserts: introduce Pool [09:13] lots of red again afaict [09:16] pedronis: do you have an example of a recent run with reds? [09:16] pedronis: what PR should I look at? [09:17] mvo: here for example: https://github.com/snapcore/snapd/pull/8436 I suppose it's the store but haven't looked at the logs [09:17] PR #8436: configcore,tests: use daemon-reexec to apply watchdog config <⛔ Blocked> [09:18] pedronis: yeah, anything that fetches core runs into timeouts it seems [09:19] I saw store returning errors [09:19] 429 [09:22] PR snapd#8565 opened: osutil: expand FileLock to support shared locks and more [09:32] re (kind of) [09:46] mvo: pedronis: any high priority 2.45 PRs that need reviews/fixes? [10:05] PR snapd#8566 opened: cmd/cmdutil: add run inhibition operations [10:12] sil2100: hey! do you have a moment to take a look at https://github.com/snapcore/core-build/pull/59 ? [10:12] PR core-build#59: initramfs: add new handle_writable_defaults() [10:13] pstolowski: hey! Is this the same as for core20? [10:15] sil2100: no, core20 is separate (https://github.com/snapcore/core20/pull/46, which you commented on) [10:15] PR core20#46: static: add new handle_writable_defaults() to handle-writable-paths [10:29] PR snapd#8567 opened: o/devicestate: core20 early config from gadget defaults === msalvatore_ is now known as msalvatore [11:32] pstolowski: yeah, I guess I'll have the same weird question here, since the code is the same - can someone explain the meaning of the -L check in that particular conditional? [11:39] sil2100: I replied in the PR, this is to ensure that broken symlinks will still get created [11:40] sil2100: as -e will follow the symlink target and return false if there is a symlink but it point's to the void [11:40] sil2100: hence we need -L in addition, there is a "unit" test for this as well [11:44] PR core#112 opened: Fix English in comment around usage of spellcheck [11:54] PR snapd#8568 opened: asserts: rest of the Pool API [11:56] PR snapd#8569 opened: o/assertstate,asserts: use bulk refresh to refresh snap-declartions [12:14] PR core#112 closed: Fix English in comment around usage of spellcheck [13:01] zyga: is there no SU today? [13:02] hey [13:02] I don't think so [13:02] but we can meet if you have something to share [13:02] pstolowski: ^ ? [13:02] I have some cool stuff [13:02] my update is just UC20 UC20 UC20 [13:02] https://meet.google.com/rni-xhsz-uqt [13:02] :) [13:03] zyga: ijohnson hi, i'm in the opening plenary [13:03] pstolowski: can you share the link in private? [13:03] perhaps when it's over? [13:03] ijohnson: join and I can share some fun stuff [13:04] sure [13:04] zyga: i can, it's from the calendar you gave me in the morning ;) [13:11] PR snapd#8570 opened: many: allow using ~/.snapdata instead of ~/snap [13:13] Is it necessary for every snap app to spam the journal with these messages:- Apr 27 14:11:21 zion2 audit[359771]: AVC apparmor="DENIED" operation="connect" profile="snap.mysql-workbench-community.mysql-workbench-community" name="/run/uuidd/request" pid=359771 comm="mysql-workbench" requested_mask="w" denied_mask="w" fsuid=1000 ouid=0 [13:16] Gargoyle: thanks for reproting, afaik access to /run/uuidd/request was part of one specific interface, but since 2.45 (to be released soon) is aprt of the base template [13:17] Seems every snap I've installed (clean install of 20.04 a few days ago) is reporting the same message over and over. [13:18] the next snapd update will fix this as mborzecki said ... [13:18] zyga, ijohnson standup? [13:18] pstolowski: we are in zyga link above [13:19] Ahh. OK [13:19] Gargoyle: yeah, seems like a popular way to obtain uuids (idk why /proc/sys/kernel/random/uuid isn't as popular and does not require a separate daemon) [13:20] pstolowski: can you join? [13:20] ijohnson, zyga 4 people are in regular standup HO [13:20] oh [13:20] I didn't get the link [13:21] you guys are entertaining today ! [13:59] ijohnson, pstolowski, cachio: I just ran another test here and it works well before the refresh, but it fails at the-tool in the post-refresh boot [13:59] I'll try it non-encrypted to be able to see what's going on [13:59] cmatsuoka: I assume that's with edge everything and remastered kernel snap ? [14:00] ijohnson: it's core20 602, gadget 96, remastered kernel 470 [14:01] ijohnson: did something important change in core20 or gadget? [14:01] I mean, core20 or kernel? [14:01] well the kernel was revved over the weekend, I see rev 473 on edge now [14:01] cmatsuoka, here it is still running [14:02] * cmatsuoka updates kernel [14:05] * zyga -> lunch [14:05] cmatsuoka: what about nested/basic core20 test? does it work [14:10] pstolowski, I am running these tests now [14:10] I'll tell you [14:10] cachio: ok, ty [14:11] pstolowski, I'll let you know [14:15] store is very, very slow right now [14:16] last run I got kill timeout downloing the kernel snap [14:24] re [14:24] cmatsuoka: yes, painfully slow :/ [14:28] release effect stilll? [14:30] mborzecki: compound effect, more than one issue [14:38] I've updated https://github.com/snapcore/snapd/pull/8570 with some more expectations [14:38] PR #8570: many: allow using ~/.snapdata instead of ~/snap [14:43] ijohnson: apparently the problem was caused by a local change that I thought it would be totally harmless [14:44] cmatsuoka: good to know, what was the change? [14:44] ijohnson: experimentations moving the snap-bootstrap bootstrap and partition packages to gadget [14:45] ah I see [14:48] cmatsuoka: that needs a bit of design btw, it's not just mechanical [14:49] there is some semi-duplication between bits of gadget and partition [14:49] that is not super clear what to do about [14:49] pedronis: yes, I noticed that it wouldn't be a simple move [14:50] cmatsuoka: maybe you can have a chat with mborzecki, he might have input on that [14:51] there's an unclear relation between the bits that existed for snap-image [14:51] and the stuff for snap-bootstrap [14:51] pedronis: the other thing I was going to examine was to integrate the new helpers from secboot [14:51] pedronis: perhaps that would be a more productive thing to do right now? [14:52] cmatsuoka: it might be easier, if you look at that my comments on the measure model PRs, there are a couple of helper in snap-boostrap [14:52] that belong in secboot I think [14:53] cmatsuoka: a question is a bit whether we want them as functions or we want some kind of wrapper around a tpm connection [14:53] that can be set up even if there's no tpm [14:53] pedronis: I'll have a look [14:53] but then the functionaliy/some functionality becomes NOPs === xnox is now known as xnox|sessions [15:09] pstolowski, tests are failing [15:09] pstolowski, can't validate the fix [15:10] cachio: kill timeouts? or store? [15:10] pstolowski, the store [15:10] getting kill timeout but downloading snaps [15:12] cachio: downloading snaps for me is incredibly slow right now [15:13] having slow snap downloads? kill it then restart. Also wait a bit, things are being tweaked right now [15:15] roadmr: I've killed it at least 5 times now and keep getting 503s eventually [15:15] then please wait a bit, things are being tweaked right now :) [15:15] sure [15:17] ijohnson, yes [15:18] ijohnson, it is making all the tests fail [15:40] is anyone upgrading to groovy yet? [15:45] hah [15:45] a [15:46] that's it, I'm moving to groovy ;) [15:47] the name conflicting with groovy is slightly unfortunate heh [15:48] "why do you move to groovy! it's java deep down inside,argh" [15:57] cachio, ijohnson : are things happier for you folks now? [15:57] roadmr: let me check now [15:59] roadmr, running tests, I'll let you know in few minutes [15:59] roadmr, thanks [15:59] well "snap download whatever" and if it doesn't 503 or be dog-slow then you're ok :) [16:00] roadmr: much much much much much much much better now [16:00] thank you! [16:00] I did nothing :) [16:00] (and the whole snapstore team in general too :-) ) [16:00] thanks ! [16:06] roadmr, seems to be working much better now [16:06] thanks for confirming cachio :) [16:16] degville: hey, not sure if you look at github notifications [16:17] degville: I've opened a small PR with two paragraphs of text [16:17] degville: I think the text I've used is terrible and I need some help [16:17] https://github.com/snapcore/snapd/pull/8571 [16:17] PR #8571: overlord/snapstate: warn of refresh/postpone events [16:17] the text is right up front in the diff [16:17] PR snapd#8571 opened: overlord/snapstate: warn of refresh/postpone events [16:17] in the tests below you can see how it would look like at runtime: https://github.com/snapcore/snapd/pull/8571/files#diff-5a74ab37983225c1b7fd74b7f20746e0R670 [16:18] zyga: thanks for letting me know! I'll take a look (I do try to keep on top of GH notifications) [16:18] if you have a moment I would love to get some better sentences instead [16:18] super, thank you :) [16:23] PR snapd#8572 opened: packaging: add the inhibit directory [16:34] PR snapd#8573 opened: overlord/snapstate: inhibit startup while unlinked [16:42] * zyga EODs [16:49] PR snapd#8560 closed: tests: disable "searching" test <⚠ Critical> <⛔ Blocked> === ergiusenss is now known as sergiusens [17:40] ijohnson: have you seen this error before? https://pasteboard.co/J5Mjrds.png [17:40] cmatsuoka: no sorry I don't have time to debug it right now [17:41] and also to be clear I haven't seen that error [17:41] thanks, just checking [18:25] cmatsuoka, still I see the smae issue https://paste.ubuntu.com/p/3C8Xg6WjcZ/ [18:26] cmatsuoka, at some point the message Press enter to configure. should appear [18:26] cmatsuoka, right? [18:26] I am runnign without the -cpu host [18:26] because it make qemu fail on google [18:26] cmatsuoka, should I try to find an alternative? [18:27] for -cpu host [18:27] I suspect yes, the default has other problems [18:31] cachio: checking the log... [18:31] I see this error. qemu-system-x86_64[62220]: qemu-system-x86_64: /build/qemu-74sXTC/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed. [18:32] when I use -cpu host [18:41] cachio: it's a bit unclear to me what could be happening there, I'm researching a little bit [18:43] thanks [18:44] jdstrand: hey, when you get a chance, could you please add `interface: home; read-all: true` to auto-connections for Multipass? [18:45] Saviq: why is that needed? (I need to document this) [18:45] jdstrand: to allow launching images coming from user's homes [18:46] jdstrand: we may avoid it long-term by amending our own profile runtime [18:46] to allow access to that one file (which could be outside of /home at that point) [18:47] Saviq: ok, thanks [18:48] cachio: debugging msr failures seems to be hard: https://bugs.launchpad.net/qemu/+bug/1661386 [18:48] Bug #1661386: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed [18:50] cachio, cmatsuoka: zyga came to us yesterday with a strangely similar issue, is this nested on VMWare Fusion? If not, please ignore :) [18:51] Saviq: it's nested but on gce [18:52] Saviq: I'm googling for similar reports and they're quite weird [18:53] Saviq: I think gce uses kvm [18:54] the last update to qemu on bionic was in February, so unlikely for that to be related http://changelogs.ubuntu.com/changelogs/pool/main/q/qemu/qemu_2.11+dfsg-1ubuntu7.23/changelog [18:55] Saviq: could you rebase on core 20? [18:56] I suspect is is the same security they paper over [18:56] Saviq: what we actually need is a qemu cpu with rdrand so we're using -cpu host, and doing that triggered the msr problem cachio reported [18:57] This is the same as I debugged [18:57] Dropping -cpu host fixes it for me [18:58] hmm, I see that vmware rejects pmu msr accesses [18:58] zyga: is your problem related to pmu? [18:59] zyga: if so -cpu host,pmu=off might help [19:00] I did not use pmu [19:55] cachio: I don't have any good idea on how to address that, I think it's better to open a bug for kernel reporting it and see if anyone suggests a flag to use or a workaround [19:55] cachio: I see similar problems reported for azure and other nested cases, but not this problem exactly [19:57] cachio: I mean, for qemu [19:59] cachio: not kernel [20:01] cmatsuoka, it is weird because it was working 2 weeks ago [20:01] then a change made that it didnt work any more [20:02] cachio: oh really? hmm, what could have changed? [20:03] cmatsuoka, not sure [20:03] last week I started to see that new behaviour [20:06] cachio: can we try an older image to see if it still works? [20:09] no [20:09] cmatsuoka, we build the image in the test [20:10] I cant re-create an old image [20:10] but I could use snapd from beta [20:10] does it make sense? [20:10] but the bug is in qemu [20:10] so it's about the host [20:10] not the uc image [20:11] pedronis, there are 2 things [20:11] unless is the kernel in the uc image triggering something [20:12] a bug in qemu and also this issue that I see that the first boot does not finish [20:12] install mode goes well [20:12] then in run mode it does not finish [20:12] and it was working 2 weeks ago with the same qemu/kvm [20:13] pedronis, I'll try with an older kernel and older snapd to see what happens [20:29] hey who knows a bit about spread? :) Cannot allocate lxd:ubuntu-18.04: cannot list lxd remotes: exec: "lxc": executable file not found in $PATH [20:57] PR snapcraft#3090 closed: ci: move staging store tests to spread [20:58] sergiusens: maybe you know about that error I got above? lxc/d is installed, though from deb, not snap [20:58] sergiusens: btw this looks much easier to set up / run than the old way :) thanks [21:13] Bug #1875493 opened: [core] log rotation doesn't properly restart rsyslogd [21:53] roadmr: are you using spread via the snap ? [21:53] oh well nvm [22:44] jdstrand, if you're about, is the gnome session manager dbus api (for uses other than screen-inhibit-control) omitted from the desktop or desktop-legacy plugs because we're expecting snaps to define their use via a custom plug? [22:44] specifically it's on the session bus at endpoint org.gnome.SessionManager member RegisterClient [22:45] possibly also an UnregisterClinet? [22:46] for an example snap that is trying to use the api:: https://forum.snapcraft.io/t/openaudible-bounty/16835/8