[02:59] <mup> PR snapcraft#3295 opened: specification: desktop extension font hook <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3295>
[05:27] <mborzecki> morning
[05:34] <mborzecki> back in a bit, need ot drive the kids to school and then hopefully a brief visit to the tax office
[06:28] <zyga> good morning
[06:28]  * zyga sees two failures on 16.04 left
[06:28] <zyga> neat :)
[06:30] <mvo> zyga: good morning
[06:32] <zyga> hey :)
[06:32] <zyga> how are you?
[06:33] <zyga> ok, tweked two more tests and started another run
[06:40] <zyga> 16.04 gets a green pass, looking at 18.04 which unlocks a few more tests
[06:41] <mborzecki> re
[06:41] <mborzecki> mvo: zyga: morning guys
[06:41] <zyga> hey mborzecki
[06:42] <mvo> hey mborzecki
[06:42] <zyga> it's not quite frosty morning stage yet
[06:42] <zyga> but it's definitely chilly morning here
[06:42] <zyga> but even more than yesterday, we're looking at crazy 28C later today
[06:49] <zyga> I'll grab some food
[07:05] <zyga> mmmm, scrambled eggs never get old
[07:07] <pstolowski> morning
[07:10] <zyga> hello Pawel
[07:10] <mvo> good morning pstolowski
[07:26] <pstolowski> o/
[07:53] <zyga> 18.04 is clean
[07:53] <zyga> 20.04 and then core
[07:54] <zyga> I think there may be a few tests left
[07:54] <zyga> and definitely selinux
[07:54] <zyga> but this looks great
[08:06] <Chipaca> zyga: 👋
[08:06] <Chipaca> zyga: is snap-confine still your bff?
[08:11] <Chipaca> asking because a friend is getting a 'sanity timeout expired' thing, https://dpaste.org/2YDP
[08:12] <Chipaca> snap version & etc, https://dpaste.org/mfb8
[08:22] <zyga-mbp> Chipaca hey
[08:22] <zyga-mbp> sorry, I was in a call
[08:22] <zyga-mbp> Chipaca ohhh
[08:22] <zyga-mbp> let me look
[08:22] <StyXman> BTW, the friend he mentions it's me
[08:22] <zyga-mbp> StyXman o/
[08:22] <Chipaca> StyXman: 🙂
[08:23] <zyga-mbp> so was the machine under some incredible load?
[08:23] <Chipaca> zyga-mbp: of note is that this isn't the first time it happens, and last time StyXman had to reboot to fix it
[08:23] <zyga-mbp> that timeout is a relatively short in human scale but really huge in kernel scale duration where we grab an exclusive lock to do something tiny
[08:23] <Chipaca> well, 'had to', tried it and it worked
[08:23] <mup> PR snapd#9400 opened: o/assertstate: support refreshing any number of snap-declarations <Created by pedronis> <https://github.com/snapcore/snapd/pull/9400>
[08:23] <zyga-mbp> I have a few minutes and then another call I cannot miss
[08:23] <zyga-mbp> if you give me more context I will think about what may be going on
[08:24] <zyga-mbp> maybe it's a stuck process dead somewhere
[08:24] <zyga-mbp> well
[08:24] <zyga-mbp> not dead but blocked
[08:24] <zyga-mbp> and holding the lock
[08:24] <zyga-mbp> we rigged it so that processes that can grab it should die
[08:24] <zyga-mbp> the only exception is snapd which also grabs the lock for brief moments
[08:24] <zyga-mbp> so any context you can provide helps
[08:24] <Chipaca> zyga-mbp: maybe running fuser on the lock shines some light on that?
[08:25] <zyga> yes
[08:25] <zyga> it's a flock-based lock
[08:25] <zyga> the locks are in /run/snapd/lock
[08:26] <Chipaca> StyXman: try: sudo fuser -v /run/snapd/lock/lxd.lock
[08:26] <zyga-mbp> there's also a global lock /run/snapd/lock/.lock
[08:27] <StyXman> zyga-mbp: o/
[08:27]  * zyga-mbp types from two accounts
[08:28] <StyXman> zyga-mbp: holding *which* lock?
[08:28] <zyga-mbp> we grab the global lock, do some setup, then we grab the per-snap lock
[08:28] <zyga-mbp> here it was definitely holding the per-snap lock
[08:28] <StyXman> nothing on fuser's ouput
[08:28] <zyga-mbp> so /run/snapd/lock/lxd.lock
[08:29] <zyga-mbp> normally snap-confine itself grabs this lock
[08:29] <StyXman> nothing either, weird, even when snapd is running
[08:29] <zyga-mbp> to synchronize access of concurrently executing snap-confine
[08:29] <zyga-mbp> sometimes snapd also grabs this lock
[08:29] <zyga-mbp> but very briefly and in specific scenarios
[08:29] <zyga-mbp> can you reproduce the failure?
[08:29] <StyXman> all the time on this machine
[08:29] <zyga-mbp> hmmm
[08:29] <zyga-mbp> ok
[08:30] <StyXman> wait
[08:30] <StyXman> all the time in this reboot of this machine
[08:30] <StyXman> I hadn't (can't) reboot it now
[08:30] <StyXman> and i had this befor on other hosts
[08:30] <StyXman> https://github.com/lxc/lxd/issues/6772#issuecomment-606597033
[08:31] <StyXman> and the next, I'm the same person
[08:31] <StyXman> and I guess you'r in a meeting again
[08:34] <StyXman> I notice that the last two messages are not prefixed with DEBUG, so I guess it's not snapd who's printing them?
[08:34] <StyXman> notice the like above is an issue opened in the lxd project
[08:35] <StyXman> load is bellow 6 on a 32 core machine
[08:35] <StyXman> less than 50% ram used by apps, so no, no load at all
[08:36] <StyXman> I had restarted snapd too,  just in case
[08:38] <mup> PR snapd#9401 opened:  gadget: allow content observer to have opinions about a change  <Simple 😃> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9401>
[08:57] <zyga> re
[08:57] <zyga> DEBUG messages are from snap-confine
[09:01] <zyga> StyXman: hmmm
[09:01] <zyga> StyXman: can you describe the setup a bit
[09:01] <zyga> like the FS used
[09:01] <zyga> or anything unusual?
[09:03] <StyXman> not much
[09:03] <StyXman> let me check this one
[09:03] <zyga> lxd on metal?
[09:03] <StyXman> yes
[09:03] <StyXman> sharing with KVM
[09:04] <StyXman> fs is xfs
[09:05] <StyXman> like I said, 32 cores, 128 of ram
[09:06] <StyXman> fs's are on LVM, opver several disks, but I don't think that's relevant
[09:07] <StyXman> how can I figure out which lock is failing?
[09:08] <StyXman> run under strace?
[09:08] <StyXman> run snapd*
[09:08] <zyga> one sec
[09:08] <StyXman> sure
[09:08] <zyga> you can run snap-confine via strace
[09:08] <StyXman> k
[09:08] <StyXman> how?
[09:09] <StyXman> I mean, which invocation (incantation?) of snap-confine?
[09:09] <zyga> hold on :)
[09:09] <StyXman> right
[09:11] <zyga> sudo SNAP_NAME=lxd SNAP_REVISION=$(readlink /snap/lxd/current) strace -f /snap/core/9993/usr/lib/snapd/snap-confine snap.lxd.lxd /bin/true
[09:11] <zyga> StyXman: ^ try this, you may also add --base core18 if lxd was using core18 as base (I forgot)
[09:11] <zyga> er
[09:12] <zyga> SNAP_INSTANCE_NAME=lxd instead of SNAP_NAME= most likely
[09:12] <StyXman> daam, that's definitely an incantation :)
[09:12] <zyga> typed dry, my setup has a custom snapd that does a lot new things and I'm not running things now
[09:13] <StyXman> something's *really* slow here
[09:13] <StyXman> I can see read()s scroll by
[09:14] <StyXman> it takes ages reading /proc/self/mountinfo
[09:14] <StyXman> WTF
[09:14] <zyga> can you read that?
[09:15] <zyga> note:
[09:15] <zyga> it may be reading the mountinfo of lxd
[09:15] <zyga> to reproduce
[09:15] <zyga> nsenter -m/run/snapd/ns/lxd.mnt cat /proc/self/mountinfo
[09:16] <zyga> do pastebin if it's curious and you have no secrets there
[09:16] <StyXman> this is insane: https://dpaste.org/viR7
[09:16] <StyXman> nsenter?
[09:16] <zyga> hmm, it doesn't load
[09:17] <zyga> nsenter is a tool that can jump into one of the many linux namespaces
[09:17] <zyga> the incantation I gave shows the mount namespace of the lxd snap
[09:17] <StyXman> something's really worng
[09:17] <zyga> StyXman: I cannot open the URL you have provided
[09:17] <StyXman> it's full of the same mount info over and over again
[09:18] <zyga> how big is the paste/
[09:18] <zyga> oh it loaded
[09:18] <StyXman> *sigh*
[09:18] <StyXman> mdione@demo-hv1:~$ sudo nsenter -m/run/snapd/ns/lxd.mnt time wc -l /proc/self/mountinfo
[09:18] <StyXman> nsenter: failed to execute time: No such file or directory
[09:18] <zyga> can you run the 2nd command and show me the vanilla output?
[09:18] <StyXman> it never ends
[09:18] <zyga> how about /bin/cat?
[09:19] <zyga> or /usr/bin/cat
[09:19] <StyXman> wait
[09:19] <zyga> (it's not running that in a shell)
[09:19] <StyXman> cat worked, it just never ends
[09:19] <zyga> so if you want time, you need sh -c ...
[09:19] <zyga> I'll make coffee
[09:19] <StyXman> the 'file' has this line repeade indefinitely:
[09:19] <StyXman> /lxd/storage-pools /mnt/kvm/lxd/storage-pools rw,relatime shared:104 master:34 - ext4 /dev/mapper/ssd-kvm rw,data=ordered
[09:19] <zyga> I think this may require lxd developers
[09:19] <zyga> lxd is very much special
[09:20] <zyga> and does things regular snaps cannot
[09:20] <zyga> StyXman: please collect the mountinfo in a bug report
[09:20] <zyga> and share it here
[09:20] <Chipaca> bug report for lxd, or snapd?
[09:20] <Chipaca> or both :-p
[09:20] <zyga-mbp> for lxd for now
[09:20] <zyga-mbp> on lp it can be both
[09:21] <zyga-mbp> so
[09:21] <zyga-mbp> it's not a bug in snap-confine
[09:21] <zyga-mbp> it's a very unusual mount namespace
[09:21] <zyga-mbp> the origin of the bug is unclear for now
[09:21] <StyXman> ack :(
[09:22] <zyga-mbp> please collect the full output of mountinfo
[09:22] <zyga-mbp> on both the host
[09:22] <zyga-mbp> and as seen from lxd point of view
[09:23] <zyga-mbp> if you need to reboot please get those first
[09:23] <zyga-mbp> you can redirect to a file to avoid losing it
[09:25] <StyXman> I waited more than a minute before I got tired of seeing the same line appear over and over again
[09:25] <StyXman> also: WTH is it doing between lines?
[09:27] <StyXman> I just send it to a log file, let's see how long it takes
[09:27] <StyXman> (now I just realized maybe the remoteness was playing a part in it)
[09:29] <StyXman> WTF
[09:29] <StyXman> the first few reads take >100µs, then it jumps up to 10s of ms
[09:32] <StyXman> cat is taking ages
[09:34] <StyXman> 3m48s
[09:34] <StyXman> wc -l /tmp/nsenter_cat_proc_self_mountinfo.log
[09:34] <StyXman> 65631 /tmp/nsenter_cat_proc_self_mountinfo.log
[09:37] <zyga-mbp> StyXman it's a virtual file
[09:38] <zyga-mbp> generating each line is non trivial in general
[09:38] <zyga-mbp> can you paste it anywhere?
[09:39] <StyXman> I don't think it makes sense
[09:40] <StyXman> oh
[09:40] <StyXman> yes, there are some interesting things
[09:40] <StyXman> we use loop files in the ocntainers
[09:40] <StyXman> and it seems it has something to do
[09:40] <StyXman> it's exponential!
[09:41] <StyXman> https://dpaste.org/6qp0
[09:41] <StyXman> before the first loop device the'res one line
[09:41] <StyXman> before the second, 2
[09:41] <StyXman> before the third, 4
[09:42] <StyXman> 8, 16,, ...
[09:42] <StyXman> 2**n
[09:42] <StyXman> there are 28 loop files on that node
[09:46] <StyXman> so at the end it should be... around 512 million lines?
[09:53] <zyga-mbp> StyXman that can take a while
[09:53]  * Chipaca has a O(no) joke
[09:53] <zyga-mbp> I think we'd like to know how it happened
[09:53] <zyga-mbp> and if it comes back after rebooting
[09:53] <StyXman> Chipaca: hehehehe
[09:54] <zyga-mbp> but before please ask stgraber for help
[09:54] <zyga-mbp> my network is experiencing big issues, I may be offline
[09:54] <Chipaca> StyXman: stgraber is -0400 so maybe in a bit
[09:56] <StyXman> is that NY?
[09:57]  * StyXman has more clocks in his wall that humanly needed
[10:01] <Chipaca> StyXman: NY is also -4, yes
[10:08] <zyga-mbp> re
[10:08] <zyga-mbp> I think my network is back
[10:08] <zyga-mbp> StyXman what's in mountinfo outside of lxd?
[10:08] <zyga-mbp> on your host, that is
[10:08] <zyga-mbp> did you do any changes to mount tables recently?
[10:09] <StyXman> on the host not really, it's been ages since it was installed
[10:09] <zyga> ok
[10:09] <StyXman> the contaioners come and go as needed
[10:10] <StyXman> it's (ab)used for testing purposes
[10:11] <StyXman> here's the lxd comments: https://github.com/lxc/lxd/issues/6772#issuecomment-697265476
[10:12] <zyga> thank you
[10:19] <mup> Issue core20#72 opened: move docker user/group to extrausers <Blocked> <Created by anonymouse64> <https://github.com/snapcore/core20/issues/72>
[10:19] <mup> PR core20#45 closed: Add arches <Created by xnox> <Closed by xnox> <https://github.com/snapcore/core20/pull/45>
[10:19] <mup> PR core20#86 closed: .travis.yml: use stable snapcraft now <Created by anonymouse64> <Merged by xnox> <https://github.com/snapcore/core20/pull/86>
[10:24] <mup> PR core20#87 opened: Revert "hooks: mv docker user/group definition to extrausers" <Created by xnox> <https://github.com/snapcore/core20/pull/87>
[10:39] <ijohnson> yay new pc-kernel in edge
[10:39] <ijohnson> and also good morning folks
[10:40] <pstolowski> hi ijohnson !
[10:41] <ijohnson> o/ pstolowski
[10:41] <zyga> ijohnson: good morning!
[10:42] <zyga> hmm
[10:42] <ijohnson> hey zyga
[10:42] <zyga> is google down?
[10:42]  * ijohnson is excitedly testing the new edge kernel
[10:43] <zyga> 2020-09-23 12:43:03 Cannot allocate google:opensuse-15.1-64: cannot perform Google request: Get https://www.googleapis.com/compute/v1/projects/computeengine/zones: Post https://oauth2.googleapis.com/token: dial tcp 172.217.16.10:443: i/o timeout
[10:43] <zyga> over and over
[10:43] <zyga> maybe my network segment somewhere is flaky
[10:50] <ijohnson> zyga: seems to be working fine for me, I just launched a spread run on gce
[10:50] <zyga> thanks
[10:57] <zyga> 2020-09-23T12:56:36+02:00 ERROR Get https://api.snapcraft.io/api/v1/snaps/assertions/snap-revision/EnYyzGIokaLNadpwYWUw0qFczPCprbUPfS3KPBv8VsDMkFLDmosiytxhmkP9hwSP?max-format=0: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[10:57] <zyga> hmmm
[10:57] <zyga> network is troubled
[11:00] <zyga> mborzecki: I've just installed gimp
[11:00] <zyga> no fonts
[11:00] <zyga> and it crashes
[11:01] <mborzecki> zyga: on fedora/opensuse?
[11:01] <zyga> on focal
[11:01] <mborzecki> ooh
[11:01] <mborzecki> zyga: try with --experimental-gdbserver maybe?
[11:02] <mborzecki> zyga: but i investigated it a bit on arch and the backtrace was pointing to fonts issues again
[11:14] <mup> PR snapd#9402 opened: tests/lib/prepare.sh: stop patching the uc20 initrd since it has been updated now <Run nested> <Simple 😃> <UC20> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9402>
[11:14] <zyga> weird
[11:14] <zyga> irc works
[11:14] <zyga> but stuff other than that falls apart
[11:16] <StyXman> ssh? http? ftp?
[11:17] <zyga> http
[11:18] <StyXman> http or https?
[11:18] <StyXman> what ISP?
[11:18] <zyga> t-mobile
[11:22]  * zyga goes to debug
[11:22] <zyga> sigh
[11:31] <zyga> (gimp:1053668): GLib-ERROR **: 13:30:51.456: ../../../../glib/gmem.c:333: overflow allocating 18446744073709551615*18446744073709551615 bytes
[11:33] <zyga> mborzecki: reinstalled, no change
[11:35] <mborzecki> zyga: yeah, iirc there was an earlier assertion that failed, try running with G_DEBUG=fatal-critical
[11:35] <mborzecki> quick errand, back in 30 or so
[11:35] <Chipaca> zyga: here's a nickle, kid, get yourself a computer that can allocate 18446744073709551615*18446744073709551615 bytes
[11:35] <zyga> Fontconfig warning: FcPattern object width does not accept value [85 100)
[11:35] <zyga>  /snap/gimp/292/usr/bin/gimp: Gimp-Text-KRYTYCZNE: gimp_font_factory_load_names: assertion 'fontset' failed
[11:35]  * Chipaca is out of nickles
[11:36] <zyga> heh
[11:36] <zyga> gimp_device_info_set_device: trying to set GdkDevice 'Ultimate Gadget Laboratories Ultimate Hacking Keyboard' on GimpDeviceInfo which already has a device
[11:50] <pedronis> degville: ijohnson: hi, a naming question here where maybe you can give some input:  https://github.com/snapcore/snapd/pull/9401#discussion_r493493467
[11:50] <mup> PR #9401:  gadget: allow content observer to have opinions about a change  <Simple 😃> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9401>
[11:50] <degville> pedronis: looking now...
[11:52] <ijohnson> pedronis I responded too
[12:01] <mborzecki> re
[12:10] <pedronis> pstolowski: hi, added some comments in #9391
[12:10] <mup> PR #9391: [RFC] o/assertstate: introduce ValidationTrackingKey/ValidationSetTracking and basic methods <Needs Samuele review> <validation-sets :white_check_mark:> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9391>
[12:19] <pedronis> ijohnson: I reviewed #9379
[12:19] <mup> PR #9379: [RFC] cmd/s-b/initramfs-mounts: use ConfigureTargetSystem for install, recover modes <Bug> <Needs Samuele review> <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9379>
[12:19]  * zyga-mbp goes for a walk
[12:33] <pstolowski> pedronis: ty
[12:35] <mborzecki> pedronis: wdyt about just using TrustedAssetsBootloader for the merged TrustedAssets* & ManagedAssets* interfaces?
[12:35] <mborzecki> or TrustedBootloader maybe?
[12:35] <pedronis> mborzecki: TrustedAssetsBootloader was what I thought
[12:37] <mborzecki> pedronis: ok
[13:40] <ijohnson> thanks pedronis I'll look at your review
[13:41] <ijohnson> btw can I get a review on https://github.com/snapcore/snapd/pull/9369 since it's green again due to new kernel snap in edge/beta ?
[13:41] <mup> PR #9369: tests/nested/manual/refresh-revert-fundamentals: re-enable test <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9369>
[13:41] <ijohnson> (from anyone though)
[13:41] <ijohnson> that pr is green now just needs reviews
[14:07] <ijohnson> man github actions new view is nice in that it jumpts to the error again like it did a few months ago initially, but now the log on that page is less than 20% of the screen :-/
[14:12] <ijohnson> tests/main/snap-run seems to be hanging in strace reliably on ubuntu 20.04, 18.04 and 16.04 since recently I wonder if something broken in strace recently that makes this hang happen
[14:24] <zyga> I'm running opensuse tests and it looks like apparmor profile for snap-confine is just not getting loaded
[14:25] <mup> PR snapd#9369 closed: tests/nested/manual/refresh-revert-fundamentals: re-enable test <Run nested> <UC20> <⚠ Critical> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9369>
[14:29] <pedronis> zyga: I looked at https://github.com/snapcore/snapd/pull/9384, one questions there
[14:29] <mup> PR #9384: overlord: export and use snapd tools <Created by zyga> <https://github.com/snapcore/snapd/pull/9384>
[14:30] <pedronis> maybe I'm confused or something can be simplified further?
[14:48] <ijohnson> #9395 needs another review and is fairly simple
[14:48] <mup> PR #9395:  o/snapstate/check_snap_test.go: mock osutil.Find{U,G}id to avoid bug in test <Bug> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9395>
[15:04] <ijohnson> pedronis: I addressed your feedback on #9379 and marked it ready for review
[15:04] <mup> PR #9379: [RFC] cmd/s-b/initramfs-mounts: use ConfigureTargetSystem for install, recover modes <Bug> <Needs Samuele review> <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9379>
[15:04] <ijohnson> pedronis: let me know if you want me to split the pr up at all
[15:17] <pedronis> ijohnson: why split it? is not large
[15:17] <ijohnson> pedronis: I dunno cause sometimes it is easier to split things
[15:17] <ijohnson> but sure not splitting is easier for me anyways :-)
[15:18] <pedronis> anything up to 500 is usually reasonable, even a bit more if a good chunk is tests
[15:20] <ijohnson> makes sense
[15:20] <mup> PR snapd#9403 opened: many: move ManagedAssetsBootloader into TrustedAssetsBootloader, drop former <Simple 😃> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9403>
[15:23] <pedronis> ijohnson: question there
[15:24] <mborzecki> pedronis: is https://github.com/snapcore/snapd/pull/9005 something we want for 1.0?
[15:24] <mup> PR #9005: boot: support setting extra command line argument, bootloader interface tweaks <UC20> <⛔ Blocked> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9005>
[15:24] <ijohnson> where?
[15:26] <pedronis> ijohnson: in the PR
[15:26] <ijohnson> pedronis: oh it was in the resolved conversation sorry I see it now
[15:26] <pedronis> mborzecki: possibly, we need to discuss
[15:26] <mborzecki> pedronis: ok, let's sync in the morning then
[15:27] <ijohnson> pedronis: https://github.com/CanonicalLtd/subiquity/blob/fe012f20dc758f1fb1641d0b5744baf00739524d/bin/console-conf-wrapper#L25
[15:27] <ijohnson> afaik writing the complete file shouldn't interfere with that at all
[15:28] <pedronis> ah, thx
[15:29] <pedronis> yes, writing the complete is not a problem. we still support the recover trigger in case anyway because of the or cond in the service
[15:34] <pedronis> ijohnson: +1
[15:34] <ijohnson> nice thanks
[16:00] <mup> PR snapd#9402 closed: tests/lib/prepare.sh: stop patching the uc20 initrd since it has been updated now <Run nested> <Simple 😃> <UC20> <⚠ Critical> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9402>
[16:03] <zyga-mbp> pedronis ack, looking
[16:04] <pedronis> zyga-mbp: I got this failure google:arch-linux-64:tests/main/cgroup-tracking:root here: https://github.com/snapcore/snapd/pull/8569/checks?check_run_id=1155234596
[16:04] <mup> PR #8569: o/assertstate,asserts: use bulk refresh to refresh snap-declarations <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8569>
[16:15] <mup> PR snapd#8569 closed: o/assertstate,asserts: use bulk refresh to refresh snap-declarations <Bulk assert refresh :scroll::scroll::scroll:> <Created by pedronis> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/8569>
[16:15] <mup> PR snapd#9404 opened: asserts: deserialize grouping only once in AddBatch if needed <Created by pedronis> <https://github.com/snapcore/snapd/pull/9404>
[16:25] <mup> PR snapd#9334 closed: tests/nested/manual: add test for grades above signed booting with testkeys <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9334>
[16:25] <mup> PR snapd#9397 closed: tests/nested: misc robustness fixes <Run nested> <Simple 😃> <Test Robustness> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9397>
[16:28] <ijohnson> cachio: did you ever look at the api for gcloud launching shielded vm's? I see that actually in their rest API they have a `shieldedInstanceInitialState` setting which can take in pk, keks , dbs, and dbxs settings, which I assume is for the secure boot configuration
[16:29] <zyga> pedronis: looking at the failure
[16:29] <ijohnson> cachio: of course to be able to set that, I assume it would need spread changes but that could probably be handled
[16:30] <cachio> ijohnson, hey
[16:31] <cachio> I already verified that a time ago
[16:31] <cachio> they confirmed that it is not possible to set the keys for the secure boot
[16:31] <ijohnson> cachio: ah ok, so this api is not really usable for that then
[16:31] <cachio> It was when we were at frankfurt
[16:31] <ijohnson> cachio: I was wondering if maybe they added it since you looked at it
[16:32] <cachio> I can chack again
[16:32] <ijohnson> right a million years ago :-D
[16:32] <ijohnson> cachio: no need, I just was waiting for test results and looked at the api
[16:32] <cachio> ijohnson, it was the ifrst thing we tried to do
[16:32] <ijohnson> I see
[16:33] <cachio> ijohnson, I fixed tumbleweed again
[16:33] <cachio> for some raeson a new kernel appeared
[16:33] <ijohnson> nice I'll keep an eye out on runs this afternoon for tumbleweed
[16:33] <cachio> I am testing right now
[16:35] <zyga> pedronis: no immediate idea, we did run a test from another suite quite recently before this test, still no idea why this failed exactly
[16:59]  * cachio afk -> go to doctor app and make study
[17:01] <cachio> ijohnson, I am leaving now, if you need any review just leave me a message, I'll do it when I am back
[17:02] <ijohnson> cachio: I'm good for now, thanks!
[17:02] <cachio> ijohnson, nice
[18:40] <ijohnson> cachio: when you're back really simple pr https://github.com/snapcore/snapd/pull/9405
[18:40] <mup> PR #9405: tests/nested/manual/grade-signed-above-testkeys-boot: enable kvm <Run nested> <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9405>
[18:41] <mup> PR snapd#9405 opened: tests/nested/manual/grade-signed-above-testkeys-boot: enable kvm <Run nested> <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9405>
[19:29] <zyga> re
[19:29] <zyga> PT is late recently
[19:29] <zyga> tired and sleepy
[19:29] <zyga> and bloody tests got network timeouts
[19:37] <zyga> some more network luck now
[19:37] <zyga> 2020-09-23 21:37:38 Error discarding google:arch-linux-64 (sep231929-708690): cannot deallocate Google server google:arch-linux-64 (sep231929-708690): cannot perform Google request: Delete https://www.googleapis.com/compute/v1/projects/computeengine/zones/us-east1-b/instances/sep231929-708690: dial tcp 216.58.208.202:443: i/o timeout
[19:37] <zyga> all the time
[19:38] <zyga> oh well
[19:38] <zyga> tomorrow I will start with my network
[19:45] <zyga-kaveri> on fallback link
[19:46] <zyga-kaveri> I need to debug the primary link tomorrow
[19:46] <zyga-kaveri> o/
[19:46]  * zyga-kaveri EODs
[19:46] <zyga-kaveri> ijohnson: if you have a minute I have a super-simple PR at 9407
[19:46] <mup> PR snapd#9406 opened: many: allow ignoring running apps for specific request <Created by zyga> <https://github.com/snapcore/snapd/pull/9406>
[19:46] <mup> PR snapd#9407 opened: overlord: explicitly set refresh-app-awareness in tests <Simple 😃> <Skip spread> <Created by zyga> <https://github.com/snapcore/snapd/pull/9407>
[19:53] <ijohnson> zyga-kaveri: sur
[19:53] <ijohnson> Sure
[21:47] <mup> PR snapcraft#3294 closed: project loader: install dirmngr prior to configuring package repositories <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3294>