[06:13] <mborzecki> morning
[06:38] <mborzecki> mvo: hey
[06:38] <mvo> good morning mborzecki
[06:41] <mup> PR snapd#9377 closed: [RFC] snap-repair: minimal uc20 support <Needs Samuele review> <UC20> <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/9377>
[06:52] <jamesh> mvo: I put together https://forum.snapcraft.io/t/adding-users-to-system-groups-on-ubuntu-core/20109 with my thoughts on handling groups, in prep for tomorrow's meeting.
[06:52] <mborzecki> mvo: heads up, google-nested:ubuntu-20.04-64:tests/nested/manual/preseed seems to be failing (i've pinged pawel in one of the prs)
[06:53] <mvo> jamesh: thanks, excellent, let me read this
[06:53] <mvo> mborzecki: oh no, any more clues yet?
[06:54] <mborzecki> mvo: the preseed key is different, https://github.com/snapcore/snapd/pull/9380#issuecomment-695933603
[06:54] <mup> PR #9380: tests: update to support nested kvm without reboots on UC20 <Run nested> <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9380>
[06:56] <mvo> mborzecki: ok, that sounds not too bad
[07:06] <zyga> re
[07:06] <zyga> good morning!
[07:07] <pstolowski> morning
[07:08] <zyga> o/
[07:09] <mvo> good morning pstolowski and zyga
[07:10] <zyga> hmm, more failures
[07:10] <zyga> but this looks like a bug in repacking core
[07:10]  * zyga looks
[07:10] <zyga> hmm
[07:11] <zyga> mvo: today started more like normal, wife has afternoon shift today
[07:11] <mborzecki> pstolowski: zyga: hey
[07:12] <mborzecki> pstolowski: pinged you about nested preseed test failure, please take a look
[07:12] <zyga> mvo: oh no, umatrix development is over
[07:12] <zyga> hey mborzecki :)
[07:12] <zyga> it's real autumn now
[07:12] <mborzecki> pstolowski: https://github.com/snapcore/snapd/pull/9380#issuecomment-695933603
[07:12] <mup> PR #9380: tests: update to support nested kvm without reboots on UC20 <Run nested> <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9380>
[07:13] <mborzecki> zyga: yeah, very much so, this is what it looked like during my afternoon ride yesterday https://dgtzuqphqg23d.cloudfront.net/O1Yi0M_6L-sWwaf-Rxs-NoxxaZjK96HwWdDgXIato6U-1536x2048.jpg
[07:13] <zyga> ohh
[07:13]  * zyga envies bike riding
[07:13] <zyga> in a few months
[07:14] <zyga> my back still hurts a little when I bend
[07:14] <zyga> forecast says 24C today so maybe I will go for a walk instead
[07:14] <mborzecki> zyga: and it gets cold fast, 20C when i was leaving, 16C when i was coming back, stopped to take a couple of photos and i felt like i was freezing for the last 7km
[07:14] <zyga> mborzecki: I'll try some selinux bits today, I got nowhere on Friday
[07:14] <zyga> mborzecki: that's evening ligth?
[07:15] <mborzecki> yeh
[07:15] <zyga> yeah, I need to find my full shoes
[07:15] <zyga> I had sporty hole-ridden shoes all summer
[07:16] <mvo> zyga: umatrix development over? nooooo, that's super sad. where did you see that?
[07:18] <zyga> mvo: ./
[07:18] <zyga> mvo: github repo is archived
[07:23] <zyga> hmm
[07:23] <zyga> weird
[07:23] <mvo> zyga: meh, so sad. I can't see anything on the readme about the why
[07:23] <zyga> ha
[07:23] <zyga> I found a bug
[07:24] <zyga> repack_snapd_snap_with_deb_content_and_run_mode_firstboot_tweaks :)
[07:24] <zyga> that's a mouthful
[07:24] <zyga> and identical copy-pasted bug in repack_snapd_snapd_with_deb_content
[07:25] <zyga> I'll send noise and bug fixes separately in a moment
[07:25] <zyga> just want to see green core tests
[07:25] <mvo> zyga: nice
[07:25] <zyga> man it's COLD in the office today
[07:25] <zyga> definitely sub 20
[07:25] <zyga> probably ~15
[07:26] <pstolowski> mborzecki: replied, you're correct there
[07:36] <mborzecki> zyga: pushed a branch again? ;) i see wip/export-manager-patch-set in the latest pull
[07:38] <zyga> hmm
[07:38] <zyga> mborzecki: gaah
[07:38]  * zyga wonders why that happens 
[07:38] <zyga> mborzecki: cleaned that up
[07:39] <zyga> we don't seem to repackage core when doing core18 tests
[07:40] <zyga> ah
[07:40] <zyga> indeed
[07:40] <zyga> okay
[07:40] <mborzecki> zyga: i normally set origin to be my fork
[07:42] <zyga> oh
[07:42] <zyga> conceptual question
[07:42] <zyga> when testing ubuntu core
[07:42] <zyga> should we repackage both core and snapd
[07:42] <zyga> so that they are in sync
[07:43] <zyga> I wrote my test assuming as much
[07:43] <mborzecki> i think the only reason we repack core is to setup the logging (and iirc we do it only in nested)
[07:44] <mborzecki> w8, core as in core20
[07:44] <mborzecki> core is probably repacked with newer snapd anyway
[07:44] <zyga> we repackage both for the same reason
[07:44] <zyga> to get new snapd
[07:44] <zyga> my question was more subtle
[07:44] <zyga> should we always have both repackaged
[07:45] <zyga> regardless of the version of ubuntu core
[07:46] <zyga> right now if you run a test with ubuntu-core-18
[07:46] <zyga> and install something that pulls in core
[07:46] <zyga> you get core tools
[07:46] <zyga> with old snap-exec
[07:46] <zyga> mvo, pedronis: ^ opinion please
[07:46] <zyga> I can adjust my test or adjust the test suite either way
[07:47]  * mvo is in a meeting
[07:52]  * zyga does the simpler thing
[08:04] <zyga> passed
[08:38] <zyga> re
[08:42] <mup> PR snapd#9345 closed: overlord: introduce the export manager, export snapd tools <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/9345>
[09:06] <zyga> quick review required for https://github.com/snapcore/snapd/pull/9382
[09:06] <mup> PR #9382: tests: copy /usr/lib/snapd/info to correct directory <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9382>
[09:07] <zyga> optionally for https://github.com/snapcore/snapd/pull/9381 but not important
[09:07] <mup> PR #9381: many: typos and small test tweak <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9381>
[09:07] <mup> PR snapd#9381 opened: many: typos and small test tweak <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9381>
[09:07] <mup> PR snapd#9382 opened: tests: copy /usr/lib/snapd/info to correct directory <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9382>
[09:12] <zyga> pstolowski, mborzecki: ^
[09:22] <mborzecki> yay, pi4 8gb with today's uc20 image is up
[09:23] <zyga> mborzecki: nice :)
[09:39] <zyga> mborzecki: trying core20 with tpm on vmware now
[09:50] <zyga-mbp> mborzecki booting now
[09:50] <zyga-mbp> no such device ubuntu boot
[09:51] <zyga-mbp> now black screen
[09:51] <zyga-mbp> now efi stub secure boot enabled
[09:51] <zyga-mbp> now wall of text
[09:51] <zyga-mbp> booting :)
[09:52] <mborzecki> so far so good
[09:52] <mborzecki> zyga-mbp: there will be a log during install that encryption is enabled
[09:52] <zyga> mborzecki: can you look at https://github.com/snapcore/snapd/pull/9382
[09:52] <mup> PR #9382: tests: copy /usr/lib/snapd/info to correct directory <Simple 😃> <Test Robustness> <Created by zyga> <https://github.com/snapcore/snapd/pull/9382>
[09:52] <zyga> I need to merge itr
[09:52] <zyga> ok
[09:52] <zyga-mbp> hmm
[09:53] <zyga-mbp> got a load of IO errors
[09:54] <zyga-mbp> tweaked vm and rebooted
[09:59] <zyga-mbp> mborzecki no luck yet
[10:00] <zyga-mbp> mborzecki it could not boot wuth disk type=nvme
[10:00] <zyga-mbp> secure boot enabled
[10:00] <zyga-mbp> wall of kernel booting
[10:00] <zyga-mbp> I switched to sata
[10:00] <zyga-mbp> it's starting
[10:01] <zyga-mbp> mborzecki tpm device detected and enabled!
[10:01] <zyga-mbp> rebooting
[10:01] <zyga-mbp> :D
[10:01] <zyga-mbp> I think it will work
[10:02] <zyga-mbp> hmm, now rebooted and looking at efi boot logo
[10:02] <zyga-mbp> with "attempting to start up from efi network"
[10:02] <zyga-mbp> it's either not clearing the buffer
[10:02] <zyga-mbp> or just booting silently
[10:02] <zyga-mbp> mborzecki are we doing anything for efi vars to select boot method?
[10:02] <zyga-mbp> kernel up!
[10:02] <pedronis> it should reboot from install to run , if it gets far enough
[10:03] <pedronis> it tries to do that automatically
[10:03] <zyga-mbp> I think it's a winner :)
[10:03] <zyga-mbp> seeding
[10:03] <zyga-mbp> I should expect console conf, right/
[10:04] <zyga-mbp> now the last thing is "stopping snap daemon"
[10:04] <zyga-mbp> (from systemd)
[10:04] <zyga-mbp> started getty!
[10:05] <zyga-mbp> it's still doing stuff
[10:05] <zyga-mbp> waiting for server to restart
[10:05] <zyga-mbp> I guess that's "snap run" message
[10:06] <zyga-mbp> more getty changes
[10:07] <zyga-mbp> my only feedback so far is lots of black screen and waiting :)
[10:08] <zyga-mbp> and no CPU usage
[10:08] <zyga-mbp> hmmm
[10:08] <zyga-mbp> brb, small break to move
[10:25] <zyga-mbp> It worked :)
[10:26] <zyga-mbp> there was a backtrace visible in console conf after initial setup
[10:26] <zyga-mbp> hmm
[10:26] <zyga-mbp> (super brief, it's gone now)
[10:26] <zyga-mbp> I wonder if I can create an account
[10:27] <pstolowski> pedronis: hey, do you have a moment (~10 mins i guess) to chat about validation-sets?
[10:27] <zyga-mbp> now I have black screen with blinky cursor and nothing else
[10:28] <zyga-mbp> I think console conf does crash
[10:28] <zyga-mbp> I'm back in the setup screen
[10:28] <zyga-mbp> yep
[10:28] <zyga-mbp> meh
[10:28] <zyga-mbp> that's anti-climactic
[10:28] <zyga-mbp> guess that is it
[10:29] <zyga-mbp> the screen flash is so quick I have no hope of seeing what crashed
[10:29] <zyga-mbp> let me see if I can record it
[10:32] <zyga-mbp> got it
[10:32] <zyga-mbp> the traceback is requests.exceptions.ConnectionError
[10:32] <zyga-mbp> Connection refused!
[10:32] <zyga-mbp> didn't someone mention that recently?
[10:41] <mvo> meh, 9383 should be a draft - oh well
[10:42] <zyga-mbp> re, stupid suspend
[10:42] <zyga-mbp> anyway, is there a way to debug a dangerous model?
[10:42] <zyga-mbp> I'd like to see what's wrong with snapd
[10:42] <mvo> zyga-mbp: yes, press ESC during boot to get grub (you need the second grub for run mode)
[10:42] <mup> PR snapd#9383 opened: [RFC] snap-repair: minimal uc20 support <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9383>
[10:43] <mvo> zyga-mbp: and then change the kernel commandline to include
[10:43] <mvo> zyga-mbp: "dangerous systemd.debug-shell=1" and maybe "snapd.debug=1"
[10:45] <zyga-mbp> mvo second grub?
[10:45] <zyga-mbp> ah
[10:45] <zyga-mbp> second menu entry?
[10:47] <zyga-mbp> I have the grub menu now
[10:47] <zyga-mbp> booting!
[10:48] <zyga-mbp> for the record, I edited the recovery boot menu
[10:48] <zyga-mbp> hmm
[10:48] <zyga-mbp> now it asks me for recovery key :-(
[10:48] <zyga-mbp> so I guess that's SOL
[10:48] <zyga-mbp> mborzecki ^ any ideas?
[10:48] <mborzecki> hm?
[10:49] <zyga-mbp> mborzecki how to debug encrypted vm
[10:49] <mborzecki> snapd isn't up yet?
[10:50] <mborzecki> zyga-mbp: i've built custom snapd snap with this patch https://github.com/bboozzoo/snapd/commit/a1ec67e26bc5fde9ef645bcc8d6e127345cee344 then you get a debug shell
[10:52] <mvo> zyga-mbp: oh, of course, you won't be able to open the disk if the cmdline changed
[10:53] <zyga-mbp> snapd was partially up
[10:53] <zyga-mbp> it seeded and console conf is a snap after all
[10:53] <zyga-mbp> let me show you the backtrace
[10:56] <zyga-mbp> mborzecki I sent the screenshot via telegram
[10:57] <mborzecki> yeah, looks like snapd isn't up yet (or not quite working)
[10:58] <zyga-mbp> mborzecki just to be clear, can I do anything right now with this VM or is it gone and buseteD?
[11:01] <mborzecki> zyga-mbp: you can wait a bit more
[11:02] <zyga-mbp> mborzecki ok, I'll give it some time
[11:02] <mborzecki> zyga-mbp: otherwise, you can try to build a new image, repack the core20 snap and add logging to console
[11:02] <zyga-mbp> but it's weird
[11:02] <zyga-mbp> console conf is up but snapd is not responding?
[11:02] <zyga-mbp> hmmmm
[11:02] <zyga-mbp> now something weird happened
[11:02] <zyga-mbp> boot fails
[11:03] <zyga-mbp> failed yo boot both default and fallback
[11:03] <zyga-mbp> and I only see one item in grub
[11:03] <zyga-mbp> like something failed and it generated different grub menu with just one broken entry
[11:04] <zyga-mbp> and I get /EFI/ubuntu/kernel.efi not found
[11:04] <zyga-mbp> repeats across reboot
[11:04] <mvo> zyga-mbp: woah, that looks bad
[11:04] <zyga-mbp> indeed
[11:05] <zyga-mbp> shut down, rebooting
[11:05] <zyga-mbp> well, booting afresh
[11:05] <zyga-mbp> same
[11:05] <zyga-mbp> I think this is busted
[11:06] <zyga-mbp> I wonder if there's some case where snapd starts up
[11:06] <zyga-mbp> and something is wrong where it seems "no systems are available"
[11:06] <zyga-mbp> and just generates garbage
[11:06] <zyga-mbp> before it broke I saw a few more entries
[11:07] <zyga-mbp> including run, recover, reinstall and EFI setup
[11:07] <zyga-mbp> now I only see one
[11:07] <zyga-mbp> and EFI setup is oddly gone
[11:07] <zyga-mbp> as if the grub config was truncated somehow?
[11:08] <mvo> speaking of strange stuff - looks like 20.10 test are broken? or is this a one-off thing?
[11:10] <zyga-mbp> mvo what did you see?
[11:13] <mup> PR snapd#9381 closed: many: typos and small test tweak <Simple 😃> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9381>
[11:13] <zyga-mbp> thanks mvo!
[11:14] <mvo> zyga-mbp: looks like XDG_RUNTIME_DIR is no longer defined
[11:14] <mvo> zyga-mbp: in 20.10
[11:14] <mvo> zyga-mbp: maybe something in the image changed?
[11:14] <mvo> zyga-mbp: I think I will ask cachio once he is around
[11:14] <pedronis> pstolowski: hi, we can do now or before the standup? also I re-reviewed #9350 (thanks)
[11:14] <mup> PR #9350: store: handle v2 error when fetching assertions <Created by stolowski> <https://github.com/snapcore/snapd/pull/9350>
[11:15] <zyga-mbp> hmm?
[11:15] <zyga-mbp> hmm
[11:16] <pstolowski> pedronis: thanks for the review. now works for me
[11:16] <pedronis> pstolowski: let's do now then
[11:17] <pedronis> pstolowski: going to the standup HO
[11:17] <pstolowski> ok
[11:21]  * zyga-mbp looks at 20.10 failure
[11:22] <zyga-mbp> mvo hmm, maybe fixing info influenced snapd reexec
[11:22] <zyga-mbp> so now we are testing what we are running
[11:22] <zyga-mbp> and something used to be maskeD?
[11:22]  * zyga-mbp debugs
[11:41] <zyga> hmm
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[1]: Starting User Manager for UID 12345...
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[39458]: user@12345.service: Failed to determine user credentials: No such process
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[39458]: user@12345.service: Failed at step USER spawning /lib/systemd/systemd: No such process
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[1]: user@12345.service: Main process exited, code=exited, status=217/USER
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[1]: user@12345.service: Failed with result 'exit-code'.
[11:41] <zyga> Sep 21 11:39:29 sep211125-748067 systemd[1]: Failed to start User Manager for UID 12345.
[11:42] <zyga> google:ubuntu-20.10-64 .../tests/main/snap-session-agent-service-control# tests.session -u test prepare
[11:42] <zyga> Failed to connect to bus: No such file or directory
[11:42] <zyga> what?
[11:43] <mup> PR snapd#9376 closed: check-pr-title.py * : allow "*" in the first part of the title <Simple 😃> <Skip spread> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9376>
[11:45] <pstolowski> zyga: could you take a look at #9270? would be great to land soon now that 2.47 was branched
[11:45] <mup> PR #9270: wrappers, systemd: allow empty root dir and conditionally do not pass --root to systemctl <Run nested> <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9270>
[11:45] <zyga> certainly
[11:47] <zyga> ah I see what changed
[11:48] <mvo> zyga re 20.10 - I suspect it might be more generic than just your pr, i run a spread run now over my lunch to see if it also happens on master
[11:49] <zyga> mvo: https://pastebin.ubuntu.com/p/9yYgZm7Dyk/
[11:49] <zyga> we're missing updates
[11:50] <mvo> zyga I see - something for sergio I guess
[11:50] <zyga> but casual update doesn't seem to fix this :/
[11:50] <zyga> I will try some more
[11:50] <zyga> pstolowski: why are interfaces using NewUnderRoot?
[11:51] <zyga> I mean, the systemd backend
[12:01] <pstolowski> zyga: mhm, i suppose i should change that as well
[12:02] <zyga> pstolowski: I don't know mainly asking
[12:02] <zyga> is the PR description up to date?
[12:07] <pstolowski> it could probably use an update as the scope of changes grew
[12:07] <zyga> pstolowski: let me know if I should review as-is or wait
[12:08] <pstolowski> zyga: i'll update in a moment
[12:08] <zyga> ok
[12:11] <zyga> mvo: I tried upgrading all the packages but that's not enough
[12:11] <zyga> mvo: shall we take 20.10 off the required list or do you want things to limp along red for now?
[12:12] <cmatsuoka> zyga: sorry, I only noticed your message now -- I never rebuilt the core20 snap
[12:12] <zyga> cmatsuoka: hey, no worries
[12:12] <zyga> cmatsuoka: on the up side the encryption was working for a while
[12:13] <cmatsuoka> I think we injected stuff into it during the spike but we never had to rebuild it
[12:14] <zyga> ok, time to get selinux patches in and asses the overal where-are-we status
[12:26] <cmatsuoka> zyga: found this in the pile of triaged-with-undecided-importance bugs, do we have a newer status? https://bugs.launchpad.net/snapd/+bug/1659861
[12:26] <mup> Bug #1659861: Failure results in messy message leaking internals <snapd:Confirmed for chipaca> <https://launchpad.net/bugs/1659861>
[12:27] <zyga> cmatsuoka: nope
[12:27] <zyga> cmatsuoka: the %q using double quotes will be a pain to change
[12:28] <mborzecki> hmm
[12:29] <mborzecki> when pi uc20 system is installed, some state is stored on ubuntu-seed?
[12:32] <zyga> mborzecki: what is stored there?
[12:34] <mborzecki> idk, i've removed ubuntu-boot and ubuntu-data partitions in hope that system would reinstall, but none of that happened
[12:34] <zyga> hmmm
[12:34] <zyga> what happened instead?
[12:36] <zyga> brb
[12:36] <zyga> mborzecki: do you have 10 minutes to look at selinux failures with me?
[12:36] <zyga> maybe in ~30 minutes from now?
[12:36] <zyga> oh
[12:36] <zyga> maybe after standup then
[12:38] <mborzecki> zyga: we have uc20 status after the standup, before maybe?
[12:40] <mborzecki> create user does soemthing weird, it says the user is uknown (not sure why?), but still creates one locally, but without the ssh key so i cannot log in
[12:41] <zyga> mborzecki: 10 minutes before?
[12:41] <mborzecki> zyga: sgtm
[12:41] <zyga> thank you
[12:50] <mborzecki> zyga: standup?
[12:50] <zyga> mborzecki: going
[12:56] <mvo> zyga: yes, I remove it now
[12:56] <pstolowski> zyga: you can look at #9270, i pushed one more change and updated the desc
[12:56] <mup> PR #9270: wrappers, systemd: allow empty root dir and conditionally do not pass --root to systemctl <Run nested> <Services ⚙️> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9270>
[12:57] <mvo> zyga did you talk to cachio about 20.10 and that the image needs an update already? maybe this cann happy quickly?
[12:57] <zyga-mbp> mvo not yet
[12:57] <zyga-mbp> in a call
[13:00] <cachio> mvo, I dont manage 20.10
[13:00] <mvo> cachio: oh, ok
[13:03] <mup> PR snapd#9382 closed: tests: copy /usr/lib/snapd/info to correct directory <Simple 😃> <Test Robustness> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9382>
[13:03] <zyga-mbp> thanks mvo
[13:13] <mup> PR snapd#9384 opened: overlord: export and use snapd tools <Created by zyga> <https://github.com/snapcore/snapd/pull/9384>
[13:53] <mup> PR snapd#9385 opened: spread.yaml: add ubuntu-20.10-64 to qemu <Simple 😃> <Skip spread> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9385>
[14:06] <zyga-mbp> trying those two extra permissions
[14:26] <mborzecki> hmm `snap debug boot-vars` does not quite work for uc20
[14:35] <zyga> mborzecki: most denials fixed
[14:35] <zyga> mborzecki: two remain mount and remount
[14:35] <ijohnson> mborzecki: shouldn't there be a --uc20 flag?
[14:36] <zyga> let me try to fix those myself
[14:36] <ijohnson> mborzecki: oh also it is currently hard-coded to only do extracted run kernel image not env kernel references like on pi
[14:36] <mborzecki> ijohnson: there is, but it's not doing anything useful
[14:37] <ijohnson> right
[14:37] <mborzecki> ijohnson: at least for uboot, it looks at boot.sel only if options is passed to bootloader.Find(), which boot-vars does not do
[14:37] <ijohnson> ahh
[14:37] <ijohnson> yeah good point
[14:37] <mborzecki> maybe we should try --recovery or --run-mode
[14:38] <mborzecki> instead of --uc20 ;)
[14:47] <ijohnson> mvo: pstolowski: should we file a PR making the nested 20.04 preseed variant not die on system-key differences so we can land things again?
[14:51] <pstolowski> ijohnson: we may have no other choice :(. that effectively means disabling it for good though, i don't think we run this test on other systems
[14:52] <zyga> I think I got it
[14:52] <ijohnson> pstolowski: well we can just disable the system-key check at the end of the test for the currently failing combo, and then after that is merged open a pr reverting that change and just keep it open retrying it every once in a while to see if it ever fixes itself with an update to the image
[14:53] <pstolowski> ijohnson: yes, i mean disabling just the system-key check of course. yes, the idea of having a revert PR around sounds good to me
[14:53] <ijohnson> pstolowski: ack I can throw up a pr in a little bit doing that then
[14:54] <pstolowski> thanks, note there are two checks there, another one uses debug api
[14:54] <ijohnson> right
[14:58] <mvo> ijohnson: +1
[15:16] <mvo> mborzecki: can I squash merge 9366?
[15:16] <mvo> mborzecki: I want to cherry pick into 2.47
[15:19] <zyga> hmm
[15:19] <zyga> 2020-09-21T13:43:20.6939339Z error: cannot get snap sections: cannot sections: got unexpected HTTP status code 403 via GET to "https://api.snapcraft.io/api/v1/snaps/sections"
[15:19] <ijohnson> mvo can you also force-merge 9375, the failures were nested preseed, cgroup tracking on arch (unrelated) and another xenial lxd failure (also unrelated)
[15:19] <mborzecki> mvo: yeah, go ahead
[15:19] <zyga> is the store in trouble?
[15:19] <ijohnson> hmm haven't seen 403s lately
[15:22] <mvo> ijohnson: sure
[15:23] <zyga> selinux clean passed!
[15:23] <mup> PR snapd#9366 closed:  gadget: resolve device mapper devices for fallback device lookup <Run nested> <UC20> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9366>
[15:24] <mup> PR snapd#9375 closed: tests/nested/cloud-init-many: simplify tests and unify helpers/seed inputs <Run nested> <Test Robustness> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9375>
[15:24] <zyga> mborzecki: ^^ :D
[15:24] <ijohnson> Thanks!
[15:24] <mvo> mborzecki: thanks, done and cherry-picked into 2.47 so the gadget update can now be tested by cmatsuoka  once the snapd beta got build and published
[15:25] <zyga> https://github.com/snapcore/snapd/pull/9384 needs a 1st pass review now
[15:25] <mup> PR #9384: overlord: export and use snapd tools <Created by zyga> <https://github.com/snapcore/snapd/pull/9384>
[15:29] <zyga> mvo: review advice is strongly encouraged
[15:29] <zyga> mvo: mainly if I should attempt to split it
[15:30] <zyga> mvo: or if I should wait for another pass first
[15:30] <mup> PR snapcraft#3292 opened: readme: remove link to Google+ <Created by timsueberkrueb> <https://github.com/snapcore/snapcraft/pull/3292>
[15:30] <mvo> zyga that the 3k one?
[15:31] <zyga> correct
[15:31] <mborzecki> mvo: yay
[15:31] <zyga> mborzecki: thank you :)
[15:31] <mborzecki> ok, need to go, drive the kids around a bit
[15:31] <zyga> o/
[15:32] <zyga> mvo: happy to have a call if that would help
[15:32] <cachio> pstolowski, hey
[15:33] <cachio> pstolowski, about the preseed test
[15:33] <cachio> pstolowski, which is the way to fix it?
[15:33] <cachio> what I need to change in the image?
[15:34] <mup> PR snapd#9350 closed: store: handle v2 error when fetching assertions <Created by stolowski> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9350>
[15:35] <pstolowski> cachio: we would need same/similiar kernel on the host and in the nested vm. if you can do anything about that then it would be awesome; until then part of the test will be disabled, see earlier discussion with ijohnson
[15:36] <pstolowski> cachio: the problem is that currently apparmor features are different because of kernel change, and this affects system-key
[15:38] <cachio> pstolowski, ok, so, this part should be just skipped in ubuntu-20.04
[15:38] <cachio> the rest shouldn't be affected
[15:39] <cachio> pstolowski, if we change the kernel in that image we have the kvm reboots
[15:39] <cachio> so I think the "skip" should be permanente
[15:39] <cachio> permanent
[15:39] <cachio> pstolowski, does it make sense?
[15:40] <cachio> ijohnson, ~
[15:41] <ijohnson> cachio: yes we should just disable this part of the test
[15:41] <ijohnson> cachio I will open a PR doing this shortly
[15:41] <pstolowski> cachio: yep
[15:43]  * zyga does a small break
[15:44] <zyga> mvo: if you want to talk just TG me
[15:44] <mvo> zyga maybe tomorrow morning? I'll have dinner soon
[15:46] <zyga> sure
[15:46] <zyga> I need to install an SSD
[15:46] <zyga> want to run linux on my ryzen box :)
[15:53] <zyga> error: cannot get snap sections: cannot sections: got unexpected HTTP status code 403 via GET to "https://api.snapcraft.io/api/v1/snaps/sections"
[15:53] <zyga> pedronis: ^ should those be reported to the store?
[16:20]  * zyga EODs
[16:20] <ijohnson> cachio: can you take a look at #9334 today?
[16:20] <mup> PR #9334: tests/nested/manual: add test for grades above signed booting with testkeys <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9334>
[16:20] <zyga> o/
[16:20] <zyga> consider disable searching test
[16:20] <zyga> as it's flaky now
[16:36] <cachio> ijohnson, sure
[16:36] <ijohnson> thanks
[16:37] <ijohnson> cachio: also I just pushed a PR disabling the system-key check on nested/manual/snap-preseed
[16:37] <ijohnson> if you could take a look at that too that would be great
[16:37] <cachio> ijohnson, sure I'll do
[16:37] <ijohnson> thanks
[16:37] <ijohnson> the pr is https://github.com/snapcore/snapd/pull/9386
[16:37] <mup> PR #9386: tests/nested/manual/preseed: disable system-key check for 20.04 image <Run nested> <Simple 😃> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9386>
[16:39] <mup> PR snapd#9386 opened: tests/nested/manual/preseed: disable system-key check for 20.04 image <Run nested> <Simple 😃> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9386>
[16:54] <cachio> ijohnson, some comments in 9334
[16:54] <ijohnson> thanks, looking now
[16:54] <cachio> t
[16:54] <cachio> x
[16:57]  * cachio lunch
[18:41] <zyga> ijohnson, cmatsuoka: I tried core 20 from cdimage , the beta directory and that's both old and did not boot correctly
[18:41] <zyga> is there a different beta I should be using?
[18:41] <zyga> or is that expected
[18:41] <ijohnson> zyga: hmm
[18:41] <ijohnson> old beta should work
[18:41] <ijohnson> zyga: which specific link did you download ?
[18:41] <zyga> let me find that
[18:41] <cmatsuoka> zyga: what image did you use exactly? I can test it here
[18:41] <zyga> http://cdimage.ubuntu.com/ubuntu-core/20/dangerous-beta/current/ubuntu-core-20-amd64.img.xz
[18:42] <ijohnson> huh that should work
[18:42] <zyga> let me boot again and see the error message
[18:42] <zyga> it was about some /dev/disk/by-uuid
[18:43] <ijohnson> ah interesting that could be a real bug in snap-bootstrap
[18:43] <ijohnson> depending on how vmware sets up the vm hard disk
[18:43] <zyga> hold on for a sec
[18:43] <zyga> wait for the ubuntu core choose trigger finished
[18:43] <zyga> now it's waiting for ubuntu-seed
[18:44] <zyga> with 32s out of 1.5m timeout
[18:44] <zyga> I have a feeling we lack kernel modules for storage
[18:44] <zyga> the same thing happened with nvme disk
[18:44] <zyga> this is a scsi virtual disk now
[18:44] <zyga> probably real hardware will have similar issues
[18:44] <zyga> it failed now
[18:45] <cmatsuoka> 0529 is the "good" one and should work, let me test just in case...
[18:45] <ijohnson> zyga: how many boots did it do?
[18:45] <zyga> two now
[18:45] <zyga> ok, I have a screenshot
[18:45] <zyga> mattermost?
[18:45] <ijohnson> zyga: was it on the first one that it failed? or did it successfully boot then reboot and fail
[18:45] <ijohnson> zyga: sure
[18:46] <ijohnson> we have a ~uc20 channel on mm
[18:46] <ijohnson> yeah that image booted fine for me in qemu on focal
[18:47] <zyga> ijohnson I wasn't paying attention all the time
[18:47] <zyga> it seems like first boot
[18:47] <zyga> sent to uc20 chann
[18:47] <ijohnson> zyga: thanks
[18:47] <ijohnson> I see the image
[18:47] <ijohnson> ah actually maybe let's just move this convo to mm
[18:48] <zyga> let me try with sata disk
[19:40] <mup> PR snapd#9385 closed: spread.yaml: add ubuntu-20.10-64 to qemu <Simple 😃> <Skip spread> <Created by mvo5> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9385>
[20:15] <mup> PR snapd#9386 closed: tests/nested/manual/preseed: disable system-key check for 20.04 image <Run nested> <Simple 😃> <⚠ Critical> <Created by anonymouse64> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9386>
[20:20] <ijohnson> thanks cachio
[20:23] <cachio> yaw
[20:25] <mup> PR snapd#9387 opened: interfaces/builtin: add usbmuxd interface <Created by qmfrederik> <https://github.com/snapcore/snapd/pull/9387>
[20:35] <mup> PR snapcraft#3292 closed: readme: remove link to Google+ <Created by timsueberkrueb> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3292>
[20:40] <mup> PR snapd#9388 opened: bootloader: lk cleanups <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9388>