=== jamesh_ is now known as jamesh [06:13] morning [06:38] mvo: hey [06:38] good morning mborzecki [06:41] PR snapd#9377 closed: [RFC] snap-repair: minimal uc20 support [06:52] mvo: I put together https://forum.snapcraft.io/t/adding-users-to-system-groups-on-ubuntu-core/20109 with my thoughts on handling groups, in prep for tomorrow's meeting. [06:52] mvo: heads up, google-nested:ubuntu-20.04-64:tests/nested/manual/preseed seems to be failing (i've pinged pawel in one of the prs) [06:53] jamesh: thanks, excellent, let me read this [06:53] mborzecki: oh no, any more clues yet? [06:54] mvo: the preseed key is different, https://github.com/snapcore/snapd/pull/9380#issuecomment-695933603 [06:54] PR #9380: tests: update to support nested kvm without reboots on UC20 [06:56] mborzecki: ok, that sounds not too bad [07:06] re [07:06] good morning! [07:07] morning [07:08] o/ [07:09] good morning pstolowski and zyga [07:10] hmm, more failures [07:10] but this looks like a bug in repacking core [07:10] * zyga looks [07:10] hmm [07:11] mvo: today started more like normal, wife has afternoon shift today [07:11] pstolowski: zyga: hey [07:12] pstolowski: pinged you about nested preseed test failure, please take a look [07:12] mvo: oh no, umatrix development is over [07:12] hey mborzecki :) [07:12] it's real autumn now [07:12] pstolowski: https://github.com/snapcore/snapd/pull/9380#issuecomment-695933603 [07:12] PR #9380: tests: update to support nested kvm without reboots on UC20 [07:13] zyga: yeah, very much so, this is what it looked like during my afternoon ride yesterday https://dgtzuqphqg23d.cloudfront.net/O1Yi0M_6L-sWwaf-Rxs-NoxxaZjK96HwWdDgXIato6U-1536x2048.jpg [07:13] ohh [07:13] * zyga envies bike riding [07:13] in a few months [07:14] my back still hurts a little when I bend [07:14] forecast says 24C today so maybe I will go for a walk instead [07:14] zyga: and it gets cold fast, 20C when i was leaving, 16C when i was coming back, stopped to take a couple of photos and i felt like i was freezing for the last 7km [07:14] mborzecki: I'll try some selinux bits today, I got nowhere on Friday [07:14] mborzecki: that's evening ligth? [07:15] yeh [07:15] yeah, I need to find my full shoes [07:15] I had sporty hole-ridden shoes all summer [07:16] zyga: umatrix development over? nooooo, that's super sad. where did you see that? [07:18] mvo: ./ [07:18] mvo: github repo is archived [07:23] hmm [07:23] weird [07:23] zyga: meh, so sad. I can't see anything on the readme about the why [07:23] ha [07:23] I found a bug [07:24] repack_snapd_snap_with_deb_content_and_run_mode_firstboot_tweaks :) [07:24] that's a mouthful [07:24] and identical copy-pasted bug in repack_snapd_snapd_with_deb_content [07:25] I'll send noise and bug fixes separately in a moment [07:25] just want to see green core tests [07:25] zyga: nice [07:25] man it's COLD in the office today [07:25] definitely sub 20 [07:25] probably ~15 [07:26] mborzecki: replied, you're correct there [07:36] zyga: pushed a branch again? ;) i see wip/export-manager-patch-set in the latest pull [07:38] hmm [07:38] mborzecki: gaah [07:38] * zyga wonders why that happens [07:38] mborzecki: cleaned that up [07:39] we don't seem to repackage core when doing core18 tests [07:40] ah [07:40] indeed [07:40] okay [07:40] zyga: i normally set origin to be my fork [07:42] oh [07:42] conceptual question [07:42] when testing ubuntu core [07:42] should we repackage both core and snapd [07:42] so that they are in sync [07:43] I wrote my test assuming as much [07:43] i think the only reason we repack core is to setup the logging (and iirc we do it only in nested) [07:44] w8, core as in core20 [07:44] core is probably repacked with newer snapd anyway [07:44] we repackage both for the same reason [07:44] to get new snapd [07:44] my question was more subtle [07:44] should we always have both repackaged [07:45] regardless of the version of ubuntu core [07:46] right now if you run a test with ubuntu-core-18 [07:46] and install something that pulls in core [07:46] you get core tools [07:46] with old snap-exec [07:46] mvo, pedronis: ^ opinion please [07:46] I can adjust my test or adjust the test suite either way [07:47] * mvo is in a meeting [07:52] * zyga does the simpler thing [08:04] passed === pedronis_ is now known as pedronis [08:38] re [08:42] PR snapd#9345 closed: overlord: introduce the export manager, export snapd tools [09:06] quick review required for https://github.com/snapcore/snapd/pull/9382 [09:06] PR #9382: tests: copy /usr/lib/snapd/info to correct directory [09:07] optionally for https://github.com/snapcore/snapd/pull/9381 but not important [09:07] PR #9381: many: typos and small test tweak [09:07] PR snapd#9381 opened: many: typos and small test tweak [09:07] PR snapd#9382 opened: tests: copy /usr/lib/snapd/info to correct directory [09:12] pstolowski, mborzecki: ^ [09:22] yay, pi4 8gb with today's uc20 image is up [09:23] mborzecki: nice :) [09:39] mborzecki: trying core20 with tpm on vmware now [09:50] mborzecki booting now [09:50] no such device ubuntu boot [09:51] now black screen [09:51] now efi stub secure boot enabled [09:51] now wall of text [09:51] booting :) [09:52] so far so good [09:52] zyga-mbp: there will be a log during install that encryption is enabled [09:52] mborzecki: can you look at https://github.com/snapcore/snapd/pull/9382 [09:52] PR #9382: tests: copy /usr/lib/snapd/info to correct directory [09:52] I need to merge itr [09:52] ok [09:52] hmm [09:53] got a load of IO errors [09:54] tweaked vm and rebooted [09:59] mborzecki no luck yet [10:00] mborzecki it could not boot wuth disk type=nvme [10:00] secure boot enabled [10:00] wall of kernel booting [10:00] I switched to sata [10:00] it's starting [10:01] mborzecki tpm device detected and enabled! [10:01] rebooting [10:01] :D [10:01] I think it will work [10:02] hmm, now rebooted and looking at efi boot logo [10:02] with "attempting to start up from efi network" [10:02] it's either not clearing the buffer [10:02] or just booting silently [10:02] mborzecki are we doing anything for efi vars to select boot method? [10:02] kernel up! [10:02] it should reboot from install to run , if it gets far enough [10:03] it tries to do that automatically [10:03] I think it's a winner :) [10:03] seeding [10:03] I should expect console conf, right/ [10:04] now the last thing is "stopping snap daemon" [10:04] (from systemd) [10:04] started getty! [10:05] it's still doing stuff [10:05] waiting for server to restart [10:05] I guess that's "snap run" message [10:06] more getty changes [10:07] my only feedback so far is lots of black screen and waiting :) [10:08] and no CPU usage [10:08] hmmm [10:08] brb, small break to move [10:25] It worked :) [10:26] there was a backtrace visible in console conf after initial setup [10:26] hmm [10:26] (super brief, it's gone now) [10:26] I wonder if I can create an account [10:27] pedronis: hey, do you have a moment (~10 mins i guess) to chat about validation-sets? [10:27] now I have black screen with blinky cursor and nothing else [10:28] I think console conf does crash [10:28] I'm back in the setup screen [10:28] yep [10:28] meh [10:28] that's anti-climactic [10:28] guess that is it [10:29] the screen flash is so quick I have no hope of seeing what crashed [10:29] let me see if I can record it [10:32] got it [10:32] the traceback is requests.exceptions.ConnectionError [10:32] Connection refused! [10:32] didn't someone mention that recently? [10:41] meh, 9383 should be a draft - oh well [10:42] re, stupid suspend [10:42] anyway, is there a way to debug a dangerous model? [10:42] I'd like to see what's wrong with snapd [10:42] zyga-mbp: yes, press ESC during boot to get grub (you need the second grub for run mode) [10:42] PR snapd#9383 opened: [RFC] snap-repair: minimal uc20 support [10:43] zyga-mbp: and then change the kernel commandline to include [10:43] zyga-mbp: "dangerous systemd.debug-shell=1" and maybe "snapd.debug=1" [10:45] mvo second grub? [10:45] ah [10:45] second menu entry? [10:47] I have the grub menu now [10:47] booting! [10:48] for the record, I edited the recovery boot menu [10:48] hmm [10:48] now it asks me for recovery key :-( [10:48] so I guess that's SOL [10:48] mborzecki ^ any ideas? [10:48] hm? [10:49] mborzecki how to debug encrypted vm [10:49] snapd isn't up yet? [10:50] zyga-mbp: i've built custom snapd snap with this patch https://github.com/bboozzoo/snapd/commit/a1ec67e26bc5fde9ef645bcc8d6e127345cee344 then you get a debug shell [10:52] zyga-mbp: oh, of course, you won't be able to open the disk if the cmdline changed [10:53] snapd was partially up [10:53] it seeded and console conf is a snap after all [10:53] let me show you the backtrace [10:56] mborzecki I sent the screenshot via telegram [10:57] yeah, looks like snapd isn't up yet (or not quite working) [10:58] mborzecki just to be clear, can I do anything right now with this VM or is it gone and buseteD? [11:01] zyga-mbp: you can wait a bit more [11:02] mborzecki ok, I'll give it some time [11:02] zyga-mbp: otherwise, you can try to build a new image, repack the core20 snap and add logging to console [11:02] but it's weird [11:02] console conf is up but snapd is not responding? [11:02] hmmmm [11:02] now something weird happened [11:02] boot fails [11:03] failed yo boot both default and fallback [11:03] and I only see one item in grub [11:03] like something failed and it generated different grub menu with just one broken entry [11:04] and I get /EFI/ubuntu/kernel.efi not found [11:04] repeats across reboot [11:04] zyga-mbp: woah, that looks bad [11:04] indeed [11:05] shut down, rebooting [11:05] well, booting afresh [11:05] same [11:05] I think this is busted [11:06] I wonder if there's some case where snapd starts up [11:06] and something is wrong where it seems "no systems are available" [11:06] and just generates garbage [11:06] before it broke I saw a few more entries [11:07] including run, recover, reinstall and EFI setup [11:07] now I only see one [11:07] and EFI setup is oddly gone [11:07] as if the grub config was truncated somehow? [11:08] speaking of strange stuff - looks like 20.10 test are broken? or is this a one-off thing? [11:10] mvo what did you see? [11:13] PR snapd#9381 closed: many: typos and small test tweak [11:13] thanks mvo! [11:14] zyga-mbp: looks like XDG_RUNTIME_DIR is no longer defined [11:14] zyga-mbp: in 20.10 [11:14] zyga-mbp: maybe something in the image changed? [11:14] zyga-mbp: I think I will ask cachio once he is around [11:14] pstolowski: hi, we can do now or before the standup? also I re-reviewed #9350 (thanks) [11:14] PR #9350: store: handle v2 error when fetching assertions [11:15] hmm? [11:15] hmm [11:16] pedronis: thanks for the review. now works for me [11:16] pstolowski: let's do now then [11:17] pstolowski: going to the standup HO [11:17] ok [11:21] * zyga-mbp looks at 20.10 failure [11:22] mvo hmm, maybe fixing info influenced snapd reexec [11:22] so now we are testing what we are running [11:22] and something used to be maskeD? [11:22] * zyga-mbp debugs [11:41] hmm [11:41] Sep 21 11:39:29 sep211125-748067 systemd[1]: Starting User Manager for UID 12345... [11:41] Sep 21 11:39:29 sep211125-748067 systemd[39458]: user@12345.service: Failed to determine user credentials: No such process [11:41] Sep 21 11:39:29 sep211125-748067 systemd[39458]: user@12345.service: Failed at step USER spawning /lib/systemd/systemd: No such process [11:41] Sep 21 11:39:29 sep211125-748067 systemd[1]: user@12345.service: Main process exited, code=exited, status=217/USER [11:41] Sep 21 11:39:29 sep211125-748067 systemd[1]: user@12345.service: Failed with result 'exit-code'. [11:41] Sep 21 11:39:29 sep211125-748067 systemd[1]: Failed to start User Manager for UID 12345. [11:42] google:ubuntu-20.10-64 .../tests/main/snap-session-agent-service-control# tests.session -u test prepare [11:42] Failed to connect to bus: No such file or directory [11:42] what? [11:43] PR snapd#9376 closed: check-pr-title.py * : allow "*" in the first part of the title [11:45] zyga: could you take a look at #9270? would be great to land soon now that 2.47 was branched [11:45] PR #9270: wrappers, systemd: allow empty root dir and conditionally do not pass --root to systemctl [11:45] certainly [11:47] ah I see what changed [11:48] zyga re 20.10 - I suspect it might be more generic than just your pr, i run a spread run now over my lunch to see if it also happens on master [11:49] mvo: https://pastebin.ubuntu.com/p/9yYgZm7Dyk/ [11:49] we're missing updates [11:50] zyga I see - something for sergio I guess [11:50] but casual update doesn't seem to fix this :/ [11:50] I will try some more [11:50] pstolowski: why are interfaces using NewUnderRoot? [11:51] I mean, the systemd backend [12:01] zyga: mhm, i suppose i should change that as well [12:02] pstolowski: I don't know mainly asking [12:02] is the PR description up to date? [12:07] it could probably use an update as the scope of changes grew [12:07] pstolowski: let me know if I should review as-is or wait [12:08] zyga: i'll update in a moment [12:08] ok [12:11] mvo: I tried upgrading all the packages but that's not enough [12:11] mvo: shall we take 20.10 off the required list or do you want things to limp along red for now? [12:12] zyga: sorry, I only noticed your message now -- I never rebuilt the core20 snap [12:12] cmatsuoka: hey, no worries [12:12] cmatsuoka: on the up side the encryption was working for a while [12:13] I think we injected stuff into it during the spike but we never had to rebuild it [12:14] ok, time to get selinux patches in and asses the overal where-are-we status [12:26] zyga: found this in the pile of triaged-with-undecided-importance bugs, do we have a newer status? https://bugs.launchpad.net/snapd/+bug/1659861 [12:26] Bug #1659861: Failure results in messy message leaking internals [12:27] cmatsuoka: nope [12:27] cmatsuoka: the %q using double quotes will be a pain to change [12:28] hmm [12:29] when pi uc20 system is installed, some state is stored on ubuntu-seed? [12:32] mborzecki: what is stored there? [12:34] idk, i've removed ubuntu-boot and ubuntu-data partitions in hope that system would reinstall, but none of that happened [12:34] hmmm [12:34] what happened instead? [12:36] brb [12:36] mborzecki: do you have 10 minutes to look at selinux failures with me? [12:36] maybe in ~30 minutes from now? [12:36] oh [12:36] maybe after standup then [12:38] zyga: we have uc20 status after the standup, before maybe? [12:40] create user does soemthing weird, it says the user is uknown (not sure why?), but still creates one locally, but without the ssh key so i cannot log in [12:41] mborzecki: 10 minutes before? [12:41] zyga: sgtm [12:41] thank you [12:50] zyga: standup? [12:50] mborzecki: going [12:56] zyga: yes, I remove it now [12:56] zyga: you can look at #9270, i pushed one more change and updated the desc [12:56] PR #9270: wrappers, systemd: allow empty root dir and conditionally do not pass --root to systemctl [12:57] zyga did you talk to cachio about 20.10 and that the image needs an update already? maybe this cann happy quickly? [12:57] mvo not yet [12:57] in a call [13:00] mvo, I dont manage 20.10 [13:00] cachio: oh, ok [13:03] PR snapd#9382 closed: tests: copy /usr/lib/snapd/info to correct directory [13:03] thanks mvo [13:13] PR snapd#9384 opened: overlord: export and use snapd tools [13:53] PR snapd#9385 opened: spread.yaml: add ubuntu-20.10-64 to qemu [14:06] trying those two extra permissions [14:26] hmm `snap debug boot-vars` does not quite work for uc20 [14:35] mborzecki: most denials fixed [14:35] mborzecki: two remain mount and remount [14:35] mborzecki: shouldn't there be a --uc20 flag? [14:36] let me try to fix those myself [14:36] mborzecki: oh also it is currently hard-coded to only do extracted run kernel image not env kernel references like on pi [14:36] ijohnson: there is, but it's not doing anything useful [14:37] right [14:37] ijohnson: at least for uboot, it looks at boot.sel only if options is passed to bootloader.Find(), which boot-vars does not do [14:37] ahh [14:37] yeah good point [14:37] maybe we should try --recovery or --run-mode [14:38] instead of --uc20 ;) [14:47] mvo: pstolowski: should we file a PR making the nested 20.04 preseed variant not die on system-key differences so we can land things again? [14:51] ijohnson: we may have no other choice :(. that effectively means disabling it for good though, i don't think we run this test on other systems [14:52] I think I got it [14:52] pstolowski: well we can just disable the system-key check at the end of the test for the currently failing combo, and then after that is merged open a pr reverting that change and just keep it open retrying it every once in a while to see if it ever fixes itself with an update to the image [14:53] ijohnson: yes, i mean disabling just the system-key check of course. yes, the idea of having a revert PR around sounds good to me [14:53] pstolowski: ack I can throw up a pr in a little bit doing that then [14:54] thanks, note there are two checks there, another one uses debug api [14:54]