[05:29] morning [06:45] PR snapd#9054 closed: bootloader/assets: generate bootloader assets from files [08:43] it's nice to see green ticks from CI again [08:48] yes [08:48] mborzecki: catching up on some things, and then I'll review 9006 first [08:48] pedronis: thanks! [08:49] Saviq: hey, multipass does not plug removable media? [08:55] good morning [08:56] PR snapd#9060 opened: interfaces/bultin/multipass: replace U+00A0 no-break space with simple space [08:58] zyga: hi [08:58] hey pedronis [08:58] sorry for starting late, it's extremely difficult to sleep lately [08:58] only one day left though [09:00] zyga: how do you want to proceed, start cutting the release this morning, or in the afternoon? I can provide a cleaned up changelog [09:01] pedronis: I would like to look at the release branch now, run tests manually, review the instructions from Mvo and get everything ready for early afternoon [09:01] zyga: ok [09:01] zyga: I can do the part of the instruction that is producing the changelong content and cleaning it up [09:01] zyga: ping me when you need that [09:01] zyga: thank you [09:01] thank you, that will be most welcome [09:05] 100 patches between 2.25.2 and release head [09:37] zyga: anything unexpected there? [09:39] mborzecki: nothing yet [09:39] I forgot we did some interface changes backports [09:41] most is just noise and test tweaks [09:44] brb [09:45] zyga: the condesed and edited changelog from mvo tool is: https://gist.github.com/pedronis/c1577c450d53ff3c9fe3086447726232 [09:46] ack, looking [09:47] nicely readable [09:50] I've gone through all the patches and everything looks good [09:50] I'll run tests two more times while reading mvo's instructions [09:58] mborzecki: did a pass on #9006, let me know if you have questions [09:58] PR #9006: bootloader: compose command line with mode and extra arguments [10:02] one thing I found in the changelogs that we don't have in mvo's doc is the SRU tracking bug [10:02] for 2.45 we used https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1875071 [10:02] Bug #1875071: [SRU] 2.45

Eoan):Fix Released> [10:02] but it seems to mention 2.45.1 as well [10:02] I wonder if I should file a placeholder bug for 2.45.3 [10:03] first run: two tests failed in fedora [10:03] well, centos and fedora [10:03] - google:centos-8-64:tests/main/selinux-clean [10:03] - google:fedora-30-64:tests/main/snap-run [10:03] but worse, preparing arch failed entirely [10:03] * zyga looks [10:08] zyga: ah, w8, there was a patch for zstd [10:08] mborzecki: I think I saw one in the patches [10:09] it's not that [10:09] zyga: 4de033c77e742c166dd075fd5e9db507a99b17dc this one? [10:09] https://paste.ubuntu.com/p/wdHDqkXFKy/ [10:09] zyga: hm that one i cannot fix [10:10] yes, that patch is in the release [10:10] heh, only on release day :) [10:10] that looks like google's own arch repo? [10:10] zyga: yeah, not sure where it's coming from [10:11] tbh i didn't know there's one even [10:16] different failure now [10:16] Found unused ./vendor packages: [10:16] vu github.com/snapcore/secboot/internal/pe1.14 [10:16] Please fix via 'govendor remove +unused' [10:16] * zyga looks [10:17] zyga: pacman -Sy seems to work, so maybe there really is a repo, but it's only accessible form the nodes? [10:21] PR snapd#9060 closed: interfaces/builtin/multipass: replace U+00A0 no-break space with simple space [10:27] zyga: does it work now? [10:33] now things pass [10:33] mborzecki: it's weird, I removed vendor bits and ran tests without them [10:33] why didn't any other system complain? [10:34] zyga: i have no idea, arch should be using go 1.14.6 atm, maybe something about it [10:37] mborzecki: about the other failures, the snap-run test hangs on strace sometimes so that's not new [10:37] mborzecki: and the centos 8 selinux policy situation has not changed, correct? [10:37] zyga: nope, i should fix that test sometime [11:16] PR snapd#9061 opened: packaging: add placeholder changelog for 2.45.3 [11:16] changelogs are ready, I created a PR for the placeholder entry, I've cloned the trello card and I'm looking at the card now [11:17] I don't think I've ever touched snapd translations on launchpad [11:17] not sure if anyone knows how that operates [11:27] zyga: mvo maybe (hopefully)? [11:27] yeah, I wish I knew [11:27] I think we should rotate this more often [11:27] feels a bit "muscle memory" [11:31] PR snapcraft#3227 opened: meta: fix content slot to set content when using source key [11:47] brb [11:52] re [12:03] pedronis: about https://github.com/snapcore/snapd/pull/9006#discussion_r460764489 you mean the same such that the order stays unchanged, i.e. mode/system as specified last? [12:03] PR #9006: bootloader: compose command line with mode and extra arguments [12:06] mborzecki: the order yes, or anything else, formulated otherwise my question is whether the final command lines are changing or are the same [12:06] vs what we have on master right now [12:09] pedronis: hm it does change, that's why i bumped the edition number to 2 initially [12:09] mborzecki: can you make a pastebin of old and new somewhere? [12:09] side by side [12:10] pedronis: grub.cfg and recovery? [12:10] the command lines [12:10] for both [12:11] pedronis: ok, let me look at the scripts again, this has changed a bunch of times during the reviews [12:18] pedronis: heh, i was wrong, it changed twice in the course of reviews, at the end of the day it's the same as in master (where are a new thing and not used atm), see https://github.com/snapcore/snapd/pull/9006/files#diff-6f9b97376e62185c79af158d2d3c9474 and https://github.com/snapcore/snapd/pull/9006/files#diff-d3f0c595846626e6f78fe9b29f0a61e6 [12:18] PR #9006: bootloader: compose command line with mode and extra arguments [12:19] pedronis: that diff chunk in boot is caused by a fix in bootloadertest mocks [12:20] mborzecki: good, my impression was also that in the end the result was the same [12:21] pedronis: while at it, i'll tweak the interface to have CommandLine(mode, system, extraArgs string) [12:22] sounds good [12:23] mborzecki: to be clear mode and system there are full "name=value", right? [12:26] PR snapcraft#3228 opened: review tools: fix snap link or copy path [12:39] pedronis: yes, just like in the PR now, the only difference is the call will be eg. CommandLine("snapd_recovery_mode=recover", "snapd_recovery_system=123123", "arg=1 other=2") [12:40] thx [12:40] pedronis: and grub bootloader implementation verifies snapd_recovery_mode= and snapd_recovery_system= prefixes [12:41] mborzecki: the extra args in cmdline should come only from gadget or also from system settings? [12:41] * zyga just got the call [12:41] 12:30 tomorrow [12:41] finally [12:42] cmatsuoka: gadget for now, i recall we discussed some system args, but so far only gadget was something we though would happen [12:43] ok. I asked because system settings is mentioned once in the doc intro but there are no details about that [12:43] brb, see you at the standup [12:43] zyga: nice, feeling anxious? [12:50] no, just wish it was tomorrow already [13:12] PR snapd#9062 opened: releasing package snapd version 2.45.3 [13:37] PR snapd#9061 closed: packaging: add placeholder changelog for 2.45.3 [13:41] zyga: sorry I was confused, obviously edge builds from master normally, but there is this: https://launchpad.net/~snappy-dev/+snap/snapd-2.45 [13:41] ah, nice, it goes to beta directly [13:42] and it builds automatically [13:43] it goes to a branch of beta to be clear [13:44] hmm, right, I see that now [13:45] zyga: and there's https://launchpad.net/~snappy-dev/+snap/core-beta [13:45] for core [13:45] that one is manual though [13:45] and this one is built on request, so we can trigger when the ppa is ready [13:45] yes [13:45] you can trigger the other one as well if needed [13:53] * zyga breaks for 30 minutes for meds and lunch [14:01] zyga: lp:snapd will need to be imported though for snapd build to work with the right version [14:01] pedronis: uc20 status meeting ? [14:02] omw [14:32] pedronis: cmatsuoka: i've updated #9006 [14:32] PR #9006: bootloader: compose command line with mode and extra arguments [14:55] mborzecki: looked, did you see this comment: https://github.com/snapcore/snapd/pull/9006#discussion_r460770887 ? is that something we should discuss? [14:55] PR #9006: bootloader: compose command line with mode and extra arguments [15:02] PR snapd#9062 closed: releasing package snapd version 2.45.3 [15:09] zyga: I triggered a repo import for that ^ [15:10] thank you [15:10] I tried looking at the patch for git-buildpackage, it's a bit unfortunate that we have to carry it [15:11] zyga: https://code.launchpad.net/~niemeyer/snapd/+git/snapd [15:11] 2.45 is still at the code from a few days ago [15:12] zyga: yes, look, it's says an import will run soon [15:12] it doesn't start immeditately [15:12] yeah, refreshing the page shows stuff is changing [15:13] "Repository scan failed" [15:13] clicking the rescan button [15:13] not allowed [15:13] fun [15:13] I guess [15:14] what does that even mean? [15:14] git import into launchpad database? [15:14] * zyga asks on #launchpad [15:17] zyga: the import says it worked, but the branches aren't updated yet [15:18] ijohnson: are you blocked on reviews by me? [15:18] the pain is under control now, I'll get some tea and be back soon [15:18] pedronis: no not at the moment [15:19] pedronis: sorting out some things locally then I will push to pawel's debug seeding branch for the devicestate changes then open another PR with the command itself [15:19] ijohnson: ok, thx [15:23] it worked now [15:23] zyga: snapd:release/2.45 is up to date on LP now [15:24] yep [15:24] thee snapd-2.45 should start on its own soon [15:24] *the snapd-2.45 build [15:25] I [15:25] I'll dput to the ppa now [15:32] * cachio lunch [15:56] lintian says we're a bit rusty on various details [15:56] but that's fine [15:57] zyga: which package isn't? :) [15:57] mborzecki: hey, my toy package is clean like a ... clean package [15:58] dput complete [15:58] sorry I took forever, I kept struggling with gdp [15:58] *gbp [16:04] zyga: so my theory was wrong, we have snapd snaps but they have ugly versions [16:07] zyga: did you create the wrong kind of tag? or wasn't it pushed? [16:07] I don't see the tag if I grab release 2.45 [16:07] hmmmmm [16:07] I opened a PR but that didn't carry the tag! [16:07] I guess I can just push it now [16:07] zyga: you need to push the tag [16:07] done [16:08] and it's there [16:08] I'm sorry, I entirely missed that the tag is a separate entity [16:08] we need to rebuild the snap now, right? [16:08] hm it ended up on a weird commit, but i guess this is not a problem [16:09] zyga: not sure, I'm still getting weird versions [16:13] zyga: no, I fear how the tag was attached matters [16:13] I get the right version if I go to the tag commit but I can the wrong one if I go to the tip of release/2.45 [16:15] hmmmm [16:15] I suspect I know why [16:15] mvo pushes that to release directly [16:15] I could not [16:16] so I created a merge PR [16:16] so we have a new commit [16:16] let me see if I can do something [16:16] ok [16:18] do we need another lp import? [16:19] I haven't solved the problem, I'm not entirely sure how mvo pushes to those branches [16:20] maybe he has extra permissions [16:21] next time we should record mvo doing all those things and discuss the details [16:24] ppa is still building [16:25] ah no, status was stale [16:25] it failed [16:25] on aarch64 [16:25] on unit tests [16:25] pedronis: https://github.com/snapcore/snapd/pull/9035 is now ready for you to review [16:25] PR #9035: o/devicestate: save seeding/preseeding times for use with debug seeding api (3/N) [16:26] TestClientTimeoutLP1837804 [16:26] ... error string = "cannot communicate with server: Post http://127.0.0.1:40893/: dial tcp 127.0.0.1:40893: i/o timeout" [16:26] ... regex string = ".* timeout exceeded while waiting for response" [16:31] zyga: now it should work, forced pushed to 2.45 the commit with the tag, instead of the merge [16:31] great [16:31] zyga: please double check that 2.45 is sane [16:31] so now we need another lp import [16:31] sure, doing so now [16:31] yes [16:32] scheduled one just now [16:32] thx [16:33] did you force push the tag? [16:33] I'm looking at https://github.com/snapcore/snapd/commit/b055d1902de062efe536f2c3f7dd7d2a35db2c61 [16:35] zyga: the tag is there: https://github.com/snapcore/snapd/tree/2.45.3 [16:35] ah, it's just UI that confused me [16:35] ok [16:36] ijohnson: thanks, I'll try to look still today [16:37] pedronis: thanks, also https://github.com/snapcore/snapd/pull/9063 is the next one which is the snap debug command bit [16:37] PR #9063: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) [16:37] (which should be the last bit, I included the spread test bit there too, it's fairly small, but if you prefer I can make the spread test another PR) [16:37] oh, ijohnson, can you look at https://github.com/snapcore/snapd/pull/9057 please - I removed the environment entry, it was unjused anyway [16:37] PR #9057: overlord: use new tracking cgroup for refresh app awareness [16:37] *unused [16:37] PR snapd#9063 opened: cmd/snap/debug: add "snap debug seeding" command for preseeding debugging (4/N) [16:38] zyga: sure, I have an in progress review I kept getting distracted from :-) [16:38] OK [16:38] ijohnson: added to my queue [16:38] zyga: and in case I don't get a chance to say it before tomorrow, good luck tomorrow! really hoping things go well for you [16:38] thanks pedronis [16:39] I will address some feedback and do reviews now, but then I will look into the re-exec bug for preseeding then [16:41] zyga: I requested a manual build of the snap [16:42] zyga: snapd is building now [16:42] great, the lp import has succeeded as well [16:48] zyga: +1d [16:48] woot [16:48] thank you [16:48] no, thank you for sticking with this, we're really close I think :-) [16:49] * zyga listens to the thunder and rain outside [16:49] ijohnson: and before the anniversary [16:49] of the v2 branch at least :) [16:49] :-D [16:49] once this lands the rest is just code removal [16:49] and some misc changes unrelated to this feature [17:06] the versions now look good [17:06] on the beta/2.45 branch that is [17:07] the ppa is still building for some architectures [17:07] once that is done we should trigger https://launchpad.net/~snappy-dev/+snap/core-beta [17:07] I will break for an hour and come back [17:07] need to double check my hospital bag [17:08] and see what's going on at home [17:11] PR snapcraft#3229 opened: errors: introduce HostToolNotFoundError exception [17:17] ijohnson, hey [17:17] hi cachio [17:17] I was just about to break for lunch [17:17] (just fyi) [17:18] do you know what happens during initial boot after the line systemd[1]: Started Journal Service. --> https://paste.ubuntu.com/p/cBK2cnG6NR/ [17:18] ijohnson, I se most of the random reboots after that line [17:18] ijohnson, np [17:18] cachio: looking quickly [17:19] ijohnson, also after line systemd-journald[185]: Received client request to flush runtime journal. [17:20] cachio: I think that just happens to be one of the last things that systemd will do before starting up snapd, etc. [17:20] I don't think what is triggering those lines specifically would be what is causing the random reboot [17:20] but I dunno maybe it is [17:20] ijohnson, so then snap starts [17:20] cachio: but now immediately after that point [17:21] cachio: but not immediately after that point [17:21] ijohnson, ok [17:21] I have some ideas for more debugging, but I need to break for lunch now === ijohnson is now known as ijohnson|lunch [17:21] ijohnson|lunch, is it any way to see a log with that info? [17:22] after that line in the log that I got from the nested vm I see [17:22] [ 14.067916] systemd-journald[539]: Received client request to flush runtime journal. [17:31] cmatsuoka, any idea? [17:32] PR snapcraft#3228 closed: review tools: fix snap link or copy path [17:32] PR snapcraft#3230 opened: lifecycle: check for snap using shutil.which() [17:32] let me see... [17:34] cachio: this is the entire log, and the system crashed after that? [17:35] cmatsuoka, after that the system works well [17:35] this is the full log [17:35] but when it crashes, it crashes at that point after userspace is running, is that correct? [17:36] [ 62.115886] systemd[1]: Starting Remount Root and Kernel File Systems... [17:36] [ 62.154860] systemd[1]: Starting udev Coldplug all Devices... [17:36] [ 62.201278] systemd[1]: Finished Create list of static device nodes for the current kernel. [17:36] [ 62.237719] systemd[1]: modprobe@drm.service: Succeeded. [17:36] [ 62.263067] systemd[1]: Finished Load Kernel Module drm. [17:36] [ 62.275837] systemd[1]: Started Journal Service. [17:37] PR snapcraft#3227 closed: meta: fix content slot to set content when using source key [17:37] after that line at some point it crashed [17:37] maybe addming something like systemd.log_level=debug systemd.log_target=console to the cmline could help? [17:37] cmatsuoka, but not log [17:38] cmatsuoka, is it possible to update the command line with tpm and secboot enabled?? [17:39] hmm, you would need to add the parameters to the cmdlines used in sealing too [17:39] unless it crashes before you unlock the encrypted device [17:40] in this case you could boot with any cmdline === ijohnson|lunch is now known as ijohnson [17:40] but you won't be able to go past beyond the writable fs mount [17:41] cmatsuoka: cachio: what I was going to propose is a test PR that changes the cmdline everywhere to include taht systemd.log_level kernel cmdline just to see the boot log [17:41] I presume it happens every time so opening a PR with "run nested" label should be enough to trigger it and get some more representative logs [17:42] ijohnson, ok [17:42] the other issue is that if it is crashing instead of rebooting, we may lose some of the logs as they don't get flushed to GCE in time [17:42] ijohnson, cachio: having an instrumented PR is probably the way to go [17:42] cachio: I will send a PR then [17:42] nice [17:42] thanks [17:42] cmatsuoka: do you know if it's sufficient to change the kernel cmdline inside snapd now with mborzecki's changes, or do I still need to make changes to the gadget grub.cfg ? [17:43] cmatsuoka: I don't know how far along mborzecki's changes are [17:43] ijohnson: uuuhm, I don't know if mborzecki's changes are already effective [17:43] if not, the hard-coded cmdline is in gadget/install/install.go [17:44] I think they might be effective already, I tried something similar late last week and found that snapd was writing the grub.cfg [17:45] at some point last week I had to change the hardcoded cmdline, but I don't know if I had master merged in that PR [17:45] * cmatsuoka checks [17:49] ah, I think you're right, in my test I edited the bootloader line by hand [17:49] (it was a local test, not a test PR) [17:51] I have a local branch where I changed the hardcoded lines with calls to maciek's command line building, hence the confusion [17:53] PR snapd#9064 opened: .github/workflows: move snap building to test.yaml as separate cached job [17:55] zyga: how's it going? [17:55] mborzecki: let me chcek [17:55]