[04:38] PR snapd#9299 closed: tests/core/uc20-recovery: fix check for at least specific calls to mock-shutdown [05:34] morning [05:56] good morning mb [05:56] mborzecki: good morning [06:06] mvo: hey [06:06] mvo: i see the PRs have landed mostly, and claudio updated #9300 [06:06] PR #9300: boot: build bootchains data for sealing <⛔ Blocked> [06:07] mborzecki: yeah [06:08] mborzecki: that's great, so just one more to go, right? [06:10] hopefully ;) [06:11] haha [06:11] yes [06:24] PR snapd#9310 opened: tests: remove "set -e" from function only shell libs [06:56] good morning [06:56] starting early although I'm still pretty sleepy [06:56] today is a strong coffee day [06:57] good morning zyga-kaveri [06:57] and good morning pstolowski [06:57] pstolowski: hey :) [06:57] hey, good morning! [06:57] pstolowski: are you by any chance using vscode? [06:58] zyga-kaveri: yes [06:58] pstolowski: I would like to try a code sharing session sometime today [06:58] zyga-kaveri: sure [06:58] I have a question and I wanted to check how this works compared to screen sharing [06:58] but please let me wake up first [07:02] pstolowski: try https://prod.liveshare.vsengsaas.visualstudio.com/join?D184D6AF59242DB6AAB93D1D2585F4E2A08E [07:04] zyga-kaveri: pstolowski: hey [07:05] hey :) [07:05] zyga-kaveri: hm interesting, i joined with a browser(?!) [07:05] mborzecki: cool [07:05] mborzecki: it's pretty neat [07:05] did you have to log in? [07:05] I've installed the audio sharing extension as well [07:05] zyga-kaveri: feels like it's pulling vs into firefox atm [07:06] so I could talk to you if you did the same [07:06] zyga-kaveri: ok, i think i joined after a few hops [07:06] kinda flaky in firefox, let me try chromium instead [07:08] mborzecki: try in vscode [07:08] one sec guys [07:08] zyga-kaveri: i think you session died [07:08] *your [07:08] yeah one sec :) [07:09] yeah. 404 [07:09] I had to reload to install the extension [07:09] mborzecki: I sent you an invite by mail now [07:10] https://prod.liveshare.vsengsaas.visualstudio.com/join?8A0945F1FF41E82128AF4EB5930C2A992FB4 [07:10] or try this [07:11] hahah working [07:12] can you see the session chat? [07:19] PR snapd#9311 opened: nested: add support to telnet to serial port in nested VM [07:22] woot [07:22] that was quick and productive, thank you both [07:22] zyga-kaveri: yw [07:23] how was the quality on your end? [07:23] was it better than video share? [07:23] zyga-kaveri: you closed the session right? cause it said i left it [07:23] zyga-kaveri: yeah i think it's much better, no artifacts [07:23] zyga-kaveri: goot that at least it works in the browser [07:23] zyga-kaveri: plus the client controls the look/fonts etc [07:23] I'm not sure, I sometimes hit a wrong key on my all-blank-keycaps keyboard [07:24] could you see my three-tab view [07:24] zyga-kaveri: i think it makes sense for longer sessions [07:24] I wonder how it adjusts from one screen size to another [07:25] pstolowski: it also supports read/write editing [07:25] zyga-kaveri: yes i saw this [07:25] also terminal can be interactive [07:26] zyga-kaveri: i don't think i saw your three-tab view [07:26] but i'm unclear what you mean by that [07:26] pstolowski: ah that's good, I have a wide monitor and I was worried that would not translate well to other aspect ratios [07:26] pstolowski: three columns of text [07:26] + terminal at the bottom [07:27] mvo: do you need the run nested label? [07:27] zyga-kaveri: see the screenshot on mattermost [07:27] zyga-kaveri: yeah, probably. let me do that [07:29] pstolowski: I see it [07:29] so the editor layout is custom per participant [07:30] ok [07:30] now that I understand the problem all I need to do is to code the fix [07:30] thank you again :) [07:31] great [07:32] i don't remember the exact reason it's done that way there. maybe we could have fixed auto-connect and make it smarter. but perhaps there was a good reason [07:32] i touched this line 2 years ago, amazing... [07:32] it seems like an upgrade path for old systems [07:33] that upgrade to new snapd and new snapd needs to run their tasks maybe? [07:34] zyga-kaveri: ah yes, there is a comment at the top [07:47] mvo: did you see? we found the (one) issue with nested [07:49] pedronis: hi, shall we have a quick chat about 9300? [07:49] mborzecki: yes [07:50] pedronis: ok, in 5/10 maybe? [07:50] ok [07:51] pedronis: yes, I'm wrapping my tweaks up now right now [07:52] pedronis: good that it's not something deeper and that nested is back to function it seems [07:52] mvo: yes, but annoying, one of those situation where splitting a PR created its own issues [07:52] pedronis: yeah, it's a combinations of issues it seems, the fact that the nested tests only run sometimes is also contributing to the issue [07:53] mvo: yes, but otoh our tests are flaky and slow already :/ [07:53] even before nested [07:53] mvo: we could add an action that runs nested every 12 hours [07:53] pedronis: yeah [07:53] it gets reported in github pretty nicely [07:53] (on master) [07:54] mvo: I mean I had a very frustrating evening, staying around to force merge things [07:54] zyga-kaveri: as long as we make sure that each nested failure is considered high priority I think that's a good one [07:54] pedronis: .( [07:54] pedronis: what tests in particular ? [07:54] pedronis: or all over the place? [07:54] mvo: I'll cook something today [07:54] a bit all over the place [07:54] pedronis: the nested suite *should* be more robust, we only tests very robust stuff there [07:54] one issue was uc20-recovery, but we have also random failures in ohter places [07:55] pedronis: well, "should" [07:55] pedronis: uc20-recover should now be fixed after merging ian PR [07:55] like arch seems to fail on cgroup-tracking:root often but not always [07:55] pedronis: but yeah, I feel your pain :( [07:55] pedronis: I saw this one too! [07:55] and then various interface-* tests failing sometimes [07:56] and snap run timeouts sometimes but not always on strace [07:56] I saw it on 18 [07:56] saw all over the place [07:56] but in places that are required [07:57] :( [07:58] pedronis: joining standup [07:58] pedronis: sry! I will watch out for failures and see if I can fix them [08:19] mborzecki: maybe I confused you: https://github.com/snapcore/snapd/pull/9300#discussion_r486149013 [08:19] PR #9300: boot: build bootchains data for sealing <⛔ Blocked> [08:20] mborzecki: the Kernel BootFile should have full paths, it's the Kernel fields that should be ok with just names [08:21] (it's not super deep, it's mostly for consistency) [08:31] * zyga-x240 caved and ran from the office upstairs [08:31] it's so much warmer in the sun [08:31] I'll go there later [08:32] PT is at home today so I can get to the bottom of remaining tests :) [08:44] PR snapd#9312 opened: o/snapstate: disk space check with snap update [08:57] mvo: I reviewed #9021, it needs a new 2nd review [08:57] PR #9021: snap: implement new `snap reboot` command [08:59] pedronis: \o/ thank you [08:59] pstolowski: are you up for a second review of 9021? [09:00] pedronis: I assume 9300 is not quite ready for a review yet(?) [09:00] mvo: sure. btw is #9284 now critical to unbreak something in nested? [09:00] PR #9284: tests: some fixes and improvements for nested execution <⚠ Critical> [09:00] mvo: mborzecki is working on it [09:00] pstolowski: I see failures with nested on master right now [09:01] pstolowski: and with 9284 it seems no failures anymore [09:02] I will also make the nested runs required, so that if they are activated we can only merge if they are green [09:02] +1 [09:02] ok, i'll conclude my review of 9284 then [09:05] PR snapd#9313 opened: boot: do not reorder boot assets when generating predictable boot chains [09:05] pedronis: mvo: https://github.com/snapcore/snapd/pull/9313 a prerequisite [09:05] PR #9313: boot: do not reorder boot assets when generating predictable boot chains [09:05] skipped spread, so that we can land it quickly [09:06] mborzecki: thanks, checking [09:09] I switched to a hybrid [09:09] with tasks for the things where undo is easier as tasks (export, unexport) [09:09] mborzecki: I am bit confused by tests there [09:09] and a pointer helper for the link [09:09] to avoid the complexity of link snap relationship [09:09] making the changes now but I think this is best of both worlds [09:12] mborzecki: I think the tests needs a cleanup, now that the assets are not ordered in themselves we don't need so many weird cases in them [09:17] if I'm making sense [09:45] pedronis: i've updated #9300 [09:45] PR #9300: boot: build bootchains data for sealing <⛔ Blocked> [10:20] mborzecki: did you forget to push to 9313 ? [10:23] pedronis: #9313 is updated now too [10:23] PR #9313: boot: do not reorder boot assets when generating predictable boot chains [10:42] mborzecki, pedronis fwiw, I just merged 9284 and nested runs are green, I also made green nested runs a requirement for merging PRs (anything without the run-nested label gets an automatic green here) [10:44] mborzecki: thanks for the update on 9313 - fwiw, looks all right to me [10:45] PR snapd#9284 closed: tests: some fixes and improvements for nested execution <⚠ Critical> [11:00] PR snapd#9314 opened: o/snapstate: disk space check on UpdateMany [11:01] Issue core20#32 closed: The machine-id file should be writable [11:01] Issue core20#72 closed: move docker user/group to extrausers [11:01] PR core20#82 closed: [RFC] hooks: mv docker user/group definition to extrausers [11:01] PR core20#83 closed: static/writable-paths: make /etc/default/crda writable [11:01] pstolowski: do we need a check on remove? [11:01] currently remove can fail if we have no space for the snapshot [11:02] zyga-x240: that's done and landed already [11:02] zyga: unless you found a bug [11:03] it's behind feature flag too [11:03] mborzecki: thanks for the changes in 9300, there are probably some tweaks that can make it easier to reuse the bits for reseal [11:05] zyga-x240: correction, single snap remove does that already in master; remove-many is about to land [11:05] pstolowski: ah, no I just realized this should be so while reading the other PR [11:05] pstolowski: that's great [11:05] ok [11:05] zyga-x240: yeah i think all major ops are covered now [11:27] mvo: mborzecki: I pushed some small tweaks/simplifcations to 9313 [11:28] pedronis: thanks! [11:29] I will look at 9300 again in a bit [11:29] pedronis: looking [11:34] pedronis: do you know whether #9032 is getting replaced by something else? [11:34] PR #9032: secboot: add call to reseal an existing key [11:35] it's not afaik, it's used by #9287 which is the one to tweaks after we are happy with 9300 [11:35] PR #9287: secboot: reseal key to parameters specified in modeenv <⛔ Blocked> [11:40] hmm wonder whether claudio has some unpushed merge with master in 9287 [11:47] * pstolowski lunch [11:49] mborzecki: anyway, I reviewed 9300, there are things to improve there [11:50] let me see [12:03] pedronis: tweaked buildRecoveryBootChain to actually takea list of systems, which in the initial install case will be just the current recovery system, or in a general case, the recovery systems we have [12:14] * zyga-x240 -> lunch [12:21] pedronis: ok, another push to #9300 [12:21] PR #9300: boot: build bootchains data for sealing <⛔ Blocked> [12:22] cmatsuoka: hi, we've hijacked a bit your PRs :) please take a look at 9300 as well, hope i've not messed up something [12:22] mborzecki: thanks, I'm glad you're working on it [12:24] * zyga-x240 also errand [12:24] mvo: my update in case I cannot connect for standup [12:25] mvo: synced with pawel and maciek in the morning to debug issue from yesterday, guys solved it instantly! [12:25] mborzecki: thanks [12:25] mvo: split tasks so that export/unexport just handles files and not the "current" symlink [12:25] mvo: used a hack to update current inside doLinkSnap, avoiding complex test failures related to task changes after link-snap [12:26] mvo: tweaked description of the exported content so that undo path has no custom code left, just loads data from task state and calls the regular remove method [12:27] mvo: I will tweak some tests and open a new PR today [12:27] now everything should pass okay [12:27] * zyga-x240 runs to do the errand [12:28] pedronis: cmatsuoka: should we sync before the standup? [12:32] mborzecki: I can, SU room? [12:39] cmatsuoka: mborzecki: I'm there [12:40] cmatsuoka: we're there [12:55] PR snapd#9313 closed: boot: do not reorder boot assets when generating predictable boot chains and other small tweaks [13:01] PR snapd#9315 opened: tests: fix snap-routime-portal-info test [13:35] ijohnson, #9208 has conflicts [13:35] PR #9208: tests/nested/core20/kernel-failover: add test for failed refresh of uc20 kernel [13:38] cachio: ack I see, I will resolve those now [13:47] ijohnson, nice, thanks [14:06] re [14:16] PR snapd#9021 closed: snap: implement new `snap reboot` command [14:16] PR snapd#9316 opened: tests: use full path to test-snapd-refresh.version binary [14:21] PR snapd#9317 opened: [RFC] devicestate: keep log from install-mode on installed system [14:29] mborzecki: cmatsuoka: 9300 seems to have passed all relevant tests, still needs the 2nd review [14:34] yay [14:35] PR snapcraft#3278 closed: cli: client side check for setting default tracks [14:36] pedronis: cmatsuoka: since it's green, and if 2nd review comments on cosmetics/messages, i think we should land it and do a simple followup with the tweka [14:40] s/tweka/tweaks/ ofc === popey8 is now known as popey [15:00] pedronis, mborzecki, cmatsuoka I have one more meeting, then I should be able to have a look [15:00] (unless someone else beats me to it :) [15:06] mvo: thanks, it would be great if you could have a look [15:11] * cmatsuoka will return after lunch [15:14] one question on 9300 [15:14] (left in the PR that is) [15:16] ijohnson: I answered [15:16] * cachio lunch [15:37] pedronis: I +1d 9300, not sure if you still want someone else like mvo to +1 it, I have not kept up to date on all the resealing pr's this week [15:40] at this point I think we should land it and cmatsuoka can work on the small follow ups, and the next ones. OTOH if mvo has time to post-review it that's also welcome [15:41] sure, let's land it [15:46] PR snapd#9312 closed: o/snapstate: disk space check with snap update [15:46] PR snapd#9314 closed: o/snapstate: disk space check on UpdateMany [15:51] PR snapd#9300 closed: boot: build bootchains data for sealing [15:51] PR snapd#9310 closed: tests: remove "set -e" from function only shell libs [16:01] PR snapd#9294 closed: o/snapstate: check available disk space in RemoveMany [16:06] PR snapd#9318 opened: .github/workflows/test.yaml: also run 20.04 nested tests with UC20 label [16:07] pedronis: I get dinner now, please let me know if I can do anything after dinner, I will read backlog [16:16] mvo: I think we need more code from cmatsuoka before anything else [16:18] pedronis: ok, thanks, I will just read backlog then :) [16:24] cmatsuoka: I don't know if you noticed yet, but 9032 itself needs update because of the change to the params [16:25] to support trees [16:25] pedronis: I'll check [16:25] it's fascinating, it doesn't conflict, but then it gets unhappy about types [16:26] pedronis: ah yes, that's true, I'll open a PR with the small fixes for 9300 and fix 9032 [16:26] * mvo hugs cmatsuoka and pedronis [16:30] I think pedronis deserves a day off, at least, for recent crunch [16:30] he seems to work in all the timezones at once [16:31] agreed [16:31] zyga-mbp: and all the complex subsystems [16:31] PR snapd#9319 opened: seal: error message and function name adjusts [16:32] that was a given [16:33] today is a bit more patchy, with errands and now remote school meeting [16:33] gosh if we ever complain about hangouts or anything like that [16:33] try putting 30 parents with poor tech know-how [16:33] and have them achieve anything [16:33] been there [16:34] #9208 is ready for a review, I de-conflicted it now [16:34] PR #9208: tests/nested/core20/kernel-failover: add test for failed refresh of uc20 kernel [16:34] cachio: ^ [16:34] ijohnson, tx [16:41] #9319 should have had skip spread [16:41] PR #9319: boot: in seal.go adjust error message and function names [16:50] ijohnson: ^ should be quick to review, and has the comment you asked now [16:50] * pedronis dinner [16:50] ok [17:26] now GH is sending (resending?) old emails ?? [17:34] hmm I didn't receive anything strange from GH recently [17:35] neither did I [17:37] PR snapd#9320 opened: boot: group SealKeyModelParams by model [17:40] afk for a short while, /me completely forgot to pay some bills yesterday [17:42] mvo, I see that is really weird [17:42] https://paste.ubuntu.com/p/3YQrFW6F22/ [17:43] mvo, starting stopping many times until it reaches the limit [18:02] PR snapd#9319 closed: boot: in seal.go adjust error message and function names [18:08] cmatsuoka: I reviewed 9032, my remark can probably be addressed in a follow up [18:09] pedronis: thanks, I'll check [18:22] cmatsuoka: are you assuming we land #9320 (I didn't think it was urgent) ? [18:22] PR #9320: boot: group SealKeyModelParams by model [18:22] PR snapd#9321 opened: cmd/snap/model: specify grade in the model command output [18:22] pedronis: not urgent, I prepared it because it looked easy enough [18:23] pedronis: now working on the new reseal call [19:19] PR snapd#9032 closed: secboot: add call to reseal an existing key [19:38] * zyga-mbp works on a few more tests [19:44] ijohnson, hey [19:44] Hi [19:45] I am checking basic20 test which fails [19:45] and see this [19:45] MATCH '# Snapd-Boot-Config-Edition: [0-9]+' < /run/mnt/ubuntu-seed/EFI/ubuntu/grub.cfg [19:45] the check fails [19:46] ijohnson, but, it metches [19:46] MATCH '# Snapd-Boot-Config-Edition: [0-9]+' < /boot/grub/grub.cfg [19:48] cachio those are two different files [19:48] One is /boot/grub/grub.cfg and the other is /run/mnt/ubuntu-seed/EFI/ubuntu/grub.cfg [19:49] cachio which test is this [19:50] core/basic20 [19:50] the check for /run/mnt/ubuntu-seed/EFI/ubuntu/grub.cfg is failing [19:50] ijohnson, https://paste.ubuntu.com/p/25wQdj2qFW/ [19:51] cachio so this test will only work if the image that was created was flashed with a new enough snapd used by ubuntu-image [19:52] cachio how was that image built? [19:52] ijohnson, I build that today, using beta [19:52] channel [19:53] ijohnson, should I update snapd? [19:53] cachio you need to point ubuntu-image to snapd from your system with an env var [19:54] cachio set UBUNTU_IMAGE_SNAP_CMD=$(which snap) [19:54] cmatsuoka: how it's going? any problems? [19:54] Then call ubuntu-image [19:55] ijohnson, ahh, nice, I'll retry with that [19:55] pedronis: I did reseal, now I'm running a quick test before opening the PR [19:55] ijohnson, thanks [19:56] cachio just to be on the safe side try with snapd from edge or beta on your host too [19:57] and... it failed! for some reason [19:57] * cmatsuoka investigates [20:00] far fewer failures [20:00] but some bugs still remain, [20:00] I think snapstate.Info description is misleading [20:00] it doesn't work for snaps that are not in state [20:02] I think i just can't do what I'm trying to do (resealing just after sealing) [20:03] cmatsuoka: do you think the TPM does not support that? [20:03] zyga-mbp: I think we lock the tpm after we seal because reseal can't even open it [20:04] PR snapd#9322 opened: boot: add call to reseal an existing key [20:05] also closing the previous PR that did the same thing [20:08] brb, need to restart wifi [20:09] PR snapd#9287 closed: secboot: reseal key to parameters specified in modeenv <⛔ Blocked> [20:11] ijohnson, I also see this change https://paste.ubuntu.com/p/Vft4S47VP4/ [20:11] which fails [20:11] it should be fixed whith the new snapd? [20:12] cachio: no that test needs to be updated for external systems [20:12] or I need to update the test [20:12] cachio: that test depends on the image used [20:12] ijohnson, ok [20:12] yes, makes sense [20:12] thanks [20:14] 2K insertions [20:14] I need about 500 more to finish [20:14] PR snapd#9248 closed: exportstate: add scaffolding for the export manager [20:14] PR snapd#9323 opened: snap/naming: upgrade TODO to TODO:UC20 [20:16] ijohnson, do you know where I can get the info to check snap recovery? [20:16] cachio: the info comes from the model assertion and the recovery system label [20:19] ijohnson, and do we store that in a file? [20:19] cachio: the 2nd column in the output of `snap recovery` will match the output of the brand from `snap model` [20:19] cachio: the 3rd column will match the output of `snap model | grep model` [20:19] ijohnson, nice [20:20] I'll get the info from there [20:20] thanks [20:26] zyga-kaveri: ha, the tpm device was left open after sealing [20:28] and now it works [20:46] time for spread [20:46] I have one unit test failure but debugging some of the state engine failures is a bit mysterious [20:46] but that's okay, let's see what spread shows [20:47] * cachio afk [20:54] PR core18#169 opened: snapcraft.yaml: use build-base for modern snapcraft, and use lxd in travis to build the snap [21:04] PR snapd#9323 closed: snap/naming: upgrade TODO to TODO:UC20 [21:04] PR snapd#9324 opened: secboot: adjust parameters to buildPCRProtectionProfile [21:05] thanks! [21:05] np, thanks for fixing it :-) [21:13] ah, [21:13] I know what to do next [21:24] PR snapd#9325 opened: strutil: add SortedListsUniqueMerge [21:30] ok, no unit test failures, [21:30] a few XXXes [21:30] running spread again [21:30] should be all green now [21:34] ijohnson: cmatsuoka: I worked a bit more on #9320, it has more tests for some of things in the previous PRs, it needs #9325 [21:34] PR #9320: boot: group SealKeyModelParams by model, improve testing [21:34] PR #9325: strutil: add SortedListsUniqueMerge [21:34] pedronis: ack, checking [21:35] pedronis what does sh stand for? [21:35] size? [21:35] where ? [21:35] line 79 [21:35] https://github.com/snapcore/snapd/pull/9325/files#diff-342c12787e3c9ff0147933bb2fe7aa7fR79 [21:35] PR #9325: strutil: add SortedListsUniqueMerge [21:36] sz [21:36] yes, size [21:37] * zyga reads sz as sh due to the pronounciation in his language which sometimes makes things confusing [21:48] pedronis why the else at the end [21:48] cannot we drain both lists? [21:48] oh [21:48] only one would be [21:48] right? [21:51] +1 [21:53] I'll wait for tests to finish [21:53] and run one more overnight [21:53] I'm mainly interested in seeing core tests [21:53] tomorrow the last stretch to complete this [21:54] zyga: yes, if they are both true we would still be in the previous loop [21:54] right [21:54] nice code :) [22:02] afk, phone call [22:02] ijohnson: I commented on your note in the standup, the difference between snap recovery and snap model is expected [22:03] we have the same difference in other places (is the difference between list vs info-like output) [22:04] cmatsuoka: I missed you had opened #9322 [22:04] PR #9322: boot: add call to reseal an existing key [22:05] PR snapd#9326 opened: tests: fix for basic20 test running on external backend and rpi [22:05] pedronis: and now I noticed it failed the macos test, probably something related to nosecboot [22:07] cmatsuoka: looks good looking quickly, but one comment, I will look closer in the morning [22:07] pedronis: ok, thanks, I'll see the macos and the other issue you raised in a moment, I need to make a few phone calls first [22:08] thx, calling it a day [22:10] good night pedronis === secretfader49 is now known as fader