[00:39] PR snapd#9257 opened: bootloader: retrieve boot chains from bootloader [05:03] good morning [05:25] morning [05:25] starting a bit later, need to drive kids to school [05:28] mborzecki: ok [05:41] brb [06:08] brb again, time to fetch charger [06:14] I have the 12v power ;) [06:15] * zyga is struggling a little with the export provider algorithm, it probably indicates I need a different data model [06:22] good morning zyga [06:22] hello :) [06:22] mvo: first day of morning shift, I'm trying to re-think the data model for exports [06:22] mvo: good luck with your meetings [06:23] mvo: ian's feedback last night was very useful, may simplify zfs support [06:24] zyga - oh, nice! in what PR was that feedback provided? [06:24] zyga or was it here on irc? [06:24] mvo: the one about exchange [06:25] mvo: ian's comments made me think that all I may need, after all, is the plain old rename [06:26] zyga oh, ok [06:27] zyga: getting away with this would be great, less-code++ [06:34] re [06:34] mvo: zyga: morning guys [06:35] good morning mborzecki [06:35] wow, the weather turned to shit pretty quickly this year [06:35] mborzecki: how was first day of school? [06:35] mborzecki: it's 2020 [06:35] zyga: 13C, raining all of yesterday and during the night [06:35] zyga: ofc, also a short power outage yday, and my primary internet link is down ;P [06:36] mborzecki: nothing like 1st of September to remind everyone that holidays are over [06:36] * zyga cannot get used to the fact that some places start school mid-August [06:36] yeah, bringing everyone up to speed with the harsh reality ;) [06:37] mborzecki: I have a question, a comment from ian relating to RENAME_EXCHANGE indicated you did some research about atomic properties of that operation [06:37] hm? [06:37] mborzecki: do you recall any problems with that? [06:37] it was in relation to AtomicSylink helpers [06:38] zyga: let me take a look at the notes [06:39] mborzecki: it's not a biggie if you don't find anything [06:39] just reading that comment made me wonder if you found something about rename+exchange that is still racy [06:42] zyga: hm not much in my notes, aside from #8045 and 8044/8043 [06:42] PR #8045: osutil: add helpers for creating symlinks and renaming in an atomic manner [06:44] * zyga was tired, re-reading Ian's comment now makes a lot more sense [06:44] mborzecki: thanks! [07:02] morning [07:03] good morning pawel [07:03] pstolowski: hey [07:05] good morning ps [07:05] good morning pstolowski [07:05] o/ [07:18] pedronis: hi, thanks for the cleanup in #9249 [07:18] PR #9249: boot,bootloader,gadget: apply new bootloader.Options.Role [07:22] mborzecki: thx [07:22] Lucy just woke up [07:22] afk [07:36] PR snapd#9255 closed: o/snapstate, features: add feature flag for disk space check on remove (2.46) [09:03] * zyga-kaveri starts the day [09:14] mvo: there's some progress on building understanding of the apparmor notification protocol [09:14] mvo: I will try to spend Fridays on that, there's a good chunk of code I can now write [09:14] mvo: so the meetings work :) [09:14] zyga-kaveri: cool [09:15] mvo: in a way the kernel is really full of murky #defines and bitfield and modes and masks and stuff [09:15] writing a good, high-level abstraction for this is only possible with hands-on access to insight not present in the .h files [09:18] * mvo nods [09:24] pedronis: i've updated #9246 [09:24] PR #9246: boot: handle canceled update [09:24] mborzecki: thx, I put it back on my queue [09:24] pedronis: thanks! [09:35] quick errand, back in 30 [09:41] PR snapd#9258 opened: devicestate: add tests around logging in RequestSystemAction [10:11] re === mborzeck1 is now known as mborzecki [10:31] PR snapd#9259 opened: client, api: handle insufficient space error [10:32] hah, nice touch from github: typing # in the text fields not only auto-completes PR numbers, it actually understands strings and finds PRs by titles. i wish I knew that earlier ;) [10:36] pstolowski: is that a new thing? [10:37] mborzecki: i have not idea, just discovered by accident [10:37] anyways nice, i would often launch a new tab, look for the PR and then use the number [10:37] i always assumed numbers [10:37] yeah, that's what i was always doing [10:40] pedronis: given that 9250 is green and there is a followup planned anyway, do you mind if I merge it (even without the doc updates?) [10:41] * pstolowski lunch [10:43] pstolowski: neat :) [10:44] pstolowski: maybe it does more, #fixthisbugplease [10:49] mvo: it's fine, let's just not forget [11:01] PR snapd#9250 closed: many: use BootFile type in load sequences [11:05] mvo: we should try to land https://github.com/snapcore/snapd/pull/8920 no? it might need a master merge, and a re-run [11:05] PR #8920: interfaces: update cups-control and add cups for providing snaps

[11:06] PR snapd#9260 opened: seed/seedwriter: test local asserted snaps with UC20 grade signed [11:06] PR snapd#9261 opened: bootloader: tweak doc comments (thanks Samuele) [11:10] * zyga-kaveri afk for 15 [11:24] pstolowski: what's the status of #8960? should I stil wait to review it? do we need to discuss about it? [11:24] PR #8960: o/snapstate,servicestate: use service-control task for service actions (9/9) [11:29] pedronis: i've a tentative fix and need to update unit tests for it, not pushed yet [11:30] pedronis: but it's not the workaround we discussed earlier this week [11:31] re [11:54] mvo: I re-reviewed, tbh not a fan of the new logging code refactor [11:54] #9210 [11:54] PR #9210: daemon: add /v2/systems "reboot" action API [12:00] pedronis: that's fine, happy to redo it, the important part is that we have tests now so it's easier [12:08] mvo: maybe it's quicker if I push a re-refactor? [12:11] pedronis: please do, in a meeting [12:25] * zyga-kaveri takes a lunch break and looks at apparmor sources [12:53] mvo: pushed [12:53] pedronis: thank you! [12:54] hopefully is not worse [12:55] https://github.com/snapcore/snapd/pull/9210/files#diff-0b649265c21137e8fd367d4a16607a82R1058 [12:55] PR #9210: daemon: add /v2/systems "reboot" action API [13:47] PR snapd#9262 opened: configcore: rework how console-conf is disabled [14:52] PR snapd#9263 opened: interfaces/fpga: add fpga interface [15:35] * cachio lunch [15:45] cachio: let me know when you're back [15:47] PR snapd#9254 closed: sysconfig/cloudinit.go: add DisableNoCloud to CloudInitRestrictOptions [15:53] PR snapd#9264 opened: [RFC] many: introduce ContentChange for tracking gadget content in observers [16:00] cmatsuoka: I did a pass on #9257 [16:00] PR #9257: bootloader: retrieve boot chains from bootloader [16:01] ijohnson, hey [16:01] I am here [16:02] pedronis: thanks, I was in the process of making the mock trusted bootloader functions consistent with the mocked trusted assets [16:03] ijohnson, I see I can't boot anymore uc20 on nested [16:04] cachio: hey I need to write a manual nested with my own model assertion when building the image, do you think I should modify the function to get the model assertion to use a global var that I define in the test? Does that make sense? [16:04] cachio: re uc20 not booting in nested, what's wrong? can you provide logs? is this on your PR or on master ? [16:04] ijohnson, about the mode, yes [16:05] a var for the model sounds good [16:05] cachio: ack I will add that then [16:06] ijohnson, https://paste.ubuntu.com/p/rstDJgrkqq/ [16:06] ijohnson, could be related to reseal? [16:07] cachio: mmm I don't think enough of that has landed to be related to resealing [16:07] cachio: was this from master or from your branch ? [16:08] it is my branch [16:08] I merged with master [16:08] not sure if it is the root cause [16:13] cachio: I'll have a look in a bit [16:15] I thnik it is something in edge channel because the iamges created from beta work well [16:18] cachio: I've got google-nested:ubuntu-20.04-64:tests/nested/core20 running from master now, let's see what happens [16:19] ijohnson, nice, manual also fails for some tests [16:19] cachio: which tests ? is it reproducible ? [16:19] minimal-smoke [16:20] cachio: yes I've seen that one fail intermittently on PR's [16:20] ijohnson, I am still waiting for more results [16:20] k [16:20] but fails to prepare, basically can't login [16:21] in the logs I don't see any reboot [16:23] ijohnson, the weird part is that I ran with kvm and there are not any reboot [16:33] cmatsuoka: have you ever seen this failure on gce before? [16:33] 2020-08-26T17:52:35.2828113Z Aug 26 17:46:06 ubuntu snapd[1116]: secboot_tpm.go:416: TPM provisioning error: cannot access resource at handle TPM_RH_LOCKOUT because an authorization check failed [16:33] 2020-08-26T17:52:35.2828332Z Aug 26 17:46:06 ubuntu snapd[1116]: taskrunner.go:271: [change 2 "Setup system for run mode" task] failed: cannot create partitions: cannot seal the encryption key: cannot provision TPM: cannot access resource at handle TPM_RH_LOCKOUT because an authorization check failed [16:33] seems like the TPM wasn't properly reset before it was used to boot this image? [16:34] cachio: yeah very odd, my run is almost done preparing the image, just waiting for it to either finish successfully or get stuck [16:34] ijohnson: yes, it happened to cachio a couple of times. according to chris this shouldn't happen, unless the TPM was not properly cleared [16:34] (or the simulator is faulty) [16:34] mmm yeah this is with the swtpm-mvo snap [16:36] ijohnson: do you see any possibility of the TPM not being cleared before this test? [16:43] ijohnson: I tried to answer your considerations in #9253 [16:43] PR #9253: sysconfig/cloudinit.go: add AllowCloudInit and use GadgetDir for cloud.conf [16:43] mmm seems my internet is out :-/ [16:44] cmatsuoka: actually yes I see how it could happen, we don't ever reset the TPM from swtpm-mvo, if it's already installed, we should reset it if it's already installed when we boot a VM [16:44] I can send a pr when my internet is back [16:45] ijohnson: ah, mystery solved then. I thought the nv data file was deleted before each test [16:45] pedronis ack thanks for clarifying, I will leave the function as-is, copying gadget config if it is found [16:46] Err rather I'll refactor to first check if AllowCloudInit, then if it's allowed then copy the gadget config [16:49] yay internet is back [16:51] ijohnson: I think the points are mostly order, rename and comments [16:52] ijohnson, cmatsuoka at some point we were reseting this [16:52] I added that when we originally had that problem and claudio told me to add that [16:52] Well apparently my internet is not back [16:52] don't know if it is gone with a refactor [16:52] * ijohnson is back to the phone [16:53] pedronis ack yes I will push up an update later today [16:53] cachio I don't see it anywhere, where were we doing this? [16:53] cachio all I see on master is that `if ! snap list swtpm-mvo ...` [16:54] ijohnson, it is not there anymore [16:54] yeah [16:54] I think during a refactor we remove that [16:54] I'll take a look to the logs [16:54] I can add that to my PR as well [16:55] so perhaps that is affecting also [16:55] cachio: please don't add it to your pr [16:55] cachio: I really want to land that pr without further iteration [16:55] cachio: that pr is really big, I am almost done with another review for you [16:55] ijohnson, sure [16:55] ijohnson, ah, nice, thanks [16:56] it started small [16:56] :) [16:56] yes it started small, but is no longer small :-) [16:59] cachio: well re the failing uc20 nested tests, I lost my internet connection so I don't know if the test failed or not, I will try again now hopefully my internet doesn't cut out again [17:03] ijohnson, np [17:04] the test in this pr could be failing because the tpm or the kernel repack thing [17:04] which are already being fixed in other prs [17:05] ijohnson, here we have a lot of problems with the internet too, all the children having zoom meetings for school [17:05] pedronis: is it expected that `snap ack` on a system-user assertion doesn't "just work" and that the system-user assertion needs to be imported by a usb drive for example? [17:05] and so many people working from home in this area as well [17:05] cachio: yeah that's what I expect for our neighborhood too, it's all virtual learning here too pretty much [17:06] pedronis: `snap ack system-user.assert` returns 0, but no user is created [17:07] pedronis: ah I see from the docs on system-user assertions: [17:07] > The simple addition of such assertions to a device assertion database should not be enough to trigger the user creation [17:07] so nvm me, it is not enough to just `snap ack` [17:14] pedronis: addressed the issues but not sure if the grub NoSlashBoot fix is what you had in mind [17:18] ijohnson: that's by design, auto import from usb does a step more, but ack alone is not alone [17:18] *is not enough [17:19] Yes I see that now [17:22] cmatsuoka: it's fine, but probably better not to use TrustedAssets in boot chain now we the vars [17:22] pedronis: thanks, will update accordingly [17:31] * cmatsuoka breaks for lunch [17:40] ijohnson: could you look at #9246 ? [17:40] PR #9246: boot: handle canceled update [17:40] pedronis: sure I can look this afternoon [17:53] cachio: both of the tests/nested/core20 from master failed for me [17:53] cachio: I think what happened is that cloud-init took too long to run and got disabled by snapd [17:53] cachio: but I need to do another run with persistent logging to know for sure what happened [17:54] it's rather annoying that we don't have persistent logging enabled by default :-/ [18:03] ijohnson, thanks for checking that [18:13] PR snapcraft#3274 closed: schema: rename package-repository's "deb-types" to "format" [18:13] ijohnson: isn't our cloud-init timeout fairly generous? [18:14] do we need to increase it? [18:14] pedronis: well without logs I don't know what happened, all I know is that cloud-init didn't create the user even though the right files were put onto /etc/cloud from ubuntu-seed, but the disabled file was installed by snapd from the previous boot [18:15] pedronis: we give cloud-init 5 minutes to finish running [18:15] pedronis: but also note that these VM's are excruciatingly slow [18:15] *nested VM's [18:15] so maybe it does actually take more than 5 minutes during a first boot for cloud-init to finish, I dunno [18:16] I have a run in progress with persistent logging enabled via the gadget so hopefully I can see what happened for sure [18:17] ijohnson, I'll be afk about 1 hour, need to go to the kinesiologist now, please let me know if you find anything else [18:17] cachio: sounds good [18:17] I'll go back to this once I am back [18:17] thanks [18:18] PR snapcraft#3259 closed: cli: introduce set-default-track [18:18] PR snapcraft#3275 closed: cli: ignore sudo warning when using multipass [18:18] PR snapcraft#3276 closed: meta: add suite/component validation checks for package-repositories [18:20] * cachio -> kinesiologist [18:33] PR snapd#9260 closed: seed/seedwriter: test local asserted snaps with UC20 grade signed [18:38] PR snapd#9261 closed: bootloader: tweak doc comments (thanks Samuele) [19:02] cachio: I +1'd 9098, I think you should merge it when it goes green [19:02] let's just handle cleanups and followups, the PR is changing a lot and it's difficult to keep tracking [19:53] mmm this is odd [19:53] devicemgr.go:702: System initialized, cloud-init reported to be done, set datasource_list to [ None ] [19:54] seems cloud-init ran, it did in face take more than 5 minutes, but also it seems that cloud-init finished without actually doing anything [19:54] Sep 02 19:25:19 ubuntu systemd[1]: Reached target Cloud-init target. [19:54] Sep 02 19:25:19 ubuntu systemd[1]: Startup finished in 52.655s (kernel) + 5min 16.330s (userspace) = 6min 8.985s. [20:21] cachio: I think I understand why nested uc20 tests are broken [20:21] cachio: it is because of a PR landed to the core20 snap which breaks cloud-init [20:40] cachio: I have debugged it and will be opening a PR shortly to fix the problem [20:45] PR core20#84 opened: static/writable-paths: use transition instead of none for /etc/cloud [20:48] ijohnson, awesome [20:48] thanks [20:50] PR core20#84 closed: static/writable-paths: use transition instead of none for /etc/cloud [21:24] ijohnson, please take a look to the answers I left in the PR #9098 [21:24] PR #9098: tests: new organization for nested tests [21:24] cachio: yes I saw, looks fine to me [21:24] ijohnson, thanks [21:24] cachio: as soon as the pr goes green, please merge I think it's ok without a second +1 [21:24] ijohnson, nice, thank [21:24] s [22:04] PR snapd#9265 opened: many: move seal code from gadget/install to boot [22:06] zyga-kaveri, when you have 5 minutes, could you please take a final view to #9098 [22:06] PR #9098: tests: new organization for nested tests [22:49] PR snapd#9266 opened: tests/lib/nested.sh: reset the TPM when we create the uc20 vm [22:56] ijohnson: is "partion" a real word or can I assume it's mispelled partition? [23:04] cmatsuoka: afaik it's not a word [23:04] ijohnson: from the context I'll assume it's intended to be "partition" [23:07] cmatsuoka: is this from one of my prs? [23:08] ijohnson: I found many instances, just grep -Ri partion. I'm preparing a PR to change them to partition [23:16] ack [23:24] PR snapd#9267 opened: many: fix partion vs partition typo