[00:25] PR snapcraft#3286 opened: [legacy backport] v1 plugins: lock godep's dependencies [01:26] PR snapcraft#3286 closed: [legacy backport] v1 plugins: lock godep's dependencies [04:08] PR snapd#9348 opened: tests: print all the serial logs for the nested test [06:09] morning [06:11] mvo: hey, saw your comment under #9347, i can start looking into this [06:11] PR #9347: tests/lib/nested.sh: use more focused cloud-init config for uc20 [06:11] mborzecki: thanks [06:11] mborzecki: might be as simple as adding something to nested_start_core_vm_unit that already waits for ssh [06:12] mhm [06:12] anyway, looks like we're becoming experts in cloud init too [06:14] mborzecki: yeah :( [06:14] mborzecki: fun, looks like your better logging PR passed except for the smoke test where the change to add logging broke the test because now snap-confine prints stuff to stderr [06:14] hahah [06:16] mvo: oh, and idk if you seen the comment about the nested suite sending 150MB of project data to the vm [06:17] maybe that's the kernel/core/snapd snaps repacked [06:18] mborzecki: yeah, I think [06:19] mborzecki: looks like we need to either put them elsewhere or clean them after the repacking [06:19] mborzecki: but yeah, looks like there is some junk around [06:21] mborzecki: given that all the other tests have passed and that is super rare - what functional change could have triggered this? or do you think it's pure coincidence? [06:21] mborzecki: i.e. could the cloud-init chnage be responsible? [06:21] feels like a stroke of luck [06:22] mborzecki: ok [06:34] mvo: as for SNAP_DEBUG=1 in environment, we could drop an override for snapd.service, i had that at some point, but then dropped it and added to /etc/environemnt (because no logs were appearing which i didn't know at the time to be caused by the MaxLevel* in journald) [06:36] mborzecki: yeah, let me quickly push an idea [06:36] ok, runing the nested suite now [06:37] mborzecki: 9349 use your idea but puts it only for the snapd unit [06:37] mvo: this is what i have for the snap command; https://paste.ubuntu.com/p/s4zr5NsKX4/ [06:38] mborzecki: that looks reasonable, a bit less central than I had hoped, we can't put it into nested_start_for_core_vm ? this waits for ssh already? [06:39] PR snapd#9349 opened: tests: change snapd.spread-tests-run-mode-tweaks.sh to add logging [06:39] mvo: hm, need to double check it wouldn't conflict the nested/manual tests [06:43] mborzecki: yeah, probably fine [06:43] mborzecki: mostly wondering [06:44] mvo: just checked and it should be fine, i've updated the patch [06:44] mborzecki: it's also true that muddeling the concept of start and wait-for-ready is a bit strange so keeping seprarate may well be better [06:44] mborzecki: cool [06:44] mborzecki: again, mostly thinking that this is okay since we wait for ssh already [06:45] maybe i should just push all of it into #9343 [06:45] PR #9343: tests: more logging for UC20 kernel test [06:45] i mean ian's path + the tweak for waiting [06:46] s/path/patch/ [06:48] mborzecki: yes [06:48] mborzecki: I think that's good [06:52] mvo: mborzecki: should we sync? [06:53] pedronis: at 9? i'll grab some tea [06:53] yes [06:54] mvo: I'm quite confused by #9349 given Ian's merge from yesterday [06:54] PR #9349: tests: change snapd.spread-tests-run-mode-tweaks.sh to add logging [06:58] NOMATCH is something that was added to spread recently? [06:58] pedronis: maybe I'm confused, let me double check what happend :( [06:59] pedronis: oh, I see :( [06:59] mvo: afaik on master we don't use the function you are changing anymore [07:00] mvo: I asked you to comment on that yesterday evening when Ian asked to merge and you said yes :) [07:00] pedronis: it is fine [07:00] I'm going to the SU [07:00] pedronis: it's just that it's all a bit of a maze and apparently I did not had enough tea yet [07:00] pedronis: joining [07:01] morning [07:04] PR snapd#9349 closed: tests: change snapd.spread-tests-run-mode-tweaks.sh to add logging [07:07] hello [07:07] good morning pstolowski and zyga-kaveri [07:07] how is core20? [07:07] I'm baby-sitting still [07:12] pstolowski: hi [07:12] pstolowski: could you please look at https://pastebin.ubuntu.com/p/QMJ3x8HK4h/ [07:13] is anything there out of place for preseeding? [07:13] ah [07:13] it's not the full list [07:13] I think I know what is going on [07:16] * zyga-kaveri fixes [07:22] zyga-kaveri: seems 3 tasks are missing (your new tasks I presume) [07:23] pstolowski: yeah exactly :D [07:25] man that run was pretty green overall [07:25] one test adjustment, some random failures [07:25] but otherwise it's pretty good [07:25] I was only running core and main tests for ubuntu 16 [08:15] mvo: pedronis: i've updated #9347 [08:15] PR #9347: tests/lib/nested.sh: use more focused cloud-init config for uc20 [08:25] mborzecki: \o/ [08:32] your video froze mvo [08:49] mborzecki: mvo: so it takes around 50-60 to a resealing in the non-accel vms, almost 80 with two run kernels I suppose [08:49] wow [08:49] mborzecki: we are hasing 2 or 3 kernels each time plus other ops plus signing [08:51] mvo: mborzecki: actually not, we are not hashing kernels yet, so it will become even slower [08:51] raspberry pi 1 speed! [08:52] mborzecki: mvo: anyway, we should be able to cut down on the number of reseals, we get so many because current unasserted kernel logic [08:53] speeding them up is a different matter, it would need help from secboot, also there are trade offs [08:59] PR snapd#9350 opened: [RFC] store: handle v2 error when fetching assertions [09:00] pedronis: hi, let me know if this is what you had in mind ^ ; i'll add tests if the idea is ok [09:03] * pstolowski back to snapshots [09:19] PR snapd#9351 opened: tests: simplify repack_snapd_snap_with_deb_content_and_run_mode_first_boot_tweaks [09:32] 2020-09-15 11:22:19 Executing google:ubuntu-20.04-64:tests/main/preseed (sep150911-454471) (1/1)... [09:32] 2020-09-15 11:22:55 Successful tasks: 1 [09:32] :D [09:41] pstolowski: we should find a quiet time [09:41] pstolowski: and demo vscode to the team [09:41] pstolowski: it's such a gamechanger [09:43] okay, onto snap-confine changes [09:43] NOW IS THE TIME :D [09:54] zyga-kaveri: is it vim vs emacs vs vscode now? ;) [09:55] pstolowski: for some people it's always vim vs emacs [09:55] pstolowski: the rest just uses code [10:03] For me, it's emacs AND code. [10:04] I'm code and vim [10:04] but vim much less often [10:04] mainly inertia [10:04] but code is just genuinely better [10:06] code's remote is nice. [10:06] Especially with low memory/disk space VM... [10:06] yeah [10:07] I only wish it had riscv and mips binaries [10:08] Too bad for Perl, it needs at least 5.18 and my platform uses 5.8.9... [10:17] mborzecki: 9347 is at 1:28h already, I wonder if that is good or bad [10:17] mvo: looks like it's slow, i was able to see the logs and minimal smoke hit a 40m timeout [10:17] * mvo nods [10:18] maybe we should have a longer kill timeout for the nested suite [10:18] yeah, might be needed [10:18] let's see if it completes, I really hope it does (successfully) [10:22] mborzecki: and it just failed in 2020-09-15T10:22:06.2176601Z - google-nested:ubuntu-20.04-64:tests/nested/manual/minimal-smoke :( [10:22] mvo: so that's it, hit a timeout on minimal/smoke, but otherwise the rest ran succesfuly [10:23] mborzecki: aha, timeout? ok, that sounds very promising [10:23] mvo: yup, a spread kill timeout [10:23] mborzecki: yeah, just saw it. sounds like we should raise it and re-run, can you do that? [10:23] mborzecki: but that is *very* encouraging, well done mborzecki and ijohnson :) [10:24] mborzecki: I really hope/think this could give us the reliable tests we need [10:27] mvo: hm i'd be leaning towards landing the PR and bumping the timeout in another one, looking at the logs, the test stated at 9:10, and at 9:49 it was executing tests/smoke/remove in the nested vm, so just running slow/late [10:28] wdyt? [10:28] mborzecki: works for me [10:29] mborzecki: done [10:29] thanks! [10:29] mborzecki: ok, let's include the longer timeout as one of the needed next steps (is more logging next?) [10:29] PR snapd#9347 closed: tests/lib/nested.sh: use more focused cloud-init config for uc20 [10:41] mvo: pedronis: #9343 is updated with a bump in kill timeouts and a little tweak for cleaning the serial log before each test (otherwise it would accumulate) [10:41] PR #9343: tests: more logging for UC20 kernel test [10:43] mvo: what's #9348 and how it relates to #9343 ? (cc mborzecki ) [10:43] PR #9348: tests: print all the serial logs for the nested test [10:43] PR #9343: tests: more logging for UC20 kernel test [10:45] pedronis: looks like a followup/tweak we could do once 9343 lands? [10:46] mborzecki: the changes to the cloud init config don't make sense anymore in it? don't we land some other code now? [10:46] heh, didn't we land [10:46] ah, damn missed those [10:47] it needs a master merge? [10:47] ah, it just had one? [10:48] mborzecki: mvo: fwiw I'm working on resealing a bit less even with unasserted kernels [10:49] mborzecki: does it make sense to land 9343 as it is or instead pull the logging changes out and land that separately and then we merge master into the kernel reseal tests? [10:49] (cc pedronis -^) [10:50] we can land the logging first maybe? [10:50] i can cherry pick the relevant commits and open a PR [10:56] mborzecki: would think it's better, reading 9343 is a bit hard [10:56] ok [10:57] hmm landing #9311 would make it smaller still [10:57] PR #9311: nested: add support to telnet to serial port in nested VM [11:01] mborzecki: that looks ok and has a lot of +1 (it's a bit repetitive but so nested.sh) [11:02] mborzecki: I merged master into 9311, once that is green we should merge to master [11:03] mborzecki: also 9311 is not really needed for gce, we can always merge later [11:03] mhm [11:03] but in a meeting so only have 10% of my brain [11:15] PR snapd#9352 opened: test: improve logging in nested tests [11:15] pedronis: mvo: ^^ just the logging [11:23] mborzecki: \o/ [11:23] * mvo is still in a meeting [11:26] I am putting together a custom-image as documented, works pretty good. I would now like to have some configurations changed like "sudo snap set somesnap foo=bar" for my custom-image, what would be the right way to do this? create a snap which is exclusively running those inside the configure hook? like a configuration-snap ? or is there some other prefered way? [11:29] dariball, the typical way is to use a custom gadget and set them in the "defaults:" in snapycraft.yaml [11:30] err. [11:30] s/snapcraft.yaml/gadget.yaml/ [11:31] dariball, https://snapcraft.io/docs/gadget-snap [11:31] therefore I would fork the default gadget snap, right, because I can only have one gadget snap? [11:31] yep [11:32] and I would add arbitrary commands to the prepare-device hook? [11:32] no [11:32] yu would use the gadget.yaml options for connections and for setting defaults [11:32] look at the linked page above [11:33] and running arbitrary commands shall be avoided? [11:34] because let's say I have some cli tool from a snap I would like to run ? [11:35] well, thats not esaily doable without forking the snap and make some daemonized script [11:36] (snaps can not easily call commands from other snaps, else you'd be able to just create a snap full of scripts) [11:37] okok, thx this already helps alot, hope I get along with connections + defaults, looks good... [11:37] if you have a brands stroe you *can* use the snapd REST API from a config or tooling snap though, that allows things beyond the limited stuff [11:37] *brand store [11:38] but use of the interface (snapd-control) that gives you access to the API is brand store bound, snaps in the global store do not get permission to use it [11:40] mvo: mborzecki: I haven't proposed but here are the changes to reseal less with unasserted kernels: https://github.com/pedronis/snappy/commit/39746b163590c1f6be0103362b8fcfeee0f32338 [11:57] ok, modified snap-confine and some small bits, running in complain snap-confine apparmor profile to see what we get [12:05] PR snapd#9185 closed: secboot: use the snapcore/secboot native recovery key type [12:05] funny when spread hangs on discarding a node [12:07] brb [12:10] pedronis: the patch looks reasonable, are you going to propose a branch? [12:12] mborzecki: probably not before we get other things in [12:12] PR snapcraft#3285 closed: v1 plugins: lock godep's dependencies [12:12] PR snapcraft#3287 opened: elf: reduce noise in the developer debug logs [12:12] pedronis: ack [12:24] heh, finally the nested suite is in progress in 9352 [12:29] exported tools work [12:29] I need to adjust apparmor profiles [12:29] but it's functional [12:29] \o/ [12:43] heh, nested tests i ran: !!!! X64 Exception Type - 0E(#PF - Page-Fault) CPU Apic ID - 00000000 !!!! [12:44] that's when rebooting from install -> run [12:44] mborzecki, I saww that yesterday as erll [12:44] well [12:44] ouch [12:45] and random, didn't happen in other tests [12:45] it is very sporadic [12:46] maybe -smp 1 could help mitigate this? [12:47] cmatsuoka, I saw that just running with kvm enabled [12:47] mborzecki, are you running with kvm enalbed right? [12:48] cachio: what version of ovmf are we using in our tests? [12:48] cmatsuoka, need to check, I tried with the latest and the once from proposed [12:51] cachio: no, this was on gcp [12:52] cmatsuoka: and the whichever version of qemu/ovmf we have in focal [12:52] mborzecki, how did you run the test on gcp? [12:52] cachio: spread -debug -v google-nested:ubuntu-20.04-64:tests/nested/... [12:53] cmatsuoka, this is the version we use of ovmf 0~20191122.bd85bf54-2ubuntu3 [12:54] mborzecki, well the tests in core20 suite will run without kvm but tests on manual suite will run with kvm enabled [12:54] mborzecki, don't know which test showed that panic [12:55] cachio: tests/nested/manual/core20-early-config so maybe it's related to kvm [12:55] mborzecki, ahh, I think so [12:58] the groovy ovmf is a bit newer, but many changes in upstream since then [13:00] PR snapd#9311 closed: nested: add support to telnet to serial port in nested VM [13:24] mborzecki: 9352 just passesd [13:24] mvo: just landed it [13:24] mborzecki: \o/ third in a row, fingers crossed [13:24] mvo: i'll cherry pick the spread test from your PR and add it to kernel reseal one [13:24] mborzecki: sounds great [13:25] mborzecki: feel free to close the test PR from me once you chrry-picked to the right place [13:25] PR snapd#9352 closed: test: improve logging in nested tests [13:42] I'm going to break for lunch now [13:42] ttyl [13:44] pedronis: mvo: i've updated https://github.com/snapcore/snapd/pull/9331 [13:44] PR #9331: boot: reseal when changing kernel [13:45] mvo: i've dropped the patch where test waits for snap to become available, we have that covered in nested setup now [13:45] PR snapd#9338 closed: tests: add nested core20 kernel reseal test [13:45] PR snapd#9351 closed: tests: simplify repack_snapd_snap_with_deb_content_and_run_mode_first_boot_tweaks [13:49] mborzecki: reviewed, it needs a 2nd review [13:49] pedronis: thanks! [13:49] mvo: ijohnson: could you take a look at #9331? [13:49] PR #9331: boot: reseal when changing kernel [13:49] sure [13:51] mborzecki: of course [13:52] my own comment on it are addressed in #9340 (which is adding some tests, a doing some refacor/renames) [13:52] PR #9340: boot: streamline bootstate20.go reseal and tests changes === tomwardill_ is now known as tomwardill === bluesabre_ is now known as bluesabre === marosg_ is now known as marosg === beisner_ is now known as beisner === coreycb_ is now known as coreycb === philroche_ is now known as philroche === urluck_ is now known as urluck === ogra_ is now known as Guest47593 [14:10] PR snapd#9321 closed: cmd/snap/model: specify grade in the model command output [14:17] mborzecki: pedronis I had a question on the unit tests in 9331 [14:17] perhaps I have misunderstood something [14:18] there is at least one test that my PR changes to do something quite different [14:18] more in line with the name [14:19] ah, something else, yes I think some of those tests are cheating a bit [14:20] because the mock bootloader has limitations [14:20] ok, is it alright that we aren't testing the same way things are being used in real life? [14:20] well, we just feed all that to a mocked secboot reseal [14:21] that just check that the processing looked alright [14:21] do we have anywhere that unit tests end to end with the actual expected boot chain? or is that just unnecessary since we will have the spread tests [14:21] so I think it's ok but might confuse us later, but it's not a blocker atm [14:21] ok, thanks [14:21] ijohnson: yes, we do have tests that are more realistic [14:21] seal_test.go own tess are more realistic [14:21] ok sounds good [14:22] the comment about checking the env seems legit though [14:22] it might be easier to pick it up in my PR tough [14:24] that's fine with me [14:24] I'm finishing up a reivew of 9340 now btw === xnox1 is now known as xnox [14:51] 16.04 failed with google:ubuntu-16.04-64:tests/main/lxd:snapd_cgroup_neither [14:53] * mvo is off for a couple of minutes to taxi kids around, tg for emergencies [14:55] mborzecki, again for me https://paste.ubuntu.com/p/6F4jypF8Wd/ [14:55] PR snapd#9353 opened: [RFC] configcore: do not error in console-conf.disable for install mode [14:59] hm looks like we still need to do something about the client timeout [15:00] mborzecki: which client timeout? [15:00] doTimeout in client.go [15:00] was doing a couple of kernel reseal runs on gcp and one failed with cannot communicate with the server, client timeout exceeded and so on [15:01] it's set to 50s atm, maybe we should have an env variable to make it longer [15:12] * cachio lunch [15:18] ah [15:18] yeah we should probably bump that timeout, there are reports of it not being long enough on the forum too on like rpi or somethin I think [15:20] mhm [15:22] PR snapcraft#3238 closed: db: introduce generalized datastore [15:29] zyga-kaveri: hey, please have snap-confine PRs still go through us. feel free to ping me and I'll assign someone [15:30] not that you weren't going to do that, just wanted to make sure people know :) [15:30] cc pedronis ^ [15:30] * jdstrand would cc mvo but he's not here atm :) [15:43] hm mvo isn't around anymore? [15:43] yeah he said he had to go taxi his kids around [15:43] ijohnson: ah, thanks for the info [15:44] mborzecki: any pr's I should review right now? [15:44] I reviewed both 9331 and 9340 this morning [15:48] ijohnson: you can try #9341 [15:48] PR #9341: tests: add nested core20 gadget reseal test