mupBug #1887153 opened: Installing network-manager drops the network <Snappy:New> <https://launchpad.net/bugs/1887153>00:53
mupBug #1887153 changed: Installing network-manager drops the network <Snappy:New> <https://launchpad.net/bugs/1887153>00:59
mupBug #1887153 opened: Installing network-manager drops the network <Snappy:New> <https://launchpad.net/bugs/1887153>01:05
mborzeckimvo: hey06:17
mvohey mborzecki06:19
mvomborzecki: do you think you could have a look at 9358? seems like we need it for 2.47 :)06:19
mvomborzecki: what did I miss otherwise?06:20
mborzeckimvo: yes, i'll push a patch dropping that fmt.Println06:20
mborzeckiotherwise it seems ok06:20
mborzeckimvo: so the gadget assets update seems to be a nogo now, maybe we should have a managers test instead?06:21
mborzeckii totally missed that we're only modifying the conf files06:21
mvomborzecki: why a no-go? wouldn't changing/resigning work?06:22
mvomborzecki: yeah, sorry for that, I messed up the test :(06:22
mborzeckimvo: i mean, in the current form it's a no go, needs more work06:23
mvomborzecki: yes, sorry, then we are in agreement06:23
mvomborzecki: I was wondering if maybe just changing the pe header timestamp would be enough06:23
mvomborzecki: I'm looking how archivable this is right now06:24
mborzeckimvo: cool, that would modify the contnt so a differnt hash, thus triggering assets to be updated and a reseal after06:24
mvomborzecki: yes, iirc secboot does understand pe binaries already to some extend so there are libs06:25
mvomborzecki: and then the same problem applies to the kernel06:25
mvomborzecki: I guess managers should not write code or tests anymore :(06:25
mvomborzecki: anyway, looking at this now06:26
mborzeckimvo: why miss out on all the fun? :)06:26
mvomborzecki: lol06:26
mborzeckimvo: fwiw, maybe we should emply reflection for modeenv at some point, when we add more fields06:28
mborzeckimvo: i mean field tags, and so on, then it'd be easier to not loose new fields by accident06:29
mvogood point06:29
zygagood morning06:32
mvozyga: good morning06:34
mborzeckizyga: hey06:35
zygahey guys06:38
zygaeveryone had breakfast but Lucy decided to go back to bed for some reason06:39
zygaso here I am06:39
zygalet me see if I can connect to the office06:39
zygaI had some issues last night, I think the switch is faulty06:39
zygait's got a flaky power cable and turns off easily when moved06:39
mvomborzecki: testing proper gadget manip now locally before pushing to avoid congesting spread06:43
zygaI'll slowly start working while Lucy is not doing much06:46
mvosounds good06:48
mborzeckimvo: tweaked #9358 a little, please take a look06:58
mupPR #9358: boot/modeenv: track unknown keys in Read and put back into modeenv during Write <UC20> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9358>06:58
zygaadjusted ExportEntryt to be a struct now07:03
mborzeckipstolowski: hey07:04
mborzeckimvo: looked at gadget assets and 9348, both ran nested core20 tests which were successful (not counting gadget assets reseal test)07:05
mvomborzecki: http://paste.ubuntu.com/p/6x7ZXFhccH/ is what I have for the gadget reseal test07:10
mvomborzecki: it's running locally now07:10
tobias_good morning07:10
mborzeckimvo: haha, can you maye do sed -i 's/This program cannot be run in DOS mode/This program cannot be run in     mode/' otherwise it's hard to spot the O/0 change07:16
mvomborzecki: sure thing07:31
mvomborzecki: the test is funny right now, I ran it but now it resealed when I installed the same gadget again, I'm confused but maybe I need to merge master or something else if funky07:31
mborzeckimvo: ok, let me try to run it07:34
mvomborzecki: I'm also running it now07:36
mvomborzecki: I changed the XXX07:36
mborzeckimvo: so you're saying it stopped after the install of unchanged gadget?07:40
mvomborzecki: http://paste.ubuntu.com/p/mCVhCYSv5t/07:41
mvomborzecki: that's the full log07:41
mvomborzecki: eh, sorry, that does not look full, one sec07:41
mvomborzecki: so SEALED_KEY_MTIME_1 and _2 are different, the original error scrolled past my logbuffer .(07:43
mvomborzecki: which is strange because the "snap change" output of the gadget install tells me that there is gadget update needed07:43
mvomborzecki: which also reminds me that I should add code that checks if a gadget update was triggered to this test07:44
mborzeckimvo: added some extra logs and running again07:58
mborzeckipedronis: Model/Brand was a problem when i tried adding tags, looked like there'd be a separate structure needed for read/write with just the eported fields (and tags)08:02
pedronismborzecki: no, we can do something simple, I will try. but in a meeting now08:03
mborzeckipedronis: ok08:03
mborzeckimvo: after reinstalling the gadget snap, the reseal count is unchaged08:29
mborzecki(at least that's what I see here)08:29
mvomborzecki: oh, so the test should work? that's great actually08:34
mborzeckimvo: not quite, https://github.com/snapcore/snapd/pull/9341/files#r48925909308:35
mupPR #9341: tests: add nested core20 gadget reseal test <Run nested> <UC20> <⛔ Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9341>08:35
mborzeckimvo: and i'm executing it manually, one by line, there's something wrong with the gadget.yaml i think08:35
mborzeckimvo: 2020-09-16T08:34:04Z ERROR bootloader not declared in any volume, need to find out why08:36
mborzeckimvo: fwiw, i can work on the test and push fixes there :)08:37
mvomborzecki: pleae, in a meeting now08:37
mvomborzecki: and then another one08:37
mvomborzecki: and then another one08:37
mvomborzecki: fwiw, this is my updated kernel reseal draft https://github.com/snapcore/snapd/compare/master...mvo5:kernel-reseal-better-test but I think this misses kernel re-signing08:41
mvomborzecki: most FYI but if the gadget stuff goes quickly it might be a good one to check out too :)08:41
mvomborzecki: and THANK YOU for this :)08:41
=== doko_ is now known as doko
mborzeckimvo: so, i got a reboot, but then this appeared: https://paste.ubuntu.com/p/jRJhY4yp7W/09:04
mborzeckii'm looking at boot chains, grubx64.efi for the recovery chain was changed (by the test), but also bootx64.efi was changed during the update09:11
mvomborzecki: oh, interessting, aha, I think it's because we resign both09:25
mvomborzecki: just resign both09:25
mvomborzecki: I will explain more, in a meeting still09:25
ijohnsonmorning folks09:52
pstolowskimorning ijohnson !09:55
tobias_morning ijohnson09:55
ijohnsonhey pstolowski tobias_ o/09:56
pstolowskipedronis: hey, i think i'm confused by one thing we discussed about snapshots - why would we keep external .hash file (apart from .idx), seems it would duplicate meta.sha3_384 or i misunderstood that?09:57
pedronispstolowski: sorry, what external hash file?09:58
pstolowskipedronis: afair we would have the .zip file, but then we would introduce .idx and .hash file (for hash of the metadata), but maybe i made a mistake in my notes09:59
pedronispstolowski: no, idx would have both the set id and the new hash09:59
pedronisnot two files09:59
pedronisdoes that make more sense?09:59
mupPR snapd#9357 closed: interfaces/process-control: add sched_setattr to seccomp <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9357>10:00
pstolowskipedronis: not really, i'm not clear why would we need to store the hash again if we have it in meta.sha3_384 (which would be caclulcated with id:0 in new code)10:02
pedronispstolowski: maybe, we need to sync again10:03
ijohnsonhuh can anyone else actually access the log for this spread run ? https://github.com/snapcore/snapd/pull/9333/checks?check_run_id=111904478210:03
mupPR #9333: tests/nested, fakestore: changes necessary to run nested uc20 signed/secured tests <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9333>10:03
ijohnsonwhen I cleck on "view raw logs" it just gives me like 4 lines of trying to allocate a self hosted worker, but the time it took to run is 1hr 39m, so spread must have run10:03
pstolowskipedronis: also, maybe i'm confused, but it seems we store timestamp inside meta, and it will affect checksum as well, we may want to do move it out as well10:05
pedronispstolowski: let's chat when we both have a moment10:10
* pedronis lunch10:10
xnoxijohnson:  so you are saying that beta-image recovery partition is now broken?10:18
ijohnsonxnox: sorry do you mean about last night's discovery about the regression after resealing ?10:18
xnoxijohnson:  meaning, it's no longer forward compatible with current snapd/sealing any more?10:18
xnoxijohnson:  yes.10:18
xnoxseed partition will always have older kernel/snapd/snap-bootstrap than what is in the ubuntu-data partition. and it currently must remain forever forward compatible with how ubuntu-data is sealed.10:19
xnoxuntil we can update seed partition with a new recovery system.10:19
ijohnsonxnox: well I don't think the recovery partition is broken because while the snap-bootstrap there will always drop new keys that it didn't know about in May from the modeenv which breaks fresh install, an old boot to recover will always work because that recovery system must have been generated with a snapd that didn't have resealing logic in it10:20
xnoxbut doesn't resealing logic, reseals things upon upgrading snapd in the ubuntu-data partition?10:20
ijohnsonxnox: hmm although I wonder if an old system refreshed to a new snapd with modeenv on the encrypted data gets the new resealing keys in the modeenv, if during recover mode we will ever modify the data modeenv10:21
ijohnsonI don't think we will10:21
ijohnsonbut I can test that10:21
ijohnsongive me a moment, I will try with a beta image refresh to edge snapd, then recover, then go back to normal run mode and see if things are broken10:21
xnoxi guess it will reseal when you also trigger a reseal event? or do we now have two codepaths depending on how the system was installed?!10:22
xnoxthat would help us understand the scope of brokeness =)10:22
ijohnsonmy hope/expectation is that recover mode won't mutate the run mode modeenv, because for example the initramfs of recover mode writes the modeenv for recover mode to the tpmfs data10:22
xnoxmaybe refresh snapd first, then refresh kernel or like gadget.10:22
ijohnsonbut maybe I'm wrong10:22
xnoxit has access to both, so it could do either, or bugs10:23
ijohnsonthe number of possible bugs is only limited by our imagination10:23
pedronismborzecki: this is what I'm doing: https://paste.ubuntu.com/p/PNXqQ5yWpw/ is it too naive? I'm not sure what you tried (I'm not happy with the model& syntax but I can change it)10:32
mborzeckipedronis: looks fine for the PR ijohnson has, i tried to do the serialization bits and ran into trouble with model being <brand>/<model>10:36
ijohnsonmborzecki: is there a bug in my PR?10:36
pedronisno, but mborzecki suggested trying to avoid mantaining the list of fields separately from the struct10:37
pedronisby using tags10:37
ijohnsonyeah that makes a lot of sense10:37
mborzeckiijohnson: no, we're trying to use tags to replace the hard coded list of keys10:37
pedronisbut then I think he tried to do even more and then it gets complicated10:37
mborzeckipedronis: anyways, the thing you have seems fine10:38
jameshijohnson: re. https://github.com/snapcore/core20/issues/72 -- my interest came from the "GDM on Ubuntu Core" experiments I've been doing.  It's the sudo group I'm more interested in than docker.10:38
ijohnsonjamesh: right understood10:39
ijohnsonjamesh: I will keep you and the issue updated with the result of the discussion we have tomorrow about this issue10:39
ijohnsonour more immediate concern is what to do for uc20 1.0 but that will involve some discussion of what to do in general for adding users to groups like that10:39
ijohnsonpedronis: hmm so if I use cloud-init to create a user on uc20 and don't run console-conf, should console-conf run when I transition to recover mode? seems wrong to me to run console-conf in recover mode, but then again I suppose since it was never run in run mode, you could just reboot the device back to run mode and then run console-conf anyways10:41
pedronisare we even running console-conf in non run mode?10:43
ijohnsonpedronis: well in this case it seems console-conf is being run in recover mode10:43
ijohnsonpedronis: it's my feeling that that's a bug10:43
ijohnsonI seem to have refreshed the kernel snap without getting a reboot scheduled10:45
ijohnsonah the same revision of the kernel snap is on edge as beta10:45
pedronisijohnson: remember though that the service also trigger the recovery menu10:45
pedronisso i'ts a bit unclear exactly what we need there10:46
ijohnsonpedronis: sure, shall I just file a bug about it and we sort out the right thing to do there later ?10:46
pedronisto be clear I'm not sure never running it is the general answer either10:47
pedronisthe problem is more if its runs what should it do/not do10:47
pedronisand when should it run10:47
ijohnsonyeah it's a bit confusing10:48
ijohnsonpedronis: how does disabling console-conf work from the gadget defaults now?10:49
pedronisit creates a the complete file (that hasn't changed)10:49
ijohnsonok, I wonder if we copy the complete file over from the run system to the recover system, I think we do10:50
ijohnsonI actually don't think we do :-(10:50
pedroniswhy would we10:51
pedronisanyway if it's disabled in the gadget we will disable it in recover10:51
pedronismvo has a PR about that triggering a bug10:51
ijohnsonpedronis: right that was my question was about whether we will disable it in recover when we seed10:51
ijohnson(when we seed the recovery system that is)10:51
ijohnsonsorry this has probably already been discussed and I'm just now discovering the fixes that were merged for it10:52
mborzeckimvo: hm so, the gadget asset test manages to get the first mtime of sealed key before snapd reseals on mark boot successful (and the branch does not have fixes from pedronis about avoiding that)10:54
mborzeckimvo: also explains why i could run the test manuall line by line10:55
pedronisijohnson: this is the PR https://github.com/snapcore/snapd/pull/9353 it actually but install, not recover10:55
mupPR #9353: configcore: do not error in console-conf.disable for install mode <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9353>10:55
pedronismborzecki: which fixes you mean? (I fixed lots of things)10:56
ijohnsonpedronis: hmm so I wonder if we need to make that work for recover mode too10:56
ijohnsonpedronis: do you want me to test that works as advertised ?10:56
mborzeckipedronis: not resealing with unasserted kernel when modeenv was not changed10:56
ijohnson(gadget default to disable console-conf in recover mode works)10:56
pedronismborzecki: ah10:56
pedronisijohnson: yes10:57
ijohnsonack will add it to my list of things to test this morning10:57
ijohnsonstill trying to see if beta image gets broken by installing edge snapd10:57
pedronismborzecki: I think maybe it's not that hard to do marshal/unmarshal as well but probably not something to do now11:06
pedroniss/probably/very likely/11:06
pedronismborzecki: ijohnson: pushed11:08
ijohnsonpedronis: hmm so I have an image built with edge snapd and old pc-kernel, but upon refreshing the pc-kernel, it did not add the current_trusted_... keys to the modeenv ever11:17
pedronisijohnson: that's expected we have no code to do that atm11:17
pedronismaybe that isn't clear11:18
ijohnsonpedronis: hmm ok11:18
ijohnsonpedronis: wait but I built the image with a new edge snapd, shouldn't it have added the current_trusted_... keys to the modeenv during install mode?11:18
pedronisbut then you said old kernel, doesnt it lose stuff as we know?11:19
ijohnsonyes I built the image with an old kernel but new snapd11:19
pedronisthere are lots of moving parts, they behave kind of predictably at least11:19
ijohnsonlet me think about this, probably this is expected then I guess11:19
pedronisijohnson: we really need to decide what to support or not in terms of backward compat11:20
pedronismvo knows that is an open question11:20
ijohnsonprobably not an issue for 1.011:20
ijohnsonsince new images for 1.0 should be fine11:20
ijohnsonah dammit I'm an idiot, too many different images I just ended up booting an old snapd with an old kernel so it worked and never tried to write the extra modeenv stuff11:27
* ijohnson tries again to not be so foolish11:27
ijohnsonok so yes booting an image with new snapd and old kernel fails to seed in run mode as expected because the old kernel there is broken, however given the fact that we never will add the resealing keys to the modeenv, an old snapd refreshing to a new snapd will not add the keys, so thus will not trigger a resealing and thus will not be broken by transitioning back to an old recover mode snapd11:32
ijohnsonxnox: so we don't have a regression where your old installed system refreshes to new snapd, then goes back into old recovery system11:32
ijohnsonpedronis: since old beta systems will never be properly resealed since we don't add those keys to the modeenv, will old beta systems ever get broken by the new secboot version format for example?11:34
pedronisijohnson: sorry, I was confused last night,  the secboot version thing works differently, v0 keys system will keep v0 keys11:34
ijohnsonok, so beta systems will keep on working as always11:35
ijohnsonthey just will never reseal11:35
pedronisthey can reseal, to v0 keys again11:35
pedronisso the problem is really more on our side about that11:35
ijohnsonbut does our code ever trigger resealing for those old systems though?11:35
pedronisit will, it just explode doing so11:35
pedronisbecause we don't have the required data11:36
ijohnson... so old systems will eventually get broken by resealing11:36
pedroniswell, I would say more by trying to reseal11:36
pedronisunless we do something11:36
ijohnsonso is that what you meant about backwards compatibility? maybe we don't care that beta systems will eventually explode themselves ?11:37
* ijohnson thought we were not supposed to let beta systems explode themselves11:37
pedroniswe said we would prefer not to, but we also didn't propose not to break them11:38
pedronisbut as I said, it's really a decision to make with mvo11:38
ijohnsonI see11:38
pedronisbut we also said we would have tests but we don't11:39
ijohnsonok well I will leave testing that aside for the moment then, we know that beta systems are not yet exploded by refreshing to new snapd11:39
ijohnsonpedronis: we do have tests tho11:39
mborzeckimvo: #9341 is green, but want to push one more commit addressing the race11:39
mupPR #9341: tests: add nested core20 gadget reseal test <Run nested> <UC20> <⛔ Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9341>11:39
ijohnsonwe build an image from beta and then refresh to edge11:39
ijohnsonerr rather I think we download the beta image then refresh to edge11:39
* zyga adjusts remaining manifest leftovers11:39
ijohnsonand see if it explodes11:39
pedronisijohnson: I see, it's probably not fully enough11:39
pedronisanyway it's a decision to make11:39
ijohnsonpedronis: yes there's an additional test we should have I think which I mentioned last night which would have caught this issue with modeenv in pr's before things landed11:40
ijohnsonI will try to propose it today11:41
ijohnsonreally this new test is about making sure old initramfs doesn't break new snapd11:41
pedroniswhat I mean, we need a test that upadtes snapd and then a kernel11:41
pedronisor simply updates snapd and then reboots (that probably would fail already)11:42
ijohnsonmmm yes we could have a test that updates snapd first then kernel, it's a bit tricky because we need to disable refreshes from the gadget otherwise the old kernel will get refreshed first before new snapd11:42
ijohnsonand as I said new snapd + old kernel right now is totally broken11:42
xnoxpedronis:  ijohnson: ignoring beta systems exploading is fine, but the problem repeats itself though cause v1 (GA) image should continue to reseal going forward, no?11:55
ijohnsonxnox: yes new images built with 1.0 will properly reseal and not explode11:55
xnoxas in new snapd snap, should be backwards compatible with older snapd that we ship on GA11:55
xnoxwhen we make v2 resealing11:55
ijohnsonxnox: yes it will be11:55
xnoxor v3 resealing etc11:55
xnoxand like still dont' have a way to update seed partition.11:55
ijohnsonxnox: did you get a chance to test my core-initrd pr yet?11:56
ijohnsonxnox: last night's discoveries led to the realization that we really really need a new snap-bootstrap in the initramfs11:56
ijohnsonpedronis: should` snap reboot --recover` without a label work? it doesn't but feels like it should :-/  it currently has an unhelpful error: `error: cannot request system reboot: method "POST" not allowed` because the client and cli doesn't validate whether systemLabel is empty11:58
pedronisijohnson: that's a bug11:58
ijohnsonpedronis: ack I can try to fix that, we could probably just do a GET to /v2/systems and then figure out the current one and then do a post to /v2/systems/<current>11:59
pedronismissing spread test11:59
pedronisijohnson: maybe11:59
ijohnsonoh hey look at that I just defeated fde on uc2011:59
pedronisI mean not sure exactly what the solution is12:00
ijohnsonpedronis: so gadget default for console-conf disable does not work at all for recover mode12:00
ijohnsonbecause console-conf is disabled for run mode and doesn't run, or doesn't let you create a user properly, but then you go to recover mode and it lets you create a user and login12:00
pedronisijohnson: we need to fix that12:01
pedronisI'm going to be in a meeting now12:01
ijohnsonyes, I think mvo's patch can just be expanded to recover mode too12:01
ijohnsonI will test this out12:01
mborzeckimvo: i think that #9341 is good to go12:01
mupPR #9341: tests: add nested core20 gadget reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9341>12:01
mborzeckidropped the blocked label12:02
ijohnsonoh actually maybe that won't work12:03
mvomborzecki: \o/12:05
ijohnsonmvo: we have a blocker for 2.47, console-conf doesn't get properly disabled in recover mode12:06
ijohnsonI'm looking at a fix now12:06
mvoijohnson: ta12:06
cachioxnox, hi, I see there are OVMF_CODE_4M and OVMF_VARS_4M in ovmf12:16
cachioxnox, I tried those with uc20 yesterday but image didn't boot12:16
cachioxnox, any idea if we need any change to qemu command line? or any other configuration to support that?12:17
xnoxcachio:  those ones are without keys, you want the ones with MS keys and 4k, no?12:18
xnoxcachio:  which filenames did you use?12:18
cachioxnox, I tried  OVMF_CODE_4M.secboot.fd and OVMF_VARS_4M.snakeoil.fd12:26
mupPR snapd#9358 closed: boot/modeenv: track unknown keys in Read and put back into modeenv during Write <UC20> <⚠ Critical> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9358>12:26
cachioxnox, and the ms as well12:27
ijohnsonmborzecki: haha so your branch bboozzoo/uc20-tweak-sid-packaging-params.go exists in the my local git tree and breaks building the snap locally because go interprets that file in the .git dir as a go file12:28
ijohnson*face palm*12:29
ijohnson2020/09/16 12:27:18 processFiles failed with: /root/parts/snapd-deb/build/.git/refs/remotes/bboozzoo/bboozzoo/uc20-tweak-sid-packaging-params.go:1:1: expected 'package', found 'IDENT' c0de648a090a1690ce9d3f6b350d5e3dc92d6bd712:29
ijohnsonfeels pretty clear to me the debian packaging should ignore everything in .git as possible go source files12:29
zygaok, running spread now12:30
zyganeed to adjust two variable names12:30
zygaand lots of related function names12:30
zygabut that's just small thing12:30
zygafirst need to check if nothing is broken12:30
* ijohnson afk for coffee 12:33
cachioxnox, do we need to you diferent ones for uc20?12:46
zygabrb, see you at the standupo12:49
ijohnsonmmm I kinda just wanna copy the complete file for console-conf from the initramfs from run mode to recover mode if it exists12:52
ijohnsonmaybe that's too naive12:52
mborzeckihm that mbr thing on encrypted device shouldn't be a big problem, i think we can make sure that we readlink of the mount source and then walk the /sys/block/<dev>/slaves up to a point where there's no more slaves13:51
mborzeckithe upside is taht it will work even if there's multiple dm devices invovled, lvm, luks and others13:51
ijohnsoncachio: could you look at #9333 again?14:03
mupPR #9333: tests/nested, fakestore: changes necessary to run nested uc20 signed/secured tests <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9333>14:03
cachioijohnson, on that14:18
mvohey, I saw in 9359 a bunch of lines like: [  124.065523] snapd[670]: 2020/09/16 13:52:57.974158 stateengine.go:150: state ensure error: devicemgr: cannot mark boot successful: cannot compose run mode boot chains: cannot find expected boot asset bootx64.efi in modeenv - does that ring any bells?14:20
ijohnsonmvo: yes that was the regression I was talking about14:20
ijohnsonmvo: that is what happens when you boot a uc20 image with old kernel and new snapd with resealing, it is broken right now14:21
ijohnsonmvo: to fix it we need new snap-bootstrap in the kernel initramfs14:21
ijohnsonsorry if I was unclear about that in SU14:21
mupPR snapd#9359 opened: tests: improve kernel reseal test <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>14:21
mvoijohnson: oh, right14:30
ijohnsonyeah this is why even without my core-initrd pr merged it would be really good to have a new ubuntu-core-initramfs built14:30
cachioijohnson, done14:30
ijohnsoncachio: what do you mean about setting the env vars in spread.yaml? the vars are set from a specific tests's environment keyword which will be set before anything in spread's prepare, etc. is executed no?14:31
ijohnsonat least that's what I see14:32
cachioijohnson, you need to add them in environment:14:32
cachio    GOHOME: /home/gopath14:32
cachio    GOPATH: $GOHOM14:32
cachioin this part14:33
ijohnsoncachio: but why14:33
cachioor in the suite specific env dict14:33
cachiospread will set just those vars in the environment for the test14:33
cachiootherwise the var will not be available for the script14:35
ijohnsoncachio: hmm but I just use those variables for a specific nested/manual test where14:35
ijohnsonI set the env vars and use them from prepare and it works14:35
cachioijohnson, which ones?14:36
ijohnsoncachio: see my other pr, https://github.com/snapcore/snapd/pull/933414:36
mupPR #9334: tests/nested/manual: add test for grades above signed booting with testkeys <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9334>14:36
cachioijohnson, that works14:37
cachiowhat doesn't work is to set the model from cli14:38
ijohnsoncachio: right but why would we need to set the model from the cli14:38
ijohnsonis that a thing you do ?14:38
cachioijohnson, I do that for spread cron14:38
cachioI think those vars are not going to be chagned14:39
ijohnsoncachio: so do you still want me to define those things in the spread.yaml environment?14:39
cachioI used to update kvm, secboot, tpm, etc14:39
cachioijohnson, no14:39
ijohnsonI can address your other feedback now though14:39
cachioijohnson, for installing jq14:42
cachiowe already have a function called get_test_snap_suffix14:42
cachioit is in core-config.hs14:42
cachioin the lib14:42
cachiothat could be usedfull14:42
ijohnsonah ok, interesting, I can take a look at that14:43
=== pedronis_ is now known as pedronis
cachiocmatsuoka, I reviewed hte msr doc and the case we have is the following https://paste.ubuntu.com/p/wNVYrWxxxj/14:50
cachiocmatsuoka, because the msr code starts with 0xffffffff14:50
mupPR snapcraft#3288 closed: cmake v2 plugin: add help for cmake generators <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3288>14:50
mborzeckigot to run to a school meeting14:55
cmatsuokacachio: I think the document I mentioned earlier is this one: https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf15:04
cachiocmatsuoka, nice, thanks15:06
ijohnsoncachio: pr updated15:06
cachioijohnson, +115:08
pedronispstolowski: I commented on #935015:19
mupPR #9350: [RFC] store: handle v2 error when fetching assertions <Created by stolowski> <https://github.com/snapcore/snapd/pull/9350>15:19
* cachio lunch15:23
mupPR snapd#9341 closed: tests: add nested core20 gadget reseal test <Run nested> <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9341>16:06
mvopedronis: do you think you have capacity to look at 9293 ?16:08
mvopedronis: and maybe 9277 (which is actually more important)?16:09
pedronismvo: do we need to merge the improved reseal tests into 9277 before merging it?16:13
pedronismvo: https://github.com/snapcore/snapd/pull/9277#pullrequestreview-48978659516:14
mupPR #9277: secboot: add boot manager profile to pcr protection profile <Run nested> <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/9277>16:14
mvopedronis: yeah, I think it makes sense to merge master once more, maybe cmatsuoka  can do it? I will then branch 2.47 in the morning most likely16:16
mvothis should give it ample time to finish :)16:16
mupPR snapd#9360 opened: tests: make gadget-reseal more robust <Run nested> <Simple 😃> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9360>16:16
mvocachio: I noticed we get some nested uc18 failures now sometimes, do you think you could have a look if you spot something (a pattern or so?)16:17
* mvo dinners16:17
pedronismvo: so we need to land #9359 and #9360 first ?16:17
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>16:17
mupPR #9360: tests: make gadget-reseal more robust <Run nested> <Simple 😃> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9360>16:17
pedronisonly #935916:21
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>16:21
cachiomvo, checking16:29
cachiomvo, do you have any log?16:30
xnoxcachio:  we have stopped using snakeoil keys months ago, so snakeoil.fd has not been working for months now.16:30
pedronisijohnson: if we had strange cla check failure on mvo PR: https://github.com/snapcore/snapd/pull/9359/checks?check_run_id=1124165015, maybe you can look when you have a moment. I reviewed your cloud-init PR16:31
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>16:31
cachioxnox, I also tried ms keys16:33
ijohnsonThanks pedronis16:33
cachioxnox, should that work?16:33
ijohnsonThe cla-check issue seems to be a problem with cmatsuoka spread machine16:33
xnoxcachio:  it would be helpful if you could paste the full thing you are trying16:33
xnoxcachio:  qemu / libvirt, and which things were initialized with which and on what series.16:34
xnoxcachio:  i'll doublecheck what i do locally, and can compare.16:34
ijohnsoncmatsuoka: perhaps you updated your spread worker python?16:34
cachioxnox, ok16:36
cachiolet me prepare the environmnent16:36
cachioI'll leave you a note with that16:36
cmatsuokaijohnson: uhm? cla-checker?16:46
ijohnsoncmatsuoka: see pedronis' link above, the log has `claudio-spread-1` in it and has a python backtrace when it tries to run cla-checker16:46
cmatsuokaugh, let's see what's happening here16:47
mupPR snapd#9333 closed: tests/nested, fakestore: changes necessary to run nested uc20 signed/secured tests <Run nested> <Test Robustness> <UC20> <Created by anonymouse64> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9333>16:52
cmatsuokamvo: should I merge #9359 into #9277?16:53
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>16:53
mupPR #9277: secboot: add boot manager profile to pcr protection profile <Run nested> <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/9277>16:53
* cmatsuoka just came back from lunch16:53
cachiomvo, I didn't find the errors on nested uc18, I'll try to reproduce it manually here16:53
cmatsuokaijohnson: is the cla-check expecting to use python 2?16:53
ijohnsoncachio: the issue I think he saw was the same problem with users disappearing when spread tries to login to the nested machine16:54
ijohnsoncmatsuoka: hmm good question, let me take a look quickyl16:54
cachioijohnson, ah, ok, I think we need to skip the configuration for the images16:55
cmatsuokathe runner only had python 3 installed, I installed a minimal python2 now16:55
cachioijohnson, i'll create a pr for that16:55
ijohnsoncachio: ah interesting maybe that's the issue indeed16:55
cachioijohnson, yes16:55
ijohnsoncmatsuoka: it is possible that it only supports python2, but that doesn't seem to be the case just looking at the code, it seems fine python3 code to me from my non-experienced python eyes16:57
cmatsuokawe'll soon find out when the next cla-check job arrives16:58
ijohnsoncmatsuoka: yes after some work, checking things the cla script does not support python317:07
ijohnsonerr it needs to be ported slightly17:07
cmatsuokaijohnson: ok, the runner now has python 2 too17:07
ijohnsoncmatsuoka: ok well now I have a branch that works17:12
ijohnsonwith python3 that is17:12
cmatsuokaah nice17:12
mupPR snapd#9361 opened: tests/lib/cl_check.py: use python3 compatible code <Simple 😃> <Skip spread> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9361>17:17
mupPR snapd#9362 opened: tests: skip nested images pre-configuration by default <Run nested> <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9362>17:22
cachioijohnson, I thhink with this #9362 that issue should be gone17:22
mupPR #9362: tests: skip nested images pre-configuration by default <Run nested> <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9362>17:22
ijohnsoncachio: thanks I'll have a look in a bit17:23
=== ijohnson is now known as ijohnson|lunch
mvocmatsuoka: I think merging master is enough17:37
cmatsuokamvo: ack17:39
cachioijohnson|lunch, I need to go to the kinesiologist now, please leave a note if you think the fix is not correct17:45
cachioI'll take a llok once I am back17:45
* cachio -> kinesiologist17:45
* zyga is physically tired but mentally ready for coding17:56
=== ijohnson|lunch is now known as ijohnson
zygaI may have fixed crashes18:35
zyganow let's run and see18:35
ijohnsoncachio: 9362 lgtm, not clear that it will fix all of our problems, but I don't think it can hurt18:35
pedronisijohnson: https://github.com/snapcore/snapd/pull/9359/checks?check_run_id=1124222729 failed on nested.sh shell stuff, could look into it?19:11
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>19:11
ijohnsonpedronis: sure19:11
pedronisfailed sadly tbh :/19:11
pedronis /home/gopath/src/github.com/snapcore/snapd/tests/lib/nested.sh: line 290: $4: unbound variable19:11
ijohnsonwell I mean to be fair it was only provided 3 argument19:12
pedronisit's still sad19:12
ijohnsonyes still sad19:12
pedronisgiven how long it took to get to that error19:12
ijohnsondo you want me to push a fix to this branch ?19:12
ijohnsonack, one moment19:12
pedronisI don't see how to avoid another run19:13
pedronisunless somebody runs it locally19:13
ijohnsonis the concern that we wanted to branch 2.47 tonight with this pr merged ?19:13
ijohnsonI can tend to this pr if it needs some poking tonight19:14
ijohnsonotherwise I could try running it locally to confirm my patch works19:15
pedronisit's a prereq to run cmatsuoka PR again19:16
pedronisI think the idea was to cut 2.47 in the morning19:16
pedronisbut this shifts a bit everything19:16
pedronisanyway I don't have particular opionions on this anymore19:17
ijohnsonok, well I'm trying a spread run locally with qemu-nested on my desktop, perhaps that will be faster, if it passes I will pastebin the results into the PR19:18
zygaijohnson: how long does it take on your desktop19:20
zygaijohnson: and what are your cpu specs roughly?19:20
ijohnsonzyga: I dunno I haven't run this particular test, but in general qemu-nested is a few minutes faster on my first gen 16 core threadripper19:21
ijohnsonfaster than gce at least19:21
zygafaster than gce?19:21
zygahey, main/snap-run passed19:22
zygaI'll run more tests and see what else needs changes, snap-confine profile is still complain mode19:22
mupPR snapd#9318 closed: .github/workflows/test.yaml: also run 20.04 nested tests with UC20 label <UC20> <Created by anonymouse64> <Closed by anonymouse64> <https://github.com/snapcore/snapd/pull/9318>19:22
cachioijohnson, left you a comment in #936219:27
mupPR #9362: tests: skip nested images pre-configuration by default <Run nested> <Simple 😃> <Created by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9362>19:27
ijohnsoncachio: ah right I didn't realize that all the other nested tests were using the generic images19:28
ijohnsoncachio: did you ever measure for non-uc20 how much speedup you get from not rebuilding/reconfiguring the image fresh for each test ?19:28
cachioif you run 2 tests with the same image in the same machine you save time19:29
cachioijohnson, thats the measurement I did for uc2019:29
ijohnsonah so you didn't measure for non-uc20 then?19:29
cachionot sure for uc16 and uc1819:29
cachioI just measured the slowest19:30
ijohnsonI see19:30
ijohnsoncachio: sure let's merge this if we keep seeing the issue with minimal-smoke on uc18, then let's revert this pr and try something else19:31
cachioijohnson, sure19:31
cachioijohnson, well19:34
cachioI thinks for uc16 and 18 the time is pretty the same19:34
cachioI just compared PRs19:34
cachioand the times are pretty similar19:35
cachioallways between 34m and 40m19:35
zygaok, snapctl adjusted19:38
mupPR snapd#9363 opened: boot: adjust comments, naming, log success around reseal <Simple 😃> <Skip spread> <UC20> <Created by pedronis> <https://github.com/snapcore/snapd/pull/9363>19:47
pedronisijohnson: cmatsuoka: ^ simple PR addressing previous feedback19:48
* cmatsuoka checks19:49
pedronisijohnson: I made a comment about your doubt around overlord and snap-bootstrap, we already import some bits of overlord, what we don't want to import are the managers19:54
pedronisthat's why we build with nomanagers19:54
ijohnsonI didn't know about nomanagers19:55
ijohnsonok this is much easier now then19:55
ijohnsonthanks for the hint19:55
mupPR snapd#9363 closed: boot: adjust comments, naming, log success around reseal <Simple 😃> <Skip spread> <UC20> <Created by pedronis> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9363>20:07
pedronisijohnson: how it's going the local run of #9359 ?20:55
mupPR #9359: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9359>20:55
ijohnsonSorry got distracted but it passed21:04
ijohnsonpedronis: ^21:05
ijohnsonI just pasted in the results so you can force-merge if you like21:05
ijohnsonthe tests are not done running yet for nested uc20 yet21:05
ijohnsonin the pr that is21:06
cmatsuokaI think we should chip in and buy google some faster machines21:09
* zyga considers going to bed now21:14
zygaI'll finish the rest in the morning21:14
zygagood luck, see you tomorrow guys21:14
pedronisijohnson: cmatsuoka: I force merged it21:15
cmatsuokapedronis: thanks, I'll merge master into #927721:15
mupPR #9277: secboot: add boot manager profile to pcr protection profile <Run nested> <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/9277>21:15
pedroniscmatsuoka: thank you, was about to ask about that21:15
* pedronis calls is a day21:15
* cmatsuoka will patiently wait for the tests to complete21:17
mupPR snapd#9359 closed: tests: improve kernel reseal test <Run nested> <UC20> <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9359>21:18
cmatsuokawhich will still be way better than the 8h layover in LHR last year21:18
ijohnsonYeah we would have been in Frankfurt again this week!21:21

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!