[00:02] <ijohnson> hey cmatsuoka what's up
[00:02]  * ijohnson is mostly there he thinks
[00:06] <cmatsuoka> hmm I forgot what I was going to say
[00:06] <cmatsuoka> anyway
[00:07] <cmatsuoka> I see many test failing because pi-kernel is not available, do you know what happened there?
[00:10] <ijohnson> cmatsuoka: hmm could be store outage
[00:10] <ijohnson> cmatsuoka: I can see the pi-kernel on my arm64 machine now
[00:11] <cmatsuoka> ijohnson: could you try to download the snap?
[00:11] <ijohnson> let me try
[00:12] <ijohnson> cmatsuoka: you need to specify a channel, there is no latest channel on the pi-kernel
[00:12] <cmatsuoka> ah I see
[00:12] <ijohnson> if you do `snap download pi-kernel --channel=20/edge` it works
[00:12] <ijohnson> but just `snap download pi-kernel` won't work
[00:13] <cmatsuoka> but did that change recently?
[00:13] <ijohnson> cmatsuoka: hmm don't think so
[00:14] <cmatsuoka> because some old tests started to fail
[00:14] <ijohnson> hmm did you check the full snapd service logs?
[00:14] <ijohnson> the spread tests will always do verbose HTTP logging for when they talk to the store
[00:15] <ijohnson> so you should see some 400s or 500s if the store was having a fit
[00:15] <cmatsuoka> let me see...
[00:18] <cmatsuoka> ijohnson: https://github.com/snapcore/snapd/pull/8495/checks?check_run_id=587219964
[00:18] <mup> PR #8495: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <https://github.com/snapcore/snapd/pull/8495>
[00:19] <cmatsuoka> ijohnson: maybe you can see something there that could tell us what's happening
[00:20] <cmatsuoka> oh it's in prepare-image
[00:20] <ijohnson> cmatsuoka: yeah looks like store having a fit
[00:20] <ijohnson> https://www.irccloud.com/pastebin/6ZKFXKKS/
[00:20] <ijohnson> store responded with 429
[00:20] <mup> PR snapcraft#3037 closed: plugins: introduce v2.CMakePlugin <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3037>
[00:21] <cmatsuoka> too many requests
[00:21] <cmatsuoka> pft, ok
[00:22] <cmatsuoka> do you know when it resets the request counter?
[00:23] <ijohnson> cmatsuoka: just restart the test, it's usually temporary
[00:24] <cmatsuoka> ok good, thanks, will do that
[00:24] <ijohnson> np
[01:08] <mup> PR snapcraft#3042 opened: repo: minor debug log tweaks <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3042>
[01:47] <mup> PR snapcraft#3034 closed: repo: add initialize_snapcraft_defaults() to set default source lists <Created by cjp256> <Closed by cjp256> <https://github.com/snapcore/snapcraft/pull/3034>
[01:50] <mup> PR snapcraft#3040 closed: V2 autotools plugin <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3040>
[02:17] <mup> PR snapcraft#3042 closed: repo: minor debug log tweaks <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3042>
[03:31] <jamesh> cjp256: I've updated PR 3031 to use a custom parameter type.  Let's see how that goes.
[03:31] <mup> PR #3031: cmd/snap-confine-suid-trampoline: add new helper <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/3031>
[04:05] <mup> PR snapd#8499 opened: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499>
[05:53] <mborzecki> morning
[06:24] <zyga> Good morning
[06:24] <mvo> good morning zyga
[06:25] <zyga> How are you guys
[06:25] <zyga> Hopefully today I will make progress on dbus
[06:27] <mborzecki> zyga: mvo: morning guys
[06:27] <mborzecki> have you seen https://github.com/snapcore/snapd/pull/8496 ?
[06:27] <mup> PR #8496: interfaces/apparmor: use differently templated policy for non-core bases <⛔ Blocked> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8496>
[06:28] <mborzecki> supper happy this is moving forward
[06:29] <mvo> good morning mborzecki
[06:36] <zyga> mborzecki: not yet
[06:36] <zyga> but we talked about this in Toronto
[07:01] <pstolowski> morning
[07:03] <zyga> hey pawel :)
[07:05] <mborzecki> pstolowski: hey
[07:39] <mborzecki> hhm trying to investigate the context deadline exceeded thing that happens in our tests randomly https://paste.ubuntu.com/p/zr6fNXMMbf/
[07:40] <mborzecki> but looking at go stdlib side, the whole timeout detection in http.Client.do() looks quite brittle
[07:48] <mborzecki> heh `time.Now().After(deadline)` heh if this is true then Client.do() returns httpError{timeout:true} otherwise it's urlError
[07:59] <pedronis> mvo: hi
[08:00] <mvo> pedronis: hi
[08:00] <pedronis> mvo: #8489 is all green but of course the required things don't exist there, it needs reviews and you will have to land it manually I suppose
[08:00] <mup> PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489>
[08:02] <pedronis> mborzecki: zyga:  ^ needs reviews
[08:03] <zyga> mm
[08:03] <mvo> pedronis: yeah, it does not have review, once it does I can land it manually and adjust the required things in the settings
[08:04] <pedronis> zyga: do you noticed/know why it seems we don't get the tooltip/panels with the detected error lines anymore?
[08:04] <zyga> yeah, I noticed
[08:04] <zyga> I was looking at why just now when looking at this diff
[08:05] <zyga> I think            echo "::add-matcher::.github/spread-problem-matcher.json" needs to be in an earlier step
[08:05] <zyga> or there's a typo
[08:05] <pedronis> but it happens  not only here
[08:05] <pedronis> also weirdly enough the emails I get have the count
[08:06] <zyga> hmm
[08:08] <pedronis> mvo: could you land #8488, it failed on the store in one test,  it will unblock various other PRs
[08:08] <mup> PR #8488: bootloader: add efi pkg for reading efi variables <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8488>
[08:10] <mvo> pedronis: sure, will do
[08:13] <mvo> pedronis: meh, I had no idea how annoying efi vars are :(
[08:13] <mvo> pedronis: anyway, landed
[08:13] <mup> PR snapd#8488 closed: bootloader: add efi pkg for reading efi variables <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8488>
[08:13] <pedronis> mvo: thank you, I will pick up now Claudio's #8463
[08:13] <pedronis> now
[08:13] <mup> PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463>
[08:40] <mvo> zyga,pedronis: re, looks like network is a bit unstable today, did I miss anything in the last 20min
[08:41] <zyga> mvo: I'm doing a review of your actions patches
[08:41] <zyga> mvo: hopefully nothing else
[08:42] <pedronis> mvo: I updated #8463 if you want to double check my changes
[08:42] <zyga> mvo: https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393570583
[08:42] <mup> PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463>
[08:42] <mup> PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489>
[08:45] <jamesh> Are updates to core20/edge held up for manual review?
[08:59] <mup> PR snapd#8498 closed: run-checks: use consistent "Checking ..." style messages <Simple 😃> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8498>
[09:00] <mborzecki> omg that github workflow PR, wish we didn't need to have this hack
[09:00] <mvo> mborzecki: you mean the amount of duplicated code?
[09:01] <mvo> mborzecki: that is very sad indeed
[09:01] <mborzecki> and the full workflow name appears in the jobs list :/
[09:01] <mborzecki> and it's controlled by repo level switches :/
[09:01] <mborzecki> and it's all duplicated
[09:01] <mborzecki> mvo: a clever hack nonetheless ;)
[09:03] <mborzecki> feels like gh actions are half baked atm
[09:04] <mvo> mborzecki: yeah, I wish we had yaml ancors or anything really
[09:05] <jamesh> rustup generates their workflows from parameterised source yaml files
[09:05] <mborzecki> jamesh: but it's done offline right?
[09:06] <mborzecki> or is there an action that does that? :)
[09:06] <jamesh> mborzecki: yes.  The generated workflows need to be committed to git
[09:07] <mborzecki> we still need the dependency on the unit tests
[09:07] <mborzecki> (even if we were to split workflows per spread ssytem)
[09:08] <zyga> mvo, mborzecki: jobs and steps can have a name and an ID
[09:09] <zyga> we can keep the ID as is but give it a shorter name
[09:21] <zyga> brb
[09:31] <mup> PR snapd#8500 opened: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8500>
[09:31] <mborzecki> shoud help with random failure in httputil retry tests ^^
[09:31] <mborzecki> still a mess, but nothing we can do, especially when sticking to older go versions
[09:38] <mborzecki> mvo: zyga: so a silly thought about gh actions, what if we just accept to run the unit tests more than once, and run it on the target system if possible (eg. via a docker container and whatnot) or a reasonmable default eg 18.04
[09:38] <zyga> mborzecki: that's what mvo did, no?
[09:38] <mborzecki> zyga: i mean workflow per spread system
[09:38] <zyga> yeah
[09:38] <zyga> as for running them only in spread
[09:38] <zyga> I disagree
[09:39] <zyga> that's somewhat slower and much more costly
[09:39] <zyga> but we can just try
[09:40] <mvo> mborzecki: you suggest one workflow per spread system and each workflow would have the unit tests first?
[09:40] <pedronis> I think we should try to live with a setup for a bit, after mvo split lands, before making other changes
[09:41] <zyga> yeah
[09:41] <pedronis> we are really blocked at the moment by everything being red and and needing full re-run or mvo override
[09:41] <zyga> I would only make cosmetic tweaks (names) so that it's nicer on a PR page
[09:41] <mvo> or we land it and do cosmetic tweaks in the followup :)
[09:41] <mvo> ?
[09:41] <mvo> (it still needs a +1)
[09:41] <zyga> yes totally
[09:42]  * zyga checks the comments
[09:42] <mborzecki> mvo: yes, the unit test could run in a docker container of the target system or some known default if there's none or just run the spread unit test job on the target system separately first
[09:43] <pedronis> that seems it would slow down things
[09:43] <pedronis> anyway it all sounds more work than we can afford atm
[09:43] <mborzecki> the downside it it'd obviously take longer, the upside is that unit tests would run on more systems where they happen o fail sometimes when mocking is lacking
[09:44] <mborzecki> but you can also restart workflows :P
[09:44] <mvo> mborzecki: we run the unit tests in spread on the target too (tests/unit/go)
[09:45] <pedronis> this all sounds a discussion to have at the earliest in 3 weeks
[09:45] <pedronis> tbh
[09:45] <mvo> the trade-off is unit tests first or the risk of spinning up 50 machines that all fail
[09:48] <zyga> FYI: I scaled down my deployment to 7 nodes (currently busy) to force some traffic onto canonistack instances
[09:48] <zyga> if something falls over I will scale back up
[09:48] <mborzecki> hmmm could we cache the job results somehow?
[09:49] <zyga> https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393621491
[09:49] <mup> PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489>
[09:49] <zyga> mborzecki: yes
[09:49] <mborzecki> as in, cache the result of unit tests or individual spread tests
[09:49] <zyga> mborzecki: we could
[09:49] <zyga> mborzecki: that's actually pretty brilliant
[09:49] <mborzecki> and then skip those jobs when workflow restarts?
[09:49] <zyga> mborzecki: if sha matches and is green
[09:49] <zyga> just go ahead :)
[09:49] <mvo> oh, that's interessting
[09:49] <zyga> we could use the cache action for that
[09:50] <zyga> mvo: https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393621491
[09:52] <mup> PR snapd#8489 closed: github: partition the github action workflows <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8489>
[09:52] <zyga> mborzecki: can you merge master or rebase https://github.com/snapcore/snapd/pull/8500
[09:52] <mup> PR #8500: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8500>
[09:52] <zyga> mborzecki: it will need the new checks to pass anyway
[09:52] <zyga> mborzecki: and I want to use it as a guinea pig
[09:53] <mborzecki> haha ok
[09:53] <zyga> with a bit of luck canonistack workers will now pick it up
[09:55] <mborzecki> hmm Error restoring google:ubuntu-18.04-64:tests/completion/ (apr150717-345956) : read tcp 10.113.57.178:48146->35.229.25.208:22: read: connection reset by peer
[09:56] <mborzecki> did we run out of some quota on gce?
[09:56] <zyga> no, probably my fault
[09:56] <zyga> let's wait for 20 minutes to see if the new setup works
[09:57] <mup> PR snapd#8494 closed: tests: preserve size for centos images on spread.yaml <Simple 😃> <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8494>
[09:59] <zyga> mborzecki: we should figure out caching for "go test"
[09:59] <zyga> 7 minutes of each run is there
[09:59] <mup> PR snapd#8495 closed: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8495>
[10:00] <mborzecki> ehh more of that `Error restoring google:ubuntu-20.04-64:tests/main/parallel-install-services (apr150717-996191) : read tcp 10.113.57.84:36100->34.74.8.8:22: read: connection reset by peer`
[10:02] <zyga> we are mow using canonistack as well as my "cloud"
[10:02] <zyga> 6 workers but hopefully with way better network
[10:02] <zyga> we have enough memory and CPU load is low
[10:03] <zyga> and network is much faster actually
[10:20] <mborzecki> hmm https://github.com/snapcore/snapd/pull/8499 wat?
[10:20] <mup> PR #8499: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499>
[10:21] <zyga> hmmm?
[10:22] <zyga> mborzecki: maybe he has a github repo
[10:22] <zyga> we can send a PR with a response
[10:22] <zyga> / thank you for this comment
[10:23] <mborzecki> it's probably some weird snap that produces lots of data in $SNAP_DATA or somesuch
[10:23] <zyga> mborzecki: lol
[10:23] <zyga> he made a typo
[10:23] <zyga> so we didn't waste money "testing" that PR
[10:23] <zyga> mvo: I guess that's one for you to handle
[10:24] <zyga> mvo: https://github.com/snapcore/snapd/pull/8499
[10:24] <mup> PR #8499: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499>
[10:25]  * mvo is in a meeting
[10:36] <zyga> brb
[10:57] <mborzecki> damn, error: cannot download snap "pi-kernel": no snap revision available as specified
[10:57] <mborzecki> keeps repating
[11:00] <mborzecki> weird, why it's using 20-pi3/edge while xnox's xnox-core-20-pi-arm64.model is using 20/edge
[11:00] <mvo> mborzecki: yeah, I think the kernel team did something to "pi-kernel"
[11:00] <pedronis> I think 20-pi3 doesn't exist anymore
[11:00] <mborzecki> xnox: do you know whether we should be using 20/edge for pi kernels now?
[11:00] <pedronis> also why are we 20 there ?
[11:00] <pedronis> we using 20 there?
[11:01] <mvo> oh, funny "pi-kernel" has no latest track anymore
[11:01] <mvo> so snap info pi-kernel gives no reply
[11:01] <mborzecki> pedronis: this is what we're using https://github.com/snapcore/snapd/blob/master/tests/lib/assertions/developer1-pi-uc20.model.json
[11:02] <pedronis> I think that's broken now
[11:03] <mborzecki> ok, the official (?) model ubuntu-core-20-pi-arm64.model is using 20/stable, so 20/edge is ok right?
[11:03] <pedronis> mborzecki: yes
[11:03] <pedronis> I think so
[11:06] <xnox> mborzecki: please stop using that, and instead use canonical signed dangerous model that is now available from models repo & from assertions service.
[11:06] <mup> PR snapd#8501 opened: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8501>
[11:06] <xnox> 20/edge is the way to go
[11:07] <mborzecki> xnox: thanks!
[11:07] <pedronis> that's our own prepare-image tests
[11:09] <ogra> xnox, a post on the forum would be really nice (where to find models etc) ... though probably post-release
[11:09] <ogra> so our customers can start rolling their own core20 imgs
[11:10] <xnox> ogra:  no =) nobody shall find out anything about core20 ever.
[11:10] <ogra> haha
[11:10]  * xnox giggles
[11:10] <pedronis> mvo: for latest you need pi2-kernel I think, pi-kernel is 18/20 only afaict
[11:16] <zyga> cachio: hey
[11:16] <zyga> cachio: around? :)
[11:38] <mup> PR snapd#8502 opened: github: try caching test results <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8502>
[11:39] <mborzecki> zyga: ^^
[11:39] <zyga> yeah, noticed :)
[11:40] <mborzecki> zyga: poor man's test result cache
[11:57] <cmatsuoka> zyga: hey
[11:57] <zyga> mmm
[11:58] <zyga> what's up :) ?
[11:58] <cmatsuoka> zyga: are you aware of snap-confine-undesired-mode-group errors in fedora? or are these fixed already in master?
[11:58] <zyga> cmatsuoka: can you show me the log please
[11:58] <cmatsuoka> sure, just a sec
[11:59] <zyga> cmatsuoka: there's one thing in master that is fixing the only known, to me, failure of this test
[11:59] <cmatsuoka> https://github.com/snapcore/snapd/pull/8495/checks?check_run_id=587546789
[11:59] <mup> PR #8495: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8495>
[11:59] <zyga> yes
[11:59] <zyga> this is fixed in master
[11:59] <cmatsuoka> excellent, thanks!
[12:09] <pedronis> cmatsuoka: hi, I worked a bit on #8463, it should be ready to land I think, but it's very red (for various reasons)
[12:09] <mup> PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463>
[12:10] <cmatsuoka> pedronis: I just checked it, thanks for the fixes/improvements
[12:12] <pedronis> cmatsuoka: Chris asked you to review: https://github.com/snapcore/secboot/pull/46
[12:12] <mup> PR secboot#46: Make SealKeyToTPM enforce a key size of 64-bytes <Created by chrisccoulson> <https://github.com/snapcore/secboot/pull/46>
[12:12] <cmatsuoka> ack
[12:40] <mup> PR snapcraft#3039 closed: build providers: setup initial apt source configuration <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3039>
[13:07] <mborzecki> mvo: can you use your superpowers and merge https://github.com/snapcore/snapd/pull/8501 ?
[13:07] <mup> PR #8501: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <⚠ Critical> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8501>
[13:23] <mvo> mborzecki: sure
[13:23] <mborzecki> mvo: thanks!
[13:24] <mup> PR snapd#8501 closed: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <⚠ Critical> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8501>
[13:27] <mvo> mborzecki: does 8476 look ok otherwise (beside this one point you made there)
[13:36] <zyga> ok, I can now turn off my home server
[13:37] <zyga> we have 38 workers in two locations
[13:38] <roadmr> zyga: maybe you can lease some DC capacity from Stéphane :)
[13:38] <zyga> roadmr: that's an interesting idea
[13:38] <zyga> mvo: ^ we probably could :)
[13:39] <mup> PR snapd#8476 closed: secboot: add tpm support helpers <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8476>
[13:41] <mvo> zyga: heh, interessting
[13:53]  * zyga breaks for lunch
[13:59] <mup> PR snapd#8481 closed: cmd/snap-update-ns: handle EBUSY when unlinking files <Bug> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8481>
[13:59] <mup> PR snapd#8485 closed: cmd/snap/debug/boot-vars: add opts for setting dir and/or uc20 vars <Simple 😃> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8485>
[14:13] <zyga> thank you mvo!
[14:16] <pedronis> cmatsuoka: not really urgent but we should probably merge secboot_tpm.go and tpm.go at some point, they are not big, it's not very clear what each does. the other option is to rename secboot_tpm.go to something else
[14:21] <mup> PR snapd#8500 closed: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8500>
[14:22] <ijohnson> zyga: so is robust mount namespace updates already enabled by default on master ?
[14:23] <cmatsuoka> pedronis: agree, I considered that myself when moving tpm.go to secboot but decided to propose that later to avoid changing too much in a single step
[14:28] <zyga> ijohnson: yes it is
[14:28] <ijohnson> nice
[14:28] <zyga> ijohnson: 2.45 will have it
[14:28]  * zyga is partially away, prepping food 
[14:33] <pedronis> ijohnson: cmatsuoka: I did some digging about the SecureBoot variable
[14:34] <cmatsuoka> pedronis: ah interesting, any new finding?
[14:34] <pedronis> cmatsuoka: see the PR comments
[14:34]  * cmatsuoka checks
[15:16] <zyga> hmmmm
[15:16] <zyga> did canonistack just die?
[15:17] <roadmr> deadstack
[15:19] <zyga> hmm
[15:20] <zyga> not sure what happened
[15:20] <mup> PR snapd#8453 closed: [RFC] travis.yml: re-enable travis <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8453>
[15:22] <ijohnson> pedronis: #8497 is ready again
[15:22] <mup> PR #8497: boot/bootstate20: re-factor kernel methods to use new interface for state <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8497>
[15:25] <pedronis> ijohnson: I was in meetings, looking now
[15:29] <ijohnson> thanks
[15:33] <pedronis> ijohnson: commented
[15:40] <ijohnson> pedronis: do you want a separate PR/rebase to extract just the change for https://github.com/snapcore/snapd/pull/8497#discussion_r408941648 ?
[15:40] <mup> PR #8497: boot/bootstate20: re-factor kernel methods to use new interface for state <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8497>
[15:41]  * zyga settles for quiet work
[15:41] <pedronis> ijohnson: maybe yes, I would expect a test change for it in some form
[15:42] <ijohnson> pedronis: ok, fwiw that was changed in the PR that's open
[15:42] <ijohnson> that will be easy to extract out though
[15:46] <mup> PR snapd#8503 opened: boot/bootstate20: fix bug in try-kernel cleanup <Bug> <Simple 😃> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8503>
[15:46] <ijohnson> pedronis: broken out as ^
[15:47] <pstolowski> mvo: i've pushed my early config test to https://github.com/stolowski/snapd/tree/core18-early-config
[15:47] <jdstrand> sil2100 (cc mvo): hey, the core20 snap is failing with https://paste.ubuntu.com/p/DrhfhjWG9y/. can you confirm this is ok or not?
[15:47] <pstolowski> mvo: to run it: google-nested:ubuntu-18.04-64:tests/nested/manual/core-early-config
[15:48] <cmatsuoka> cachio: unfortunately we still need sb enabled to be able to run the full encryption test
[15:48] <cmatsuoka> cachio: error: cannot seal the encryption key: cannot store encryption key: cannot add EFI secure boot policy profile: cannot compute secure boot policy digests: the current boot was performe
[15:48] <cmatsuoka> d with secure boot disabled in firmware
[15:51] <jdstrand> sil2100 (mvo): I'm going to approve it to unblock the queue, but please look into it
[15:53] <mvo> pstolowski: thanks, in a meeting right now, will have a look once that is over
[15:54] <cachio> cmatsuoka, is that on classic?
[15:54] <pedronis> ijohnson: thanks, it wasn't clear that the calls number where not a consquences of some of the refactorings
[15:54] <mvo> jdstrand: thanks, problably https://paste.ubuntu.com/p/DrhfhjWG9y/ is something for xnox to review
[15:54] <ijohnson> pedronis: yes the refactor actually exposed the bug :-)
[15:55] <cmatsuoka> cachio: mm, yes, the test is supposed to run on classic
[15:55] <cachio> cmatsuoka, ok, I'll take a look
[15:56] <xnox> mvo:  can you explain that output?
[15:56] <cmatsuoka> cachio: thanks, I'll run a few more tests here and will be back after lunch
[15:56] <cachio> cmatsuoka, just first let me finish the tests changes I did
[15:56] <cachio> could you give me a link to your tests?
[15:56] <xnox> mvo:  i.e. these files were removed from .deb's in focal, and hence new revisions of core20 snap don't have them......
[15:57] <mvo> xnox: jdstrand runs a tool that compares the files from one base to the next base and when stuff is removed it alerts and blocks the snap
[15:57] <ijohnson> mborzecki: around?
[15:57] <xnox> mvo:  why are you asserting contents of .deb's that are part of core20 snap?
[15:57] <cmatsuoka> cachio: yes, of course. I'm running the test with PR #8409
[15:57] <mvo> xnox: I don't, the store-review-tools do
[15:57] <mup> PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409>
[15:57] <mvo> xnox: it's there to ensure that bases are stable and drop support
[15:58] <xnox> jdstrand:  those files are dropped in focal, despite equivalent functionality still present.
[15:58] <xnox> jdstrand:  you should see new added files, like subiquitycore/palette.py
[15:58] <xnox> jdstrand:  and that systemd-timesyncd is now a standalone package, which is included.
[15:59] <jdstrand> xnox: ack, that's fine. thanks!
[15:59] <jdstrand> xnox: just wanted to make sure it was intentional
[15:59] <xnox> jdstrand:  it was =)
[15:59] <xnox> jdstrand:  if you have questions like these in the future, core20 bug tracker is on github snapcore/core20 issues
[16:00] <xnox> jdstrand:  or foundations-crew@lists.canonical.com as foundations owns/builds core20
[16:00] <cachio> cmatsuoka, nice, I'll take a look after lunch as well
[16:00] <jdstrand> xnox: right, that is why I pinged sil2100, but I can change the workflow
[16:01] <xnox> jdstrand:  he mostly deals with "stable" like core18 / core16
[16:01] <xnox> jdstrand:  core20 will be dead to me next thursday! and onto bootstrapping core22
[16:01] <jdstrand> xnox: heh, so should foundations-crew@lists.canonical.com be used for core16 and core18?
[16:01] <sil2100> I like how xnox said "stable" ;)
[16:02] <jdstrand> mvo: ok, so core20 revisions are now passing automated reviews again and should be caught up in a few minutes
[16:02] <xnox> jdstrand:  yes you can. foundations-crew@ is one stop shop for core snaps, subiquity, netplan, etc.
[16:03] <jdstrand> ok, thanks
[16:03]  * cachio lunch
[16:04] <jdstrand> xnox: that list is internal only, right?
[16:05] <xnox> jdstrand:  yes.
[16:34] <mvo> cmatsuoka: looks like 8463 is ready to land but one core20 test is failing, i.e. google:ubuntu-core-20-64:tests/core/basic20, could you please disable this part of the test and/or move to nested (if disable, TODO:UC20: is probably fine). then this can land
[16:44] <mvo> cmatsuoka: nevermind
[16:45] <mvo> cmatsuoka: pushed it
[16:56] <cachio> cmatsuoka, I don't see any nested test on #8409
[16:57] <mup> PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409>
[16:57] <cachio> cmatsuoka, is that the PR?
[17:02] <cmatsuoka> cachio: yes, the test is currently in tests/main/uc20-create-partitions-encrypted, it will be moved to nested
[17:04] <cachio> cmatsuoka, ah
[17:04] <cachio> ok
[17:06] <pedronis> this failure is weird: https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/1163/signedlogcontent/31?urlExpires=2020-04-15T17%3A04%3A50.7188367Z&urlSigningMethod=HMACV1&urlSignature=n1FukofiSafDmCcRwGnoncpLJS8mfgpiWZRJABATpvc%3D
[17:07] <ijohnson> pedronis: that url seems to be expired
[17:07] <ijohnson> which PR was that from?
[17:07] <pedronis> ijohnson: your PR
[17:08] <ijohnson> I noticed an odd lxd error from 8503, but I saved the logs and restarted the check
[17:08] <ijohnson> ah okay so same thing I saw
[17:08] <ijohnson> I'm looking into the issue now
[17:08] <pedronis> yes, weird failure in a lxd regression test
[17:10] <pedronis> ijohnson: it failed also in your other pr
[17:10] <pedronis> so maybe it's real
[17:10] <ijohnson> that's silly
[17:10]  * ijohnson will look more into it after lunch
[17:17] <zyga> what was the failure?
[17:17] <zyga> (in the lxd regression test)
[17:40] <cmatsuoka> pedronis: what would be a good place to define the kernel command line used in tpm sealing?
[17:46] <pedronis> ijohnson: that test is failing for master too
[17:46] <zyga> pedronis: do you have logs?
[17:46] <zyga> which test is it?
[17:48] <pedronis> something thinks we should have snapd around
[17:48] <pedronis> but actually we have core
[17:48] <pedronis> # lxc file push /usr/bin/snap bionic/usr/bin/snap
[17:48] <pedronis> Error: open /snap/snapd/current/usr/bin/snap: no such file or directory
[17:48] <zyga> ah
[17:48] <zyga> I think I know that this is
[17:48] <zyga> one moment
[17:49] <zyga> I think it's a race, I've seen things like that today
[17:49] <zyga> lxc launch ...
[17:49] <pedronis> it's google:ubuntu-18.04-64:tests/regression/lp-1871652
[17:49] <zyga> lxc file push ...
[17:49] <zyga> that fails
[17:49] <pedronis> it fails on master, running alone
[17:49] <zyga> I added lxc exec -- ls -l ...
[17:49] <zyga> that makes the container "ready" somehow
[17:49] <zyga> pedronis: thanks, I'll run it now
[17:52] <pedronis> cmatsuoka: the bootloaders should know about this, but for now you can hard code it in devicestate if it's easier
[17:53] <cmatsuoka> pedronis: ok, thanks
[17:53] <pedronis> cmatsuoka: actually is not devicestate, right? this goes into snap-boostrap?
[17:54] <pedronis> anyway you can hard code it in snap-boostrap for now
[17:54] <cmatsuoka> mm, yes, it's in snap-bootstrap
[17:59]  * ijohnson is back
[18:00] <ijohnson> zyga: did you reproduce/fix the issue?
[18:13] <zyga> ijohnson: yes, not yet
[18:14] <ijohnson> ah finally my spread run just got there I've got it now too
[18:14] <zyga> so
[18:14] <zyga> this is weird
[18:14] <zyga> there are two systems
[18:14] <zyga> the host and the container
[18:15] <zyga> neither has snapd snap
[18:15] <zyga> yet
[18:15] <zyga> lxc file push says
[18:15] <zyga> Error: open /snap/snapd/current/usr/bin/snap: no such file or directory
[18:15] <zyga> this is lxd 4.0.0
[18:15] <zyga> maybe lxd bug?
[18:15] <stgraber> you must have SNAP or SNAP_NAME in the environment
[18:16] <ijohnson> stgraber: we don't
[18:16] <zyga> indeed
[18:16] <ijohnson> https://www.irccloud.com/pastebin/dtcv3kfp/
[18:16] <zyga> and this passed recently as it's a fix for the bug that was plaguing lxd (from snapd side)
[18:16] <stgraber> what's the "lxc file push" you're running?
[18:16] <zyga> lxc file push /usr/bin/snap bionic/usr/bin/snap
[18:16] <zyga> I'm copying the snap command from the host (deb) to the guest
[18:17] <zyga> since the host has locally built snapd
[18:17] <stgraber> hmm, odd
[18:18] <ijohnson> stgraber: if I switch to 3.23/stable channel the same command works
[18:18] <ijohnson> and actually switching back to 4.0/stable channel again the same command fails again
[18:18] <ijohnson> looks like a lxd regression to me
[18:18] <stgraber> hmm, we didn't touch any of the file code between 3.23 and 4.0
[18:18] <zyga> yeah, I just tried that too
[18:19] <zyga> stgraber: did you release 4.0 recently?
[18:19] <stgraber> we did just switch the snap over to core18 though
[18:19] <ijohnson> zyga: did you see the same behavior?
[18:19] <zyga> yes
[18:19] <zyga> works perfectly on 3.23
[18:20] <zyga> ijohnson: shall we update LXD channel and call it a day?
[18:20] <ijohnson> zyga: yeah I think that's fine for now
[18:20] <ijohnson> I'll file a PR
[18:20] <zyga> thank you!
[18:20] <zyga> may I suggest something?
[18:20] <zyga> ijohnson: adjust the test to respect a variable
[18:20] <stgraber> testing with current 4.0 with same file push here to see if it reproduces easily
[18:20] <zyga> and update the variable in spread.yaml
[18:20] <zyga> I think I missed that we have this in the first place when I wrote the test
[18:21] <ijohnson> zyga: that's a great idea sure
[18:21] <zyga> thanks!
[18:21] <stgraber> root@sateda:~# nsenter --mount=/run/snapd/ns/lxd.mnt  -- ls -lh /usr/bin/snap
[18:22] <stgraber> lrwxrwxrwx 1 root root 32 Mar 11 05:46 /usr/bin/snap -> /snap/snapd/current/usr/bin/snap
[18:22] <stgraber> that would be why ^
[18:22] <zyga> !!!
[18:22] <zyga> that's weird
[18:22] <zyga> how did we get that symlink?
[18:22] <stgraber> presumably this wasn't that way on core16 so that wasn't a problem
[18:22] <zyga> is that a core18 regression?
[18:22] <zyga> is 3.23 using a different base?
[18:22] <stgraber> I mean, it's still a bug, it shouldn't actually pull the thing from inside the snap, it needs to pull it from the host
[18:22] <stgraber> yeah, we switched based from core to core18 a few hours ago
[18:22] <zyga> indeed
[18:23] <zyga> this makes sense
[18:23] <stgraber> root@sateda:~# nsenter --mount=/run/snapd/ns/lxd.mnt  -- ls -lh /var/lib/snapd/hostfs/usr/bin/snap
[18:23] <stgraber> -rwxr-xr-x 1 root root 14M Oct 30 12:17 /var/lib/snapd/hostfs/usr/bin/snap
[18:23] <zyga> cool, do you need a bug report for tracking?
[18:23] <stgraber> that's still fine and it should be using that
[18:23] <stgraber> you can if you want, but I'll probably have it fixed in 5min or so
[18:23] <zyga> ok, no need then
[18:27] <stgraber> INFO[04-15|18:27:30] Pushing /var/lib/snapd/hostfs/usr/bin/snap to /usr/bin/snap (file)
[18:27] <stgraber> that's more like it
[18:27] <stgraber> I absolutely hate that logic...
[18:28] <stgraber> such a mess to support absolute and relative paths, with and without symlinks in the middle, ...
[18:29] <stgraber> zyga, ijohnson: https://github.com/lxc/lxd/pull/7198
[18:29] <mup> PR lxc/lxd#7198: shared/util: Never look into the snap <Created by stgraber> <https://github.com/lxc/lxd/pull/7198>
[18:29] <stgraber> will be in the stable snap in the next 4-5 hours I'd say
[18:29] <ijohnson> cool thanks stgraber
[18:29] <zyga> stgraber: you like to live dangerously ;-)
[18:29] <zyga> but I admire that
[18:29] <stgraber> I've been doing daily stable cherry-picks lately
[18:29] <zyga> ah
[18:29] <zyga> I see
[18:29] <zyga> that's interest
[18:29] <zyga> *interesting
[18:29] <zyga> perhaps something we could do?
[18:30] <ijohnson> ahhhh we would have many folks very mad if we did as many stable releases as often as lxd :-)
[18:30] <stgraber> well, we have pretty reasonable testing on all distros and architecture, that's why it takes 4-5 hours before we're pretty confident it's not horribly broken :)
[18:30] <zyga> stgraber: having a regression test for push would be great tho :D
[18:31] <stgraber> yeah, not very practical to test in CI since we don't use any snaps there, but I'll sneak some file push testing into our snap tests
[18:33] <stgraber> test added
[18:33] <zyga> thank you :)
[18:36] <ijohnson> zyga: fancy a +1 before you EOD? https://github.com/snapcore/snapd/pull/8504
[18:36] <mup> PR #8504: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8504>
[18:36] <ijohnson> :-)
[18:36] <zyga> looking
[18:36] <zyga> I'm not EODing yet
[18:36] <zyga> deeeeply in dbus
[18:36] <ijohnson> one might say you are dbusly in dbus
[18:36]  * ijohnson stops
[18:36] <mup> PR snapd#8504 opened: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8504>
[18:36] <zyga> hahaha
[18:37] <zyga> +1
[18:38] <zyga> there's something magic about the words
[18:38] <zyga> "in progress"
[18:38] <zyga> that's so much better than "queued" we had before
[18:44] <mup> PR snapd#8505 opened: spread.yaml: switch back to latest/candidate for lxd snap <Test Robustness> <⛔ Blocked> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8505>
[18:46] <zyga> ijohnson: ^ nice
[18:46] <ijohnson> in progress does sound much better
[18:47] <ijohnson> I think in general whenever we need to do things like this, i.e. revert something to unbreak master while something is fixed, we should always open a PR undoing that change at the same time so at least we remember because there's an open PR about changing
[18:47] <zyga> yes
[18:47] <ijohnson> i.e. same thing when we move a system to unstable, we should open another PR which moves it back to stable
[18:48] <ijohnson> pedronis: we fixed the lxd issue on master
[18:48] <ijohnson> well we have a PR open
[18:48] <mup> PR snapd#8463 closed: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8463>
[18:49] <pedronis> ijohnson: I see
[18:50] <pedronis> ijohnson: anyway proves why I'm not a fan of dangling symlinks in bases
[18:51] <ijohnson> pedronis: yes but also why we should be very strict about letting snaps access things in bases too
[18:52] <pedronis> yes
[18:52] <ijohnson> pedronis: also did you discuss with maciej about picking up the necessary changes to gadget pkg et al to enable mbr support for create-partitions? should I try to push that forward? I was kinda hoping that he would pick that up but I keep forgetting to ask him before he EODs
[18:53] <pedronis> ijohnson: I don't know, it's actually a prereq to make the run mode changes relevant, right? because it's install time?
[18:53] <ijohnson> pedronis: yes
[18:54] <ijohnson> yes it's a blocker I mean
[18:54] <pedronis> ijohnson: have you started the other bootloaderKernelState ?
[18:55] <ijohnson> I'm wrapping it up now, but I need to figure out a way to write tests for it, I'd rather not rewrite the dozens of tests we already have for bootstate20 to use a mock uboot bootloader, etc.
[18:56] <pedronis> ijohnson: maybe just write direct tests for it?
[18:56] <pedronis> we probably need to reorg some of the other tests as table tests so that they are easier to reuse
[18:56] <ijohnson> pedronis: yes perhaps, need to think about it a little bit
[18:56] <pedronis> or have a actual test mixins
[18:57] <pedronis> we don't have a lot of the latter but we don sometimes
[18:58] <ijohnson> yeah we could refactor some of the tests big chunks to use helpers too
[18:58] <ijohnson> lots of the tests are just bootloader setup anyways
[18:59] <pedronis> cmatsuoka: is #8409 ready for Chris to look at it?
[18:59] <mup> PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409>
[19:00] <cmatsuoka> pedronis: will be in a minute, pushing a final commit
[19:01] <cmatsuoka> pedronis: pushed
[19:23] <mup> Issue core20#36 closed: SSH login shell says the system was minimized and to run "unminimize", but should not <Created by anonymouse64> <Closed by xnox> <https://github.com/snapcore/core20/issue/36>
[19:23] <mup> PR core20#42 closed: drop `unminimize` instructions that are not applicable on Core <Created by xnox> <Merged by xnox> <https://github.com/snapcore/core20/pull/42>
[19:52] <mup> PR snapd#8284 closed: config: add system.store-certs.[a-zA-Z0-9] support <Needs Samuele review> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8284>
[19:58] <mup> PR snapd#8503 closed: boot/bootstate20: fix bug in try-kernel cleanup <Bug> <Simple 😃> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8503>
[20:05] <mup> PR snapd#8504 closed: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8504>
[20:37] <mup> PR snapd#8506 opened: Add libnvidia-opticalflow as Nvidia library <Created by joedborg> <https://github.com/snapcore/snapd/pull/8506>
[20:37] <joedborg> hey everyone, if someone could take a quick look at ^ it'd be much appreciated
[21:36] <mwhudson> what determines what ends up in parts/$PART/install/usr/lib/python3/dist-packages using the python plugin?
[22:17] <mup> PR snapcraft#3043 opened: package-repositories: initial schema and meta read/write support <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3043>
[22:45] <mup> PR snapcraft#3044 opened: build providers: use ubuntu-ports mirrors for non-x86 platforms <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3044>