| ijohnson | hey cmatsuoka what's up | 00:02 |
|---|---|---|
| * ijohnson is mostly there he thinks | 00:02 | |
| cmatsuoka | hmm I forgot what I was going to say | 00:06 |
| cmatsuoka | anyway | 00:06 |
| cmatsuoka | I see many test failing because pi-kernel is not available, do you know what happened there? | 00:07 |
| ijohnson | cmatsuoka: hmm could be store outage | 00:10 |
| ijohnson | cmatsuoka: I can see the pi-kernel on my arm64 machine now | 00:10 |
| cmatsuoka | ijohnson: could you try to download the snap? | 00:11 |
| ijohnson | let me try | 00:11 |
| ijohnson | cmatsuoka: you need to specify a channel, there is no latest channel on the pi-kernel | 00:12 |
| cmatsuoka | ah I see | 00:12 |
| ijohnson | if you do `snap download pi-kernel --channel=20/edge` it works | 00:12 |
| ijohnson | but just `snap download pi-kernel` won't work | 00:12 |
| cmatsuoka | but did that change recently? | 00:13 |
| ijohnson | cmatsuoka: hmm don't think so | 00:13 |
| cmatsuoka | because some old tests started to fail | 00:14 |
| ijohnson | hmm did you check the full snapd service logs? | 00:14 |
| ijohnson | the spread tests will always do verbose HTTP logging for when they talk to the store | 00:14 |
| ijohnson | so you should see some 400s or 500s if the store was having a fit | 00:15 |
| cmatsuoka | let me see... | 00:15 |
| cmatsuoka | ijohnson: https://github.com/snapcore/snapd/pull/8495/checks?check_run_id=587219964 | 00:18 |
| mup | PR #8495: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <https://github.com/snapcore/snapd/pull/8495> | 00:18 |
| cmatsuoka | ijohnson: maybe you can see something there that could tell us what's happening | 00:19 |
| cmatsuoka | oh it's in prepare-image | 00:20 |
| ijohnson | cmatsuoka: yeah looks like store having a fit | 00:20 |
| ijohnson | https://www.irccloud.com/pastebin/6ZKFXKKS/ | 00:20 |
| ijohnson | store responded with 429 | 00:20 |
| mup | PR snapcraft#3037 closed: plugins: introduce v2.CMakePlugin <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3037> | 00:20 |
| cmatsuoka | too many requests | 00:21 |
| cmatsuoka | pft, ok | 00:21 |
| cmatsuoka | do you know when it resets the request counter? | 00:22 |
| ijohnson | cmatsuoka: just restart the test, it's usually temporary | 00:23 |
| cmatsuoka | ok good, thanks, will do that | 00:24 |
| ijohnson | np | 00:24 |
| mup | PR snapcraft#3042 opened: repo: minor debug log tweaks <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3042> | 01:08 |
| mup | PR snapcraft#3034 closed: repo: add initialize_snapcraft_defaults() to set default source lists <Created by cjp256> <Closed by cjp256> <https://github.com/snapcore/snapcraft/pull/3034> | 01:47 |
| mup | PR snapcraft#3040 closed: V2 autotools plugin <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3040> | 01:50 |
| mup | PR snapcraft#3042 closed: repo: minor debug log tweaks <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3042> | 02:17 |
| jamesh | cjp256: I've updated PR 3031 to use a custom parameter type. Let's see how that goes. | 03:31 |
| mup | PR #3031: cmd/snap-confine-suid-trampoline: add new helper <Created by zyga> <Closed by zyga> <https://github.com/snapcore/snapd/pull/3031> | 03:31 |
| mup | PR snapd#8499 opened: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499> | 04:05 |
| mborzecki | morning | 05:53 |
| zyga | Good morning | 06:24 |
| mvo | good morning zyga | 06:24 |
| zyga | How are you guys | 06:25 |
| zyga | Hopefully today I will make progress on dbus | 06:25 |
| mborzecki | zyga: mvo: morning guys | 06:27 |
| mborzecki | have you seen https://github.com/snapcore/snapd/pull/8496 ? | 06:27 |
| mup | PR #8496: interfaces/apparmor: use differently templated policy for non-core bases <⛔ Blocked> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8496> | 06:27 |
| mborzecki | supper happy this is moving forward | 06:28 |
| mvo | good morning mborzecki | 06:29 |
| zyga | mborzecki: not yet | 06:36 |
| zyga | but we talked about this in Toronto | 06:36 |
| pstolowski | morning | 07:01 |
| zyga | hey pawel :) | 07:03 |
| mborzecki | pstolowski: hey | 07:05 |
| mborzecki | hhm trying to investigate the context deadline exceeded thing that happens in our tests randomly https://paste.ubuntu.com/p/zr6fNXMMbf/ | 07:39 |
| mborzecki | but looking at go stdlib side, the whole timeout detection in http.Client.do() looks quite brittle | 07:40 |
| mborzecki | heh `time.Now().After(deadline)` heh if this is true then Client.do() returns httpError{timeout:true} otherwise it's urlError | 07:48 |
| pedronis | mvo: hi | 07:59 |
| mvo | pedronis: hi | 08:00 |
| pedronis | mvo: #8489 is all green but of course the required things don't exist there, it needs reviews and you will have to land it manually I suppose | 08:00 |
| mup | PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489> | 08:00 |
| pedronis | mborzecki: zyga: ^ needs reviews | 08:02 |
| zyga | mm | 08:03 |
| mvo | pedronis: yeah, it does not have review, once it does I can land it manually and adjust the required things in the settings | 08:03 |
| pedronis | zyga: do you noticed/know why it seems we don't get the tooltip/panels with the detected error lines anymore? | 08:04 |
| zyga | yeah, I noticed | 08:04 |
| zyga | I was looking at why just now when looking at this diff | 08:04 |
| zyga | I think echo "::add-matcher::.github/spread-problem-matcher.json" needs to be in an earlier step | 08:05 |
| zyga | or there's a typo | 08:05 |
| pedronis | but it happens not only here | 08:05 |
| pedronis | also weirdly enough the emails I get have the count | 08:05 |
| zyga | hmm | 08:06 |
| pedronis | mvo: could you land #8488, it failed on the store in one test, it will unblock various other PRs | 08:08 |
| mup | PR #8488: bootloader: add efi pkg for reading efi variables <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8488> | 08:08 |
| mvo | pedronis: sure, will do | 08:10 |
| mvo | pedronis: meh, I had no idea how annoying efi vars are :( | 08:13 |
| mvo | pedronis: anyway, landed | 08:13 |
| mup | PR snapd#8488 closed: bootloader: add efi pkg for reading efi variables <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8488> | 08:13 |
| pedronis | mvo: thank you, I will pick up now Claudio's #8463 | 08:13 |
| pedronis | now | 08:13 |
| mup | PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463> | 08:13 |
| mvo | zyga,pedronis: re, looks like network is a bit unstable today, did I miss anything in the last 20min | 08:40 |
| zyga | mvo: I'm doing a review of your actions patches | 08:41 |
| zyga | mvo: hopefully nothing else | 08:41 |
| pedronis | mvo: I updated #8463 if you want to double check my changes | 08:42 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393570583 | 08:42 |
| mup | PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463> | 08:42 |
| mup | PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489> | 08:42 |
| jamesh | Are updates to core20/edge held up for manual review? | 08:45 |
| === pedronis_ is now known as pedronis | ||
| === pedronis_ is now known as pedronis | ||
| mup | PR snapd#8498 closed: run-checks: use consistent "Checking ..." style messages <Simple 😃> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8498> | 08:59 |
| mborzecki | omg that github workflow PR, wish we didn't need to have this hack | 09:00 |
| mvo | mborzecki: you mean the amount of duplicated code? | 09:00 |
| mvo | mborzecki: that is very sad indeed | 09:01 |
| mborzecki | and the full workflow name appears in the jobs list :/ | 09:01 |
| mborzecki | and it's controlled by repo level switches :/ | 09:01 |
| mborzecki | and it's all duplicated | 09:01 |
| mborzecki | mvo: a clever hack nonetheless ;) | 09:01 |
| mborzecki | feels like gh actions are half baked atm | 09:03 |
| mvo | mborzecki: yeah, I wish we had yaml ancors or anything really | 09:04 |
| jamesh | rustup generates their workflows from parameterised source yaml files | 09:05 |
| mborzecki | jamesh: but it's done offline right? | 09:05 |
| mborzecki | or is there an action that does that? :) | 09:06 |
| jamesh | mborzecki: yes. The generated workflows need to be committed to git | 09:06 |
| mborzecki | we still need the dependency on the unit tests | 09:07 |
| mborzecki | (even if we were to split workflows per spread ssytem) | 09:07 |
| zyga | mvo, mborzecki: jobs and steps can have a name and an ID | 09:08 |
| zyga | we can keep the ID as is but give it a shorter name | 09:09 |
| zyga | brb | 09:21 |
| mup | PR snapd#8500 opened: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8500> | 09:31 |
| mborzecki | shoud help with random failure in httputil retry tests ^^ | 09:31 |
| mborzecki | still a mess, but nothing we can do, especially when sticking to older go versions | 09:31 |
| mborzecki | mvo: zyga: so a silly thought about gh actions, what if we just accept to run the unit tests more than once, and run it on the target system if possible (eg. via a docker container and whatnot) or a reasonmable default eg 18.04 | 09:38 |
| zyga | mborzecki: that's what mvo did, no? | 09:38 |
| mborzecki | zyga: i mean workflow per spread system | 09:38 |
| zyga | yeah | 09:38 |
| zyga | as for running them only in spread | 09:38 |
| zyga | I disagree | 09:38 |
| zyga | that's somewhat slower and much more costly | 09:39 |
| zyga | but we can just try | 09:39 |
| mvo | mborzecki: you suggest one workflow per spread system and each workflow would have the unit tests first? | 09:40 |
| pedronis | I think we should try to live with a setup for a bit, after mvo split lands, before making other changes | 09:40 |
| zyga | yeah | 09:41 |
| pedronis | we are really blocked at the moment by everything being red and and needing full re-run or mvo override | 09:41 |
| zyga | I would only make cosmetic tweaks (names) so that it's nicer on a PR page | 09:41 |
| mvo | or we land it and do cosmetic tweaks in the followup :) | 09:41 |
| mvo | ? | 09:41 |
| mvo | (it still needs a +1) | 09:41 |
| zyga | yes totally | 09:41 |
| * zyga checks the comments | 09:42 | |
| mborzecki | mvo: yes, the unit test could run in a docker container of the target system or some known default if there's none or just run the spread unit test job on the target system separately first | 09:42 |
| pedronis | that seems it would slow down things | 09:43 |
| pedronis | anyway it all sounds more work than we can afford atm | 09:43 |
| mborzecki | the downside it it'd obviously take longer, the upside is that unit tests would run on more systems where they happen o fail sometimes when mocking is lacking | 09:43 |
| mborzecki | but you can also restart workflows :P | 09:44 |
| mvo | mborzecki: we run the unit tests in spread on the target too (tests/unit/go) | 09:44 |
| pedronis | this all sounds a discussion to have at the earliest in 3 weeks | 09:45 |
| pedronis | tbh | 09:45 |
| mvo | the trade-off is unit tests first or the risk of spinning up 50 machines that all fail | 09:45 |
| zyga | FYI: I scaled down my deployment to 7 nodes (currently busy) to force some traffic onto canonistack instances | 09:48 |
| zyga | if something falls over I will scale back up | 09:48 |
| mborzecki | hmmm could we cache the job results somehow? | 09:48 |
| zyga | https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393621491 | 09:49 |
| mup | PR #8489: github: partition the github action workflows <Created by mvo5> <https://github.com/snapcore/snapd/pull/8489> | 09:49 |
| zyga | mborzecki: yes | 09:49 |
| mborzecki | as in, cache the result of unit tests or individual spread tests | 09:49 |
| zyga | mborzecki: we could | 09:49 |
| zyga | mborzecki: that's actually pretty brilliant | 09:49 |
| mborzecki | and then skip those jobs when workflow restarts? | 09:49 |
| zyga | mborzecki: if sha matches and is green | 09:49 |
| zyga | just go ahead :) | 09:49 |
| mvo | oh, that's interessting | 09:49 |
| zyga | we could use the cache action for that | 09:49 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/8489#pullrequestreview-393621491 | 09:50 |
| mup | PR snapd#8489 closed: github: partition the github action workflows <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8489> | 09:52 |
| zyga | mborzecki: can you merge master or rebase https://github.com/snapcore/snapd/pull/8500 | 09:52 |
| mup | PR #8500: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8500> | 09:52 |
| zyga | mborzecki: it will need the new checks to pass anyway | 09:52 |
| zyga | mborzecki: and I want to use it as a guinea pig | 09:52 |
| mborzecki | haha ok | 09:53 |
| zyga | with a bit of luck canonistack workers will now pick it up | 09:53 |
| mborzecki | hmm Error restoring google:ubuntu-18.04-64:tests/completion/ (apr150717-345956) : read tcp 10.113.57.178:48146->35.229.25.208:22: read: connection reset by peer | 09:55 |
| mborzecki | did we run out of some quota on gce? | 09:56 |
| zyga | no, probably my fault | 09:56 |
| zyga | let's wait for 20 minutes to see if the new setup works | 09:56 |
| mup | PR snapd#8494 closed: tests: preserve size for centos images on spread.yaml <Simple 😃> <Created by sergiocazzolato> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8494> | 09:57 |
| zyga | mborzecki: we should figure out caching for "go test" | 09:59 |
| zyga | 7 minutes of each run is there | 09:59 |
| mup | PR snapd#8495 closed: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8495> | 09:59 |
| mborzecki | ehh more of that `Error restoring google:ubuntu-20.04-64:tests/main/parallel-install-services (apr150717-996191) : read tcp 10.113.57.84:36100->34.74.8.8:22: read: connection reset by peer` | 10:00 |
| zyga | we are mow using canonistack as well as my "cloud" | 10:02 |
| zyga | 6 workers but hopefully with way better network | 10:02 |
| zyga | we have enough memory and CPU load is low | 10:02 |
| zyga | and network is much faster actually | 10:03 |
| mborzecki | hmm https://github.com/snapcore/snapd/pull/8499 wat? | 10:20 |
| mup | PR #8499: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499> | 10:20 |
| zyga | hmmm? | 10:21 |
| zyga | mborzecki: maybe he has a github repo | 10:22 |
| zyga | we can send a PR with a response | 10:22 |
| zyga | / thank you for this comment | 10:22 |
| mborzecki | it's probably some weird snap that produces lots of data in $SNAP_DATA or somesuch | 10:23 |
| zyga | mborzecki: lol | 10:23 |
| zyga | he made a typo | 10:23 |
| zyga | so we didn't waste money "testing" that PR | 10:23 |
| zyga | mvo: I guess that's one for you to handle | 10:23 |
| zyga | mvo: https://github.com/snapcore/snapd/pull/8499 | 10:24 |
| mup | PR #8499: Adding comment on system instability caused by a privileged cp <Created by TZubiri> <https://github.com/snapcore/snapd/pull/8499> | 10:24 |
| * mvo is in a meeting | 10:25 | |
| zyga | brb | 10:36 |
| mborzecki | damn, error: cannot download snap "pi-kernel": no snap revision available as specified | 10:57 |
| mborzecki | keeps repating | 10:57 |
| === pedronis_ is now known as pedronis | ||
| mborzecki | weird, why it's using 20-pi3/edge while xnox's xnox-core-20-pi-arm64.model is using 20/edge | 11:00 |
| mvo | mborzecki: yeah, I think the kernel team did something to "pi-kernel" | 11:00 |
| pedronis | I think 20-pi3 doesn't exist anymore | 11:00 |
| mborzecki | xnox: do you know whether we should be using 20/edge for pi kernels now? | 11:00 |
| pedronis | also why are we 20 there ? | 11:00 |
| pedronis | we using 20 there? | 11:00 |
| mvo | oh, funny "pi-kernel" has no latest track anymore | 11:01 |
| mvo | so snap info pi-kernel gives no reply | 11:01 |
| mborzecki | pedronis: this is what we're using https://github.com/snapcore/snapd/blob/master/tests/lib/assertions/developer1-pi-uc20.model.json | 11:01 |
| pedronis | I think that's broken now | 11:02 |
| mborzecki | ok, the official (?) model ubuntu-core-20-pi-arm64.model is using 20/stable, so 20/edge is ok right? | 11:03 |
| pedronis | mborzecki: yes | 11:03 |
| pedronis | I think so | 11:03 |
| xnox | mborzecki: please stop using that, and instead use canonical signed dangerous model that is now available from models repo & from assertions service. | 11:06 |
| mup | PR snapd#8501 opened: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8501> | 11:06 |
| xnox | 20/edge is the way to go | 11:06 |
| mborzecki | xnox: thanks! | 11:07 |
| pedronis | that's our own prepare-image tests | 11:07 |
| ogra | xnox, a post on the forum would be really nice (where to find models etc) ... though probably post-release | 11:09 |
| ogra | so our customers can start rolling their own core20 imgs | 11:09 |
| xnox | ogra: no =) nobody shall find out anything about core20 ever. | 11:10 |
| ogra | haha | 11:10 |
| * xnox giggles | 11:10 | |
| pedronis | mvo: for latest you need pi2-kernel I think, pi-kernel is 18/20 only afaict | 11:10 |
| zyga | cachio: hey | 11:16 |
| zyga | cachio: around? :) | 11:16 |
| mup | PR snapd#8502 opened: github: try caching test results <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8502> | 11:38 |
| mborzecki | zyga: ^^ | 11:39 |
| zyga | yeah, noticed :) | 11:39 |
| mborzecki | zyga: poor man's test result cache | 11:40 |
| === pedronis_ is now known as pedronis | ||
| cmatsuoka | zyga: hey | 11:57 |
| zyga | mmm | 11:57 |
| zyga | what's up :) ? | 11:58 |
| cmatsuoka | zyga: are you aware of snap-confine-undesired-mode-group errors in fedora? or are these fixed already in master? | 11:58 |
| zyga | cmatsuoka: can you show me the log please | 11:58 |
| cmatsuoka | sure, just a sec | 11:58 |
| zyga | cmatsuoka: there's one thing in master that is fixing the only known, to me, failure of this test | 11:59 |
| cmatsuoka | https://github.com/snapcore/snapd/pull/8495/checks?check_run_id=587546789 | 11:59 |
| mup | PR #8495: cmd/snap-bootstrap: switch to a 64-byte key for unlocking <UC20> <Created by chrisccoulson> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8495> | 11:59 |
| zyga | yes | 11:59 |
| zyga | this is fixed in master | 11:59 |
| cmatsuoka | excellent, thanks! | 11:59 |
| pedronis | cmatsuoka: hi, I worked a bit on #8463, it should be ready to land I think, but it's very red (for various reasons) | 12:09 |
| mup | PR #8463: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463> | 12:09 |
| cmatsuoka | pedronis: I just checked it, thanks for the fixes/improvements | 12:10 |
| pedronis | cmatsuoka: Chris asked you to review: https://github.com/snapcore/secboot/pull/46 | 12:12 |
| mup | PR secboot#46: Make SealKeyToTPM enforce a key size of 64-bytes <Created by chrisccoulson> <https://github.com/snapcore/secboot/pull/46> | 12:12 |
| cmatsuoka | ack | 12:12 |
| mup | PR snapcraft#3039 closed: build providers: setup initial apt source configuration <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3039> | 12:40 |
| mborzecki | mvo: can you use your superpowers and merge https://github.com/snapcore/snapd/pull/8501 ? | 13:07 |
| mup | PR #8501: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <⚠ Critical> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8501> | 13:07 |
| mvo | mborzecki: sure | 13:23 |
| mborzecki | mvo: thanks! | 13:23 |
| mup | PR snapd#8501 closed: tests/lib/assertions/developer1-pi-uc20.model: use 20/edge for kernel and gadget <Test Robustness> <UC20> <⚠ Critical> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8501> | 13:24 |
| mvo | mborzecki: does 8476 look ok otherwise (beside this one point you made there) | 13:27 |
| zyga | ok, I can now turn off my home server | 13:36 |
| zyga | we have 38 workers in two locations | 13:37 |
| roadmr | zyga: maybe you can lease some DC capacity from Stéphane :) | 13:38 |
| zyga | roadmr: that's an interesting idea | 13:38 |
| zyga | mvo: ^ we probably could :) | 13:38 |
| mup | PR snapd#8476 closed: secboot: add tpm support helpers <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8476> | 13:39 |
| mvo | zyga: heh, interessting | 13:41 |
| * zyga breaks for lunch | 13:53 | |
| mup | PR snapd#8481 closed: cmd/snap-update-ns: handle EBUSY when unlinking files <Bug> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8481> | 13:59 |
| mup | PR snapd#8485 closed: cmd/snap/debug/boot-vars: add opts for setting dir and/or uc20 vars <Simple 😃> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8485> | 13:59 |
| zyga | thank you mvo! | 14:13 |
| pedronis | cmatsuoka: not really urgent but we should probably merge secboot_tpm.go and tpm.go at some point, they are not big, it's not very clear what each does. the other option is to rename secboot_tpm.go to something else | 14:16 |
| mup | PR snapd#8500 closed: httputil: fix client timeout retry tests <Simple 😃> <Test Robustness> <Created by bboozzoo> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8500> | 14:21 |
| ijohnson | zyga: so is robust mount namespace updates already enabled by default on master ? | 14:22 |
| cmatsuoka | pedronis: agree, I considered that myself when moving tpm.go to secboot but decided to propose that later to avoid changing too much in a single step | 14:23 |
| zyga | ijohnson: yes it is | 14:28 |
| ijohnson | nice | 14:28 |
| zyga | ijohnson: 2.45 will have it | 14:28 |
| * zyga is partially away, prepping food | 14:28 | |
| pedronis | ijohnson: cmatsuoka: I did some digging about the SecureBoot variable | 14:33 |
| cmatsuoka | pedronis: ah interesting, any new finding? | 14:34 |
| pedronis | cmatsuoka: see the PR comments | 14:34 |
| * cmatsuoka checks | 14:34 | |
| zyga | hmmmm | 15:16 |
| zyga | did canonistack just die? | 15:16 |
| roadmr | deadstack | 15:17 |
| zyga | hmm | 15:19 |
| zyga | not sure what happened | 15:20 |
| mup | PR snapd#8453 closed: [RFC] travis.yml: re-enable travis <Created by mvo5> <Closed by mvo5> <https://github.com/snapcore/snapd/pull/8453> | 15:20 |
| ijohnson | pedronis: #8497 is ready again | 15:22 |
| mup | PR #8497: boot/bootstate20: re-factor kernel methods to use new interface for state <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8497> | 15:22 |
| pedronis | ijohnson: I was in meetings, looking now | 15:25 |
| ijohnson | thanks | 15:29 |
| pedronis | ijohnson: commented | 15:33 |
| ijohnson | pedronis: do you want a separate PR/rebase to extract just the change for https://github.com/snapcore/snapd/pull/8497#discussion_r408941648 ? | 15:40 |
| mup | PR #8497: boot/bootstate20: re-factor kernel methods to use new interface for state <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8497> | 15:40 |
| * zyga settles for quiet work | 15:41 | |
| pedronis | ijohnson: maybe yes, I would expect a test change for it in some form | 15:41 |
| ijohnson | pedronis: ok, fwiw that was changed in the PR that's open | 15:42 |
| ijohnson | that will be easy to extract out though | 15:42 |
| mup | PR snapd#8503 opened: boot/bootstate20: fix bug in try-kernel cleanup <Bug> <Simple 😃> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8503> | 15:46 |
| ijohnson | pedronis: broken out as ^ | 15:46 |
| pstolowski | mvo: i've pushed my early config test to https://github.com/stolowski/snapd/tree/core18-early-config | 15:47 |
| jdstrand | sil2100 (cc mvo): hey, the core20 snap is failing with https://paste.ubuntu.com/p/DrhfhjWG9y/. can you confirm this is ok or not? | 15:47 |
| pstolowski | mvo: to run it: google-nested:ubuntu-18.04-64:tests/nested/manual/core-early-config | 15:47 |
| cmatsuoka | cachio: unfortunately we still need sb enabled to be able to run the full encryption test | 15:48 |
| cmatsuoka | cachio: error: cannot seal the encryption key: cannot store encryption key: cannot add EFI secure boot policy profile: cannot compute secure boot policy digests: the current boot was performe | 15:48 |
| cmatsuoka | d with secure boot disabled in firmware | 15:48 |
| jdstrand | sil2100 (mvo): I'm going to approve it to unblock the queue, but please look into it | 15:51 |
| mvo | pstolowski: thanks, in a meeting right now, will have a look once that is over | 15:53 |
| cachio | cmatsuoka, is that on classic? | 15:54 |
| pedronis | ijohnson: thanks, it wasn't clear that the calls number where not a consquences of some of the refactorings | 15:54 |
| mvo | jdstrand: thanks, problably https://paste.ubuntu.com/p/DrhfhjWG9y/ is something for xnox to review | 15:54 |
| ijohnson | pedronis: yes the refactor actually exposed the bug :-) | 15:54 |
| cmatsuoka | cachio: mm, yes, the test is supposed to run on classic | 15:55 |
| cachio | cmatsuoka, ok, I'll take a look | 15:55 |
| xnox | mvo: can you explain that output? | 15:56 |
| cmatsuoka | cachio: thanks, I'll run a few more tests here and will be back after lunch | 15:56 |
| cachio | cmatsuoka, just first let me finish the tests changes I did | 15:56 |
| cachio | could you give me a link to your tests? | 15:56 |
| xnox | mvo: i.e. these files were removed from .deb's in focal, and hence new revisions of core20 snap don't have them...... | 15:56 |
| mvo | xnox: jdstrand runs a tool that compares the files from one base to the next base and when stuff is removed it alerts and blocks the snap | 15:57 |
| ijohnson | mborzecki: around? | 15:57 |
| xnox | mvo: why are you asserting contents of .deb's that are part of core20 snap? | 15:57 |
| cmatsuoka | cachio: yes, of course. I'm running the test with PR #8409 | 15:57 |
| mvo | xnox: I don't, the store-review-tools do | 15:57 |
| mup | PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409> | 15:57 |
| mvo | xnox: it's there to ensure that bases are stable and drop support | 15:57 |
| xnox | jdstrand: those files are dropped in focal, despite equivalent functionality still present. | 15:58 |
| xnox | jdstrand: you should see new added files, like subiquitycore/palette.py | 15:58 |
| xnox | jdstrand: and that systemd-timesyncd is now a standalone package, which is included. | 15:58 |
| jdstrand | xnox: ack, that's fine. thanks! | 15:59 |
| jdstrand | xnox: just wanted to make sure it was intentional | 15:59 |
| xnox | jdstrand: it was =) | 15:59 |
| xnox | jdstrand: if you have questions like these in the future, core20 bug tracker is on github snapcore/core20 issues | 15:59 |
| xnox | jdstrand: or foundations-crew@lists.canonical.com as foundations owns/builds core20 | 16:00 |
| cachio | cmatsuoka, nice, I'll take a look after lunch as well | 16:00 |
| jdstrand | xnox: right, that is why I pinged sil2100, but I can change the workflow | 16:00 |
| xnox | jdstrand: he mostly deals with "stable" like core18 / core16 | 16:01 |
| xnox | jdstrand: core20 will be dead to me next thursday! and onto bootstrapping core22 | 16:01 |
| jdstrand | xnox: heh, so should foundations-crew@lists.canonical.com be used for core16 and core18? | 16:01 |
| sil2100 | I like how xnox said "stable" ;) | 16:01 |
| jdstrand | mvo: ok, so core20 revisions are now passing automated reviews again and should be caught up in a few minutes | 16:02 |
| xnox | jdstrand: yes you can. foundations-crew@ is one stop shop for core snaps, subiquity, netplan, etc. | 16:02 |
| jdstrand | ok, thanks | 16:03 |
| * cachio lunch | 16:03 | |
| jdstrand | xnox: that list is internal only, right? | 16:04 |
| xnox | jdstrand: yes. | 16:05 |
| mvo | cmatsuoka: looks like 8463 is ready to land but one core20 test is failing, i.e. google:ubuntu-core-20-64:tests/core/basic20, could you please disable this part of the test and/or move to nested (if disable, TODO:UC20: is probably fine). then this can land | 16:34 |
| mvo | cmatsuoka: nevermind | 16:44 |
| mvo | cmatsuoka: pushed it | 16:45 |
| cachio | cmatsuoka, I don't see any nested test on #8409 | 16:56 |
| mup | PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409> | 16:57 |
| cachio | cmatsuoka, is that the PR? | 16:57 |
| cmatsuoka | cachio: yes, the test is currently in tests/main/uc20-create-partitions-encrypted, it will be moved to nested | 17:02 |
| cachio | cmatsuoka, ah | 17:04 |
| cachio | ok | 17:04 |
| pedronis | this failure is weird: https://pipelines.actions.githubusercontent.com/xS8oSnypZkPEQZqiZgDaRp2kdvQJKbOY08TesHp7E8vn7g4hYR/_apis/pipelines/1/runs/1163/signedlogcontent/31?urlExpires=2020-04-15T17%3A04%3A50.7188367Z&urlSigningMethod=HMACV1&urlSignature=n1FukofiSafDmCcRwGnoncpLJS8mfgpiWZRJABATpvc%3D | 17:06 |
| ijohnson | pedronis: that url seems to be expired | 17:07 |
| ijohnson | which PR was that from? | 17:07 |
| pedronis | ijohnson: your PR | 17:07 |
| ijohnson | I noticed an odd lxd error from 8503, but I saved the logs and restarted the check | 17:08 |
| ijohnson | ah okay so same thing I saw | 17:08 |
| ijohnson | I'm looking into the issue now | 17:08 |
| pedronis | yes, weird failure in a lxd regression test | 17:08 |
| pedronis | ijohnson: it failed also in your other pr | 17:10 |
| pedronis | so maybe it's real | 17:10 |
| ijohnson | that's silly | 17:10 |
| * ijohnson will look more into it after lunch | 17:10 | |
| === pedronis_ is now known as pedronis | ||
| zyga | what was the failure? | 17:17 |
| zyga | (in the lxd regression test) | 17:17 |
| cmatsuoka | pedronis: what would be a good place to define the kernel command line used in tpm sealing? | 17:40 |
| pedronis | ijohnson: that test is failing for master too | 17:46 |
| zyga | pedronis: do you have logs? | 17:46 |
| zyga | which test is it? | 17:46 |
| pedronis | something thinks we should have snapd around | 17:48 |
| pedronis | but actually we have core | 17:48 |
| pedronis | # lxc file push /usr/bin/snap bionic/usr/bin/snap | 17:48 |
| pedronis | Error: open /snap/snapd/current/usr/bin/snap: no such file or directory | 17:48 |
| zyga | ah | 17:48 |
| zyga | I think I know that this is | 17:48 |
| zyga | one moment | 17:48 |
| zyga | I think it's a race, I've seen things like that today | 17:49 |
| zyga | lxc launch ... | 17:49 |
| pedronis | it's google:ubuntu-18.04-64:tests/regression/lp-1871652 | 17:49 |
| zyga | lxc file push ... | 17:49 |
| zyga | that fails | 17:49 |
| pedronis | it fails on master, running alone | 17:49 |
| zyga | I added lxc exec -- ls -l ... | 17:49 |
| zyga | that makes the container "ready" somehow | 17:49 |
| zyga | pedronis: thanks, I'll run it now | 17:49 |
| pedronis | cmatsuoka: the bootloaders should know about this, but for now you can hard code it in devicestate if it's easier | 17:52 |
| cmatsuoka | pedronis: ok, thanks | 17:53 |
| pedronis | cmatsuoka: actually is not devicestate, right? this goes into snap-boostrap? | 17:53 |
| pedronis | anyway you can hard code it in snap-boostrap for now | 17:54 |
| cmatsuoka | mm, yes, it's in snap-bootstrap | 17:54 |
| * ijohnson is back | 17:59 | |
| ijohnson | zyga: did you reproduce/fix the issue? | 18:00 |
| zyga | ijohnson: yes, not yet | 18:13 |
| ijohnson | ah finally my spread run just got there I've got it now too | 18:14 |
| zyga | so | 18:14 |
| zyga | this is weird | 18:14 |
| zyga | there are two systems | 18:14 |
| zyga | the host and the container | 18:14 |
| zyga | neither has snapd snap | 18:15 |
| zyga | yet | 18:15 |
| zyga | lxc file push says | 18:15 |
| zyga | Error: open /snap/snapd/current/usr/bin/snap: no such file or directory | 18:15 |
| zyga | this is lxd 4.0.0 | 18:15 |
| zyga | maybe lxd bug? | 18:15 |
| stgraber | you must have SNAP or SNAP_NAME in the environment | 18:15 |
| ijohnson | stgraber: we don't | 18:16 |
| zyga | indeed | 18:16 |
| ijohnson | https://www.irccloud.com/pastebin/dtcv3kfp/ | 18:16 |
| zyga | and this passed recently as it's a fix for the bug that was plaguing lxd (from snapd side) | 18:16 |
| stgraber | what's the "lxc file push" you're running? | 18:16 |
| zyga | lxc file push /usr/bin/snap bionic/usr/bin/snap | 18:16 |
| zyga | I'm copying the snap command from the host (deb) to the guest | 18:16 |
| zyga | since the host has locally built snapd | 18:17 |
| stgraber | hmm, odd | 18:17 |
| ijohnson | stgraber: if I switch to 3.23/stable channel the same command works | 18:18 |
| ijohnson | and actually switching back to 4.0/stable channel again the same command fails again | 18:18 |
| ijohnson | looks like a lxd regression to me | 18:18 |
| stgraber | hmm, we didn't touch any of the file code between 3.23 and 4.0 | 18:18 |
| zyga | yeah, I just tried that too | 18:18 |
| zyga | stgraber: did you release 4.0 recently? | 18:19 |
| stgraber | we did just switch the snap over to core18 though | 18:19 |
| ijohnson | zyga: did you see the same behavior? | 18:19 |
| zyga | yes | 18:19 |
| zyga | works perfectly on 3.23 | 18:19 |
| zyga | ijohnson: shall we update LXD channel and call it a day? | 18:20 |
| ijohnson | zyga: yeah I think that's fine for now | 18:20 |
| ijohnson | I'll file a PR | 18:20 |
| zyga | thank you! | 18:20 |
| zyga | may I suggest something? | 18:20 |
| zyga | ijohnson: adjust the test to respect a variable | 18:20 |
| stgraber | testing with current 4.0 with same file push here to see if it reproduces easily | 18:20 |
| zyga | and update the variable in spread.yaml | 18:20 |
| zyga | I think I missed that we have this in the first place when I wrote the test | 18:20 |
| ijohnson | zyga: that's a great idea sure | 18:21 |
| zyga | thanks! | 18:21 |
| stgraber | root@sateda:~# nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -lh /usr/bin/snap | 18:21 |
| stgraber | lrwxrwxrwx 1 root root 32 Mar 11 05:46 /usr/bin/snap -> /snap/snapd/current/usr/bin/snap | 18:22 |
| stgraber | that would be why ^ | 18:22 |
| zyga | !!! | 18:22 |
| zyga | that's weird | 18:22 |
| zyga | how did we get that symlink? | 18:22 |
| stgraber | presumably this wasn't that way on core16 so that wasn't a problem | 18:22 |
| zyga | is that a core18 regression? | 18:22 |
| zyga | is 3.23 using a different base? | 18:22 |
| stgraber | I mean, it's still a bug, it shouldn't actually pull the thing from inside the snap, it needs to pull it from the host | 18:22 |
| stgraber | yeah, we switched based from core to core18 a few hours ago | 18:22 |
| zyga | indeed | 18:22 |
| zyga | this makes sense | 18:23 |
| stgraber | root@sateda:~# nsenter --mount=/run/snapd/ns/lxd.mnt -- ls -lh /var/lib/snapd/hostfs/usr/bin/snap | 18:23 |
| stgraber | -rwxr-xr-x 1 root root 14M Oct 30 12:17 /var/lib/snapd/hostfs/usr/bin/snap | 18:23 |
| zyga | cool, do you need a bug report for tracking? | 18:23 |
| stgraber | that's still fine and it should be using that | 18:23 |
| stgraber | you can if you want, but I'll probably have it fixed in 5min or so | 18:23 |
| zyga | ok, no need then | 18:23 |
| stgraber | INFO[04-15|18:27:30] Pushing /var/lib/snapd/hostfs/usr/bin/snap to /usr/bin/snap (file) | 18:27 |
| stgraber | that's more like it | 18:27 |
| stgraber | I absolutely hate that logic... | 18:27 |
| stgraber | such a mess to support absolute and relative paths, with and without symlinks in the middle, ... | 18:28 |
| stgraber | zyga, ijohnson: https://github.com/lxc/lxd/pull/7198 | 18:29 |
| mup | PR lxc/lxd#7198: shared/util: Never look into the snap <Created by stgraber> <https://github.com/lxc/lxd/pull/7198> | 18:29 |
| stgraber | will be in the stable snap in the next 4-5 hours I'd say | 18:29 |
| ijohnson | cool thanks stgraber | 18:29 |
| zyga | stgraber: you like to live dangerously ;-) | 18:29 |
| zyga | but I admire that | 18:29 |
| stgraber | I've been doing daily stable cherry-picks lately | 18:29 |
| zyga | ah | 18:29 |
| zyga | I see | 18:29 |
| zyga | that's interest | 18:29 |
| zyga | *interesting | 18:29 |
| zyga | perhaps something we could do? | 18:29 |
| ijohnson | ahhhh we would have many folks very mad if we did as many stable releases as often as lxd :-) | 18:30 |
| stgraber | well, we have pretty reasonable testing on all distros and architecture, that's why it takes 4-5 hours before we're pretty confident it's not horribly broken :) | 18:30 |
| zyga | stgraber: having a regression test for push would be great tho :D | 18:30 |
| stgraber | yeah, not very practical to test in CI since we don't use any snaps there, but I'll sneak some file push testing into our snap tests | 18:31 |
| stgraber | test added | 18:33 |
| zyga | thank you :) | 18:33 |
| ijohnson | zyga: fancy a +1 before you EOD? https://github.com/snapcore/snapd/pull/8504 | 18:36 |
| mup | PR #8504: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8504> | 18:36 |
| ijohnson | :-) | 18:36 |
| zyga | looking | 18:36 |
| zyga | I'm not EODing yet | 18:36 |
| zyga | deeeeply in dbus | 18:36 |
| ijohnson | one might say you are dbusly in dbus | 18:36 |
| * ijohnson stops | 18:36 | |
| mup | PR snapd#8504 opened: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8504> | 18:36 |
| zyga | hahaha | 18:36 |
| zyga | +1 | 18:37 |
| zyga | there's something magic about the words | 18:38 |
| zyga | "in progress" | 18:38 |
| zyga | that's so much better than "queued" we had before | 18:38 |
| mup | PR snapd#8505 opened: spread.yaml: switch back to latest/candidate for lxd snap <Test Robustness> <⛔ Blocked> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8505> | 18:44 |
| zyga | ijohnson: ^ nice | 18:46 |
| ijohnson | in progress does sound much better | 18:46 |
| ijohnson | I think in general whenever we need to do things like this, i.e. revert something to unbreak master while something is fixed, we should always open a PR undoing that change at the same time so at least we remember because there's an open PR about changing | 18:47 |
| zyga | yes | 18:47 |
| ijohnson | i.e. same thing when we move a system to unstable, we should open another PR which moves it back to stable | 18:47 |
| ijohnson | pedronis: we fixed the lxd issue on master | 18:48 |
| ijohnson | well we have a PR open | 18:48 |
| mup | PR snapd#8463 closed: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8463> | 18:48 |
| pedronis | ijohnson: I see | 18:49 |
| pedronis | ijohnson: anyway proves why I'm not a fan of dangling symlinks in bases | 18:50 |
| ijohnson | pedronis: yes but also why we should be very strict about letting snaps access things in bases too | 18:51 |
| pedronis | yes | 18:52 |
| ijohnson | pedronis: also did you discuss with maciej about picking up the necessary changes to gadget pkg et al to enable mbr support for create-partitions? should I try to push that forward? I was kinda hoping that he would pick that up but I keep forgetting to ask him before he EODs | 18:52 |
| pedronis | ijohnson: I don't know, it's actually a prereq to make the run mode changes relevant, right? because it's install time? | 18:53 |
| ijohnson | pedronis: yes | 18:53 |
| ijohnson | yes it's a blocker I mean | 18:54 |
| pedronis | ijohnson: have you started the other bootloaderKernelState ? | 18:54 |
| ijohnson | I'm wrapping it up now, but I need to figure out a way to write tests for it, I'd rather not rewrite the dozens of tests we already have for bootstate20 to use a mock uboot bootloader, etc. | 18:55 |
| pedronis | ijohnson: maybe just write direct tests for it? | 18:56 |
| pedronis | we probably need to reorg some of the other tests as table tests so that they are easier to reuse | 18:56 |
| ijohnson | pedronis: yes perhaps, need to think about it a little bit | 18:56 |
| pedronis | or have a actual test mixins | 18:56 |
| pedronis | we don't have a lot of the latter but we don sometimes | 18:57 |
| ijohnson | yeah we could refactor some of the tests big chunks to use helpers too | 18:58 |
| ijohnson | lots of the tests are just bootloader setup anyways | 18:58 |
| pedronis | cmatsuoka: is #8409 ready for Chris to look at it? | 18:59 |
| mup | PR #8409: snap-bootstrap: seal and unseal encryption key using tpm <UC20> <⛔ Blocked> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8409> | 18:59 |
| cmatsuoka | pedronis: will be in a minute, pushing a final commit | 19:00 |
| cmatsuoka | pedronis: pushed | 19:01 |
| mup | Issue core20#36 closed: SSH login shell says the system was minimized and to run "unminimize", but should not <Created by anonymouse64> <Closed by xnox> <https://github.com/snapcore/core20/issue/36> | 19:23 |
| mup | PR core20#42 closed: drop `unminimize` instructions that are not applicable on Core <Created by xnox> <Merged by xnox> <https://github.com/snapcore/core20/pull/42> | 19:23 |
| mup | PR snapd#8284 closed: config: add system.store-certs.[a-zA-Z0-9] support <Needs Samuele review> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8284> | 19:52 |
| mup | PR snapd#8503 closed: boot/bootstate20: fix bug in try-kernel cleanup <Bug> <Simple 😃> <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8503> | 19:58 |
| mup | PR snapd#8504 closed: spread.yaml,tests/many: use global env var for lxd channel <Test Robustness> <⚠ Critical> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8504> | 20:05 |
| mup | PR snapd#8506 opened: Add libnvidia-opticalflow as Nvidia library <Created by joedborg> <https://github.com/snapcore/snapd/pull/8506> | 20:37 |
| joedborg | hey everyone, if someone could take a quick look at ^ it'd be much appreciated | 20:37 |
| mwhudson | what determines what ends up in parts/$PART/install/usr/lib/python3/dist-packages using the python plugin? | 21:36 |
| mup | PR snapcraft#3043 opened: package-repositories: initial schema and meta read/write support <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3043> | 22:17 |
| mup | PR snapcraft#3044 opened: build providers: use ubuntu-ports mirrors for non-x86 platforms <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3044> | 22:45 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!