[05:50] <mborzecki> morning
[06:20] <zyga> o/
[06:22] <mborzecki> zyga: hey
[06:22] <mborzecki> zyga: some trouble with the cla-check job
[06:22] <zyga> uc20-snap-recovery failed
[06:22] <mborzecki> zyga: where?
[06:22] <zyga> but it ran on 19.10
[06:22] <zyga> https://github.com/snapcore/snapd/pull/8440/checks?check_run_id=569193826
[06:22] <mup> PR #8440: github: move spread to self-hosted workers <Created by zyga> <https://github.com/snapcore/snapd/pull/8440>
[06:23] <mborzecki> zyga: uh, merge master
[06:23] <zyga> is that even expected?
[06:23] <zyga> known issue?
[06:23] <mborzecki> zyga: yes, it's fixed already
[06:24] <zyga> k
[06:25] <zyga> how did cla check fail?
[06:25] <zyga> it passed on my branch just now
[06:25] <zyga> 38seconds
[06:26] <zyga> meanwhile, travis is broken
[06:26] <zyga> https://t.co/h3UEAleWVW?amp=1
[06:27] <zyga> I think I can just go back to bed
[06:27] <mborzecki> zyga: if you open a PR with a commit right on top of the master so that no merge commit is generated it will fail
[06:27] <zyga> I see
[06:57] <mup> PR snapd#8439 closed: secboot: import secboot on ubuntu, provide dummy on !ubuntu <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8439>
[07:01] <pstolowski> morning
[07:01] <mvo> good morning pstolowski
[07:02] <mvo> zyga: quick question, do we have a 32bit machine in travis actions?
[07:03] <zyga> mvo: travis actions?
[07:03] <mvo> zyga: sorry, gh actions
[07:03] <zyga> mvo: as I said yesterday I didn't add a 32bit xenial machine to github actions
[07:03] <zyga> mvo: though it's a one-liner in the matrix, it slipped through the cracks in the initial PRs
[07:04] <zyga> good morning :)
[07:04] <zyga> last night store went belly up
[07:04] <zyga> and everything running failed one way or another
[07:04] <zyga> so I just called it quits and went to sleep (too late anyway)
[07:05] <mborzecki> mvo: pstolowski: hey
[07:05] <mup> PR snapd#8455 opened: tests/lib/cla_check: expect explicit commit range <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8455>
[07:05] <mborzecki> zyga: can we skip the spread jobs?
[07:05] <zyga> mborzecki: in principle yes but it's not something we coded, we should try that if: ... expression I pasted before
[07:05] <zyga> one sec
[07:05] <zyga> maybe add that to your PR
[07:06] <mborzecki> contains(github.event.issue.labels.*.name, 'skip-spread') or somesuch?
[07:06] <zyga> yes
[07:06] <zyga> if: !contains ...
[07:06] <mborzecki> idk tho, just copied and pasted from the docs :P
[07:06] <zyga> :)
[07:06] <zyga> I tried to get https://github.com/snapcore/snapd/pull/8440 green
[07:06] <mup> PR #8440: github: move spread to self-hosted workers <Created by zyga> <https://github.com/snapcore/snapd/pull/8440>
[07:07] <zyga> but each time something random failed
[07:07] <mup> PR snapd#8456 opened: tests: add 32 bit machine to GH actions <Simple 😃> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8456>
[07:07] <zyga> some desktop service, some store bits, some reboot tests
[07:07] <zyga> so tough luck
[07:18] <zyga> mvo: could you please merge https://github.com/snapcore/snapd/pull/8454
[07:18] <mup> PR #8454: tests/session-tool: session ordering is non-deterministic <Created by zyga> <https://github.com/snapcore/snapd/pull/8454>
[07:22] <mborzecki> zyga: hm the docs are kinda meh
[07:23] <mup> PR snapd#8457 opened: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[07:32] <zyga> mborzecki: interesting, except that the status check is required
[07:32] <zyga> mborzecki: perhaps instead wrap that in ${{  }}
[07:32] <zyga> and have the worker essentially do nothing?
[07:33] <zyga> mborzecki: ${{ .. }} is required in run blocks
[07:33] <mborzecki> zyga: hm which pr?
[07:33] <zyga> your pr
[07:33] <mborzecki> there's 2 ;)
[07:33] <zyga> 8457
[07:33] <zyga> and there's a syntax error
[07:34] <zyga> I would drop the first part
[07:34] <zyga> as all events are pull reqeusts
[07:34] <zyga> let me pull the docs
[07:35] <zyga> if: contains(github.event.pull_request.labels.*.name, 'Skip spread')
[07:35] <zyga> then just negate
[07:35] <zyga> if: !contains(github.event.pull_request.labels.*.name, 'Skip spread')
[07:35] <zyga> but as we learned, that should not go into if because then the status check wont report
[07:35] <zyga> so maybe:
[07:36] <zyga> run: | echo ${{ !contains(...) }}
[07:36] <zyga> and see what that prints (probably true as that is just js)
[07:36] <mborzecki> heh
[07:36] <zyga> then wrap that into a shell
[07:36] <zyga> and should be good
[07:36] <mborzecki> i mean, wtf are the docs about labels?
[07:36] <zyga> they are there
[07:36] <zyga> hold on
[07:36] <zyga> it's somewhat confusing because they are not in the action docs
[07:36] <zyga> but in the bigger github docs
[07:36] <zyga> the whole object model is documented
[07:37] <zyga> https://developer.github.com/v3/issues/labels/
[07:37] <zyga> by doing ${{ ... }} you're effectively tapping into that
[07:38] <mborzecki> zyga: the pull request event is this: https://developer.github.com/v3/activity/events/types/#pullrequestevent  doesn't list the label there but it's in the example
[07:38] <mborzecki> and it's an empty array
[07:39] <mborzecki> however, there's actually an example in the issues event payload
[07:39] <zyga> https://developer.github.com/v3/pulls/ has the labels listed
[07:40] <mvo> zyga: re 8454 sure, I will merge once the spread tests finished, they are still running
[07:40] <zyga> thanks
[07:41] <zyga> one test already failed
[07:41] <zyga> on portal info
[07:41] <mvo> zyga: oh, ok. is james aware of the flakiness here?
[07:41] <zyga> I don't know
[07:41] <zyga> it's in spread-unstable so perhaps nobody noticed?
[07:42] <zyga> jamesh: can you please check if this is expected
[07:42] <mvo> aha, could be
[07:42] <zyga> https://github.com/snapcore/snapd/pull/8454/checks?check_run_id=569207445#step:4:814
[07:42] <mup> PR #8454: tests/session-tool: session ordering is non-deterministic <Created by zyga> <https://github.com/snapcore/snapd/pull/8454>
[07:43] <zyga> fedora failed to prepare, network error
[07:44] <mborzecki> zyga: idk, i think that the labels is not actually included there
[07:44] <zyga> mborzecki: where specifically?
[07:44] <mborzecki> zyga: is the pull_request object is the same as pull_request in https://developer.github.com/v3/activity/events/types/#pullrequestevent then the label is not htere
[07:44] <mborzecki> but should be?
[07:44] <mborzecki> idk
[07:44] <zyga> pull request *event*
[07:44] <jamesh> zyga: It isn't expected.  If you're seeing this error, then it can't map the process ID to a snap via cgroups
[07:44] <zyga> refers to pull request
[07:45] <zyga> that has labels
[07:45] <zyga> jamesh: fun, I guess it is debug time then
[07:56] <zyga> mvo: https://github.com/snapcore/snapd/pull/8456/files
[07:56] <zyga> is the vendor change expected?
[07:56] <mup> PR #8456: tests: add 32 bit machine to GH actions <Created by mvo5> <https://github.com/snapcore/snapd/pull/8456>
[07:56] <zyga> mvo: https://github.com/snapcore/snapd/pull/8440 is green
[07:56] <mup> PR #8440: github: move spread to self-hosted workers <Created by zyga> <https://github.com/snapcore/snapd/pull/8440>
[07:56] <zyga> but let's chat about that in the call
[07:56] <mborzecki> zyga: don't think that check works https://github.com/snapcore/snapd/pull/8457 looks like the spread jobs are still schedule
[07:56] <mup> PR #8457: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[07:56] <mborzecki> d
[07:57] <zyga> mborzecki: how do you determine that?
[07:57] <zyga> mborzecki: they are required, so they are marked as expected
[07:57] <zyga> mborzecki: note that normally you don't get any jobs until the previous pass is successful
[07:58] <zyga> so I don't believe this is accurate as measurement
[07:58] <mborzecki> ok, let's wait then
[07:58] <mvo> zyga: yeah
[07:59] <zyga> ah
[07:59] <zyga> I see the 2nd commit now
[07:59] <zyga> cool
[07:59] <zyga> thanks
[08:11] <mup> PR snapd#8440 closed: github: move spread to self-hosted workers <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8440>
[08:15] <jamesh> mborzecki: one option would be to move the if: clause down to the step level
[08:16] <mborzecki> zyga: have you seen the 'cancel workflow' request to have any effect?
[08:16] <mborzecki> jamesh: supposedly job level `if` is supported now https://github.blog/changelog/2019-10-01-github-actions-new-workflow-syntax-features/
[08:16] <jamesh> mborzecki: it's not quite as efficient since a job would still be sent to a runner, but it would mean the job would be considered successful
[08:17] <mborzecki> unless it isn't :/ idk, maybe i just need to wait
[08:17] <jamesh> mborzecki: yes, but if the conditional causes the job not to run, then it isn't considered successful
[08:18] <jamesh> if you want to get rid of the "Some checks haven’t completed yet" message, the jobs need to at least do something
[08:25] <zyga> mvo: there's a problem with the -32 bit build
[08:25] <zyga> src/github.com/snapcore/snapd/vendor/github.com/chrisccoulson/go-tpm2/mu.go:267:17: constant 4294967295 overflows int
[08:26] <zyga> chrisccoulson: ^ FYI
[08:26] <zyga> mborzecki: IIRC cancelling works but spread doesn't cancel and the worker is killed
[08:33] <mvo> zyga: I know, I updated the PR that adds 32bit works, it should have a fix
[08:34] <zyga> maybe the hash is wrong?
[08:34] <mvo> zyga: oh, let me double check :(
[08:34] <mvo> zyga: could be that govendor confused me
[08:34] <zyga> when you push again merge master please
[08:35] <mvo> zyga: sorry, I'm an idiot, I updated go-tpm instead go-tpm2
[08:35]  * zyga hugs mvo
[08:36] <zyga> https://github.com/snapcore/snapd/pull/8403 needs a 2nd review
[08:36] <mup> PR #8403: sandbox/cgroup: avoid making arrays we don't use <Skip spread> <Created by zyga> <https://github.com/snapcore/snapd/pull/8403>
[08:37] <zyga> it failed on store traffic: - Fetch and check assertions for snap "test-snapd-content-slot-no-content-attr" (1) (error reading assertion headers: read tcp 10.240.1.50:58298->91.189.92.20:443: use of closed network connection (Client.Timeout exceeded while reading body))
[08:41] <mup> PR snapd#8458 opened: github: allow cached debian downloads to restore <Created by zyga> <https://github.com/snapcore/snapd/pull/8458>
[08:41] <zyga> jamesh: https://github.com/snapcore/snapd/pull/8458
[08:41] <mup> PR #8458: github: allow cached debian downloads to restore <Created by zyga> <https://github.com/snapcore/snapd/pull/8458>
[08:41] <zyga> this should fix the cache
[08:42] <zyga> though I think it looks only in the scope of the PR, there's still more opportunity to cache things than we exploit
[08:42] <zyga> (caches are associated with objects and are not global)
[08:44] <zyga> brb
[08:46] <jamesh> I suspect caches are probably scoped  to the (repo, user) pair
[08:53]  * zyga monitors https://github.com/snapcore/snapd/actions?query=is%3Aqueued
[09:01] <mup> PR snapd#8421 closed: tests: enable unit tests on debian-sid again <Simple 😃> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8421>
[09:03] <zyga> mvo: that seems to have fixed things
[09:03] <zyga> oh, I spoke too soon
[09:03] <zyga> mvo: src/github.com/snapcore/snapd/vendor/github.com/snapcore/secboot/utils.go:73:37: cannot call non-function he.TPMError.Code (type tpm2.ErrorCode)
[09:03] <zyga> I think this commit is not good :/
[09:04] <zyga> why didn't this get flagged by the unit test run?
[09:04] <zyga> are we not building / testing secboot?
[09:04] <zyga> ahh wait
[09:04] <zyga> that's weird
[09:04] <zyga> ah, snapcore/secboot is a different repository
[09:04] <zyga> oh well
[09:05] <zyga> (we don't seem to test anything there in CI)
[09:06] <mvo> zyga: meh
[09:06] <zyga> but at least the tests were quick now :)
[09:08] <mvo> zyga: haha, yes. but that's slightly annoying that this fails
[09:10] <mvo> zyga: one more try
[09:10] <zyga> ok
[09:10] <zyga> still 0 queued
[09:11] <zyga> (which is good)
[09:15] <zyga> mborzecki: thanks for the suggestion in https://github.com/snapcore/snapd/pull/7614
[09:15] <zyga> updated
[09:15] <mup> PR #7614: cmd/snap-confine: implement snap-device-helper internally <Created by zyga> <https://github.com/snapcore/snapd/pull/7614>
[09:16] <zyga> still 0 queued
[09:16] <zyga> mvo: I also wonder if actions are more heavily used in US, making afternoon "harder"
[09:17] <jamesh> I've always found CI runs faster before you Europeans wake up
[09:18] <zyga> mborzecki: could you look at https://github.com/snapcore/snapd/pull/7825 and tell me if you think it's work splitting
[09:18] <jamesh> I think it is more a case of two groups of users using CI at once
[09:18] <mup> PR #7825: many: use transient scope for tracking apps and hooks <Security-High> <Created by zyga> <https://github.com/snapcore/snapd/pull/7825>
[09:18] <zyga> I could take the go bits that do cgroup scanning out and push separately
[09:18] <zyga> jamesh: haha, yeah
[09:20] <mborzecki> heh, as jamesh commented, https://github.com/snapcore/snapd/pull/8457 does appear to be stuck
[09:20] <mup> PR #8457: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[09:20] <mborzecki> the unit tests job should run though, but it hasn't yet
[09:22] <mborzecki> wierd, i'll wait a little bit longer
[09:22] <jamesh> could it have rejected the workflow entirely?
[09:27] <mborzecki> idk, clearly something is off
[09:27] <zyga> one job queued
[09:28] <zyga> (all 32 spread workers are busy)
[09:28] <zyga> mborzecki: werid
[09:28] <zyga> mborzecki: can you rebase on master and push?
[09:29] <zyga> at 32 spread runs I'm seeing roughly 1MB/s in and 1MB/s out
[09:29] <zyga> that's not too terrible
[09:29] <zyga> it spikes to 10MB/s
[09:29] <zyga> especially when new jobs kick in and there's the initial sync
[09:29] <mborzecki> zyga: where do you see that?
[09:29] <zyga> spread has an inefficiency where the starting worker pushes the same tarball to each node
[09:29] <zyga> mborzecki: on the machine running spread workers
[09:30] <zyga> we could optimize that traffic down by just sending the tarball once and then fetching it from the cloud
[09:36] <pstolowski> pedronis: hi. currently FilesystemOnlyApply skips core-only handlers if release is classic; i think this needs to be relaxed for image/setupSeed with a flag passed down to FilesystemOnlyApply; makes sense?
[09:39] <pedronis> pstolowski: let me look
[09:44] <pedronis> pstolowski: yes, the cleanest thing is probably for the package not use release.OnClassic at all, and get info through some options
[09:45] <pstolowski> pedronis: k, thanks for confirming
[10:05] <zyga> core 18 revert tests failed: https://github.com/snapcore/snapd/pull/8454/checks?check_run_id=570248002
[10:05] <mup> PR #8454: tests/session-tool: session ordering is non-deterministic <Created by zyga> <https://github.com/snapcore/snapd/pull/8454>
[10:06] <zyga> + snap list
[10:06] <zyga> error: cannot list snaps: cannot communicate with server: timeout exceeded while waiting for response
[10:10] <mup> PR snapd#8454 closed: tests/session-tool: session ordering is non-deterministic <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8454>
[10:15] <popey> ogra where should bugs about ubuntu core images be filed?
[10:16] <popey> actually, probably a bug in the installer, is that subiquity on core? (the first run thing)
[10:21]  * popey starts a forum thread.
[10:26] <zyga> mborzecki: TBH I really wish there were type annotations
[10:26] <zyga> reading foreign python code is like "where are the types" :(
[10:30] <zyga> mborzecki: did you try adding any annotations?
[10:34] <mborzecki> zyga: not really, i've had enough fun with implementing the chooser ui
[10:34] <mborzecki> zyga: anyways if you want to play with it, better talk to mwhudson first
[10:37] <zyga> mborzecki: https://github.com/CanonicalLtd/subiquity/pull/692#pullrequestreview-389844549
[10:37]  * zyga goes upstaris to make tea
[10:37] <mup> PR CanonicalLtd/subiquity#692: console_conf: various recover chooser tweaks <Created by bboozzoo> <https://github.com/CanonicalLtd/subiquity/pull/692>
[10:37] <zyga> we are running at 23/32 workers now
[10:37] <zyga> we've reached saturation once for about 20 minutes
[10:39] <pedronis> mvo: I made some comments in #8325, some are really general hindsight questions
[10:39] <mup> PR #8325: snap-bootstrap: copy auth data from real ubuntu-data in recovery mode <UC20> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8325>
[10:45] <mup> PR snapd#8458 closed: github: allow cached debian downloads to restore <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8458>
[10:47] <mup> PR snapd#8448 closed: tests/session-tool: add session-tool --dump <Simple 😃> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8448>
[10:47] <zyga> thanks!
[10:48] <mvo> pedronis: thanks, will look in a wee bit, looks like it is closed, I will try to get it to a landable point today :)
[10:51] <ogra> popey, yeah, subiquity is correct
[10:52] <ogra> popey, but the issue is indeed the clock ...
[11:02] <pedronis> mvo: I don't know, there are some open questions
[11:03] <zyga> mborzecki: https://github.com/CanonicalLtd/subiquity/pull/692#pullrequestreview-389870317
[11:03] <mup> PR CanonicalLtd/subiquity#692: console_conf: various recover chooser tweaks <Created by bboozzoo> <https://github.com/CanonicalLtd/subiquity/pull/692>
[11:04] <mborzecki> zyga: thanks!
[11:07] <popey> ogra ok
[11:22] <pedronis> mvo: can you merge #8449, it's all green but travis never came back or started, afaict ?
[11:22] <mup> PR #8449: dirs: don't depend on osutil anymore, mv apparmor vars to apparmor pkg <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8449>
[11:25] <mvo> pedronis: sure
[11:26] <mup> PR snapd#8449 closed: dirs: don't depend on osutil anymore, mv apparmor vars to apparmor pkg <Simple 😃> <Test Robustness> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8449>
[11:30] <zyga> core 20 recovery design
[11:30] <zyga> MAGA - make appliance good again
[11:30]  * zyga hides
[11:31] <zyga> we are at 3/32 workers
[11:31] <zyga> though it will go back to ~20 once canary jobs are done
[11:32] <ogra> MAGA ? so should we deny it exists until it hits us hard ? :)
[11:37] <mborzecki> ogra: you mean another customs war?
[11:39] <zyga> I started implementing snapctl refresh-available
[11:39] <zyga> should have a simple version today
[11:39] <zyga> but first, *hot* tea
[11:39] <zyga> the office is horribly cold even today
[11:40] <zyga> I need a 2nd review for https://github.com/snapcore/snapd/pull/8403
[11:40] <mup> PR #8403: sandbox/cgroup: avoid making arrays we don't use <Skip spread> <Created by zyga> <https://github.com/snapcore/snapd/pull/8403>
[12:08] <mup> PR snapd#8459 opened: asserts: it should be possible to omit many snap-ids if allowed, fix <Created by pedronis> <https://github.com/snapcore/snapd/pull/8459>
[12:15] <zyga> pedronis: ^ gofmt
[12:17] <diddledan> no, you go fmt!
[12:26] <mup> PR snapd#8460 opened: tests/session-tool: kill cron session, if any <Created by zyga> <https://github.com/snapcore/snapd/pull/8460>
[12:26] <zyga> pedronis: ta
[12:42] <pedronis> I'm seeing failures on core-16-64, that are not obviously bogus
[12:44] <zyga> what kind of failures?
[12:44] <zyga> I saw two kinds today:
[12:44] <zyga> - reboot that went nowhere
[12:45] <zyga> - snap rollback and timeout on "snap list"
[12:45] <zyga> that felt really broken
[12:45] <zyga> mborzecki: https://github.com/snapcore/snapd/pull/8457/checks?check_run_id=570718915 <- cache of debian deps worked!
[12:45] <mup> PR #8457: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[12:45] <pedronis> zyga: possibly, yes, reboot that went nowhere, but it seems new and real
[12:45] <zyga> mborzecki: I wonder if we can set cache scope to "global" to make sure everyone benefits
[12:45] <zyga> pedronis: I saw the reboot failure about twice last week as well
[12:46] <zyga> but never when testing with -debug to see :/
[12:46] <zyga> mborzecki: spread-canary started on your skip label PR
[12:46] <zyga> mborzecki: and it works!!!
[12:46] <zyga> mborzecki: cool
[12:47] <zyga> mborzecki: with some extra love you could set a status label that shows it was skipped
[12:47] <zyga> but the feature works :)
[12:47] <mborzecki> zyga: uhh i don't like it though
[12:47] <zyga> mborzecki: why?
[12:47] <mborzecki> zyga: we still need to take as many workers as distros
[12:47] <zyga> mborzecki: but not spread Vms
[12:47] <zyga> mborzecki: that's nearly free
[12:47] <zyga> mborzecki: they all passed now
[12:47] <zyga> mborzecki: it adds ~30 seconds
[12:48] <zyga> and it's green - except for "pending travis"
[12:48] <mborzecki> hahah
[12:48] <zyga> mvo: https://github.com/snapcore/snapd/pull/8457 <-
[12:48] <mup> PR #8457: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[12:48] <mborzecki> no surprises there
[12:48]  * zyga hugs maciek
[12:48] <zyga> thank you :)
[12:53] <mborzecki> now, i still need to figure out that cla check
[12:54] <mborzecki> looks like there's a difference in what gets merged where between gh and travis
[12:56] <zyga> mvo: src/github.com/snapcore/snapd/vendor/github.com/chrisccoulson/go-tpm2/mu.go:267:17: constant 4294967295 overflows int
[12:57] <zyga> mvo: this now breaks master .deb builds
[12:57] <zyga> mborzecki: you can change how we check out things
[12:57] <zyga> mborzecki: there's also plenty of 3rd party solutions for this but I didn't look deeper
[12:57] <zyga> mborzecki: one was cool though, each CLA signature was a signed file in the repo
[12:57] <zyga> mborzecki: so the check was entirely offline
[12:57] <pedronis> zyga: it's because we are getting a pc-kernel update in the middle of the tests
[12:58] <zyga> ohh
[12:58] <zyga> pedronis: did you reproduce it
[12:58] <mborzecki> zyga: there's an action ready for that, broought to you by SAP (?!)
[12:58] <pedronis> zyga: no, but the log is obvious
[12:58] <zyga> pedronis: we should probably hold refreshes for snaps that cause reboots
[12:58] <pedronis> once you look at it and that the tests
[12:58] <zyga> mborzecki: yes, SAP
[12:58] <mborzecki> https://github.com/cla-assistant/github-action
[12:58] <zyga> mborzecki: fun world :)
[12:58] <pedronis> zyga: we don't have single snaps holding
[12:58] <zyga> I read that one
[12:58] <pedronis> but it's still strange
[12:58] <zyga> pedronis: oh... right
[12:58] <zyga> hmm
[12:58] <zyga> but why doesn't it come back?
[12:58] <pedronis> why would we try to refresh kernel anyway
[12:58] <zyga> maybe really buggy?
[12:58] <pedronis> something is off
[12:58] <zyga> oh
[12:58] <zyga> standup time
[12:59]  * zyga needs to check one thing first
[12:59] <pedronis> zyga: because we make reboot slow explicitly
[12:59] <pedronis> the test don't really support unexplicit
[12:59] <pedronis> reboots
[12:59] <zyga> ah, indeed
[12:59] <pedronis> these tests are not meant to reboot
[13:00] <mup> PR snapd#8457 closed: github: skip spread jobs when corresponding label is set <Skip spread> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/8457>
[13:00] <pedronis> zyga: anyway I do think we want to add per-snap holding at some point, just not clear when
[13:09] <rbasak> I just accidentally published a snap to beta, when nothing was published in beta before. Can I undo that?
[13:10] <roadmr> rbasak: yes
[13:10] <roadmr> rbasak: snapcraft close your-snap beta
[13:10] <rbasak> Ah, I found a "close" option?
[13:10] <rbasak> Got it. Thanks!
[13:10] <roadmr> 👍
[13:12] <rbasak> While I'm on the topic, is there any way I can unpublish the i386 snaps (on a different snap)? We don't build those now, so the ones that are there are way behind and probably useless now.
[13:18] <mvo> zyga: yeah, trying to fix it in 8456
[13:23] <jdstrand> zyga: note, I re-read through PR 8408 yesterday (though after it was merged; it was fine)
[13:23] <mup> PR #8408: snap/naming: add validator for snap security tag <Skip spread> <Created by zyga> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8408>
[13:23] <zyga> jdstrand: thank you!
[13:24] <jdstrand> zyga: PR 7614 and PR 7825 are very high on the list
[13:24] <zyga> jdstrand: thanks
[13:24] <mup> PR #7614: cmd/snap-confine: implement snap-device-helper internally <Created by zyga> <https://github.com/snapcore/snapd/pull/7614>
[13:24] <mup> PR #7825: many: use transient scope for tracking apps and hooks <Security-High> <Created by zyga> <https://github.com/snapcore/snapd/pull/7825>
[13:24] <jdstrand> zyga: the apparmor upload and some training I gave set me back a bit, but I will be getting to them
[13:24] <zyga> jdstrand: both fail in CI now, one on silly thing and one (f31 or f32) fails on something that seems real, I'll invesetigate soon
[13:24] <zyga> jdstrand: but any feedback would be great
[13:25] <zyga> jdstrand: in the branch about refresh-app-awareness please note if I should split up the cgroup scanning code to a separate PR, it could be reviewed faster and land, aiding further review
[13:26]  * jdstrand nods
[13:49] <zyga> uh
[13:50] <zyga> my daughter's school friend is at a hospital
[13:50] <zyga> he lives next door :/
[13:50] <roadmr> ohnoes
[13:51] <zyga> FYI we run at capacity now, saturated 32 workers
[13:51] <zyga> but the queue is empty
[13:51] <zyga> and we should see a drop to ~ half of that in a few minutes
[13:52] <zyga> I will look at implementing the ideas we had today, that should reduce the queue load significantly
[13:52] <zyga> well, worker load
[13:52] <zyga> we're still not queueing because we manage to process everything (for now)
[13:56] <zyga> and we are at 24/32
[14:06] <pstolowski> zyga: oh
[14:06] <mup> PR snapcraft#3019 closed: static: consolidate tooling setup to setup.cfg <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3019>
[14:06] <mup> PR snapcraft#3020 closed: spread tests: default base for local plugin tests <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3020>
[14:06] <mup> PR snapcraft#3022 opened: plugins: introduce v2.PluginV2 and v2.NilPlugin <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3022>
[14:07] <zyga> pstolowski: yeah :/
[14:07]  * zyga is hungry and breaks for lunch
[14:07] <zyga> o/
[14:11] <mup> PR snapd#8411 closed: boot: cleanup more things, simplify code <Reviewed> <UC20> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/8411>
[14:11] <stgraber> zyga: back to what I asked you about yesterday, any reason why "snap run --command=stop" would block on snapd.socket?
[14:11] <stgraber> zyga: if so, you need to find a way to make snapd keep running until the last snap has been stopped
[14:11] <zyga> yes, it does so when system key is different to the one on disk
[14:12] <zyga> it normally happens when you boot a new kernel
[14:12] <zyga> we ask snapd to generate new profiles and wait until it does so
[14:12] <stgraber> zyga: the current situation means LXD is never stopped properly, causes a 10min shutdown delay and data loss
[14:12] <stgraber> *never stopped properly in those cases
[14:13] <stgraber> I'm not sure it's just about kernel updates, I have a system that seems to reproduce it every time, let me try it again today
[14:13] <zyga> stgraber: please raise a bug to mvo
[14:13]  * zyga is at lunch
[14:13] <zyga> (well almost)
[14:13] <zyga> mborzecki: https://github.com/snapcore/snapd/pull/8461
[14:13] <mup> PR #8461: github: run non-canary if label is present <Created by zyga> <https://github.com/snapcore/snapd/pull/8461>
[14:13]  * zyga is gone
[14:14] <mup> PR snapd#8461 opened: github: run non-canary if label is present <Created by zyga> <https://github.com/snapcore/snapd/pull/8461>
[14:16] <zyga> stgraber: FYI I experienced this issue but was unable to debug it at the time
[14:16] <stgraber> yeah, got it easily reproducible on an arm64 system somehow, seems to happen at every single reboot
[14:17] <stgraber> the new kernel thing would explain why other users only get it somewhat randomly though
[14:17] <zyga> For me it was x86
[14:17] <stgraber> filing a critical bug against snapd claiming data loss
[14:17] <zyga> Perhaps system key is buggy
[14:17] <zyga> Please!
[14:22] <stgraber> mvo: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1871652
[14:22] <mup> Bug #1871652: Daemon snaps not properly stopped in some cases <champagne> <snapd (Ubuntu):Triaged> <https://launchpad.net/bugs/1871652>
[14:23] <stgraber> mvo: as you know, every single server and cloud instances of 20.04 will use the LXD snap and all upgrading users of 18.04 snap will upgrade to the snap too, so we really really need this resolved or we're in for a lot of data loss / corruption issues.
[14:25] <mvo> stgraber: looking
[14:25] <stgraber> mvo: thanks!
[14:25] <stgraber> I'm creating a test VM on that arm64 system which can be played with as much as needed, should make fixing this easier
[14:27] <mvo> stgraber: I think zyga is on to something here, snap run will wait for snapd to re-generate the profiles, if snapd is already stopped this of course won't work, I need to see why this happens/how to do fix it
[14:40] <stgraber> mvo: I've updated the LP bug with my reproducer on arm64
[14:40] <stgraber> mvo: I'm happy to sort out a way for someone from your team to access that system if that helps
[14:41]  * cachio afk
[14:42]  * cachio afk
[14:42] <mvo> stgraber: in various meetings right now, need to find someone to look at this while I'm "off"
[14:42] <zyga> re
[14:42] <zyga> stgraber: I have plenty of arm64 boards
[14:43] <zyga> I can look later today
[14:43] <stgraber> zyga: VM capable?
[14:43] <zyga> stgraber: hmmmm
[14:43] <zyga> stgraber: good question
[14:43]  * zyga checks
[14:43] <stgraber> zyga: the system I'm testing this on is a 48 core, 128GB RAM, arm64 server :)
[14:43] <zyga> stgraber: I think you win :)
[14:43] <stgraber> (which Qualcomm kindly forgot in my basement before firing the entire team who designed it)
[14:44] <zyga> GCE had a hiccup, restarted a job to see if it was temporary
[14:53] <mup> PR snapd#8459 closed: asserts: it should be possible to omit many snap-ids if allowed, fix <UC20> <⚠ Critical> <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/8459>
[14:55] <mup> PR snapd#8460 closed: tests/session-tool: kill cron session, if any <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8460>
[14:56] <cjwatson> that's one way to get hardware
[14:58] <zyga> stgraber: if you _ever_ want to throw it out
[14:58] <zyga> just remember
[14:59] <zyga> bring it to europe on a plane and I can relieve you of it ;-)
[15:00] <stgraber> :)
[15:00] <zyga> stgraber: are there any arm servers available that don't require a datacenter contract?
[15:04] <zyga> stgraber: looking at the bug now
[15:04] <zyga> stgraber: would it be possible for me to get a shell on a machine where this can be reproduced?
[15:05] <zyga> alternatively, I'd love to see the system key snapd writes
[15:05] <stgraber> zyga: happen to have IPv6 on your side?
[15:05] <zyga> stgraber: if is in /var/lib/snapd/system-key
[15:05] <zyga> stgraber: unfortunately no :/
[15:05] <zyga> maybe don't open access for now,
[15:05] <zyga> system-key is ... well, the key
[15:05] <mvo> zyga: thanks for looking, I'm a bit busy with meetings
[15:05] <zyga> it may be revealing
[15:08] <stgraber> {"version":10,"build-id":"799a88b406b245795da51b18f6224003020c6fb9","apparmor-features":["caps","dbus","domain","file","mount","namespaces","network","network_v8","policy","ptrace","query","rlimit","signal"],"apparmor-parser-mtime":1538072454,"apparmor-parser-features":["unsafe"],"nfs-home":false,"overlay-root":"","seccomp-features":["allow","errno","kill_process","kill_thread","log","trace","trap","user_noti
[15:08] <stgraber> f"],"seccomp-compiler-version":"66988dd2c3fb0abf9b1fb29be212771d7c38ae85 2.4.1 8c73f36d3de1f71977107bf6687514f16787f639058b4db4c67b28dfdb2fd3af bpf-actlog","cgroup-version":"1"}
[15:08] <zyga> thanks, let me inspect things now
[15:11] <zyga> stgraber: so, why does lxd stop itself using snap run?
[15:12] <zyga> this is not a bug on your side, I think, I'm just curious
[15:12] <stgraber> that's how the systemd units are generated
[15:12] <stgraber> all Commands in there wrap using snap run I think
[15:12] <zyga> ah, I see,
[15:13] <zyga> indeed
[15:15] <zyga> stgraber: is it possible to reproduce this with SNAPD_DEBUG=1 set
[15:16] <zyga> stgraber: if so please attach that
[15:16] <zyga> I need to break now, my 1yo daughter just woke up
[15:16] <zyga> but I have a hunch I know what it is
[15:16] <zyga> having that will confirm
[15:18] <pstolowski> so, fun fact about persistent journal; restarting systemd-journald triggers snapd restart (?) and since this is happening from config hook, bad things happen :/
[15:21] <pedronis> pstolowski: that's a problem for sure :/
[15:22] <pstolowski> pedronis: it's annoying, because core16 seems to need journald restart
[15:25] <zyga> Whaaat?
[15:25] <zyga> Why do we restart?
[15:25] <zyga> Can we reload it instead?
[15:26] <pstolowski> zyga: i need to try
[15:39] <pedronis> pstolowski: core18 and 20 work without?
[15:41] <pstolowski> pedronis: core18 - yes. i haven't checked 20
[15:43] <pstolowski> Failed to reload systemd-journald.service: Job type reload is not applicable for unit systemd-journald.service.
[15:43] <pstolowski> :}
[15:43] <pstolowski> there you go
[15:43] <pstolowski> only systemctl restart does it
[15:44] <pedronis> pstolowski: if it works with 18 and 20 without, I would just go without, restarting the journal is kind of weird anyway
[15:45] <pstolowski> pedronis: ok. i'll double check if i wasn't dreaming
[15:46] <pedronis> pstolowski: anyway what you could try is kill USR1
[15:46] <pedronis> pstolowski: see man systemd-journald
[15:47] <pstolowski> pedronis: aaha, thanks!
[15:49] <pedronis> pstolowski: as usual is not super clear what it does
[15:49] <pedronis> from the man
[15:55] <pedronis> zyga: mvo: I got again a bunch of allocation problems: https://github.com/snapcore/snapd/pull/8436/checks?check_run_id=570978994
[15:55] <mup> PR #8436: configcore,tests: use daemon-reexec to apply watchdog config <Reviewed> <Squash-merge> <Created by pedronis> <https://github.com/snapcore/snapd/pull/8436>
[15:56] <zyga> pedronis: looking
[15:56] <zyga> pedronis: happened once today
[15:56] <zyga> it looks like some permission issue
[15:56] <zyga> it was mentioned on the internal channel
[15:56] <zyga> please restart the workflow, it's not a capacity problem
[15:56] <zyga> we don't know what caused it
[15:57] <zyga> better yet, merge master for more fixes :)
[15:58] <pedronis> zyga: I merged master many times
[16:10] <pstolowski> pedronis: systemctl kill --signal=SIGUSR1 systemd-journald does the job on core16
[16:10] <pedronis> pstolowski: good
[16:10] <pedronis> pstolowski: that seems safe everywhere
[16:11] <pedronis> systemd has Kill I think, right?
[16:11] <pedronis> I mean systemd our package
[16:12] <pstolowski> pedronis: yes, i'm just looking at it right now
[16:41] <mvo> zyga: do you have anything to share about the lxd bug ? anything you figured out already that is worth for me to know?
[16:41] <zyga> mvo: I'm still partially AFK but give me some more time
[16:42] <zyga> mvo: I have conditions to reproduce it
[16:42] <zyga> mvo: and I _suspect_ I know what the problem is
[16:43] <mvo> zyga: nice, keep me updated please
[16:47] <ijohnson> pedronis: did you want me to change to use a string pointer for mockedMountInfo in 8451 ?
[16:48] <pedronis> ijohnson: yes, it's it not too annoying
[16:48] <pedronis> heh
[16:48] <pedronis> if it's not
[16:48] <ijohnson> sure I mean I'll only have to re-start the workflows 1000 more times anyways so it's not a big deal
[16:49] <pedronis> ijohnson: about the selinux tests, yes, that's fine, anyway is a different package, it was really testing two levels
[16:49] <ijohnson> right
[17:37] <zyga> let's merge https://github.com/snapcore/snapd/pull/8456#pullrequestreview-390189749
[17:37] <zyga> it needs a 2nd review
[17:38] <mup> PR #8456: tests: add 32 bit machine to GH actions <Created by mvo5> <https://github.com/snapcore/snapd/pull/8456>
[17:38] <zyga> ijohnson: any issues?
[17:39] <ijohnson> mm?
[17:39] <ijohnson> oh the PR you just mentioned?
[17:39] <zyga> with CI
[17:40] <ijohnson> I haven't been looking at CI in the past hour or two, just seems annoying that every time I look at one of my PR's exactly one check out of the 17 failed and so I have to restart everything
[17:41] <zyga> ijohnson: I'll prepare the quad workflows for tomorrow
[17:41] <ijohnson> I reviewed 8456
[17:41] <zyga> ijohnson: it's late and I'm looking at something else
[17:41] <ijohnson> yes that would be much appreciated
[17:41] <ijohnson> also did you see the mount ns bug I assigned to you last night ?
[17:41]  * zyga needs coffee and checks
[17:41] <ijohnson> I couldn't reproduce it with robust-mount-namespace-updates=true with a small reproducer snap, but with the full snap I can still reproduce the EBUSY
[17:42] <ijohnson> anyways I can send you the snap when you have time to look at the issue
[17:42] <mup> PR snapd#8456 closed: tests: add 32 bit machine to GH actions <Created by mvo5> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8456>
[17:43] <zyga> ijohnson: cannot find it, let me check my mail
[17:50] <pedronis> did we get a newer systemd recently in 20.04 ?
[18:01] <mvo> ijohnson: I can override failures in merges fwiw
[18:02] <mvo> ijohnson: we are a bit timezone challenged so not ideal but do ping me if you have such a case
[18:04] <ijohnson> mvo ack maybe I'll send you an email at my EOD if needed
[18:04] <mvo> ijohnson: sure thing!
[18:08] <zyga> re, back to work
[18:09] <zyga> pedronis: 244 was in Feburary
[18:09] <zyga> February
[18:09] <zyga> 245 was in March
[18:10] <zyga> we are now on 245.2
[18:13] <pedronis> ok, just confused because a test that I tried failed now, anyway it indeeds needs tweaking for systemd >=243
[19:17] <zyga> mvo: I debugged the issue related to lxd and snapd
[19:19] <zyga> cachio: 19.10 images also have GDM
[19:19] <zyga> cachio: it would be good to regenerate them so that we don't have the desktop
[19:20] <ijohnson> zyga: what was the issue with lxd and snapd ?
[19:20] <ijohnson> I'm curious
[19:25] <zyga> ijohnson: https://bugs.launchpad.net/snapd/+bug/1871652
[19:25] <zyga> it's all there
[19:25] <mup> Bug #1871652: Daemon snaps not properly stopped in some cases <champagne> <snapd:Confirmed for zyga> <snapd (Ubuntu):Confirmed for zyga> <https://launchpad.net/bugs/1871652>
[19:25] <zyga> ijohnson: but tl;dr; is in the last comment
[19:25]  * ijohnson reads
[19:25] <zyga> ijohnson: it's pretty interesting actually
[19:25] <zyga> stgraber: thank you for the debugging environment
[19:25] <zyga> stgraber: it's late so unless it's very urgent I will fix it first thing  tomorrow after discussing with the team
[19:26] <stgraber> zyga: it's been happening for a long long time, we can wait another day :)
[19:26] <zyga> I hope one last day
[19:26] <zyga> let me do one more test today
[19:26] <stgraber> it's just that now that we understand it, we also understand the danger from it (containers aren't stopped at all, filesystem isn't unmounted, so data loss potential)
[19:26] <stgraber> it actually explains why we've seen some odd db corruption in the past which we couldn't really explained based on logs
[19:27] <zyga> yes, I think the bug is well marked as critical
[19:28] <mvo> zyga: ohhhh, what did you find out?
[19:28] <mvo> zyga: ok, how involved is the fix :) ?
[19:28] <zyga> stgraber: as a small note, setting
[19:28] <zyga> SNAPD_DEBUG_SYSTEM_KEY_RETRY=0
[19:28] <zyga> should work around it
[19:29] <zyga> mvo: it depends
[19:29] <zyga> mvo: please read https://bugs.launchpad.net/snapd/+bug/1871652
[19:29] <zyga> mvo: it's probably something we can fix tomorrow
[19:29] <mup> Bug #1871652: Daemon snaps not properly stopped in some cases <champagne> <snapd:Confirmed for zyga> <snapd (Ubuntu):Confirmed for zyga> <https://launchpad.net/bugs/1871652>
[19:29] <zyga> mvo: tl;dr; is https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1871652/comments/7
[19:29] <mvo> zyga: nice
[19:29] <mvo> zyga: but it does sounds like the fix will not be entirely easy
[19:30] <zyga> mvo: it's actually very easy
[19:30] <zyga> mvo: just the if ( ... ) part needs discussing
[19:30] <mvo> I like the sound of that
[19:30] <zyga> we cannot wait for system key on shutdown
[19:30] <zyga> and we probably should depend on core/snapd
[19:30] <zyga> and not let them go away / unmounted
[19:30] <zyga> this was never expressed in systemd terms
[19:30] <zyga> but let's discuss that tomorrow
[19:30] <mvo> ok
[19:30] <zyga> it's late and I'd love to get off my chair :)
[19:31] <mvo> zyga: sounds great, thank you so much
[19:31] <zyga> we know exactly how to reproduce this
[19:31] <zyga> and we know what to change to fix it, we need to discuss how to introduce the changes
[19:31] <zyga> mvo: I suspect we _can_ do a minimal fix tomorrow
[19:31] <zyga> without ill effects
[19:31] <zyga> and work on a more proper fix for +1
[19:32] <zyga> the minimal fix will just detect the shutdown and ignore system key
[19:32] <zyga> the proper fix will introduce dependencies
[19:32] <zyga> so lxd will not stop after core is unmounted
[19:32] <mvo> zyga: sounds good to me
[19:32] <zyga> but that's more iffy for the reasons you probably know about (wrappers and ensure)
[19:32]  * zyga waves and takes a break
[19:33] <cachio> zyga, I'll run the update7
[19:33] <zyga> cachio: let me know if you need mount-ns test changes for that
[19:33] <mvo> zyga: thank again and good night
[19:33] <zyga> cachio: it slipped from my radar but I will send the patches tomorrow
[19:34] <cachio> zyga, sure
[19:34] <zyga> stgraber: and I know why I cannot reproduce it, for development I disabled reexec on my main machine
[19:34] <stgraber> ah, that'd do it
[19:37] <cachio> zyga, job running
[19:38] <cachio> zyga, https://travis-ci.org/github/snapcore/spread-cron/builds/672666095
[19:38] <cachio> it would be ready in about 40 minutes
[19:49] <mup> PR snapcraft#3021 closed: remote-build: remove artifact sanity check <Created by cjp256> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3021>
[19:50] <zyga> stgraber: ^
[19:50] <zyga> https://github.com/snapcore/snapd/pull/8462
[19:50] <zyga> actually ^
[19:50] <mup> PR #8462: cmd/snap: don't wait for system key when stopping <Created by zyga> <https://github.com/snapcore/snapd/pull/8462>
[19:50] <zyga> it should do the job, we need to package it with tests and stuff
[19:50] <mup> PR snapd#8462 opened: cmd/snap: don't wait for system key when stopping <Created by zyga> <https://github.com/snapcore/snapd/pull/8462>
[19:51] <zyga> that code didn't seem to have unit tests before so it will take me more
[19:51] <zyga> now I'm really gone
[19:51] <zyga> mvo: T
[19:51] <zyga> ^
[19:51] <stgraber> looks simple enough :)
[20:50] <mup> PR snapcraft#3023 opened: pluginhandler: move attributes to PluginHandler <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3023>
[21:16] <mup> PR snapd#8463 opened: secboot: key sealing also depends on secure boot enabled <UC20> <Created by cmatsuoka> <https://github.com/snapcore/snapd/pull/8463>