[05:08] <mborzecki> morning
[08:50] <mborzecki> heh, go templates are quite annoying
[09:37] <mborzecki> pedronis: https://github.com/snapcore/snapd/pull/9054
[09:37] <mup> PR #9054: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>
[09:38] <mup> PR snapd#9054 opened: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>
[09:38] <pedronis> thx
[09:46] <mborzecki> quick errand, back in 30
[10:04] <zyga> o/
[10:04] <zyga> back now
[10:04] <zyga> all tests done
[10:04] <zyga> just surgery left
[10:05] <zyga> how are things?
[10:08] <pedronis> zyga: hi, we have some PR to fix some spread test issues, but otoh stuck/queued tests are definitely a frequent thing: https://github.com/snapcore/snapd/pull/9052 for example, it was already closed/reopened once
[10:08] <mup> PR #9052: release/2.45: backport _writable_defaults dir changes <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[10:08] <zyga> hmm, I wonder if the fact that our workers are on older release is a factor
[10:08] <zyga> perhaps I should switch all workers to the current release
[10:08] <zyga> and see if that makes it better
[10:13] <zyga> pedronis: do you know if all kinds of tests were uniformly stuck or were spread jobs more likely to be stuck?
[10:13] <zyga> github-hosted workers are updated automatically
[10:13] <zyga> I wonder if there's any indication that version skew is a factor
[10:21] <mborzecki> re
[10:21] <mborzecki> zyga: can you take a look at https://github.com/snapcore/snapd/pull/9051 ?
[10:21] <mup> PR #9051: tests/main/interfaces-pulseaudio: enable debug logging in pulseaudio <Squash-merge> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9051>
[10:22] <zyga> syre
[10:22] <zyga> sure
[10:23] <mborzecki> omg, debian packageing is calling go generate
[10:23] <zyga> what is - in <<-EOF?
[10:24] <mborzecki> discards leading space
[10:24] <zyga> oh neat
[10:24] <zyga> cool
[10:25] <zyga> thank you for using tests.cleanup!
[10:25] <zyga> +1
[10:27] <zyga> mborzecki: should I restart tests there?
[10:27] <mborzecki> zyga: restarted just now
[10:28] <zyga> ok
[10:30] <zyga> I'll fire up more workers, just let me automate the upgrade process
[10:48] <zyga> ok, we should have 32 more workers now, running the latest version
[10:50] <mborzecki> when we're building snapd deb, any clue how _build directory comes to life?
[10:51] <mborzecki> it gets populated with go files, but clearly the data files are not copied, so go generate files
[11:01] <zyga> it's a part of debhelper IIRC
[11:01] <zyga> it constructs a gopath
[11:01] <zyga> it only considers .go files
[11:01] <zyga> similar issues are present in fedora and suse helper stacks
[11:01] <pedronis> mborzecki: we do copy some things by hand already though
[11:02] <pedronis> cp -a cmd/snap/test-data/*.gpg _build/src/$(DH_GOPKG)/cmd/snap/test-data/ for example
[11:05] <mborzecki> pedronis: yeah, added a line there now
[11:33] <mborzecki> pedronis: can you use your superpowers and merge https://github.com/snapcore/snapd/pull/9051 ? the failure in uc20 prepare is a known problem
[11:33] <mup> PR #9051: tests/main/interfaces-pulseaudio: enable debug logging in pulseaudio <Squash-merge> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9051>
[11:34] <mborzecki> feels wasteful to do another spread run hoping it succeeds this time
[11:36] <pedronis> mborzecki: I'll look in a little bit
[11:37] <mborzecki> pedronis: thanks
[11:42] <zyga> mborzecki: systemd can now support v2 freezer
[11:42] <zyga> and load apparmor policy
[11:42] <zyga> on early boot
[11:42] <zyga> wow
[11:44] <zyga> systemd can also verify veriety hashes for filesystem images
[11:44] <zyga> lots of very interesting things there
[11:51] <pedronis> mborzecki: merging, do you know though why that failure log is unreadable? where is the journal printing done? is the debug stuff or some toher code
[11:51] <mborzecki> pedronis: the failure log when prepare on a 20.04 host fails?
[11:51] <pedronis> yes
[11:52] <mborzecki> pedronis: it's all the debugging i and mvo added, https://github.com/snapcore/snapd/pull/8883 should fix the problem so we'll not need the debug output anymore
[11:52] <mup> PR #8883: packaging: stop snapd early on purge <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8883>
[11:53] <pedronis> mborzecki: to be clear if we need to keep we need to do something about it, because it doesn't make sense to a casual reader
[11:54] <mborzecki> pedronis: i've pinged zyga for a review in 8883, maybe you could add it to your queue as well?
[11:54] <mborzecki> there were some concerns where to call the helpers in postrm
[11:54] <zyga> I'll look in 10 minutes
[11:54] <mborzecki> s/helpers/DEBHELPER snippets/
[11:56] <pedronis> mborzecki: I merged 9051, changed a bit description giving it has a fix, not just more logging now
[11:56] <mborzecki> pedronis: cool, thanks
[11:58] <mborzecki> i've updated https://github.com/snapcore/snapd/pull/9053 hopefully groovy spread jobs will be green now
[11:58] <mup> PR #9053: many/tests/preseed: reset the preseeded images before preseeding them <Bug> <Preseeding 🍞> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9053>
[11:59] <mup> PR snapd#9051 closed: tests/main/interfaces-pulseaudio: disable start limit checking for pulseaudio service / enable debug logging <Squash-merge> <Test Robustness> <Created by bboozzoo> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9051>
[11:59] <pedronis> mborzecki: oops, I missed the squash-merge label on it :/
[12:00] <mborzecki> fairly self contained, should be a problem picking all 4 when we need to
[12:00] <mborzecki> all 4 commits from that pr i mean
[12:03] <mup> PR # closed: core18#148, core18#156, core18#163, core18#164, core18#165
[12:14]  * zyga lunch and back to reviews
[12:29] <pedronis> mborzecki: I review #8982 btw, you were asking about the other day
[12:29] <mup> PR #8982: snapshots: export of snapshots <Needs Samuele review> <Created by slimjim777> <https://github.com/snapcore/snapd/pull/8982>
[12:29] <pedronis> *reviewed
[12:29] <mborzecki> pedronis: thanks, there's a sibling PR https://github.com/snapcore/snapd/pull/9036 which could use your review too
[12:29] <mup> PR #9036: snapshots: import of a snapshot set <Created by slimjim777> <https://github.com/snapcore/snapd/pull/9036>
[12:30] <pedronis> mborzecki: I know, not today
[12:33] <zyga> re
[12:57] <pedronis> mborzecki: #9054 looks good to me with small naming nitpick
[12:57] <mup> PR #9054: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>
[12:57] <pedronis> it needs a 2nd review
[12:57] <mborzecki> pedronis: thanks
[13:09] <mup> PR snapd#9055 opened: tests: fix incorrect check in smoke/remove test <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9055>
[13:43] <mborzecki> cmatsuoka: can you take a look at  https://github.com/snapcore/snapd/pull/9054 later on?
[13:43] <mup> PR #9054: bootloader/assets: generate bootloader assets from files <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9054>
[13:43] <cmatsuoka> mborzecki: sure
[13:43] <mborzecki> cmatsuoka: thx
[13:52] <zyga> afk for 15 min then back to reviews starting with maciek's fix for purge
[14:06] <mborzecki> need to go and pick up some stuff fromt he hardware store, bbl
[14:09] <mup> PR snapd#9049 closed: cmd/snap-bootstrap,seed: backport of uc20 PR's (2.45) <UC20> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9049>
[14:21] <ogra_> now this is a curious suggestion ...
[14:21] <ogra_> Log: apparmor="DENIED" operation="chmod" profile="snap.acsccid.pcscd" name="/var/snap/acsccid/common/" pid=26404 comm="pcscd" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[14:21] <ogra_> File: /var/snap/acsccid/common/ (write)
[14:21] <ogra_> Suggestion:
[14:21] <ogra_> * adjust program to write to $SNAP_DATA, $SNAP_COMMON, $SNAP_USER_DATA or $SNAP_USER_COMMON
[14:21] <ogra_> (and despite the denial the app actually creates its socket in SNAP_COMMON as required)
[14:23] <jdstrand> ogra_: the idea is that is a difference to writing in SNAP_COMMON and writing SNAP_COMMON
[14:23] <jdstrand> ogra_: I'll adjust the suggestion to make that clearer
[14:24] <ogra_> oh, i'll make sure it writes to a subdir then ...
[14:24] <zyga> re
[14:24] <ogra_> jdstrand, the worse pare with this snap is this though:
[14:24] <ogra_> Log: apparmor="DENIED" operation="open" profile="snap.acsccid.pcscd" name="/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-2/idVendor" pid=26404 comm="pcscd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[14:24] <ogra_> File: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-2/idVendor (read)
[14:25] <jdstrand> hardware-observe should cover that
[14:25] <ogra_> jdstrand, raw-usb is connected (this is on a Pi4 which has the USB ubs attached via PCI)
[14:25] <ogra_> ah
[14:25] <ogra_> let me try
[14:25] <jdstrand> let me check raw-usb
[14:26] <zyga> uh, that hurt
[14:26] <ogra_> and i have another one ...
[14:26] <ogra_> Log: apparmor="DENIED" operation="open" profile="snap.acsccid.set-power" name="/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/power/control" pid=26098 comm="set-power" requested_mask="wc" denied_mask="wc" fsuid=0 ouid=0
[14:26] <ogra_> File: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/power/control (write)
[14:26] <ogra_> also with raw-usb connected
[14:26] <jdstrand> yeah, the raw-usb interface should cover that, but doesn't: soc vs scb
[14:27] <ogra_> yep, thats what i thought
[14:27] <ogra_> i bet it would work on a regular pi or x86
[14:27] <jdstrand> ogra_: the first was read, that is hardware-observe, but the second is write, which should be raw-usb. I've added this to the next batch of updates
[14:28] <jdstrand> ogra_: what device is that? (for the git commit)
[14:29] <ogra_> jdstrand, Pi4 trying to use an NFC reader (i'm actually trying to assemble a pcscd snap to provide a content interface for others snaps to use later ... thats longstanding missing)
[14:29] <jdstrand> ogra_: ok, thanks :)
[14:30] <ogra_> jdstrand, one issue i cant reall solve is that most of these usb readers actually require you to blacklist the pn533_usb module ... not sure what to do with that (i can modprobe -r when using kernel-module-control indeed)
[14:30] <jdstrand> ogra_: if you get the pcscd to a point that you'd like to share, I can (finally) do the pcscd interface and we can iterate on it in the PR together
[14:31] <ogra_> this is super ugly ... it seems the kernel added a module that most of the readers can not actually use ... they all come with userspace drivers instead ... and indeed that clashes
[14:32] <jdstrand> ogra_: huh. that is something a gadget could do. we'll have to think through this more with pedronis if we want to expose this via interfaces somehow
[14:32] <ogra_> yeah
[14:32] <jdstrand> ogra_: or maybe a snap set :system... (would help the gadget scenario and anyone with snapd-control)
[14:32] <ogra_> i'm not really keen on using kernel-module-control anywhere ...
[14:32] <ogra_> not even in a gadget
[14:33] <ogra_> i dotn ge why the kernel even defaults to the module ... it seems useless in 90% of the cases
[14:33] <ogra_> *dont get
[14:33] <jdstrand> ogra_: oh, that isn't what I meant. today, with the gadget, we could allow you via system-files to write out to a blacklist. a better solution would be a snap set option to unload/disable loading of the module
[14:33] <ogra_> (each nfc howto you find starts with "add this to /etc/modprobe.d/blacklist.conf 🙂 )
[14:34] <jdstrand> heh
[14:34] <ogra_> sure ... but i wonder if we shouldnt rather fix this on a kernel level 🙂
[14:34] <ogra_> (though this might take even longer ...)
[14:35] <jdstrand> that would be ideal. but a module disable facility is generally usable. can you create a topic?
[14:35] <ogra_> will do
[14:37] <jdstrand> thanks!
[14:38] <jdstrand> pedronis: hey, do you have a rough estimate on when 2.46 will go to beta? (trying to plan the 2.46 batch of policy updates)
[14:39] <jdstrand> pedronis: also, we've merge a tool for the store to use for verifying snap decls and roadmr is aware (and part of the review process)
[14:39] <ijohnson> jdstrand: at least not until mvo is back in a week and a half
[14:39] <pedronis> jdstrand: we plan to have a 2.45.3 in beta next week
[14:39] <jdstrand> pedronis: and I plan to answer your forum questions today
[14:39] <jdstrand> ijohnson, pedronis: thanks! :)
[14:40] <jdstrand> pedronis: I think that is everything I have for you :)
[14:40] <pedronis> jdstrand: thx
[14:43] <pedronis> ijohnson: mmh, 9052 has no reviews atm
[14:43] <ijohnson> yes that is true
[14:47] <ijohnson> pedronis: otoh, https://github.com/snapcore/snapd/pull/9055 is basically ready to be merged I think
[14:47] <mup> PR #9055: tests: fix incorrect check in smoke/remove test (2.45) <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9055>
[14:47] <ijohnson> the tests ran especially quick this morning
[14:54] <pedronis> ijohnson: I'm confused by this change:  https://github.com/snapcore/snapd/pull/9052/files#diff-cfcdf502f888d073b649e9c2e2140932R62  master seems to have only part of it, and there's still / in master
[14:54] <mup> PR #9052: many: backport _writable_defaults dir changes (2.45) <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[14:54] <pedronis> there are similar things
[14:55] <ijohnson> mmm interesting seems I was too naive in my patch
[14:55] <ijohnson> let me quick put down what I was coding and look at this again
[14:56] <pedronis> ijohnson: to be clear I don't think, it's material, but I'm confused why master has still the spurious /
[14:58] <pedronis> #9055 can be merged
[14:58] <mup> PR #9055: tests: fix incorrect check in smoke/remove test (2.45) <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9055>
[14:59] <ijohnson> pedronis: shall I merge that or will you ?
[14:59] <pedronis> please go ahead, I have a meeting now
[14:59] <ijohnson> done
[14:59] <mup> PR snapd#9055 closed: tests: fix incorrect check in smoke/remove test (2.45) <Simple 😃> <Test Robustness> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9055>
[15:02] <ijohnson> pedronis: so regarding master only having part of the patch, boot.InstallHostWritableDir is used both has the TargetRootDir and what we check, so master tests pass, but master tests should be using boot.InstallWritableDir like this PR has. I think probably what happened is that one of the writable-defaults PR's was opened before we adjusted all of those vars and then adjusted only the runtime vars, not the test vars
[15:02] <ijohnson> before being merged to master
[15:02] <ijohnson> I will prepare a small PR which fixes the dirs on master to make it more obvious
[15:03] <pedronis> ijohnson: thx
[15:05] <ijohnson> ahhh sorry no I misread parts of the code :-/
[15:05] <ijohnson> let me finish reading this to give a full explanation
[15:06]  * ijohnson is not a big fan of writable-paths
[15:10]  * zyga hurts a bit, 50 min till next pill
[15:29] <ijohnson> pedronis: so all of what I said above is inverted, master is correct (except the "/" at the end, seems testutil silently ignores that), but my patch to 2.45 is wrong, but again because it's wrong both in the TargetRootDir and what we check the test still passes
[15:29] <pedronis> ijohnson: ok, sounds we want small tweaks for both master and 2.45
[15:29] <pedronis> then
[15:30] <ijohnson> to be clear, ConfigureRunSystem when called from devicemgr _should_ be writing to the _writable_defaults dir and from the initramfs when we call DisableCloudInit directly, we also want to write to _writable_defaults dir
[15:30] <ijohnson> but! for fixing the cloud-init CVE, we want to not use _writable_defaults dir because at that point in time, writable-paths(5) has already applied the _writable_defaults stuff to writable/etc/cloud
[15:31] <ijohnson> pedronis: but yes small patches to both 2.45 and master
[15:31] <ijohnson> pedronis: I will rebase and force push the 2.45 branch to make it a simpler git history and hopefully reduce chance of conflicts when merging 2.45 back to master
[15:34] <pedronis> ijohnson: ok
[15:39] <ijohnson> pedronis: PR 9052 is updated
[15:39] <mup> PR #9052: many: backport _writable_defaults dir changes (2.45) <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[15:45] <mup> PR snapd#9056 opened: sysconfig/cloudinit_test.go: add test for initramfs case, rm "/" from path <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9056>
[15:54] <pedronis> ijohnson: reviewed
[15:54] <ijohnson> pedronis: who else could review it so it could land today? cmatsuoka maybe ?
[15:57] <zyga>  I will be available for review in ~ 45 min
[15:57] <zyga> just under pain now
[15:57] <ijohnson> zyga: no worries, even if we can't get it reviewed, it's fine to get it merged on monday too
[15:57] <ijohnson> aiui it's the last PR with nothing blocking on it
[15:58] <mborzecki> ijohnson: 9056?
[15:58] <mborzecki> o 9052?
[15:59] <ijohnson> mborzecki: 9052
[15:59] <ijohnson> sorry I thought you had EOD'd already :-)
[16:00] <mborzecki> ijohnson: yeah, haven't been to good at eod'ing recently
[16:00] <ijohnson> haha
[16:15] <mup> PR snapd#8977 closed: cmd/snap: track started apps and hooks <Created by zyga> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8977>
[16:15] <mborzecki> ijohnson: had to take a look at core20 pr again, but lgtm
[16:15] <mborzecki> and a kernel update just finished, right on time
[16:15] <ijohnson> Thanks mborzecki
[16:15] <ijohnson> Ah right on time indeed
[16:26] <zyga> +1 on https://github.com/snapcore/snapd/pull/9052
[16:26] <mup> PR #9052: many: backport _writable_defaults dir changes (2.45) <Run nested> <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[16:31] <ijohnson> thanks zyga!
[16:32] <zyga> ijohnson: is that the last of the backports?
[16:32] <ijohnson> yeah looks like we will be able to land this today then, all the test are almost done
[16:32] <ijohnson> yes
[16:32] <zyga> that's great
[16:32] <ijohnson> yeah should be in good order for monday
[16:32] <zyga> I see all workers are lit up
[16:33] <zyga> that's good
[16:33] <zyga> man this hurts
[16:33] <zyga> I'm really ready for Tuesday
[16:34] <zyga> ijohnson: will you update https://github.com/snapcore/snapd/pull/8998 to test both variants?
[16:34] <mup> PR #8998: tests/cmd/snap-bootstrap/initramfs-mounts: fix typo <Simple 😃> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/8998>
[16:36] <ijohnson> zyga yes it's in my queue
[16:55] <mup> PR snapd#9052 closed: many: backport _writable_defaults dir changes (2.45) <Run nested> <UC20> <Created by anonymouse64> <Merged by zyga> <https://github.com/snapcore/snapd/pull/9052>
[16:55] <zyga> all is in
[17:05] <mup> PR snapd#9056 closed: sysconfig/cloudinit_test.go: add test for initramfs case, rm "/" from path <Simple 😃> <Test Robustness> <Created by anonymouse64> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/9056>
[17:18] <zyga> xnox: do you recall if there's something that makes systemd not mount /sys/fs/cgroup/unified?
[17:21] <zyga> though I suspect it's just one of our tests breaking something
[17:33] <xnox> zyga:  depends on the kernel you run on, and systemd you run, and kernel commandline, and systemd compiled time default.
[17:33] <xnox> zyga:  so one has to check all that, all version numbers, etc.
[17:33] <zyga> xnox: just noticing it magically goes away randomly on our 16.04 systems
[17:33] <xnox> it has changed over time, when it was introduced, etc.
[17:33] <zyga> so I really think that's another test messing up something
[17:33] <xnox> zyga:  which kernel.
[17:33] <zyga> stock 16.04 up-to-date kernel
[17:33] <zyga> it's a vanilla xenial image
[17:33] <xnox> zyga: which kernel
[17:33]  * zyga looks
[17:34] <xnox> zyga:  all our public clouds can have stock, stock-hwe, stock-cloud-specific kernel none of which match "generic"
[17:34] <zyga> ah, I see
[17:35] <zyga> eh, uname output is not in the log
[17:35] <zyga> oh well
[17:35] <zyga> I'll debug this somehow
[17:35] <ijohnson> xnox: it should be a gce kernel
[17:36] <zyga> linux/4.15.0-1078-gcp
[17:36] <zyga> found it
[17:40] <mup> PR snapd#9057 opened: overlord: use new tracking cgroup for refresh app awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/9057>
[17:43] <zyga> not very creative, but last of the content-rich patch from the big integration branch
[17:54] <zyga> ijohnson: in how many hours do you plan to EOD?
[17:54] <zyga> I will upgrade the other half of the workers then
[17:54] <ijohnson> probably 3-4
[17:54] <ijohnson> but feel free to upgrade things now,
[17:55] <ijohnson> I have some meetings and more iteration locally before pushing up branches
[17:55] <ijohnson> and all the PR's I would have restarted jobs on already landed this morning :-D
[17:55] <zyga> ok
[17:56] <zyga> that out-of-date worker is idle anyway
[17:56] <zyga> let me take them offline first
[17:58] <zyga> upgrading
[17:58] <ijohnson> 👍
[18:01] <zyga> upgrade complete
[18:05] <mup> PR snapd#8949 closed: tests: new fs-state which replaces the files.sh helper <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8949>
[18:06] <zyga> ijohnson: https://github.com/snapcore/snapd/pull/9058
[18:06] <mup> PR #9058: tests: drop accidental accents from e <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9058>
[18:06] <zyga> :D
[18:06] <ijohnson> +1d
[18:08] <zyga> thanks
[18:10] <mup> PR snapd#9058 opened: tests: drop accidental accents from e <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9058>
[18:26] <xnox> ijohnson:  note that 4.15 is based off bionic kernel. Or like an hwe kernel. If you expect to test "xenial core16 as UC16 is shipped" try booting v4.4 based kernel somehow.
[18:26] <ijohnson> xnox: those tests were for desktop/server
[18:26] <xnox> ijohnson:  for which series.
[18:26] <ijohnson> xnox: the tests we do for core boot an actual UC16 vm with an actual uc16 kernel snap
[18:27] <xnox> ijohnson:  default server on xenial are v4.4 kernel too.
[18:27] <zyga> I don't think we test 4.4 then
[18:27] <ijohnson> xnox: for all the series ? I dunno we have 14.04, 16.04 18.04 20.04, 20.10, etc.
[18:27] <ijohnson> for non-ubuntu core it's just whatever is on gce afaik
[18:27] <zyga> though I was mostly after the mystery of why it changes sometimes (across re-run of same test)
[18:28] <zyga> I'll write a detector to spot that though, should be uncovered soon
[18:28] <xnox> ijohnson:  each series has their own base kernel. which doesn't match public cloud images kernel version. Because all public cloud images chose to control their own kernel versions on rolling basis.
[18:30] <ijohnson> xnox: we aren't particulary interested in testing every kernel under the sun for ubuntu, we just want to test the ubuntu releases, i.e. we care about 16.04, if there's some kernel with 16.04 that acts weird it's not really our testing's job to find that
[18:30] <ijohnson> I would expect the kernel team to be on top of testing that 4.15 and 4.4 and 5.whatever kernels all work well with xenial
[18:47] <zyga> ijohnson: one more trivial https://github.com/snapcore/snapd/pull/9059
[18:47] <mup> PR #9059: tests: replace _wait_for_file_change with retry <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9059>
[18:50] <zyga> thank you for the review cmatsuoka
[18:50] <cmatsuoka> zyga: that was a quite strange typo :)
[18:50] <mup> PR snapd#9059 opened: tests: replace _wait_for_file_change with retry <Simple 😃> <Created by zyga> <https://github.com/snapcore/snapd/pull/9059>
[18:51] <zyga> cmatsuoka: I suspect a side effect of dead keys
[18:54] <zyga> ijohnson: updated https://github.com/snapcore/snapd/pull/9057
[18:54] <mup> PR #9057: overlord: use new tracking cgroup for refresh app awareness <Created by zyga> <https://github.com/snapcore/snapd/pull/9057>
[18:56]  * zyga returns to https://github.com/snapcore/snapd/pull/8883/files 
[18:56] <mup> PR #8883: packaging: stop snapd early on purge <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8883>
[19:02] <xnox> ijohnson:  you asked me about why /unified is sometimes mounted and sometimes not.
[19:02] <ijohnson> xnox: well I didn't ask
[19:02] <xnox> the answer to that is that it depends on the host kernel.
[19:02] <ijohnson> sure
[19:02] <ijohnson> that makes sense
[19:02] <xnox> and xenial ships both kernels that have support for it, and dont'.
[19:04] <xnox> but even if one uses lxd, things like that leak. Cause one can boot focal without unified in a lxd container. and vice versa boot xenial with unified in a lxd container on new enough host.
[19:05] <xnox> ijohnson:  yes sorry. the conversation originally was a ping from zyga.
[19:05] <zyga> xnox: that's interesting
[19:05] <zyga> xnox: can you tell me which kernel would do unified and which would not?
[19:05] <xnox> i can't remember. check yourself
[19:05] <zyga> ok
[19:05] <xnox> google it?
[19:05] <zyga> :D
[19:06] <zyga> I'll just look at systemd
[19:06] <zyga> I'll do a quick invariant check that you gave me an idea for
[19:06] <zyga> ijohnson: do you think we could be getting two sets of kernels?
[19:06] <zyga> usually one but sometimes another?
[19:06] <xnox> zyga:  i think there is "requirements" text file or some such.
[19:07] <xnox> zyga:  i think that does mention kernel levels for various things. i hope it is mentioned for unified cgroups too.
[19:07] <zyga> thanks, I'll look it up
[19:07] <xnox> or like log -p history of it.
[19:19]  * zyga takes a break
[19:19] <zyga> I pushed a comment to https://github.com/snapcore/snapd/pull/8883 and will come back to see if it passes with latest master
[19:19] <mup> PR #8883: packaging: stop snapd early on purge <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/8883>
[19:49] <cmatsuoka> ijohnson: I have a theory for the game controller problem in core
[19:50] <cmatsuoka> ijohnson: I think /dev/input/js0 is readable but SDL doesn't read it because it didn't receive the controller connection event
[19:50] <ijohnson> cmatsuoka: I read that as "you have a game theory problem in core" and was simultaneously excited and terrified at the same time
[19:51] <ijohnson> ah that's a good point
[19:51] <cmatsuoka> ahah
[19:52] <cmatsuoka> ijohnson: to test that I wrote a very simple SDL program that just reports button presses. It works nicely on classic but doesn't report anything on core.
[19:52] <cmatsuoka> ijohnson: but the interesting thing is: it doesn't even try to read /dev/input
[19:55] <ijohnson> cmatsuoka: mmm so you strace'd it?
[19:55] <cmatsuoka> yes I did
[19:55] <cmatsuoka> it doesn't open /dev/input/xx
[19:55] <ijohnson> hmm yeah that does sound like a SDL problem indeed then
[19:56] <cmatsuoka> I think it's not exactly a SDL problem, but rather something related to udev or USB hotplugging
[19:57] <cmatsuoka> or how the confined application receives those events
[19:57] <ijohnson> I confirmed with popey that the joystick is in the device cgroup, so I don't see how udev could be the problem
[19:57] <ijohnson> unless SDL listens for udev events directly itself ?
[19:58] <cmatsuoka> well that's a good question. I think I'll have to check how SDL does that
[20:08] <cmatsuoka> ijohnson: it can use different methods to detect the joystick: libudev, HID directly, or maybe some other ways (I see some ioctls I don't recognize there)
[20:08] <cmatsuoka> libudev is optional
[20:08] <ijohnson> I mean if it is asking udev directly (i.e. not monitoring for events) if there exist joystick devices that should work
[20:09] <ijohnson> did you minimal SDL app use libudev ?
[20:09] <ijohnson> *your
[20:10] <cmatsuoka> not explicitly, the code is: https://pastebin.ubuntu.com/p/m5Kpz7JmW4/
[20:10] <cmatsuoka> it's also not in the ldd output
[20:11] <ijohnson> hmm so it's probably using HID
[20:51] <mup> PR snapd#8789 opened: interfaces/docker: use implicitOnClassic: true <Needs Samuele review> <⛔ Blocked> <Created by jdstrand> <https://github.com/snapcore/snapd/pull/8789>
[23:57] <mup> PR snapd#9059 closed: tests: replace _wait_for_file_change with retry <Simple 😃> <Created by zyga> <Merged by cmatsuoka> <https://github.com/snapcore/snapd/pull/9059>