[03:28] <mup> Bug #1888600 opened: snap run should autocomplete <Snappy:New> <https://launchpad.net/bugs/1888600>
[05:13] <mborzecki> morning
[05:25] <mup> PR snapd#9042 closed: interfaces: optimize rules of multiple connected iio/i2c/spi plugs (2.45) <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9042>
[05:56] <mborzecki> so weird, some ci jobs appear stuck for no clear reason, and restarting the workflow makes those previously stuck jobs run immediately
[06:01] <mup> PR snapd#9050 opened: Avoid exit when nested type var is not defined  (2.45) <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9050>
[06:11] <mup> PR snapd#9047 closed: cmd/snap-preseed: backport fixes (2.45) <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9047>
[06:20] <jamesh> yeah.  I had the ubuntu-nested-* jobs all waiting to start for a run I set going last night.
[06:33] <mborzecki> jamesh: which PR? can you restart the workflow (instead of closing and reopening the PR)?
[06:33] <jamesh> mborzecki: I restarted it, and the missing jobs ran
[06:33] <jamesh> mborzecki: it was this one: https://github.com/snapcore/snapd/pull/8861
[06:34] <mup> PR #8861: data,packaging,wrappers: extend D-Bus service activation search path <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/8861>
[06:34] <mborzecki> jamesh: cool
[06:34] <mborzecki> jamesh: it's funny cause you can restart the workflow if it's seemingly stuck and it runs
[06:34] <jamesh> I'd like to merge it, but the ubuntu-20.10 job is failing on the preseed and pulseaudio tests.
[06:35] <jamesh> mborzecki: maybe the communication link with the self hosted runners got confused?
[06:35] <mborzecki> jamesh: that's fine, preseed is known (the groovy images are preseeded now and that confuses the test)
[06:35]  * jamesh hits merge then.
[06:36] <mup> PR snapd#8861 closed: data,packaging,wrappers: extend D-Bus service activation search path <Created by jhenstridge> <Merged by jhenstridge> <https://github.com/snapcore/snapd/pull/8861>
[06:37] <mborzecki> jamesh: pulseaudio is a known thing to, i'll do some digging today but if i fail to uncover anything useful we may need to poke someone frm the desktop team
[06:38] <jamesh> mborzecki: I'm looking into this bug report from my end: https://bugs.launchpad.net/ubuntu/+source/pulseaudio/+bug/1886854 -- it doesn't obviously tie in with the spread failure though.
[06:38] <mup> Bug #1886854: Race in load-module snap policy check in classic confinement <focal> <rls-ff-incoming> <pulseaudio (Ubuntu):New for jamesh> <https://launchpad.net/bugs/1886854>
[06:41] <mborzecki> jamesh: interesting, i'll try to debug the test today, i'll let you know if it's related (though i suspect it's rather something about our user session setup)
[06:56] <mup> PR snapd#9030 closed: bootloader/assets: helpers for registering per-edition snippets, register snippets for grub <UC20> <Created by bboozzoo> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9030>
[07:21] <mup> PR snapd#9026 closed: tests/nested/manual: add spread tests for cloud-init vuln <Run nested> <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9026>
[07:37] <mborzecki> pedronis: hey, i've opened one more cherry-pick for 2.45 https://github.com/snapcore/snapd/pull/9050
[07:37] <mup> PR #9050: tests: avoid exit when nested type var is not defined  (2.45) <Run nested> <Simple 😃> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9050>
[07:37] <pedronis> mborzecki: I saw it, trying to run it with nested tests on
[07:38] <pedronis> mborzecki: I merged master into #9013 which is now small, it needs a re-review
[07:38] <mup> PR #9013: many: merge 2.45.2 fixes back into master <⛔ Blocked> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9013>
[07:38] <mborzecki> pedronis: ok, will do
[07:39] <pedronis> thx
[07:58] <zyga> o/
[07:58] <zyga> 2AM schedule is not fun
[08:41] <mup> PR snapd#8973 closed: tests: moving journalctl.sh to a new journal-state tool <Created by sergiocazzolato> <Merged by zyga> <https://github.com/snapcore/snapd/pull/8973>
[08:47] <mborzecki> pedronis: i've updated #9006
[08:47] <mup> PR #9006: bootloader: compose command line with mode and extra arguments <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9006>
[08:52] <pedronis> mborzecki: can we do the "go generate" thing?  looking at scripts changes in the middle of go changes is not great
[08:56] <mup> PR snapd#9050 closed: tests: avoid exit when nested type var is not defined  (2.45) <Run nested> <Simple 😃> <Created by bboozzoo> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9050>
[09:05] <zyga> brb coffee
[09:08] <mborzecki> pedronis: i'll drop that commit bumping the edition and force push
[09:54] <mborzecki> jamesh: https://paste.ubuntu.com/p/NRkvhFfVQZ/ hm interestingly haven't seen it before
[09:56]  * jamesh looks
[09:59] <jamesh> mborzecki: That is from today's landing.  I wonder why it didn't show up before?
[10:01] <jamesh> mborzecki: I don't think we patched core to make /etc/dbus-1/session.d writable because I thought the core18 code didn't run there
[10:02] <pedronis> jamesh: no, but we have some tests that try the core -> snapd/core18 transition
[10:02] <pedronis> and that code will get involved
[10:02] <jamesh> pedronis: right.  I'm wondering why it didn't show up in spread test runs for my PR
[10:03] <jamesh> I realise we probably need to add the extra writable path
[10:03] <mborzecki> jamesh: that would be core16, so /etc/dbus-1/ is not writable
[10:03] <mborzecki> duh, dbus-1/session.d
[10:03] <mborzecki> (dbus-1/systemd.d is)
[10:04] <pedronis> anyway we need either a quick fix or to revert something temporarly
[10:04] <pedronis> I fear
[10:05] <jamesh> It is a one line change to the core snap build script.  Could that be landed fast enough?
[10:12] <jamesh> pedronis: here is a PR if that is acceptable: https://github.com/snapcore/core/pull/114
[10:12] <mup> PR core#114: writable-paths: make /etc/dbus-1/session.d writable <Created by jhenstridge> <https://github.com/snapcore/core/pull/114>
[10:13] <zyga> I wonder if this will affect any tests back in snapd
[10:13] <zyga> oh well
[10:13] <zyga> I'm okay with the ping pong if that's what it takes
[10:14] <jamesh> I'm still not sure why the test didn't fail in my own PR
[10:15] <mup> PR core#114 opened: writable-paths: make /etc/dbus-1/session.d writable <Created by jhenstridge> <https://github.com/snapcore/core/pull/114>
[10:16] <mborzecki> maybe the restore of tests doing remodel 16 -> 18 is not cleaning up all of the state properly?
[10:17] <mup> PR snapd#9051 opened: tests/main/interfaces-pulseaudio: enable debug logging in pulseaudio <Precious Logs> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9051>
[10:18] <zyga> mborzecki: possibly
[10:18] <zyga> mborzecki: could we write an invariant check somehow?
[10:19] <zyga> brb
[10:48] <pedronis> jamesh: the change in itself seems fine
[11:26] <zyga> I've updated https://github.com/snapcore/snapd/pull/8977
[11:26] <mup> PR #8977: cmd/snap: track started apps and hooks <Created by zyga> <https://github.com/snapcore/snapd/pull/8977>
[11:37] <pedronis> zyga: I have put it back into my queue
[11:37] <zyga> thanks!
[12:05] <ijohnson> morning folks
[12:16] <zyga> ijohnson: hey :)
[12:16] <zyga> brb
[12:16] <ijohnson> hey zyga feeling better today ?
[12:16] <ijohnson> also the github actions stuff is really exciting, I didn't realize globs work, and there's even a download artifact action too
[12:16] <ijohnson> still a bit unclear how we feed things from the github actions into spread, but I don't have time to work on that anyways
[12:23] <zyga> ijohnson: yeah
[12:24] <zyga> ijohnson: the tests today don't require any special precautions so I'm my normal dose of painkillers
[12:24] <zyga> ijohnson: but the surgery is so close I cannot wait!
[12:24] <ijohnson> that's good to hear
[12:24] <zyga> ijohnson: I may be able to stand at the tgif next week :)
[12:24] <ijohnson> nice!
[12:24] <zyga> I won't be able to sit for a while but that's really minor
[12:24] <ijohnson> but certainly take it as slow as you need to
[12:24] <zyga> after all the stuff heals it will be amazing
[12:24] <zyga> ijohnson: as for actions, yeah they come in pairs
[12:25] <zyga> ijohnson: we could even build snapd on a dedicated fast machine, with deps, with preserved go cache
[12:25] <zyga> ijohnson: and just upload the artefacts for spread cloud
[12:25] <zyga> ijohnson: so many options
[12:25] <zyga> actions have great design
[12:25] <ijohnson> my initial poc idea for how to feed that into spread would be to download the actions into the repo filesystem, so that when we build the tarball of the repo to send to the workers, they already have all the artifacts in them
[12:25] <zyga> they are different from random CI solutions but the benefits are really nicw
[12:26] <zyga> yeah
[12:26] <zyga> exactly
[12:26] <ijohnson> that would have increased network costs though
[12:26] <zyga> you need the token to fetch that
[12:26] <ijohnson> right so that avoids the token problem
[12:26] <zyga> so it must be done by the action runner we host
[12:26] <zyga> well, 16MB is not much
[12:26] <zyga> those machines stream a *LOT* of uncompressed traffic
[12:26] <ijohnson> well but if we really wanted to save time, we'd build the rpm, debs, etc.
[12:26] <zyga> 16MB is like a drop in an ocean
[12:26] <zyga> yeah, we could build remaining packages
[12:27] <zyga> we could even build deb and use that to build the snap
[12:27] <zyga> I think that's the way forward
[12:27] <ijohnson> right, cause actually the snapd snap isn't really used anywhere in the tests right now
[12:27] <zyga> and I think this will cut build time by about 20 minutes
[12:27] <ijohnson> in fact the places we need to use the snapd snap for UC18 and UC20, we just stuff the snapd deb into the snapd snap :-)
[12:27] <zyga> haha, right!
[12:27] <zyga> so I think there are options to explore
[12:27] <ijohnson> oh also I was going to ask you, for hosting a runner, what kind of network usage do you see ?
[12:28] <zyga> but even having a .deb and a .rpm and a .snap on a PR is great
[12:28] <zyga> it's easy to pick up and explore
[12:28] <zyga> ijohnson: let me pull the stats
[12:28] <zyga> network is pretty heavy
[12:28] <zyga> constant ram + spiky ram and CPU on startup (a lot actually) and then loads of network
[12:28] <zyga> go ssh does't implement any compression
[12:28] <zyga> so we pay a lot for traffic that's silly
[12:28] <zyga> it's easily in the megabit a sec for worker range
[12:29] <ijohnson> I have some machines I could dedicate to runners, but I'm concerned about network performance, my network is fast, but it goes down quickly when there are lots of outgoing network connections, like my ISP seems to think that I am DDOS'ing the world when I try to run a full spread suite and my network entirely disappears
[12:29] <ijohnson> but as soon as I kill that spread run, network comes right back
[12:29] <zyga> I think running a few is probably okay
[12:29] <zyga> but it adds up
[12:29] <zyga> I used my TB quota that month when I hosted 32 workers at home
[12:29] <zyga> (but it overflowed my other house usage, not a TB by itself)
[12:30] <zyga> I think one major improvement would be to implement ssh compression
[12:30] <zyga> but there's a bug for that for a few years
[12:30] <zyga> and nobody touched that
[12:30] <zyga> we send lots of text back and forth
[12:30] <zyga> and that adds up
[12:31] <zyga> https://github.com/golang/go/issues/31369
[12:32] <zyga> ijohnson: another possibility might be to shell-out to real ssh
[12:32] <zyga> ah, drat, I don't have the other laptop
[12:32] <zyga> I'll ask my wife to bring it later if that's okay
[12:32] <ijohnson> mmm
[12:32] <zyga> I can then give you precise network usage figures
[12:33] <ijohnson> sure no rush, just curious
[12:40] <zyga> ha
[12:40] <zyga> I was about to ask if we should merge that
[12:40] <zyga> :D
[12:40] <zyga> pedronis: can we trigger a core build faster?
[12:41] <mup> PR core#114 closed: writable-paths: make /etc/dbus-1/session.d writable <Created by jhenstridge> <Merged by pedronis> <https://github.com/snapcore/core/pull/114>
[12:46] <pedronis> zyga: maybe, but first the code needs to get to LP
[12:46] <zyga> ah, via imporyt
[12:50] <pedronis> zyga: yes
[13:07] <zyga-x240> lost network, trying to rejoin
[13:49] <mborzecki> trying to run pulseaudio tests with a fix now
[13:50] <pedronis> https://github.com/snapcore/snapd/blob/master/cmd/snap-bootstrap/cmd_initramfs_mounts.go#L314
[13:51] <mup> PR core18#164 opened: Core18 next <Created by xnox> <https://github.com/snapcore/core18/pull/164>
[13:51] <mup> PR core18#165 opened: Cleanup /run in the core <Created by xnox> <https://github.com/snapcore/core18/pull/165>
[14:08] <mborzecki> pushed a tweak to https://github.com/snapcore/snapd/pull/9051 let's see if the pulseaudio tests are happy this time
[14:08] <mup> PR #9051: tests/main/interfaces-pulseaudio: enable debug logging in pulseaudio <Precious Logs> <Test Robustness> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9051>
[14:18] <mup> PR snapd#9048 closed: .github/workflows/snap-build.yaml: build the snapd snap via GH Actions too <Simple 😃> <Created by anonymouse64> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9048>
[14:22] <ijohnson> mborzecki: I just noticed a pulseaudio and audio-playback failure on focal, have you seen that before ?
[14:24] <ijohnson> mborzecki: logs are here https://pastebin.ubuntu.com/p/zxTZtQnVmf/
[14:27] <mborzecki> ijohnson: hm looks similar
[14:29] <ijohnson> pedronis: btw looks like #9013 is ready
[14:29] <mup> PR #9013: many: merge 2.45.2 fixes back into master <Created by mvo5> <https://github.com/snapcore/snapd/pull/9013>
[14:30] <ijohnson> (but needs sudo git merge perms)
[14:45] <mborzecki> ijohnson: do you have a debian-10 vm?
[14:45] <ijohnson> mborzecki: not handy no
[14:45] <ijohnson> mborzecki: I think jdstrand has some instructions on his blog on how to easily get a cloud image VM
[14:46] <mborzecki> hm seems like the preset on sid is enabled for both snapd.socket and .service
[14:46] <mborzecki> so i'm not sure how op in https://forum.snapcraft.io/t/snapd-not-start-automatic/19039/ got both in 'disabled'
[14:48] <mup> PR snapd#9052 opened: release/2.45: backport writable_defaults dir changes <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[14:48] <mborzecki> anyways, time to start packing for a drive back home
[14:48] <jdstrand> I do! https://blog.strandboge.com/2019/04/16/cloud-images-qemu-cloud-init-and-snapd-spread-tests/
[14:49] <ijohnson> ah 3 seconds too late!
[14:49] <jdstrand> ah bummer :)
[14:49]  * jdstrand hopes that page is useful to others. I certainly have found it handy
[14:52] <ijohnson> pedronis: I opened https://github.com/snapcore/snapd/pull/9049 with the manually ported patch from master for that bit we talked about
[14:52] <mup> PR #9049: release/2.45: backport of uc20 PR's <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9049>
[14:52] <ijohnson> oh whoops wrong PR number
[14:53] <ijohnson> #9052 is the right PR
[14:53] <mup> PR #9052: release/2.45: backport writable_defaults dir changes <Run nested> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9052>
[14:54] <pedronis> thx
[15:12] <mup> PR core20#76 closed: hooks/001-extra-packages.chroot: add gdbserver <Created by anonymouse64> <Merged by xnox> <https://github.com/snapcore/core20/pull/76>
[15:17] <mup> PR core20#77 opened: Drop console-conf hack, fixed in subiquity. core18 pr#148 <Created by xnox> <https://github.com/snapcore/core20/pull/77>
[16:03] <mup> PR snapd#9013 closed: many: merge 2.45.2 fixes back into master <Created by mvo5> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9013>
[16:05] <pedronis> ijohnson: I merged that ^ and reviewed #9049
[16:05] <mup> PR #9049: release/2.45: backport of uc20 PR's <UC20> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9049>
[16:09] <ijohnson> thanks pedronis, looking now
[20:29] <mup> PR snapcraft#3225 opened: specifications: default tracks <specification> <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3225>
[20:30] <mup> PR snapd#9053 opened: many/tests/preseed: reset the preseeded images before preseeding them <Bug> <Preseeding 🍞> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9053>
[21:30] <mup> PR snapd#9029 closed: api: seeding debug api (2/N) <Needs Samuele review> <Preseeding 🍞> <Created by stolowski> <Merged by pedronis> <https://github.com/snapcore/snapd/pull/9029>
[23:20] <mup> PR snapcraft#3226 opened: extensions: kde-neon: use gtk3 platform theme on gtk-based DE's <Created by galgalesh> <https://github.com/snapcore/snapcraft/pull/3226>