[00:03] <mup> PR snapd#9326 closed: tests: fix for basic20 test running on external backend and rpi <Simple 😃> <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9326>
[01:59] <mup> PR snapd#9362 closed: tests: skip nested images pre-configuration by default <Run nested> <Simple 😃> <Created by sergiocazzolato> <Merged by sergiocazzolato> <https://github.com/snapcore/snapd/pull/9362>
[06:20] <mup> PR snapd#9360 closed: tests: make gadget-reseal more robust <Run nested> <Simple 😃> <UC20> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9360>
[06:35] <mborzecki> morning
[06:37] <mvo> good morning mborzecki
[06:37] <mborzecki> mvo: hey
[06:38] <mborzecki> mvo: so are we out of the woods yet?
[06:39] <mvo> mborzecki: yeah, I think it's time for a (small) bottle of champagne
[06:39] <mvo> mborzecki: resealing+pcr4 just landed
[06:40] <mvo> mborzecki: so we should be good
[06:40] <mup> PR snapd#9277 closed: secboot: add boot manager profile to pcr protection profile <Run nested> <UC20> <Created by cmatsuoka> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9277>
[06:40] <mup> PR snapd#9356 closed: sysconfig,o/devicestate: mv DisableNoCloud to DisableAfterLocalDatasourcesRun <UC20> <Created by anonymouse64> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9356>
[06:40] <mvo> mborzecki: I guess the next step is to test gadget udpate with the pc branch 20/beta/edition3 for real
[06:40] <mvo> mborzecki: and if that works it's really time to celebrate
[06:41] <mvo> mborzecki: I will branch 2.47 now
[06:41] <mborzecki> yay
[06:41] <mvo> mborzecki: afaict all is merged and the only missing pr needs a samuele review first
[06:41] <mborzecki> mvo: i'll work on the fix for updating structures without partitions on an encrypted device
[06:42] <mvo> mborzecki: sounds great, thanks for this
[07:01] <pstolowski> morning
[07:02] <mvo> good morning pstolowski
[07:02] <mvo> pstolowski: do we happen to have a forum topic for disk-space awareness that I could link to?
[07:02] <pstolowski> mvo: i don't think so
[07:03] <pstolowski> mvo: i could publish the notes i sent to degville
[07:04] <mvo> pstolowski: not urgent if degville publishes them I can link to that
[07:04] <mvo> pstolowski: mostly wondering because I'm working on 2.47 release notes
[07:05] <degville> mvo / pstolowski: the disk awareness doc is here: https://forum.snapcraft.io/t/disk-space-awareness/20007
[07:07] <pstolowski> degville: nice, thank you!
[07:07] <degville> np!
[07:07] <pstolowski> degville: there is a mistake there re snapd version (it says 2.37+ )
[07:08] <degville> pstolowski: ah, thanks. I'll fix it now.
[07:08] <mvo> degville: \o/ thank you! I updated the snapd roadmap page now
[07:10] <zyga> good morning :)
[07:10] <zyga> I patched all the profiles last night and ran a test before EODing
[07:10] <zyga> I see that lxd and gdbserver failed, let me look at those
[07:12] <zyga> let's debug those
[07:12] <zyga> mvo: is 2.47 ready now?
[07:13] <zyga> I saw ian and samuele fight some issues last night
[07:13] <mvo> good morning zyga
[07:13] <mvo> zyga: yeah, I'm on 2.47~pre1 right now
[07:13] <zyga> I slept for 5 hours but I'm okay surprisingly
[07:13] <zyga> cool, fingers crossed no more surprises
[07:15] <mup> PR snapd#9361 closed: tests/lib/cl_check.py: use python3 compatible code <Simple 😃> <Skip spread> <Created by anonymouse64> <Merged by bboozzoo> <https://github.com/snapcore/snapd/pull/9361>
[07:27] <zyga> ah missing snap-gdbserver-shim
[07:27] <zyga> fixed
[07:27] <zyga> let's look at lxd now
[07:35] <mborzecki> zyga: can you take a loot at https://github.com/snapcore/snapd/pull/9364 ? you like them python bits ;)
[07:35] <mup> PR #9364: tests/lib/cla_check: default to Python 3, tweaks, formatting <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9364>
[07:35] <zyga> sure
[07:35] <zyga> looking
[07:35] <mborzecki> zyga: fwiw, do you remember what distro is used for the workers/
[07:35] <mup> PR snapd#9364 opened: tests/lib/cla_check: default to Python 3, tweaks, formatting <Simple 😃> <Skip spread> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9364>
[07:36] <zyga> re-formatted with black I presume
[07:36] <mborzecki> zyga: yeah, the trademark () style it uses
[07:36] <zyga> +1
[07:37] <mborzecki> zyga: are you in power to install python3-launchpadlib in the containers used to run the cla workflow?
[07:37] <zyga> on mine, yes
[07:37] <zyga> one moment
[07:39] <mborzecki> zyga: who manages the other workers? that workflow runs on self-hosted (whatver that means)
[07:39] <zyga> mborzecki: claudio
[07:39] <zyga> installing now
[07:39] <mborzecki> zyga: cool, i'll ping him
[07:40] <zyga> mborzecki: self hosted are workers which are managed buy us, not github
[07:40] <mborzecki> zyga: right, i though that maybe is runs/manages some of those for us
[07:40] <zyga> that's two nodes from me and one from claudio
[07:40] <zyga> it's self-managed
[07:40] <zyga> it will be ready in a few minutes
[07:41] <mborzecki> zyga: cool, thank you
[07:55] <zyga> mborzecki: done, all workers under my control are ready
[07:56] <mborzecki> zyga: thanks
[07:56] <zyga> mborzecki: as for self-hosted workers
[07:56] <zyga> mborzecki: we could tweak actions to run cla checks on a pi easily
[07:56] <zyga> mborzecki: no need for spread or root access
[07:56] <zyga> mborzecki: just a small container with python and git
[07:57] <zyga> mborzecki: so that's entirely viable
[07:58] <zyga> nothing is running now
[07:58] <zyga> rebooting for kernel update
[07:59] <zyga> workers will be back in a minute or two
[08:00] <zyga> ok, fixed the profile bit affecting lxd
[08:00] <zyga> it's quite curious
[08:00] <zyga> MS_BIND really matters for MS_REMOUNT
[08:00] <zyga> testing again
[08:00] <zyga> all of main should pass now
[08:00] <zyga> I'll do core next and then add the feature flag I need for safety
[08:01] <zyga> and that should be it :)
[08:01] <zyga> mvo: I'll start working on enabling raa today as we talked
[08:09] <mborzecki> hm built a uc20 image and it gets stuck in the efi shell
[08:09] <mborzecki> pretty sure i'm using the same ovmf that worked the last time
[08:14] <zyga> Removing snapd (2.45.1ubuntu0.2) ...
[08:14] <zyga> Failed to execute operation: Connection timed out
[08:14] <zyga> inside containers
[08:15] <mup> PR snapd#9365 opened: release: 2.47~pre1 <Simple 😃> <Skip spread> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9365>
[08:18] <zyga> \o/
[08:24] <zyga> lxd passed
[08:24] <zyga> regression and core are in progress
[08:47] <zyga> afk, lucy needs me
[08:47] <zyga> both core and regression passed
[08:47] <zyga> just the feature to restore old behavior in case we need it
[08:48] <zyga> but that's after wife is back in a moment
[08:48] <zyga> o/
[08:48] <pedronis> mvo: I made a comment in https://github.com/snapcore/snapd/pull/9293
[08:48] <mup> PR #9293: snap: auto-import will not try to auto-create users on managed devices <Needs Samuele review> <Created by mvo5> <https://github.com/snapcore/snapd/pull/9293>
[08:48] <mvo> pedronis: \o/
[08:48] <mvo> pedronis: sounds sensible
[08:49] <pedronis> mvo: sorry, it's a bit more work, but maybe put us in a better spot
[08:49] <mvo> pedronis: yeah, totally agreed, I like this
[09:07] <mborzecki> duh, console-conf drops a backtrace when i try to configure the device
[09:08] <zyga-mbp> mborzecki physical or vm?
[09:08] <mborzecki> or snapd isn't quite up yet https://paste.ubuntu.com/p/3RZY7vmC4m/
[09:08] <mborzecki> vm
[09:09] <zyga-mbp> well, console conf should handle that error
[09:09] <zyga-mbp> and probably just retry
[09:09] <zyga-mbp> maybe we just got restarted
[09:34] <mvo> mborzecki: please file a bug and we need to put it onto the foundations radar
[09:35] <mborzecki> mvo: investigating still a bit, maybe it's snapd that's tarting slow
[09:36] <mborzecki> mvo: and seriously takes a bit of work to get a debug shell on the system :/
[09:36] <mborzecki> mvo: but i guess you already know the pain :)
[09:36] <mvo> mborzecki: yes, but it's okay to just file a bug
[09:37] <mborzecki> uhh, mark boot successful fails, `cannot identify kernel snap with bootloader grub: cannot read dangling symlink kernel.efi`
[09:37] <mvo> mborzecki: and not dive too deep, if it's really a matter of retrying. but sorry, in a meeting so only 20% of my brain is available
[09:38] <mvo> mborzecki: this is not the issue that ian identified the other day that snap-bootstrap older versions eat modeenv keys?
[09:38] <mborzecki> yeah, omg, need to repack the kernel
[09:38] <mborzecki> ehh
[09:39] <mborzecki> i suppose i have what i wanted now anyway
[09:40] <mborzecki> mvo: a random thought, it'd be great to have at least a debug version of the snapd snap which has `systemd.debug-shell=1 dangerous` baked in
[09:43] <mvo> mborzecki: yeah, I was wondering something similar - and snapd.debug=1 too :)
[09:44] <mvo> mborzecki: maybe we should have this as part of the +testkeys or a similar new build-tag?
[09:48] <zyga> re :)
[10:36] <zyga> I've updated mount-ns test to cope with the new tmpfs
[11:22] <zyga> now added the feature flag
[11:22] <zyga> now for some tests
[11:28] <zyga> brb
[11:55] <zyga> re
[12:08] <ijohnson> mborzecki: imho the proper way to set systemd.debug-shell=1 dangerous for tests is to make gadget specified kernel cmdlines work, then when we already repack the gadget we can set the extra cmdlines from there
[12:09] <mborzecki> ijohnson: well, i'm using https://github.com/snapcore/snapd/commit/a1ec67e26bc5fde9ef645bcc8d6e127345cee344 for now ;)
[12:09] <mborzecki> ijohnson: but yeah, i'd prefer the gadget way
[12:19] <mborzecki> login.ubuntu.com is down?
[12:19] <mborzecki> cannot create users in a UC20 vm
[12:20] <zyga> WFM
[12:22] <mborzecki> ehh, dropped too many modules when repacking the kernel
[12:26] <mup> PR snapd#9366 opened:  gadget: resolve device mapper devices for fallback device lookup <Run nested> <UC20> <Created by bboozzoo> <https://github.com/snapcore/snapd/pull/9366>
[12:27] <pedronis> zyga: any news on google:ubuntu-16.04-64:tests/main/lxd:snapd_cgroup_neither ?
[12:27] <zyga> pedronis: oh? is that broken?
[12:27] <zyga> I ran it today several times
[12:28] <pedronis> it failed here: https://github.com/snapcore/snapd/pull/9270/checks?check_run_id=1123940072
[12:28] <mup> PR #9270: wrappers, systemd: allow empty root dir and conditionally do not pass --root to systemctl <Run nested> <Services ⚙️> <⛔ Blocked> <Created by stolowski> <https://github.com/snapcore/snapd/pull/9270>
[12:28] <zyga> let me look
[12:29] <zyga> ah, wait, is that the issue we discussed recently
[12:29] <zyga> I remember now,
[12:29] <cmatsuoka> zyga: I had multiple errors in that test yesterday
[12:29] <zyga> it's weird, I did see the failure messages but in my case the test ultimately passed (it was slow though)
[12:30] <zyga> I'll check with stgraber about it and re-check with -shell-after, to see if it really passed for me
[12:39] <pstolowski> pedronis: there is no api to append to zip in Go... but gess what, there is an old juju package from rogpeppe to do this.. https://godoc.org/github.com/juju/zip#Reader.Append
[12:39] <pedronis> pstolowski: mmh
[12:39] <pstolowski> pedronis: it's a fork of stdlib though ...
[12:39] <pstolowski> and old
[12:40] <pedronis> pstolowski: we need to chat again
[12:40] <pstolowski> for context: https://groups.google.com/g/golang-dev/c/ZSubqlF2G4k?pli=1
[12:40] <pstolowski> and https://github.com/golang/go/issues/15626
[12:40] <pstolowski> it got nowwhere in stdlib of go
[12:46] <pedronis> pstolowski: I also I was confused about something
[12:46] <pedronis> pstolowski: anyway let's chat again
[12:46] <pstolowski> pedronis: ok, now/later?
[12:47] <pedronis> pstolowski: as you prefer, but not a lot of time before standup
[12:47] <pstolowski> pedronis: ok, after then
[13:01] <mup> PR snapd#9364 closed: tests/lib/cla_check: default to Python 3, tweaks, formatting <Simple 😃> <Skip spread> <Created by bboozzoo> <Merged by anonymouse64> <https://github.com/snapcore/snapd/pull/9364>
[14:01] <mborzecki> hmm, maybe we still need to improve error handling in nested prepare: https://paste.ubuntu.com/p/FBBRKSKBGH/
[14:01] <ijohnson> huh
[14:01] <ijohnson> if the snap command is available why would it be connection refused talking to the socket
[14:02] <ijohnson> I guess sudo snap wait system seed.loaded in the nested_retry_until_success with a much shorter timeout on that
[14:06] <mborzecki> ijohnson: could it be that this ran after the socket was created by system, but before snapd.service started listening (or the service started)?
[14:07] <ijohnson> mborzecki: but I thought that was the whole point of socket activation
[14:08] <ijohnson> is that the stuff sent on the socket gets queued and "saved" so that when snapd is ready and starts listening using systemd's fd we just get what was sent before snapd finished starting
[14:10] <mup> Bug #1887153 changed: Installing network-manager drops the network <Snappy:Invalid> <snappy-hwe-snaps:New> <https://launchpad.net/bugs/1887153>
[14:13] <mup> Bug #1887153 opened: Installing network-manager drops the network <Snappy:Invalid> <snappy-hwe-snaps:New> <https://launchpad.net/bugs/1887153>
[14:19] <mup> Bug #1887153 changed: Installing network-manager drops the network <Snappy:Invalid> <snappy-hwe-snaps:New> <https://launchpad.net/bugs/1887153>
[14:51]  * cachio lunch
[15:27] <mup> PR snapd#9365 closed: release: 2.47~pre1 <Simple 😃> <Skip spread> <Created by mvo5> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/9365>
[15:47] <mup> PR snapd#9367 opened: tests: misc nested changes <Run nested> <Simple 😃> <Test Robustness> <Created by anonymouse64> <https://github.com/snapcore/snapd/pull/9367>
[15:47] <ijohnson> hmm does anybody remember the version of snapd we supported parallel installs of classic snaps?
[15:51] <ijohnson> degville: when you have time could you review my update to https://forum.snapcraft.io/t/parallel-installs/7679 mentioning classic parallel snap support in 2.43+ ?
[15:52] <ijohnson> hopefully I didn't botch it too badly :-)
[15:53] <degville> ijohnson: yes, of course! Looks great - thanks for updating it.
[15:53] <ijohnson> thanks
[16:00] <zyga> woot, new test passes
[16:21] <zyga> a few more variants to cover everything
[16:21] <zyga> I'll go make tea
[16:21] <zyga> I like days like this
[16:21] <zyga> quiet focus
[16:43] <cachio> mvo, 2.47 is comming today, right?
[16:54] <zyga> -debug time
[17:22] <ijohnson> cachio: could you look at my 9364 again, it seems my change to the debug: section for each suite didn't take when the nested tests failed, so I force pushed a different change which should work now I think
[17:23] <cachio> ijohnson|lunch, sure
[17:23] <ijohnson|lunch> thanks
[17:24] <cachio> ijohnson|lunch, it is already merged
[17:24] <ijohnson|lunch> cachio: sorry I meant 9367
[17:24] <cachio> ijohnson|lunch, hehe, ok
[17:49] <mvo> cachio: it's already in beta - at least the snapd snap
[17:50] <mvo> cachio: systemd is yet again hold in the review queue
[17:50] <cachio> mvo, ok, I'll start with snapd, thnanks
[17:51] <mvo> cachio: great, thank you
[17:51]  * zyga starts to prepare commits
[18:57] <zyga> ok, one more case fixed
[18:57] <zyga> this test acts as extra documentation
[19:11] <zyga> still some failures on some systems
[19:17] <zyga> centos/fedora/amazon
[19:18] <zyga> and core
[19:18] <zyga> let's dig at the classic set first
[19:18] <mup> PR snapcraft#3289 opened: build providers: fix issues running on Windows <Created by sergiusens> <https://github.com/snapcore/snapcraft/pull/3289>
[19:19] <zyga> but hey 29 / 40 passed :)
[19:27] <cmatsuoka> ijohnson: did you try to build and install an edge system recently?
[19:27] <ijohnson> cmatsuoka: yes, many
[19:27] <ijohnson> cmatsuoka: let me read your mind
[19:27] <ijohnson> cmatsuoka: it's broken isn't it?
[19:27] <ijohnson> :-)
[19:27] <cmatsuoka> ijohnson: hum, I just tried it and it failed to install
[19:27] <cmatsuoka> ah ok
[19:28] <ijohnson> the issue is that we need a kernel snap respin
[19:28] <ijohnson> ubuntu-core-initramfs was updated with new snap-bootstrap
[19:28] <ijohnson> but the kernel snap has not been rebuilt yet
[19:28] <cmatsuoka> ah I see
[19:28] <ijohnson> so edge is currently broken
[19:28] <cmatsuoka> ah I thought it was published already
[19:28] <cmatsuoka> ok
[19:29] <ijohnson> not yet
[19:30]  * zyga takes the dog out
[19:31] <cmatsuoka> my idea was to build a edge image with maciek's gadget update snapd branch and try it with the edition3 gadget
[19:31] <cmatsuoka> and see if it reseals correctly
[19:32] <ijohnson> yeah you'll need to repack the initramfs with the new version of ubuntu-core-initramfs
[19:33] <cmatsuoka> do we have any estimate on when the kernel snap will be ready?
[19:33] <ijohnson> no idea
[19:43] <zyga> okay
[19:44] <zyga> back and looking at the debug prompt
[19:48] <zyga> huh
[19:48] <zyga> looks like real byg
[19:50] <zyga> printf debug time
[20:06] <zyga> mmm
[20:22] <cachio> ijohnson, cmatsuoka I found something really insteresting about the issue on kvm
[20:22] <cachio> if I use the host system = debian 9
[20:23] <cachio> no reboots
[20:23] <cmatsuoka> oh
[20:23] <cmatsuoka> that's interesting
[20:23] <cachio> the only diff is that on debian 9 I coundt make tpm work
[20:23] <zyga> ok, interesting
[20:23] <zyga> debugging
[20:23] <cachio> I need to take a look to that
[20:23] <zyga> but looks like forced host tools
[20:24] <zyga> but why, maybe some silly bug
[20:24] <cachio> but tmp does not affect on the reboots
[20:24] <zyga> anyway, thank for integration tests
[20:24] <zyga> I'll know on the next cycle
[20:24] <cachio> I tested ubuntu without tmp and ti reboots
[20:25] <cachio> so perhaps a solution for us is to replace ubuntu focal by debian 9
[20:25] <zyga> cachio: debian 9 has really old qemu
[20:25] <zyga> perhaps it doesn't do something fancy enough
[20:25] <cachio> zyga, I am gonna try usign the same qemu on focal
[20:25] <zyga> or doesn't interact with recent security mitigations
[20:26] <cachio> zyga, both things could be the problem
[20:27] <ijohnson> cachio: but if you can't make tpm work with debian then it seems not useful since we need tpm
[20:27] <cachio> zyga, I'll try with debian sid
[20:27] <cachio> ijohnson, that qemu does not support tpm emulation
[20:27] <ijohnson> cachio: right but my point is that we need nested vm's with tpm to test uc20
[20:28] <cachio> ijohnson, yes yes agre
[20:28] <ijohnson> and afaik the reboots only happen with tpm
[20:28] <cachio> ijohnson, no
[20:28] <ijohnson> but yes anyways very interesting point about using debian 9
[20:28] <cachio> ijohnson, reboots happen when we use ovmf
[20:28] <cachio> independently if we use tpm
[20:29] <cachio> I tried removing tpm and the reboots still happen
[20:29] <ijohnson> cachio: ah I see
[20:29] <cachio> ijohnson, I'll try now with debian sid
[20:30] <cachio> it should support tpm by software
[20:30] <cachio> if sid fails the problem is in qemu/kvm version used
[20:32] <ijohnson> Yeah good find nonetheless
[20:32] <cmatsuoka> sid has qemu 5.1 I believe
[20:34] <mup> PR snapcraft#3290 opened: build providers: unified provider refactoring for provider setup <Created by cjp256> <https://github.com/snapcore/snapcraft/pull/3290>
[20:34] <cachio> I am creating a new image to support virtualization on sid
[20:34] <cachio> let see
[20:39] <zyga> ah
[20:39] <zyga> what is /etc/sysconfig/snapd:SNAP_REEXEC=0
[20:39]  * zyga looks
[20:41] <zyga> ah
[20:41] <zyga> packaging/fedora/snapd.spec:echo
[20:41] <zyga> why?
[20:42] <zyga> at least that explains it
[20:43] <zyga> the test needs adjusting
[20:58] <zyga> if this passes the next error is on core
[20:59] <zyga> I'll keep pushing, open the PR and go to sleep
[20:59] <zyga> but I may be a bit tired tomorrow
[21:04] <mup> PR snapcraft#3289 closed: build providers: fix issues running on Windows <Created by sergiusens> <Merged by sergiusens> <https://github.com/snapcore/snapcraft/pull/3289>
[21:04] <zyga> fedora passed
[21:04] <zyga> ok, core
[21:08] <zyga> need to stretch back
[21:08] <zyga> brb
[21:16] <zyga> back with water,
[21:16] <zyga> core is rebooting
[21:16] <zyga> I'll know soon
[21:20] <zyga> ah
[21:20] <zyga> hh
[21:20] <zyga> ok, weird
[21:20] <zyga> let's check
[21:20] <zyga> I think the test has a mistake
[21:25] <cachio> ijohnson, cmatsuoka, zyga fails on debian sid
[21:25] <cachio> it reboots
[21:25] <zyga> mmm
[21:25] <cachio> so the qemu version
[21:26] <cachio> I need to find a version with supports tpm and no reboots
[21:26] <cmatsuoka> is it using kvm or just software qemu?
[21:26] <cachio> kvm
[21:27] <cmatsuoka> if using kvm it should depend on the kernel too
[21:27] <cachio> cmatsuoka, in tha tpast I tried with a very old kernel
[21:27] <cachio> and it rebooted
[21:28] <cachio> an old kernel which supported kvm
[21:28] <cachio> I think I should find a version of qemu which works
[21:30] <zyga> I think we should ask some qemu people
[21:33] <cmatsuoka> our case is very specific even for qemu people (nested with ovmf)
[21:34] <cmatsuoka> the last time I asked some kvm engineers they said they never worked with that specific scenario :|
[21:53] <cachio> cmatsuoka,just ran in xenial and no reboots
[21:53] <cachio> qemu 2.5
[21:54] <cmatsuoka> hum
[21:54] <cmatsuoka> and does it work with tpm?
[21:54] <cachio> no
[21:55] <cachio> now I'' try to find which version works with tpm
[21:55] <cachio> I nee dto go now
[21:55] <cachio> I'll continue after I am back
[21:55] <cachio> need to buy some stuff
[21:57] <zyga> figured it out
[21:57] <zyga> and documented
[21:57] <zyga> man so many sharp edges
[21:58] <zyga> restarting another run
[21:58] <zyga> all systems now
[22:11] <zyga> ok
[22:11] <zyga> I bail
[22:11] <zyga> tomorrow
[22:11] <zyga> see you all