[00:03] PR snapcraft#3323 opened: electron-builder spread test: sync expected snapcraft.yaml [00:13] PR snapcraft#3311 closed: storeapi: add releases endpoint [00:18] PR snapcraft#3306 closed: storeapi: add support for reporting status of progressive releases [01:28] PR snapcraft#3323 closed: electron-builder spread test: sync expected snapcraft.yaml === benfrancis9 is now known as benfrancis [06:05] morning [06:39] good morning [07:04] morning [07:05] pstolowski: good morning [07:13] hey mvo [07:13] good morning zyga [07:14] sorry for the lag, a bit ucooperative kids today [07:14] *uncooperative [07:14] zyga: no worries [07:27] mborzecki can you approve https://github.com/snapcore/snapd/pull/9499 - it'd be my most approved change ever ;-) [07:27] PR #9499: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too [07:27] also reviewing ijohnson's fsck change for uc20 would be really good [07:35] I've de-conflicted and updated https://github.com/snapcore/snapd/pull/9422 [07:35] PR #9422: overlord: add link participant for linkage transitions [07:35] I hope it's in a good shape now and we can make progress on the export manager [07:35] zyga: hm am i missing something? https://github.com/snapcore/snapd/pull/9499#discussion_r508273373 [07:35] PR #9499: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too [07:35] whooops [07:35] how did I do that [07:35] sorry [07:36] drat [07:36] I understand what my editor asked me to save now [07:36] meh [07:36] I resolved and closed without saving [07:37] ushed [07:37] pushed [07:49] pedronis good morning [07:49] pedronis I've updated https://github.com/snapcore/snapd/pull/9422 [07:49] PR #9422: overlord: add link participant for linkage transitions [07:50] pedronis I think it should be good now, at least in principle [07:51] thx, I put it back into my queue [07:53] lxd snap was fixed yesterday, #9468 is unblocked [07:53] PR #9468: tests: lxd smoke test [07:56] pedronis thanks :) [07:56] * zyga warmed up the tea [07:56] it's still very cold in the office, heating up slowly [07:57] pstolowski reviewed [07:59] I'll work on the pi kerenl thing now [08:18] zyga: thanks, adding some comments. not sure about restore, snap remove is the main check, i think the new comments will make it clear [08:50] pedronis: i've updated #9476 with some additional tweaks to make nosecboot happy [08:50] PR #9476: many: have install return encryption keys for data and save, improve tests [09:19] google:ubuntu-20.10-64:tests/main/snapd-reexec-snapd-snap fails on something silly, probalby assumes core is installed [09:19] 20.10 has snapd preinstalled [09:19] should we disable that test there? [09:19] mvo ^ [09:19] zyga: in a meeting right now [09:19] k [09:20] zyga: sorry, can think in a bit but not right now [09:20] ok [09:21] PR snapd#9499 closed: tests: add tests for fsck; cmd/s-b/initramfs-mounts: fsck ubuntu-seed too [09:21] PR snapd#9522 opened: o/snapstate: ignore remove errors in clear-snap handler, only log them [09:23] pstolowski: check my comment please [09:34] pedronis: hi, if you have a moment, what's your take on warnings vs task log in #9522? [09:34] PR #9522: o/snapstate: ignore remove errors in clear-snap handler, only log them [09:44] booting rogrer's configuration now [09:44] gosh I love serial lines [09:44] is this a bug: /init: /scripts/init-bottom/ORDER: line 3: /scripts/init-bottom/disable-getty: Permission denied [09:44] ? [09:44] that's from initrd [09:44] seems like missing +x [09:45] second error: /usr/share/subiquity/console-conf-wrapper: line 13: snap: command not found [09:47] third error (trying to use wifi) https://pastebin.ubuntu.com/p/qXbB66tfjz/ [09:52] hmmm [09:52] I cannot get past this [09:53] mborzecki do you recall this subiquity erorr ^ / [09:53] zyga: thanks for finishing off the snapd/apparmor3 stuff :) [09:54] amurray you were right after all :) [09:55] hmm, I wonder if subiquity will refresh [09:57] does anyone know if there's something I can set in uboot to bypass subiquity on core18? [09:58] hmmmm [09:58] I think the snapd snap I'm getting crashes on startup [09:58] [FAILED] Failed to start /tmp/tmp.spBbI7OTQH/usr/lib/snapd/snapd. [09:58] eh [09:58] pstolowski ^ any ideas? [09:58] ideally on how to debug any of this [10:00] zyga: is this a private build of snapd? [10:01] pstolowski nope [10:01] pstolowski ubuntu-image with the three snaps I got from mvo [10:01] and snapd downloaded from the store (automatically) [10:01] zyga: still in a meeting but i can give you more snaps if you need them [10:02] mvo: I think whatever is wrong here also is interesting [10:02] zyga: maybe mount the image and see if the snaps looks sane ? [10:02] yeah, trying now [10:02] seems like something fundamental failed [10:06] I think snapd did not seed [10:06] I've enabled journal [10:06] let's see what we get [10:07] mborzecki: do you have a moment for #9468? [10:07] PR #9468: tests: lxd smoke test [10:08] pstolowski: sure, will take a look [10:08] huh [10:08] Oct 20 10:05:57 localhost run-snapd-from-snap[607]: /usr/lib/core18/run-snapd-from-snap: 42: /usr/lib/core18/run-snapd-from-snap: /usr/bin/snap: not found [10:08] pstolowski I'd like to "reset" seeding [10:09] can I just remove state json? [10:09] hmmm [10:09] wait there is no state.json [10:09] I only see this [10:09] https://pastebin.ubuntu.com/p/MCX46Dnsf2/ [10:09] pstolowski note that the snaps/* are symlink [10:09] *symlinks [10:09] and there are only two [10:12] zyga: try talking to ondra, he probably has some hacks for postprocessing the image to add users etc [10:13] I've added a ssh key for root [10:13] should be enough [10:14] pstolowski: does the test need to launch a container? [10:14] yes, symlinks are expected, this is what makes runnable on first boot, but i'm not sure why snapd isn't symlinked; indeed it seems seeding did't run [10:15] mborzecki: no [10:15] mborzecki: it was failing pretty reliably with these steps [10:15] ok [10:16] I'm in [10:16] fuuuck [10:16] mborzecki: thanks! [10:16] PR snapd#9468 closed: tests: lxd smoke test [10:16] exec format error [10:16] ubuntu image pulled x86-64 snapd!??! [10:17] woot [10:17] no sorry [10:17] well [10:17] I have arm7l kernel [10:17] and aarch64 snapd [10:17] that's fun [10:18] my fault [10:21] booting fixed image [10:21] eh [10:21] :D [10:24] reported https://github.com/snapcore/pi-gadget/issues/53 [10:30] I went for coffee [10:30] and the device is updating and rebooting [10:30] but still no console-conf [10:31] I'll keep it running [10:31] weird [10:32] it's progressing [10:32] it's mounting snapd now [10:33] hmmm [10:33] weird [10:33] https://pastebin.ubuntu.com/p/jpPTtCDsc8/ [10:33] this is what I got on the serial log so far [10:33] I've hit newline a few times when it took a long time to make any progerss [10:34] all this time no console conf [10:34] more getty changes [10:35] hmmm hmm [10:35] I wonder if this makes any sense to anyone [10:36] getty changes again [10:36] I'll reboot [10:36] this makes zero sense [10:37] ok, I have console conf now [10:37] and it crashes like before [10:37] heh [10:37] WTF [10:37] * zyga debugs [10:40] the system cannot seed [10:40] * zyga looks at why [10:43] ok, I'll enable journal and ssh and try again [10:43] there's nothing I can see that explains what failed [10:43] state.json doesn't have any useful messages [10:46] trying again [10:53] ok, I'm in while it's still trying to seed [10:54] snapd is not seeded correctly again [10:55] this is so weird [10:56] huh [10:56] pstolowski [10:56] Oct 20 10:56:31 localhost snapd[750]: taskrunner.go:271: [change 2 "Make snap \"snapd\" (9731) available to the system" task] failed: open /etc/dbus-1/session.d/snapd.session-services.conf.7PTDJQPMf [10:56] it fails on this! [10:57] PR snapd#9523 opened: store, snap-repair: fix use of retry strategies [10:57] right, because the fixed snapd is only later [10:57] (edge) [10:57] huh [10:57] but wait [10:57] with old core18 it should be fine? [10:57] btw [10:57] but this is old core18 and new snapd [10:57] only a problem once you refresh no? [10:57] btw, offtopic, why do we have to seeding services? [10:57] snapd-seeding.service loaded active running /tmp/tmp.jFHQH5ZiJr/usr/lib/snapd/snapd [10:57] snapd.seeded.service loaded active exited Wait until snapd is fully seeded (core18) [10:57] perhaps we should give them better names (cc mvo) [10:58] "two"? [10:58] mvo: i've set 2.47 milestone on #9523 [10:58] PR #9523: store, snap-repair: fix use of retry strategies [10:58] pstolowski one with dot, one with dash [10:58] pstolowski snapd-seeding and snapd.seeding [10:59] zyga: yeah, got it, you said "to", wasn't sure [10:59] ah, sorry [10:59] switching between macbook and hacker keyboard [10:59] mind confused :) [10:59] ? one is "seeded" not seeding [11:00] ah [11:00] that's right [11:02] they do very different things [11:02] yes, just have inconsistent naming [11:04] Oct 20 11:03:20 localhost systemd[1]: snapd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functional until restarted. [11:05] snap tasks NNN is super slow due to locks [11:05] it's not a good experience :/ [11:05] it's practically unusable when there are changes in progress downloading snaps [11:06] pstolowski: thank you [11:07] zyga: still in meeting :/ but +1 to anything that adds clarity [11:07] mvo hard to change that [11:07] * mvo has not read backlog :/ [11:08] managed to get snap tasks after several minutes of waiting [11:08] refreshing core18 explicitly [11:08] let's see if we learn something [11:10] I've set refresh time now [11:11] so no auto updates for a while [11:11] I'll get core18 refreshed in a few minutes [11:11] with a bit of luck that will show what was failing for Roger [11:16] here we go, rebooting [11:17] [ 1839.110777] systemd-shutdown[1]: Failed to wait for process: Protocol error [11:17] ha [11:17] reproduced [11:17] pstolowski, mborzecki [11:18] mvo I've reproduced the failure on that pi [11:18] yay [11:18] zyga: so old kernel/gadget? [11:19] check this out [11:20] https://pastebin.ubuntu.com/p/JVW6c3kZK2/ [11:21] mborzecki but before rebooting [11:21] there was a message about system-shutdown [11:21] it seems it really did fail [11:21] and now fat is fcks up [11:21] zyga: \o/ you rock! [11:21] let me plug the device back [11:22] zyga: you think that boot may be corrupted? [11:22] ha [11:22] fuck yeah [11:22] two power cycles [11:22] and it's back [11:22] I'm not kidding [11:22] this is exactly what happened on that other pi [11:22] I'll paste the more complete log now [11:23] https://pastebin.ubuntu.com/p/hmYzXHW4rf/ [11:24] so now, some research time [11:25] I'll install core to have fsck available [11:27] https://www.raspberrypi.org/forums/viewtopic.php?t=242510 [11:27] hm what mounts /run/mnt/ubuntu-seed on /var/lib/snapd/seed? [11:28] mborzecki dunno [11:28] core20 bit for sure [11:30] haha :P [11:31] yeah, this is totally reproducible [11:31] core18 cannot refresh, we shutdown uncleanly [11:31] w[FAILED] Failed to start Ubuntu core (all-sn…tem shutdown helper setup service. [11:31] See 'systemctl status snapd.system-shutdown.service' for details. [11:32] hmm [11:33] I'll report a detailed bug capturing this [11:33] I think core18 is good later [11:33] but not yet [11:33] so i mount ubuntu-save under /run/mnt/ubuntu-save, and bind mount it under /run/data/system-data/var/lib/snapd/device/save [11:33] remember this is old coe18 [11:33] mborzecki it's a propagating mount [11:34] quite sure there's some rbind missing somewhere [11:35] probably sysroot-writable.mount (in core20-initrds aka ubuntu-core-initramfs), or/and handle-writable-paths [11:35] but i see /var/lib/snapd/seed is mounted, and maybe we could do the same instead, not sure though what mounts it there [11:39] zyga: amazing stuff! [11:39] https://bugs.launchpad.net/snapd/+bug/1900693 [11:39] Bug #1900693: snapd cannot refresh on some SD cards due to uboot bug [11:39] mborzecki: is done here I think (via fstab): https://github.com/snapcore/core20/blob/master/static/usr/lib/core/handle-writable-paths#L183 [11:39] ok, I'll try to update uboot now [11:40] pedronis: thanks! [11:41] mborzecki who is our PI maintainer? [11:41] heh, noticed it's at the end of the file and was looking for something that may modify fstab after populate-writable runs [11:41] zyga: try poking waveform [11:41] thanks [11:42] I suspect we could use a repair assertion [11:43] well, we can see [11:43] dang, i'm not sure my diagnosis re retry strategies was correct [12:01] meh, shellcheck is unhappy about handle-writable-paths [12:10] mborzecki, hey [12:10] about the tumbleweed image [12:10] cachio: hey [12:11] mborzecki, this is the script we use to create that image [12:11] https://github.com/snapcore/spread-images/blob/master/tasks/google/add-opensuse-tumbleweed-64/task.yaml [12:11] mborzecki, what could we do about that to make tumbleweed work better? [12:11] cachio: we start from leap? [12:12] mborzecki, yes opensuse-15.2 [12:13] mborzecki, I am importing this image opensuse-cloud/opensuse-leap-15-2-v20200702 [12:13] cachio: i guess there's no tumbleweed image we could use? [12:14] mborzecki, no [12:14] let me check again [12:14] just to make sure [12:17] mborzecki, 15 15.1 and 15.2 [12:17] FYI I asked in #opensuse-admin [12:18] that sucks a bit [12:18] I asked sysrich about this a while ago [12:18] and he did produce a link to TW images [12:18] but said they are not QAd [12:18] hmm [12:19] i mean, upgrading from a stable release is supposed to be supported so maybe we just need to figure out which pacakges are incorrect? [12:19] zyga: got link? [12:19] searching [12:19] cachio: when was the last time you built an image? [12:19] https://twitter.com/zygoon/status/1098257262102106112 [12:19] my memory was rusty [12:20] mborzecki, the current tumbleweed is opensuse-tumbleweed-64-v20200923 [12:20] and the base we use opensuse-tumbleweed-64-base-v20200831 [12:20] so 1 month ago is the last one [12:21] cachio IIRC the problem was that it had a leap kernel [12:21] and was some sort of frankenstein that should not be needed if we had a real TW system [12:21] (real = vanilla) [12:21] zyga, yes [12:21] so i ran zypper dup and it wants to install this kernel: `kernel-default 5.8.14-1.2 x86_64 repo-oss openSUSE` [12:22] which is tw [12:23] cachio: so maybe the image build should actually run more often (weekly?), and run zypper dup once more after the first reboot? [12:23] it runs weekly [12:24] but it is failing with differnet errors [12:24] I can trigger a new one [12:24] maybe we could work through those errors [12:24] are you talking about test failures? [12:24] or about upgrade failures [12:25] zyga, upgrade failures [12:25] some incompatibilities with google packages [12:25] etc [12:25] this is the kernel in tumbleweed now [12:25] 5.3.18-lp152.19-default [12:25] pstolowski: upgrading the kernel is equally futile [12:26] pstolowski and as I discussed with maciek, upgrading the gadget is impossible now [12:26] mvo: pi is busted :P [12:26] mvo: needs some coordination to unbreak [12:27] zyga, mborzecki this is the last error upgrading [12:27] https://paste.ubuntu.com/p/sbNgmZgyrH/ [12:27] zyga: that's really terrible [12:28] zyga: maybe add your findings under the original bug from Roger? [12:28] yeah, I will, still looking at some ideas [12:34] cachio: hm not sure why you're seeing those, i'm running zypper dup on the tw image we have and it seems tow ork [12:35] mborzecki, I think it is because we force the leap 15.2 kernel [12:35] cachio: why do we have that? [12:35] mborzecki, cachio: https://download.opensuse.org/tumbleweed/appliances/ [12:35] this is because of a bug [12:35] I think we want to try https://download.opensuse.org/tumbleweed/appliances/openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2 [12:36] mborzecki, https://github.com/snapcore/spread-images/blob/master/tasks/google/update-opensuse-tumbleweed/task.yaml [12:36] it used to get stuck when doing govendor sync [12:36] so we had to force that kernel [12:37] cachio: hm can we try with the current kernel? [12:37] mborzecki, yes [12:55] mborzecki, in few minutes the new image will be available [13:38] cachio: perhaps missing call to lxd cleanup [13:52] * zyga breaks for lunch [13:52] cachio I can talk later [13:52] zyga, sure, I am reviewing the code now [13:52] after lunch I will fix xdg-settings [13:52] thanks [13:52] the test fails for real with app tracking on fedora [13:52] but probably for good reasons [13:59] off to pick up the kids [14:20] * zyga is back and looks at xdg-settings [14:20] PR snapcraft#3322 closed: package repositories: drop $SNAPCRAFT_APT_HOST_ARCH variable