[00:21] PR snapcraft#3329 opened: cli: update revisions to use releases API [05:28] morning [06:06] PR snapd#9476 closed: many: have install return encryption keys for data and save, improve tests [06:37] mvo: hey [06:38] hey mborzecki [06:38] good morning [07:02] pstolowski: hey [07:03] hi [07:05] good morning pstolowski [07:05] hey mvo [07:05] mborzecki: anything I should review for you? I'm doing zygas 9446 right now [07:06] mvo: no thanks, the reelvant prs have landed and i haven't opened anything new yet [07:06] mborzecki: cool [07:10] good morning everyone [07:14] zyga: goooood morning [07:14] indeed, it's a morning to be good on [07:15] zyga: heya [07:15] hey hey [07:15] I'm looking at tumbleweed failures [07:15]

[07:15] zyga: more of those? [07:15] on quiet govendor sync [07:16] I'll debug interactively in a moment [07:16] zyga: that most likely is the kernel panic again [07:16] heh [07:16] fun [07:16] maybe kernel modules and kernel itself are skewed? [07:25] zyga: idk, last time the panic susgested problems with fs [07:25] hmm [07:26] I'll know soon [07:46] mborzecki passed without hanging or crashing [07:47] zyga: make sure there are no packages in the vendor directory (whcih should be the default with the changes to spread.yaml in master) [07:47] oh, good point [07:48] running again [08:18] huh, i think the download stalling issue is simply due to missing timeout, we create http.Client with default timeout 0 there, for reference https://golang.org/src/net/http/client.go?s=3300:3996#L97 [08:21] pstolowski: hm, timeout is not easy because we just don't know how long it might take :/ [08:24] mvo: pstolowski: will have to think [08:25] mvo, pedronis i'd hope that this is smart enough to timeout of no response from transport, and otherwise be not affected by actual download time; func (b *cancelTimerBody) Read(p []byte) might be relevant there [08:25] zyga: 9446 has a review [08:25] ohh [08:25] thanks [08:26] pstolowski: no, see the definition of timeout in the pkg, at least that timeout is not like that [08:26] pstolowski, pedronis I guess we would need something like "timeout-for-no-new-data-from-the-remote". not sure if that is available in go(?) [08:27] mvo: pstolowski: we need to think a bit [08:27] pedronis: sorry, timeout in which pkg? [08:27] ok [08:27] pstolowski: I mean, look at the doc of Client.Timeout [08:28] mvo: I'll go over the feedback after uboot [08:29] fwtw I found retry logic fine and tried with separate standalone code that uses same combo of strategies as snapd, and hits limit on every strategy correctly. so getting stuck in io.Copy is my main suspect now [08:30] ah, "... and will interrupt reading of the Response.Body". ok, I think i misunderstood the meaning of this [08:36] pedronis: some ideas here https://medium.com/@simonfrey/go-as-in-golang-standard-net-http-config-will-break-your-production-environment-1360871cb72b - see Variable timeout section, with ltime.AfterFunc, a loop around CopyN and timer.Reset that pushes timer back [08:38] pstolowski: yea, let's discuss after standup or something, basically the issue is that using io.Copy for something that can take 10s of minutes was/is probably a bit naive [08:42] PR snapd#9525 opened: packaging: make sure that static binaries are indeed static, fix openSUSE [08:50] pedronis: a peek at 9373 would be good for uc1.0. it's a bit of a extra but it would be nice if console-conf could have the ability to show the recovery key. but your earlier comments made me wonder if this should be a proper API in snpad instead of a client command. [08:52] PR snapd#9523 closed: store, snap-repair: fix use of retry strategies <⛔ Blocked> [08:52] mvo: the blocker there is more that we need a bit more progress on save to know where exactly things are [08:53] pedronis: oh, good point [09:57] PR snapd#9525 closed: packaging: make sure that static binaries are indeed static, fix openSUSE [10:32] PR snapd#9526 opened: snapshotstate: add clenup of abandonded snapshot imports [10:39] ok [10:52] pedronis: mvo: we may need to have larger ubuntu-save to be able to encrypt it [10:53] cryptsetup open complains about `Requested offset is beyond real size of device /dev/vda4`, probably something about the backup header [11:03] mborzecki: yes, moderm LUKS default use quite a bit of space [11:04] pedronis: ran luksFormat with --debug-json, if i'm not mistaken, just the keyslot area is 7MB [11:11] mvo I've managed to build uboot from the old tree [11:11] with the patch [11:11] going to boot it now [11:14] mborzecki: there's --luks2-metadata-size and --luks2-keyslot-size , but the doc are not very clear what are reasonable values [11:14] pedronis: the docs aren't very clear at all :P [11:14] pedronis: this is what i'm looking at currently: https://gitlab.com/cryptsetup/LUKS2-docs/blob/master/luks2_doc_wip.pdf [11:15] mborzecki: afaik that one hasn't been updated to the larger defaults [11:16] in 2.1 they: The default size of the LUKS2 header is increased to 16 MB. [11:16] but that doc still uses the smaller old values iirc [11:22] PR snapd#9527 opened: o/snapstate: implement undo handler for unlink-snap [11:30] pstolowski tiny feedback on your branch [11:47] zyga: yes, saw that, thanks! [11:47] +1 [11:50] zyga: hmm, did you mean boot.Participant ? [11:51] pstolowski no no [11:51] sorry I meant link participant [11:51] pstolowski merge master, there's a new call in all link* methods [11:51] near the end [11:51] aah [11:52] zyga: righ, I can see it now [11:54] mvo: I was thinking, snapd could perform this test as a part of seeding, at least on pi, we could try to reboot and see the bootloader increment the boot counter [11:54] mvo then we'd known instantly [11:54] and not at the next rare kernel update [11:54] or base update [12:05] won't the kernel cache get in the way of naive testing? [12:06] ah, you said try to reboot [12:06] yeah [12:06] pedronis there's one bit missing that is essential [12:07] pedronis I had a call with mvo where I described an extension to our boot protocol [12:07] pedronis where apart from setting snappy_mode, the compatible bootloader would operate on a snapd-provided cookie [12:08] pedronis the goal would be to be able to detect that bootloader can or cannot write to storage even on rollback [12:08] pedronis so snappy_write_cookie=pong on snappy_write_cookie=ping [12:08] or something similar [12:08] we could be smarter but something extremely basic like that [12:08] the bootloader script could contain another hint that this is supported at all [12:09] and if we detect that it is supported but bootloader could not write, we could report to the store [12:10] I've booted my build of uboot! [12:16] zyga: to me, it's a bit unclear what's our goal here [12:16] pedronis to detect that bootloader cannot write to the storage and cannot ever upgrade certain snap types that require such writes [12:17] pedronis so that we can report this and get statistics about it, right now we are in the blind [12:17] zyga: but we don't have a way to report statistics [12:17] atm [12:17] one thing at a time [12:17] we could send error reports [12:18] it's better than nothing [12:18] we could send MMC card information as well [12:18] PR snapd#9528 opened: cmd/snap-bootstrap: mount ubuntu-save during boot if present [12:26] mvo: it's fixed [12:27] mborzecki: question/comment in #9528 [12:27] PR #9528: cmd/snap-bootstrap: mount ubuntu-save during boot if present [12:31] mvo: refreshed core18 just now [12:31] no write issues [12:32] refreshing the rest (kernel) to check if it's good in general [12:34] mvo: heh, updated kernel doesn't boot :D [12:34] hangs after starting kernel [12:34] pi and pi kernel well both following 18-pi/stable [12:35] zyga: oh no! why is it hanging? [12:42] mvo no idea, it rolls back the kernel on reboot [12:42] mvo the only message I get is "Starting kernel" [12:42] my hunch is as followw [12:42] *follows [12:42] if you build a new image using stable 18 everything [12:42] it will work [12:43] but upgrades are hitting dtb skew [12:43] and fail [12:43] I can check that easily by upgrading manually [12:43] (I'm running u-boot with fixed SD write support but otherwise matching revision 17) [12:46] refreshed gadget assets by hand, re-trying kernel update [12:51] so, the latest uboot we ship still has the write error [12:51] so it cannot make progress past that boot cycle [12:51] I'll build update u-boot with the patch and verify [12:52] but this is also interesting, it means that _no_ current uboot we have is good on this [12:52] and we can make images, and test them, that will just fail later [12:58] zyga, hangs after starting kernel usually points to mistmatch of ctb [12:58] *dtb [12:59] yeah [12:59] I'm sure that's the problem [12:59] well, you can easily test (on pre UC20 at least) by simply replacing it and moving the original one to *.bak [12:59] yeah :) [13:00] just one sec, no network ;) [13:00] standup first [13:50] zyga: what's the gadget and original kernel are you testing? pi2 pi3 ? [13:51] pedronis: this is a pi3b+ with the snaps listed in https://bugs.launchpad.net/snapd/+bug/1900693 -- pi-kernel and pi (gadget) from 18-pi/stable tracks [13:51] Bug #1900693: snapd cannot refresh on some SD cards due to uboot bug [13:52] pedronis the particular affected revisions are listed ion the bug [13:54] this is unexpected `2020-10-21 13:54:01 Cannot allocate google-nested:ubuntu-18.04-64: cannot find any Google image matching "ubuntu-1804-64-virt-enabled" on project "computeengine" or "ubuntu-os-cloud"` [14:58] mvo: https://github.com/snapcore/snapd/pull/9149#pullrequestreview-513775434 [14:58] PR #9149: gadget: provide new gadget.ResolveContentPaths() (2/N) [15:13] cachio: could you please check https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1895929 and sru validate it? [15:13] Bug #1895929: [SRU] 2.47

[15:13] [15:14] * zyga breaks for now, more later [15:14] need to look after Lucy [15:28] mvo, sure [15:34] * cachio lunch [18:19] PR snapcraft#3330 opened: package repositories: fix case where formats is empty [18:29] PR snapcraft#3331 opened: spread tests: move package-repositories test snaps into own dir [19:19] PR snapcraft#3324 closed: set ROS_PYTHON_VERSION for rosdep [19:19] PR snapcraft#3326 closed: lxd unit tests: simplify command checking pattern [20:11] cmatsuoka, ijohnson hey [20:11] I see the test uc20-create-partitions failing in 20.04 [20:11] cachio: do you have logs? [20:11] ijohnson, yes [20:12] ijohnson, https://paste.ubuntu.com/p/qC5RvpCdVv/ [20:13] since snapd isn't available for my distro, can i compile it from source? i don't know if the fedora RPMs would work well with it... [20:14] i know you can have snapd install flatpak on ubuntu.. is there a reverse way of doing that? (using flatpak to install snapd) [20:14] cachio: the test needs to be updated, there was an update to the gadget snap [20:14] RingtailedFox: what distro are you on ? [20:14] ijohnson, ah, nice, do you know which update does it need? [20:14] Mageia Linux (aa descendent of Mandrake, later, Mandriva) [20:15] cachio: let me take a quick look [20:15] it's related to fellow forks ROSA, OpenMandriva and PCLinuxOS [20:15] RingtailedFox: does your distro use systemd as it's init system ? [20:15] yes [20:15] it has since the days of Mandriva 2011.0 :) [20:16] i sometimes miss sysvinit, but systemd has matured well enough to no longer suck :P [20:16] cachio: yes I know what to change, it's a simple 1-liner I will file a PR shortly [20:16] ijohnson, nice, thanks a lot [20:16] RingtailedFox: then you should be able to compile snapd from source if you want to, not sure what kind of package management that distro uses, you mentioned something about rpms ? [20:17] yeah it uses RPM and dnf [20:17] along with urpmi [20:17] github.com/snapcore/snapd/releases/tag/2.47.1/ shows three different versions (no-vendor, only-vendor, and vendor).. which would i use? [20:17] or should i just download the source tarball and try my luck with that? [20:17] you should be able to just compile/build the snapd rpm on your machine, you probably want the vendor version [20:17] i never had much luck with compiling from source to RPM... [20:17] hmm I'm not actually sure which version fedora builds from regarding vendorization ... [20:18] yeah I'm not an expert at building rpms at all either tbh [20:18] i have gotten some programs/libraries to work in the past by using fedora RPMs, but i try not to do that unless i have no other options [20:18] * ijohnson is barely been able to build debs :-) [20:18] i do have alien installed, so i can try to import a deb and convert it to rpm [20:19] maybe that works better? I dunno I have not uesed alien either [20:19] sorry I can't be more helpful [20:19] no no, you're still helpful [20:19] i'll try to compile the vendor version and see if that works. if not, compile from source. if not that... then try to see if someone has a flatpak containing snapd :P [20:19] i still find it hilarious i was able to get flatpak onto ubuntu through snap lol [20:20] ok. so snapd-2.47.1 is extracted... it looks like it's already coimpiled into binary format [20:22] cachio: see 9529 [20:22] ijohnson, checking [20:23] +1 [20:25] PR snapd#9529 opened: tests/uc20-create-partitions: don't check for grub.cfg [20:31] thanks cachio [20:40] so what do i do with snapd-2.47.1? shove it in /usr/bin/ or /opt/? [21:29] ijohnson: in core, what systemd unit file ultimately launches snapd? [21:29] cmatsuoka: which release of ubuntu core [21:29] 20 [21:29] cmatsuoka: so after the system has been seeded that will be snapd.service which is shipped by the snapd snap [21:29] before that point, there is a systemd service file in the core20 snap which unpacks the snapd snap to run initially [21:30] let me find the name of that [21:30] cmatsuoka: it's `core.start-snapd.service` [21:31] uuhm ok, thanks, I'll have a look [21:31] it seems that the systemd keyring mode setting failed for some reason [21:32] cmatsuoka: is this for install mode or run mode? [21:32] it's run mode [21:32] and it seemed that snapd was executed by some other way and snapd.service was not used [21:33] cmatsuoka: well initially snapd in run mode for first seeding will be run from that `core.start-snapd.service` unit [21:33] so if you need to modify snapd.service, you may need to adjust that one too [21:34] but the core service should do minimal amount of work to set up the real snapd snap service, then re-exec there [21:34] thanks, I'll see what would be the best approach here [21:34] cmatsuoka: is the issue that you need to read something from systemd upon startup to save it? [21:35] cmatsuoka: if so, you might want your startup method to recognize if it's being run from core.start-snapd.service and if so then just skip that, and let the real snapd read that data [21:35] ijohnson: yes, so if this first execution doesn't attempt to reseal anything, I won't try to read the auth key [21:35] ijohnson: are you aware of a good method to detect that? [21:35] cmatsuoka: let me take a look [21:36] the way snapd bootstraps isn't exactly straightforward I see [21:42] cmatsuoka: what part of snapd uses the data that you are getting from systemd ? [21:43] in other words, what depends on it? [21:44] cmatsuoka: because what might be easiest is instead of a StartUp method on the devicemgr is to instead add a ensureSomething() that gets called as part of the devicemgr.Ensure(), and that ensureSomething checks to make sure that the device is seeded, and then gets the data it needs from systemd, because the ephemeral snapd that is started by the core.start-snapd won't ever finish seeding [21:44] I collect the data in the device manager startup and use it when resealing [21:44] pedronis may disagree with me on this however, not sure how important it is that we get this data from systemd asap after starting up [21:45] cmatsuoka: ok, so we wouldn't reseal until at least after seeding is done [21:45] so my proposal doesn't sound too terrible [21:45] Ok, it sounds interesting, let's try that [21:45] thanks Ian [21:46] np, let me know if you need any help, but for example you coudl take a look at how ensureCloudInitRestricted is implemented, that is essentially doing the same thing, waiting for snapd to be seeded then doing something [21:46] I think the idea of doing it asap is to block access to the keyring before we run too much stuff [21:47] yeah that might be the reason why my ensure proposal isn't a good idea [21:47] cmatsuoka: so after getting this data we essentially destroy it or otherwise prevent ourselves from accessing it from snapd again ? [21:48] so you didn't find a good way to determine where snapd was started from? [21:48] ijohnson: yes, after reading it the user keyring is unlinked from the systemd session keyring [21:48] so it can't be accessed again [21:50] PR snapd#9529 closed: tests/uc20-create-partitions: don't check for grub.cfg [21:51] cmatsuoka: hmm so the other thing you could do in StartUp is to check what systemd service you are running as, the ephemeral one will be under "snapd-seeding" instead of snapd [21:51] ah interesting [22:04] * ijohnson EODs [22:19] PR snapcraft#3330 closed: package repositories: fix case where formats is empty [23:04] PR snapcraft#3329 closed: cli: update revisions to use releases API [23:16] PR snapd#9519 closed: tests: moving the lib section from systems.sh helper to os.query tool