=== ivo is now known as ivo_cavalcante === ivo is now known as ivo_cavalcante === jamesh_ is now known as jamesh [06:46] morning [07:01] hi mborzecki [07:01] jamesh: hey, happy 2021! [07:34] mborzecki: could you have a look at https://github.com/snapcore/snapd/pull/9806 ? This fixes the CI after a change to one of the static check tools it runs. [07:34] PR #9806: run-checks: update to match new argument syntax of ineffassign [07:36] I guess this is another thing modules would give us: the ability to pin versions of these tools [07:36] PR snapd#9806 opened: run-checks: update to match new argument syntax of ineffassign [07:37] heh, not too happy about the change in command line argument behavio [07:39] too bad everyone will have to go get a new version now [07:39] if backward compatibility wasn't an issue, then the new syntax has the benefit of matching the "go" tool [07:40] good morning mborzecki and jamesh ! hope you had a nice break [07:40] mvo: hey, welcome back [07:41] hrm, entropy ftw, 2 weeks and things fall apart: [07:41] Checking for ineffective assignments [07:41] # golang.org/x/tools/go/analysis [07:41] ../../../golang.org/x/tools/go/analysis/validate.go:119:8: undefined: strings.Builder [07:47] So the ineffassign changes added a dep on golang.org/x/tools/go/analysis, which is uses features not found in the Go 1.9 standard library [07:53] pfff [07:53] fun [07:55] at least it reaffirms that go 1.9 is kinda obsolete at the point [07:56] I'm thinking to just disable the check on old Go [07:56] we still get the benefit from the run on latest [07:56] we could move in 2.49, will need a bit of work on our side to backport focal to the previous lts [07:56] PR snapd#9800 closed: tests: use apiBaseSuite for snapshots tests, fix import endpoint path [08:05] mvo: do you still need to care about xenial? [08:05] jamesh: unfortunately for a while for ESM [08:07] I guess it's a good thing we're not targeting vivid-overlay :-) [08:09] I think one of my favourite features of pkg.go.dev is the ability to view the documentation for old versions of a package. It's really useful in determining when a particular stdlib feature was introduced [08:12] i actually had to enable that auto redirect from godoc.org to pkg.go.dev, just can't force myself to type in that address properly [08:15] nice! wasn't aware of this even [08:15] jamesh: some naming suggestion in https://github.com/snapcore/snapd/pull/9805 best to consult it with pedronis when he's around [08:15] PR #9805: interfaces: add an optional mount-font-cache plug attribute to the desktop interface [08:16] mborzecki: thanks [08:16] mvo: can we land https://github.com/snapcore/snapd/pull/9804 ? [08:16] PR #9804: interfaces/opengl: allow RPi MMAL video decoding [08:17] mborzecki: I think so, that's how I read what seth wrote, would be nice to have a slightly more explicit "vote" but should be fine, worst case we just revert [08:17] mvo: maybe we need an official 'security team seal of approval' ;) [08:32] PR snapd#9806 closed: run-checks: update to match new argument syntax of ineffassign [08:43] brb, need to reboot, `[drm:dm_restore_drm_connector_state [amdgpu]] *ERROR* Restoring old state failed with -12` in dmesg and the display does not wake up :/ [08:44] that's on 5.10.4 kernel, which is clearly plagued by differnt issues from the day it was released :/ [08:59] good morning [09:00] mborzecki, happy new year [09:01] after work today I'll resume export manager tweaks [09:02] zyga: hey, happy 2021, may it be the year that they successfully patch cyperpunk 2077 xD [09:02] mborzecki, haha, at some point they will [09:02] hopefully before 2077 :P [09:02] I didn't play cyberpunk all that much, I was coding and reading about fiber optic network [09:03] mborzecki, I got a new router [09:21] good morning pedronis [09:21] hello [09:21] FYI it seems my fosdem talk has been accepted [09:22] I sent a proposal to the testing and automation devroom [09:22] the talk is about interactive debugging in CI systems, specifically spread [09:29] pedronis: hey [09:30] hi all [09:31] and happy new year 😊 [09:34] Where can I check for more details for a failing core18 snap refresh? It rolls back every time after a reboot, snap tasks shows an error at “Automatically connect eligible plugs and slots of snap "core18"” [09:34] dot-tobias, on what machine? [09:34] dot-tobias, x86 or arm? [09:35] zyga: arm, Raspberry Pi Model 3B [09:36] dot-tobias, perhaps that's the uboot bug [09:36] dot-tobias, do you have serial console logs from early boot? [09:36] zyga: I can attach a serial cable and reboot the device, hang on [09:40] dot-tobias, there was a bug, now fixed but not updated in the field, which on some systems prevents uboot from writing to the SD card [09:40] making all core/kernel updates fail [09:47] zyga: I see. Anything I should look out for in the boot log? There's “Failed to load 'uEnv.txt', rest seems fine. [09:53] zyga: FYI, here's the full log via serial https://paste.ubuntu.com/p/TmM3NHyzvb/ [09:57] looking [09:58] dot-tobias, that's a normal boot [09:58] dot-tobias, try refreshing core now [09:58] and collect the log again [10:09] zyga: Done, https://paste.ubuntu.com/p/ZS9spYFYcf/ → noticed the help output for env/save [10:09] huh [10:09] that looks like a new bug [10:09] mborzecki, ^^ [10:09] it looks like bad uboot script [10:09] Looks like `env` and `save` are called separately? [10:09] yeah [10:10] as if someone has accidentially injected a space into "saveenv" [10:10] /o\ [10:11] ogra, dot-tobias: can you guys please work on reporting that, it's a terrible typo that should not have been made but it needs to be fixed [10:11] mvo, ^ FYI [10:11] For clarity, this is on a custom gadget snap forked from the pi-gadget. Let me check my git history if I brought in any inadvertent changes first [10:12] dupadupa [10:12] dot-tobias, ah [10:12] mborzecki, is that your password ;D ? [10:13] pfff [10:13] yeah, very hardcode vm password [10:13] pff :D [10:14] damn, hit that amdgu issue again, looks like it's time to boot the old kernel [10:14] zyga, ogra, mborzecki: I pulled in this commit from pi-gadget recently, replaces saveenv with a custom function … https://github.com/snapcore/pi-gadget/commit/d837f0cfcdc84b92b3de8a380bafd14e55e65f6a [10:14] mvo, https://fosdem.org/2021/schedule/track/testing_and_automation/ [10:15] my talk is scheduled now :) [10:15] zyga: thanks! looks like something that waveform should be aware of, i.e. if uboot fails like in https://paste.ubuntu.com/p/ZS9spYFYcf/ [10:15] ogra, dot-tobias thanks for raising this, a LP bug would be great, once we have that we can ping foundations :) [10:16] mvo, waveform: see https://github.com/snapcore/pi-gadget/commit/d837f0cfcdc84b92b3de8a380bafd14e55e65f6a → guess this is the cause? Still checking my modifications though [10:16] ogra: Will do once I have definitely ruled out mishaps in my customizations 😅 [10:19] dot-tobias, are you using kernel_addr_r at all (i think we used to use kernel_addr without _r in the past or something like that) [10:21] i also wonder where it is supposed to get filesize from .. that var usually automatically contains the size of the last loaded file (whcih is typically the initrd) [10:28] ogra: You mean in the gadget? It's currently based on the 18-armhf branch of pi-gadget, there are `kernel_addr_r` references in uboot.env.in. The commit referenced above indirectly uses it as well, but it seems to have been used before that change as well. [10:30] ogra: https://github.com/snapcore/pi-gadget/blob/18-armhf/uboot.env.in#L20 → I'm not familiar with uboot, is it safe to define export_env before kernel_addr_r? [10:31] back on 5.10.3 [10:37] google:ubuntu-18.04-64:tests/main/cohorts is failing consistently everywhere [10:38] seems like the yq tool the test uses has changed something [10:38] mvo: ^^ [10:40] mvo, dot-tobias - interesting -- will take a look at that a bit later today; looks like the env export command fails (which is why the save then fails as $filesize won't be set) [10:44] waveform: appreciated 😊 [10:46] thanks waveform ! [10:46] waveform, do you have a muxpi setup at home? I've been working with that recently and I found it very useful for automation and testing [10:47] and, I know of a place that sells pre-made units [10:47] mborzecki: yq> amazing how much attention things take. thanks for taking care of this :) [10:49] maybe we can just get rid of the tool [10:50] dot-tobias, and to answer your question, the order wont matter ... vars are loaded to the runtime environment all together and export_env is also called only at the end when "snappy_boot" runs [10:50] waveform, ogra: for completeness, this is the source for my gadget snap based on pi-gadget 18-armhf: https://gitlab.com/glancr/gadget-snap-pi-kiosk (though I haven't modified uboot.env.in, see https://gitlab.com/glancr/gadget-snap-pi-kiosk/-/commits/18-armhf-mirros-customizations/uboot.env.in) In case the issues stems from a renamed volume or whatnot. [10:53] dot-tobias, hmm, seems thats new, but gitlab doesnt let me see your stuff without having an account [10:53] zyga, waveform: Should I file the LP bug for snap-core18 like https://bugs.launchpad.net/snap-core18/+bug/1900879, or is there a separate tracker for the gadget? [10:53] Bug #1900879: MAC addresses hard-coded in bootloader

[10:53] ogra: sec [10:53] * zyga doesn't recall [10:55] ogra: should be public now [10:55] better 🙂 [11:01] dot-tobias, your original issue was that the core snap did fail the configure hook though ... i'm not actually sure thats related to what waveform does in the uboot script ... [11:02] (but lets see and have this fixed first) [11:04] zyga, no - I've been looking at juergh's rather neat setup for automated testing, but piwheels kept me busy over the hols so I haven't had the chance to do much with it yet [11:04] waveform, I'm not familiar with that, can you share any pointers please? [11:04] ogra: Yeah, though the problem with save/env arose after a core18 refresh attempt (see diff of the two boot logs I pasted earlier), so I guess this section is only triggered on core refreshes? Just making semi-educated guesses at this point 😄 [11:05] zyga, https://github.com/juergh/bottle-pi https://github.com/juergh/rpi-install and https://github.com/juergh/usb-power-switch (for PSU control on a 400) [11:06] dot-tobias, you are correct that that section is only triggered on core refreshes [11:06] waveform, thanks, I'll go through that [11:06] (well, kernel or core updates) [11:06] waveform, does that rely on netboot in any way? [11:06] dot-tobias, yeah, might be ... thats why i said we should wait for the fix and then see if it still occurs ... i do wonder why gitlab shows "refresh-timer" in red in your gadget.yaml though ... and if thats an issue with the yaml highlighting in GL or if there is a tab instead of spaces [11:07] zyga, no - it's essentially using a GPIO to select a different boot sequence that boots into a minimal initrd to reflash the local card [11:07] ah [11:07] that makes sense [11:08] you can key off the gpio state in boot config [11:08] yup [11:08] waveform, I'll be working on extending spread to drive muxpi directly [11:08] but if you reflash the whole card, isnt your initrd then gone ? [11:08] and add support for serial line connections [11:08] (mainly for zephyr though) [11:09] ogra, not from ram, so the initrd can "re-inject itself" into the image post-flash, although I don't think the current version does that (it's one of the things I wanted to investigate over xmas but didn't get the time) [11:09] ah [11:10] well, that wouldnt work in UC20 testing i guess [11:10] ogra, hence muxpi :) [11:11] provided there's ~30M free on the first FAT partition it ought to [11:12] what it can't do (which I assume muxpi can) is to implicitly "remove" the SD card; i.e. it always boots off the SD card (various caveats there with 4's EEPROM, blah blah) [11:12] * ogra used to have setup simply using a transcend WIFI-SD card ... with replaced linux for the WIFI side ... that only had busybox wget and dd on it and could simply stream an img to the SD and reboot the Pi ... [11:13] waveform, yeah, muxpi setup can do that [11:13] waveform, order yours today ;) they arrive pretty quickly, modulo brexit border issues [11:13] waveform, and I'm happy to support muxpi on pi as that's one of my targets as well [11:14] waveform, and the tooling is really nice as well [11:17] zyga, if you've got a link for pre-made units I'll certainly take a look, but I do like the simplicity of juerg's setup - something to be said for making it easy/cheap for anyone to build [11:17] waveform, you can order them from https://shop.3mdeb.com/ [11:17] waveform, grab https://shop.3mdeb.com/product/muxsd-adapter/ and https://shop.3mdeb.com/product/muxpi/ [11:18] waveform, just expense it if your manager approves [11:18] waveform, I'd love to get it spreadified [11:18] so that spread can run from pi-related repos [11:18] you can deploy a gitub worker on muxpi so that it processes tests [11:18] zyga, will do [11:20] waveform, I've started looking at the software that drives muxpi, I'll be working on a PPA that has all of the things ready so that it's easier to prepare a new instance without reliance on magic images from anywhere [11:20] so far I've reproduced the microcontroller image [11:20] and all the system services that talk to it [11:21] I didn't prepare packaging yet but I also didn't miss any dependency [11:21] so it should be doable [11:21] in reasonable time that is [11:21] I'll prepare a bunch of muxpi-* packages and ask mvo to help me sponsor it to debian [11:21] this should allow vanilla Ubuntu/Debian installation with apt-get to run the stack correctly [11:22] PR snapd#9807 opened: tests/main/cohorts: replace yq with a Python snippet <⚠ Critical> [11:24] mborzecki, +1 [11:24] * zyga is very grateful that his team membership was not revoked [11:37] happy new year cachio [11:40] zyga, hey [11:40] thanks [11:40] happy new year for you as well [11:42] cachio: hey, happy 2021 [11:42] mborzecki, hey [11:42] cachio: hey, happy 2021! [11:43] happy new year [11:43] mvo, hey [11:43] happy new year [11:43] happy to be working again :) [12:23] ogra: re refresh-timer in gadget.yaml (overlooked your message): you mean https://gitlab.com/glancr/gadget-snap-pi-kiosk/-/blob/18-armhf-mirros-customizations/gadget.yaml#L6 ? For me, it shows a green-ish string, so maybe really a GitLab issue? My local editor says spaces, anyway. [12:23] okay ... was just curious [12:26] ogra: Always grateful for hints on little details, especially for this part of my stack – there [12:27] …'s a lot of d'oh potential with a side of really undesirable consequences with gadget snap snafus 😄 [12:27] ogra, waveform, zyga: https://bugs.launchpad.net/snap-core18/+bug/1910094 [12:27] Bug #1910094: uboot fails to save env after core18 refresh [12:28] dot-tobias, ta - I've stuck it on my list [12:29] * dot-tobias still misses emoji reactions like. +1 on IRC to avoid log bloat 😉 [12:29] dot-tobias, well, it is a bit strange that it fails in the configure hook, modifying u-boot vars lives directly in snapd (unless that changed) ... so the hook should not be involved here at all ... but it might just be fallout from the actual upgrad failing or so [12:36] mvo: hi, are the debian --help test also failing with a panic? [12:36] we are getting panics also on F32/33 [12:40] pedronis: hey, happy 2021 :) the debian tests just format differently, no panic there [12:49] sigh ... this is really curious ... https://github.com/ogra1/zoom-snap/issues/64 [12:50] i cant reproduce it on a 20.04 install here ... did we secretly update 20.04 to a new fontconfig or so ? [12:55] hmmm https://paste.ubuntu.com/p/jSTTCmBJ96/ [12:55] panic in snap validate --help on fedora 33 [12:57] and f32 [12:59] mborzecki: yes [13:00] mborzecki, outdated go-flags [13:00] mborzecki, are you a committer in fedora? I can try to rev it [13:05] weird, that version has been there since july 2020 [13:05] mborzecki, is the test new? [13:05] so what changed that it started panicking just now? [13:05] I don't think it's a new problem [13:05] I recall this panic long ago as well [13:05] it just slipped through the cracks then [13:08] mborzecki, anyway, let me know if you need updates [13:08] mborzecki, I wanted to upload zmk to Fedora and I can get my packaging memory refreshed [13:14] i have a vague recollection we hit that once before [13:19] hmm there have been some commits in go-flags since the revision we list in vendor.json [13:19] ha, even my fix got in :P [13:27] hm bisect tells me this is the first bad commit https://github.com/jessevdk/go-flags/commit/71064e1638ea2c34ad561fd1ec92a23944673ff4 [13:27] and is the next commit after the revision we listed in vendor.json [13:34] heh [13:34] so we are on old but good version? [13:34] FYI: some notes from my work on static analysis from last year: https://listed.zygoon.pl/21463/integrating-pvs-studio-and-coverity-with-make [13:43] heheh, so https://github.com/snapcore/snapd/commit/85900470da735d61e7c520b27dfc82172a6af911 triggered an apparent bug in go-flags [13:50] mborzecki, nice :) === msalvatore_ is now known as msalvatore [14:18] PR snapd#9807 closed: tests/main/cohorts: replace yq with a Python snippet <⚠ Critical> [15:08] PR snapd#9808 opened: tests/main/snap-validate-basic: disable test on Fedora due to go-flags panics [15:16] dot-tobias: did you report a bug about this saveenv issue? just curious so that I can keep track of uc20 issues [15:18] PR snapd#9809 opened: snap: skip help output tests for go-flags v1.4.0 [15:29] * cachio lunch [16:49] PR snapd#9810 opened: tests: allow to set a specific PROJECT_PATH === ijohnson is now known as ijohnson|lunch [18:19] PR snapd#9810 closed: tests: allow to set a specific PROJECT_PATH [20:16] anybody on core team able to talk a bit on recovering to previously know good version sets of snaps? I am looking to get some owa updates standardized for a project and I am not sure what already exists [20:23] hardwire: do you mean you want to revert your installed snap to a previous version [20:24] core20 itself if needed along with any other snaps that were part of a potentially failed update (asseen by health check after reboot) [20:24] if for instance a reboot doesn’t even complete then next boot will revert [20:24] that sort of setup [20:25] I am working on my own code now for an owa agent early on in boot and initramfs [20:26] but it seems like there may be room for uboot to handle some of this if for instance the kernel snap fails or initramfs is munged somehow [20:26] just trying to find what exists out there === ijohnson|lunch is now known as ijohnson [20:41] hardwire: are you specifically wondering about how ubuntu core handles trying a new revision of a kernel snap ? [20:42] hardwire: what ubuntu core implements for trying (aka a-b boot) is to extract / mount the kernel snap and then set a boot environment variable to "try", which the bootloader must set to "trying" after rebooting to try the update, then if the boot is successful userspace sees "trying" and assumes the try was successful [20:43] if the boot fails and we get rebooted, the bootloader will see the value of "trying" and reset that to "", which userspace then sees and interprets "" as meaning that the update failed [20:43] trying is in cmdline right? That's similar to what I was attempting. [20:43] you can see more info on this at https://snapcraft.io/docs/ubuntu-core-boot-modes [20:43] it's slightly different for uc20, but the same basic idea [20:43] hardwire: no think like uboot env file or grubenv [20:44] ah gotcha. I will be using uboot. [20:44] well you also put some of the info on the cmdline, but the main variable about "try"/"trying"/"" has to be in uboot env in your case then [20:44] how did I miss this when reviewing the docs.... [20:45] ok so this is ideally just for core and kernel [20:45] I'll want to dovetail on those a bit for what I'm doing for customer managed snaps [20:45] yes just the base snap (i.e. core, core18, core20) and the kernel snap [20:46] I'll review the code on those and do some testing before I go forward. thank you for the info [20:46] how app snaps are updated is a bit different, but actually much easier since app snaps don't require a reboot [20:46] in most cases for sure. [20:46] I was happy to see that ufw was made into a snap [20:47] A lingering question with my own work is why snaps don't typically pull from the closest LTS release to the core for their stage packages. [20:48] I was able to make a snap using stage-packages. But it seems like that is not at all preferred and I'm not sure if that has to deal with rpath related issues. [20:48] s/deal/do/ [20:50] well it depends on the snap, some snaps are indeed just using the stage-packages from the LTS release of their base snap exactly, but note that you can have snaps with different base snaps than the actual host system release [20:50] for example on an ubuntu core 20 system, I can have app snaps which use base snaps from the "core" snap which is based on xenial 16.04 [20:51] in the converse, on a xenial 16.04 ubuntu server system, I can use snaps that have a base snap from the "core20" snap which is based on ubuntu 20.04 [20:52] indeed and I believe that is to be expected [20:53] just not really something I've witnessed. I was reviewing the NetworkManager snap and was a bit surprised to see it being built. [20:53] same with some others. [20:53] I'm working on a GATT server right now that will eventually be a snap. It will be responsible for setting os level settings (via snap settings) for wifi info. [20:59] I joined the uboot mailing list to see if there was a special way to run memtest and store the results for a specific ephemeral boot [20:59] man that mailing list is busy [22:03] ijohnson, hey, could you please take a quick look to this #9811 ? [22:03] PR #9811: tests: fix library path used for tests.pkgs [22:03] sure [22:05] PR snapd#9811 opened: tests: fix library path used for tests.pkgs [22:55] cachio: approved