[06:05] morning [06:08] PR snapd#9078 closed: boot: fancy marshaller for modeenv values [06:32] good morning [06:33] PR snapd#9107 opened: tests: remove End-Of-Life releases from spread.yaml [06:35] zyga-x240: hey [06:36] hey :) [06:36] good morning zyga-x240 and mborzecki [06:36] I'm experimenting with a small idea that will give us a bit more CI speed [06:36] hey mvo :) [06:36] we can run the cla-check on self-hosted workers [06:37] this will release a slot for the more expensive unit test jobs [06:37] mvo: hey [06:38] zyga-x240: do we need to move the check to the tests yaml file or can this be changed in the github repo actions configuration? [06:38] just a tweak to the yaml [06:38] no need to change anything else [06:38] PR snapd#9088 closed: cmd/snap-preseed: use snapd from the deb if newer than from seeds [06:38] I'm preparing the rest for this though [06:38] as the containers are unprivileged and devoid of sudo [06:38] though now that we have mvo around [06:38] I have one more idea :) [06:38] but that's for later [06:38] we can label workers [06:39] so we can create a subset of workers without spread keys [06:39] but with sudo [06:39] and we could use those for running unit tests (I have 4 spare cores at home, soon will have more) [06:39] with fast apt proxy and preinstalled snaps it will be speedy [06:40] on par with those over-provisioned xeons [06:43] PR snapd#9092 closed: interfaces/udev: do not reload udevadm rules when preseeding [06:43] PR snapd#9093 closed: interfaces/kmod: don't load kernel modules in kmod backend when preseeding [06:43] PR snapd#9101 closed: interfaces/systemd: use emulation mode when preseeding [06:47] mvo: https://github.com/snapcore/snapd/pull/9107#pullrequestreview-463069566 [06:47] PR #9107: tests: remove End-Of-Life releases from spread.yaml [06:47] mvo: bw. we have two entries for debian-sid-64 in qemu backend [06:47] mborzecki: haha - fun [06:48] mvo: oh, and while at it, i think we can drop fedora-30 from google backend, it's EOL anyway [06:48] PR snapd#9108 opened: gadget, osutil: use atomic file copy, adjust tests (2.45) [06:48] mvo: left a comment there [06:51] zyga-x240: \o/ [06:53] we should try to get that 20.04-on-zfs image [06:53] I'd love to see it fail [07:02] mvo: hi, I fixed the conflicts in #9097, could you re-review it ? [07:02] PR #9097: boot/modeenv: add deepEqual, Copy helpers to simplify bootstate20 refactor [07:03] PR snapd#9020 closed: cmd: add new "snap recovery" command [07:05] zyga: https://github.com/snapcore/snapd/pull/9043 is probably in good shape to review now [07:05] PR #9043: daemon: replace access control flags on commands with access checkers [07:06] morning [07:07] hey pedronis, jamesh, pstolowski [07:07] ack, I'll look in a moment [07:07] pstolowski: good morning! does master have all the preseed bits we need for cpc? if so I will release a 2.46~pre1.gitXXX to groovy [07:08] PR snapd#9109 opened: github: run CLA checks on self-hosted workers [07:08] PR snapd#9110 opened: preseed: cherry-pick #8704, #8709, #9088 (2.45) [07:08] pedronis: thanks for pushing fixes to my PRs yesterday [07:09] np [07:10] mvo: yes [07:10] pstolowski: cool, releasing now then [07:10] ty! [07:13] woah [07:13] today is a good LTE day [07:13] I was just uploading at 200MBit rate [07:15] zyga-x240: maybe everyone else is on vacation? hence lots of available bandwidth [07:15] mborzecki: maybe [07:15] I was wondering if getting an external antenna would help [07:15] the BTS I'm talking to is ~ 30 meters away [07:16] maybe 50 [07:16] but I have to go through a bit of wall and glass [07:16] we could affix an antenna to the side of the house and just run the wires, the modem has two SMA connectors [07:17] mvo: the 2.45 thing is broken [07:17] src/github.com/snapcore/snapd/cmd/snap-preseed/preseed_linux.go:31:2: imported and not used: "github.com/snapcore/snapd/cmd/cmdutil" [07:17] src/github.com/snapcore/snapd/cmd/snap-preseed/preseed_linux.go:167:22: undefined: snapdtool [07:17] src/github.com/snapcore/snapd/cmd/snap-preseed/preseed_linux.go:175:21: un [07:17] something is not right [07:18] zyga-x240: oh no [07:18] zyga-x240: yeah, it's a PITA, it diverged quite a bit [07:18] zyga-x240: maybe the answer is really 2.46 :/ [07:18] *exactly* [07:18] we should really try [07:18] even if it goes nowhere but beta [07:19] we'd be backporting less [07:19] well, as long we do release .3.1 to stable [07:20] mvo: did you see the SANFU with systemd yesterday? [07:20] it's a bit unfortunate [07:20] I hope it doesn't affect regular users [07:20] upgrading in-place seems less and less supported [07:20] *SNAFU [07:20] how could I typo that :P [07:21] also jdstrand wanted things into 2.46 that he hasn't finished yet [07:21] I think having an early outlook would be great, after all it's all just a number, we could really finally push to stable something more like 2.46.3 [07:21] as I said, it's fine, we do need to release .3.1 though [07:22] I agree [07:24] * zyga-x240 reviews https://github.com/snapcore/snapd/pull/9043 [07:24] PR #9043: daemon: replace access control flags on commands with access checkers [07:25] zyga-x240: if you're uploading at 200mbit/s i doubt antennas would help [07:25] I wonder what's the limit of this class of modem [07:25] I, for one, will be on 5G the moment it is avaliable [07:26] zyga-x240: but, if your bts is heavily occupied, you may want to check other frequencies and force one that's rarely used by mobiles [07:26] unfortunately my firmware is pretty locked so I have almost no choice [07:26] on the upside I get pretty good overall speed so I think it's a pretty lucky location [07:26] there are BTSes all around here, maybe there are just enough [07:27] zyga-x240: usually 800mhz is quite busy, but 1.8ghz and 2.6 are not :P [07:32] brb [07:44] pstolowski: if I set "run-nested" will that also run the nested/manual suite? [07:48] mvo: heh, i don't know of "run-nested"... i was always invoking them manually with spread ... google-nested:...:tests/nested/manual/... [07:48] PR snapd#9111 opened: releases: release 2.46~pre1 to groovy [07:49] wow, it's warm today [07:49] pstolowski: I updated #9086 after your feedback [07:49] PR #9086: many: reorg cmd/snapinfo.go into snap and new client/clientutil [07:49] pedronis: yes, looking atm, ty [07:50] could we change things so that without run-nested there are "skips" not green ticks [08:00] * zyga-x240 returns to review after a small interrupt to fix a test [08:02] mvo, pedronis: could you please look at https://github.com/snapcore/snapd/pull/7825 and +1/-1 merging as-is [08:02] PR #7825: many: use transient scope for tracking apps and hooks [08:02] it's +2 technically but I wanted to triple check [08:02] zyga-x240: you told me yesterday not to look at it :) [08:02] it's a +13 -355 leftover from the "backend" work [08:02] pedronis: yeah because yesterday it was not relevant :) [08:02] I mean, the other PRs were more interesting [08:03] as they contain new work that needs direction [08:03] this is just pushing a stone up the hill till it's done [08:03] PR snapd#9112 opened: tests: run as hightest via tests.session [08:29] jamesh: around? [08:29] jamesh: https://github.com/snapcore/snapd/pull/9043#pullrequestreview-463128041 (partial to ask a question) [08:29] PR #9043: daemon: replace access control flags on commands with access checkers [08:29] I'm reading the rest [08:29] zyga-x240: yeah [08:33] zyga-x240: the ucred == nil case would be true if the REST API was available via TCP. It should always be non-nil for unix domain sockets [08:34] I see [08:34] in that case I think we should do what I suggested in the second comment [08:34] it's safer this way [08:34] zyga-x240: this effectively makes the existing GuestOK vs UserOK distinction meaningless [08:34] we can revisit this once we have http [08:34] I'll review the rest carefully [08:34] if you want and agree please push that change to the existing checkers [08:34] I would feel much safer with an early != nil check [08:36] zyga-x240: probably a good idea, on the basis of not making access decisions prematurely [08:40] mvo: we could set 'run nested' label on #9102 [08:40] PR #9102: corecfg: add "system.timezone" setting to the system settings [08:42] pstolowski: yeah, that's why I was asking earlier [08:42] mvo: aaah, sorry, i didn't have enough coffee, i though it was about run-checks etc [08:48] PR snapd#9113 opened: tests: port regression-home-snap-root-owned to tests.session [10:42] jamesh: https://github.com/snapcore/snapd/pull/9043#pullrequestreview-463139716 [10:42] PR #9043: daemon: replace access control flags on commands with access checkers [10:42] sorry for taking this long, I was really trying to think through the various consequences [11:08] brb [11:08] quick coffee :) [11:10] pedronis: i updated #9001 [11:10] PR #9001: o/snapshotstate: helpers for calculating disk space needed for an automatic snapshot (2/N) <⛔ Blocked> [11:12] hmm, wot, unit test failed on FAIL: cmd_sign_test.go:55: SnapKeysSuite.TestHappyNonDefaultKey, seems like it called real gpg? [11:13] what was the rest of the failure? [11:14] I saw something like this today as well [11:14] PR snapd#9114 opened: tests: fix debug section of appstream-id test [11:15] zyga-x240: value *errors.errorString = &errors.errorString{s:"cannot sign assertion: bad GPG produced signature: it does not verify: openpgp: invalid signature: RSA verification failure" [11:16] zyga-x240: i'm looking into it [11:16] yeah, same error [11:16] cool [11:16] thanks! [11:16] * zyga-x240 stops coding and goes for that coffee [11:37] mborzecki: we are hitting some kind of new issue on centos related to selinux and package versions: https://github.com/snapcore/snapd/pull/9097/checks?check_run_id=957590501 [11:37] PR #9097: boot/modeenv: add deepEqual, Copy helpers to simplify bootstate20 refactor [11:38] we also got again the weird 16.04 with different cgroup setup [11:38] pedronis: hmm, let me check that [11:39] pedronis: uhh, yeah that's the usual upgrade thing where centos is lagging behind rhel again [11:44] PR snapd#9097 closed: boot/modeenv: add deepEqual, Copy helpers to simplify bootstate20 refactor [11:44] PR snapd#9105 closed: tests: work around bug in systemd/debian <⚠ Critical> [11:46] pstolowski: https://github.com/snapcore/snapd/pull/9115 [11:46] PR #9115: interfaces: check !b.preseed earlier [11:46] pedronis: I'll look at the cgroup mystery as well [11:47] mborzecki: do you have any ideas? I'd like to ensure we boot the right kernel - depending on the spread system [11:47] +1, ty [11:47] mborzecki: and then depending on the kernel that some super basic things hold (I know it intersects with systemd so I'd like to just constrain this to xenial for now) [11:47] morning folks [11:47] ijohnson: good morning! [11:48] hey zyga-x240 [11:48] zyga-x240: simple sanity checks are probably ok [11:48] how are tests today [11:48] seems like centos and cgroups are giving us trouble still [11:48] zyga-x240: i mean, those distros are kind of fixed, so we know what to expect on each system [11:49] PR snapd#9115 opened: interfaces: check !b.preseed earlier [11:49] yeah [11:50] mborzecki: I'll do that [11:50] I'm very curious to find out what we get [11:52] mvo: in case you didn't figure it out adding the run-nested label to a PR only works if you add the label before opening it, or close and re-open the PR after adding the label [11:52] at least that's been my experience [11:53] hmm i cannot repro cmd_sign_test issue [11:53] hey ijohnson ! [11:53] hey pstolowski [11:54] thanks for all the iface backend PR's [11:54] sure thing, thanks for reviews [11:58] pstolowski: it's very rare [11:58] pstolowski: maybe leave it on repeat -1000 [12:03] grr mounted fs updater [12:03] ? [12:03] what? [12:04] zyga-x240: have you seen it outside of 16.04? [12:04] pstolowski: I don't recall where I saw that [12:10] zyga-x240: the changes for the content update observer are slightly annoying [12:10] so are the tests [12:10] hmm? [12:10] content update observer> [12:10] ? [12:11] zyga: yes, the resealing & gadget updates things i'm working on [12:12] zyga: hm pagure badges? wtf? [12:13] haha, for updates? [12:14] yeah, but it looks buggy, i haven't pushed 1000 commits into pagure [12:24] mborzecki, cmatsuoka i've requested your re-reviews on #9001 because of the changes after addressing Samuele's comment [12:24] PR #9001: o/snapshotstate: helpers for calculating disk space needed for an automatic snapshot (2/N) [12:24] pstolowski: ack, will do [12:26] thx [12:26] pedronis: btw. in the morning in saw 408 with a GET request from a run that happend within the last 24h [12:29] pstolowski: checking [12:29] hey cmatsuoka [12:29] zyga: hi [12:31] zyga: how's the pain? feeling better? [12:31] yeah it's really comfortable now [12:31] a bit hot today but now I'm just looking for execuses [12:32] haha, ok [12:39] PR snapd#9108 closed: gadget, osutil: use atomic file copy, adjust tests (2.45) [12:39] PR snapd#9114 closed: tests: fix debug section of appstream-id test [12:39] PR snapd#9115 closed: interfaces: check !b.preseed earlier [12:39] PR snapd#9116 opened: tests: adding system information and image information when debug info is equired [12:39] thanks mvo [12:42] thanks for the review pedronis [12:43] zyga: thank you! [12:44] mvo if you are reviewing could you please advice on the last comment on https://github.com/snapcore/snapd/pull/7825 [12:44] PR #7825: many: use transient scope for tracking apps and hooks [12:44] PR snapd#9107 closed: tests: remove End-Of-Life releases from spread.yaml [12:44] PR snapd#9117 opened: tests: remove End-Of-Life opensuse/fedora releases [12:44] should I keep splitting? [12:47] zyga: looking [12:48] zyga: that seems fine - but 7825 just rmeoves code now it seems from a casual look? [12:49] yes, and removes a workaround needed earlier [12:49] zyga: what would you split from it? all the additions to make it removal only? [12:49] should I get more reviews, split it further or do something else? [12:49] just the removal for the explicit review [12:50] and leave the few odd tweaks (+13) as-is [12:50] zyga: the problem with that PR at moment is that it has 197 commits and a description that I'm not sure matches the content anymore [12:51] that's fine, I really want to merge it to keep the history in place [12:51] as I said, I'm happy to shave it further [12:51] just looking for advice [12:51] isn't the history in all the PR that landed before this one? [12:52] they contain a subset [12:53] also I would love to just merge it eventuallu [12:53] it's almost complete subset, no, if all we are left is removing and +13 ? [12:53] eventyally* [12:53] yes but commit history here is more detailed [12:53] *eventually, sorry [12:53] still learning the new keyboard [12:54] you are saying that in master it will look like things came from two places? [12:54] that seems confusing to me [12:55] no, master will contain a merge commit [12:55] you can dig deeper to see that [12:55] if you annotate master it will show what it shows currently [12:55] as that is not changed [12:55] but I think there's some useful information in that history [12:57] I think I need to play with it a bit, I'm trying to understand the useful vs confusing factor here [12:59] git merge that into a test branch and tell me if that is confusing [12:59] anyway, time for standup [13:04] PR snapd#9118 opened: tests: detect unexpected xenial kernels [13:30] PR snapd#9119 opened: many: remove usage and creation of hijacked pid cgroup [13:31] pstolowski speaking of gpg it just failed [13:31] https://github.com/snapcore/snapd/pull/9118/checks?check_run_id=958332953 [13:31] PR #9118: tests: detect unexpected xenial kernels [13:31] zyga: wow [13:31] in azure-hosted unit tests [13:31] so perhaps running it in spread is useless [13:31] as it fails in a different env [13:32] zyga: seems to be more frequent than before then [13:43] errand, bbl [13:56] Hmm.. still don't have anything in calendar for the next hour [13:57] ijohnson: Doesn't seem to have worked [13:57] niemeyer: hmm let me try again [13:58] Got the one from mvo though [13:59] niemeyer: ok, well I just re-sent it, so hopefully you can see the link to join [14:09] niemeyer I see your're invited there [14:09] but not accepted [14:09] zyga: It's been fixed [14:09] Thanks [14:09] (I've been in the call) [14:10] ah, good [14:12] I think I need a shower [14:12] 32C [14:12] it's a hot day [14:20] zyga, this is the error that I saw https://paste.ubuntu.com/p/DrXNkRyNxm/ [14:21] and this https://paste.ubuntu.com/p/KrFm3H9Xt6/ [14:46] pedronis: wrt 2.46, I'm actively working on various items for k8s, I will be picking up the cups-control/cups pr after that, the pickup 8301, then go through the list of misc policy updates. that should all be happening in the next week [14:48] pedronis: I also need to review jamesh' dbus PR and go down the list of whatever needs security review [14:54] cachio looking [15:15] jdstrand: ok, I think mvo might have more sense about 2.46 timelines early next week [15:15] pedronis, jdstrand in a meeting but yes, first 2.46~pre early next week, groovy has it already [15:16] ijohnson: I did another pass, my main comment is that more bits probably need "On commit" preambles [15:17] pedronis: thanks yeah I saw the other places I will try to do a full pass over all relevant comments [15:25] PR snapd#9120 opened: interfaces: add kernel-crypto-api interface [15:26] pedronis: at your convenience, would you mind at least reviewing the interface name in ^ [15:26] * jdstrand adds appropriate label for that [15:27] cachio I hope we can detect the cause of that cgroup weirdness soon [15:27] hey jdstrand :-) [15:27] good to see you again [15:27] I'll break for lunch because I'm starving [15:28] zyga, I re executed but couldn't reproduce so far [15:28] I'll ping you if I have more info [15:28] jdstrand: thx, I'll try to look earl next week. My initial comment is that we don't have any other *-api named interface, though many are for apis though [15:28] kenvandine: boy, I don't know what has been going on lately but on my focal host and firefox snap, periodically the fonts show the little utf-8 boxes and I have to restart the browser. I think jamesh was looking at that some (or at least commented on something similar in the forum)? [15:29] pedronis: yeah. I picked that because that appears to be how everyone refers to it [15:29] pedronis: ie, I wasn't using -api as a new suffix, I was thinking of 'kernel-crypto-api' as like 'mir' [15:29] pedronis: fyi for your review next week [15:30] s/fyi/context/ [15:31] mvo: actually, iirc, you did an fc-cache update lately for that ^ (is this the libfreetype mismatch?) [15:31] hey zyga :) [15:31] jdstrand: yeah, i've seen that once myself. I could not figure out what's going on there [15:31] zyga: you too! hey, I've been looking for us being on at the same time. did you see my comment in https://forum.snapcraft.io/t/alternate-home-workaround-request/18679/11 ? [15:32] it only started happening when we updated firefox to use the new platform [15:32] kenvandine: it basically happens at least once a day for me [15:32] wow [15:32] sometimes more often [15:32] it hasn't happened to me in months [15:32] jdstrand: if you could try to debug it... i would really appreciate it :) [15:32] very interesting that it's that common for you [15:33] kenvandine: I don't really know what to look for... do you have debugging instructions? [15:33] i think to start with run firefox from a terminal [15:33] also, I don't really have time atm. but I guess if it keeps happening, I can try [15:33] and grab stdout when it happens [15:33] when you have time, i would appreciate it [15:34] i can't figure out what triggers it [15:34] jdstrand, let me check [15:34] I didn't read that yet [15:34] but first food [15:35] kenvandine: ok, I restarted it from a terminal. let's hope I don't accidentally close it :) [15:35] jdstrand: thanks! [15:41] jdstrand: kenvandine: if it is the issue that mvo fixed with freetype update to the fc-cache binary in the snapd snap, it will happen when snapd updates _any_ snap [15:42] so if you see snap changes that happen around the time that those font boxes show up, that could probably be it [15:42] ijohnson: has that fix landed? [15:42] and also I don't think mvo's fix has made it to stable, I think it's still on edge iirc [15:42] ah [15:42] i'm on edge [15:42] so maybe why i haven't seen it [15:42] perhaps :) [15:43] we are a bit behind getting things to stable with 2.46, had many things come up with required 2.45.x :-) [15:43] kenvandine: well that's great actually since it seems to imply that the fix worked [15:43] jdstrand: what version of snapd have you been tracking? if it is not edge then probably you have this bug and it may go away if you track snapd edge [15:43] question is what's jdstrand running :) [15:46] re [15:50] I'm using snapd from edge and I still get boxes in firefox sometimes [15:52] pedronis: but do you get them _less often_ :-) ? [15:52] maybe [15:52] mvo: 19.10 removal has weird effects [15:53] better boxes than immedia segfaults [15:53] 19.10 fails on https://github.com/snapcore/snapd/pull/9119 [15:53] PR #9119: many: remove usage and creation of hijacked pid cgroup [15:54] but it's not there [15:55] cachio: 8942 needs another review, it changed significantly since pstolowski's review unfortunately I think [15:55] cachio: but I +1d it just now [15:56] ijohnson, nice, thanks a lot [15:56] pstolowski, if you could take a look it should be awasome [15:57] * mvo is in a meeting [16:05] * cachio lunch [16:10] PR snapd#9121 opened: github: remove Ubuntu 19.10 from actions workflow <⚠ Critical> [16:13] cachio: i'll, but on monday at this point, eow now [16:14] ijohnson: snapd 2.45.3.1+git2475.g9bb0e0c from edge [16:14] 8916 [16:15] * jdstrand wonders if it is happening daily since he is tracking edge and the caches are getting out of date [16:15] out of sync* [16:15] jdstrand: mmm ok so your bug is probably not fixed by that PR [16:15] sorry, I have 2.45.3.1+git2463.gaf15176 installed, snap refresh hasn't yet happened today [16:16] But also yes tracking edge would result in more font cache re builds [16:16] 8906 is what is installed [16:18] actually, I could test that theory [16:18] * jdstrand snap refreshes snapd [16:21] that alone didn't seem to cause the problem [16:22] unless the fix came in between 8906 and 8916 [16:22] jdstrand: purge cache, restart all apps [16:22] and see if something breaks [16:24] jdstrand: the PR was not to snapd directly but rather to fc-cache-static-builder [16:24] https://github.com/snapcore/fc-cache-static-builder/pull/2 [16:24] PR fc-cache-static-builder#2: build freetype from the security pocket too [16:28] ijohnson: right-- and where does my system get that? [16:29] jdstrand it's built in the snapd snap [16:29] that was merged more than a month ago. certainly that was in 8906... [16:29] So shortly after that PR was merged the next edge snapd snap build would have got it [16:29] Yes it certainly is in 8906 [16:29] * jdstrand nods [16:30] one more tweak to the update observer branch and eow [16:31] zyga-x240: are you talking about ./.cache/fontconfig/ ? how would removing all that be a valid reproducer? (ie, I'm personally not doing that, is something else?) [16:31] jdstrand: I think cache is only written to when absent, [16:32] and that some combination of apps write the cache that other apps cannot read [16:32] this would just give you another chance to try [16:32] (alternatively stash the cache) [16:32] No, the cache is always rewritten when snap update happens iirc [16:33] So removing the cache and refreshing a snap would show that snapd is writing a broken cache [16:33] That was the bug mvo fixed by building with free type [16:33] Unclear that jdstrand's bug is a corrupt cache or not [16:33] ijohnson: global cache you mean? [16:34] ijohnson: there's also ~/.cache/fontconfig which some of the desktop clue copies to the $SNAP_USER_COMMON/.cache/fontconfig [16:34] ETOOMUCHFONTCONFIG [16:35] Ah yes that's right there's multiple caches [16:35] Yes snapd will just regenerate the global cache [16:35] ijohnson: global as in /var/cache or ~/.cache/fontconfig? [16:36] * jdstrand would be surprised if snapd went into ~/.cache/fontconfig [16:36] /var/cache [16:37] I do have a few files in there that were touched [16:37] (when I did the refresh a minute ago) [16:37] few minutes* [16:46] ijohnson: how do I trigger the snapd regeneration? [16:47] Refresh any snap [16:47] ijohnson: would an install do? [16:48] Mmm probably yes [16:49] * jdstrand refreshes to another channel [16:49] well, that didn't trigger it [16:49] * jdstrand takes some notes [16:53] ok, if it happens again, I'll see if there is a discrepency between /var/cache/fontconfig, ~/.cache/fontconfig and ~/snap/firefox/common/cache/fontconfig [16:57] * jdstrand jots down fc-cat -v [16:58] (that let's me see what font corresponds to what cache file (among other things) [16:58] * jdstrand moves along [17:06] hi... when someone uninstalls a package all app data gets deleted... so when u reinstall the app u lose all data... thats a huge bug! [17:09] dust: have you looked at `snap saved` at all ? [17:10] dust: snapd will automatically create shapshots before removing a snap that can be restored by the user if they want to after reinstalling the snap [17:10] jdstrand: just to confirm you don't have any special font setup like manually installed fonts that you configured firefox to use everywhere right ? [17:10] ijohnson, where to find that? [17:19] ijohnson, in ubuntu 20.04 [17:26] PR snapd#9117 closed: tests: remove End-Of-Life opensuse/fedora releases [17:36] dust: run `snap saved` to show snapshots that have been created automatically, and `snap restore` to restore snapshots [17:37] ijohnson: the only thing I would say is special about this machine is tha it is pretty old so I've gone through many do-release-upgrades, but I've not installed any fonts from anywhere. all just from debs [17:37] and debs from the archive [17:37] dust: see also https://snapcraft.io/docs/snapshots [17:37] jdstrand: hmm then yeah I don't think there should be anything there [17:37] PR snapcraft#3240 opened: cli: skip sudo check for lxd/multipass if not running in tty === ijohnson is now known as ijohnson|lunch [17:41] ijohnson|lunch, thx === ijohnson|lunch is now known as ijohnson [18:53] /me wonders about - Make snap "test-snapd-user-service" (unset) available to the system (Post http://0/v1/service-control: dial unix /run/user/0/snapd-session-agent.socket: connect: connection refused) [18:54] what is that http://0/ [18:54] is that something we synthesize [18:56] 2020-08-07T15:54:53.3788099Z - google:fedora-32-64:tests/main/snap-user-service [18:56] 2020-08-07T15:54:53.3788764Z - google:fedora-32-64:tests/main/snap-user-service-socket-activation [18:56] 2020-08-07T15:54:53.3797466Z - google:fedora-32-64:tests/main/snap-user-service-start-on-install [18:56] 2020-08-07T15:54:53.3798599Z - google:fedora-32-64:tests/main/snap-user-service-upgrade-failure [18:56] those seem to fail often [18:57] zyga-x240: yes I've seen those a lot today as well and was wondering about that path we are posting to [18:57] seems like that 0 should not be there [18:58] I'll look [18:58] i.e. I think it should be doing http://v1/service-control on /run/user/0/snapd-session-agent.socket [19:03] the curious bit is [19:03] that the test tries to exercise the test user [19:03] but we really fail for root [19:03] as we don't have perfect root "restore" path [19:04] that socket is surely corrupted [19:04] I'll add some logic [19:04] IIRC we had some of this already [19:04] in another case [19:04] where we realized something was needed [19:04] like restarting the socket [19:04] or something alike [19:04] we had something similar in pulseaudio but that was only on the surface, I suspect [19:04] as there pulse tried to create socket [19:05] and systemd tried to create a socket for activation [19:05] and that was racy [19:05] I'll add some better debug to that test [19:05] and add preconditions [19:05] to all four actually [19:20] mmm that makes sense [19:41] PR snapd#9122 opened: mkversion.sh: if the changelog version has git in it, don't add git version info [19:43] +1 [19:48] hey, reproduced [19:48] nice [19:48] let's examine [19:49] ha [19:49] funny [19:50] google:fedora-32-64 .../tests/main/snap-mgmt# ls -ld /run/user/0/snapd-session-agent.socket [19:50] srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/snapd-session-agent.socket [19:50] google:fedora-32-64 .../tests/main/snap-mgmt# systemctl --user status snapd-session-agent.socket [19:50] Failed to connect to bus: Connection refused [19:50] wat? [19:50] google:fedora-32-64 .../tests/main/snap-mgmt# ls /run/user/0/bus -l [19:50] srw-rw-rw-. 1 root root 0 Aug 7 19:46 /run/user/0/bus [19:51] funny that those are around [19:51] root 45646 0.0 0.2 27040 9888 ? Ss 19:47 0:00 /usr/bin/python3 /snap/test-snapd-service/x1/bin/start-stop-mode sighup [19:51] we leak those from another test [19:51] ijohnson, hey, beta image is not booting foc uc20 [19:51] is it known? [19:51] connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/bus"}, 18) = -1 ECONNREFUSED (Connection refused) [19:52] cachio: uhhhh [19:52] no? [19:52] cachio: how is it not booting ? [19:52] zyga-x240: that's really weird [19:52] sealing problem I see [19:52] systemd --user for root is down [19:52] no session, no bus [19:53] weird [19:53] ijohnson, https://paste.ubuntu.com/p/TRCnfF6h6c/ [19:53] cmatsuoka, any idea? [19:53] this is the test sequence: google:fedora-32-64:tests/main/degraded google:fedora-32-64:tests/main/interfaces-broadcom-asic-control google:fedora-32-64:tests/main/login google:fedora-32-64:tests/main/snapshot-cross-revno google:fedora-32-64:tests/main/interfaces-content-mimic google:fedora-32-64:tests/main/refresh-undo google:fedora-32-64:tests/main/media-sharing google:fedora-32-64:tests/main/snap-switch google:fedora-32-64:tests/main/try-snap-goes-awa [19:53] y:test_snapd_service google:fedora-32-64:tests/main/security-seccomp google:fedora-32-64:tests/main/core18-with-hooks google:fedora-32-64:tests/main/security-dev-input-event-denied google:fedora-32-64:tests/main/snap-run google:fedora-32-64:tests/main/interfaces-content-mkdir-writable:snap google:fedora-32-64:tests/main/retryable-error google:fedora-32-64:tests/main/snap-handle-link google:fedora-32-64:tests/main/interfaces-location-control [19:53] google:fedora-32-64:tests/main/interfaces-netlink-audit google:fedora-32-64:tests/main/snap-mgmt google:fedora-32-64 .../tests/runtime-state# [19:54] I'm tired, let's fight this next week [19:54] have a great weekend cachio, ijohnson, cmatsuoka! [19:54] ttyl zyga-x240 [19:54] zyga-x240, you too [19:54] you have a good weekend too! [19:54] cachio: looking now [19:55] ijohnson, something related to tpm [19:55] cachio: where did you see that? in a nested VM ? [19:55] ijohnson, yes [19:55] when booting a beta image [19:55] hmm, which image? built locally or from cdimage ? [19:55] built locally [19:55] cachio: this is a strange error [19:56] sergio@cachiomachine:~/workspace/snapcore/snapd$ export SPREAD_BUILD_SNAPD_FROM_CURRENT=true [19:56] sergio@cachiomachine:~/workspace/snapcore/snapd$ export SPREAD_ENABLE_KVM=true [19:56] sergio@cachiomachine:~/workspace/snapcore/snapd$ export SPREAD_ENABLE_KVM=false [19:56] sergio@cachiomachine:~/workspace/snapcore/snapd$ spread -debug google-nested:ubuntu-20.04-64:tests/nested/manual/refresh-revert-fundamentals:base [19:56] this is for reproduce it [19:56] is it happening in master because it failed during the nightly suite [19:56] cachio: let me try to reproduce, is this with master ? [19:57] cachio: got it [19:57] cachio: it seems that it's a signature mismatch [19:57] use kvm = false [19:57] cachio: running now, let's see what happens [19:58] ijohnson: maybe something related to the dual signed components? [19:58] cachio: I'll try to reproduce it here [19:58] ijohnson, cmatsuoka thanks [19:58] cmatsuoka: i saw xnox say that dual signed shim? I think was ready and he wanted to upload it, could be he uploaded it and things actually aren't ready for it [19:59] cmatsuoka: I notice that pc gadget has new snap from yesterday [19:59] it is happening just when I create a beta image [19:59] if I create an image from edge works well [20:00] cachio: any special step you're taking? are you just re-signing the beta gadget? [20:00] cmatsuoka, we are not modiying nothing on that test [20:01] it has defined this: [20:01] BUILD_SNAPD_FROM_CURRENT: false [20:01] USE_CLOUD_INIT: true [20:01] ENABLE_SECURE_BOOT: true [20:01] ENABLE_TPM: true [20:01] so, no modifications for kernel or gadget [20:02] cachio: so it's a locally built beta image, with snapd from master? [20:02] yes [20:02] ok, I'll build one here [20:02] I have the command line used in the test [20:03] this -> /bin/ubuntu-image --image-size 10G /home/gopath/src/github.com/snapcore/snapd/tests/lib/assertions/nested-20-amd64.model --channel beta --output /tmp/work-dir/image/ubuntu-core-20-beta.img [20:03] cachio: so snapd is from master, and you're not injecting snap-bootstrap, is that correct? [20:04] snapd is from beta [20:04] kernel gadget and core20 are from beta [20:04] as well [20:04] ah, nothing changed then [20:04] ok [20:04] no [20:04] it is an imgage from beta [20:05] can you paste nested-20-amd64.model somewhere, so I can use the same model? [20:07] cmatsuoka, 1 sec [20:08] cmatsuoka, https://github.com/snapcore/snapd/blob/master/tests/lib/assertions/nested-20-amd64.model [20:09] thanks [20:10] yaw [20:22] cachio: ok, reproduced the problem here, I'll investigate [20:22] cmatsuoka, good, thanks [20:25] yeah I reproduced as well, I will let cmatsuoka investigate [20:34] cachio. ijohnson: I suspect this is the dual signed shim and snapd in beta still doesn't know how to deal with it [20:34] cmatsuoka: any idea when snapd edge would have gotten support for it? [20:35] cmatsuoka: I assume probably with an update to secboot vendor.json ? [20:35] ijohnson: let me check the secboot history [20:36] ijohnson: edge has a reasonably updated secboot but I don't know about snapd in beta [20:36] cmatsuoka: right what I mean is we should probably try to update secboot vendor.json to what's on edge before 2.45.4 is released [20:37] err before 2.45.4 is uploaded to beta channel [20:38] ijohnson: let me test it here with a newer snapd... [20:38] cmatsuoka: hmm actually that may not be enough [20:39] cmatsuoka: I see that secboot vendor.json sha on master was last updated with https://github.com/snapcore/snapd/pull/8651 [20:39] PR #8651: release: 2.45 [20:39] cmatsuoka: ah wait nvm I was on the release/2.45 branch haha [20:39] one moment [20:40] cmatsuoka: ok so last update to secboot vendor.json sha on master was from you with https://github.com/snapcore/snapd/pull/8972 [20:40] PR #8972: gadget/install,secboot: use snapcore/secboot luks2 api [20:40] cmatsuoka: so we should back-port that PR to release/2.45 [20:42] ahhhhhh but there's conflicts :-/ [20:43] ijohnson: I'm building an image to check if it actually fixes this issue [20:43] cmatsuoka: thanks I'll try to prep a PR operating under the assumption that it does fix the issue [20:46] ijohnson, cachio: it installed correctly with snapd from edge [20:46] cmatsuoka, yes [20:46] just fails with beta [20:48] I just verified that [20:57] ugh we need to squash merge more PR's [20:57] like the amount of time wasted on trying to cleanly cherry-pick commits is ridiculous [21:00] ijohnson: did the API change in a way that we can't just update secboot? I don't remember really [21:00] cmatsuoka: maybe we could just try to update secboot [21:01] cmatsuoka: I don't fully understand all the changes that have happened, as there are numerous PR's from you and Maciej that are "related" to using secboot / gadget / partitioning that are pre-reqs for the single most recent PR where I assume that secboot was updated with [21:01] ijohnson: let me check there, just a sec [21:01] cmatsuoka: perhaps it would just be quicker to try and build a snapd with an updated secboot, do you want to try that quickly ? [21:01] thank you [21:09] ijohnson: this is interesting, I think the version we have there is from 2020-05-12 and the fix was commited in 2020-05-13 [21:09] haha wow [21:11] ijohnson: so should we just apply that patch to secboot, or update all the way to the current version? [21:11] cmatsuoka: can you build snapd from release/2.45 branch with just updating the version of secboot in vendor.json ? [21:11] I can also try if it's too late for you [21:11] ijohnson: I can do it [21:11] in fact I'm very curious about the outcome of this test [21:12] cool, yeah just checkout release/2.45, then `govendor update github.com/snapcore/secboot` or something to update the dependency in the vendor.json and then build snapd exe and inject it into the snapd snap [21:32] ijohnson: ok, it worked [21:33] ijohnson: I'll format a PR for that [21:42] cmatsuoka: awesome, thanks! [21:42] Have a nice weekend [21:42] * ijohnson EODs [21:45] ijohnson, good weekend [21:47] PR snapd#9123 opened: vendor: update secboot to support dual signed EFI binaries [21:49] cmatsuoka, with that PR the issue in beta will be fixed at some point right? [21:49] cachio: that's the idea [21:49] cmatsuoka, nice, thanks for htat [21:49] that [21:50] I hope it works :) Perhaps you could test it to see if it fixes the installation for you [21:51] cmatsuoka, how should be the test? [21:51] I need to build snapd with the that branch? [21:52] cachio: uhm, inject snapd built from that branch into a the snapd snap from beta and run the test [21:52] yes [21:52] or you can build the entire snap instead of injecting [21:52] ok, I'll try that [21:52] I don't know what method fits better in your workflow [21:55] I'll try to inject the code in the snapd snap [21:55] I need to see [21:56] cachio: either use published beta images, or self build edge ones. [21:56] cachio: there will be new snapd in beta on Monday, which will work with newly built beta images. [21:56] is it failing with the images which are built during the test [21:57] xnox, yes, but cmatsuoka did a change, not sure how it will affect next beta [21:58] cmatsuoka, is it needed that change you did for 2.46? [21:59] cachio: it only fails for 2.45 AFAIK [22:00] cmatsuoka, yes, but next week mvo will create a 2.46 and send it to beta [22:00] ah in this case it should work automatically [22:00] so perhaps your change needs to be on that beta [22:01] cmatsuoka, ok [22:01] so perhaps it is better to wait for that beta? [22:01] I opened a PR against release/2.45, is there a branch for 2.46 already? [22:02] cmatsuoka, I think mvo will create it early next week [22:02] ah ok [22:02] and use all what we have in master [22:02] I thought there would be one more 2.45 going to beta, but if that's not the case then the patch is unnecessary [22:03] cmatsuoka, yes, but today mvo said next week he was going to branch and send 2.46 pre release to beta [22:04] cmatsuoka, but not sure which day [22:04] but it is ok to have that pr on 2.45 [22:04] because it we need a new point release on beta it is going to be required [22:04] ah ok if 2.46 goes to beta we can just discard that patch [22:05] lets discuss it on Monday with mvo [22:06] xnox: but the snapd that we were going to put in beta won't work hence we need cmatsuoka's PR [22:06] ijohnson: 😭 [22:07] cachio: cmatsuoka: yes we need to discuss with mvo if we will do snapd 2.46 as beta or if we will go ahead as planned with 2.45.4, the latter does not currently have the fix for this dual-signed shim issue [22:07] ijohnson, agree [22:08] if we do end up not doing a 2.45.4, then we don't need to fix release/2.45, but aiui we are still waiting on a couple things before we could branch 2.46, so beta would be broken until we land those other things (which are not uc20 related) [22:08] ok [22:09] my wife politely suggests that's time for me to handle SIGEOW, so I think I should do that [22:09] have a nice weekend! [22:09] cmatsuoka, you too [22:10] ijohnson: I have gadget that is single signed shim, with BootHole proof grub. I can revert that too, if you need to do 2.45.4 [22:10] ijohnson: but from a meeting earlier today I thought the plan was to have 2.46 beta in beta channel on Monday. [22:13] xnox, that's the idea, but also could be a 2.45.4 [22:17] cachio: well, sync on Monday. [22:20] xnox, sure, good weekend [22:22] xnox ah well I wasn't in that meeting, and that meeting happened after we talked about it during standup [22:24] * cachio EOW