[00:06] PR snapd#8513 opened: snap-bootstrap: lock access to sealed keys [05:19] Good morning [05:19] * zyga is tired after working past midnight [05:51] morning [05:52] Hey [05:57] Made some more progress last night [05:57] But tired now [05:57] I’ll start at around 9 [06:14] mvo: zyga: hey [06:20] mborzecki: good morning [06:24] Hey mvo [06:25] hey zyga [06:25] How is bit cacher? [06:25] zyga: seems to be working, running a real spread run this morning to see if it behaves [06:26] Cool [06:26] I guess we could then return to phased testing, with one build of snapd deb/snap [06:27] zyga: yes, I think those can be distributed via artifcats [06:28] That would be superb. [06:32] PR snapd#8510 closed: boot/bootstate20: small changes to bootloaderKernelState20 [06:40] PR snapd#8507 closed: packaging: fix build on Centos8 to support BUILDTAGS [06:50] mvo: i'm going to file a bug in rhel about the build tags macro [06:50] just upadting my rhel8 vm [06:53] mborzecki: nice, thank you! fwiw, the git report referenced in the code has what we need [07:01] morning [07:07] * zyga starts the day [07:13] PR snapcraft#3047 closed: repos: fix returned strings for install_stage_packages() [07:20] pstolowski: hey [08:08] my dbus test passes without hanging for the first time [08:08] let me clean it up a little and propose :) [08:28] mvo: can you grant me edit rights to the doc? [08:28] pstolowski: please reload, you should have it [08:31] PR snapd#8514 opened: Extend openvswitch interface to support ovs-appctl [08:32] mvo: ty [08:35] mborzecki: hey, just fyi - ondra shared some fun initrd building snippet https://git.launchpad.net/~ondrak/ondras-snaps/+git/linux-kernel/tree/snapcraft.yaml?h=ubuntu-focal-qcomlt-uc20#n50 [08:36] mvo: mhm, right, that's what we do too [08:36] mborzecki: aha, ok [08:37] otherwise it's lots of pain [08:39] pedronis: hey, not sure if you saw my yesterday's finding (pastebin from the evening)? mvo figured out the reason (it's due to how writable is handled) and proposed how to handle it; we discussed it in a HO this morning. i'm sharing this notes with you and we can discuss when you have a moment today [08:40] pstolowski: mvo: did you discuss what to do about /etc/cloud, it has the same problem? [08:47] pedronis: the soution outlined in the gdoc (number c) should apply to all of them equally [08:47] pedronis: it's defined as synced per https://github.com/snapcore/core18/blob/master/static/etc/system-image/writable-paths#L54 [08:49] mvo: ok, I'll wait for the doc to be finished [08:51] pedronis: it's finished, (d) is really a bit of a distraction [08:51] pedronis: it's super interessting but I'm not sure if it's worth it given that we also need to support uc16/uc18 [08:52] mvo: c for 16/18 needs initramfs changes there, right? [08:52] pedronis: correct [08:52] huh, whole session froze [08:52] mvo: maybe if we need to do changes, we should do the same thing as uc20 and call a hook inside the bases? [08:53] but upgrades [08:53] there's probably a way to do that though [08:54] reexec like [08:54] pedronis: that's a very interessting idea, let me look at the old initramfs code [08:55] pedronis: I like this, should be totally doable (also in a backward compat way). it's a bit risky as it involves initrd changes but would be nice and clean and more uniform [08:56] mvo: notice though that we can't probably remove any synced from core18 writable-paths [08:57] because we don't know what people did with ubuntu-image hooks [08:57] we can/should remove them from core20 though [08:58] pedronis: +1 [08:59] mvo: there is probably some other wrinkle as well, related to prepare-image [09:00] I'll write some notes in the doc, when I have a moment. Finishing something else atm [09:00] pedronis: thanks! adding some notes myself too [09:04] mvo: ok, yes, one point I wanted to make was indeed iv [09:07] mvo: pstolowski: made a couple of comments [09:07] pedronis: thanks for adding the comment about the fallback! [09:08] thanks [09:16] PR snapd#8515 opened: testutil: add NewDBusTestConn [09:21] I pushed the test helper alone here [09:21] and the tests are in https://github.com/snapcore/snapd/pull/7825 [09:21] PR #7825: many: use transient scope for tracking apps and hooks [09:24] mborzecki: hey, what's the plan with #7414? [09:24] PR #7414: tests: keep track of installed packages and restore the state after the test [09:24] I will write a few more tests to exercise all the interactions [09:24] pstolowski: probably needs a merge from master and making sure it still works [09:24] but now I need a break, my back is killing me [09:26] need to run an errand, back in 1.5-2h, hopefully sooner [09:38] PR snapcraft#3048 opened: V2 dump plugin [10:24] * zyga is pissed off at go fmt [10:24] can we move to non-ancient go fmt [10:24] it doesn't matter for compiling [10:24] is anyone in the team still using "go fmt" from 1.6 because they cannot use newer go? [10:25] I have to fight my editor to hand-craft restore some alignment [10:30] I made an alias go-fmt that calls the old go fmt from a snap installation fwiw, I'm also happy to discuss how to switch to something newer, but probably after the beta at this point [10:37] PR snapd#8516 opened: github: format with modern go [10:42] PR snapd#8516 closed: github: format with modern go [10:43] pedronis: huh [10:43] zyga: what part of until beta you didn't get? [10:43] pedronis: I just read you writing that [10:43] pedronis: and you could have just marked it as blocked or something [10:43] pedronis: not nice [10:47] zyga: if we don't land it, it will need tending, so I don't see how marking it blocked helps [10:47] zyga: I don't know what people workflows are like, I don't think I want to even discuss that before beta [11:12] it seems that jenkins deb is not authenticated in special-home-can-run-classic-snaps, despite adding the key in the test? i saw this test failing on some PRs [11:13] re [11:19] PR core-build#59 opened: initramfs: add new handle_writable_defaults() [11:20] pstolowski: ^- that is probably all we need, the interessting part if of course the next step where we test this :) [11:20] pstolowski: we can look into this after the standup [11:21] zyga: fwiw 8508 works, thanks again for your help with that [11:21] mvo: oh thank you, i didn't expect you to take this! thanks! [11:22] pstolowski: no worries, it's just a tiny step [11:22] * mvo gets some lunch [11:22] mborzecki: (8505 - your caching idea ftw!) [11:22] * mvo really gets lunch [11:47] PR snapd#8517 opened: tests: temporarily allow unauthenticated jenkins deb [11:48] ^ a bit scary, perhaps we should disable the test for now.. [12:00] so much weirdness [12:04] pstolowski: we probably need mvo input on that one [12:25] cmatsuoka: hi, i seem to be getting a differnt output from lsblk --fs --json than the tests expect [12:26] mborzecki: that sounds bad, what test? [12:27] cmatsuoka: here's the test data vs what i get from the tool https://paste.ubuntu.com/p/qSCBCC44zF/ [12:28] cmatsuoka: the parsing only handles blockdevice.[i] elements, does not descend into blockdevice.[i].children[j] [12:28] mborzecki: I think the test output is simulating the output when lsblk is called directly on the partition and not the disk [12:28] aah w8 [12:28] right [12:29] does it make sense? [12:29] yeah, it does now [12:29] meh, more mocking in tests :/ [12:29] ah ok good [12:53] morning folks [12:54] pstolowski, hei, there is a beg assigned to you, lp:1872115 [12:55] pstolowski, did you see that? is there a priority for that? [12:57] hey ijohnson [12:57] hey cmatsuoka o/ [12:59] cachio: I had problems with the tpm branches yesterday, will try the sb test again after the SU [12:59] cmatsuoka, sure [12:59] cachio: no, i didn't see that, thanks. i guess zyga assigned it to me because of my other rsyslog work, need to check [12:59] cmatsuoka, ping me if you need any help [13:01] ugggh pulseaudio [13:01] Bug #1873452 opened: 'snap install' | "INFO Waiting for restart..." [13:24] hi diddledan are you around? [13:24] ello [13:25] diddledan: I got a request from the mycroft devs to have the snap transferred to them. Are you OK with that? [13:25] yup [13:25] sweet, ok thanks, all I needed was confirmation :) I'll take it from here [13:25] happy Friday [13:25] coolbeans :-) [13:28] PR snapd#8517 closed: tests: temporarily allow unauthenticated jenkins deb [13:31] PR snapd#8518 opened: tests: disable special-home-can-run-classic-snaps due to jenkins repo issue [13:41] PR snapcraft#3046 closed: plugins: introduce v2.GoPlugin [14:01] ijohnson: to be clear I'm not sure if there are different scenarios were we need that divergent snap, we don't with the current tests [14:02] I don't think we will need it, I'm not sure what scenarios we could end up with that, where the kernel the bootloader boots is not the one we want as the "normal" kernel on the bootloader after MarkSuccessful runs [14:03] ijohnson: that sounds right [14:03] we still want to check before and after though [14:03] just for the same value, to be clear [14:03] what do you mean check before and after? [14:03] before and after the MarkSuccessful ? [14:04] yes [14:04] I suppose [14:04] hmm ok I can make that work [14:05] ijohnson: I thougth, it's an easy change, we are probably talk cross-purpose [14:11] PR snapcraft#3049 opened: build providers: rename default sources [14:26] jdstrand: hi, are you around? [14:46] hi, I'm having an issue. my lxd snap suddenly broke, looking at /snap/lxd there's no snap dir there, just a broken "current" link [14:49] roadmr: I'm here today but stepping away for a little bit. please ask and I'll answer when I get back [14:49] and no lxd snap file in /var/lib/snapd/snaps [14:50] jdstrand: remember https://dashboard.snapcraft.io/snaps/inkscape/feedback/ timing out? it's because the page lists thousands of items. Are you OK with paginating that by e.g. 100 items at a time? [14:50] PR snapcraft#3050 opened: meta: split up package repository sanity checks [14:51] roadmr: I can answer that now [14:51] roadmr: yes. please yes [14:51] :) [14:51] \o/ hehe ok jdstrand we'll do it that way then [14:52] zyga: you have mail :) [14:52] * jdstrand steps away [14:54] is there a way I can make snapd happy and mount the lxd snap again? [15:00] error: cannot communicate with server: timeout exceeded while waiting for response [15:01] My machine ^. Can't install or update anything. [15:12] PR snapcraft#3051 opened: project: introduce 'keys' for project assets [15:15] popey: journalctl -u snapd [15:15] ? [15:16] ijohnson: pushed the changes to https://github.com/snapcore/snapd/pull/8487 and marked it as ready for review [15:16] PR #8487: [RFC] gadget, cmd/snap-bootstrap: hack some things into place to get rpi4 semi-working [15:16] https://paste.ubuntu.com/p/mDrRqBJNrZ/ pstolowski [15:16] thanks mborzecki [15:16] I'll give it a try on my pi today [15:17] popey: can you still run snaps? [15:18] popey: ok, not much interesting there, anything else in the logs? [15:18] yes, i can run snaps [15:19] i just can't refresh, or it does, but i get that error [15:20] apparently deamon is not running [15:20] mvo, zyga: https://discuss.linuxcontainers.org/t/unable-to-start-snap-lxd-on-debian-10/7456/17?u=stgraber [15:20] this feels like pretty crap experience :) [15:20] stgraber: in a meeting right now, but I though we did test lxd on debian :( [15:21] mvo: reexec [15:21] mvo: backports [15:21] stgraber: can you add assumes to lxd? [15:21] stgraber: and at least put a FAQ that you need to snap install core [15:21] it will break in a nicer way IMO [15:22] unless we update backports we have few optiosn [15:22] *options [15:22] stgraber: I totally agree about the UX [15:22] popey: 'snap changes' fails the same I presume? [15:22] yeah, assumes should fix things [15:22] zyga: where is the doc on assumes? [15:22] pstolowski snap changes returns immediately [15:23] https://snapcraft.io/docs/snapcraft-top-level-metadata#heading--assumes [15:23] stgraber: ^ add the assumess snapdX.YY version that meets your expectations [15:23] ah yeah, just found it too [15:23] zyga: so what's the minimum version that's known to behave? [15:24] stgraber: perhaps we could patch snapd in debian 10 to say something like [15:24] PR snapcraft#3049 closed: build providers: rename default sources [15:24] stgraber: anyone that you pick as right for you [15:24] stgraber: it depends on a snap [15:24] zyga: well, in this case the issue is apparently the snapd didn't connect any of the interfaces, so I have no idea when you fixed that :) [15:24] popey: anything else in the logs? a crash/panic? [15:24] stgraber: perhaps we could patch debian 10 snapd to say "you need snapd X.YY, you may try to get it from backports or by installing core" [15:24] zyga: we purposefuly don't use anything new/fancy in the snap [15:24] pstolowski not that I have seen :( [15:25] stgraber: put a version that you tested as ok and that's enough [15:25] stgraber: it's fine to depend on new snapd [15:25] popey: and what does 'systemctl status snapd.service snapd.socket' say? [15:25] pstolowski, popey: this is assertion check being >> than snap client timeout [15:25] IIRC we ran into it before and debugged it to that [15:25] but I could be wrong [15:29] zyga: looks like 2.39 and higher based on the versions our CI sees on the distros we test [15:31] pstolowski https://paste.ubuntu.com/p/zPcXzjtpNJ/ [15:34] zyga: can you elaborate on the assertion check, what do you mean? [15:34] pstolowski: that's for stgraber [15:34] pstolowski: it's unrelated to the issue you are looking at for popey [15:34] I can explain if you want to, just wanted to make this clear [15:35] ack [15:35] popey: and 'snap --version' ? [15:35] https://paste.ubuntu.com/p/zNtPDNF8ns/ [15:36] pstolowski: to reproduce install 300 snaps [15:36] and refresh [15:36] I have 113 installed [15:36] pstolowski: if I'm correct your work on assertion refreshes will help immensly [15:36] pstolowski: but would be good to undestand this in depth, perhaps there are other inefficiences [15:37] (each from the store, not local) [15:37] hmm, we could actually use [15:37] zyga: i'm not working on assertion refreshes afaik ;) [15:37] test-snapd-manysnap-$(seq...) [15:37] in the store [15:37] pstolowski: oh, sorry, I though that was in the new store API work [15:37] anyway [15:37] no i was new search v2 api [15:38] *it* [15:38] I see [15:39] PR snapcraft#3052 opened: build providers: wait for systemd and better nameserver setup on LXD [15:39] zyga: I thought pedronis was working on bulk assertion refreshes [15:40] yes [15:40] ijohnson: yeah, I must have confused the store work with "anything related to store" [15:40] is this issue related to the internet? must be the store [15:40] and that folks is how you do investigative work [15:41] ijohnson: IIRC just that we cannot grab the lock [15:41] ijohnson: needed to produce most "snap foo" responses [15:41] * ijohnson was joking [15:41] ah :) [15:41] sorry [15:41] I have no idea honestly what this issue is [15:41] * zyga should really EOD [15:42] I've reviewed half of James' user session services PR [15:42] yeah, let's rest [15:42] PR snapd#8518 closed: tests: disable special-home-can-run-classic-snaps due to jenkins repo issue <⚠ Critical> [15:42] alright master should be at least not red due to jenkins again [15:43] pstolowski: you wanna quick open up the PR re-enabling the jenkins test? [15:43] if you want I can do it too [15:43] popey: does ps aux|grep apparmor show anything? [15:44] no [15:44] ijohnson: please do, thanks, trying to figure out popey's issue [15:44] ack [15:48] popey: how about 'sudo snap debug state /var/lib/snapd/state.json' ? [15:48] (that looks like snap changes)? [15:48] https://paste.ubuntu.com/p/wHXFrdQhJ2/ [15:49] popey: yes, but it doesn't talk to the daemon [15:50] ahh [15:54] popey: does 'snap list' work? [15:55] yes [15:58] popey: ah, so snapd does respond, i got confused by that first communication error. perhaps store is acting up? how about 'snap find ...'? [15:58] snap find works [15:58] bah, now I'm not getting the error when I refresh! [15:58] pstolowski: my theory is wrong, based on the debug output [16:00] popey: i bet that was a store problem, maybe of limited scope (i was testing with same snapd as you have without issues) [16:01] popey: there was no sign of snapd errors/malfunction in the info you got me so far [16:01] popey: how performant is this machine with 116 snaps? and also what kind of disk is it on? [16:02] popey: given that it keeps finding bugs, maybe turning on debug logging for snapd would be a good idea, even maybe http debug logging too [16:02] it's an ThinkPad T450, i7, 32GB RAM [16:02] brb, meeting [16:02] pedronis: if you have time yet today, 8511 is updated with your comments [16:03] ah actually I forgot one thing, let me push that quickly [16:10] eow, cu all! [16:15] PR snapd#8409 closed: snap-bootstrap: seal and unseal encryption key using tpm [16:16] \o/ [16:17] ijohnson: yes, I'll try to wrap up a couple of things and some reviews after dinner [16:17] thanks pedronis [16:17] congrats cmatsuoka ! [16:17] ijohnson: many fixes still to come [16:18] cmatsuoka: can you merge master into your locking keys PRs now [16:19] pedronis: doing that right now [16:19] thx [16:34] cachio: ok, ready to run the sb/tpm tests [16:34] nice [16:34] cachio: I'm using your spread, which back-end should I use? [16:36] PR snapcraft#3051 closed: project: introduce 'keys' for project assets [16:36] cachio: and should I run on master or tests-extend-uc20-nested-suite? [16:37] cmatsuoka, you should use the google-tpm I sent yesterday [16:38] which test do you want to run? [16:38] uc20-create-partitions-encrypted [16:38] you should add this new tpm backend to the spread.yaml [16:38] ah yes, I remember now, let's do this [16:39] and run spread -debug google-tpm:ubuntu-20.04-64:tests/main/uc20-create-partitions-encrypted [16:39] cmatsuoka, when you run this please tell me so I can monitor the vm [16:48] PR snapcraft#3050 closed: meta: split up package repository sanity checks [16:57] PR snapcraft#3053 opened: snap_packaging: quote final LD_LIBRARY_PATH [17:03] PR snapcraft#3054 opened: Add note about docker snap [17:06] PR snapcraft#3048 closed: V2 dump plugin [17:27] PR snapcraft#3052 closed: build providers: wait for systemd and better nameserver setup on LXD [17:39] PR snapcraft#3055 opened: repo: add identifiers for gpg keys and sources [18:31] hello [18:32] I'm trying to release a snap that uses classic confinement, and need an approval to release it to the snap store. I have started a discourse post here https://forum.snapcraft.io/t/classic-confinement-request-for-nhc-snap [18:33] is there someone from the reviewers team available to possibly give some feedback ^? [18:33] thanks! [18:39] PR snapcraft#3056 opened: remote-build: package up local sources with source-type 'git' [18:41] PR snapd#8325 closed: snap-bootstrap: copy auth data from real ubuntu-data in recovery mode [18:42] PR snapd#8514 closed: interfaces/openvswitch: support use of ovs-appctl [18:42] PR snapcraft#3057 opened: repo: format $SNAPCRAFT_APT_RELEASE instead of ${release} for suites [19:00]