[05:56] <zyga-mbp> good morning
[07:01] <pstolowski> morning
[07:14] <mvo> good morning pstolowski 
[07:15] <pstolowski> mvo: hi! could you please merge https://github.com/snapcore/snapd/pull/10447 ? it only failed on the lxd test that cachio fixed with the other PR
[07:15] <mvo> pstolowski: sure
[07:16] <mborzecki> meh, damn matrix bridge
[07:16] <mborzecki> mvo: pstolowski: hey
[07:17] <mborzecki> i've been sitting there since 630, wondering why nobody has showed up
[07:19] <zyga-mbp> mborzecki hahaha
[07:19] <zyga-mbp> I've been here :)
[07:19] <zyga-mbp> well, since around 8
[07:19] <zyga-mbp> I've filed https://git.ostc-eu.org/OSTC/tools/oh-spread/-/issues/8
[07:20] <mvo> mborzecki: good morning to you as well :)
[07:20] <mvo> mborzecki: hahaha
[07:21] <mvo> zyga-mbp: ! good morning
[07:21] <mborzecki> mvo: yeah, heh
[07:21] <zyga-mbp> and also https://git.ostc-eu.org/OSTC/tools/oh-spread/-/issues/9
[07:21] <zyga-mbp> if anyone from the snapd team is interested I would love to include you in code review
[07:21] <zyga-mbp> if not that's fine, I'll just push on 
[07:21] <mborzecki> mvo: mardy is around 730 usually, then you join ~830 and pawel joins around 9, so something didn't add up ;)
[07:23] <mborzecki> heh, so some people in #irc:matrix.org have noticed equally confusing behavior
[07:27] <mborzecki> mvo: can you take a look at https://github.com/snapcore/snapd/pull/10432 ?
[07:31] <mborzecki> heh, so i'm confused, libera.chat are running their own bridge to matrix, but it seems like it's out of sync :/
[07:32] <zyga-mbp> mborzecki matrix is out of sync!
[07:32] <zyga-mbp> mborzecki the agents are coming ;)
[07:32] <mborzecki> haha, took the wrong pill this morning ;)
[07:32] <mborzecki> waiting for agent smith to knock on my door
[07:38] <mborzecki> ok, going to school, back in 1.5h or os
[07:38] <mborzecki> so
[07:40] <pedronis> mvo: hi, we are getting more? random core 20 failures on https://github.com/snapcore/snapd/pull/10446, I don't think it's the new code (because it deterministic) but it's annoying
[07:41] <mvo> mborzecki I will look at 10432 sure! 
[07:41] <mvo> pedronis: let me check
[07:47] <pstolowski> mvo: can you land that one too? https://github.com/snapcore/snapd/pull/10428 (also, only lxd test failure fixed by cachio)
[07:49] <mvo> pstolowski: sure
[07:50] <pstolowski> mvo: thank you!
[08:14] <mvo> 10448 needs a second review (should be simple)
[08:36] <pstolowski> pedronis: i've updated error handling with ignore flag in https://github.com/snapcore/snapd/pull/10408, you should probably look at the last commit
[08:47] <pedronis> pstolowski: I'll look in a bit
[08:49] <pstolowski> ty
[09:14] <pedronis> mvo: question in https://github.com/snapcore/snapd/pull/10449/ ?
[09:26] <mvo> pedronis_: looking
[09:27] <mvo> pedronis_: I can remove the random things in the end, I misunderstood the initial suggestion. will fix it, sorry
[09:41] <mborzecki> re
[09:44] <mborzecki> hm still doesn't work
=== pedronis_ is now known as pedronis
[10:34] <dot-tobias> hi everyone
[11:05] <rbasak> Is this a core snap regression? https://bugs.launchpad.net/snapd/+bug/1933392
[11:05] <rbasak> I've not tested, but I'm not sure if that report is in the right place
[11:05] <rbasak> It's apparently regressing the certbot snap on Debian.
[11:09] <dot-tobias> Building a fresh Core image with ubuntu-image, I currently get this fatal error: invalid layout of volume <myvolume>: cannot resolve content for structure #0 at index 0: cannot find "dtbs" in kernel info from "/tmp/tmplm3ypa5a/unpack/kernel"
[11:09] <ogra> rbasak, you could never "run snapctl on the host" ... that only works internally 
[11:11] <dot-tobias> Building an image worked yesterday around 3pm UTC+1, but I saw that the pi-kernel snaps from the 18-pi channel have been updated yesterday. Found https://forum.snapcraft.io/t/failed-to-build-pi-uc20-image/24918 but using edge snaps isn't really an option for prod images.
[11:11] <ogra> not sure about the REST API stuff, but the call to snapctl is definitely nt supposed to work unles you run it from "snap run --shell ..." from inbside some snap 
[11:11] <ogra> geez ... my typing ... must be friday ... 
[11:14] <pstolowski> jamesh: hi, there is a todo: check g_icon_serialize in your notification prototype; is it optional? is gtk notification api able to deal both with icon names & serialized icon?
[11:15] <ogra> dot-tobias, https://forum.snapcraft.io/t/changes-in-pi-pi-kernel-dtb-handling/24369 update your gadget 
[11:15] <rbasak> ogra: can you glance at the github issue linked please? In that case, it _is_ running from inside the snap. The reporter tried to generalise it in the LP report.
[11:15] <ogra> aha 
[11:20] <dot-tobias> ogra: I've cherrypicked https://github.com/snapcore/pi-gadget/commit/e452ccb3c6608e6f1de85e51e161706e780f01b1 and https://github.com/snapcore/pi-gadget/commit/fd40afabf0f5bcfcebb673a16a4046a0aa4234e6 to my gadget, that resolved the original error (can't remember the wording) and produced a working image. The same gadget, which includes said changes and worked yesterday, now suddenly fails … I'll triple-check my gadget to be sure
[11:35] <mborzecki> dot-tobias: it's a new thing, pi-kernel must have finally caught up https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1907056
[11:35] <mborzecki> tbh the feature was made specifically for pi
[11:53] <pstolowski> cachio_: do you have a system where snap debug state hangs?
[11:54] <cachio_> pstolowski, let me reserve that
[11:55] <pstolowski> cachio_: i actually just need state.json from such system, if you can download one that would be enough
[11:55] <cachio_> pstolowski, I already reserved 1 device
[11:55] <pstolowski> ok thanks
[11:55] <cachio_> it is being provisioned
[12:14] <dot-tobias> mborzecki: Thanks, I reverted the changes to my gadget, image is building again. Looking forward to have DTBS from the kernel snap, though 😊 
[12:17] <ogra> yeah, it is definitely the best thing since sliced bread !
[12:19] <ogra> rbasak, (sorry, got dragged into a meeting) if you suspect a core regression the first thing i'd do is a snap revert core and compare ... ad given the curl command in the bug i'd also check if snap login makes a difference 
[12:19] <ogra> i doubt the core changes are that big nowadays given 16.04 is ESM ... so such a regression is not super likely (possible still indeed)
[12:22] <rbasak> ogra: I don't have a reproducer here readily available unfortunately - I was just asked because I wrote the snap originally.
[12:22] <rbasak> So I don't have anything to revert.
[12:22] <rbasak> According to https://github.com/certbot/certbot/issues/8922#issuecomment-868451477 microk8s is affected too
[12:23] <rbasak> I suspect maybe a regression in snapd rather than core? Anyway I'm just passing on the message.
[12:23] <ogra> yeah, perhaps ... 
[12:23] <ogra> in either case soeone should ask in the issue to revert core to the former version to see if it is really the new core 
[12:24] <ogra> its the easiest bit to verify this 
[12:24] <rbasak> If it reproduces as reported then that's a bug, no?
[12:24] <rbasak> I'd like for upstream to do the debugging :-/
[12:24] <ogra> it is 
[12:25] <ogra> ... but it helps a lot if the reporter can nail down the two revision of core so you know where to lokk 
[12:25] <ogra> *look
[12:26] <rbasak> It's not possible to do a bisection in the general case, because previous snap binaries are not made available except to the snap developer.
[12:27] <rbasak> That decision has a cost: you can't reasonably expect affected users to do that kind of debugging.
[12:58] <ogra> rbasak, the last local revision is still there ... indeed that doesnt help when you install snapd/core from scratch 
[12:59] <ogra> so affected users shuld be always able to revert ... just *new* users wont be (as the only have the current core version they just freshly installed)
[13:01] <rbasak> ogra: well, OK, but affected users can't delegate - eg. I can't help.
[13:02] <rbasak> Or, at least, it's artificially made difficult for me to help
[13:02] <ogra> yes
[13:48] <pedronis> rbasak: ogra: fwiw, we have asked somebody on the team to investigate this
[13:48] <ogra> 👍
[13:49] <rbasak> Thanks!
[14:11] <mborzecki> pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/10460 ? sergiusens reported this in the forum https://forum.snapcraft.io/t/snap-download-slips-in-extra-logging-with-download-size/25151
[14:12] <pstolowski> looking
[14:20] <mborzecki> cachio_: can you try to update the tumbleweed image?
[14:20] <cachio_> mborzecki, sure
[14:21] <mborzecki> cachio_: thanks, hoepfully the image will be better this time ;) the builds in obs keep on working
[14:21] <cachio_> mborzecki, nice
[14:22] <cachio_> I'll create a new one right now
[15:35] <pstolowski> pedronis: the debian issue seems to be caused by /run/snapd-snap.socket vs /run/snapd.socket rename; debian 9 has a very old snapd 2.21 deb, the old snapctl from the deb on the host can't talk to new snapd
[15:36] <pedronis> pstolowski: that's not a rename, or I'm cofunsed we have two sockets
[15:37] <pedronis> pstolowski: one is for snapctl and one is for the general api
[15:37] <pedronis> pstolowski: /run/snapd.socket  is the general API socket (use by the snap command for example
[15:37] <pstolowski> pedronis: yeah, sorry, no rename, we have two indeed. what i see with strace is the old snapctl is trying to talk to /run/snapd.socket
[15:38] <pstolowski> did we have just one socket in the past?
[15:38] <pedronis> not sure, we need to go dig into the release branches I suppose
[15:40] <pstolowski> good idea
[15:41] <pedronis> pstolowski: it was already like that:  https://github.com/snapcore/snapd/blob/release/2.29/cmd/snapctl/main.go#L38
[15:41] <pedronis> so maybe there's no socket or something is wrong with apparmor?
[15:41] <pedronis> deos the socket exist?
[15:41] <pedronis> I mean which sockets are in /run
[15:42] <pstolowski> pedronis: that's 2.29 though, we should be looking at 2.21
[15:42] <pedronis> ah, sorry I misread
[15:43] <pedronis> seems we didn't have two sockets then
[15:43] <pedronis> but I'm still confused why is the snapctl new though?
[15:46] <pstolowski> pedronis: we have two sockets because snapd 2.51 comes from core. but afaiu the code of snapctl from 2.21 (which comes from the deb) expects to talk to /run/snapd.socket
[15:47] <pedronis> pstolowski: sorry, you mean snapctl comes from core, but snapd comes from the 2.21 deb?
[15:49] <pstolowski> pedronis: no. snapd comes from core, snap --versin shows 2.51. but for snapctl we don't reexec, so the one from deb 2.21 is used
[15:49] <pedronis> because this is a classic snap?
[15:50] <pstolowski> pedronis: i only tried snapctl manually so far, let me try a classic snap
[15:51] <pedronis> I suppose you want to install a confined snap and a classic one, and try snapctl from their snap run --shell
[15:52] <pstolowski> pedronis: i tried confined snap it was fine, it used snapctl from core
[15:54] <pstolowski> pedronis: yep, classic snap uses snapctl from the host, so fails
[15:54] <pedronis> this is not a regression though, I expect things behaves like this since a long time
[15:54] <pedronis> I wonder what changes, and why a classic snap is using snapctl at all?
[15:55] <pedronis> what changed
[15:55] <pedronis> pstolowski: I suppose we need to try to snap mentioned in the bug? I don't know why it is using snapctl
[15:56] <pedronis> rbasak: did certbot always use snapctl? this doesn't seem a regression
[15:58] <rbasak> pedronis: it has done for a long time
[15:58] <rbasak> Let me find the thread
[15:58] <pedronis> anyway I'm not quite sure how it ever worked on debian 9
[15:59] <pstolowski> pedronis: yes this seems to be very old
[16:01] <rbasak> pedronis: see https://forum.snapcraft.io/t/certbot-request-for-classic-snap-approval/6240/19?u=rbasak and the following post
[16:01] <pstolowski> pedronis: it uses snapctl unset from prepare-plug-plugin, i think that's all
[16:01] <rbasak> That's why it's using snapctl - it was a condition of classic snap approval to check for that
[16:02] <pedronis> anyway as I said unsure it ever worked on debian 9 given the version of snapd it shipped with. We could think whether there's a way to fix this, but is not new afaict
[16:02] <rbasak> pedronis: I might have jumped the gun in assuming it was a regression. Now that I look at the report, I don't see the reporter actually saying that.
[16:03] <rbasak> Apologies for that
[16:04] <rbasak> I suppose it's possible that nobody has tried using the certbot snap with a plugin on Debian 9 before?
[16:04] <pstolowski> one sec
[16:04] <rbasak> Given how well used it is, that does seem odd to me still though.
[16:05] <rbasak> But also, installing core seems to regress it? The certbot snap doesn't need core - IIRC it's based on core20.
[16:05] <pedronis> it might also depends what other snaps are installed
[16:06] <pedronis> though given that old snapd it might always install core as well
[16:06] <pedronis> not sure
[16:07] <pstolowski> the bug report isn't complete afaict
[16:07] <pstolowski> the github issue it links to is complete
[16:07] <pstolowski> snap set certbot trust-plugin-with-root=ok followed by snap install certbot-dns-cloudflare
[16:07] <pstolowski> triggers it
[16:08] <pstolowski> because of prepare-plug hook uses snapctl
[16:09] <pstolowski> anyway.. it's pretty clear the old snapctl from the deb can't talk to new snapd and must be like that for a few years already. i can also reproduce it with snap run --shell certbot