[07:09] <pstolowski> morning
[07:18] <zyga> goooood morning :-)
[07:27] <mvo> good morning pstolowski and zyga!
[07:45] <pstolowski> cjwatson: hi, do you have any eta for the rollout of your cla fix?
[07:53] <jamesh> pstolowski: I think the difference is that the new action doesn't whitelist existing committers, so ends up hitting LP more often
[07:53] <jamesh> Setting the accept-existing-contributors input parameter might get it to pass as often as the old workflow
[07:54] <pstolowski> jamesh: ah, interesting
[07:57] <pstolowski> jamesh: would that be an argument to github action (not familiar with them)?
[07:58] <jamesh> pstolowski: yeah.  I'll put together a simple PR so we have something to discuss
[07:58] <pstolowski> jamesh: thanks
[08:14] <mup> PR snapd#10104 opened: Use x-gvfs-hide mount option to hide squashfs loopback devices in Gnome gvfs <Created by lhotari> <https://github.com/snapcore/snapd/pull/10104>
[08:44] <mup> PR snapd#10105 opened: ci: set the accept-existing-contributors parameter for the cla-check action <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/10105>
[08:45] <jamesh> pstolowski: ^^^ this one will probably get you unstuck
[08:48] <pstolowski> jamesh: +1, thank you!
[08:49] <pstolowski> mvo: ^ can you given 2nd review?
[09:45] <cjwatson> pstolowski: Hopefully today - we have some QA to do on other revisions that are in the way first
[09:46] <pstolowski> cjwatson: great, thank you. in the meantime we may have a remedy from jamesh to lessen the stress on lp (PR above)
[09:46] <cjwatson> pstolowski: Well, fewer requests are always good, but it's not a stress thing
[09:47] <pstolowski> cjwatson: yeah i understand it's a suboptimal query; but at least this should unblock our own landings
[09:59] <jamesh> It's interesting the Launchpad API documentation says "If you want a method to check if a given person is a member of a team, you should probably look at IPerson.inTeam()", which is not exposed by the API
[10:00] <jamesh> Maybe exposing that would let scripts like this perform simpler queries
[10:08] <cjwatson> jamesh: That certainly occurred to me, but I wanted to put out the immediate fire first, especially since it's a short week
[10:43]  * pstolowski lunch
[12:33] <pstolowski> mvo: can you merge https://github.com/snapcore/snapd/pull/10105 manually?
[12:33] <mup> PR #10105: ci: set the accept-existing-contributors parameter for the cla-check action <Skip spread> <Created by jhenstridge> <https://github.com/snapcore/snapd/pull/10105>
[12:51] <pedronis> pstolowski: are you blocked on me atm? I looked at https://github.com/snapcore/snapd/pull/10053 but don't think I can do a full review today
[12:51] <mup> PR #10053: o/snapstate: helper for getting snaps affected by refresh, define new hook <Needs Samuele review> <Refresh control> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10053>
[12:52] <pstolowski> pedronis: no i'm not, thanks for asking
[12:59] <cjwatson> pstolowski,mvo: CLA checks should be fixed now
[12:59] <pstolowski> \o/
[12:59] <pstolowski> cjwatson: thank you!
[13:02] <mvo> \o/
[13:05] <mup> PR snapd#10100 closed: github: revert cla-check action <Skip spread> <Created by bboozzoo> <Closed by stolowski> <https://github.com/snapcore/snapd/pull/10100>
[13:31] <pedronis> pstolowski: there are a couple of failure of the degraded test in https://github.com/snapcore/snapd/pull/10077 and I'm not sure if they are related to the PR or just random (degraded does fail sometimes)
[13:31] <mup> PR #10077: o/configstate: fix panic with a sequence of config unset ops over same path <Bug> <Squash-merge> <Created by stolowski> <https://github.com/snapcore/snapd/pull/10077>
[13:31] <pstolowski> pedronis: ok, i'll investigate, thanks for heads up
[13:35] <jam> with snapd 2.49.2 we are seeing failures for our juju-db snap to be able to start correctly. did apparmor rules change between 2.49.0 and 2.49.2 ?
[13:35] <mup> PR snapd#10106 opened: secboot,boot: provide fde-hooks v2 API interface to hooks <Created by mvo5> <https://github.com/snapcore/snapd/pull/10106>
[13:35] <ijohnson> jam: what system are you seeing this on ?
[13:36] <jam> focal
[13:36] <ijohnson> jam: with stock ubuntu kernel ?
[13:36] <jam> yes
[13:36] <ijohnson> hmm
[13:37] <ijohnson> jam: if you can share more details on a forum post or on a launchpad bug and we'll take a look
[13:37] <ijohnson> please include system journal logs from snapd as well as the output of `snap version` and `snap changes`
[13:38] <ijohnson> also if you are seeing juju-db not starting with 2.49.2, please try reverting with `snap revert` and then see if it works
[13:48] <mvo> pedronis: I will push a PR that *may* help with the prepare issue, I have found an issues here
[13:57] <mvo> pedronis: I opned 10107, let's see if it helps, I will re-run if needed
[13:57] <mvo> pedronis: (if it passes I will re-run to ensure it's not chance but a real effect we see)
[14:00] <mup> PR snapd#10107 opened: packaging: stop snapd.{socket,service} before purging data <Test Robustness> <Created by mvo5> <https://github.com/snapcore/snapd/pull/10107>
[14:08] <pedronis> ijohnson: jam: we also need apparmor denial logs if any in the system log
[14:08] <ijohnson> yes that too
[14:08] <jam> thanks, I will try to get some context for you.
[14:10] <pstolowski> pedronis: i've run degraded test manually on 20.04 and 20.10 and it passed
[14:10] <pedronis> mvo: we need to tell sergio not to promote until we have clarified these issues, though maybe it was already 2.49.1
[14:11] <pedronis> jam: you moved from 2.49.0 to 2.49.2,  so you don't know if the issue was already in 2.49.1 ?
[14:11] <jam> pedronis, we know that we are seeing it in 2.49.2, I'm not able to reproduce locally, so I'm proxying for people in AU that were trying and didn't get it working.
[14:11] <jam> my understanding is they tried reverting to 2.49.1 but that didn't get it to work.
[14:54] <jam> pedronis, so I've been able to reproduce it, is it just /var/log/syslog that you are looking for the apparmor issues?
[14:54] <ijohnson> jam: use `journalctl --no-pager | grep DENIED` for denials
[14:56] <jam> ijohnson, what is the best place to report this, open a launchpad bug and start including logs there, or a forum post (and which forum :)
[14:56] <ijohnson> jam: let's just use the launchpad, bugs.launchpad.net/snapd
[14:56] <ijohnson> thanks
[14:57] <jam> ijohnson, so if I revert snapd, what steps do I need to do to make sure the snap is installed correctly with the old logic?
[14:57] <ijohnson> jam: what is `snap list` on this machine ?
[14:58] <jam> I just did 'snap revert snapd' but I might have old/new install of the other snap
[14:58] <jam> https://paste.ubuntu.com/p/DH6cN2yYJb/
[15:05] <ijohnson> jam: ok so you have both the `core` and `snapd` snaps installed, which gets a bit confusing, can you also revert the core snap wit h`snap revert core`, so that you are on 2.49 for both core and snapd, and then see if you can reproduce still ?
[15:09] <jam> ijohnson, sure. I should also note that 'journalct | grep DENIED' is about 3.8M lines. And while I can just paste the end of it for you, is there a way I can get a reset so I can be sure I'm giving you fresh information?
[15:09] <ijohnson> jam: what about `journalctl -b 0 --no-pager | grep DENIED` ?
[15:09] <ijohnson> is that any shorter?
[15:10] <ijohnson> that should just be the current boot
[15:10] <jam> 600k
[15:10] <jam> shorter
[15:10] <jam> but still a bit unweildy
[15:11] <jam> but still helpful, yes
[15:11] <ijohnson> let me figure out hte options to journalctl to limit it to just the past couple of hours
[15:11] <jam> ijohnson, so juju-db is using core18, presumably I should revert that one as well?
[15:11] <jam> or maybe you only get snapd from core
[15:11] <ijohnson> jam: no core18 is fine
[15:12] <ijohnson> yeah exactly, snapd can come from either the core snap or the snapd snap, whichever is the newest iirc
[15:12] <ijohnson> though maybe we do have rules to always prefer the snapd snap over the core snap, I don't recall off the top of my head, but I know that re-exec like this is confusing :-|
[15:12] <jam> core                 16-2.49            10859  latest/stable    canonical✓  core
[15:12] <jam> snapd                2.49               11107  latest/stable    canonical✓  snapd
[15:13] <jam> juju-db              4.0.9              29     4.0/stable       juju-qa     -
[15:13] <jam> so with those, I am still seeing the issue, though I have re installed a second time juju-db
[15:13] <jam> after the revert
[15:17] <ijohnson> jam: ok, here's for the past 1 hour `journalctl --no-pager --since "$(date -d '1 hour ago' "+%Y-%m-%d %H:%M:%S")" | grep DENIED`
[15:17] <ijohnson> you can change that to however many hours you need to demonstrate when juju-db fails to start
[15:18] <jam> sure. I can actually use that to 'now' go run the thing, and then give you just the recent content. Thank you
[15:20] <jam> so just with the reproducer, I get:
[15:20] <jam> $ sudo journalctl --no-pager --since "$dt" | grep DENIED
[15:20] <jam> 42:Apr 01 10:20:25 focal audit[3611588]: AVC apparmor="DENIED" operation="open" profile="snap.juju-db.mongod" name="/sys/block/" pid=3611588 comm="mongod" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0
[15:20] <jam> 44:Apr 01 10:20:25 focal kernel: audit: type=1400 audit(1617290425.219:322223): apparmor="DENIED" operation="open" profile="snap.juju-db.mongod" name="/sys/block/" pid=3611588 comm="mongod" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0
[15:22] <ijohnson> hmm doesn't seem like it would have been related, so to beclear after core at 16-2.49 and snapd at 2.49, you can still reproduce the issue with juju-db ?
[15:23] <jam> reverting to core 16-2.49 and snapd 2.49 I'm still seeing the failure, yes. so it doesn't seem related to 2.49.2 at least.
[15:23] <ijohnson> jam: can you confirm the output of `snap version` too ?
[15:23] <jam> $ snap version
[15:23] <jam> snap    2.49
[15:23] <jam> snapd   2.49
[15:23] <jam> series  16
[15:23] <jam> ubuntu  20.04
[15:23] <jam> kernel  5.4.0-70-generic
[15:24] <ijohnson> ok, so I think we can confirm this is not related to the 2.49.2 release at least
[15:24] <ijohnson> but please file a bug on launchpad with all these details and we'll have a look
[15:24] <ijohnson> also journal of the failing unit would be useful, you can get that for the last hour with:
[15:24] <ijohnson> `journalctl -u snap.juju-db.* --no-pager --since "$(date -d '1 hour ago' "+%Y-%m-%d %H:%M:%S")" `
[15:27] <jam> ijohnson, so just to confirm, the snap I'm running appears to be:   4.0/stable:       4.0.9  2019-06-05 (29) 62MB -
[15:28] <jam> which means it hasn't been changed for about 1.5years, thus isn't the source of it not working now.
[15:28] <ijohnson> right agreed, but seeing how it is failing to start will be helpful to understand what else might have changed to break it
[15:29] <jam> sure. I just wanted to make sure my understanding of what I was running was the right one. :)
[15:29] <diddledan> ello folks. I'm looking into WSL2's native gui support - they put the wayland socket at /mnt/wslg/runtime-dir/wayland-0 and the pulseaudio runtime at /mnt/wslg/runtime-dir/pulse (containing native and pid)
[15:30] <ijohnson> hey diddledan
[15:30] <diddledan> if we want to ever support GUI snaps on WSL2 we'll need to allow those locations methinks
[15:31] <diddledan> (I have an issue with XWayland via their native support right now, but I think that's a WSL bug rather than because I've started systemd)
[15:31] <jam> ijohnson, the specific failures seem to revolve around: 2021-04-01T11:29:21.722-0400 W FTDC     [initandlisten] Error getting directory iterator '/sys/block': Permission denied
[15:32] <diddledan> if you want to play along at home, I enable systemd with https://github.com/diddledan/one-script-wsl2-systemd
[15:32] <ijohnson> jam: right, certainly seems related
[15:32] <diddledan> specifically https://github.com/diddledan/one-script-wsl2-systemd/tree/build-21286+
[15:33] <ijohnson> diddledan: so you think the wayland interface needs to also allow permission to read/write to /mnt/wslg/runtime-dir/wayland-0 and the audio-playback interface needs to allow permission to /mnt/wslg/runtime-dir/pulse ?
[15:33] <diddledan> that's what I was thinking, yes. The alternative would be to expose those via bind-mounts into the snap world view
[15:34] <ijohnson> ah right because probably most linux apps are not going to look there they are probably going to look in the existing one in `/run/user/[0-9]*/wayland-[0-9]*`
[15:35] <diddledan> right. although WSL does set XDG_RUNTIME_DIR to point to /mnt/wslg/runtime-dir but that gets nuked when we launch a strictly confined snap
[15:36] <ijohnson> hmm yeah I guess I would be curious to hear what jamesh has to say about that, I admit to being fairly ignorant about wayland specifics
[15:36] <diddledan> me too :-)
[15:36] <diddledan> enthusiatic amateur covers me ;-p
[15:37] <diddledan> PULSE_SERVER=/mnt/wslg/PulseServer is set in the environment of WSL too
[15:38] <diddledan> that's another socket file so probably also needs to be considered
[15:39] <diddledan> they also look to be starting a dbus daemon and putting it's runtime dir at /mnt/wslg/runtime-dir/dbus-1 and also dconf at /mnt/wslg/runtime-dir/dconf
[15:44] <jam> ijohnson, I was trying to create a reproducer with a more minimal configuration. however, I can't figure out where the charm is *supposed* to be able to write data. $SNAP_DATA it complains is read-only
[15:45] <diddledan> jam: are you running as root?
[15:45] <diddledan> $SNAP_DATA is only writable by root, see
[15:46] <jam> k. Well historically we use 'juju-db.mongod' to initialize our test suite, but any dir under $HOME it is now telling me that it is readonly
[15:46] <jam> so I was trying to find a better dir for it to play in
[15:46] <diddledan> hmm, $HOME should be writable
[15:49] <jam> so I was able to 'snap run --shell juju-db.mongod mkdir /tmp/foo' and create stuff there (which looks to be /tmp/snap.juju-db/tmp/foo) which at least gets me past the readonly error
[15:52] <ijohnson> jam if you can't get a minimal reproducer that's fine, the full reproducer should be useful anyways
[15:52] <ijohnson> I do note that you are using charms, which adds a whole other layer of fun on top of snaps
[15:53] <diddledan> ;-)
[15:54] <jam> ijohnson, so in this case, I'm just running a test suite locally, which means it is *just* snaps
[15:54] <ijohnson> ah perfect
[16:04] <diddledan> ijohnson: I've documented all my findings about the WSL gui support in a bug for snapd https://bugs.launchpad.net/snapd/+bug/1922262
[16:04] <mup> Bug #1922262: Add support for WSL2 Native GUI capability <snapd:New> <https://launchpad.net/bugs/1922262>
[16:04] <ijohnson> nice thanks for that
[16:35] <mvo> pedronis, ijohnson 10107 looks promising, first run did not error in prepare anywhere, I triggered a re-run
[16:37] <ijohnson> Nice
[16:49]  * mvo calls it a day but I will check mattermost and tg
[19:06] <mup> PR snapd#10102 closed: daemon: introduce apiBaseSuite.(json|sync|async|error)Req  (and some apiBaseSuite cosmetics) <Created by pedronis> <Merged by mvo5> <https://github.com/snapcore/snapd/pull/10102>
[22:11] <diddledan> who do I have to sweet-talk into making it so the store won't issue a 500 error when you upload a >~1GB snap?!