[00:22] I'll EODnow [00:22] tests in spread are more comprehensive [00:22] tomorrow I'll rebase on master and propose [02:31] ijohnson, nice, thanks [06:28] Uh [06:28] Tired [06:38] morning [07:48] hey [07:48] oooh [07:48] my tests passed [07:48] mborzecki: I massaged that silly session tool more [07:48] mborzecki: and learned a few interesting things along the way [07:48] zyga: hey [07:49] mborzecki: for instance, certain invocations of su/sudo spawn all the session services [07:49] mborzecki: and that may include pulseaudio, bluetooth, gpg daemon, apt daemon and a hell lot more [07:49] mborzecki: and they stick around [07:49] mborzecki: they are shut down asynchronously, roughly 10-20 seconds later [07:50] mborzecki: I think this could explain some cases of tests seeing weird processes [07:50] hahah [07:50] what is worse is that this is per-distro [07:50] it depends on PAM configuration [07:50] how come it evolved to be so complex [07:50] the upside is I know how to handle that [07:50] but it's not clear how to handle that for root [07:50] for non-root that's just loginctl kill-user test [07:51] for root I think we could enumerate sessions, kill all but the ssh incoming remote session [07:51] that would reliably shut stuff we run between invocations [07:51] _except_ if that populates the remote session [07:51] then we're SOL [07:54] in the meantime i'm trying differnt daily images in hope of triggering https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1865063 [07:54] Bug #1865063: snapd package hangs on deb postinst [07:54] mborzecki: ready for the sprint? [07:55] oh [07:55] I remember hearing about it [07:55] mborzecki: what does postinst do? [07:55] mborzecki: could this be arm64? [08:02] morning [08:02] hey [08:02] is master better [08:03] I heard some horror stories from yesterday [08:04] hmm nothing super werid about postinst [08:05] mborzecki: does it interact with snap? [08:05] as in, run anything that would have it wait for snapd that's panicking [08:07] I stopped working at 2AM [08:07] please don't expect me to be around much today [08:07] I need to prepare for the sprint, specifically the apparmor prompting demo needs some work [08:07] zyga: no not really, there's services started ofc, so unless snapd was stopped it will be started on install and may do things it does on startup, like rebuilding seccomp profiles etc [08:07] and I have a haircut at 10:30 [08:08] wow, haircut? :) [08:08] mborzecki: anything in autogen postinst? [08:08] mborzecki: I call it yak shaving [08:09] hmmm [08:09] mborzecki: and speaking of PAM [08:10] mborzecki: my test passed everything but debian [08:10] both sid and 9 hang [08:10] time to debug [08:10] oh well [08:10] I'll go upstairs and make another coffee [08:10] I feel so sleepy still [08:10] daughter woke me up at 6:40 by kicking me in the head [08:22] zyga: can you take a look at https://github.com/snapcore/snapd/pull/8083 ? [08:22] PR #8083: spread, data/selinux: add CentOS 8, update policy [08:23] hope we could land it today, as there's some selinux fixes included [08:43] mvo: hey [08:46] hey mborzecki [09:08] mborzecki: yep [09:09] hi, #7999 and #8185 need 2nd review. #8185 shows some problems with test/mock code org, but I don't think it should be blocked on that. [09:09] PR #7999: devicestate: allow encryption regardless of grade [09:09] PR #8185: tests: add uc20 kernel snap upgrade managers test, fix bootloadertest bugs [09:10] see my comment here: https://github.com/snapcore/snapd/pull/8185#discussion_r385579873 [09:10] mborzecki: done [09:11] I'm off for a yak shaving session [09:11] o/ [09:30] zyga.shave() [09:31] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/8083 ? [09:32] PR #8083: spread, data/selinux: add CentOS 8, update policy [09:32] pstolowski: there's some selinux bits in there [09:32] mborzecki: will do [09:32] pstolowski: thanks! [10:03] mborzecki: hi, I made an high-level comment in #8191 [10:03] PR #8191: [RFC] cmd/snap-recovery-chooser: add a recovery chooser [10:05] pedronis: thanks, it's totally up for discussion, we can surely brainstorm in FRA [10:09] mborzecki: ok, also to clear we quite likely will need that kind of JSON I described somewhere in snapd, because snapd will a API endpoint to list the recovery systems [10:09] *will have [10:27] #8187 also needs a 2nd review [10:27] PR #8187: boot: misc UC20 changes [10:39] PR snapd#8083 closed: spread, data/selinux: add CentOS 8, update policy [10:39] mvo: can you cherry-pick the patch? ^^ [10:39] re [10:39] I think I know why debian is failing now [10:40] awk/gawk probably [10:40] mborzecki: yes, thank you! [10:40] running a test to confirm [10:40] I'll take a shower, my face is full of sharp pieces of hair [10:40] mborzecki: done [10:42] mvo: thanks! [10:47] PR snapd#8214 opened: tests: test that after "remove-user" the system is unmanaged [11:05] zyga, sorry, i was at embeddedworld this week, i havent used snapd-docker for a long time, technically it should still work, practically i think some defaults would need updating (IIRC it points to old releases etc) [11:11] mvo: hi, still fixing tests issues? as I mentioned there are UC20 things needing 2nd review, but maybe mborzecki is on those already [11:13] pedronis: in a meeting right now, was mostly on tests+2.44 cherry-picks today [11:16] pedronis: i've updated 8187, looking at 8185 now [11:16] thx [11:18] zyga: so review tools is updated now? [11:18] mborzecki: dunno, I didn't hear anything about that [11:19] mborzecki: updated for which issue? [11:19] I think it has been updated for the setgid changes [11:19] pedronis: snapd setgid [11:19] shall we merge the revert then? [11:21] in principle yes, but really for mvo to decide, I see the PR is running tests atm, #8181 [11:21] PR #8181: packaging: revert "work around review-tools and snap-confine" <⛔ Blocked> [11:22] pedronis: yeah, restated them just now, fwiw the change looks ok [11:28] PR snapd#8215 opened: interfaces: make the network-status interface implicit on classic [11:32] pedronis: heh, missed that odd wording thing when i moved the comments around [11:37] * pstolowski early lunch and sprint-related errand [11:41] pedronis, zyga +1 for reverting the setgid change [11:41] interestin failure with nonce is invalid https://paste.ubuntu.com/p/6F9z9rWZDy/ [11:41] * zyga debugs the debian issue [11:42] easy to repro, just unclear why it hangs in something that passes elsewhere [11:44] heh when POSTing to /api/v1/snaps/auth/sessions there's sometimes 503 service unavailable recived back [11:46] so it's not mawk vs gawk [11:46] something more subtle ... [11:49] huh [11:49] NOW it works? [11:49] what did I fix [12:03] https://forum.snapcraft.io/t/error-cannot-communicate-with-server-timeout-exceeded-while-waiting-for-response/15726 [12:03] :( [12:04] popey: that log is cut [12:04] could you get more with journalctl -u snapd.service please [12:04] ideally [12:04] ok [12:04] journalctl -u snapd.service -o ... # e.g. -o cut [12:04] or something similar [12:04] this should give you all the logs [12:04] sorry, give me one command [12:04] sure, one sec [12:05] sudo journalctl -u snapd.service -o short | cat [12:05] that ought to work [12:05] i do not believe you will learn more from that output :D [12:06] but okay :D [12:06] popey: current output is cut [12:06] but also, memory 1.7G [12:06] ouch :/ [12:07] ok, added to the forum post [12:07] popey: does snap version now work? [12:08] snap version always worked [12:08] snap refresh didnt [12:08] ok [12:08] just checking id daemon is up [12:09] how many snaps do you have? [12:09] 311 [12:10] how many do you have? :) [12:10] PR snapd#8203 closed: netlink: fix panic on arm64 with the new rawsockstop code [12:10] popey: about a dozen per system [12:10] popey: I think this is a real bug, just one we are not experiencing ourselves :) [12:10] we have performance improvements related to systems with many snaps coming this cycle thanks to changes the store is doing, but nothing immediate [12:10] I guess the client times out, waiting for snapd to compute the response [12:11] oh dear [12:11] popey: can you snap refresh foo # replace foo with one snap [12:11] zyga: yeah, that'd be my guess, that's the client deadline context bit [12:11] PR snapd#8181 closed: packaging: revert "work around review-tools and snap-confine" [12:11] iirc the deadline is quite generous though, 10s or so [12:11] popey: how big is your state.json for reference [12:12] but that's weird, everything is async [12:12] it must be slow to get the ansync response! [12:12] mborzecki: for historical and complexity reasons we do some things synchronously in the refresh case, it's an issue, it will improve a bit this cycle [12:13] 1269004 [12:14] popey: if you gzip it, how big would it be? [12:14] anyway, that's probably not going to help [12:14] it's not the problem anyway, not mainly [12:14] here [12:14] 183847 [12:15] right, I suppose we block on O(N) requests somehow [12:16] zyga: yes, the bulk assert refresh work will help here [12:35] hmmm [12:35] I managed to reproduce a hand once on my machine [12:35] then it's gone [12:35] reproduces ok in spread [12:35] ... what am I missing [12:47] now it breaks on focal [12:47] ok, feels like it's buffering [12:47] added fflush [12:47] PR snapd#8173 closed: tests: adding arch-linux execution [12:49] I found a workaround [12:49] https://forum.snapcraft.io/t/error-cannot-communicate-with-server-timeout-exceeded-while-waiting-for-response/15726/3 [13:05] popey, sosumi ?!? wow ! [13:05] * diddledan sues ogra [13:05] hah [13:06] live feed of new snaps as they appear: https://twitter.com/snapstats_org :-) [13:06] * diddledan promotes [13:07] it's on a 30minute cycle [13:07] so the longest you'll be behind live is 29 minutes :-) [13:08] the day it gets overwhelmed in a 30 min period is a good day :D [13:08] aye. usually only one or two snaps per weekday [13:09] rarely zero per day [13:09] there was a day recently when something like 22 landed in the store [13:09] wow [13:10] the graph is fairly linear: https://snapstats.org/ [13:11] those are weekly ticks [13:17] diddledan: what are "developer counts"? [13:17] number of developers publishing snaps? [13:17] the number of developers that have a published snap [13:17] oh wow [13:17] i.e. the number of unique developer names on publicly visible snaps [13:18] I wished for 2^X but [13:19] diddledan: nice! [13:19] thanks :-) [13:19] diddledan: how often is the snap count updated? daily? [13:19] every 30 minutes [13:20] but then it gets thinned out to one per day [13:20] ... at the end of each day [13:20] hmm, who is admin on the forum ? i would like to edit the pi4 thread to remove the links to my experimental images (and point to the new official ones) but i seem not to be able to edit my own posts there [13:21] ogra: hmmm [13:21] not sure [13:21] I suspect popey can do that? [13:21] mvo: ^ [13:21] mvo would know for sure. good call. [13:21] link? [13:22] this one? https://forum.snapcraft.io/t/raspberry-pi-4-chromium-mir-kiosk/15725 [13:22] dont think so [13:22] oh no. not that one [13:22] https://forum.snapcraft.io/t/support-for-raspberry-pi-4/11970/4 ? [13:23] right ... [13:23] you know what would be better [13:23] put a readme in that folder and move the content out the way [13:23] speaking of pis. you see they dropped the 1GB model and reduced the 2GB price? [13:23] yeah, i wanted to do that additionally [13:24] but most people might find the thread so i wouldnt want to detour them through people.c.c [13:24] https://www.raspberrypi.org/blog/new-price-raspberry-pi-4-2gb/ [13:24] but people might find people.c.c too [13:25] yes, thats why i wanted to do it there additionally :) [13:25] hey [13:25] (do it: update readme etc) [13:25] tests passed!!! [13:25] awk is such a beast [13:25] use sed :P [13:25] tests? pass? that's unpossible! [13:25] there must be something wrong [13:25] awk is just for people that fear sed :) [13:26] diddledan: I was telling myself that for a few hours [13:26] i have marked Steve's most recent post as "solution" so it's right at the top [13:26] which should help [13:26] +1 [13:27] i have never tried to edit such an old post, is it normal that it loses editing capabilities over time ? [13:28] i guess [13:28] discourse be discourse [13:28] hmpf .. [13:46] mborzecki: https://github.com/snapcore/snapd/pull/8216 [13:46] PR #8216: tests: add session-tool, a su / sudo replacement [13:46] I think I cannot believe this really works [13:46] * zyga runs more tests [13:46] PR snapd#8216 opened: tests: add session-tool, a su / sudo replacement [13:53] pedronis: sorry I didn't finish those docs yesterday, please take a look at the booting doc again I finished the expected kernel mounting section just now [13:54] pedronis: also I went back and move some things that were in yellow in the boot envs doc to actual sentences and such in the "Run /var/lib/snapd/modeenv" section [13:55] PR snapd#8187 closed: boot: misc UC20 changes [14:17] zyga, https://forum.snapcraft.io/t/requesting-uninstall-reason-via-popup/15655 ... shouldnt the teardown of the namespace on uninstall kill all processes and their kids ? [14:17] in a call [14:18] I'll check soon [14:27] cmatsuoka: that was the bult-in laptop mic [14:27] cmatsuoka: I often use the external mic [14:27] cmatsuoka: and I sometimes use the built-in imac mic [14:31] ogra: could be a bug with systemd timers that are created for snaps [14:36] zyga: the sound quality was quite good frequency-wise, but it picked some background noise [14:36] https://www.reddit.com/r/linux/comments/fa8ewy/ubuntu_2004_lts_to_revert_gnome_calculator_and/ heh [14:37] regarding that link, zyga, and ogra, does snapd not set up a cgroup to collect a snap's processes and kill the cgroup when it removes a snap? [14:38] diddledan: no [14:38] diddledan: not yet [14:38] aah. that's coming? [14:39] diddledan: not exactly that [14:39] diddledan: we cannot kill some [14:39] oh, I see [14:39] diddledan: because of compatibility [14:39] diddledan: but we can probably kill most (all desktop apps) [14:39] bah. compatibility be damned :-p [14:40] diddledan: what will more likely happen is reverse [14:40] diddledan: snapd will tell you "you cannot remove this snap because it is still running" [14:40] diddledan: close "app foo" and try again [14:40] diddledan: this is compatible and in line with other work [14:41] regarding uninstall surveys. if we want to support such a feature we could add a snap.yaml field for "uninstall url" that fires up the system web browser or prints the url on the terminal for non-gui installs [14:41] that makes sense. I like that, rather than just killing stuff, give the user the choice [14:42] I think uninstall surveys are a bit of an antipattern though [14:42] as long as there is also a --force option for admins that dont want to give heir users choices ;) [14:43] I see them as the developer saying "I've annoyed you enough to make you uninstall my app. let me annoy you even more with this obnoxious survey." [14:44] I get that the developer wants to understand what they did wrong. but an uninstall survey is the wrong way IMO [14:45] however, snap needs to be pragmatic, and it is a common enough pattern that we might be wise to bend to the will [14:52] IMHO people should be able to uninstall snaps without going through a survey [14:52] OTOH, we should at least enable that use case for developers that want it [14:52] that's my view, ijohnson [14:52] yeah agreed diddledan [14:52] not sure what the right technical solution is though [14:52] stop stealing my ideas. get your own ;-p [14:53] haha but it's so hard to think on my own [14:53] I hear that! [14:53] * diddledan goes to find a sheep to follow [14:53] 🐑 [14:54] aww, that emoji doesn't work here :-( [14:54] looks to be brave-specific failure thouhg [14:55] it works fine in my terminal [14:55] :( [14:55] * diddledan tries firefox [14:55] diddledan: do you use the lounge snap thing I seem to remember you worked on a while ago? [14:55] yes, I do [14:55] seemed really cool but I haven't given it a try yet [14:55] I'm chatting in it right now o/ [14:56] popey and Wimpress did a great job enabling ssl/tls support the other day [14:56] nice [14:56] which means that if you access it through a mobile device you can enable push notifications [14:57] * cachio lunch and school [15:32] re [15:34] * zyga had a bowl of żurek for dinner [15:37] PR snapd#8170 closed: snap-preseed: support for preseeding of snapd and core18 [15:57] ijohnson: I made some suggestion and add some bits to the doc [15:57] ijohnson: thank you for the review! [15:58] thanks pedronis looking now [15:58] yaw yga [15:58] zyga [16:21] PR pc-amd64-gadget#38 opened: grub.cfg-recovery: fix typo, filter vars when loading them [16:25] ijohnson: check this out :) https://github.com/snapcore/snapd/pull/8216/files#diff-a8fce2de6a6829d98f798117050b6c93R59 [16:25] PR #8216: tests: add session-tool, a su / sudo replacement [16:26] zyga: ah much easier to read thanks [16:26] it's not a heredoc but has the same property [16:26] just a multi-line value :) [16:26] yes multi-line is what I was after, heredoc was just the first thing I could think of that was multi-line :-) [16:46] I want to make sure this really works [16:46] spawned this test across the fleet, with ... -repeat 100 [16:46] it's a quick test [16:46] but this gives me more confidence [16:46] brb, coffee [16:55] back [16:55] daughter asleep :) [16:55] so quiet upstairs [16:56] let's try to solve the issue I've built this for :) [17:10] holly molly [17:10] zyga@focal:~/go/src/github.com/snapcore/snapd/tests$ git grep 'su -l' | wc -l [17:10] 114 [17:10] that's a lot of tests to fix [17:10] ijohnson: ^ [17:11] oh boy [17:11] there's one interesting thing [17:11] linger [17:11] linger? [17:11] https://www.freedesktop.org/software/systemd/man/loginctl.html < check for "linger" [17:11] Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller. [17:11] it means we'd have a "test" user session that we can always hop into [17:12] ah seems cool [17:12] probably faster than spawning and shutting one for each command [17:12] I'll explore some more [17:12] I wonder if this means I can ssh into a system [17:12] pfft ... ln -s /usr/bin/nohup /usr/bin/linger [17:12] start a systemd user session [17:12] :P [17:12] and log out [17:13]