[03:25] PR snapcraft#2952 opened: spread: introduce appstream parse-info test === mcphail3 is now known as mcphail === popey8 is now known as popey === Eleventh_Doctor is now known as Pharaoh_Atem [06:11] morning [06:40] school run [07:08] re [07:40] PR snapd#8184 closed: cmd/libsnap, tests: fix C unit tests failing as non-root <⚠ Critical> [07:41] ijohnson: meh, it looks like 2020-02-24 22:05:33 Error executing google:ubuntu-core-18-64:tests/core/snapd-failover : is still failing on master even after merging your latest fix for it :( [07:42] ijohnson: https://api.travis-ci.org/v3/job/654625002/log.txt which is a run after 8171 got merged [07:42] mvo: hey [07:42] hey mborzecki [07:43] ouch, failover still failing? :/ [07:43] mborzecki: at least one log still has it [07:43] mvo: there's one oustanding PR with fixes iirc [07:43] mborzecki: aha, there is? [07:44] mborzecki: 8169 ? [07:44] hm i think it was #8140 [07:44] PR #8140: tests: enable more tests for UC20/UC18 [07:44] let me check, it was referenced in one that got merged [07:45] mvo: right 8169 [08:03] good morning [08:04] * zyga is sleepy today, sorry for starting late [08:04] zyga: hey! [08:04] mvo: hey, I fixed the error from last evening [08:04] zyga: I saw, thanks for this! merged already [08:04] ah, good [08:04] mvo: I wanted to raise https://github.com/snapcore/snapd/pull/8123 [08:04] PR #8123: interfaces/network-control: bring /var/lib/dhcp from host (approach b) [08:05] mvo: jamie gave it a +1 and expressed ok to put this in 2.44 [08:05] my reasoning for raising it is that I prefer one variant rather than two [08:05] morning [08:05] mvo: if 2.44 ships (a) (merged) and then we switch to (b) in 2.45 that's more churn than just going with (b) [08:05] mvo: at the same time, it's not a priority fix, just for your consideration [08:05] zyga: pstolowski: hey guys [08:05] hey maciek! [08:06] * zyga is sleepy, couldn't sleep last night [08:07] I'll make a 2nd coffee and be back soon [08:10] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/8081 ? [08:10] PR #8081: tests/main/user-session-env: add test verifying environment variables inside the user session [08:10] sure [08:13] PR core20#23 closed: hooks: ensure console-conf@ is started after snapd.recovery-chooser-trigger.service [08:13] pstolowski: thanks! [08:14] zyga: thanks, it looks like samuele wants to have another look, I'm not against this (at all) [08:14] yep, I saw, [08:14] PR snapd#8188 opened: spread.yaml: make qemu ubuntu-core-20-64 use ubuntu-20.04-64 [08:14] what's the timeline for 2.44? [08:16] PPA is busted? [08:17] https://www.irccloud.com/pastebin/BCFak9n5/ [08:17] * zyga wonders if we could just get on with slack [08:21] * mborzecki prepares to buy another 16GB of ram [08:22] mborzecki: that's easy though :) [08:22] mborzecki: remember when vista was around [08:22] mborzecki: all that free memory [08:30] i think i wouldn't mind matrix really [08:30] mborzecki: +1 with small remark [08:30] pstolowski: thanks, pushing the update now [08:33] seems like there's a bunch of mobile apps, probably none as well supported as slack app, but still looks better than mattermost [08:34] mvo: so, about the chooser ui, i think we should have some intiial prompt rather than jumping straight away to options that do something [08:35] mvo: for instnace, on serial you may have trouble with baud rate setting, or the output may not apepar right away, if the user presses enter and that activates some 'funky' option there may be trouble ahead [08:35] mborzecki: does matrix have a good phone app? [08:35] mborzecki: good point [08:36] zyga: hmm https://play.google.com/store/apps/details?id=im.vector.app&hl=en_US [08:36] 677 reviews, heh [08:36] seems poor [08:36] there's this one too https://play.google.com/store/apps/details?id=im.vector.riotx&hl=en_US [08:37] 5k installs [08:37] yeah [09:19] zyga / mborzecki: I've used Riot.im for some time, and it has really improved over the last 12 months. I use it mostly as an IRC bridge though. RiotX looks good - it would be great to see how Mozilla gets on with Matrix, because it's scale that worries me and can't test. [09:24] degville: I wonder if matrix can be summarized as "good desktop experience, irrelevant mobile experience" [09:24] degville: which might explain why mozilla is OK with it [09:25] degville: I would probably not use slack on my phone outside of conferences and occasional "let's check that one thing from bed" moments [09:26] zyga: me neither. my phone is too old and it will probably reduce battery to 30 mins. I used the web client when I needed to use slack. It does seem a little 'over engineered'. It would be great to get behind an open replacement for IRC imo, because it's not just us who could benefit. [09:27] though I did get the weechat (IRC client) to slack plugin working, which is a nice compromise. [09:37] degville: I think one thing has changed though, the acceptance level has shifted up [09:37] degville: people expect a more polished experience [09:37] degville: and in the endless quarrels of FOSS people we have not managed to raise above that to produce an excellent opinionated product that is also free [09:38] degville: I fear we'll go to a non-free product just because it is actually working well [09:38] degville: a bit like going with google for mai [09:38] *mail [09:38] can we run our own mail server, sure [09:38] is it nice to most people, not as much [09:38] I think IRC will be exactly the same thing [09:39] (and email has much nicer desktop apps than IRC ever did) [09:39] yeah, you're right. I really don't h [09:39] have a problem with non-free at all. It is about the best tool. [09:40] but... it's also a change to differentiate, and also flex some open source credentials when other people may be going the other way. [09:40] I wish there was a monetarily supported free product that's really nice and offers good experience [09:41] I wonder what suse and rh are doing in this regard [09:41] yep. I was seriously considering getting in touch with a couple of friends to see if they wanted to create a startup :) [09:42] I think being able to host your own FOSS server would be 90% of what we want [09:42] well, RH has just gone with Slack. [09:42] over MS teams (https://www.theverge.com/2020/2/10/21132060/ibm-slack-chat-employee-rollout-microsoft-teams-competition) [09:42] the apps could just be closed source for all I care (mobile) and web apps running from the same server [09:42] oooooh [09:42] RH slack? [09:42] degville: as in 'just now' or just (as in no other options considered) ? [09:43] are they using the snap? ;) [09:43] ah ok, IMB [09:43] ahahah... [09:43] IBM [09:44] yeah, true. I didn't realise it was only 350k employees either, which is a drop in the IBM ocean. [09:44] wait, what's 350k employees? [09:44] IBM's employees. [09:45] ouch [09:45] that must cost some pretty penny [09:45] ...actually, all of it's employees. [09:45] its [09:45] * mborzecki wonders about uinput for snap-boostrap recovery-chooser-trigger spread tests [09:45] uinput? [09:46] what's that? [09:46] zyga: userspace driven input device, basically create /dev/input/ from userspace and inject input events [09:46] aha [09:46] all this input devices are now yours [09:46] except evdev [09:46] attempt no open there [09:46] ;) [09:47] * zyga goes back to hitting his head on sessions [09:47] zyga, degville: there are multiple options for self-hosted FOSS team chat... RocketChat, Mattermost, etc. - curious why you're dismissing them (iirc this community used to be on RocketChat, what happened?) [09:48] I've been working on FOSS chat stuff for ~15 years, so I'm 100% curious to hear from folk (and happy to chat in private if it's too off-topic here) [09:48] MattJ: I don't know - do any of those offer identity management and mobile apps? [09:48] * zyga wonders what the requirements are [09:48] Yes [09:50] hm mattermost mobile app was pretty crappy tbh [09:51] I've never used their mobile app myself, but the rest of Mattermost is generally fairly polished (it's essentially a Slack clone) [09:51] Main issue with Mattermost is that they do the do the "enterprise edition" thing, and refuse to add (or accept contributions for) certain features in the community edition [09:52] typical open core [09:52] is that bad tough [09:52] otherwise the open part would not be there either [09:52] *shrug* - that's an entire debate in itself :) [09:53] I'm personally not a fan of it, but I do also find it weird how many stories I've heard of companies running the open-source version with bunches of patches applied because they don't want to pay for the commercial version [09:53] but then they basically have to maintain their own fork [09:54] but I guess dev/ops time spent on that doesn't appear explicitly on balance sheets, so it's ok [09:54] combinations of imperfect realities [09:55] we should all use git irc [09:55] like irc [09:55] just with the usability of git ;) [09:55] commit your messages [09:55] don't rebase public channels [09:55] that kind of stuff [09:55] for usability it comes with a man page generator [09:55] There are plenty of blockchain chat solutions if that really tickles your fancy :D [09:56] ICO included ;) [09:57] looks like it's a choice between lest crappy solution [09:58] Well there are plenty of annoyances with Slack and other proprietary solutions too, so... welcome to software :) [09:58] I'm looking to possibly release something in this space later in the year, so I've been doing a fair amount of research [10:05] * zyga is feeling so so today [10:06] PR snapd#8177 closed: cmd/snap-bootstrap/initramfs-mounts: mount the snapd snap in run-mode too [10:06] [Tue Feb 25 08:37:42 2020] audit: type=1400 audit(1582619862.212:29075): apparmor="DENIED" operation="open" profile="snap.test-snapd-audio-record.play" name="/etc/pulse/client.conf" pid=11535 comm="paplay" requested_mask="r" denied_mask="r" fsuid=12345 ouid=0 [10:07] https://www.irccloud.com/pastebin/06KY87al/ [10:30] hi, is there a way to programmatically (from a script) check for snapd support for default tracks? [10:32] ackk: apart from version query, probably not [10:32] zyga, ok. so >=2.44 ? [10:32] ackk: I don't know, please ask mvo about the details [10:32] * zyga is deep in systemd-pam [10:33] * ackk looks around for mvo :) [10:36] ackk: yeah, 2.44 is the best bet right now [10:36] mvo, ok, thanks [10:37] ackk: we could build something programmatic if needed [10:37] mvo, are versions guaranteed to be x.y (just two numbers)? [10:38] PR snapd#8189 opened: seed,cmd/snap-boostrap: introude seed.Snap.EssentialType, simplify bootstrap code [10:51] mvo: mborzecki: ^ this is relatively simple, the size is mostly from test updates that change indent [10:51] pedronis: thanks [10:51] PR snapd#8156 closed: cmd/snap-bootstrap: subcommand to detect UC chooser trigger [10:51] ackk: following the ubuntu versioning schema, so 2.44 or 2.44.1 [10:52] is not the step that reduces validations but is on the way there [10:57] mvo, thanks [11:14] mborzecki: thanks for review and for spotting the image_test.go issue [11:27] image_test.go fixed [11:43] PR snapd#8190 opened: overlord, taskrunner: exit on task/ensure error when preseeding [12:00] pedronis: hi! curious, is there a store api to determine the highest snap-declaration format the store supports? [12:08] jdstrand: not at the moment [12:08] ok, thanks [12:08] it could be added if needed [12:09] but needs to involve the store (obviously) [12:11] pedronis: sure, it isn't important [12:13] pedronis: I'm just writing a small tool to help reviewers working with store apis. I can hardcode it with a pointer to maxSupportedFormat in asserts/asserts.go [12:13] jdstrand: fwiw, it doesn't support 4 yet [12:13] I know [12:13] I'll ask for a bump during 2.44 release cycle [12:14] * jdstrand nods [12:26] PR snapd#8191 opened: [RFC] cmd/snap-recovery-chooser: add a recovery chooser === Eighth_Doctor is now known as Conan_Kudo === Conan_Kudo is now known as Eighth_Doctor [12:35] * zyga is deep into PAM but feels like it is actually working now [12:36] mborzecki: fun stuff [12:36] zyga: pam? [12:36] zyga: the session/scope thing? [12:37] PR core20#25 opened: hooks/200-console-conf-after.chroot: perform console-conf ordering checks [12:40] hmm signal? [12:41] mborzecki: yeah, I feel I'm starting to understand WTF when you log in [12:41] I need to take the dog out [12:41] ttyl [12:53] I'm taking off [12:53] cannot stand smell of food and cooking at home [12:54] $motherinlaw is on a cooking spree [13:08] hmm looks like all spread jobs are failing? [13:09] mborzecki: the error message also looks wrong [13:10] 'If that is intentional, please update the package list in the' [13:10] cmatsuoka: the error is way up in the log [13:10] ah ok, it continues after in a new echo [13:10] never mind [13:11] but i meant that snapd spread jobs are failing due to snapd-failover [13:12] mborzecki: make install error, that's strange [13:13] did we land ian's snapcraft.yaml fix? [13:16] anyway, I'm not familiar with this usr/share/snappy/dpkg.list [13:18] mborzecki: ah, failover again? I thought mvo fixed that a few days ago [13:23] cmatsuoka: I thought so too :( [13:23] cmatsuoka: investigating this again, it's a PITA [13:25] mvo: btw dimitri's fix worked and now we have run mode working, with and without tpm [13:26] * zyga is at a cafeteria [13:26] finally no food smell [13:28] what :) [13:30] cmatsuoka: \o/ [13:35] zyga: must be a bad cafeteria if you're no smelling any food [13:36] mborzecki: are you kidding me? [13:36] mborzecki: there's coffee and food but you don't smell any of it [13:36] mborzecki: it's not a kitchen [13:41] zyga: sure, it doesn't smell like beef steaks, but these places have a very distinct smell too (one i actually enjoy) [13:46] mborzecki: I think I'm just overly happy that it doesn't smell like the back of a kitchen anymore :) [13:46] hahaha ;) fair enough [13:47] mborzecki: but on topic, pam is really key to my understanding of the whole problem [13:47] mborzecki: building pam with debugging and setting up the debug log [13:47] mborzecki: also forkstat [13:47] that opened my eyes as to what's going on on login [13:48] in various contexts [13:48] I'm going through the config and the pam modules that interact with default sessions you may open, piecing together the picture of how it works [13:48] zyga: you'll have to do a write up for everyone [13:48] mborzecki: yep, including instructions on how to get stuff into debuggable state :) [13:49] and not break your system in the process [13:50] mborzecki: pam_systemd.c (in systemd's tree) is really 99% of what I care about [13:50] although some other bits are also relevant in other modules [13:51] heh [13:51] this is a fun comment [13:51] https://www.irccloud.com/pastebin/zk6NdIwe/ [13:52] PR snapd#8192 opened: tests: add more debug output to the snapd-failure handling [13:55] hmm, how to set syslog priority level [13:58] mvo: I'll skip standup; my update is a deep-dive into logind, pam, ssh, getty and systemd in order to write proper tests for the issue discovered by jamie [13:59] hey folks [13:59] mvo: I'll update the standup notes later today, as I've been taking notes on with pen&paper [13:59] zyga: sounds like fun :-) [13:59] ijohnson: yeah, and useful [13:59] I didn't know this at all [14:00] but I'm slowly building towards a tool that will let us run stuff as a user in a session of given properties [14:07] zyga: ok [14:10] ijohnson: looks like we need to sync about core20/subiquity PRs [14:11] ijohnson: opened this one https://github.com/CanonicalLtd/subiquity/pull/638 to master [14:11] PR CanonicalLtd/subiquity#638: debian/console-conf.service: ensure correct order with respect to snapd services [14:11] mborzecki: hmm talk more after we talk after SU about the other thing? [14:11] ijohnson: and this one to core20 https://github.com/snapcore/core20/pull/25 [14:11] PR core20#25: hooks/200-console-conf-after.chroot: perform console-conf ordering checks [14:12] ijohnson: yeah, let's stay after standup [14:18] mborzecki: this is cool [14:18] https://www.irccloud.com/pastebin/XACC4Nqv/ [14:20] for example, this is running "true" via ssh [14:20] https://www.irccloud.com/pastebin/9Bmj2q1M/ [14:21] (so much noise and garbage) [14:21] it's also interesting as to what PATH is set to specific components, e.g. update-motd [14:22] or the number of times python has to run before you get the shell prompt [14:22] each time to run lsb_release [14:33] mborzecki: comparison of /etc/pam.d/{su,sudo} is interesting [14:34] one involves common-session and the other common-session-noninteractive [14:34] common-session invokes pam_systemd.so while common-session-noninteractive does not [14:45] cmatsuoka: hey, you're probably aware of https://bugs.launchpad.net/snapd/+bug/1863886 ? [14:45] Bug #1863886: snap-bootstrap should validate which ubuntu-data is mounted [14:46] pstolowski: ah yes [14:46] pstolowski: we still must check the best way to do that [14:48] sure [14:49] but yes, it's in my todo list [14:49] cmatsuoka: ok, i'll assign it to you then [14:49] pstolowski: ok, thanks! [14:56] ijohnson: added your patch to https://github.com/CanonicalLtd/subiquity/pull/638 [14:56] PR CanonicalLtd/subiquity#638: debian/console-conf.service: ensure correct order with respect to snapd services [15:01] mborzecki: cool sounds good, I hadn't noticed your branch didn't have the other old uc18 specific service [15:02] ijohnson: can you do a quick review of https://github.com/snapcore/core20/pull/25 maybe? [15:02] PR core20#25: hooks/200-console-conf-after.chroot: perform console-conf ordering checks [15:02] mborzecki: sure looking now [15:03] ijohnson: thanks! [15:05] PR core20#24 closed: hooks: delete console-conf hack [15:32] * zyga has a small eureka moment!!! [15:48] in case I get hit by a bus on my way home [15:48] the magic to run a command in a session [15:48] is [15:49] systemd-run /sbin/runuser -l USER -c COMMAND [15:49] one can also pass --send-sighup and --tty to systemd-run, for instant gratification [15:49] the rationale is as follows: [15:50] systemd-run runs the rests as a servicem outside of our existing logind session [15:50] runuser -l invokes pam_systemd which asks logind for a new session - this normally fails if we already have one (hence systemd-run) [15:50] the session lives for the duration of the executed command [15:50] I'll work on making this nice and useful for writing tests [15:51] CC ^ mborzecki [15:51] zyga: nice work that's pretty great to know [15:52] *magic* [15:52] I'll make some test primitives [15:52] and adjust our test code where it matters [15:53] I have extra notes on how to get debug information [15:53] the critical bit was tracing logind code paths to understand that it fails to create a session if the requesting PID already inhabits one [15:53] I'm happy about this day now :) [15:54] I think I deserve lunch now [15:54] I'll head home [15:54] and sit some more to write down the code in the evening [15:54] CC jdstrand ^ (how to test in a user session above) [15:54] I need to check this on more distributions as PAM config matters [15:54] mainly to check if there's any important skew from one distro to another [16:01] zyga: nice! :) [16:01] jdstrand: the journey through PAM was interesting [16:01] jdstrand: one-more-way-to-setenv [16:01] perhaps a way out of some of our environment problems [16:07] * jdstrand nods [16:18] ogra: is this reliable? https://github.com/ogra1/snapd-docker === alan_g_ is now known as alan_g [16:38] hm, I get "remote failed to report status" when I try to push to github [16:38] anyone else seeing this? pushing a new branch [16:39] systemd-run --pipe --quiet --wait /sbin/runuser -l test -c "COMMAND" [16:39] wooot [16:39] that's what we need :) [16:39] https://status.github.com/ [16:40] increased error rate [16:40] so probably broken [16:40] partial outage [16:40] aha [16:40] it's very annoying, it *may* be the fix for the failover [16:42] hahaha [16:42] that's indeed a bit annoying [16:43] ok, I need to stop coding and wrap up [16:43] * zyga waits for spread to finish [16:43] PR snapd#8193 opened: snapstate: do not restart in undoLinkSnap unless on first install [16:44] ijohnson: I pushed 8193 with something that worked for me at least once, let's hope it proves to be ok [16:44] mvo: nice I'll try that in combination with 8169 [16:44] oooh [16:44] wow [16:45] mvo: is this intentional? https://github.com/snapcore/snapd/pull/8193/files#diff-5a87044d1fc20ea2664be6c683755bffR129 [16:45] PR #8193: snapstate: do not restart in undoLinkSnap unless on first install [16:45] zyga: I have that in my branch too [16:46] zyga: it's quite useful I think because if we are at the point where snap-failure is running, something has went wrong so having debug info from snapd when it is invoked from snap-failure is very useful [16:46] I don't doubt usefulness [16:46] I wonder what the consequences of running with debug are [16:46] it's rather quiet because in this case snapd should only revert the snapd snap and exit [16:48] I'll review stuff with clear head later [16:48] I think working outside is a bit more tiresome than indoors [16:48] but this was a good day [16:48] (at least for me) [16:50] * zyga EODs [16:51] zyga: yeah, it's on prupose, like ijohnson says, the output is actually quite short but very useful [17:23] zyga: FYI I released man-db 2.9.1 with the /snap/man change [17:23] as discussed [17:28] mvo: I've done 3-4 spread runs of your branch with mine on top on core18 qemu and haven't run into the race yet [17:29] but to be fair I was only seeing it once every 30 or so runs, so not sure how excited to get about that data point [17:33] mvo: ack, I just wanted to check [17:33] * zyga is back with fixed bikes [17:33] cjwatson: ack, thank you [17:33] cjwatson: I'll look at the state on our side soon [17:34] * zyga spawns some more tests and goes for dinner [17:34] * zyga is starving [17:38] ijohnson: ohhhhhh [17:38] ijohnson: fingers crossed - I will stop irc now and pretend the bug is fixed :) [17:38] haha [17:41] * mvo hugs ijohnson [18:28] drat [18:28] my laptop suspended while running tests [18:28] oh well [18:49] Hey zyga, when I run `snap login`, does that macaroon ONLY have the ability to install snaps? Nothing else? [18:49] kyrofa: mmmm, no [18:49] kyrofa: I think that is equivalent to root [18:49] What other possibilities are there? [18:49] kyrofa: the details are in api.go [18:49]