[02:06] PR snapd#8152 opened: managers_test: add uc20 kernel snap update happy and panic tests [06:15] morning [06:38] school run, back in 30 [07:06] re === abeato_ is now known as abeato [07:45] mvo: morning [07:48] hey mborzecki ! good morning [08:01] morning [08:14] re [08:14] good morning [08:16] tests are somewhat unhappy [08:16] I've seen snapd-failover fail a lot yesterdat [08:16] I know Ian was looking into that but perhaps it needs more eyes [08:18] and snap-run hangs, oh my [08:18] zyga: right, i saw snapd-failower failures as well [08:22] so [08:22] I need bionic [08:22] and I need tests that run as user [08:22] * zyga gets to work [08:40] ok, test written, let's run it [09:10] mborzecki: do you expect https://bugs.launchpad.net/snapd/+bug/1863859 to be true given our environment generators? [09:10] Bug #1863859: Using su removes /snap/bin from PATH [09:10] mborzecki: I just noticed this in tests [09:13] zyga: su - ? [09:13] su test -c "snap run foo" vs su test -c "foo" [09:13] zyga: try su -l [09:13] mborzecki: k [09:14] or . $TESTSLIB/user.sh and then as_user_simple ? [09:14] zyga: simple will not load the profile [09:16] I'll look at some tests that suffer from this [09:32] * zyga loves calls with baby crying next door [09:33] PR snapd#8153 opened: [RFC] "snap run --explain" with different formating [09:33] mvo: I'll review in a moment, need to check what the crying is all about [09:34] zyga: no rush, thank you! [09:55] re [10:10] * zyga debugs all kinds of things through one failing test [10:46] mborzecki: quick question [10:46] https://www.irccloud.com/pastebin/B2mlykUV/ [10:46] (this is test-snapd-sh without arguments [10:46] spawning endless loop of bad substitution [10:46] why would our argv[0] cause this? [10:48] I'm patching snap-exec to show what's going on at that layer [10:48] it is driving me crazy, I want to finally fix it [10:53] mborzecki: it's a dash specific message [10:53] that's interesting [10:53] I think we do something incorrectly [10:53] oh well, let me grab coffee [10:55] desktop 18.04 install in progress [10:55] I think we have a problem with systemd there :/ [10:55] but I think my test setup is not doing what is right [11:08] brb [11:14] ha [11:14] got it [11:14] found a bug in spread [11:17] PS1 breaks dash [11:17] becuase SPREAD_PATH is unset [11:18] is it just me or is this a quiet channel? [11:30] zyga: hey :-) [11:30] hey [11:30] how are you? [11:30] zyga: i'm fine. thank you. Little bit tired. But fine :-) And you ? [11:30] a bit scared I missed something big that came up yesterday evening [11:31] debugging it now [11:31] oh, good luck [11:32] i'm just started to fix the snap for the github actions. [11:34] zyga: on which point it's best to do chmod of a script ? I have a /run.sh and snapcraft says it's not executable. I use chmod at `override-build` currently. [11:35] sdhd-sascha: are you building on windows? [11:35] `Failed to generate snap metadata: The specified command 'run.sh' defined in the app 'runner' is not executable.` [11:35] sdhd-sascha: it's best to keep it executable in the tree [11:35] I wonder, why the generated tar isn't executable ... i will see [11:38] zyga: if i go into `multipass exec ` ... Then inside of `/root/parts/runner/build/run.sh` it's executable. [11:38] ``` [11:38] apps: [11:38] runner: [11:38] command: run.sh [11:38] ``` [11:38] But i still get: `Failed to generate snap metadata: The specified command 'run.sh' defined in the app 'runner' is not executable.` [11:39] hmm [11:39] oh great [11:39] killing dash killed gnome-terminal [11:39] wow [11:40] anyway [11:40] just more bugs [11:43] ijohnson: thanks for the PR [11:43] ijohnson: it seems that tests failed in a strange way [11:46] command systemctl is-active snapd.failure.service keeps failing after 30 attempts [11:47] and snapd.service: Failed to execute command: Exec format error [11:47] zyga: does that sound familiar to you? [11:47] cmatsuoka: hey [11:47] yeah, there's something really broken [11:47] it's worse recently perhaps the test is new or something related changed [11:47] it fails left and right [11:48] EXEC spawning /snap/snapd/x2/usr/lib/snapd/snapd: Exec format error [11:48] I haven't looked into it yet [11:48] cmatsuoka: yeah, the test is trying to install corrupt snapd [11:48] to check failover [11:48] but failover never kicks in [11:48] the exec format error is expected and harmless noise (for the test) [11:49] ah ok [12:02] PR snapd#8154 opened: tests: reset PS1 before possibly interactive dash [12:02] mborzecki: ^ [12:02] this caused me some grief [12:02] cmatsuoka: I can look after I investigate more systemd stuff [12:03] cmatsuoka: zyga: yes I was looking into this a bit yesterday, I think that the patch mborzecki recommended to snap-failure is useful here, I had that small change proposed in my larger core spread PR, but perhaps I should break it out just to get that merged [12:03] and morning folks btw [12:03] ijohnson: hey! [12:03] hey mborzecki [12:03] * ijohnson is here for real this morning [12:04] `The dash bug has been reported to the dash mailing list (there is no bug tracker).` hahah welp [12:08] mborzecki: oh well [12:09] mborzecki: it's very unixy [12:09] mborzecki: mailing list archive, CCing people [12:09] mborzecki: 3-character names [12:09] thank you for the review guys [12:09] I see a lot of red in the most recent spread runs, has anyone investigated yet? [12:09] mvo: snapd-failover [12:09] it's failing ... over and ... I'll shut up now ;) [12:10] :D [12:10] * zyga returns to debugging systemd [12:14] PR snapd#8155 opened: tests: mv ubuntu-core-snapd{,-failover} to core/ suite [12:14] mvo: I opened ^ which I think will help with the snapd-failover test [12:15] ijohnson, zyga failover test started failing a time ago but with external backend [12:15] now is started failing on travis too [12:15] cachio: interesting you mean on the lab machines ? [12:15] cachio: do you have logs? [12:15] ijohnson, http://paste.ubuntu.com/p/DXD9WN7CtJ/ [12:16] this is from the 2.43.3 [12:16] line v [12:16] 1330 [12:16] I though it was related to our configuration for the external backend [12:16] because it was not possible to reproduce on travis [12:16] using gce [12:17] but seems to be something else [12:17] yeah that seems to fail the same it's failing on travis now [12:17] the issue is that snapd.failure.service is not being started ever by systemd [12:17] the weird part is that when using the external backend the test is not failing 100% of the times [12:17] right [12:18] yes I don't think it is failing 100% of the time on travis either, indeed when I tried reproducing yesterday I didn't hit it [12:18] a time ago I shared more info about the error, I am trying to find it [12:19] I remember I sent the error to mvo [12:20] failover was trying to find core and core snap was not installed [12:20] ijohnson, but I cant find the logs [12:20] cachio: hmm that sounds different [12:21] the error is the same you see in the log I sent you [12:21] I'll reproduce it [12:22] and share the logs again, give me 15 minutes [12:22] google:ubuntu-18.04-64 .../tests/main/cgroup-tracking# test-snapd-sh.sh [12:22] $ [12:22] google:ubuntu-18.04-64 .../tests/main/cgroup-tracking# [12:22] ^ getting this feels like this day is already worth it [12:24] thanks cachio [12:25] mborzecki: but it was worth reporting, it's a known problem [12:25] https://patchwork.kernel.org/patch/11343121/ [12:25] and there's a patch [12:28] nice [12:40] PR snapd#8156 opened: [RFC] cmd/snap-bootstrap: subcommand to detect if we want a chooser to run [12:41] mvo: ^^ the key-watching thing [12:41] pedronis: mvo: we should probably discuss the naming there [12:46] oh [12:46] I wanted to mention that yesterday [12:46] about recovery and prompt and stuff [12:46] something to keep an open mind [12:46] smart IOT bulbs [12:46] do recovery [12:46] if ... [12:46] you turn them on and off five times in a row with specific time delay [12:47] our discussion about key detection feels incorrect [12:47] it should not be something we mandate [12:47] Hello all. I instaleld snapd on a raspbian based pi today. Unfortunately, none of the built-in interfaces/slots are available. What creates these? [12:47] zyga: we don't mandate it, we need a reference experience [12:48] Since they dont exist, snaps are unable to connect to anything [12:48] codingpanic: did you install snapd as a deb pkg ? [12:48] ijohnson: yes [12:48] hrm [12:48] ubuntu@localhost:~$ snap version [12:48] snap 2.43.3+git1047.g6b52b37 [12:48] snapd unavailable [12:48] series - [12:48] ubuntu@localhost:~$ snap list [12:48] error: cannot list snaps: cannot communicate with server: timeout exceeded while waiting for response [12:48] ubuntu@localhost:~$ [12:48] codingpanic: what's `snap list` ? [12:48] ubuntu@localhost:~$ ps ax|grep snapd [12:48] 8932 ? Ssl 0:00 /snap/snapd/6521/usr/lib/snapd/snapd [12:48] ogra: why did you break your snapd [12:49] this is how i found one of my qemu machines this morning [12:49] well don't leave it like that :-) [12:49] seems it got some reboot-triggering update yesterday evening [12:49] ogra: journalctl -u snapd.service [12:49] ogra: anything in the system journal for snapd? [12:49] pi@raspberrypi:/var/log $ snap list [12:49] Name Version Rev Tracking Publisher Notes [12:49] core18 20200124 1671 stable canonicalâ base [12:49] scummvm 2.1.1 3126 stable snapcrafters - [12:50] I've also tried retroarch [12:51] codingpanic: snap install core [12:51] https://paste.ubuntu.com/p/xvRDVNPW4S/ [12:51] the last part simply repeats over and over [12:51] codingpanic: it's a known bug [12:51] zyga: got it, thank you! [12:51] ogra: Feb 19 08:11:22 localhost snapd[824]: panic: cannot checkpoint even after 5m0s of retries every 3s: write /var/ [12:52] ogra: did you run out of disk space [12:52] ? [12:52] zyga: that fixed it [12:52] zyga, bah, yeah [12:53] * zyga hugs ogra [12:53] been there done that (too many times) [12:53] I need a coffee [12:53] brb [12:54] pedronis: I need to run an errand right after SU (3:30 PM your time to be specific), can we put something on the calendar for 4:30 PM your time ? [12:54] zyga: I thought we fixed that bug though ? [12:54] ijohnson: no, that's a separate bug [12:54] ijohnson: look at snap list output [12:54] ijohnson: according to our design you have 0 interfaces in that case [12:55] ijohnson: you must get snapd or core [12:55] zyga: but I thought we fixed the bug where the core snap was not installed when you to go install only a snap with `base: core18` ? [12:55] ijohnson: also this is raspbian so probably runs an older copy [12:55] zyga: ah right yes probably older version of snapd with the bug still in place [12:55] yep [12:56] zyga: we should probably update snapd in raspbian, seems to happen somewhat frequently now I think [12:56] ijohnson: I don't think we can [12:56] ijohnson: we should update snapd in debian [12:56] it very old [12:56] zyga: who handles snapd update in debian? [12:56] nobody [12:56] us [12:56] sometimes [12:56] when $fire [12:57] also doesn't raspbian have it's own archives based on, but separate from debian? perhaps I'm misremembering [12:57] hrm ... freeing up 120MB and rebooting just gets me back to that state :/ [12:57] ijohnson: it does [12:57] ijohnson: but we don't have any way to change it [12:57] bah ... because it is secretly upgrading ... it just rebooted again [13:00] hooray now ogra's ephemeral qemu VM is more secure [13:07] zyga: hey, fyi, https://github.com/snapcore/snapd/pull/7825#discussion_r381277068 [13:07] PR #7825: many: use transient scope for tracking apps and hooks [13:07] hye [13:07] hey* [13:07] ack [13:07] I'm still looking into this [13:07] and especially what happened to your interactive blackbox testing [13:10] zyga: that's fine. I just wanted you to be aware of that weird thing I saw where I semi-started a transient unit and couldn't get it to go away in case it saves you some time [13:11] thank you [13:12] zyga: otoh, can you also make sure that these tests aren't skipped on core? you mentioned trusty but I thought I remembered something about user sessions on sore.... that said, xenial is starting systemd --user so hopefully it is all ok [13:13] jdstrand: I will [13:13] jdstrand: I think this approach was picked specifically to support 16.04 desktop [13:13] that sounds familiar :) [13:13] that was a long time ago :) [13:14] zyga: fwiw, I'm really bought into the technique and like the PR overall :) [13:14] just need to make sure systemd does what we need everywhere [13:14] jdstrand: yeah, I just hope I didn't miss anything [13:17] zyga: I did notice yesterday that runc was using StartTransientUnit for at least something... could look there to build confidence. but one of the things I like about this is we are just asking systemd to take care of things for us. so long as we use the api correctly (which it appears you are), we should be good (barring bugs (systemd or otherwise)) [13:17] jdstrand: the only thing I'm worried about is --user [13:17] the API is sold and supported for a long time [13:17] but user session is more complex [13:17] systemd --user is asked to stuff [13:17] but cannot [13:17] so asks system systemd to do it [13:17] zyga: kinda like how we ask it to manage starting and stopping services. we tell it a little bit and say "thanks for doing this for me!". same here [13:17] and that path I believe is somewhat new [13:18] I'm digging into this now [13:18] * jdstrand nods [13:18] zyga: well, it isn't *terribly* new if it is in bionic (I didn't test xenial), but there is definitely some different behavior [13:19] which might be cherrypicks, etc [13:19] I think it's all the way back to xenial, but give me a moment to debug it [13:19] zyga: we always have in out back pocket runtime detection and gracefully degrading too (not saying we need it yet) [13:19] s/in out/in our/ [13:20] yes, it's not essential in a way [13:22] ijohnson, this is the output https://paste.ubuntu.com/p/qNnRgrHktg/ [13:22] mborzecki: thanks, looking [13:23] from a local run on core-18 [13:23] ijohnson, at the end you can see the issue that I raised a time ago snap-failure[9015]: cmd_snapd.go:124: restoring invoking snapd from: /snap/core/current/usr/lib/snapd/snapd [13:23] thanks cachio, I'm looking now [13:24] also the error snapd.service: Failed at step EXEC spawning /snap/snapd/x1/usr/lib/snapd/snapd: Exec format error [13:25] cachio: the EXEC spawning issue is very odd, because it should be x2 that has the Exec format error [13:26] zyga: so it is crystal clear: the reproducer is: take a desktop vm install, install snapd/enable the feature/blah, in the graphical session, open a terminal, use hello-world.sh/snap run --shell/etc and it works (find /sys/fs/cgroup -name '*.scope'). ssh in/console login and try the same and it doesn't (but not errors in SNAPD_DEBUG=1 or anywhere else I could find [13:26] ) [13:26] * zyga nods [13:26] let me check that quickly [13:26] I was going via spread and I hit another issue there [13:26] * jdstrand nods [13:27] ijohnson, I have a debug session opened [13:27] ijohnson, if you need any output please tell me [13:27] I figured it might be easier to see what to recreate if you saw the problem [13:27] I think there is a known issue in that snap-failure should be more robust in trying to restart the socket, and isn't [13:27] cachio: ^ [13:27] * jdstrand leaves zyga to it [13:28] cachio: sorry let me keep reading I'm not sure what I would like you to run in the debug session [13:32] jdstrand: wwo [13:32] jdstrand: so [13:32] jdstrand: it works on a fresh install of bionic [13:32] jdstrand: desktop session [13:32] jdstrand: I recorded bustle (dbus traffic analyzer) dumps [13:32] zyga: right, in the desktop session, it works [13:33] and even confirmed it the simple way [13:33] https://www.irccloud.com/pastebin/pENvRQqK/ [13:33] oh? I missed that [13:33] when didn't it work? [13:33] zyga: ssh in or do console login and it doesn't [13:33] ah [13:33] I see [13:33] okay that's much clearer now [13:33] thanks, I'll focus on that [13:33] zyga: that is what made it tricky for me to diagnose :) [13:33] jdstrand: but I'm 99% sure it's because there's no session bus there [13:33] and this is a dbus interaction [13:34] zyga: I saw systemd --user started with just ssh [13:34] jdstrand: ok, I'll debug both cases (console login and ssh) [13:34] jdstrand: were you logging into the same user as the interactive session? [13:34] jdstrand: or another user [13:34] zyga: but yeah. maybe there is an activation thing or something. idk [13:34] I sopped looking when I ran out of day [13:34] stopped [13:34] ok [13:34] I'll explore those [13:35] because previously it was a bit magic [13:35] zyga: I was [13:35] as to why it worked in places I checked [13:35] (same user) [13:35] ok [13:35] that would explain the systemd --user aspect [13:36] zyga: well, no. I logged into core a few minutes ago before asking about adding uc tests and there was --user [13:36] jdstrand: ok, please give me a day to get to the bottom of this [13:36] jdstrand: if you could review any of the other PRs I mentioned I could perhaps make progress on those [13:36] jdstrand: but for this one, I know what to do [13:36] that doesn't mean there is a dbus session bus... [13:36] jdstrand: right [13:36] yes, that was the plan [13:36] jdstrand: perhaps 18.04 doesn't have enough activation [13:36] testing is paramount [13:37] I'll try to cover those [13:37] * jdstrand nods [13:42] mvo: fyi. github runner tries to create config files in the binary directory. This needs to be fixed first. Seems to be hardcoded path's. [13:42] https://github.com/sd-hd/runner-snap/releases [13:43] jdstrand, mvo: fyi https://github.com/snapcore/snapd/pull/7825#discussion_r381296772 [13:43] PR #7825: many: use transient scope for tracking apps and hooks [13:46] zyga: 0.05s? that is not bad :) [13:48] mvo: that's wall clock time [13:48] mvo: the actual new code is 1ms [13:57] you have a wall clock that measures 0.05s ?!? [13:58] ogra: well if you look hard enough a lot of wristwatches measure 0.25s (seconds hand moves at 4hz) [13:58] that's mechanical wind-up wristwatches mind you [13:58] * roadmr now looks really old and anachronistic ⌚ [14:00] haha, yeah [14:08] jdstrand: reproduced [14:08] https://www.irccloud.com/pastebin/lg1cEV4Q/ [14:08] jdstrand: ^ that's bionic over ssh as user [14:08] jdstrand: running systemd-run --user --scope ls [14:09] jdstrand: on the up side, I now have systemd --user running as my user (remote ssh activated it) [14:09] jdstrand: on the down side, no idea why it failed yet [14:09] but I can dig [14:10] * jdstrand nods [14:10] I <3 reproducers [14:10] jdstrand: but that's really weird, raspbian (older) works okay [14:10] jdstrand: but I'll dig [14:12] jdstrand: I'll check 20.04 and 16.04 as well [14:12] jdstrand: _complexity_ :) [14:14] zyga: yes. hopefully it isn't too deep (I recall how the snap_daemon spun out cause of bugs, bugs, bugs. I hope very much that isn't the case here (I don't think it is)) [14:16] I think it's just configuration (famous last words?) [14:18] jdstrand: zyga: your ssh shell should already be part of a user slice/session scope (?) [14:18] it is [14:19] zyga: same when you log in on a tty, though no clue who arranges that, pam & logind maybe? [14:19] s/who/what/ [14:20] mborzecki: the problem is why remote cannot systemd-run [14:20] smells like polkit to me [14:20] I'll look after the standup [14:23] hmm hmm intersting [14:23] jdstrand, pedronis : is there a non-system-files interface that would allow write access to /etc/ssh/ssh{,d}_config ? [14:24] if not, is it acceptable to grant system-files in this case (it's a device-specific snap in a brand store, so mostly well-scoped) [14:30] mborzecki: it's polkit [14:30] mborzecki: give me a while to get the bits in place [14:30] zyga: polkit? [14:30] yep [14:30] it's always polkit ;) [14:30] it's always /pluseaudio/polkit/snapd/ [14:30] heh forgot systemd [14:35] I need to take the dog out [14:38] who let the dogs out? woof wof... [14:42] zyga: doesn't look like polkit to me [14:42] openat(AT_FDCWD, "/sys/fs/cgroup/unified/user.slice/user-1000.slice/user@1000.service/run-rbcaa5f3010d646e38684382cacc4aa70.scope/cgroup.procs", O_WRONLY|O_NOCTTY|O_CLOEXEC) [14:42] = 34 [14:42] fcntl(34, F_GETFL) = 0x8001 (flags O_WRONLY|O_LARGEFILE) [14:42] write(34, "2821\n", 5) = -1 EACCES (Permission denied) [14:42] uh, w8 that's on focal ;) [14:46] hm weird, there's this sequence [14:46] openat(AT_FDCWD, "/sys/fs/cgroup/unified/user.slice/user-1000.slice/user@1000.service/run-r51cceeed963d4c33ace8cf029e2657d5.scope/cgroup.procs", O_WRONLY|O_NOCTTY|O_CLOEXEC) [14:46] = 30 [14:46] write(30, "2504\n", 5) = -1 EACCES (Permission denied) [14:47] close(30) [14:47] followed by opening *.scope directory and getdents() [14:48] anyways, the message is Failed to add PIDs to scope's control group: Permission denied which kind of makes sense in this context [14:53] re [14:53]