=== diddledan0 is now known as diddledan [00:32] PR core20#55 closed: hooks: make systemd-modules-load depend on mounts at /usr/lib/{firmware,modules} [00:57] PR snapd#8623 opened: tests/lib/prepare.sh: delete patching of the initrd <⚠ Critical> [05:49] morning [05:57] PR snapd#8622 closed: cmd/snap-bootstrap/initramfs-mounts: add sudoers to dirs to copy as well [06:30] mvo: hey [06:31] mvo: we can land https://github.com/snapcore/snapd/pull/8623 [06:31] PR #8623: tests/lib/prepare.sh: delete patching of the initrd <⚠ Critical> [06:31] mborzecki: cool [06:32] mborzecki: done [06:32] mvo: thanks [06:33] PR snapd#8623 closed: tests/lib/prepare.sh: delete patching of the initrd <⚠ Critical> [06:33] seems like that go 1.9 failure in wrappers is random after all [06:51] good morning [06:51] eventful evening? [06:52] zyga: good morning - we did all the bits missing for beta I think [06:52] zyga: snapd snap just building, that was the last piece, then we hopefully have a working beta [06:52] that's impressive [06:53] mvo: I'll do what claudio mentioned [06:53] boot it and leave it running [06:53] an aging test [06:53] zyga: nice [06:53] zyga: hey [06:54] :-) [06:54] guys, if you have daughters [06:54] hm? [06:54] then spend 30zł / < 10 euro [06:55] and play it with them [06:55] https://www.gog.com/game/foxtail [06:55] this game is incredibly beautiful and fun [06:55] localized, with click-to-advance in dialogue, so kids can read at their own pace [06:55] zyga: hmm looks like an old school adventure game? [06:55] it is [06:56] zyga: do you remember teenagent? :P [06:56] and the mood and looks are just so nicely crafted [06:56] mborzecki: yes but this is way better :) [06:56] (I bought teenagent as a teen) [06:56] no way xD [06:56] yeah :D [06:57] too bad gog doesn't support their client on linux [06:57] there's no DRM and it supports linux as well [06:57] mborzecki: yeah but the game is just a zip to download and run [06:59] zyga: well, it is, but it's clearly a work in progress title, so on linux you'll have trouble keeping up with the updates, none of the open source clients i tried seem to support updates nicely and there's some issue with gog updates api that apaprently makes it hard to integrate [06:59] mborzecki: this game is updated roughly once a year [06:59] mborzecki: anyway, just consider it [07:00] it really is worth the money [07:00] mborzecki: each update brings the next episode (3 out of 7 are available now) [07:00] it is made by a tiny studio [07:03] morning [07:03] good morning pawel! [07:13] anyone able to reproduce the unit test timeout in wrappers package? [07:13] mborzecki: on master? [07:15] mborzecki: I started go test -count 1000 ./wrappers/ [07:15] I'll let you know if it triggers [07:16] pedronis: hi, were you able to reproduce the unit test timeout in wrappers by any chance? [07:16] no [07:17] hm intersting, also the PRs that ijohnson opened yesterday were green today [07:17] it's definitely weird [07:17] also it might be that something else failed and the panic covers it because it doesn't get printed [07:18] yeah, that's true [07:18] mvo: hi, it's not needed for the beta but this also needs to be backported https://github.com/snapcore/snapd/pull/8602 I don't see it in 2.45 yet [07:18] PR #8602: configcore: only reload journald if systemd is new enough [07:21] mvo: this also wasn't ported https://github.com/snapcore/snapd/pull/8622/ ? [07:21] PR #8622: cmd/snap-bootstrap/initramfs-mounts: add sudoers to dirs to copy as well [07:30] pedronis: yes, that is right, let me fix that [07:34] pedronis: I moved https://github.com/snapcore/snapd/pull/8566 to snaplock [07:34] PR #8566: cmd/cmdutil: add run inhibition operations [07:34] once this is +1 I will adjust the dependencies [07:34] thx [07:35] need to do some other things before getting back to reviews [07:35] sure [07:35] I have plenty of things to write so not blocked [07:35] PR snapd#8624 opened: cmd/snap: fix the order of positional parameters in help output [07:37] trivial fix ^^ [07:38] mborzecki: I reproduced the panic [07:38] https://www.irccloud.com/pastebin/vmigrXPM/ [07:46] mborzecki: so [07:46] mborzecki: this trace is funny [07:46] mborzecki: we clearly exec something [07:46] yet the SetUpTest says [07:46] s.systemctlRestorer = systemd.MockSystemctl(func(cmd ...string) ([]byte, error) { [07:46] s.sysdLog = append(s.sysdLog, cmd) [07:46] return []byte("ActiveState=inactive\n"), nil [07:46] }) [07:46] which seems to suggest this mock is one thing [07:47] but calling "systemctl" executable is another [07:47] zyga: which go version is that? [07:47] 1.13.8 [07:47] brb [07:48] ok, so not go specific, probably just borked setup [07:48] or test [08:35] bugs, bugs, bugs [08:35] but also progres [08:40] #8564 got a +1 from pstolowski and now I applied his suggestions and is ready for 2nd reviews [08:40] PR #8564: asserts: introduce Pool [08:53] pedronis: what is the 'h' assertion header for? [08:53] mvo, hi, I canceled today sync as there isn't really anything to talk about [08:53] ackk: ok, I see there is a bug mentioned, I have a look at that [08:54] mvo, ah yeah, not sure if it's really relevant for snapd, but if you want we can talk about it. it's just me today [08:54] mvo, mostly, I added it there to ask if there was a best practice for knowing the underlying OS details from a snap [08:55] ackk: no need to meet just for that :) [08:55] ackk: we can cover it next time [08:55] right [09:01] zyga: they are test-only assertions, they look a bit like snap-declarations and snap-revisions [09:02] zyga: but unless you reviewed the other ones in this chain, I wouldn't recommend looking a tit [09:02] heh [09:02] at it [09:04] mborzecki: so that test failed again with the timeout [09:05] mvo: some the unit tests changes that came with jamesh PR that was merged yesterday are failing regularly but is not super clear what is going on [09:06] mvo: do you need to do a pre4 ? [09:08] mvo, I have a very basic snap with just meta/snap.yaml (to test the system-files interface). I snap pack it but when I install I get - Run configure hook of "testsnap" snap if present (run hook "configure": cannot snap-exec: no such file or directory) [09:08] mvo, why is it trying hooks when there are none? [09:09] ackk: do you have meta/hooks/configure? [09:09] zyga, nope [09:09] though the snap-exec error is also curious [09:09] just meta/snap.yaml and "script" , which is a bash script [09:09] I'm using base: bare if that mattes [09:09] *matters [09:09] ah [09:10] bare base has no shell? [09:10] no /bin/sh [09:10] oh, [09:10] though the configure hook is weird still [09:10] it's bare for a reason :D [09:10] in fact, it has nothing [09:10] you must provide a static binary [09:10] right, I forgot [09:11] pedronis: can you reproduce it? [09:11] pedronis: or was that in some PR? [09:11] mborzecki: a PR [09:12] pedronis: not sure if we need a pre4, I hope we can test what we have in beta and if it's all working fine I can do a 2.45 for real but it depends a bit. do you know already something that is broken? [09:12] mvo: no, just wondered because sudoers bits are not in pre3 [09:13] so you cannot really test recover [09:13] afaiu [09:13] pedronis: yeah, that was silly of me, I cherry picked it and pushed to release/2.45 and rebuild the beta. so it will be ~pre3+git but that should be ok for testing :) [09:14] pedronis: commit d16c44f0aa mentiones some issues with shutting down the session agent [09:15] I know [09:15] maybe we should try undoing and then it happens for often and try to really understand [09:15] s/for/more/ [09:16] anyway this is very annoying [09:25] hm wonde rif it's possible that the agent hits idle timeout and stops listening [09:26] mborzecki: ah [09:26] maybe should be possible to write a test to see what happens in that case [09:26] trying to provoke that in tests [09:26] but there's shoould be an Accept around [09:26] in that case, no? [09:26] sorry, there shouldn't be [09:27] ah right [09:30] zyga: I did another pass on #8566 [09:30] PR #8566: c/snaplock/runinhibit: add run inhibition operations [10:48] hmm we could use systemd version check to avoid LazyUnmount on core16 (which generates a lot of noise in system log) [10:49] pedronis: thank you for the review, I replied to one comment and I'll adjust the rest later today [10:49] pstolowski: +1 [10:49] pstolowski: but then we never rewrite .mount units [10:49] pstolowski: so -1 unless you know how to do that on systemd upgrades [10:50] ah, hmm [10:50] anyway, something to think about [11:01] PR snapd#8625 opened: wrappers: tweak the order of restoring, use client timeout in services test teardown [11:01] pedronis: ^^ maybe this will give us more insight [11:07] hm removed that thing poking the client in teardown and can reproduce the timeout now [11:11] zyga: maybe we could make systemd version part of systemkey [11:12] pstolowski: we don't rewrite mount units on system key [11:12] zyga: yes, but maybe then we would [11:12] pstolowski: what happens when you rewrite mount unit [11:13] I tried to propose that but it got stuck on this discussion [11:13] do we remount things? we cannot really [11:13] allright... just thinking aloud [11:13] no no, that's good [11:13] it's just we need to think how to do that in practice [11:15] zyga: interesting, firefox does this HOME: $SNAP_USER_COMMOM in its snap.yaml [11:15] I hope it's COMMON :) [11:15] but interesting, yeah [11:15] we give people the means and they get creative [11:26] meh, go test -timeout is kinda silly [11:26] PR snapd#8626 opened: tests: fix passing stdin via session-tool [11:27] mborzecki: ^ a trivial patch and an annoying discovery === pedronis_ is now known as pedornis === pedornis is now known as pedronis [11:27] mborzecki: your PR is green, so maybe we were trying to talk to the real agent? [11:27] do we need that poking code? [11:28] yeah, but it'd hard to explain why there would be another agent [11:28] maybe we activate it via socket? [11:28] do we mask the real agent? [11:28] however, go test ./... runs all package tests in parallel, if there's another package starting the agent, we could try to talk to taht other agent ? [11:28] ah, unit tests [11:28] (assuming mocking elewhere is incorrect too) [11:29] mborzecki: that sounds fragile, well there are agents own tests of course [11:30] mborzecki: anyway I'm still unsure why we need that poking code [11:30] right [11:31] perhaps we could land #8625 while i'll try to poke further [11:31] PR #8625: wrappers: tweak the order of restoring, use client timeout in services test teardown [11:31] mborzecki: yes, that's fine [11:38] it's finally warm enough to use the standing desk in the colder corner of the office :) [11:44] zyga, hi [11:44] hi [11:44] yesteday I left some errors related to session tool [11:44] I see you created a new PR with fixes [11:44] oh? [11:44] I didn't see those [11:45] which fixes? :) [11:45] zyga, is it related to this? https://paste.ubuntu.com/p/9Svw3Qd7yB/ [11:45] https://paste.ubuntu.com/p/k7fyPJch78/ [11:45] no [11:45] restore EOFs? [11:45] zyga, ahh, these fixes #8626 [11:45] PR #8626: tests: fix passing stdin via session-tool [11:46] that is related to another PR but not to the pastebin [11:46] is this EOF a one-off or something that always happens/ [11:46] zyga, restore EOF and also the other test [11:46] zyga, the test failing on this https://paste.ubuntu.com/p/k7fyPJch78/ [11:48] zyga, cannot connect to server -> curl --unix-socket /run/user/12345/snapd-session-agent.socket -D- -X POST -H 'Content-Type: application/json' -d '{"action": "daemon-reload"}' http://localhost/v1/service-control [11:50] zyga, this is also failing on uc20 https://paste.ubuntu.com/p/NgHhWVVPFW/ [11:50] zyga, the problem is that as it is failing on restore, it breaks the whole test suite [11:50] zyga, my concern is why it is not failing in google execution [11:51] but fails on edge and beta validation [12:06] re [12:06] cachio: probably random [12:07] cachio: unless it happens each time when executing a specific test [12:07] cachio: can you reproduce the problem in tests/main/snap-session-agent-service-control? [12:07] zyga, mborzecki it happens 100% of the time [12:07] mborzecki, yes I can [12:07] cachio: did you collect information about the other session? [12:08] cachio: from the session-tool failure? [12:08] ah [12:08] wait [12:08] sorry, I misread [12:08] it's the EOF [12:08] so no new data [12:08] perhaps the session agent isn't running yet/already/at all [12:08] zyga, no, but I can do it [12:08] it's only interesting iff we can reproduce it in isolation [12:08] cachio: if you can run session-tool test _alone_ [12:09] zyga, sure [12:09] ok [12:10] zyga, running [12:10] gime me 5 minutes until it fails [12:21] zyga, I reproduced the error [12:21] is any info which I could provide? [12:22] I have an ssh opened [12:22] what's the last thing that was logged in the spread run? [12:24] 2020-05-08 09:15:58 Error restoring external:ubuntu-core-18-64:tests/main/session-tool:test (external:ubuntu-core-18-64) : + session-tool --restore -u test [12:24] 2020-05-08 09:15:58 Error debugging external:ubuntu-core-18-64:tests/main/session-tool:test (external:ubuntu-core-18-64) : EOF [12:24] 2020-05-08 09:15:58 Restoring external:ubuntu-core-18-64:tests/main/ (external:ubuntu-core-18-64)... [12:24] 2020-05-08 09:15:58 Error restoring external:ubuntu-core-18-64:tests/main/ (external:ubuntu-core-18-64) : EOF [12:25] I could run with the flag to show the output [12:25] cachio: how do you log into the system? [12:25] it is a vm here [12:25] cachio: how does spread log into the system? [12:25] I it does not log [12:26] well, does it execute test by magic? [12:26] I have a spreac which show ssh on real time [12:26] it must connect to the system somehow [12:26] zyga, I have an image to reproduce the error if you want [12:26] no [12:26] I'd like to understand if this is a normal spread run that uses ssh to connect [12:27] zyga, it is a normal spread run [12:27] ok [12:27] using external backend [12:27] is it using ssh to log in as the root user on the device under test? [12:28] or is there any other user used [12:29] zyga, it uses root [12:29] same as in google backend [12:29] PR snapd#8627 opened: c/snap-bootstrap: port mount state mocking to the new style on master (2.45) [12:29] ok [12:29] cachio: can you log into the system [12:29] and run, as root [12:30] session-tool --prepare -u test [12:30] session-tool --restore -u test [12:32] zyga, done [12:32] did it EOF? [12:32] no [12:32] ok [12:32] is there anything in journal from the time of the failure? [12:34] zyga, https://paste.ubuntu.com/p/bfJYGthXwB/ [12:35] this is the only I see which seems to be relevant [12:35] I see it many times [12:35] cachio: what specifically? [12:36] May 08 12:15:58 localhost systemd[1]: session-tool-c24e7dda-59d7-4fbd-a871-fc2431b4f5d1.service: Main process exited, code=exited, status=1/FAILURE [12:36] the test is running "false" [12:37] Starting session-tool running false as test... [12:37] to check that exit status is forwarded [12:37] ah [12:39] cachio: K w [12:39] I wonder if this is just a network timeout over ssh [12:39] this test takes a while to run [12:39] maybe tweak it so that there's fewer iterations [12:39] and see if that changes anything [12:40] I am running again but now showing the output [12:40] so I'll see on which line fails [12:41] zyga, perhaps it helps [12:41] yeah, let's try [12:43] it is looping [12:56] cachio: any failures? [12:56] still preparing [12:56] ok [12:56] zyga, am I doing something wrong here https://bugs.launchpad.net/maas/+bug/1876217/comments/5 ? [12:56] I had to re-execute [12:56] Bug #1876217: Controllers report Ubuntu Core version in the Snap [12:56] because once it fails, then the session is not correctly cleaned and fails the next run [12:57] zyga, (wrt system-files plug) [12:57] zyga, I see same behaviour [12:57] as it calls itself it goes into an infinite loop [12:58] ackk: yeah [12:58] ackk: probably ..l [12:58] ls -ld /etc/os-release [12:58] is it a symlink? [12:58] zyga, outside or inside the snap? [12:59] well [12:59] outside [12:59] the hostfs one is failing [12:59] /etc/os-release -> ../usr/lib/os-release [12:59] so [12:59] ah I see [12:59] so I have to use the actual file? [12:59] apparmor grants you rights to read /var/lib/snapd/hostfs/etc/os-release [12:59] but you that is a symlink [12:59] so you need /var/lib/snapd/hostfs/usr/lib/os-release [13:00] and you need to understand this in your app [13:00] it kind of sucks because symlinks seen via hostfs are "hard" [13:00] it's good we don't have absolute paths in them [13:00] extend the read section [13:00] to say /var/lib/snapd/hostfs/usr/lib/os-release [13:00] (in addition to the etc line) [13:00] and you should be good [13:00] zyga, so I suspect that won't work for the purpose of being os-agnostic, as on other OSes that might not be a symlink [13:00] yeah [13:00] zyga, so it won't work even if the interface allows both? [13:00] the denial would also tell you [13:00] well [13:01] the symlink can in theory point anywhere [13:01] in practice it either is not a symlink [13:01] or it points to a finite set of files [13:01] which do differ by OS flavour (I recommend booting a fedora workstation) [13:02] zyga, I see, thanks [13:02] good luck! :) [13:02] oh [13:02] standup!! [13:03] ackk: I would prefer a snapctl os-release -like API [13:03] where you could just ask [13:04] and we'd tell you without this mess [13:04] zyga, yeah, that'd be good [13:05] zyga, alternatively the actual /etc/os-release content could be exposed in some other file, like /etc/os-release-host or something [13:05] although that would only work for core* bases [13:05] ackk: that's tricky too [13:05] it's easier to have an API [13:05] zyga, yeah I assumed it would be trickier :) [13:06] zyga, this is the last part of the output [13:06] https://paste.ubuntu.com/p/mysVkdXQ4M/ [13:06] "last" :D [13:07] I don't know where to look [13:07] the end looks like just cleanup [13:08] I am logging the full run now [13:08]