=== chihchun_afk is now known as chihchun [06:10] morning [07:04] Good morning [07:21] zyga: hey [07:21] zyga: do you know who controls strace-static snap? [07:22] is it only mvo? [07:22] I believe so [07:22] What do you need? [07:23] well, it doesn't work on fedora, something about unexpected wait status value, i believe it's the recent kernel that's causing this [07:23] the edge version is 4.23 and stil not good enough [07:23] fedora ships strace 4.24 and that one works [07:23] i've worked around it in the spread suite, but we'll need to update the snap at some point [07:27] Mmm [07:28] Is 4.24 compatible with other distributions? [07:32] zyga: we'll see when we end up using it in the spread suite [07:32] zyga: fwiw i have 4.25 in arch [07:34] one last spread run and i'll be opening a PR with Fedora 29 [07:50] Ok [07:51] I will be around in 30 minutes [07:53] As usual, worked late last night === pstolowski|afk is now known as pstolowski [08:12] heyas [08:20] pstolowski: hey [08:35] back now [08:35] good morning pawel [08:36] how are you guys? [08:39] sorry for being sleepy, I was pushing some things last night [08:40] are we still affected by the issue where our logs are too long for travis? [08:45] PR snapd#6264 opened: spread, tests: add Fedora 29 [08:47] mborzecki: what is your stance on the repackaging issue? [08:47] I have not read the PR yet so if it is stated there please refer me instead [08:50] zyga: imo it's benign as long as we don't reexec on those system, i think we could easily not repack the core snap on such systems [08:50] why is it related to reexec? [08:50] like it happened with snap-device-helper, some things are always used from core [08:51] zyga: oh, w8, it's not, right reexec would still pull the libs from host so that's fine [08:51] or am I mistaken, that the issue was that repackaged, incompatible script was used [08:51] zyga: the spread test for dns error (#6222) is driving me crazy.. been trying a couple of tweaks yesterday evening. sounds crazy, but it looks like iptables-save+restore on debian doesn't actually restore the rules sometimes - some other test fails and when poking around in the system after failure i see the rule is active.. [08:51] PR #6222: cmd/snap: handle DNS error gracefully [08:51] zyga: so it's probably only snapctl at this point [08:51] which we could easily build statically [08:52] mborzecki: would the issue also surface on future ubuntu 20.04 distribution testing a core16 snaps? [08:52] pstolowski: can you dump the rules to compare them in text form? [08:53] mborzecki: when snap-confine, reexecuting, is linked to libc not present in core16 [08:54] zyga: afaict only tools that are executed inside snap mount ns would be affected, those are snapctl and snap-device-helper, snap-update-ns (but that's already linked statically) [08:54] indeed [08:54] that's a good observation [08:54] snapctl can be linked statically, it's go :) [08:55] as other tools will still execute on the outside [08:55] thanks [08:57] zyga: that shebang being rewritten from #!/bin/sh to #!/usr/bin/sh is probably some usr-merge thing again [08:58] yes [08:58] zyga: iptables-save dumps them in txt form; let me actually inspect that when some other test fails [08:58] pstolowski: yeah, just dump them in prepare and restore [08:58] and compare what we get [08:58] guys, I'm almost convinced there's something seriously broken [08:58] so consider, when inspecting magic failures, that set -e is indeed not working [08:59] so perhaps something failed earlier on and the script nonetheless advanced [08:59] though I still don't grasp the details [09:04] pstolowski: silly thought, maybe we should start our own dns server for that test [09:04] mborzecki: hmm but we would need to modify the test environment anyway, so potentially effect something else? [09:05] mborzecki: it seems like a primitive approach with just iptables -I .., followed by iptables -D ... works. or maybe i'm just lucky. [09:05] hehe :) if it works then it's fine [09:15] mborzecki, zyga: ha, reproduced. run all spread tests. a radom test failed (not the one that messes up with iptables). the rules dump file created by iptables-save is empty, but iptables has the dns reject rule [09:16] intriguing, how do you save and observe the tables? [09:16] and where do you store the dump? [09:16] mborzecki, zyga grr, ignore me. the rules should be empty :D [09:16] ah [09:16] mborzecki, zyga apparently iptables-restore didn't work or wasn't invoked [09:19] hmm, ok, not even that. rules dump is empty (as completely empty), but it should actually have some iptables-save noise [09:19] weird... [09:19] how do you dump the rules? [09:20] pstolowski: were there any rules in place when iptables-save was called? [09:22] mborzecki: no (and in such case iptables-save creates a kind empty file with empty chains) [09:23] is the file overwritten somewhere? [09:23] pstolowski: hm empty? shouldn't there be a timestamp and a list of all chanins (all empty) [09:24] mborzecki: yes [09:24] zyga: interesting theory [09:25] trying without iptables save/restore now, if it works i'm gonna leave it like that [09:30] #6264 is green :) [09:30] PR #6264: spread, tests: add Fedora 29 [09:45] damn, hitting unrelated errors in the PRs [09:45] appstream-id, refresh-all :/ [09:45] and mount thing obviously too [09:50] yes [09:50] bane of random failures [09:50] eh :/ [09:50] I really encourage everyone to attack them [09:50] they are crippling our ability to make progress === chihchun is now known as chihchun_afk [10:09] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/6261 ? [10:09] PR #6261: tests/lib/prepare: make sure that SELinux context of repacked core snap is controlled [10:09] it's finally green [10:10] k [10:17] zyga: about that xdg-desktop-portal: https://paste.ubuntu.com/p/nkFc5FJ3Qc/ [10:18] mmm [10:18] but i don't see what requires that really === chihchun_afk is now known as chihchun [10:18] we require it [10:19] zyga: well, it's not listed anywhere in the spec [10:20] hmm [10:20] perhaps I was mistaken [10:20] #6222 green as well; kicking it again to double check [10:20] but I recall something like that [10:20] PR #6222: cmd/snap: handle DNS error gracefully [10:21] pstolowski: ack [10:21] zyga: https://paste.ubuntu.com/p/Yf3bRmcjBr/ [10:21] pstolowski: is it safe to merge or do you anticipate tests breaking in the future? [10:21] hm [10:21] mborzecki: then I have no idea [10:22] * zyga -> tea [10:24] zyga: the test is dead simple and doesn't shouldn't have any side-effects, but you never know. i'll give it one more run to see. if it causes problem in the future i'll disable it [10:24] ok [10:25] zyga: heh i can actually remove xdg-desktop-portal and none of snapd components are removed [10:42] PR snapd#6261 closed: tests/lib/prepare: make sure that SELinux context of repacked core snap is controlled [10:49] pstolowski: if you could do this one too: https://github.com/snapcore/snapd/pull/6260 [10:50] PR #6260: spread, tests: use checkpoints when dumping audit log [10:51] zyga: no clue what's happening with dependencies there, left a note in the PR for now [10:51] sounds good [10:57] random mount failure on 6222 - Mount snap "test-snapd-tools_instance" (7) ([start snap-test\x2dsnapd\x2dtools_instance-7.mount] failed with exit status 1: Job for snap-test\x2dsnapd\x2dtools_instance-7.mount failed. [10:58] heh [11:01] on whihc system? [11:01] *which system [11:01] moved downstairs to the office [11:01] sorry, I don't feel well today [11:08] zyga: google:ubuntu-18.04-64:tests/main/refresh-all [11:08] 18.04 [11:08] mmm [11:08] has anyone seen this on 18.10? [11:20] PR snapd#6265 opened: cmd/snap: attempt to restore SELinux context of snap user directories [11:21] PR snapd#6260 closed: spread, tests: use checkpoints when dumping audit log [11:24] pstolowski: thanks! [11:27] zyga, hey, I tested the systemd fix but still failing [11:27] yw [11:27] there is a test interfaces-hostname-control [11:28] which found that error [11:28] which still fail [11:29] cachio: can you report on the SRU bug please [11:29] cachio: and talk to xnox about the details [11:29] sure [11:29] thank you [11:29] zyga, np [11:29] I have a branch with the change I did to check that [11:29] hi, is there a "hook" to do stuff like push files/configs to the multipass VM snapcraft uses? [11:29] if you want to take a look just tell me [11:31] cachio: that's fine, unless you want me to [11:32] ok, I'll explain more during the standup [11:51] #6222 is green again, merging [11:51] PR #6222: cmd/snap: handle DNS error gracefully [11:52] PR snapd#6222 closed: cmd/snap: handle DNS error gracefully [11:53] jdstrand: 6244 is read [11:53] *ready [11:53] pstolowski: https://github.com/snapcore/snapd/pull/6251 needs a 2nd review [11:53] PR #6251: cmd/snap-confine: refactor calling snapd tools into helper module [11:54] mborzecki: https://github.com/snapcore/snapd/pull/6149 needs a partially second review [11:54] PR #6149: cmd/snap-confine: capture initialized per-user mount ns [11:54] k, looking [11:54] thank you :) [11:56] mborzecki: any action needed on F29 PR in light of the weak dependency fact? [11:57] zyga: i'm running the fix under spread right now [11:57] ok [12:01] that umask thing keeps popping up [12:01] hmmm [12:01] tests leaking [12:01] or something else? [12:02] zyga: the test that passed on travis and locally before failed right now [12:04] * cachio afk [12:04] I'm back to debugging set -e bug [12:11] waaat?!!? [12:17] zyga: would it be possible to get this change merged? https://code.launchpad.net/~jamesh/snappy-hub/test-snapd-dbus-service/+merge/360118 -- this test package isn't currently used in snapd master, and I needed to update it a bit for the new version of the dbus activation changes [12:17] looking [12:18] if this is merged and published to the store, would snapd tests be affected straight away? [12:18] I mean, would any tests that used to pass start to fail? [12:19] zyga: there is no reference test-snapd-dbus-service in the tests/ directory of snapd master [12:19] ok [12:20] it originates from mvo's old abandoned PR [12:22] jdstrand: is there a merge service running against that BZR branch? [12:22] er [12:22] jamesh: ^ [12:22] sorry, bad tab completion [12:23] zyga: I don't think so. [12:23] ok, I'll merge it manually then [12:24] thank you. I think I'm getting close to being able to remove that "WIP" tag from my PR [12:24] (and I understand it might take a while to get reviewed) [12:25] hopefully more reviews will resume next week [12:26] I'm sure there's still problems to discover once my spread tests run on all distros [12:32] hmm [12:32] * zyga doesn't recall bzr specific weirdness [12:32] jamesh: I _think_ it is merged [12:33] yep [12:43] mborzecki: can you re-review https://github.com/snapcore/snapd/pull/6259/files [12:43] the set -e mystery is solved [12:43] PR #6259: tests: reduce verbosity around package installation <⚠ Critical> [12:43] I was so stuck looking at why dpkg-query failure was ignored [12:43] it was not [12:43] it was never executed [12:43] the exit above made sure of that [12:43] so the test would fail but it was returning early [12:58] zyga: something fishy about tests, i have no idea how that fedora 29 pr is green [12:58] haha [12:58] in which sense [12:58] maybe it is manual? [12:59] omg, damn, it is [13:03] PR snapd#6250 closed: data: set KillMode=process for snapd [13:06] advice welcome on https://github.com/snapcore/snapd/pull/6190 [13:06] PR #6190: overlord/configstate,features: expose features to snapd tools <⛔ Blocked> [13:06] is that good as is [13:06] or should I iterate on some aspect [13:06] should I split some parts out to land quicky [13:31] PR snapd#6259 closed: tests: reduce verbosity around package installation <⚠ Critical> [13:31] PR snapd#6266 opened: tests: make security-device-cgroups-{devmode,jailmode} work on arm devices [13:34] updated https://github.com/snapcore/snapd/pull/6149#discussion_r239058308 [13:34] PR #6149: cmd/snap-confine: capture initialized per-user mount ns [13:42] mborzecki: do you mean to make F29 non-manual in https://github.com/snapcore/snapd/pull/6264 [13:42] PR #6264: spread, tests: add Fedora 29 [13:43] zyga: yes, running with the latest batch of fixes, i'll label the PR as blocked for now [13:43] thank you [13:43] PR snapd#6267 opened: overlord: move Conf interface to configstate/config to avoid cyclic imports [13:43] mborzecki, pstolowski: I pushed a very simple PR to shrink my other feature PR, could you guys please look ^ [13:51] pstolowski: need a 2nd review on https://github.com/snapcore/snapd/pull/6149 :) [13:51] PR #6149: cmd/snap-confine: capture initialized per-user mount ns [13:51] feel that this is finally moving forward :) [13:53] zyga: ok, will do [14:16] zyga: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-ba8845b83b [14:24] xnox: hey, what going to happen now that the SRU verification of https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1778936 is failed? [14:25] Bug #1778936: please re-add Support-system-image-read-only-etc.patch

[14:30] zyga, https://github.com/sergiocazzolato/snapd/tree/tests-validate-lp-1778936 [14:30] cachio: can you add a note to that bug please, that explains how to reproduce the results of that test [14:31] cachio: specifically how to build a system image with core18 which contains systemd from proposed [14:31] ok [14:31] thank you :) [14:32] cachio: I'm not sure how what you changed there tests systemd form proposed [14:32] since our testing process doesn't involve unpacking systemd into the repackaged core18 snap [14:32] cachio: so you would still get the vanilla core18 from the store [14:32] which has systemd from stable [14:32] (well, from the archive) [14:33] cachio: may I ask how you checked the version of systemd that you observed? [14:34] cachio: in the effective image built with ubuntu-image? [14:34] when the test fails [14:34] zyga, it is possible to check systemd version in a debug session [14:34] mhm [14:34] and what is the version you saw? [14:35] 237-3ubuntu10.10 [14:35] and how does that relate to the version in main and in proposed? [14:35] in main is 237-3ubuntu10.7 [14:35] irrc [14:35] the version that is mentioned in the bug report as containing the fix is different [14:36] it is 239-7ubuntu7 [14:36] basically, what I do it to enable proposed and refresh systemd during the project preparation [14:36] or 237-3ubuntu10.8 in bionic [14:36] Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.10 [14:36] cachio: but that changes the starting throwaway image [14:36] not the core18 image you build [14:37] I don't see how the changes you did affect the core18 _snap_ that is used to build the image where tests execute [14:37] it may be that this is correct [14:37] but I don't understand it yet [14:37] and I want to since the verification failed and this is an urgent issue [14:38] let me create a debug session [14:38] zyga, perhaps it works [14:38] to understand better [14:38] you can create one but can you argument how this is expected to work [14:38] I want to see how the proposed package ends up in the core18 snap === chihchun is now known as chihchun_afk [14:40] so, we build core18 snap we use the deb snapd.deb we already created [14:41] how do we build the core18 snap? can you show me please? [14:41] zyga, sure, 1 sec [14:41] thanks [14:42] jdstrand: https://github.com/snapcore/snapd/pull/6244 is ready, if you have some time to look at the last patch I would love to land that and have that bug fixed :) [14:42] https://github.com/snapcore/snapd/blob/master/tests/lib/prepare.sh#L339 [14:42] PR #6244: release: detect too old apparmor_parser [14:42] zyga, ~ [14:42] cachio: looking [14:43] so [14:43] we unpack the _snapd_ snap [14:43] right? [14:43] yes [14:43] does the snapd snap contain systemd? [14:44] AFAIK systemd is inside the core18 snap [14:44] unless we are repackaing that one elsewhere, and based on the package from the host [14:44] I don't see how this was testing proposed [14:45] zyga, what I did is to enable proposed on this instance [14:46] and add a dependency on snapd.deb to the package on proposed [14:46] Sorry, in which instance? [14:46] so then if we build the image based on out .deb [14:46] instance is the ubuntu-18.04-64 that we use to create the image [14:46] The test was checking snapd on core18 or on Ubuntu 18.04? [14:47] zyga, the test checks on core 18 instance [14:47] which is build on the fly using a ubuntu bionic image [14:47] So explain how the choice of the host instance affects the system image built with Ubuntu image that is eventually tested? [14:48] My understanding is that the core18 snap is coming from the store [14:48] And that it contains systemd [14:49] kenvandine: oSoMoN: hi guys! a mozilla employee found an odd behavior regarding assertions (even though he installed the snap with `--dangerous`). He left more details in https://bugzilla.mozilla.org/show_bug.cgi?id=1297531. Would you know who would be suited to help him out? [14:49] And because we do not alter it in the test, that is the core18 snap is as-is from the store then the changes ran against the same systemd version as before [14:49] mmm, I see in the code that we modify the snapd snap to use what we have in the snapd.deb [14:49] Yes [14:49] But snapd snap does not contain systemd [14:49] hey jlorenzo [14:51] hey jlorenzo [14:51] reading that bug [14:51] I though that by setting a dependency for the snapd.deb to use the systemd from proposed is enough to force the image gets this systemd [14:51] but based on the code I am not 10% sure [14:51] 100% [14:52] It would if this was using apt [14:52] But we unpack just snapd.deb [14:52] Still [14:52] I believe you saw the right version of systemd [14:52] Can you tell me which channel of core18 was used in the test [14:53] Was it edge? [14:53] And is core18 edge using main or proposed? [14:53] we use edge [14:53] Perhaps you tested the SRU despite making changes that have no effect on the produced image [14:53] oSoMoN: thanks [14:54] Can you please check the core18 edge snap recipe [14:54] It should have this information [14:54] sure [14:54] Thank you :-) [14:56] jlorenzo, no need for an assertion, for local tests the browser-support interface can be manually connected, I'll comment on the bug to explain [14:59] who can kick the store review thingy for me? gog-galaxy-wine revision 62 is stuck [15:01] zyga, at least core is not built from proposed [15:01] zyga, I can't see the recipy for core18 [15:02] oSoMoN: got it! thank you very much for the help :) [15:03] jlorenzo, you're welcome [15:03] zyga, so, I'll try to build a snapd.snap with systemd from proposed [15:03] zyga, that should work right? [15:06] re [15:06] cachio: no, build core18 with systemd from proposed [15:06] cachio: snapd.snap version is irrelevant [15:07] cachio: can you ask foundations for help, they should be able to point you to the recipe [15:07] zyga, great, thanks [15:07] I'll try that [15:07] sure, let me know if you cannot find the recipe, I can look as well [15:08] cachio: and please add a comment to the bug report that testing is still ongoing [15:08] so that xnox doesn't take action on incomplete information [15:11] zyga, sure, comment left [15:11] super, thank you [15:11] zyga, thanks for the help [15:23] cachio, if you give me something i can boot in a VM, i'm happy to validate things =) [15:23] w.r.t. etc-writable stuff. [15:29] xnox: who builds the core18 images now, is that foundations? [15:29] sorry, I was imprecise, I meant core18.snap [15:31]