[04:58] morning [05:00] mborzecki: fire fire [05:00] zyga: hi, yeah, looking into it [05:00] https://bugzilla.redhat.com/show_bug.cgi?id=1716397 [05:01] Thank you! [05:14] zyga: makes me wonder, why snapd attempts to remount /var/lib/snapd/snap [05:14] zyga: maybe it's because it's trying to put some services in separate ns? [05:23] mborzecki: must be because of Private* directives in service files [05:24] zyga: ^^ [05:25] Remount? [05:25] Aha! [05:25] zyga: PrivateMounts=yes probably [05:25] Private directives is a systemd feature [05:25] Is it on by default? [05:25] zyga: then systemd sets up a mount namespace for the service [05:26] zyga: i think now (at least not on arch), but some services, like resolved have it enabled [05:26] So what is the interplay with snapd? [05:27] snaps are mounted with snappy_snap_t context, but init_t does not have permission to remount this [05:27] Ahhhh [05:27] Meh [05:28] Selinux is too complex! [05:28] So now we have to fix systemd policy? [05:28] zyga: no, just selinux [05:29] zyga: http://paste.ubuntu.com/p/fZ2hp7Y9hC/ the fix [05:29] i supose that's the best we can do outside of core/contrib policy [05:31] Why did our test suite not catch that? [05:31] And how does it cause people’s machines to become unresponsive? Is systemd hanging? [05:32] zyga: we don't reboot in the tests mostly, and it affects external services that need to be restarted, eg resolved, which we don't do [05:32] zyga: i guess sddm and such have PrivateHome or similar [05:34] mborzecki: let’s add a test that reboots [05:34] It is essential to provide good service [06:36] zyga: running a regression test spread test [06:39] hey mborzecki and zyga! [06:39] mvo: hey, some fire in fedora-land [06:39] mborzecki: uhoh, whats up? [06:39] mvo: https://bugzilla.redhat.com/show_bug.cgi?id=1708991 [06:40] * mvo looks [06:43] Hey mvo [06:43] I didn’t manage to do that review for avahi changes [06:44] Sorry, fighting uphill battle with kids [06:44] I’ll start around 10 [06:47] zyga: ok [06:47] zyga: about that spurious SELinux failure in tests, it's caused by the upgrade test, which installs snapd from the repo, a version we know to break with symlinks in /etc/ [06:48] mborzecki: I read the bugzilla now, do we have an idea about the fix already? [06:50] mvo: yes, i'm running a regression test under spread now [06:50] cool === pstolowski|afk is now known as pstolowski [07:03] morning [07:13] hey pstolowski ! good morning [07:18] pstolowski: hey [07:18] mvo: zyga: https://github.com/snapcore/snapd/pull/6946 [07:23] mborzecki: we will do a 2.39.2 for this one [07:23] mborzecki: it sound critical enough, yes? [07:24] mvo: yeah, neal asked whether we could do .2 [07:28] funny that there is no issue on centos [07:48] mborzecki: do you need the logs from #6944 ? I would like to restart it [07:49] mvo: no, go ahead [07:49] pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/6946 ? [07:49] Arriving in 10 min [07:51] mborzecki: ok [07:55] pstolowski: thanks! [07:58] Almost ready [08:00] re [08:00] okay [08:00] mborzecki: looking [08:02] mborzecki: I added some comments for 6946 [08:03] mborzecki: reviewed [08:03] zyga: can't push to fedora [08:03] why not? [08:03] ENOPERM? [08:04] if so we should fix that [08:04] but I can help meantime [08:05] zyga: yeah, no permissions :P [08:06] trying to log in on https://bugzilla.redhat.com/show_bug.cgi?id=1716736 [08:07] gets bounced via id.fedora* [08:07] then nothing [08:07] still not logged in [08:07] eh [08:08] browser woes [08:09] zyga: you can mark it as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1708991 [08:12] pstolowski: are you using the nested spread suite? [08:13] mborzecki: i used it for hotplug test, yes [08:13] mborzecki: sure [08:14] pstolowski: that's good [08:14] mborzecki: done [08:26] mborzecki: i haven't run it for a while; but i believe it's now run overnight (unless Sergio changed something) [08:30] pstolowski: hm no clue where those are started [08:31] pstolowski: there's no indication in travis.yaml or travis project page, probably sergio knows [08:33] got a quick errand to run, back in 30 [08:41] mborzecki: https://github.com/snapcore/snapd/pull/6946 is green [08:43] mborzecki: i don't know how this is all wired up with spread-cron, but i found a run from 6h ago here: https://travis-ci.org/snapcore/spread-cron/builds/541031798, you can see nested stuff is run there [08:51] * zyga moves somewhere quiet [09:01] pedronis: can https://github.com/snapcore/snapd/pull/6935 land (has +2 now) or do you want to take a look? [09:02] pstolowski: it's fine [09:02] ty [09:04] pedronis: btw, load-state was too fast in my tests to get saved in the state and was not reported, likely due to small size [09:11] re [09:35] * zyga is back home [09:35] apparently no good place for work today [09:35] ETOOHOT ETOOLOUD [09:36] ogra@localhost:~$ sudo hostnamectl set-hostname pi2 [09:36] Could not set property: Failed to set static hostname: Read-only file system [09:36] ogra@localhost:~$ ls /etc/writable/ [09:36] ogra@localhost:~$ ls -l /etc/hostname [09:36] lrwxrwxrwx 1 root root 17 Feb 11 14:50 /etc/hostname -> writable/hostname [09:36] mvo, i thought that was fixed (core18) [09:36] (just downloaded from cdimage stable) [09:39] ogra: can you please refresh core18? maybe the image is old? [09:39] it did just auto-refresh it seems [09:39] creating writable/hostname works fine btw [09:39] ogra: is there an /etc/writable ? [09:40] Chipaca, yes, but not the empty files [09:40] just touching hostname, localtime and timezone in the dir usually makes it work [09:45] oh, lovely ... and while i was just apt installing something under the classic snap, snapd decided to reboot in the middle of it [09:45] * Chipaca hands ogra a wand of shutdown -c [09:45] Chipaca, hard to do if the console is occupied by apt output :P [09:46] ogra: and 'tmux' is a classic snap :-( [09:46] (i know i can open a second ssh session but you have to be really fast since the shutdown happens so quickly in 18) [09:51] * Chipaca goes for coffee [09:53] * zyga is tired today [09:53] sorry everyone, next year I'm getting A/C [09:53] I'll get that coffee [09:54] but first some paperwoor [09:54] work [10:09] mvo: can you commit this suggestion please https://github.com/snapcore/snapd/pull/6945#discussion_r290222845 [10:10] zyga: already done [10:10] thank you [10:18] grah, there's a bug in a test :-( [10:18] * Chipaca grahs [10:18] * Chipaca restarts the test because it's a racy bug but looks into fixing it anyway [10:38] looks like i've hijacked all build slots on travis :) [10:39] mborzecki: *used* :) [10:39] it's good to keep things going [10:39] * zyga iterates locally === sparkieg` is now known as sparkiegeek [10:55] my mind is melting today [10:56] zyga: well +30 in shade here, summer came early? [10:56] mborzecki: I'm working from within an oven [10:57] tried to work from a cafeteria but the insane requirement to keep ultra loud music drove me away [10:57] I really hate that brand owners force branded stores to play loud music all day long [10:57] I got a headache and left [10:58] zyga: should have played some vader on your laptop to counter the ambient noise :) [10:58] the staff hates it [10:58] but they cannot turn it down [10:58] they literally cannot I was told [11:09] mvo, digging a bit deeper regarding the hostname/timezone/localtime stuff, i see the files in /snap/core/current/etc/writable/ and also only have one core installed yet ... seems the content of /etc/writable did not get copied for me [11:12] ogra@pi2:~$ grep etc/writable /etc/system-image/writable-paths [11:12] /etc/writable auto persistent none none [11:13] hmm, shouldnt the second arg ne "transition" ? [11:20] ok ... here is core16: [11:21] ogra@stream:~$ grep /etc/writable /etc/system-image/writable-paths [11:21] /etc/writable auto synced none none [11:21] why did we remove the "synced" between core 16 and 18 ? [11:24] ogra: perhaps, I recall some changes there [11:24] mvo: do you remember? [11:25] ogra: the problem with writable paths is that it is non-obvious what happens when a file is removed [11:25] well, "persistent none" is definitely wrong [11:25] and that it's pretty niche so we don't all have experience with it [11:26] well, if persistent is set nothing happens if a file is removed from the writable area ... if synced is set the one from the core(18) snap will be copied in place [11:26] http://manpages.ubuntu.com/manpages/xenial/man5/writable-paths.5.html is pretty clear about the bahaviour [11:26] *behaviour [11:28] or do you mean removed from the readonly part (it will not have any effect, we had a userspace removal service for that on the phone to make sure the writable side gets removed as well if needed) [11:30] in any case the current behaviour is wrong, the dir needs to be copied once on firstboot (whihc either "synced none" or "persistent transition" achieves", with synced you have a permaent connection between the two while "persistent transition" will only copy once on first boot) [11:31] ogra: I'm deep in another place, sorry I cannot help with this now [11:32] i didnt expect you to ... but i had the impression that was fixed a while (months ?) ago [11:32] * pstolowski lunch [11:38] mvo: https://github.com/snapcore/snapd/pull/6946 landed === ricab is now known as ricab|lunch [11:45] * zyga debugs stuff [11:45] gnome, how I hate your interactions [11:45] I cannot wait for WSL2 [11:45] I will happily run WSL2 and windows terminal [11:45] sorry gnome, you're full of great tech and you were surely made with love [11:46] it's me, it's not you [11:46] mvo, hey [11:47] zyga: how about garden gnomes? :) [11:47] mborzecki: same level of adoration [11:54] mvo, is it ok if I do some modifications to the autopkgtests change? [12:03] cachio: I think so, what do you want to change? [12:06] zyga, move the test to the new nightly suite [12:06] cachio: what are you aiming to achieve with that move? [12:06] off to pick up the kids [12:09] zyga, it is mainly because the find of tests we have in the nightly suite [12:11] zyga, but we can do it in a following PR [12:16] zyga, I am running the test now to see how it works within the nested suite [12:27] cachio: sure, what do you have in mind? [12:28] zyga: i failed to rework retry error handling, but come up with this https://github.com/snapcore/snapd/pull/6949 ; keen to see how travis likes it. local runs with network ns & spread tests on select machines were succesful [12:29] mborzecki: thanks, cherry-picked [12:30] ogra: hey, sorry for the delay. re "synced"> we don't have synced anymore as it was a source of confusion and issues, having transition there is fine if its missing of course [12:31] ogra: let me look at the issue at hand [12:32] mvo, yeah, we should definitely add transition to get the dir initially filled with files ... interestingly hostnamectl seems to behave differently on x86 (for lool ) and actually just creates the missing file (while it seems to try to modify /etc/hostname directly on my pi install) [12:33] so there seems to be an additional issue that doesnt show up if the filesystem is set up as expected [12:33] hmm [12:33] mborzecki: do you know when you are in debug shell [12:33] and then you run a command like foo.sh [12:33] when sh is a simple app that does exec /bin/sh "$@" [12:34] and then you end up in a loop with /bin/sh: 0: Bad substitution [12:34] and you cannot interrupt that [12:34] is that that app running itself? [12:34] I bump into this once in a while [12:34] and it is infuriating when it happens [12:34] (I just lost all state I had in this machine) [12:35] mvo, as the test is failing at least on ubuntu-16.04 running on nested vm [12:36] I though on moving it to the nightly suite [12:36] ogra: I created #131 against core18 [12:36] and there run spread with autopackgtest backend [12:37] cachio: meh, ok, I ran it only on 16.04 - what is the error? [12:37] mvo line 65: autopkgtest-buildvm-ubuntu-cloud: command not found [12:38] cachio: aha, ok - I think I know what to do [12:38] * zyga tries some more debugging [12:39] cachio: nightly is an interessting idea, we can test on more distro release then too, right? lets talk about it in a bit, need to talk to zyga first about something unrelated [12:39] mvo, sure [12:41] mvo, thanks ! [12:46] ogra: thanks for raising it [13:05] mborzecki: it's curious, just running spread -shell-before and running "sh" kills the system [13:05] checking === ricab|lunch is now known as ricab [13:15] mborzecki: it's PS1 [13:15] mborzecki: it's not compatible with sh [13:45] zyga, mborzecki: so are we going to have a 2.39.2, or should I cherry-pick on top of 2.39.1? [13:45] Eighth_Doctor: I would recommend cherry picking [13:45] we just discussed this [13:45] .2 would be for autopkg test [13:45] that's not really anything relevant [13:45] 🤷‍♂️ [13:46] and I think it's really better to cherry pick in both cases (maybe .2 is not needed at all) [13:46] I'll work on cherry-picking later today then [13:46] Eighth_Doctor: .1 also has a fix for symlinks in /etc/ [13:46] yeah, I saw that [14:02] Eighth_Doctor: thank you! [14:03] I'm suspending my desktop and going to look for shade in the bedroom; this place is at +40C now [14:03] mvo, hey [14:03] zyga: you should get air conditioning [14:03] I would have figured after last year, you would have bought an AC unit and set that up [14:03] so am also getting out of snpace issues runing autopkgtests on gce [14:04] mvo, perhaps the best is to remove the test from that PR, and I'll continue wotking on this on a following PR [14:04] otherise it is going to break the nested suite [14:05] otherwise [14:05] mvo, does it makes sense? [14:06] mvo, I'll need to create new images to support this new test [14:27] hey folks, is it possible to get https://github.com/snapcore/snapd/pull/6805 pulled into a 2.39.x ? or do we need to wait until 2.40 hits stable? [14:27] zyga: #6949 is green; are you able to test it on a suse build infra? [14:28] pstolowski: sure, let me try [14:29] cachio: yes, I can do this - fwiw, I got the same error you got - testbed failure due to timeout, I think we need to run "autopkgtest --timeout-factor=3" or something [14:29] cachio: I will remove the test now and force push and then we can do that in a separate PR [14:29] ijohnson: hey, let me look [14:29] ijohnson: that looks ok for 2.39.2 [14:30] thanks mvo! do you want me to file a PR against release/2.39 for that branch? [14:30] ijohnson: let me see if I can cherry pick cleanly, if so no need for a PR [14:31] mvo, nice, thanks [14:31] ijohnson: cherry pick worked, so all good, its part of release/2.39 now [14:31] mvo: thank you! [14:31] mvo, in the nightly suite is not giving that error but at some point the vm is going out of space [14:32] mvo and breaks [14:32] ijohnson: thank *you* for the PR :) [14:32] cachio: oh, woah [14:32] cachio: the nested VM? or the "parent" VM? [14:32] I think the nested [14:33] mvo, https://paste.ubuntu.com/p/G9qhxwcn5W/ [14:33] I mean the parent [14:34] cachio: thanks and *meh* thats annyoing, I can see why, the nested vm will also need quite a bit of space :/ [14:34] cachio: I removed all but the "integrationtest" commit [14:34] mvo, I'll updates those images with more space [14:34] cachio: thanks [14:34] mvo, nice [14:35] cachio: I will run adt manually now on all our supported distro releases to unblock this PR [14:35] nice [14:35] mvo, thanks [14:36] pstolowski: building now [14:41] zyga: ty [14:45] pstolowski: I need to rebase the patch, hold on [14:48] mborzecki: can you take a look at #6900 if you have a moment? [14:49] pstolowski: sure [14:49] pstolowski: some typos in the patch, check your editor please [14:51] pstolowski: rebased, not sure which other patches should be present [14:51] testing now [14:51] * cachio lunch [14:51] zyga: typos? [14:52] pstolowski: yes, commit message has several [14:52] pstolowski: the tests have failed: [14:52] https://www.irccloud.com/pastebin/GlYDnpJR/ [14:52] pstolowski: also the 1st line is too long for git commit messages [14:53] pstolowski: anyway, the essence is different [14:53] pstolowski: I pushed opensuse/2.39.1-patches [14:53] this is what I tested [14:53] please look at the patches, I can nuke any easily [14:55] zyga: ah, but this is something else, i haven't seen such failure when running with net ns [14:56] pstolowski: correct, I think inside obs there are more things changed [14:56] specifically the DNS server [14:56] wonderful [14:56] pstolowski: fun, right? [14:56] pstolowski: if you want I can show you how to set up for local testing [14:56] it's not complex actually [14:57] zyga: okay, i need some way of triggering this [14:58] zyga: need to got to vet now, bbiab [14:58] pstolowski: k, ping me later please [15:32] zyga: back [15:33] in a call now [15:33] pstolowski: install tumbleweed in a vm [15:33] pstolowski: maybe pick ext4 over btrfs [15:33] pstolowski: that's all you'll need [15:33] k [15:57] zyga: ok, i've it nwo [15:57] *now [16:01] pstolowski: you want to install whatever provides osc [16:01] pstolowski: osc checkout system:snappy [16:01] pstolowski: then osc build [16:01] it's fairly well documented [16:01] you will likely need to create a suse account [16:01] it's hand-holdy all the way so I think it should be smooth [16:01] once you have that I can show you how to trigger the failure [16:02] k [16:08] pstolowski: pstolowski: we can HO if you want to do that interactively [16:08] zyga: i've a checkout now [16:08] pstolowski: try osc build [16:08] that should pass [16:09] if you comment-out some of the patches you can osc build again and see the failures happen [16:16] zyga: ok, got successfull build as you said, trying without patches now [16:16] pstolowski: ok [16:16] pstolowski: you only need to change two lines [16:16] one that says Patch: ... at the top of the file [16:17] and another one slightly lower that says %patch3 ... [16:17] pstolowski: using a git branch, like the one I sent, is convenient [16:17] pstolowski: because you can then git format-patch 2.39.1..HEAD and simply copy them over [16:18] pstolowski: you can also tweak the spec file, so when it runs tests it will run that test only, that's somewhere around %check [16:18] zyga: good hint with git branch, thanks [16:19] zyga: ok, got test failure now as expected. good [16:19] pstolowski: ok, great! [16:19] pstolowski: if you want I can dig around to understand what exactly is going on when osc build runs [16:19] pstolowski: I didn't look that deep yet [16:23] zyga: looks like a chroot in /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 ? [16:23] pstolowski: yep, likely [16:26] * zyga jumps back at mount issues [16:29] will continue digging tomorrow === pstolowski is now known as pstolowski|afk [16:33] sure, thank you pstolowski|afk [16:36] mvo, should we merge the autopkgtests PR now? [16:40] cachio: I think so, we need a test too but that can be in a followup, I will do 2.39.2 with those fixes in my morning then [16:41] mvo, perfect [17:23] EOD here 👋 [18:21] zyga: 2.39.1 updates proposed [18:21] F30: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3c4eec8250 [18:21] F29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-02e160af16 [18:21] EL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-91a080fd55 [19:05] * cachio afk [19:26] Pharaoh_Atem: thank you [19:40] zyga: I need testing and karma [19:40] otherwise this update will not ship anytime soon