[04:58] <mborzecki> morning
[05:00] <zyga> mborzecki: fire fire
[05:00] <mborzecki> zyga: hi, yeah, looking into it
[05:00] <zyga> https://bugzilla.redhat.com/show_bug.cgi?id=1716397
[05:01] <zyga> Thank you!
[05:14] <mborzecki> zyga: makes me wonder, why snapd attempts to remount /var/lib/snapd/snap
[05:14] <mborzecki> zyga: maybe it's because it's trying to put some services in separate ns?
[05:23] <mborzecki> mborzecki: must be because of Private* directives in service files
[05:24] <mborzecki> zyga: ^^
[05:25] <zyga> Remount?
[05:25] <zyga> Aha!
[05:25] <mborzecki> zyga: PrivateMounts=yes probably
[05:25] <zyga> Private directives is a systemd feature
[05:25] <zyga> Is it on by default?
[05:25] <mborzecki> zyga: then systemd sets up a mount namespace for the service
[05:26] <mborzecki> zyga: i think now (at least not on arch), but some services, like resolved have it enabled
[05:26] <zyga> So what is the interplay with snapd?
[05:27] <mborzecki> snaps are mounted with snappy_snap_t context, but init_t does not have permission to remount this
[05:27] <zyga> Ahhhh
[05:27] <zyga> Meh
[05:28] <zyga> Selinux is too complex!
[05:28] <zyga> So now we have to fix systemd policy?
[05:28] <mborzecki> zyga: no, just selinux
[05:29] <mborzecki> zyga: http://paste.ubuntu.com/p/fZ2hp7Y9hC/ the fix
[05:29] <mborzecki> i supose that's the best we can do outside of core/contrib policy
[05:31] <zyga> Why did our test suite not catch that?
[05:31] <zyga> And how does it cause people’s machines to become unresponsive? Is systemd hanging?
[05:32] <mborzecki> zyga: we don't reboot in the tests mostly, and it affects external services that need to be restarted, eg resolved, which we don't do
[05:32] <mborzecki> zyga: i guess sddm and such have PrivateHome or similar
[05:34] <zyga> mborzecki: let’s add a test that reboots
[05:34] <zyga> It is essential to provide good service
[06:36] <mborzecki> zyga: running a regression test spread test
[06:39] <mvo> hey mborzecki and zyga!
[06:39] <mborzecki> mvo: hey, some fire in fedora-land
[06:39] <mvo> mborzecki: uhoh, whats up?
[06:39] <mborzecki> mvo: https://bugzilla.redhat.com/show_bug.cgi?id=1708991
[06:40]  * mvo looks
[06:43] <zyga> Hey mvo
[06:43] <zyga> I didn’t manage to do that review for avahi changes
[06:44] <zyga> Sorry, fighting uphill battle with kids
[06:44] <zyga> I’ll start around 10
[06:47] <mvo> zyga: ok
[06:47] <mborzecki> zyga: about that spurious SELinux failure in tests, it's caused by the upgrade test, which installs snapd from the repo, a version we know to break with symlinks in /etc/
[06:48] <mvo> mborzecki: I read the bugzilla now, do we have an idea about the fix already?
[06:50] <mborzecki> mvo: yes, i'm running a regression test under spread now
[06:50] <mvo> cool
[07:03] <pstolowski> morning
[07:13] <mvo> hey pstolowski ! good morning
[07:18] <mborzecki> pstolowski: hey
[07:18] <mborzecki> mvo: zyga: https://github.com/snapcore/snapd/pull/6946
[07:23] <mvo> mborzecki: we will do a 2.39.2 for this one
[07:23] <mvo> mborzecki: it sound critical enough, yes?
[07:24] <mborzecki> mvo: yeah, neal asked whether we could do .2
[07:28] <mborzecki> funny that there is no issue on centos
[07:48] <mvo> mborzecki: do you need the logs from #6944 ? I would like to restart it
[07:49] <mborzecki> mvo: no, go ahead
[07:49] <mborzecki> pstolowski: can you take a look at https://github.com/snapcore/snapd/pull/6946 ?
[07:49] <zyga> Arriving in 10 min
[07:51] <pstolowski> mborzecki: ok
[07:55] <mborzecki> pstolowski: thanks!
[07:58] <zyga> Almost ready
[08:00] <zyga> re
[08:00] <zyga> okay
[08:00] <zyga> mborzecki: looking
[08:02] <mvo> mborzecki: I added some comments for 6946
[08:03] <zyga> mborzecki: reviewed
[08:03] <mborzecki> zyga: can't push to fedora
[08:03] <zyga> why not?
[08:03] <zyga> ENOPERM?
[08:04] <zyga> if so we should fix that
[08:04] <zyga> but I can help meantime
[08:05] <mborzecki> zyga: yeah, no permissions :P
[08:06] <zyga> trying to log in on https://bugzilla.redhat.com/show_bug.cgi?id=1716736
[08:07] <zyga> gets bounced via id.fedora*
[08:07] <zyga> then nothing
[08:07] <zyga> still not logged in
[08:07] <zyga> eh
[08:08] <zyga> browser woes
[08:09] <mborzecki> zyga: you can mark it as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1708991
[08:12] <mborzecki> pstolowski: are you using the nested spread suite?
[08:13] <pstolowski> mborzecki: i used it for hotplug test, yes
[08:13] <zyga> mborzecki: sure
[08:14] <mborzecki> pstolowski: that's good
[08:14] <zyga> mborzecki: done
[08:26] <pstolowski> mborzecki: i haven't run it for a while; but i believe it's now run overnight (unless Sergio changed something)
[08:30] <mborzecki> pstolowski: hm no clue where those are started
[08:31] <mborzecki> pstolowski: there's no indication in travis.yaml or travis project page, probably sergio knows
[08:33] <mborzecki> got a quick errand to run, back in 30
[08:41] <zyga> mborzecki: https://github.com/snapcore/snapd/pull/6946 is green
[08:43] <pstolowski> mborzecki: i don't know how this is all wired up with spread-cron, but i found a run from 6h ago here: https://travis-ci.org/snapcore/spread-cron/builds/541031798, you can see nested stuff is run there
[08:51]  * zyga moves  somewhere  quiet
[09:01] <pstolowski> pedronis: can https://github.com/snapcore/snapd/pull/6935 land (has +2 now) or do you want to take a look?
[09:02] <pedronis> pstolowski: it's fine
[09:02] <pstolowski> ty
[09:04] <pstolowski> pedronis: btw, load-state was too fast in my tests to get saved in the state and was not reported, likely due to small size
[09:11] <mborzecki> re
[09:35]  * zyga is back home
[09:35] <zyga> apparently no good place for work today
[09:35] <zyga> ETOOHOT ETOOLOUD
[09:36] <ogra> ogra@localhost:~$ sudo hostnamectl set-hostname pi2
[09:36] <ogra> Could not set property: Failed to set static hostname: Read-only file system
[09:36] <ogra> ogra@localhost:~$ ls /etc/writable/
[09:36] <ogra> ogra@localhost:~$ ls -l /etc/hostname
[09:36] <ogra> lrwxrwxrwx 1 root root 17 Feb 11 14:50 /etc/hostname -> writable/hostname
[09:36] <ogra> mvo, i thought that was fixed (core18)
[09:36] <ogra> (just downloaded from cdimage stable)
[09:39] <mvo> ogra: can you please refresh core18? maybe the image is old?
[09:39] <ogra> it did just auto-refresh it seems
[09:39] <ogra> creating writable/hostname works fine btw
[09:39] <Chipaca> ogra: is there an /etc/writable ?
[09:40] <ogra> Chipaca, yes, but not the empty files
[09:40] <ogra> just touching hostname, localtime and timezone in the dir usually makes  it work
[09:45] <ogra> oh, lovely ... and while i was just apt installing something under the classic snap, snapd decided to reboot in the middle of it
[09:45]  * Chipaca hands ogra a wand of shutdown -c
[09:45] <ogra> Chipaca, hard to do if the console is occupied by apt output :P
[09:46] <Chipaca> ogra: and 'tmux' is a classic snap :-(
[09:46] <ogra> (i know i can open a second ssh session but you have to be really fast since the shutdown happens so quickly in 18)
[09:51]  * Chipaca goes for coffee
[09:53]  * zyga is tired today
[09:53] <zyga> sorry everyone, next year I'm getting A/C
[09:53] <zyga> I'll get that coffee
[09:54] <zyga> but first some paperwoor
[09:54] <zyga> work
[10:09] <zyga> mvo: can you commit this suggestion please https://github.com/snapcore/snapd/pull/6945#discussion_r290222845
[10:10] <mvo> zyga: already done
[10:10] <zyga> thank you
[10:18] <Chipaca> grah, there's a bug in a test :-(
[10:18]  * Chipaca grahs
[10:18]  * Chipaca restarts the test because it's a racy bug but looks into fixing it anyway
[10:38] <mborzecki> looks like i've hijacked all build slots on travis :)
[10:39] <zyga> mborzecki: *used* :)
[10:39] <zyga> it's good to keep things going
[10:39]  * zyga iterates locally
[10:55] <zyga> my mind is melting today
[10:56] <mborzecki> zyga: well +30 in shade here, summer came early?
[10:56] <zyga> mborzecki: I'm working from within an oven
[10:57] <zyga> tried to work from a cafeteria but the insane requirement to keep ultra loud music drove me away
[10:57] <zyga> I really hate that brand owners force branded stores to play loud music all day long
[10:57] <zyga> I got a headache and left
[10:58] <mborzecki> zyga: should have played some vader on your laptop to counter the ambient noise :)
[10:58] <zyga> the staff hates it
[10:58] <zyga> but they cannot turn it down
[10:58] <zyga> they literally cannot I was told
[11:09] <ogra> mvo, digging a bit deeper regarding the hostname/timezone/localtime stuff, i see the files in /snap/core/current/etc/writable/ and also only have one core installed yet ... seems the content of /etc/writable did not get copied for me
[11:12] <ogra> ogra@pi2:~$ grep etc/writable /etc/system-image/writable-paths
[11:12] <ogra> /etc/writable                           auto                    persistent  none        none
[11:13] <ogra> hmm, shouldnt the second arg ne "transition" ?
[11:20] <ogra> ok ... here is core16:
[11:21] <ogra> ogra@stream:~$ grep /etc/writable /etc/system-image/writable-paths
[11:21] <ogra> /etc/writable                           auto                    synced      none        none
[11:21] <ogra> why did we remove the "synced" between core 16 and 18 ?
[11:24] <zyga> ogra: perhaps, I recall some changes there
[11:24] <zyga> mvo: do you remember?
[11:25] <zyga> ogra: the problem with writable paths is that it is non-obvious what happens when a file is removed
[11:25] <ogra> well, "persistent none" is definitely wrong
[11:25] <zyga> and that it's pretty niche so we don't all have experience with it
[11:26] <ogra> well, if persistent is set nothing happens if a file is removed from the writable area ... if synced is set the one from the core(18) snap will be copied in place
[11:26] <ogra> http://manpages.ubuntu.com/manpages/xenial/man5/writable-paths.5.html is pretty clear about the bahaviour
[11:26] <ogra> *behaviour
[11:28] <ogra> or do you mean removed from the readonly part (it will not have any effect, we had a userspace removal service for that on the phone to make sure the writable side gets removed as well if needed)
[11:30] <ogra> in any case the current behaviour is wrong, the dir needs to be copied once on firstboot (whihc either "synced none" or "persistent transition" achieves", with synced you have a permaent connection between the two while "persistent transition" will only copy once on first boot)
[11:31] <zyga> ogra: I'm deep in another place, sorry I cannot help with this now
[11:32] <ogra> i didnt expect you to ... but i had the impression that was fixed a while (months ?) ago
[11:32]  * pstolowski lunch
[11:38] <mborzecki> mvo: https://github.com/snapcore/snapd/pull/6946 landed
[11:45]  * zyga debugs stuff
[11:45] <zyga> gnome, how I hate your interactions
[11:45] <zyga> I cannot wait for WSL2
[11:45] <zyga> I will happily run WSL2 and windows terminal
[11:45] <zyga> sorry gnome, you're full of great tech and you were surely made with love
[11:46] <zyga> it's me, it's not you
[11:46] <cachio> mvo, hey
[11:47] <mborzecki> zyga: how about garden gnomes? :)
[11:47] <zyga> mborzecki: same level of adoration
[11:54] <cachio> mvo, is it ok if I do some modifications to the autopkgtests change?
[12:03] <zyga> cachio: I think so, what do you want to change?
[12:06] <cachio> zyga, move the test to the new nightly suite
[12:06] <zyga> cachio: what are you aiming to achieve with that move?
[12:06] <mborzecki> off to pick up the kids
[12:09] <cachio> zyga, it is mainly because the find of tests we have in the nightly suite
[12:11] <cachio> zyga, but we can do it in a following PR
[12:16] <cachio> zyga, I am running the test now to see how it works within the nested suite
[12:27] <mvo> cachio: sure, what do you have in mind?
[12:28] <pstolowski> zyga: i failed to rework retry error handling, but come up with this https://github.com/snapcore/snapd/pull/6949 ; keen to see how travis likes it. local runs with network ns & spread tests on select machines were succesful
[12:29] <mvo> mborzecki: thanks, cherry-picked
[12:30] <mvo> ogra: hey, sorry for the delay. re "synced"> we don't have synced anymore as it was a source of confusion and issues, having transition there is fine if its missing of course
[12:31] <mvo> ogra: let me look at the issue at hand
[12:32] <ogra> mvo, yeah, we should definitely add transition to get the dir initially filled with files ... interestingly hostnamectl seems to behave differently on x86 (for lool ) and actually just creates the missing file (while it seems to try to modify /etc/hostname directly on my pi install)
[12:33] <ogra> so there seems to be an additional issue that doesnt show up if the filesystem is set up as expected
[12:33] <zyga> hmm
[12:33] <zyga> mborzecki: do you know when you are in debug shell
[12:33] <zyga> and then you run a command like foo.sh
[12:33] <zyga> when sh is a simple app that does exec /bin/sh "$@"
[12:34] <zyga> and then you end up in a loop with /bin/sh: 0: Bad substitution
[12:34] <zyga> and you cannot interrupt that
[12:34] <zyga> is that that app running itself?
[12:34] <zyga> I bump into this once in a while
[12:34] <zyga> and it is infuriating when it happens
[12:34] <zyga> (I just lost all state I had in this machine)
[12:35] <cachio> mvo, as the test is failing at least on ubuntu-16.04 running on nested vm
[12:36] <cachio> I though on moving it to the nightly suite
[12:36] <mvo> ogra: I created #131 against core18
[12:36] <cachio> and there run spread with autopackgtest backend
[12:37] <mvo> cachio: meh, ok, I ran it only on 16.04 - what is the error?
[12:37] <cachio> mvo line 65: autopkgtest-buildvm-ubuntu-cloud: command not found
[12:38] <mvo> cachio: aha, ok - I think I know what to do
[12:38]  * zyga tries some more debugging
[12:39] <mvo> cachio: nightly is an interessting idea, we can test on more distro release then too, right? lets talk about it in a bit, need to talk to zyga first about something unrelated
[12:39] <cachio> mvo, sure
[12:41] <ogra> mvo, thanks !
[12:46] <mvo> ogra: thanks for raising it
[13:05] <zyga> mborzecki: it's curious, just running spread -shell-before and running "sh" kills the system
[13:05] <zyga> checking
[13:15] <zyga> mborzecki: it's PS1
[13:15] <zyga> mborzecki: it's not compatible with sh
[13:45] <Eighth_Doctor> zyga, mborzecki: so are we going to have a 2.39.2, or should I cherry-pick on top of 2.39.1?
[13:45] <zyga> Eighth_Doctor: I would recommend cherry picking
[13:45] <zyga> we just discussed this
[13:45] <zyga> .2 would be for autopkg test
[13:45] <zyga> that's not really anything relevant
[13:45] <Eighth_Doctor> 🤷‍♂️
[13:46] <zyga> and I think it's really better to cherry pick in both cases (maybe .2 is not needed at all)
[13:46] <Eighth_Doctor> I'll work on cherry-picking later today then
[13:46] <mborzecki> Eighth_Doctor: .1 also has a fix for symlinks in /etc/
[13:46] <Eighth_Doctor> yeah, I saw that
[14:02] <zyga> Eighth_Doctor: thank you!
[14:03] <zyga> I'm suspending my desktop and going to look for shade in the bedroom; this place is at +40C now
[14:03] <cachio> mvo, hey
[14:03] <Eighth_Doctor> zyga: you should get air conditioning
[14:03] <Eighth_Doctor> I would have figured after last year, you would have bought an AC unit and set that up
[14:03] <cachio> so am also getting out of snpace issues runing autopkgtests on gce
[14:04] <cachio> mvo, perhaps the best is to remove the test from that PR, and I'll continue wotking on this on a following PR
[14:04] <cachio> otherise it is going to break the nested suite
[14:05] <cachio> otherwise
[14:05] <cachio> mvo, does it makes sense?
[14:06] <cachio> mvo, I'll need to create new images to support this new test
[14:27] <ijohnson> hey folks, is it possible to get https://github.com/snapcore/snapd/pull/6805 pulled into a 2.39.x ? or do we need to wait until 2.40 hits stable?
[14:27] <pstolowski> zyga: #6949 is green; are you able to test it on a suse build infra?
[14:28] <zyga> pstolowski: sure, let me try
[14:29] <mvo> cachio: yes, I can do this - fwiw, I got the same error you got - testbed failure due to timeout, I think we need to run "autopkgtest --timeout-factor=3" or something
[14:29] <mvo> cachio: I will remove the test now and force push and then we can do that in a separate PR
[14:29] <mvo> ijohnson: hey, let me look
[14:29] <mvo> ijohnson: that looks ok for 2.39.2
[14:30] <ijohnson> thanks mvo! do you want me to file a PR against release/2.39 for that branch?
[14:30] <mvo> ijohnson: let me see if I can cherry pick cleanly, if so no need for a PR
[14:31] <cachio> mvo, nice, thanks
[14:31] <mvo> ijohnson: cherry pick worked, so all good, its part of release/2.39 now
[14:31] <ijohnson> mvo: thank you!
[14:31] <cachio> mvo, in the nightly suite is not giving that error but at some point the vm is going out of space
[14:32] <cachio> mvo and breaks
[14:32] <mvo> ijohnson: thank *you* for the PR :)
[14:32] <mvo> cachio: oh, woah
[14:32] <mvo> cachio: the nested VM? or the "parent" VM?
[14:32] <cachio> I think the nested
[14:33] <cachio> mvo, https://paste.ubuntu.com/p/G9qhxwcn5W/
[14:33] <cachio> I mean the parent
[14:34] <mvo> cachio: thanks and *meh* thats annyoing, I can see why, the nested vm will also need quite a bit of space :/
[14:34] <mvo> cachio: I removed all but the "integrationtest" commit
[14:34] <cachio> mvo, I'll updates those images with more space
[14:34] <mvo> cachio: thanks
[14:34] <cachio> mvo, nice
[14:35] <mvo> cachio: I will run adt manually now on all our supported distro releases to unblock this PR
[14:35] <cachio> nice
[14:35] <cachio> mvo, thanks
[14:36] <zyga> pstolowski: building now
[14:41] <pstolowski> zyga: ty
[14:45] <zyga> pstolowski: I need to rebase the patch, hold on
[14:48] <pstolowski> mborzecki: can you take a look at #6900 if you have a moment?
[14:49] <mborzecki> pstolowski: sure
[14:49] <zyga> pstolowski: some typos in the patch, check your editor please
[14:51] <zyga> pstolowski: rebased, not sure which other patches should be present
[14:51] <zyga> testing now
[14:51]  * cachio lunch
[14:51] <pstolowski> zyga: typos?
[14:52] <zyga> pstolowski: yes, commit message has several
[14:52] <zyga> pstolowski: the  tests have failed:
[14:52] <zyga> https://www.irccloud.com/pastebin/GlYDnpJR/
[14:52] <zyga> pstolowski: also the 1st line is too long for git commit messages
[14:53] <zyga> pstolowski: anyway, the essence  is different
[14:53] <zyga> pstolowski: I pushed opensuse/2.39.1-patches
[14:53] <zyga> this is what  I tested
[14:53] <zyga> please look at  the patches, I can nuke any easily
[14:55] <pstolowski> zyga: ah, but this is something else, i haven't seen such failure when running with net ns
[14:56] <zyga> pstolowski: correct, I think inside obs there are  more things changed
[14:56] <zyga> specifically the DNS server
[14:56] <pstolowski> wonderful
[14:56] <zyga> pstolowski: fun, right?
[14:56] <zyga> pstolowski: if you want I can show you how to set up for local testing
[14:56] <zyga> it's not complex actually
[14:57] <pstolowski> zyga: okay, i need some way of triggering this
[14:58] <pstolowski> zyga: need to got to vet now, bbiab
[14:58] <zyga> pstolowski: k, ping me later please
[15:32] <pstolowski> zyga: back
[15:33] <zyga> in a call now
[15:33] <zyga> pstolowski: install tumbleweed in a vm
[15:33] <zyga> pstolowski: maybe pick ext4 over btrfs
[15:33] <zyga> pstolowski: that's all you'll need
[15:33] <pstolowski> k
[15:57] <pstolowski> zyga: ok, i've it nwo
[15:57] <pstolowski> *now
[16:01] <zyga> pstolowski: you want to install whatever provides osc
[16:01] <zyga> pstolowski: osc checkout system:snappy
[16:01] <zyga> pstolowski: then osc build
[16:01] <zyga> it's fairly well documented
[16:01] <zyga> you will likely need to create a suse account
[16:01] <zyga> it's hand-holdy all the way so I think it should be smooth
[16:01] <zyga> once you have that I can show you how to trigger the failure
[16:02] <pstolowski> k
[16:08] <zyga> pstolowski: pstolowski: we can HO if you want to do  that interactively
[16:08] <pstolowski> zyga: i've a checkout now
[16:08] <zyga> pstolowski: try osc build
[16:08] <zyga> that should  pass
[16:09] <zyga> if you comment-out some of the patches you can osc build again and see the failures happen
[16:16] <pstolowski> zyga: ok, got successfull build as you said, trying without patches now
[16:16] <zyga> pstolowski: ok
[16:16] <zyga> pstolowski: you only need to change two lines
[16:16] <zyga> one that says Patch: ... at the top of the file
[16:17] <zyga> and another one  slightly lower that says %patch3 ...
[16:17] <zyga> pstolowski: using a git branch, like the one I sent, is convenient
[16:17] <zyga> pstolowski: because you can then git format-patch 2.39.1..HEAD and simply copy them over
[16:18] <zyga> pstolowski: you can also tweak the spec file, so when it runs tests it will run that test only, that's somewhere around %check
[16:18] <pstolowski> zyga: good hint with git branch, thanks
[16:19] <pstolowski> zyga: ok, got test failure now as expected. good
[16:19] <zyga> pstolowski: ok, great!
[16:19] <zyga> pstolowski: if you want I can  dig around to understand what exactly is going on when osc build runs
[16:19] <zyga> pstolowski: I didn't look that deep yet
[16:23] <pstolowski> zyga: looks like a chroot in /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 ?
[16:23] <zyga> pstolowski: yep, likely
[16:26]  * zyga jumps back at mount issues
[16:29] <pstolowski> will continue digging tomorrow
[16:33] <zyga> sure, thank you pstolowski|afk
[16:36] <cachio> mvo, should we merge the autopkgtests PR now?
[16:40] <mvo> cachio: I think so, we need a test too but that can be in a followup, I will do 2.39.2 with those fixes in my morning then
[16:41] <cachio> mvo, perfect
[17:23] <Chipaca> EOD here 👋
[18:21] <Pharaoh_Atem> zyga: 2.39.1 updates proposed
[18:21] <Pharaoh_Atem> F30: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3c4eec8250
[18:21] <Pharaoh_Atem> F29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-02e160af16
[18:21] <Pharaoh_Atem> EL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-91a080fd55
[19:05]  * cachio afk
[19:26] <zyga> Pharaoh_Atem: thank you
[19:40] <Pharaoh_Atem> zyga: I need testing and karma
[19:40] <Pharaoh_Atem> otherwise this update will not ship anytime soon