/srv/irclogs.ubuntu.com/2019/06/04/#snappy.txt

mborzeckimorning04:58
zygamborzecki: fire fire05:00
mborzeckizyga: hi, yeah, looking into it05:00
zygahttps://bugzilla.redhat.com/show_bug.cgi?id=171639705:00
zygaThank you!05:01
mborzeckizyga: makes me wonder, why snapd attempts to remount /var/lib/snapd/snap05:14
mborzeckizyga: maybe it's because it's trying to put some services in separate ns?05:14
mborzeckimborzecki: must be because of Private* directives in service files05:23
mborzeckizyga: ^^05:24
zygaRemount?05:25
zygaAha!05:25
mborzeckizyga: PrivateMounts=yes probably05:25
zygaPrivate directives is a systemd feature05:25
zygaIs it on by default?05:25
mborzeckizyga: then systemd sets up a mount namespace for the service05:25
mborzeckizyga: i think now (at least not on arch), but some services, like resolved have it enabled05:26
zygaSo what is the interplay with snapd?05:26
mborzeckisnaps are mounted with snappy_snap_t context, but init_t does not have permission to remount this05:27
zygaAhhhh05:27
zygaMeh05:27
zygaSelinux is too complex!05:28
zygaSo now we have to fix systemd policy?05:28
mborzeckizyga: no, just selinux05:28
mborzeckizyga: http://paste.ubuntu.com/p/fZ2hp7Y9hC/ the fix05:29
mborzeckii supose that's the best we can do outside of core/contrib policy05:29
zygaWhy did our test suite not catch that?05:31
zygaAnd how does it cause people’s machines to become unresponsive? Is systemd hanging?05:31
mborzeckizyga: we don't reboot in the tests mostly, and it affects external services that need to be restarted, eg resolved, which we don't do05:32
mborzeckizyga: i guess sddm and such have PrivateHome or similar05:32
zygamborzecki: let’s add a test that reboots05:34
zygaIt is essential to provide good service05:34
mborzeckizyga: running a regression test spread test06:36
mvohey mborzecki and zyga!06:39
mborzeckimvo: hey, some fire in fedora-land06:39
mvomborzecki: uhoh, whats up?06:39
mborzeckimvo: https://bugzilla.redhat.com/show_bug.cgi?id=170899106:39
* mvo looks06:40
zygaHey mvo06:43
zygaI didn’t manage to do that review for avahi changes06:43
zygaSorry, fighting uphill battle with kids06:44
zygaI’ll start around 1006:44
mvozyga: ok06:47
mborzeckizyga: about that spurious SELinux failure in tests, it's caused by the upgrade test, which installs snapd from the repo, a version we know to break with symlinks in /etc/06:47
mvomborzecki: I read the bugzilla now, do we have an idea about the fix already?06:48
mborzeckimvo: yes, i'm running a regression test under spread now06:50
mvocool06:50
=== pstolowski|afk is now known as pstolowski
pstolowskimorning07:03
mvohey pstolowski ! good morning07:13
mborzeckipstolowski: hey07:18
mborzeckimvo: zyga: https://github.com/snapcore/snapd/pull/694607:18
mvomborzecki: we will do a 2.39.2 for this one07:23
mvomborzecki: it sound critical enough, yes?07:23
mborzeckimvo: yeah, neal asked whether we could do .207:24
mborzeckifunny that there is no issue on centos07:28
mvomborzecki: do you need the logs from #6944 ? I would like to restart it07:48
mborzeckimvo: no, go ahead07:49
mborzeckipstolowski: can you take a look at https://github.com/snapcore/snapd/pull/6946 ?07:49
zygaArriving in 10 min07:49
pstolowskimborzecki: ok07:51
mborzeckipstolowski: thanks!07:55
zygaAlmost ready07:58
zygare08:00
zygaokay08:00
zygamborzecki: looking08:00
mvomborzecki: I added some comments for 694608:02
zygamborzecki: reviewed08:03
mborzeckizyga: can't push to fedora08:03
zygawhy not?08:03
zygaENOPERM?08:03
zygaif so we should fix that08:04
zygabut I can help meantime08:04
mborzeckizyga: yeah, no permissions :P08:05
zygatrying to log in on https://bugzilla.redhat.com/show_bug.cgi?id=171673608:06
zygagets bounced via id.fedora*08:07
zygathen nothing08:07
zygastill not logged in08:07
zygaeh08:07
zygabrowser woes08:08
mborzeckizyga: you can mark it as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=170899108:09
mborzeckipstolowski: are you using the nested spread suite?08:12
pstolowskimborzecki: i used it for hotplug test, yes08:13
zygamborzecki: sure08:13
mborzeckipstolowski: that's good08:14
zygamborzecki: done08:14
pstolowskimborzecki: i haven't run it for a while; but i believe it's now run overnight (unless Sergio changed something)08:26
mborzeckipstolowski: hm no clue where those are started08:30
mborzeckipstolowski: there's no indication in travis.yaml or travis project page, probably sergio knows08:31
mborzeckigot a quick errand to run, back in 3008:33
zygamborzecki: https://github.com/snapcore/snapd/pull/6946 is green08:41
pstolowskimborzecki: i don't know how this is all wired up with spread-cron, but i found a run from 6h ago here: https://travis-ci.org/snapcore/spread-cron/builds/541031798, you can see nested stuff is run there08:43
* zyga moves somewhere quiet08:51
pstolowskipedronis: can https://github.com/snapcore/snapd/pull/6935 land (has +2 now) or do you want to take a look?09:01
pedronispstolowski: it's fine09:02
pstolowskity09:02
pstolowskipedronis: btw, load-state was too fast in my tests to get saved in the state and was not reported, likely due to small size09:04
mborzeckire09:11
* zyga is back home09:35
zygaapparently no good place for work today09:35
zygaETOOHOT ETOOLOUD09:35
ograogra@localhost:~$ sudo hostnamectl set-hostname pi209:36
ograCould not set property: Failed to set static hostname: Read-only file system09:36
ograogra@localhost:~$ ls /etc/writable/09:36
ograogra@localhost:~$ ls -l /etc/hostname09:36
ogralrwxrwxrwx 1 root root 17 Feb 11 14:50 /etc/hostname -> writable/hostname09:36
ogramvo, i thought that was fixed (core18)09:36
ogra(just downloaded from cdimage stable)09:36
mvoogra: can you please refresh core18? maybe the image is old?09:39
ograit did just auto-refresh it seems09:39
ogracreating writable/hostname works fine btw09:39
Chipacaogra: is there an /etc/writable ?09:39
ograChipaca, yes, but not the empty files09:40
ograjust touching hostname, localtime and timezone in the dir usually makes  it work09:40
ograoh, lovely ... and while i was just apt installing something under the classic snap, snapd decided to reboot in the middle of it09:45
* Chipaca hands ogra a wand of shutdown -c09:45
ograChipaca, hard to do if the console is occupied by apt output :P09:45
Chipacaogra: and 'tmux' is a classic snap :-(09:46
ogra(i know i can open a second ssh session but you have to be really fast since the shutdown happens so quickly in 18)09:46
* Chipaca goes for coffee09:51
* zyga is tired today09:53
zygasorry everyone, next year I'm getting A/C09:53
zygaI'll get that coffee09:53
zygabut first some paperwoor09:54
zygawork09:54
zygamvo: can you commit this suggestion please https://github.com/snapcore/snapd/pull/6945#discussion_r29022284510:09
mvozyga: already done10:10
zygathank you10:10
Chipacagrah, there's a bug in a test :-(10:18
* Chipaca grahs10:18
* Chipaca restarts the test because it's a racy bug but looks into fixing it anyway10:18
mborzeckilooks like i've hijacked all build slots on travis :)10:38
zygamborzecki: *used* :)10:39
zygait's good to keep things going10:39
* zyga iterates locally10:39
=== sparkieg` is now known as sparkiegeek
zygamy mind is melting today10:55
mborzeckizyga: well +30 in shade here, summer came early?10:56
zygamborzecki: I'm working from within an oven10:56
zygatried to work from a cafeteria but the insane requirement to keep ultra loud music drove me away10:57
zygaI really hate that brand owners force branded stores to play loud music all day long10:57
zygaI got a headache and left10:57
mborzeckizyga: should have played some vader on your laptop to counter the ambient noise :)10:58
zygathe staff hates it10:58
zygabut they cannot turn it down10:58
zygathey literally cannot I was told10:58
ogramvo, digging a bit deeper regarding the hostname/timezone/localtime stuff, i see the files in /snap/core/current/etc/writable/ and also only have one core installed yet ... seems the content of /etc/writable did not get copied for me11:09
ograogra@pi2:~$ grep etc/writable /etc/system-image/writable-paths11:12
ogra/etc/writable                           auto                    persistent  none        none11:12
ograhmm, shouldnt the second arg ne "transition" ?11:13
ograok ... here is core16:11:20
ograogra@stream:~$ grep /etc/writable /etc/system-image/writable-paths11:21
ogra/etc/writable                           auto                    synced      none        none11:21
ograwhy did we remove the "synced" between core 16 and 18 ?11:21
zygaogra: perhaps, I recall some changes there11:24
zygamvo: do you remember?11:24
zygaogra: the problem with writable paths is that it is non-obvious what happens when a file is removed11:25
ograwell, "persistent none" is definitely wrong11:25
zygaand that it's pretty niche so we don't all have experience with it11:25
ograwell, if persistent is set nothing happens if a file is removed from the writable area ... if synced is set the one from the core(18) snap will be copied in place11:26
ograhttp://manpages.ubuntu.com/manpages/xenial/man5/writable-paths.5.html is pretty clear about the bahaviour11:26
ogra*behaviour11:26
ograor do you mean removed from the readonly part (it will not have any effect, we had a userspace removal service for that on the phone to make sure the writable side gets removed as well if needed)11:28
ograin any case the current behaviour is wrong, the dir needs to be copied once on firstboot (whihc either "synced none" or "persistent transition" achieves", with synced you have a permaent connection between the two while "persistent transition" will only copy once on first boot)11:30
zygaogra: I'm deep in another place, sorry I cannot help with this now11:31
ograi didnt expect you to ... but i had the impression that was fixed a while (months ?) ago11:32
* pstolowski lunch11:32
mborzeckimvo: https://github.com/snapcore/snapd/pull/6946 landed11:38
=== ricab is now known as ricab|lunch
* zyga debugs stuff11:45
zygagnome, how I hate your interactions11:45
zygaI cannot wait for WSL211:45
zygaI will happily run WSL2 and windows terminal11:45
zygasorry gnome, you're full of great tech and you were surely made with love11:45
zygait's me, it's not you11:46
cachiomvo, hey11:46
mborzeckizyga: how about garden gnomes? :)11:47
zygamborzecki: same level of adoration11:47
cachiomvo, is it ok if I do some modifications to the autopkgtests change?11:54
zygacachio: I think so, what do you want to change?12:03
cachiozyga, move the test to the new nightly suite12:06
zygacachio: what are you aiming to achieve with that move?12:06
mborzeckioff to pick up the kids12:06
cachiozyga, it is mainly because the find of tests we have in the nightly suite12:09
cachiozyga, but we can do it in a following PR12:11
cachiozyga, I am running the test now to see how it works within the nested suite12:16
mvocachio: sure, what do you have in mind?12:27
pstolowskizyga: i failed to rework retry error handling, but come up with this https://github.com/snapcore/snapd/pull/6949 ; keen to see how travis likes it. local runs with network ns & spread tests on select machines were succesful12:28
mvomborzecki: thanks, cherry-picked12:29
mvoogra: hey, sorry for the delay. re "synced"> we don't have synced anymore as it was a source of confusion and issues, having transition there is fine if its missing of course12:30
mvoogra: let me look at the issue at hand12:31
ogramvo, yeah, we should definitely add transition to get the dir initially filled with files ... interestingly hostnamectl seems to behave differently on x86 (for lool ) and actually just creates the missing file (while it seems to try to modify /etc/hostname directly on my pi install)12:32
ograso there seems to be an additional issue that doesnt show up if the filesystem is set up as expected12:33
zygahmm12:33
zygamborzecki: do you know when you are in debug shell12:33
zygaand then you run a command like foo.sh12:33
zygawhen sh is a simple app that does exec /bin/sh "$@"12:33
zygaand then you end up in a loop with /bin/sh: 0: Bad substitution12:34
zygaand you cannot interrupt that12:34
zygais that that app running itself?12:34
zygaI bump into this once in a while12:34
zygaand it is infuriating when it happens12:34
zyga(I just lost all state I had in this machine)12:34
cachiomvo, as the test is failing at least on ubuntu-16.04 running on nested vm12:35
cachioI though on moving it to the nightly suite12:36
mvoogra: I created #131 against core1812:36
cachioand there run spread with autopackgtest backend12:36
mvocachio: meh, ok, I ran it only on 16.04 - what is the error?12:37
cachiomvo line 65: autopkgtest-buildvm-ubuntu-cloud: command not found12:37
mvocachio: aha, ok - I think I know what to do12:38
* zyga tries some more debugging12:38
mvocachio: nightly is an interessting idea, we can test on more distro release then too, right? lets talk about it in a bit, need to talk to zyga first about something unrelated12:39
cachiomvo, sure12:39
ogramvo, thanks !12:41
mvoogra: thanks for raising it12:46
zygamborzecki: it's curious, just running spread -shell-before and running "sh" kills the system13:05
zygachecking13:05
=== ricab|lunch is now known as ricab
zygamborzecki: it's PS113:15
zygamborzecki: it's not compatible with sh13:15
Eighth_Doctorzyga, mborzecki: so are we going to have a 2.39.2, or should I cherry-pick on top of 2.39.1?13:45
zygaEighth_Doctor: I would recommend cherry picking13:45
zygawe just discussed this13:45
zyga.2 would be for autopkg test13:45
zygathat's not really anything relevant13:45
Eighth_Doctor🤷‍♂️13:45
zygaand I think it's really better to cherry pick in both cases (maybe .2 is not needed at all)13:46
Eighth_DoctorI'll work on cherry-picking later today then13:46
mborzeckiEighth_Doctor: .1 also has a fix for symlinks in /etc/13:46
Eighth_Doctoryeah, I saw that13:46
zygaEighth_Doctor: thank you!14:02
zygaI'm suspending my desktop and going to look for shade in the bedroom; this place is at +40C now14:03
cachiomvo, hey14:03
Eighth_Doctorzyga: you should get air conditioning14:03
Eighth_DoctorI would have figured after last year, you would have bought an AC unit and set that up14:03
cachioso am also getting out of snpace issues runing autopkgtests on gce14:03
cachiomvo, perhaps the best is to remove the test from that PR, and I'll continue wotking on this on a following PR14:04
cachiootherise it is going to break the nested suite14:04
cachiootherwise14:05
cachiomvo, does it makes sense?14:05
cachiomvo, I'll need to create new images to support this new test14:06
ijohnsonhey folks, is it possible to get https://github.com/snapcore/snapd/pull/6805 pulled into a 2.39.x ? or do we need to wait until 2.40 hits stable?14:27
pstolowskizyga: #6949 is green; are you able to test it on a suse build infra?14:27
zygapstolowski: sure, let me try14:28
mvocachio: yes, I can do this - fwiw, I got the same error you got - testbed failure due to timeout, I think we need to run "autopkgtest --timeout-factor=3" or something14:29
mvocachio: I will remove the test now and force push and then we can do that in a separate PR14:29
mvoijohnson: hey, let me look14:29
mvoijohnson: that looks ok for 2.39.214:29
ijohnsonthanks mvo! do you want me to file a PR against release/2.39 for that branch?14:30
mvoijohnson: let me see if I can cherry pick cleanly, if so no need for a PR14:30
cachiomvo, nice, thanks14:31
mvoijohnson: cherry pick worked, so all good, its part of release/2.39 now14:31
ijohnsonmvo: thank you!14:31
cachiomvo, in the nightly suite is not giving that error but at some point the vm is going out of space14:31
cachiomvo and breaks14:32
mvoijohnson: thank *you* for the PR :)14:32
mvocachio: oh, woah14:32
mvocachio: the nested VM? or the "parent" VM?14:32
cachioI think the nested14:32
cachiomvo, https://paste.ubuntu.com/p/G9qhxwcn5W/14:33
cachioI mean the parent14:33
mvocachio: thanks and *meh* thats annyoing, I can see why, the nested vm will also need quite a bit of space :/14:34
mvocachio: I removed all but the "integrationtest" commit14:34
cachiomvo, I'll updates those images with more space14:34
mvocachio: thanks14:34
cachiomvo, nice14:34
mvocachio: I will run adt manually now on all our supported distro releases to unblock this PR14:35
cachionice14:35
cachiomvo, thanks14:35
zygapstolowski: building now14:36
pstolowskizyga: ty14:41
zygapstolowski: I need to rebase the patch, hold on14:45
pstolowskimborzecki: can you take a look at #6900 if you have a moment?14:48
mborzeckipstolowski: sure14:49
zygapstolowski: some typos in the patch, check your editor please14:49
zygapstolowski: rebased, not sure which other patches should be present14:51
zygatesting now14:51
* cachio lunch14:51
pstolowskizyga: typos?14:51
zygapstolowski: yes, commit message has several14:52
zygapstolowski: the  tests have failed:14:52
zygahttps://www.irccloud.com/pastebin/GlYDnpJR/14:52
zygapstolowski: also the 1st line is too long for git commit messages14:52
zygapstolowski: anyway, the essence  is different14:53
zygapstolowski: I pushed opensuse/2.39.1-patches14:53
zygathis is what  I tested14:53
zygaplease look at  the patches, I can nuke any easily14:53
pstolowskizyga: ah, but this is something else, i haven't seen such failure when running with net ns14:55
zygapstolowski: correct, I think inside obs there are  more things changed14:56
zygaspecifically the DNS server14:56
pstolowskiwonderful14:56
zygapstolowski: fun, right?14:56
zygapstolowski: if you want I can show you how to set up for local testing14:56
zygait's not complex actually14:56
pstolowskizyga: okay, i need some way of triggering this14:57
pstolowskizyga: need to got to vet now, bbiab14:58
zygapstolowski: k, ping me later please14:58
pstolowskizyga: back15:32
zygain a call now15:33
zygapstolowski: install tumbleweed in a vm15:33
zygapstolowski: maybe pick ext4 over btrfs15:33
zygapstolowski: that's all you'll need15:33
pstolowskik15:33
pstolowskizyga: ok, i've it nwo15:57
pstolowski*now15:57
zygapstolowski: you want to install whatever provides osc16:01
zygapstolowski: osc checkout system:snappy16:01
zygapstolowski: then osc build16:01
zygait's fairly well documented16:01
zygayou will likely need to create a suse account16:01
zygait's hand-holdy all the way so I think it should be smooth16:01
zygaonce you have that I can show you how to trigger the failure16:01
pstolowskik16:02
zygapstolowski: pstolowski: we can HO if you want to do  that interactively16:08
pstolowskizyga: i've a checkout now16:08
zygapstolowski: try osc build16:08
zygathat should  pass16:08
zygaif you comment-out some of the patches you can osc build again and see the failures happen16:09
pstolowskizyga: ok, got successfull build as you said, trying without patches now16:16
zygapstolowski: ok16:16
zygapstolowski: you only need to change two lines16:16
zygaone that says Patch: ... at the top of the file16:16
zygaand another one  slightly lower that says %patch3 ...16:17
zygapstolowski: using a git branch, like the one I sent, is convenient16:17
zygapstolowski: because you can then git format-patch 2.39.1..HEAD and simply copy them over16:17
zygapstolowski: you can also tweak the spec file, so when it runs tests it will run that test only, that's somewhere around %check16:18
pstolowskizyga: good hint with git branch, thanks16:18
pstolowskizyga: ok, got test failure now as expected. good16:19
zygapstolowski: ok, great!16:19
zygapstolowski: if you want I can  dig around to understand what exactly is going on when osc build runs16:19
zygapstolowski: I didn't look that deep yet16:19
pstolowskizyga: looks like a chroot in /var/tmp/build-root/openSUSE_Tumbleweed-x86_64 ?16:23
zygapstolowski: yep, likely16:23
* zyga jumps back at mount issues16:26
pstolowskiwill continue digging tomorrow16:29
=== pstolowski is now known as pstolowski|afk
zygasure, thank you pstolowski|afk16:33
cachiomvo, should we merge the autopkgtests PR now?16:36
mvocachio: I think so, we need a test too but that can be in a followup, I will do 2.39.2 with those fixes in my morning then16:40
cachiomvo, perfect16:41
ChipacaEOD here 👋17:23
Pharaoh_Atemzyga: 2.39.1 updates proposed18:21
Pharaoh_AtemF30: https://bodhi.fedoraproject.org/updates/FEDORA-2019-3c4eec825018:21
Pharaoh_AtemF29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-02e160af1618:21
Pharaoh_AtemEL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-91a080fd5518:21
* cachio afk19:05
zygaPharaoh_Atem: thank you19:26
Pharaoh_Atemzyga: I need testing and karma19:40
Pharaoh_Atemotherwise this update will not ship anytime soon19:40

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!