[00:48] <tinoco> .
[06:10] <cpaelzer> kstenerud: ping me for your build issues when you are arund
[06:11] <cpaelzer> kstenerud: they are odd to be sure, but we need to get them unblocked so give me a ping to talk about them
[07:23] <lordievader> Good morning
[07:45] <kstenerud> cpaelzer: ok. This build got stuck. I left it overnight and now it's a repeat of the other build
[07:46] <kstenerud> successful build yesterday with only i386 and amd64 enabled: https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-testing/+build/16508098
[07:46] <kstenerud> later build of the same thing (different version in changelog but no other changes), with all archs enabled: https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-support-new-libicu/+build/16508933
[07:48] <kstenerud> I don't know how the lp build system works internally, but from the log comparison, they're doing what looks like the same thing right up until that one test hangs
[07:48] <kstenerud> so something is interfering between parallell running tests?
[08:01] <cpaelzer> kstenerud: there is nothing interefering between the different arches
[08:02] <cpaelzer> kstenerud: they are on different VMs which are on different hosts - they only get together again when their results are collected
[08:02] <cpaelzer> kstenerud: I'd think it was a red herring that it worked when not having the other architectures enabled
[08:05] <kstenerud> OK, I've queued up exactly the same .changes file that failed yesterday, to a new ppa: https://launchpad.net/~kstenerud/+archive/ubuntu/support-new-icu-amdonly
[08:08] <cpaelzer> kstenerud: not for now but please do note that even in the "good" case there are 301 failed unexpectedly
[08:08] <cpaelzer> it seems not to gate on those tests (probably because they failed all the time)
[08:09] <cpaelzer> but then it is even more painful that a failing test now kills your build
[08:09] <cpaelzer> at some point those should be cleared to either work or be masked out from testing - and then those tests should be made gating IMHO
[08:09] <cpaelzer> maybe nacc has looked at that in the past and can share some thoughts on that with you kstenerud
[08:10] <cpaelzer> kstenerud: I see no tests at all in https://buildd.debian.org/status/fetch.php?pkg=php7.3&arch=all&ver=7.3.3-1&stamp=1551993939&raw=0 - do you happyn to know why?
[08:11] <kstenerud> no
[08:13] <cpaelzer> ah that was arch all
[08:14] <cpaelzer> kstenerud: it is failing there "to the same amount" https://buildd.debian.org/status/fetch.php?pkg=php7.3&arch=amd64&ver=7.3.3-1&stamp=1551996189&raw=0
[08:14] <cpaelzer> ok that aside, I don't know enough about it to discuss why it is that way
[08:14] <cpaelzer> kstenerud: but back to your case
[08:14] <cpaelzer> kstenerud: do you have a second failing build log please?
[08:16] <kstenerud> https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-fix-0050
[08:19] <kstenerud> With what I'm building now, it fails at test 3443. That other log gets well past that
[08:19] <cpaelzer> kstenerud: yep I've seen it
[08:20] <cpaelzer> kstenerud: the test is an intentional crash "Crash when file pointers passed to curl are closed before calling curl_multi_exec"
[08:20] <cpaelzer> kstenerud: my assumption is that there is a race that this kills the test runner instead of the epected test binary or so
[08:20] <cpaelzer> kstenerud: I'm reading through d/rules and the source how the tests work in general
[08:21] <cpaelzer> if you have 2.4k skipped tests and ignore 310 really failing ones we can as well eliminate this problematic single case and be good with it
[08:23] <kstenerud> yeah. Also that test isn't intentionally crashing. It's testing that PHP handles it by falling back to stdout
[08:24] <kstenerud> A test from 2009: https://bugs.php.net/bug.php?id=48203
[08:25] <kstenerud> The test itself looks pretty benign. Unless maybe the broken pipe is stalling the entire process?
[08:27] <cpaelzer> kstenerud: it would be nice to resolve this for real, but that might be better with your php7.3 work next cycle
[08:27] <cpaelzer> kstenerud: for now lets cut that test out and see if it makes the build reliable - as it works "sometimes" we at least know it is not generally failing
[08:27]  * cpaelzer sigh
[08:28] <cpaelzer> kstenerud: in d/rules they skip whole modules, but I'd not want to skip all other 227 tests of curl (even though sicne the build ignores failures the worth is to debate)
[08:28] <cpaelzer> kstenerud: instead you could just patch our the ext/curl/tests/bug48203_multi.phpt file
[08:28] <kstenerud> yeah, looks like it just blindly runs whatever's in the dir
[08:28] <kstenerud> so we'd have to patch it out
[08:28] <cpaelzer> exactly
[08:29] <cpaelzer> kstenerud: once you have done that throw it in three PPAs at once, if 3/3 build fine lets review and upload that before freezes hit us
[08:29] <kstenerud> ok
[08:29] <cpaelzer> kstenerud: for 7.3 I'd ask you to NOT retain this change on the Merge and have a look again there
[08:29] <kstenerud> It'll be a separate patch file that we can just drop
[08:30] <cpaelzer> yes
[08:30] <cpaelzer> kstenerud: mention that it is supposed to be dropped int he dep3 header please
[08:30] <kstenerud> ok
[08:33] <cpaelzer> nice, I filed a Debian bug and someone subscribed to the package has a https://bitbounce.com/?ref=bitbounce blocker set up
[08:33] <cpaelzer> 0.90$ to deliver the email, no thanks
[08:50] <kstenerud> Yup, the test hung again: https://launchpad.net/~kstenerud/+archive/ubuntu/support-new-icu-amdonly/+build/16511712
[08:50] <kstenerud> OK, starting 3 new builds with the test disabled...
[11:29] <rbasak> cpaelzer: I'm not sure if I pinged you about this previously. I'm just clearing out an old card left over from an old triage session. Bug 1817027 needs a response I think, because the reporter did what you said and changed the bug status back to New.
[11:30] <rbasak> But I'm not sure quite how to respond.
[11:35] <cpaelzer> rbasak: reading ...
[11:36] <rbasak> Sorry I should have followed up on this earlier
[11:36] <rbasak> No rush now though
[11:44] <cpaelzer> rbasak: I need a brain break from MIRs anyway I'll take a look
[11:44] <cpaelzer> also doko was so kind to suggest others might take over some MIRs, so I'm living in hope for ~3h that I don't have to do them all
[11:48] <cpaelzer> hi lagarcia, are you around enough that I might ask you some ppc kvm questions?
[12:37] <cpaelzer> rbasak: bug 1817027 is answered, it was straight forward debugging but ended up in too special samba-foo
[12:37] <cpaelzer> rbasak: but I think I got the case one step further and we can wait if the reporter is willing to move it to an upstream bug discussion
[12:38] <cpaelzer> ahasenack: ^^ if you want to take a look as well if it is a known case to you
[12:40] <ahasenack> doesn't ring a bell
[12:40] <ahasenack> samba 4.10.0 final was released today, btw
[12:40] <ahasenack> I need to update disco
[12:42] <ahasenack> strlen on a null pointer?
[12:42] <ahasenack> which happens to be a password
[12:55] <rbasak> cpaelzer: thank you for taking care of it!
[12:55] <rbasak> I closed my Trello card. If he responds it should come up in triage again
[13:00] <ahasenack> cpaelzer: I reopened the zeromq ftbfs mp, and with some questions at the end
[13:03] <cpaelzer> ahasenack: reading the zmq MP after I completed some tests on a qemu case I'm on
[13:04] <ahasenack> ok
[13:20] <kstenerud> woohoo! php7.2 works!!! MP submitted
[13:23] <cpaelzer> kstenerud: could you abandon the old MP pelase
[13:24] <kstenerud> sure
[13:25] <ahasenack> rbasak: samba 4.10.0 final is out, what should I call our version/tarball: 4.10.0+dfsg (what debian would call it), 4.10.0~dfsg (so we can "upgrade" to 4.10.0+dfsg when debian releases it?), or something else?
[13:26] <cpaelzer> ahasenack: last time you had a dfsk-ubuntu something version (I remember because I complained about it being so long)
[13:26] <cpaelzer> ahasenack: what became of that string for the tarball?
[13:26] <ahasenack> cpaelzer: yeah, I dropped that, it became 4.10.0~rc4+dfsg
[13:27] <ahasenack> I thought the risk of conflicting with debian was minimal at that stage, and I new a final release would still come
[13:27] <ahasenack> so I postponed that question to now :)
[13:27] <ahasenack> what will happen in LP when we create a 4.10.0+dfsg orig tarball, and debian creates one slightly different (diff md5), and launchpad tries to ingest it?
[13:28] <ahasenack> would it reject it, like it does when we upload a tarball with same name but different content?
[13:28] <cpaelzer> interesting thought, I don't want to find out as that smells like trouble
[13:29] <ahasenack> with 4.7.6, the last time we went ahead of debian, I used samba_4.7.6+dfsg~ubuntu.orig.tar.gz
[13:30] <cpaelzer> yeah why not that again
[13:30] <ahasenack> I assumed you wouldn't like the long name :)
[13:30] <cpaelzer> as I said, I dislike that it has ubuntu in the version twice then eventually
[13:31] <cpaelzer> but not knowing a better solution I'd at least stay consistent to last time then
[13:31] <ahasenack> what about ~dfsg? I haven't seen that before, I just made that up
[13:31] <ahasenack> or +dfsg~1
[13:32] <ahasenack> no ubuntu
[13:32] <cpaelzer> and shorter
[13:32] <cpaelzer> I don't know, we need people to chime in that did such things more often
[13:33] <ahasenack> hm
[13:33] <ahasenack> ii  ghostscript                                9.26~dfsg+0-0ubuntu0.18.04.7                 amd64        interpreter for the PostScript language and for PDF
[13:33]  * ahasenack checks that one
[13:34] <ahasenack> nah, came from debian already (the ~dfsg)
[13:41] <frickler> jamespage: coreycb: I'm still seeing messages in syslog like in this bug, not only for designate but also other services. the reason seems to be that the dh_systemd helper actually generates pretty messed up service definitions https://bugs.launchpad.net/charm-designate/+bug/1782644 http://paste.openstack.org/show/748022/
[14:20] <coreycb> frickler: is the package in need of permissions changes?
[14:21] <frickler> coreycb: no, this has nothing to do with permission IIUC, the "*Directory" stanzas in the systemd .service file are just bogus, see the paste
[14:27] <coreycb> frickler: ok yeah RuntimeDirectory doesn't look valid. it's possible that's an issue with openstack-pkg-tools.
[14:28] <frickler> coreycb: actually this is valid, but I had to look up the docs, too. arguments is "a whitespace-separated list of directory names" https://www.freedesktop.org/software/systemd/man/systemd.exec.html
[14:29] <frickler> also the service manager should create these dirs if they don't exist
[14:30] <frickler> so actually I'm back to not being sure yet what's wrong with it
[14:30] <coreycb> frickler: ok i'll target that at the designate package for now as well
[14:44] <frickler> coreycb: hmm, I'm now assuming that systemd on xenial is too old and doesn't support these features
[14:46] <lordcirth> frickler, coreycb Added RuntimeDirectory was added in systemd 211 https://github.com/systemd/systemd/blob/master/NEWS#L6157
[14:46] <lordcirth> My xenial container here is 229
[14:46] <lordcirth> However, more features were added to it later
[14:47] <frickler> lordcirth: thanks, I've been trying to find that. the version in xenial seems to support RuntimeDirectory, but only a single one, and also no CacheDirectory
[14:51] <frickler> the latter comes in with 235
[14:52] <lordcirth> frickler, indeed. What do you need cachedirectory for?
[14:54] <frickler> lordcirth: I don't necessarily need it, the systemd helper writes it into the service definition, causing errors in syslog on daemon-reload. this is for openstack packages, but probably others
[14:55] <jamespage> coreycb: flooding stein-proposed from stein-staging
[14:55] <lordcirth> frickler, ah, I see. This is a new system just for openstack? Why not run 18.04?
[14:55] <jamespage> coreycb: last few snapshots are working their way through - octavia and networking-ovn will block until we get octavia-lib
[14:55] <jamespage> I might wedge that into stein-staging for now
[14:55] <frickler> lordcirth: we are migrating to that, but UCA packages are intended to run fine on 16.04, too
[14:58] <frickler> jamespage: coreycb: actually the tool in question is pkgos-gen-systemd-unit from openstack-pkg-tools , so that's a pure openstack-internal problem probably
[14:59] <jamespage> frickler: so that issue is from xenial-queens?
[15:00]  * jamespage reads backscroll to catchup
[15:01] <frickler> jamespage: yes, the tools generates service definitions that would need systemd >=235, but xenial has 229
[17:11] <jamespage> coreycb: have you seen a tempest failure on tempest.api.identity.v3.test_domains.DefaultDomainTestJSON.test_default_domain_exists ?
[17:11] <jamespage> permission denied - I think its a change in tempest which requires a cloud level admin permission
[17:12] <coreycb> jamespage: yes but have not dug into it yet. i think admcleod may have.
[17:12] <coreycb> jamespage: i'll look today. i'm going to test rocky sru's.
[17:12] <jamespage> coreycb: okies
[17:12] <jamespage> coreycb: was doing some bionic/queens catchup for ceph
[17:13] <jamespage> coreycb: documented the failure here - https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1813807 but detailed it was unrelated to ceph
[17:13] <coreycb> jamespage: ok great
[17:13] <rbasak> ahasenack: I would just call it what Debian would call it.
[17:13] <ahasenack> rbasak: ok
[17:13] <rbasak> ahasenack: but probably should check with #ubuntu-devel or the security team if there's something else special done for that case
[17:19] <ahasenack> rbasak: emailed
[17:33] <rbasak> cpaelzer: did you get any guidance on your custom maintscript handling in https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/364760 from anywhere?
[17:33] <rbasak> Or should I just review from the perspective of it having been written from scratch?
[17:38] <ahasenack> rbasak: I got this ping from him earlier: <cpaelzer> andreas-lunch: check /usr/bin/dpkg-maintscript-helper sa it is copy and modified from there
[17:57] <rbasak> ahasenack: cpaelzer: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/364760 reviewed but it looks like I ended up grabbing the ~canonical-server review slot and can't create it again.
[17:57] <rbasak> (as I'm not a member of ~ubuntu-virt)
[17:58] <ahasenack> thanks for reviewing it
[17:58] <jamespage> coreycb: ok stein proposed is baking - I'll test first thing tomorrow
[17:59] <jamespage> I've shimmed in octavia-lib for now pending aa acceptance
[17:59] <coreycb> jamespage: great, sounds good
[21:58] <jerichowasahoax> Why am I getting "Fatal: Can't create directory /var/run/dovecot/login: Permission denied" even after running chmod -R 777 /var/run
[21:58] <jerichowasahoax> 18.04
[22:00] <sarnold> first things first, double-check that's a symlink to /run
[22:00] <jerichowasahoax> sarnold: yes
[22:00] <sarnold> next, set /run back to 755.
[22:01] <jerichowasahoax> sarnold: done. dovecot still does not start.
[22:02] <sarnold> alright, cool. what's /run/dovecot's owner:group and permissions?
[22:02] <jerichowasahoax> sarnold: dovecot:dovecot
[22:13] <sarnold> sigh. why don't more programs give instructions about the owner, group, and permissions?
[22:14] <sarnold> jerichowasahoax: what permissions does that have? I think it's supposed to be 755
[22:15] <jerichowasahoax> 755
[22:16] <sarnold> hmm. alright .. how about /run/dovecot/login ?
[22:16] <jerichowasahoax> sarnold: doesn't exist
[22:16] <sarnold> does ls -ln on these directories show the same numeric userid as ps auxwn ?
[22:17] <sarnold> .. or the numeric number that *would* be used if they started? heh
[22:19] <jerichowasahoax> sarnold: i'm *assuming* if a dovecot user exists then dovecot would be started as that user, given that this kind of work is generally handled by apt, yes?
[22:19] <jerichowasahoax> or, rather, by the package maintainer
[22:19] <jerichowasahoax> because i didn't compile this package by hand or anything
[22:21] <sarnold> jerichowasahoax: yeah, that's my hope too. but if there's two users with the same name and different numbers, you'll get errors that are nearly impossible to track down without ls -ln :)
[22:22] <jerichowasahoax> sarnold: grep dovecot /etc/passwd returns only one user
[22:23] <jerichowasahoax> sarnold: who's UID matches that listed by ls -ln
[22:32] <sarnold> jerichowasahoax: alright hmm
[22:32] <sarnold> jerichowasahoax: how about any DENIED messages in dmesg or auditd logs?
[22:34] <jerichowasahoax> sarnold: https://paste.ubuntu.com/p/RNZwRsNnSm/
[22:35] <sarnold> cuuuuuuuute
[22:36] <jerichowasahoax> sarnold: is this some new-to-bionic thing? this all happened after a dist-upgrade from xenial
[22:38] <sarnold> jerichowasahoax: do you have any /etc/apparmor.d/usr.sbin.dovecot*dpkg* files?
[22:39] <jerichowasahoax> sarnold: i have /etc/apparmor.d/usr.sbin.dovecot
[22:39] <sarnold> jerichowasahoax: here's the usr.sbin.dovecot profile that I grabbed from the bionic apparmor-profiles package: http://paste.ubuntu.com/p/mn82zMsMGg/
[22:39] <sarnold> it's got both the attach_disconnected flag that should handle the "Failed name lookup - disconnected path" messages
[22:40] <sarnold> and the /{,var/}run/dovecot/** rw,  that should have allowed the mkdir
[22:40] <jerichowasahoax> sarnold: existing file looks like nothing like the one you provided me, doing a complete replacement
[22:41] <jerichowasahoax> sarnold: dovecot still does not start, same error
[22:42] <sarnold> jerichowasahoax: be sure to copy over any local changes to /etc/apparmor.d/local/usr.sbin.dovecot
[22:42] <sarnold> jerichowasahoax: did you reload the profile?
[22:43] <jerichowasahoax> sarnold: why does it have to be in local
[22:43] <sarnold> jerichowasahoax: be sure to run apparmor_parser --replace /etc/apparmor.d/usr.sbin.dovecot
[22:43] <jerichowasahoax> i don't like apparmor
[22:43] <jerichowasahoax> can i disable it
[22:43] <sarnold> jerichowasahoax: if you edit this file in place, then it probably won't be upgraded in the future
[22:44] <sarnold> would you mind a half-way step of just apt-get  purge apparmor-profiles? those profiles are less well-tested than the others
[22:45] <jerichowasahoax> sarnold: apparmor-profiles was not installed
[22:45] <sarnold> jerichowasahoax: oh curious. that means the profile came from somewhere else, perhaps someone else on site?
[22:46] <jerichowasahoax> sarnold: only me on site, seems like a remnant from something else
[22:47] <jerichowasahoax> sarnold: apparmor stopped and disabled in systemd. dovecot still does not start.
[22:47] <jerichowasahoax> should i reformat
[22:47] <sarnold> no
[22:47] <jerichowasahoax> i think the dist-upgrade broke my bo\x
[22:47] <sarnold> apparmor_parser --remove /etc/apparmor.d/usr.sbin.dovecot may help
[22:48] <sarnold> it's possible but most of the rough edges on the upgrades should have been fixed by now
[22:49] <jerichowasahoax> sarnold: apparmor_parser [...] ran. dovecot starts.
[22:50] <jerichowasahoax> i'm kind of distrustful that apparmor profiles are enforced even after stopping apparmor in systemd, but maybe the "start/stop" verbs aren't being used in the way i expect them to be.
[22:52] <sarnold> indeed, 'stop' doesn't unload profiles, because 'start' can't confine already running processes