[00:48] . === ykarel is now known as ykarel|afk === ykarel|afk is now known as ykarel [06:10] kstenerud: ping me for your build issues when you are arund [06:11] kstenerud: they are odd to be sure, but we need to get them unblocked so give me a ping to talk about them [07:23] Good morning [07:45] cpaelzer: ok. This build got stuck. I left it overnight and now it's a repeat of the other build [07:46] successful build yesterday with only i386 and amd64 enabled: https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-testing/+build/16508098 [07:46] later build of the same thing (different version in changelog but no other changes), with all archs enabled: https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-support-new-libicu/+build/16508933 [07:48] I don't know how the lp build system works internally, but from the log comparison, they're doing what looks like the same thing right up until that one test hangs [07:48] so something is interfering between parallell running tests? [08:01] kstenerud: there is nothing interefering between the different arches [08:02] kstenerud: they are on different VMs which are on different hosts - they only get together again when their results are collected [08:02] kstenerud: I'd think it was a red herring that it worked when not having the other architectures enabled [08:05] OK, I've queued up exactly the same .changes file that failed yesterday, to a new ppa: https://launchpad.net/~kstenerud/+archive/ubuntu/support-new-icu-amdonly [08:08] kstenerud: not for now but please do note that even in the "good" case there are 301 failed unexpectedly [08:08] it seems not to gate on those tests (probably because they failed all the time) [08:09] but then it is even more painful that a failing test now kills your build [08:09] at some point those should be cleared to either work or be masked out from testing - and then those tests should be made gating IMHO [08:09] maybe nacc has looked at that in the past and can share some thoughts on that with you kstenerud [08:10] kstenerud: I see no tests at all in https://buildd.debian.org/status/fetch.php?pkg=php7.3&arch=all&ver=7.3.3-1&stamp=1551993939&raw=0 - do you happyn to know why? [08:11] no [08:13] ah that was arch all [08:14] kstenerud: it is failing there "to the same amount" https://buildd.debian.org/status/fetch.php?pkg=php7.3&arch=amd64&ver=7.3.3-1&stamp=1551996189&raw=0 [08:14] ok that aside, I don't know enough about it to discuss why it is that way [08:14] kstenerud: but back to your case [08:14] kstenerud: do you have a second failing build log please? [08:16] https://launchpad.net/~kstenerud/+archive/ubuntu/disco-php7.2-fix-0050 [08:19] With what I'm building now, it fails at test 3443. That other log gets well past that [08:19] kstenerud: yep I've seen it [08:20] kstenerud: the test is an intentional crash "Crash when file pointers passed to curl are closed before calling curl_multi_exec" [08:20] kstenerud: my assumption is that there is a race that this kills the test runner instead of the epected test binary or so [08:20] kstenerud: I'm reading through d/rules and the source how the tests work in general [08:21] if you have 2.4k skipped tests and ignore 310 really failing ones we can as well eliminate this problematic single case and be good with it [08:23] yeah. Also that test isn't intentionally crashing. It's testing that PHP handles it by falling back to stdout [08:24] A test from 2009: https://bugs.php.net/bug.php?id=48203 [08:25] The test itself looks pretty benign. Unless maybe the broken pipe is stalling the entire process? [08:27] kstenerud: it would be nice to resolve this for real, but that might be better with your php7.3 work next cycle [08:27] kstenerud: for now lets cut that test out and see if it makes the build reliable - as it works "sometimes" we at least know it is not generally failing [08:27] * cpaelzer sigh [08:28] kstenerud: in d/rules they skip whole modules, but I'd not want to skip all other 227 tests of curl (even though sicne the build ignores failures the worth is to debate) [08:28] kstenerud: instead you could just patch our the ext/curl/tests/bug48203_multi.phpt file [08:28] yeah, looks like it just blindly runs whatever's in the dir [08:28] so we'd have to patch it out [08:28] exactly [08:29] kstenerud: once you have done that throw it in three PPAs at once, if 3/3 build fine lets review and upload that before freezes hit us [08:29] ok [08:29] kstenerud: for 7.3 I'd ask you to NOT retain this change on the Merge and have a look again there [08:29] It'll be a separate patch file that we can just drop [08:30] yes [08:30] kstenerud: mention that it is supposed to be dropped int he dep3 header please [08:30] ok [08:33] nice, I filed a Debian bug and someone subscribed to the package has a https://bitbounce.com/?ref=bitbounce blocker set up [08:33] 0.90$ to deliver the email, no thanks [08:50] Yup, the test hung again: https://launchpad.net/~kstenerud/+archive/ubuntu/support-new-icu-amdonly/+build/16511712 [08:50] OK, starting 3 new builds with the test disabled... === Wryhder is now known as Lucas_Gray [11:29] cpaelzer: I'm not sure if I pinged you about this previously. I'm just clearing out an old card left over from an old triage session. Bug 1817027 needs a response I think, because the reporter did what you said and changed the bug status back to New. [11:29] bug 1817027 in samba (Ubuntu) "samba crashes when uploading files" [Undecided,New] https://launchpad.net/bugs/1817027 [11:30] But I'm not sure quite how to respond. === Wryhder is now known as Lucas_Gray [11:35] rbasak: reading ... [11:36] Sorry I should have followed up on this earlier [11:36] No rush now though [11:44] rbasak: I need a brain break from MIRs anyway I'll take a look [11:44] also doko was so kind to suggest others might take over some MIRs, so I'm living in hope for ~3h that I don't have to do them all [11:48] hi lagarcia, are you around enough that I might ask you some ppc kvm questions? [12:37] rbasak: bug 1817027 is answered, it was straight forward debugging but ended up in too special samba-foo [12:37] bug 1817027 in samba (Ubuntu) "samba crashes when uploading files" [Undecided,New] https://launchpad.net/bugs/1817027 [12:37] rbasak: but I think I got the case one step further and we can wait if the reporter is willing to move it to an upstream bug discussion [12:38] ahasenack: ^^ if you want to take a look as well if it is a known case to you [12:40] doesn't ring a bell [12:40] samba 4.10.0 final was released today, btw [12:40] I need to update disco [12:42] strlen on a null pointer? [12:42] which happens to be a password [12:55] cpaelzer: thank you for taking care of it! [12:55] I closed my Trello card. If he responds it should come up in triage again [13:00] cpaelzer: I reopened the zeromq ftbfs mp, and with some questions at the end [13:03] ahasenack: reading the zmq MP after I completed some tests on a qemu case I'm on [13:04] ok [13:20] woohoo! php7.2 works!!! MP submitted [13:23] kstenerud: could you abandon the old MP pelase [13:24] sure [13:25] rbasak: samba 4.10.0 final is out, what should I call our version/tarball: 4.10.0+dfsg (what debian would call it), 4.10.0~dfsg (so we can "upgrade" to 4.10.0+dfsg when debian releases it?), or something else? [13:26] ahasenack: last time you had a dfsk-ubuntu something version (I remember because I complained about it being so long) [13:26] ahasenack: what became of that string for the tarball? [13:26] cpaelzer: yeah, I dropped that, it became 4.10.0~rc4+dfsg [13:27] I thought the risk of conflicting with debian was minimal at that stage, and I new a final release would still come [13:27] so I postponed that question to now :) [13:27] what will happen in LP when we create a 4.10.0+dfsg orig tarball, and debian creates one slightly different (diff md5), and launchpad tries to ingest it? [13:28] would it reject it, like it does when we upload a tarball with same name but different content? [13:28] interesting thought, I don't want to find out as that smells like trouble [13:29] with 4.7.6, the last time we went ahead of debian, I used samba_4.7.6+dfsg~ubuntu.orig.tar.gz [13:30] yeah why not that again [13:30] I assumed you wouldn't like the long name :) [13:30] as I said, I dislike that it has ubuntu in the version twice then eventually [13:31] but not knowing a better solution I'd at least stay consistent to last time then [13:31] what about ~dfsg? I haven't seen that before, I just made that up [13:31] or +dfsg~1 [13:32] no ubuntu [13:32] and shorter [13:32] I don't know, we need people to chime in that did such things more often [13:33] hm [13:33] ii ghostscript 9.26~dfsg+0-0ubuntu0.18.04.7 amd64 interpreter for the PostScript language and for PDF [13:33] * ahasenack checks that one [13:34] nah, came from debian already (the ~dfsg) [13:41] jamespage: coreycb: I'm still seeing messages in syslog like in this bug, not only for designate but also other services. the reason seems to be that the dh_systemd helper actually generates pretty messed up service definitions https://bugs.launchpad.net/charm-designate/+bug/1782644 http://paste.openstack.org/show/748022/ [13:41] Launchpad bug 1782644 in OpenStack Designate Charm "Charm blocked on designate services with Runtime directory is not valid reported in syslog" [Medium,Fix released] [14:20] frickler: is the package in need of permissions changes? [14:21] coreycb: no, this has nothing to do with permission IIUC, the "*Directory" stanzas in the systemd .service file are just bogus, see the paste [14:27] frickler: ok yeah RuntimeDirectory doesn't look valid. it's possible that's an issue with openstack-pkg-tools. [14:28] coreycb: actually this is valid, but I had to look up the docs, too. arguments is "a whitespace-separated list of directory names" https://www.freedesktop.org/software/systemd/man/systemd.exec.html [14:29] also the service manager should create these dirs if they don't exist [14:30] so actually I'm back to not being sure yet what's wrong with it [14:30] frickler: ok i'll target that at the designate package for now as well [14:44] coreycb: hmm, I'm now assuming that systemd on xenial is too old and doesn't support these features [14:46] frickler, coreycb Added RuntimeDirectory was added in systemd 211 https://github.com/systemd/systemd/blob/master/NEWS#L6157 [14:46] My xenial container here is 229 [14:46] However, more features were added to it later [14:47] lordcirth: thanks, I've been trying to find that. the version in xenial seems to support RuntimeDirectory, but only a single one, and also no CacheDirectory [14:51] the latter comes in with 235 [14:52] frickler, indeed. What do you need cachedirectory for? [14:54] lordcirth: I don't necessarily need it, the systemd helper writes it into the service definition, causing errors in syslog on daemon-reload. this is for openstack packages, but probably others [14:55] coreycb: flooding stein-proposed from stein-staging [14:55] frickler, ah, I see. This is a new system just for openstack? Why not run 18.04? [14:55] coreycb: last few snapshots are working their way through - octavia and networking-ovn will block until we get octavia-lib [14:55] I might wedge that into stein-staging for now [14:55] lordcirth: we are migrating to that, but UCA packages are intended to run fine on 16.04, too [14:58] jamespage: coreycb: actually the tool in question is pkgos-gen-systemd-unit from openstack-pkg-tools , so that's a pure openstack-internal problem probably [14:59] frickler: so that issue is from xenial-queens? [15:00] * jamespage reads backscroll to catchup [15:01] jamespage: yes, the tools generates service definitions that would need systemd >=235, but xenial has 229 === dpawlik is now known as dpawlik_ === lotuspsychje_ is now known as lotuspsychje [17:11] coreycb: have you seen a tempest failure on tempest.api.identity.v3.test_domains.DefaultDomainTestJSON.test_default_domain_exists ? [17:11] permission denied - I think its a change in tempest which requires a cloud level admin permission [17:12] jamespage: yes but have not dug into it yet. i think admcleod may have. [17:12] jamespage: i'll look today. i'm going to test rocky sru's. [17:12] coreycb: okies [17:12] coreycb: was doing some bionic/queens catchup for ceph [17:13] coreycb: documented the failure here - https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1813807 but detailed it was unrelated to ceph [17:13] jamespage: ok great [17:13] Launchpad bug 1813807 in Ubuntu Cloud Archive pike "[SRU] ceph 12.2.11" [Medium,Triaged] [17:13] ahasenack: I would just call it what Debian would call it. [17:13] rbasak: ok [17:13] ahasenack: but probably should check with #ubuntu-devel or the security team if there's something else special done for that case [17:19] rbasak: emailed [17:33] cpaelzer: did you get any guidance on your custom maintscript handling in https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/364760 from anywhere? [17:33] Or should I just review from the perspective of it having been written from scratch? [17:38] rbasak: I got this ping from him earlier: andreas-lunch: check /usr/bin/dpkg-maintscript-helper sa it is copy and modified from there [17:57] ahasenack: cpaelzer: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/364760 reviewed but it looks like I ended up grabbing the ~canonical-server review slot and can't create it again. [17:57] (as I'm not a member of ~ubuntu-virt) [17:58] thanks for reviewing it [17:58] coreycb: ok stein proposed is baking - I'll test first thing tomorrow [17:59] I've shimmed in octavia-lib for now pending aa acceptance [17:59] jamespage: great, sounds good [21:58] Why am I getting "Fatal: Can't create directory /var/run/dovecot/login: Permission denied" even after running chmod -R 777 /var/run [21:58] 18.04 [22:00] first things first, double-check that's a symlink to /run [22:00] sarnold: yes [22:00] next, set /run back to 755. [22:01] sarnold: done. dovecot still does not start. [22:02] alright, cool. what's /run/dovecot's owner:group and permissions? [22:02] sarnold: dovecot:dovecot [22:13] sigh. why don't more programs give instructions about the owner, group, and permissions? [22:14] jerichowasahoax: what permissions does that have? I think it's supposed to be 755 [22:15] 755 [22:16] hmm. alright .. how about /run/dovecot/login ? [22:16] sarnold: doesn't exist [22:16] does ls -ln on these directories show the same numeric userid as ps auxwn ? [22:17] .. or the numeric number that *would* be used if they started? heh [22:19] sarnold: i'm *assuming* if a dovecot user exists then dovecot would be started as that user, given that this kind of work is generally handled by apt, yes? [22:19] or, rather, by the package maintainer [22:19] because i didn't compile this package by hand or anything [22:21] jerichowasahoax: yeah, that's my hope too. but if there's two users with the same name and different numbers, you'll get errors that are nearly impossible to track down without ls -ln :) [22:22] sarnold: grep dovecot /etc/passwd returns only one user [22:23] sarnold: who's UID matches that listed by ls -ln [22:32] jerichowasahoax: alright hmm [22:32] jerichowasahoax: how about any DENIED messages in dmesg or auditd logs? [22:34] sarnold: https://paste.ubuntu.com/p/RNZwRsNnSm/ [22:35] cuuuuuuuute [22:36] sarnold: is this some new-to-bionic thing? this all happened after a dist-upgrade from xenial [22:38] jerichowasahoax: do you have any /etc/apparmor.d/usr.sbin.dovecot*dpkg* files? [22:39] sarnold: i have /etc/apparmor.d/usr.sbin.dovecot [22:39] jerichowasahoax: here's the usr.sbin.dovecot profile that I grabbed from the bionic apparmor-profiles package: http://paste.ubuntu.com/p/mn82zMsMGg/ [22:39] it's got both the attach_disconnected flag that should handle the "Failed name lookup - disconnected path" messages [22:40] and the /{,var/}run/dovecot/** rw, that should have allowed the mkdir [22:40] sarnold: existing file looks like nothing like the one you provided me, doing a complete replacement [22:41] sarnold: dovecot still does not start, same error [22:42] jerichowasahoax: be sure to copy over any local changes to /etc/apparmor.d/local/usr.sbin.dovecot [22:42] jerichowasahoax: did you reload the profile? [22:43] sarnold: why does it have to be in local [22:43] jerichowasahoax: be sure to run apparmor_parser --replace /etc/apparmor.d/usr.sbin.dovecot [22:43] i don't like apparmor [22:43] can i disable it [22:43] jerichowasahoax: if you edit this file in place, then it probably won't be upgraded in the future [22:44] would you mind a half-way step of just apt-get purge apparmor-profiles? those profiles are less well-tested than the others [22:45] sarnold: apparmor-profiles was not installed [22:45] jerichowasahoax: oh curious. that means the profile came from somewhere else, perhaps someone else on site? [22:46] sarnold: only me on site, seems like a remnant from something else [22:47] sarnold: apparmor stopped and disabled in systemd. dovecot still does not start. [22:47] should i reformat [22:47] no [22:47] i think the dist-upgrade broke my bo\x [22:47] apparmor_parser --remove /etc/apparmor.d/usr.sbin.dovecot may help [22:48] it's possible but most of the rough edges on the upgrades should have been fixed by now [22:49] sarnold: apparmor_parser [...] ran. dovecot starts. [22:50] i'm kind of distrustful that apparmor profiles are enforced even after stopping apparmor in systemd, but maybe the "start/stop" verbs aren't being used in the way i expect them to be. [22:52] indeed, 'stop' doesn't unload profiles, because 'start' can't confine already running processes