[00:00] doko (or anyone else): any chance of uploading the co-installable go stuff today? [00:01] ibm are suggesting i update the s390x patch [00:01] mwhudson, it's there for a few hours ... [00:01] oh! [00:02] i wonder why i didn't get bugmail [00:02] doko: thanks [00:02] not yet migrated [00:02] oh because the bug wasn't reported against the right source packages, because they didn't exist yet [00:03] doko: migrated 18 mins ago apparently... [00:03] slangasek: ah, debian has already dropped it as a dep for php-letodms-lucene [00:03] ah -defaults hasn't migrated yet [00:04] should in the next run though [00:05] slangasek: LP: #1560737 [00:05] Launchpad bug 1560737 in php-letodms-lucene (Ubuntu) "[FFe] Please sync php-letodms-lucene 1.1.1-1" [Undecided,New] https://launchpad.net/bugs/1560737 [00:08] jdstrand: so, bug #1020598 is interesting; we've been carrying a delta to iptables to drop ipq support since quantal because "the kernel no longer supports ip_queue", yet the Debian package builds libipq just fine against xenial (and unblocks the perlipq package hanging around in -proposed). Do you remember this? [00:08] bug 1020598 in iptables (Ubuntu Quantal) "iptables ftbfs due to ip_queue obsolete on 3.5 kernels" [High,Fix released] https://launchpad.net/bugs/1020598 [00:09] nacc: php-letodms-lucene has an Ubuntu delta which seems to still be relevant. Maybe needs to be a merge instead? [00:09] slangasek: ack, sorry! was looking at the wrong output [00:11] how often does britney run for xenial-proposed -> xenial-release? [00:14] slangasek: ah nm, no longer depends on zend-framework [00:15] nacc: ok [00:15] pitti, more locale issues: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/amd64/p/python3.5/20160322_210741@/log.gz [00:16] doko: probably fixed by infinity's upcoming glibc upload [00:17] doko: or at least, worked around - but the actual test failure in there doesn't look locale related? [00:17] Now running lintian... [00:17] Complex regular subexpression recursion limit (32766) exceeded at /usr/share/lintian/checks/cruft.pm line 939. [00:17] nacc: ^^ ugh, what is this package [00:17] (icinga-web) [00:19] slangasek, yes, test_cmath failing, but only in debug mode... will have to wait until tomorrow [00:19] slangasek: a fairly old framework for monitoring systems and networks [00:19] slangasek: that's the web interface part [00:20] i think there is a 2.x at this point, but not packaged anywhere [00:20] afaict [00:21] slangasek: err, the web interface isn't [00:21] there is a icinga2 package [00:21] yeah; and the icinga2 package has a wrong dep and is blocked in -proposed [00:22] slangasek: oh i see it now [00:22] mwhudson: britney runs every minute, sees that a lock is taken, and goes back into its hole for 4 more weeks of winter [00:22] slangasek: i see, this is a "how long is a piece of string" question then [00:24] slangasek: so on consideration, if we can fix up icingaweb2, we can just remove icinga-web, i think? [00:24] no rev-deps? [00:24] nacc: probably, if icinga2 is something you care about [00:32] slangasek: ok, patch for icingaweb2 posted in LP: #1544352 [00:32] Launchpad bug 1544352 in adminer (Ubuntu) "[PHP7] After bootstrapping, these PHP packages can be rebuilt" [Wishlist,New] https://launchpad.net/bugs/1544352 [00:33] stgraber: the lxd test on i386 on http://autopkgtest.ubuntu.com/running.shtml looks unhappy? [00:33] nacc: ok. is there a bug requesting removal of dh-make-php? [00:33] oh no, nm, it's going again [00:33] (thought it was stuck) [00:33] slangasek: i'll add it to LP : #1547183 if that's ok? [00:33] nacc: sure [00:37] nacc: re: icingaweb2, I will note that it seems suboptimal that Debian and Ubuntu have different spellings of 'zendframework'. it looks like there are half a dozen packages in Debian that depend on it. rather than needing to carry a delta for each of these, should we (eventually) sync up the zendframework package names? [00:38] slangasek: yeah it's quite annoying [00:39] we definitely should, i wonder why it happened [00:39] slangasek: i think they might be idfferent packages :/ [00:40] hmm, well, then this package that just got uploaded maybe won't dtrt ;) [00:40] well, i mean, different sources [00:40] they are the "same" but ubuntu has its own [00:40] i don't know why [00:40] sure, I think because someone in Ubuntu was interested in packaging it before it was available in Debian [00:40] yeah, could be, and now we're out of date :) [00:41] we don't have to get the packages in sync, but synchronizing on the package name would save us from having to carry delta on /other/ packages [00:41] we're older than oldstable at this point :) [00:41] agreed on the naming issue, though, i'll try to figure that out [00:46] golang-defaults has migrated too w00t [00:53] mwhudson: So, guessing we can remove golang, then? [00:53] infinity: yes [00:53] infinity: and golang-race-detector-runtime [00:54] infinity: want a bug or will you JFDI? [00:55] mwhudson: What produces golang-race-detector-runtime now? [00:55] infinity: golang-defaults [00:55] and it depends on the new golang-1.6-race-detector-runtime [00:55] Ta. Both removed. [00:55] cool [00:56] infinity: golang-go.tools while you're there? :) [00:56] that was replaced by golang-golang-x-tools aaages ago [00:57] (the latter packages makes transitional dep packages) [00:58] golang-golang-x-tools ... What a lovely redundant name. [00:58] (And removed) [00:59] mwhudson: Thanks for doing this. I really hope it makes future golang maintenance slightly less painful. [00:59] mwhudson: And we need to revisit trusty really soon, I guess. :/ [00:59] infinity: so do i! [00:59] infinity: slangasek put himself on the hook for that, the fool :-p [00:59] Hah. [00:59] Excellent. [00:59] One more thing off my plate. [01:00] To be fair, now that you have the versioned golang-1.6 in xenial, it should be pretty much a straight backport. Almost. [01:00] I'm guessing. [01:00] And then making $stuff in trusty build with the versioned paths. [01:01] oh yes that's a good point, we should backport the xenial golang-1.6, not the hacked up one i made [01:01] um [01:02] infinity: how do we do this and end up with a golang-1.6 package in trusty that has a lower version than the one in xenial? :) [01:02] I should clearly buy galangal.com, so I can have a package called golang-galangal-root-dev [01:02] mwhudson: add new changelog entry, copy previous version number, append ~14.04 [01:02] slangasek: but this process starts by copying-with-binaries from xenial to trusty [01:03] mwhudson: Russian nesting ppas [01:03] you can build-depend on packages in a ppa not your own [01:03] ah right [01:04] and then you can upload a version that has a lower version than the ppa you b-d on [01:04] yep [01:04] Yeah, should be nice and simple now. [01:04] Extra bonus to the defaults thing I forced on you. :P [01:05] jdstrand: ignore my previous inquiry; I've confirmed that the ipq support Debian added back to iptables by copying a kernel header is useless on modern kernels [01:06] infinity: heh the only forced thing was getting it done in time for x, i think it's a good idea :-) [01:06] mwhudson: s/forced/encouraged/ :) [01:13] slangasek: ok, i think i'm stopping with updates for the day ... only 19 done out of 400 or so, but ... something! [01:14] slangasek: Pharaoh_Atem: we need to figure out a plan for drupal [01:14] drupal7 is in the archive & in debian, but doesn't support php7 [01:14] drupal8 does, but isn't pacakged [01:14] * sarnold *shudders* [01:14] drupal should never have been packaged in the first place. [01:14] * nacc tends to agree [01:15] http://people.canonical.com/~ubuntu-security/cve/pkg/drupal7.html [01:15] * mwhudson checks that golang-1.6 doesn't need to breaks the golang in trusty [01:15] and was going to ask if we could just drop it [01:15] * Pharaoh_Atem agrees violently [01:15] mwhudson: I'd assume all the new versioned paths won't have any conflicts. But nice to be sure. [01:15] can we drop *all* the web apps? [01:15] infinity: yeah, i'd certainly hope not [01:15] but assume make a etc etc [01:16] mwhudson: I, too, spell assume "etcetc". [01:17] infinity: sarnold: ok, drupal7 is close to a leaf package, so in terms of that, should be easy to drop (along with drupal7-mod-libraries). Just one update to civicrm to not build a drupal7 module. [01:19] nacc: what about phpbb3 and wordpress? [01:19] the former doesn't even HAVE a version that supports php7, and wp is just a nightmare [01:20] My personal opinion is that no webapps should ever be packaged, but I'm not sure I should be making that call alone. :P [01:20] I've certainly been losing that war for years. === JanC_ is now known as JanC [01:21] Pharaoh_Atem: on my list to look at [01:21] at least owncloud isn't in the archive anymore [01:23] nacc: heh, the auto-sed of cacti results in debian/README.Debian making some interesting claims about suhosin support [01:24] well. not actually auto in this case [01:25] slangasek: argh, sorry! i had dropped that bit in my local tree [01:25] i think that whole paragraph is wrong [01:25] yep ;) [01:26] http://askubuntu.com/questions/298689/i-cant-get-apt-get-install-php5-suhosin-to-work [01:26] :) [01:26] slangasek: i am trying to review the pacakges at that level, which is making the process pretty slow [01:26] slangasek: i can send an updated debdiff for cacti tmrw with that removed [01:27] nacc: I'm not sure that level of review is required, tbh. I just noticed it in passing [01:31] ah now the game of "waiting for publisher runs" begins [02:03] slangasek: yeah, it may not be, but i wanted to minimize my own mistakes [02:22] suhosin is unnecessary these days [02:22] just use fpm and protect it with SELinux or AppArmor [02:25] what's the deal with all my PPAs using SHA-1, which is showing a warning as being weak when I update my computer.. [02:25] *computer... [02:25] tsimonq2: the switch was flipped to enforce SHA-2 hashes for signatures [02:25] tsimonq2: the fix is apparently in progress [02:26] well I've seen the blog post from the apt team, but the fix is in progress for PPAs? [02:26] yeah [02:26] if you add e.g. a new xenial release to your ppa the signatures ought to be fixed quickly [02:26] but they'll bulk-resign all ppas soon [02:27] soon being hours, days, weeks, months, years, what? [02:28] probably "days" [02:28] https://lists.ubuntu.com/archives/ubuntu-devel/2016-March/039287.html [02:29] okay, thank you sarnold :) [02:30] sarnold: It already hit, if you'd upload a new source package (Eg, make it re-publish the PPA.) === juliank_ is now known as juliank [02:30] Source: The bug report I don't remember the number. :3 [02:30] hehe [02:32] Just looking now to re-publish all of 'em. [02:43] you guys got an API function to republish a PPA or do you have to do it all by hand? [02:43] the latter would be painful [02:43] *very* painful [02:52] Do we have a Haskell guy? libghc-gnutls-dev 0.2-2 in Xenial ships /usr/lib/haskell-packages/ghc/lib/x86_64-linux-ghc-7.10.3/gnutls-0.2-BYccZTzDgCw38R5GwM0fsE/libHSgnutls-0.2-BYccZTzDgCw38R5GwM0fsE-ghc7.10.3.so which needs libgnutls-deb0.so.28 which doesn't exist in the archive. [02:53] Rebuilding src:haskell-gnutls gives me a libghc-gnutls-dev that uses libgnutls.so.30 instead. [02:53] I could just upload a no-change rebuild, but I wonder if this indicates that some wider check is needed. [02:54] rbasak: i've got a vague feeling infinity handled the last pile of haskell changes [02:55] Thanks. I've just found another with the same issue: libghc-network-protocol-xmpp-dev [02:55] So I think we need a bunch of rebuilds. [02:59] Where a bunch appears to equal two. === beisner- is now known as beisner [06:40] good morning [07:15] Good morning [07:15] good evening [07:21] infinity, doko: after the two latest responses in bug 1534263 I'm inclined to approve this; WDYT? [07:21] bug 1534263 in cmake (Ubuntu) "[FFe] [merge request] Import cmake-3.4 series to Ubuntu Xenial 16.04LTS" [High,Incomplete] https://launchpad.net/bugs/1534263 [07:52] pitti: [07:52] Setting up udev (229-3ubuntu1) ... [07:52] addgroup: The group `input' already exists as a system group. Exiting. [07:52] /var/lib/dpkg/info/udev.postinst: 109: [: Illegal number: * [07:55] infinity: yep, fixed in git already, bug 1560112 [07:55] bug 1560112 in systemd (Ubuntu) " /var/lib/dpkg/info/udev.postinst: 109: [: Illegal number: *" [Medium,Fix committed] https://launchpad.net/bugs/1560112 [07:56] pitti: Does that break anything? [07:56] but it's just cosmetical, thus no reason to panic-upload [07:56] pitti: Kay. [08:11] good morning [08:25] beta coming today? [08:25] :) [08:26] * asac considers to do the LTS->LTS upgrade test and wonders when is best time to be most useful :) [08:26] ah its tomorrow... nevermind [09:54] tsimonq2: We will not have to do it by hand. The thing I'm working on is some adjustments to the publishing scripts so that we can do a bulk run over everything without it taking all week. [09:54] Still, thanks for the fix so far. [10:05] infinity, seen on amd64: https://launchpadlibrarian.net/249606658/buildlog_ubuntu-xenial-amd64.python3.5_3.5.1-8_BUILDING.txt.gz [10:06] (search for: final link failed) [10:19] ok, this complaint seems to be valid, _math.o is built without -fPIC [11:08] slangasek, upstream plymouth has proper hidpi support in git now (but no release as yet) do we want to cherry-pick the patches for 16.04? ricotz has test packages at https://launchpad.net/~ricotz/+archive/ubuntu/staging/+packages?field.name_filter=&field.status_filter=published&field.series_filter=xenial [11:09] hi folks, can anybody please explain this? https://launchpad.net/ubuntu/+source/fpc/3.0.0+dfsg-4/+build/9390047 [11:10] Setting up fp-compiler-3.0.0 (3.0.0+dfsg-2) ... [11:10] An unhandled exception occurred at $0FE502D0: [11:10] EAccessViolation: [11:10] $0FE502D0 [11:10] $0FE5E684 [11:10] $0FE152A8 [11:10] $0FE1537C [11:11] the build doesn't even start [11:12] LocutusOfBorg, its failing before that though, [11:12] dpkg: dependency problems prevent configuration of sbuild-build-depends-fpc-dummy: [11:12] sbuild-build-depends-fpc-dummy depends on fp-compiler; however: [11:12] Package fp-compiler is not configured yet. [11:12] Package fp-compiler-3.0.0 which provides fp-compiler is not configured yet. [11:13] pitti: hi! can you take a look at the comment I've added in https://bugs.freedesktop.org/show_bug.cgi?id=85477 please? [11:13] Freedesktop bug 85477 in operations "Creating new partition and filesystem sometimes only creates the partition" [Normal,New] [11:14] darkxst, that part "Setting up fp-compiler-3.0.0" is before what your wrote :) [11:15] something really bad is happening on boost / ppc64el [11:15] https://launchpad.net/ubuntu/+source/clblas/2.10-2/+build/9384486 [11:18] LocutusOfBorg, oh yes seems I skipped to the second error [11:18] :) [11:18] seems some sort of apt/dpkg/chroot/sbuild issue? [11:20] how would that be an sbuild issue? /usr/include/boost/math/policies/policy.hpp:818:18: error: ‘__float128’ was not declared in this scope [11:20] LocutusOfBorg, seems like an arch issue to me [11:25] Huh. I wasn’t expecting “sudo usermod -l 😌 testaccount” to work, but it does [11:27] cjwatson, sbuild issue is fpc [11:28] boost issue clblas [11:28] :) [11:28] sure, but you weren't desperately specific in the one I was replying to [11:28] yes, sorry [11:28] anyway, the fpc thing is also not an apt/dpkg/chroot/sbuild issue; the build-depended-on fpc itself is apparently blowing up [11:33] I don't understand sorry :) [11:38] LocutusOfBorg: dpkg tries to install fp-compiler-3.0.0; its postinst fails with the output you quoted. this is not the fault of dpkg or any layer above it, but the fault of whatever the postinst is calling that's exploding, which is probably some bit of fp-compiler-3.0.0 itself. [11:40] and only on ppc64el :( [11:40] I have no porterstuff on ubuntu, and Debian is fine [11:40] LocutusOfBorg: is that update-alternatives? [11:40] nice and trivial to debug :( [11:42] LocutusOfBorg: i have a VM at OpenPower, lemme try... [11:42] that would be *awesome* [11:43] https://sources.debian.net/src/fpc/3.0.0%2Bdfsg-4/debian/fp-compiler.postinst.in/ [11:43] this seems to the failing script? [11:46] the failing build you quoted is powerpc, not ppc64el [11:47] my guess would be that the failing thing is something itself implemented in Pascal, so probably fpcmkcfg [11:47] but that's a guess [11:47] the boost stuff is failing on ppc64el [11:47] the fpc on powerpc :) [11:47] see "not desperately specific" above [11:47] aargh, i'm installing fpc on ppc64el :( [11:48] probably my bad connection is messing up the messages [11:48] LocutusOfBorg: try to break one thing at a time === alvesadrian is now known as adrian [11:55] ginggs, I'm leaving because of bad connection, please upload/fix/send me a mail or whatever if you have updates <3 [12:13] hmm rebuilding clblas 2.10-1 also fails with ‘__float128’ was not declared in this scope on ppc64el [12:23] ginggs, can you check if this is triggered by glibc 2.23? [12:32] doko: sure, i'll have a look === _salem is now known as salem_ [12:35] doko: i don't think so, clblas FTBFS on 2016-03-11 already http://people.ubuntuwire.org/~wgrant/rebuild-ftbfs-test/test-rebuild-20160226-xenial.html [12:54] update_excuses.html says "libmysqlclient-dev/amd64 unsatisfiable Depends: libmysqlclient20 (= 5.7.11-0ubuntu2)" [12:54] But libmysqlclient20 5.7.11-0ubuntu2 is in xenial-proposed for amd64. [12:54] Some help please? [12:54] rbasak: check components [12:55] Ah [12:55] That'll be it, thanks. [12:55] slangasek: mind moving libmysqlclient20 and mysql-{client,server}-5.7 to main, please? [12:56] They will replace libmysqlclient18 and 5.6 when we're done. [13:00] rbasak: done [13:00] Thank you! [13:23] LocutusOfBorg, cjwatson: same access violation when installing fp-compiler on real powerpc hardware [13:27] and just running 'fpcmkcfg-3.0.0' also [13:38] mvo, https://launchpad.net/ubuntu/+source/golang-github-mvo5-goconfigparser/0.2-0ubuntu5/+build/9322535 [13:40] doko: thanks, I have a look [13:46] mvo: morning! so I see that ubuntu-snappy FTBFS on powerpc with a test timeout; do you know if it's likely to build on a retry? [13:50] slangasek: good morning! I doubt it, it seems to be a bug in the gccgo, I will probably hvae to disable the test [13:51] mvo: is it the test that's broken or the code under test? should I just delete the old powerpc binaries instead? [13:51] slangasek: its hard to say, but if we could remove powerpc for now, that would be fine with me as well, its the only problematic arch [13:56] slangasek: sorry, I missed mysql-client-core-5.7. Did you notice mysql-server-core-5.7 since that seems to be OK? [13:57] rbasak: I noticed mysql-server-core, yeah, and missed the other. fixing now [13:57] Thanks [13:57] Skuggen: FYI ^^ === lynxman_ is now known as lynxman [14:25] mvo: oops, ubuntu-snappy did build successfully on powerpc after a give-back; sorry ;) [14:28] rbasak: is 5.7 intended to replace 5.6 for beta? [14:33] pitti: 2 users reporting udev 229-3ubuntu1 failing postinst with "/var/lib/dpkg/info/udev.postinst: 109: [: Illegal number: * " " [14:35] slangasek: does Ubuntu not support UsrMerge? [14:36] I distinctly remember that it was something Ubuntu looked into in the quantal days, but I don’t see any indications of it being supported [14:36] TJ-: already fixed in git, it's just cosmetical [14:36] TJ-: the postinst doesn't fail, it just spits out this warning [14:37] pitti: ahhh, OK :) couldn't find a bug report for it [14:37] TJ-: bug 1560112 [14:37] bug 1560112 in systemd (Ubuntu) " /var/lib/dpkg/info/udev.postinst: 109: [: Illegal number: *" [Medium,Fix committed] https://launchpad.net/bugs/1560112 [14:37] pitti: strange; the launchpad search on the udev package didn't show that [14:38] TJ-: the source package is systemd [14:38] udev has been built from that since, hmm, looong ago [14:39] pitti: oh, of course! I was caught out because there are some very recent 16.04 reports on the udev package [14:44] slangasek: heh, nice [14:49] superm1: replace yes, not sure about beta. [14:49] Which is a point. I wasn't expecting it to hit the archive this late. Am I OK to proceed with the transition during the beta freeze? [14:49] rbasak: ok i'm hoping not by beta because mythbuntu has some hardcoded stuff in the ubiquity plugin to configure 5.6 mysql [14:50] it's not a problem to change it to 5.7 other than it's a timing thing to make sure they both land in the right ISO image [14:51] superm1: noted. I tagged block-proposed in bug 1528583 to ensure we hold off until I understand what we should do. [14:51] bug 1528583 in mysql-5.7 (Ubuntu) "[FFe] Please update to MySQL 5.7 series" [Wishlist,In progress] https://launchpad.net/bugs/1528583 [14:52] rbasak: ok thanks. after beta no problem for me to adjust our code [14:54] Skuggen: FYI ^^ [14:54] infinity: ^^ thoughts? Should we hold off? [14:55] I guess at this point the only way to hold it off is by blocking it in britney [14:56] if not already too late [14:56] rbasak: can you open a proposed-migration blocking bug? [14:56] slangasek: already done [14:56] ok [14:56] I think it's in time because the dep8 tests take quite a while. [14:56] oh, also freeze, which means it needs hinting anyway, so good [14:57] errr except it wasn't previously seeded so it's not in the freeze hint, oops ;) [14:57] so blocking bug is it [14:57] I used the "update to 5.7" bug but moved it to the new src:mysql-5.7 first. So I presume it'll work. [14:58] should do, yes [15:01] rbasak: Ok [15:13] nacc: so… I sorta/kinda cajoled Remi into porting libvirt-php to work on php7: https://gitlab.com/Conan_Kudo/libvirt-php7/commits/php7 [15:13] I’m going to try to see if I can get it upstreamed [15:13] I really, really, really want to get it upstreamed [15:14] I’m not a huge fan of maintaining a fork like this [15:28] Son_Goku: cool, also on my list to look at === sergiusens_ is now known as sergiusens [16:01] cjwatson: so fpc 3.0.0+dfsg-2 has become uninstallable on powerpc. I installed 2.6.4+dfsg-8 from wily and tried building 3.0.0+dfsg-4. It got to 99% and then hit an access violation [16:02] [ 99%] Compiled package zorba [16:02] An unhandled exception occurred at $0FE502D0: [16:02] EAccessViolation: [16:02] $0FE502D0 [16:02] $0FE5E684 [16:02] $0FE152A8 [16:02] $0FE1537C [16:03] Son_Goku: no. UsrMerge is upside-down from the way a merged /usr should be done [16:03] slangasek: how so? [16:03] things should be merged into /, not into /usr [16:03] why [16:03] err, no [16:05] slangasek: merging things into / sounds like it’d make things harder, not easier [16:07] ginggs: I'm not going to be able to help further with this, I was just pointing in the right general direction to start with [16:08] cjwatson: ok, i pinged you because i recall you have bootstrapped fpc in the past [16:09] ginggs: I can help with rebootstrapping given specific directions if that's what it needs, but not with the investigation along the way [16:09] I probably just did something mechanical based on Debian's binaries, if it was me [16:32] bdmurray, can you reject the wily nova upload that is in the queue? I am going to combine it with another sru. [16:33] coreycb: done [16:33] bdmurray, thanks [16:41] slangasek: do you happen to know about watershed? [16:42] bdmurray, can you also reject the python-oslo.messaging upload to wily dated 2016-03-18? [16:43] pitti: sure [16:44] slangasek: if I run "/lib/udev/watershed sleep 5" n times, are the other n-1 instances actually supposed to hang until the first one finishes? [16:44] pitti: yes [16:44] I wonder if that's a regression, or I just misunderstand how watershed works [16:44] watershed serializes, but doesn't skip the calls [16:44] ah, then it's rather pointless to use watershed in an udev rule [16:44] well [16:45] the purpose is to avoid multiple calls to a thing running in parallel that shouldn't [16:45] slangasek: thanks, so I guess we'll either need to change watershed (it's only being used by that lvm rule and apport-noui) or find something else that's non-blocking [16:45] but we can't just /drop/ the extra calls, because of races [16:45] why non-blocking? because of udev rule timeouts? [16:46] that too, but blocking an udev rule blocks the worker, and there's only finitely many of them [16:46] so in that bug 1560710 with 50 LVs this leads to a deadock [16:46] bug 1560710 in udev (Ubuntu) "ISST-SAN:KVM:R3-0:Unable to get the login prompt after reinstallation of Ubuntu16.04" [Undecided,New] https://launchpad.net/bugs/1560710 [16:46] right [16:46] ok, I'll think about that, thanks for confirming that it's not a regression in watershed itself [16:47] but are there expectations elsewhere in the system that the rule be synchronous? [16:47] How did I know that was going to be an IBM bug as soon as you said "50 LVs" [16:47] yeah, it's not a regression, it's just IBM testing at scale [16:47] ;) [16:47] slangasek: not really -- the only point is to run vgchange after the last pending LVM event got processed [16:47] ok [16:47] cjwatson: good :) [16:48] that can happen asynchronously (and it might cause more uevents due to the new block devices) [16:48] slangasek: at scale> yeah, not even smb managed to hit that limit :) [16:48] pitti: so if the actual calls to vgscan/vgchange need to be serialized, maybe watershed itself should just be backgrounded in the rule? [16:49] pitti: In IBM's world, this isn't "at scale", this is "normal". [16:49] no, you can't background stuff from an udev rule [16:49] :) [16:49] pitti, I would not be sure :-P [16:49] it'll get killed right after the main process exits (to avoid zombies) [16:49] Even our teeny tiny POWER8 machines from them are insanely overspecced. [16:49] pitti: wellll that what would it matter if watershed "hung"? or did you just not understand the purpose of watershed? [16:49] s/that/then/ [16:50] slangasek: obviously; I thought it woudl detect if the same command was already running, just re-inc the "refcount" of the first instance, and quit itself immediately [16:50] but apparently it's not doing that [16:50] pitti: ah. can't do that, because not guaranteed that the running command picked up the latest events [16:50] that's the race I meant === kickinz1 is now known as kickinz1|eod [16:51] slangasek: right, every further call would need to update some stamp file/thingy somewhere which resets the first instance's idea of "after" [16:51] pitti: now, maybe what watershed should actually do is detect when there are *two* other watersheds running, and quietly exit any others [16:52] slangasek: hm, but how is that guaranteeing that the first or second one defers the re-run enough for the n-th invocation? [16:52] because if you have 1 running, and 5 queued, when the first one exits you only need 1 of the other 5 to run to catch up [16:52] not all 5 of them back-to-back [16:52] right, but the second one is just going to wait for #1, not for #17, no? [16:52] and actually this would also address the boot-time load bugs we saw with apport-noui [16:53] well, I don't know how watershed detects duplicates of itself, maybe that already works [16:53] it doesn't work [16:53] I know for sure ;) [16:53] so, somewhere in teh system we need some kind of lock/stamp file with the command [16:53] and duplicate invocations need to touch that [16:54] but if we made it do so, it would improve the scaling of watershed with AFAIK no downsides [16:54] No need for lock files. [16:54] and some running instance needs to defer the second run until after the current timestamp [16:54] having N-2 of them exit may only work when the command that's called is idempotent [16:54] sarnold: it is [16:54] it's always exactly the same "/sbin/lvm vgscan; /sbin/lvm vgchange -a y" [16:55] the point is just to run it *after* the last pending LVM uevent [16:55] and avoid running multiple copies while a whole bunch of them is coming in in parallle [16:55] well, this is an ugly hack which we've been carrying for a decade, maybe this is all moot with modern versions, not sure [16:58] infinity: well, not necessarily lock files, but the watersheds must communicate with each other somehow if we don't want all of them to stick around indefinitely? [16:58] pitti: Instead of lockfiles, I'd think all it needs is a SIGUSR* handler. New invocation of watershed sends signal to all running watersheds. If running watershed is in "wait" state and /proc//cmdline matches our own, exit immediately. [17:00] Once we transition from wait to running, just ignore SIGUSR and carry on to a normal exit. [17:00] bdmurray, thanks. if you or someone else on the sru team has some time we could use reviews of nova, python-oslo.messaging, and keystone in the wily queue. [17:01] slangasek who should I talk to about auto package test SRU failures http://people.canonical.com/~ubuntu-archive/pending-sru.html ?? The failures for coreutils and initramfs-tools all look to be infrastructure issues. [17:03] infinity: nice idea; it currently communicates via /run/watershed/ files, but comparing /proc/pid/cmdline sounds easier indeed [17:04] does ibm have any systems where 64k pids aren't enough? :) it might not be fun to find all watershed processes if there's a few hundred thousand or million processes running [17:05] well, hopefully with a change like this they would die much quicker [17:05] and thus there should only be at most one long-running one around [17:05] sorry, I meant if the system has a few hundred thousand -other- processes running [17:05] oh, that's right [17:05] say if someone goes nuts and runs a few hundred thousand ubuntu lxds [17:05]