[08:33] morning gmb [08:33] Hi frankban [08:35] gmb: how did you know that I was going to create an MP for for bug 985338? ;-) [08:35] <_mup_> Bug #985338: lp.testing.tests.test_pgsql.TestPgTestSetupTuning.testOptimization fails intermittently in parallel tests with ProgrammingError < https://launchpad.net/bugs/985338 > [08:35] frankban, :D. Lucky oops-I-got-the-wrong-card guess. [08:36] frankban, Let me know when it's ready and I'll review it. [08:36] cool gmb [08:38] gmb: it's ready: https://code.launchpad.net/~frankban/launchpad/bug-985338-programming-error/+merge/102815 [08:39] frankban, Okay, will look presently. [08:41] frankban, Approved. [08:41] thanks gmb [08:54] frankban, Can you give https://code.launchpad.net/~gmb/launchpad/bug-984312/+merge/102812 a once-over please? (Raphael is your mentor, right?) [08:54] gmb: yes, but we will start the next month, he is now busy with Maas [08:55] frankban, Okay. Well, take a look anyway, and I'll self-review / mentor; I just want someone else to glance at it for me. [08:56] ok gmb, this instead is just a formality: https://code.launchpad.net/~frankban/launchpad/setuplxc-install-subunit/+merge/102817 [08:56] ok [08:57] Approved [09:05] gmb: your changed seem good to me [09:05] frankban, Thanks. [09:05] s/changed/changes [11:06] Oh, spectacular - connection problems a-go-go. [11:06] * gmb lunches, hopes they resolve themselves. [12:11] bac benji frankban gmb who is not here IIRC, call asap [12:12] I'm here... [13:02] I'm looking over the go tour for a few more minutes and then I'm going to figure out what to take up next. [13:15] benji, I'd like to work with you on the buildbot results card, if that's of interest to you. [13:16] we need to merge jml's branch to our subunit one [13:16] add that to our ppa [13:16] verify it all works [13:16] gary_poster: were those results of any value? [13:16] well, verify it works [13:16] (I'm not really here) [13:16] bac_, yes, did you see my reply to you (mistakenly) and then the list? [13:16] the results showed things are actually working as we'd hoped [13:17] No, haven't seen email this morn [13:17] gary_poster: sounds good [13:17] Great [13:17] thank you bac_ [13:17] benji, then we should modify the buidbot config [13:17] to do the stuff we've talked about [13:17] should be kinda fun, I think :-) [13:18] and probably pretty easy [13:18] yep [13:18] cool [13:18] gary_poster: let me make some coffee and I'll ping you when that's ready [13:19] benji, I also moved your "add worker ID tagging to testrepository" card to Done-done [13:19] because it has been merged [13:19] cool benji sounds good [13:19] k [13:19] I'll restart my system post-update during that time. biab [13:27] gary_poster: shall we? [13:27] benji, cool, I'm already in https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde [15:00] gary_poster: http://paste.ubuntu.com/938374/ [17:18] benji, how goes it? [17:19] gary_poster: I only have a few unread emails left. In other words I got distracted. :) [17:21] benji, heh [17:21] benji. I beat you on the emails. I'm going to try jono's branch really quickly... [17:21] maybe we should move the bug card back, if it works [17:23] gary_poster: I'm confused. I thought the leakage was fixed on the trunk. If not, what branch are you referring to? [17:23] specifically, lp:~jml/testtools/tag-leakage has been merged to trunk [17:23] benji, oh, cool! [17:23] oh no benji [17:23] this is the subunit thing [17:24] in jml's subunit branch [17:24] ah1 [17:24] not reviewed or merged [17:24] ! [17:24] so we need to merge his branch into our subunit/latest [17:24] benji, it works [17:24] for me [17:24] I'm merging now [17:24] to see if it works for me [17:26] benji, I undid the previous merge to latest and merged fresh [17:26] may have been a bad idea [17:26] but so afaik now we have to resolve the fact that the recipe run failed [17:27] benji, agree that I'll push card for bug 974617 back to tasks? [17:27] <_mup_> Bug #974617: test_operational_view_integration fails intermittently in parallel tests

< https://launchpad.net/bugs/974617 > [17:27] gary_poster: yep [17:28] I get the same bad behavior with the updated subunit. [17:29] benji, really! weird [17:29] I wonder what I'm doing wrong. [17:30] ah! I figured it out. I was accedentally running the system subunit, not the checkout. [17:30] it works for me now [17:30] cool [17:30] --without-tag works for me too [17:31] cat ~/Downloads/testrepo-0.txt | env PYTHONPATH=python ./filters/subunit-filter -s --with-tag='worker-1' --without-tag="zope:layer" | subunit-ls [17:31] that does exactly what I think we want [17:32] well, except for the occasional "Running in a subprocess." [17:33] gary_poster: you can use --no-passthrough and non-subunit bits will be elided [17:33] benji, not in my experience. [17:33] just tried this: [17:33] heh [17:34] $ cat ~/Downloads/testrepo-0.txt | env PYTHONPATH=python ./filters/subunit-filter -s --with-tag='worker-1' --without-tag="zope:layer" --no-passthrough | subunit-ls | grep 'subprocess' [17:34] 20 instances of "Running in a subprocess." [17:34] should we file a bug report and run away? [17:35] +1 for running away. I don't think it will bite us. Of course, it's not clear where the bug report should go. Is this subunit, for not sufficiently handling extra bits? Or or zope.testing fork, for not outputting the right stuff? [17:36] test: Running in a subprocess. [17:37] in fact: [17:37] test: Running in a subprocess. [17:37] time: 2012-04-19 21:43:50.769725Z [17:37] tags: zope:info_suboptimal worker-4 [17:37] successful: Running in a subprocess. [ multipart [17:37] ] [17:37] from the original output [17:37] (from testrepository) [17:38] maybe --without-tag zope:info_suboptimal is sufficient [17:39] This does the trick, benji: [17:39] cat ~/Downloads/testrepo-0.txt | env PYTHONPATH=python ./filters/subunit-filter -s --with-tag='worker-1' --without-tag="zope:layer" --without-tag="zope:info_suboptimal" --no-passthrough | subunit-ls [17:41] yeah the "test: Running in a subprocess." bit seems like a bug, that's not really a test. We do lie about tests a lot because subunit doesn't have a general "something bad happened" channel [17:41] exactly [17:42] benji, this is interesting, I think: http://pastebin.ubuntu.com/938615/ [17:42] counts of each worker [17:42] suspicious for a round-robin test assignment [17:43] aren't we now keeping info for better balancing in subsequent runs? [17:43] benji, yes, but bac's file name implied that it was the first run [17:43] no balancing info available [17:43] hmm, if so it is very suspicious [17:43] yeah [17:43] we should keep an eye on this :-/ [17:45] The last test in worker 3 is lp.testing.tests.test_zope_test_in_subprocess.TestZopeTestInSubProcess.test [17:45] fwiw [17:52] gary_poster: shall we consider our card unblocked and resume working on it? [17:53] benji, yes. This is another interesting "test" [17:53] test: replicated-development/launchpad.conf [17:53] time: 2012-04-19 21:40:58.711127Z [17:53] tags: worker-2 [17:53] successful: replicated-development/launchpad.conf [ multipart [17:53] ] [17:53] not sure what that means [17:53] heh, yeah; that doesn't make much sense to me [17:54] I was trying to find reasons why worker-2 would have more tests [17:56] or why worker 3 has so many fewer [17:56] benji > http://pastebin.ubuntu.com/938635/ ? [17:56] worker 3, yeah. one hypothesis is that the last test kills the process for some reason [17:57] that's why I noted the last test in worker 3, above [17:57] in case we see that again. [17:57] benji, I need to prepare [17:57] that's odd, but it does suggest a hypothesis: the list of tests were reasonably ballanced, but the number of non-test tests for each run differed (layers, and those other things you found) [17:57] me too, back in a minute [17:57] could be [18:15] I think I figured out the sigogglin test distribution. [18:16] (I just learned that new word a week or to ago. It's a beaut.) [18:17] * gary_poster looks it up [18:17] the numbers at http://pastebin.ubuntu.com/938615/ filter too much out, we don't want to remove tests as tagged in the zope layer or those with the other crazy tags [18:17] heh [18:17] ("sigogglin") [18:18] we do, however want to remove tests with *names* that end with :setUp or :tearDown [18:18] benji, I think/hope you are wrong [18:18] heh [18:18] why do you think so? [18:18] shall we hangout? [18:18] good idea [18:19] * gary_poster hangs [18:19] out [18:29] gary_poster: http://paste.ubuntu.com/938680/