[12:08] bac benji frankban gmb call in 2 [12:09] ok [12:22] gary_poster: i'd like to see the rabbit timeout failure logs if you still have them. [12:35] * gmb -> lunch [12:41] bac, http://pastebin.ubuntu.com/992273/ [12:41] for me, an interesting aspect of watching _Portlandia_ is imagining our Portland-based colleagues interacting with them. [12:41] heh [12:45] gary_poster: that log is interesting in that it shows it used the new rabbitfixture and the timeout loop worked properly. given that, perhaps we do need to try upping the timeout, though 15 seconds is pretty long already. [12:46] gary_poster: and next time it may be worth looking at the rabbit log file mentioned in the exception, if the machine doesn't get torn down [12:46] bac, ok cool. For reference, on eight core concurrency, 30 seconds is usually enough to start up all the lxc containers and 60 is always enough; with 32 cores, 240 was the minimum [12:46] and this failure was on 32 cores [12:46] sorry, not 240, 160 was the minimum [12:47] so more than 5 * [12:49] gary_poster: so perhaps we make an experimental version with absurdly long startup timeout and log the actual start up time so we can see how this one service is impacted by number of cores [12:52] ok bac. sounds like a plan. I wish getting the logs were easier. :-/ [12:56] frankban, lpbuildbot approved. looks good, thank you [12:57] thanks gary_poster [12:57] welcome [13:00] gary_poster: https://dev.launchpad.net/yellow/Termbeamer [13:00] I did the basics and added a "Motivation" section mostly cribbed from the email I sent about the old way of sharing terminals [13:21] benji, looks great [13:41] frankban, sorry, late. joining. [14:12] gmb, you around? [14:23] would someone like to review https://code.launchpad.net/~bac/lpbuildbot/cleanup-testr/+merge/106197 -- to add the functionality to delete old testr data ? [14:31] gmb yoohoo [14:31] gary_poster: Yep, around. [14:32] benji's ping issue is infefectious. [14:32] heh [14:32] cool gmb. frankban is looking for a next task and I thought pairing with you would be wunderbar [14:32] we have our call now [14:32] would it be worth talking with him to introduce the problem [14:32] so he can investigate a bit [14:32] and then we can call [14:32] and then you can rejoin? [14:32] gmb, note that frankban will not be here tomorrow [14:33] bac, if no one has taken it by 11 or 11:30 I'll be able to [14:33] and you guys can collaborate remotely via terminal , [14:33] :-) yeah that too [14:33] gary_poster: That works for me; I can pick something else up tomorrow or make what headway I can or... something. [14:33] gmb, cool. Your manager [14:33] 's pipedream is that you two identify the problem today [14:34] and then you have a chance to code it up tomorrow :-) [14:34] and look at that caterpiller smoking the hookah outside! [14:34] Oooookay. [14:34] pipedream? crazy? you know? oh ok. :-P [14:35] gary_poster: So, should frankban and I talk and then you and I can talk when I'm free, or do you want in on this, since you have more context and I've only given it a tentative prodding? [14:35] eh, I wasn't intending to participate, but sure, I'll join the party [14:36] gmb, frankban https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde ? [14:37] gary_poster: joining [14:37] cool [15:07] frankban: So, I'm going to grab coffee and then I'll be right with you. [15:07] ok gmb [15:08] * benji reboots [15:32] frankban, I'm in https://plus.google.com/hangouts/_/extras/zope.testing.die.in.a.fire whenever you're ready. [16:28] gmb: http://pastebin.ubuntu.com/992640/ [16:28] frankban, Thanks. [17:03] benji, hey. So, I'm trying to get my head around the implications of your most recent results for bug 992184 [17:03] <_mup_> Bug #992184: lib/lp/services/database/doc/textsearching.txt fails intermittently/rarely on parallel tests < https://launchpad.net/bugs/992184 > [17:04] - previous tests are irrelevant [17:04] - all code other than the python call to nl_term_candidates is irrelevant? [17:04] gary_poster: grr/yay! by beep bug is back again, now I can test my latest theory [17:04] heh [17:05] gary_poster: I believe both points are correct. Another one: there are other calls to nl_term_candidates in that test, but this one is the only one that ever fails [17:05] (I need to write that in the bug.) [17:05] huh [17:07] benji, can you dupe in bin/harness alone? [17:07] * benji tries to remember what bin/harness is. [17:07] benji,it's like bin/py except the db is initialized [17:08] I think :-P [17:08] I'll try to repro with that. [17:08] cool [17:15] gary_poster: not so much, utility lookup error [17:15] benji, which utility? [17:15] it executes zcml before starting AFAICT [17:16] gary_poster: lp.services.webapp.interfaces.IStoreSelector [17:16] benji, so you mean bin/harness didn't even start for you? [17:16] (it does for me) [17:16] (but I'm several revisions behind the times I bet) [17:17] gary_poster: it does, but since it doesn't accept a script on the command line to run I just hacked it to run the code I want and that code generates the error [17:17] the docstring suggests that it should work, I'm still looking at it [17:17] oh ok [17:19] I got it to work, now to run it in a loop for a while. [17:19] cool [17:35] benji, that's really weird that only the running the test in a loop worked [17:35] I imagine you already investigated the test's setup? [17:36] gary_poster: nope, that would be a good thing to look at. [17:36] cool [17:37] grr, no python file mentions the doctest in question; someone must be a smarty pants and loading it by looking at the files in a directory [17:41] ah, found it [17:41] there's no setup, it is run in the DatabaseFunctionalLayer [17:54] huh [17:54] weird [17:54] mm [17:54] benji, I think layers define pretest and posttest code [17:54] or, they can, I should say [17:55] My system updated, and wants a restart. back in a mo. [17:56] I'm looking at the various layers triggered by that test now, I'm going to try to see which bits can be removed without making the test fail. Maybe some will keep it passing but never let it generate the error and we'll have a clue. [18:32] benji, no rush, https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhordeoneonone is ready when you are [18:32] gary_poster: sure, one minute [19:31] gary_poster: ping me when ready [19:31] ack thanks bac almost [19:32] * bac -> refill [19:45] bac, ping (https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhordeoneonone) [19:49] * benji reboots to test missing beep hypothesis. [20:08] bac, approved with only the comment we discussed on the call [20:08] gary_poster: great, thanks [20:11] gary_poster: i've added rabbitfixture-0.3.3-bac-exp.tar.gz to download-cache with 90 second timeout [20:18] ok bac, I'll give it a whirl tomorrow and report back [20:18] ok [20:20] so apple is going all renewable at the NC data center [20:21] they claim a 20 megawatt solar site will produce 42 million kwh [20:21] and a 5 megawatt fuel cell will produce 40 million kwh [20:21] how's that work? do they base the solar output on 6 hr/shine/day?