[00:09] zopelessappserverlayer [00:19] wgrant: you're going to laugh at this [00:19] wgrant: in between the tears. [00:19] I hope so. [00:19] wgrant: ZASL runs up the codehosting server. [00:19] wgrant: that calls into private-xmlrpc [00:19] wgrant: thats raising oopses because of disconnected stores [00:20] ZASL itself shouldn't start codehosting... [00:20] Unless it's pretty different from AppServerLayer, which it shouldn't be since I made ZopelessLayer and FunctionalLayer almost equal. [00:21] wait, there is more [00:22] we also get a disconnectionerrror: terminating connection due to administrator command [00:22] Where are you running this? [00:22] wgrant: -> because the test db reset code pulls the db out from under the slave appserver [00:22] Locally? [00:22] wgrant: yes, totally reproducable [00:22] You need the new storm. [00:22] on lucid ? [00:22] Yes. [00:23] libpq5 8.4.9 breaks everything. [00:23] Destroyed buildbot on Friday. [00:23] ec2 isn't affected yet. [00:23] this is 8.4.3-1 [00:23] New Storm is in devel now. [00:23] Ah. [00:23] Maybe your analysis is correct, then. [00:23] maybe :) [00:23] the reason it happens only when two tests are run [00:24] is that we have a brand new appserver if we only run one test [00:24] Yep. [00:24] Still, this looks similar to the libpq5 issue... [00:24] Exactly the same thing. [00:25] librarian was broken after the first test. [00:25] how so ? [00:25] I thought that was it not detecting the disconnect ? [00:25] Right. [00:25] this detects it fine [00:25] If this is a DisconnectionError, it's probably working. [00:25] its raised from storm.database line 376 _check_disconnect [00:25] yes, it is detecting it [00:26] we're just oopsing - correctly - because the db was yanked without telling the appserver [00:28] so I should say, rather than 'codehosting is started' the test using bzr transports and these are backing onto xmlrpc requests in some fashion [00:29] and bingo [00:29] the db is being dropped after the test, rather than the sequences reset [00:29] I'm now totally confident of the analysis, but wtf to do about it [00:30] I think I'll expect 1 or 9 oopses [00:30] along with an XXX about this [00:30] wgrant: as my victim^Wreviewer, how do you feel about this [00:34] lifeless: The DB will be dropped if it's dirty. [00:34] I think that's probably fine. [00:34] As long as it is really 1 or 9 :) [00:35] bug 884036 [00:35] <_mup_> Bug #884036: db layer test resets interact badly with slave appserver instances < https://launchpad.net/bugs/884036 > [00:36] see if that convinces you [00:36] second oops added now [00:39] Sounds good. [00:43] lifeless: https://code.launchpad.net/~wgrant/launchpad/cte-it-up/+merge/80767 [00:44] 2.5s->1.6s on qastaging [00:44] wgrant: does this change +bugs as well? or just BugTask:* ? [00:45] lifeless: +bugs as well. Do you want that checked too? [00:45] uhm yes. [00:46] just cause, well, EPIC TIMEOUTS [00:46] It should in theory be a significant improvement. But let's see. [00:46] rev 14156 of my branch is up; Could you please incremental review anything you haven't, and I'll land. [00:47] Ermm. [00:47] This isn't landable... [00:47] There is no rabbit on prod, and the fallback is flaky at best. [00:48] with no rabbit it will behave as it did befre [00:48] *before* [00:48] And the moment we have rabbit, all OOPSes will disappear. [00:48] Also, BSON. [00:48] whats flaky about the fallback ? [00:48] If rabbit is up but amqp2disk hasn't set up its queue bindings, OOPSes go to nowhere. [00:48] And that's just what I've noticed. [00:49] thats how loosely coupled works. [00:49] Yes, but it's not how production should work. [00:49] Its not a defect, its expected. I'm hoping to get that side of it sorted later today. [00:49] eg. qastaging and staging have rabbit, no amqp2disk. [00:49] we can disable the config for them. Or get amqp2disk up and live. [00:49] Do that before landing. [00:50] sure [00:50] but whats flaky? [00:50] or was that code for 'new world order does not have the same guarantees' ? [00:50] It only works if rabbit is entirely unconnectable, and it's not exactly well-tested. [00:51] wgrant: its tested in oops-amqp; the tests there pull rabbit out from under it [00:51] wgrant: rabbit config isn't, and shouldn't, be the business of the oops producer [00:51] Sure. [00:51] But as your branch is now, as soon as rabbitmq is live on production then we lose all error reporting. [00:52] IFF the queue isn't setup [00:52] Which it probably won't be. [00:52] separate issue; we can toggle off the oops exchange very very easily. [00:52] (in prodconfigs) [00:52] lol prodconfigs easily [00:53] more easily than landing this [00:53] change, push, review, pqm-submit. [00:55] lifeless: Why is UnknownRemoteError now permitted in test_runner.py? [00:56] wgrant: because python OOM is nondeterministic [00:56] wgrant: I ran it in a loop and didn't get MemoryError every time [00:56] But that invalidates the test... [00:57] Also, isn't the setUpQueue call in sync() racy? [00:57] I can put it back, but I expect some buildbot failures as a result [00:57] I don't see why it needs a new queue. [00:57] wgrant: because auto_delete [00:57] They would be legitimate failures. [00:57] The current test is not flaky. [00:58] Ah. [00:58] Fair point. [00:58] wgrant: I bet it is [00:58] Still racy, but acceptable. [00:58] lifeless: It's never been a problem before. [00:58] wgrant: poolie ran into a race the other day, last happened 4 years ago. [00:58] wgrant: doesn't make it less of a race [00:58] Even so. [00:58] I can put it back. [00:58] It makes it an invalid test. [00:58] But I think the test lies today [00:58] the test isn't valid [00:58] lifeless: two races [00:59] Better to have it failing and then disabled. [00:59] Than left entirely invalid. [00:59] i guess it is the spring racing carnival in launchpad too [00:59] lifeless: Perhaps it's because you were running it i386? [01:00] wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it? [01:00] wgrant: yes, I am [01:01] wgrant: I have no particular interest in this; the runner code doesn't generate MemoryError synthetically, by checking for ulimit violations etc - it depends on the python runtime only ever raising MemoryError AFAICT (grepping for MemoryError ..) [01:01] wgrant: so pick a preferred resolution, and I'll do it. [01:02] lifeless: Your INTERACTIVE_TESTS change concerns me. Doesn't that mean we're just not going to see any oopses from the appserver? [01:02] wgrant: myself, I'm quite sure that we will eventually hit whatever point in CPython caused UnknownRemoteError [01:03] lifeless: Sure, but I'd prefer to have a flaky or disabled test, rather than one that doesn't actually test. [01:03] wgrant: no tests look for them, and its *only* yuixhr that uses this code path [01:03] wgrant: depends on the intent of the test; I think 'raises promptly' is sufficiently narrow. [01:03] Oh, it's in testapp, right. [01:03] wgrant: we don't depend on catching MemoryError anywhere (damn straight) [01:04] wgrant: so the -only- impact on broadening the exceptions is that the test suite lines up with the OOPS reports [01:04] wgrant: [if/when we see this in prod] [01:04] lifeless: Hm, UnknownRemoteError is probably txamqp... [01:04] wgrant: txamp do you mean ? [01:04] Yes. [01:04] I fail. [01:05] So, I'd prefer it if you reverted that change. [01:05] If it is a problem, we can unrevert it. [01:05] it is in Twisted AMP yes [01:05] sure, I will (it is a problem) [01:06] It's not a problem. [01:06] It may be on i386, but nobody's noticed that before :) [01:08] let me just thrash carob's pagecache [01:09] Oh? [01:09] launchpad.net-logs$ grep UnknownRemoteError . -r [01:10] Ah [01:10] That seems unwise... [01:10] It's like 150GB or more. [01:10] Or was it 250GB [01:10] I can't quite remember. [01:10] serial IO ftw [01:10] I still contend that carob's disks are actually a NAS in NZ. [01:11] I could grep faster if they were [01:11] anything else ? [01:12] * StevenK prods wallyworld_. [01:12] ouch [01:13] [12:00] < StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it? [01:13] sure. [01:13] lifeless: I think that's about it. [01:15] ok, so I'll push th revert to test_memory_hob_job, toss at ec2, and we'll see what we see [01:15] StevenK: '%s/+activate-ppa' % canonical_url(team) -> canonical_url(team, view_name='+activate-ppa') [01:17] other than that, looks good thanks [01:17] wgrant: can as approved vote? [01:20] wallyworld_: Thank you for the Approve. [01:20] np. [01:20] lifeless: Done. [01:24] thanks [01:25] lifeless: Erm. [01:26] What did I say about not landing this. [01:26] I approved mostly given that you were going to get at least (qa)staging fixed first :) [01:26] (and no, "It'll be done soon" is not valid) [01:31] wgrant: I'm working on that right now [01:31] If it's not done by the time ec2 finishes, please kill the ec2 instance. [01:32] that would be a waste [01:32] Let's not get into another 2-month cart-before-the-horse situation like we did with fastdowntime. [01:32] if its not ready I'll change prodconf to disable the key in staging [01:32] wgrant: you exaggerate there [01:32] No... [01:32] wgrant: I'm not about to block the deployment pipeline for any extended period [01:33] wgrant: I'm well aware of its impact, given I've done most of the analysis and discussion to demonstrate that [01:33] You are about to significantly hinder the crisis analysis process, though. [01:33] I'm about to change the exact recipe needed [01:33] grep still works, data can still be easily view [01:33] ed [01:34] No, grep doesn't still work. [01:34] I can't grep for ^Exception-Type: RequestExpired now. [01:34] yes it does, bson stores strings as length: prefix, so you can do exactly that. [01:34] just phrased -very slightly- differently. [01:35] I am close to rescinding my approval. [01:36] why? [01:36] 1) all (qa)staging OOPSes will be lost unless you sort things out in the next few hours. [01:37] 1) hystrionics, I've outlined 2 solid workarounds and taken responsibility for having them sorted. [01:37] 2) oops-tools is mostly useless, and this breaks shell pipelines. [01:37] 2) has had 3? more? weeks for folk to respond to the get-out-and-try aspect, and there are also recipes in that same thread to deal. [01:38] We already had three expressions of disapproval. [01:38] In addition to my initial one. [01:41] aaron asked why; stuart noted about readability with an anecdote from ISD who are doing noone-knows-what, and gary was -0.5 accepting my argument about greppability [01:43] I have been trying and playing around, and I'm satisfied I can do the analyses I was previously doing. [01:43] Have you experimented, or are you just concerned about the possible impact? [01:44] I know that my usual method for analysing performances crises now has to be rewritten in Python, or invoke Python thousands of times. [01:45] how do you know that ? [01:45] Because it relies on selecting on and extracting fields and parts of fields from OOPSes. [01:46] so, its an assumption. Thats ok, but its useful to be clear. [01:47] you are assuming, without trying, that grep and sed can't extract what you want from bson. [01:47] you may be right, you may not be. [01:49] I'm more worried about the url interactions than the change to bson; oops-tools has shocking understanding of urls-are-text [01:49] for instance our appserver generated urls are unicode [01:49] I'm not too concerned about how it interacts with oops-tools. [01:49] so we may end up double encoding for a day or two, which is tolerable but annoying [01:49] oops-tools isn't critical, and can be fixed. [01:53] lifeless: +bugs seems to be much faster on qastaging, with the CTE patch. [01:53] FTI Ubuntu searches are still timing out, but they were before too so I suspect the index is just cold. [01:53] ok. [01:53] uhm [01:53] just for insurance, lets FF this [01:53] its a particularly sensitive bit of code [01:54] k [01:54] It's all about to become a lot simpler, but until then... [01:55] I'm sure the FF will be overkill [01:55] but I'll feel a whole lot happier having it :) [01:55] Indeed. [02:08] test_bugtask_search is terrible :( [02:08] A slow base test case with dozens of derivatives. [02:09] yes [02:16] lifeless: It's now behind a flag, with a few strategic tests duplicated to test both paths. The new function is ugly as hell, but at least it should be safe :) [02:16] wgrant: thank you very much! [02:17] Bah, but now I have to ec2 it :( [02:17] what could possibly go wrong [02:17] The FF check will probably introduce a new query sometimes :( [02:17] Which could break some query limit tests. [02:18] most of them have some deliberate fat [02:18] with a generous backstop, but yes, it could [02:19] I've been meaning to do a test run with the matcher ensuring a lower bound as well. [02:19] Some of them are waaaaay high now. [02:27] Bah, I can't have my change in create_initialized_view, too many things can't be canonical_url()'d. [03:22] what's a CTE? [03:25] poolie: Common Table Expression [03:25] Defined by the WITH clause. [03:27] ok [03:36] wgrant: Is QA for r14209 on your list this afterrnoon? [03:37] StevenK: That's the userCanView change? [03:37] Yeah [03:37] It's semi-ok. [03:37] qa-ok-ish [03:37] ? [03:38] Liveable, but not ideal. I'm hoping that an improvement will land before someone tries to deploy. [03:38] We have a large amount of European QA before that rev. [03:38] Yes. [04:08] http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx is a good read [04:09] can someone quickly review http://readthedocs.org/docs/judge/en/latest/ for me [04:11] lifeless: that is interesting [04:12] it would be interesting to study: how good are developers at predicting which factors drive quality on their projects [04:22] indeed [04:28] what should i do now? [04:29] partay [04:30] heh [04:30] morning! :) [04:35] hi nigelb [04:37] Hey poolie [04:47] o/ nigelb [04:47] ohai spm :) [05:24] Ugh, I now officially hate the expiring membership email. [05:50] yeah? ok then. INSERT into teams VALUES nibelb WHERE team has expiringing_membership; [05:50] spelling to be optional. [05:51] spm: Let me give you a cronjob that will reset nigelb's memberships to expire in seven days. Every day. [05:51] StevenK: approved. [05:51] this wouldn't be excessive abuse of power would it? :sincerity: [06:13] hah [06:13] StevenK, spm: Love you too :P [06:14] :-) [07:00] wgrant: you said txlongpoll was bust [07:00] wgrant: in that it didn't accept a queue name prefix, right ? [07:00] lifeless: Yes. [07:00] is bigjools aware of this? And that its a blocker to the deploy ? [07:00] I believe Red is aware. [07:01] However, AFAIK they are unaware that anybody is using rabbit except them. [07:01] So they are probably unaware that the fix is time-critical. [07:06] wgrant: there is no bug on https://bugs.launchpad.net/txlongpoll [07:07] wgrant: as you know the details, could you make one please? [07:08] Hmm, rvba said he was going to file one. [07:08] I left it with him, since it was well after midnight. [07:08] the rt claims that read is only on ^longpoll.* [07:09] so there is access control in place [07:09] That is a start. [07:10] Personally I'd prefer the old behaviour to be revived. [07:11] I'm not sure what that is; thus my encouragement for you to file a bug :) [07:11] hmm [07:12] my branch has been running from 0140 through to 0708 [07:12] I suspect a stupidly overloaded host [07:12] lifeless: I just had a branch take 4:45 [07:12] Which is longer than normal. [07:13] hmm [07:13] I suspect that that sync() in cleanUp is a problem [07:13] I have another trace here that took 6.5 hours [07:13] I'll land a branch immediately to remove the trailing sync() [07:14] [given I only added it while debugging that rosetta zopelessappserver test issue, I know its not needed know] [07:14] Ah. [07:15] its the only think I can think of that would add 100% test time [07:16] unless the rabbit layer reset is heinous, but I didn't change that, nor add it to more than a few dozen tests [07:18] wgrant: I suspect both the sync and the existence/presence of poor ec2 loading [07:19] wgrant: did we decide setting mandatory wouldn't work ? [07:20] lifeless: We can't necessarily receive the response. [07:20] wgrant: did we do an experiment ? [07:20] No. [07:21] * StevenK has clearly forgotten how to drive subunit [07:21] But we'd need to asynchronously detect the basic.return, and then $something. [07:21] But I don't think amqplib exposes basic.returns. [07:21] I have a subunit stream and I want to see which tests failed or gave an error [07:21] StevenK: subunit-filter --no-skip --no-passthrough | subunit-ls [07:22] StevenK: or testr load < stream; testr failing [07:22] (or testr failing --list) [07:22] StevenK: or tribunal < stream [07:22] subunit-filter --no-success --no-passthrough --error --failure < lp.log | subunit-ls [07:22] That gives me a traceback [07:23] StevenK: you're making it hard for yourself :) [07:23] StevenK: 3 of those options are the defaults [07:23] StevenK: also whats the tb you get? [07:24] Broken pipe [07:24] subunit-filter is probably erroring [07:24] So one of subunit-filter or subunit-ls is misbehaving [07:24] try one of my recipes above === jbepsilon_ is now known as jbepsilon [07:29] StevenK: ? [07:30] Sorry, got distracted. Your recipe works fine [07:30] "subunit-filter --no-success --no-passthrough --error --failure < lp.log 1>/dev/null" doesn't produce any output on stderr [07:31] its a legitimate set of options [07:31] I'm not sure why you had a tb [07:32] conceivably the pipe was closed with no output I guess [07:32] it works for me, fwiw [07:33] lifeless: http://pastebin.ubuntu.com/724060/ [07:33] wgrant: so, you'll file a bug| get rvba to ? [07:33] StevenK: the first tb is the issue [07:33] lifeless: I'll ask rvba for the latest news. [07:33] StevenK: Can you reproduce with a current subunit ? [07:33] StevenK: (what does dpkg -l python-subunit say) [07:34] lifeless: 0.0.6-3 [07:35] try 0.0.7 [07:35] Such a pity, Mr Bond. [07:38] heh [07:50] lifeless: that reminds me, can you add a tag for 0.0.7 on lp:subunit? [07:51] jelmer: probably, there isn't one ? [07:51] jelmer: it'll be just tip [07:51] jelmer: :!bzr log -r -1 [07:51] ------------------------------------------------------------ [07:51] revno: 152 [07:51] tags: 0.0.7 [07:51] lifeless: "bzr tags -d lp:subunit" doesn't list a 0.0.7 [07:51] lifeless: perhaps that tag is local? [07:52] jelmer: if so, I blame bzr [07:53] jelmer: hmm, diverged branches. funky. [07:54] yes, somethings up. I'm going to push --overwrite and then merge trunk back in [07:57] jelmer: there should be a tag now [07:57] lifeless: great, thanks! [08:17] useoops has landed \o/ [08:20] Has the fix landed? [08:23] My next branch is going to be called The Eagle [08:23] Just to announce its landing? [08:23] correct [08:24] I'll call dibs on Airforce one and Canonical one ;) [08:26] Who's at UDS? francis, matt, dan..and? [08:28] that's about it IIRC [08:28] oh one of the US guys too [08:32] good morning [08:32] wgrant: its about to start ec2 [08:45] nigelb: I thought it was Francie, Matt, Deryck and Curtis. Didn't know Dan was going too. [08:46] new kid, doing the rounds :) [08:54] bigjools: is rvba in today? [09:01] lifeless: doesn't look like it [09:16] bigjools / wgrant - Thanks :) [09:16] I'd quite like to be there again, was the best UDS hotel in ages [09:17] heh [09:17] I wish I could be there. I regret skipping this UDS> [09:18] bigjools: Will you be at the next one? :) [09:19] Too bad I missed you last UDS. [09:19] you walked right past me there :) [09:19] :( [09:19] That's the UDS where I asked Francis "Are you a launchpad guy?" :D [09:19] as for the next one, no idea, we don't decide until shortly before who's going [09:20] Ah! [09:20] <\sh> moins [09:21] <\sh> dear lp-devs how difficult will it be, to change someones launchpad id from X to Y ? [09:21] Its easy to do the change. [09:21] THe repurcusions might make some time to fix ^-^ [09:21] \sh: the answer is "it depends" [09:21] And its not a launchpad problem. Its an SSO thing. [09:22] SSO actually steals that data from us. [09:22] It's a Launchpad thing. [09:22] \sh: Unless you have a PPA, you can just change it. [09:22] <\sh> bigjools, let's say, I would like to request to change my launchpad-id from shermann to sadig [09:22] \sh: then as wgrant says [09:23] wgrant: Meh. I think most of the time its our openid client which isn't implemented properly. [09:23] <\sh> wgrant, ok...my ppa is not a problem...so when I delete my ppa I can just change it? [09:23] if you have any PPAs then they need to be deleted [09:23] then yes [09:23] \sh: Yes. May have to wait up to 10 minutes for it to be completely deleted. [09:23] \sh: delete the ppa, wait for a publishing cycle, visit +edit and rename it [09:23] I think its +edit [09:24] <\sh> wgrant, lifeless , thx...I'm waiting now ;) [09:24] * bigjools has a buildd-manager that will cancel builds \\o/ [09:25] bigjools: Does it also impact on the production builders too :-P [09:25] bigjools: niiice! [09:25] bigjools: You know that's been possible for recipe builds for a year, right? :P [09:25] whatever [09:26] running out of memory != cancelling :) [09:26] lol [09:26] ZING. [09:26] They have a cancel button which sets them to SUPERSEDED. Which actually works pretty well. [09:26]