| wgrant | StevenK: I return | 03:01 |
|---|---|---|
| lifeless | and therefor you are? | 03:01 |
| StevenK | wgrant: Just as I'm about to go to lunch, nice work. :-P | 03:02 |
| StevenK | wgrant: I thought ? and params=(list,) were allergic? | 03:02 |
| wgrant | StevenK: Ah, indeed. Though you can easily Stormify the outer bit of that and just use SQL for the inner bit, or even add FOR UPDATE/SHARE support to Select in about 4 lines of code. | 03:04 |
| StevenK | Or I could ignore it and make you twitch | 03:06 |
| StevenK | wgrant: http://pastebin.ubuntu.com/5685858/ btw | 03:46 |
| wgrant | StevenK: Does that work? | 03:52 |
| wgrant | Oh, I guess it does. | 03:53 |
| wgrant | Because all the SSs that we touch will have BSFs at the start. | 03:53 |
| StevenK | Exactly. And possibly none of them will at the end. | 03:53 |
| wgrant | StevenK: However, does that query at the start perform adequately? | 03:54 |
| wgrant | It would be more reliable to filter on SS ID in the outer query | 03:55 |
| wgrant | Rather than relying on the planner to guess that the inner query results in extremely good selectivity. | 03:55 |
| StevenK | Requires a JOIN in the outer query, ids == BSF not SS | 03:55 |
| wgrant | Or obtain the IDs first, since you need them anyway | 03:56 |
| StevenK | I had that, you said I should get them from the FOR SHARE query ... :-) | 03:56 |
| StevenK | Bleh, do I need execute_zcml_for_scripts to do IStore(BranchMergeProposal) ? | 03:56 |
| wgrant | You need the component architecture, yes. | 03:57 |
| * StevenK stabs Zope | 03:57 | |
| === tasdomas_afk is now known as tasdomas | ||
| wgrant | StevenK: http://pastebin.ubuntu.com/5685908/ | 04:36 |
| StevenK | Haha | 04:37 |
| StevenK | I see you took the easy way of using FOR SHARE too | 04:37 |
| StevenK | wgrant: Your filter_expr is too late | 04:38 |
| StevenK | We've already deleted some BSFs by then | 04:38 |
| wgrant | StevenK: Why is that a problem? | 04:38 |
| wgrant | It's a NOT EXISTS at that point | 04:38 |
| StevenK | Ah | 04:39 |
| wgrant | It only locks the first row that it finds, since a NOT EXISTS obviously fails as soon as it finds a row, but that's fine for our needs. | 04:42 |
| StevenK | wgrant: My MP script of doom is almost done too | 04:47 |
| StevenK | Then we'll see PreviewDiffPruner likes a large dataset | 04:47 |
| wgrant | :) | 04:51 |
| StevenK | 2 PDs for MP 863 | 05:08 |
| StevenK | 4 PDs for MP 861 | 05:08 |
| StevenK | Come on! | 05:08 |
| StevenK | 290 | 05:08 |
| StevenK | Oh, I think it's commint | 05:09 |
| StevenK | *commiting | 05:09 |
| wgrant | StevenK: Going OK? | 05:17 |
| StevenK | I think it's still commiting | 05:19 |
| wgrant | That's very unlikely | 05:19 |
| wgrant | Committing is cheap | 05:19 |
| wgrant | SELECT * FROM pg_stat_activity; | 05:20 |
| StevenK | <IDLE> in transaction | 05:21 |
| wgrant | What's the query start time? | 05:21 |
| StevenK | 2013-05-21 04:09:56.441673+00 | 2013-05-21 04:09:56.448991+00 | 2013-05-21 05:21:26.137095+00 | 05:21 |
| wgrant | That's session | transaction | query | 05:22 |
| wgrant | ? | 05:22 |
| StevenK | backend_start | xact_start | query_start | 05:22 |
| wgrant | So yes | 05:22 |
| wgrant | You're not running $script with LP_DEBUG_SQL? | 05:23 |
| StevenK | Nope | 05:23 |
| wgrant | If this is local, enable statement logging in postgres | 05:23 |
| StevenK | DF | 05:23 |
| wgrant | If it's on DF, god help you. | 05:23 |
| wgrant | Ah | 05:23 |
| StevenK | It's adding an average of 3 PDs for every MP | 05:23 |
| StevenK | (But only one actual diff, so the rest are just rows in PD) | 05:24 |
| wgrant | I just caught it doing an INSERT | 05:24 |
| wgrant | MP 128897 | 05:24 |
| wgrant | It's very slow | 05:24 |
| wgrant | Where's the code? | 05:24 |
| StevenK | codebase-current/foo, it's incredible naive | 05:25 |
| StevenK | incredibly | 05:25 |
| StevenK | Awwww | 05:25 |
| StevenK | It just threw MemoryError | 05:26 |
| wgrant | Yeah | 05:26 |
| wgrant | It used all RAM | 05:26 |
| wgrant | I saw it at 75% | 05:26 |
| wgrant | Commit in batches | 05:26 |
| StevenK | I would if you'd close it | 05:27 |
| wgrant | Done | 05:27 |
| StevenK | wgrant: Look sensible? | 05:28 |
| wgrant | StevenK: I'd do more like 2000, but doesn't really matter, so go ahead :) | 05:32 |
| wgrant | StevenK: It might still fail due to StupidCache, but it's worth a try | 05:33 |
| StevenK | wgrant: I'd bump it to 2000, but you like leaving vim running | 05:34 |
| wgrant | StevenK: E! | 05:34 |
| StevenK | Hah | 05:36 |
| StevenK | 3 PDs for MP 160882 | 05:36 |
| StevenK | 4 PDs for MP 160881 | 05:36 |
| StevenK | 5 PDs for MP 160880 | 05:36 |
| StevenK | Commit | 05:36 |
| StevenK | Ah, look at that | 06:06 |
| wgrant | Hm? | 06:07 |
| StevenK | It finished | 06:07 |
| StevenK | And I think we have OMGLOTS of previewdiffs to kill | 06:08 |
| StevenK | SELECT count(*) FROM previewdiff WHERE EXTRACT(year FROM date_created) = 2001; => 570852 | 06:10 |
| wgrant | :) | 06:11 |
| StevenK | 2013-05-21 06:12:14 DEBUG2 [PreviewDiffPruner] Iteration 14 (size 10000.0): 1.226 seconds | 06:12 |
| StevenK | 2013-05-21 06:13:06 DEBUG2 [PreviewDiffPruner] Iteration 64 (size 10000.0): 0.002 seconds | 06:13 |
| StevenK | 2013-05-21 06:13:06 DEBUG2 [PreviewDiffPruner] Done. 584116 items in 65 iterations, 68.688814 seconds, average size 8986.402205 (8503.80299128/s) | 06:13 |
| StevenK | wgrant: So, I will put up a branch tomorrow with 44-4 that adds the two indices on pd and id. | 06:13 |
| wgrant | StevenK: Sounds good | 06:14 |
| StevenK | And PDP is already hellishly fast, so that's handy | 06:14 |
| StevenK | It took 30 minutes to add them and 60 seconds to delete them | 06:14 |
| wgrant | The deletion is optimised SQL | 06:14 |
| wgrant | The insertion was awful Python | 06:14 |
| StevenK | s/awful/naive/ :-) | 06:15 |
| wgrant | No, the script wasn't bad, but the LP infrastructure is | 06:15 |
| StevenK | wgrant: News at 11 :-P | 06:22 |
| StevenK | Haha | 06:22 |
| StevenK | DiffPruner punted the lone diff I added too | 06:22 |
| wgrant | :) | 06:22 |
| StevenK | 2013-05-21 06:14:10 DEBUG2 [DiffPruner] Iteration 0 (size 1.0): 132.183 seconds | 06:23 |
| StevenK | I'm guessing DF wasn't so happy | 06:23 |
| StevenK | Oh, it might have been transaction blocked | 06:23 |
| StevenK | Given that time is extremly close to the entire runtime | 06:23 |
| wgrant | More likely that it includes the time to calculate the condemned IDs | 06:24 |
| wgrant | StevenK: Are these in -daily? | 06:24 |
| StevenK | They both are, yes | 06:24 |
| wgrant | Good | 06:24 |
| StevenK | I wasn't sure how heavyweight they would be, and we don't buy anything with them in hourly | 06:25 |
| StevenK | wgrant: If you're happy to accept r=sinzui on that MP, the changes you suggested have been pushed for a while. | 06:29 |
| wgrant | I was hoping to relayer lp.bugs.m.bsf and .ss, but they're too intertangled, so I guess that will have to do | 06:45 |
| wgrant | StevenK: r=me | 06:46 |
| wgrant | StevenK: For QA, I'd recomment running a harness with LP_DEBUG_SQL and trying a deletion | 06:47 |
| wgrant | The queries are designed to be very fast, but we should check that I didn't miss something | 06:47 |
| === liam_ is now known as Guest39447 | ||
| czajkowski | jtv: you alive.... | 09:00 |
| jtv | czajkowski: barely... this particular jet lag always gets me in a weird way, where for a while I think I'm off the hook and then it just comes back. | 09:12 |
| czajkowski | jtv: tomorrow could you look at https://answers.launchpad.net/launchpad/+question/229157 please.... | 09:12 |
| jtv | OK | 09:12 |
| czajkowski | thanks | 09:13 |
| jtv | StevenK: any chance you can get at any oopses generated from this error? https://answers.launchpad.net/launchpad/+question/229157 | 10:17 |
| czajkowski | wgrant: ^ | 10:20 |
| wgrant | Ah | 10:20 |
| wgrant | I have tracebacks about that | 10:20 |
| jtv | I thought it'd be the plurals, but I see nothing wrong with them. | 10:20 |
| jtv | Nor does the empty Last-Translator cause it. | 10:21 |
| wgrant | No, it's something to do with the query logging, IIRC | 10:21 |
| wgrant | Yeah | 10:21 |
| wgrant | The statement issued by _getPOTMsgSetBy causes the statement logger to choke when trying to expand parameters | 10:21 |
| wgrant | It first started happening yesterday | 10:22 |
| wgrant | Only on empathy and webtrees | 10:22 |
| wgrant | Haven't had a chance to investigate yet | 10:24 |
| jtv | wgrant: sorry for the silence — pre-imp call. Do we have a bug for that problem? | 10:38 |
| wgrant | jtv: Not yet | 10:41 |
| jtv | wgrant: it sounds as if there isn't much I can do for this question... would it be OK if I handed it off to you? | 10:41 |
| wgrant | jtv: I'd love to say no, but sadly I cannot. | 10:42 |
| wgrant | :) | 10:42 |
| jtv | "To the first part or the second part?" :) | 10:42 |
| jtv | If there's a way I can usefully take a portion of this problem off your hands, I'd be happy to. | 10:43 |
| * jtv has to relocate | 10:48 | |
| wgrant | jtv: You can't take it off my hands, so I can't say it's not OK to hand it off to me, as much as I'd love to have it not be my problem :P | 10:48 |
| jtv | wgrant: then we both feel entirely the same. That is one bright spot in the situation. :) | 10:49 |
| wgrant | Heh | 10:49 |
| * jtv runs | 10:50 | |
| === gary_poster is now known as gary_poster|away | ||
| === gary_poster|away is now known as gary_poster | ||
| === tasdomas is now known as tasdomas_afk | ||
| === tasdomas_afk is now known as tasdomas | ||
| === wedgwood_away is now known as wedgwood | ||
| === deryck is now known as deryck[afk] | ||
| === BradCrittenden is now known as bac | ||
| === stub1 is now known as stub | ||
| === tasdomas is now known as tasdomas_afk | ||
| === deryck[afk] is now known as deryck | ||
| === gary_poster is now known as gary_poster|away | ||
| === gary_poster|away is now known as gary_poster | ||
| Laney | hmm | 18:41 |
| === BradCrittenden is now known as bac | ||
| === wedgwood is now known as wedgwood_away | ||
| === wedgwood_away is now known as wedgwood | ||
| czajkowski | Laney: what have you broken... | 22:52 |
| === wedgwood is now known as wedgwood_away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!