| === danilo_ is now known as danilos | ||
| gary_poster | hey gmb, when you have time, I'd love reviews of https://code.launchpad.net/~gary/launchpad/bug724025/+merge/72484 and https://code.launchpad.net/~gary/launchpad/sqlprofiler/+merge/72491. The second one is big (1168 lines). I can divide it in two if you like; take a glance and let me know then if you want me to. | 11:38 |
|---|---|---|
| gmb | gary_poster: Sure. I've got an oversized branch to review for Henning first, but I'll take a look once that's done. | 12:10 |
| gary_poster | cool thanks gmb | 12:10 |
| gary_poster | note to self: make schema != make harness :-/ | 12:21 |
| gary_poster | In a novel twist, I'll announce the call a few minutes before it is supposed to happen. | 12:26 |
| gary_poster | bac benji danilos (danilo_) gmb, call in 3 | 12:27 |
| gmb | k | 12:27 |
| bac | o | 12:27 |
| gary_poster | :-) | 12:27 |
| benji | r | 12:27 |
| danilos | n | 12:27 |
| gary_poster | e | 12:27 |
| gary_poster | a | 12:27 |
| benji | it's like there was a cat-on-keyboard epidemic | 12:28 |
| gary_poster | If I had only gotten my "a" in faster, it would have been "korea" -> "korean" | 12:28 |
| gary_poster | boys are going to their first tae kwon do lesson today, so it is on the mind | 12:29 |
| bac | i thought you just didn't know how to spell your eye parts | 12:29 |
| gary_poster | heh | 12:29 |
| bac | cool, make them call you "dear leader" from now on | 12:29 |
| gary_poster | heh, I suspect they are south korean teachers, though maybe not | 12:30 |
| gmb | gary_poster: bug724025 branch approved. | 13:26 |
| * gmb starts on the other one. | 13:26 | |
| gary_poster | sweet, thanks gmb. gmb, I'll probably head out in 20 if that makes a difference to your planning | 13:27 |
| gmb | gary_poster: It shouldn't. I can always queue up questions ;) | 13:27 |
| gary_poster | heh, ok cool thx | 13:27 |
| gary_poster | danilos, you agree that for the purpose of getting merge proposals for all the bugtasks of a given bug, I need the IAllBranches branch collection, right? I don't think I can narrow it down any more. (I will accept "I have no idea" as an answer ;-) ) | 13:50 |
| gary_poster | it reduced queries by 200 (1998 -> 1752) but no change on actual time of page. | 13:56 |
| * gary_poster to doctor | 13:56 | |
| bac | danilos could you use your alleged super powers to run http://pastebin.ubuntu.com/673160/ on qastaging? | 14:25 |
| bac | danilos: actually this would be better: | 14:26 |
| bac | select person.name, emailaddress.email, account.* from person, emailaddress, account where emailaddress.account=account.id and account.status=10 and person.account=account.id and emailaddress.email like '%@list%'; | 14:26 |
| bac | nm danilos i'll ask a losa | 14:47 |
| * gmb -> running an errand; bbiab | 15:05 | |
| danilos | bac, oh, sorry, deep into code | 15:09 |
| danilos | gary_poster, sorry to you as well :) generally, you'd provide a separate branch collection for a bug which fetches all branches for a bug, that should be simple | 15:11 |
| danilos | bac, do you still need stuff to happen on staging? | 15:12 |
| danilos | gary_poster, perhaps you need to increase storm_cache_size as well, though | 15:12 |
| bac | danilos, no, thanks | 15:14 |
| gary_poster | doctor confirms that I'm still alive, as far as he can tell. | 16:27 |
| gary_poster | danilos, separate branch collection for bug, oh ok. I'll see how to do that. I did increase storm cache size to 10000 before I started my experiments | 16:28 |
| danilos | gary_poster, well, the idea is to have something like "getAllBranchesForBug()" where you could pass in eager_load when you want it to fetch things like merge proposals and such | 16:45 |
| danilos | gary_poster, though, if you've already got a list of branches for a bug with a single query, and you pass those branches into getMergeProposals(for_branches=..., eager_load=True), then there's not much you will really get from that | 16:46 |
| gary_poster | danilos, right, I do. | 16:47 |
| gary_poster | This is what I added: | 16:47 |
| gary_poster | list(getUtility(IAllBranches).getMergeProposals( | 16:47 |
| gary_poster | for_branches=linked_branches, eager_load=True)) | 16:47 |
| gary_poster | code already calculates linked_branches pretty efficiently | 16:47 |
| danilos | gary_poster, yeah, that looks good then | 16:47 |
| gary_poster | danilos, I still saw calls I didn't expect, perhaps naively. Lemme get a few... | 16:47 |
| danilos | gary_poster, maybe it's about some other stuff | 16:48 |
| danilos | gary_poster, sure | 16:48 |
| * danilos dines while gary is at it :) | 16:48 | |
| gary_poster | :-) | 16:48 |
| gary_poster | danilos, I see these: | 16:48 |
| gary_poster | mm, going to pastebin instead | 16:49 |
| danilos | sure :) | 16:49 |
| gary_poster | danilos, this is one set of repeated queries. I get these for each MP, I think. | 16:49 |
| gary_poster | http://pastebin.ubuntu.com/673239/ | 16:49 |
| gary_poster | I wonder if something is running was SQL or otherwise explicitly bypassing the cache | 16:50 |
| * gary_poster going to get tracebacks; this takes > 1 minute | 16:51 | |
| danilos | gary_poster, interesting, I expected 1-3 and 5-7 (inclusive) to not be there | 16:51 |
| gary_poster | that's what I had hoped too :-) | 16:51 |
| gary_poster | it is interesting that it still cut 250 queries out | 16:52 |
| danilos | gary_poster, yeah, that's why I thought it might have been the cache size thing | 16:53 |
| gary_poster | It's at 10000 though | 16:54 |
| gary_poster | I suppose it could be bigger | 16:54 |
| danilos | gary_poster, yeah, understood (just explaining where I was coming from) | 16:54 |
| gary_poster | what's production, do you remember off-hand? | 16:54 |
| danilos | well, production is 10k | 16:54 |
| gary_poster | yeah ok | 16:54 |
| danilos | so no need for bigger, this should be good enough | 16:54 |
| danilos | gary_poster, I don't have any suggestions on what to look for though :/ | 16:56 |
| gary_poster | danilos, that's because you haven't seen my new super cool ++profile++supertrace with nitro burning funny cars | 16:56 |
| danilos | gary_poster, do note that these values are usually fetched from cached properties associatedProductSeries and associatedSourcePackages on IBranch or something, so perhaps look if these are being used properly | 16:56 |
| danilos | gary_poster, heh, true, true, I'll have to try that one out | 16:56 |
| gary_poster | ++profile++sqltrace, that is | 16:57 |
| danilos | gary_poster, and eager_load in getMergeProposals fills the caches for the above properties, so something going wonky there is the only thing I can think of | 16:58 |
| gary_poster | interesting, I wonder if it is cache exhaustion... | 16:58 |
| gary_poster | I have 160 mps in my example... | 16:58 |
| danilos | I've tested with 300 for +activereviews page | 16:58 |
| danilos | though, you have bugtasks and such with all their data and objects | 16:59 |
| danilos | (though, I think I only used something like 2000 for the cache size) | 16:59 |
| gary_poster | before, that led to three sets of 160 repeated queries, two of which were related to branches. | 16:59 |
| gary_poster | Now some of them are 118 | 16:59 |
| danilos | well, it should be easy to try upping the cache size to a gazillion or something like that :) | 16:59 |
| danilos | just to rule that out | 16:59 |
| gary_poster | yeah, I'll try that | 17:01 |
| gary_poster | while I'm looking at the results though... | 17:01 |
| gary_poster | Only this line is repeated for awhile: | 17:01 |
| gary_poster | SELECT CodeReviewVote.branch_merge_proposal, CodeReviewVote.vote_message, | 17:01 |
| gary_poster | (etc.) | 17:01 |
| gary_poster | then it joins in with ... | 17:02 |
| gary_poster | SELECT SeriesSourcePackageBranch.branch, SeriesSourcePackageBranch.date_created, | 17:02 |
| gary_poster | (etc.) | 17:02 |
| gary_poster | and the Product Series branch, and the BranchMergeProposal | 17:03 |
| gary_poster | I wonder if, instead of relying on the cache, I could use the getMergeProposals call directly for what I want... | 17:04 |
| danilos | well, you'd have to rework the code logic for that, but sure you could | 17:06 |
| gary_poster | yeah | 17:08 |
| gary_poster | no change in sql calls after upping the cache to 100K (1752) so that's out. Looking deeper. | 17:09 |
| danilos | gary_poster, btw, all things kept in cached properties are there because the queries are not by ID (eg. by source_branch=... or something), so perhaps they are populated too late | 17:09 |
| danilos | gary_poster, on re-reading that, let me clarify it: "stuff in cached properties is in cached properties because just keeping it in storm caches isn't sufficient since queries are not by ID" | 17:10 |
| danilos | gary_poster, anyway, I am really off now, hopefully you figure something out :) | 17:11 |
| gary_poster | danilos, but in my experiment I put the MP calculation within the code that calculates...OK :-) thanks for your help and thoughts. Have a great evening | 17:11 |
| benji | umm... earthquake? | 17:54 |
| bac | er? | 17:55 |
| benji | we just had a 5.8 earthquake | 17:56 |
| benji | http://maps.google.com/maps?f=q&hl=en&q=37.8750,-77.9080%28M5.8+-+Virginia+-+2011+August+23+17:51:03+UTC%29&t=h&z=7&iwloc=A | 17:56 |
| benji | that was... exciting | 17:56 |
| bac | http://earthquake.usgs.gov/earthquakes/recenteqsww/Quakes/at00lqe6x3.php | 18:03 |
| bac | wow, benji, that was real close to you. felt nothing here | 18:06 |
| benji | bac: yep, we're roughly 25 miles north of where the quake was centered | 18:07 |
| bac | since the dog didn't wake up i'll assume it wasn't just me being inattentive | 18:11 |
| bac | benji: http://jmckinley.posterous.com/dc-earthquake-devastation | 19:06 |
| benji | bac: heh; that's about the extent of it; my worst damage is that the (rather large) fountain we have haning by our front door is all crooked now | 19:07 |
| bac | well we have a big crack in the ceiling in the kitchen! | 19:08 |
| bac | but that showed up last week. | 19:08 |
| bac | we have a very prescient kitchen | 19:08 |
| * bac is still confused. a 17 year old house shouldn't be settling | 19:08 | |
| benji | yeah, that's kind of surprising; I wonder if there is an erosion problem. Do you live on an incline? | 19:09 |
| gary_poster | eek on earthquake | 19:43 |
| * bac chrs... | 20:06 | |
| bac | wow RT was full of spam | 20:19 |
| benji | yep, I cleared some of it out but ran out of time (and energy) | 20:22 |
| * gary_poster didn't make up for being away for the doctor during my time | 21:10 | |
| gary_poster | unfortunately my second doctor appt this week does not conflict with CHR :-P | 21:10 |
| * bac learns about +recipes | 21:18 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!