=== danilo_ is now known as danilos [11:38] hey gmb, when you have time, I'd love reviews of https://code.launchpad.net/~gary/launchpad/bug724025/+merge/72484 and https://code.launchpad.net/~gary/launchpad/sqlprofiler/+merge/72491. The second one is big (1168 lines). I can divide it in two if you like; take a glance and let me know then if you want me to. [12:10] gary_poster: Sure. I've got an oversized branch to review for Henning first, but I'll take a look once that's done. [12:10] cool thanks gmb [12:21] note to self: make schema != make harness :-/ [12:26] In a novel twist, I'll announce the call a few minutes before it is supposed to happen. [12:27] bac benji danilos (danilo_) gmb, call in 3 [12:27] k [12:27] o [12:27] :-) [12:27] r [12:27] n [12:27] e [12:27] a [12:28] it's like there was a cat-on-keyboard epidemic [12:28] If I had only gotten my "a" in faster, it would have been "korea" -> "korean" [12:29] boys are going to their first tae kwon do lesson today, so it is on the mind [12:29] i thought you just didn't know how to spell your eye parts [12:29] heh [12:29] cool, make them call you "dear leader" from now on [12:30] heh, I suspect they are south korean teachers, though maybe not [13:26] gary_poster: bug724025 branch approved. [13:26] * gmb starts on the other one. [13:27] sweet, thanks gmb. gmb, I'll probably head out in 20 if that makes a difference to your planning [13:27] gary_poster: It shouldn't. I can always queue up questions ;) [13:27] heh, ok cool thx [13:50] danilos, you agree that for the purpose of getting merge proposals for all the bugtasks of a given bug, I need the IAllBranches branch collection, right? I don't think I can narrow it down any more. (I will accept "I have no idea" as an answer ;-) ) [13:56] it reduced queries by 200 (1998 -> 1752) but no change on actual time of page. [13:56] * gary_poster to doctor [14:25] danilos could you use your alleged super powers to run http://pastebin.ubuntu.com/673160/ on qastaging? [14:26] danilos: actually this would be better: [14:26] select person.name, emailaddress.email, account.* from person, emailaddress, account where emailaddress.account=account.id and account.status=10 and person.account=account.id and emailaddress.email like '%@list%'; [14:47] nm danilos i'll ask a losa [15:05] * gmb -> running an errand; bbiab [15:09] bac, oh, sorry, deep into code [15:11] gary_poster, sorry to you as well :) generally, you'd provide a separate branch collection for a bug which fetches all branches for a bug, that should be simple [15:12] bac, do you still need stuff to happen on staging? [15:12] gary_poster, perhaps you need to increase storm_cache_size as well, though [15:14] danilos, no, thanks [16:27] doctor confirms that I'm still alive, as far as he can tell. [16:28] danilos, separate branch collection for bug, oh ok. I'll see how to do that. I did increase storm cache size to 10000 before I started my experiments [16:45] gary_poster, well, the idea is to have something like "getAllBranchesForBug()" where you could pass in eager_load when you want it to fetch things like merge proposals and such [16:46] gary_poster, though, if you've already got a list of branches for a bug with a single query, and you pass those branches into getMergeProposals(for_branches=..., eager_load=True), then there's not much you will really get from that [16:47] danilos, right, I do. [16:47] This is what I added: [16:47] list(getUtility(IAllBranches).getMergeProposals( [16:47] for_branches=linked_branches, eager_load=True)) [16:47] code already calculates linked_branches pretty efficiently [16:47] gary_poster, yeah, that looks good then [16:47] danilos, I still saw calls I didn't expect, perhaps naively. Lemme get a few... [16:48] gary_poster, maybe it's about some other stuff [16:48] gary_poster, sure [16:48] * danilos dines while gary is at it :) [16:48] :-) [16:48] danilos, I see these: [16:49] mm, going to pastebin instead [16:49] sure :) [16:49] danilos, this is one set of repeated queries. I get these for each MP, I think. [16:49] http://pastebin.ubuntu.com/673239/ [16:50] I wonder if something is running was SQL or otherwise explicitly bypassing the cache [16:51] * gary_poster going to get tracebacks; this takes > 1 minute [16:51] gary_poster, interesting, I expected 1-3 and 5-7 (inclusive) to not be there [16:51] that's what I had hoped too :-) [16:52] it is interesting that it still cut 250 queries out [16:53] gary_poster, yeah, that's why I thought it might have been the cache size thing [16:54] It's at 10000 though [16:54] I suppose it could be bigger [16:54] gary_poster, yeah, understood (just explaining where I was coming from) [16:54] what's production, do you remember off-hand? [16:54] well, production is 10k [16:54] yeah ok [16:54] so no need for bigger, this should be good enough [16:56] gary_poster, I don't have any suggestions on what to look for though :/ [16:56] danilos, that's because you haven't seen my new super cool ++profile++supertrace with nitro burning funny cars [16:56] gary_poster, do note that these values are usually fetched from cached properties associatedProductSeries and associatedSourcePackages on IBranch or something, so perhaps look if these are being used properly [16:56] gary_poster, heh, true, true, I'll have to try that one out [16:57] ++profile++sqltrace, that is [16:58] gary_poster, and eager_load in getMergeProposals fills the caches for the above properties, so something going wonky there is the only thing I can think of [16:58] interesting, I wonder if it is cache exhaustion... [16:58] I have 160 mps in my example... [16:58] I've tested with 300 for +activereviews page [16:59] though, you have bugtasks and such with all their data and objects [16:59] (though, I think I only used something like 2000 for the cache size) [16:59] before, that led to three sets of 160 repeated queries, two of which were related to branches. [16:59] Now some of them are 118 [16:59] well, it should be easy to try upping the cache size to a gazillion or something like that :) [16:59] just to rule that out [17:01] yeah, I'll try that [17:01] while I'm looking at the results though... [17:01] Only this line is repeated for awhile: [17:01] SELECT CodeReviewVote.branch_merge_proposal, CodeReviewVote.vote_message, [17:01] (etc.) [17:02] then it joins in with ... [17:02] SELECT SeriesSourcePackageBranch.branch, SeriesSourcePackageBranch.date_created, [17:02] (etc.) [17:03] and the Product Series branch, and the BranchMergeProposal [17:04] I wonder if, instead of relying on the cache, I could use the getMergeProposals call directly for what I want... [17:06] well, you'd have to rework the code logic for that, but sure you could [17:08] yeah [17:09] no change in sql calls after upping the cache to 100K (1752) so that's out. Looking deeper. [17:09] gary_poster, btw, all things kept in cached properties are there because the queries are not by ID (eg. by source_branch=... or something), so perhaps they are populated too late [17:10] gary_poster, on re-reading that, let me clarify it: "stuff in cached properties is in cached properties because just keeping it in storm caches isn't sufficient since queries are not by ID" [17:11] gary_poster, anyway, I am really off now, hopefully you figure something out :) [17:11] danilos, but in my experiment I put the MP calculation within the code that calculates...OK :-) thanks for your help and thoughts. Have a great evening [17:54] umm... earthquake? [17:55] er? [17:56] we just had a 5.8 earthquake [17:56] http://maps.google.com/maps?f=q&hl=en&q=37.8750,-77.9080%28M5.8+-+Virginia+-+2011+August+23+17:51:03+UTC%29&t=h&z=7&iwloc=A [17:56] that was... exciting [18:03] http://earthquake.usgs.gov/earthquakes/recenteqsww/Quakes/at00lqe6x3.php [18:06] wow, benji, that was real close to you. felt nothing here [18:07] bac: yep, we're roughly 25 miles north of where the quake was centered [18:11] since the dog didn't wake up i'll assume it wasn't just me being inattentive [19:06] benji: http://jmckinley.posterous.com/dc-earthquake-devastation [19:07] bac: heh; that's about the extent of it; my worst damage is that the (rather large) fountain we have haning by our front door is all crooked now [19:08] well we have a big crack in the ceiling in the kitchen! [19:08] but that showed up last week. [19:08] we have a very prescient kitchen [19:08] * bac is still confused. a 17 year old house shouldn't be settling [19:09] yeah, that's kind of surprising; I wonder if there is an erosion problem. Do you live on an incline? [19:43] eek on earthquake [20:06] * bac chrs... [20:19] wow RT was full of spam [20:22] yep, I cleared some of it out but ran out of time (and energy) [21:10] * gary_poster didn't make up for being away for the doctor during my time [21:10] unfortunately my second doctor appt this week does not conflict with CHR :-P [21:18] * bac learns about +recipes