[11:38] <gary_poster> hey gmb, when you have time, I'd love reviews of https://code.launchpad.net/~gary/launchpad/bug724025/+merge/72484 and https://code.launchpad.net/~gary/launchpad/sqlprofiler/+merge/72491.  The second one is big (1168 lines).  I can divide it in two if you like; take a glance and let me know then if you want me to.
[12:10] <gmb> gary_poster: Sure. I've got an oversized branch to review for Henning first, but I'll take a look once that's done.
[12:10] <gary_poster> cool thanks gmb
[12:21] <gary_poster> note to self: make schema != make harness :-/
[12:26] <gary_poster> In a novel twist, I'll announce the call a few minutes before it is supposed to happen.
[12:27] <gary_poster> bac benji danilos (danilo_) gmb, call in 3
[12:27] <gmb> k
[12:27] <bac> o
[12:27] <gary_poster> :-)
[12:27] <benji> r
[12:27] <danilos> n
[12:27] <gary_poster> e
[12:27] <gary_poster> a
[12:28] <benji> it's like there was a cat-on-keyboard epidemic
[12:28] <gary_poster> If I had only gotten my "a" in faster, it would have been "korea" -> "korean"
[12:29] <gary_poster> boys are going to their first tae kwon do lesson today, so it is on the mind
[12:29] <bac> i thought you just didn't know how to spell your eye parts
[12:29] <gary_poster> heh
[12:29] <bac> cool, make them call you "dear leader" from now on
[12:30] <gary_poster> heh, I suspect they are south korean teachers, though maybe not
[13:26] <gmb> gary_poster: bug724025 branch approved.
[13:26]  * gmb starts on the other one.
[13:27] <gary_poster> sweet, thanks gmb.  gmb, I'll probably head out in 20 if that makes a difference to your planning
[13:27] <gmb> gary_poster: It shouldn't. I can always queue up questions ;)
[13:27] <gary_poster> heh, ok cool thx
[13:50] <gary_poster> danilos, you agree that for the purpose of getting merge proposals for all the bugtasks of a given bug, I need the IAllBranches branch collection, right?  I don't think I can narrow it down any more.  (I will accept "I have no idea" as an answer ;-) )
[13:56] <gary_poster> it reduced queries by 200 (1998 -> 1752) but no change on actual time of page.
[13:56]  * gary_poster to doctor
[14:25] <bac> danilos could you use your alleged super powers to run http://pastebin.ubuntu.com/673160/ on qastaging?
[14:26] <bac> danilos: actually this would be better:
[14:26] <bac> select person.name, emailaddress.email, account.* from person, emailaddress, account where emailaddress.account=account.id and account.status=10 and person.account=account.id and emailaddress.email like '%@list%';
[14:47] <bac> nm danilos i'll ask a losa
[15:05]  * gmb -> running an errand; bbiab
[15:09] <danilos> bac, oh, sorry, deep into code
[15:11] <danilos> gary_poster, sorry to you as well :) generally, you'd provide a separate branch collection for a bug which fetches all branches for a bug, that should be simple
[15:12] <danilos> bac, do you still need stuff to happen on staging?
[15:12] <danilos> gary_poster, perhaps you need to increase storm_cache_size as well, though
[15:14] <bac> danilos, no, thanks
[16:27] <gary_poster> doctor confirms that I'm still alive, as far as he can tell.
[16:28] <gary_poster> danilos, separate branch collection for bug, oh ok.  I'll see how to do that.  I did increase storm cache size to 10000 before I started my experiments
[16:45] <danilos> gary_poster, well, the idea is to have something like "getAllBranchesForBug()" where you could pass in eager_load when you want it to fetch things like merge proposals and such
[16:46] <danilos> gary_poster, though, if you've already got a list of branches for a bug with a single query, and you pass those branches into getMergeProposals(for_branches=..., eager_load=True), then there's not much you will really get from that
[16:47] <gary_poster> danilos, right, I do.
[16:47] <gary_poster> This is what I added:
[16:47] <gary_poster>         list(getUtility(IAllBranches).getMergeProposals(
[16:47] <gary_poster>             for_branches=linked_branches, eager_load=True))
[16:47] <gary_poster> code already calculates linked_branches pretty efficiently
[16:47] <danilos> gary_poster, yeah, that looks good then
[16:47] <gary_poster> danilos, I still saw calls I didn't expect, perhaps naively.  Lemme get a few...
[16:48] <danilos> gary_poster, maybe it's about some other stuff
[16:48] <danilos> gary_poster, sure
[16:48]  * danilos dines while gary is at it :)
[16:48] <gary_poster> :-)
[16:48] <gary_poster> danilos, I see these:
[16:49] <gary_poster> mm, going to pastebin instead
[16:49] <danilos> sure :)
[16:49] <gary_poster> danilos, this is one set of repeated queries.  I get these for each MP, I think.
[16:49] <gary_poster> http://pastebin.ubuntu.com/673239/
[16:50] <gary_poster> I wonder if something is running was SQL or otherwise explicitly bypassing the cache
[16:51]  * gary_poster going to get tracebacks; this takes > 1 minute
[16:51] <danilos> gary_poster, interesting, I expected 1-3 and 5-7 (inclusive) to not be there
[16:51] <gary_poster> that's what I had hoped too :-)
[16:52] <gary_poster> it is interesting that it still cut 250 queries out
[16:53] <danilos> gary_poster, yeah, that's why I thought it might have been the cache size thing
[16:54] <gary_poster> It's at 10000 though
[16:54] <gary_poster> I suppose it could be bigger
[16:54] <danilos> gary_poster, yeah, understood (just explaining where I was coming from)
[16:54] <gary_poster> what's production, do you remember off-hand?
[16:54] <danilos> well, production is 10k
[16:54] <gary_poster> yeah ok
[16:54] <danilos> so no need for bigger, this should be good enough
[16:56] <danilos> gary_poster, I don't have any suggestions on what to look for though :/
[16:56] <gary_poster> danilos, that's because you haven't seen my new super cool ++profile++supertrace with nitro burning funny cars
[16:56] <danilos> gary_poster, do note that these values are usually fetched from cached properties associatedProductSeries and associatedSourcePackages on IBranch or something, so perhaps look if these are being used properly
[16:56] <danilos> gary_poster, heh, true, true, I'll have to try that one out
[16:57] <gary_poster> ++profile++sqltrace, that is
[16:58] <danilos> gary_poster, and eager_load in getMergeProposals fills the caches for the above properties, so something going wonky there is the only thing I can think of
[16:58] <gary_poster> interesting, I wonder if it is cache exhaustion...
[16:58] <gary_poster> I have 160 mps in my example...
[16:58] <danilos> I've tested with 300 for +activereviews page
[16:59] <danilos> though, you have bugtasks and such with all their data and objects
[16:59] <danilos> (though, I think I only used something like 2000 for the cache size)
[16:59] <gary_poster> before, that led to three sets of 160 repeated queries, two of which were related to branches.
[16:59] <gary_poster> Now some of them are 118
[16:59] <danilos> well, it should be easy to try upping the cache size to a gazillion or something like that :)
[16:59] <danilos> just to rule that out
[17:01] <gary_poster> yeah, I'll try that
[17:01] <gary_poster> while I'm looking at the results though...
[17:01] <gary_poster> Only this line is repeated for awhile:
[17:01] <gary_poster> SELECT CodeReviewVote.branch_merge_proposal, CodeReviewVote.vote_message,
[17:01] <gary_poster> (etc.)
[17:02] <gary_poster> then it joins in with ...
[17:02] <gary_poster> SELECT SeriesSourcePackageBranch.branch, SeriesSourcePackageBranch.date_created,
[17:02] <gary_poster> (etc.)
[17:03] <gary_poster> and the Product Series branch, and the BranchMergeProposal
[17:04] <gary_poster> I wonder if, instead of relying on the cache, I could use the getMergeProposals call directly for what I want...
[17:06] <danilos> well, you'd have to rework the code logic for that, but sure you could
[17:08] <gary_poster> yeah
[17:09] <gary_poster> no change in sql calls after upping the cache to 100K (1752) so that's out.  Looking deeper.
[17:09] <danilos> gary_poster, btw, all things kept in cached properties are there because the queries are not by ID (eg. by source_branch=... or something), so perhaps they are populated too late
[17:10] <danilos> gary_poster, on re-reading that, let me clarify it: "stuff in cached properties is in cached properties because just keeping it in storm caches isn't sufficient since queries are not by ID"
[17:11] <danilos> gary_poster, anyway, I am really off now, hopefully you figure something out :)
[17:11] <gary_poster> danilos, but in my experiment I put the MP calculation within the code that calculates...OK :-) thanks for your help and thoughts.  Have a great evening
[17:54] <benji> umm... earthquake?
[17:55] <bac> er?
[17:56] <benji> we just had a 5.8 earthquake
[17:56] <benji> http://maps.google.com/maps?f=q&hl=en&q=37.8750,-77.9080%28M5.8+-+Virginia+-+2011+August+23+17:51:03+UTC%29&t=h&z=7&iwloc=A
[17:56] <benji> that was... exciting
[18:03] <bac> http://earthquake.usgs.gov/earthquakes/recenteqsww/Quakes/at00lqe6x3.php
[18:06] <bac> wow, benji, that was real close to you.  felt nothing here
[18:07] <benji> bac: yep, we're roughly 25 miles north of where the quake was centered
[18:11] <bac> since the dog didn't wake up i'll assume it wasn't just me being inattentive
[19:06] <bac> benji: http://jmckinley.posterous.com/dc-earthquake-devastation
[19:07] <benji> bac: heh; that's about the extent of it; my worst damage is that the (rather large) fountain we have haning by our front door is all crooked now
[19:08] <bac> well we have a big crack in the ceiling in the kitchen!
[19:08] <bac> but that showed up last week.
[19:08] <bac> we have a very prescient kitchen
[19:08]  * bac is still confused.  a 17 year old house shouldn't be settling
[19:09] <benji> yeah, that's kind of surprising; I wonder if there is an erosion problem.  Do you live on an incline?
[19:43] <gary_poster> eek on earthquake
[20:06]  * bac chrs...
[20:19] <bac> wow RT was full of spam
[20:22] <benji> yep, I cleared some of it out but ran out of time (and energy)
[21:10]  * gary_poster didn't make up for being away for the doctor during my time
[21:10] <gary_poster> unfortunately my second doctor appt this week does not conflict with CHR :-P
[21:18]  * bac learns about +recipes