/srv/irclogs.ubuntu.com/2011/10/30/#launchpad-dev.txt

=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
cjohnstonany devs around? Have an issue between LP and summit that we need some assistance figuring out please.14:41
lifelesscjohnston: shoot19:27
nigelblifeless: ohai. We're having an issue wwith the https://blueprints.launchpad.net/sprints/uds-p/+temp-meeting-export thingy.19:28
cjohnstonyay!19:28
cjohnstongo nigelb!19:28
cjohnstonits like nigelb has my nick on highlight19:28
nigelbThe subscribers listed in https://blueprints.launchpad.net/linaro/+spec/linaro-summits-server-1 don't show up on the meeting export page :(19:29
lifelessshould really switch to using API's :)19:38
cjohnstonlifeless: feel free to join our session on wednesday ;-)19:39
nigelblifeless: I don't think there's an appropriate API for this.19:40
nigelbIf there is, I'm willing to make the switch.19:40
lifelesscjohnston: if its in your afternoon, I may; I'm not at UDS though19:40
cjohnstonIt's at noon EST IIRc19:40
lifelessnigelb: just export sprints on the API; there doesn't need to be a tailored one19:40
nigelbHrm. How do I get subscribers to a BP on the API?19:41
* nigelb looks at docs19:41
lifelessanyhow19:41
lifelessuhm, are all bp's faulty, or just that linaro one ?19:41
cjohnstonI believe there are atleast a couple19:42
lifelessworth checking what they have in common then, if its not all19:42
nigelblinaro one is the one we have a complaint about.19:42
lifelesse.g. unicode names (loïc)19:43
lifelessor being in two different sprints (uds-p and lcq4.11)19:43
lifelessobviously there is a bug somewhere; I presume you've checked the meeting export page to determine that the data is missing there, vs a summit bug ?19:43
nigelbYeah, we have :)19:43
nigelbsummit uses the meeting export page.19:44
nigelbAnd the data isn't there.19:44
lifelessok; and what another faulty blueprint ?19:44
nigelbcjohnston: Know of another faulty one?19:44
cjohnstonya19:45
cjohnstonlooking for it19:45
cjohnstonhttps://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-lc4.11-improving-ubuntu-upstream-relationship19:45
nigelbcjohnston: that one is fine breakage.19:45
nigelbNot accepted into uds-p19:46
nigelband hence won't show up in meeting export.19:46
cjohnstonits on the schedule nigelb19:46
nigelbwha.19:46
lifelessit shouldn't be :)19:46
nigelblifeless: sprints are exposed over the API? I don't see anything in API doc.19:46
cjohnstontuesday at 11 in grand serra h19:46
cjohnstonits approved for lcq4.1119:47
lifelesscjohnston: it may be manually locked in but the lp data is inconsistent19:47
lifelessnigelb: may need to be added...19:47
cjohnstonso if its approved for lcq4.11 doesnt that make it valid?19:47
nigelblifeless: Agreed. I'll work on that next cycle and get it done.19:47
nigelbcjohnston: meeting export is only run for uds-p, and not being approved there doesn't get it added into summit automatically.19:48
nigelbI think someone added at manually.19:48
lifelesseasy fix - get a uds driver to ack it19:48
nigelbJorge!19:48
cjwatsonI can do it19:48
nigelbcjwatson: Please do :)19:49
cjwatsondone19:49
nigelbThanks!19:49
cjohnstonthere are more that are only approved for lcq19:49
lifelessthat might be a distraction then, lets get that one acked and refresh summit, see if that fixes it.19:49
cjohnstonhow are they getting some of the participants but not all of them19:49
cjwatsonthough, um, it's pretty nasty that we have to basically make uds-p a superset of lcq4.1119:49
lifelesscjohnston: well, the next candidate is a unicode bug19:50
lifelesse.g. the first has loïc the second has Данило Шеган19:50
lifelessI'd hope we were well beyond such things, but ... you never know your luck in the big city19:51
nigelbI'm going to bet on a unicode bug.19:51
nigelbBecause I've seen 2 BPs with danilos in it that broke updates :)19:51
cjohnstonhttps://code.launchpad.net/~james-w/summit/usability/+merge/8010819:51
lifelesshave you guys filed a bug yet ?19:51
cjohnstonlifeless: nigelb ^19:51
cjohnstoni wonder if thats what that one is for19:51
nigelbNope.19:52
cjohnstonthe unicode part of it19:52
nigelbDjango's __unicode__ has nothing to do with what lifeless is talking about.19:52
cjohnstonk19:52
lifelesscjwatson: does man-db hit all man page inodes ?19:53
cjwatsonprobably yes, e.g. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=62094719:54
cjwatson(well, that's cat page directories)19:54
cjwatsonit'll skip input directories that weren't modified since the last run19:54
cjwatsoncjohnston: if the LP export is wrong then I doubt it's worth looking for relevant changes to summit ...19:55
cjohnstonthat was a new change cjwatson and i didnt really ever look at it, so i was just wondering if that could be it19:56
lifelesscjwatson: heh, so yah i was watching man db on my freshly booted desktop and going 'how can that be measurable time' :)19:56
cjwatsoncjohnston: but logically it can't have affected the content of the LP export19:56
* nigelb dons his LP community developer hat on mucks around in sprint code.19:56
lifelessnigelb: cjohnston: is there a bug for this yet ?19:56
cjohnstonhttps://bugs.launchpad.net/summit/+bug/883407 is our bug19:57
_mup_Bug #883407: Summit fails to show all my subscribed talks <Summit:Incomplete> < https://launchpad.net/bugs/883407 >19:57
cjwatsonlifeless: the thing that really kills man-db performance is forking subprocesses when it doesn't need to; I've been working (in my CFT) on adding better in-process support to libpipeline, sort of coroutine-like19:57
cjwatsonlifeless: microbenchmarks suggest that that accounts for the overwhelming majority of the runtime19:58
lifelesscjwatson: interesting; it must create bazjillions of subprocesses - my desktop is a very recent performance cpu w/ tonnes of RAM19:58
cjwatsonseveral per page19:58
lifelesscjwatson: that, or microbenchmarks are ignoring IO time (a common issue)19:59
nigelbFUUUU.19:59
nigelbI found the problem.19:59
cjwatsonhttp://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630799#2019:59
nigelb            if subscription.personID not in attendee_set:19:59
nigelb                continue19:59
nigelbIF someone is not registered on lp for uds-p sprint, their name doesn't show up in temp-meeting-export.19:59
lifelessof course19:59
cjwatsonlifeless: I doubt that's it, because the "fork roughly the right number of processes" microbenchmark I ran had runtime very similar to that of man-db itself20:00
nigelbThat yes, the people missing don't seem to be registered for uds-p.20:00
nigelbs/That/And20:00
lifelesscjwatson: ah, thats good corroboration20:00
cjwatsoner, wait, misremembering slightly20:00
nigelbYay for mucking in launchpad source :)20:00
cjwatsondepends how you run the microbenchmark, but at any rate details in that bug message20:00
nigelb*so* glad I started hacking on LP.20:00
lifelesscjwatson: thanks, yes I see20:01
cjwatsonin /usr/share/man, 'find -type f | xargs cat | zcat >/dev/null' => 2.5s, 'find -type f | xargs -n1 zcat >/dev/null' => 98s, mandb => 100s20:01
lifeless20K processes \o/20:01
cjwatsons/98/88/20:02
lifelesscjwatson: the libpipeline thing would also be a good point to introduce concurrency20:02
cjwatsonnow all I need is less work to do so that I can have time to finish this ;-)20:02
cjwatsonmaybe, but my gut feel based on the 'find -type f | xargs cat | zcat >/dev/null' test is that it doesn't need it20:03
lifelesscjwatson: won't help on cold cache situations much/at all, but would on hot20:03
lifelesscjwatson: depends how fast you want it to be :)20:03
lifelesscjwatson: on cold cache situations it may permit some extra disk concurrency if depth(disk tag queue) <= cpucount20:04
lifelessif thats false, you might overwhelm the scheduler20:04
cjwatsonI think shaving off 80% is eminently possible simply by avoiding the process storm, and that would bring it down to an entirely acceptable runtime in my book; I would rather not introduce concurrency when I don't need to20:04
lifeless:) shaving off 80% would be awesome, and your plan sounds eminently sensible20:05
lifelesscjohnston: nigelb: so, long story short - folk that haven't registered are silently dropped.20:05
nigelblifeless: Yep.20:05
lifelesscjohnston: nigelb: workaround is to get them to register if the are planning to attend remotely, I think there is a flag fo rthat20:05
nigelblifeless: No, no. The problem is the 2 sprints20:06
nigelbThe issue is because linaro has a sprint of its own and a lot of linaro folks are only registered on the linaro sprint.20:06
nigelbWe need to get them to register on both.  That's the least painful way.20:06
nigelbAnything else involves a non-trival summit code change less than 24 hours before UDS :D20:07
cjohnstonlifeless: http://summit.ubuntu.com/uds-p/2011-11-01/  why would riku-voipio show up in linaro server summit (grand sierra I at 9) but not Linaro Ubuntu LEB (grand sierra H at 11)20:07
lifelesscjohnston: summit has local overrides for stuff; check that hasn't been done first.20:08
nigelbIs that the one cjwatson just approved. They may be some manual thingies there.20:08
nigelbI don't think it was in +temp-meeting-export until about 10 minutes back.20:08
cjohnstonnigelb: he was already in that one.. so cjwatson just approving it wasn't relevent20:09
cjwatsonno, I approved https://blueprints.launchpad.net/linaro-ubuntu/+spec/linaro-platforms-lc4.11-improving-ubuntu-upstream-relationship20:09
nigelbYeah, that's the Linaro Ubuntu LEB20:09
cjwatsonoh right20:09
nigelbcjohnston: It is. Someone manually added the BP and probably a few relevant people.20:09
nigelbFairly sure ricardo is a track lead.20:09
cjohnstonadded releevant people where? in summit? because he wouldn't get imported on that one if he isnt an attendee for uds-p20:10
nigelbhe's a summit attendee.20:10
nigelbBut if the BP and its people were manually imported...20:10
lifelessthis is why having summit manually override is nuts ;)20:10
lifelesseither move the data store to summit20:11
lifelessor enhance LP to directly do what you want20:11
lifelessthat will avoid all this skew20:11
lifelesswhich completely confuses folk :)20:11
nigelbI'll take option 2 :)20:11
nigelbwell, the problem is, people don't trust summit20:11
nigelband override stuff t work20:11
nigelbInstead of *asking*20:12
lifelessnigelb: no reason the overrides can't be stored in LP20:12
nigelbI thought this stuff was not going to be maintained in the future?20:12
lifelesshuh?20:12
lifelessissuetracker says we're consolidating concepts, not eliminating functionality.20:12
nigelbah.20:13
nigelbIn wwhich case, I will bug fracis at the summit session and figureout something20:13
nigelbI'm definitely will to put the LP work in :)20:13
nigelb(Sigh, I wish I was there in person)20:13
lifelessit should be fairly straight forward: identify the places that summit is duplicating data from LP, and nuke with vengeance.20:13
nigelbheh20:14
lifelessthe cost of changing LP has dropped a bit already20:14
lifelessI'm confident the next 3-4 months will see another drop20:14
nigelb\o/20:14
nigelband 2 of us have commited patches to LP.20:14
nigelbSo, we're good on that front mostly.20:14
lifelessyah20:14
lifelessthe alternative is to move attendance totally out of LP20:15
lifelessor things like that20:15
lifelessnow, I don't have a preconceived 'right' solution20:15
nigelbHrm, I'm open to that as well.20:15
lifelessmy criteria are:20:15
nigelbJust import relevant people from BPs. Not from sprint attendance only.20:15
lifeless - no duplication once its done (mirroring is ok, but duplicate isn't)20:15
lifeless - LP isn't left with stub functionality: what remains should make sense on its own [or with summit as a blessed official addon that anyone can run]20:16
lifelessMy *suspicion* is that moving session times into LP, to permit LP storing pinned sessions, is the simplest change.20:17
nigelb..20:17
lifelessSeparately we need to address this two-sprint thing.20:17
nigelbYou'll let us do that? :)20:17
lifelessnigelb: why wouldn't I ?20:17
nigelb\o/20:17
lifelessnigelb: LP was *born* to support Ubuntu20:17
nigelbI will completely jump with joy if we can get that done :)20:17
lifelesspics or it didn't happen20:18
lifeless:)20:18
nigelbheh20:18
nigelbTo be honest, I thought those bits were not changeable from launchpad end.20:19
lifelessok, well I'm going to put nose to the grindstone and put this amqp oops workaround in place20:19
nigelbThis opens up a whole new avenue of possiblities.20:19
lifelessnigelb: Ok, so big picture and little picture.20:19
lifelessbig picture: we don't want to add random new features to LP without them tying in sensible. Just because it needs doing doesn20:19
lifeless't mean it needs doing the LP DB20:19
lifeless -> Look at the results tracker, separate DB, will become part of LP UI soon though.20:20
lifeless -> fixing an existing feature is not adding a random new feature.20:20
nigelbRight.20:21
lifeless -> supporting Ubuntu is a primary role for LP [explicitly Ubuntu in addition to its broader role of supporting the free software community]20:21
lifelesslittle picture:20:21
lifeless -> summit and lp sprints have overlapping models that leads to skew and confusion20:21
lifeless -> this is a problem20:21
lifeless -> lets fix :)20:22
nigelbIt took me half a day to track this down. I've worked on both summit *and* LP.20:22
nigelbSo, yes. Skew and confusion indeed.20:22
lifelessone approach is to let LP store the entire model20:22
lifelessanother approach is to remove from LP the parts of the model that are maintained by summit20:22
nigelbMost of wwhat summit does is mirroring.20:23
nigelbBut summit lets overriding of stuff. Like creating a meeting without a BP.20:23
lifelessyes, but ELOCALOVERRIDES20:23
nigelbExactly.20:23
lifelessso I think if we say 'LP sprints are not able to represent what happens at UDS', thats fairly clearly a defect in LP20:23
nigelbIndeed.20:24
lifeless'What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer. '20:27
lifeless*interesting*20:27
lifelesswgrant: when you are around, I'd like that incremental review.21:55
lifelesswgrant: other than this weird rosetta isolation thing, i think the branch is GTG21:55
wgrantlifeless: Sure, will look shortly.21:55
* nigelb blinks.21:56
nigelbOh. Its a Monday.21:56
nigelbRight.21:56
lifelessI'm thinking of making TestCase.setUp create an oops-message with the testid21:57
lifelessI think that might make debugging some of these things easier21:58
wgrantWhat do you mean?21:58
lifelessErrorReportingUtility.oopsMessage('test %s' % (self.id(),))22:03
wgrantlifeless: Ah!22:04
wgrantHmm, perhaps.22:04
wgrantlifeless: Could you please explananlyze https://pastebin.canonical.com/55125/ on qastaging?22:05
wgrant1-2ms on DF, 400-600ms on qas.22:05
wgrantAt least that's what ++profile++sql says.22:05
lifeless4.5ms22:06
lifeless6.1ms 'total runtime'22:06
lifelessah22:06
wgrantPerhaps ++profile++sql is buggy.22:06
lifelesswgrant: when you say 1-2ms on DF22:07
lifelesswgrant: are you executing the query, or the explain analyze ?22:07
wgrantexplain analyze22:07
lifeless ?column?22:07
lifeless----------22:07
lifeless        122:07
lifeless(1 row)22:07
lifelessTime: 745.439 ms22:07
lifelesswhen I execute it22:07
lifelesstry executing it22:07
wgrantStill lightning fast.22:07
lifelessexplaining is lightning fast, executing is slow22:07
wgrantWell, 8ms, but still fast.22:07
lifelesson qas22:07
wgrantOdd.22:07
lifeless                                                     Index Cond: ((public.teamparticipation.team = public.distribution.owner) AND (public.teamparticipation.person = 21997))22:08
wgrantAny idea how that could be?22:08
lifeless Total runtime: 0.754 ms22:08
lifeless(64 rows)22:08
lifelessunreported time is usually either startup or wire-transmission overheads22:08
wgrantBut the returned value is [(1,)]...22:08
lifelessI didn't say it fitted ;)22:08
lifelesswhat does this intend to do ?22:10
wgrantIt's the legacy bug visibility query.22:11
lifeless0.9ms to profile on wild22:11
wgrantprofile == explanalyze?22:12
lifeless386ms to execute22:12
lifelessyes22:12
wgrantWTF22:12
lifelessthat was local machine so no networking etc possible22:12
wgrantWith \timing on, how long does the EXPLAIN ANALYZE take?22:14
lifeless6.1ms22:14
wgrantNot the time it gives, but the time it takes?22:14
lifeless6.1ms!22:15
wgrantBah.22:15
lifelessRepeating the question presumes I misunderstood22:15
lifelessI have timing on always ;)22:15
wgrantSo it's not some pathological query parsing or anything. it's just insane.22:15
lifelessplease tell me you're not going to put this into every bug query ?22:15
wgrantWhat about https://pastebin.canonical.com/55126/?22:17
wgrantThat's a query that's already there.22:17
wgrantVery similar.22:17
lifelessI'd factor out the TP stuff into a CTE22:17
wgrantIs it still slow?22:17
wgrantSo would I, but this whole thing is being deleted shortly.22:17
lifeless580ms22:17
lifeless0.7ms to profile it22:17
wgrantexplanalyze gives a quick plan, though?22:18
wgrantRight.22:18
wgrantWTF?22:18
wgrantThis making our bug pages 600ms slower than they have any justification for being.22:18
lifelessthat should be an explainanalyze22:18
lifelessbah22:18
lifelessa union all22:18
lifelessbut it doesn't correct the issue22:18
wgrantThe first query I gave you is now on qastaging. I landed it because it performed really well on DF :)22:18
wgrantRight, I didn't write those conditions.22:18
wgrantThe second query is an existing one which is on prod.22:19
wgrantThe first is the condition for the second one being reused to implement Bug.userCanView.22:19
lifelessdropping the bug columns doesn't help22:20
wgrantSo, any leads on the slowness? I can't reproduce on DF.22:20
wgrantHmm.22:20
lifelessfiddling22:20
wgrantThanks.22:20
lifeless43ms do ?22:25
lifelesshttps://pastebin.canonical.com/55127/22:25
wgrantThat's still crap, but how?22:25
lifelessCTE22:25
wgrantIndeed, that is faster...22:26
lifeless12ms hot22:26
wgrantUnsurprising.22:26
wgrantBut how?22:26
lifelessCTE, you can ignore the inner limit 1, thats not needed22:26
wgrantWell, not surprising that it's faster.22:26
wgrantBut I didn't bother optimising because it was sub-ms.22:26
wgrantExcept that it's not.22:26
wgrantBecause EXPLAIN ANALYSE is a lie.22:27
wgrantlifeless: https://pastebin.canonical.com/55128/22:28
lifelessso, there is probably an execution bug in pg22:28
lifelessand its probably fixed in 9.322:28
wgrant9.3 is out!?22:28
wgrantYou mean 9.1?22:28
lifelessno22:28
lifelessThat was called humour22:29
wgrantHah22:29
lifeless181ms cold22:29
lifelessand hot22:29
wgrantThis is the real one that will be executed for !~launchpad on prod.22:29
wgrantHmm.22:29
wgrantNot sure if worth rolling back...22:29
lifelesshah22:31
lifelessI fluffed my edits22:31
lifelesscheck the bugsub clause22:31
wgrantHeh22:31
lifelessstill 12ms22:31
wgrantDoes that fix it back to the old speed?22:31
lifelessreturns 0 rows22:32
wgrantThat's wrong.22:32
wgrantI have a subscription through a team.22:32
lifelesson qas ?22:32
wgrant~ubuntuone, in particular.22:32
wgrantYes.22:33
lifeless13ms 1 row22:33
lifelessfixed22:33
wgrantDamn.22:33
lifelesshttps://pastebin.canonical.com/55129/22:33
lifeless4ms for the new one CTEified22:34
wgrant"new one" == only two things being unified?22:34
lifelessyes22:34
wgrantThanks.22:34
lifeless4 with and without UNION ALL22:34
lifelesshttps://pastebin.canonical.com/55130/22:35
wgrantSo, this is a bit of a worry :/22:36
lifelesswhats the current live query ?22:36
wgrantBug.userCanView on prod is currently an undefined number of queries, depending on caching.22:36
wgrantMight get a ++oops++ to find out.22:37
lifelessremember that we don't cache across requests22:37
wgrantYes.22:37
wgrantBut I mean it's not obvious from the method what will need to be done.22:38
lifelesssure22:38
lifelessok, I'm going to go start doing limited test run bisects to find this rosetta issue22:38
wgrantGood plan.22:38
lifelessits reported test 12751. I think I'll power-of two working up, rather than shrinking22:40
lifelesswow22:56
lifelesspickledb is -not- what you'd think it is. Way to confuse.22:56
lifelessit may still be terrible, but its a different sort of terrible22:57
wgrantlifeless: How is https://pastebin.canonical.com/55131/ on prod? sub-ms on DF, 340ms in ++oops++.23:02
lifeless253ms23:04
lifeless160 repeated23:04
lifeless250ms third time23:05
wgrant1 row?23:05
lifelessso very variable23:05
lifelessyes23:05
wgrantWTF?23:05
lifelessI have drop lynne up the street23:06
wgrantLet's run LP on mawson :)23:06
wgrantk23:06
wgrantWell.23:08
wgrantIndeed, that bug renders in 0.81s on DF.23:08
wgrant2.5s on prod.23:08
wgrantAh, because I'm an admin on DF, so it skips the bits of the queries that make everything slow except not.23:09
wgrant(but the queries I was timing before were from qastaging)23:10
wgrantHah.23:10
wgrant4 extra queries, but still 0.88s.23:10
wgrantSo, DF is three times faster than prod.23:11
wgrantprod is pretty fucked.23:11
pooliehi all23:19
wgrantMorning poolie.23:20
poolieso that was a bit of an anticlimax that the buildd upgrades did not fix all the memory issues23:20
pooliewe can investigate more23:20
poolielocally23:20
pooliei hope a second attempt won't take so long23:21
lifelesswgrant: thats interesting23:22
lifelesswgrant: unpleasant. But interesting.23:22
lifelesswgrant: I suspect table/index bloat23:22
wgrantThat would be the leading theory, I expect, but the high variance suggests that it may be something else.23:23
lifelessnot really23:23
lifelessI have a unified theory ;)23:23
lifelesstable bloat means more pages are consulted23:23
lifelessconsulting a *page* involves a lock23:24
wgrantAh.23:24
wgrantYeah.23:24
lifelesspg lock scalability is known suboptimal (and under improvement)23:24
wgrantWe really should upgrade to 9.1 at some point.23:24
wgrantI guess that can happen soon after slony1-2.23:24
lifelessincreasing the page count by (say) 100 fold would increase lock overhead by 1000 fold23:24
lifelessbah 100 fold23:24
lifelessanyhow, that would provide lots of room on a busy server (e.g. prod) for happening to not contend on those page locks23:25
lifelessthe CTE causes a more efficient scan of the persons rows after after that stops consulting that table at all23:26
wgrantYes.23:26
lifelessthats my theory anyhow23:26
wgrantQuite reasonable.23:26
wgrantSo, sounds like we should upgrade postgres and continue fixing LP to not be ORMish.23:26
lifelessits a 72MB table23:26
wgrantWhat is?23:27
wgrantTP?23:27
lifelessyes23:27
lifelessconceivable packable during FDT23:27
wgrantSurely that's all going to be hot.23:27
lifelessoh it will be23:27
lifelessno disk IO at all I expect23:27
pooliehuwshimi: hi, maybe we can have a talk about markdown together some time23:29
lifelessbah, can't find the blog post23:33
lifelesswgrant: anyhow, fseek() takes out a kernel lock23:33
lifelesswgrant: and postgresql fseeks(-1) to get the size of files it 'reads'23:33
pooliea per-file lock, or a larger scope than that?23:35
lifelesspoolie: I don't remember the analysis23:35
wgrantlifeless: Can you check bloat on tables?23:38
lifelessyes, if I poke around a bit23:38
wgrant00113-00338@SQL-main-slave SELECT DISTINCT ON (Person.name, BugSubscription.person) Person.account, Person.creation_comment, Person.creation_rationale, Person.datecreated, Person.defaultmembershipperiod, Person.defaultrenewalperiod, Person.displayname, Person.hide_email_addresses, Person.homepage_content, Person.icon, Person.id, Person.logo, Person.mailing_list_auto_subscribe_policy, Person.merged, Person.mugshot, Person.name, ...23:38
wgrant... Person.personal_standing, Person.personal_standing_reason, Person.registrant, Person.renewal_policy, Person.subscriptionpolicy, Person.teamdescription, Person.teamowner, Person.verbose_bugnotifications, Person.visibility, BugSubscription.bug, BugSubscription.bug_notification_level, BugSubscription.date_created, BugSubscription.id, BugSubscription.person, BugSubscription.subscribed_by FROM Bug, BugSubscription, Person, TeamParticipation ...23:39
wgrant... WHERE (Bug.id = 597155 OR Bug.duplicateof = 597155) AND BugSubscription.bug = Bug.id AND TeamParticipation.person = 21997 AND (TeamParticipation.team = BugSubscription.person) AND (Person.id = TeamParticipation.team) ORDER BY Person.name23:39
wgrantFail.23:39
wgrantDidn't meant to paste all that.23:39
wgrantBut anyway, that's 225ms on prod, 4ms on DF (from ++oops++)23:39
lifelessoh cool23:39
lifelessindex-only scans is in tip23:39
wgrantNice.23:40
StevenKwgrant: When I hung around on #linux/ircnet years ago -- we had a term for that; "slammermaus": a mouse that randomly selects great gobs of text and pastes it into the channel.23:40
wgrantAnd then the big query I asked about second ("SELECT BugTask.blah ... WHERE (massive union)") is 420ms vs 5ms,23:41
wgrantThose two queries make up half the render of time of the bug when prod is hot.23:42
lifelesswgrant: http://pgsnaga.blogspot.com/2011/10/index-only-scans-and-heap-block-reads.html23:42
wgrantAnd there's still another 200ms I need to track down.23:42
wgrantAhhh very nice.23:43
wgrantAlthough this is going to make vacuums even more critical for performance.23:44
lifelessopen question is if autovaccuum manages23:44
wgrantExactly.23:44
wgrantlifeless: I guess it's good that we can reproduce this on qastaging.23:47
lifelesstes23:48
wgrantWithout those two bad queries, we could reasonably easily get such bugs below 0.5s.23:49
wgrantWhich is still terrible, but not as bad as it is now.23:49
wgrantAnyhow.23:49
wgrantI might land that CTE.23:49
wgrantWhich will fix one of the queries.23:49
wgrantAnd add another instance of it.23:49
* lifeless headdesks23:58
lifelessrunning test_rosetta_branches_script before $1_oops causes the too-many-oops situation23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!