/srv/irclogs.ubuntu.com/2010/12/04/#launchpad-dev.txt

cody-somervillewgrant, Although it is unfortunate for several reasons that we have to support obsolete series, I'm curious why you care as it isn't you that has to do the work to support the obsolete release. :P00:41
wgrantcody-somerville: Publisher performance.00:42
wgrantcody-somerville: We can save a lot of time if we have to publish 6 releases, not 14.00:42
wgrantWe should probably be querying for dirty series anyway, but it's a whole lot easier to just skip obsolete ones.00:43
cody-somervillewgrant, sounds like a scaling issue.00:46
wgrantcody-somerville: It is.00:47
wgrantBut I'd also like to reject uploads to obsolete series.00:47
lifelesswe has a lot of them00:47
lifelessI can haz perf problems00:47
wgrantI now haz full publisher log, so I can actually see where the problems lie.00:47
lifelesshttps://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html is updated00:49
wgrantNeed simple eager loading :(00:49
lifelesswgrant: well, I've specced it now.,.. you can write if you like :)01:01
wgrantHmm.01:04
wgrantI'd really like to see how much faster the PPA publisher is if we turn off cache flushing.01:04
lifelesselmo: have you seen http://blog.benstrong.com/2010/11/google-and-microsoft-cheat-on-slow.html ?01:05
elmoflacoste: I wasn't arguing against the desiloization in general or arguing for soyuz retaining specific people - as I think I've said to you and certainly to lifeless before now, I'm all for desiloization, I think it's important.  what I was reacting to was the idea that soyuz is no different to e.g. bugs or jml's "be bold, be confident, do what the fuck you want" (I may be paraphrasing) and wanted to reinforce the idea that soyuz is not like the rest of L01:24
elmolifeless: I hadn't - that's interesting01:24
lifelesselmo: your long statement cut off at 'the rest of L'01:25
elmolala01:25
elmoP - it a)  doesn't have the safety net of the browser in terms of impact on systems and b) can impact orders of magnitudes more  systems in an automated way compared to other LP systems that also don't have the browser safety net (e.g. code)01:25
elmoflacoste: ^--01:25
elmoman, now i'm going to be reading this tcpm list for hours01:33
lifelesselmo: sorry!02:00
lifeless=== Top 10 Time Out Counts by Page ID ===02:06
lifeless    Hard / Soft  Page ID02:06
lifeless     168 / 6026  Archive:+index02:06
lifeless      94 /  256  POFile:+translate02:06
lifeless      90 /  269  BugTask:+index02:06
lifeless      19 /  268  Distribution:+bugs02:06
lifeless      16 /  118  ProjectGroupSet:CollectionResource:#project_groups02:06
lifeless      15 /  282  Distribution:+bugtarget-portlet-bugfilters-stats02:06
lifeless      15 /    2  ProjectGroup:+milestones02:06
lifeless      10 /   21  DistroSeries:+queue02:06
lifeless      10 /   19  DistroSeriesLanguage:+index02:06
lifeless       8 /    0  ProductSeries:+edit02:06
jcsackettlifeless, what exactly is the difference between a hard and a soft timeout? is it where it ran up against our time limits vs when the server actually gives up and dies?02:07
jcsackettthat may be a stupid question, but i just realized i don't actually know.02:08
lifelesssoft timeouts are requests that complete ok, longer than X where X is configured in lp-production-configs02:12
lifelesshard timeouts are requests that trip the timeout code (grep for RequestExpired) and get interrupted02:12
lifelessthe default hard timeout is currently 15 seconds02:13
lifelessand some page ids have higher ones, while we recover from pg8.402:13
wgrantlifeless: Any chance you could throw one of the recent DistroSeries:+queue timeout query logs my way?02:30
lifelesswgrant: sorry no03:45
lifelesswgrant: the grouping in the report means that high frequency things crowd out the details for lower ones03:45
lifelesswgrant: so I don't have a +queue OOPS that is a timeout.03:46
lifelesswgrant:  https://bugs.launchpad.net/soyuz/+bug/276950 is the bug03:46
_mup_Bug #276950: DistroSeries:+queue Timeout accepting many packages queue page <queue-page> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/276950>03:46
lifelesswgrant: and it has a one month old bug03:46
lifelessone month old OOPS03:46
wgrantlifeless: Ah, OK.03:49
wgrantThanks.03:49
lifelessI'll pull out data from it in a sec03:49
lifelesswgrant: updated03:54
wgrantlifeless: Thanks.03:55
lifelesswow03:58
wgrantHm?03:58
lifelesshttps://bugs.launchpad.net/soyuz/+bug/276950/comments/1203:59
_mup_Bug #276950: DistroSeries:+queue Timeout accepting many packages queue page <queue-page> <timeout> <Soyuz:Triaged> <https://launchpad.net/bugs/276950>03:59
wgrantUh.04:00
wgrantWhy didn't it use packageupload__distroseries__status__idx?04:01
wgrantOr does it really need a full (archive, distroseries, status) index?04:01
wgrant(that should be the index anyway, but I still think it should have been able to get some benefit from the existing one)04:02
lifelessit may be outdated statistics04:02
lifelessits the order by04:03
lifelessyou need an index with the order key, to use the index04:03
lifelessSELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM PackageUpload WHERE packageupload.distroseries = 103 AND  packageupload.archive IN (1, 534) AND packageupload.status IN (0) LIMIT 31 OFFSET 0;04:03
wgrantInteresting.04:03
lifeless(1 row)04:03
wgrantDespite there being only one row.04:04
lifelessTime: 1.601 ms04:04
lifelessstandard sql rules04:04
wgrantSo I guess we want an (archive, distroseries, status, id) index?04:04
lifeless-id04:04
lifelessbut yeah04:04
lifeless Limit  (cost=0.00..90.66 rows=31 width=36) (actual time=58.704..58.709 rows=1 loops=1)04:05
lifeless   ->  Index Scan using packageupload__distroseries__status__idx on packageupload  (cost=0.00..8946.13 rows=3059 width=36) (actual time=58.701..58.704 rows=1 loops=1)04:05
lifeless         Index Cond: ((distroseries = 103) AND (status = 0))04:05
lifeless         Filter: (archive = ANY ('{1,534}'::integer[]))04:05
lifelesswithout the order by04:05
wgrantThat's a little better.04:05
lifeless1.5ms hot :)04:06
lifelessanyhow04:06
lifelesstheres another index that I created, you do archive, distroseries, status, id DESC04:06
lifelessor something like that04:06
wgrantThat's not the normal timeout, but it'll do for now.04:06
lifelesswe may be able to do04:06
lifelessshaving 1.5s of normal is WIN04:07
lifelesspackageupload__distroseries__status__idx" btree (distroseries, status)04:07
lifelesspackageupload__distroseries__status__idx" btree (distroseries, status, id DESC)04:07
lifelessmay be enough04:07
lifelessexperiment in your local machine04:07
lifelessdisable all scans04:07
wgrantHow do I disable scans?04:07
lifelesshttp://www.postgresql.org/docs/8.4/interactive/runtime-config-query.html#RUNTIME-CONFIG-QUERY-ENABLE04:08
wgrantThanks.04:09
lifelessso its set04:09
lifelessset enable_seqscan 004:09
lifelesswgrant: did that help?04:35
lifelesswgrant: also see http://www.postgresql.org/docs/8.4/static/sql-createindex.html04:36
lifelesslook for ORDER BY04:36
wgrantlifeless: Debugging Soyuz DB corruption issues at the moment.04:40
lifeless\o/04:40
lifelesslooking at this04:41
lifelessI'd do04:41
lifelessdistroseries, status, archive, id04:41
lifelessas the index, replace status__idx04:42
wgrantlifeless: I've tried a few combinations, but it will always use an explicit sort step. If I disable explicit sort steps, it reverts to using distroreleasequeue_pkey04:44
wgrantRegardless of sort direction.04:45
wgrantOdd.04:45
wgrantOh.04:45
wgrantThere we go.04:45
wgrant.. no.04:45
wgrantIt will not use the index.04:45
wgrantCan I deprioritise distroreleasequeue_pkey somehow?04:46
lifelesswell04:50
lifelessthe stats will play into what it uses04:50
wgrantI guess I really need to try on real data.04:50
wgrantRight.04:50
lifelesshey stub05:12
lifelessreckon you create an index on qastaging to test a query improvement for wgrant and I ?05:12
lifelesscreate index packageupload__distroseries__status2__idx on packageupload(distroseries, status, archive, id);05:12
lifelessthis is the query 'SELECT PackageUpload.archive, PackageUpload.changesfile, PackageUpload.date_created, PackageUpload.distroseries, PackageUpload.id, PackageUpload.pocket, PackageUpload.signing_key, PackageUpload.status FROM PackageUpload WHERE packageupload.distroseries = 103 AND  packageupload.archive IN (1, 534) AND packageupload.status IN (0) ORDER BY packageupload.id desc LIMIT 31 OFFSET 0;05:13
wgrantsinzui: The bugjam doesn't start for another 9 days. You don't need to start drowning us already :P05:18
stublifeless: yo05:43
stubCurrently that is a crap query (28 sec)05:46
wgrantIt wasn't always :(05:46
stubyer - surprised it isn't using the distroseries,status index05:47
wgrantExactly.05:47
wgrantThat should really be archive, distroseries, status nowadays, but still.05:47
stubWe are missing an index on archive05:47
stub(different problem)05:48
wgrantOh, true, I misread the fkey as an index.05:48
stubJust adding an archive, status index brings it down to 1.6 seconds05:50
stubOh... must be warm cache now. It isn't using that index ;)05:50
lifelessstub: it needs an index on id05:52
lifelessstub: because of the order byu05:52
stubyeah - warming up to that05:52
lifelessstub: thus the 4-element one I'm proposing05:52
stubIts always wanting to use the primary key index05:55
wgrantThat's what I found locally.05:58
wgrantI hoped real data would work better :(05:58
stubDefine 'better'. PG is sometimes smarter than we are when picking plans. It might be correct that this approach is faster with the real data05:59
stubOk... id needs to be first06:00
lifelessstub: ?!06:01
wgrant... what?06:01
lifelessstub: win06:01
stubLimit  (cost=0.00..518.75 rows=31 width=36) (actual time=0.061..92.017 rows=1 loops=1)06:01
stub   ->  Index Scan Backward using pu_test7 on packageupload  (cost=0.00..52209.53 rows=3120 width=36) (actual time=0.059..92.014 rows=1 loops=1)06:01
stub         Index Cond: ((distroseries = 103) AND (status = 0))06:01
stub         Filter: (archive = ANY ('{1,534}'::integer[]))06:01
stub Total runtime: 92.065 ms06:01
stub(5 rows)06:01
wgrantHow does that make sense?06:01
lifelessstub: whats the defn of that index?06:01
stubIt doesn't have to materialize the whole thing before truncating 'first 30 items'06:01
stubIt can pull suitably filtered rows straight off the index in order.06:02
wgrantAhh.06:02
lifelessstub: thats still way worse than if you drop the ORDER BY :(06:02
stubcreate index pu_test7 on packageupload(id, distroseries, status, archive);06:02
lifelessstub: 50* worse06:02
stubLooking at that, it doesn't bother using the index for archive06:03
lifelesswhat about06:04
stub(not surprising if those archives are big ones... it might get used for small archives so we can leave that column in the index)06:04
wgrantGiven the distribution that might not be unreasonable.06:04
lifelesscreate index packageupload__distroseries__status2__idx on packageupload(distroseries, status, archive, id desc);06:04
stublifeless: tried that one06:04
lifelessstub: with the DESC ?06:05
stub    "pu_test1" btree (archive, status)06:05
stub    "pu_test2" btree (archive, status, id)06:05
stub    "pu_test3" btree (distroseries, archive, status, id)06:05
stub    "pu_test4" btree (archive, status, id DESC)06:05
stub    "pu_test5" btree (distroseries, status, archive, id)06:05
stub    "pu_test6" btree (distroseries, status, archive, id DESC)06:05
stub    "pu_test7" btree (id, distroseries, status, archive)06:05
lifelesskk06:05
stubDESC shouldn't make a difference in this case - it can use the index backwards just as easily. If you were ordering by 'archive, id DESC' that would be different.06:05
lifelessyeah, theory n prac though06:06
stubYup. I always pick my theory after empirical results are in ;)06:06
stubI sound smarter that way.06:07
stubDo we need the index for this rollout?06:08
wgrantIt's not a recent regression.06:09
wgrantSo I'd imagine not.06:09
stubI can apply the patch live if we want to skip getting a patch on the production branch06:10
stubTomorrow though - already got some indexes building on production for translations06:10
lifelessno panic06:10
lifelessbut we should figure it out :)06:10
stubIf you put in a db patch, please add the index on archive too... I don't like having ON DELETE CASCADE foreign key reference without the index to support that quickly.06:11
stubShall I drop these indexes on qastaging now so we are qaing what we think we are qaing?06:11
lifelessstub: please06:15
lifelessstub: I don't know that we've got a good enough index yet.06:16
lifelessstub: for the archive one, tell wgrant what it should be, he can add a patch for that for your happiness :)06:16
stubCREATE INDEX packageupload__archive__idx ON PackageUpload(archive);06:16
stubI can't see how to make that index better06:18
stuband 91ms is pretty good06:19
lifelessunordered its 1.6ms :)06:19
stubYes, but that is pulling rows in disk order rather than randomly - far fewer pages to pull up off disk.06:20
stubOne block vs 30 odd.06:20
stubI can make an index that is used that runs slower ;-)06:21
lifelessoh?06:21
stubyer - you want the terms that filter the most rows on the left.06:22
lifelessbbiafew hours06:23
stubBut with timings this low, we are dealing with shape of the data for those parameters06:23
stubDitto06:23
sinzuiwgrant, sorry. This is not about the bugjam. I want every registry bug tagged with something I can find before the bugs are moved to launchpad.15:11
jelmer'morning15:21
lifelesssinzui: I think its great that you're tagging separately from 'closing'... I'm expecting the bugjam to close bugs it shouldn't really.18:08
lifelesstheres not much nuance in the message :)18:08
sinzuilifeless, I filter all bug mail from me to junk :)18:09
lifelesssinzui: I read all bugmail on all lp projects :)18:10
lifeless+ all bzr projects18:10
lifeless+ my own things18:10
sinzuiI think bug mail is easy to read with the right filers. Maybe we should start a project/wiki of bug mail filters18:11
lifelessmight be interesting18:13
lifelessI'm reading it without filters atm18:13
sinzuiwow. There are toddlers who are faster at reading than myself. I cheat to keep up18:14
lifelessI've got fairly good at short circuiting things as I go18:14
lifelesssinzui: what do you think of alexanders comments on bug 52523518:25
_mup_Bug #525235: milestone page: expected release said XX hours ago, when scheduled for today <releases> <Launchpad Registry:Triaged> <https://launchpad.net/bugs/525235>18:25
sinzuiI have put some thought to that18:26
sinzuiwe use a date field, not a date time, but our formatter assumes it is a datetime18:26
sinzuiI think we want to assume the date + 23:59:59.9 seconds so all the UI cases look right18:27
sinzuilifeless,  the case of "today" is like the case of "minutes ago" our formatters are ridiculously precise (2 seconds ago).18:29
lifelessright18:29
lifelessanother bug is open about that :)18:30
sinzuiI think we are tracking that issue in launchpad-web tagged with tales18:30
lifeless\o/ one bug trackker18:30
sinzuiI think an engineer could close all the tales bugs in 2 days18:30
lifelessthat would be awesome18:30
lifelessthen we could delete the project :)18:30
lifelesssinzui: what about bug 212439 - application vs acceptance ?18:32
_mup_Bug #212439: Incorrect date for "member since" on +members, round 2 <Launchpad Registry:Won't Fix> <https://launchpad.net/bugs/212439>18:32
lifelesssinzui: it seems reasonable to me that until you're accepted, you're not a member.18:32
sinzuiI think there are really two issues here. One is an effort to fix the existing data with data that does not exist. We really do not know the answer to when you were accepted18:34
lifelesssinzui: poll support is being yanked, right?18:34
lifelesssinzui: so, I accept that we can't retrospectively fix, but when someone goes through the process now, do we show 'accepted' or 'applied' in the 'member since' field ?18:35
sinzuiAs to recording that information, we could...but Launchpad is not hosting the team. The correct were is register a team. Lp could learns of the team, and the membership years too lae18:35
lifelesssinzui: if we show applied, I think its a legit bug - we should update the application date when they are accepted.18:35
sinzuiPolls will be removed. Someone will get to close twenty bugs soon18:36
lifelesssinzui: you've already started :)18:36
lifelesssinzui: so, https://edge.launchpad.net/~pythonistas/+members *is* an lp team18:36
lifelesssinzui: this leaves me confused18:37
sinzuithe member since/accepted issue assumes the team only exists in Lp. I can be a member of a team before Lp learns of it and before I decide it is now time to register with Lp and join the team18:37
lifelesssinzui: right, but thats a bit meta18:37
lifelessI mean, its addressable by saying 'member of (in launchpad)'18:38
lifelessas the column heading18:38
sinzuiThe admin will need a new field to track that data....18:38
lifelesssinzui: why?18:38
sinzuinote that we do not have a date when a team was registered.18:38
lifelesssinzui: I'm really confused.18:38
lifelesssinzui: the bug doesn't talk about membership *outside* lp.18:38
sinzuiThe issue is that there is an underlying assumption that Lp knows about these dates18:39
lifelesssinzui: thats *an* issue, not the only one.18:39
lifelesssinzui: I agree that addressing the issue with the underlying assumption might also address the issue the users are talking about.18:40
sinzuiWe do not know when a team was registered. membership dates in a team are mutable, they change often as the user's role changes18:40
lifelesssinzui: mmm, holding-a-role-dates change, part-of-team doesn't fluctuate like that, neither in reality or as lp models it18:41
lifelessjoin-exit-join does cause terrible modelling issues.18:41
sinzuilifeless, I think my tagging of "teams" now shows that that is the most common problem in the registry domain. Teams do not work as users expect.18:41
lifelesssinzui: ok18:41
lifelesssinzui: so, I guess I'm saying: bug 212439 sounds like a reasonable thing we could fix, even if its not the big picture. You seem to disagree (its marked wont fix); why?18:42
_mup_Bug #212439: Incorrect date for "member since" on +members, round 2 <Launchpad Registry:Won't Fix> <https://launchpad.net/bugs/212439>18:42
sinzuilifeless, I cannot fix that user's data. that is why.18:42
lifeless(indeed, it sounds like a trivial thing to fix)18:42
sinzuiThe data does not exist18:43
lifelesssinzui: its not a request for data fixing18:43
sinzuiWe could record the data, then tell admins that we have a field that can be set to correct the lost data18:43
lifelessthey are asking why lp didn't record the time that the admin clicked on 'accept' in the team.18:43
lifelesssinzui: separate the concerns. a) fix existing data, b) do better in future.18:44
lifelessI'm talking about b) only18:44
sinzuiThe reason is because all membership changes over write the date18:44
lifelesssinzui: they seem to be saying that they *don't*18:44
sinzuisee the other bugs the not that that underlying issue is wrong18:44
lifelessor are you saying 'if they had joined recently the acceptance date is what would have been shown' ?18:44
sinzuibugger18:45
sinzuiI misread the other bugs18:45
sinzuiThe other bugs are about *expiration* date, not acceptance date18:46
lifelesssinzui: I might reopen this one ?18:49
sinzuiThere is still an issue that we do not  know how to fix the data18:49
lifelessright18:50
lifelessbut I think might be able to do better going forward18:50
sinzui(date_review, date_accepted) -> date_joined. They are used based on subscription policy, and that too is mutable18:50
lifelesswgrant: care to put a patch up for CREATE INDEX packageupload__archive__idx ON PackageUpload(archive);21:13
wgrantlifeless: Will do it later today.22:03
lifelesswgrant: kk22:06
wgrantHmmmmmmmmmmmmmmmmmm.22:30

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!