=== StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | Critical bugs: 269 [00:43] hm [00:44] hmm indeed! [00:44] there is a LOT of crack in canonical.launchpad.webapp :( [00:46] is read only mode officially dead now? can we just nuke the relevant code from orbit yet? [00:46] lifeless: ^ ? [00:46] * mwhudson settles for nuking 6 year old crack instead [01:06] mwhudson: There is, yes :( [01:06] mwhudson: We're preserving RO-mode for a few months. [01:06] Just in case. [01:06] ok [01:06] The rest of canonical.* is nearly cleaned out, but canonical.launchpad.webapp is particularly bad and mostly untouched. [01:07] wgrant: i just pushed a branch called i-dont-believe-for-a-minute-that-steveIsFixingThis [01:07] Heh. [01:08] Hey, it's not yet 7 years old. [01:09] it was hilariously old and stale _when i started working on launchpad_ though [01:09] Yes. [01:09] There's lots of that :) [01:09] Every couple of weeks I spend a few hours cleaning up canonical.*... eventually the maintenance squads will do that, I suppose, but it's a very long-term "eventually". [01:11] luckily i don't have the constraint of being paid to work on lp :-) [01:11] mwhudson: not quite dead yet [01:11] lifeless: You're not here -- go away. [01:11] mwhudson: but yeah, I just need to tweak mthaddon a bit about it :) [01:11] lifeless: ok [01:11] wgrant: I had a 6 hour unexpected medical thing yesterday [01:11] wgrant: so I'm partly here [01:11] Ah. [01:11] Carry on, then. [01:12] wgrant: if you have a spare minute, can you search for 'thread_locals' in canonical.launchpad.webapp.publication? [01:12] i think it's complete crack and don't see at all why it's stored on a thread locals object [01:12] ... unless publications are process global or something... [01:13] Used by publication and servers. This can't be anything good. [01:13] Oh, thread_locals is only publication. [01:14] wgrant: i think the one in servers is different [01:14] Yes. [01:14] and is there because of zope crack rather than old pointless crack [01:14] (i think) [01:14] Yeah. [01:14] I don't think the publication is actually per-request. [01:15] i think it is :-) [01:15] the only way i can imagine checking is a print statement though :/ [01:16] It's been several years since I last dealt with this stuff. I forget how it is stuck together. [01:16] yeah fair enough [01:16] (this was before LP was open) [01:16] don't waste too much time on it [01:17] Ewww canonical_url constructs its own request. [01:17] Forgot about that bit. [01:17] haha\ [01:17] yes that is particularly terrible [01:17] (i found that earlier too) [01:19] Hmm, so you're right, the publication is normally per-request. [01:20] But then why do most of its methods take a request... [01:22] the word baroque comes to mind [01:22] oh my yes [01:23] i wish the person who came up with all this was a bit less able to cope with complexity [01:23] aaaaaaa [01:23] wgrant: i'm not sure publications _are_ request local [01:24] publication = self._publication_cache.get(publication_class) [01:24] if publication is None: [01:24] publication = publication_class(self._db) [01:24] self._publication_cache[publication_class] = publication [01:24] (this is in zope.app.publication.httpfactory) [01:24] mwhudson: But the RequestPublicationFactories take the class as a factory... [01:24] Are they lying? [01:25] "sort of" [01:27] Stuff like this is why I've been focusing more on the layer above. The cruft that integrates with our Zope integration cruft. [01:27] Not our Zope integration cruft itself. [01:27] yeah fair enough [01:27] Because that is messy and awful :( [01:33] wgrant: do you want to do the honours? https://code.launchpad.net/~mwhudson/launchpad/i-dont-believe-for-a-minute-that-steveIsFixingThis/+merge/79047 [01:34] mwhudson: Done. [01:36] wgrant: :-) [01:38] mwhudson: I came across that code the week I started on LP ... it was unsettling [01:49] anyone else getting an oops when visiting https://code.launchpad.net/~launchpad-pqm/launchpad/devel/+activereviews ? [01:50] NotOneError: one() used with more than one result available [01:50] Whee [01:50] That may be from the staticdiff migration... [01:50] But I don't see how. [01:50] huwshimi: https://code.launchpad.net/launchpad-project/+activereviews [01:50] They've been on the OOPS reports for a couple of weeks. [01:50] But https://code.launchpad.net/launchpad/+activereviews still works. [01:51] wgrant: You said you didn't agree with Julian's meta-lp-deps change? [01:51] StevenK: Correct. But he hasn't withdrawn it yet. [01:51] I think I agree with you, it's unnecessary [01:52] wgrant, StevenK: I got that link from: https://code.launchpad.net/~launchpad-pqm/launchpad/devel and then clicking on "56 branches proposed for merging into this one." [01:52] huwshimi: Bug [01:52] There may already be a bug for it. [01:52] It's been on the OOPS reports for ages. [01:52] * StevenK waits for wgrant's celeberal LP link to fire up. [01:53] it means there are diffs that are linked to by more than one mp? [01:53] cerebral, even [01:53] It's somewhat concerning that PreviewDiff.branch_merge_proposal is meant to be a single object, but there's apparently no UNIQUE constraint on the table. [01:55] There are 35 offending diffs. [01:56] 33 have two references, 2 have 3. [01:56] But we dropped them all [01:56] Our changes couldn't have caused this. [01:56] Ah [01:57] 3 failures. 2 are doctests. -nearly- there. [01:58] lifeless: oops-tools? [01:59] yes [01:59] then its just purging / getting an ack on bundled code [01:59] and we're done. muahahahahahhahahahahaa [01:59] hahhaa [02:00] hahahah [02:00] hahahahahahaha [02:01] and the errors finally all fit in my terminal scrollback [02:05] StevenK: Actually, I think it may have been the staticdiff dropping that did it :( [02:06] Oh? [02:07] There's an undeclared uniqueness requirement on BranchMergeProposal.preview_diff [02:07] Whereas I believe staticdiffs were meant to be shared if they were between the same revisions. [02:08] This is upsetting. [02:08] So we will have converted a single shared staticdiff into a single shared previewdiff. [02:08] Which is illegal, although permitted by the DB. [02:09] Perhaps we should purge the illegalness and toss up a DB patch to add a UNIQUE constraint [02:10] wgrant: did you and sinzui talk about private N-pillar bug tasks ? [02:10] lifeless: A little, but the call was cut short. [02:11] lifeless: He intends to talk to stakeholders tomorrow about eliminating them. [02:12] ack [02:13] 1 test [02:14] at the loss of a little test coverage. Sob, don't know how ai shall cope [02:14] (no, I didn't just rm the test ..) [02:15] Ah, so you commented it out instead [02:17] I trimmed the sample data rather vigorously [02:17] the report has a lot less to show now:) [02:54] lifeless: just for my education, could we cluster bugaffectsuser by user? [02:55] to make the paging behaviour better? [02:55] i guess the main objection would be if it commonly is queried by bug? [02:55] offhand - yes [02:55] but i think not, or only to generate the counts [02:55] no, that wouldnt be an objection [02:55] oh, why not? [02:55] generally indices are what matters [02:56] its only in rare cases (perhaps like this) that the table page order matters [02:56] currently there is no clustering [02:57] if you mean 'an alternative cluster would be by bug' - well yes [02:57] poolie: but clustering is a one-off in psql [02:58] you mean it's not automatically maintained? i read that. [02:59] so to try this i could: ask stub to on staging make the per-person index be clustered, and then reorder the table to cluster, and then see if it's faster? [03:00] well, on the next downtime [03:00] and we'd need to test on staging to see how long it takes, but it should be fast. [03:00] right, but we could test it on staging fairly freely? [03:01] then you'd need to wait for a cold cache, and then you could try [03:01] yes [03:01] it wouldn't need a particular downtime window to do it there? [03:01] its ok to tie staging up for a few minutes [03:03] obviously the timing is pretty noisy, but if it's worth doing hopefully the effect would stand out from the noise [03:05] poolie: whats the current query you run? I want to check its plan [03:05] it's in bug 866100 [03:05] <_mup_> Bug #866100: bug search with affects_me=on times out

< https://launchpad.net/bugs/866100 > [03:08] that's a paste of the actual sql generated by my updated branch [03:08] ok, so it seems affected_user_count is maintained redundantly (as it should be) [03:09] well, not that it probably matters [03:09] poolie: so you have a problem [03:10] ignore the times for now [03:10] http://paste.ubuntu.com/706443/ [03:10] different plans for user 2 and user 16 [03:10] using different indices [03:11] clustering by person won't help for user 2 - that plan uses the bug,person index [03:11] well, won't help - it won't align completely. [03:11] it will mean the backing table data is in one place, but the index pages will still be scattered all over [03:13] just looking at that query i'm a bit surprised [03:15] i was mostly working on the 'affecting bugs in a product' but that query is all bugs for a user [03:15] which is fine [03:15] just thought i pasted for the other one [03:15] so why a different plan for different users? [03:15] I've clustered staging and qastaging on bugaffectsperson__person__idx for testing purposes [03:15] perhaps because it thinks selecting only bugs affecting user 2 won't reduce it much? [03:16] it seems like a poor choice [03:16] it may be. This is where checking the stats targets may influence things [03:16] we can get more info out of new pg's as well - the explain/analyze stuff got sexy in 9.x [03:17] is it on the roadmap for lp to switch to that? [03:18] yes [03:18] are the stats going to be reasonably up to date on qas? [03:19] before the cluster yes [03:19] I need to updat ethem, one sec [03:20] so https://bugs.qastaging.launchpad.net/~gz/+affectingbugs with 42 results, works ok in 2.66s [03:20] ah, you and i work now too, though perhaps it's just warmed up [03:21] btw did you read of http://www.thumbtack.com/engineering/googlebot-makes-post-requests-via-ajax/ [03:21] win! [03:24] poolie: yes, its warm [03:24] right, and mpt for instance fails once then works [03:24] poolie: it seems faster when hot, though there are confounding issues (clustering also compresses pages, which reduces bloat, which itself helps with performance - less parsing of data structures) [03:25] it seems at the moment to consistently work first go on users with small lists [03:25] poolie: I would wait and see how your thing goes on prod [03:25] https://bugs.qastaging.launchpad.net/~silbs/+affectingbugs :) [03:25] https://bugs.qastaging.launchpad.net/~sabdfl/+affectingbugs - only 40! [03:25] ok [03:26] note that fully *half* of the poor performance on me cold is in bugtask: [03:26] -> Index Scan Backward using bugtask_importance_idx on bugtask (cost=0.00..153181.30 rows=341880 width=161) (actual time=189.469..46810.549 rows=5993 loops=1) [03:26] Filter: ((status = 10) OR (status = ANY ('{15,13,14}'::integer[])) OR (status = 20) OR (status = 21) OR (status = 22) OR (status = 25)) [03:26] 46 seconds walking bugtask backwards by importance [03:27] well, that would be very big, right? [03:27] 6000 rows [03:27] that's going to be pulling out ~500,000 rows? [03:27] oh, why so few? [03:27] LIMIT 8 OFFSET [03:27] 0 [03:28] apparently I have 8 open affecting me bugs in the last 5993 filed bugs [03:28] huh, i wonder how that got in there [03:28] if we raise that 8 (launchpad.dev) to 75 (launchpad) [03:29] we may see different plans etc [03:29] yeah, limit 75 switches the plan to the same as the original one for user 16 [03:29] pg is pretty smart ;) [03:30] i'm a bit confused why my code running on dev would have put a 'limit 8' into the query [03:30] how would it know there are 8 [03:31] the batch code does a count(*) [03:31] and if < limits to match [03:31] abel's work helps stop this crazy behaviour, but we have to roll it out over each collection individually [03:32] wow, i see [03:33] anyhow, just a thing to watch for when copying test plans around :) [03:33] so it really seems to me it ought to be using bugaffectsperson__person__idx [03:33] it does for user=2 with limit 75 offset 0 [03:33] even for you, that should cut down the bugtask table quite a lot [03:33] but not for limit 8 offset 0 [03:34] its going to be assessing linear IO costs in there vs random [03:34] I think we need to do some benching and tweaking of these params [03:36] ok, so, like you suggest i'll just wait for it on prod [03:37] and - should we do anything further on clustering, or just if it's passing on prod leave it alone? [03:38] if it works don't touch it [03:38] :) [03:38] :) yeah [03:39] i have more context to understand abel's mail now, thanks [03:39] so if this does time out, one thing i could try is to make that update in the bug list pages [03:44] ok i might take a break, thanks [03:45] [A3 [03:46] +1 === almaisan-away is now known as al-maisan [06:01] wgrant: you know stderr is meant for non-exceptional stuff :) [06:01] wgrant: per POSIX :) [06:02] lifeless: Meh. [06:03] It's Launchpad. Everything is non-exceptional because it's all NORMAL. :-( [06:03] wgrant: explicitly even. [06:04] StevenK: actually this is referring to apt-ftparchive [06:04] Sigh. [06:04] Bring on NMAF [06:04] StevenK: which does progress chatter on stderr (which is ok, but it should have a -q to get it to bare bones) [06:04] wgrant: (and btw, did you file a bug upstream ?) [06:05] lifeless: I didn't. It may be silencable. [06:05] (this is what was already done for the publisher, it just needed to be done for generate-contents-files too) [06:05] -q Quiet [06:05] -qq --shut-the-hell-up [06:05] from its help :) [06:20] Who's orange team? [06:20] The bug list mocks up look... WOW [06:25] nigelb: Deryck and the rest of the mafia [06:25] StevenK: heh [06:25] StevenK: I guess Teal is the assasin squad, seeing the amount of code you kill :P [06:26] Hahaha [06:26] I shall have to propose that to Curtis [06:26] heh [06:27] StevenK: "The Teal Assasins" [06:28] Yes, I was thinking that [06:28] * G joins the group "Save the Ducks" and starts protesting :) [06:28] nigelb: "Assassinating bad code, one line at a time." [06:29] StevenK: bwahaha, I <3 it. [06:29] wgrant: ^^ [06:29] Heh [06:29] G: You like duck typing? [06:30] StevenK: no, but I happen to like all ducks, Teals included, not keen on the idea of assassinating them :) [06:30] wgrant: globalErrorUtility is defined in errorlog, and not exported :-( [06:31] StevenK: in fact, the server that my irssi connection is located on, is named after the brown teal (Pateke) [06:31] StevenK: You're not meant to use getUtility(IErrorReportingUtility)? [06:31] wgrant: I have no idea :-( [06:32] G: I name my machines after things that can befall them that make them stop working [06:32] G: My mail server is 'mangled', my desktop is 'liquified' [06:32] You work machine is 'deleted'? :P [06:32] heh, subconciously all my VPS' have been named after ducks :) [06:33] *your [06:33] nigelb: No, I do my work on liquified [06:33] StevenK: ha :) [06:34] liquified might prove to be true, sadly. [06:34] I think the lone fan needs a clean [06:34] StevenK: you are going to put in water-cooling? [06:34] No, I'm just commenting that it is running a little too hot at the moment. [06:39] anyone tried to create_initialised_view(some_person, name='some_view') and got a LocationError: 'link-display-name-id'? === al-maisan is now known as almaisan-away [07:02] Bah. [07:02] Unity stole my text editor [07:02] It is still running, but nowhere to be found. [07:48] wgrant: want to take this question? https://answers.launchpad.net/launchpad/+question/173985 [07:52] jtv: Julian should be here in 10, right? [07:58] wgrant: what's the easiest way to get a mp on qas that i can play around with? [07:59] wallyworld_: Create one fresh. [07:59] ok. i think last time i tried it didn't work but i'll try again [08:00] It should work OK, but you might need to get the runner run manually if you want a diff. [08:00] ah, i think i remember - i never get a diff [08:01] what's the runner script called? [08:01] Should be merge-proposal-jobs.py [08:01] thanks [08:02] wgrant: yes [08:03] * StevenK grumbles at the existance of cronscripts/run_jobs.py and cronscripts/process-job-source.py [08:07] good morning [08:07] bigjools: lp:python-oops-amqp :) [08:07] good morning [08:07] g'morning [08:08] :) [08:08] lifeless: someone's been busy [08:08] bigjools: things are coming together [08:11] bigjools: I'm going to do a spool-from-disk module for oops-datedir-repo, and then we can use that as the backing store when rabbit is down [08:11] bigjools: at which point, it will be time to land it all in LP trunk :) [08:12] lifeless: backing store? [08:12] fallback [08:12] if rabbit is awol and an oops is raised, what should happen... [08:12] depends what raises the oops [08:13] (answer: we write it to disk, and we have a separate mini task that picks it up from disk and sends it to amqp when amqp comes back) [08:13] oh wait - you're talking about pushing OOPSes down to rabbit?! [08:13] danilos: I think I saw movement from your direction. Want to review a very small translations change? https://code.launchpad.net/~jtv/launchpad/bug-872646/+merge/79053 [08:13] bigjools: its configurable (just depends on what publishers we setup), but this seems like a sane setup for all our services/scripts to me [08:13] bigjools: yes ! [08:14] lifeless: ah! I thought you mean picking up rabbit problems and filing an oops [08:14] bigjools: no, though that is needed too [08:14] bigjools: that wouldn't get a generic name though :) === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [08:25] wgrant: sadly, running the job got rid of the "Updating Diff" message but no diff has been created [08:26] wallyworld_: Are the source and target branches in place? [08:27] not sure. the target branch is lp trunk, the source is one i did a few weeks ago [08:27] LP trunk probably isn't there. [08:27] wallyworld_: if it's similar to what happens locally, you need to run the script *twice* to get a diff. [08:29] rvba: hi! twice? could be, although i suspect it's more due to qas not having the source and/or target branches available === danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK, danilos | Critical bugs: 269 [08:35] jtv, r=me, as for the expiration period, 6 months is fine (we can certainly go lower, but doesn't really matter imho) [08:36] danilos: that was my thinking too. Thanks. [08:36] Thought I might as well grab you as long as I still can. :) [08:38] jtv, heh, yeah, good point, I am also making a point of fixing some of the important translations issues :) [08:40] bigjools, hi, do you happen to know if there are actual reasons for +configure-translations page to have builder-related stuff? (see https://translations.launchpad.net/ubuntu/+configure-translations) [08:41] danilos: saw that, including the policy change on approval permissions. Good news. [08:41] danilos: bigjools & I are otop atm :-P [08:41] jtv, ask him if configuring arm builders is part of translations settings then :P [08:41] danilos: there is no reason at all for those 2 options [08:41] bigjools, cool, thanks, I'll remove them from that page [08:41] danilos: ubuntu uses non-virt, everyone else virtual === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [09:54] Pending team membership requests don't ever expire, do they? [09:54] No. [09:54] Shouldn't they? [09:55] Why? [09:55] I'd like to answer that in 2 ways. [09:55] "Here, let me your question with another question." ? [09:55] 1. Why should they? → Because if a request hasn't been reviewed after, say, a year, that amounts to “no you can't join.” [09:56] 2. Why do you ask? → Because we've got teams with eternally pending membership requests, caused by problem with person merger, that currently can't be got rid of. [09:56] That wasn't so bad, was it? [09:57] I know, I know, we should fix person merger. But that's marked as Low. We'll never have time for it unless it becomes super-urgent. [09:58] It also won't help people who are affected by this problem already. === StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: danilos | Critical bugs: 269 [10:06] "here, let me answer your question with unicode glyphs" [10:07] poolie: oh ☣ off [10:11] .... [10:11] Never a dull moment in this channel [10:13] jtv: As a team admin, I'd want to set a limit of number of days. If a person isn't approved within that time, they're not getting approved. [10:13] 1 year sounds arbitary [10:13] Oh, just saw your m/l post. /me will reply to it. [10:18] nigelb: thanks for thinking about it though. [10:19] :) [10:47] danilos: fancy a review of a trivial MP: https://code.launchpad.net/~adeuring/launchpad/feature-flag-for-custom-bug-listings/+merge/79103 ? [10:48] adeuring, r=me ;) [10:48] danilos: thanks :) [10:50] jtv: btw, I was looking at the person merger, and I might take a stab at it? [10:51] We have a person merger :-P [10:51] G: you mean fixing the bug itself? That'd be great, but it looks like it could get complicated. [10:52] awesome, generate-contents-files is producing 646 oopses a day [10:52] For example: person A is in team X, and person B is in team Y, and team Y is in team X, and you mere A & B. What happens to each of the TeamParticipations? [10:52] What happens when A and B are in the same team, but with different statuses? [10:52] Etc. [10:53] and yeah, I agree w/ nigelb that groups should set the expiration period per group, but really, shouldn't the default be more like 4-6 months, anything longer kind of implies an inactive group (or I guess indecision) === Guest82578 is now known as jelmer [10:54] jtv: a valid point, but is there actually any harm for the merged user to be a member in both groups? [10:55] jtv: Perefect interview question ;) [10:55] *perfect (oh the irony) [10:55] G: Teams. I would expect it to be simply impossible; it would probably violate a database constraint. [10:55] nigelb: does sound like it, doesn't it? [10:56] err yeah teams, sorry [10:56] It sounds like a fun problem to solve, because its non-trivial. [10:56] Meanwhile, person A departs Manchester in a train heading towards London, where person B… [10:56] lol [10:56] There can be especially nasty bugs if you delete one record; update another; move on; and later hit a database error when the two changes get flushed to the database in the wrong order. [10:57] ah. replication. [10:57] as far as A & B in the same group w/ different status' shouldn't the higher status be the one kept? After all they'd have that access anyway [10:57] ooooh yeah, replication [10:58] nigelb: careful—replication is a different matter altogether. I'm talking about a different layer of caching, within the ORM itself. [10:58] The replication machinery won't make that mistka.e [10:58] *mistake [10:58] (speaking of irony) [10:59] bug 860245 [10:59] <_mup_> Bug #860245: generate-contents-files.py should be a LaunchpadCronScript < https://launchpad.net/bugs/860245 > [10:59] that ^^ is causing 1000+ oops a day [10:59] jtv: Oh right. I didn't realize ther was an ORM layer of caching [11:00] bigjools: the bug is, or the fix is? [11:00] the fix [11:00] Ouch. [11:00] the script logs at WARNING [11:00] sigh [11:00] Ah that. [11:00] bigjools, jtv: Fix landed a couple of hours ago. [11:00] You say critical, I say cheap karma. [11:01] wgrant: ah, good. What's the bug? I just filed a dupe [11:01] ~> [11:01] ~> [11:01] It was a 5-character fix so I didn't file one :P [11:01] :/ [11:07] jtv: hmmm I certainly get what you are saying, might try and think of a way to counter it [11:08] G: looking at the bug again, I'm not even sure if there are TeamParticipations for these particular membership records. It seems such a simple problem, but… [11:08] jtv: yeah, I haven't looked at the code, my checked out branch is a few weeks old, so I'm just updating it now [11:09] jtv: It may depend in which directions the accounts are merged wouldn't it? [11:09] now if only postgres had a databasediff tool :P [11:09] (I'm guessing this is merging imported automatically created accounts with real ones) [11:09] nigelb: probably, too. Lots of choices to be made. Sometimes you just have to start coding to find out what the choices are, and then you have to be willing to throw away your code and start over if need be. [11:09] jtv: Or start writing tests :) [11:09]