[01:43] StevenK: Did you ec2 and pre-QA that twisted change? [01:47] (it's a bit awkward that we're upgrading to a new major Twisted release, but cannot QA buildd-manager) [01:48] wgrant: I did ec2 it. It was lightly-QA'd. IE, with a local dev instance, I could ssh to codehost [01:48] OK [01:49] If that doesn't ally your concerns, I understand. :-) [01:50] look at it this way [01:50] its UDS, noone needs builders this week [01:50] Hah [01:58] lifeless: Oh, so no gems about Twisted's upstream or anything? :-) [01:59] StevenK: twisted are pretty good at backward compat [02:00] StevenK: you probably don't even need to qa it [02:00] Indeed, so I'm not hugely concerned [02:00] But it's still something we should really be testing [02:11] StevenK: wgrant: do you know where the tests for the nascent uploads overrides are done? i can't see anywhere where the tests set up various scenarios, run the uploads, and check the overrides. but then again, i'm not sure where to look either [02:12] wallyworld_: lib/lp/archiveuploader [02:12] * wallyworld_ looks [02:13] wgrant: Can I land https://code.launchpad.net/~stevenk/launchpad/db-destroy-potmsgset-potemplate/+merge/131294 ? [02:13] wallyworld_: A disturbingly large number of nascentupload tests are somewhat incidental results of specific prebuilt sample packages being uploaded [02:13] I'm not sure about this particular case; it may just be untested [02:13] There are also some archiveuploader-covering doctests in lib/lp/soyuz [02:14] StevenK: Sure [02:14] oh, will look around a bit. thanks. [02:14] I think the override policy bits are tested, but can't think of an end-to-end test [02:14] archiveuploader doesn't really use override policies today [02:15] Oh, it's the copier [02:15] i am looking for tests which set up various source packages etc and then do uploads to test the various scenarios [02:15] With the uploader being "Eh, later" IIRC [02:15] they mostly seem to be testing hard coded things [02:16] wallyworld_: Both wgrant and I dislike the archiveuploader tests [02:17] sadly, i made a change and ran all the tests with nascent in the test file/name and nothing failed [02:17] They're archaic, slow, mostly doctests and use vast amounts of sampledata. [02:17] yeah, so it seems [02:21] uploadprocessor tests a lot of things [02:21] You really have to run all the archiveuploader tests [02:21] + soyuz-set-of-uploads [02:21] and a couple of others [02:22] ok, will try thise [02:28] SSoU is horrid [03:09] wallyworld_: Have you killed yourself yet? [03:18] StevenK: wtf [03:18] StevenK: Bug #1013013 somehow got tagged as part of r16208 [03:18] <_mup_> Bug #1013013: Timeout viewing my own translations

< https://launchpad.net/bugs/1013013 > [03:18] Oh [03:19] misread revno [03:19] wgrant: Reading is a good skill. [03:19] 16208 and 12088 are both translations changes and adjacent in my mailbox [03:19] Last digit matches, must be right :) [03:19] Haha [03:19] wgrant: Well, first and last. Who cares about the middle bit. [03:22] StevenK: no [03:22] wallyworld_: That's good to hear, at least. [03:22] just trying to see the best place to add some new tests. making progress [03:22] * StevenK loads up OOPS-e8f60c07d46b84f6e939a46d4bc8966c and reaches for the rusty spoon [03:23] it does seem that things are scattered in a few places [03:56] wgrant: TranslationsPerson:+index on qas does not love me [03:57] The query that constantly shows up is ITranslationsPerson.suggestReviewableTranslationFiles() :-( [03:57] That's the one we didn't fix on Friday, right? [03:57] Indeed [03:57] Oh look, it finally worked [03:57] 8 page loads later [03:58] Is my query fast? [03:58] * StevenK greps out an OOPS ID [03:59] wgrant: Based on the fact that suggest is called after get, I'm going to hazzard a 'yes', but confirming [03:59] StevenK: You should be able to use the inline query log [03:59] Just expand it, /WITH, look for MASSIVE BLOB OF PAIN [03:59] 11.434ms [03:59] SQL-main-slave: WITH recent_pofiles AS (SELECT POFile.id FROM POFileTranslator ... [03:59] Yay [04:00] 1023.917ms [04:00] SQL-main-slave: SELECT DISTINCT POFile.currentcount, POFile.date_changed, [04:00] That's suggest [04:00] Aya [04:00] Terrible query is still utterly terrible, news at 11 [04:00] wgrant: I was thinking about it over the weekend ... [04:01] wgrant: Could we use the 4 CTEs of doom with an anti-join and limiting to like 50? [04:01] We only want to suggest a small number of projects, so limiting one of the CTEs to a small number of pofiles doesn't seem like a bad move [04:04] Do we only want to suggest a small number of projects? [04:04] We need a specific plan in mind for this sort of thing [04:04] For the previous one, it was that we only cared about recently translated pofiles. [04:05] Do we have such a shortcut here? [04:05] We have a no_older_than that might be specified [04:05] I'm re-checking what the view does, since Friday was a sleep-induced haze [04:06] fetch = 5 * (list_length - len(overall)) [04:06] suggestions = self._suggestTargetsForReview(fetch) [04:07] And then, pofiles = person.suggestReviewableTranslationFiles( [04:07] no_older_than=self.history_horizon)[:max_fetch] [04:08] So a quicker fix might be tossing max_fetch down one level so the database can actually LIMIT [04:09] Do you have an explain analyze for the existing query? [04:09] But I like the CTEs better than the ten joins of utter doom, so I'd like to use them if we can [04:09] The CTEs we used for the previous query are somewhat irrelevant here, as they existed to coerce postgres into choosing a plan style that cannot work here [04:10] wgrant: No, and running queries on mawson is somewhat useless given it has no disk at all currently [04:10] It's also not completely indexed [04:10] Fortunately we have people with access to fast production databases :) [04:10] Let's try staging [04:11] Or prod, right [04:12] About 2/3 of the way through indexing the restore, then we will have DF back [04:12] The big indices are done [04:12] If you started it on Friday, then it's grown another day [04:26] wgrant: So we have an explain analyze now [04:30] StevenK: What's this query doing? [04:30] wgrant: It's virtually indentical to the get query, except it wants pofiles that the person hasn't contributed to [04:31] StevenK: Right, so you can probably see why the previous optimisation is no longer useful [04:32] Yes, lots and lots of pofiles [04:33] Which is why I suggested limiting the number of pofiles that the CTE returns [04:33] It wants the last 50. We can't really limit earlier than we do now, because then filtering my give us less than 50 [04:34] Bleh [04:35] Blah, I can't check on DF due to the DB rebuild [04:36] So, we want pofiles that are enabled/preferred for translation, that the user has review privileges for, and that the user has not translated recently. [04:36] I was going to have webops repurpose a bunch of pofiles to me so I can see if qas will still load that page [04:36] StevenK: What are you wanting to check? [04:37] I wanted a user id with a 'average' number of pofiles [04:37] Well [04:37] So I could UPDATE pofile set ... [04:37] We don't really need that yet [04:37] We know the query performs terribly for even you now [04:37] True [04:37] So it's qa-ok, but it doesn't fix the bug [04:37] Right [04:38] We know from experiments last week that the fixed query performs well across a wide range of users, and we have good reason to believe it will behave for everyone. [04:38] Right [04:39] It's either quick or blistering fast if the person has no pofiles at all [04:39] Right [04:39] Because we're able to choose a small set to start with [04:39] That's the key to a fast query [04:40] It seems the kicker is 'that the user has no translated recently.' [04:41] Well, sort of [04:41] The kicker is the combination of that and the sort [04:44] Right [04:44] wgrant: So I guess the question is, how can we choose a small set? [04:45] StevenK: Not really [04:45] The better question is "is the result of this query useful?" [04:47] wgrant: You think suggesting things is not useful? [04:49] StevenK: Well, it's not really a binary thing, but a query is not always sufficiently useful to justify the effort expended in its calculation. [04:50] Right [04:50] We can't even tell if users actually do go "Oh, yeah, I should do some." [04:53] .. I can't even see the suggestions on Person:+translations [04:54] You probably don't have review privs in the sense that the query wants them [04:54] Ah ha [04:55] wgrant: So, should we kill it entirely? [04:55] If we can't thoroughly optimise it, deletion is one option [04:57] I can't think of how to optimise it, sadly. [06:12] steven@undermined:~/launchpad/lp-branches/remove-suggested-translations% bzr di | diffstat -s [06:12] 5 files changed, 8 insertions(+), 155 deletions(-) [06:44] wgrant, wallyworld_: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? Or shall I shake jtv until an opinion falls out? [06:45] StevenK: best to check with a SME i think [06:45] StevenK: I'd like to confirm with a rosettay person [06:45] wallyworld_: A what? [06:46] subject matter expert [06:46] wgrant: They're kinda thin on the ground. :-( [06:46] they should be online in a few minutes [06:47] StevenK: You need to talk to dpm anyway :) [06:53] nightly.sh might have actually run update-stats by now [06:54] Not sure when it's kicked off in the chain === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [09:06] jtv: Hai. Do you have one sec to cast your eye over https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? I'm not after a review at all, just your opinion on what the MP is doing. [09:06] I probably do at that. Hi. === yofel_ is now known as yofel [09:08] StevenK: best thing I can do right now is go over why it is the way it is. [09:08] The purpose was to guide translators towards where their efforts can do the most good. [09:09] jtv: Right, and I understand. I just don't think it helps if the query that is trying to do that takes 4 seconds and causes continous timeouts. :-) [09:10] Absolutely. Just going over what it tries to achieve so as to help inform the decision. [09:10] The obvious thing to do would be to steer translators to the highest-priority unfinished translations. But that leads to duplicate effort, for several reasons: [09:11] - multiple translators will be looking at the same page at the same time. [09:11] - unreviewed translations pile up before reviewers can get to them. [09:12] - viewers are incorrectly concentrated on translations that may already be finished until the “100% complete” statistics have filtered through. [09:12] So we wanted a bit of random spread. [09:14] jtv: If you do have some suggestions on how to stop the query asking that question having to join through ten tables, that would help to. [09:14] I don't think I'll have time to look into that, I'm afraid; but I'm sure there are more efficient ways of achieving something that's similar enough. [09:15] We built that functionality from the UI down, and so we may have overspecified it slightly. [09:15] jtv: In which case, some clues about where to look would be nice. [09:15] Hang on, I'll ask huwshimi [09:15] I'm just playing in the shallow end of translations. :-) [09:19] StevenK: talking irl with Huw here [09:23] StevenK: interestingly, it looks as if the suggestions are horribly broken. [09:23] Huw gets a bunch of suggestions to review Low German translations. [09:23] Haha [09:23] haha, whaat [09:23] He is not now, nor ever has been, associated with Low German [09:24] Oh! === almaisan-away is now known as al-maisan [09:24] He just discovered that he is a member of some translation teams for complicated and as-yet unexplored reasons. [09:27] StevenK: Huw just asked a good question — is the query's performance very sensitive to the number of results? [09:27] Nope [09:28] It's terrible no matter who you're asking about [09:28] Didn't think so. :/ [09:28] Well it's particularly timeout-prone in my case. :) [09:28] I bet [09:30] StevenK: here's a solution we just discussed... [09:31] We could stop filling out the list with random suggestions, but underneath the list, have a “take me to a random project that needs my help” link. [09:33] jtv: Is there a way to fetch that quickly? [09:33] StevenK: that means you never execute the query until the user requests it, but its functionality is still there _and_ it's explicit, not cryptically mixed in with more obvious functionality. [09:34] Well, moving the query from one page to another just means another page will timeout [09:34] StevenK: Wouldn't need to be in the generated HTML — could link to a redirect. [09:34] So at worst, the redirect would time out. But there'd be a lot fewer hits on it. [09:38] jtv: Right, which means another page will timeout, and means I fix a critical bug by creating another. [09:38] IE, not on. [09:38] Yes, but! [09:38] It'd allow us to remove some of the selection criteria. [09:38] As Huw just put it, take a slot-machine approach. [09:38] If we have a link to a random project to work on, people may be a lot more acceptable of iffy guesses. [09:39] For example, we won't need to exclude projects that the user has already been translating on. [09:39] Oh, so we just one random project/distro where rosetta_usage == ServiceUsage.LAUNCHPAD ? [09:39] Because that's fast [09:39] And if it isn't, it can indexed [09:40] A bit more than that, but in principle, yes. [09:40] jtv: Do you have time to scribble all this down on the MP? [09:40] Huw suggested we might make the query less sensitive to the user's identity. [09:40] Should be able to, yes. [09:41] Instead of checking translation permissions, we could just limit the suggestions to projects with sufficiently liberal permission models. [09:41] Right [09:41] Which is nice and easy to make performant. [09:43] The core reason the query was so slow, really, was that we're pushing these suggestions in the user's face without explanation. [09:43] The suggestions have to work hard to avoid including absolute nonsense. [09:43] jtv: So, we should be able to 1) Grab a number of projects that fit that, and then grab a random unreviewed POFile from those and job's done? [09:45] StevenK: more or less — there's still the problem that you've got to look at multiple tables, several of which may provide a reason why a particular record is not a candidate. [09:46] So you can't quite say “give me one project, and then we'll look for a POFile there.” [09:46] It's still not a trivial problem. [09:46] But there's a lot of room for compromise once we make it clear that it's a random choice. [09:46] Honesty has its benefits for both parties. :) [09:46] Heh [09:47] As Huw notes, that listing, hard as it tries not to present nonsense, already comes up with some very surprising stuff. [09:48] It will come up with at least 9 pofiles you have worked on, and then at least one suggestion [09:49] Yeah. Filling out that list with suggestions isn't really a very good thing to do. [09:51] StevenK: it'd be fine with me to vary the length of that list. It's tested behaviour. [09:51] huwshimi, StevenK: shall I just summarize on the MP? [09:52] jtv: Please do. [09:52] huwshimi also concurs [09:52] (irl shortcuts irc but it can get confusing) === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [10:19] StevenK: commented on the MP. [10:27] czajkowski: I don't know. jam? [10:28] abentley: I don't have czajkowski's question in my backlog, can you give context? [10:28] jam: czajkowski- does anyone know if we have an intro to bzr video anywhere done ? [10:29] rick_h_: http://wiki.bazaar.canonical.com/Documentation with a section on "Screencasts & Presentations" and an "Installing Bazaar introduction from EmmaJane." [10:30] czajkowski: ^ [10:30] ahhh [10:30] thanks you === shadeslayer is now known as udsslayer [10:49] Can a private team be a member of a public team? [10:51] nigelb: why on earth would you want that! [10:52] czajkowski: I can think of a bunch of use cases, the most immediate one that I thought of was the possibility of having ~canonical a member of ~ubuntu-etherpad. [10:53] nigelb: it's not need ed for canonical most are already on there and the rest join like everyone else. [10:54] czajkowski: I'm not saying it's needed. I was asking if it were possible on LP. === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === Ursinha is now known as Ursinha-afk === al-maisan is now known as almaisan-away === broder_ is now known as broder [14:32] sinzui: is there a way to check LP for Linkedin spam like we do for others like nude or prescription, save people creating questions https://dev.launchpad.net/MaintenanceRotationSchedule [14:34] czajkowski, just search for it [14:34] if it is an issue, we add it to the list of searches to check [14:35] nods it is [14:36] czajkowski, I think this is it: https://launchpad.net/+search?field.text=%22LinkedIn+Corporation%22&field.actions.search=Search [14:37] yes, that shows a lot of comments I would hide [14:37] sinzui: bah I was chaanging the old search and it wsnt returning that [14:37] thanks [14:37] this will make mpt happy [14:37] czajkowski, it will take 30-60 days for google to drop that match from its index [14:38] ok good to know thanks curtis [14:39] sinzui: at uds, so am working on answers and bugs, though the two current one stump me, only thing not able to do is keep an eye on irc [14:39] yes, there are are some hard questions at the moment [14:40] I am waiting for caffeine to rattle me some more [14:40] sinzui: bit stumped on https://support.one.ubuntu.com/Ticket/Display.html?id=24331 can you point me in the rihght direction [14:42] The user does not understand that Lp requires a registered project [14:42] either he needs to register it, or he uses the one already registered [14:42] ack [14:43] wow, this guy has been in Lp since 2005 === benji___ is now known as benji === Ursinha-afk is now known as Ursinha [15:04] Oops, I broke buildbot. Fixing. [15:16] cjwatson: naughty [15:44] cjwatson, ping me if you want a review. === matsubara is now known as matsubara-lunch [16:22] sinzui: Could you just revert the broken rev for me for now, perhaps? I do want to fix this, but it's going to require pretty extensive changes to this horrible doctest [16:22] :( [16:22] sinzui: And I'm going to have difficulty finding enough time to fix in the remaining half-hour of UDS today [16:22] I will queue the revert [16:22] So I'd rather unblock buildbot [16:22] Thanks [16:22] I think I am the only developer today, and I am writing, not coding [16:23] The whole of xx-copy-packages.txt is utterly built around synchronous copies [16:23] It can probably be patched up by just inserting lots of job.run() kinds of things [17:00] hi there, got a question for some launchpad expert [17:00] I need to pull the binary debs for a given build in a private PPA [17:00] I'm logged in and I got credentials set up, but I cannot find a way to expose the binary file through lplib directly... only by getting the url with binaryFileUrls() [17:01] but that the latter I have to do it as a normal browser session, except this is a headless system [17:01] is there a way to maybe generate a one-time cookie so I can use urllib to pull the file? or is there a lplib-approved method? === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [17:16] ricmm, I don't know the answer to your question. I am sure there is no approved way. I think this approach looks viable: http://stackoverflow.com/questions/5082128/how-do-i-authenticate-a-urllib2-script-in-order-to-access-https-web-services-fro === matsubara-lunch is now known as matsubara [17:49] sinzui: dashing out again, mailed you re info I was unsure about [17:49] if you have a chance can you look over it [17:49] please [17:50] okay [22:22] wgrant: StevenK: sinzui: https://pastebin.canonical.com/77402/ [22:56] wallyworld_, sinzui: https://www.youtube.com/watch?v=6TiXUF9xbTo ? === Ursinha is now known as Ursinha-afk