wgrant | StevenK: Did you ec2 and pre-QA that twisted change? | 01:43 |
---|---|---|
wgrant | (it's a bit awkward that we're upgrading to a new major Twisted release, but cannot QA buildd-manager) | 01:47 |
StevenK | wgrant: I did ec2 it. It was lightly-QA'd. IE, with a local dev instance, I could ssh to codehost | 01:48 |
wgrant | OK | 01:48 |
StevenK | If that doesn't ally your concerns, I understand. :-) | 01:49 |
lifeless | look at it this way | 01:50 |
lifeless | its UDS, noone needs builders this week | 01:50 |
wgrant | Hah | 01:50 |
StevenK | lifeless: Oh, so no gems about Twisted's upstream or anything? :-) | 01:58 |
lifeless | StevenK: twisted are pretty good at backward compat | 01:59 |
lifeless | StevenK: you probably don't even need to qa it </wink> | 02:00 |
wgrant | Indeed, so I'm not hugely concerned | 02:00 |
wgrant | But it's still something we should really be testing | 02:00 |
wallyworld_ | StevenK: wgrant: do you know where the tests for the nascent uploads overrides are done? i can't see anywhere where the tests set up various scenarios, run the uploads, and check the overrides. but then again, i'm not sure where to look either | 02:11 |
StevenK | wallyworld_: lib/lp/archiveuploader | 02:12 |
* wallyworld_ looks | 02:12 | |
StevenK | wgrant: Can I land https://code.launchpad.net/~stevenk/launchpad/db-destroy-potmsgset-potemplate/+merge/131294 ? | 02:13 |
wgrant | wallyworld_: A disturbingly large number of nascentupload tests are somewhat incidental results of specific prebuilt sample packages being uploaded | 02:13 |
wgrant | I'm not sure about this particular case; it may just be untested | 02:13 |
wgrant | There are also some archiveuploader-covering doctests in lib/lp/soyuz | 02:13 |
wgrant | StevenK: Sure | 02:14 |
wallyworld_ | oh, will look around a bit. thanks. | 02:14 |
StevenK | I think the override policy bits are tested, but can't think of an end-to-end test | 02:14 |
wgrant | archiveuploader doesn't really use override policies today | 02:14 |
StevenK | Oh, it's the copier | 02:15 |
wallyworld_ | i am looking for tests which set up various source packages etc and then do uploads to test the various scenarios | 02:15 |
StevenK | With the uploader being "Eh, later" IIRC | 02:15 |
wallyworld_ | they mostly seem to be testing hard coded things | 02:15 |
StevenK | wallyworld_: Both wgrant and I dislike the archiveuploader tests | 02:16 |
wallyworld_ | sadly, i made a change and ran all the tests with nascent in the test file/name and nothing failed | 02:17 |
StevenK | They're archaic, slow, mostly doctests and use vast amounts of sampledata. | 02:17 |
wallyworld_ | yeah, so it seems | 02:17 |
wgrant | uploadprocessor tests a lot of things | 02:21 |
wgrant | You really have to run all the archiveuploader tests | 02:21 |
wgrant | + soyuz-set-of-uploads | 02:21 |
wgrant | and a couple of others | 02:21 |
wallyworld_ | ok, will try thise | 02:22 |
StevenK | SSoU is horrid | 02:28 |
StevenK | wallyworld_: Have you killed yourself yet? | 03:09 |
wgrant | StevenK: wtf | 03:18 |
wgrant | StevenK: Bug #1013013 somehow got tagged as part of r16208 | 03:18 |
_mup_ | Bug #1013013: Timeout viewing my own translations <lp-translations> <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by stevenk> < https://launchpad.net/bugs/1013013 > | 03:18 |
wgrant | Oh | 03:18 |
wgrant | misread revno | 03:19 |
StevenK | wgrant: Reading is a good skill. | 03:19 |
wgrant | 16208 and 12088 are both translations changes and adjacent in my mailbox | 03:19 |
wgrant | Last digit matches, must be right :) | 03:19 |
StevenK | Haha | 03:19 |
StevenK | wgrant: Well, first and last. Who cares about the middle bit. | 03:19 |
wallyworld_ | StevenK: no | 03:22 |
StevenK | wallyworld_: That's good to hear, at least. | 03:22 |
wallyworld_ | just trying to see the best place to add some new tests. making progress | 03:22 |
* StevenK loads up OOPS-e8f60c07d46b84f6e939a46d4bc8966c and reaches for the rusty spoon | 03:22 | |
wallyworld_ | it does seem that things are scattered in a few places | 03:23 |
StevenK | wgrant: TranslationsPerson:+index on qas does not love me | 03:56 |
StevenK | The query that constantly shows up is ITranslationsPerson.suggestReviewableTranslationFiles() :-( | 03:57 |
wgrant | That's the one we didn't fix on Friday, right? | 03:57 |
StevenK | Indeed | 03:57 |
StevenK | Oh look, it finally worked | 03:57 |
StevenK | 8 page loads later | 03:57 |
wgrant | Is my query fast? | 03:58 |
* StevenK greps out an OOPS ID | 03:58 | |
StevenK | wgrant: Based on the fact that suggest is called after get, I'm going to hazzard a 'yes', but confirming | 03:59 |
wgrant | StevenK: You should be able to use the inline query log | 03:59 |
wgrant | Just expand it, /WITH, look for MASSIVE BLOB OF PAIN | 03:59 |
StevenK | 11.434ms | 03:59 |
StevenK | SQL-main-slave: WITH recent_pofiles AS (SELECT POFile.id FROM POFileTranslator ... | 03:59 |
wgrant | Yay | 03:59 |
StevenK | 1023.917ms | 04:00 |
StevenK | SQL-main-slave: SELECT DISTINCT POFile.currentcount, POFile.date_changed, | 04:00 |
StevenK | That's suggest | 04:00 |
wgrant | Aya | 04:00 |
StevenK | Terrible query is still utterly terrible, news at 11 | 04:00 |
StevenK | wgrant: I was thinking about it over the weekend ... | 04:00 |
StevenK | wgrant: Could we use the 4 CTEs of doom with an anti-join and limiting to like 50? | 04:01 |
StevenK | We only want to suggest a small number of projects, so limiting one of the CTEs to a small number of pofiles doesn't seem like a bad move | 04:01 |
wgrant | Do we only want to suggest a small number of projects? | 04:04 |
wgrant | We need a specific plan in mind for this sort of thing | 04:04 |
wgrant | For the previous one, it was that we only cared about recently translated pofiles. | 04:04 |
wgrant | Do we have such a shortcut here? | 04:05 |
StevenK | We have a no_older_than that might be specified | 04:05 |
StevenK | I'm re-checking what the view does, since Friday was a sleep-induced haze | 04:05 |
StevenK | fetch = 5 * (list_length - len(overall)) | 04:06 |
StevenK | suggestions = self._suggestTargetsForReview(fetch) | 04:06 |
StevenK | And then, pofiles = person.suggestReviewableTranslationFiles( | 04:07 |
StevenK | no_older_than=self.history_horizon)[:max_fetch] | 04:07 |
StevenK | So a quicker fix might be tossing max_fetch down one level so the database can actually LIMIT | 04:08 |
wgrant | Do you have an explain analyze for the existing query? | 04:09 |
StevenK | But I like the CTEs better than the ten joins of utter doom, so I'd like to use them if we can | 04:09 |
wgrant | The CTEs we used for the previous query are somewhat irrelevant here, as they existed to coerce postgres into choosing a plan style that cannot work here | 04:09 |
StevenK | wgrant: No, and running queries on mawson is somewhat useless given it has no disk at all currently | 04:10 |
wgrant | It's also not completely indexed | 04:10 |
wgrant | Fortunately we have people with access to fast production databases :) | 04:10 |
StevenK | Let's try staging | 04:10 |
StevenK | Or prod, right | 04:11 |
wgrant | About 2/3 of the way through indexing the restore, then we will have DF back | 04:12 |
wgrant | The big indices are done | 04:12 |
StevenK | If you started it on Friday, then it's grown another day | 04:12 |
StevenK | wgrant: So we have an explain analyze now | 04:26 |
wgrant | StevenK: What's this query doing? | 04:30 |
StevenK | wgrant: It's virtually indentical to the get query, except it wants pofiles that the person hasn't contributed to | 04:30 |
wgrant | StevenK: Right, so you can probably see why the previous optimisation is no longer useful | 04:31 |
StevenK | Yes, lots and lots of pofiles | 04:32 |
StevenK | Which is why I suggested limiting the number of pofiles that the CTE returns | 04:33 |
wgrant | It wants the last 50. We can't really limit earlier than we do now, because then filtering my give us less than 50 | 04:33 |
StevenK | Bleh | 04:34 |
StevenK | Blah, I can't check on DF due to the DB rebuild | 04:35 |
wgrant | So, we want pofiles that are enabled/preferred for translation, that the user has review privileges for, and that the user has not translated recently. | 04:36 |
StevenK | I was going to have webops repurpose a bunch of pofiles to me so I can see if qas will still load that page | 04:36 |
wgrant | StevenK: What are you wanting to check? | 04:36 |
StevenK | I wanted a user id with a 'average' number of pofiles | 04:37 |
wgrant | Well | 04:37 |
StevenK | So I could UPDATE pofile set ... | 04:37 |
wgrant | We don't really need that yet | 04:37 |
wgrant | We know the query performs terribly for even you now | 04:37 |
StevenK | True | 04:37 |
StevenK | So it's qa-ok, but it doesn't fix the bug | 04:37 |
wgrant | Right | 04:37 |
wgrant | We know from experiments last week that the fixed query performs well across a wide range of users, and we have good reason to believe it will behave for everyone. | 04:38 |
StevenK | Right | 04:38 |
StevenK | It's either quick or blistering fast if the person has no pofiles at all | 04:39 |
wgrant | Right | 04:39 |
wgrant | Because we're able to choose a small set to start with | 04:39 |
wgrant | That's the key to a fast query | 04:39 |
StevenK | It seems the kicker is 'that the user has no translated recently.' | 04:40 |
wgrant | Well, sort of | 04:41 |
wgrant | The kicker is the combination of that and the sort | 04:41 |
StevenK | Right | 04:44 |
StevenK | wgrant: So I guess the question is, how can we choose a small set? | 04:44 |
wgrant | StevenK: Not really | 04:45 |
wgrant | The better question is "is the result of this query useful?" | 04:45 |
StevenK | wgrant: You think suggesting things is not useful? | 04:47 |
wgrant | StevenK: Well, it's not really a binary thing, but a query is not always sufficiently useful to justify the effort expended in its calculation. | 04:49 |
StevenK | Right | 04:50 |
StevenK | We can't even tell if users actually do go "Oh, yeah, I should do some." | 04:50 |
StevenK | .. I can't even see the suggestions on Person:+translations | 04:53 |
wgrant | You probably don't have review privs in the sense that the query wants them | 04:54 |
StevenK | Ah ha | 04:54 |
StevenK | wgrant: So, should we kill it entirely? | 04:55 |
wgrant | If we can't thoroughly optimise it, deletion is one option | 04:55 |
StevenK | I can't think of how to optimise it, sadly. | 04:57 |
StevenK | steven@undermined:~/launchpad/lp-branches/remove-suggested-translations% bzr di | diffstat -s | 06:12 |
StevenK | 5 files changed, 8 insertions(+), 155 deletions(-) | 06:12 |
StevenK | wgrant, wallyworld_: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? Or shall I shake jtv until an opinion falls out? | 06:44 |
wallyworld_ | StevenK: best to check with a SME i think | 06:45 |
wgrant | StevenK: I'd like to confirm with a rosettay person | 06:45 |
StevenK | wallyworld_: A what? | 06:45 |
wallyworld_ | subject matter expert | 06:46 |
StevenK | wgrant: They're kinda thin on the ground. :-( | 06:46 |
lifeless | they should be online in a few minutes | 06:46 |
wgrant | StevenK: You need to talk to dpm anyway :) | 06:47 |
StevenK | nightly.sh might have actually run update-stats by now | 06:53 |
StevenK | Not sure when it's kicked off in the chain | 06:54 |
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
StevenK | jtv: Hai. Do you have one sec to cast your eye over https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? I'm not after a review at all, just your opinion on what the MP is doing. | 09:06 |
jtv | I probably do at that. Hi. | 09:06 |
=== yofel_ is now known as yofel | ||
jtv | StevenK: best thing I can do right now is go over why it is the way it is. | 09:08 |
jtv | The purpose was to guide translators towards where their efforts can do the most good. | 09:08 |
StevenK | jtv: Right, and I understand. I just don't think it helps if the query that is trying to do that takes 4 seconds and causes continous timeouts. :-) | 09:09 |
jtv | Absolutely. Just going over what it tries to achieve so as to help inform the decision. | 09:10 |
jtv | The obvious thing to do would be to steer translators to the highest-priority unfinished translations. But that leads to duplicate effort, for several reasons: | 09:10 |
jtv | - multiple translators will be looking at the same page at the same time. | 09:11 |
jtv | - unreviewed translations pile up before reviewers can get to them. | 09:11 |
jtv | - viewers are incorrectly concentrated on translations that may already be finished until the “100% complete” statistics have filtered through. | 09:12 |
jtv | So we wanted a bit of random spread. | 09:12 |
StevenK | jtv: If you do have some suggestions on how to stop the query asking that question having to join through ten tables, that would help to. | 09:14 |
jtv | I don't think I'll have time to look into that, I'm afraid; but I'm sure there are more efficient ways of achieving something that's similar enough. | 09:14 |
jtv | We built that functionality from the UI down, and so we may have overspecified it slightly. | 09:15 |
StevenK | jtv: In which case, some clues about where to look would be nice. | 09:15 |
jtv | Hang on, I'll ask huwshimi | 09:15 |
StevenK | I'm just playing in the shallow end of translations. :-) | 09:15 |
jtv | StevenK: talking irl with Huw here | 09:19 |
jtv | StevenK: interestingly, it looks as if the suggestions are horribly broken. | 09:23 |
jtv | Huw gets a bunch of suggestions to review Low German translations. | 09:23 |
StevenK | Haha | 09:23 |
nigelb | haha, whaat | 09:23 |
jtv | He is not now, nor ever has been, associated with Low German | 09:23 |
jtv | Oh! | 09:24 |
=== almaisan-away is now known as al-maisan | ||
jtv | He just discovered that he is a member of some translation teams for complicated and as-yet unexplored reasons. | 09:24 |
jtv | StevenK: Huw just asked a good question — is the query's performance very sensitive to the number of results? | 09:27 |
StevenK | Nope | 09:27 |
StevenK | It's terrible no matter who you're asking about | 09:28 |
jtv | Didn't think so. :/ | 09:28 |
jtv | Well it's particularly timeout-prone in my case. :) | 09:28 |
StevenK | I bet | 09:28 |
jtv | StevenK: here's a solution we just discussed... | 09:30 |
jtv | We could stop filling out the list with random suggestions, but underneath the list, have a “take me to a random project that needs my help” link. | 09:31 |
StevenK | jtv: Is there a way to fetch that quickly? | 09:33 |
jtv | StevenK: that means you never execute the query until the user requests it, but its functionality is still there _and_ it's explicit, not cryptically mixed in with more obvious functionality. | 09:33 |
StevenK | Well, moving the query from one page to another just means another page will timeout | 09:34 |
jtv | StevenK: Wouldn't need to be in the generated HTML — could link to a redirect. | 09:34 |
jtv | So at worst, the redirect would time out. But there'd be a lot fewer hits on it. | 09:34 |
StevenK | jtv: Right, which means another page will timeout, and means I fix a critical bug by creating another. | 09:38 |
StevenK | IE, not on. | 09:38 |
jtv | Yes, but! | 09:38 |
jtv | It'd allow us to remove some of the selection criteria. | 09:38 |
jtv | As Huw just put it, take a slot-machine approach. | 09:38 |
jtv | If we have a link to a random project to work on, people may be a lot more acceptable of iffy guesses. | 09:38 |
jtv | For example, we won't need to exclude projects that the user has already been translating on. | 09:39 |
StevenK | Oh, so we just one random project/distro where rosetta_usage == ServiceUsage.LAUNCHPAD ? | 09:39 |
StevenK | Because that's fast | 09:39 |
StevenK | And if it isn't, it can indexed | 09:39 |
jtv | A bit more than that, but in principle, yes. | 09:40 |
StevenK | jtv: Do you have time to scribble all this down on the MP? | 09:40 |
jtv | Huw suggested we might make the query less sensitive to the user's identity. | 09:40 |
jtv | Should be able to, yes. | 09:40 |
jtv | Instead of checking translation permissions, we could just limit the suggestions to projects with sufficiently liberal permission models. | 09:41 |
StevenK | Right | 09:41 |
StevenK | Which is nice and easy to make performant. | 09:41 |
jtv | The core reason the query was so slow, really, was that we're pushing these suggestions in the user's face without explanation. | 09:43 |
jtv | The suggestions have to work hard to avoid including absolute nonsense. | 09:43 |
StevenK | jtv: So, we should be able to 1) Grab a number of projects that fit that, and then grab a random unreviewed POFile from those and job's done? | 09:43 |
jtv | StevenK: more or less — there's still the problem that you've got to look at multiple tables, several of which may provide a reason why a particular record is not a candidate. | 09:45 |
jtv | So you can't quite say “give me one project, and then we'll look for a POFile there.” | 09:46 |
jtv | It's still not a trivial problem. | 09:46 |
jtv | But there's a lot of room for compromise once we make it clear that it's a random choice. | 09:46 |
jtv | Honesty has its benefits for both parties. :) | 09:46 |
StevenK | Heh | 09:46 |
jtv | As Huw notes, that listing, hard as it tries not to present nonsense, already comes up with some very surprising stuff. | 09:47 |
StevenK | It will come up with at least 9 pofiles you have worked on, and then at least one suggestion | 09:48 |
jtv | Yeah. Filling out that list with suggestions isn't really a very good thing to do. | 09:49 |
jtv | StevenK: it'd be fine with me to vary the length of that list. It's tested behaviour. | 09:51 |
jtv | huwshimi, StevenK: shall I just summarize on the MP? | 09:51 |
StevenK | jtv: Please do. | 09:52 |
jtv | huwshimi also concurs | 09:52 |
jtv | (irl shortcuts irc but it can get confusing) | 09:52 |
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
jtv | StevenK: commented on the MP. | 10:19 |
abentley | czajkowski: I don't know. jam? | 10:27 |
jam | abentley: I don't have czajkowski's question in my backlog, can you give context? | 10:28 |
rick_h_ | jam: czajkowski- does anyone know if we have an intro to bzr video anywhere done ? | 10:28 |
jam | rick_h_: http://wiki.bazaar.canonical.com/Documentation with a section on "Screencasts & Presentations" and an "Installing Bazaar introduction from EmmaJane." | 10:29 |
rick_h_ | czajkowski: ^ | 10:30 |
czajkowski | ahhh | 10:30 |
czajkowski | thanks you | 10:30 |
=== shadeslayer is now known as udsslayer | ||
nigelb | Can a private team be a member of a public team? | 10:49 |
czajkowski | nigelb: why on earth would you want that! | 10:51 |
nigelb | czajkowski: I can think of a bunch of use cases, the most immediate one that I thought of was the possibility of having ~canonical a member of ~ubuntu-etherpad. | 10:52 |
czajkowski | nigelb: it's not need ed for canonical most are already on there and the rest join like everyone else. | 10:53 |
nigelb | czajkowski: I'm not saying it's needed. I was asking if it were possible on LP. | 10:54 |
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
=== Ursinha is now known as Ursinha-afk | ||
=== al-maisan is now known as almaisan-away | ||
=== broder_ is now known as broder | ||
czajkowski | sinzui: is there a way to check LP for Linkedin spam like we do for others like nude or prescription, save people creating questions https://dev.launchpad.net/MaintenanceRotationSchedule | 14:32 |
sinzui | czajkowski, just search for it | 14:34 |
sinzui | if it is an issue, we add it to the list of searches to check | 14:34 |
czajkowski | nods it is | 14:35 |
sinzui | czajkowski, I think this is it: https://launchpad.net/+search?field.text=%22LinkedIn+Corporation%22&field.actions.search=Search | 14:36 |
sinzui | yes, that shows a lot of comments I would hide | 14:37 |
czajkowski | sinzui: bah I was chaanging the old search and it wsnt returning that | 14:37 |
czajkowski | thanks | 14:37 |
czajkowski | this will make mpt happy | 14:37 |
sinzui | czajkowski, it will take 30-60 days for google to drop that match from its index | 14:37 |
czajkowski | ok good to know thanks curtis | 14:38 |
czajkowski | sinzui: at uds, so am working on answers and bugs, though the two current one stump me, only thing not able to do is keep an eye on irc | 14:39 |
sinzui | yes, there are are some hard questions at the moment | 14:39 |
sinzui | I am waiting for caffeine to rattle me some more | 14:40 |
czajkowski | sinzui: bit stumped on https://support.one.ubuntu.com/Ticket/Display.html?id=24331 can you point me in the rihght direction | 14:40 |
sinzui | The user does not understand that Lp requires a registered project | 14:42 |
sinzui | either he needs to register it, or he uses the one already registered | 14:42 |
czajkowski | ack | 14:42 |
sinzui | wow, this guy has been in Lp since 2005 | 14:43 |
=== benji___ is now known as benji | ||
=== Ursinha-afk is now known as Ursinha | ||
cjwatson | Oops, I broke buildbot. Fixing. | 15:04 |
czajkowski | cjwatson: naughty | 15:16 |
sinzui | cjwatson, ping me if you want a review. | 15:44 |
=== matsubara is now known as matsubara-lunch | ||
cjwatson | sinzui: Could you just revert the broken rev for me for now, perhaps? I do want to fix this, but it's going to require pretty extensive changes to this horrible doctest | 16:22 |
sinzui | :( | 16:22 |
cjwatson | sinzui: And I'm going to have difficulty finding enough time to fix in the remaining half-hour of UDS today | 16:22 |
sinzui | I will queue the revert | 16:22 |
cjwatson | So I'd rather unblock buildbot | 16:22 |
cjwatson | Thanks | 16:22 |
sinzui | I think I am the only developer today, and I am writing, not coding | 16:22 |
cjwatson | The whole of xx-copy-packages.txt is utterly built around synchronous copies | 16:23 |
cjwatson | It can probably be patched up by just inserting lots of job.run() kinds of things | 16:23 |
ricmm | hi there, got a question for some launchpad expert | 17:00 |
ricmm | I need to pull the binary debs for a given build in a private PPA | 17:00 |
ricmm | I'm logged in and I got credentials set up, but I cannot find a way to expose the binary file through lplib directly... only by getting the url with binaryFileUrls() | 17:00 |
ricmm | but that the latter I have to do it as a normal browser session, except this is a headless system | 17:01 |
ricmm | is there a way to maybe generate a one-time cookie so I can use urllib to pull the file? or is there a lplib-approved method? | 17:01 |
=== Ursinha is now known as Ursinha-afk | ||
=== Ursinha-afk is now known as Ursinha | ||
sinzui | ricmm, I don't know the answer to your question. I am sure there is no approved way. I think this approach looks viable: http://stackoverflow.com/questions/5082128/how-do-i-authenticate-a-urllib2-script-in-order-to-access-https-web-services-fro | 17:16 |
=== matsubara-lunch is now known as matsubara | ||
czajkowski | sinzui: dashing out again, mailed you re info I was unsure about | 17:49 |
czajkowski | if you have a chance can you look over it | 17:49 |
czajkowski | please | 17:49 |
sinzui | okay | 17:50 |
wallyworld_ | wgrant: StevenK: sinzui: https://pastebin.canonical.com/77402/ | 22:22 |
StevenK | wallyworld_, sinzui: https://www.youtube.com/watch?v=6TiXUF9xbTo ? | 22:56 |
=== Ursinha is now known as Ursinha-afk |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!