/srv/irclogs.ubuntu.com/2012/10/29/#launchpad-dev.txt

wgrantStevenK: Did you ec2 and pre-QA that twisted change?01:43
wgrant(it's a bit awkward that we're upgrading to a new major Twisted release, but cannot QA buildd-manager)01:47
StevenKwgrant: I did ec2 it. It was lightly-QA'd. IE, with a local dev instance, I could ssh to codehost01:48
wgrantOK01:48
StevenKIf that doesn't ally your concerns, I understand. :-)01:49
lifelesslook at it this way01:50
lifelessits UDS, noone needs builders this week01:50
wgrantHah01:50
StevenKlifeless: Oh, so no gems about Twisted's upstream or anything? :-)01:58
lifelessStevenK: twisted are pretty good at backward compat01:59
lifelessStevenK: you probably don't even need to qa it </wink>02:00
wgrantIndeed, so I'm not hugely concerned02:00
wgrantBut it's still something we should really be testing02:00
wallyworld_StevenK: wgrant: do you know where the tests for the nascent uploads overrides are done? i can't see anywhere where the tests set up various scenarios, run the uploads, and check the overrides. but then again, i'm not sure where to look either02:11
StevenKwallyworld_: lib/lp/archiveuploader02:12
* wallyworld_ looks02:12
StevenKwgrant: Can I land https://code.launchpad.net/~stevenk/launchpad/db-destroy-potmsgset-potemplate/+merge/131294 ?02:13
wgrantwallyworld_: A disturbingly large number of nascentupload tests are somewhat incidental results of specific prebuilt sample packages being uploaded02:13
wgrantI'm not sure about this particular case; it may just be untested02:13
wgrantThere are also some archiveuploader-covering doctests in lib/lp/soyuz02:13
wgrantStevenK: Sure02:14
wallyworld_oh, will look around a bit. thanks.02:14
StevenKI think the override policy bits are tested, but can't think of an end-to-end test02:14
wgrantarchiveuploader doesn't really use override policies today02:14
StevenKOh, it's the copier02:15
wallyworld_i am looking for tests which set up various source packages etc and then do uploads to test the various scenarios02:15
StevenKWith the uploader being "Eh, later" IIRC02:15
wallyworld_they mostly seem to be testing hard coded things02:15
StevenKwallyworld_: Both wgrant and I dislike the archiveuploader tests02:16
wallyworld_sadly, i made a change and ran all the tests with nascent in the test file/name and nothing failed02:17
StevenKThey're archaic, slow, mostly doctests and use vast amounts of sampledata.02:17
wallyworld_yeah, so it seems02:17
wgrantuploadprocessor tests a lot of things02:21
wgrantYou really have to run all the archiveuploader tests02:21
wgrant+ soyuz-set-of-uploads02:21
wgrantand a couple of others02:21
wallyworld_ok, will try thise02:22
StevenKSSoU is horrid02:28
StevenKwallyworld_: Have you killed yourself yet?03:09
wgrantStevenK: wtf03:18
wgrantStevenK: Bug #1013013 somehow got tagged as part of r1620803:18
_mup_Bug #1013013: Timeout viewing my own translations <lp-translations> <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by stevenk> < https://launchpad.net/bugs/1013013 >03:18
wgrantOh03:18
wgrantmisread revno03:19
StevenKwgrant: Reading is a good skill.03:19
wgrant16208 and 12088 are both translations changes and adjacent in my mailbox03:19
wgrantLast digit matches, must be right :)03:19
StevenKHaha03:19
StevenKwgrant: Well, first and last. Who cares about the middle bit.03:19
wallyworld_StevenK: no03:22
StevenKwallyworld_: That's good to hear, at least.03:22
wallyworld_just trying to see the best place to add some new tests. making progress03:22
* StevenK loads up OOPS-e8f60c07d46b84f6e939a46d4bc8966c and reaches for the rusty spoon03:22
wallyworld_it does seem that things are scattered in a few places03:23
StevenKwgrant: TranslationsPerson:+index on qas does not love me03:56
StevenKThe query that constantly shows up is ITranslationsPerson.suggestReviewableTranslationFiles() :-(03:57
wgrantThat's the one we didn't fix on Friday, right?03:57
StevenKIndeed03:57
StevenKOh look, it finally worked03:57
StevenK8 page loads later03:57
wgrantIs my query fast?03:58
* StevenK greps out an OOPS ID03:58
StevenKwgrant: Based on the fact that suggest is called after get, I'm going to hazzard a 'yes', but confirming03:59
wgrantStevenK: You should be able to use the inline query log03:59
wgrantJust expand it, /WITH, look for MASSIVE BLOB OF PAIN03:59
StevenK11.434ms 03:59
StevenKSQL-main-slave: WITH recent_pofiles AS (SELECT POFile.id FROM POFileTranslator ...03:59
wgrantYay03:59
StevenK1023.917ms 04:00
StevenKSQL-main-slave: SELECT DISTINCT POFile.currentcount, POFile.date_changed,04:00
StevenKThat's suggest04:00
wgrantAya04:00
StevenKTerrible query is still utterly terrible, news at 1104:00
StevenKwgrant: I was thinking about it over the weekend ...04:00
StevenKwgrant: Could we use the 4 CTEs of doom with an anti-join and limiting to like 50?04:01
StevenKWe only want to suggest a small number of projects, so limiting one of the CTEs to a small number of pofiles doesn't seem like a bad move04:01
wgrantDo we only want to suggest a small number of projects?04:04
wgrantWe need a specific plan in mind for this sort of thing04:04
wgrantFor the previous one, it was that we only cared about recently translated pofiles.04:04
wgrantDo we have such a shortcut here?04:05
StevenKWe have a no_older_than that might be specified04:05
StevenKI'm re-checking what the view does, since Friday was a sleep-induced haze04:05
StevenK        fetch = 5 * (list_length - len(overall))04:06
StevenK        suggestions = self._suggestTargetsForReview(fetch)04:06
StevenKAnd then,         pofiles = person.suggestReviewableTranslationFiles(04:07
StevenK            no_older_than=self.history_horizon)[:max_fetch]04:07
StevenKSo a quicker fix might be tossing max_fetch down one level so the database can actually LIMIT04:08
wgrantDo you have an explain analyze for the existing query?04:09
StevenKBut I like the CTEs better than the ten joins of utter doom, so I'd like to use them if we can04:09
wgrantThe CTEs we used for the previous query are somewhat irrelevant here, as they existed to coerce postgres into choosing a plan style that cannot work here04:09
StevenKwgrant: No, and running queries on mawson is somewhat useless given it has no disk at all currently04:10
wgrantIt's also not completely indexed04:10
wgrantFortunately we have people with access to fast production databases :)04:10
StevenKLet's try staging04:10
StevenKOr prod, right04:11
wgrantAbout 2/3 of the way through indexing the restore, then we will have DF back04:12
wgrantThe big indices are done04:12
StevenKIf you started it on Friday, then it's grown another day04:12
StevenKwgrant: So we have an explain analyze now04:26
wgrantStevenK: What's this query doing?04:30
StevenKwgrant: It's virtually indentical to the get query, except it wants pofiles that the person hasn't contributed to04:30
wgrantStevenK: Right, so you can probably see why the previous optimisation is no longer useful04:31
StevenKYes, lots and lots of pofiles04:32
StevenKWhich is why I suggested limiting the number of pofiles that the CTE returns04:33
wgrantIt wants the last 50. We can't really limit earlier than we do now, because then filtering my give us less than 5004:33
StevenKBleh04:34
StevenKBlah, I can't check on DF due to the DB rebuild04:35
wgrantSo, we want pofiles that are enabled/preferred for translation, that the user has review privileges for, and that the user has not translated recently.04:36
StevenKI was going to have webops repurpose a bunch of pofiles to me so I can see if qas will still load that page04:36
wgrantStevenK: What are you wanting to check?04:36
StevenKI wanted a user id with a 'average' number of pofiles04:37
wgrantWell04:37
StevenKSo I could UPDATE pofile set ...04:37
wgrantWe don't really need that yet04:37
wgrantWe know the query performs terribly for even you now04:37
StevenKTrue04:37
StevenKSo it's qa-ok, but it doesn't fix the bug04:37
wgrantRight04:37
wgrantWe know from experiments last week that the fixed query performs well across a wide range of users, and we have good reason to believe it will behave for everyone.04:38
StevenKRight04:38
StevenKIt's either quick or blistering fast if the person has no pofiles at all04:39
wgrantRight04:39
wgrantBecause we're able to choose a small set to start with04:39
wgrantThat's the key to a fast query04:39
StevenKIt seems the kicker is 'that the user has no translated recently.'04:40
wgrantWell, sort of04:41
wgrantThe kicker is the combination of that and the sort04:41
StevenKRight04:44
StevenKwgrant: So I guess the question is, how can we choose a small set?04:44
wgrantStevenK: Not really04:45
wgrantThe better question is "is the result of this query useful?"04:45
StevenKwgrant: You think suggesting things is not useful?04:47
wgrantStevenK: Well, it's not really a binary thing, but a query is not always sufficiently useful to justify the effort expended in its calculation.04:49
StevenKRight04:50
StevenKWe can't even tell if users actually do go "Oh, yeah, I should do some."04:50
StevenK.. I can't even see the suggestions on Person:+translations04:53
wgrantYou probably don't have review privs in the sense that the query wants them04:54
StevenKAh ha04:54
StevenKwgrant: So, should we kill it entirely?04:55
wgrantIf we can't thoroughly optimise it, deletion is one option04:55
StevenKI can't think of how to optimise it, sadly.04:57
StevenKsteven@undermined:~/launchpad/lp-branches/remove-suggested-translations% bzr di | diffstat -s06:12
StevenK 5 files changed, 8 insertions(+), 155 deletions(-)06:12
StevenKwgrant, wallyworld_: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? Or shall I shake jtv until an opinion falls out?06:44
wallyworld_StevenK: best to check with a SME i think06:45
wgrantStevenK: I'd like to confirm with a rosettay person06:45
StevenKwallyworld_: A what?06:45
wallyworld_subject matter expert06:46
StevenKwgrant: They're kinda thin on the ground. :-(06:46
lifelessthey should be online in a few minutes06:46
wgrantStevenK: You need to talk to dpm anyway :)06:47
StevenKnightly.sh might have actually run update-stats by now06:53
StevenKNot sure when it's kicked off in the chain06:54
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
StevenKjtv: Hai. Do you have one sec to cast your eye over https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? I'm not after a review at all, just your opinion on what the MP is doing.09:06
jtvI probably do at that.  Hi.09:06
=== yofel_ is now known as yofel
jtvStevenK: best thing I can do right now is go over why it is the way it is.09:08
jtvThe purpose was to guide translators towards where their efforts can do the most good.09:08
StevenKjtv: Right, and I understand. I just don't think it helps if the query that is trying to do that takes 4 seconds and causes continous timeouts. :-)09:09
jtvAbsolutely.  Just going over what it tries to achieve so as to help inform the decision.09:10
jtvThe obvious thing to do would be to steer translators to the highest-priority unfinished translations.  But that leads to duplicate effort, for several reasons:09:10
jtv - multiple translators will be looking at the same page at the same time.09:11
jtv - unreviewed translations pile up before reviewers can get to them.09:11
jtv - viewers are incorrectly concentrated on translations that may already be finished until the “100% complete” statistics have filtered through.09:12
jtvSo we wanted a bit of random spread.09:12
StevenKjtv: If you do have some suggestions on how to stop the query asking that question having to join through ten tables, that would help to.09:14
jtvI don't think I'll have time to look into that, I'm afraid; but I'm sure there are more efficient ways of achieving something that's similar enough.09:14
jtvWe built that functionality from the UI down, and so we may have overspecified it slightly.09:15
StevenKjtv: In which case, some clues about where to look would be nice.09:15
jtvHang on, I'll ask huwshimi09:15
StevenKI'm just playing in the shallow end of translations. :-)09:15
jtvStevenK: talking irl with Huw here09:19
jtvStevenK: interestingly, it looks as if the suggestions are horribly broken.09:23
jtvHuw gets a bunch of suggestions to review Low German translations.09:23
StevenKHaha09:23
nigelbhaha, whaat09:23
jtvHe is not now, nor ever has been, associated with Low German09:23
jtvOh!09:24
=== almaisan-away is now known as al-maisan
jtvHe just discovered that he is a member of some translation teams for complicated and as-yet unexplored reasons.09:24
jtvStevenK: Huw just asked a good question — is the query's performance very sensitive to the number of results?09:27
StevenKNope09:27
StevenKIt's terrible no matter who you're asking about09:28
jtvDidn't think so.  :/09:28
jtvWell it's particularly timeout-prone in my case.  :)09:28
StevenKI bet09:28
jtvStevenK: here's a solution we just discussed...09:30
jtvWe could stop filling out the list with random suggestions, but underneath the list, have a “take me to a random project that needs my help” link.09:31
StevenKjtv: Is there a way to fetch that quickly?09:33
jtvStevenK: that means you never execute the query until the user requests it, but its functionality is still there _and_ it's explicit, not cryptically mixed in with more obvious functionality.09:33
StevenKWell, moving the query from one page to another just means another page will timeout09:34
jtvStevenK: Wouldn't need to be in the generated HTML — could link to a redirect.09:34
jtvSo at worst, the redirect would time out.  But there'd be a lot fewer hits on it.09:34
StevenKjtv: Right, which means another page will timeout, and means I fix a critical bug by creating another.09:38
StevenKIE, not on.09:38
jtvYes, but!09:38
jtvIt'd allow us to remove some of the selection criteria.09:38
jtvAs Huw just put it, take a slot-machine approach.09:38
jtvIf we have a link to a random project to work on, people may be a lot more acceptable of iffy guesses.09:38
jtvFor example, we won't need to exclude projects that the user has already been translating on.09:39
StevenKOh, so we just one random project/distro where rosetta_usage == ServiceUsage.LAUNCHPAD ?09:39
StevenKBecause that's fast09:39
StevenKAnd if it isn't, it can indexed09:39
jtvA bit more than that, but in principle, yes.09:40
StevenKjtv: Do you have time to scribble all this down on the MP?09:40
jtvHuw suggested we might make the query less sensitive to the user's identity.09:40
jtvShould be able to, yes.09:40
jtvInstead of checking translation permissions, we could just limit the suggestions to projects with sufficiently liberal permission models.09:41
StevenKRight09:41
StevenKWhich is nice and easy to make performant.09:41
jtvThe core reason the query was so slow, really, was that we're pushing these suggestions in the user's face without explanation.09:43
jtvThe suggestions have to work hard to avoid including absolute nonsense.09:43
StevenKjtv: So, we should be able to 1) Grab a number of projects that fit that, and then grab a random unreviewed POFile from those and job's done?09:43
jtvStevenK: more or less — there's still the problem that you've got to look at multiple tables, several of which may provide a reason why a particular record is not a candidate.09:45
jtvSo you can't quite say “give me one project, and then we'll look for a POFile there.”09:46
jtvIt's still not a trivial problem.09:46
jtvBut there's a lot of room for compromise once we make it clear that it's a random choice.09:46
jtvHonesty has its benefits for both parties.  :)09:46
StevenKHeh09:46
jtvAs Huw notes, that listing, hard as it tries not to present nonsense, already comes up with some very surprising stuff.09:47
StevenKIt will come up with at least 9 pofiles you have worked on, and then at least one suggestion09:48
jtvYeah.  Filling out that list with suggestions isn't really a very good thing to do.09:49
jtvStevenK: it'd be fine with me to vary the length of that list.  It's tested behaviour.09:51
jtvhuwshimi, StevenK: shall I just summarize on the MP?09:51
StevenKjtv: Please do.09:52
jtvhuwshimi also concurs09:52
jtv(irl shortcuts irc but it can get confusing)09:52
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
jtvStevenK: commented on the MP.10:19
abentleyczajkowski: I don't know.  jam?10:27
jamabentley: I don't have czajkowski's question in my backlog, can you give context?10:28
rick_h_jam:  czajkowski- does anyone know if we have an intro to bzr video anywhere done ?10:28
jamrick_h_: http://wiki.bazaar.canonical.com/Documentation with a section on "Screencasts & Presentations" and an "Installing Bazaar introduction from EmmaJane."10:29
rick_h_czajkowski: ^10:30
czajkowskiahhh10:30
czajkowskithanks you10:30
=== shadeslayer is now known as udsslayer
nigelbCan a private team be a member of a public team?10:49
czajkowskinigelb: why on earth would you want that!10:51
nigelbczajkowski: I can think of a bunch of use cases, the most immediate one that I thought of was the possibility of having ~canonical a member of ~ubuntu-etherpad.10:52
czajkowskinigelb: it's not need ed for canonical most are already on there and the rest join like everyone else.10:53
nigelbczajkowski: I'm not saying it's needed. I was asking if it were possible on LP.10:54
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
=== Ursinha is now known as Ursinha-afk
=== al-maisan is now known as almaisan-away
=== broder_ is now known as broder
czajkowskisinzui: is there a way to check LP for Linkedin spam like we do for others like nude or prescription, save people creating questions https://dev.launchpad.net/MaintenanceRotationSchedule14:32
sinzuiczajkowski, just search for it14:34
sinzuiif it is an issue, we add it to the list of searches to check14:34
czajkowskinods it is14:35
sinzuiczajkowski, I think this is it: https://launchpad.net/+search?field.text=%22LinkedIn+Corporation%22&field.actions.search=Search14:36
sinzuiyes, that shows a lot of comments I would hide14:37
czajkowskisinzui: bah I was chaanging the old search and it wsnt returning that14:37
czajkowskithanks14:37
czajkowskithis will make mpt happy14:37
sinzuiczajkowski, it will take 30-60 days for google to drop that match from its index14:37
czajkowskiok good to know thanks curtis14:38
czajkowskisinzui: at uds, so am working on answers and bugs, though the two current one stump me, only thing not able to do is keep an eye on irc14:39
sinzuiyes, there are are some hard questions at the moment14:39
sinzuiI am waiting for caffeine to rattle me some more14:40
czajkowskisinzui: bit stumped on https://support.one.ubuntu.com/Ticket/Display.html?id=24331 can you point me in the rihght direction14:40
sinzuiThe user does not understand that Lp requires a registered project14:42
sinzuieither he needs to register it, or he uses the one already registered14:42
czajkowskiack14:42
sinzuiwow, this guy has been in Lp since 200514:43
=== benji___ is now known as benji
=== Ursinha-afk is now known as Ursinha
cjwatsonOops, I broke buildbot.  Fixing.15:04
czajkowskicjwatson: naughty15:16
sinzuicjwatson, ping me if you want a review.15:44
=== matsubara is now known as matsubara-lunch
cjwatsonsinzui: Could you just revert the broken rev for me for now, perhaps?  I do want to fix this, but it's going to require pretty extensive changes to this horrible doctest16:22
sinzui:(16:22
cjwatsonsinzui: And I'm going to have difficulty finding enough time to fix in the remaining half-hour of UDS today16:22
sinzuiI will queue the revert16:22
cjwatsonSo I'd rather unblock buildbot16:22
cjwatsonThanks16:22
sinzuiI think I am the only developer today, and I am writing, not coding16:22
cjwatsonThe whole of xx-copy-packages.txt is utterly built around synchronous copies16:23
cjwatsonIt can probably be patched up by just inserting lots of job.run() kinds of things16:23
ricmmhi there, got a question for some launchpad expert17:00
ricmmI need to pull the binary debs for a given build in a private PPA17:00
ricmmI'm logged in and I got credentials set up, but I cannot find a way to expose the binary file through lplib directly... only by getting the url with binaryFileUrls()17:00
ricmmbut that the latter I have to do it as a normal browser session, except this is a headless system17:01
ricmmis there a way to maybe generate a one-time cookie so I can use urllib to pull the file? or is there a lplib-approved method?17:01
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
sinzuiricmm, I don't know the answer to your question. I am sure there is no approved way. I think this approach looks viable: http://stackoverflow.com/questions/5082128/how-do-i-authenticate-a-urllib2-script-in-order-to-access-https-web-services-fro17:16
=== matsubara-lunch is now known as matsubara
czajkowskisinzui: dashing out again, mailed you re info I was unsure about17:49
czajkowskiif you have a chance can you look over it17:49
czajkowskiplease17:49
sinzuiokay17:50
wallyworld_wgrant: StevenK: sinzui: https://pastebin.canonical.com/77402/22:22
StevenKwallyworld_, sinzui: https://www.youtube.com/watch?v=6TiXUF9xbTo ?22:56
=== Ursinha is now known as Ursinha-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!