[01:43] <wgrant> StevenK: Did you ec2 and pre-QA that twisted change?
[01:47] <wgrant> (it's a bit awkward that we're upgrading to a new major Twisted release, but cannot QA buildd-manager)
[01:48] <StevenK> wgrant: I did ec2 it. It was lightly-QA'd. IE, with a local dev instance, I could ssh to codehost
[01:48] <wgrant> OK
[01:49] <StevenK> If that doesn't ally your concerns, I understand. :-)
[01:50] <lifeless> look at it this way
[01:50] <lifeless> its UDS, noone needs builders this week
[01:50] <wgrant> Hah
[01:58] <StevenK> lifeless: Oh, so no gems about Twisted's upstream or anything? :-)
[01:59] <lifeless> StevenK: twisted are pretty good at backward compat
[02:00] <lifeless> StevenK: you probably don't even need to qa it </wink>
[02:00] <wgrant> Indeed, so I'm not hugely concerned
[02:00] <wgrant> But it's still something we should really be testing
[02:11] <wallyworld_> StevenK: wgrant: do you know where the tests for the nascent uploads overrides are done? i can't see anywhere where the tests set up various scenarios, run the uploads, and check the overrides. but then again, i'm not sure where to look either
[02:12] <StevenK> wallyworld_: lib/lp/archiveuploader
[02:12]  * wallyworld_ looks
[02:13] <StevenK> wgrant: Can I land https://code.launchpad.net/~stevenk/launchpad/db-destroy-potmsgset-potemplate/+merge/131294 ?
[02:13] <wgrant> wallyworld_: A disturbingly large number of nascentupload tests are somewhat incidental results of specific prebuilt sample packages being uploaded
[02:13] <wgrant> I'm not sure about this particular case; it may just be untested
[02:13] <wgrant> There are also some archiveuploader-covering doctests in lib/lp/soyuz
[02:14] <wgrant> StevenK: Sure
[02:14] <wallyworld_> oh, will look around a bit. thanks.
[02:14] <StevenK> I think the override policy bits are tested, but can't think of an end-to-end test
[02:14] <wgrant> archiveuploader doesn't really use override policies today
[02:15] <StevenK> Oh, it's the copier
[02:15] <wallyworld_> i am looking for tests which set up various source packages etc and then do uploads to test the various scenarios
[02:15] <StevenK> With the uploader being "Eh, later" IIRC
[02:15] <wallyworld_> they mostly seem to be testing hard coded things
[02:16] <StevenK> wallyworld_: Both wgrant and I dislike the archiveuploader tests
[02:17] <wallyworld_> sadly, i made a change and ran all the tests with nascent in the test file/name and nothing failed
[02:17] <StevenK> They're archaic, slow, mostly doctests and use vast amounts of sampledata.
[02:17] <wallyworld_> yeah, so it seems
[02:21] <wgrant> uploadprocessor tests a lot of things
[02:21] <wgrant> You really have to run all the archiveuploader tests
[02:21] <wgrant> + soyuz-set-of-uploads
[02:21] <wgrant> and a couple of others
[02:22] <wallyworld_> ok, will try thise
[02:28] <StevenK> SSoU is horrid
[03:09] <StevenK> wallyworld_: Have you killed yourself yet?
[03:18] <wgrant> StevenK: wtf
[03:18] <wgrant> StevenK: Bug #1013013 somehow got tagged as part of r16208
[03:18] <_mup_> Bug #1013013: Timeout viewing my own translations <lp-translations> <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by stevenk> < https://launchpad.net/bugs/1013013 >
[03:18] <wgrant> Oh
[03:19] <wgrant> misread revno
[03:19] <StevenK> wgrant: Reading is a good skill.
[03:19] <wgrant> 16208 and 12088 are both translations changes and adjacent in my mailbox
[03:19] <wgrant> Last digit matches, must be right :)
[03:19] <StevenK> Haha
[03:19] <StevenK> wgrant: Well, first and last. Who cares about the middle bit.
[03:22] <wallyworld_> StevenK: no
[03:22] <StevenK> wallyworld_: That's good to hear, at least.
[03:22] <wallyworld_> just trying to see the best place to add some new tests. making progress
[03:22]  * StevenK loads up OOPS-e8f60c07d46b84f6e939a46d4bc8966c and reaches for the rusty spoon
[03:23] <wallyworld_> it does seem that things are scattered in a few places
[03:56] <StevenK> wgrant: TranslationsPerson:+index on qas does not love me
[03:57] <StevenK> The query that constantly shows up is ITranslationsPerson.suggestReviewableTranslationFiles() :-(
[03:57] <wgrant> That's the one we didn't fix on Friday, right?
[03:57] <StevenK> Indeed
[03:57] <StevenK> Oh look, it finally worked
[03:57] <StevenK> 8 page loads later
[03:58] <wgrant> Is my query fast?
[03:58]  * StevenK greps out an OOPS ID
[03:59] <StevenK> wgrant: Based on the fact that suggest is called after get, I'm going to hazzard a 'yes', but confirming
[03:59] <wgrant> StevenK: You should be able to use the inline query log
[03:59] <wgrant> Just expand it, /WITH, look for MASSIVE BLOB OF PAIN
[03:59] <StevenK> 11.434ms 	
[03:59] <StevenK> SQL-main-slave: WITH recent_pofiles AS (SELECT POFile.id FROM POFileTranslator ...
[03:59] <wgrant> Yay
[04:00] <StevenK> 1023.917ms 	
[04:00] <StevenK> SQL-main-slave: SELECT DISTINCT POFile.currentcount, POFile.date_changed,
[04:00] <StevenK> That's suggest
[04:00] <wgrant> Aya
[04:00] <StevenK> Terrible query is still utterly terrible, news at 11
[04:00] <StevenK> wgrant: I was thinking about it over the weekend ...
[04:01] <StevenK> wgrant: Could we use the 4 CTEs of doom with an anti-join and limiting to like 50?
[04:01] <StevenK> We only want to suggest a small number of projects, so limiting one of the CTEs to a small number of pofiles doesn't seem like a bad move
[04:04] <wgrant> Do we only want to suggest a small number of projects?
[04:04] <wgrant> We need a specific plan in mind for this sort of thing
[04:04] <wgrant> For the previous one, it was that we only cared about recently translated pofiles.
[04:05] <wgrant> Do we have such a shortcut here?
[04:05] <StevenK> We have a no_older_than that might be specified
[04:05] <StevenK> I'm re-checking what the view does, since Friday was a sleep-induced haze
[04:06] <StevenK>         fetch = 5 * (list_length - len(overall))
[04:06] <StevenK>         suggestions = self._suggestTargetsForReview(fetch)
[04:07] <StevenK> And then,         pofiles = person.suggestReviewableTranslationFiles(
[04:07] <StevenK>             no_older_than=self.history_horizon)[:max_fetch]
[04:08] <StevenK> So a quicker fix might be tossing max_fetch down one level so the database can actually LIMIT
[04:09] <wgrant> Do you have an explain analyze for the existing query?
[04:09] <StevenK> But I like the CTEs better than the ten joins of utter doom, so I'd like to use them if we can
[04:09] <wgrant> The CTEs we used for the previous query are somewhat irrelevant here, as they existed to coerce postgres into choosing a plan style that cannot work here
[04:10] <StevenK> wgrant: No, and running queries on mawson is somewhat useless given it has no disk at all currently
[04:10] <wgrant> It's also not completely indexed
[04:10] <wgrant> Fortunately we have people with access to fast production databases :)
[04:10] <StevenK> Let's try staging
[04:11] <StevenK> Or prod, right
[04:12] <wgrant> About 2/3 of the way through indexing the restore, then we will have DF back
[04:12] <wgrant> The big indices are done
[04:12] <StevenK> If you started it on Friday, then it's grown another day
[04:26] <StevenK> wgrant: So we have an explain analyze now
[04:30] <wgrant> StevenK: What's this query doing?
[04:30] <StevenK> wgrant: It's virtually indentical to the get query, except it wants pofiles that the person hasn't contributed to
[04:31] <wgrant> StevenK: Right, so you can probably see why the previous optimisation is no longer useful
[04:32] <StevenK> Yes, lots and lots of pofiles
[04:33] <StevenK> Which is why I suggested limiting the number of pofiles that the CTE returns
[04:33] <wgrant> It wants the last 50. We can't really limit earlier than we do now, because then filtering my give us less than 50
[04:34] <StevenK> Bleh
[04:35] <StevenK> Blah, I can't check on DF due to the DB rebuild
[04:36] <wgrant> So, we want pofiles that are enabled/preferred for translation, that the user has review privileges for, and that the user has not translated recently.
[04:36] <StevenK> I was going to have webops repurpose a bunch of pofiles to me so I can see if qas will still load that page
[04:36] <wgrant> StevenK: What are you wanting to check?
[04:37] <StevenK> I wanted a user id with a 'average' number of pofiles
[04:37] <wgrant> Well
[04:37] <StevenK> So I could UPDATE pofile set ...
[04:37] <wgrant> We don't really need that yet
[04:37] <wgrant> We know the query performs terribly for even you now
[04:37] <StevenK> True
[04:37] <StevenK> So it's qa-ok, but it doesn't fix the bug
[04:37] <wgrant> Right
[04:38] <wgrant> We know from experiments last week that the fixed query performs well across a wide range of users, and we have good reason to believe it will behave for everyone.
[04:38] <StevenK> Right
[04:39] <StevenK> It's either quick or blistering fast if the person has no pofiles at all
[04:39] <wgrant> Right
[04:39] <wgrant> Because we're able to choose a small set to start with
[04:39] <wgrant> That's the key to a fast query
[04:40] <StevenK> It seems the kicker is 'that the user has no translated recently.'
[04:41] <wgrant> Well, sort of
[04:41] <wgrant> The kicker is the combination of that and the sort
[04:44] <StevenK> Right
[04:44] <StevenK> wgrant: So I guess the question is, how can we choose a small set?
[04:45] <wgrant> StevenK: Not really
[04:45] <wgrant> The better question is "is the result of this query useful?"
[04:47] <StevenK> wgrant: You think suggesting things is not useful?
[04:49] <wgrant> StevenK: Well, it's not really a binary thing, but a query is not always sufficiently useful to justify the effort expended in its calculation.
[04:50] <StevenK> Right
[04:50] <StevenK> We can't even tell if users actually do go "Oh, yeah, I should do some."
[04:53] <StevenK> .. I can't even see the suggestions on Person:+translations
[04:54] <wgrant> You probably don't have review privs in the sense that the query wants them
[04:54] <StevenK> Ah ha
[04:55] <StevenK> wgrant: So, should we kill it entirely?
[04:55] <wgrant> If we can't thoroughly optimise it, deletion is one option
[04:57] <StevenK> I can't think of how to optimise it, sadly.
[06:12] <StevenK> steven@undermined:~/launchpad/lp-branches/remove-suggested-translations% bzr di | diffstat -s
[06:12] <StevenK>  5 files changed, 8 insertions(+), 155 deletions(-)
[06:44] <StevenK> wgrant, wallyworld_: Do either of you want to review https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? Or shall I shake jtv until an opinion falls out?
[06:45] <wallyworld_> StevenK: best to check with a SME i think
[06:45] <wgrant> StevenK: I'd like to confirm with a rosettay person
[06:45] <StevenK> wallyworld_: A what?
[06:46] <wallyworld_> subject matter expert
[06:46] <StevenK> wgrant: They're kinda thin on the ground. :-(
[06:46] <lifeless> they should be online in a few minutes
[06:47] <wgrant> StevenK: You need to talk to dpm anyway :)
[06:53] <StevenK> nightly.sh might have actually run update-stats by now
[06:54] <StevenK> Not sure when it's kicked off in the chain
[09:06] <StevenK> jtv: Hai. Do you have one sec to cast your eye over https://code.launchpad.net/~stevenk/launchpad/remove-suggested-translations/+merge/131820 ? I'm not after a review at all, just your opinion on what the MP is doing.
[09:06] <jtv> I probably do at that.  Hi.
[09:08] <jtv> StevenK: best thing I can do right now is go over why it is the way it is.
[09:08] <jtv> The purpose was to guide translators towards where their efforts can do the most good.
[09:09] <StevenK> jtv: Right, and I understand. I just don't think it helps if the query that is trying to do that takes 4 seconds and causes continous timeouts. :-)
[09:10] <jtv> Absolutely.  Just going over what it tries to achieve so as to help inform the decision.
[09:10] <jtv> The obvious thing to do would be to steer translators to the highest-priority unfinished translations.  But that leads to duplicate effort, for several reasons:
[09:11] <jtv>  - multiple translators will be looking at the same page at the same time.
[09:11] <jtv>  - unreviewed translations pile up before reviewers can get to them.
[09:12] <jtv>  - viewers are incorrectly concentrated on translations that may already be finished until the “100% complete” statistics have filtered through.
[09:12] <jtv> So we wanted a bit of random spread.
[09:14] <StevenK> jtv: If you do have some suggestions on how to stop the query asking that question having to join through ten tables, that would help to.
[09:14] <jtv> I don't think I'll have time to look into that, I'm afraid; but I'm sure there are more efficient ways of achieving something that's similar enough.
[09:15] <jtv> We built that functionality from the UI down, and so we may have overspecified it slightly.
[09:15] <StevenK> jtv: In which case, some clues about where to look would be nice.
[09:15] <jtv> Hang on, I'll ask huwshimi
[09:15] <StevenK> I'm just playing in the shallow end of translations. :-)
[09:19] <jtv> StevenK: talking irl with Huw here
[09:23] <jtv> StevenK: interestingly, it looks as if the suggestions are horribly broken.
[09:23] <jtv> Huw gets a bunch of suggestions to review Low German translations.
[09:23] <StevenK> Haha
[09:23] <nigelb> haha, whaat
[09:23] <jtv> He is not now, nor ever has been, associated with Low German
[09:24] <jtv> Oh!
[09:24] <jtv> He just discovered that he is a member of some translation teams for complicated and as-yet unexplored reasons.
[09:27] <jtv> StevenK: Huw just asked a good question — is the query's performance very sensitive to the number of results?
[09:27] <StevenK> Nope
[09:28] <StevenK> It's terrible no matter who you're asking about
[09:28] <jtv> Didn't think so.  :/
[09:28] <jtv> Well it's particularly timeout-prone in my case.  :)
[09:28] <StevenK> I bet
[09:30] <jtv> StevenK: here's a solution we just discussed...
[09:31] <jtv> We could stop filling out the list with random suggestions, but underneath the list, have a “take me to a random project that needs my help” link.
[09:33] <StevenK> jtv: Is there a way to fetch that quickly?
[09:33] <jtv> StevenK: that means you never execute the query until the user requests it, but its functionality is still there _and_ it's explicit, not cryptically mixed in with more obvious functionality.
[09:34] <StevenK> Well, moving the query from one page to another just means another page will timeout
[09:34] <jtv> StevenK: Wouldn't need to be in the generated HTML — could link to a redirect.
[09:34] <jtv> So at worst, the redirect would time out.  But there'd be a lot fewer hits on it.
[09:38] <StevenK> jtv: Right, which means another page will timeout, and means I fix a critical bug by creating another.
[09:38] <StevenK> IE, not on.
[09:38] <jtv> Yes, but!
[09:38] <jtv> It'd allow us to remove some of the selection criteria.
[09:38] <jtv> As Huw just put it, take a slot-machine approach.
[09:38] <jtv> If we have a link to a random project to work on, people may be a lot more acceptable of iffy guesses.
[09:39] <jtv> For example, we won't need to exclude projects that the user has already been translating on.
[09:39] <StevenK> Oh, so we just one random project/distro where rosetta_usage == ServiceUsage.LAUNCHPAD ?
[09:39] <StevenK> Because that's fast
[09:39] <StevenK> And if it isn't, it can indexed
[09:40] <jtv> A bit more than that, but in principle, yes.
[09:40] <StevenK> jtv: Do you have time to scribble all this down on the MP?
[09:40] <jtv> Huw suggested we might make the query less sensitive to the user's identity.
[09:40] <jtv> Should be able to, yes.
[09:41] <jtv> Instead of checking translation permissions, we could just limit the suggestions to projects with sufficiently liberal permission models.
[09:41] <StevenK> Right
[09:41] <StevenK> Which is nice and easy to make performant.
[09:43] <jtv> The core reason the query was so slow, really, was that we're pushing these suggestions in the user's face without explanation.
[09:43] <jtv> The suggestions have to work hard to avoid including absolute nonsense.
[09:43] <StevenK> jtv: So, we should be able to 1) Grab a number of projects that fit that, and then grab a random unreviewed POFile from those and job's done?
[09:45] <jtv> StevenK: more or less — there's still the problem that you've got to look at multiple tables, several of which may provide a reason why a particular record is not a candidate.
[09:46] <jtv> So you can't quite say “give me one project, and then we'll look for a POFile there.”
[09:46] <jtv> It's still not a trivial problem.
[09:46] <jtv> But there's a lot of room for compromise once we make it clear that it's a random choice.
[09:46] <jtv> Honesty has its benefits for both parties.  :)
[09:46] <StevenK> Heh
[09:47] <jtv> As Huw notes, that listing, hard as it tries not to present nonsense, already comes up with some very surprising stuff.
[09:48] <StevenK> It will come up with at least 9 pofiles you have worked on, and then at least one suggestion
[09:49] <jtv> Yeah.  Filling out that list with suggestions isn't really a very good thing to do.
[09:51] <jtv> StevenK: it'd be fine with me to vary the length of that list.  It's tested behaviour.
[09:51] <jtv> huwshimi, StevenK: shall I just summarize on the MP?
[09:52] <StevenK> jtv: Please do.
[09:52] <jtv> huwshimi also concurs
[09:52] <jtv> (irl shortcuts irc but it can get confusing)
[10:19] <jtv> StevenK: commented on the MP.
[10:27] <abentley> czajkowski: I don't know.  jam?
[10:28] <jam> abentley: I don't have czajkowski's question in my backlog, can you give context?
[10:28] <rick_h_> jam:  czajkowski- does anyone know if we have an intro to bzr video anywhere done ?
[10:29] <jam> rick_h_: http://wiki.bazaar.canonical.com/Documentation with a section on "Screencasts & Presentations" and an "Installing Bazaar introduction from EmmaJane."
[10:30] <rick_h_> czajkowski: ^
[10:30] <czajkowski> ahhh
[10:30] <czajkowski> thanks you
[10:49] <nigelb> Can a private team be a member of a public team?
[10:51] <czajkowski> nigelb: why on earth would you want that!
[10:52] <nigelb> czajkowski: I can think of a bunch of use cases, the most immediate one that I thought of was the possibility of having ~canonical a member of ~ubuntu-etherpad.
[10:53] <czajkowski> nigelb: it's not need ed for canonical most are already on there and the rest join like everyone else.
[10:54] <nigelb> czajkowski: I'm not saying it's needed. I was asking if it were possible on LP.
[14:32] <czajkowski> sinzui: is there a way to check LP for Linkedin spam like we do for others like nude or prescription, save people creating questions https://dev.launchpad.net/MaintenanceRotationSchedule
[14:34] <sinzui> czajkowski, just search for it
[14:34] <sinzui> if it is an issue, we add it to the list of searches to check
[14:35] <czajkowski> nods it is
[14:36] <sinzui> czajkowski, I think this is it: https://launchpad.net/+search?field.text=%22LinkedIn+Corporation%22&field.actions.search=Search
[14:37] <sinzui> yes, that shows a lot of comments I would hide
[14:37] <czajkowski> sinzui: bah I was chaanging the old search and it wsnt returning that
[14:37] <czajkowski> thanks
[14:37] <czajkowski> this will make mpt happy
[14:37] <sinzui> czajkowski, it will take 30-60 days for google to drop that match from its index
[14:38] <czajkowski> ok good to know thanks curtis
[14:39] <czajkowski> sinzui: at uds, so am working on answers and bugs, though the two current one stump me, only thing not able to do is keep an eye on irc
[14:39] <sinzui> yes, there are are some hard questions at the moment
[14:40] <sinzui> I am waiting for caffeine to rattle me some more
[14:40] <czajkowski> sinzui: bit stumped on https://support.one.ubuntu.com/Ticket/Display.html?id=24331 can you point me in the rihght direction
[14:42] <sinzui> The user does not understand that Lp requires a registered project
[14:42] <sinzui> either he needs to register it, or he uses the one already registered
[14:42] <czajkowski> ack
[14:43] <sinzui> wow, this guy has been in Lp since 2005
[15:04] <cjwatson> Oops, I broke buildbot.  Fixing.
[15:16] <czajkowski> cjwatson: naughty
[15:44] <sinzui> cjwatson, ping me if you want a review.
[16:22] <cjwatson> sinzui: Could you just revert the broken rev for me for now, perhaps?  I do want to fix this, but it's going to require pretty extensive changes to this horrible doctest
[16:22] <sinzui> :(
[16:22] <cjwatson> sinzui: And I'm going to have difficulty finding enough time to fix in the remaining half-hour of UDS today
[16:22] <sinzui> I will queue the revert
[16:22] <cjwatson> So I'd rather unblock buildbot
[16:22] <cjwatson> Thanks
[16:22] <sinzui> I think I am the only developer today, and I am writing, not coding
[16:23] <cjwatson> The whole of xx-copy-packages.txt is utterly built around synchronous copies
[16:23] <cjwatson> It can probably be patched up by just inserting lots of job.run() kinds of things
[17:00] <ricmm> hi there, got a question for some launchpad expert
[17:00] <ricmm> I need to pull the binary debs for a given build in a private PPA
[17:00] <ricmm> I'm logged in and I got credentials set up, but I cannot find a way to expose the binary file through lplib directly... only by getting the url with binaryFileUrls()
[17:01] <ricmm> but that the latter I have to do it as a normal browser session, except this is a headless system
[17:01] <ricmm> is there a way to maybe generate a one-time cookie so I can use urllib to pull the file? or is there a lplib-approved method?
[17:16] <sinzui> ricmm, I don't know the answer to your question. I am sure there is no approved way. I think this approach looks viable: http://stackoverflow.com/questions/5082128/how-do-i-authenticate-a-urllib2-script-in-order-to-access-https-web-services-fro
[17:49] <czajkowski> sinzui: dashing out again, mailed you re info I was unsure about
[17:49] <czajkowski> if you have a chance can you look over it
[17:49] <czajkowski> please
[17:50] <sinzui> okay
[22:22] <wallyworld_> wgrant: StevenK: sinzui: https://pastebin.canonical.com/77402/
[22:56] <StevenK> wallyworld_, sinzui: https://www.youtube.com/watch?v=6TiXUF9xbTo ?