[00:06] <StevenK> Somebody should blow sampledata up
[01:18] <StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/performant-getSourcesForDeletion/+merge/153704
[01:21] <wgrant> StevenK: Have you checked that my theory is correct?
[01:21] <StevenK> It sounded plausible to me
[01:22] <wgrant> A query to check that scheduleddeletiondate is set iff (published or has published (or unremoved? can't remember) binaries) would give me more confidence
[01:22] <wgrant> I guess if you check scheduleddeletiondate on the binaries it might work
[01:25] <wgrant> Though it's not going to corrupt data if my assumption is incorrect, we should still try to verify it against existing data.
[01:36] <StevenK> wgrant: Query is http://pastebin.ubuntu.com/5624173/, results http://pastebin.ubuntu.com/5624172/
[01:46] <wgrant> StevenK: https://pastebin.canonical.com/86983/
[01:48] <StevenK> wgrant: http://pastebin.ubuntu.com/5624196/
[01:48] <wgrant> I know
[01:48] <StevenK> Right
[01:48] <wgrant> So it looks like it might be right, but maybe not
[01:48] <wgrant> We should probably investigate those cases to see what might be up
[01:48] <wgrant> They may indicate publisher bugs
[01:48] <StevenK> Which cases?
[01:49] <wgrant> Those 20
[01:49] <wgrant> It's also possible that a publisher run will fix those
[01:49] <wgrant> Because this invariant is only eventually consistent
[01:49] <StevenK> Well, let's see on prod
[01:49] <StevenK> Since qas is a bit odd
[01:49] <wgrant> Oh
[01:50] <wgrant> You also check that the binary is actually published
[01:50] <wgrant> Rather than condemned
[01:50] <StevenK> That was what the old method did
[01:50] <wgrant> And I only changed the archive in one part of the query
[01:50] <StevenK> Haha
[01:50] <StevenK> Oh
[01:51] <StevenK> Right, so the results are fairly useless
[01:52] <wgrant> That also explains why it ran so quickly
[01:52] <StevenK> If you're running it on sour, I'll kill mine
[01:52] <wgrant> I am
[01:53] <StevenK> Mine has been killed
[02:06] <StevenK> wgrant: That query with 32 for both archives is 0 rows
[03:14] <StevenK> wgrant: What did that query return for PRIMARY?
[03:15] <wgrant> StevenK: 20000 rows
[03:16] <StevenK> Bloody hell
[03:16] <StevenK> I don't think Archive:+delete-packages works for PRIMARY anyway?
[03:16] <wgrant> Sure
[03:16] <wgrant> But the primary archive and PPAs use the same domination code.
[03:17] <wgrant> Any problem in the primary archive can apply to another archive.
[03:17] <wgrant> So it is a useful pathological test case.
[03:17] <StevenK> Right, so do you want to split up the work, or what is your plan?
[03:20] <wgrant> Hmm, overrides, maybe
[03:20]  * wgrant gives up
[03:20] <StevenK> Overrides will indeed fuck that up
[03:20] <StevenK> Thinking about it
[03:20] <StevenK> No overrides for PPAs
[03:32] <StevenK> wgrant: So I think it's plausible that overrides won't be doing that query any favours.
[03:34] <wgrant> Running it over the first 10k PPAs gives about 500 results
[03:35] <wgrant> Possibly explainable by copies
[03:35] <wgrant> I guess it'll do
[03:36] <StevenK> wgrant: Wait, how do copies explain it?
[03:36] <wgrant> StevenK: Undeletions
[03:36] <wgrant> If I delete something, let it be removed, then revive it
[03:37] <StevenK> Oh, the necromancy style of copies
[03:37] <wgrant> The old pubs may have been removed from disk, despite there now being live binaries
[03:51] <StevenK> wgrant: If it will do, can I get a review, then?
[03:55] <wgrant> StevenK: r=me with a comment
[03:56] <StevenK> wgrant: Your comment assumes I understand the reasons :-P
[03:57] <wgrant> StevenK: We only expose deletion of sources in the UI
[03:57] <wgrant> But you may need to delete a superseded source's binaries
[03:57] <wgrant> So we want to show a source if can itself be deleted, or if it has binaries that can be.
[03:58] <wgrant> Due to the GPL and sanity, we don't condemn sources until their binaries are gone.
[03:58] <StevenK> Right
[03:58] <StevenK> Yeah, I just recalled GPL compliance
[03:59] <StevenK> +        # We will return sources that can be deleted, or deleted sources that
[03:59] <StevenK> +        # still have published binaries. We can use scheduleddeletiondate
[04:00] <StevenK> +        # rather than linking through BPB, BPR and BPPH since we don't condemn
[04:00] <StevenK> +        # sources until their binaries are all gone due to GPL compliance.
[04:00] <StevenK> wgrant: ^
[04:00] <wgrant> Sounds reasonable.
[04:04] <StevenK> wgrant: How goes your TranslationMessage denorm?
[04:12] <wgrant> StevenK: Given that it's a fairly large table, I'm looking for other things I can do at the same time to avoid rewriting it more than once
[04:13] <StevenK> wgrant: It requires a full table rewrite?
[04:20] <wgrant> StevenK: Denorming a column does, yes.
[04:20] <wgrant> We need to populate it.
[04:24] <wgrant> Resolving the message sharing performance debacle is probably a much more difficult matter, but we'll see
[04:24] <wgrant> I'm reasonably convinced that the feature was primarily designed to make me cry years later.
[04:25] <StevenK> Haha
[04:25] <StevenK> Given TM is a 6GiB table, we should only rewrite it once, yes.
[04:34] <wgrant> A statement count of 328 doesn't seem too bad until you realise that the page only shows 10 items and times out before it's done half the queries.
[04:36] <StevenK> Hah
[04:36] <StevenK> wgrant: I wonder if we want tmpreaper for ackee and loganberry to combat bug 881255 ?
[04:36] <_mup_> Bug #881255: bzr-limbo-?????? and gpg-?????? in /tmp is not being cleaned up and taking up a lot of space <canonical-losa-lp> <Launchpad itself:Triaged> < https://launchpad.net/bugs/881255 >
[04:37] <wgrant> StevenK: Combat? Conceal.
[04:38] <StevenK> Conceal works too
[05:15] <StevenK> wgrant: Hmmmm
[05:15] <wgrant> The failure?
[05:15] <StevenK> wgrant: Deleting publications does not set scheduleddeletiondate, so the sources turn up in the view again
[05:16] <wgrant> StevenK: Bah, yes.
[05:16] <wgrant> Damn
[05:16] <wgrant> Although that's probably not necessarily a huge problem
[05:16] <wgrant> Except for tests.
[05:16] <StevenK> It will get set by the publisher?
[05:16] <wgrant> It'll show the status correctly.
[05:16] <wgrant> Yes
[05:17] <wgrant> It's done during domination.
[05:18] <StevenK> It will show the status correctly, but there is probably nothing stopping users from selecting a DELETED publication and trying to do so again and then getting a OOPS
[05:18] <wgrant> It shouldn't oops
[05:19] <StevenK> Hm, indeed
[05:19] <wgrant> Recall that we show them when they have published binaries exactly so that an already deleted source can be deleted again
[05:20] <StevenK> I thought I saw an assert for !DELETED
[05:20] <StevenK> But I can't see one
[05:20] <StevenK> wgrant: I think the easiest thing for the tests is to set scheduleddeletiondate
[05:21] <wgrant> Depends on what the test wants to achieve
[05:21] <wgrant> But probably, yes.
[05:21] <StevenK> archive-views.txt is checking a double POST
[05:22] <StevenK> We can probably just destroy the view.has_sources_for_display check