[00:06] Somebody should blow sampledata up [01:18] wgrant: https://code.launchpad.net/~stevenk/launchpad/performant-getSourcesForDeletion/+merge/153704 [01:21] StevenK: Have you checked that my theory is correct? [01:21] It sounded plausible to me [01:22] A query to check that scheduleddeletiondate is set iff (published or has published (or unremoved? can't remember) binaries) would give me more confidence [01:22] I guess if you check scheduleddeletiondate on the binaries it might work [01:25] Though it's not going to corrupt data if my assumption is incorrect, we should still try to verify it against existing data. [01:36] wgrant: Query is http://pastebin.ubuntu.com/5624173/, results http://pastebin.ubuntu.com/5624172/ [01:46] StevenK: https://pastebin.canonical.com/86983/ [01:48] wgrant: http://pastebin.ubuntu.com/5624196/ [01:48] I know [01:48] Right [01:48] So it looks like it might be right, but maybe not [01:48] We should probably investigate those cases to see what might be up [01:48] They may indicate publisher bugs [01:48] Which cases? [01:49] Those 20 [01:49] It's also possible that a publisher run will fix those [01:49] Because this invariant is only eventually consistent [01:49] Well, let's see on prod [01:49] Since qas is a bit odd [01:49] Oh [01:50] You also check that the binary is actually published [01:50] Rather than condemned [01:50] That was what the old method did [01:50] And I only changed the archive in one part of the query [01:50] Haha [01:50] Oh [01:51] Right, so the results are fairly useless [01:52] That also explains why it ran so quickly [01:52] If you're running it on sour, I'll kill mine [01:52] I am [01:53] Mine has been killed [02:06] wgrant: That query with 32 for both archives is 0 rows [03:14] wgrant: What did that query return for PRIMARY? [03:15] StevenK: 20000 rows [03:16] Bloody hell [03:16] I don't think Archive:+delete-packages works for PRIMARY anyway? [03:16] Sure [03:16] But the primary archive and PPAs use the same domination code. [03:17] Any problem in the primary archive can apply to another archive. [03:17] So it is a useful pathological test case. [03:17] Right, so do you want to split up the work, or what is your plan? [03:20] Hmm, overrides, maybe [03:20] * wgrant gives up [03:20] Overrides will indeed fuck that up [03:20] Thinking about it [03:20] No overrides for PPAs [03:32] wgrant: So I think it's plausible that overrides won't be doing that query any favours. [03:34] Running it over the first 10k PPAs gives about 500 results [03:35] Possibly explainable by copies [03:35] I guess it'll do [03:36] wgrant: Wait, how do copies explain it? [03:36] StevenK: Undeletions [03:36] If I delete something, let it be removed, then revive it [03:37] Oh, the necromancy style of copies [03:37] The old pubs may have been removed from disk, despite there now being live binaries [03:51] wgrant: If it will do, can I get a review, then? [03:55] StevenK: r=me with a comment [03:56] wgrant: Your comment assumes I understand the reasons :-P [03:57] StevenK: We only expose deletion of sources in the UI [03:57] But you may need to delete a superseded source's binaries [03:57] So we want to show a source if can itself be deleted, or if it has binaries that can be. [03:58] Due to the GPL and sanity, we don't condemn sources until their binaries are gone. [03:58] Right [03:58] Yeah, I just recalled GPL compliance [03:59] + # We will return sources that can be deleted, or deleted sources that [03:59] + # still have published binaries. We can use scheduleddeletiondate [04:00] + # rather than linking through BPB, BPR and BPPH since we don't condemn [04:00] + # sources until their binaries are all gone due to GPL compliance. [04:00] wgrant: ^ [04:00] Sounds reasonable. [04:04] wgrant: How goes your TranslationMessage denorm? [04:12] StevenK: Given that it's a fairly large table, I'm looking for other things I can do at the same time to avoid rewriting it more than once [04:13] wgrant: It requires a full table rewrite? [04:20] StevenK: Denorming a column does, yes. [04:20] We need to populate it. [04:24] Resolving the message sharing performance debacle is probably a much more difficult matter, but we'll see [04:24] I'm reasonably convinced that the feature was primarily designed to make me cry years later. [04:25] Haha [04:25] Given TM is a 6GiB table, we should only rewrite it once, yes. [04:34] A statement count of 328 doesn't seem too bad until you realise that the page only shows 10 items and times out before it's done half the queries. [04:36] Hah [04:36] wgrant: I wonder if we want tmpreaper for ackee and loganberry to combat bug 881255 ? [04:36] <_mup_> Bug #881255: bzr-limbo-?????? and gpg-?????? in /tmp is not being cleaned up and taking up a lot of space < https://launchpad.net/bugs/881255 > [04:37] StevenK: Combat? Conceal. [04:38] Conceal works too [05:15] wgrant: Hmmmm [05:15] The failure? [05:15] wgrant: Deleting publications does not set scheduleddeletiondate, so the sources turn up in the view again [05:16] StevenK: Bah, yes. [05:16] Damn [05:16] Although that's probably not necessarily a huge problem [05:16] Except for tests. [05:16] It will get set by the publisher? [05:16] It'll show the status correctly. [05:16] Yes [05:17] It's done during domination. [05:18] It will show the status correctly, but there is probably nothing stopping users from selecting a DELETED publication and trying to do so again and then getting a OOPS [05:18] It shouldn't oops [05:19] Hm, indeed [05:19] Recall that we show them when they have published binaries exactly so that an already deleted source can be deleted again [05:20] I thought I saw an assert for !DELETED [05:20] But I can't see one [05:20] wgrant: I think the easiest thing for the tests is to set scheduleddeletiondate [05:21] Depends on what the test wants to achieve [05:21] But probably, yes. [05:21] archive-views.txt is checking a double POST [05:22] We can probably just destroy the view.has_sources_for_display check === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === wedgwood_away is now known as wedgwood === matsubara is now known as matsubara-lunch === deryck is now known as deryck[lunch] === Ursinha_ is now known as Ursinha === matsubara-lunch is now known as matsubara === deryck[lunch] is now known as deryck === Ursinha is now known as Ursinha-afk === BradCrittenden is now known as bac === Ursinha-afk is now known as Ursinha === Ursinha_ is now known as Ursinha === wedgwood is now known as wedgwood_away