[00:26] And I lose buildbot bingo rather spectacularly. [00:26] rick_h_: No fair if you force it. [00:40] StevenK: I didn't touch it :P === slank is now known as slank_away [01:46] wgrant: Haha, it takes DF 13 minutes to work out the source size of the primary archive. [01:48] I daresay binary size is far worse. [01:48] Quite [01:49] StevenK: Oh, why didn't you just trivially fix that rev rather than reverting it? [01:49] I saw multiple issues in the failures [01:49] Wasn't it all just the webservice.py import removals? [01:50] I wasn't sure [01:50] Quicker to revert and reland [01:50] (After checking, if I care) [01:51] wgrant: So we can't do the repository size in the request, and we can't do it in a job, and it's too slow for the publisher :-( [01:52] Indeed [01:52] We could possibly denorm published size onto spph and bpph [01:52] And then just traverse that [01:52] That would change the definition substantially (and probably fatally), and not be hugely fast anyway [01:53] We're running out of places we can do it :-( [01:53] It will probably require a redefinition of what an archive's size means, or a (archive, filename)ish denormalised table [01:53] Neither of which is easy [01:53] Pick another bug :) [01:54] Eh, it answers the question you asked last time I bought it up [01:54] What was the question? [01:54] I'm pretty sure I said it was probably too slow to ever calculate fully in one hit very much [01:55] "How long does it take to calculate the size for the primary archive, if we do it in the publisher?" [01:55] Or so [01:55] Well [01:55] If I did pose that question, it was mostly to demonstrate that it was dreadfully slow [01:55] Which makes the answer 'an eon or so' [01:55] And so your approach was invalid [01:56] wgrant: Are you working on the +sharing critical? [01:56] No [01:56] But it's trivial, so feel free [01:57] While you're there you should also check that the unused AP pruner will DTRT in terms of Product.information_type [01:58] * StevenK watches kanban choke his browser [02:38] StevenK: That doesn't look very constant... [02:46] its slow cause its cold cache IO [02:46] Nope [02:46] Unless cold cache magically causes 40 repetitions [02:46] repo size stuff [02:47] back up an hour [02:47] Oh, yeah, obviously [02:47] There's a lot of rows [02:47] Most of which won't be poked much [02:47] O(files) will always be slow for that [02:47] So the cache will always be slow [02:47] s/cache/query/ [02:47] you need sublinear scaling for that one. [02:47] Yes [02:47] Of course [02:47] That's what i said :) [02:47] It needs a complete redesign [02:47] hence defer [02:47] e.g. event based recalc and a cached value. [02:47] Precisely [02:48] But it's too slow to even do event-based full recalc in the current model [02:48] So it needs to be reworked even further [02:48] wgrant: !cite [02:48] [its not clear to me that thats the case] [02:50] Let's say 20k binaries per DAS [02:50] Average of 5 or 6 DASes per series [02:50] Maybe 6 published series at a time [02:50] + 20k sources per series [02:51] So that's >600k pubs for the primary archive [02:51] Which isn't huge [02:51] And an average of 3 files per publication [02:51] But we don't need that hot [02:51] So we should keep it cold if we can [02:51] StevenK: Not quite, due to binaries it'll be more like 1.5-2 I suspect [02:52] wgrant: Then you have things like linux and firefox that build 10,000 binaries [02:52] There are better things to keep in wildcherry's cache than millions of Ubuntu rows [02:52] When we can relatively easily cache [02:53] Similar to branchrevision [02:53] how often does the primary archive do deletes (vs updates) ? [02:53] Having to keep that hot for anything to work is completely completely stupid [02:53] lifeless: > hourly [02:53] Since stuff gets superseded [02:53] sure [02:53] p-d-r is seperate from the publisher [02:54] so what I'd be inclined to do is to use something like twitters graph thing to keep this hot [02:54] and have a size calculating spout [02:54] So [02:54] hot as in live in-memory on a small number of machines [02:54] Things we can deploy in the next 5 years [02:54] heh :) [02:54] - Not a Twitter graph thing [02:54] We need a solution to this in the next 12 months [02:54] So I think the current query works fine for small PPAs. But it scales horribly [02:55] So deploying anything exotic is entirely out of the question [02:55] And wildcherry has enough cache issues as it is [02:57] you don't need to query wildcherry [03:04] wgrant: So we create a proprietary project, set it to public and make certain a PUBLIC AccessPolicy wasn't created? [03:06] StevenK: Yes [03:08] StevenK: So, more work required on +queue :( [03:08] :-( [03:08] It still makes too many queries? [03:09] DONE precise on qas is 129 [03:09] Which is like 1/3rd [03:09] 24 PackageBuild queries, plus lots of other stuff [03:10] https://oops.canonical.com/oops.py/?oopsid=OOPS-566218b1efc27aca3c77c68091a09d83 [03:11] Yeah, I just read the query log on qas [03:11] The lowest hanging fruit is certainly PackageBuild [03:14] wgrant: Hm, now I remember. I couldn't get my test cases to make the PB queries so I moved on. [03:18] You clearly need more PUBs [03:19] 15 isn't enough? [03:21] So [03:21] Work out where the PB queries come from [03:21] Then work out why they aren't appearing in your test [03:23] wgrant: However, we can slam the two bugs back to In Progress and qa-ok [03:23] Yes [03:31] wgrant: Based on my reading of the code, the 16 PUBLIC APs won't be deleted, there are probably APGs holding them in [03:34] StevenK: APGs don't hold them [03:34] Only APAs and sharing policies do [03:35] But the sharing policies will probably keep them [03:36] We can fix those up without a code change, though? [03:36] Well, we'll delete them manually [03:37] _pruneUnusedPolicies keeps anything that's permitted by a sharing policy. [03:37] Including any public types that shouldn't have had an AP in the first place. [03:39] wgrant: I'm confused, do we need a code change to clean up? Or when you say delete you mean "DELETE FROM" ? [03:40] We could switch _pruneUnusedPolicies to always prune policies that are public, but that's pointless because they shouldn't exist in the first place [03:40] It would be best to SQL them away manually [03:50] 'if not self.obj.archive.private' causes PB lookups? :-( [03:50] Sure [03:50] Archive is on PB [03:51] wgrant: Sure, but from ViewSourcePackageRelease ? [03:51] Um [03:51] SPR.archive doesn't exist [03:51] So no, you're wrong :) [03:52] Sounds more like ViewBinaryPackageBuild [03:52] Indeed [03:52] Right [03:54] Perhaps self.factory.makeBuildPackageUpload is missing some bits, then [04:35] wgrant: https://code.launchpad.net/~stevenk/launchpad/no-public-for-all-projects/+merge/141700 [04:43] StevenK: _prepare_to_set_sharing_policy already knows how to restrict it to just private types [04:43] We shouldn't duplicate that logic. [04:46] Let me move into _ensurePolicies then [04:48] wgrant: http://pastebin.ubuntu.com/1490688/ [04:49] Sounds reasonable [04:50] wgrant: Actually, http://pastebin.ubuntu.com/1490691/ [04:53] Indeed [05:09] Oh wonderful, I think I've broken diff generation for that MP [05:17] Ah, look at that. [05:17] wgrant: Can haz +1 now? [05:31] MismatchError: queries do not match: 52 != 87 [05:31] Success [05:33] StevenK: Done [05:33] What did you eliminate? [05:34] I eliminated nothing. I reached into the BPB's BPR and called addFile [05:35] Ah [05:35] wgrant: And sorry for my OCD forcing +39/-41 on you :-) [05:37] Right, preloading PB gets us down to 73 [05:46] wgrant: http://pastebin.ubuntu.com/1490769/ gets us to 30 queries on the API, and 57 on the web === almaisan-away is now known as al-maisan [05:48] We don't really need to preload PBs, Components and Sections in getAll(), but it was trival. [05:49] getAll should basically only be used by getPackageUploads and +queue, right? [05:49] So it's probably not too bad to aggressively preload [05:49] Pretty much, yes [05:50] I'd like to preload away the PackageDiff stuff, but that doesn't look easy [05:50] I thought you already looked at doing that [05:50] Oh right, you gave up after I told you your trivial implementation wouldn't work? [05:50] I did, yes [05:51] wgrant: If you're happy enough with that diff I can put together an MP? [05:52] IIRC it just iterates over SPR.package_diffs [05:52] Which is going to be relatively easily to cachedpropertyify [05:52] * StevenK peers [05:52] It's been a while since I was over there [05:53] But I don't see why it would do otherwise [05:53] Yeah [05:53] wgrant@lplucid:~/launchpad/lp-branches/devel$ bzr grep package_diffs lib/lp/soyuz/templates/ [05:53] lib/lp/soyuz/templates/distroseries-queue.pt: lib/lp/soyuz/templates/sourcepackagerelease-diffs.pt: tal:define="diffs context/package_diffs" [05:53] cachedproperty [05:53] done [05:53] (well, plus a couple of lines here and there to prepopulate and invalidate) [05:54] It's a SQLMutlipleJoin, so I can cheat by _package_diffs, or rewrite to something actually using IStore [05:54] SQLMultipleJoin, even [05:55] It's unlikely that much actually cares that it's an SQLMultipleJoin [05:55] It can probably be turned into a simple cachedproperty list, but you'll have to look at the other callsites [05:57] Seem to be only 5 non-test callsites [05:57] ez [05:58] wgrant: Eh, easy enough to copy the pattern that PackageUpload uses with sources vs _sources [05:59] As I said in the last MP, that's pretty pointless [05:59] If nothing needs a resultset then a resultset need not be exposed at all [05:59] Never expose unnecessary interfaces [05:59] Or you end up with something like Launchpad :) [06:00] (it appears that none of the callsites will notice if you make it a lisT) [06:00] + @cachedproperty [06:00] + def package_diffs(self): [06:00] + return list(self._package_diffs) [06:01] Bad StevenK [06:01] Ah ha, so you *do* want the destruction of the SQLMultipleJoin [06:01] It's pointless to keep it [06:02] return list(Store.of(self).find(PackageDiff, to_source=self).order_by(Desc(PackageDiff.date_requested))) [06:02] done [06:04] Right, which I was in the middle of typing when you said that ... [06:17] wgrant: Not sure where to invalidate SPR.package_diffs, but that preloading gets us to 53 queries. [06:17] Except that the LFA and LFC preloading helps not at all [06:20] DId you preload the correct LFAs and LFCs? [06:20] wgrant@lplucid:~/launchpad/lp-branches/devel$ bzr grep '\bPackageDiff\(' [06:20] lib/lp/soyuz/model/packagediff.py:class PackageDiff(SQLBase): [06:20] lib/lp/soyuz/model/sourcepackagerelease.py: return PackageDiff( [06:20] lib/lp/testing/factory.py: PackageDiff( [06:22] Easy [06:39] Huh [06:40] I just removed all the writable bits of the BuildFarmJob and PackageBuild security configuration [06:40] And the only tests that appear to fail are those that test the permissions directly [06:53] \o/ [08:50] aloha === al-maisan is now known as almaisan-away [09:11] ood morning [09:12] adeuring: hopefuly not an odd morning either :) [09:13] czajkowski: yeah: i need to clean my keayboard from tobacco crumbs... [09:15] brb === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === yofel_ is now known as yofel === slank_away is now known as slank === Ursinha-afk is now known as Ursinha [16:05] rick_h_: http://i.imgur.com/OWtoF.jpg [16:05] czajkowski: I'm sending help, hold on tight! === deryck is now known as deryck[lunch] === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === BradCrittenden is now known as bac === deryck[lunch] is now known as deryck === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [21:00] buildbot is seriously harshing my mellow [22:32] sinzui: I'm going to go with that's not a good mellow