[01:08] cjwatson: How's the performance looking? [03:03] wgrant: About a minute for each run [03:03] cjwatson: Nice. [03:04] Down from three and a bit [03:05] Looks like reasonably consistently 53s or nearby [03:32] wgrant: Is there any way to sensibly preload things that are modelled using SQLMultipleJoin(..., orderBy=)? Storm doesn't seem to cache that in a way that can be satisfied by a previous load_referencing, not even if I expand out load_referencing's query to include a matching ORDER BY. [03:34] cjwatson: No, Storm only caches by PK [03:35] You'll need to turn the SQLMultipleJoin into a cachedproperty [03:35] Aha [03:35] Depending on the callsites, that can be very easy or very hard. [03:35] Which one is this? [03:35] SPR.files [03:35] Ah, most things should be happy with that being a list. [03:35] What are you trying to accelerate? [03:35] I got pissed off with PPA publisher performance and thought I'd try the obvious preloads [03:36] Ah, yeah [03:37] OK, this is really confusing. I'm watching a UOS session where you're talking, while talking to you about something totally different on IRC. [03:37] Everything else from source publication is easy to make constant query count by number of source packages. Haven't tried binaries yet [03:37] Heh [03:38] It's not a huge amount of work now that model cachedproperties aren't verboten. [03:38] eg. SPR.files probably only needs invalidation in SPR.addFile [03:38] But gina needs checking. [03:39] Right, I assume I just grep for SourcePackageReleaseFile as the starting point [03:40] Yep. [03:59] Yay, that makes my record_two_runs test for source publishing pass. Thanks. [03:59] I'll try the same for binaries in the morning. === timrc is now known as timrc-afk [04:35] Or maybe I'll have insomnia and JFDI. Should actually try sleeping while more of the tests run though ... [05:54] cjwatson: Did you go to bed? :) === alexlist` is now known as alexlist [08:08] wgrant: For a bit, yeah [08:09] Disappointing. [08:10] Well, I'd written the binaries code before saying that above [08:10] Should have that up for you to look at a bit later today [08:11] Any luck with livefs*? [08:16] Spent most of the day on log analysis and scalingstack stuff, but I'll hopefully have that sorted out tomorrow in the unlikely event that nothing else comes up :/ [09:29] cjwatson: How many queries were actually executed per pub? [09:30] I guess one for the xPR, one for the xPR?Fs, then one for each LFA and one for each LFC. [09:30] Plus the xPN [09:30] And sometimes the component and section. [09:31] And maybe one for the BPB, its SPPH, its SPR, its SPN and its component in the case of binaries? [09:31] Though that might be done by postgres the view, I can't quite recall. [09:31] I think it was about five or six per pub [09:31] in the view [09:32] Ugh, some of that lint is mine. [09:32] Seven per source and six per binary if I'm reading my terminal scrollback correctly [09:33] SPR, SPN, SPRF, number of files * (LFA, LFC) [09:33] BPR, BPB, BPF, LFA, LFC, (maybe SPN), BPN [09:34] Also sometimes Section. Component is effectively constant [09:34] I was wondering how SPRs had more. Of course, multiple files. [09:34] Two in my test but of course often more. [09:34] Hm [09:34] We need to check the Storm cache config. [09:35] I think it's capped at 10000 objects [09:35] It is for the [09:35] webapp, at least, I think [09:35] I might have uncapped it for scripts a few years ago/ [09:35] How many packages are there in the largest PPA? [09:35] A few thousand [09:35] 3000ish IIRC [09:36] We probably want to do this in reasonably sized chunks, but your diff is better than what we have now, so meh. [09:37] I guess if we were to chunk it the preloading would have to move into archivepublisher [09:37] As it bloody well should. [09:37] None of that ridiculous publishing code belongs in registry :) [09:37] Or even soyuz. [09:37] Oh yes let's have the webapp publication objects know how to serialise themselves to disk, that's a good idea. [09:37] Well, I mean into the caller rather than the callee [09:38] Those get*PackagePublishing methods only have one call site each [09:38] Yeah [09:38] (going out for a bit before UOS) [09:41] I've been meaning to investigate chunking DecoratedResultSet.__iter__ for a while. [09:41] It already does a bit of evil, so it wouldn't be too bad. [09:42] Have it iterate through chunks of a caller-specified size, so it does 10 bulk queries each over 1000 objects rather than one over 100000. [09:44] Ignore the missing order of magnitude there. [09:57] Inline comments are nice. [10:24] wgrant: it is capped, but you can override it in the launchpad.conf per script === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === jelmer_ is now known as jelmer === BradCrittenden is now known as bac [12:05] cjwatson: How'd the source caching test go, modulo ENOSPC? [13:31] wgrant: Looked fine, was about four minutes quicker [13:32] Results looked correct [13:32] I've asked mvo to do a sourceful trusty backport so that we can then build that lot on precise [13:38] :) [13:43] 15:53 dists.in-progress/utopic/main/source/: 6392 pkgs in 35s [13:43] 15:53 dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s [13:43] 15:53 dists.in-progress/utopic/universe/source/: 39628 pkgs in 3min 37s [13:43] 15:53 dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 8s [13:43] 15:53 then second run: [13:43] 15:53 dists.in-progress/utopic/main/source/: 6392 pkgs in 0s [13:43] 15:53 dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s [13:43] 15:53 dists.in-progress/utopic/universe/source/: 39628 pkgs in 6s [13:44] (both with the new apt-ftparchive, but the first with no sources cache) [13:44] 15:53 dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 0s [13:44] So does exactly what it says on the tin, I think. [13:46] I wonder where I should put ubuntu-archive-publishing config, re https://code.launchpad.net/~cjwatson/launchpad/configurable-germinate-base/+merge/194340 . Maybe I should just have a file where we look things up by os.environ["LPCONFIG"] [13:49] cjwatson: That or a config file named by LPCONFIG sounds reasonable. === olli_ is now known as olli === mbarnett` is now known as mbarnett [15:43] wgrant: https://code.launchpad.net/~cjwatson/launchpad/preload-publisher-indexes/+merge/222768 should be ready for re-review now. [15:50] cjwatson: Thanks. [15:51] cjwatson: We'll want to test variously sized archives on DF to check for any weird postgres decisions on the huge queries, but that's nice and easy with NMAF (no pool required). [15:51] no pool required? [15:52] Testing a-f at scale can only be done on pepo, as it depends on the pool being intact. [15:52] oh, right, you mean we can let NMAF put the pool in place [15:52] No, we don't need a pool. [15:52] I'm obviously not awake. I understand now. [15:52] Well considering you were off IRC for two hours. [15:53] Andreas had a didn't-want-to-sleep night. [15:53] :( [15:53] Also, <3 caffeine [15:53] Heh [15:54] OK, that a-f smoke-test on DF would have been more effective if I'd changed something. [15:54] Clean suites are quick to publish, at least. [15:54] Oh, it did change. Good. [15:54] Just looking at the wrong publish-distro run. [15:54] Ah [15:55] And see your previous comments about needing a pool. Well, didn't explode. [15:55] It won't explode. [15:56] But the indices'll be pretty empty. [15:56] Yeah, just memtest86+ in main. Shrug. [15:57] === modified file 'finalize.d/20-germinate' (properties changed: +x to -x) [15:57] thanks, bzr [15:58] Anyway, we can deploy once lpqateam catches up. Shall I request that? [15:58] Sounds like a good plan. [15:59] Have you done one lately? [15:59] webops will ask for an RT [15:59] But otherwise as usual. [15:59] Not lately, but I'd seen you switching to RTing things. [16:00] Eh, what the heck did generate-extra-overrides just do on labbu [16:01] Oh, maybe it's talking to production for its series information [16:01] Ah, heh, probably. [16:02] Wonder what to do about that. I could just cowboy -l dogfood in cron.germinate on labbu for now ... [16:08] wgrant: bug 1328355 was applied live? [16:08] <_mup_> Bug #1328355: Distribution.guessPublishedSourcePackageName times out frequently

[16:08] cjwatson: It was. [16:12] Grr, api.dogfood.launchpad.net [16:12] I guess we should SRU python-launchpadlib [16:13] I need to move the stagings at some point as well. [16:13] Luckily -l https://api.dogfood.launchpad.net/ works [16:13] er. [16:13] -l https://api.dogfood.paddev.net/ [16:14] Yeah, I was just thinking and checking that [16:19] OK, cowboyed to death now. Getting rid of lp-query-distro will kill one of the cowboys. [16:20] :) [16:20] I'll probably shove the endpoint URI into the config infrastructure I'm writing for bug 1248869. [16:20] <_mup_> Bug #1248869: Stop the publisher from talking to people.canonical.com

[16:20] Yup, makes sense. [16:20] You have the ndt etc. under control? [16:20] I might wander off. [16:20] Yeah, go ahead. [16:20] Thanks. === Ursinha is now known as Ursinha-afk [18:42] wgrant: I self-reviewed a testfix to preload-publisher-indexes to unblock buildbot. Hopefully it looks right to you. === Ursinha-afk is now known as Ursinha === jml- is now known as jml [21:18] stub: is there a way in pytz to get a tzinfo object given a utc offset? [23:10] cjwatson: Looks good, thanks. [23:12] I'm trying some test runs with it now. [23:12] "/srv/launchpad.net/codelines/current/scripts/publish-distro.py -vv -d ubuntu --ppa --ms -A", hacked to publish only a single PPA, taking time between "Step C'" and "Step D", second run to ensure hot cache [23:12] cjwatson/ppa, canonical-kernel-team/ppa, launchpad/ppa, kubuntu-ninjas/ppa [23:13] Any other suggestions? [23:13] Those are pretty good options. [23:13] -A publishes all series, but the more the merrier. [23:14] (Actually I'm currently getting control data from r17041.) [23:33] cjwatson/ppa (2.835) 1.260 (1.412) 1.375 [23:33] canonical-kernel-team/ppa (33.896) 23.570 (9.972) 10.152 [23:33] launchpad/ppa (9.644) 3.739 (3.332) 3.367 [23:33] kubuntu-ninjas/ppa (69.837) 43.198 (19.578) 20.031 [23:33] wgrant: ^- owner-name/ppa-name (r17041 first run) r17041 second run (r17043 first run) r17043 second run [23:34] archives compare functionally identical (aside from compression timestamps and suchlike) [23:34] looks pretty good to me [23:38] cjwatson: Still very slow, but quite an improvement. Is that just index generation time? [23:38] I suspect the remaining time would bear some profiling, since it certainly shouldn't be maxing out on I/O there, but still [23:38] ie. C' -> D time? [23:38] Right, just step C' [23:39] Maybe lacking indexes for some of the rest? [23:39] All the queries you touched yesterday have sensible indices, but yeah, it's probably worth investigating now [23:39] I'll have a poke when you're done with labbu. [23:40] I think I'm done now; that much is enough for me to say qa-ok I think [23:40] Agreed. [23:42] * wgrant investigates. [23:43] Released my vim lock on lib/lp/soyuz/scripts/publishdistro.py which remains cowboyed [23:43] Feel free to mangle/revert [23:43] Thanks. [23:44] 335 queries [23:45] Hm [23:45] I might hack it to only do one suite. [23:46] Unless you have logs from the original run? [23:47] kubuntu-ninjas trusty binary-amd64 took 2.2s in my latest test run. [23:47] cjwatson: Did you keep your logs? [23:49] 6.1s with the old code. [23:50] er, not in a file but I think I have scrollback [23:50] Bah, not enough of it, sorry. [23:50] Nevermind, quick enough to rerun :) [23:51] I guess I should disable compression and see how much that shaves off [23:51] It was the limiting factor in my heavy caching experiments years ago. [23:54] bzip2 is only 5-10% of the time [23:56] The {Plain,Gzip,Bzip2}TempFile scheme could be fixed to not do tempfile.mkstemp when it's just going to discard the result (in the gzip and bzip2 cases) [23:56] I expect that's epsilon though [23:56] Yeah [23:56] Oh, just the bzip2 case, not gzip [23:57] Will be interesting to see how much this improves things on prod [23:57] Still, stupid code layout [23:57] We have network latency and auth etc. there. [23:57] Yeah [23:57] So the many->notmany queries will be a much bigger win. [23:57] Plus wildcherry is a potato. [23:59] kubuntu-ninjas trusty binary-amd64 took 23s on prod on 2014-05-13 [23:59] dunno how close to the same set of packages that is [23:59] Roughly 46000 additional queries in step C' across all suites