| wgrant | cjwatson: How's the performance looking? | 01:08 |
|---|---|---|
| cjwatson | wgrant: About a minute for each run | 03:03 |
| wgrant | cjwatson: Nice. | 03:03 |
| cjwatson | Down from three and a bit | 03:04 |
| cjwatson | Looks like reasonably consistently 53s or nearby | 03:05 |
| cjwatson | wgrant: Is there any way to sensibly preload things that are modelled using SQLMultipleJoin(..., orderBy=)? Storm doesn't seem to cache that in a way that can be satisfied by a previous load_referencing, not even if I expand out load_referencing's query to include a matching ORDER BY. | 03:32 |
| wgrant | cjwatson: No, Storm only caches by PK | 03:34 |
| wgrant | You'll need to turn the SQLMultipleJoin into a cachedproperty | 03:35 |
| cjwatson | Aha | 03:35 |
| wgrant | Depending on the callsites, that can be very easy or very hard. | 03:35 |
| wgrant | Which one is this? | 03:35 |
| cjwatson | SPR.files | 03:35 |
| wgrant | Ah, most things should be happy with that being a list. | 03:35 |
| wgrant | What are you trying to accelerate? | 03:35 |
| cjwatson | I got pissed off with PPA publisher performance and thought I'd try the obvious preloads | 03:35 |
| wgrant | Ah, yeah | 03:36 |
| wgrant | OK, this is really confusing. I'm watching a UOS session where you're talking, while talking to you about something totally different on IRC. | 03:37 |
| cjwatson | Everything else from source publication is easy to make constant query count by number of source packages. Haven't tried binaries yet | 03:37 |
| cjwatson | Heh | 03:37 |
| wgrant | It's not a huge amount of work now that model cachedproperties aren't verboten. | 03:38 |
| wgrant | eg. SPR.files probably only needs invalidation in SPR.addFile | 03:38 |
| wgrant | But gina needs checking. | 03:38 |
| cjwatson | Right, I assume I just grep for SourcePackageReleaseFile as the starting point | 03:39 |
| wgrant | Yep. | 03:40 |
| cjwatson | Yay, that makes my record_two_runs test for source publishing pass. Thanks. | 03:59 |
| cjwatson | I'll try the same for binaries in the morning. | 03:59 |
| === timrc is now known as timrc-afk | ||
| cjwatson | Or maybe I'll have insomnia and JFDI. Should actually try sleeping while more of the tests run though ... | 04:35 |
| wgrant | cjwatson: Did you go to bed? :) | 05:54 |
| === alexlist` is now known as alexlist | ||
| cjwatson | wgrant: For a bit, yeah | 08:08 |
| wgrant | Disappointing. | 08:09 |
| cjwatson | Well, I'd written the binaries code before saying that above | 08:10 |
| cjwatson | Should have that up for you to look at a bit later today | 08:10 |
| cjwatson | Any luck with livefs*? | 08:11 |
| wgrant | Spent most of the day on log analysis and scalingstack stuff, but I'll hopefully have that sorted out tomorrow in the unlikely event that nothing else comes up :/ | 08:16 |
| wgrant | cjwatson: How many queries were actually executed per pub? | 09:29 |
| wgrant | I guess one for the xPR, one for the xPR?Fs, then one for each LFA and one for each LFC. | 09:30 |
| wgrant | Plus the xPN | 09:30 |
| wgrant | And sometimes the component and section. | 09:30 |
| wgrant | And maybe one for the BPB, its SPPH, its SPR, its SPN and its component in the case of binaries? | 09:31 |
| wgrant | Though that might be done by postgres the view, I can't quite recall. | 09:31 |
| cjwatson | I think it was about five or six per pub | 09:31 |
| wgrant | in the view | 09:31 |
| wgrant | Ugh, some of that lint is mine. | 09:32 |
| cjwatson | Seven per source and six per binary if I'm reading my terminal scrollback correctly | 09:32 |
| cjwatson | SPR, SPN, SPRF, number of files * (LFA, LFC) | 09:33 |
| cjwatson | BPR, BPB, BPF, LFA, LFC, (maybe SPN), BPN | 09:33 |
| cjwatson | Also sometimes Section. Component is effectively constant | 09:34 |
| wgrant | I was wondering how SPRs had more. Of course, multiple files. | 09:34 |
| cjwatson | Two in my test but of course often more. | 09:34 |
| wgrant | Hm | 09:34 |
| wgrant | We need to check the Storm cache config. | 09:34 |
| wgrant | I think it's capped at 10000 objects | 09:35 |
| wgrant | It is for the | 09:35 |
| wgrant | webapp, at least, I think | 09:35 |
| wgrant | I might have uncapped it for scripts a few years ago/ | 09:35 |
| cjwatson | How many packages are there in the largest PPA? | 09:35 |
| wgrant | A few thousand | 09:35 |
| wgrant | 3000ish IIRC | 09:35 |
| wgrant | We probably want to do this in reasonably sized chunks, but your diff is better than what we have now, so meh. | 09:36 |
| cjwatson | I guess if we were to chunk it the preloading would have to move into archivepublisher | 09:37 |
| wgrant | As it bloody well should. | 09:37 |
| wgrant | None of that ridiculous publishing code belongs in registry :) | 09:37 |
| wgrant | Or even soyuz. | 09:37 |
| wgrant | Oh yes let's have the webapp publication objects know how to serialise themselves to disk, that's a good idea. | 09:37 |
| cjwatson | Well, I mean into the caller rather than the callee | 09:37 |
| cjwatson | Those get*PackagePublishing methods only have one call site each | 09:38 |
| wgrant | Yeah | 09:38 |
| cjwatson | (going out for a bit before UOS) | 09:38 |
| wgrant | I've been meaning to investigate chunking DecoratedResultSet.__iter__ for a while. | 09:41 |
| wgrant | It already does a bit of evil, so it wouldn't be too bad. | 09:41 |
| wgrant | Have it iterate through chunks of a caller-specified size, so it does 10 bulk queries each over 1000 objects rather than one over 100000. | 09:42 |
| wgrant | Ignore the missing order of magnitude there. | 09:44 |
| wgrant | Inline comments are nice. | 09:57 |
| stub | wgrant: it is capped, but you can override it in the launchpad.conf per script | 10:24 |
| === Ursinha is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| === jelmer_ is now known as jelmer | ||
| === BradCrittenden is now known as bac | ||
| wgrant | cjwatson: How'd the source caching test go, modulo ENOSPC? | 12:05 |
| cjwatson | wgrant: Looked fine, was about four minutes quicker | 13:31 |
| cjwatson | Results looked correct | 13:32 |
| cjwatson | I've asked mvo to do a sourceful trusty backport so that we can then build that lot on precise | 13:32 |
| wgrant | :) | 13:38 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/main/source/: 6392 pkgs in 35s | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/universe/source/: 39628 pkgs in 3min 37s | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 8s | 13:43 |
| cjwatson | 15:53 <cjwatson> then second run: | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/main/source/: 6392 pkgs in 0s | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s | 13:43 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/universe/source/: 39628 pkgs in 6s | 13:43 |
| cjwatson | (both with the new apt-ftparchive, but the first with no sources cache) | 13:44 |
| cjwatson | 15:53 <cjwatson> dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 0s | 13:44 |
| cjwatson | So does exactly what it says on the tin, I think. | 13:44 |
| cjwatson | I wonder where I should put ubuntu-archive-publishing config, re https://code.launchpad.net/~cjwatson/launchpad/configurable-germinate-base/+merge/194340 . Maybe I should just have a file where we look things up by os.environ["LPCONFIG"] | 13:46 |
| wgrant | cjwatson: That or a config file named by LPCONFIG sounds reasonable. | 13:49 |
| === olli_ is now known as olli | ||
| === mbarnett` is now known as mbarnett | ||
| cjwatson | wgrant: https://code.launchpad.net/~cjwatson/launchpad/preload-publisher-indexes/+merge/222768 should be ready for re-review now. | 15:43 |
| wgrant | cjwatson: Thanks. | 15:50 |
| wgrant | cjwatson: We'll want to test variously sized archives on DF to check for any weird postgres decisions on the huge queries, but that's nice and easy with NMAF (no pool required). | 15:51 |
| cjwatson | no pool required? | 15:51 |
| wgrant | Testing a-f at scale can only be done on pepo, as it depends on the pool being intact. | 15:52 |
| cjwatson | oh, right, you mean we can let NMAF put the pool in place | 15:52 |
| wgrant | No, we don't need a pool. | 15:52 |
| cjwatson | I'm obviously not awake. I understand now. | 15:52 |
| wgrant | Well considering you were off IRC for two hours. | 15:52 |
| cjwatson | Andreas had a didn't-want-to-sleep night. | 15:53 |
| wgrant | :( | 15:53 |
| cjwatson | Also, <3 caffeine | 15:53 |
| wgrant | Heh | 15:53 |
| cjwatson | OK, that a-f smoke-test on DF would have been more effective if I'd changed something. | 15:54 |
| wgrant | Clean suites are quick to publish, at least. | 15:54 |
| cjwatson | Oh, it did change. Good. | 15:54 |
| cjwatson | Just looking at the wrong publish-distro run. | 15:54 |
| wgrant | Ah | 15:54 |
| cjwatson | And see your previous comments about needing a pool. Well, didn't explode. | 15:55 |
| wgrant | It won't explode. | 15:55 |
| wgrant | But the indices'll be pretty empty. | 15:56 |
| cjwatson | Yeah, just memtest86+ in main. Shrug. | 15:56 |
| cjwatson | === modified file 'finalize.d/20-germinate' (properties changed: +x to -x) | 15:57 |
| cjwatson | thanks, bzr | 15:57 |
| cjwatson | Anyway, we can deploy once lpqateam catches up. Shall I request that? | 15:58 |
| wgrant | Sounds like a good plan. | 15:58 |
| wgrant | Have you done one lately? | 15:59 |
| wgrant | webops will ask for an RT | 15:59 |
| wgrant | But otherwise as usual. | 15:59 |
| cjwatson | Not lately, but I'd seen you switching to RTing things. | 15:59 |
| cjwatson | Eh, what the heck did generate-extra-overrides just do on labbu | 16:00 |
| cjwatson | Oh, maybe it's talking to production for its series information | 16:01 |
| wgrant | Ah, heh, probably. | 16:01 |
| cjwatson | Wonder what to do about that. I could just cowboy -l dogfood in cron.germinate on labbu for now ... | 16:02 |
| cjwatson | wgrant: bug 1328355 was applied live? | 16:08 |
| _mup_ | Bug #1328355: Distribution.guessPublishedSourcePackageName times out frequently <easy> <qa-ok> <timeout> <Launchpad itself:Fix Released by wgrant> <https://launchpad.net/bugs/1328355> | 16:08 |
| wgrant | cjwatson: It was. | 16:08 |
| cjwatson | Grr, api.dogfood.launchpad.net | 16:12 |
| cjwatson | I guess we should SRU python-launchpadlib | 16:12 |
| wgrant | I need to move the stagings at some point as well. | 16:13 |
| wgrant | Luckily -l https://api.dogfood.launchpad.net/ works | 16:13 |
| wgrant | er. | 16:13 |
| wgrant | -l https://api.dogfood.paddev.net/ | 16:13 |
| cjwatson | Yeah, I was just thinking and checking that | 16:14 |
| cjwatson | OK, cowboyed to death now. Getting rid of lp-query-distro will kill one of the cowboys. | 16:19 |
| wgrant | :) | 16:20 |
| cjwatson | I'll probably shove the endpoint URI into the config infrastructure I'm writing for bug 1248869. | 16:20 |
| _mup_ | Bug #1248869: Stop the publisher from talking to people.canonical.com <soyuz-publish> <ubuntu-archive-publishing:In Progress by cjwatson> <https://launchpad.net/bugs/1248869> | 16:20 |
| wgrant | Yup, makes sense. | 16:20 |
| wgrant | You have the ndt etc. under control? | 16:20 |
| wgrant | I might wander off. | 16:20 |
| cjwatson | Yeah, go ahead. | 16:20 |
| wgrant | Thanks. | 16:20 |
| === Ursinha is now known as Ursinha-afk | ||
| cjwatson | wgrant: I self-reviewed a testfix to preload-publisher-indexes to unblock buildbot. Hopefully it looks right to you. | 18:42 |
| === Ursinha-afk is now known as Ursinha | ||
| === jml- is now known as jml | ||
| lifeless | stub: is there a way in pytz to get a tzinfo object given a utc offset? | 21:18 |
| wgrant | cjwatson: Looks good, thanks. | 23:10 |
| cjwatson | I'm trying some test runs with it now. | 23:12 |
| cjwatson | "/srv/launchpad.net/codelines/current/scripts/publish-distro.py -vv -d ubuntu --ppa --ms -A", hacked to publish only a single PPA, taking time between "Step C'" and "Step D", second run to ensure hot cache | 23:12 |
| cjwatson | cjwatson/ppa, canonical-kernel-team/ppa, launchpad/ppa, kubuntu-ninjas/ppa | 23:12 |
| cjwatson | Any other suggestions? | 23:13 |
| wgrant | Those are pretty good options. | 23:13 |
| cjwatson | -A publishes all series, but the more the merrier. | 23:13 |
| cjwatson | (Actually I'm currently getting control data from r17041.) | 23:14 |
| cjwatson | cjwatson/ppa (2.835) 1.260 (1.412) 1.375 | 23:33 |
| cjwatson | canonical-kernel-team/ppa (33.896) 23.570 (9.972) 10.152 | 23:33 |
| cjwatson | launchpad/ppa (9.644) 3.739 (3.332) 3.367 | 23:33 |
| cjwatson | kubuntu-ninjas/ppa (69.837) 43.198 (19.578) 20.031 | 23:33 |
| cjwatson | wgrant: ^- owner-name/ppa-name (r17041 first run) r17041 second run (r17043 first run) r17043 second run | 23:33 |
| cjwatson | archives compare functionally identical (aside from compression timestamps and suchlike) | 23:34 |
| cjwatson | looks pretty good to me | 23:34 |
| wgrant | cjwatson: Still very slow, but quite an improvement. Is that just index generation time? | 23:38 |
| cjwatson | I suspect the remaining time would bear some profiling, since it certainly shouldn't be maxing out on I/O there, but still | 23:38 |
| wgrant | ie. C' -> D time? | 23:38 |
| cjwatson | Right, just step C' | 23:38 |
| cjwatson | Maybe lacking indexes for some of the rest? | 23:39 |
| wgrant | All the queries you touched yesterday have sensible indices, but yeah, it's probably worth investigating now | 23:39 |
| wgrant | I'll have a poke when you're done with labbu. | 23:39 |
| cjwatson | I think I'm done now; that much is enough for me to say qa-ok I think | 23:40 |
| wgrant | Agreed. | 23:40 |
| * wgrant investigates. | 23:42 | |
| cjwatson | Released my vim lock on lib/lp/soyuz/scripts/publishdistro.py which remains cowboyed | 23:43 |
| cjwatson | Feel free to mangle/revert | 23:43 |
| wgrant | Thanks. | 23:43 |
| wgrant | 335 queries | 23:44 |
| wgrant | Hm | 23:45 |
| wgrant | I might hack it to only do one suite. | 23:45 |
| wgrant | Unless you have logs from the original run? | 23:46 |
| wgrant | kubuntu-ninjas trusty binary-amd64 took 2.2s in my latest test run. | 23:47 |
| wgrant | cjwatson: Did you keep your logs? | 23:47 |
| wgrant | 6.1s with the old code. | 23:49 |
| cjwatson | er, not in a file but I think I have scrollback | 23:50 |
| cjwatson | Bah, not enough of it, sorry. | 23:50 |
| wgrant | Nevermind, quick enough to rerun :) | 23:50 |
| wgrant | I guess I should disable compression and see how much that shaves off | 23:51 |
| wgrant | It was the limiting factor in my heavy caching experiments years ago. | 23:51 |
| wgrant | bzip2 is only 5-10% of the time | 23:54 |
| cjwatson | The {Plain,Gzip,Bzip2}TempFile scheme could be fixed to not do tempfile.mkstemp when it's just going to discard the result (in the gzip and bzip2 cases) | 23:56 |
| cjwatson | I expect that's epsilon though | 23:56 |
| wgrant | Yeah | 23:56 |
| cjwatson | Oh, just the bzip2 case, not gzip | 23:56 |
| wgrant | Will be interesting to see how much this improves things on prod | 23:57 |
| cjwatson | Still, stupid code layout | 23:57 |
| wgrant | We have network latency and auth etc. there. | 23:57 |
| cjwatson | Yeah | 23:57 |
| wgrant | So the many->notmany queries will be a much bigger win. | 23:57 |
| wgrant | Plus wildcherry is a potato. | 23:57 |
| cjwatson | kubuntu-ninjas trusty binary-amd64 took 23s on prod on 2014-05-13 | 23:59 |
| cjwatson | dunno how close to the same set of packages that is | 23:59 |
| wgrant | Roughly 46000 additional queries in step C' across all suites | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!