[01:08] <wgrant> cjwatson: How's the performance looking?
[03:03] <cjwatson> wgrant: About a minute for each run
[03:03] <wgrant> cjwatson: Nice.
[03:04] <cjwatson> Down from three and a bit
[03:05] <cjwatson> Looks like reasonably consistently 53s or nearby
[03:32] <cjwatson> wgrant: Is there any way to sensibly preload things that are modelled using SQLMultipleJoin(..., orderBy=)?  Storm doesn't seem to cache that in a way that can be satisfied by a previous load_referencing, not even if I expand out load_referencing's query to include a matching ORDER BY.
[03:34] <wgrant> cjwatson: No, Storm only caches by PK
[03:35] <wgrant> You'll need to turn the SQLMultipleJoin into a cachedproperty
[03:35] <cjwatson> Aha
[03:35] <wgrant> Depending on the callsites, that can be very easy or very hard.
[03:35] <wgrant> Which one is this?
[03:35] <cjwatson> SPR.files
[03:35] <wgrant> Ah, most things should be happy with that being a list.
[03:35] <wgrant> What are you trying to accelerate?
[03:35] <cjwatson> I got pissed off with PPA publisher performance and thought I'd try the obvious preloads
[03:36] <wgrant> Ah, yeah
[03:37] <wgrant> OK, this is really confusing. I'm watching a UOS session where you're talking, while talking to you about something totally different on IRC.
[03:37] <cjwatson> Everything else from source publication is easy to make constant query count by number of source packages.  Haven't tried binaries yet
[03:37] <cjwatson> Heh
[03:38] <wgrant> It's not a huge amount of work now that model cachedproperties aren't verboten.
[03:38] <wgrant> eg. SPR.files probably only needs invalidation in SPR.addFile
[03:38] <wgrant> But gina needs checking.
[03:39] <cjwatson> Right, I assume I just grep for SourcePackageReleaseFile as the starting point
[03:40] <wgrant> Yep.
[03:59] <cjwatson> Yay, that makes my record_two_runs test for source publishing pass.  Thanks.
[03:59] <cjwatson> I'll try the same for binaries in the morning.
[04:35] <cjwatson> Or maybe I'll have insomnia and JFDI.  Should actually try sleeping while more of the tests run though ...
[05:54] <wgrant> cjwatson: Did you go to bed? :)
[08:08] <cjwatson> wgrant: For a bit, yeah
[08:09] <wgrant> Disappointing.
[08:10] <cjwatson> Well, I'd written the binaries code before saying that above
[08:10] <cjwatson> Should have that up for you to look at a bit later today
[08:11] <cjwatson> Any luck with livefs*?
[08:16] <wgrant> Spent most of the day on log analysis and scalingstack stuff, but I'll hopefully have that sorted out tomorrow in the unlikely event that nothing else comes up :/
[09:29] <wgrant> cjwatson: How many queries were actually executed per pub?
[09:30] <wgrant> I guess one for the xPR, one for the xPR?Fs, then one for each LFA and one for each LFC.
[09:30] <wgrant> Plus the xPN
[09:30] <wgrant> And sometimes the component and section.
[09:31] <wgrant> And maybe one for the BPB, its SPPH, its SPR, its SPN and its component in the case of binaries?
[09:31] <wgrant> Though that might be done by postgres the view, I can't quite recall.
[09:31] <cjwatson> I think it was about five or six per pub
[09:31] <wgrant> in the view
[09:32] <wgrant> Ugh, some of that lint is mine.
[09:32] <cjwatson> Seven per source and six per binary if I'm reading my terminal scrollback correctly
[09:33] <cjwatson> SPR, SPN, SPRF, number of files * (LFA, LFC)
[09:33] <cjwatson> BPR, BPB, BPF, LFA, LFC, (maybe SPN), BPN
[09:34] <cjwatson> Also sometimes Section.  Component is effectively constant
[09:34] <wgrant> I was wondering how SPRs had more. Of course, multiple files.
[09:34] <cjwatson> Two in my test but of course often more.
[09:34] <wgrant> Hm
[09:34] <wgrant> We need to check the Storm cache config.
[09:35] <wgrant> I think it's capped at 10000 objects
[09:35] <wgrant> It is for the
[09:35] <wgrant> webapp, at least, I think
[09:35] <wgrant> I might have uncapped it for scripts a few years ago/
[09:35] <cjwatson> How many packages are there in the largest PPA?
[09:35] <wgrant> A few thousand
[09:35] <wgrant> 3000ish IIRC
[09:36] <wgrant> We probably want to do this in reasonably sized chunks, but your diff is better than what we have now, so meh.
[09:37] <cjwatson> I guess if we were to chunk it the preloading would have to move into archivepublisher
[09:37] <wgrant> As it bloody well should.
[09:37] <wgrant> None of that ridiculous publishing code belongs in registry :)
[09:37] <wgrant> Or even soyuz.
[09:37] <wgrant> Oh yes let's have the webapp publication objects know how to serialise themselves to disk, that's a good idea.
[09:37] <cjwatson> Well, I mean into the caller rather than the callee
[09:38] <cjwatson> Those get*PackagePublishing methods only have one call site each
[09:38] <wgrant> Yeah
[09:38] <cjwatson> (going out for a bit before UOS)
[09:41] <wgrant> I've been meaning to investigate chunking DecoratedResultSet.__iter__ for a while.
[09:41] <wgrant> It already does a bit of evil, so it wouldn't be too bad.
[09:42] <wgrant> Have it iterate through chunks of a caller-specified size, so it does 10 bulk queries each over 1000 objects rather than one over 100000.
[09:44] <wgrant> Ignore the missing order of magnitude there.
[09:57] <wgrant> Inline comments are nice.
[10:24] <stub> wgrant: it is capped, but you can override it in the launchpad.conf per script
[12:05] <wgrant> cjwatson: How'd the source caching test go, modulo ENOSPC?
[13:31] <cjwatson> wgrant: Looked fine, was about four minutes quicker
[13:32] <cjwatson> Results looked correct
[13:32] <cjwatson> I've asked mvo to do a sourceful trusty backport so that we can then build that lot on precise
[13:38] <wgrant> :)
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/main/source/: 6392 pkgs in 35s
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/universe/source/: 39628 pkgs in 3min 37s
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 8s
[13:43] <cjwatson> 15:53 <cjwatson> then second run:
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/main/source/: 6392 pkgs in 0s
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/restricted/source/: 24 pkgs in 0s
[13:43] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/universe/source/: 39628 pkgs in 6s
[13:44] <cjwatson> (both with the new apt-ftparchive, but the first with no sources cache)
[13:44] <cjwatson> 15:53 <cjwatson>  dists.in-progress/utopic/multiverse/source/: 1102 pkgs in 0s
[13:44] <cjwatson> So does exactly what it says on the tin, I think.
[13:46] <cjwatson> I wonder where I should put ubuntu-archive-publishing config, re https://code.launchpad.net/~cjwatson/launchpad/configurable-germinate-base/+merge/194340 .  Maybe I should just have a file where we look things up by os.environ["LPCONFIG"]
[13:49] <wgrant> cjwatson: That or a config file named by LPCONFIG sounds reasonable.
[15:43] <cjwatson> wgrant: https://code.launchpad.net/~cjwatson/launchpad/preload-publisher-indexes/+merge/222768 should be ready for re-review now.
[15:50] <wgrant> cjwatson: Thanks.
[15:51] <wgrant> cjwatson: We'll want to test variously sized archives on DF to check for any weird postgres decisions on the huge queries, but that's nice and easy with NMAF (no pool required).
[15:51] <cjwatson> no pool required?
[15:52] <wgrant> Testing a-f at scale can only be done on pepo, as it depends on the pool being intact.
[15:52] <cjwatson> oh, right, you mean we can let NMAF put the pool in place
[15:52] <wgrant> No, we don't need a pool.
[15:52] <cjwatson> I'm obviously not awake.  I understand now.
[15:52] <wgrant> Well considering you were off IRC for two hours.
[15:53] <cjwatson> Andreas had a didn't-want-to-sleep night.
[15:53] <wgrant> :(
[15:53] <cjwatson> Also, <3 caffeine
[15:53] <wgrant> Heh
[15:54] <cjwatson> OK, that a-f smoke-test on DF would have been more effective if I'd changed something.
[15:54] <wgrant> Clean suites are quick to publish, at least.
[15:54] <cjwatson> Oh, it did change.  Good.
[15:54] <cjwatson> Just looking at the wrong publish-distro run.
[15:54] <wgrant> Ah
[15:55] <cjwatson> And see your previous comments about needing a pool.  Well, didn't explode.
[15:55] <wgrant> It won't explode.
[15:56] <wgrant> But the indices'll be pretty empty.
[15:56] <cjwatson> Yeah, just memtest86+ in main.  Shrug.
[15:57] <cjwatson> [15:57] <cjwatson> thanks, bzr
[15:58] <cjwatson> Anyway, we can deploy once lpqateam catches up.  Shall I request that?
[15:58] <wgrant> Sounds like a good plan.
[15:59] <wgrant> Have you done one lately?
[15:59] <wgrant> webops will ask for an RT
[15:59] <wgrant> But otherwise as usual.
[15:59] <cjwatson> Not lately, but I'd seen you switching to RTing things.
[16:00] <cjwatson> Eh, what the heck did generate-extra-overrides just do on labbu
[16:01] <cjwatson> Oh, maybe it's talking to production for its series information
[16:01] <wgrant> Ah, heh, probably.
[16:02] <cjwatson> Wonder what to do about that.  I could just cowboy -l dogfood in cron.germinate on labbu for now ...
[16:08] <cjwatson> wgrant: bug 1328355 was applied live?
[16:08] <_mup_> Bug #1328355: Distribution.guessPublishedSourcePackageName times out frequently <easy> <qa-ok> <timeout> <Launchpad itself:Fix Released by wgrant> <https://launchpad.net/bugs/1328355>
[16:08] <wgrant> cjwatson: It was.
[16:12] <cjwatson> Grr, api.dogfood.launchpad.net
[16:12] <cjwatson> I guess we should SRU python-launchpadlib
[16:13] <wgrant> I need to move the stagings at some point as well.
[16:13] <wgrant> Luckily -l https://api.dogfood.launchpad.net/ works
[16:13] <wgrant> er.
[16:13] <wgrant> -l https://api.dogfood.paddev.net/
[16:14] <cjwatson> Yeah, I was just thinking and checking that
[16:19] <cjwatson> OK, cowboyed to death now.  Getting rid of lp-query-distro will kill one of the cowboys.
[16:20] <wgrant> :)
[16:20] <cjwatson> I'll probably shove the endpoint URI into the config infrastructure I'm writing for bug 1248869.
[16:20] <_mup_> Bug #1248869: Stop the publisher from talking to people.canonical.com <soyuz-publish> <ubuntu-archive-publishing:In Progress by cjwatson> <https://launchpad.net/bugs/1248869>
[16:20] <wgrant> Yup, makes sense.
[16:20] <wgrant> You have the ndt etc. under control?
[16:20] <wgrant> I might wander off.
[16:20] <cjwatson> Yeah, go ahead.
[16:20] <wgrant> Thanks.
[18:42] <cjwatson> wgrant: I self-reviewed a testfix to preload-publisher-indexes to unblock buildbot.  Hopefully it looks right to you.
[21:18] <lifeless> stub: is there a way in pytz to get a tzinfo object given a utc offset?
[23:10] <wgrant> cjwatson: Looks good, thanks.
[23:12] <cjwatson> I'm trying some test runs with it now.
[23:12] <cjwatson> "/srv/launchpad.net/codelines/current/scripts/publish-distro.py -vv -d ubuntu --ppa --ms -A", hacked to publish only a single PPA, taking time between "Step C'" and "Step D", second run to ensure hot cache
[23:12] <cjwatson> cjwatson/ppa, canonical-kernel-team/ppa, launchpad/ppa, kubuntu-ninjas/ppa
[23:13] <cjwatson> Any other suggestions?
[23:13] <wgrant> Those are pretty good options.
[23:13] <cjwatson> -A publishes all series, but the more the merrier.
[23:14] <cjwatson> (Actually I'm currently getting control data from r17041.)
[23:33] <cjwatson> cjwatson/ppa                    (2.835) 1.260           (1.412) 1.375
[23:33] <cjwatson> canonical-kernel-team/ppa       (33.896) 23.570         (9.972) 10.152
[23:33] <cjwatson> launchpad/ppa                   (9.644) 3.739           (3.332) 3.367
[23:33] <cjwatson> kubuntu-ninjas/ppa              (69.837) 43.198         (19.578) 20.031
[23:33] <cjwatson> wgrant: ^- owner-name/ppa-name (r17041 first run) r17041 second run (r17043 first run) r17043 second run
[23:34] <cjwatson> archives compare functionally identical (aside from compression timestamps and suchlike)
[23:34] <cjwatson> looks pretty good to me
[23:38] <wgrant> cjwatson: Still very slow, but quite an improvement. Is that just index generation time?
[23:38] <cjwatson> I suspect the remaining time would bear some profiling, since it certainly shouldn't be maxing out on I/O there, but still
[23:38] <wgrant> ie. C' -> D time?
[23:38] <cjwatson> Right, just step C'
[23:39] <cjwatson> Maybe lacking indexes for some of the rest?
[23:39] <wgrant> All the queries you touched yesterday have sensible indices, but yeah, it's probably worth investigating now
[23:39] <wgrant> I'll have a poke when you're done with labbu.
[23:40] <cjwatson> I think I'm done now; that much is enough for me to say qa-ok I think
[23:40] <wgrant> Agreed.
[23:42]  * wgrant investigates.
[23:43] <cjwatson> Released my vim lock on lib/lp/soyuz/scripts/publishdistro.py which remains cowboyed
[23:43] <cjwatson> Feel free to mangle/revert
[23:43] <wgrant> Thanks.
[23:44] <wgrant> 335 queries
[23:45] <wgrant> Hm
[23:45] <wgrant> I might hack it to only do one suite.
[23:46] <wgrant> Unless you have logs from the original run?
[23:47] <wgrant> kubuntu-ninjas trusty binary-amd64 took 2.2s in my latest test run.
[23:47] <wgrant> cjwatson: Did you keep your logs?
[23:49] <wgrant> 6.1s with the old code.
[23:50] <cjwatson> er, not in a file but I think I have scrollback
[23:50] <cjwatson> Bah, not enough of it, sorry.
[23:50] <wgrant> Nevermind, quick enough to rerun :)
[23:51] <wgrant> I guess I should disable compression and see how much that shaves off
[23:51] <wgrant> It was the limiting factor in my heavy caching experiments years ago.
[23:54] <wgrant> bzip2 is only 5-10% of the time
[23:56] <cjwatson> The {Plain,Gzip,Bzip2}TempFile scheme could be fixed to not do tempfile.mkstemp when it's just going to discard the result (in the gzip and bzip2 cases)
[23:56] <cjwatson> I expect that's epsilon though
[23:56] <wgrant> Yeah
[23:56] <cjwatson> Oh, just the bzip2 case, not gzip
[23:57] <wgrant> Will be interesting to see how much this improves things on prod
[23:57] <cjwatson> Still, stupid code layout
[23:57] <wgrant> We have network latency and auth etc. there.
[23:57] <cjwatson> Yeah
[23:57] <wgrant> So the many->notmany queries will be a much bigger win.
[23:57] <wgrant> Plus wildcherry is a potato.
[23:59] <cjwatson> kubuntu-ninjas trusty binary-amd64 took 23s on prod on 2014-05-13
[23:59] <cjwatson> dunno how close to the same set of packages that is
[23:59] <wgrant> Roughly 46000 additional queries in step C' across all suites