=== slank is now known as slank_away
StevenKwgrant: Can you poke at http://pastebin.ubuntu.com/1451410/ and see if you can make sense of why my preloading changes don't seem to help at all?02:52
wgrantStevenK: What's not working?03:23
StevenKwgrant: It seems to me like the preloading PackageDiff call isn't working at all, since the code still loops loading PackageDiffs one by one03:24
wgrantWell, that's not really surprising03:25
wgrantHow would that work?03:25
wgrantThe PK is ID, not (from, to)03:25
wgrantSo simply loading them into the Storm cache isn't going to do much good03:25
wgrantAs the view is going to be querying by (from, to)03:25
StevenKwgrant: Eh? I'm using getDiffsToReleases() which should okay, no?03:26
wgrantSure, that'll load the right ones03:26
wgrantBut how will subsequent calls know that they're already loaded?03:26
wgrantHow will they find them?03:26
StevenKwgrant: What's more concerning is I can't even see the preloading query in the OOPS03:28
wgrantStevenK: It's there for me03:33
wgrantI see the initial diff grabbing the actual packagediff rows03:33
wgrantThen a subsequent query looking for the existence of a packagediff for the single source03:33
wgrantSo it all seems fine03:33
cjohnstonQuestion guys.. the bugs I fixed are starting to have qa-needstesting, am I allowed to test them? if so, what's the process for testing them?03:37
StevenKcjohnston: Try it on qastaging03:38
wgrantJust check that they don't obviously break anything on qastaging.launchpad.net03:38
wgrantIf the fix is good, change qa-needstesting to qa-ok03:38
cjohnstonok.. cool.. wasn't sure if as the code fixer I was allowed to do the reviews03:39
wgrantWe normally QA our own stuff if we're around03:39
cjohnstonok.. anyone know what the URL is to create a new distro?03:40
StevenKI think you need to be an admin for that.03:40
wgrantAh, yeah, normal users can't create them03:41
cjohnstondo you two have admin access?03:41
cjohnstonI can't test line 46 then https://code.launchpad.net/~chrisjohnston/launchpad/181725/+merge/14065303:42
wgrantI'm not sure that such a trivial change warrants the effort of obtaining admin privileges03:43
wgrantThe test suite is a pretty good indication that it works fine03:43
cjohnstonok.. the public views look fine.. so I can mark it qa-ok?03:43
cjohnstonits just a text change anyway03:43
cjohnstonok..thanks wgrant and StevenK03:43
StevenKcjohnston: You're welcome03:44
StevenKwgrant: Hmm, yeah. Now I see the preload03:44
cjohnston4 coomits in one week.. thats a record for me.. by about 303:44
StevenKHeh, I don't track how many commits I do03:45
cjohnstonStevenK: your kind-of on the LP team... I'm community :-P03:47
StevenKSo it appears my attempts at preloading to drop the query count don't really help.03:51
StevenKI take that back. Preloading changesfile drops from 237 to 213.03:53
* StevenK stabs zope.tal03:59
StevenKIt looks like the tal has a condition on not: packagediff or so, and that's forcing storm to grab each packagediff, LFA and then LFC04:00
wgrantStevenK: I'd expect a packagediff existence check, then a packagediff retrieval query, then LFA and LFC queries for each diff04:10
StevenKYeah, I'm trying to staple packagediff into CPU so I can stop those04:11
StevenKWhy it chains through to LFA and LFC, though ...04:13
StevenKpackage_diffs is a SQLMultipleJoin, and the TAL seems to only want a fmt:link04:13
wgrantWhy wouldn't that want LFA and LFC?04:14
wgrantIt needs LFA for the link04:14
wgrantAnd LFC for the size04:14
StevenKwgrant: 237 down to 186.05:04
wgrantGetting there05:06
StevenKNow 16705:07
StevenK(214-186: Preloading PUBs, BPBs and binary SPRs, 186-167: Preloading all SPNs for source and binary SPRs)05:08
StevenK153, preloading all DASes.05:11
StevenKHmmm, looks like preloading PCJs is causing duplicate queries against PackageUploadBuild05:40
StevenKNot sure how, given uploads should be a list of objects, and builds is a cachedproperty05:41
wgrantCheck what is generating the queries/05:42
StevenK    keys.update(map(partial(getattr, owning_object), foreign_keys)) in load_releated05:42
StevenKload_related, sigh05:42
StevenKOh, wait a minute, why is that calling checkPermission?05:43
wgrantIt may be trying to access a secured attribute to find the related IDs05:46
StevenKBleh, rewritting the PCJ query just pushes the calls to my LFA preloading05:51
StevenKAnd changing the LFA preloading makes the query count jump to 17805:56
StevenKBack down to 154, but this is terrible :-(05:56
StevenKBleh, and that just pushes it down to 'for item in uploads' :-(06:04
wgrantWhat are you doing?06:05
wgrantStevenK: How's it pushing it down? It should be cached06:06
StevenKwgrant: I'm trying to avoid the stacks of queries that the permission check is causing06:16
StevenKWhat I mean by pushing down is further along in the method -- it was inside loadPackageCopyJobs, and then in the bulk loading for LFAs for changesfile, and now it's at the end06:17
wgrantStevenK: For the same objects?06:21
StevenKuploads = list(self.batchnav.currentBatch())06:22
wgrantThey must be cached06:22
StevenKupload_ids = [upload.id for upload in uploads]06:22
wgrantIf checkPermission succeeds once, it will be cached06:22
wgrantSo I don't see how you're pushing them back06:22
StevenKwgrant: I can upload an OOPS if you wish06:22
wgrantHow did you do the permission preloading?06:23
StevenKI haven't06:23
wgrantHow'd you move it, then?06:23
StevenKBy avoiding load_related() and using upload_ids06:24
wgrantDon't you want to check permissions?06:24
StevenKHave you read the PU permission check? It's hideous06:25
wgrantBut that means you might want to prepopulate the permissions another way06:25
wgrantNot just somehow attempt to work around ever accessing an attribute of the protected object06:25
wgrantIn the end you are going to have to access a PackageUpload06:26
wgrantSo there's no point introducing ugly hacks to attempt to avoid checking permissions06:26
StevenKwgrant: I'm curious why the upload_ids = [upload.id for upload in uploads] doesn't cause them06:26
wgrantStevenK: id is presumably zope.Public06:27
StevenKRight, since we have the SPRs for each PU preloaded06:28
StevenKwgrant: load_related() avoidance hacks removed06:30
wgrantStevenK: You're fixing bug #835645, aren't you?08:17
_mup_Bug #835645: DistroSeries:+queue timeout paginating <lp-soyuz> <queue-page> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/835645 >08:17
StevenKwgrant: Hopefully08:20
=== yofel_ is now known as yofel
=== _mup__ is now known as _mup_
=== apw` is now known as apw
=== StevenK_ is now known as StevenK
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== gary_poster is now known as gary_poster|away
=== Ursinha is now known as Ursinha-afk
=== gary_poster|away is now known as gary_poster
=== Ursinha-afk is now known as Ursinha
=== deryck is now known as deryck[lunch]
=== deryck[lunch] is now known as deryck
lifelessgary_poster: btw19:53
lifelessgary_poster: I know you guys are not focused on LP any more, but I've added some more stuff to testr that may be of interest :) -19:53
gary_posterlifeless, hiya. :-)19:53
lifelessgary_poster: hiya :)19:53
lifelesshttps://testrepository.readthedocs.org/en/latest/MANUAL.html#remote-or-isolated-test-environments and19:54
gary_posterlifeless, so this could be used to build test runs across machines the way statik talked about way back when, yeah?  among many other things.  cool!19:55
gary_posternice lifeless.  I'm guessing you are using testr in anger, then :-)19:56
lifelessgary_poster: well I have been for ages; I got some employer sponsored time to step it up a level though.19:57
lifelessgary_poster: openstack's nova project now uses testr as its official test runner19:58
gary_posterlifeless, awesome!  great.  thanks for highlighting the changes--I'll keep 'em in my back pocket19:58
lifelessgary_poster: and yeah, multiple machines would be easier with this I think. Aaron's wrapper should become simpler too.19:58
lifelessgary_poster: the isolation analyzer is my favourite bling right now though. I think it would have made your analysis work a lot easier.19:59
gary_posterah yes, this is what I was going to do in Go and lost interest in :-)19:59
lifelessHeh :).20:00
sinzuijcsackett, you about?20:19
czajkowskisinzui: would you mind looking at https://bugs.launchpad.net/launchpad/+bug/1092643  if you have time please.20:24
_mup_Bug #1092643: Release Management and Downstream Tracking <feature> <Launchpad itself:New> < https://launchpad.net/bugs/1092643 >20:24
czajkowskisinzui: cheers20:36
czajkowskijcsackett: if you like mashups http://bootiemashup.com/blog/21:46
jcsackettczajkowski: cool!21:47
sinzuiwgrant, I revised the branch to convert the 400 to a 404 https://code.launchpad.net/~sinzui/launchpad/path-info-bad-request/+merge/14069822:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!