[00:01] wgrant: mumble? [00:02] sinzui: You can't hear me? [00:02] Once again I think I need to restart to hear [00:09] I'm amazed our test suite works at all [00:10] os.environ['HOME'] = self.root [00:11] in the buildd tactestsetup affects all subsequent tests. [00:11] but [00:11] it helpfully deletes that directory [00:11] Heh [00:12] so when rabbit tries to start up [00:12] it uses that as home [00:12] which doesn't exist [00:12] Ahh [00:12] two bugs: rabbit should isolate itself. [00:12] and we should restore global state we futz with [00:13] \o/ [00:14] bug 780794 if you're interested [00:14] <_mup_> Bug #780794: tests of/using BuilddSlaveTestSetup mangle global state for other tests < https://launchpad.net/bugs/780794 > [00:30] wgrant: I am perplexed. My test script only passes the person changes. None of the question_target methods work [00:30] sinzui: Did you change the inheritance hierarchy? [00:31] sinzui: Each object has a single interface exported. [00:31] So eg. IProduct needs to inherit IQuestionTarget. [00:31] No, But I am wondering I I need to [00:31] Product cannot implement IProduct and IQuestionTarget directly. [00:32] okay, that is my mistake [00:33] wgrant The branch is okay to release, but I need to followup with another branch to pass my test [00:33] sinzui: Great, thanks. [00:36] Yippie, build fixed! [00:36] Project db-devel build #535: FIXED in 5 hr 31 min: https://lpci.wedontsleep.org/job/db-devel/535/ [00:49] does Failure in test testEpochNonInteger (lp.archivepublisher.tests.test_debversion.VersionTests) [00:49] happen for anyone else? [00:51] Have you run update-sourcecode lately? [00:51] That test is new. [00:53] 30 rev deploy time :( [00:55] no, I haven't [00:59] lifeless: It was a bug in python-debian. [00:59] So update-sourcecode and try again. [01:01] Greeeeeeeen [01:02] zomg [01:02] Assignee picker on +filebug works on qastaging. [01:20] Project windmill-devel build #63: STILL FAILING in 44 min: https://lpci.wedontsleep.org/job/windmill-devel/63/ [01:40] Hmm. [01:41] I wonder how many users the API export of archive.checkUpload has. [01:42] * wgrant checks the PPR. [01:44] \o/\o/\o/\o/\o/\o/ [01:44] ? [01:44] Rabbit working? [01:44] yup [01:44] lp-land in a second [01:44] !!! [01:45] Did you decrapify the U1 harness a bit? [01:45] a tad more [01:45] Great. [01:45] Next stop: wildcherry. [01:45] but I want the darn thing in tree [01:45] Yeah. [01:45] so that the next 'if we had queues' comment I see I can drop the bomb on [01:45] Yes yes yse. [01:45] of course, we have 1 person (jtv) with srs queue experience [01:46] so its not at cargo cult point yet [01:46] :( 127 checkUpload calls last month. [01:46] Maybe I can grep logs and destroy whoever is doing that. [01:46] Stereotactic RadioSurgery? [01:46] Because it's a shit API. [01:47] jtv: serious [01:47] lifeless: ah... so, exciting rabbit things happening? Gimme! [01:50] :) [01:52] oh damn crappy shitty damn bad connection I smegging hate this [01:52] :> [01:52] so I answered you with [01:52] :) [01:55] lifeless: got the smileys, and would be joining in the smiles if it weren't for the broken connections! [01:55] o/ jtv [01:55] Hi spm! [01:56] And the good thing about bad connections is you get to keep saying hello to your friends. :) [01:56] jtv: the next step is a Layer for rabbit, which should be trivial with the fixture [01:56] Aigh! Not a layer! We want resources and fixtures! [01:56] jtv: and after that we probably need some test support - things to say 'a message was sent' and so on [01:56] jtv: we have a fixture [01:56] jtv: but it needs to fit in the current test layout [01:57] Which we all hate :) [01:57] sure [01:57] orthogonal problem [01:58] True. [01:58] anyhow [01:58] pull devel [01:58] its there [01:58] Great [01:58] Meanwhile, I thought I was being clever but am running into problems that you'd be able to help with. [01:58] shoot [01:59] My test takes 6s or so. I thought I could speed it up by re-using test SPPH objects across tests, because creating those seems to be taking most of the time. [01:59] I had a really nice idea for that: a generator that caches a list of SPPHs. [01:59] You request one, it gives you the next one from its cached list. [01:59] the db layer rolls back the db [01:59] Yes. :( [02:00] either have one test, or accept the cost, or test narrower layers that don't need the db (oops the last one is [for now] science fiction) [02:00] I suppose I might produce fakes, but it gets a bit hackey. [02:00] (Thank you, alarm clock, but I've been at work for hours) [02:01] This particular one is pretty painful. It'd have been nice to avoid it. [02:01] I do have a plan B: [02:01] see if somewhere in the factory we do commits just for the Librarian's sake. [02:02] If so, use FakeLibrarian and replace the commit with the fake. [02:02] It's really ridiculous: at one point I'm spending 0.4s on creating 2 test objects! [02:02] yeah [02:02] thats about 10 tests in bzr :> [02:03] Hmm not a lot of committing going on in the factory these days. That's good, but it means the remaining fruit hangs higher. [02:03] jtv: Have you profiled? [02:03] That's hideous. [02:04] Alternatively use makeSourcePackagePublishingHistory, which doesn't create any files AFAIK. [02:04] Only print-profiling so far. [02:04] At least I was able to use it in DatabaseFunctionalLayer without a problem. [02:04] It _is_ makeSourcePackagePublishingHistory that's taking up that time. [02:04] It seems to be a bit irregular somehow. [02:06] Some apparently trivial utility operations can be really slow. [02:06] NFI why. [02:08] I'm going to try it in "make harness" with SQL logging on. [02:08] See if it's doing any outrageous DB work. [02:09] Yeah. [02:09] There's lots of SQL. [02:10] Huh. [02:10] makeSPPH repeatably takes 0.3s [02:10] That's awful :/ [02:11] Quite. [02:11] Imagine what we could save if we had an easy way around that. [02:11] I guess it creates lots of people. [02:11] Which is slow. [02:11] Because of Account. [02:12] I was going to ask about that: how does that currently work? [02:12] Which? [02:12] Account? [02:12] Account. I half expected to see commits to synchronize the account/LP dbs. [02:12] Right, that was removed a while ago. [02:12] Since we no longer have a HA auth store. [02:12] Because SSO is a separate DB. [02:13] So you no longer need to commit after creating people. [02:13] and session is also not replicated [02:13] if we fail over its trashed [02:13] Looks like creating a person takes 15 queries :( [02:13] Still should be quick, though. [02:13] * wgrant fires up a profiler. [02:14] :( no context manager profiler [02:16] Can't start one of the regular python profilers in "make harness"? [02:17] Looks like makeArchive() takes up a significant portion of the time of a makeSPPH(). [02:17] Yes. But a context manager would probably be easier. [02:18] bzrlib.lsprof.profile(...) is your friend [02:18] and yes, it works in harness [02:18] Can we just merge bzrlib into the standard library or something? [02:18] *blink* [02:19] lp.testing.storm [02:19] * lifeless sobs [02:19] Does not exist. [02:19] lib/lp/testing/storm.py [02:19] pull devel [02:20] its quite buggy for new generic code [02:20] I pull in trepidation. [02:21] sinzui: when you said hack, I didn't imagine this [02:22] sinzui: (sorry for being negative; it is a bit of a tricky problem) [02:26] 28% in the DB, 20% clearing lazr.restful memcache... [02:26] the representation cache? [02:26] Yes. [02:26] turn the f*er off [02:26] 10% on top of the 28% in the storm tracer. [02:27] I thought the representation cache was off in prod anyhow [02:27] It is. [02:29] Killing the representation cache only shaved off a bit under 10%. [02:30] Now 50% in storm, 40% in the DB. [02:31] OOPS-1957DY36 [02:32] actually filing a bug should not lookup similar bugs >< [02:32] right, bug 780835 [02:32] <_mup_> Bug #780835: representation cache a pessimism < https://launchpad.net/bugs/780835 > [02:33] Pessimism or pessimization? [02:33] bah [02:33] yes [02:33] ESLEEPFFFFAIL [02:34] Hmm I got some savings of a few tens of seconds by eliding Person constructions. [02:34] fixed [02:35] jtv: createPersonAndEmail seems to take 33% of the time. [02:36] And a third of that is EmailAddress.new... [02:36] How. [02:37] [02:37] if self.getByEmail(email) is not None: [02:37] raise EmailAddressAlreadyTaken( [02:37] "The email address '%s' is already registered." % email) [02:37] LBYL :( [02:37] Yeah. [02:37] assert status in EmailAddressStatus.items [02:37] is inefficient (should be a type check) [02:38] Still trivial. [02:39] valid_email looks sane enough [02:39] how many persons are you creating? [02:39] 5, looks like. [02:39] oh [02:39] that LBYL isn't indexed. [02:39] Aha. [02:39] possibly [02:39] It must be, surely. [02:39] need to check one more thing [02:39] "emailaddress__lower_email__key" UNIQUE, btree (lower(email)) [02:40] ah, it is [02:40] 6% in PersonSet.getByName :( [02:40] At <100 persons in the sample data, would it matter anyway? [02:40] jtv: psosibly ;> [02:40] I nearly halved my test time just by eliminating makePerson calls. [02:40] but [02:40] its stupid to have an index on a function [02:40] lifeless: [02:40] when we can apply the function on data insertion [02:40] Sort (cost=6.65..6.66 rows=1 width=39) (actual time=0.146..0.146 rows=0 loops=1) [02:41] Sort Key: email [02:41] Sort Method: quicksort Memory: 25kB [02:41] But should still be fast, given how small it is :/ [02:41] -> Seq Scan on emailaddress (cost=0.00..6.64 rows=1 width=39) (actual time=0.096..0.096 rows=0 loops=1) [02:41] Filter: (lower(email) = 'email755122@example.com'::text) [02:41] That's the plan. [02:41] So WTF is it so slow. [02:41] It's sub-ms. [02:41] lifeless: absolutely... especially because last I heard, function indexes weren't quite as effective as I expected. [02:41] specifically, all queries on email come from our appservers [02:41] joins are on id [02:42] so we should be able to lower() in the appserver before querying rather than in the db [02:42] anyhoo [02:42] wgrant: how many ms does your profiling think it is? [02:42] wgrant: and what profiler did you use? [02:42] 5% of roughly 300ms. [02:42] So 15ms. [02:43] I used bzrlib.lsprof. [02:43] ok [02:43] The query is 200µs hot. [02:43] so @ 15ms that suggtests 14ms in python [02:43] wgrant: what exactly was slow? [02:43] wgrant: how long does the profiler think the cursor get took ? [02:44] jtv: EmailAddressSet.getByEmail [02:44] Weirdness. [02:44] Unexpected ASCII/Unicode conversion issue? [02:45] (Since this is one of the few cases where ASCII would probably be fine) [02:45] kcachegrind confuses me. [02:47] Or maybe it's just lying to me. It's not showing any callees beyond selectOne. [02:47] you've turned min-node down ? [02:51] As low as it goes, yes. But it's still showing some nodes without children. Hrm. [02:51] C modules add some fun [02:51] Stripped? [02:51] Or otherwise without debug info? [02:52] they show as a monolithic block in the profiler [02:52] because the profiler works by running code between every opcode [02:52] (thats a lie, its every line, shrug) [02:53] and when C modules call back into python the callgraph can be fun [02:53] 'perf' can sometimes show you where C modules are consuming time. [02:53] (Not tied into the Python callstack at all, of course) [02:53] I think kcachegrind might just be broken. [02:53] gprof2dot sees calls into EmailAddress.new as expected. kcachegrind sees 0. [02:54] win [02:55] Hmm. kcachegrind shows two createPersonAndEmail functions. [02:55] I wonder if some C in the middle is breaking everything. [02:55] One of them has no calls. [02:55] And the one with no calls calls the uncalled functions! [02:55] That's not very nice. [02:55] \o/ [02:55] that might be fixed by jams recent lsprof tweak in bzrlib actually [02:56] Ahh, and the harness will be using ancient bzr... [02:56] 2.2.2 :/ [02:57] We should really fix that. [02:57] yes! [02:57] bzr team have been doing that [02:57] Have they? [02:57] thats my understanding [02:57] IMBW [02:58] I don't think it's been updated since Code was disbanded and then everyone ran away,. [02:58] It was last changed 2011-02-09, when I applied a patch for a stacking bug. [02:58] So it hasn't been upgraded since Code evaporated. [02:59] => bzr probably doesn't do it. [03:12] lifeless: Ahh, that's much better. [03:12] The tree is no longer lots of trees. [03:13] (after a quick port to 2.4b2) [03:18] Perhaps I will toss it at ec2 and see what breaks. [03:18] After I work out how to use weave_fmt properly as a plugin. [03:24] Ahh [03:24] 2/3 of the getByEmail time is flushing writes. [03:24] Before it does the find. [03:25] It would be more illuminating if we could turn off the write cache... [03:37] Project windmill-devel build #64: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill-devel/64/ [03:42] Project windmill-db-devel build #262: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/262/ [03:54] wgrant: which write cache [03:54] lifeless: Storm's. [03:54] the same one that broke commits on the archive thingy the other week ? [03:55] Hm? [03:56] Anyway, I emulated disabling it by flushing after adding each new object. [03:56] when I eager loaded source packages [03:57] commits crawled to a halt because of O(cache size) behaviour in commit [03:57] Ah. [03:57] No, this is the dirty object cache. [04:09] I'm trying to decide between [04:09] fix a timeout [04:09] write a rabbit layer [04:09] $other [04:09] Rabbit. [04:10] Rabbit will fix lots of timeouts. [04:17] lifeless: staging would like to have http://paste.ubuntu.com/605983/ run on it with an EXPLAIN ANALYZE. [04:18] [qa]? [04:18] Either. [04:18] http://paste.ubuntu.com/605985/ [04:20] lifeless: And hot? [04:20] 340.216 [04:20] Slower than DF :( [04:20] 205ms hot. [04:20] -> Hash (cost=13594.89..13594.89 rows=5168 width=8) (actual time=112.326..112.326 rows=5189 loops=1) [04:20] But more like what I was expecting. [04:21] -> Index Scan using securesourcepackagepublishinghistory__distroseries__idx on sourcepackagepublishinghistory (cost=0.00..19355.35 rows=16746 width=24) (actual time=0.259..214.622 rows=17420 loops=1) [04:21] Hold on, my world is crashing down. [04:22] We'll have to revise the term "dog [food] slow" [04:22] wgrant: you just restored; less bloat [04:23] staging should have just done that, too? [04:24] hahahahaahhahahahahaha [04:24] no, restore failed [04:24] :-( [04:24] also this is qastaging [04:24] Ah [04:25] which is better to test on for things that aren't coupled to schema changes [04:25] But still only restores on demand :-( [04:25] (same hardware, longer lifetime - it gets its own bloat going, and its where we qa upcoming deploys) [04:26] wgrant: did you eliminate that silliness in packagecopier where a permissions error was suppressed if the user had Append privileges? [04:27] "if reason is not None: raise CannotCopy(reason)" instead of a nested "if" to give it another chance? === jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jtv | https://code.launchpad.net/launchpad-project/+activereviews [04:28] Or maybe it was Gavin. [04:29] And by the way wgrant, would you care to review my "copy asynchronously if there's more than 100 packages to copy" branch? https://code.launchpad.net/~jtv/launchpad/bug-780319/+merge/60571 [04:29] jtv: 100?! [04:29] jtv: we struggle to copy 2 in the webapp [04:29] That's what's been specced. [04:30] well, reality is going to intervene [04:30] Then feel free to lower the threshold to 1. :) [04:31] Er, doesn't it make use of PackageCopyJob? [04:31] Yes. [04:31] Then that's completly different. [04:31] lifeless: ^ [04:31] so if its using a job [04:32] its already asynchronous [04:32] isn't it? [04:33] jtv: the time budget for backend tasks that hold transactions is effectively 7-8 seconds (allowing time for webapp changes after they start actually doing writes [04:33] This implements a choice between synchronous and asynchronous. [04:34] does synchronous mean in-webapp [04:34] Yes. [04:34] if that is set to 1, it might work sometimes. [04:35] Is someone planning to speed that up massively, perhaps? [04:35] yes, by making it async [04:35] 964 OOPS-1955CE140 Archive:+copy-packages [04:35] I mean the actual copying. [04:35] 11 / 8 Archive:+copy-packages [04:36] Copying is very fast. [04:36] Copy checking is still slow sometimes. [04:36] and when its slow its glacial [04:36] 11 failures on 416 copies yesterday [04:36] 2.5% failure rate [04:36] Lots of those were probably delayed copies. [04:37] 99th percentile is 8.3 seonds [04:37] mean is 3 seconds [04:37] Non-sql time: 4333 ms [04:37] when you say 'fast' [04:37] lifeless: +copy-packages or syncSource? [04:37] Archive:+copy-packages [04:37] That's got UI crap. [04:37] Which is unoptimised and constant. [04:37] Archive:EntryResource:sync has its 99th percentile 11 seconds [04:38] mean 2.15 seconds [04:38] Right, because it's used a lot for delayed copies, or copies from -proposed to -updates. [04:38] Which close bugs. [04:38] None of which is optimised. [04:38] 99th percentil of db time is 6.5 seconds [04:38] Actual copying is fast. Delayed copies are not. Bug closing is not. [04:38] Checking when the source already exists in the target archive is not. [04:38] sure, but does 'do a copy' trigger these other htings? [04:38] Yes. They can and should be optimised. [04:39] if it does, then a discussion about copying has to assume that copying is slow [and optimisable perhaps] [04:39] In any case, my branch isolates a policy choice for when to do which. [04:39] which is great [04:39] I guess I'm saying that right now, I would put the policy at 0 [04:39] I figure improving and tuning that choice is a separate step. [04:40] jtv: in fact, there is a good request: please make sure its possible to disable synchronous copying entirely. [04:40] As long as these both use pretty much the same code. [04:40] And it isn't like delayed copies. [04:40] If it is anything like delayed copies I will rs it out now. [04:40] rs? [04:41] Delete without review, because delayed copies are just that hideous. [04:42] Where do delayed copies come into the picture? [04:42] They are the old asynchronous copy mechanism. [04:43] I think I saw bits of that, deeper down in the call tree. [04:43] Which is not even slightly like the synchronous copy mechanism. [04:43] If both of these use the direct copy function, I am happy. [04:43] When do we use which? [04:43] They use do_copy. [04:43] Delayed copies are used for copies from private to public archives. [04:43] So that's a policy decision, not some kind of time management? [04:44] Half and half and half of some more hacks. [04:44] Delayed copies are *hideous*, and I'm going to remove them [04:44] It was initially because those copies needed to reupload files to the public librarian. [04:44] Which was slow. [04:44] Then more hacks were layered on top that were unrelated. [04:44] And then it became clear that the initial rationale was wrong: you can just toggle the restricted flag in the DB. [04:44] But it all remains. [04:45] So... it might make sense for the sync/async choice to check whether any of the source archives are private and the dest archive is public? [04:45] This UI cannot go anywhere near production until delayed copies are dead and buried. [04:45] Or we'll have doubly async copies. [04:45] And badness. [04:47] If you're worried about doubly async copies, then I guess that would be a blocker for using PackageCopyJobs. [04:47] Doubly async is stupid, and the behaviour is inconsistent. a PackageCopyJob should never use a delayed copy. [04:47]