[00:01] lifeless, I'm signing off for now, will push the branch once the upgrade finishes [00:02] (which would be now) [00:09] mars: cool [00:23] lifeless: I think basic auth is still enabled, just using the old passwords. [00:23] Which is probably useless and a security risk. [00:27] oh, hahaha. [00:27] I just realised another interaction/issue with lowering hard timeouts [00:27] * lifeless sighs [00:27] Oh? [00:27] lifeless: ? [00:27] a single rogue page that renders slowly can cause 3 other requests to timeout [00:28] because python + threads is epic fail. [00:29] so the lower the hard timeout, the more boundary-case slow pages will cause *other* pages to have inexplicable timeouts like - https://lp-oops.canonical.com/oops.py/?oopsid=1681EB3871 [00:30] OTOH this may just be a genuinely slow template [00:33] lifeless: Maybe you should use that as an excuse to spawn a push to minimise Python time throughout the application. [00:33] wgrant: we're doing that by lowering the timeout :P [00:33] Is the performance report public again yet? [00:34] any losa around that can't sleep? I need a kcachegrind profile from staging please. [00:34] wgrant: no, it needs log filtering applied to strip private urls. [00:34] wgrant: technically even staff can't look at it atm. [00:34] Ahh. [00:35] well, thats a slight exaggeration, but you know what I mean: there *can* be stuff turn up that normally you'd need DB access to see [00:36] which is undesirable at best [00:38] OOPS reports don't expose quite as much? [00:38] they are also staff only [00:38] and also need to have this issue addressed [00:38] with the issue addressed we could start publicising them, which I would dearly love to do [00:40] * thumper is frustrated [00:40] damn code didn't work as I expected [00:40] which one? [00:41] QAing the fix I did for processing merge directives [00:41] wgrant: publishedpackage is what you nuked yes? [00:41] wgrant: what revision did it land in? has it landed? [00:41] I fixed one bug, but now it isn't actually pulling in the directive (it seems) [00:42] how often are the staging logs synced to devpad? [00:43] 7 minutes I think [00:43] perhaps 6 [00:45] :( known 2a and MD bug check failed: Cannot add revision(s) to repository: missing text keys: [00:45] * thumper thinks [00:45] only way to confirm fix is not to use 2a [00:45] lifeless: db-devel r9628 [00:46] damn [00:46] Not on staging yet? [00:46] wgrant: I wish you had done that as two separate patches, db + other [00:47] wgrant: no, 10 timeouts from it in sundays edge report [00:47] :( [00:48] Which views? [00:48] +distrotask? [00:48] +filebug [00:49] Ah. [00:59] tell me about SPPPH [00:59] sorry, SPPH [01:00] also, ugh, pagination *hate* [01:00] lifeless: got a stutter there? [01:01] can anyone push to staging right now? [01:09] lifeless: What about SPPH? [01:10] SSPPH died a few months ago. [01:12] wgrant: timeouts [01:14] lifeless: Ah, the getBuildRecords timeout? [01:15] wgrant: yeah, it's a separate class so we can port a bit at a time. It has a different API as I think this is better than the old one. [01:15] james_w: It looks like it, yes. [01:16] STP is pretty old and was put together over a long time. [01:16] yeah, it's understandable. IMO it needs to mature now that it is a central part of Soyuz's testing [01:16] Definitely. [01:16] And bonus points for eliminating sample data while we're at it. [01:17] yeah, that's my main aim, but as I started working with it I wanted to clean up the API [01:17] 0*$ bash Menu: [01:17] some changes are gratuitous, but others will help stop being the tests so tightly bound to the implementation, which is one of the problems that we have with sampledata, so I wouldn't want to replicate it at a different level [01:18] Indeed. [01:18] We have model code working around bad sampledata :( [01:18] So it will be good to eliminate it. [01:18] grah [01:26] hmm [01:26] this is nonoptimal [01:26] lpmain_staging=> SELECT Bug.heat FROM Bug, Bugtask, DistroSeries WHERE Bugtask.bug = Bug.id AND Bugtask.distroseries = DistroSeries.id AND DistroSeries.distribution = 3 ORDER BY Bug.heat DESC LIMIT 1; [01:26] heat [01:27] ------ [01:27] (0 rows) [01:27] Time: 235347.492 ms [01:27] ... [01:27] that's awfully long [01:27] you think? [01:27] It is staging, though. [01:27] But yeah, just slightly slow. [01:27] 14 seconds 'hot' [01:27] which matches production timing [01:29] http://pastebin.com/G8sctDVF [01:30] Hmm. [01:31] If I read it correctly its scanning every bug ever in descending heat and one by one comparing their distroseries distribution [01:32] what we'd really rather it do is expand the set of distroseries ids to be 'IN select distroseries.id from distroseries where distroseries.distribution = 3' [01:32] i don't suppose it matters here, but can there really not be an index on distroseries.distribution? [01:32] of course, also to do that magically. [01:32] maybe it's just such a small table that seqscan is quicker [01:32] mwhudson: it may be that that is needed. [01:32] It's a tiny table at the moment. [01:32] yes, but query analyser magic. [01:34] * wgrant wishes for an import fixer. [01:35] sed's handy for moving stuff, but then you have to reorder and merge lines :( [01:35] edwin has a vim script for it [01:36] I have his set of vim scripts. [01:36] Maybe I have it but just don't know about it. [01:36] * wgrant looks. [01:37] i want to write something that does all my import statement editing for me [01:37] (based on tags or something) [01:37] rope apparently has something to do this, but i never got ropemacs to work [01:37] Hm, so, yes, it has Ctrl+L which uses tags and adds the magical import. [01:37] And Ctrl+P supposedly reformats all the imports. [02:02] wgrant: if you're interested in performance stuff [02:02] DistroSeries:EntryResource:getBuildRecords [02:03] https://bugs.launchpad.net/soyuz/+bug/590708 [02:03] <_mup_> Bug #590708: DistroSeries.getBuildRecords often timing out

[02:03] wgrant: ^ the soyuz task on it needs to change to a better query in the short term [02:04] wgrant: danilo has crafted one, but its not been transcribed; should be very low hanging fruit, and its the top timeout from mondahy [02:05] poolie: shall we continue with that bug ? [02:24] hmm, when do we toggle from 'fix committed' to 'fix released' ? [02:31] sinzui: hi, btw Ursinha requested that we use oops and timeout tags as mutually exclusive things [02:32] lifeless, after the rollout? [02:32] Ursinha: no, a couple of weeks ago [02:32] Ursinha: you pinged me about filing bugs with oops & timeout tags, as said 'one or other' [02:32] lifeless, I answered your first question [02:32] hmm, when do we toggle from 'fix committed' to 'fix released' ? [02:33] Ursinha: ah, cool, ECONFUSED :) [02:33] hehe :) [02:33] lifeless, akc [02:33] or ack [02:35] Ursinha, When I see the release announcement, and I see the registry change is indeed on lpnet, I run the close_developer_bugs for the registry team. When I was RM, I ran it for all launchpad-project [02:36] sinzui, that's cool [02:36] lifeless, ^ [02:36] cool [02:36] I was asking because I wanted to fit in ;) [02:37] I know I sent it to the list a few months back. you can pass a list of launchpad ids to update, or ALL to DTRT [02:38] sinzui: the portlet thing [02:38] yes that thing [02:38] when I looked at an oops for the timeout [02:38] it shows a very long non-sql gap taking place [02:39] do you think that that is just the parsing and construction of the objects returned by the earlier query ? [02:40] lifeless, yes. EdwinGrubbs can to that conclusion [02:40] we only want a number too [02:40] right [02:40] I'm with you on that, just wanting to be sure we fix the core issue :) [02:41] sinzui: the thing is, that there are 2155 rows [02:41] sinzui: I find it hard to believe that creating 2155 objects takes 18 seconds of python time [02:41] sorry, 16 seconds [02:42] sinzui: I'm looking at https://lp-oops.canonical.com/oops.py/?oopsid=1681EB2995 [02:42] the last query that completed is at 960ms into the request, and has no limit - its the one you want to convert to get a count. [02:43] the distroseries is 103 based on another place in that page that has the literal [02:43] That is the one I was looking at [02:46] I think it is getting packaging, spn, product_series, product) because in most cases, you want to see all the items...but most cases also make the request with a batch navigatot [02:46] lifeless, ^ [02:47] so, that makes sense to me [02:47] lifeless, The property must use a decorated resultset too. It is iterating over *all* the items [02:48] it does, or it should ? [02:48] It should I think [02:48] makes sense to me [02:48] whats the property called ? [02:48] distroseries.packaging [02:50] lifeless, last year this property only had to content with a few hundred packages. It is time to review all the callsites and decide wha the model really needs to provide [02:50] s/content/contend/ [02:50] thanks [02:50] yes, I agree [02:50] scaling - this is part of my design guidelines, and its past time I wrote them up on the wiki [02:51] I almost got rid of distroseries.packaging 6 months ago too :( [02:51] I wanted to get some actual concrete fingers-dirty things done before making real suggestions [02:51] sinzui: :) [02:53] lifeless, we are down to 4 callsite, 3 of which is the distroseries model itself [02:54] so is the attribute the issue [02:54] or the table joins etc? [02:54] I guess I mean, whichi thing are yo uworking to reduce-the-use-of-and-delete? [02:54] lifeless, email sent [02:54] I am wrong, one callsite, just the one getting the count! we can replace it without packaging_count [02:54] (about oops/bug change) [02:55] Ursinha: sweet, thanks. [02:56] * sinzui sees EdwinGrubbs move the other callsites to use _all_packagings via specialised menthods [02:56] is '++profile++' the name of soom tool? [02:56] *some [02:57] It's a magic URL path segment you'll be able to add to Launchpad URLs to make it generate profiling data. [02:57] (once the relevant work lands, if it hasn't already) [03:00] lifeless: hi, yes, let's [03:00] scutwork of expenses and sysadmin is done [03:00] i wonder if paused kvms survive reboots? [03:00] probably not [03:01] The defn does [03:01] the instance doesn't [03:01] spiv: everything but the glue has landed [03:02] Are there any examples in Launchpad where the privacy of an object is determined by an 'owning' object? [03:02] lifeless, as of last night, i put that null object into the testrequest and that had a trivial failure [03:02] so let's try again [03:03] cody-somerville: not polished ones, no. [03:04] cody-somerville: but private bug attachments will be kindof this soon [03:04] cody-somerville: and branches already are via the 'branch policy' stuff [03:04] poolie: cool; I has caffeine and I has toast cooking [03:04] Thats not really what I meant [03:04] although branch policy describes the default privacy for a new branch, privacy checks aren't actually delegated to it [03:04] * poolie tries again [03:05] have you seen any deprecation warning failures in bzr builder to do with 'debian' vs 'debian_bundle'? [03:05] cody-somerville: sinzui's team will be working on something very much like this soon [03:05] cody-somerville: P3As do that sort of thing. [03:05] or is that just out of date sourcecode or something? [03:05] All the publications and builds within the P3A inherit their privacy from it. [03:05] poolie: jelmer did something related to that recently [03:05] wgrant, Is this a feature inherent to zope or something the soyuz folks bolted on? [03:06] The security adapters just delegate to the archive. [03:06] You could write a security adapter which just by default delegates to the parent, but that's not the default. [03:06] In a normal Zope app that would work fine, since everything has a parent. [03:06] Not so much in LP. [03:06] wgrant, Do you actually write a security adapter or is it all done via zcml? [03:06] god no [03:07] cody-somerville: One just needs to throw an adapter in lib/canonical/launchpad/security.py. No ZCML required. [03:07] * ajmitch is reminded too much of acquisition at the moment [03:07] ajmitch: Hm? [03:07] cody-somerville: its a four letter word around here :P [03:07] though that's more the containing object [03:08] ajmitch: its more uncles [03:08] how do you typically run tests? [03:08] make check? [03:08] or bin/test? [03:08] poolie: The former runs everything, so you can't use it. [03:09] ? [03:09] You want to use bin/test, unless you want to run the whole test suite locally, in which case you might be crazy. [03:09] poolie: testr run -- -t PATTERN [03:10] cos.. it'll take a long time? [03:10] poolie: ~4 hours on a fast machine. [03:10] that's ok [03:10] I just 'bin/test -1cvvt [sometestspec]' [03:10] Ah. [03:10] i have a decently fast machine [03:11] i don't see a good reason to run it on ec2 and leave this idle [03:11] but also, i want to run specific tests too [03:11] If you want to run the whole thing, do use 'make check'. It wraps things in xvfb-run so you don't get Firefox windows popping up everywhere. [03:11] on a separate vm [03:12] wgrant: so does testr run :) [03:12] lifeless: Hm, so it does. [03:12] Handy. [03:13] The good thing about waking up at decent times for my timezone is the first thing I see online is my antipodean friends. The bad thing is: there is a risk that they may be talking windmill. [03:13] wgrant: so what do you think about doing that query change? [03:13] lifeless: I was hoping that stub could come up with something better. [03:13] But apparently not. [03:14] poolie: ok, so where does that leave us - are you making headway, or fighting friction? [03:15] wgrant: I'm about to start hacking on another branch; you're better equiped to do that one, but I don't want to ignore the high-oopser [03:16] wgrant: which is why I'm looking for either 'I'm on it' or 'not me' from you :) [03:16] still moving forward [03:16] lifeless: I'm on it. [03:16] wgrant: \o/ [03:18] hm apparently 2GB is not enough to run the tests [03:22] lifeless: Can you please explain analyze these two queries a few times on staging? http://paste.ubuntu.com/475721/ [03:26] http://pastebin.com/AT04PS1P [03:27] WTF [03:28] poolie, I run the whole suite in 2G on a dual core 1.8Gh [03:29] sinzui: so this is the problem line ? [03:29] lifeless: http://paste.ubuntu.com/475724/ [03:29]

[03:29] ? [03:29] before that [03:30] sinzui: its my first time (in many years) debugging from the pt down, so please excuse my fail :) [03:30] lifeless, count view/num_linked_packages >> len(self.context.packagings) [03:31] ok [03:31] and that would be found by looking at the view class [03:31] yes? [03:31] (just so I know how the dots are joined) [03:32] lifeless, the line you were looking at is intended to return the 5 latest packagings...I noted that we may need an index on datecreated too [03:32] lifeless, yes, I see the view class doing len() [03:32] wgrant: http://pastebin.com/1mdW53Yv [03:32] sinzui: ok, cool. [03:33] sinzui: I'll have a poke [03:33] lifeless: Aha, thanks. [03:33] The query planner is stupid. [03:33] wgrant: remember that staging stats can be different to prod too, but yes - thats what that bug seemed to say, to me. [03:33] wgrant: do you think you've got a better offering than danilos? [03:34] lifeless: I'm trimming a couple of other bits out of the query that don't make sense. [03:34] ah, ye old holistic approach [03:35] sinzui: this looks odd to me [03:35] sinzui: my mistake, the vm only had 1gb [03:35] and no swap [03:35] results = results.order_by(Desc(Packaging.datecreated), SourcePackageName.name)[:5] [03:36] oh foo, ignore me. [03:36] ECAFFEINE> [03:37] would it make any sense to share ~/launchpad across VMs over nfs? [03:37] no [03:37] test isolation is - and I use the word advisedly - terrible in Launchpad [03:37] bzr doesn't work across nfs, remember! [03:37] * mwhudson hides [03:37] mwhudson: :P [03:37] :-P [03:38] might make sense to share your repo and use a checkout over nfs [03:38] the sourcedeps caches are pretty small and I wouldn't stress about them [03:39] well [03:39] my download-cache is twice the size of my launchpad repo [03:40] lightweight checkout time ;) [03:40] and the eggs dir about 1.5x the size [03:40] its actually a case where that works well [03:40] i guess that would save about 50% yar [03:40] the eggs dir I have to nuke regularly [03:40] i have a 1.5GB download cache [03:40] how do you clean it up? just randomly delete stuff? [03:40] that seems very large [03:41] mine is 415 megs and is a full branch i think [03:41] rm -rf eggs/*; make build (but of course be sure to delete a couple of things I never remember because the Makefile + buildout combo is ... traumatic [03:41] oh eggs can get huge [03:42] 611MB eggs; 470MB sourcecode; 416MB download-cache (of which 217MB .bzr) === Ursinha is now known as Ursinha-afk [03:44] sinzui: whats a good test case to run to be sure I haven't broken this ? [03:44] * sinzui thinks [03:47] lifeless, I do not see a single test for num_linked_packages [03:47] sinzui: well, anything that uses 'packagings' will be fine. [03:49] lifeless, lp/registry/doc/distroseries.txt has 3 tests [03:49] thanks [03:49] http://pastebin.com/xUwxKGC1 [03:50] hah, forbidden attribute len() [03:53] ok, so with all eggs deleted, i get 'no module ZConfig' [03:53] i guess it depends on actually having some eggs present? [03:53] yes [03:53] :/ [03:53] as I said - traumatic [03:53] you need to delete the generated files from bin/ [03:53] (bzr can tell you those) [03:54] and then do python ./bootstrap [03:54] and then 'make build [03:54] bin in my branch, or elsewhere? [03:54] in your working tree [03:54] lifeless, I do not think we need .packagings any more. No callsite wants the data returned. The only callsite wants count [03:54] sinzui: \o/ tests pass [03:54] sinzui: whats the page that shows it ? [03:55] just any distroseries ? [03:56] Upstream packaging [03:56] 3 source packages are linked to registered upstream projects. 1 need linking. [03:56] \o/ [03:56] any ubuntu or debian series [03:57] sinzui: well, this was very shallow, so I'll push it up and we can see ;) [03:58] fab [03:58] sinzui: I don't know enough to know what the right/wrong fix is here [03:58] other than that this should help [03:58] ok, re-bootstrapping [04:01] sinzui: https://code.edge.launchpad.net/~lifeless/launchpad/registry-packagings/+merge/32164 [04:10] lifeless, r=me [04:12] lifeless: so if 'testr run' mentions no failures, there were no failures? [04:13] sinzui: cool [04:13] poolie: thats right [04:13] sinzui: do you think this is rc ? [04:14] what's the invocation to re-run failures? [04:14] testr run --faling [04:14] bah [04:14] testr run --failing [04:17] lifeless, yes.I think it is RC. I expect it is also a CP candidate since we really want users seeing the distroseries page [04:18] well as rc I can shoot it off now and it should be in the rollout ? [04:19] ok now http://pastebin.ubuntu.com/475731/ is strange [04:19] poolie: how so ? the fact its failing? [04:20] yes [04:20] maybe it's not my fault [04:22] how would i re-run just a doctest? -t views_txt doesn't seem to work [04:22] maybe with a dot? [04:22] teset [04:23] yes [04:23] e.g. -t doc/views.txt [04:23] or whatever [04:24] do you have any guesses about the memcached test failure? [04:25] none sorry [04:25] it looks like it uses a subclassed test or something ? [04:25] sinzui: throwing it at ec2 [04:25] subclassed view I mean [04:25] woot! [04:26] maybe memcached is sad? i'm also getting "WARNING:root:Memcache set failed"... [04:26] just thinking aloud here [04:27] poolie, I think that means the test needs to run on the LaunchpadFunctionalLayer [04:28] Oh, it is on the correct layer [04:30] poolie, Did you change something on the distroseries-needs-packaging.pt page? [04:32] sinzui: he changed base-layout [04:33] I do not think that should have any impact on the test. The test does not care about layout/markup. [04:35] poolie, when you saw ""WARNING:root:Memcache" did you see the the memcache layer was brought up before the test? [04:35] poolie: you can do 'subunit-ls<.testrepository/$testid' to answer that question [04:36] lifeless: would it be easy to implement the fake librarian as a test resource? I could install it from setUp/tearDown but it'd be such a chore. [04:36] jtv: we don't have testresources glued in with layers yet [04:36] jtv: so, setUp/tearDown will be easier, for now. [04:37] afk for a bit [04:37] bummer [04:37] sinzui: it did say the layer had been started up [04:38] poolie, I see the test requires LaunchpadFunctionalLayer. I have see the warning reported when authoring a test on the wrong layer [04:38] poolie, memcache layer is brought up just before the LaunchpadFunctionalLayer... [04:39] do we need another scummy loop to make sure memcache has started accepting connections? [04:40] poolie, I think (speculate) that I have seen memcached startup/shutdown contentions between tests. I think I saw memcached not come up because it was not fully torn down in a previous test [04:40] I saw it only once, and the issue was only on my machine when I was refactoring [04:42] so maybe it's something about the specific tests i was asking for [04:42] :( [04:44] poolie, you may want to try --layer=LaunchpadFunctional to restrict tests to a certain layer [04:44] that runs only the tests in that layer? [04:44] correct [04:44] ok [04:44] i'm now trying to just run everything [04:51] everything seems to be passing [04:52] :) [04:53] though who knows what will fail later [04:53] and i found another tribunal bug :) [04:53] it should set stdin into nonblocking mode to cope better with slowly-fed pipes [04:53] jtv: did you get the mail 'Translation template import - typewriter in pidgin-typewriter trunk' just now ? [04:54] lifeless: I don't think I did. [04:54] forwarded [04:54] its a little worrying that it exposes all the email addresses [04:54] and [04:54] that we're getting it at all :) [04:55] That was a little side job I didn't quite get around to doing at Epic. [04:55] We want to disable these success notices for Ubuntu imports at the least. [04:55] i get a lot of these [04:55] i don't know about yours in particular [04:56]