=== salgado is now known as salgado-afk [00:04] I suspect Distribution:+bugtarget-portlet-bugfilters-stats is going to be mondo painful [00:04] Yes. [00:04] also we need to remove the override on Archive:+index [00:05] How low is it now? [00:05] The page, I mean. [00:05] Not the 16000 override. [00:05] 8.26 [00:05] Nice. [00:05] https://devpad.canonical.com/~lpqateam/ppr/lpnet/2011-03/daily_2011-03-15_2011-03-16/pageids.html [00:05] Kill it. [00:05] Yes, but that takes a couple of minutes to load. [00:07] Hmm. Maybe I want to look at Person:+contactuser. [00:07] Since its 99% is twice the next worst. [00:08] 0.11 [00:08] mean sql [00:08] For +contactuser? [00:08] hmm, 99th percentile forsql is mising [00:08] wgrant: yes [00:08] Right. [00:08] It'll be mail. [00:08] wgrant: its probably mail sending [00:09] wgrant: we would benefit a lot if we had instrumentation on how long sending each email takes [00:09] its been hard getting that evaluated by is (for fairly good reasons) - but it still an open question around where the issues live [00:10] archive:+index should wait for monday [00:20] lifeless: can I get you to mentor wallyworld? https://code.launchpad.net/~thumper/launchpad/blueprint-linked-bug-tasks/+merge/53734 [00:22] thumper: I need to pop up to the shops for lunch stuff [00:22] thumper: will review after [00:22] lifeless: ok, thanks [00:30] Does anyone know of a way to use feature flags in JavaScript? [00:31] yes [00:31] export the value of the flag in the .pt file [00:31] interpret it in your js [00:34] lifeless: That was my plan, but I was having trouble getting some context values. Maybe I'll try again. [00:35] its been done before :) [00:35] huwshimi: e.g. showing the render time in the top right [00:45] lifeless: heh no wonder it wasn't working. The feature flag wasn't set. The shame. [00:45] huwshimi: rotfl [00:46] lifeless: I didn't realise that it didn't keep them between make runs [00:46] huwshimi: make schema will zerg them from your dev insance [00:47] huwshimi: 'make run' won't reset them [00:47] lifeless: I didn't think I'd done a make schema... but maybe I had [00:47] huwshimi: make schema completly resets the db to 'pristine sampledata' [00:47] yeah [00:47] Now to wash my eyes out for saying 'pristine sampledata' [00:52] * thumper is mildly confused by a comment in code [00:53] model/person.py line 1797 [00:54] thumper: i guess _getPrecachedPersons isn't in the interface? [00:54] it doesn't need to be [00:54] lifeless, i liek your idea about incomplete (bug 737008) [00:54] the self object isn't security wrapped [00:54] <_mup_> Bug #737008: old incomplete bugs are not treated as incomplete < https://launchpad.net/bugs/737008 > [00:54] once inside the object [00:54] maybe we should make that bug ask for what you said (as Low) [00:54] you are free of the security proxy on self [00:54] poolie: care to do that ? [00:54] sure [00:55] thumper: sure, but if you did getUtility(IPersonSet) you get a proxied object [00:55] thumper: you can use the proxied set and see what happens.... [00:55] mwhudson: yes you do [00:55] but once you call a method on the object (that is in the interface) [00:55] the self passed through isn't wrapped [00:56] otherwise that'd be insane [00:56] sure [00:56] * thumper is 98% sure of this [00:57] bbs [01:49] lifeless: you back? [01:49] anyone know how to clear the storm cache for tests? [01:58] store.reset [01:58] store.invalidate [01:58] IIRC [01:59] hmm [01:59] I should have been on leave today [02:00] oh yes, the memorial [02:01] I'll talk to flacoste, do a day off next week [02:03] thumper: ^ store answers for you === statik_ is now known as statik [02:17] lifeless: ta [02:17] lifeless: do I need both, or just the second? [02:18] either [02:19] one is deeper than the other [02:19] invalidate will cause existing objects to be refreshed from the db on next use [02:19] reset will make them unusable and cause errors if you try to carry an object from before the reset forward [02:24] ok [02:24] anyone remember off the top of their head what the storm equivalent of the sqlobject SQLRelatedJoin is? [02:25] lifeless, i wonder if you (or mrevell or someone) should post to the blog to explain about the critical/high/low categories [02:25] re all the "why is this now Low" comments [02:26] hmm [02:26] possibly [02:26] but there have only been 3 or 4 [02:26] in several hundred bug updates [02:26] (one bug - the wiki one - had a longer discussion that I'm counting as one) [02:27] I don't think this is really all that interesting to most users of lp [02:29] maybe not [02:34] I think people are responding because they get notified [02:34] and would do so even if they knew that medium is equivalent to low, because the thing they care about is having their change made [02:40] OMFG [02:40] this code is soooooo... bad [02:40] * thumper enfixorates [02:43] thumper: Which code? [02:44] sprints [02:44] Heh. [02:45] like the way to add an attendee, it iterated through all the attendees to see if the person matched [02:46] huwshimi, did you think you fixed the bug where you have to press enter twice to save the tag list? [02:46] doesn't seem fixed for me on lpnet [02:47] one enter selects the tag, the other saves [02:47] huwshimi: btw [02:47] yes, i know [02:47] huwshimi: try putting a non-typeahead tag into a bug [02:47] but that makes no sense if i've already completely entered a tag [02:47] poolie: The fix is in lazr-js which I haven't got around to releasing a new version of yet. [02:48] huwshimi: its not done till its done [02:48] ok thanks [02:48] just wondered if it was in-progess or failed validation [02:48] 1029 OOPS-1902EE937 BugTask:+nominate [02:48] 1029 statements on +nominate!? [02:48] win [02:48] sorry just one more thing there, what controls which tags show up in the portlet? [02:49] oh, deja vu, this is a caching bug, isn't it? [02:49] Probably, yes. [03:08] :( [03:08] Sprints are that bad? [03:09] Can I burn bug heat? [03:09] wgrant: no [03:10] :( [03:10] Does anybody use it enough that the performance penalty is worth it? [03:10] the performance penalty is not intrinsic to having heat [03:11] No. [03:11] But we need to either optimise or destroy. [03:11] optimise [03:11] distro are asking for heat improvements [03:11] Well, destruction is the ultimate optimisation :) [03:11] :( [03:11] getUtility(IPersonSet).getPrecachedPersonsFromIDs isn't doing what I'd expect [03:11] if we don't deliver the (thing) then its not optimisation anymore [03:12] thumper: list() [03:12] thumper: are you listifying it and passing need_validity=True ? [03:12] thumper: You need to evaluate the result. [03:12] wgrant: yeah got that [03:12] and I ahve need_validity=True [03:12] ok [03:12] so what are you seeing [03:12] but it is still doing the valid query for some of them [03:12] and if I add more people [03:12] then it does shit loads more [03:12] even getting some people [03:12] I'm wondering if I'm hitting the storm cache limit [03:12] where are you doing the preloading ? [03:13] there is no cache limit [03:13] test says different [03:13] lifeless: Even in tests? [03:13] its much more likely that something is triggering loading and doing a person:fmt before your eager load happens [03:13] I'm preloading in Sprint.attendances [03:13] * thumper tries something [03:13] wgrant: if you test with getUserBrowser, should be same as prod [03:14] thumper: you're using the matcher for testing this, right ? [03:14] yep [03:14] and the DatabaseFunctionalLayer [03:14] BrowsersWithQueryLimit or whatever it was [03:23] Hmm. [03:23] I wonder. [03:23] wgrant: dangerous [03:23] Should I turn recalculateBugHeat into something that adds a commit hook? [03:23] As a quick fix for all this heat badness. [03:23] what do you mean [03:24] At the moment any change to a bug calls recalculateBugHeat, which then sets maxheat everywhere and blah. [03:24] Some views call addChange several times. [03:26] so [03:26] uhm [03:26] I don't know what you should do [03:26] I can share the thoughts I had looking at this previously [03:26] Well, I should move heat updating to a job. [03:26] But that's a bit hard. [03:27] hi thumper [03:27] updating the heat in a context should be exactly two single-row queries always. [03:27] bac: hi [03:27] thumper: hey could you re-review my branch? i addressed all of your issues. [03:27] ok [03:27] wgrant: [in the current design/implementation] [03:28] bac: diff is updating [03:28] thumper: despite what i said in the earlier comment, i've removed the line-height altogether [03:28] wgrant: the fact that its now looked like a very low hanging fruit [03:28] s/now/not/ [03:28] Yippie, build fixed! [03:28] Project db-devel build #464: FIXED in 4 hr 44 min: https://hudson.wedontsleep.org/job/db-devel/464/ [03:28] lifeless: if I reduce the number of people I add to the sprint, I get expected behaviour. with 30 people, it is fine, with 50 I get some validation queries, with 100 I get people and validation queries [03:28] lifeless: so I'm guessing it is a caching issue [03:31] thumper: I've done some pretty large datasets and not see that, but you might be right [03:31] thumper: is it constant regardless of people in launchpad.dev (I use the harness to populate data for that sort of manual test) [03:32] wgrant: does that make sense, or have I been to minimal in my description ? [03:32] lifeless: It makes sense. [03:33] lifeless: I guess it should be reasonably cheap, but it's still going to be somewhat duplicated. [03:33] I'll fix the underlying stuff first. [03:33] (mostly the fact that DSP's pk isn't (distribution, sourcepackagename), so Storm likes to reretrieve it lots) [03:33] wgrant: things like: why do we query 'hottest bug' - if this bug has higher heat, update the cache, if not do nothing unless our heat lowered, and in that case [rare] query highest bug and set if and only if different [03:35] lifeless: Right. [03:35] wgrant: I don't see any reason to move heat to a job [03:36] wgrant: I wouldn't object to it per se, I just don't see anything driving it [03:38] lifeless: so where is this storm cache defined? [03:38] rendered locally, a sprint with 100 attendees was 30 queries [03:38] 200 attendees was 30 queries [03:38] 400 attendees was 102 [03:39] wondering if I was hitting the limit again [03:39] heh, do we even use stormsugar.py [03:39] I was looking for it [03:39] but didn't end up using it [03:40] thumper: grep for StupidCache [03:40] Is that seriously its name? [03:40] bac: I think you were too aggressive in your code removal [03:40] bac: with no link it should be a and it is no more [03:43] lifeless: Well, for one thing it seems pretty unwise to be locking Ubuntu during every change to an Ubuntu bug. [03:44] wgrant: right, but only *one bug* can cause that [03:44] wgrant: note that all bug updates will take a row lock preventing deletes on ubuntu [03:44] lifeless: We hope. [03:45] wgrant: if there are two bugs fighting for peak heat [03:45] wgrant: we can deal [03:45] Maybe. [03:46] StevenK: What are you doing to Jenkins? [03:46] Moving it [03:46] Well, moving its DNS name to lpci.w.o [03:46] Aha. [03:47] hmm [03:48] does bug expiry look a the project group ? [03:48] if so, that would explain why its not working for launchpad [03:48] and there is no ui for setting it on project gourp [03:49] It doesn't seem to use the project group setting. [03:50] wgrant: It should announce URLs as lpci.w.o anyway. hudson.w.o is now a CNAME. [03:50] wth defines 'fixed elsewhere' [03:50] zomg [03:50] lifeless: Where have you seen that? [03:50] we support *per distroseries* bug expiry [03:50] Er, what? [03:50] Do we? [03:50] https://lp-oops.canonical.com/oops.py/?oopsid=1857C157#longstatements query 4 [03:51] That seems like crack. [03:51] Pure crack. [03:51] its mindblowingly insane [03:51] lifeless: Er. [03:51] That's not per-distroseries bug expiry. [03:51] That's finding bugs in distroseries in distributions that are expirable. [03:52] Sigh, lp-oops DIAF [03:52] It always misbehaves for me [03:52] Authentication drops the query string, yes. [03:52] https://bugs.launchpad.net/ubuntu/+bugtarget-portlet-bugfilters-stats - thats where the 'fixed elsewhere' thing shows up [03:52] wgrant: thats true, but the query is nutjob noddy anyway [03:52] lifeless: Welcome to Launchpad. [03:53] wgrant: And https://lp-oops.canonical.com/ is the most unhelpful front page EVER [03:54] argh [03:55] extra_clauses.append("BugTask.bug IN " "(SELECT DISTINCT bug FROM BugCve)") [03:55] -not win- [03:56] StevenK: Ouch, whatever server is running that site must be really hurting if that's running in debug mode [03:57] huwshimi: That's devpad. [03:57] the fixed elsewhere queries are buggy too - they ignore product series bugs [03:58] wgrant: I'm guessing it doesn't have much ram free. [04:01] Meh, it's only 220MiB into swap [04:05] well we probably shouldn't be running it in dev mode anyway [04:05] or rather debug mode [04:05] huwshimi: why not? [04:06] lifeless: In debug mode Django holds every query in ram [04:06] huwshimi: since start up? [04:06] huwshimi: or for a request lifetime? [04:06] lifeless: I think so [04:07] The only thing that isn't playing so nice with carob is the PPR [04:07] generation ? [04:07] Yes [04:07] yeah, its started failing again [04:08] lifeless: Sorry I mean, I think it is since start up [04:08] huwshimi: right, well that could be cause for concern, yes. [04:08] lifeless: Let me just check the docs [04:09] lifeless: "It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you are debugging, but on a production server, it will rapidly consume memory." from: http://docs.djangoproject.com/en/dev/ref/settings/#debug [04:10] hahahahahahha [04:10] my respect for django just increased [04:10] so, could you please file a bug on oops-tools noting this? [04:12] lifeless: You had some? :-) [04:12] StevenK: its gone so far in one direction that things that make it worse, make it better [04:13] wgrant: Last SPR processed by p-s-c was for edgy. [04:13] StevenK: We are getting somewhere. [04:13] Slowly. [04:14] The OO.o SPRs are taking 1 minute. [04:18] lifeless: We might have to check our other Django deploys too. [04:19] (I believe we have others) [04:20] * wgrant stabs postgres in the eye. [04:21] * mwhudson has one too.... [04:21] (but it hardly uses the database so that might be ok) [04:22] lifeless: https://bugs.launchpad.net/oops-tools/+bug/737327 [04:22] <_mup_> Bug #737327: django should not be running in debug mode on production, it will eat all the RAM < https://launchpad.net/bugs/737327 > [04:23] https://code.launchpad.net/~thumper/launchpad/sprint-attendees/+merge/53941 [04:24] lifeless: and other review poke [04:24] * thumper goes to get friday takeaways [04:30] thumper: reviewing [04:30] thumper: I have suggestions fwiw, but nothing mandatory so far [04:32] ok, I'll look later [04:32] * thumper afk [04:38] wgrant: do you know where the contents of /srv/launchpad.net/ubuntu-archive/ubuntu-distscopy come from? [04:42] jtv: It's maintained by the script. [04:43] It keeps two copies of the dists tree around. [04:43] So it can do reasonably atomic updates. [04:43] wgrant: what is "the" script? [04:43] "the script" being cron.publish-ftpmaster [04:43] Ah [04:43] It grabs the backed up dists tree ubuntu-distscopy, and moves it to ubuntu/dists.new [04:44] It then runs the publisher on that. [04:44] I see it copying _from_ ubuntu-distscopy… does it write _to_ ubuntu-distscopy further on? [04:44] See cleanup() near the top. [04:44] So this is a chicken-and-egg situation? [04:45] Howso? [04:45] # Create backup dists folder, if this is the first time. [04:46] That copies _from_ ubuntu-distscopy [04:46] Then it will move the empty folder to dists.new, swap dists and dists.new bring the new dists.new up to date, then move dists.new into ubuntu-distscopy [04:46] s/ bring/, bring/ [04:46] But publish-distro is failing to write a Release file there [04:46] Oh? [04:47] grumpy? [04:47] No, a freshly-created distroseries. [04:47] Does it have any ComponentSelections? [04:47] Errr [04:47] You need ComponentSelections in all your publishable series. [04:47] what's a component selection? [04:47] Or you'll have no components. [04:47] And the publisher will crash. [04:47] "Components? But Andy Tanenbaum told me that Linux was monolithic!" [04:47] (not optimal, but it's better than lucilleconfig) [04:48] Bug #675042 [04:48] <_mup_> Bug #675042: Release file generation fails for series without components

< https://launchpad.net/bugs/675042 > [04:48] Does that look like what you're seeing? [04:49] Yes, except it's not in /var/tmp/archive. [04:49] So, try adding a ComponentSelection for main to your new series. [04:50] Does the test publisher know how to do that for me? [04:51] I doubt it. [04:51] Not even the LOF does. [04:51] The Line Of Flight? [04:51] Launchpad Object Factory. [04:52] I'm tempted to fix the bug, but in this case that'd probably be hiding incomplete test coverage for cron.publish-ftpmaster, wouldn't it? [04:53] It would. [04:58] * jtv sings hi-ho, making componentselections we go [05:04] lifeless: Hi. [05:06] wgrant: hi [05:06] hi ho [05:06] hi ho [05:06] its off to work we go? [05:06] Ha ha. [05:06] Could you please explain analyze query 106 from https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1902A2028? [05:06] It is a wonderful 9978ms abomination. [05:06] \o/ [05:07] It takes about 30s on mawson, and adding a very sensible index brings it down to 2s. [05:07] what bug [05:08] Bug #276950, probably. [05:08] <_mup_> Bug #276950: DistroSeries:+queue Timeout accepting many packages queue page

< https://launchpad.net/bugs/276950 > [05:08] Although this is probably only bad on GETs. [05:08] still running [05:08] Heh [05:09] I guess the index was not terribly bad when it was created. [05:10] still going [05:10] I'm going to go fiddle this portlet a bit more and check back in a bit [05:10] Thanks. [05:11] ok, on the bug [05:11] hot is 600ms [05:12] Bah. [05:12] Thanks. [05:12] cold is 201945.176 [05:13] Over 2 minutes? [05:13] wgrant: so you're adding distinct there [05:13] Over three minutes. [05:13] to stop redundant rows? [05:13] I presume so. [05:14] lets see [05:14] 53 rows [05:16] 720 rows without the distinct [05:18] wgrant: so its doing about 10 times the work needed [05:18] wgrant: because the non distinct version is still 600ms [05:19] lifeless: It must think that a full scan is a better idea. It's probably right. [05:23] wgrant: rephrasing it may work [05:24] wgrant: its doing index scans [05:24] wgrant: an index that permits doing distinct inline may make it a lot better, we can ask stub to play with that later [05:29] wgrant: I'm adding a makeComponentSelection to the factory. [05:29] jtv: Why? [05:29] the constructor is fairly trivial. [05:29] The only bonus is omitting an import. [05:30] wgrant: makes it easy to say self.factory.makeComponentSelection(distroseries, "main") [05:31] Ah [05:31] Accepts component, component name, or none. [05:31] Also, it means that yokels such as yours truly won't have to worry about just how trivial they are to create. :-) [05:32] And meanwhile I'm on to the next hurdle: missing Sources files [05:33] Even with a component? [05:33] It may be the thing you ran into with my parallelization branch—it's a-f. [05:33] WARNING a-f: E: Could not open file /var/tmp/archive/distribution510022/dists.new/distroseries510031/main/source/Sources.bz2.new - open (2: No such file or directory) [05:34] Hmmm. [05:34] Does that dir exist? [05:35] (Luckily I rigged this test with a "now tar up the temp directory so I can have a look" finale) [05:35] Heh [05:35] oh wait, this is in /var/tmp [05:36] It'd be nice to avoid messing with /var/tmp, wouldn't it? [05:36] Yuck, crud piling up there! [05:37] wgrant: the dists.new directory isn't present [05:37] What's in /var/tmp/archive? [05:37] Besides dozens of distribution dirs. [05:39] Dozens of distribution-like dirs with "-cache," "-misc," "-overrides," "-partner," or "-temp" suffixed to their names. [05:40] What evil have to done to cron.publish-ftpmaster? It sounds like it's not moving the backup in. [05:40] s/have to/have you/ [05:42] I made it use a temp directory instead of /srv/launchpad.net, I sabotaged the lockfile usage, and I faked dsync-flist and commercial-compat.sh. That's about it, really. [05:43] echo "$(date -R): Moving backup dists into place..." [05:43] Does it do that? [05:44] wgrant: so its about 300ms in the bpr lookup [05:44] and 300-500 (depending on plan) in the bpph filtering [05:44] lifeless: Right. [05:45]