[00:19] * StevenK takes ILaunchBag out the back, shoots it in the head, and then stabs it for good measure. [00:34] * mwhudson cheers StevenK on [00:34] StevenK: for real? or is that just wishful thinking? [00:35] mwhudson: Wishful thinking [00:35] boo [01:16] Damn [01:16] branch is down more than 90%, but it's still there :/ [02:39] http://dilbert.com/strips/comic/2007-11-26/ [03:29] wgrant: 100000 loops, best of 3: 7.32 usec per loop [03:31] bah, typo [03:40] ah, thats better [03:40] :!bash ./profile [03:40] 10 loops, best of 3: 21.7 msec per loop [03:41] lifeless: Heh, that's slightly less implausible. [03:42] Is that compile? [03:44] pybars [03:44] 10 loops, best of 3: 21.9 msec per loop [03:44] pystache [03:44] 10 loops, best of 3: 30.8 msec per loop [03:46] lols [03:55] comfortably faster b/4 optimisation [03:56] could do 1500 items of that scale without worry within a second, I suspect [03:56] will want some detailed tuning [03:59] The previous timings were compile, I assume? [04:00] no [04:00] running the same template [04:00] but with no data [04:00] because lp.cache != what we feed the template [04:00] so all the top level lookups found nothing [04:01] 1500 * 30.8 > 1000 [04:01] thats pystache [04:01] 1500 * 21.9 > 1000 [04:01] you are forgetting the /75 [04:01] Oh, so that's the template for the whole list? [04:02] yes [04:02] (also, lpnet average down to 395% from the 560% it was 2 weeks ago) [04:31] wgrant: Do you want to review https://code.launchpad.net/~stevenk/launchpad/no-active-for-dsd-spr/+merge/93762 ? [04:33] StevenK: r=me [04:33] So, that's OOPS #3 (not #2 as I thought earlier) fixed. [04:34] If you fix #1 (easy) and #2 (probably trivial) as well, we'll be at around 100 exceptions per day. [04:34] And #4 isn't real [04:35] StevenK: Actually [04:35] wgrant: Hmm? [04:35] Mm, I guess it's fine. [04:35] Since it picks the most recent. [04:35] And filters by version [04:36] So there shouldn't be an issue with too many or non-latest resolds. [04:36] results [04:36] Right [04:36] * StevenK looks for our #1 OOPS [04:36] It's None: None [04:36] Which is usually log.warning or log.error [04:36] In most cases it's garbo whinging that a task timed out. [04:36] Can probably be demoted. [04:37] Ah, bug 888049 [04:37] <_mup_> Bug #888049: garbo-frequently logged an oops < https://launchpad.net/bugs/888049 > [04:37] Yeah [04:38] gnargh I hate security proxies breaking object identity. [04:38] Garbo tasks timing out are a concern - we need to ensure it keeps up, even if that means increasing time slices or moving hourly jobs to daily or on both lists [04:38] stub: Mm [04:38] We care that it keeps up in the end. [04:39] At present to care that it keeps up daily we have to put it in garbo-daily. [04:39] When in fact we may want to do it -hourly, but don't care if it lags for two hours. [04:39] Or more, thanks to the backups [04:39] Right [04:39] Yes, and the only way we have of doing that without adding features is by ensuring it always finishes in its allotted time slice (which might be changed) [04:40] Backups were an issue, and we fixed that. [04:40] They where a problem [04:40] They still are AFAIK [04:40] Backups are happening on a slave and garbo isn't looking at that slave afaik. If it is, that is a problem we need to fix. [04:41] I thought that didn't matter. [04:41] The OOPSes don't look like a problem of not running [04:41] They are hitting the slice limit and being killed [04:41]

: [BugHeatUpdater] Task aborted after 3299 seconds. [04:42] Ah [04:42] keeps getting blocked by librariangc and autovacuum now :/ [04:42] Instead of backups. [04:42] But yes, bugheatupdater is terrible. [04:42] I'm not sure why. [04:43] librariangc? That is a new one. How is that blocking? Long running transactions? [04:43] But the whole lot often get blocked by autovacuum for an hour [04:43] Long running transactions on both counts, yeah [04:43] 2012-02-20 02:53:23 INFO [DuplicateSessionPruner] Blocked on 1:42:21.965885 old xact postgres@launchpad_prod_3/25895 - . [04:43] 2012-02-20 02:53:23 INFO [DuplicateSessionPruner] Blocked on 0:44:56.982871 old xact postgres@launchpad_prod_3/11353 - autovacuum: VACUUM pg_catalog.pg_index. [04:43] oh... librariangc is a victim, not a cause [04:43] 2012-02-20 00:02:06 INFO [BugHeatUpdater] Blocked on 1:07:30.239287 old xact librariangc@launchpad_prod_3/32758 - . [04:43] Possibly true, indeed. [04:44] Right, so it's using self.log.warn() [04:44] So we can silence these (don't report a task timeout if we got blocked for reasons beyond our control) [04:45] Or we can make dblooptuner ignore autovacuum, which scares me a little. [04:45] Perhaps we record the total time each task has been blocked. [04:45] stub: My plan was to fix looptuner to not warn, so it stops causing >100 OOPSes a day [04:45] And if it's more than $threshold, don't warn. [04:46] That is easy enough to do IIRC [04:46] We can go with that. [04:46] Monitoring long running transactions can be done better than watching batch jobs produce errors [04:47] Haha [04:49] haha, I just read flacoste's email babout serial tab openers team. lol. [04:50] nigelb: Did you see the cricket? :-P [04:50] stub: That BranchLookup.getByUniqueName fix freed 1.5 cores on wildcherry. [04:51] Which might almost mean that we're not screwed if we have to failover. [04:51] StevenK: Nope, been offline helping run an conference. [04:51] nigelb: Aus 5/288, India 178 [04:51] fuuuuu. [04:51] now? [04:51] Heh [04:51] That was last night [04:52] Ah. I was a zombie last night after being on the road all day. [04:52] Just woke up after a 12-hour glorious sleep. [04:52] Hmm, switching to self.log.info() means it doesn't get printed by the doctest now. :-( [04:54] * StevenK can't even work out how the warnings are injected into the stream [04:56] wgrant: \o/ [05:09] StevenK: http://www.devsigh.com/sigh/67 [05:13] Hah [05:16] * StevenK sighs at looptuner.txt === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [05:44] grar [05:44] how does AMD manage to make fglrx into such a piece of shit [05:44] Bwaha [05:44] one bug at a time [05:45] Perhaps they're actually benevolent [05:45] They just make it unstable in order to improve FS crash handling cases. [05:47] Or they just can't write drivers for their own hardware [05:47] Never attribute to stupidity that which can be adequately explained by them being proprietary arses. [05:48] And yet you say AMD are *better*? :-P [05:50] In some ways :) [06:25] what's fashionable these days for running lp dev? [06:26] lxc, bare (precise) metal? [06:26] LXC is in fashion, and there's even a script to set it up, but I'm not sure if it works. [06:26] Yellow is making it really easy to set up at present. [06:26] hm [06:26] does it run ok on precise? [06:27] That's the target. [06:27] perhaps i'll just go with bare metal for now [06:27] Oh [06:27] LP on precise? [06:27] Mostly, yes. [06:27] A few tests fail, but it otherwise works. [06:31] poolie: the target for the paralleltests project is lxc (lucid) hosted on precise [06:31] poolie: the setuplxc script in tree should do a decent job of putting that together, starting from a precise 'bare metal' [06:32] lifeless, i reinstalled my precise laptop a while ago so i have to choose between putting lp deps on the base os, or sometihng else [06:32] perhasp i'll try that [06:32] *setuplxc [06:47] would i be right in guessing less-oops.sh is obsolete now? === Ursinha` is now known as Ursinha === Ursinha is now known as Guest37626 [07:06] setuplxc is stuck at [07:06] mbp@lptests's password: [07:07] Hm. It should automatically let your key in, AFAIK. [07:07] or rather, i'm not sure what password it's expecting [07:07] But it's the same as your normal system password. [07:08] if it is, ssh is rejecting it [07:08] :/ [07:12] ok [07:12] Feb 20 07:05:19 localhost sshd[457]: User mbp not allowed because shell /usr/bin/zsh does not exist [07:12] Ah, heh. [07:12] i can fix that, but later [08:26] Oh no [08:26] Surely not [08:34] ? [08:34] lifeless: Found out why the heat updater is slow [08:34] lifeless: Goal time is 2s, as normal. [08:35] The initial query is 2.8s. [08:35] So it degrades to roughly 1 update per 3 seconds. [08:35] And reads 154000 bug rows every 3 seconds. [08:36] I bet if I fix that it updates the whole lot in 5 minutes. [08:36] why is it reading 154K rows? [08:37] Probably because it doesn't update duplicates, but the index doesn't exclude duplicates. [08:37] wgrant: as in, why doesn't it have a LIMIT batch_size there ? [08:38] lifeless, well, i pruned one small bit of cruft for you [08:38] not yet the timeline stuff though, due to EYAK [08:38] thank you [08:39] wgrant: uhm, you know we're running a branch of storm already.... [08:39] wgrant: so why you copy code? [08:39] Mmm, I guess. [08:40] I prefer to be non-invasive where possible. [08:40] dropping the non-null requirement on duplicates, and updating them appropriate, makes it fast [08:40] launchpad_qastaging=> EXPLAIN ANALYZE SELECT id FROM bug WHERE (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13') LIMIT 1; [08:40] Time: 1.906 ms [08:42] lifeless: What if you add back the duplicateof and leave the LIMIT 1? [08:42] 7.7 seconds, table scan [08:42] Yep [08:42] with dup is null and order by heat_last_updated nulls last -> 2.2 sec [08:43] I think we probably just fix the index to be duplicateof IS NULL. [08:43] * wgrant tries. [08:43] erm [08:43] actually [08:43] did you get the actual results as well ? [08:44] there is a pattern, when you see a big slow scan even with a limit [08:44] Oh, there's meant to be an ORDER BY id there too [08:44] Missed that [08:44] Just to probably make it even slower. [08:44] nah, not that [08:44] I didn't get the results. [08:44] I know [08:44] But thought I should note it. [08:44] so, ordering by the index will force it to use it [08:45] (often) [08:45] the reason its slow is that it is finished [08:45] tada. [08:45] SELECT id FROM bug WHERE duplicateof IS NULL AND (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13') order by heat_last_updated desc LIMIT 1; [08:45] id [08:45] ---- [08:45] (0 rows) [08:45] wgrant: it is slow, and consulting all rows, because it can't fulfill the request; even when limited [08:46] Hah, true. [08:46] wgrant: it has an estimate of 155K rows which is bong; it *is* examining all million rows [08:47] so, I think there are two bugs: if it is not limiting its results to the current batch size, that is bong. [08:47] And if it continues to try to do shit when nothing is returned, that is bong. [08:47] wgrant: this is the same slowness that batches that run past the end of big data sets like BPR exhibit [08:47] wgrant: where it suddenly goes off into lala land and reads through to the end of the table [08:48] I think it is limiting, actually, I misread. [08:48] ok cool [08:48] Yeah [08:48] in which case, this is a nuisance but also a symptom - it should just stfu and stop :) [08:48] But all this would not be an issue if the index was right. [08:49] the index is fine [08:49] change the query to have order by heat_last_updated [08:49] 3 bongs, a shit and lala land. [08:49] that will update the oldest rows first (which is strictly correct anyhow) [08:49] lifeless: The index is less than ideal. [08:49] Because it has to do a bitmap heap scan to determine duplicateness. [08:50] Avoiding order by heat_last_updated as that isn't stable ordering [08:50] And it will have to read all those rows first. [08:50] wgrant: I don't see a bitmap heat scan in the explain [08:50] Because they'll be older. [08:50] Bitmap Heap Scan on bug (cost=12576.08..141492.57 rows=154605 width=4) (actual time=2861.017..2861.017 rows=0 loops=1) [08:50] Recheck Cond: ((heat_last_updated IS NULL) OR (heat_last_updated < '2012-02-13 00:00:00'::timestamp without time zone)) [08:50] Filter: (duplicateof IS NULL) [08:50] That looks like a bitmap heap scan filtering on duplicateof to me :) [08:50] that was without the limit right ? [08:51] explain analyze SELECT id FROM bug WHERE duplicateof IS NULL AND (heat_last_updated IS NULL OR heat_last_updated < '2012-02-13') LIMIT 100; [08:51] QUERY PLAN [08:51] ------------------------------------------------------------------------------------------------------------------------------------------------------- [08:51] Limit (cost=0.00..100.82 rows=100 width=4) (actual time=1261.132..1261.132 rows=0 loops=1) [08:51] -> Seq Scan on bug (cost=0.00..155602.61 rows=154338 width=4) (actual time=1261.131..1261.131 rows=0 loops=1) [08:51] Filter: ((duplicateof IS NULL) AND ((heat_last_updated IS NULL) OR (heat_last_updated < '2012-02-13 00:00:00'::timestamp without time zone))) [08:51] With the limit it seq scans here [08:51] Total runtime: 1261.219 ms [08:51] Which is hardly better. [08:51] shrug, I'm fine with you improving it. [08:52] Just saying, doesn't seem terrible given there are no rows [08:52] So, it either has to do a seq scan or it has to do an index scan. [08:52] If it does a seq scan it's probably going to take forever. [08:52] If we make the index partial, we have to ensure that all of the queries also have the duplicateof IS NULL clause [08:52] If it does an index scan, it has every dupe bug ever earlier on in the index. [08:53] stub: All one query? :) [08:53] stub: tis already partial [08:53] It's not partial. [08:53] oh, not its not [08:53] But the one query that uses it will work if it's partial. [08:53] Right, I should sleep. [08:53] That's why I want it partial :) [08:53] "bug__heat_last_updated__idx" btree (heat_last_updated) [08:53] Anyway, I'm right, will submit DB patch with a few more indices later. [08:53] * wgrant dines. [08:54] its very odd that its not using the index today [08:54] as the index is a superset of the data needed; it must think there are an overwhelming number of stale-hat duplicates [08:54] wgrant: I suggest changing the code to update the heat on duplicates. [08:54] wgrant: make it simpler, leave the index as-is [08:55] wgrant: (yes, the update will be trivial..) [08:56] As there are 0 rows matching the clause, there should be 0 rows in the stats matching that clause. So the planner should think that, if they exist, they are unlikely and hit the indexes. [08:57] wgrant: so yea, [08:57] select id, duplicateof, heat, heat_last_updated from bug where heat_last_updated in (select min(heat_last_updated) from bug); [08:57] id | duplicateof | heat | heat_last_updated [08:57] --------+-------------+------+---------------------------- [08:57] 199626 | 199600 | 0 | 2010-05-28 17:50:04.055114 [08:57] (1 row) [08:57] we've been carefully shoving all the duplicate rows to the front of the desired index [08:57] stub : but the planner doesn't know this [08:58] stub: what it knows is that the index *starts* with a lot of data - 196K rows - [08:58] lifeless: I'm talking from the pov of the planner. It has to assume that the rows might be there, and that they are uncommon because they are not in the stats, so index lookups would be a win unless it needs to scan for some other reason. [08:58] stub: me too [08:58] stub: the thing is, they are common :) [08:59] stub: the first 196K rows in the index are for duplicateof is null bugs [08:59] setuplxc has new doctests? [08:59] wgrant: updating duplicates would be more in line with using this as an occasional re-fresh-the-data script too - we might not always have duplicate imply 0, so this would remove a booby trap. [09:00] stub: select count(*) from bug where heat_last_updated < '2012-02-13'; [09:00] count [09:00] -------- [09:00] 196094 [09:01] Aloha [09:02] Oh right. [09:03] so I think the planner is more or less saying '1/5 of the bugs in the table are relevant, which means I probably will find what they want on the first page -> scan' [09:03] of course, then it gets horribly disappointed :) [09:04] lifeless: That's another option. [09:04] wgrant: I favour it, for the reasons I mention above [09:04] Yep [09:04] Glad you saw the light about the index :) [09:04] Its saying there are 196094 bugs where heat_last_updated < '2012-02-13', and zero bugs where heat_last_updated is null. But it is too stupid to know that there are 0 bugs where head_last_updated is null or < '2012-02-13' [09:05] stub: s/heat_last_updated/duplicateof/ :P and yes. [09:05] stub: The assumption in that second sentence is false. [09:05] Right [09:05] What lifeless said. [09:05] wgrant: well, I see that the index isn't performing, I don't think we should change it, all things considered. [09:05] anyhoo; i leave you and stub to debate it :> I'm offtosleep [09:06] Night. [09:06] Thanks! [09:06] Ah, crap. [09:06] That will probably need a DB patch. [09:07] wgrant: what will? [09:07] Or I workaround it by wrapping the calculate_bug_heat calls with a guard for dupeness. [09:07] lifeless: calculate_bug_heat doesn't return 0 for dupes. [09:07] https://pastebin.canonical.com/60572/ [09:07] That's implemented in updateHeat [09:07] wgrant: you wouldn't just call updateHeat(bug) ? [09:08] lifeless: It currently uses RS.set() [09:08] I guess this will still be a thousand times faster. [09:08] So could just use updateHeat [09:08] yeah [09:08] think I'm on the wrong query [09:08] I mean, we're doing two things [09:08] a) moving to update-once, ever, so not caring about 'older than one week' rather 'older than constant value' [09:09] and b) making the operation decently efficient [09:09] wgrant: or we could store non-zero heat for duplicates [09:10] wgrant: I don't think it would break anything [09:10] Hello [09:10] lifeless: That's true. [09:10] It has always seemed odd to me that a bug suddenly gets 0 heat when it becomes a dupe; dupes are filtered from results, so the heat should be orthogonal; and if someone searches for bugs including dupes, sorted by heat... [09:10] Would remove a fair chunk of the remaining heat code [09:10] (ie. 5 lines) [09:10] why would we shove all the dupes to the bottom of the list [09:11] hi mrevell === almaisan-away is now known as al-maisan [09:11] So, I'll remove the dupe => 0, and remove the non-dupe filter? [09:11] mrevell: I've been meaning to catchup with you, but ETERRIBLETIMING [09:11] wgrant: that works for me; see if you can convince yourrandomreviewer of the sanity of it ;) [09:12] wgrant: you should also land either a knob, or a hard coded value, to replace the '1 week old' thing, unless you've done that already. [09:13] good morning [09:13] lifeless: I haven't. Might do it in the same branch. [09:13] A flag to set the earliest valid timestamp, if null then nothing happens. [09:15] yah [09:15] -> be [09:15] d [09:15] noight [09:17] night wgrant [09:17] Not me, lifeless :) [09:19] meh, wgrant as no days and nights. he's awake throughout. [09:20] Heh [09:23] lifeless, I'm available 22.30 UTC my Monday (11.30 your Tuesday, I believe). Does that work for you? [09:39] bigjools: But only because Queenslanders wouldn't know what daylight savings time was if it bit them on the bum. :-) [09:46] StevenK: quite! [09:46] StevenK: I am quite happy with that though, means I stay closer to UK time === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [14:01] daft question time, I do aplogise in advance [14:02] if a person requests ownership of a project and the registery are already the owners and seems to have some clue as to what they are talking about and an interest in helping the project [14:02] is it ok to hand it over once fully checking they will maintain it? [14:08] czajkowski: yes [14:09] flacoste: aloha there :) [14:09] thanks [14:09] hi! [14:09] someone wants to take over ownership of Mandriva [14:18] interesting [14:19] czajkowski: note that Mandriva is a distribution, and what you can do with this in LP is kind of limited [14:19] distribution not using Soyuz [14:19] mainly you can only track bugs [14:19] nods [14:19] which doesn't make sense here since Mandriva doesn't use LP for bug tracking :-) [14:19] flacoste: https://answers.launchpad.net/launchpad/+question/188064 [14:20] flacoste: which is why I'm kinda confused and trying to work it out, will get there eventually :) [14:23] czajkowski: seems like a reasonable request, what is interesting is that he thinks that the Mandriva community would use LP for project planning, like the derivative he's involve dwith [14:23] nods [14:23] I did meet one of the folks leading the derivative at FOSDEM and he was asking how useful LP was. So maybe there is a plan B === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado [17:51] I'm really not sure how to asnwer this question or if even is a lp Question : https://answers.launchpad.net/launchpad/+question/188094 [17:55] czajkowski: yea, that's a major 'not applicable' to LP. Might suggest they try ask ubuntu if they've having issues with getting some PHP magic to work with ubuntu versios of php/apache. [17:55] rick_h: thanks [17:55] want to start on a clena page tomorrow so trying to clear up the last few [17:55] bene looking at that trying to work it out [17:56] yea, basically if register globals is on, then things sent in a POST request to the server should be auto variables in his script and it's not working. but that stuff has been so bad practice for so long, it might just not even work in 5.3 I don't know [17:56] * rick_h has php dev flashbacks ...shudder [18:02] I was wondering if people knew the reasons behind the lack of opera support on LP - https://answers.launchpad.net/launchpad/+question/188200 [18:02] manpower [18:02] / womenpower [18:05] lifeless: while I'd agreew ith you, writing that on a questions page isn't really going to be helpful now is it :) [18:05] and the question shall remain open and I dont like open things in my queue :) [18:05] * czajkowski has beaten RT into submission today [18:07] czajkowski: we've got a bug for the opera issue and I hope to take a peek at it, as we'll need it to fix IE8 support as well, so it's not that we're not going to fix it. Just not yet [18:08] ok that seems more of a reasonable answer to give someone [18:08] czajkowski: yea, looking for the bug now, thought I was watching it [18:08] czajkowski: so, opera isn't in our list of target browsers. [18:09] czajkowski: its not in the list because (usually) what works for chromium (which is in the list) works for it, and we don't have the resources to test all browsers; opera isn't in the top-3 by volume browsers out there. [18:10] czajkowski: I think its a fine answer to give, its true :). Also, please don't commit to us fixing opera, because chances are we won't, unless or until someone steps up and maintains opera support as a volunteer. [18:12] nods [18:12] let me forward you the stakeholders thread on this [18:17] lifeless: thanks [18:25] czajkowski: there was also a related discussion, on -dev I think, about how YUI no longer categorise browers the same way [18:25] nods [18:25] thanks for the mail [18:26] right time for some foods and relaxing befor eI switch brain over to locoteams work [18:26] nn [18:26] nn [20:49] flacoste: can you hear me ? [21:07] flacoste: http://bazaar.launchpad.net/~canonical-launchpad-branches/pybars/trunk/view/head:/pybars/_compiler.py === salgado is now known as salgado-afk [21:23] lifeless: ISP woes again? [21:28] ADSL apparently stands for approximately down standard link [21:35] flacoste: http://www.yuiblog.com/blog/2011/07/12/gbs-update/#grades-deprecated is what I was babbling about [21:48] lifeless: yes, i saw, but their main page still explains the three tier grade approach, they just don't specify which browser gets which experience [21:48] because that should be left to each team to dictate [21:48] they just specify which browsers they test [21:48] i think for us, it still makes sense [21:48] since we are one such team that has to decide of the experience users get based on their browser and our resources to support it [21:59] kk [22:11] Hey lifeless. Did you want to talk today? [22:16] mrevell: sometime, but not today ;) [22:16] mrevell: i have to pop out for a bit, and it's already way late for you [22:17] lifeless, Okay, great :) I shall go to bed. [22:17] Night all [22:59] * wgrant needs opinions. [22:59] Bug #933911 is somewhat qa-bad [22:59] <_mup_> Bug #933911: Remove PersonLocation-related code