[00:18] lifeless, sinzui: I was planning to do a deployment this morning. [00:19] But then a certain monstrous completely un-QAable undisable hopeless merge failure buggy criss-cross branch landed. [00:19] We won't be deploying for a few days :/ === wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld (*jtv) | Critical bugs: 202 - 0:[######=_]:256 === wallyworld_ changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 202 - 0:[######=_]:256 [00:37] wgrant: ^^^^^^ which branch out of interest? [01:21] It's tempting to roll it back on principle. [01:21] There are bugs. [01:21] Not fatal if it was flagged. [01:21] But... [01:21] It's not. [01:27] Wasn't it rolled back once already? [01:27] Yes. [01:30] Yay [01:30] It crashes in Firefox 3.5, at least sometimes. [01:32] :/ this is a big UI change. [01:33] No longer obvious how to subscribe, and you can't tell who is directly subscribed to the bug... [01:45] Project parallel-test build #44: STILL FAILING in 1 hr 17 min: https://lpci.wedontsleep.org/job/parallel-test/44/ [01:51] Project windmill-db-devel build #398: STILL FAILING in 1 hr 6 min: https://lpci.wedontsleep.org/job/windmill-db-devel/398/ [01:51] wallyworld_: the bugsubscription one [01:52] the 12000 line one i assume [01:53] wgrant: Can haz link to notify() bug? [01:54] StevenK: There doesn't seem to be one. Check #launchpad from 21:48 yesterday. [02:02] Project db-devel build #642: STILL FAILING in 5 hr 41 min: https://lpci.wedontsleep.org/job/db-devel/642/ [02:05] wgrant: if its buggy, its a rollback [02:27] lifeless: The ordering is apparently arbitrary and the JS crashes when not logged in. [02:27] And it completely changes the UI of a feature we announced a week ago :/ [02:29] crashing when not logged in is a pretty big blocker === Ursula is now known as Ursinha-afk [02:31] It's not a regression from what we have now. [02:31] Ohhh, yes it is, actually. [02:32] Since at least now the direct subscribers are shown. [02:33] wgrant: crashing is a regression [02:33] wgrant: the UI aspect is deliberate AIUI. IMBW [02:37] lifeless: Deliberate, but untested and a break from what we announced :/ [02:50] lifeless: ITMFTFOWTFYW [02:50] huwshimi: ? [02:51] lifeless: It takes me forever to figure what you write [02:51] lifeless: Sentences like this: "lifeless: wgrant: the UI aspect is deliberate AIUI. IMBW" [02:51] huwshimi: sorry ! [02:52] lifeless: haha, it's ok [02:52] lifeless: Most people on Launchpad do it, I just spend half my time opening a browser and googling what everyone is saying [02:52] s/Launchpad/IRC/ [02:52] wgrant: True === wgrant changed the topic of #launchpad-dev to: Performance Tuesday ! | https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256 === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 205 - 0:[######=_]:256 [03:21] Project windmill-devel build #241: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/241/ [03:30] StevenK: Any luck with the notifications? [03:31] wgrant: I've been a little distracted, sorry. Still digging. [03:35] wgrant: Unsigned changes file, since it comes from a buildd, so it adds Changed-By. [03:36] StevenK: Ah. [03:36] StevenK: How did this not happen before? [03:37] I suspect the old code only did that if it wasn't a PPA. [03:37] Probably. [03:37] I also suspect we need more unit tests. [03:41] wgrant: I think the bug is firmly in get_recipients(), I was just going to read the old code [03:47] wgrant: Hm, the bug seems present in the old code too [03:47] StevenK: But I know it didn't happen :/ [03:47] StevenK: Possibly archiveuploader just didn't notify? [03:47] I'll look. [03:47] But I didn't change that bit! [03:48] wgrant: What I mean is _getRecipients() always added Changed-By if packageupload.signing_key is None [03:48] Yeah. [03:51] # If this is not a PPA, we also consider maintainer and changed-by. [03:51] if self.signing_key and not self.isPPA(): [03:51] StevenK: From the original code. [03:52] wgrant: But look above that [03:52] wgrant: debug(packageupload.logger, [03:52] "Changes file is unsigned, adding changer as recipient") [03:52] candidate_recipients.append(changer) [03:53] Fair point. [03:53] Oh. [03:54] sec [03:54] Hm. [03:54] The buildds override Maintainer, but not Changed-By. [03:54] So Changed-By should be the source Changed-By... hm. [03:55] This will all have to be rewritten soon anyway :/ [03:56] 2011-06-17 00:49:35 DEBUG Subject: [PPA stackapplet-dev-stackapplet] stackapplet_1.5+147~maverick1_source.changes rejected [03:56] 2011-06-17 00:49:35 DEBUG Sender: Ubuntu Installer [03:56] 2011-06-17 00:49:35 DEBUG Recipients: Launchpad Beta Testers [03:56] what [03:59] Another rewrite? :-/ [03:59] StevenK: For DDs, yes. [03:59] Yippie, build fixed! [03:59] Project devel build #813: FIXED in 5 hr 24 min: https://lpci.wedontsleep.org/job/devel/813/ [03:59] StevenK: Once we're copying between primary archives. [03:59] StevenK: Otherwise Debian people will probably be notified, which would be Bad. [04:00] And Ubuntu people would definitely be notified about Linaro builds. [04:00] Indeed [04:01] wgrant: So, shall we just remove the else bit that adds Changed-By for unsigned uploads? [04:01] StevenK: No. [04:01] StevenK: We need to work out what has changed. [04:02] Between a refactor a month ago, a rewrite 2 weeks ago, and 2 tweaks. :-/ [04:03] Interesting. Logs start on the 19th, and the first occurence was the 30th. [04:03] It was broken then. [04:04] The 30th of May? That's the rewrite [04:06] So it was. [04:06] So. [04:06] I can't see why it wasn't notifying before :/ [04:06] What do the DEBUG logs show for occurances of this problem prior to the rewrite? [04:06] There are none. [04:07] But they only go back 11 days earlier. [04:07] Bugger. [04:07] That exception is somehow not seen at all before then. [04:07] (which is not unthinkable; it also wasn't seen between the 3rd and the 10th of this month) [04:07] Yes, they don't happen very often. [04:09] https://lists.ubuntu.com/archives/ubuntu-reviews/2010-April/000281.html [04:09] That's a log from a year ago. [04:09] There are more recent ones, but that came up quickly in Google. [04:10] That doesn't trip the Changes file is unsigned, adding changer as recipien [04:10] Sigh. But you get the idea. [04:11] In fact, that doesn't send a notification at all. [04:12] I wonder... [04:13] It's possible that this is unrelated. [04:13] process-upload was originally run by b-m, with -M [04:13] So no mail was sent. [04:13] But then late last year that changed... [04:14] Could this have been happening for 6 months? Odd that we only heard about it now. [04:14] Indeed. [04:16] If only we kept logs for a while :/ [04:16] What we keep is only 13MB :( [04:16] I don't understand why we don't keep pretty much all our logs for just about ever :( [04:41] hi wallyworld_, I'm back. Any reviews to mentor? [04:43] jtv: hi. none yet [04:43] ok [04:45] wgrant: is there something similar to the qa-tagging going on with bug statuses? I notice you've marked a few bugs Fix Released before all their branches had landed. [05:45] wgrant: Since we don't have logs, we have no way to be certain -- but the work that divorced process-upload and b-m could have caused this? So I'm off the hook? [05:46] StevenK: I think so :/ [05:46] jtv: Those didn't look like multi-branch bugs, and they weren't landed with [incr], so I didn't notice, sorry. [06:00] jtv: i'm off to pick up the kid from school. still no reviews. will check again later [06:09] ok wallyworld_ [06:10] wgrant: I just knew there was something I wasn't thinking of… incr. It's getting altogether a bit finicky for me, this tagged-double-bookkeeping. [06:11] Indeed. [06:11] OMG my branch made it in. Starting at 07:30 pays off. [06:11] I think that's about 8K of today's 9K lines landed. :) [06:12] so, I have the sort bug still pending. where is a good place to add a util function? [06:14] Project parallel-test build #45: STILL FAILING in 1 hr 16 min: https://lpci.wedontsleep.org/job/parallel-test/45/ [06:19] Project windmill-db-devel build #399: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-db-devel/399/ === _mup__ is now known as _mup_ [06:42] Does anyone know lib/lp/{bugs,code,soyuz}/notifications/ is used for? All three of them contain a zero-length __init__.py [07:07] stub: hi [07:07] Yo [07:08] stub: I did that lock experiment; the blocked alter table blocked out new readers. [07:08] session one: begin; set transaction isolation level serializable; select id from bug; [07:08] session two: being; alter table bug add column foo int; [07:08] session three: begin; set transaction isolation level serializable; select id from bug; [07:09] two and three both block [07:09] doing a rollbackl in session one - andtwo and three unblock [07:09] well, once two is committed. [07:11] ok. There might be something more complex going on (transactions working with related tables causing the ALTER TABLE to block). [07:11] Or not [07:12] stub: staging has displayed one of these typical 'fail to apply' all day [07:12] stub: hloeung and I have been gathering data and poking at it [07:12] But so far, I've never had success doing anything that needed to get a lock on Person on production. 'ALTER TABLE BugSummary ADD CONSTRAINT bugsummary__viewed_by__fk FOREIGN KEY (viewed_by) REFERENCES Person' for instance. [07:12] stub: the appservers and scripts were running [07:13] Has the lag check rung any alarms? [07:13] yeah [07:14] So the script is either blocked for some reason (eg. a DB patch trying to grab a lock on the Person table ;) ), or the daemons have fallen over, or replication is broken [07:14] Good the lag check is alerting us [07:14] https://pastebin.canonical.com/48634/ [07:14] pending locks [07:15] https://pastebin.canonical.com/48634/ [07:15] bah [07:15] https://pastebin.canonical.com/48632/ [07:16] It seems to be currently running my bugsummary update. This is the first run on a production like system... the query might just be way slow. [07:16] stub: 5 hours ? [07:16] stub: 2208-00-0.sql is current, or is that the amalgamation of 2207? Happy to wait until you're done digging. [07:16] stub: if its the one I prepared, it runs into a temp table in a few minutes [07:17] There is no 2207 in our tree. launchpad-2208-00-0.sql is the current baseline. [07:19] stub: Right, but 2207 used to exist [07:19] lifeless: I didn't notice or decode your one (need to feed that stuff through a formatter). [07:19] yes [07:19] Project windmill-devel build #242: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-devel/242/ [07:19] stub: IE, where did 2207-* move to? 2208-00-0, or somewhere else? [07:19] It was either moved to 2208 or deleted [07:20] annotate on 2208 should give you the revno [07:20] I'd like to drop a view from 2208-00-0 [07:20] (Not yet, mind) [07:20] That needs a DB patch. You don't modify the baseline - it is a snapshot of production at a point in time. [07:21] stub: Okay. So for an ec2 run (for instance), the view will get created and then removed? [07:21] If you hack the baseline, it will never get applied to production [07:21] Yes [07:21] stub: Okay, thanks. [07:21] Until we rollup the next baseline. [07:22] (probably next month - we are in the 70's already) [07:22] Bring on 2209 [07:23] Any particular reason? Applying those patches takes about a second. [07:23] No, just forward progress :-) [07:23] Going back to 1 is backwards progress ;) [07:23] Haha === almaisan-away is now known as al-maisan [07:25] lifeless: So it is my bugsummary population query running. Possibly an earlier one was slow - I won't get timings until (if) it completes. [07:26] stub: no, it was that one [07:26] just digging up the query I had [07:26] lifeless: We can either wait or kill it - things should rollback fine at the current point in time. [07:26] stub: I'd like to debug whats up with it [07:26] Its chewing 100% of a core. [07:27] the EXISTS always is dangerous to performance. [07:27] Your version had that? [07:27] Anyway - we can fix it directly in the existing patch since it hasn't landed yet... [07:27] yeah [07:28] https://bugs.launchpad.net/launchpad/+bug/793848/comments/4 [07:28] <_mup_> Bug #793848: Distribution:+bugtarget-portlet-bugfilters-stats timeouts < https://launchpad.net/bugs/793848 > [07:28] had my query [07:28] But the cpu utilization means it isn't blocked. [07:28] yah [07:28] Maybe something stoopid like a missing join condition [07:28] so we need to diff the two queries [07:29] stub: it was blocked for a while [07:29] stub: was waiting for a lock on the sl_event [07:29] Some other process inserting data that needed to be replicated. [07:32] stub: it looked like a slon had started up badly and was deadlocking itself [07:33] lifeless: So how long has this patch been running do you think? [07:34] stub: https://pastebin.canonical.com/48632/ [07:34] stub: submitted 11 hours or so ago [07:35] postgres 32012 99.9 2.2 2280364 726944 ? Rs Jun16 567:55 \_ postgres: slony lpmain_staging [local] INSERT [07:36] lifeless: looking at the slony stuff being executed, it was blocked then on the event lock [07:37] stub: so, tha db patch has used 567 cpu minutes [07:37] One slony process had been waiting 4 minutes [07:37] ok, and nothing else would have had that process before hand. [07:37] right [07:38] (won't be the case after we install pgbouncer) [07:39] stub: slonik will be direct won't it ? [07:39] True [07:45] lifeless: So yeah, the queries are different where it matters. I created a new CTE with clause for bugs_fixed_upstream, and that obviously sucks. [07:45] bbs, grabbing dinner [07:48] Yer, so the CTE will be executed once for the query (which is why we normally use them), then every insert is doing a full scan on that set to see if the EXISTS clause works. Your version with it inline will be doing an index lookup. [07:49] Yippie, build fixed! [07:49] Project db-devel build #643: FIXED in 5 hr 47 min: https://lpci.wedontsleep.org/job/db-devel/643/ [08:03] stub: yah [08:11] stub: you probably need to merge db-devel and push [08:11] stub: the diff looks horrible [08:14] Whoops... lp-land already away [08:14] ah well [08:14] we'll see what happens ;) [08:15] Pumped db-devel... must have had an older version? [08:17] something :) [08:18] It landed (!) [08:20] I think it is wierd because I ran bzr pump --from db-devel in my second pipe, so it had more db-devel than my first pipe [08:22] Landed diff is sane anyway [08:22] So how to we get this on staging? [08:22] 4 hours for it to get through BB [08:22] I can kill the existing job which should fail safely [08:23] its a bit hacky [08:23] but I'd kill the existing slonik run [08:23] Given it has taken so long so far to apply the patch on the master, I'll assume we can't be arsed waiting for it to complete and run a second time on the slave. [08:23] yup [08:23] drio your new file in [08:23] and do the deploy thing yourself [08:24] select pg_cancel_backend(32012); [08:25] That should have killed the existing run [08:26] losaping: Can we kick of a staging code update without the code update? [08:26] stub: what do you mean ? [08:27] After modifying the live tree, we don't want to have the scripts blat it [08:28] stub: so, disable the cron job for a few minutes? [08:28] stub: oh, I see where you are going; ignore me [08:29] I can try running upgrade.py manually live. Not sure exactly what happens :) [08:46] stub: as long as we avoid staging-restore auto running the old patch I'm happy ;) [08:52] good morning [08:53] I can't see how to override the code update. [08:54] apart from directly editing the script [09:02] +1? [09:22] stub: think you've got the patch right now ;> [09:22] I'm sure there are other exciting ways I can screw up [09:30] Was Not A Challenge! :) [09:35] Project windmill-db-devel build #400: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-db-devel/400/ [10:00] stub: did you benchmark 2208.75.0 on staging yet? [10:01] not the new version, no [10:01] running for 24h on dogfood so far :/ === adeuring changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv), adeuring | Critical bugs: 205 - 0:[######=_]:256 [10:02] wgrant: Hi, I know you're in fire-fighting mode right now ... but if you have time after that (I know it's going to be late for you), I'll be happy if you could take a look at my (never-ending) multi parent thing. [10:02] wgrant: I you don't have time, that's really fine and we could talk about it on monday. [10:02] bigjools: yes, kill it [10:03] lifeless: ? [10:03] you are reverting it? [10:04] no, stub landed a fix [10:04] its in db-devel [10:04] grar [10:07] rvba: I'll try to get to it tonight, but no promises :/ [10:07] wgrant: that's fine, thanks a lot. [10:10] stub: whats the state of staging, given you're going ? [10:14] Needs a losa [10:15] ping for one in -ops ? ;) [10:15] I've patched the tree but not sure how to continue. I could improvise... but I'd rather someone who knows the correct processes does it [10:15] losaping here from ages ago... [10:15] they don't watch -dev [10:15] tom requested that we only do such pings in -ops [10:16] Project windmill-db-devel build #401: STILL FAILING in 40 min: https://lpci.wedontsleep.org/job/windmill-db-devel/401/ [10:26] good morning [10:26] stub: how do I tell the difference between the old and new patch? I just diffed DF's version against the one in db-devel and it's the same still. [10:26] jml: morning [10:31] bigjools: the fixedtasks CTE should be gone [10:31] line 78 of the patch [10:32] lifeless: 78 for me says "OR (" [10:32] bigjools: did you pull db-stable or db-devel [10:32] devel [10:32] *db-devel* [10:32] yes db-devel [10:32] what revno do you have? [10:32] it's hardly likely to be in devel is it :) [10:33] 10679 [10:33] is the one I am using now, but it says no diff to 10685 [10:34] danilos: I played around with the new subscription listing thing this morning, and noticed a couple of things. The JS crashes for anonymous users, the subscriptions don't seem to be sorted, and I can't work out who is directly subscribed. [10:34] bigjools: I see a huge diff [10:34] wgrant, well, everyone except the "Maybe notified" section are direct subscribers [10:34] danilos: Oh. [10:35] That's less than obvious. [10:35] wgrant, some of them might choose not to receive some emails [10:35] I assumed they were just people who had selective structural subscriptions :( [10:35] wgrant, you can have selective direct subscriptions as well [10:35] True. [10:36] So "Always notified" is sort of a misnomer? [10:36] wgrant, no, always notified is those who are always notified (i.e. "receive all" [10:36] wgrant, see eg. https://bugs.qastaging.launchpad.net/ubuntu/+source/network-manager/+bug/152754 for example of other sections [10:36] <_mup_> Bug #152754: NetworkManager fails and crashes after resume from suspend < https://launchpad.net/bugs/152754 > [10:37] wgrant, (you can see my different subscriptions on different levels) [10:37] danilos: aaaaaa [10:37] So "Maybe notified" is all the structsubs? I seeeee. [10:37] Hmm. [10:37] wgrant, yeah, "maybe" is all struct subs, all dupe subs [10:38] wgrant, for your own subscriptions, you are supposed to use "edit bug mail" link instead [10:38] lifeless: weird, a bzr ancestor diff shows nothing, but bzr missing shows 6 revisions [10:38] danilos: I've never liked that link... shouldn't it be "Edit subscriptions" or something that doesn't sound like it's editing emails? :/ [10:38] wgrant, basically, this is something that's hard (and potentially expensive) to display in succint form with enough details (while staying correct) [10:38] But yeah. [10:39] bigjools: uhm an ancestor diff should show nothing [10:39] bigjools: because your current rev is the common ancestor [10:39] I understand it's hard to calculate, yes. But it's not obvious what it's actually showing. [10:39] gah, roight [10:39] lifeless: I am stupid [10:39] :P [10:39] wgrant, yeah, I think the attempt was to clarify that this is about controlling the email you get from that particular bug [10:39] bigjools: didn't you just implement a vcs layer for distros? [10:39] wgrant, and I think we just copied whatever was there already on the pillar pages which was "Edit bug mail" [10:40] I'd like to think not :) [10:40] wgrant, I do see your point, but maybe you are thinking too much in terms of implementation, rather than trusting what the text says? [10:40] Project parallel-test build #46: STILL FAILING in 1 hr 13 min: https://lpci.wedontsleep.org/job/parallel-test/46/ [10:40] wgrant, "Notified of all changes" are, indeed, people/teams who get all the bug mail for this bug [10:40] danilos: Links like that have always talked about subscriptions before, or "Subscribe to bug mail", not "Edit bug mail" [10:41] Since editing bug mail doesn't really make sense. [10:41] danilos: wgrant: was saying earlier that the js crashes for anonymous users [10:41] wgrant, hey, I am not going to argue over that, I don't like "Edit bug mail" anymore than you do [10:41] Heh [10:41] k [10:41] lifeless, yeah, I'm going to look into it [10:41] But yes, what lifeless said. [10:41] wgrant, ack [10:41] danilos: if it needs fixing we should roll the branch back [10:42] ... and land the unrelated pieces separately? :) [10:42] danilos: we have 30! revisions to deploy already, and more will queue up over the weekend [10:42] wgrant: orthogonal discussion [10:42] lifeless: Not quite... I rolled out up to that rev this morning. [10:42] wgrant: danilos: grah sorry; stale info [10:42] There's only 8 in the queue ATM [10:42] lifeless, if it needs fixing, we'll fix it in the next few hours [10:43] lifeless, if subscribers list doesn't render for anonymous users, it's not a big deal since none of the JS works for them anyway [10:43] lifeless, iow, that should not block the rollout, but it should be fixed asap [10:44] Why shouldn't it block the rollout? [10:44] lifeless, I was also hoping to get more QA on this and try to deploy it on Monday instead (and any such problems should be fixed by then) [10:45] wgrant, why should it? I don't consider it a critical feature, and I value the time it takes to get something landed much more than this particular problem (if it's not as severe, like I said) [10:45] danilos: Some browser react to JS errors with violent popups. [10:45] And we treat JS errors as OOPSes now. [10:46] I think we'd consider it a blocker if the most commonly accessed page on LP OOPSed for anonymous users. [10:46] wgrant, right, and fixing the problem is probably going to take 30 mins as soon as I get to it, whereas just doing a revert is going to take me 10-20 mins [10:46] I don't place much value on rushing a 12000 line branch through, TBH. [10:46] It deserves a lot of caution. [10:46] wgrant, I agree, and I want to exercise some [10:47] wgrant, but I want to be realistic as well [11:04] stub: Do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5? [11:05] allenap: Nope. Last minute scramble for a plane. [11:05] stub: Okay, have a good trip :) [11:06] lifeless: I know it's silly o'clock for you, but do you have some time to help analyse the slow queries in https://lp-oops.canonical.com/oops.py/?oopsid=1993DY5? [11:07] allenap: lets get an analyse for the 3 second one off of prod, after gnuoy finishes fixing staging [11:10] Okay, cool. I have it ready on devpad. [11:44] lifeless: do you know how much quicker that new db patch is? 30 minutes and counting on DF so far. [11:45] 157 seconds when it ran on a temp table [11:45] are you sure you got the new one ? [11:45] yes [11:45] :) [11:45] allenap: ok so [11:46] 6.6 seconds [11:46] Ouch. [11:46] see the actual time [11:46] hash join on spr to dsd [11:46] 351K rows [11:47] adeuring, wallyworld_: hi, I need a review of a short branch (151 line), any takers? :) [11:47] the stats suggested to postgresql it would see 2.2K rows [11:47] danilos: I'll look [11:47] could be a stats issue [11:47] adeuring, thanks [11:47] lifeless: Can we get stats recalculated online? [11:48] yeah [11:48] analyze tablename; [11:48] allenap: how many rows are you expecting to return ? [11:49] lifeless: Around 1:1 with DistroSeriesDifference rows. [11:50] So around the same as the big IN (5, 7, ...) clause. === al-maisan is now known as almaisan-away [11:50] right, so the plan is problematic, or the model is problematic or the query is [11:50] Hehe. [11:51] Project windmill-devel build #243: STILL FAILING in 1 hr 5 min: https://lpci.wedontsleep.org/job/windmill-devel/243/ [11:51] lifeless: Is an analyze cheap enough that we could try that first? DistroSeriesDifference until recently was empty for example, that might be a candidate for refreshing stats? [11:52] Although that doesn't appear in the bad part of the plan :-/ [11:52] yes [12:00] bigjools: in other news, DF sucks. [12:00] lifeless: tell me something I don't know [12:00]