[00:00] :-) [00:00] Its beautiful! [00:00] Also, there is this space "&commit= Send Upstream" [00:01] it should not exist I think, so the commit button would change to Send upstream? [00:01] ok, tested. No. [00:02] Anyway its the most awesome thing I've seen. If LP does this by default, we'd all hug you :) [00:02] I think that's fine, that value isn't actually used by bugzilla [00:02] Yeah, I just noticed [00:04] well, it may be a while before LP does this by default, but I'll be working on it. meanwhile the cgi is going to be available for folks [00:04] I'll be interested in seeing how much use it gets [00:06] I'd also be interested in hearing about tangible time savings. I calculated after spending one day forwarding bugs without this tool, vs. one day forwarding bugs with it, I was able to send 3 times as many bugs upstream in a given period of time [00:07] the main constraint for me in achieving an even better ratio was having to manually copy over the attachments [00:09] nigelb, thanks for finding that bug. embarrassing! [00:09] clearly I need to write some tests [00:10] heh [00:10] another advantage when you have a tool like this, people don't go, "Oh, I'll forward it later" [00:10] nigelb, did you install the greasemonkey script? [00:10] they'll just. "hey, it'll take only 10 minutes, why not do it" [00:10] yep [00:11] Nope, I did it manually. [00:11] (this is work computer, can't mess with it :D) [00:12] mm, if/when you can install that gm script, you'll find it REALLY makes it fun and easy to upstream bugs [00:12] Oh! [00:14] how tough would it be to make it available for gnome bugzilla too? [00:14] it should work with gnome bugzilla already [00:14] I only tested on a couple bugs though so am not 100% sure [00:14] gnome has more people forwarding since the apps are mostly in user space [00:14] but I put the support for GNOME in last week. I'd love to get verification that it works [00:14] yep [00:15] it should work for gnome packages that have exactly the same name in launchpad and in gnome bugzilla [00:16] which afaik should cover most packages [00:16] where in the lp branch is the code? [00:16] it's in the scripts directory [00:18] its a very nifty script :) [00:19] thanks, yeah I've gotten a lot of good out of it, and looking forward to it being handy for others [00:19] nigelb, let me know if you test it on a gnome bug [00:22] I will as soon as I can find the time. gotta start working :) [00:22] nigelb, cheers :-) === Ursinha-nom is now known as Ursinha [03:18] lifeless, missed your request -- "partial" oops summaries, as we call, are regenerated every hour === Ursinha is now known as Ursinha-afk [04:26] Ursinha-afk: thanks [04:41] spm: can you time the statement in https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1662O772 ? - the 17 second one, on a slave, and on prod, please? [04:41] sure [04:42] if its slow, grab an explain analyze too :) [04:42] wow. that's... inconsistent. [04:43] enter query into cmd line; hit enter; runs, completes. uparrow+enter; repeat. [04:43] . [04:43] . [04:43] . [04:43] 1.7s, 13.1s, 0.6s, 1.0 [04:44] we might want to fix this [04:44] wtf (where) did 13.1 come from!!! [04:44] was that on a slave or main ? [04:44] main [04:45] what makes that 13.1s even more interesting, i'd have thought the query was still within postgres caching. ie it should be zomg fast. [04:45] so [04:45] guesses: load, index fail, index lock contention [04:45] cache fail... [04:45] if we're under that much pressure, sure. [04:46] we're seeing the oops 180 times a week [04:46] load, unlikely - load average: 8.31, 7.91, 8.14 <== which is pretty low for wildc [04:46] * lifeless mutters about perspective [04:46] note the 'unlikely' qualifier. :-) [04:47] how many cores does it have? [04:47] when you'd had dual 650Mhz CPU proxy firewalls on a 'normal' load of 50-80; still working at 450... [04:47] 16 [04:48] ok, so thats still 8 db threads or similar not executing for a full second [04:48] its not actually great [04:48] later we will have to go into the performance knee for wildc [04:49] right now, we already know that backsup cause an oops spike :P [04:49] ? I thought load was a (rough) measure at the end of a 'tick' of what's executing right now; regardless of how much of the tick they've had? imbw.... [04:49] ie it avergaes out to a vague approximation. [04:49] its blocked ready-to-run processes [04:50] http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages - not telling you to suck eggs, but I've quickly scanned it and its a clear explanation [04:50] heh, I suspect we're using different words to describe the same thing. :-) [04:50] ta [04:52] ok anyhow [04:52] one thread at a time [04:52] :-) [04:52] can you see if it does that slow pattern on a slave [04:54] huh, I was almost going to say o it doesn't, but persistence paid off. most (~ 10 runs ish) were sub 0.3s, 1 x 1.0, 1x 4.5. [04:54] s/ o / no / [04:54] ok [04:54] 8 cores, load average: 1.01, 1.18, 1.15 [04:54] can you get an explain analyse of it slow in both cases ? [04:55] Oooer. it changed! [04:57] wow. I didn't notice. it changed twice. https://pastebin.canonical.com/34873/ [04:57] actually... i think it was different on every run. repasting the entire test [04:58] https://pastebin.canonical.com/34874/ [04:58] *boom* [05:00] yeah. something is *really* screwy, based on my DB understandings if the explain plan can be changed so dramatically like that. I thought these things were cached for high seconds to low minutes for precisely this sort of thing. [05:00] https://pastebin.canonical.com/34875/ to complete the set, wildcherry showed similar behaviour [05:01] ah [05:01] actually I quite distinctly do recall faffing around with Oracle... 10g istr and forcibly getting it to regen plans 'offline' as it were to maximise the speed of certain queries [05:02] its a damned view [05:02] omfg [05:02] excuse me [05:02] no perfectly understandable :-) [05:02] \d publishedpackage :P [05:02] It is pretty unthinkable that'd I've used oracle and actually like it. ;-) [05:03] * spm takes a few deep breaths [05:04] Oh god, publishedpackage? [05:04] Kill it. [05:04] oh not too bad. only 12 joins. [05:04] The *cache table* is being slow? [05:04] in the fast one is its a nested loop [05:04] pg doesn't hve materialized views does it? [05:04] unknown to me [05:04] don't think so [05:05] You can emulate them using triggers, but it doesn't do them itself, no [05:05] so the fast case loops on binarypackagename_name_key on binarypackagename [05:06] the slow case Seq Scan on buildfarmjob (cost=0.00..54641.45 rows=1799245 width=12) (actual time=0.019..1470.653 rows=1799572 loops=1) [05:06] and it gets worse from there [05:06] That table has only been around for a couple of months. [05:06] So the query probably hasn't been looked at in this form before. [05:07] I need to review this, I smell OOD rather than table layout optimisation [05:07] I saw a similar thing last night [05:08] spm: so I'm guessing slight statistics variation -> boom [05:09] lifeless: I guess, but... I'm kinda left wondering why plans aren't locked in stone more; if they even can be. [05:11] spm: do I have sql access to staging, to poke at the tables a bit ? [05:11] perhaps comparing oranges with mandarin's; in oracle land - as I recall it; you'd have a generic query plan with placeholders for the values. there's a specific name, but time and memory fades. the idea being that you just replace the bits you care about; the plan itself is indentical; so you win big on lots of repititions by not doing the analyze phase(s) [05:11] lifeless: perhaps r/o? if not, I can create now and retroactively ask francis for you to have same. [05:12] spm: whatever bjorn had? [05:12] I have 99.99999% no doubt he'll be fine with that. [05:12] everyone except losas and stub has r/o [05:12] I'm sure that will be fine [05:12] rephrase, everyone *with access* has r/o [05:12] if its ever not, well, efuture [05:44] wgrant: so why do you say to delete hte view? [05:45] lifeless: I was wrong. I was thinking it was one of the cache tables. [05:46] But it's not; it's just a view. [05:53] 10 million rows in bpph [05:53] I can has delete? [05:53] You can has indices. [05:53] meh [05:53] I has indices. [05:53] I wants delete. [05:54] also fti needs some serious love [05:54] Deleting history makes me sad. [05:54] What's the FTI on? [05:55] wgrant: keeping stuff isn't free [05:55] and even logn perf hurts after a while [05:55] lifeless: Says he who developed a DVCS which preserves all history :P [05:55] its all about use cases [05:56] actually, I want a good history edit mechanism - I wrote that up a year back or so [06:00] * wgrant still needs someone to land a branch. [06:19] spm: are you able to do iostat/cpu on staging for a query for me? [06:20] wgrant: whats the mp [06:22] lifeless: One of the ones you reviewed a couple of days ago. [06:22] * wgrant hunts. [06:22] https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667 [06:22] oh, I might need a newer tree [06:22] we'll see [06:23] herh [06:23] lp.registry.windmill.tests.test_person_picker.TesPersonPickerWidget.test_person_picker_widget is failing in my branch [06:24] nice test class name! [06:26] wgrant: its mid kickoff [06:26] lifeless: Thanks. [06:26] losa ping [06:35] how do you debug a failing windmill test? [06:35] WITH A SHOTGUN [06:39] wgrant: its in flight [06:39] lifeless: Even better. [06:40] It *is* the right branch? [06:40] Last night it wasn't. [06:40] i just cp'd your link [06:41] lifeless: it seems "run it until it passes" works :( [06:42] mwhudson: My policy is "run until it passes. if it still doesn't pass, ignore it, because I didn't break it" [06:47] lifeless: sure, on staging DB server as in? [06:52] yes [06:53] the qyry is [06:53] \timing [06:53] SELECT COUNT(CASE WHEN Bug.fti @@ ftq('sphinx') THEN TRUE ELSE null END) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1; [06:56] lifeless: I forgot to time() it; but ~ 10ish seconds. https://pastebin.canonical.com/34879/ for the vmstat [06:57] \timing is your friend :) [06:57] aye... :-/ [06:58] 2nd run - just shy of 9 secs [07:00] ok [07:00] a fucktonne of IO [07:00] spm: can you check whether lucene, solr & pylucene are available in the DC, in the event we were to want to use them. [07:01] are they debug type programs? servers? [07:01] ah the first is the search engine? [07:03] yes [07:03] one stack [07:03] lifeless: looks like, not really. some possibly defaultish libraries; but I'd suggest backporting would be needed. [07:04] rmadison was hopefyul [07:07] i know the divmod folks became extremely angry with lucene [07:07] several years ago now, mind [07:11] spm: it appears to be in universe, to me [07:13] lifeless: that would be my lack of experience with dealing with packages then [07:13] hahahha [07:13] or possibly we don't enable universe on the servers... [07:13] spm: whats our policy on stuff in univers [07:13] e [07:14] actually I can see the solr stuff, I sit corrected. [07:14] heh, lalala me. universe is fine. [07:15] spm: how hard am I hammering the server ? [07:16] 100% on one cpu only [07:16] \o/ [07:16] select * from stat('select fti from Bug') order by ndoc desc, nentry desc,word limit 10; [07:16] how many cpus are there ? [07:16] 8 cores [07:17] only 32g of ram, so i'd expect more io than prod [07:17] whee that local run is evil you've made me do. also 100% [07:18] 0.0% wait state tho [07:39] spm: I'd like to try reindexing bug.fti on staging. [07:39] spm: how do we go about this? [07:40] depends on how intrusive a reindex you mean? if you just mean 'reindex table blah', that's pretty easy. [07:40] spm: or should I do that on dogfood [07:40] spm: we need to drop the bug_fti index and make a new gin one [07:40] something like [07:40] I'd suggest dogfood in the first instance to get an idea of timing [07:40] lifeless: when did test_suite() stop being necessary in launchpad tests? [07:40] (also yay!) [07:41] mwhudson: I don't know, but jtv tested and current zope.testing doesn't seem to need it. [07:41] lifeless: it'd also be my personal recommendation to run any reindexing past stub [07:41] spm: how do I get access to dogfood ? [07:41] maybe zope.testing grew a new feature then [07:41] lifeless: if you don't have already? via RT [07:41] spm: I've spoken to stub about the fti stuff; its basically untuned - he expressed frustration at not having had time to just sit down and zen it === almaisan-away is now known as al-maisan [07:42] heh [07:42] mwhudson: hell, it may be needed still - remove it and check :) [07:42] lifeless: i did [07:42] * spm can picture stub doing the OOOOOmmmmm thing before a DB layout diagram [07:43] spm: dogfood is *completely* useless for timing. [07:43] You know what sort of hardware it's on, right? [07:44] so the thing is [07:44] we have a fast-insert slow-query fti index [07:44] wgrant: I'm think of more an 'minutes vs hours vs days' idea [07:44] there is an alternative index [07:44] not 17.65 minutes. [07:44] the gin index [07:44] spm: dogfood cannot always give a reasonable answer over those three. [07:44] anyhow this isn't urgent [07:44] haha [07:44] I'm just starting to qualify simple-complex answers in this space [07:44] Recall that a DB restore took two weeks a few months ago. [07:45] And the primary publisher takes several hours. [07:45] lifeless: this talk of gin indexes rings faint bells, i think we already use them for something... [07:45] or tried [07:46] still if you talked to stub you're likely much more up to date than me :) === al-maisan is now known as almaisan-away [08:01] mwhudson: yes :) [08:11] lifeless: Are Forbbiddens logged? [08:11] /builders is 403ing sometimes. [08:11] should be [08:11] I don't know if they raise oopses [08:12] they aren't logged as oopses when you're logged in, i think [08:12] mwhudson: So we can't debug them? [08:13] Awesome. [08:13] wgrant: thats not entirely true [08:13] wgrant: i may have been lying there [08:13] lifeless, jam and i looked at the memcached graphs last night [08:13] wgrant: i think bigjools knows the problem though [08:13] as i suppoe you already have before [08:13] wgrant: it's private builds, unsurprisingly [08:13] they seem clearly size-limited [08:13] mwhudson: He suspects private teams. [08:13] wgrant: however, I encourage you, in the theme of improvign things, to add trace logs for this [08:13] and hitting only about 20% of the time [08:14] yeah [08:14] mwhudson: But I can't see any private builds, so I shouldn't see any private teams. [08:14] I don't really have memcache on my radar atm [08:14] so caching more things is likely to make the cache perform worse :) [08:14] its a tool we're not ready for [08:14] It's already overused. [08:14] Caching things that should not be cached. [08:14] i know [08:14] lpmain_staging=> SELECT COUNT(CASE WHEN Bug.fti @@ ftq('sphinx') THEN TRUE ELSE null END) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1; [08:14] count [08:14] i was thinking of letting it be refreshed from the browser by cache-control [08:14] ------- [08:14] 40 [08:14] (1 row) [08:14] Time: 7659.528 ms [08:15] lpmain_staging=> SELECT COUNT(*) from Bug, bugtask where bug.fti @@ ftq('sphinx') and BugTask.bug = Bug.id AND BugTask.distribution = 1; [08:15] count [08:15] i now feel we should add a switch to jut turn it off, and see if the oops rate changes [08:15] ------- [08:15] 40 [08:15] (1 row) [08:15] Time: 526.478 ms [08:15] we can just turn it off [08:15] and no, not pastbinning that :P [08:15] *thats* easy [08:15] poolie: There is lots of stuff that should be cached. [08:15] poolie: But it's not the stuff that's cached at the moment. [08:16] lifeless: those times are, um, different [08:16] lifeless: What's the difference in the plan? [08:16] lifeless, so tbf there are a lot of cache effects on the db server [08:16] wgrant: give you a single guess [08:16] running two queries one after the other and seeing the second is faster does not necessarily prove anything [08:17] (in my limited experience) [08:17] i agree the second looks more sensible [08:17] of course [08:17] I've run 10 of each [08:17] running 4 and having the 4th run an order of magnitude slower tho.... [08:17] but thats a lot more noisy ;) [08:17] * mwhudson gets some 503s and 504s from the apache frontend :( [08:17] ah :) and it's consistent? fairy nuf [08:17] sometimes the slow one is worse [08:18] https://pastebin.canonical.com/34874/ <== for mindblowing enjoyment [08:18] ok, so edge deployment just failed to migrate smoothly ? [08:18] mwhudson: please file a bug [08:18] it's working now [08:18] edge was just updating [08:18] yes [08:18] ah [08:18] this is a bug [08:18] we should fix it, we're going to be doing this *much* more often [08:19] mwhudson: please file it ;) [08:19] lifeless: ok [08:19] * mwhudson waits to see if the bug search times out... [08:20] short of automatically adjusting the 'load balancing' away from each server and waiting for all existing connections to same to timeout, there will be glitches for some folks? [08:20] spm: thats exactly what we should be doing [08:20] Hm. Is there no way to see all of a project's branches? [08:20] code.lp.net/project [08:20] No. [08:21] It's batched. [08:21] But without a navigator. [08:21] So it only shows the first n. [08:21] lifeless: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/608065 [08:21] <_mup_> Bug #608065: get occasional 503s and 504s from apache frontend when edge is updating [08:21] _huh_ [08:21] Ah! [08:21] +branches is batched and has a navigator. [08:21] But isn't linked from anywhere. [08:21] ouch [08:22] mwhudson: Do you know why this is so? [08:22] * mwhudson afks [08:22] wgrant: no [08:22] wgrant, use an api [08:22] :/ [08:23] poolie: There's UI. It's just got no links to it. [08:23] wgrant: you may find https://bugs.edge.launchpad.net/malone/+bug/607776 interesting [08:23] <_mup_> Bug #607776: +filebug-show-similar FTI query timeouts [08:23] then how does it smell? :) [08:24] 'Seq Scan on bugtask' [08:24] WTF [08:24] yes [08:24] case statement fail [08:25] lifeless, why would you even write such a query? [08:25] ORM [08:25] or confusion [08:25] is there some innocuous stortm thing that produces it? [08:25] I don't know; the question is how long it will take to find and fix them all [08:25] * poolie wonders if storm should sometimes generate Warnings [08:26] lifeless: Well, if we take the timeout down to 1s, they should all become fairly clear :P [08:26] wgrant: I want to get the timeout to 5s [08:26] I really love that you are actually concerned about speed, BTW. [08:26] 1 would be ok if we generally completed in 0.1s [08:26] It is something that has been missing from LP for... well... 5 years now. [08:27] mmm [08:27] All I'm offering is a detailed view [08:27] istm storm could also warn about 1=2 [08:27] not sure if it's worth it [08:27] the whole team has cared and been fixing stuff [08:27] its important not to forget that [08:28] yeah it's really inspiring how much the atmosphere was different last week to previously [08:28] True. [08:28] if I am lucky enough to have some experience solving performance issues and dealing with interlocking pessimisations - its because I've created my fair share! [08:28] Haha. [08:30] damn. that's an admission and a half. /me takes a copy for later blackmail potential ;-) [08:31] lifeless: Are you doing DB reviews now in Björn's place? [08:31] yes [08:32] two votes, stub & I [08:32] we've agreed that with one vote it can progress to the tree [08:32] two votes for deployment [08:32] Ah, handy. [08:36] rockstar: Hm, what's particularly difficult about fixing the factory? I saw it broke quite a few tests (which was why i didn't fix it then and there), but nothing that looked terribly hard to fix. [08:36] mwhudson: I'm not going to get to your review - sorry, I'd love to:) - I'll try to drop a couple of thoughts in it today though. [08:36] wgrant, most of our tests are wrong... [08:37] Hah. [08:37] Awesome. [08:37] wgrant, so the tests that aren't breaking are the worst. [08:37] wgrant, it's not "hard" per ce, just tedious, and probably not the best use of my time currently. [08:37] Yep. [08:39] wgrant, there are all sorts of hidden constraints inside soyuz that aren't really documented that I didn't know, and that we're not really exercising in tests anywhere. [08:40] rockstar: lalala not part of Soyuz any more lalala [08:40] wgrant: you're not? [08:40] lifeless: The code isn't. [08:40] soyuz.add(wgrant) [08:40] :P [08:40] lp.buildmaster's owned by everyone now :P [08:40] wgrant, for instance, the build farm gets DOSed if you submit a build that isn't against a distroseries that isn't part of the ubuntu distro. [08:40] wgrant: all code is owned by everyone anyway, in principle. [08:41] I'd like to see more application of that principle :) [08:41] rockstar: Hm, because they have no chroots? [08:41] Or is there something else? [08:41] bad error handling logic [08:41] AIUI [08:42] lifeless: fairy nuff [08:42] Well, of course, it's ex-Soyuz code. [08:42] lifeless, probably a little bit of everything. [08:43] whats the easiest way to join the dots from a page id to code [08:43] I guess which app it's likely to reside in, then grep the app's browser/configure.zcml. [08:43] lifeless, this is another question I had when I first started working on Launchpad that would have been immensely helpful in my ramp up. [08:43] There must be a better way. [08:44] why does Distribution:+filebug need to know SELECT COUNT(*) FROM Bug, BugTask WHERE BugTask.bug = Bug.id AND BugTask.distribution = 1 AND (1=1) ? === almaisan-away is now known as al-maisan [08:51] so next ramp up question [08:52] lib/lp/bugs/templates/bugtarget-filebug-search.pt calls into +filebug with id filebug-search-form [08:52] what maps that back to a new template / api call ? [08:52] grep only finds a js file [08:52] The dupefinder is JS. [08:53] ok [08:53] so its setting up a call into a js function [08:53]