=== StevenK changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews === matsubara is now known as matsubara-afk [02:06] sinzui: ping [02:06] Project windmill build #91: FAILURE in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/91/ [02:45] thats better, [02:45] more closed bugs [02:45] and one more hidden critical unearthed [02:55] * lifeless resists the urge to start at bug oldest and walk forward [02:55] wgrant: ping [02:56] I'm pinging you cause curtis had better be asleep by now :) - where is Cve:+index - is jc still on it, or would it be ok if I pick it up? its our #2 frequency OOPS [02:56] lifeless: He's still on it. [02:56] kk [03:08] wgrant: have you poked at +needs-packaging at all ? [03:08] wgrant: I'm struggling to find top-10 timeouts that are not fixed-undeployed, bugtask:+index, or in-progress by someone [03:10] lifeless: Not at all. [03:11] You seem to love BugTask:+index [03:11] wgrant: Not at all. [03:12] wgrant: the next obvious thing is a paginated message display [03:12] wgrant: which is to big for me: I get called away all over the place. [03:12] wgrant: I mean, making a vertical page-reloading batchnav is easy. [03:13] wgrant: making one that uses js to incrementally pull in new messages and actions and shows the comment form once you read to the bottom - a bit harder. [03:14] Yeah. [03:15] wgrant: so other than ratcheting scaling down a bit more, I'm not likely to really touch it for a bit [03:15] dammit, I need to finish my analysis of deletable bugtasks etc [03:15] * lifeless adds a note to talk to jml about said [03:16] wgrant: (oh, and I'd be delighted for someone else to do the next iteration on bugtask:+index) [03:18] wow [03:18] 400ms overhead per page atm [03:19] just doing traversal [03:19] https://lp-oops.canonical.com/oops.py/?oopsid=1907C1055 [03:19] probably a busy appserver contributing [03:27] i worry a bit sometimes about the minimum time launchpad can produce a page in [03:27] (i think it's quite a while, like 100ms) [03:27] maybe not that bad [03:28] mwhudson: well, the page I'm looking at - 400ms minimum. [03:28] mwhudson: trace the queries down to the session update after traversal [03:29] lifeless: erk [03:29] we're still in phase one though - drive the 99th% percentile down by fixing poor scaling in pages [03:30] oh that's the server machine wgrant is always complaining about isn't it? [03:30] mwhudson: well, its not the hardware. [03:30] It's partly the hardware. [03:30] mwhudson: its a 4-thread appserver with maybe 2-4 additional requests parsed and awaiting handoff [03:30] wgrant: it really isn't :) [03:30] lifeless: soybean has lots of them as well. [03:30] But we see most of our timeouts from the old appservers. [03:31] lifeless: yeah, something smells off about that that's not entirely related to the code in lp:launchpad [03:31] anyhow [03:31] tonight is pounce on the sysadmins night [03:31] Poor sysadmins [03:31] (not nastily) [03:32] just that tom is the guy working this progression [03:34] lifeless: if there's a bug with total crap in it (random text), do we just mark it as incomplete or do we need to get it deleted? [03:36] invalid probably. [03:36] whats the bug #? [03:37] wallyworld_: ^ [03:37] wallyworld_: Sorry, I meant to look at that earlier, but got sorta distracted :/ [03:37] 738023 [03:38] lol - classic - http://programming-motherfucker.com/ [03:38] wallyworld_: that seems to be outside the launchpad project? [03:39] yes [03:39] but it's an open lp question [03:39] ah [03:39] so i thought i'd clean it up since it's clearly a crap bug [03:39] so there is no spam in it [03:39] incomplete leaves it open [03:40] I would make it invalid [03:40] no. just crap [03:40] ok [03:40] thanks [03:47] Why does the series filter on the main pages of PPAs have such huge text? lol [03:47] Someone embedded a form inside a heading, and this was exposed in the recent style unifications. [03:48] H1 is big [04:45] I can has review https://code.launchpad.net/~lifeless/launchpad/bug-722794/+merge/54466 ? [04:51] lifeless: Done, r=me [04:52] thanks [05:36] wgrant: wtf: https://lp-oops.canonical.com/oops.py/?oopsid=1907C2337 [05:37] wgrant: I'm pinging you cause you added the transaction status to the timeline :) [05:40] lifeless: Huh. [05:40] But API timeouts normally work fine :/ [05:41] Although leonardr did change them recently. [05:41] https://bugs.launchpad.net/launchpad/+bug/730396 [05:41] <_mup_> Bug #730396: DistroSeries:EntryResource:getBuildRecords timeouts < https://launchpad.net/bugs/730396 > === jtv is now known as jtv-eat === almaisan-away is now known as al-maisan [05:54] what does BuildFarmJob.status <> 1 OR BuildFarmJob.date_finished IS NOT NULL imply [05:54] not fullybuilt and finished ? [05:54] the query I'm looking at also checks for FAILEDTOBUILD [05:54] can I drop the BuildFarmJob.date_finished IS NOT NULL condition without breaking semantics ? [06:03] thumper: https://bugs.launchpad.net/launchpad/+bug/730396/comments/9 [06:03] <_mup_> Bug #730396: DistroSeries:EntryResource:getBuildRecords timeouts < https://launchpad.net/bugs/730396 > [06:15] lifeless: That query is stupid, yes. For some of the cases you can drop the first BFJ <> AND date_finished IS NOT NULL check, yes. [06:15] lifeless: But we need to keep date_finished IS NOT NULL if status == 1. [06:18] sure [06:18] so over here [06:18] uhm [06:18] if we don't gc builds [06:18] there is no point reading all *FTBFS* rows every time [06:18] full stop [06:18] Hmmm, looks like I didn't kill dogfood [06:19] lifeless: If we move to an incremental model, sure. [06:19] lifeless: But then we'd need to read all recent successes too. [06:19] wgrant: why? [06:20] lifeless: Because we need to be able to remove failures that are no longer failures. [06:20] wgrant: do they get removed from this collection? [06:20] If we're not reading the whole failure collection, we need to read every other collection. [06:23] wgrant: do they get removed from this collection? [06:25] lifeless: Which collection? [06:25] fbtfs [06:25] lifeless: A retried build is not a new build. [06:25] A retried build will disappear from this page. [06:25] (unless it fails again) [06:25] wgrant: so, we do gc [06:25] Pardon? [06:25] or are you asying we reuse the BFJ ? [06:26] We reuse the BFJ. [06:26] * lifeless headdesks [06:26] Yes. [06:26] Someone made this decision 6 years ago, and uh, yeah. [06:26] anyhow [06:26] I'd read all collections incrementally [06:27] that seems cheap and effective as long as you don't shutdown for a month [06:28] wgrant: what rev has leonards change for timeouts ? [06:28] lifeless: It was a lazr.restful upgrade. [06:29] meh, I'll raise a critical [06:29] I'm pretty sure this worked after my changes, although it's clearly somewhat related. === al-maisan is now known as almaisan-away [06:33] https://bugs.launchpad.net/launchpad/+bug/740750 [06:33] <_mup_> Bug #740750: API timeouts broken and returning no useful data... < https://launchpad.net/bugs/740750 > [07:05] big f* jools? [07:11] poolie: ? [07:15] We reuse the BFJ. [07:16] Heh' [07:16] poolie: before his time [07:16] poolie: Build Farm Job [07:17] https://bugs.launchpad.net/launchpad/+bug/740759 might amuse you [07:17] <_mup_> Bug #740759: creating bzrdir on launchpad is slow (BzrDirFormat.initialize_ex_1.16)

< https://launchpad.net/bugs/740759 > [08:28] good morning === StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews === almaisan-away is now known as al-maisan [09:03] stub: hi [09:03] stub: how are you feeling? [09:03] yo [09:03] Better today... slept 13 hours straight last night. === jtv-eat is now known as jtv [09:04] stub: good timing - we tried your live index docs but got the patch number wrong: https://code.launchpad.net/~wgrant/launchpad/fix-patch-57/+merge/54481 is meant to fix it, but wanted to confirm: is the db patch *in the tree* meant to be a .1 patch, or a .0 with the same major as the one done live ? [09:04] I think the numbers have to match./ [09:04] Or it will be reapplied on the next db-deploy and then blow up. [09:04] I think. [09:04] db patch in the tree is supposed to be a 2208-xx-01 (or -02 etc.) patch. The number should match what you inserted into LaunchpadDatabaseRevision on production [09:05] hi [09:05] would i be right in expecting that the qastaging database will be generally faster than lpnet? [09:05] because of less load [09:05] It's a much smaller DB server. [09:06] And the cache will be colder. [09:06] Why this works - Launchpad appservers check LaunchpadDatabaseRevision for patches that have been applied that are not in the local source tree, and for patches in the local source tree that have not been applied. It ignores all patches that are non-zero in the 'patch' field. [09:06] So it's generally slower [09:06] hm [09:07] lifeless, wgrant: So that patch seems correct, and matches what I see on the production db. Did the patch also get applied to launchpad_prod_2 and launchpad_prod_1 ? [09:08] stub: yup [09:08] So it worked out best with id in the first field of the index? [09:08] stub: It was applied as -0 to all three. [09:08] stub: it was a lopsided problem [09:08] I don't see -0 in LaunchpadDatabaseRevision - I only see 57-1 [09:08] stub: Right, we had to fix it when an appserver failed to start. [09:08] stub: we tried a number of permutations on qastaging, and found that a partial index on the queues that are driven to 0 (the NEW and INCOMPLETE) performed very well [09:08] lifeless: NEW and UNAPPROVED [09:09] I thought the index would have worked better with id in the last field, as the ordering would have been done last, but I hadn't checked and wasn't sure. [09:09] stub: As would I. I didn't follow lifeless' trials very closely, though. [09:09] can't do id last [09:09] not without a bunch of headaches [09:09] interesting [09:10] right now we use one query for both: [09:10] - queue processing [09:10] - historical data analsis [09:10] pg won't use the indices we tried with id last, unless the full index sort was given [09:10] archive desc distroseries desc setc [09:11] it *will*, but not for status=0 [or 1 we are pretty sure] [09:11] because its a lopsided data problem [09:11] status | count [09:11] --------+--------- [09:11] 4 | 5777 [09:11] 1 | 182 [09:11] 2 | 47494 [09:11] 3 | 2289831 [09:11] 0 | 40961 [09:12] Um what? [09:12] That's about 41000 more with status 0 than I would expect. [09:12] wgrant: indeed, but not enough to worry the index [09:12] Enough to worry me, though... [09:13] wgrant: sure, separate problem [09:13] *blink* [09:13] Similar on mawson. [09:13] Hmm. [09:14] 0 | 97 | 40877 [09:14] Distroseries 97 [09:14] stub: so the headaches are: we would have to either: split out a separate query for the historical data, if we changed the status 0 case to use archive desc distroseries desc status 0 desc id desc [09:14] karmic [09:15] (because the historical data /wants/ id as the first sort key - and pg does a decent job there today) [09:15] Oh. [09:15] or [09:15] bigjools: Old archive rebuilds :( [09:15] well, no or [09:15] lifeless: Sure. I recall the query. I'm interested in you found it works best with the id in the sort key. I think we have similar cases elsewhere - bugs or bugtask I think. [09:15] basically, there are two totally different use cases in the one table [09:16] stub: only because its essentially a micro-table embedded in the bigger one [09:16] stub: thats why adding a partial index gave good results - -very- fast results once applied [to that particular query] [09:17] wgrant: grar [09:17] bigjools: 2009ish [09:17] let's clear those out [09:17] So long-fixed. [09:17] But yes. [09:17] stub: its possible you can come up with something better [09:17] Also, accepted is pretty big... probably newer copy archives. [09:18] Indeed, the newer copy archives. [09:19] lifeless: If you tried the partial index on (distroseries, archive, id) instead of (id, distroseries, archive), that would be all I'm interested in at the moment. [09:19] stub: but the nice thing here was that the partial index was small - 1280 kB [09:20] lifeless: I think in earlier versions of PG, the index would not have been used at all because the first field in the index was not used until the very end. [09:20] stub: we didn't try that variation, no [09:21] wgrant: we need a way of cleaning out copy archives entirely [09:21] bigjools: Yes. [09:21] wgrant: I want to make derived distros supersede them [09:21] stub: Limit (cost=0.00..104.34 rows=31 width=36) (actual time=0.132..83.965 rows=1 loops=1) [09:21] -> Index Scan Backward using packageupload__distroseries__status__archive__idx on packageupload (cost=0.00..5287.65 rows=1571 width=36) (actual time=0.131..83.961 rows=1 loops=1) [09:21] bigjools: An interesting plan. [09:21] Index Cond: (distroseries = 104) [09:21] Filter: ((archive = ANY ('{1,534}'::integer[])) AND (status = 0)) [09:22] Total runtime: 84.023 ms [09:22] stub: ignore the index name, thats on qastaging where we were experimenting [09:22] wgrant: they're essentially the same thing, but copy archives have a spethial UI [09:22] stub: also - thats *cold* runtime [09:22] bigjools: Indeed. [09:23] lifeless: Right. So in PG 8.4, it might well be more efficient to put the field we are ordering on first, and the ordering and filtering done in one index scan. [09:23] stub: at least sometimes, yeah [09:23] Which is different to how we optimized stuff before elsewhere in the schema. Might be some room for improvement there. [09:25] wgrant: OOPS-1907BUILDMASTER3 .... [09:25] lifeless: Where the live rollout docs good enough for the losas to apply the patch without hiccups? [09:25] bigjools: What about it? [09:25] stub: yup, except we got the patch # wrong [09:26] bigjools: It's annoying, but ordinary and harmless. [09:26] wgrant: it should not be an oops [09:26] bigjools: Sure. [09:27] well, we could see if someone deleted the source [09:27] Someone uploaded one to supersede it. [09:27] or that [09:27] lifeless: Right. Applying a -0 patch live would mean the appservers fail to restart, so wouldn't even be immediately noticeable. [09:27] that oops happened 3 times [09:27] bigjools: Different archs? [09:28] ah might be [09:28] Hm, no. [09:28] Different series. [09:28] https://launchpad.net/~kxstudio-team/+archive/ppa/+packages?field.name_filter=lv2core&field.status_filter=&field.series_filter= [09:28] stub: we only caught it when we tested dropping the appserver name from the lpnet configs and found it broke [09:28] ok [09:28] stub: so the name in db-devel is meant to be -1 ? [09:28] lifeless: Yes. [09:29] cool [09:29] lifeless: Anything non-zero in that field. A non-zero in the 'patch' field means ignore this on runtime checks - it is a minor patch and you shouldn't care if it has been applied or not. Matches our life updates usecase exactly, so yay for prescience. [09:30] stub: kk [09:31] jtv: Hi! [09:31] hi henninge [09:31] jtv: Are your reviewing today? [09:31] henninge: oh, completely forgot about it. :( OTP now. [09:32] np [09:32] lifeless: Hmm.... in fact, I can't see why -1+ patches can't land on launchpad/devel .... [09:32] danilos should be reviewing today, too. ;-) [09:32] stub: that would be great [09:33] Where does that rule get changed now? [09:33] the pqm config [09:33] its out of the tree [09:34] Heh, the first time it's needed a change in almost three years, a month after I argued that it never needed changing... [09:34] :-) [09:36] I think we just need to block *-0.sql from landing on launchpad/devel now. No problems landing updates to the scripts, security.cfg or minor db patches. [09:36] Right. [09:36] Anyone see a problem with that rule? [09:36] Nope. We should also drop the conflict check while we're there, since it doesn't work. [09:36] And is pointless. [09:36] fine with me [09:38] wgrant: Do you want to file an RT explaining that conflict check thing, or do you want to explain the conflict check thing to me so I can file the RT ? [09:39] stub: https://pastebin.canonical.com/45073/ is the current hook. [09:40] MY EYES! [09:40] stub: The second bit (bzr ls -0 [...]) is meant to check for conflicts, but doesn't actually work. I considered removing it when we migrated the check from the tree to PQM, since it seems pretty pointless, but decided to make as few changes as possible. [09:43] So precommit_hook=[ -z "$(bzr status -S database/schema/ | grep '-0.sql$')" ] ?? [09:44] stub: Don't need to protect fti.py or anything? [09:45] wgrant: fti.py is 50/50 - adding a new index could futz things, changing an existing one will not (the column will exist, it just won't have the expected data in it) [09:47] Sounds good, then. [09:47] RT time. [09:47] I guess we should block it - we likely won't change anything there anyway until we know where our search is heading. [09:47] stub: Thanks for the review. [09:55] precommit_hook=[ -z "$(bzr status -S database/schema/ | grep -e '$-0\.sql\|/fti.py$$')" ] [09:55] is what I end up with [09:56] Looks sensible, as long as it works. === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan === danilos changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: danilos | https://code.launchpad.net/launchpad-project/+activereviews [10:55] danilos! ;) [10:55] henninge, :) [10:55] henninge, I don't know what you've been talking about ;) [10:55] danilos: I do, though! Can I please have a review? [10:56] henninge, sure thing [10:56] danilos: diff is only 1075 lines [10:56] https://code.launchpad.net/~henninge/launchpad/devel-605924-hastranslationtemplates/+merge/54470 [10:56] henninge, ha-ha-ha [10:56] henninge, ok, I'll take it [10:56] danilos: thanks === al-maisan is now known as almaisan-away [10:58] henninge, btw, just looking through proposal text, "shortlist" is especially recommended when you expect something to return a short list [10:58] henninge, because then it catches bugs early [10:58] danilos: oh, i thought it was about cutting a list off. [10:59] henninge, no, it throws an OOPS/exception when hard limit is broken [10:59] ah [10:59] so using shortlist is saying "tell me if this ever gets longer than X" [11:00] danilos: thanks, I did not know that. I am happy to put it back in, then. [11:00] henninge, yeah, it's a developer tool :) [11:00] henninge, I don't know in what context did you drop it, so it all depends on that, but just clarifying what shortlist is about [11:01] Project windmill build #92: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/92/ [11:03] Morning, folks. [11:05] henninge, good job clearing the inter-dependency between template count and uses-translations [11:06] yeah, that was strange to see [11:16] henninge, not yours, but line 226 of the diff: "hidding" === jtv1 is now known as jtv [11:29] henninge, in TranslationTemplatesCollection, I am surprised target_pillar is not used anywhere, but since it's apparently not, good cleanup there as well :) [11:30] danilos: yes, that was only used to access translation_usage in getCurrentTranslationTemplates [11:31] henninge, also, instead of listifying stuff in tests, I suggest you use assertContentEqual instead [11:32] henninge, unless where you want to compare the order as well [11:32] henninge, fwiw, this is probably irrelevant comment when you get shortlist back :) [11:33] henninge, also, I'd rather see test_has_obsolete_translation_templates as 3 separate tests [11:33] henninge, or 4 [11:36] henninge, and also, you probably inserted list() on diff line 116 because you removed the shortlist as well, so that can probably go (and if you decided not to re-insert shortlist, this is where you should use it) [11:38] henninge, r=me [11:38] danilos: yup, that's just because of shortlist. [11:38] danilos: thanks [11:39] henninge, up to you to decide if you want shortlist back or not, not a big deal either way [11:39] danilos: I felt the same about test_translation_templates but I played along with what the rest of the file was doing. [11:39] I may just split up the other tests, too. [11:40] henninge, right, we did write tests that way in the past [11:40] danilos: And I was always the one saying "one assert per test"! [11:40] henninge, that'd be great, but if it gets complex, it might mean a new branch [11:40] * bigjools waves at danilos [11:40] I'll have a look [11:40] henninge, heh, well, that is tricky because we sometimes use asserts to make sure input data is right [11:41] bigjools, hi there [11:41] danilos: ok, that is only a guideline. [11:41] ;-) [11:41] bigjools, I hope you just look forward to hearing my lovely voice :) [11:41] danilos: are you reviewing? I have a short branch [11:41] bigjools, yeah, I do (but I am disappointed ;) [11:41] bigjools, pass it on, just done with henninge's [11:41] danilos: I miss your booming presence on the TL calls :) [11:41] henninge, yep, cheers [11:42] bigjools, ok, now I feel better, thanks :) [11:42] danilos: https://code.launchpad.net/~julian-edwards/launchpad/localdiff-sync-bug-739525/+merge/54496 [11:44] bigjools, branch sounds like a nice feature, but you've got a single lint issue so please fix it (I understand it may not be from your changes :) [11:44] looking at the code now [11:45] :/ [11:45] Project windmill build #93: STILL FAILING in 43 min: https://lpci.wedontsleep.org/job/windmill/93/ [11:45] it was my lint [11:48] bigjools, I am sad to see a message like "The following sources *would have been* synced if..." go from the real UI [11:50] :) [11:50] bigjools: Have you worked out how permissions are going to work? [11:50] no [11:50] Heh [11:50] bigjools, r=me, but I'd also remove the "XXX" from a comment about a thing you are not really going to fix unless it proves to be broken [11:50] ok [11:50] thanks danilos [11:51] wgrant: right now it's launchpad.edit on the series [11:52] bigjools: launchpad.Edit and launchpad.Append (since you need Append to run syncSources) [11:52] right [11:52] this will all fall out in the wash when we start doing intensive testing [11:52] Yeah. [11:52] And then we realise we need queues. [11:53] which Steve is doing tomorrow :) [11:53] And then you have an excuse to fix copying. [11:53] queues would be nice [11:53] I might have to badger flacoste and jml about that [11:53] mushroom! mushroom! [11:53] snaaaaaaaake === matsubara-afk is now known as matsubara === almaisan-away is now known as al-maisan [13:29] jcsackett: I Untriaged bug 740225. The former rosetta team need to explain if this was inteded and why or if it is a regression. The behaviour is correct for Ubuntu so it appears to be by design [13:29] <_mup_> Bug #740225: The differences between New and Translated for upstreams was removed

< https://launchpad.net/bugs/740225 > [13:36] Are lp-propose-merge bugs bzr bugs or something else? [13:39] sinzui: understood. [13:42] sinzui: do you remember the issue you helped oem with regarding filing bugs in IE? [13:43] i believe it was an issue with an error popping up on the summary field? i've got an open question about it, and was wondering if there was a bug filed. i can't find one. [13:44] jcsackett: The JS to populate the matches and enable the continue actions does not work in IE. It is impossible to report a bug since the user is forced to use the guided process [13:45] jcsackett: The bug is filed and has many dupes. tag: javascript, text ie or ie8 or ie7 [13:45] ah right, tags! [13:45] * jcsackett never remembers tags when bug searching. [13:46] first match [13:46] I always use tags but I hate the default ANY, I am not sure I have ever used ANY in a search === mrevell is now known as mrevell-lunch [14:02] adeuring1: bzr+ssh://bazaar.launchpad.net/~abentley/launchpad/js-translation [14:03] adeuring1: https://code.launchpad.net/~abentley/launchpad/js-translation/+merge/54067 [14:06] deryck: I'll restart [14:10] adeuring1: do you want to chat in mumble about the template changes? [14:10] abentley: no yet, let me first read a bit more [14:10] adeuring1: okay, ping me whenever. [14:15] danilos, would you take a look at https://code.launchpad.net/~leonardr/lazr.restful/integrate-strict-versioning/+merge/54530 ? [14:16] abentley: so your idea is to always render both variants of "X is configured/not configured" and to make the "wrong" variant invisible via CSS, right? [14:16] adeuring1: right. [14:16] ok, so no need to mess with the menu:overview stuff, i think [14:16] abentley: ^^^ [14:18] adeuring1: Actually, we probably do. [14:18] abentley: why? [14:18] leonardr, after lunch, sorry [14:18] adeuring1: The icon is supposed to vary between "plus" and "edit". [14:18] abentley: ah, right [14:18] adeuring1: Depending on whether there's already a value. [14:19] adeuring1: But that state is frozen when the template is rendered. [14:19] right [14:19] adeuring1: So either we get two plusses or we get two edits. [14:19]