[00:25] jelmer, hi [00:25] hello again :) [00:26] jelmer, so i think the lep is as far forward as makes sense for something that's just queued up [00:26] if we're going to actually start it, it should be made a bit tighter about requirements [00:26] probably make sure all the issues people raised are addressed [00:26] and then break out some particular bugs [00:26] all with a tag [00:27] poolie: ah, ok [00:28] poolie: is it still waiting for anything, or should one or more of us just go over it to make sure all requirements are addressed? [00:29] nothing else [00:29] that would be great [00:29] if you'd like to start on it [00:30] might as well [00:36] Project windmill build #101: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/101/ [00:37] sleep first though, g'night! [00:38] wgrant: how much work is sitting unreleased on lh trunk ? [00:43] lifeless: 11 revs... start/stop-loggerhead are gone (deprecated long ago), a couple of logging changes that seem harmless. [00:43] Otherwise just optimisations and UI tweaks. [00:44] wgrant: feel like doing a 1.19 ? [00:44] wgrant: my thinking is: we have radical stuff in the pipeline [00:44] be nice to get the incremental stuff 'out there' [00:44] lifeless: i might talk to jam about that tonight. [00:44] and you're already fiddling with all the policy etc etc [00:45] lifeless: Oh, you mean "unreleased" as in everything since 1.18, not since our last rev? [00:45] wgrant: yes [00:45] wgrant: the stuff on trunk we should deploy, thats not done because of unfamiliarity with the mechanism for jam [00:46] There's only a few revs more. 1.18 was 421. There's the new theme, what look like bzr compat fixes, and the HEAD changes. [00:46] And a favicon change. [00:46] So all releasable. [00:46] If we're going to start doing more radical stuff, we should indeed release. [00:46] historydb is more radical [00:46] Right, but I'm not sure how close that is. [00:47] I think it would be nice to have a modern loggerhead equivalentish to our deploy before that tracks forward [00:47] wgrant: not all that far off [00:47] who did the new theme? [00:47] lifeless: https://code.launchpad.net/~wgrant/launchpad/new-loggerhead/+merge/54803 [00:47] james_w: huwshimi, I believe. [00:47] thanks huwshimi [00:48] james_w: Yeah you can blame me :) [00:48] huwshimi, far from it :-) it's a great and sorely needed improvement [00:49] james_w: Hopefully it'll be integrated with Launchpad further at some point, but it's a start [00:49] yeah [00:50] huwshimi, two questions, do we need all the duplication of the (i) icons? and can we have the commit message wrap if it's too long? [00:50] e.g. http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/files [00:50] (for me at least with 1280px width) [00:51] james_w: Do you mean the icons for "view revision" and "view branch changes"? [00:51] yeah [00:52] there's more on the file pages [00:52] http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/view/head:/Makefile [00:52] that's a lot of informative links [00:55] james_w: It's not great, but that's the way we do sub navigation elsewhere in Launchpad [00:55] james_w: Can you please file a bug about the commit message length? [00:56] huwshimi, fair enough [00:56] sure [00:56] james_w: I will be address a lot of things like that soon [00:56] *addressing [00:56] huwshimi, do you want a separate one for http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/changes [00:56] where if you expand the top revision it gets loooooong [00:57] and http://bazaar.launchpad.net/~offspring-hackers/offspring/trunk/revision/60 I guess, which is the worst yet :-) [00:57] also, bugs against loggerhead or launchpad? tags? subscribers? [00:57] they should be bugs against loggerhead [00:58] just tag them as ui and you can add me as a subscriber [00:58] ok, thanks [00:58] james_w: Thank you for noticing (and caring) [00:59] it's great to have someone to raise these with again :-) [00:59] james_w: oh and with the lengths you can probably put it all in one bug but link to all those pages [01:00] bug 742178 [01:00] <_mup_> Bug #742178: doesn't handle long commit messages well < https://launchpad.net/bugs/742178 > [01:01] james_w: Thanks heaps [01:16] ah it would be good to have someone who knows some css look at those [01:29] I want a teddy bear [01:29] to talk webservice collection offset tuning with [01:29] to the sound of o/~ I need a hero [01:31] nothing here but us crickets [01:32] crickets are good [01:32] spm: hey, can you explain analyze a query on prod for me - looking for cold cache again [01:32] https://bugs.launchpad.net/launchpad/+bug/736008/comments/1 [01:32] <_mup_> Bug #736008: Product:+code-index timeouts < https://launchpad.net/bugs/736008 > [01:34] lifeless: https://pastebin.canonical.com/45213/ [01:34] booyah [01:36] spm: thanks [01:36] np [01:38] spm: could you run the same query, not explained - the actual query - on another server? [01:38] sure, one sec. [01:42] lifeless: count: 168, Time: 125.128 ms lp_1 [01:46] spm: woo,t hanks. [01:46] hot shit query [01:47] as opposed to warm? [01:47] run it again :) [01:47] Time: 62.076 ms [01:47] yup [01:47] I have oopses with it at 700ms [01:47] 110; then 3 x 171's. [01:47] the older version [01:47] let me grab that [01:48] is stabilising at 171ish. [01:48] https://lp-oops.canonical.com/oops.py/?oopsid=1906G1635#longstatements [01:49] 1521ms + 589ms [01:49] http://pastebin.com/8H8tha0g [01:49] is the original [01:50] which is 4.6 seconds coldish on qas [01:50] ouchy [01:50] and 650 hot [01:50] so yeah, I'm pretty happy with the improvment [01:51] the new plan is missing the sequential scan on branch :) [01:52] ! [01:53] the tuple read rates that stub produced seem to indicate that something was sequential scanning branch about once a second [01:53] lifeless: could you have found that? [01:56] yes [01:56] code:+index is the page [01:56] ppr will tell us hits [01:56] I would expect multiple causes [01:56] i guess any branch filtering is similar queries [01:57] yeah [01:57] the double subqueries were either too complex or constrained for some corner case - I think the thing was that they were different [01:57] * mwhudson wonders what fraction of branches are private... [01:57] so pgsql wasn't coalescing them [01:58] mwhudson: 3 5ths of bugger all [01:58] yeah [01:58] if they hit 10% of the table or whatever the threshold is i wildly speculate that we'd have the sequential scans back [01:58] but that's unlikely [01:59] problem for another day [01:59] ye [01:59] you'd actually need 10% of the table for *one context* [01:59] oh righ t:) [01:59] mwhudson: I'm so glad you are here though, you can answer something flummoxing me [01:59] mwhudson: why does the original (and the new one because i kept it) only constraint the source branch to the product [02:00] lifeless: don't know [02:00] i guess you can end up with the products being different if you move one of the branches after the mp is created [02:01] right, but why does moving the target not hide the count; but moving the source does ? [02:01] constraining the target branch would seem a bit less wrong though ... [02:01] *hige it from the count* [02:02] lifeless: i bet that whoever wrote this (possibly named after a disney rabbit) assumed that source target and target target are always the same and 'optimized' it [02:02] lifeless: just a guess though [02:02] hahaha [02:02] tactfully put [02:02] see the thing is [02:02] if the target was constrained the original query is fine [02:03] uhm, if 'source and target are constrained' [02:03] heh heh heh [02:03] DistributionSourcePackage:+code-index 540 [02:03] yes, my guess doesn't make much sense on that level does it? [02:03] Product:+code-index 4965 [02:04] more constraints -> faster query (generally) [02:04] mwhudson: yes [02:04] mwhudson: so, 5500 hits a day [02:04] mwhudson: 2 runs of the query each hit [02:04] 11K query runs a day [02:05] 710983.85 is pretty high [02:05] lifeless: it's one of these slightly horrible things, we /try/ fairly hard to enforce target target == source target, but i don't think we can completely [02:05] mwhudson: we can choose to just not report the exceptions :> [02:05] yeah, i think that would be fine [02:05] lifeless: what is 710983.85 ? [02:05] branch tuples read per second [02:06] ah right [02:06] https://devpad.canonical.com/~lpqateam/dbr/last-15-mins.txt [02:06] yes it is [02:06] > 3 * bugtask reads [02:06] and bugs are much higher user activity than branches [02:06] oh [02:06] I remember one of the branch thoughts [02:06] 'url -> disk mapping' [02:07] still several OOM too high [02:07] You should not import patch_entry_explicit_version from canonical.launchpad.components.apihelpers: [02:07] i'm pretty sure that doesn't seqscan... [02:07] Did that change recently? [02:07] StevenK: add it to __all__ [02:07] mwhudson: it doesn't [02:08] or even read more than a handful of tuples, so yeah, like you said, an OOM out [02:08] at least [02:09] i wonder how much cpu 396125.47 tuples/sec uses, it must be more than 0.01% though, and 'rewrite' isn't a user on https://devpad.canonical.com/~lpqateam/dbr/last-15-mins.txt [02:11] lifeless: the pillarname read count is demented too, but at least it's obviously a table that gets accessed an awful lot [02:11] remember that foreign keys on updates take locks [02:12] possibly on reads too,but I'd need to double check that [02:20] sinzui: ping [02:21] sinzui: if you're around; how would you and jc feel about my stabbing at CVE:+index; its consitently 2nd or third on the oops report at the moment [02:26] What is the scanner doing these last few days? [02:27] 1600 OOPSes... [02:35] Project db-devel build #488: FAILURE in 4 hr 58 min: https://lpci.wedontsleep.org/job/db-devel/488/ [02:37] wgrant: bug is filed ;) [02:56] lifeless: go ahead on the cve:+index jcsacket and I were to busy to work on it today [02:56] lifeless: Just taking a look at the AJAX logging stuff. What's involved in getting the URL into the header? [02:57] huwshimi: one cheap hack would be to change the publisher to add an X-header unconditionally on every reponse with the url it got [02:57] huwshimi: e.g. probably not much [03:00] lifeless: Is this going to be bad for us in any way? [03:01] huwshimi: slightly more response overhead [03:01] huwshimi: if someone sends a 4K request, they'll get 4K more back [03:01] huwshimi: (4K URL I mean) [03:02] lifeless: Will it be enough of an impact for us not to do it? [03:02] You can't get at the request from the response? [03:04] wgrant: Well this is not really the response. It's an event that fires every time an AJAX even completes, with no other knowledge about the request except what is returned [03:05] huwshimi: I think we can do it, will just want a brief walk-by the usual suspects to look for gotchas [03:06] huwshimi: :( [03:07] lifeless: I may have just thought of a way to do it without the header, thanks to wgrant's question. I'll have a quick try [03:07] huwshimi: care to file a bug asking for an X-LP-URL header, and mail the -dev list for feedback. [03:07] huwshimi: if you can't get it, of course [03:07] lifeless: Sure [03:41] grrr [03:41] Collections may be great for adhoc assembly [03:41] but they are terrible for optimisation. [03:41] * lifeless puts on the wet blanket hat [03:47] * StevenK grumbles. [03:48] I think python-debian has a bug in it's regex to parse valid versions [03:48] oof [03:48] StevenK: Is it meant to only parse valid versions? [03:49] Also, why is the solution to AJAX being too slow to limit it to only a few bugtasks? :( [03:49] wgrant: It accepts : in the upstream version all the time, but that's only valid if there is a epoch. [03:49] lifeless: OK I can't reliably get it this way. Is the code for managing the headers outside of Launchpad? [03:49] (Pulling from my memory, so I could be wrong.) [03:50] wgrant: It has a re_valid_version regex defined, so I doubt it. I think the regex is naive. [03:50] StevenK: I don't think it's valuable for it to be terribly pedantic, really. [03:51] huwshimi: we'd put it in lib/canonical/launchpad/(publisher|publication).py I think. [03:51] wgrant: In this case, the version that sorts the highest is invalid, and so we can't set it in the DB, due to integry rules. [03:51] huwshimi: there are some interactions with the api, but I'm 99% sure that starting there would get what we need easily enough [03:52] wgrant: So I'd like to filter invalid versions from the ancestry sets. [03:52] StevenK: Right. [03:53] wgrant: And I'm having trouble convincing debian_support.BaseVersion that "a1:1.8.8-070403-1~priv1" really is invalid. [03:54] Due to it accepting : in upstream version, it thinks epoch is None, and the upstream version is "a1:1.8.8-070403", which is just wrong. [03:54] Yeah. [03:54] Er, thinks the epoch is None? [03:54] That's a bug. [03:54] Missing epoch is 0. [03:55] But it isn't missing, it's invalid :-) [03:57] % bin/py foo [03:57] Version: a1:1.8.8-070403-1~priv1 [03:57] Epoch: None [03:57] Ah, true. [03:58] IIRC, you can only have :'s in the upstream version iff there is an epoch. [03:58] That's right. [03:58] Otherwise it's ambiguous. [03:58] "This is a single (generally small) unsigned integer. It may be omitted, in which case zero is assumed. If it is omitted then the upstream_version may not contain any colons." [03:59] wgrant: Have a look at python-debian in sourcecode, the re_valid_version is wrong, then. [04:00] StevenK: I can't really fault it on that. Be conservative in what you do, be liberal in what you accept from others, and all that. [04:01] wgrant: Suggestions on how to filter invalid versions out of ancestry, then? [04:02] ohdogohdogohdog [04:02] StevenK: I'm not sure what's best. [04:02] lifeless: Oh? [04:02] lifeless: What completely naïve and non-performant code have you found now? [04:02] wgrant: GenericBranchCollection uses storm expresses to express constraints [04:03] wgrant: so I have to unwind them to get the clause to regenerate it with class aliases. [04:03] because it discards the original context [04:03] in the adapter [04:03] :( [04:03] wgrant: I'm struggling to come up with *any* solution that doesn't involve hacking debian_support.BaseVersion [04:03] wgrant: its Yet Another Compiler. [04:03] I should suggest a code review rule : no more compilers. [04:04] gcc 4 lyfe? [04:04] StevenK: collections are compilers; wadl creation is a compiler, storm is a compiler. [04:04] StevenK: BugTaskSet.searchTasks is a compiler [04:05] StevenK: nearly everything that is easy to use and hard to make fast, is a compiler. [04:05] I didn't need the explanation, I was trolling. [04:15] lifeless, i'm glad you asked for a simple reproduction because i no longer seem to be able to hit it :) [04:16] poolie: heh [04:16] this is actually astonishingly fast [04:16] i think something must be wrong [04:17] Haha [04:21] mm i wasn't measuring the hard part [04:21] damn all software that tries to pretend the network is transparent [04:21] i wonder if i was just measuring some transient misbehaviour before [04:22] anyhow, the good thing is that if this works, it may make kanbans much faster [04:30] lifeless, the numbers from that script are just astonishingly good [04:30] 0.96s for some queries [04:31] i never saw sub-second calls from australia before [04:31] (i guess that's on a warm socket) [04:31] Happiness is a warm socket. [04:31] yeah apparently so [04:31] Maybe gandwana and potassium finally decided to die. [04:31] ? [04:32] what would that mean, wgrant? [04:32] poolie: Well, it would speed up requests. [04:32] Since those two are multithreaded, slow, and the source of most of our timeouts. [04:33] because they're slow appservers? [04:33] Slow and misconfigured. [04:41] wgrant: Didn't we port some version validator from somewhere in Soyuz? [04:42] StevenK: You mean the one that's in the database? [04:43] wgrant: That one feels a little heavyweight for validating versions [04:45] wgrant: Dpkg::Version::version_check would be nice, but I can't use that [04:51] Project windmill build #102: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/102/ [04:53] wallyworld: Those two Windmill failures look like yours, I think. [04:53] wgrant: oh :-( [04:54] i'msure they passed locally :-( [04:54] wgrant: https://code.launchpad.net/~wallyworld/launchpad/recipe-text-xss/+merge/54638 [04:54] ahhh [04:54] 1 file changed, 31 insertions(+), 35 deletions(-) [04:54] \o/ [04:54] * wallyworld has to go get kid from school [04:54] deleting code *and* making things faster [04:57] lifeless: Any suggestions on how I can break an AJAX request so it returns an exception? [04:57] lifeless: I mean returns an oops id [04:58] break the server side code [04:58] add [04:58] raise Exception('zomg') [04:58] to the code path [04:58] lifeless: ok [04:58] lifeless, i think some other change must have fixed it [04:59] huwshimi: your demo was -really really shiny- [04:59] i'm running the same code i was then and i no longer see the effect [04:59] poolie: sure [04:59] poolie: I believe that you saw it [04:59] poolie: I *suspect* it was doing a full text search for '' or something equally daft [04:59] (we repacked the fti index for bug recently) [05:00] [amongst other fixes such as improving component fixes, related bugs yada yada yada] [05:00] i certainly wasn't asking for that [05:00] it's possible that lplib or the server did end up doing it [05:00] poolie: thats what I'm sketching [05:00] yeah [05:00] its the only thing I can think of that might vary [05:00] anyhow, the bottom line on performance is actually quite interesting: [05:00] because the collections are really just function calls to searchTasks [05:00] single calls are quite impressively fast [05:01] however, if you have to do 600 of them, not so much [05:01] thus my braindump [05:01] poolie: sure; that aspect has been heavily sketched out in the filter+expand API LEP [05:01] yep [05:24] lifeless: OK, I don't know enough about Launchpad's internals to break this request. Any clues about where to look? [05:37] lifeless: Also are there any time recommendations that we have for AJAX requests? E.g. over 5 seconds is very bad under 2 seconds is good === almaisan-away is now known as al-maisan [05:48] huwshimi: ajax requests are no different than others [05:48] huwshimi: we're (long term) aiming at subsecond for 99% of requests and under 5 seconds for all [05:48] huwshimi: current timeout limit is 11 seconds [05:48] huwshimi: I think colouring anything > 1 second a vivid shade would be sensible ;) [05:49] lifeless: OK, I'll do >1 second for now [05:50] most ajax things should be very targeted operations [05:50] unlike e.g. a bug listing page [05:50] at least for now [05:51] hi lifeless [05:52] hiya === al-maisan is now known as almaisan-away [05:55] lifeless: so, several years ago, I wrote some javascript code to output an HTML table that needed to be displayed multiple times on a page (i.e. output with document.write) and that sped up load time tremendously, could something like that work as a stop gap for the repetitive JS code until it can be refactored properly? === almaisan-away is now known as al-maisan [05:56] micahg: that sounds like it would be refactoring it properly [05:56] lifeless: so, any clues on how API internals work. Like, where does the API code for bugs live? === wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews === wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews [05:57] huwshimi: lib/lp/bugs/model [05:57] huwshimi: whats the url you are calling ? [05:57] lifeless: well, no this would just output the javascript inline as opposed to using a function w/params and have one copy of the JS code in the rendered page [05:57] lifeless: But if I raise the exception there won't it be broken for the normal request as well? [05:57] huwshimi: yes [05:57] huwshimi: are you trying to check it works, or write an automated test ? [05:58] lifeless: I'm trying to check it works [05:58] huwshimi: because, for the latter, its a bit complex: we don't want /any/ broken apis in the system, so you have to register a broken one temporarily, use it, then unregister in your test. [05:58] lifeless: Are you sure your +needs-packaging change is OK? [05:58] huwshimi: right, so pick something - like mark as duplicate [05:59] wgrant: if I marked it as such, I tested it on qastaging [05:59] wgrant: why do you ask ? [05:59] lifeless: It's not QA'd yet, but it's gone from timing out on lpnet to 0.2s on qas. [05:59] So I'm a bit suspicious. [05:59] wgrant: sounds right [05:59] Seriously? [06:00] The results look fine. [06:00] But the time is not plausible. [06:00] oh ye of little faith [06:02] https://answers.qastaging.launchpad.net/ubuntu/karmic/+needs-packaging was 2.43 on qastaging [06:02] * lifeless looks at a profile [06:03] natty's first batch is 0.20 to 0.25s. [06:03] 0.17 just now. [06:04] 00294-01256 SELECT COUNT(*) FROM SourcePackageName, (SELECT ... [06:04] 01259-02197@SQL-launchpad-main-slave SELECT DISTINCT ON (score ... [06:04] it looks plausible to me [06:04] wgrant: you do realise it has memcache in there [06:04] wgrant: which I didn't touch [06:04] Oh. [06:05] Indeed. a more plausible 1.7s on the next batch.. [06:05] wgrant: 2.5 seconds cold, 0.2 in memcache [06:05] I think its qa ok indeed [06:05] marked as such [06:05] wgrant: I'll whip through and do all mine [06:06] lifeless: Great. I'm doing the others'. [06:07] done [06:16] lifeless: so, I commented on bug 735966, but almost instantly regretted it, is that the same bug? [06:16] <_mup_> Bug #735966: Product:+bugs timeouts < https://launchpad.net/bugs/735966 > [06:16] I hit the crash again, so I figured I'd confirm [06:19] * StevenK nails wgrant to the channel [06:19] Huh. [06:19] Thanks Optus. [06:20] We were throttled for almost 10 minutes. [06:20] To 28kbps. [06:20] Despite being at 70% of our quota... [06:20] Haha [06:24] lifeless: Gotta run in a minute, but this is where I'm leaving it for today: http://people.canonical.com/~huwshimi/ajax_time_log.ogv [06:29] * huwshimi is gone [06:38] micahg: may be, may not be :P [06:46] lifeless: should I file a new bug with my new oops code and you can dupe if appropriate? [06:46] no need [06:47] ok, do you want the oops? [06:47] you put it in the comment already [06:47] lifeless: no, I got a new one today [06:59] wgrant: http://pastebin.ubuntu.com/585234/ makes me sad. [07:01] StevenK: Doesn't that forbid upstream versions with colons? [07:03] wgrant: Probably. [07:04] wgrant: I looked at the check_version sub in Dpkg::Version, and I didn't feel like porting it to Python so I can use it. But I suspect it's the better answer. [07:05] StevenK: So python-debian doesn't have a useful checker? What about python-apt? [07:05] wgrant: I couldn't see one in apt_pkg [07:05] wgrant: python-debian's checker is too lax for our needs, since the DB's own checker is strict. [07:06] Doing it in one regex is not the way [07:08] :( qa-bad [07:08] wgrant: Which rev? :-( [07:09] StevenK: The Soyuz translations change. [07:09] Missing DB perms. [07:10] Which points to missing tests, too. [07:14] I think the qa-tagger needs to run more often [07:21] micahg: probably hasn't changed [07:21] lifeless: ok [07:23] Huh, translations sharing works. [07:24] wgrant: Hi! [07:24] henninge: Hi. [07:25] wgrant: why did you mark bug 741571 bad? [07:25] <_mup_> Bug #741571: Translation templates should always be uploaded from soyuz builds.