[00:06] * thumper still writing tests === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews/ [00:22] grr bug heat updates [00:23] wgrant: ID OOPS-1901QS5 [00:24] lifeless: you'd like my current tests [00:24] thumper: I will? [00:24] lifeless: they all have self.assertThat(recorder, HasQueryCount(Equals(0))) [00:34] with NoStormStatements: [00:36] lifeless: How? I changed the target of the original bug to a different project, and it sent me to the bug page as expected... [00:36] Using both +editstatus and the inline form of deprecation. === StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews/ [00:42] wgrant: I think I hit a redeploy [00:46] lifeless: How would that change things? [00:46] wgrant: transaction committed, appserver killed, refresh - boom [00:47] Mmm, I guess. [00:47] wgrant: have you seen https://devpad.canonical.com/~lpqateam/ppr/lpnet/2011-03/daily_2011-03-13_2011-03-14/categories.html - lots of requests :) [00:47] Anyway, once my branch lands it will just redirect instead. [00:47] Of course it conflicted with leonard's... [00:47] wgrant: even on post ? [00:47] lifeless: HTTP is just that awesome. [00:48] We already do it in some circumstances :/ [00:48] wgrant: sarcasm doth not befit thee [00:48] lifeless: That's daily? [00:49] lifeless: Is "sarcasm doth not befit thee" or did you make it up? [00:49] lifeless: Do we have a graph of 99% time over time? [00:49] Sigh. Is it a quote [00:50] mwhudson: Hi! [00:53] wgrant: thats one day, yes [00:55] wgrant: https://lpstats.canonical.com/graphs/PPR/ [00:55] lifeless: The translations graph is nice. [00:55] And should be even nicer tomorrow, I suppose. [00:56] wgrant: because of the drop ? [00:56] POFile:+translate was at 9.7seconds 99th percentile [00:56] lifeless: Right. [00:56] hopefully it will be better now ;) [00:56] but todays ppr failed to generate [00:57] lifeless: +translate seems to mostly complete in under 2 seconds. Nicely done. [00:58] wgrant: I think so [00:58] pure fluke that I bothered to try a with plan [00:58] there may be a fat index impeding regular queries; not sure [01:02] I want https://lpstats.canonical.com/graphs/PPR/ to be on a log scale [01:03] Indeed, most of our graphs should be. [01:04] spm: can tuolumne do that ? [01:05] log scale? not that I'm aware of. [01:07] Looks like pycairochart might not support it. [01:07] StevenK: hello? [01:09] mwhudson: I was talking to Julian yesterday about DSDs, and he was mentioning that we might want to delete DSDs -- which would remove comments associated with the DSD. Would you prefer the DSD hung around to preserve them? [01:10] Under what circumstances would you remove DSDs? [01:10] That is what I'm not certain about, but Julian seems to think the DSDJ runner needs to do it. [01:10] StevenK: dsd? [01:10] mwhudson: DistroSeriesDifference [01:10] I don't think removing them before a series is finished would be a good idea... [01:11] StevenK: you are asking me because i have linaro in my cloak i presume? [01:11] mwhudson: And you're in my TZ. :-) [01:11] mwhudson: we hear you are slightly technical [01:12] i'm not sure i'm the right person to ask about this though sadly [01:13] StevenK: i think i plead EINSUFFICIENTCONTEXT [01:13] StevenK: maybe get julian to talk to james_w ? [01:14] StevenK: or you can try to explain a bit more context i guess [01:14] Or I can in my morning [01:14] like, as wgrant says, "when would you remove them" ? [01:15] They need to be kept even between series if the difference persists. [01:15] Like I say, I'm unclear on why Julian wants to do that, and I want to get the simple cases working for the DSDJ runner before I bother diving into the harder stuff. [01:17] StevenK: i suggest we acquire some more clarity before discussing further then :) [01:17] * wgrant declares war on Makefile. [01:17] mwhudson: Do you know where to find some? :-P [01:17] StevenK: well, find out why julian wants to delete them? [01:18] data that is no longer interesting shouldn't be kept around in our primary tables [01:18] * StevenK teaches mwhudson about "humour" [01:18] mwhudson: ^ I suspect he's thinking about what is/isn't interesting [01:18] StevenK: heh [01:19] mwhudson: Something you might be able to help with -- I'm writing a run() method for a IRunableJob and I'd like to get at the log, so I can add messages to it, is that easy? [01:20] StevenK: isn't there a self.logger or something? [01:20] I have never worked out how to get the log from outside a RunableJob, either. [01:20] That would be really handy. [01:20] maybe i'm not thinking of the right object [01:20] maybe there just should be :-) [01:22] sigh, a coherent logging strategy would be nice [01:22] ... yes [01:22] a good logging framework would be too [01:22] thumper helped that by binning about 15 loggers from the tree [01:22] hi all [01:23] Hi poolie. [01:23] logging is one of those things that seems easy, but i've never seen it done satisfactorily [01:24] It doesn't matter how much it logs, you always want it to log more. [01:24] (or less, if it never breaks) [01:24] well [01:24] StevenK, wgrant: is a SourcePackage object _ever_ created without a distroseries? [01:24] jml added some overenthusiastic logging to the sftp server once [01:25] when the log got to 8 gigs within a day or something, that was decided to be "a bit too much" [01:25] Haha [01:25] StevenK, wgrant: see lp.bugs.model.bug line 1135 [01:25] ish [01:26] thumper: SourcePackage should be called DistroSeriesSourcePackage [01:26] lifeless: I realise that [01:26] thumper: there is a DistributionSourcePackage, which shouldn't exist [01:26] and a missing table which should exist [01:26] lifeless: but I've found some weird code in bugs [01:26] lifeless: it assumes that distroseries may be None in SourcePackage [01:27] I didn't think it could be [01:27] in ISourcePackage, not SourcePackage [01:27] one sec [01:28] thumper: I think you're right. [01:28] required = True [01:28] in which case, I'll simplify the bug code [01:28] thumper: But our sample data and/or tests are probably fucked. [01:28] Remove it and see what breaks, then fix it. [01:28] * thumper nods [01:29] thumper: ok, that looks bogus [01:29] lifeless: I've simplified the bug code [01:29] lifeless: as part of my work [01:30] what I don't understand [01:30] is the requirements of certain tasks [01:30] like... [01:30] what is needed for a source package task to exist? [01:30] a distroseries task [01:30] or a distro source package task [01:30] or either [01:30] or both [01:30] there seems to be implicit rules [01:30] A SourcePackage task has a DistroSeries and a SourcePackageName. [01:30] not enforced by the model code [01:30] A DistributionSourcePackage task has a Distribution and a SourcePackageName. [01:31] See determine_target [01:31] wgrant: yes... I understand that [01:31] wgrant: there are implicit expectations of certain tasks though [01:31] thumper: Ah, with regard to conjoinment? [01:31] you can't have a distroseries task without a distro task [01:31] Right. [01:31] Well, you can, but you're not meant to. [01:31] wgrant: well, exactly [01:31] eg. bug #43150 will make you cry. [01:31] <_mup_> Bug #43150: [SRU] maxima frontends fail to connect < https://launchpad.net/bugs/43150 > [01:31] the database nor model doesn't enforce it [01:32] thumper: \d bugtask [01:32] thumper: see the checl constraints [01:32] The current series task model only came about in late 2006, early 2007. [01:32] So there are old bugs (like 43150) that have invalid things. [01:32] lifeless: we don't have check constraints of other rows [01:32] indeed [01:32] I'd like to delete the conjoined concept [01:33] I see no benefit in cross-row rules [01:33] better to have a more powerful concept of 'target' and fix our seach [01:33] *search* [01:34] wgrant: wow, the rendering of that bug is --broken-- [01:34] OTOH, 2.38 seconds [01:34] I still don't understand the conjoined thing [01:34] kinda slow [01:35] lifeless: What's broken? [01:35] wgrant: no package name shown for the first two tasks [01:35] (I use that bug a lot, since it has lots of subscribers, lots of strucsubs, a few dupes, a few tasks, some series tasks, and various bits of activity) [01:35] lifeless: Right. [01:36] lifeless: There is no DSP task for them. [01:36] lifeless: Because they predate the new release management stuff. [01:36] Well, "new" meaning "4 years ago" [01:38] We need to discuss it with Ubuntu (to convince them to use milestones instead), but I think we can probably fairly sensibly just remove conjoinment. [01:42] Is there a function around to mash a string into valid_name? [01:43] not that I know of [01:44] Ah, sanitize_name. [01:46] wgrant: well, they don't have the conjoined layout [01:46] wgrant: but we could render them nicely regardless [01:46] lifeless: They are not meant to exist, so rendering them nicely probably isn't desired. [01:46] But I filed a bug on this a long time ago. [01:47] wgrant: I think we're either completely agreeing, or talking past each other [01:47] someone should write a script to add the missing bugtasks [01:47] wgrant: my point is that conjoined masters don't express anything that single bugtasks can't [01:47] Right. [01:47] wgrant: there is a /behaviour/ change needed when the default series is altered [01:48] possibly [01:48] anyhow - milestones are orthogonal I think [01:48] thumper: or we could just simplify things and get rid of the redundant data [01:48] lifeless: They're not orthogonal. [01:48] lifeless: Do you know how Ubuntu uses series tasks? [01:49] for guiding developer effort in backports [01:49] not 'backports' [01:49] Hah. [01:49] No. [01:49] I know what I mean [01:49] That's one use. [01:49] They are used for security updates and SRUs. [01:49] yes [01:49] But also for indicating release criticality. [01:50] Targetting something to the dev series makes it release critical. [01:50] thumper: wgrant: I am exhausted, but I can see that this conversation is a sequel to my predicament from last week. thumper the rules and sample data are bad. We may have fixed the factory in the last 12 hours. Have fun storming the castle [01:50] wgrant: so, conjoined masters are not needed for that [01:50] lifeless: Eeeeh. [01:50] lifeless: If there is no DSP task, then the bug won't show up in most listings. [01:51] wgrant: that is - and always was - a bug in those listings [01:52] our concept of 'bug target' doesn't map directly to obvious queries [01:53] if we fix that: e.g. search on distribution=1 or distroseries = distribution.development_focus - a lot of pain just disappears [01:53] Right. [01:54] make it so [01:54] arguably we should search on distribution=1 or distroseries in (distribution.distro_serieses) [01:54] that would be a behaviour change [01:54] my point here is that we can, without changing behaviour, eliminate conjoined bugtasks [01:54] make the bugtask table smaller [01:55] and make a bunch of code cleaner [01:55] Behaviour is still slightly changed. [01:55] But I agree fully. [01:57] I suppose I should go and get my kids from school [01:58] * thumper wanders off [01:58] and that using milestones, or not, is a different discussion, *because* we could do this separately [01:58] wow [01:59] 252 Time Outs [01:59] even *with* that search issue [01:59] Nice. [01:59] Less than 1500 critical OOPSes. [01:59] and a lovely long list of page ids [02:00] 2 / 0 https://api.launchpad.net [02:00] And full listings. [02:00] Yeah. [02:00] >< wadl [02:00] === Top 50 Exceptions (total of 96 unique items) === [02:00] :( [02:00] I guess that is potentially less bounded than the pageids. [02:01] nah, oops-tools code breaks my brain [02:01] sinzui: Did you get anywhere with the TP corruption? [02:01] We should be just below 1000 exceptions tomorrow. [02:01] Er, day after. [02:01] wgrant: I did not. I wanted to talk to henning. I wondered if he knew something about changes to launchpad translations coordinators [02:02] might drop us another second tomorrow [02:02] lifeless: Go all the way to 10. [02:02] wgrant: no [02:02] :( [02:02] wgrant: 1. I believe we could fix the data with a stealthy add remove using the UI, but I want to know how this state happened first [02:02] sinzui: The SQL breaks my brain. [02:02] lifeless: Isn't this our third drop in like a week? [02:02] sinzui: I assume there must be a bug there. [02:02] StevenK: Only the second. [02:02] Hence me wanting to go to 10. [02:03] wgrant: a second at a time mitigates the risk of multiple pages right on the boundary and not clear in the ppa [02:03] ppr [02:03] bah [02:03] Meh [02:03] oh, I can get jcsackett involved. It might bring back traumatic memories, but I think it will make him stronger [02:03] wgrant: this strategy is working, so I take your 'Meh' and raise you 'Meh' [02:04] lifeless: Meh. [02:04] 147.19s OOPS-1900K1313 Person:+contactuser <> [02:04] Someone's trying to talk to big teams. [02:04] wow 973 OOPS-1900M785 Product:+download [02:04] wgrant: yes, the 'spam me' button is alive and well :> [02:04] K, M, C, Q are all different appservers? [02:04] StevenK: yes [02:04] 2 ProgrammingError: permission denied for relation revisionauthor [02:05] In branchmail. [02:05] Huh. [02:05] lifeless: Okay, but we don't really care which appserver, right? [02:05] StevenK: if you suspect GIL contention [02:05] StevenK: then you will want to lookup and see if its a 1/2 thread server, or a 4-thread server [02:05] in lp-production-configs [02:05] We have had a mechanism for queuing emails to users for 2.5 years and it has exactly one call site, person.merge. I think we can speed up a lot of registry and answers reusing it [02:06] I think we can speed up some of soyuz by also reusing it [02:06] sinzui: +1 without looking at the code ;) [02:07] StevenK: Oh? [02:07] StevenK: Only queue accepts send email. [02:07] And then only two per package. [02:07] It's not great, but it's like 0.1% of the time. [02:07] POTemplate:+index 12529 37194.36 8.03 [02:08] translations will still have a high 99th percentile I think [02:08] till that page gets improved [02:08] PreviewDiff:EntryResource has a 54 second 99th percentile >< [02:09] Distribution:+edit - 33seconds [02:09] We need to use on the timing information if it exceeds the 99% [02:11] https://code.launchpad.net/~wgrant/launchpad/bug-456616/+merge/53556, anyone? [02:12] wgrant: do you mean by the uid ? and if its over our target 99th percentile? [02:14] lifeless: The one by the UID, but anything over the current 99th percentile. [02:14] That way it decreases without everything flashing. [02:14] With red 72pt bold. [02:15] wgrant: if you want to do it, I can describe how I'd do it. [02:15] No. [02:15] wgrant: I was thinking of a background stylesheet - like demo - with the word 'SLOW' on it. [02:15] Hah. [02:15] Oh, even better, an animated GIF background! [02:15] no [02:15] We can take slow pages back to the 90s. [02:16] green on black baby [02:16] switch the stylesheet [02:16] so - seriously. [02:16] in my original concept for this, I wanted a pervasive, unintrusive message on problem pages [02:17] I was sort of thinking an FF which highlights the time in red if it's exceeded. [02:17] the watermark idea was one way to do it [02:17] But your suggestion might be nice. [02:17] lifeless the background image is really easy. weather configured or using a onload event (life the query count), we can add a class to the body like we do private. huwshimi is landing a replacement for the private background soon too [02:17] yes, a new FF which sets the time to alert-if-exceeded would be the implementation I'd choose [02:17] sinzui: can you stack background images ? [02:18] no [02:18] sinzui: so I wouldn't want to break private pages and make them appear public [02:18] sinzui: how is that handled on staging etc? [02:18] but we also have the maincontent div which allows us to have multiple backgrounds with a little more effort [02:19] sinzui: if anyone wants to wire this up, I'd be ecstatic; if they need help or input on getting a value in the main template to trigger this - I can do that in ~ 5 minutes, and its -dead- easy. [02:19] sinzui: lifeless: css3 does actually give us the ability to have multiple background images, but they need to be defined differently. [02:19] lifeless: config.is_demo is checking in the base-template and it adds a class to body [02:19] like the time in the top right, it will need to be onload based. [02:44] lifeless: at least this one is solved :-) 2 / 0NullBugTask:+index [02:44] thumper: That class no longer exists. [02:45] It's unfortunate that it didn't time out much on its last day. [02:45] wgrant: exactly. Solved. [02:45] Heh. [02:49] wgrant, do PPAs not generate kernel udebs or something? [02:49] cody-somerville: AFAIK they should. [02:51] wallyworld_: ping [02:51] thumper: hi. hope things are ok with your friend [02:52] wallyworld_: he is feeling much better [02:52] wallyworld_: got mumble? [02:52] yep. give me a sec [03:32] poolie: what was the search you were making that timed out? [also, my preference is for new bugs please] [03:33] its much easier to dup than to split [03:34] poolie: bugsearch, for instance, has about 10/20 different components, all of which can be a problem, so theres no particular reason to think that a timeout in one search is meaningful for a timeout in a different search, unless its hitting the same code path with the same data [03:36] surely the search is in the oops? [03:37] for EOFError in /bzr iirc [03:38] poolie: oopses take up to 20 minutes to get visibility [03:38] 2.45 seconds for me : https://bugs.launchpad.net/bzr/+bugs?field.searchtext=eoferror&orderby=-importance&search=Search&field.status:list=NEW&field.status:list=INCOMPLETE_WITH_RESPONSE&field.status:list=INCOMPLETE_WITHOUT_RESPONSE&field.status:list=CONFIRMED&field.status:list=TRIAGED&field.status:list=INPROGRESS&field.status:list=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_pa [03:39] (if that didn't truncate) [03:41] Did [03:41] &field.has_no_pa [03:41] https://bugs.launchpad.net/bzr/+bugs?field.searchtext=eoferror [03:45] wgrant: hah, I had just started to look at 607954 [03:45] wgrant: enjoy :) [03:45] lifeless: Just finishing it up now. [03:45] The page is actually very lucky. [03:46] In most cases now it avoids another three queries per diff. [03:46] Because it's already preloaded them accidentally. [03:46] hahahahah [03:47] Oh, I hate circular imports. [03:47] ok, all soyuz errors are clearly visible in the new report [03:47] you should try circular servers some time. server A depends on server B being alive and working; B depends on C; C depends on A. Boom. [03:47] if it happens persistently i'll file a new bug [03:47] poolie: thanks [03:47] it does not seem as regular as it was yesterday [03:48] spm: we do that to keep you on your toes [03:48] i guess the one off will show up in the oops report? [03:48] poolie: we're down 50% on timeouts from yesterday [03:48] poolie: yes, it will [03:48] poolie: and 75% on timeouts from the week before [03:51] way to go [03:56] * StevenK looks for the critical bug graph [03:57] StevenK: http://webnumbr.com/.join(launchpad-oops-bugs.all,launchpad-timeout-bugs.all,launchpad-critical-bugs.all) ? [03:59] Huh. [03:59] I was going to say that that expression could be simplified. I didn't realise it was a real URL. [04:00] StevenK: its about to jump [04:00] StevenK: as I'm going to file bugs for all the timeouts [04:00] Heh [04:01] lifeless: Requested the timeout drop yet? [04:01] wgrant: *tomorrow* [04:01] wgrant: I want a full day without search index fail etc [04:02] Ah, right. [04:02] wgrant: so when the oops reports come out tomorrow, I'll get spm to drop it, and awaaaay we go [04:03] wgrant: I assure you, I'm ask keen as you are to get the cap down low [04:03] s/ask/as/ [04:04] Heh [04:04] bugs domain was 50% of our page renders last month [04:04] 20M/40M [04:05] lifeless: does that exclude robots? [04:05] mwhudson: its apparently bong [04:05] Or API scripts? [04:05] lifeless: :) [04:05] https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-monthly-categories.html [04:05] API scripts are excluded. [04:05] All Launchpad except operational pages [04:05] (? I don't trust that number, though. [04:05] API [04:05] ^https?://api\. 74901732 [04:05] Launchpad [04:05] . 164821045 [04:06] so, raw count is 164M [04:06] api 75M [04:06] but 'non opstats' is only 40M. [04:06] wtf [04:06] Yeah.; [04:06] That's what I'm wondering. [04:06] thus - 'bong' [04:06] fraaaaaaaanics [04:06] Since the daily non-opstats counts are much higher than 39000000/30 [04:06] And Bugs - search is zero [04:06] Which amuses me [04:06] thats a whack reges is all [04:07] lifeless: Tautology [04:07] ah [04:07] down the bottom [04:07] All launchpad except opstats [04:07] (? which is santer than the non-haproxy inclusive one >< [04:07] -no-freaking-idea [04:07] But look at daily categories.html [04:07] Registry is bong, too [04:07] 6.5m non-opstats-non-haproxy [04:08] * StevenK deletes shipit [04:08] Oh. But that might include API. [04:08] Yeah. [04:08] https://devpad.canonical.com/~lpqateam/ppr/lpnet/2011-03/daily_2011-03-05_2011-03-06/categories.html appears to have 3.2m non-API non-haproxy non-opstats requests. [04:09] wgrant: yes [04:09] wgrant: according to the daily ppr, most of our growth was scripts [04:09] Even assuming a conservative 20 working days a month, that's still 64m page views. [04:09] So where is 40m coming from,. [04:09] ah, I see what you're cross referencing [04:09] yes, its bong [04:36] lifeless: I'm looking at the feature flag controller. The handling of per_thread.features looks pretty messy — so I'm wondering if maybe there's some good reason why we're manipulating per_thread.features ad-hoc in so many places? (If not, I'd like to clean it up) [04:37] https://code.launchpad.net/~wgrant/launchpad/bug-607954/+merge/53567, anyone? [04:37] wgrant: oh, I'm OCR so I might as well === jtv changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: StevenK, jtv | https://code.launchpad.net/launchpad-project/+activereviews/ [04:37] jtv: the right way would be a participation annotation, but - meh [04:38] lifeless: what's a participation annotation? [04:38] maybe I mean interaciton annotation [04:39] You probably want the interaction. [04:39] Participation could work too, but I like to pretend that participations don't exist. [04:41] jtv: anyhow, it doesn't seem like an overhaul here would help you at all [04:41] Would improving that be complicated in any way if I replaced all the various reads from and writes to per_thread.features with calls to the existing getter and a new setter? [04:41] jtv: and you were concerned about rabbit holes the other day [04:42]