[00:01] wgrant: yes, we should celebrate this joyous occasion. [00:02] Although the branch is a bit of a read :/; [00:10] LOSA ping: Can someone please run http://paste.ubuntu.com/511984/ on prod? We seem to have corrupt more builds like the ones we found when trying to open Natty. [00:10] wooo [00:10] We had to fix those quickly, so the evidence was destroyed. [00:11] So it is probably... good... that there is more breakage. [00:11] FSVO 'good' [00:11] wgrant: (4510 rows) but they're all looking like " 4789 | | |" [00:11] (missing BuildQueues in both cases) [00:11] Er. [00:11] Did I miss a join somewhere... [00:12] Whoops. Left out the WHERE clause in that version of the query. [00:12] WHERE sourcepackagerecipebuild.id IN (4256, 4257); [00:12] *Two* rows. [00:13] ha [00:14] id | id | id | id [00:14] ------+------+---------+---- [00:14] 4256 | 4255 | 6311021 | [00:14] 4257 | 4256 | 6311022 | [00:14] wgrant: ^^ [00:14] Precisely as I suspected. [00:14] Thanks. [00:14] Now to work out why we have various builds without BuildQueues. [00:20] How I hate this model. [00:22] thumper: https://devpad.canonical.com/~stub/ppr/lpnet/latest-daily-pageids.html [00:24] How far do buildd-manager logs go back? [00:29] A long while [00:29] * StevenK goes for breakfast before his stomach breaks out on its own [00:34] rooooar [00:34] holy cow [00:34] hwsubmission || 525012.40 tuples/sec [00:35] 'release time anyone' ? [00:35] except, its not writes. hmmm [00:36] Yeah, that can't really be writes, as it would go through the last released count of Ubuntu users in 24 seconds. [00:36] wgrant: s/count/estimate/ [00:36] :-P [00:37] elmo: True that. [00:52] thumper: -> https://bugs.edge.launchpad.net/bzr/+bug/93609 <- poolie: [00:52] <_mup_> Bug #93609: Better error messages for bzr lp:// [00:53] wgrant: Do you need logs? [00:55] StevenK: I'd like to see what the logs have to say about BinaryPackageBuilds 1968526, 1968544, 1969032, and SourcePackageRecipeBuilds 4256, 4257. [00:55] Because they somehow ended up NEEDSBUILD without a BuildQueue. [00:55] wgrant: How far back are we talking? [00:55] StevenK: The BPBs are a little over three weeks ago. The SPRBs 11 days ago. [00:57] In two months I will be able to stop pestering people for logs, yay. [01:09] wgrant: in that losas aren't people? or??? [01:09] Heh. [01:36] losa ping [01:37] I need someone to delete "hard_timeoutpageid:BugTask:+index15000" from https://staging.launchpad.net/+feature-rules [01:39] lifeless: you have been unfeatured [01:41] thank you [02:08] I *so* want to not get mail for invalid bug tasks. [02:09] There are two types, though. [02:09] Of Invalid task. [02:10] lifeless: do i also need to ask for a separate review in #launchpad-reviews for that menu branch? or is your stamp of approval enough? [02:12] wallyworld: IIRC userdict provides better hook points than dict, though its not really relevant here [02:12] wgrant: bad and worse? [02:12] wgrant: I've clickly clicked for you [02:12] bah [02:12] wallyworld: [02:12] lifeless: yo [02:13] ^ 3 lines [02:13] ack. so i'll leave the code as is... [02:13] lifeless: "This isn't a valid bug", and "This isn't a valid bug here" [02:13] uhm [02:14] I think you're conflating task and bug [02:14] something like : a bug is invalid if all tasks are invalid [02:14] It is necessary to conflate them. [02:15] ‽ [02:15] We have only per-task statuses. [02:15] But, as you say, a bug could be considered invalid if all its tasks are. [02:15] The interactions with notifications may depend on both. [02:16] well, for my needs, if I'm structurally notified via an invalid task, i don't want to be. [02:16] But what if the user is arguing that the bug is valid? [02:16] You do want to be notified. [02:16] hi lifeless [02:16] 5-digit bug hey [02:16] i agree that looking at the cases is good [02:16] wgrant: can't they toggle it to new? [02:17] lifeless: That's impolite. [02:17] poolie: hey, so, thumper and I were talking. [02:17] poolie: I promised to file a bug describing a design to get better messages out with hopefully backwards compat [02:17] poolie: and suggested that spiv would be a great person to look at the detail. [02:17] poolie: when I wwent to file it I found such a closely aimed bug that I reused it. [02:18] wow [02:18] 1026 / 119 Distribution:+ppas [02:18] ^- unhappy pages R us. [02:18] Yeah, known 8.4 regression. [02:18] Not sure of the details. [02:18] yeah [02:18] But there are a few bugs on it. [02:18] count(*) [02:18] we've dupped them all I think [02:19] bad query is listed in the remaining active [02:19] yep, i thought a bug would be good too [02:19] and that's it [02:20] I don't see the bug. [02:20] * wgrant searches other projects. [02:20] https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.tag=timeout [02:20] wgrant: ^ [02:20] Ah, on launchpad. [02:20] fixed [02:21] such a nuisance [02:21] I was trying to fix it. [02:21] But it kept using the non-AJAX form. [02:21] Something odd is going on. [02:21] you clicked too quickly [02:21] gotta wait for yui to initialize and overlay [02:21] I thought so. [02:21] But apparently not. [02:21] odd indeed then [02:21] I can wait a few seconds after the status picker works, and the project one still doesn't. [02:22] oh, it never has [02:22] Hm? [02:22] there isn't an ajax project picker is there? just the drop-down if you click on the expander [02:22] There is an AJAX project picker. [02:22] I've never seen it :P [02:22] It's often buggy, but it usually at least appears. [02:36] what dor str(Person) do (when Person is a Launchpadlib object) [02:41] lifeless: IIRC str()ing an Entry will give you its URL. [02:41] blah, thanks. [02:42] Ah, now I see the relevance. [02:43] wgrant: ? [02:44] The qa-tagger bug. [02:44] ah yes [02:44] I really should add that to my sieve script, since it ends up in my inbox. [02:44] open sourcing by incremental pastebin :P [02:45] I learnt a lot about LP through dogfood back when it had public tracebacks :P [02:45] But then Julian fixed that :( [03:03] aw [03:03] i wonder why not enable them? [03:04] They leak data. [03:04] in variable values etc? [03:04] Although not much worse than API exceptions can do. [03:04] The most obvious one is object reprs in Unauthorized exceptions. [03:04] But there are probably others. [03:09] As an odd question, I wonder why ohloh doesn't have any updates from Launchpad branches since Jan [03:09] really? [03:10] you could mail them [03:10] I'm checking again [03:10] Hmm. I have a PPA build finished 13 minutes ago and still not published [03:11] Last commit it shows was 4 months ago, but the code metrics seem much older [03:15] 18 minutes :-/ [03:16] The publisher runs every 20 minutes, so that is the worse case, right? [03:16] The *PPA* publisher used to run every 5 minutes, last time I was told [03:18] maxb: Which PPA? [03:19] bzr-beta-ppa/ppa - it does seem to have just published now [03:59] lifeless, ping, if you have a moment, I am in need of some help with a zope-related implementation question [04:02] or thumper, if you have a moment to spare? [04:04] mars: whazzup? [04:04] Hi thumper, I have a question about storing state for the duration of a Zope request [04:04] right... [04:05] and what is the question? [04:06] The profiling code needs to store the active profiler information for the duration of the request. I could throw that information onto the request object itself, or take the current approach of using a module-level threading.local() [04:07] One approach turns the request object into a cesspool, the other uses a sort-of-magic secret module level variable - is one approach preferred over the other? [04:07] mars: does this relate to the existing timeline code? [04:08] no [04:08] mars: my suggestion would be to do what the timeline code does [04:09] mars: the request object is already a cesspool [04:09] heh, ok [04:10] thumper, where would I find the timeline code? [04:10] mars: I think there is a part of the request object used to store "stuff" [04:10] mars: lp.services ? [04:10] right [04:13] thank you for the help thumper [04:13] mars: hi [04:13] mars: webapp.adapter [04:13] mars: as thumper says, see what lp.services.timeline.requesttimeline does [04:14] (which is to shove it into webapp.adapter and cry with XXX's.) [04:14] lifeless, cool, thanks [04:14] mars: doesn't the profiler already work though? [04:14] [I'm interested in what you're hacking on] [04:15] lifeless, the profiler worked when using lazr.config. The setting was static for the duration of the request, so profiling was either on or off [04:15] mars: are you working on flags to enable profiling? [04:16] mars: if so, i think it still needs to be either on or off, just evaluated earlier. [04:16] lifeless, but feature flags are set and cleared with request events, and lo, so is profiling. If the features get nuked before the profiling end hook is run, no profile is generated [04:17] I am dealing with an event processing order bug [04:17] mars: sure, but wouldn't 'just' caching the result of 'should we profile' be all thats needed (and in fact I thought that that is what it did?) [04:17] win 67 [04:18] lifeless, no, it did not cache the 'profiling is on' flag (unless you count the aforementioned thread-local storage as the flag) [04:18] lifeless, and I was asking about the best implementation for said cache [04:18] either thread-local storage, or a request.atttribute [04:19] end_requqest does this [04:19] _profilers.profiler is not None [04:19] so it caches (by virtue of using an object) [04:20] mars: I expect you'll have a great deal of trouble changing the cache implementation for profiling [04:20] until we fix two bugs: [04:20] hmm, ok [04:20] - many tests call in-request-context-code outside of a reqest-context [04:20] - scripts don't establish a request-context in th esame way the webapp does [04:21] the second point won't really affect you atm as we don't profile scripts, but the first one will hurt - a lot- [04:21] so if I change this then I'll probably trip over yet more test-related bugs [04:21] see the 'except AttributeError:' on line 78 [04:21] I'd be delighted to see the root issue here fixed. [04:21] but I don't know if you have the time budgeted for that :) [04:22] this has rotted on the kanban board already (as I am sure you and Martin have noticed). I think I'll just try and land it :) [04:22] ok, so perhaps I can help solve your immediate issue. [04:22] what is it ? [04:23] well, I think it is solved then - I just have to add a comment string or two to that thread-local storage to make it less magic [04:23] name it _active_profilers or some such obvious thing, etc. [04:24] mmm [04:24] there are no inactive_profilers [04:24] well, _active_profile then [04:24] if _active_profile.actions = None [04:25] mars: if you come up with a good name; great. Myself, I'd just add a comment to the assignment and the module dostring. [04:25] anyway, something like that [04:25] lifeless, yes, will do [04:25] lifeless, thank you for the help, and for pointing past the minefield [04:28] de nada [04:39] If I have a db-devel-based branch that will land in devel after the release, how should I propose it? With db-devel as a prereq? [04:40] propose it to devel [04:40] after the release, the diff will be normal :) [04:41] If I somehow convince it to update after the release, sure. [04:41] I guess I could just wait until after the release. [04:41] one day [04:41] Although I have to wonder why we don't merge db-devel into devel at the time devel freezes. [04:41] I suggested that on the lit ;) [04:41] Ah, I'm a bit behind at the moment. [04:49] EdwinGrubbs, pong? [04:49] EdwinGrubbs, holiday here today, sorryu [04:49] *sorry [04:56] Ursinha-afk: no problem, we sorted out some issues we were having with the qatagger. It's working now, and we opened bugs for the problems that should be fixed later. [05:10] StevenK: re. Ohloh: The import has been tried a few times, but always fails after working for a couple of weeks. [05:10] StevenK: Their bzr importer doesn't seem to be awfully good. [05:30] * wgrant requests that it be kicked. [05:49] I wonder if the regular failures are because of the frequent LP downtime. [05:50] sush :-) [05:51] Well, we are the only hosting site I know of that has regular downtime... [06:20] lifeless: http://sourcefrog.wordpress.com/2010/10/13/59/ about ssl performance may interest you [06:29] poolie: thaks [06:45] stub: question about the BranchRevision patch… [06:45] ? [06:45] The problem is too much downtime when setting a new primary key, right? [06:46] The change requires creation of a huge new unique index, even though we already have one. [06:47] So I'm wondering: why do we need to declare a primary key in the first place? Does Slony require it? [06:47] Storm does. [06:48] Storm requires us to declare a primary key _to Storm_. But does it require us to declare a primary key _in the schema_? [06:48] Ah, fair point. [06:48] Because AFAIK this whole primary key business is little more than convention, a bit of optional syntactic sugar. [06:49] "Slony-I needs to have a primary key or candidate thereof on each table that is replicated." [06:49] Hm. [06:49] Okay, what does it consider a candidate? Because we have unique index and we have not-null. [06:50] One can override it to another candidate key. [06:50] So it should be doable, yes. [06:50] (just needs UNIQUE and NOT NULL) [06:52] stub: we already have the index and constraint we need… what do you think? [06:52] jtv: Slony requires a primary key [06:53] It says it requires a primary key or a candidate key. [06:53] stub: see what wgrant just dug up—we can define a primary key there without declaring it in the schema. [06:54] I like to stick to best practices, and from the best practices section " Discussed in the section on Replication Sets, it is ideal if each replicated table has a true primary key constraint; it is acceptable to use a "candidate primary key." [06:54] Well, it's either use a candidate key or have several hours of downtime... [06:55] there's also the 'make branchrevision2' plan isn't there? [06:55] or has that been shot down for some reason [06:55] That would work too. [06:55] At the expense of a crapload of disk, I guess. [06:55] yeah [06:56] Candidate key is certainly an option, but we need to teach or maintenance scripts about it. [06:56] Everyone bring your spare disks to the DC for Operation Dunkirk. :) [06:57] There is also the ignore-it-until-we-drop-branchrevision route. There might be faster, smaller and better solutions than the big table in the RDB. [06:57] How long until we run out of rows? [06:57] yeah, that's probably a reasonable approach [06:58] wgrant: it's at 600 million ish and runs out at 2 billion [06:58] so, no massive rush [06:58] Oh, I didn't realise it was only 600 million. [06:58] What's the next id though? [06:58] The table is pretty much INSERT only [06:59] good question, but probably not noticeably more than the row count [06:59] Except on branch deletion and overwrite, I guess. [06:59] Highest id is 757259996 [06:59] And aborted transactions, and maintenance. [06:59] True. [06:59] So that's okay. [07:00] according to the graph (which is only an estimate i think) we've added 300 million ish rows in the last year [07:00] Well then, let's just take one bit off the id for now and save 1/32nd of the space :) [07:00] jtv: don't muck about eh? drop database. [07:01] at last we'll have some disk space free [07:01] Also, since the only thing that inserts rows into that table is the branch scanner, we could do magic and have everything live while we do the maintenance and only have the branch scanner disabled. [07:02] stub: I don't suppose there's a way to make the new index be created concurrently like you can with a regular index? [07:02] The new primary key index? No. [07:04] And the *other* other alternative is to talk to the pg-hackers and update the system catalog to promote an existing unique index to the primary key index. [07:06] oooh getting dirty now [07:08] I keep getting this when I run tests on any branch locally: [07:08] Test-modules with import problems: [07:08] lp.services.mailman.tests.test_lpmoderate [07:08] (And one other) [07:08] It can't find blah.blah.mailman.monkeypatches, or something like that? [07:09] That error mysteriously vanished when I reinstalled and did a clean rf-setup. [07:38] wgrant: Ah ha [07:45] wgrant: Just to be clear, you mailed ohloh and asked them to kick their LP import? [07:46] StevenK: I posted in the forum as they suggest. [07:46] They seem to kick them pretty quickly. [07:47] I wonder if they'll have to do another full re-import (which takes like a month) [08:02] stub: http://paste.ubuntu.com/512180/ is what I have so far—running tests now [08:03] jtv: That looks rather evil. [08:04] wgrant: stop, you're making me blush [08:04] Less evil, however, than an email I just received from my university. [08:04] ? [08:04] It was apparently generated by a recent Microsoft Office product, and is a fairly short largely textual email. The text/plain version is <1KiB. The text/html version is 14KiB, contains 53 XML namespace declarations, and 6KiB of CSS. [08:05] It is evil, but also something I was considering :) Certainly needs to go past upstream though - everything that needs to be updated may not be obvious. [08:05] stub: absolutely—I only found out about the pg_index part because \d didn't pick up the change. [08:06] wgrant: that's actually pretty good. Last time I did that sort of analysis, it went something like 10:1 for some very basic HTML cleanups such as not declaring the font for every paragraph, and then it was another 10:1 for the cleaned-up HTML against the text. [08:10] jtv: 53 NAMESPACES. [08:10] That may be a tad on the high side, yes. [08:11] They probably just couldn't be bothered to see which ones were really needed. [08:11] Indeed. [08:11] After all, who doesn't want flashing Microsoft logos with a half-functional copy of their website embedded in their letters? [08:11] Or something. [08:23] lifeless: When you say 'the OOPS rate has tripled', is that ongoing or are you referring to the spikes during and just after the upgrades? [08:27] stub: its ongoing [08:27] Hmm... wasn't aware it was that much. [08:28] daily oops reports - we were at a few hundred, we're at 1.mumble K [08:28] its noisy too, not flat [08:28] +ppas and the person picker could well make up most of that, though. [08:28] Or maybe we have just tripled our throughput - the DB load is still half what we used to see and I don't know why ;) [08:29] stub: we should normalise by requests/day - I have a bug open for that on the oops summaries [08:29] ok, tuolumne is doing some -weird- aggregation here - https://lpstats.canonical.com/graphs/OopsEdgeHourly/20100914/20101014/ [08:30] lifeless: Oh... and we might be getting a lot of OOPS reports from newly enabled cronscripts, including the doubling up of OOPSes from some scripts. If we just look at timeouts though, the cronscripts won't contribute to that. [08:30] stub: the upgrades - https://lpstats.canonical.com/graphs/OopsLpnetHourly/20100914/20101014/ [08:32] stub: I mispoke, I should have said 'timeout rates have tripled' [08:33] Yer, and those spikes helpfully making the rest of the graph unreadable :-P I agree it looks like hard timeouts have increased significantly, and user generated errors too [08:33] stub: the ppa search is a major one [08:33] So this was for pre-release, so we can't blame increased traffic from 10/10 [08:33] 1000 a day [08:34] I haven't followed the PPA search issues - do you have a bug # handy? [08:34] https://bugs.edge.launchpad.net/soyuz/+bug/659129 [08:34] <_mup_> Bug #659129: Distribution:+ppas timeout in PPA search [08:34] I tagged it dba :) [08:34] (only this morning though - not digging at you) [08:38] isn't SourcepackagePublishingHistory a big table ? [08:40] Maybe 10-15 million records? [08:40] yup [08:40] SELECT DISTINCT archive FROM SourcepackagePublishingHistory WHERE status IN (2, 1) [08:40] Is that the package cache updater? [08:40] thats the ppa search query [08:41] That can't be the whole query. [08:41] no, its a in clause [08:41] wgrant: see the bug I linked [08:41] But still, that's indexed. [08:41] Time: 1265.807 ms [08:42] 6690 rows [08:42] its not an obvious cause for 15 second queries, but its a tad slow. [08:42] whats annoying is that the query in the OOPS runs in 1.5s on staging [08:43] so I can't reproduce [08:54] stub: how long does it take on the prod slaves? [08:55] Just over 2 seconds [08:57] >< [08:57] on all ? [08:57] Yup. I can run the query and return the 6733 results to Bangkok in 2 seconds. [08:58] damn [08:59] 2.7 seconds on a cold slave [08:59] actually hitting up https://edge.launchpad.net/ubuntu/+ppas?name_filter=munin times out though [09:00] That would be doing a different query, or is the filtering done client side? [09:00] slightly different yes [09:00] (Error ID: OOPS-1747ED330) === almaisan-away is now known as al-maisan [09:06] waiting for that to sync [09:06] Hello [09:07] hi === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [09:12] morning jml [09:13] grrr oops syncing :< [09:13] lifeless: good morning [09:17] mwhudson: thanks for doing the branch-distro thing [09:18] jml: it's probably still failing intermittently [09:18] mwhudson: I had been meaning to ask, the failures in the bug report, do they require manual intervention? [09:18] jml: no [09:19] actually seems to be going currently [09:19] jml: well, the script needs restarting, that's all [09:19] mwhudson: what's the intermittent failure then? [09:19] mwhudson: ahh... that's manual enough for me :) [09:19] jml: intermittent, as in, until the losas notice [09:19] lifeless, wgrant: Huh. SourcepackagePublishingHistory is only 1.2 million records. I thought it was bigger too. [09:20] Think I must have been confused with BinarypackagePublishingHistory, which is 11 million [09:20] mwhudson: I can't tell you how keen I am to make this much a fire-and-forget operation for next time [09:21] jml: well, you saw my advice on that front i guess [09:21] mwhudson: rather than a multi-person project [09:21] jtv: Hi. Edwin asked me to check with you about the bug you're still qa-ing. Any news? [09:21] mwhudson: yeah, thanks [09:21] gmb: yes, the build farm is now finally up and running. And now something else is broken. [09:22] Ah. Doubleplusungood. [09:22] gmb: I'm leaving out some other breakages for brevity. [09:22] losa ping - where is OOPS 1747ED330 ? [09:23] lifeless: erm, context? [09:23] jtv: Anything i can do to help? [09:23] stub: Er, yes, I mixed them up. [09:23] Oops. [09:23] mthaddon: sorry; we're trying to track down the highest timeout since pg8.4 [09:23] mthaddon: thats an OOPS I triggered 24 minutes ago [09:23] but lp-oops isn't showing it [09:23] lifeless: and what do you mean by "where is"? which server is it on, or you can't find it? [09:24] mthaddon: I guess I mean 'has it rsynced across, has the injection into lp-oops completed' - I'm very vague about where to look to figure this out myself :( [09:25] oh great, edge4 and edge5 are using the same OOPS prefix [09:25] lifeless: you should familiarise yourself with the OOPS prefix system I think - very useful for tracking this kind of stuff down [09:26] mthaddon: I'll file a bug about the fact the overlap wasn't detected [09:26] mthaddon: I rewrote part of it recently ;) [09:26] mthaddon: but the operational deployment is a bit different [09:27] That 'stable has test failures' in the topic is old now, right? [09:27] I don't know :) === stub changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.10 | PQM is Release-Critical; devel is closed (Release manager: EdwinGrubbs) | firefighting: - | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting [09:28]