[00:08] wallyworld ping [00:08] lifeless: hello [00:08] wallyworld: we're blocked on qa for Revision 12671 can not be deployed: needstesting - [00:09] lifeless: thanks. i'll look now [00:09] wallyworld: thanks! [00:09] StevenK: you have one as well - or is it behind a featureflag? [00:22] lifeless: on qastaging it has a problem which is makes deployment a no go (not a blocker locally). it should be fixed by a branch already in ec2. can we wait till it goes though? [00:23] wallyworld: so it would break prod ? [00:23] lifeless: yes [00:23] tag it bad-commit-NNNN [00:23] ok [00:23] I've fine with waiting for the followup branch thats already playing [00:27] bac: I think the test name changed; to run all the tests I've been using just "xvfb-run ./bin/test --layer=RegistryWindmillLayer -c" [00:27] oops, wrong chan [00:43] lifeless: It appears we can't get oops info for javascript at this point either. Would you be happy with me trying to land the ajax log branch as is, and then we can add the url and oops as an improvement when they become available? [00:44] huwshimi: I love incremental improvements [00:44] huwshimi: we don't get the http headers? [00:45] lifeless: You mean for the url or the oops? [00:45] huwshimi: the oops - its a response header [00:46] lifeless: We get the headers, but I didn't see anything about an oops... [00:46] huwshimi: its an X header when an oops occurs [00:46] lifeless: Let me take another look [00:46] I'm just digging up the header id [00:47] lib/lp/app/javascript/client.js:690: if (o.getResponseHeader('X-Lazr-OopsId')) { [00:47] lib/lp/app/javascript/client.js:691: server_error = server_error + ' OOPS ID:' + o.getResponseHeader('X-Lazr-OOPSid'); [00:49] huwshimi: ^ [00:49] lifeless: OK great [00:51] lifeless: Ugh, ok you're right, it's there [00:54] for extra credit [00:54] clicking on the oops id could take one to pad.lv/OOPS-xxxxx [00:54] totally optional of course [00:56] lifeless: It's trivial to add [01:00] lifeless: I've added all this under the "visible_render_time" flag that you added. Is that ok or is it better to have a separate flag for this? [01:02] huwshimi: thats ideal [01:02] huwshimi: simplest thing, can always refactor [01:02] lifeless: OK great. Do you want to review this branch? [01:03] huwshimi: I'm not a gun js guy, but I'll certainly give it a shot [01:03] lifeless: Sure. Thanks [01:03] whiskers are so ticklish [01:08] lifeless: Let me translate: "feed me" [01:11] wgrant: more 'you make a nice bed' [01:11] face aimed at my face [01:14] wow [01:14] cve:+index templates are smoking something [01:14] lifeless: Most old templates are :/ [01:14] no really [01:15] eg? [01:15] show the bug [01:15] format a link to it [01:15] then include the bugtasks-and-nominations-view for the bug [01:15] lifeless: https://code.launchpad.net/~huwshimi/launchpad/ajax-time/+merge/55257 for when you get to it. [01:16] wgrant: ^ - I'll mentor you [01:16] lifeless: k [01:19] :( structure [01:20] huwshimi: You can't put that JS somewhere other than template, with a call in the template to activate it? [01:20] JS in templates makes me very sad; "structure" doubly so. [01:23] lifeless: I'm not sure. I would like it if we had not javascript in templates (and I think we should move towards that). I just don't know where it should live. [01:26] wgrant: I meant that for you ^ [01:26] The log also highlights if the AJAX event takes more than 1 minute. [01:26] That's quite a long request! [01:26] However, this is really nice. [01:27] Particularly with the OOPS ID stuff. [01:27] james_w: Ping! [01:27] huwshimi: I was suggesting 1 second, not 1 minute, for slow ;) [01:27] lifeless: Oh right [01:27] lifeless: That's pretty short! [01:27] huwshimi: goal is 99% completing withing that time [01:28] huwshimi: perhaps start at 5s [01:28] lifeless: I think it's great [01:29] actually I think I did make it 1 second [01:30] wgrant: Sorry the description is wrong. The code is for 1 second [01:30] wgrant: My brain failed me when I was writing the description [01:31] huwshimi: Ah, that is less completely crazy. Excellent. [01:31] I think 2-3s is probably more sensible at the moment, but I guess we'll see. [01:31] Could FF it, but probably YAGNI... [01:33] wgrant: Yeah, lifeless suggested 5 seconds. I don't mind changing it. [01:36] wgrant: Also if you know of a better place for the javascript to live I can do that as well [01:41] hi StevenK [01:43] james_w: Hi. :-) [01:44] james_w: I've been dealing with python-debian for the derived distros stuff -- did you know that the BaseVersion class in debian_support isn't strict enough with parsing versions? [01:44] no, I did not know that [01:45] james_w: The regex allows a : in upstream version, but that's only allowed if there is an epoch [01:45] erm, eh? [01:46] james_w: Heh, have I lost you? [01:46] huwshimi: Reviewed, with a few comments. [01:46] I'm trying to work out if what you said is a tautology or not [01:46] wgrant: Thanks, will take a look [01:47] james_w: 'foo:bar-1' isn't valid, but '1:foo:bar-1' is. [01:47] ah, ok [01:47] epoch can only be digits [01:47] james_w: Right. I have a patch: http://pastebin.ubuntu.com/586681/ [01:48] huwshimi: The YUI IO events are more limiting than I thought :( [01:50] lifeless: Yes, r12677 is FF-protected. [01:50] lifeless: I can mark it qa-untestable, if you wish [01:51] StevenK: up to you [01:51] StevenK: we've a bad commit to rectify (its landing via ec2 now) before we can deploy [01:51] Once I've shaken out more of the stuff on dogfood, I'll be QAing it there, for the short-term [01:51] StevenK: long as your commit is either known-bad, or deployable in the next 4 hours, we should be fine [01:52] lifeless: Which is the bad commit, btw? [01:52] lifeless: Next 4 hours? That's not even enough for buildbot. [01:52] wgrant: fuzz [01:52] StevenK: wallyworlds security fix [01:52] StevenK, I'm very glad you didn't try and fix that bug using regexes :-) [01:52] StevenK, hunks 2 and 3 look good to me, but I don't think I like hunk 1 [01:53] james_w: Which would have made the regex about 4 times more complex. I'm masochistic, but not *that* much. [01:53] StevenK, also, I think the comment should be more explicit about what it is checking for, and the error could be clearer about the problem too [01:53] StevenK, aren't you a perl guy though? ;-) [01:54] Haha [01:54] james_w: Hunk one is a seperate, but related problem -- the code currently throws a ValueError if the changelog you parsed contains an invalid version [01:56] james_w: And I've fixed the comment. [01:59] james_w: The meat of the patch is hunks 2 and 3. Hunk 1 is at best, a workaround, since I don't want invalid versions in my list of versions. [02:02] StevenK, I think that the issue with hunk 1 is better solved at a different level, so I'd leave it out. [02:02] StevenK, please send the patch as a bug report to Debian and hopefully someone will commit it [02:04] james_w: And solve it in a seperate patch? [02:04] StevenK, what is the calling code that falls over a ValueError? [02:04] presumably it involves a loop over .blocks? [02:05] ._blocks, yes [02:05] james_w: get_versions() [02:05] ah, I forgot about that [02:05] don't use that then I would say :-) [02:06] Haha [02:06] james_w: But it's ._blocks, it isn't mine to touch :-P [02:06] currently the library is designed to give you good data or nothing [02:06] I think it's public in newer versions [02:06] Right [02:06] what library are you discussing? [02:07] python-debian [02:07] I was slightly naïve in my estimation of how much crap was in all the changelogs in Debian/Ubuntu though [02:07] in old versions etc. [02:07] * StevenK is finding out [02:08] james_w: Crumbs, do you know how long it's been since I filed a bug on Debian :-) [02:11] MIA! [02:13] whoa [02:13] 366 Time Outs [02:13] haproxy reconf FTW [02:14] The BugTask:+index change is striking. [02:14] * lifeless blames making things more efficient [02:15] We've 25% fewer timeouts than Sunday. That is pretty nice. [02:15] james_w: Well, ... duh [02:18] Gaaaaah [02:19] ShipIt needs to die. [02:19] ? [02:19] EmailAddress handling is impossible to do correctly. [02:19] Because they are linked to both accounts and persons. [02:20] So you may have an email address without a Person because the user used ShipIt. [02:20] You can't then create a properly unactivated Person for that email address, since the email address is active. [02:22] well, we know the email is valid [02:22] whats the issue [02:22] We are creating an active Launchpad person for somebody who has never used Launchpad. [02:23] join the dots for me [02:26] lifeless: A user creates an SSO account and requests CDs from ShipIt. The parasite creates an Account and an EmailAddress in the LP DB. [02:27] The user performs some action visible to LP. This could be maintaining a package in Debian, commenting in an upstream bugtracker, etc. [02:27] LP tries to create an unactivated Person for that email address, so it can be referenced in the DB. [02:27] But the EmailAddress is already active, and just needs to be linked to the Person. [02:28] So the Person is now activated, and the user apparently uses Launchpad. [02:34] wgrant: in other words 'person activation is implicit not explicit' [02:36] lifeless: Yes. [02:36] sinzui disagrees that that is a bug. === jamesh_ is now known as jamesh [02:45] wgrant: why ? [02:45] lifeless: I don't recall. It's been a while. [02:46] wgrant: also, if you want to rotflas - lib/lp/bugs/browser/buglinktarget.py - look for except Unauthorized [02:47] lifeless: :( [02:47] yes [02:47] Easy fixes, at least. [02:48] we were disclosing private bug *existance* and generating links [02:48] Existence and linkage. [03:06] content="structure batch_navigator/@@+table-view" -> I want to check precisely what this means [03:06] its 'the view called +table-view on the object batch_navigator' rendered into the contents of this element [03:06] right ? [03:07] Yes. [03:24] fingers crossed [03:25] wgrant: what are you hacking on today [03:27] lifeless: Bug #740594 at the moment. [03:27] <_mup_> Bug #740594: ProgrammingError: permission denied for relation emailaddress < https://launchpad.net/bugs/740594 > [03:31] bah [03:31] how is the +table-view hooked up [03:31] lib/canonical/launchpad/zcml/batchnavigator.zcml? [03:34] thanks [03:34] grepping to deep [03:34] however [03:34] its not rendering >< [03:34] Grar, ensurePerson is buggy. [03:35] If you call it with an EmailAddress with only an Account, it will create a Person, but forget to link the EmailAddress to it. [03:35] Call it with that EmailAddress a second time, and it will link the EmailAddress to the Person. [03:35] \o/ [03:36] also [03:36] \o/ [03:36] batchnavigator fixed? [03:36] tal:replace [03:48] can has review? https://code.launchpad.net/~lifeless/launchpad/bug-727023/+merge/55266 [03:49] * thumper looks [03:53] thanks [03:53] lifeless: question, why create a BugListingBatchNavigator for each bug? [03:53] thumper: the page shows [03:53] bug X [03:53] tasks [03:53] bug Y [03:53] tasks [03:55] thumper: it was easy to split it up like this without triggering more queries, so I just kept the prior page layout [03:55] ok [03:56] r=me [03:56] its a bit noddy really given that there is a portlet showing the exact same stuff :) [03:56] thanks [04:23] Project windmill build #114: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/114/ [04:27] ah, started working when I put my lp keys into the .ssh folder of the virtual machine [04:46] lifeless: Thanks. https://code.launchpad.net/~wgrant/launchpad/bug-740594/+merge/55269 is a follow-up. [04:51] wgrant: r=me [04:53] StevenK: thanks [04:53] StevenK: Thanks. [05:17] * StevenK ponders how one changes a branch in sourcecode [05:19] StevenK: Get the change into the branch, fix utilities/sourcedeps.conf. [05:21] wgrant: Ah, and then toss the branch that changes sourcedeps.conf through ec2 to make sure there isn't fallout? [05:21] StevenK: Yes [05:22] * StevenK looks at updating python-debian to the latest [05:24] How long does the fetch source code stage take? [05:24] Its been running for an hour :| [05:24] What's it up to? [05:24] Depends on your connection, and if you have any of it, the phase of the moon [05:25] did we remove the special case of using http fo rlaunchpad clones? [05:25] Its branching something I guess [05:25] nigelb: But what? [05:25] lifeless: That's not update-sourcecode. [05:26] "Making local branch of Launchpad trunk, this may take a while..." [05:26] Oh. [05:26] Not update-sourcecode. I see. [05:26] lifeless: I don't see hot_products in lp-production-configs, so it seems it's gone. [05:29] nigelb: it should be about 200MB [05:30] lifeless: oh, ugh. Its at something like 90% I think [05:30] nigelb: lots of code + lots of history [05:31] when we get around to shrinking the codebase [05:31] might do a fresh tree and massively compact it [05:31] but not for $quitesometime [05:31] I wonder if there's a way to grab it without history [05:31] or make it notshow history by default [05:32] Oh. I should get around to signing that agreement I guess. [05:35] wgrant: https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/55277 [05:38] StevenK: Deleting changelog entries: wgrant says no. [05:38] wgrant: Think of it as a sync? [05:38] It was a cherry-pick from upstream anyway [05:39] k [05:39] Well, a sync + a small patch :-) [05:46] lifeless: You mind reviewing wgrant's review? https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/55277 [05:50] StevenK: Actually, any reason the ".find(':') != -1" can't be "':' in "? [05:52] I'm not sure, does that work? [05:53] I'd hope so... group() returns a str. [05:54] Right, my string manupliation stuff is dated back to Linda. [05:55] wgrant: Good catch: http://pastebin.ubuntu.com/586728/ [05:56] StevenK: Looks reasonable. [05:58] wgrant: Pushing [05:58] can has review? https://code.launchpad.net/~lifeless/launchpad/bug-741092/+merge/55278 [05:58] huwshimi: fwiw I agree with wgrants review [05:59] huwshimi: if you need a hand with any of it just shout [05:59] lifeless: Yeah, I think they were all good comments. [05:59] lifeless: Looking. [06:00] lifeless: Is this to fix the insane 7000 subscriber thing? [06:00] wgrant: I'm happy to review your review [06:00] I think we probably want to batch as well, but OK. [06:01] Ah, you mention that in the commit message. [06:01] lifeless: You drop an XXX, can the bug related to that be closed (or is it)? [06:02] It looks like it's the ancient Storm __nonzero__ bug. [06:02] I suspect. [06:02] By the age. [06:02] So it doesn't need closing. [06:02] Anyway, looks good. [06:03] StevenK: it was a bogus XXX [06:03] StevenK: never ever needed [06:03] wgrant: its to improve it [06:03] lifeless: Improve/destroy [06:04] Although I guess retrieving all of them should be fairly quick. [06:04] We'll see! [06:04] I had a glitch in there [06:05] A glitch? [06:05] I've just pushed a trivial tweak to cache the subscription list too [06:05] bah bah [06:05] gimme a sec, to undo my fail [06:05] Ah, good point. [06:06] right, pushed === almaisan-away is now known as al-maisan [06:12] StevenK: Want to mentor? [06:13] wgrant: Done [06:14] Thanks. [06:17] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1914M186 is going to be fun to fix [06:22] lifeless: Delete BranchRevision, problem solved. [06:22] wgrant: how so? [06:22] lifeless: It's probably a scanner lock. [06:22] on branchmergeproposal ? [06:23] CONTEXT: SQL statement "SELECT 1 FROM ONLY "public"."branch" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x"\\n',) [06:24] wgrant: where are you getting that ? [06:24] lifeless: End of the exception message. [06:25] Hm. Now what do I do ... [06:25] I doubt I can lp-land a python-debian PQM branch [06:25] StevenK: pqm-submit [06:28] lifeless: We're running with the old appserver configs, but a new single-threading haproxy one? [06:33] wgrant: partly [06:33] wgrant: depending on whether mbarnett did the tweak requested, we're running on 4 or 6 concurrency from haproxy, and the unaltered appserver configs [06:34] lifeless: 4 or 6 for the 12-thread servers, 1 for the 2-thread? [06:34] 4 or 6 for the 4-thread servers [06:34] 1 for the 2-thread [06:34] Oh, right, it was 12 for 4 before. [06:34] + xmlrpc at 1 per backend that is listening for xmlrpc [07:04] lifeless: First self-review \o/ [07:04] \o/ [07:05] wgrant: I took a look at your comments and have pushed some changes. Your review were very helpful btw. [07:06] huwshimi: Great, will look in a sec once the diff is upgraded. [07:06] updated. [07:07] wgrant: or downgraded, depending on the quality of the changes [07:07] lifeless: Could you please db https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279? [07:08] wgrant: I do not like the use of \ at line 151 [07:09] I'm not sure whether \ or parentheses are more vile. [07:09] wgrant: And I don't think the commit() is needed at 187 [07:09] StevenK: It's needed to invalidate the DistroSeries. [07:10] StevenK: So it pulls in the changes from the script. [07:10] StevenK: \ is fine [07:10] wgrant: But the script commits? [07:10] StevenK: its more cromulent than () just to do a newline [07:10] *I* would rather we stopped cutting at 80 characters for such things [07:10] StevenK: Right, but then we have to invalidate the local object to see the committed changes. [07:11] Fair enough [07:11] * spiv waits for autopack of his launchpad repo to finish [07:14] Hmm. 12 minutes, but it did take 30% off the space consumed. [07:15] huwshimi: Your branch hasn't scanned yet :/ [07:16] Ah, there we go. [07:18] huwshimi: [07:18] 7 /* Can't use LPS here as it doesn't exist yet. [07:18] 8 */ [07:18] 9 LPS.use('node', 'lazr.anim', function(Y) { [07:18] Heh [07:18] wgrant: hahahaha [07:19] wgrant: Guess I forgot to take that out when I refactored some things === al-maisan is now known as almaisan-away [07:35] lifeless: Thanks. [07:35] StevenK: Are you code reviewing my branch? [07:35] huwshimi: Bear with me here, I'm experimenting with YUI. [07:36] wgrant: I wasn't, but I can [07:36] wgrant: There's no hurry. I'll be heading out soonish so I'll probably just take a look in the morning [07:36] StevenK: Ah, you were whinging about backslashes and other stuff, so I thought you might have been. [07:36] It would be great if you could. [07:37] 7/win 3 [07:37] Grar. [07:38] wgrant: Done. You just need a number from stub [07:38] StevenK: Thanks. [07:54] huwshimi: A couple of suggestions, but it looks good. [07:59] wgrant: Thanks for that. I'll take a proper look tomorrow [08:07] Project windmill build #115: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/115/ [08:17] stub: Hi. [08:17] wgrant: yo [08:18] stub: Could you DB-bless https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279? [08:19] Just done [08:19] Thanks. [08:19] stub: The apt flag is negative :/ [08:20] ok [08:20] Just keep away from the double negatives :) [08:26] stub: ping [08:26] lifeless: pong [08:26] stub: two things I'd like your help evaluating [08:27] stub: first is the bug heat thing - I saw different results on the candidate index (but only /great/ for ubuntu) [08:27] stub: I'd like to debug why we saw different results; and evaluate denormalising heat down into bugtask [08:28] we might be able to squeeze that into this db cycle even [08:29] secondly, BranchRevision doesn't support range queries by date - but merge proposals (and other interesting queries) are all date based, so I'd like to evaluate the impact of denormalising the revision date into branch revision [08:29] I'm thinking denormalizing into a separate table would be better - I think the regular heat recalculation is contributing to bloat. [08:30] https://bugs.launchpad.net/launchpad/+bug/742916 is a bug showing the merge proposal query (for a merge proposal - old data, will be out of cache a lot) [08:30] <_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan < https://launchpad.net/bugs/742916 > [08:30] I don't have a problem denormalizing the heat. It is a cache anyway - ideally it would be calculated on the fly but that is far to expensive. [08:30] stub: the concern I have with a different table is that the plan problem with sorting on a different table would still exist, unless that table had (heat, status, bug target) [08:31] For BranchRevision, this has been suggested in the past but not acted on (grand plans of removing the table alltogether and all that) [08:31] lifeless: True. Such a table is what we might end up with for bug search anyway - all the searchable fields denormalized into a BugSearch table, but that is a much larger task. [08:32] stub: can we add a datetime column, and an appropriate index to handle the query in https://bugs.launchpad.net/launchpad/+bug/742916 (e.g. (branch, sequence, revision_date)) on staging/qastaging [08:32] <_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan < https://launchpad.net/bugs/742916 > [08:32] stub: so that we can determine the disk impact? [08:32] s/we/you/ [08:32] lifeless: we can. It will take several hours. [08:33] stub: no time like the present (if we're going to do it, it would be good to add it (NULLable) this cycle [08:33] lifeless: There is work we need to test to drop the id column that should counteract the disk space impact. [08:34] stub: the column add should be instant, right? and the index able to be done CONCURRENT ? [08:34] Yes. A job to populate the column will take ages though. [08:35] stub: sure, I think I can make it tolerably efficient [08:35] stub: would like to parallelise the garbo perhaps to help it and othe migrations carry on a little faster [08:35] We need that job too - can't see the disk space impact until the rows have all been rewritten with data in that column. [08:36] stub: ah, well for the test we can do it with an update [08:36] Adding --threads= to garbo should be trivial. [08:36] update branchrevision set revision_date = revision.revision_date from branch_revision, revision where branch_revision.revision=revision.id; [08:37] ^ untested, but approximate [08:37] lifeless: The script will likely be faster, as it works in batches rather than attempting to do it all in one transaction. Also, we have to do it in chunks or we will double the space used by that table [08:38] (all the old rows and all the new rows will exist, and then vacuum will clean up all the old rows giving 50% bloat) [08:38] staging might run out of space [08:38] ok [08:38] I guess branchrevision should wait for next week then - I suspect we couldn't qa in time [08:39] bugtask.heat I'll whip up a patch for tonight. [08:39] stub: do you have any idea why you didn't see the performance benefits on the heat query I did? [08:40] stub: my only WAG is that your test query wasn't adjusted, or something [08:40] I was testing ordering by heat, id. You had a performance boost by dropping id from the sort. [08:40] stub: I tried on qastaging with a (heat, id desc) index and a query sorting by bug.heat desc, bug.id [08:41] I managed to get the existing query to use the new index once, and it saved a few seconds, but the query was still two orders of magnitude slower than your no id sort query. [08:41] ah [08:42] stub: so, I think then we weren't testing quite the same query - because I did adjust the query; I will try to be less ambiguous in future [08:42] No, I was expecting different results :-) It would have been nice to drop in an index and keep stable ordering, but I don't see that happening. [08:43] ah, ok - then *I* didn't understand :) [08:44] anyhow, I think its clear to make it snappy we need heat in the same table [08:44] otherwise it has to traverse *tonnes* of bugs to find the first bug in a context with no really hot bugs [08:44] I can guess that having lots of bugs often equates to having some very hot ones [08:45] but the plan - I pasted it in - for the updated query + new index on launchpad was pretty expensive [08:46] Limit (cost=0.00..146128.04 rows=40 width=1017) (actual time=1.302..500.965 rows=40 loops=1) [08:46]