lifeless wallyworld ping00:08
wallyworldlifeless: hello00:08
lifelesswallyworld: we're blocked on qa for Revision 12671 can not be deployed: needstesting -00:08
wallyworldlifeless: thanks. i'll look now00:09
lifelesswallyworld: thanks!00:09
lifelessStevenK: you have one as well - or is it behind a featureflag?00:09
wallyworldlifeless: on qastaging it has a problem which is makes deployment a no go (not a blocker locally). it should be fixed by a branch already in ec2. can we wait till it goes though?00:22
lifelesswallyworld: so it would break prod ?00:23
wallyworldlifeless: yes00:23
lifelesstag it bad-commit-NNNN00:23
lifelessI've fine with waiting for the followup branch thats already playing00:23
benjibac: I think the test name changed; to run all the tests I've been using just "xvfb-run ./bin/test --layer=RegistryWindmillLayer -c"00:27
benjioops, wrong chan00:27
huwshimilifeless: It appears we can't get oops info for javascript at this point either. Would you be happy with me trying to land the ajax log branch as is, and then we can add the url and oops as an improvement when they become available?00:43
lifelesshuwshimi: I love incremental improvements00:44
lifelesshuwshimi: we don't get the http headers?00:44
huwshimilifeless: You mean for the url or the oops?00:45
lifelesshuwshimi: the oops - its a response header00:45
huwshimilifeless: We get the headers, but I didn't see anything about an oops...00:46
lifelesshuwshimi: its an X header when an oops occurs00:46
huwshimilifeless: Let me take another look00:46
lifelessI'm just digging up the header id00:46
lifelesslib/lp/app/javascript/client.js:690:                if (o.getResponseHeader('X-Lazr-OopsId')) {00:47
lifelesslib/lp/app/javascript/client.js:691:                    server_error = server_error + ' OOPS ID:' + o.getResponseHeader('X-Lazr-OOPSid');00:47
lifelesshuwshimi: ^00:49
huwshimilifeless: OK great00:49
huwshimilifeless: Ugh, ok you're right, it's there00:51
lifelessfor extra credit00:54
lifelessclicking on the oops id could take one to pad.lv/OOPS-xxxxx00:54
lifelesstotally optional of course00:54
huwshimilifeless: It's trivial to add00:56
huwshimilifeless: I've added all this under the "visible_render_time" flag that you added. Is that ok or is it better to have a separate flag for this?01:00
lifelesshuwshimi: thats ideal01:02
lifelesshuwshimi: simplest thing, can always refactor01:02
huwshimilifeless: OK great. Do you want to review this branch?01:02
lifelesshuwshimi: I'm not a gun js guy, but I'll certainly give it a shot01:03
huwshimilifeless: Sure. Thanks01:03
lifelesswhiskers are so ticklish01:03
wgrantlifeless: Let me translate: "feed me"01:08
lifelesswgrant: more 'you make a nice bed'01:11
lifelessface aimed at my face01:11
lifelesscve:+index templates are smoking something01:14
wgrantlifeless: Most old templates are :/01:14
lifelessno really01:14
lifelessshow the bug01:15
lifelessformat a link to it01:15
lifelessthen include the bugtasks-and-nominations-view for the bug01:15
huwshimilifeless: https://code.launchpad.net/~huwshimi/launchpad/ajax-time/+merge/55257 for when you get to it.01:15
lifelesswgrant: ^ - I'll mentor you01:16
wgrantlifeless: k01:16
wgrant:( structure01:19
wgranthuwshimi: You can't put that JS somewhere other than template, with a call in the template to activate it?01:20
wgrantJS in templates makes me very sad; "structure" doubly so.01:20
huwshimilifeless: I'm not sure. I would like it if we had not javascript in templates (and I think we should move towards that). I just don't know where it should live.01:23
huwshimiwgrant: I meant that for you ^01:26
wgrantThe log also highlights if the AJAX event takes more than 1 minute.01:26
wgrantThat's quite a long request!01:26
wgrantHowever, this is really nice.01:26
wgrantParticularly with the OOPS ID stuff.01:27
StevenKjames_w: Ping!01:27
lifelesshuwshimi: I was suggesting 1 second, not 1 minute, for slow ;)01:27
huwshimilifeless: Oh right01:27
huwshimilifeless: That's pretty short!01:27
lifelesshuwshimi: goal is 99% completing withing that time01:27
lifelesshuwshimi: perhaps start at 5s01:28
huwshimilifeless: I think it's great01:28
huwshimiactually I think I did make it 1 second01:29
huwshimiwgrant: Sorry the description is wrong. The code is for 1 second01:30
huwshimiwgrant: My brain failed me when I was writing the description01:30
wgranthuwshimi: Ah, that is less completely crazy. Excellent.01:31
wgrantI think 2-3s is probably more sensible at the moment, but I guess we'll see.01:31
wgrantCould FF it, but probably YAGNI...01:31
huwshimiwgrant: Yeah, lifeless suggested 5 seconds. I don't mind changing it.01:33
huwshimiwgrant: Also if you know of a better place for the javascript to live I can do that as well01:36
james_whi StevenK01:41
StevenKjames_w: Hi. :-)01:43
StevenKjames_w: I've been dealing with python-debian for the derived distros stuff -- did you know that the BaseVersion class in debian_support isn't strict enough with parsing versions?01:44
james_wno, I did not know that01:44
StevenKjames_w: The regex allows a : in upstream version, but that's only allowed if there is an epoch01:45
james_werm, eh?01:45
StevenKjames_w: Heh, have I lost you?01:46
wgranthuwshimi: Reviewed, with a few comments.01:46
james_wI'm trying to work out if what you said is a tautology or not01:46
huwshimiwgrant: Thanks, will take a look01:46
StevenKjames_w: 'foo:bar-1' isn't valid, but '1:foo:bar-1' is.01:47
james_wah, ok01:47
james_wepoch can only be digits01:47
StevenKjames_w: Right. I have a patch: http://pastebin.ubuntu.com/586681/01:47
wgranthuwshimi: The YUI IO events are more limiting than I thought :(01:48
StevenKlifeless: Yes, r12677 is FF-protected.01:50
StevenKlifeless: I can mark it qa-untestable, if you wish01:50
lifelessStevenK: up to you01:51
lifelessStevenK: we've a bad commit to rectify (its landing via ec2 now) before we can deploy01:51
StevenKOnce I've shaken out more of the stuff on dogfood, I'll be QAing it there, for the short-term01:51
lifelessStevenK: long as your commit is either known-bad, or deployable in the next 4 hours, we should be fine01:51
StevenKlifeless: Which is the bad commit, btw?01:52
wgrantlifeless: Next 4 hours? That's not even enough for buildbot.01:52
lifelesswgrant: fuzz01:52
lifelessStevenK: wallyworlds security fix01:52
james_wStevenK, I'm very glad you didn't try and fix that bug using regexes :-)01:52
james_wStevenK, hunks 2 and 3 look good to me, but I don't think I like hunk 101:52
StevenKjames_w: Which would have made the regex about 4 times more complex. I'm masochistic, but not *that* much.01:53
james_wStevenK, also, I think the comment should be more explicit about what it is checking for, and the error could be clearer about the problem too01:53
james_wStevenK, aren't you a perl guy though? ;-)01:53
StevenKjames_w: Hunk one is a seperate, but related problem -- the code currently throws a ValueError if the changelog you parsed contains an invalid version01:54
StevenKjames_w: And I've fixed the comment.01:56
StevenKjames_w: The meat of the patch is hunks 2 and 3. Hunk 1 is at best, a workaround, since I don't want invalid versions in my list of versions.01:59
james_wStevenK, I think that the issue with hunk 1 is better solved at a different level, so I'd leave it out.02:02
james_wStevenK, please send the patch as a bug report to Debian and hopefully someone will commit it02:02
StevenKjames_w: And solve it in a seperate patch?02:04
james_wStevenK, what is the calling code that falls over a ValueError?02:04
james_wpresumably it involves a loop over .blocks?02:04
StevenK._blocks, yes02:05
StevenKjames_w: get_versions()02:05
james_wah, I forgot about that02:05
james_wdon't use that then I would say :-)02:05
StevenKjames_w: But it's ._blocks, it isn't mine to touch :-P02:06
james_wcurrently the library is designed to give you good data or nothing02:06
james_wI think it's public in newer versions02:06
lifelesswhat library are you discussing?02:06
james_wI was slightly naïve in my estimation of how much crap was in all the changelogs in Debian/Ubuntu though02:07
james_win old versions etc.02:07
* StevenK is finding out02:07
StevenKjames_w: Crumbs, do you know how long it's been since I filed a bug on Debian :-)02:08
lifeless366 Time Outs02:13
lifelesshaproxy reconf FTW02:13
wgrantThe BugTask:+index change is striking.02:14
* lifeless blames making things more efficient02:14
wgrantWe've 25% fewer timeouts than Sunday. That is pretty nice.02:15
StevenKjames_w: Well, ... duh02:15
wgrantShipIt needs to die.02:19
wgrantEmailAddress handling is impossible to do correctly.02:19
wgrantBecause they are linked to both accounts and persons.02:19
wgrantSo you may have an email address without a Person because the user used ShipIt.02:20
wgrantYou can't then create a properly unactivated Person for that email address, since the email address is active.02:20
lifelesswell, we know the email is valid02:22
lifelesswhats the issue02:22
wgrantWe are creating an active Launchpad person for somebody who has never used Launchpad.02:22
lifelessjoin the dots for me02:23
wgrantlifeless: A user creates an SSO account and requests CDs from ShipIt. The parasite creates an Account and an EmailAddress in the LP DB.02:26
wgrantThe user performs some action visible to LP. This could be maintaining a package in Debian, commenting in an upstream bugtracker, etc.02:27
wgrantLP tries to create an unactivated Person for that email address, so it can be referenced in the DB.02:27
wgrantBut the EmailAddress is already active, and just needs to be linked to the Person.02:27
wgrantSo the Person is now activated, and the user apparently uses Launchpad.02:28
lifelesswgrant: in other words 'person activation is implicit not explicit'02:34
wgrantlifeless: Yes.02:36
wgrantsinzui disagrees that that is a bug.02:36
=== jamesh_ is now known as jamesh
lifelesswgrant: why ?02:45
wgrantlifeless: I don't recall. It's been a while.02:45
lifelesswgrant: also, if you want to rotflas - lib/lp/bugs/browser/buglinktarget.py - look for except Unauthorized02:46
wgrantlifeless: :(02:47
wgrantEasy fixes, at least.02:47
lifelesswe were disclosing private bug *existance* and generating links02:48
wgrantExistence and linkage.02:48
lifelesscontent="structure batch_navigator/@@+table-view" -> I want to check precisely what this means03:06
lifelessits 'the view called +table-view on the object batch_navigator' rendered into the contents of this element03:06
lifelessright ?03:06
lifelessfingers crossed03:24
lifelesswgrant: what are you hacking on today03:25
wgrantlifeless: Bug #740594 at the moment.03:27
_mup_Bug #740594: ProgrammingError: permission denied for relation emailaddress  <bugwatch> <oops> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/740594 >03:27
lifelesshow is the +table-view hooked up03:31
lifelessgrepping to  deep03:34
lifelessits not rendering ><03:34
wgrantGrar, ensurePerson is buggy.03:34
wgrantIf you call it with an EmailAddress with only an Account, it will create a Person, but forget to link the EmailAddress to it.03:35
wgrantCall it with that EmailAddress a second time, and it will link the EmailAddress to the Person.03:35
wgrantbatchnavigator fixed?03:36
lifelesscan has review? https://code.launchpad.net/~lifeless/launchpad/bug-727023/+merge/5526603:48
* thumper looks03:49
thumperlifeless: question, why create a BugListingBatchNavigator for each bug?03:53
lifelessthumper: the page shows03:53
lifelessbug X03:53
lifeless  tasks03:53
lifelessbug Y03:53
lifeless  tasks03:53
lifelessthumper: it was easy to split it up like this without triggering more queries, so I just kept the prior page layout03:55
lifelessits a bit noddy really given that there is a portlet showing the exact same stuff :)03:56
LPCIBotProject windmill build #114: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/114/04:23
nigelbah, started working when I put my lp keys into the .ssh folder of the virtual machine04:27
wgrantlifeless: Thanks. https://code.launchpad.net/~wgrant/launchpad/bug-740594/+merge/55269 is a follow-up.04:46
StevenKwgrant: r=me04:51
lifelessStevenK: thanks04:53
wgrantStevenK: Thanks.04:53
* StevenK ponders how one changes a branch in sourcecode05:17
wgrantStevenK: Get the change into the branch, fix utilities/sourcedeps.conf.05:19
StevenKwgrant: Ah, and then toss the branch that changes sourcedeps.conf through ec2 to make sure there isn't fallout?05:21
wgrantStevenK: Yes05:21
* StevenK looks at updating python-debian to the latest05:22
nigelbHow long does the fetch source code stage take?05:24
nigelbIts been running for an hour :|05:24
wgrantWhat's it up to?05:24
StevenKDepends on your connection, and if you have any of it, the phase of the moon05:24
lifelessdid we remove the special case of using http fo rlaunchpad clones?05:25
nigelbIts branching something I guess05:25
wgrantnigelb: But what?05:25
wgrantlifeless: That's not update-sourcecode.05:25
nigelb"Making local branch of Launchpad trunk, this may take a while..."05:26
wgrantNot update-sourcecode. I see.05:26
wgrantlifeless: I don't see hot_products in lp-production-configs, so it seems it's gone.05:26
lifelessnigelb: it should be about 200MB05:29
nigelblifeless: oh, ugh.  Its at something like 90% I think05:30
lifelessnigelb: lots of code + lots of history05:30
lifelesswhen we get around to shrinking the codebase05:31
lifelessmight do a fresh tree and massively compact it05:31
lifelessbut not for $quitesometime05:31
nigelbI wonder if there's a way to grab it without history05:31
nigelbor make it notshow history by default05:31
nigelbOh. I should get around to signing that agreement I guess.05:32
StevenKwgrant: https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/5527705:35
wgrantStevenK: Deleting changelog entries: wgrant says no.05:38
StevenKwgrant: Think of it as a sync?05:38
StevenKIt was a cherry-pick from upstream anyway05:38
StevenKWell, a sync + a small patch :-)05:39
StevenKlifeless: You mind reviewing wgrant's review? https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/5527705:46
wgrantStevenK: Actually, any reason the ".find(':') != -1" can't be "':' in "?05:50
StevenKI'm not sure, does that work?05:52
wgrantI'd hope so... group() returns a str.05:53
StevenKRight, my string manupliation stuff is dated back to Linda.05:54
StevenKwgrant: Good catch: http://pastebin.ubuntu.com/586728/05:55
wgrantStevenK: Looks reasonable.05:56
StevenKwgrant: Pushing05:58
lifelesscan has review? https://code.launchpad.net/~lifeless/launchpad/bug-741092/+merge/5527805:58
lifelesshuwshimi: fwiw I agree with wgrants review05:58
lifelesshuwshimi: if you need a hand with any of it just shout05:59
huwshimilifeless: Yeah, I think they were all good comments.05:59
wgrantlifeless: Looking.05:59
wgrantlifeless: Is this to fix the insane 7000 subscriber thing?06:00
StevenKwgrant: I'm happy to review your review06:00
wgrantI think we probably want to batch as well, but OK.06:00
wgrantAh, you mention that in the commit message.06:01
StevenKlifeless: You drop an XXX, can the bug related to that be closed (or is it)?06:01
wgrantIt looks like it's the ancient Storm __nonzero__ bug.06:02
wgrantI suspect.06:02
wgrantBy the age.06:02
wgrantSo it doesn't need closing.06:02
wgrantAnyway, looks good.06:02
lifelessStevenK: it was a bogus XXX06:03
lifelessStevenK: never ever needed06:03
lifelesswgrant: its to improve it06:03
wgrantlifeless: Improve/destroy06:03
wgrantAlthough I guess retrieving all of them should be fairly quick.06:04
wgrantWe'll see!06:04
lifelessI had a glitch in there06:04
wgrantA glitch?06:05
lifelessI've just pushed a trivial tweak to cache the subscription list too06:05
lifelessbah bah06:05
lifelessgimme a sec, to undo my fail06:05
wgrantAh, good point.06:05
lifelessright, pushed06:06
=== almaisan-away is now known as al-maisan
wgrantStevenK: Want to mentor?06:12
StevenKwgrant: Done06:13
lifelesshttps://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1914M186 is going to be fun to fix06:17
wgrantlifeless: Delete BranchRevision, problem solved.06:22
lifelesswgrant: how so?06:22
wgrantlifeless: It's probably a scanner lock.06:22
lifelesson branchmergeproposal ?06:22
wgrantCONTEXT: SQL statement "SELECT 1 FROM ONLY "public"."branch" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x"\\n',)06:23
lifelesswgrant: where are you getting that ?06:24
wgrantlifeless: End of the exception message.06:24
StevenKHm. Now what do I do ...06:25
StevenKI doubt I can lp-land a python-debian PQM branch06:25
wgrantStevenK: pqm-submit06:25
wgrantlifeless: We're running with the old appserver configs, but a new single-threading haproxy one?06:28
lifelesswgrant: partly06:33
lifelesswgrant: depending on whether mbarnett did the tweak requested, we're running on 4 or 6 concurrency from haproxy, and the unaltered appserver configs06:33
wgrantlifeless: 4 or 6 for the 12-thread servers, 1 for the 2-thread?06:34
lifeless4 or 6 for the 4-thread servers06:34
lifeless1 for the 2-thread06:34
wgrantOh, right, it was 12 for 4 before.06:34
lifeless+ xmlrpc at 1 per backend that is listening for xmlrpc06:34
StevenKlifeless: First self-review \o/07:04
huwshimiwgrant: I took a look at your comments and have pushed some changes. Your review were very helpful btw.07:05
wgranthuwshimi: Great, will look in a sec once the diff is upgraded.07:06
huwshimiwgrant: or downgraded, depending on the quality of the changes07:07
wgrantlifeless: Could you please db https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279?07:07
StevenKwgrant: I do not like the use of \ at line 15107:08
wgrantI'm not sure whether \ or parentheses are more vile.07:09
StevenKwgrant: And I don't think the commit() is needed at 18707:09
wgrantStevenK: It's needed to invalidate the DistroSeries.07:09
wgrantStevenK: So it pulls in the changes from the script.07:10
lifelessStevenK: \ is fine07:10
StevenKwgrant: But the script commits?07:10
lifelessStevenK: its more cromulent than () just to do a newline07:10
lifeless*I* would rather we stopped cutting at 80 characters for such things07:10
wgrantStevenK: Right, but then we have to invalidate the local object to see the committed changes.07:10
StevenKFair enough07:11
* spiv waits for autopack of his launchpad repo to finish07:11
spivHmm.  12 minutes, but it did take 30% off the space consumed.07:14
wgranthuwshimi: Your branch hasn't scanned yet :/07:15
wgrantAh, there we go.07:16
wgrant 7    /* Can't use LPS here as it doesn't exist yet.07:18
wgrant 8    */07:18
wgrant 9    LPS.use('node', 'lazr.anim', function(Y) {07:18
huwshimiwgrant: hahahaha07:18
huwshimiwgrant: Guess I forgot to take that out when I refactored some things07:19
=== al-maisan is now known as almaisan-away
wgrantlifeless: Thanks.07:35
wgrantStevenK: Are you code reviewing my branch?07:35
wgranthuwshimi: Bear with me here, I'm experimenting with YUI.07:35
StevenKwgrant: I wasn't, but I can07:36
huwshimiwgrant: There's no hurry. I'll be heading out soonish so I'll probably just take a look in the morning07:36
wgrantStevenK: Ah, you were whinging about backslashes and other stuff, so I thought you might have been.07:36
wgrantIt would be great if you could.07:36
wgrant7/win 307:37
StevenKwgrant: Done. You just need a number from stub07:38
wgrantStevenK: Thanks.07:38
wgranthuwshimi: A couple of suggestions, but it looks good.07:54
huwshimiwgrant: Thanks for that. I'll take a proper look tomorrow07:59
LPCIBotProject windmill build #115: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/115/08:07
wgrantstub: Hi.08:17
stubwgrant: yo08:17
wgrantstub: Could you DB-bless https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279?08:18
stubJust done08:19
wgrantstub: The apt flag is negative :/08:19
stubJust keep away from the double negatives :)08:20
lifelessstub: ping08:26
stublifeless: pong08:26
lifelessstub: two things I'd like your help evaluating08:26
lifelessstub: first is the bug heat thing - I saw different results on the candidate index (but only /great/ for ubuntu)08:27
lifelessstub: I'd like to debug why we saw different results; and evaluate denormalising heat down into bugtask08:27
lifelesswe might be able to squeeze that into this db cycle even08:28
lifelesssecondly, BranchRevision doesn't support range queries by date - but merge proposals (and other interesting queries) are all date based, so I'd like to evaluate the impact of denormalising the revision date into branch revision08:29
stubI'm thinking denormalizing into a separate table would be better - I think the regular heat recalculation is contributing to bloat.08:29
lifelesshttps://bugs.launchpad.net/launchpad/+bug/742916 is a bug showing the merge proposal query (for a merge proposal - old data, will be out of cache a lot)08:30
_mup_Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >08:30
stubI don't have a problem denormalizing the heat. It is a cache anyway - ideally it would be calculated on the fly but that is far to expensive.08:30
lifelessstub: the concern I have with a different table is that the plan problem with sorting on a different table would still exist, unless that table had (heat, status, bug target)08:30
stubFor BranchRevision, this has been suggested in the past but not acted on (grand plans of removing the table alltogether and all that)08:31
stublifeless: True. Such a table is what we might end up with for bug search anyway - all the searchable fields denormalized into a BugSearch table, but that is a much larger task.08:31
lifelessstub: can we add a datetime column, and an appropriate index to handle the query in https://bugs.launchpad.net/launchpad/+bug/742916 (e.g. (branch, sequence, revision_date)) on staging/qastaging08:32
_mup_Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >08:32
lifelessstub: so that we can determine the disk impact?08:32
stublifeless: we can. It will take several hours.08:32
lifelessstub: no time like the present (if we're going to do it, it would be good to add it (NULLable) this cycle08:33
stublifeless: There is work we need to test to drop the id column that should counteract the disk space impact.08:33
lifelessstub: the column add should be instant, right? and the index able to be done CONCURRENT ?08:34
stubYes. A job to populate the column will take ages though.08:34
lifelessstub: sure, I think I can make it tolerably efficient08:35
lifelessstub: would like to parallelise the garbo perhaps to help it and othe migrations carry on a little faster08:35
stubWe need that job too - can't see the disk space impact until the rows have all been rewritten with data in that column.08:35
lifelessstub: ah, well for the test we can do it with an update08:36
stubAdding --threads= to garbo should be trivial.08:36
lifelessupdate branchrevision set revision_date = revision.revision_date from branch_revision, revision where branch_revision.revision=revision.id;08:36
lifeless^ untested, but approximate08:37
stublifeless: The script will likely be faster, as it works in batches rather than attempting to do it all in one transaction. Also, we have to do it in chunks or we will double the space used by that table08:37
stub(all the old rows and all the new rows will exist, and then vacuum will clean up all the old rows giving 50% bloat)08:38
lifelessstaging might run out of space08:38
lifelessI guess branchrevision should wait for next week then - I suspect we couldn't qa in time08:38
lifelessbugtask.heat I'll whip up a patch for tonight.08:39
lifelessstub: do you have any idea why you didn't see the performance benefits on the heat query I did?08:39
lifelessstub: my only WAG is that your test query wasn't adjusted, or something08:40
stubI was testing ordering by heat, id. You had a performance boost by dropping id from the sort.08:40
lifelessstub: I tried on qastaging with a (heat, id desc) index and a query sorting by bug.heat desc, bug.id08:40
stubI managed to get the existing query to use the new index once, and it saved a few seconds, but the query was still two orders of magnitude slower than your no id sort query.08:41
lifelessstub: so, I think then we weren't testing quite the same query - because I did adjust the query; I will try to be less ambiguous in future08:42
stubNo, I was expecting different results :-) It would have been nice to drop in an index and keep stable ordering, but I don't see that happening.08:42
lifelessah, ok - then *I* didn't understand  :)08:43
lifelessanyhow, I think its clear to make it snappy we need heat in the same table08:44
lifelessotherwise it has to traverse *tonnes* of bugs to find the first bug in a context with no really hot bugs08:44
lifelessI can guess that having lots of bugs often equates to having some very hot ones08:44
lifelessbut the plan - I pasted it in - for the updated query + new index on launchpad was pretty expensive08:45
lifeless Limit (cost=0.00..146128.04 rows=40 width=1017) (actual time=1.302..500.965 rows=40 loops=1)08:46
lifeless(this is actually 50% cheaper than the *current* plan, but still awful)08:46
lifelessstub: would you like a bug about the garbo parallel processing ?08:48
lifelessstub: I don't think we need a --threads option08:48
lifelessstub: multiprocessing.cpu_count() will return the # of cpus08:48
lifelessstub: we can just run that many threads08:48
stubGarbo jobs are almost certainly database bound, so not sure if that is so useful.08:49
stubI've considered adding options to only run specific tasks (but how to handle lock files is unclear), and the --threads.08:49
stubI suspect we want a lock file per task08:50
stubRather than a single lock for the script.08:50
stubBug #74480308:56
_mup_Bug #744803: Better garbo locking <Launchpad itself:Triaged> < https://launchpad.net/bugs/744803 >08:56
adeuringgood morning08:58
stubBug #74480409:03
_mup_Bug #744804: Parallel garbo <Launchpad itself:Triaged> < https://launchpad.net/bugs/744804 >09:03
lifelessstub: if we run N garbos we'll need per-tuneable locks yes; if we run one master that parallelises I think we can ignore the locking for now09:05
lifelessstub: I think many garbo jobs are overhead bound - I mean, time to pull stuff out, mangle it, shove it back in; others are going to be db bound - but we should remember we have 8 available cores on master atm, and most garbo jobs are completely up to date, we'll only have a few migrations in flight at any one time09:06
lifelessstub: I don't particularly care about implementation, the reason I suggested using cpu count is that it avoids manual selection (at the cost of not being tunable)09:07
stubSure. I'll probably add --threads and have it default to cpu count :)09:08
lifelessstub: all that said, I'd love to see things like stevenk's migrator, and the branchrevision one next week, get more of a timeslice than they do at the moment - ideally 55minutes per hour09:08
StevenKThat sounds good.09:11
lifelessstub: so I think that is the key goal: more time for migrations in the garbo09:11
lifelessstub: everything else is tactics09:11
stubI won't suggest to jtv that if looptuner became thread aware it could tune the number of instances dynamically to maximize throughput based on database load ;)09:12
lifelessstub: another thing, probably EFUTURE, would be allowing the loops themselves to parallelise09:12
lifelessstub: please don't :P09:12
jtvstub: as if you'd need to!09:13
jtvYou think I don't think about these things?09:13
stubI know you do :-)09:13
stubActually, garbo 2.0 is looking like a twisted system... but I think we are several years away from that.09:14
lifelessbug 744808 is a browser/X bug right ?09:14
_mup_Bug #744808: Redrawing of Bug View page very slow in Firefox <Launchpad itself:New> < https://launchpad.net/bugs/744808 >09:15
jtvIt does sound like one we had.09:15
lifelessstub: can you do a quick timing check for me on staging ?09:18
=== almaisan-away is now known as al-maisan
lifelessstub: add a heat column to bugtask and copy the heat over from bug09:18
lifelessstub: bugtask is only 250MB in size09:18
lifelessstub: so just a single sql statement should be fine - I want to see if its fast enough to do during downtime09:19
lifelessstub: can I take 59 for this ?09:27
stubgah... distracted09:50
stublifeless: 59... sure.09:51
lifelessstub: pastbinnning a script09:51
lifelessstub: http://pastebin.com/fUQN4JvF09:53
stubThe productseries index seems to have a bogus WHERE clause09:54
lifelessyou pass :P09:54
lifeless[bad copypaste, thanks]09:54
lifelessstub: so, did that work, how long to do the update?10:05
stublifeless: I'll shove that through slony now10:05
lifelessstub: thanks10:06
lifelessstub: can a trigger update columns in a related table?10:09
stublifeless: yes10:09
stublifeless: I'd rather just remove Bug.heat though10:09
lifelesshow do we update stuff in trusted.sql ?10:09
lifelessstub: I'll keep that in mind as I look through this10:10
stubYou just edit it. And things can break if you remove things.10:10
lifelesscalculate_bug_heat is what I'm changing, for obvious reasons10:10
lifeless# Bugs decay over time. Every day the bug isn't touched its heat  ---- needs to die10:11
stubJust change it inplace then. You shouldn't need to alter the name or signature.10:11
lifelessits not even a trigger10:11
lifelessits just an in-db function10:11
lifelesswhich of course... headdesk10:12
lifelessstub: might need a hand doing a trigger for this - we don't want to calculate the heat once per task, so keeping the bug.heat seems easier.... but we need to copy any changes to bug.heat to the bugs bugtasks.heat10:13
stubWe need to shutdown most of the staging systems - its too busy to apply the patch live.10:19
lifelessok; lets do it10:20
=== gmb changed the topic of #launchpad-dev to: erformance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | https://code.launchpad.net/launchpad-project/+activereviews
lifelessstub: triggered up10:34
lifelessgmb: you lost a P10:34
bigjoolsthe irony.  I'm just going for one.10:35
=== gmb changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: gmb | https://code.launchpad.net/launchpad-project/+activereviews
bigjoolslifeless: you'll like this: http://1.bp.blogspot.com/-YhwtjjSfRf4/TZBOYuLTNVI/AAAAAAAAAP0/0P7KUdJ1RBI/s1600/James.png10:35
lifelessbigjools: rotfl10:36
stublifeless: Do you want that trigger now? If so, we should not try and monkey patch this - just land it as normal and back things out if the timings suck10:36
lifelessstub: don't need it for the timings10:37
lifelessstub: Just want your eyeballs on the updated db side of the patch10:37
stublifeless: So the alternative of changing the UPDATE Bug SET heat=calculate_bug_heat(...) WHERE id=... to UPDATE BugTask SET heat=calculate_bug_heat WHERE bug=... and dropping Bug.heat sucks?10:38
lifelessstub: some bugs have hundreds of tasks10:38
lifelessstub: and bug heat calculation isn't, uhm, efficient10:39
stublifeless: How does this change that?10:39
lifelessstub: oh, and I need a check for NEW.heat != OLD.heat10:40
lifelessstub: uhm, it copies the one calculated value from the bug, rather than doing it once per row - or would pgsql optimise that out ?10:40
stubUPDATE BugTask SET heat=calculate_bug_heat($bug) WHERE bug=$bug; calls the stored procedure once.10:41
stubWell... assuming the function hasn't been declared to volatile10:42
stubIts declared stable, so should only be invoked once.10:43
lifelessstub: ok; I'm inclined to go inchwise for now anyway, simply because there are a few hundred references to heat.10:43
lifelessand I'd like to get this in tomorrow, which a more aggressive branch is more likely to fail10:43
jmlis there still a MOTU liaison to Launchpad?10:55
lifelesswgrant last I heard :P10:57
gmbjtv: https://code.launchpad.net/~jtv/launchpad/db-bug-55798/+merge/55174 says "This is still a work in progress; expect a proper cover letter later when it goes up for review" yet it's marked "needs review". Should it be marked "work in progress"?10:57
jtvgmb: I thought I did mark it WIP… thanks for noticing.10:57
gmbjtv: I'll do it now.10:58
lifelessstub: updated with the if condition - http://pastebin.com/bYHpVBWH10:58
* gmb just marked it Approved for a second. Would've been funny if we'd been doing automated merges10:58
gmbjtv: Done.10:58
lifelessstub: if I'm correct, this is landable once we confirm the migration time10:58
lifelessstub: and we can then do the better query post db deploy10:59
wgrantjml: So, yeah, not really.11:00
jmlwgrant: I didn't think so.11:00
wgrantjml: People basically all just know to poke me.11:00
lifelessstub: I'm going to propose this for merge11:11
henningeI get this when running "make lint": http://paste.ubuntu.com/586801/11:13
henningeUpdating my sourcedeps now ...11:13
henningeno joy ;(11:15
stublifeless: I'd say land it anyway so we don't block on losas.11:15
stublifeless: We can rework it if the rollout time sucks11:15
=== al-maisan is now known as almaisan-away
stublifeless: The trigger should be INSERT OR UPDATE11:16
jmlhenninge: there's nothing there that will come from sourcedeps11:17
lifelessstub: why? INSERT never has anything to do11:17
wgranthenninge: That class is new in 2.7 :(11:17
jmlhenninge: what version of pocketlint do you have?11:17
lifelessstub: because bugtask has a foreign key on bug11:17
henningejml: yes, I realised11:17
wgranthenninge: Could you file a bug on pocketlint?11:17
wgranthenninge: sinzui will fix it quickly I'm sure.11:17
wgrantOtherwise downgrade python-pocketlint.11:17
jmlahh, there you go.11:17
henningewgrant: Will do, thanks.11:18
StevenKOh, is that the upgraded python-pocketlint I haven't done yet? Bonus.11:18
stublifeless: oic. yes.11:18
bigjoolsallenap: in your review you said that stuff returned from canonical_url needs to be escaped.  Really?11:18
stublifeless: So when a bugtask is created, it should get the heat of the attached bug. Probably better having that in Python land rather than another trigger.11:18
lifelessstub: as we only need to sort on the high heat tasks initially, as long as new tasks have heat 0, the heat garbo job will sort it out11:18
lifelessstub: and yes, we can also sort it out in python; the column is non-null11:19
wgrantEvery problem can be solved with another layer of triggers.11:19
lifelesswgrant: except having too many triggers11:19
stublifeless: So as the patch stands at the moment, creating a bugtask will fail because it will have a NULL heat.11:19
lifelessstub: default 0 ?11:20
wgrantlifeless: We can have a trigger to generate the triggers on demand and delete them when they're not needed any more!11:20
stubMight want a DEFAULT 0 in there if you want to land this separate from code changes.11:20
lifelessyeah, I'll just tweak the alter11:20
stublifeless: Set the default at the same time or after setting the NOT NULL - not in the initial ALTER TABLE11:20
lifelessstub: https://code.launchpad.net/~lifeless/launchpad/bug-618406/+merge/5530311:22
lifelessstub: yeah, I did :)11:22
stubIts a non-obvious footgun :)11:22
lifelessbah, I left 58 in the .sql file11:22
lifelesschanging to 5911:22
stublifeless: 58 is fine11:22
lifelessstub: the .sql file is called 5911:23
lifelessstub: they need to match, don't they ?11:23
lifelessstub: I can rename both to 58, or leave - up to you11:23
stubI don't care. I have many integers :)11:23
henningewgrant: bug 74487311:26
_mup_Bug #744873: lint fails on Python 2.6 <pocket-lint:New> < https://launchpad.net/bugs/744873 >11:26
allenapbigjools: Yes. Everything that goes into HTML/XML/XHTML should be escaped unless it has been escaped already.11:26
bigjoolsallenap: ok11:27
wgrantbigjools, allenap: canonical_url should always be safe, but if you don't escape it then you have a bad habit.11:27
lifelessstub: https://code.launchpad.net/~lifeless/launchpad/bug-618406/+merge/55303 has its fidd11:27
wgrantAnd if it's easier to not escape it than it is to escape it, then we have a systemic problem.11:28
lifelessoh fail, EWRONGTARGET11:28
bigjoolswgrant: my point11:28
allenapI think we do have a systemic problem then :)11:28
lifelesstake two11:28
wgrantWe do :)11:28
* stub waits again for the diff11:29
allenapwgrant: One of my suggestions for bigjools in the review was to use lxml.html to build fragments. For very small things it appears heavyweight, but imo it soon pays off, and the cssselect() method makes it similar in use to Y.Node.11:30
stubwgrant: I wonder if we can hack structure to explode if it is passed something not known to be escaped? We already have monkey patches in place for the cache: stuff.11:31
bigjoolsallenap: I liked that but I chickened out and used cgi.escape11:31
lifelessbigjools: you know thats incomplete right ?11:31
wgrantcgi.escape(foo) is fine for element contents.11:31
wgrantIt is not fine for attributes.11:31
allenapI pointed out the quote argument too.11:31
wgrantYou need cgi.escape(foo, True) for attributes, and you cannot use single quotes around the attribute.;11:32
wgrantstub: I am considering changing zope.tal to use markupsafe for escaping, which will allow us to just pass markupsafe objects into the template without structure.11:32
wgrantstub: Then any use of structure that isn't a widget will be easily visible.11:32
lifelessstub: diff up11:32
* lifeless breaks the build11:39
lifelessgnight all11:39
lifelesshmm, wonder if wallyworld's fix landde11:40
lifelesslooks like it, we should be able to deploy if someone wants to coordinate11:40
lifelessnight all11:40
adeuringgmb: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/js-translation/+merge/55309 ?11:44
deryckMorning, all.12:02
=== almaisan-away is now known as al-maisan
adeuringmorning deryck12:03
adeuringgmb, ping12:03
=== henninge_ is now known as henninge
henningegmb: Hi!12:06
henningegmb: Can you please review my branch? I will be having lunch and switching locations now but I'll be happy to answer questions after that.12:08
=== henninge is now known as henninge-lunch
wallyworldlifeless: fix has landed12:14
wgrantFix is going to be deployed once the staging mailbox loads.12:16
LPCIBotProject windmill build #116: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/116/12:21
wgrantStupid Pacific.12:32
wgrantAnd/or Optus.12:32
wgrantI would like more than 10KB/s to international hosts tonight, thanks...12:32
gmbadeuring, henninge-lunch: Sure.12:48
adeuringgmb: thanks12:49
jmlwgrant: on cable? maybe there's a popular show on the telly12:49
=== henninge-lunch is now known as henninge
henningegmb: thanks a lot12:54
* henninge reloates12:54
jml(no, I don't understand anything about how physical networks work)12:59
wgrantjml: Sadly they don't work that way.13:01
wgrantAnd it's only international.13:01
wgrantcjwatson: The buildbot slaves are updated, and I've hopefully built a sufficient EC2 AMI, which I'm about to test your binary xz branch against. If it goes well I'll land it tomorrow.13:01
StevenKwgrant: And added your ID to ec2 land?13:03
wgrantStevenK: In this branch, yes.13:03
gmbadeuring: r=me13:05
LPCIBotProject windmill build #117: STILL FAILING in 45 min: https://lpci.wedontsleep.org/job/windmill/117/13:07
cjwatsonwgrant: excellent.  is that the data.tar.xz one?13:23
wgrantcjwatson: Yeah.13:24
wgrantjml: I don't know how I lived without ec2 list.13:25
wgrantcjwatson: Should both be deployed on the 6th, I guess.13:25
jmlwgrant: :)13:25
cjwatsonI was just about to ask that.13:25
wgrantcjwatson: We'll hopefully be able to get cocoplum into the daily nodowntime rollouts eventually, but it's more difficult than just about anything else.13:26
cjwatsonI can imagine.13:26
gmbhenninge: r=me13:27
jmlI'm not hopeful for an answer but it's worth asking...13:31
jmlis there a programmatic way to find out what, say, a distribution owner is allowed to do?13:32
bigjoolssecurity.py nicely obfuscates that :/13:36
wgrantAt least most Soyuz stuff is neatly encapsulated in security.py13:37
wgrantFor other stuff (bugs come to mind).. uh, good luck.13:37
bigjoolsI'm not that sure it is13:37
bigjoolsthere's a few methods that take user args13:38
wgrantSoyuz has an entirely separate permission system for it's complex stuff.13:38
wgrantMm, true.13:38
wgrantBut not many.13:38
bigjoolsseparate - as in "do what you like!"13:38
adeuringgmb: thanks!13:38
bigjoolswhich I will endeavour to fix for some cases for DDs13:39
wgrantbigjools: Oh?13:39
bigjoolsyes, I want to make distro management done in the UI/API as much as possible13:40
wgrantOh, I was referring to ArchivePermission.13:40
wgrantNot ssh cocoplum.13:40
bigjoolsthe coming year is going to be painful13:40
jmlwgrant: manual inspection hasn't proved too hard at this point13:40
jmljust... manual13:40
wgrantbigjools: I was looking forward to working on some of that stuff, TBH :/13:41
jmlnext time I hear someone say "for audit purposes", I'm going to ask them how they are going to audit.13:41
bigjoolswgrant: what's stopping you? :)13:41
wgrantIt's out of scope of the maintenance squads :(13:42
henningegmb: Thanks for the review! ;) I had already filed bug 744873 about pocketlint.13:53
_mup_Bug #744873: lint fails on Python 2.6 <pocket-lint:New> < https://launchpad.net/bugs/744873 >13:53
gmbhenninge: Cool, thanks.13:53
deryckhenninge: ping for standup14:01
henningederyck: coming14:01
jmlwhy is https://blueprints.qastaging.launchpad.net/launchpad showing different data to https://blueprints.launchpad.net/launchpad ?14:31
jmlapparently because they are configured differently14:34
jmlsinzui: hey, you had a XXX analyzer, right?14:57
sinzuijml: I wrote the script in utilities14:59
sinzuiI am uncertain of its value15:00
jmlsinzui: I was wondering which XXX comments refer to fixed bugs.15:01
jmlsinzui: your script helps15:01
sinzuijml: :) I found one a few days ago actually15:01
bacsinzui: when i run 'make lint' i get an import error for ParseError.  did you miss a dependency on pocketlint?15:03
sinzuijml: utilities/xxxreport.py -f c ./xxx-report.csv15:03
jmlsinzui: ./utilities/xxxreport.py -f csv  . | cut -d, -f5 | sort -n | uniq15:03
sinzuijml: did did something similar to feed and api script that added tech-debt to each of those bugs15:04
jmlshame there's no way to get all of those bugs in one API request.15:04
henningebac: bug 744873 ;)15:04
sinzuisome most bugs are in the launchpad-project. A few are in Zope15:04
_mup_Bug #744873: lint fails on Python 2.6 <pocket-lint:Fix Committed by sinzui> < https://launchpad.net/bugs/744873 >15:04
jmlsinzui: I imagine a few are in Bazaar or Twisted too.15:05
sinzuioh, yes, some are in twisted15:05
bacthanks henninge15:05
jmlhow do you get a bug by id in the API?15:08
jmlhuh, just add it to the end of the bugs collection15:08
sinzuijml: something like this: http://pastebin.ubuntu.com/586883/15:10
jmlsinzui: ta15:11
abentleyhenninge: if you'd like to discuss my changes, I'm free.15:12
henningeabentley: your code page is timing out on me ...15:18
abentleyhenninge: the loggerhead page?15:18
henningeabentley: which branch should I look at?15:18
abentleyhenninge: the cumulative changes are in lp:~abentley/launchpad/js-translation-215:19
henningeabentley: thanks15:20
henningeabentley: I'll play around with it a bit15:20
abentleyhenninge: Cool15:20
jmlsinzui: http://paste.ubuntu.com/586884/ all of the bugs in XXX comments that have a closed task15:21
jmlsinzui: that's over a quarter of all bugs mentioned.15:22
sinzuijml: :) Maybe we should have had an action item to followup on this15:22
sinzuiThis is very good progress15:22
LPCIBotProject db-devel build #503: FAILURE in 43 min: https://lpci.wedontsleep.org/job/db-devel/503/15:22
jmlsinzui: an action item for who to follow up on this, exactly.15:23
sinzuijml: We concluded the report was not driving the XXX to zero, but it would have been nice to check the report every thunderdome to see if we are making progress15:24
jmlsinzui: ahh yes.15:24
jmlsinzui: well, that's probably a good idea for next thunderdome, especially if those XXXs for closed bugs are removed from the tree15:24
jmlwell, that's enough play for me. back to the grindstone15:26
sinzuihenninge: pocket-lint is fixed for python 2.6  and published in launchpad's ppa15:26
henningesinzui: cool, thanks for the prompt fix15:27
sinzuiIt was just as prompt as my fix for python 2.7...but this time both work. I was incompetent.15:28
bacsinzui: i'm getting an ExpatError from pocketlint (the new one) http://pastebin.ubuntu.com/586887/15:35
bacsinzui: you want my branch for testing?15:36
sinzuibac: I did not see those changes in the diff :(. I made this change a long time ago15:37
sinzuibac: henninge: you will need to wait another 60min for a fix. I see the second change in annotate and it is simple to treat both kinds of errors separately.15:40
LPCIBotProject windmill build #118: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/118/15:40
sinzuiI sense some irony in this mess. I was slow to release pocket-lint fixes in the past because I lost the test suite when I divorced the linter from gedit-developer-plugins. I rebuilt the test suite so I now release quickly. The test suite runs with the default python, which for my 2.715:43
bacsinzui: i made a wee fix to formatcheck.py that allows me to go forward15:44
bacsinzui: it is also ironic that formatcheck.py has lint errors in it.  :)15:46
sinzuibac: it did not until I had to backport :)15:46
sinzuipyflakes hates import redefinitions15:47
=== al-maisan is now known as almaisan-away
jmlthere are ways to avoid redefinitions15:52
jmldoes "Contact this team" send emails to teams if the team-within-a-team has a contact address?15:52
jmllooks like it contacts members with preferred emails15:55
jmlcan teams have preferred emails?15:55
jmlmrevell: do you know how this works?15:57
* mrevell reads up15:57
mrevelljml, Teams can have preferred emails. They can choose from "Email every member individually", "email the team's list, if it exists", "email some other address that the admin has specified"15:58
mrevellSo, I assume the team-within-a-team thing will respect the team-within-a-team's choice.15:58
mrevellBut I don't know for sure.15:59
jmlfrustrating that this is so hard to figure out.15:59
henningeabentley: Shouldn't all the items of the check list be active already?16:13
abentleyNo, that depends on the state of the sourcepackage.16:14
henningeabentley: by 'active' I meant 'using ajax'16:15
abentleyhenninge: You mean that the page where users select the target branch for a sourcepackage already uses ajax, for example?16:17
henningeabentley: I can set the targe branch for the upstream project using ajax.16:18
henningeabentley: and I can search for a upstream product series16:18
henningebut the edit icons next to the other two items in the list take me to the edit pages for those (web 1.0)16:19
henningeabentley: I guess I am just asking if you have pushed your latest code ... ;)16:19
abentleyhenninge: Yes, I have pushed all my latest code, and the last two items still need to be done.16:20
abentleyhenninge: I am currently fixing up the windmill tests for selecting a productseries.16:20
henningeabentley: np, I was just wondering because I see that there is already code for those in the js.16:21
abentleyhenninge: There is model code, but I wan't sure what to do for the UI code until I spoke with deryck yesterday afternoon.16:21
LPCIBotProject db-devel build #504: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/db-devel/504/16:23
=== almaisan-away is now known as al-maisan
henningeabentley: thanks. I will EOD now and continue tomorrow.16:35
abentleyhenninge: cool16:35
jmllooks like the build failure is a legit one16:36
jmlanyone addressing it?16:37
=== salgado is now known as salgado-lunch
=== Ursinha is now known as Ursinha-afk
LPCIBotProject windmill build #119: STILL FAILING in 35 min: https://lpci.wedontsleep.org/job/windmill/119/16:58
jmlquick review for testfix? http://pastebin.ubuntu.com/586935/17:00
=== al-maisan is now known as almaisan-away
jmltaking that as a "no"17:06
jmllanding w/out review17:06
bigjoolsjml: r=me17:07
jmlbigjools: thanks.17:07
* bigjools pines for the r=trivial days17:08
jmlheh. this isn't trivial, per se17:08
bigjoolsI actually think testfixes should be mandatory trivial - else back the revision out17:09
jmlbzr pqm-submit --submit-branch=bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel -m '[testfix][r=bigjools] Be explicit about which table we get heat from.'17:14
jmlwhat's wrong with that line?17:14
jmlit's acting as if it works but nothing happens on pqm.launchpad.net or lpbuildbot17:14
bigjools[no-qa] missing?17:14
bigjoolsor bug=17:14
=== deryck is now known as deryck[lunch]
jmlbigjools: seems to work now, thanks. maybe it was just a delay (or fast PQM processing + slow buildbot)17:22
jmlbigjools: interesting idea re mandatory trivial testfixes. certainly a good rule of thumb.17:22
nigelbflacoste: thanks.17:24
bigjoolsjml: indeed - the idea being that all the time it's not fixed, everyone is blocked, so we need to spend the minimum amount of time fixing it.17:25
jmlthere's a launchpad.Driver permission17:29
jmlfor some reason, that makes me wish I could swear like a 19th century Parisian.17:29
bigjoolsjust look up the Merovingian's line in The Matrix17:31
jmlthere's a thought.17:31
jmlalthough I doubt that whatever the real Merovingians spoke sounded like that.17:32
bigjools"It's like wiping your arse with silk"17:32
jmlAn analogy for which I lack direct experience17:33
jml(since I've never sworn in French, of course)17:33
bigjoolsof course17:33
* jml is filling in https://wiki.ubuntu.com/LaunchpadPermissions very very slowly17:42
jmlok. I am sick of surprises.17:49
jmlg'night all17:59
jmlthe build is failing again18:05
jmllooks like a hangover from the previous test failure. forcing a new build after the failure ought to fix it.18:06
jmlI know there are plenty of North Americans around to read this in a timely fashion, so I'm not going to send an email.18:06
=== salgado-lunch is now known as salgado
=== deryck[lunch] is now known as deryck
=== beuno is now known as beuno-lunch
=== beuno-lunch is now known as beuno
lifelessjml: thank you19:53
lifelesssinzui: ping20:04
sinzuihi lifeless20:04
lifelessyou helped me get an oops for https://bugs.launchpad.net/launchpad/+bug/74109220:04
_mup_Bug #741092: Archive:+subscriptions times out with many subscribers <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/741092 >20:04
lifelesscould you perhaps help me qa it - see if it lets you view it cold, and after a few retries, on qastaging ?20:04
lifelessI haven't batched it20:05
* sinzui types the url20:05
lifelessbut in theory we can handle 7K in one shot; if it doesn't render even after a few refreshes we probably are running out of storm cache or something and will need to batch :(20:05
sinzuilifeless: I do not have permission to see that one, but I think the beta-font team has an equally pathological subscriber page. I will check production first20:07
lifelessI found the design there a little confusing too, given it doesn't use teamparticipation, but presumably we need to permit new members to see automatically20:09
=== mtaylor_ is now known as mtaylor
sinzuilifeless: I asked achuni to help since he reported the bug. The cold cache does timeout: OOPS-1914QS128 and the second try does too: OOPS-1914QS12920:14
lifelesssinzui: 'darn'20:17
lifelesssinzui: can you get him to try 6 times20:17
lifelesssinzui: (6 times because if its /just/ cold cache in the DB, we are only walking Person records 10 seconds worth at a time20:17
lifelessI'm syncing qastaging to look at the oops20:18
sinzui:) I explained the same to achuni20:18
lifelesssinzui: \o/20:18
sinzuilifeless: all oopes: oops ids are OOPS-1914QS133 to 13920:22
LPCIBotProject devel build #588: FAILURE in 5 hr 0 min: https://lpci.wedontsleep.org/job/devel/588/20:23
lifelessoops ids are being reused on qastaging20:39
lifelessgarbo hourly is overwriting20:39
=== almaisan-away is now known as al-maisan
lifelessok, I got a trace https://lp-oops.canonical.com/oops.py/?oopsid=1914QS13320:48
lifelessStatement Count: 1491 ><20:48
lifeless1467 person lookups20:48
* lifeless thinks the storm cache is too small20:49
lifelessno, thats not it, its clearly doing single loads20:51
* lifeless failed 20:51
lifelessthe traceback is really weird21:05
lifelessids = set(map(attrgetter('subscriber_id'), result)) is triggering late evaluation. EWTF21:06
lifelessModule canonical.launchpad.webapp.authorization, line 218, in checkPermission21:08
lifeless    principal.account)21:08
=== al-maisan is now known as almaisan-away
lifelesssinzui: what do you think http://pastebin.com/70gVSwSt ?21:13
sinzuilifeless: we were doing the expensive check first?21:24
lifelesssinzui: we loaded the subscriptions21:25
lifelessthen in the line I pasted - the attrgetter triggered permission check via the security proxy trying o read the subscriber_id21:25
lifelessthe person hasn't been loaded at that point21:26
lifelessso it lazy evaluated the person21:26
lifelessfound that the user wasn't the subscriber, found the user was archive admin, and we go around again21:26
lifelessI'm doing two changes21:27
lifelessone to change the order - common case will be the archive admin managing subscriptions21:27
lifelessI think we should allow subscriber_id to any code21:27
lifelessI think http://pastebin.com/HTPWc93e will do it21:27
sinzuithat is sensible21:28
lifelessbut i don't know - because the _id is also in the interface21:28
lifelesswill the lp.View declaration override the allow declaration ?21:28
sinzuilifeless: I do not have an issue with using the id. (I have some issues with is appearing in the ui/forms). I think the security.py change is good to land...21:30
lifelesssinzui: the id shouldn't appear in any ui/forms21:30
LPCIBotProject windmill build #120: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/120/21:31
sinzuilifeless:  I do not think id will be exposed anywhere. Do we need a test to verify access to subscriber_id? Is it implicitly tested?21:32
lifelesssinzui: do you know if subscribers use this page as well ?21:32
lifelesssinzui: we know it works via the security proxy; the question is whether it avoids triggering a security check at all21:33
lifelesshowever its a bit of a stopgap21:33
* sinzui checks the alternate path21:33
lifelessthe basic problem is that when subscribers view this page we should /only/ load the subscribers subscription21:33
lifelessnot all21:33
lifelessby which I mean, the security.py change fixes admins/archive owners21:34
lifelessbut won't fix 'archive subscriber' viewing the page21:34
lifelesssinzui: so, I'll land the security.py thing21:35
lifelesswe might not have subscribers view the page21:35
lifelessand if we do, we can do the root cause change rather than a bandaid21:35
sinzuilifeless: subscribers cannot access that page/view. They have a separate view21:35
lifelesssinzui: in which case, though its a little more expensive than idea, the change I've made is sufficient21:36
lifelesssinzui: do you want to review, or should I self review?21:37
sinzuiYou have my r=,me21:37
sinzuiI can mark it reviewed if you point me to an mp21:37
lifelessjust filing it21:38
lifelesssinzui: it has a diff now21:42
lifelesssinzui: wrong widget :P21:43
sinzuiIt will come21:43
lifelessI really want callbacks21:44
lifelesswould make our UI /so much better/21:44
lifelessnow to see what my db patch is up to21:48
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
lifelessallenap: you've seen load_related, right ?22:03
thumperhow do I find out what files get installed by a package?22:07
wallyworld_benji: thanks for the review. i run the tests using "bin/test" and assumed that would "Do The Right Thing"22:12
wallyworld_benji: are you saying it is using the wrong python doing that?22:13
lifelessbenji: whats wrong with testtools as a dep ?22:13
wallyworld_benji: btw, i only ran the one new test i added and it did pass for me locally22:13
benjiwallyworld_: it's more that it's the wrong Python to run the buildout with; you want a "clean" python, one without anything installed in it's site-packages22:13
wallyworld_benji: ah ok. in that case yes i was using the system python22:14
wallyworld_to do the buildout22:14
benjilifeless: nothing, it just looked accidental so I did the easiest thing that came to mind to get past the test failures22:14
benjiwallyworld_: I wouldn't be surprised if the tests fail on the current trunk, if you can verify that for me then tomorrow I'll take a look at straightening them out22:15
wallyworld_benji: ok. you want me to do a fully clean build not using the system python?22:17
benjiright; you can use a virtualenv to get a clean python22:17
wallyworld_benji: will do. thanks again.22:18
benjimy pleasure22:18
lifelessbenji: theres lots of test goodies in there; perhaps we should just add it as a dep to everything22:18
benjilifeless: that'd be fine with me; the problem was that it was an _undeclared_ dependency.  I don't have an opinion one way or the other on it being an actual dependency.  Well, I take that back; it wasn't really used for anything so I'd rather it wait until we actually want it to use it for something.  But that's a pretty low bar.22:20
LPCIBotYippie, build fixed!22:22
LPCIBotProject db-devel build #505: FIXED in 5 hr 6 min: https://lpci.wedontsleep.org/job/db-devel/505/22:22
* wallyworld_ starts a full windmill test run and runs off for a bit to see the accountant so the tax man doesn't come knocking22:48
sinzuiI know how to fix the spastic mhonarc message encoding22:54
sinzuiwgrant: can you see this https://code.launchpad.net/~sinzui/launchpad/mailing-list-archiving-0/+merge/5543523:02
sinzuiwgrant: mumble?23:02
wgrantsinzui: I can.23:03
thumperwgrant: how do I find out what files get installed by a package?23:28
lifelessdpkg -L packagename23:29
allenaplifeless: Yes, I saw load_related recently. Neat :) What about it?23:31
lifelessallenap: seems like the load recipe you mentioned in a recent review would also be generalisable23:33
wgrantsinzui: Are there docs on running a local mailman? I recall I did it 18 months ago, but don't recall how.23:35
allenaplifeless: I included a way to do it with load_related() too. Or are you saying there's scope for another helper? In that case the code turned out to be more concise just using load(), but maybe I missed a trick.23:35
lifelessallenap: I think theres room to improve our tooling furhter23:36
lifelessallenap: not quite sure how, its just a feeling23:36
allenaplifeless: Yeah, I feel the same way.23:38
allenapRight, properly off to bed now.23:38
wgrantNight allenap.23:39
=== StevenK changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: StevenK | https://code.launchpad.net/launchpad-project/+activereviews
* StevenK assumes it is no longer Tuesday or that gmb is actively reviewing. Pointers about both assumptions welcome.23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!