[00:08] <lifeless>  wallyworld ping
[00:08] <wallyworld> lifeless: hello
[00:08] <lifeless> wallyworld: we're blocked on qa for Revision 12671 can not be deployed: needstesting -
[00:09] <wallyworld> lifeless: thanks. i'll look now
[00:09] <lifeless> wallyworld: thanks!
[00:09] <lifeless> StevenK: you have one as well - or is it behind a featureflag?
[00:22] <wallyworld> lifeless: on qastaging it has a problem which is makes deployment a no go (not a blocker locally). it should be fixed by a branch already in ec2. can we wait till it goes though?
[00:23] <lifeless> wallyworld: so it would break prod ?
[00:23] <wallyworld> lifeless: yes
[00:23] <lifeless> tag it bad-commit-NNNN
[00:23] <wallyworld> ok
[00:23] <lifeless> I've fine with waiting for the followup branch thats already playing
[00:27] <benji> bac: I think the test name changed; to run all the tests I've been using just "xvfb-run ./bin/test --layer=RegistryWindmillLayer -c"
[00:27] <benji> oops, wrong chan
[00:43] <huwshimi> lifeless: It appears we can't get oops info for javascript at this point either. Would you be happy with me trying to land the ajax log branch as is, and then we can add the url and oops as an improvement when they become available?
[00:44] <lifeless> huwshimi: I love incremental improvements
[00:44] <lifeless> huwshimi: we don't get the http headers?
[00:45] <huwshimi> lifeless: You mean for the url or the oops?
[00:45] <lifeless> huwshimi: the oops - its a response header
[00:46] <huwshimi> lifeless: We get the headers, but I didn't see anything about an oops...
[00:46] <lifeless> huwshimi: its an X header when an oops occurs
[00:46] <huwshimi> lifeless: Let me take another look
[00:46] <lifeless> I'm just digging up the header id
[00:47] <lifeless> lib/lp/app/javascript/client.js:690:                if (o.getResponseHeader('X-Lazr-OopsId')) {
[00:47] <lifeless> lib/lp/app/javascript/client.js:691:                    server_error = server_error + ' OOPS ID:' + o.getResponseHeader('X-Lazr-OOPSid');
[00:49] <lifeless> huwshimi: ^
[00:49] <huwshimi> lifeless: OK great
[00:51] <huwshimi> lifeless: Ugh, ok you're right, it's there
[00:54] <lifeless> for extra credit
[00:54] <lifeless> clicking on the oops id could take one to pad.lv/OOPS-xxxxx
[00:54] <lifeless> totally optional of course
[00:56] <huwshimi> lifeless: It's trivial to add
[01:00] <huwshimi> lifeless: I've added all this under the "visible_render_time" flag that you added. Is that ok or is it better to have a separate flag for this?
[01:02] <lifeless> huwshimi: thats ideal
[01:02] <lifeless> huwshimi: simplest thing, can always refactor
[01:02] <huwshimi> lifeless: OK great. Do you want to review this branch?
[01:03] <lifeless> huwshimi: I'm not a gun js guy, but I'll certainly give it a shot
[01:03] <huwshimi> lifeless: Sure. Thanks
[01:03] <lifeless> whiskers are so ticklish
[01:08] <wgrant> lifeless: Let me translate: "feed me"
[01:11] <lifeless> wgrant: more 'you make a nice bed'
[01:11] <lifeless> face aimed at my face
[01:14] <lifeless> wow
[01:14] <lifeless> cve:+index templates are smoking something
[01:14] <wgrant> lifeless: Most old templates are :/
[01:14] <lifeless> no really
[01:15] <wgrant> eg?
[01:15] <lifeless> show the bug
[01:15] <lifeless> format a link to it
[01:15] <lifeless> then include the bugtasks-and-nominations-view for the bug
[01:15] <huwshimi> lifeless: https://code.launchpad.net/~huwshimi/launchpad/ajax-time/+merge/55257 for when you get to it.
[01:16] <lifeless> wgrant: ^ - I'll mentor you
[01:16] <wgrant> lifeless: k
[01:19] <wgrant> :( structure
[01:20] <wgrant> huwshimi: You can't put that JS somewhere other than template, with a call in the template to activate it?
[01:20] <wgrant> JS in templates makes me very sad; "structure" doubly so.
[01:23] <huwshimi> lifeless: I'm not sure. I would like it if we had not javascript in templates (and I think we should move towards that). I just don't know where it should live.
[01:26] <huwshimi> wgrant: I meant that for you ^
[01:26] <wgrant> The log also highlights if the AJAX event takes more than 1 minute.
[01:26] <wgrant> That's quite a long request!
[01:26] <wgrant> However, this is really nice.
[01:27] <wgrant> Particularly with the OOPS ID stuff.
[01:27] <StevenK> james_w: Ping!
[01:27] <lifeless> huwshimi: I was suggesting 1 second, not 1 minute, for slow ;)
[01:27] <huwshimi> lifeless: Oh right
[01:27] <huwshimi> lifeless: That's pretty short!
[01:27] <lifeless> huwshimi: goal is 99% completing withing that time
[01:28] <lifeless> huwshimi: perhaps start at 5s
[01:28] <huwshimi> lifeless: I think it's great
[01:29] <huwshimi> actually I think I did make it 1 second
[01:30] <huwshimi> wgrant: Sorry the description is wrong. The code is for 1 second
[01:30] <huwshimi> wgrant: My brain failed me when I was writing the description
[01:31] <wgrant> huwshimi: Ah, that is less completely crazy. Excellent.
[01:31] <wgrant> I think 2-3s is probably more sensible at the moment, but I guess we'll see.
[01:31] <wgrant> Could FF it, but probably YAGNI...
[01:33] <huwshimi> wgrant: Yeah, lifeless suggested 5 seconds. I don't mind changing it.
[01:36] <huwshimi> wgrant: Also if you know of a better place for the javascript to live I can do that as well
[01:41] <james_w> hi StevenK
[01:43] <StevenK> james_w: Hi. :-)
[01:44] <StevenK> james_w: I've been dealing with python-debian for the derived distros stuff -- did you know that the BaseVersion class in debian_support isn't strict enough with parsing versions?
[01:44] <james_w> no, I did not know that
[01:45] <StevenK> james_w: The regex allows a : in upstream version, but that's only allowed if there is an epoch
[01:45] <james_w> erm, eh?
[01:46] <StevenK> james_w: Heh, have I lost you?
[01:46] <wgrant> huwshimi: Reviewed, with a few comments.
[01:46] <james_w> I'm trying to work out if what you said is a tautology or not
[01:46] <huwshimi> wgrant: Thanks, will take a look
[01:47] <StevenK> james_w: 'foo:bar-1' isn't valid, but '1:foo:bar-1' is.
[01:47] <james_w> ah, ok
[01:47] <james_w> epoch can only be digits
[01:47] <StevenK> james_w: Right. I have a patch: http://pastebin.ubuntu.com/586681/
[01:48] <wgrant> huwshimi: The YUI IO events are more limiting than I thought :(
[01:50] <StevenK> lifeless: Yes, r12677 is FF-protected.
[01:50] <StevenK> lifeless: I can mark it qa-untestable, if you wish
[01:51] <lifeless> StevenK: up to you
[01:51] <lifeless> StevenK: we've a bad commit to rectify (its landing via ec2 now) before we can deploy
[01:51] <StevenK> Once I've shaken out more of the stuff on dogfood, I'll be QAing it there, for the short-term
[01:51] <lifeless> StevenK: long as your commit is either known-bad, or deployable in the next 4 hours, we should be fine
[01:52] <StevenK> lifeless: Which is the bad commit, btw?
[01:52] <wgrant> lifeless: Next 4 hours? That's not even enough for buildbot.
[01:52] <lifeless> wgrant: fuzz
[01:52] <lifeless> StevenK: wallyworlds security fix
[01:52] <james_w> StevenK, I'm very glad you didn't try and fix that bug using regexes :-)
[01:52] <james_w> StevenK, hunks 2 and 3 look good to me, but I don't think I like hunk 1
[01:53] <StevenK> james_w: Which would have made the regex about 4 times more complex. I'm masochistic, but not *that* much.
[01:53] <james_w> StevenK, also, I think the comment should be more explicit about what it is checking for, and the error could be clearer about the problem too
[01:53] <james_w> StevenK, aren't you a perl guy though? ;-)
[01:54] <StevenK> Haha
[01:54] <StevenK> james_w: Hunk one is a seperate, but related problem -- the code currently throws a ValueError if the changelog you parsed contains an invalid version
[01:56] <StevenK> james_w: And I've fixed the comment.
[01:59] <StevenK> james_w: The meat of the patch is hunks 2 and 3. Hunk 1 is at best, a workaround, since I don't want invalid versions in my list of versions.
[02:02] <james_w> StevenK, I think that the issue with hunk 1 is better solved at a different level, so I'd leave it out.
[02:02] <james_w> StevenK, please send the patch as a bug report to Debian and hopefully someone will commit it
[02:04] <StevenK> james_w: And solve it in a seperate patch?
[02:04] <james_w> StevenK, what is the calling code that falls over a ValueError?
[02:04] <james_w> presumably it involves a loop over .blocks?
[02:05] <StevenK> ._blocks, yes
[02:05] <StevenK> james_w: get_versions()
[02:05] <james_w> ah, I forgot about that
[02:05] <james_w> don't use that then I would say :-)
[02:06] <StevenK> Haha
[02:06] <StevenK> james_w: But it's ._blocks, it isn't mine to touch :-P
[02:06] <james_w> currently the library is designed to give you good data or nothing
[02:06] <james_w> I think it's public in newer versions
[02:06] <StevenK> Right
[02:06] <lifeless> what library are you discussing?
[02:07] <StevenK> python-debian
[02:07] <james_w> I was slightly naïve in my estimation of how much crap was in all the changelogs in Debian/Ubuntu though
[02:07] <james_w> in old versions etc.
[02:07]  * StevenK is finding out
[02:08] <StevenK> james_w: Crumbs, do you know how long it's been since I filed a bug on Debian :-)
[02:11] <james_w> MIA!
[02:13] <lifeless> whoa
[02:13] <lifeless> 366 Time Outs
[02:13] <lifeless> haproxy reconf FTW
[02:14] <wgrant> The BugTask:+index change is striking.
[02:14]  * lifeless blames making things more efficient
[02:15] <wgrant> We've 25% fewer timeouts than Sunday. That is pretty nice.
[02:15] <StevenK> james_w: Well, ... duh
[02:18] <wgrant> Gaaaaah
[02:19] <wgrant> ShipIt needs to die.
[02:19] <lifeless> ?
[02:19] <wgrant> EmailAddress handling is impossible to do correctly.
[02:19] <wgrant> Because they are linked to both accounts and persons.
[02:20] <wgrant> So you may have an email address without a Person because the user used ShipIt.
[02:20] <wgrant> You can't then create a properly unactivated Person for that email address, since the email address is active.
[02:22] <lifeless> well, we know the email is valid
[02:22] <lifeless> whats the issue
[02:22] <wgrant> We are creating an active Launchpad person for somebody who has never used Launchpad.
[02:23] <lifeless> join the dots for me
[02:26] <wgrant> lifeless: A user creates an SSO account and requests CDs from ShipIt. The parasite creates an Account and an EmailAddress in the LP DB.
[02:27] <wgrant> The user performs some action visible to LP. This could be maintaining a package in Debian, commenting in an upstream bugtracker, etc.
[02:27] <wgrant> LP tries to create an unactivated Person for that email address, so it can be referenced in the DB.
[02:27] <wgrant> But the EmailAddress is already active, and just needs to be linked to the Person.
[02:28] <wgrant> So the Person is now activated, and the user apparently uses Launchpad.
[02:34] <lifeless> wgrant: in other words 'person activation is implicit not explicit'
[02:36] <wgrant> lifeless: Yes.
[02:36] <wgrant> sinzui disagrees that that is a bug.
[02:45] <lifeless> wgrant: why ?
[02:45] <wgrant> lifeless: I don't recall. It's been a while.
[02:46] <lifeless> wgrant: also, if you want to rotflas - lib/lp/bugs/browser/buglinktarget.py - look for except Unauthorized
[02:47] <wgrant> lifeless: :(
[02:47] <lifeless> yes
[02:47] <wgrant> Easy fixes, at least.
[02:48] <lifeless> we were disclosing private bug *existance* and generating links
[02:48] <wgrant> Existence and linkage.
[03:06] <lifeless> content="structure batch_navigator/@@+table-view" -> I want to check precisely what this means
[03:06] <lifeless> its 'the view called +table-view on the object batch_navigator' rendered into the contents of this element
[03:06] <lifeless> right ?
[03:07] <wgrant> Yes.
[03:24] <lifeless> fingers crossed
[03:25] <lifeless> wgrant: what are you hacking on today
[03:27] <wgrant> lifeless: Bug #740594 at the moment.
[03:27] <_mup_> Bug #740594: ProgrammingError: permission denied for relation emailaddress  <bugwatch> <oops> <Launchpad itself:In Progress by wgrant> < https://launchpad.net/bugs/740594 >
[03:31] <lifeless> bah
[03:31] <lifeless> how is the +table-view hooked up
[03:31] <wgrant> lib/canonical/launchpad/zcml/batchnavigator.zcml?
[03:34] <lifeless> thanks
[03:34] <lifeless> grepping to  deep
[03:34] <lifeless> however
[03:34] <lifeless> its not rendering ><
[03:34] <wgrant> Grar, ensurePerson is buggy.
[03:35] <wgrant> If you call it with an EmailAddress with only an Account, it will create a Person, but forget to link the EmailAddress to it.
[03:35] <wgrant> Call it with that EmailAddress a second time, and it will link the EmailAddress to the Person.
[03:35] <lifeless> \o/
[03:36] <lifeless> also
[03:36] <lifeless> \o/
[03:36] <wgrant> batchnavigator fixed?
[03:36] <lifeless> tal:replace
[03:48] <lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/bug-727023/+merge/55266
[03:49]  * thumper looks
[03:53] <lifeless> thanks
[03:53] <thumper> lifeless: question, why create a BugListingBatchNavigator for each bug?
[03:53] <lifeless> thumper: the page shows
[03:53] <lifeless> bug X
[03:53] <lifeless>   tasks
[03:53] <lifeless> bug Y
[03:53] <lifeless>   tasks
[03:55] <lifeless> thumper: it was easy to split it up like this without triggering more queries, so I just kept the prior page layout
[03:55] <thumper> ok
[03:56] <thumper> r=me
[03:56] <lifeless> its a bit noddy really given that there is a portlet showing the exact same stuff :)
[03:56] <lifeless> thanks
[04:23] <LPCIBot> Project windmill build #114: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/114/
[04:27] <nigelb> ah, started working when I put my lp keys into the .ssh folder of the virtual machine
[04:46] <wgrant> lifeless: Thanks. https://code.launchpad.net/~wgrant/launchpad/bug-740594/+merge/55269 is a follow-up.
[04:51] <StevenK> wgrant: r=me
[04:53] <lifeless> StevenK: thanks
[04:53] <wgrant> StevenK: Thanks.
[05:17]  * StevenK ponders how one changes a branch in sourcecode
[05:19] <wgrant> StevenK: Get the change into the branch, fix utilities/sourcedeps.conf.
[05:21] <StevenK> wgrant: Ah, and then toss the branch that changes sourcedeps.conf through ec2 to make sure there isn't fallout?
[05:21] <wgrant> StevenK: Yes
[05:22]  * StevenK looks at updating python-debian to the latest
[05:24] <nigelb> How long does the fetch source code stage take?
[05:24] <nigelb> Its been running for an hour :|
[05:24] <wgrant> What's it up to?
[05:24] <StevenK> Depends on your connection, and if you have any of it, the phase of the moon
[05:25] <lifeless> did we remove the special case of using http fo rlaunchpad clones?
[05:25] <nigelb> Its branching something I guess
[05:25] <wgrant> nigelb: But what?
[05:25] <wgrant> lifeless: That's not update-sourcecode.
[05:26] <nigelb> "Making local branch of Launchpad trunk, this may take a while..."
[05:26] <wgrant> Oh.
[05:26] <wgrant> Not update-sourcecode. I see.
[05:26] <wgrant> lifeless: I don't see hot_products in lp-production-configs, so it seems it's gone.
[05:29] <lifeless> nigelb: it should be about 200MB
[05:30] <nigelb> lifeless: oh, ugh.  Its at something like 90% I think
[05:30] <lifeless> nigelb: lots of code + lots of history
[05:31] <lifeless> when we get around to shrinking the codebase
[05:31] <lifeless> might do a fresh tree and massively compact it
[05:31] <lifeless> but not for $quitesometime
[05:31] <nigelb> I wonder if there's a way to grab it without history
[05:31] <nigelb> or make it notshow history by default
[05:32] <nigelb> Oh. I should get around to signing that agreement I guess.
[05:35] <StevenK> wgrant: https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/55277
[05:38] <wgrant> StevenK: Deleting changelog entries: wgrant says no.
[05:38] <StevenK> wgrant: Think of it as a sync?
[05:38] <StevenK> It was a cherry-pick from upstream anyway
[05:39] <wgrant> k
[05:39] <StevenK> Well, a sync + a small patch :-)
[05:46] <StevenK> lifeless: You mind reviewing wgrant's review? https://code.launchpad.net/~stevenk/python-debian/merge-and-stricter/+merge/55277
[05:50] <wgrant> StevenK: Actually, any reason the ".find(':') != -1" can't be "':' in "?
[05:52] <StevenK> I'm not sure, does that work?
[05:53] <wgrant> I'd hope so... group() returns a str.
[05:54] <StevenK> Right, my string manupliation stuff is dated back to Linda.
[05:55] <StevenK> wgrant: Good catch: http://pastebin.ubuntu.com/586728/
[05:56] <wgrant> StevenK: Looks reasonable.
[05:58] <StevenK> wgrant: Pushing
[05:58] <lifeless> can has review? https://code.launchpad.net/~lifeless/launchpad/bug-741092/+merge/55278
[05:58] <lifeless> huwshimi: fwiw I agree with wgrants review
[05:59] <lifeless> huwshimi: if you need a hand with any of it just shout
[05:59] <huwshimi> lifeless: Yeah, I think they were all good comments.
[05:59] <wgrant> lifeless: Looking.
[06:00] <wgrant> lifeless: Is this to fix the insane 7000 subscriber thing?
[06:00] <StevenK> wgrant: I'm happy to review your review
[06:00] <wgrant> I think we probably want to batch as well, but OK.
[06:01] <wgrant> Ah, you mention that in the commit message.
[06:01] <StevenK> lifeless: You drop an XXX, can the bug related to that be closed (or is it)?
[06:02] <wgrant> It looks like it's the ancient Storm __nonzero__ bug.
[06:02] <wgrant> I suspect.
[06:02] <wgrant> By the age.
[06:02] <wgrant> So it doesn't need closing.
[06:02] <wgrant> Anyway, looks good.
[06:03] <lifeless> StevenK: it was a bogus XXX
[06:03] <lifeless> StevenK: never ever needed
[06:03] <lifeless> wgrant: its to improve it
[06:03] <wgrant> lifeless: Improve/destroy
[06:04] <wgrant> Although I guess retrieving all of them should be fairly quick.
[06:04] <wgrant> We'll see!
[06:04] <lifeless> I had a glitch in there
[06:05] <wgrant> A glitch?
[06:05] <lifeless> I've just pushed a trivial tweak to cache the subscription list too
[06:05] <lifeless> bah bah
[06:05] <lifeless> gimme a sec, to undo my fail
[06:05] <wgrant> Ah, good point.
[06:06] <lifeless> right, pushed
[06:12] <wgrant> StevenK: Want to mentor?
[06:13] <StevenK> wgrant: Done
[06:14] <wgrant> Thanks.
[06:17] <lifeless> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1914M186 is going to be fun to fix
[06:22] <wgrant> lifeless: Delete BranchRevision, problem solved.
[06:22] <lifeless> wgrant: how so?
[06:22] <wgrant> lifeless: It's probably a scanner lock.
[06:22] <lifeless> on branchmergeproposal ?
[06:23] <wgrant> CONTEXT: SQL statement "SELECT 1 FROM ONLY "public"."branch" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x"\\n',)
[06:24] <lifeless> wgrant: where are you getting that ?
[06:24] <wgrant> lifeless: End of the exception message.
[06:25] <StevenK> Hm. Now what do I do ...
[06:25] <StevenK> I doubt I can lp-land a python-debian PQM branch
[06:25] <wgrant> StevenK: pqm-submit
[06:28] <wgrant> lifeless: We're running with the old appserver configs, but a new single-threading haproxy one?
[06:33] <lifeless> wgrant: partly
[06:33] <lifeless> wgrant: depending on whether mbarnett did the tweak requested, we're running on 4 or 6 concurrency from haproxy, and the unaltered appserver configs
[06:34] <wgrant> lifeless: 4 or 6 for the 12-thread servers, 1 for the 2-thread?
[06:34] <lifeless> 4 or 6 for the 4-thread servers
[06:34] <lifeless> 1 for the 2-thread
[06:34] <wgrant> Oh, right, it was 12 for 4 before.
[06:34] <lifeless> + xmlrpc at 1 per backend that is listening for xmlrpc
[07:04] <StevenK> lifeless: First self-review \o/
[07:04] <lifeless> \o/
[07:05] <huwshimi> wgrant: I took a look at your comments and have pushed some changes. Your review were very helpful btw.
[07:06] <wgrant> huwshimi: Great, will look in a sec once the diff is upgraded.
[07:06] <wgrant> updated.
[07:07] <huwshimi> wgrant: or downgraded, depending on the quality of the changes
[07:07] <wgrant> lifeless: Could you please db https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279?
[07:08] <StevenK> wgrant: I do not like the use of \ at line 151
[07:09] <wgrant> I'm not sure whether \ or parentheses are more vile.
[07:09] <StevenK> wgrant: And I don't think the commit() is needed at 187
[07:09] <wgrant> StevenK: It's needed to invalidate the DistroSeries.
[07:10] <wgrant> StevenK: So it pulls in the changes from the script.
[07:10] <lifeless> StevenK: \ is fine
[07:10] <StevenK> wgrant: But the script commits?
[07:10] <lifeless> StevenK: its more cromulent than () just to do a newline
[07:10] <lifeless> *I* would rather we stopped cutting at 80 characters for such things
[07:10] <wgrant> StevenK: Right, but then we have to invalidate the local object to see the committed changes.
[07:11] <StevenK> Fair enough
[07:11]  * spiv waits for autopack of his launchpad repo to finish
[07:14] <spiv> Hmm.  12 minutes, but it did take 30% off the space consumed.
[07:15] <wgrant> huwshimi: Your branch hasn't scanned yet :/
[07:16] <wgrant> Ah, there we go.
[07:18] <wgrant> huwshimi:
[07:18] <wgrant>  7    /* Can't use LPS here as it doesn't exist yet.
[07:18] <wgrant>  8    */
[07:18] <wgrant>  9    LPS.use('node', 'lazr.anim', function(Y) {
[07:18] <StevenK> Heh
[07:18] <huwshimi> wgrant: hahahaha
[07:19] <huwshimi> wgrant: Guess I forgot to take that out when I refactored some things
[07:35] <wgrant> lifeless: Thanks.
[07:35] <wgrant> StevenK: Are you code reviewing my branch?
[07:35] <wgrant> huwshimi: Bear with me here, I'm experimenting with YUI.
[07:36] <StevenK> wgrant: I wasn't, but I can
[07:36] <huwshimi> wgrant: There's no hurry. I'll be heading out soonish so I'll probably just take a look in the morning
[07:36] <wgrant> StevenK: Ah, you were whinging about backslashes and other stuff, so I thought you might have been.
[07:36] <wgrant> It would be great if you could.
[07:37] <wgrant> 7/win 3
[07:37] <wgrant> Grar.
[07:38] <StevenK> wgrant: Done. You just need a number from stub
[07:38] <wgrant> StevenK: Thanks.
[07:54] <wgrant> huwshimi: A couple of suggestions, but it looks good.
[07:59] <huwshimi> wgrant: Thanks for that. I'll take a proper look tomorrow
[08:07] <LPCIBot> Project windmill build #115: STILL FAILING in 1 hr 9 min: https://lpci.wedontsleep.org/job/windmill/115/
[08:17] <wgrant> stub: Hi.
[08:17] <stub> wgrant: yo
[08:18] <wgrant> stub: Could you DB-bless https://code.launchpad.net/~wgrant/launchpad/backports-not-automatic/+merge/55279?
[08:19] <stub> Just done
[08:19] <wgrant> Thanks.
[08:19] <wgrant> stub: The apt flag is negative :/
[08:20] <stub> ok
[08:20] <stub> Just keep away from the double negatives :)
[08:26] <lifeless> stub: ping
[08:26] <stub> lifeless: pong
[08:26] <lifeless> stub: two things I'd like your help evaluating
[08:27] <lifeless> stub: first is the bug heat thing - I saw different results on the candidate index (but only /great/ for ubuntu)
[08:27] <lifeless> stub: I'd like to debug why we saw different results; and evaluate denormalising heat down into bugtask
[08:28] <lifeless> we might be able to squeeze that into this db cycle even
[08:29] <lifeless> secondly, BranchRevision doesn't support range queries by date - but merge proposals (and other interesting queries) are all date based, so I'd like to evaluate the impact of denormalising the revision date into branch revision
[08:29] <stub> I'm thinking denormalizing into a separate table would be better - I think the regular heat recalculation is contributing to bloat.
[08:30] <lifeless> https://bugs.launchpad.net/launchpad/+bug/742916 is a bug showing the merge proposal query (for a merge proposal - old data, will be out of cache a lot)
[08:30] <_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >
[08:30] <stub> I don't have a problem denormalizing the heat. It is a cache anyway - ideally it would be calculated on the fly but that is far to expensive.
[08:30] <lifeless> stub: the concern I have with a different table is that the plan problem with sorting on a different table would still exist, unless that table had (heat, status, bug target)
[08:31] <stub> For BranchRevision, this has been suggested in the past but not acted on (grand plans of removing the table alltogether and all that)
[08:31] <stub> lifeless: True. Such a table is what we might end up with for bug search anyway - all the searchable fields denormalized into a BugSearch table, but that is a much larger task.
[08:32] <lifeless> stub: can we add a datetime column, and an appropriate index to handle the query in https://bugs.launchpad.net/launchpad/+bug/742916 (e.g. (branch, sequence, revision_date)) on staging/qastaging
[08:32] <_mup_> Bug #742916: BranchMergeProposal:+index timeouts - slow query plan <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/742916 >
[08:32] <lifeless> stub: so that we can determine the disk impact?
[08:32] <lifeless> s/we/you/
[08:32] <stub> lifeless: we can. It will take several hours.
[08:33] <lifeless> stub: no time like the present (if we're going to do it, it would be good to add it (NULLable) this cycle
[08:33] <stub> lifeless: There is work we need to test to drop the id column that should counteract the disk space impact.
[08:34] <lifeless> stub: the column add should be instant, right? and the index able to be done CONCURRENT ?
[08:34] <stub> Yes. A job to populate the column will take ages though.
[08:35] <lifeless> stub: sure, I think I can make it tolerably efficient
[08:35] <lifeless> stub: would like to parallelise the garbo perhaps to help it and othe migrations carry on a little faster
[08:35] <stub> We need that job too - can't see the disk space impact until the rows have all been rewritten with data in that column.
[08:36] <lifeless> stub: ah, well for the test we can do it with an update
[08:36] <stub> Adding --threads= to garbo should be trivial.
[08:36] <lifeless> update branchrevision set revision_date = revision.revision_date from branch_revision, revision where branch_revision.revision=revision.id;
[08:37] <lifeless> ^ untested, but approximate
[08:37] <stub> lifeless: The script will likely be faster, as it works in batches rather than attempting to do it all in one transaction. Also, we have to do it in chunks or we will double the space used by that table
[08:38] <stub> (all the old rows and all the new rows will exist, and then vacuum will clean up all the old rows giving 50% bloat)
[08:38] <lifeless> staging might run out of space
[08:38] <lifeless> ok
[08:38] <lifeless> I guess branchrevision should wait for next week then - I suspect we couldn't qa in time
[08:39] <lifeless> bugtask.heat I'll whip up a patch for tonight.
[08:39] <lifeless> stub: do you have any idea why you didn't see the performance benefits on the heat query I did?
[08:40] <lifeless> stub: my only WAG is that your test query wasn't adjusted, or something
[08:40] <stub> I was testing ordering by heat, id. You had a performance boost by dropping id from the sort.
[08:40] <lifeless> stub: I tried on qastaging with a (heat, id desc) index and a query sorting by bug.heat desc, bug.id
[08:41] <stub> I managed to get the existing query to use the new index once, and it saved a few seconds, but the query was still two orders of magnitude slower than your no id sort query.
[08:41] <lifeless> ah
[08:42] <lifeless> stub: so, I think then we weren't testing quite the same query - because I did adjust the query; I will try to be less ambiguous in future
[08:42] <stub> No, I was expecting different results :-) It would have been nice to drop in an index and keep stable ordering, but I don't see that happening.
[08:43] <lifeless> ah, ok - then *I* didn't understand  :)
[08:44] <lifeless> anyhow, I think its clear to make it snappy we need heat in the same table
[08:44] <lifeless> otherwise it has to traverse *tonnes* of bugs to find the first bug in a context with no really hot bugs
[08:44] <lifeless> I can guess that having lots of bugs often equates to having some very hot ones
[08:45] <lifeless> but the plan - I pasted it in - for the updated query + new index on launchpad was pretty expensive
[08:46] <lifeless>  Limit (cost=0.00..146128.04 rows=40 width=1017) (actual time=1.302..500.965 rows=40 loops=1)
[08:46] <lifeless> (this is actually 50% cheaper than the *current* plan, but still awful)
[08:48] <lifeless> stub: would you like a bug about the garbo parallel processing ?
[08:48] <lifeless> stub: I don't think we need a --threads option
[08:48] <lifeless> stub: multiprocessing.cpu_count() will return the # of cpus
[08:48] <lifeless> stub: we can just run that many threads
[08:49] <stub> Garbo jobs are almost certainly database bound, so not sure if that is so useful.
[08:49] <stub> I've considered adding options to only run specific tasks (but how to handle lock files is unclear), and the --threads.
[08:50] <stub> I suspect we want a lock file per task
[08:50] <stub> Rather than a single lock for the script.
[08:56] <stub> Bug #744803
[08:56] <_mup_> Bug #744803: Better garbo locking <Launchpad itself:Triaged> < https://launchpad.net/bugs/744803 >
[08:58] <adeuring> good morning
[09:03] <stub> Bug #744804
[09:03] <_mup_> Bug #744804: Parallel garbo <Launchpad itself:Triaged> < https://launchpad.net/bugs/744804 >
[09:05] <lifeless> stub: if we run N garbos we'll need per-tuneable locks yes; if we run one master that parallelises I think we can ignore the locking for now
[09:06] <lifeless> stub: I think many garbo jobs are overhead bound - I mean, time to pull stuff out, mangle it, shove it back in; others are going to be db bound - but we should remember we have 8 available cores on master atm, and most garbo jobs are completely up to date, we'll only have a few migrations in flight at any one time
[09:07] <lifeless> stub: I don't particularly care about implementation, the reason I suggested using cpu count is that it avoids manual selection (at the cost of not being tunable)
[09:08] <stub> Sure. I'll probably add --threads and have it default to cpu count :)
[09:08] <lifeless> stub: all that said, I'd love to see things like stevenk's migrator, and the branchrevision one next week, get more of a timeslice than they do at the moment - ideally 55minutes per hour
[09:10] <stub> k
[09:11] <StevenK> That sounds good.
[09:11] <lifeless> stub: so I think that is the key goal: more time for migrations in the garbo
[09:11] <lifeless> stub: everything else is tactics
[09:12] <stub> I won't suggest to jtv that if looptuner became thread aware it could tune the number of instances dynamically to maximize throughput based on database load ;)
[09:12] <lifeless> stub: another thing, probably EFUTURE, would be allowing the loops themselves to parallelise
[09:12] <lifeless> stub: please don't :P
[09:13] <jtv> stub: as if you'd need to!
[09:13] <jtv> You think I don't think about these things?
[09:13] <stub> I know you do :-)
[09:14] <stub> Actually, garbo 2.0 is looking like a twisted system... but I think we are several years away from that.
[09:14] <lifeless> bug 744808 is a browser/X bug right ?
[09:15] <_mup_> Bug #744808: Redrawing of Bug View page very slow in Firefox <Launchpad itself:New> < https://launchpad.net/bugs/744808 >
[09:15] <jtv> It does sound like one we had.
[09:18] <lifeless> stub: can you do a quick timing check for me on staging ?
[09:18] <lifeless> stub: add a heat column to bugtask and copy the heat over from bug
[09:18] <lifeless> stub: bugtask is only 250MB in size
[09:19] <lifeless> stub: so just a single sql statement should be fine - I want to see if its fast enough to do during downtime
[09:27] <lifeless> stub: can I take 59 for this ?
[09:50] <stub> gah... distracted
[09:51] <stub> lifeless: 59... sure.
[09:51] <lifeless> stub: pastbinnning a script
[09:53] <lifeless> stub: http://pastebin.com/fUQN4JvF
[09:54] <stub> The productseries index seems to have a bogus WHERE clause
[09:54] <lifeless> you pass :P
[09:54] <lifeless> [bad copypaste, thanks]
[10:05] <lifeless> stub: so, did that work, how long to do the update?
[10:05] <stub> lifeless: I'll shove that through slony now
[10:06] <lifeless> stub: thanks
[10:09] <lifeless> stub: can a trigger update columns in a related table?
[10:09] <stub> lifeless: yes
[10:09] <stub> lifeless: I'd rather just remove Bug.heat though
[10:09] <lifeless> how do we update stuff in trusted.sql ?
[10:10] <lifeless> stub: I'll keep that in mind as I look through this
[10:10] <stub> You just edit it. And things can break if you remove things.
[10:10] <lifeless> calculate_bug_heat is what I'm changing, for obvious reasons
[10:11] <lifeless> # Bugs decay over time. Every day the bug isn't touched its heat  ---- needs to die
[10:11] <stub> Just change it inplace then. You shouldn't need to alter the name or signature.
[10:11] <lifeless> hah
[10:11] <lifeless> its not even a trigger
[10:11] <lifeless> its just an in-db function
[10:12] <lifeless> which of course... headdesk
[10:13] <lifeless> stub: might need a hand doing a trigger for this - we don't want to calculate the heat once per task, so keeping the bug.heat seems easier.... but we need to copy any changes to bug.heat to the bugs bugtasks.heat
[10:19] <stub> We need to shutdown most of the staging systems - its too busy to apply the patch live.
[10:20] <lifeless> ok; lets do it
[10:34] <lifeless> stub: triggered up
[10:34] <lifeless> http://pastebin.com/xMfWZWFE
[10:34] <lifeless> gmb: you lost a P
[10:35] <gmb> Ha.
[10:35] <bigjools> the irony.  I'm just going for one.
[10:35] <lifeless> TMI
[10:35] <bigjools> lifeless: you'll like this: http://1.bp.blogspot.com/-YhwtjjSfRf4/TZBOYuLTNVI/AAAAAAAAAP0/0P7KUdJ1RBI/s1600/James.png
[10:36] <lifeless> bigjools: rotfl
[10:36] <stub> lifeless: Do you want that trigger now? If so, we should not try and monkey patch this - just land it as normal and back things out if the timings suck
[10:37] <lifeless> stub: don't need it for the timings
[10:37] <lifeless> stub: Just want your eyeballs on the updated db side of the patch
[10:38] <stub> lifeless: So the alternative of changing the UPDATE Bug SET heat=calculate_bug_heat(...) WHERE id=... to UPDATE BugTask SET heat=calculate_bug_heat WHERE bug=... and dropping Bug.heat sucks?
[10:38] <lifeless> stub: some bugs have hundreds of tasks
[10:39] <lifeless> stub: and bug heat calculation isn't, uhm, efficient
[10:39] <stub> lifeless: How does this change that?
[10:40] <lifeless> stub: oh, and I need a check for NEW.heat != OLD.heat
[10:40] <lifeless> stub: uhm, it copies the one calculated value from the bug, rather than doing it once per row - or would pgsql optimise that out ?
[10:41] <stub> UPDATE BugTask SET heat=calculate_bug_heat($bug) WHERE bug=$bug; calls the stored procedure once.
[10:42] <stub> Well... assuming the function hasn't been declared to volatile
[10:43] <stub> Its declared stable, so should only be invoked once.
[10:43] <lifeless> stub: ok; I'm inclined to go inchwise for now anyway, simply because there are a few hundred references to heat.
[10:43] <lifeless> and I'd like to get this in tomorrow, which a more aggressive branch is more likely to fail
[10:55] <jml> is there still a MOTU liaison to Launchpad?
[10:57] <lifeless> wgrant last I heard :P
[10:57] <gmb> jtv: https://code.launchpad.net/~jtv/launchpad/db-bug-55798/+merge/55174 says "This is still a work in progress; expect a proper cover letter later when it goes up for review" yet it's marked "needs review". Should it be marked "work in progress"?
[10:57] <jtv> gmb: I thought I did mark it WIP… thanks for noticing.
[10:58] <gmb> jtv: I'll do it now.
[10:58] <jtv> Thanks.
[10:58] <lifeless> stub: updated with the if condition - http://pastebin.com/bYHpVBWH
[10:58]  * gmb just marked it Approved for a second. Would've been funny if we'd been doing automated merges
[10:58] <gmb> jtv: Done.
[10:58] <lifeless> stub: if I'm correct, this is landable once we confirm the migration time
[10:59] <lifeless> stub: and we can then do the better query post db deploy
[11:00] <wgrant> jml: So, yeah, not really.
[11:00] <jml> wgrant: I didn't think so.
[11:00] <wgrant> jml: People basically all just know to poke me.
[11:11] <lifeless> stub: I'm going to propose this for merge
[11:13] <henninge> Hi!
[11:13] <henninge> I get this when running "make lint": http://paste.ubuntu.com/586801/
[11:13] <henninge> Updating my sourcedeps now ...
[11:15] <henninge> no joy ;(
[11:15] <stub> lifeless: I'd say land it anyway so we don't block on losas.
[11:15] <stub> lifeless: We can rework it if the rollout time sucks
[11:16] <stub> lifeless: The trigger should be INSERT OR UPDATE
[11:17] <jml> henninge: there's nothing there that will come from sourcedeps
[11:17] <lifeless> stub: why? INSERT never has anything to do
[11:17] <wgrant> henninge: That class is new in 2.7 :(
[11:17] <jml> henninge: what version of pocketlint do you have?
[11:17] <lifeless> stub: because bugtask has a foreign key on bug
[11:17] <henninge> jml: yes, I realised
[11:17] <wgrant> henninge: Could you file a bug on pocketlint?
[11:17] <wgrant> henninge: sinzui will fix it quickly I'm sure.
[11:17] <wgrant> Otherwise downgrade python-pocketlint.
[11:17] <jml> ahh, there you go.
[11:18] <henninge> wgrant: Will do, thanks.
[11:18] <StevenK> Oh, is that the upgraded python-pocketlint I haven't done yet? Bonus.
[11:18] <wgrant> python-pocket-lint
[11:18] <stub> lifeless: oic. yes.
[11:18] <bigjools> allenap: in your review you said that stuff returned from canonical_url needs to be escaped.  Really?
[11:18] <stub> lifeless: So when a bugtask is created, it should get the heat of the attached bug. Probably better having that in Python land rather than another trigger.
[11:18] <lifeless> stub: as we only need to sort on the high heat tasks initially, as long as new tasks have heat 0, the heat garbo job will sort it out
[11:19] <lifeless> stub: and yes, we can also sort it out in python; the column is non-null
[11:19] <wgrant> Every problem can be solved with another layer of triggers.
[11:19] <lifeless> wgrant: except having too many triggers
[11:19] <stub> lifeless: So as the patch stands at the moment, creating a bugtask will fail because it will have a NULL heat.
[11:20] <lifeless> stub: default 0 ?
[11:20] <wgrant> lifeless: We can have a trigger to generate the triggers on demand and delete them when they're not needed any more!
[11:20] <stub> Might want a DEFAULT 0 in there if you want to land this separate from code changes.
[11:20] <lifeless> yeah, I'll just tweak the alter
[11:20] <stub> lifeless: Set the default at the same time or after setting the NOT NULL - not in the initial ALTER TABLE
[11:22] <lifeless> stub: https://code.launchpad.net/~lifeless/launchpad/bug-618406/+merge/55303
[11:22] <lifeless> stub: yeah, I did :)
[11:22] <stub> Its a non-obvious footgun :)
[11:22] <lifeless> bah, I left 58 in the .sql file
[11:22] <lifeless> changing to 59
[11:22] <stub> lifeless: 58 is fine
[11:23] <lifeless> stub: the .sql file is called 59
[11:23] <stub> ok
[11:23] <lifeless> stub: they need to match, don't they ?
[11:23] <stub> yes
[11:23] <lifeless> stub: I can rename both to 58, or leave - up to you
[11:23] <stub> I don't care. I have many integers :)
[11:23] <lifeless> \o/
[11:26] <henninge> wgrant: bug 744873
[11:26] <_mup_> Bug #744873: lint fails on Python 2.6 <pocket-lint:New> < https://launchpad.net/bugs/744873 >
[11:26] <allenap> bigjools: Yes. Everything that goes into HTML/XML/XHTML should be escaped unless it has been escaped already.
[11:27] <bigjools> allenap: ok
[11:27] <wgrant> bigjools, allenap: canonical_url should always be safe, but if you don't escape it then you have a bad habit.
[11:27] <lifeless> stub: https://code.launchpad.net/~lifeless/launchpad/bug-618406/+merge/55303 has its fidd
[11:28] <wgrant> And if it's easier to not escape it than it is to escape it, then we have a systemic problem.
[11:28] <lifeless> oh fail, EWRONGTARGET
[11:28] <bigjools> wgrant: my point
[11:28] <allenap> I think we do have a systemic problem then :)
[11:28] <lifeless> take two
[11:28] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-618406/+merge/55306
[11:28] <wgrant> We do :)
[11:29]  * stub waits again for the diff
[11:30] <allenap> wgrant: One of my suggestions for bigjools in the review was to use lxml.html to build fragments. For very small things it appears heavyweight, but imo it soon pays off, and the cssselect() method makes it similar in use to Y.Node.
[11:31] <stub> wgrant: I wonder if we can hack structure to explode if it is passed something not known to be escaped? We already have monkey patches in place for the cache: stuff.
[11:31] <bigjools> allenap: I liked that but I chickened out and used cgi.escape
[11:31] <lifeless> bigjools: you know thats incomplete right ?
[11:31] <wgrant> cgi.escape(foo) is fine for element contents.
[11:31] <wgrant> It is not fine for attributes.
[11:31] <allenap> I pointed out the quote argument too.
[11:32] <wgrant> You need cgi.escape(foo, True) for attributes, and you cannot use single quotes around the attribute.;
[11:32] <wgrant> stub: I am considering changing zope.tal to use markupsafe for escaping, which will allow us to just pass markupsafe objects into the template without structure.
[11:32] <wgrant> stub: Then any use of structure that isn't a widget will be easily visible.
[11:32] <lifeless> stub: diff up
[11:32] <stub> looking
[11:38] <lifeless> thanks
[11:39]  * lifeless breaks the build
[11:39] <lifeless> gnight all
[11:40] <lifeless> hmm, wonder if wallyworld's fix landde
[11:40] <lifeless> looks like it, we should be able to deploy if someone wants to coordinate
[11:40] <lifeless> night all
[11:44] <adeuring> gmb: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/js-translation/+merge/55309 ?
[12:02] <deryck> Morning, all.
[12:03] <adeuring> morning deryck
[12:03] <adeuring> gmb, ping
[12:06] <henninge> gmb: Hi!
[12:08] <henninge> gmb: Can you please review my branch? I will be having lunch and switching locations now but I'll be happy to answer questions after that.
[12:08] <henninge> https://code.launchpad.net/~henninge/launchpad/devel-737418-remove-sharing-links/+merge/55313
[12:14] <wallyworld> lifeless: fix has landed
[12:16] <wgrant> Fix is going to be deployed once the staging mailbox loads.
[12:21] <LPCIBot> Project windmill build #116: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/116/
[12:32] <wgrant> Grar.
[12:32] <wgrant> Stupid Pacific.
[12:32] <wgrant> And/or Optus.
[12:32] <wgrant> I would like more than 10KB/s to international hosts tonight, thanks...
[12:48] <gmb> adeuring, henninge-lunch: Sure.
[12:49] <adeuring> gmb: thanks
[12:49] <jml> wgrant: on cable? maybe there's a popular show on the telly
[12:54] <henninge> gmb: thanks a lot
[12:54]  * henninge reloates
[12:54] <henninge> relocates
[12:54] <henninge> even
[12:59] <jml> (no, I don't understand anything about how physical networks work)
[13:01] <wgrant> jml: Sadly they don't work that way.
[13:01] <wgrant> And it's only international.
[13:01] <wgrant> cjwatson: The buildbot slaves are updated, and I've hopefully built a sufficient EC2 AMI, which I'm about to test your binary xz branch against. If it goes well I'll land it tomorrow.
[13:03] <StevenK> wgrant: And added your ID to ec2 land?
[13:03] <wgrant> StevenK: In this branch, yes.
[13:05] <gmb> adeuring: r=me
[13:07] <LPCIBot> Project windmill build #117: STILL FAILING in 45 min: https://lpci.wedontsleep.org/job/windmill/117/
[13:23] <cjwatson> wgrant: excellent.  is that the data.tar.xz one?
[13:24] <wgrant> cjwatson: Yeah.
[13:25] <cjwatson> great.
[13:25] <wgrant> jml: I don't know how I lived without ec2 list.
[13:25] <wgrant> cjwatson: Should both be deployed on the 6th, I guess.
[13:25] <jml> wgrant: :)
[13:25] <cjwatson> I was just about to ask that.
[13:26] <wgrant> cjwatson: We'll hopefully be able to get cocoplum into the daily nodowntime rollouts eventually, but it's more difficult than just about anything else.
[13:26] <cjwatson> I can imagine.
[13:27] <gmb> henninge: r=me
[13:31] <jml> I'm not hopeful for an answer but it's worth asking...
[13:32] <jml> is there a programmatic way to find out what, say, a distribution owner is allowed to do?
[13:35] <bigjools> haha
[13:36] <bigjools> security.py nicely obfuscates that :/
[13:37] <wgrant> At least most Soyuz stuff is neatly encapsulated in security.py
[13:37] <wgrant> For other stuff (bugs come to mind).. uh, good luck.
[13:37] <bigjools> I'm not that sure it is
[13:38] <bigjools> there's a few methods that take user args
[13:38] <wgrant> Soyuz has an entirely separate permission system for it's complex stuff.
[13:38] <wgrant> Mm, true.
[13:38] <wgrant> But not many.
[13:38] <bigjools> separate - as in "do what you like!"
[13:38] <adeuring> gmb: thanks!
[13:39] <bigjools> which I will endeavour to fix for some cases for DDs
[13:39] <wgrant> bigjools: Oh?
[13:40] <bigjools> yes, I want to make distro management done in the UI/API as much as possible
[13:40] <wgrant> Oh, I was referring to ArchivePermission.
[13:40] <wgrant> Not ssh cocoplum.
[13:40] <bigjools> the coming year is going to be painful
[13:40] <jml> wgrant: manual inspection hasn't proved too hard at this point
[13:40] <jml> just... manual
[13:41] <wgrant> bigjools: I was looking forward to working on some of that stuff, TBH :/
[13:41] <jml> next time I hear someone say "for audit purposes", I'm going to ask them how they are going to audit.
[13:41] <bigjools> wgrant: what's stopping you? :)
[13:41] <wgrant> '
[13:42] <wgrant> It's out of scope of the maintenance squads :(
[13:53] <henninge> gmb: Thanks for the review! ;) I had already filed bug 744873 about pocketlint.
[13:53] <_mup_> Bug #744873: lint fails on Python 2.6 <pocket-lint:New> < https://launchpad.net/bugs/744873 >
[13:53] <gmb> henninge: Cool, thanks.
[14:01] <deryck> henninge: ping for standup
[14:01] <henninge> deryck: coming
[14:31] <jml> why is https://blueprints.qastaging.launchpad.net/launchpad showing different data to https://blueprints.launchpad.net/launchpad ?
[14:34] <jml> apparently because they are configured differently
[14:57] <jml> sinzui: hey, you had a XXX analyzer, right?
[14:59] <sinzui> jml: I wrote the script in utilities
[15:00] <sinzui> I am uncertain of its value
[15:01] <jml> sinzui: I was wondering which XXX comments refer to fixed bugs.
[15:01] <jml> sinzui: your script helps
[15:01] <sinzui> jml: :) I found one a few days ago actually
[15:03] <bac> sinzui: when i run 'make lint' i get an import error for ParseError.  did you miss a dependency on pocketlint?
[15:03] <sinzui> jml: utilities/xxxreport.py -f c ./xxx-report.csv
[15:03] <jml> sinzui: ./utilities/xxxreport.py -f csv  . | cut -d, -f5 | sort -n | uniq
[15:03] <jml> http://pastebin.ubuntu.com/586882/
[15:04] <sinzui> jml: did did something similar to feed and api script that added tech-debt to each of those bugs
[15:04] <jml> shame there's no way to get all of those bugs in one API request.
[15:04] <henninge> bac: bug 744873 ;)
[15:04] <sinzui> some most bugs are in the launchpad-project. A few are in Zope
[15:04] <_mup_> Bug #744873: lint fails on Python 2.6 <pocket-lint:Fix Committed by sinzui> < https://launchpad.net/bugs/744873 >
[15:05] <jml> sinzui: I imagine a few are in Bazaar or Twisted too.
[15:05] <sinzui> oh, yes, some are in twisted
[15:05] <bac> thanks henninge
[15:08] <jml> how do you get a bug by id in the API?
[15:08] <jml> huh, just add it to the end of the bugs collection
[15:10] <sinzui> jml: something like this: http://pastebin.ubuntu.com/586883/
[15:11] <jml> sinzui: ta
[15:12] <abentley> henninge: if you'd like to discuss my changes, I'm free.
[15:18] <henninge> abentley: your code page is timing out on me ...
[15:18] <abentley> henninge: the loggerhead page?
[15:18] <henninge> abentley: which branch should I look at?
[15:18] <henninge> https://code.launchpad.net/~abentley
[15:19] <abentley> henninge: the cumulative changes are in lp:~abentley/launchpad/js-translation-2
[15:20] <henninge> abentley: thanks
[15:20] <henninge> abentley: I'll play around with it a bit
[15:20] <abentley> henninge: Cool
[15:21] <jml> sinzui: http://paste.ubuntu.com/586884/ all of the bugs in XXX comments that have a closed task
[15:22] <jml> sinzui: that's over a quarter of all bugs mentioned.
[15:22] <sinzui> jml: :) Maybe we should have had an action item to followup on this
[15:22] <sinzui> This is very good progress
[15:22] <LPCIBot> Project db-devel build #503: FAILURE in 43 min: https://lpci.wedontsleep.org/job/db-devel/503/
[15:23] <jml> sinzui: an action item for who to follow up on this, exactly.
[15:24] <sinzui> jml: We concluded the report was not driving the XXX to zero, but it would have been nice to check the report every thunderdome to see if we are making progress
[15:24] <jml> sinzui: ahh yes.
[15:24] <jml> sinzui: well, that's probably a good idea for next thunderdome, especially if those XXXs for closed bugs are removed from the tree
[15:26] <jml> well, that's enough play for me. back to the grindstone
[15:26] <sinzui> henninge: pocket-lint is fixed for python 2.6  and published in launchpad's ppa
[15:27] <henninge> sinzui: cool, thanks for the prompt fix
[15:28] <sinzui> It was just as prompt as my fix for python 2.7...but this time both work. I was incompetent.
[15:35] <bac> sinzui: i'm getting an ExpatError from pocketlint (the new one) http://pastebin.ubuntu.com/586887/
[15:36] <sinzui> bugger
[15:36] <bac> sinzui: you want my branch for testing?
[15:36] <sinzui> no
[15:37] <sinzui> bac: I did not see those changes in the diff :(. I made this change a long time ago
[15:40] <sinzui> bac: henninge: you will need to wait another 60min for a fix. I see the second change in annotate and it is simple to treat both kinds of errors separately.
[15:40] <LPCIBot> Project windmill build #118: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill/118/
[15:43] <sinzui> I sense some irony in this mess. I was slow to release pocket-lint fixes in the past because I lost the test suite when I divorced the linter from gedit-developer-plugins. I rebuilt the test suite so I now release quickly. The test suite runs with the default python, which for my 2.7
[15:44] <bac> sinzui: i made a wee fix to formatcheck.py that allows me to go forward
[15:46] <bac> sinzui: it is also ironic that formatcheck.py has lint errors in it.  :)
[15:46] <sinzui> bac: it did not until I had to backport :)
[15:47] <sinzui> pyflakes hates import redefinitions
[15:52] <jml> there are ways to avoid redefinitions
[15:52] <jml> hey,
[15:52] <jml> does "Contact this team" send emails to teams if the team-within-a-team has a contact address?
[15:55] <jml> looks like it contacts members with preferred emails
[15:55] <jml> can teams have preferred emails?
[15:57] <jml> mrevell: do you know how this works?
[15:57]  * mrevell reads up
[15:58] <mrevell> jml, Teams can have preferred emails. They can choose from "Email every member individually", "email the team's list, if it exists", "email some other address that the admin has specified"
[15:58] <mrevell> So, I assume the team-within-a-team thing will respect the team-within-a-team's choice.
[15:59] <mrevell> But I don't know for sure.
[15:59] <jml> thanks
[15:59] <jml> frustrating that this is so hard to figure out.
[16:13] <henninge> abentley: Shouldn't all the items of the check list be active already?
[16:14] <abentley> No, that depends on the state of the sourcepackage.
[16:15] <henninge> abentley: by 'active' I meant 'using ajax'
[16:17] <abentley> henninge: You mean that the page where users select the target branch for a sourcepackage already uses ajax, for example?
[16:18] <henninge> abentley: I can set the targe branch for the upstream project using ajax.
[16:18] <henninge> abentley: and I can search for a upstream product series
[16:19] <henninge> but the edit icons next to the other two items in the list take me to the edit pages for those (web 1.0)
[16:19] <henninge> abentley: I guess I am just asking if you have pushed your latest code ... ;)
[16:20] <abentley> henninge: Yes, I have pushed all my latest code, and the last two items still need to be done.
[16:20] <henninge> ah!
[16:20] <abentley> henninge: I am currently fixing up the windmill tests for selecting a productseries.
[16:21] <henninge> abentley: np, I was just wondering because I see that there is already code for those in the js.
[16:21] <abentley> henninge: There is model code, but I wan't sure what to do for the UI code until I spoke with deryck yesterday afternoon.
[16:23] <LPCIBot> Project db-devel build #504: STILL FAILING in 42 min: https://lpci.wedontsleep.org/job/db-devel/504/
[16:35] <henninge> abentley: thanks. I will EOD now and continue tomorrow.
[16:35] <abentley> henninge: cool
[16:36] <jml> looks like the build failure is a legit one
[16:37] <jml> anyone addressing it?
[16:58] <LPCIBot> Project windmill build #119: STILL FAILING in 35 min: https://lpci.wedontsleep.org/job/windmill/119/
[17:00] <jml> quick review for testfix? http://pastebin.ubuntu.com/586935/
[17:06] <jml> taking that as a "no"
[17:06] <jml> landing w/out review
[17:07] <bigjools> jml: r=me
[17:07] <jml> bigjools: thanks.
[17:08]  * bigjools pines for the r=trivial days
[17:08] <jml> heh. this isn't trivial, per se
[17:09] <bigjools> I actually think testfixes should be mandatory trivial - else back the revision out
[17:13] <jml> hmm
[17:14] <jml> bzr pqm-submit --submit-branch=bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel -m '[testfix][r=bigjools] Be explicit about which table we get heat from.'
[17:14] <jml> what's wrong with that line?
[17:14] <jml> it's acting as if it works but nothing happens on pqm.launchpad.net or lpbuildbot
[17:14] <bigjools> [no-qa] missing?
[17:14] <bigjools> or bug=
[17:22] <jml> bigjools: seems to work now, thanks. maybe it was just a delay (or fast PQM processing + slow buildbot)
[17:22] <jml> bigjools: interesting idea re mandatory trivial testfixes. certainly a good rule of thumb.
[17:24] <nigelb> flacoste: thanks.
[17:25] <bigjools> jml: indeed - the idea being that all the time it's not fixed, everyone is blocked, so we need to spend the minimum amount of time fixing it.
[17:29] <jml> ...
[17:29] <jml> there's a launchpad.Driver permission
[17:29] <jml> for some reason, that makes me wish I could swear like a 19th century Parisian.
[17:31] <bigjools> just look up the Merovingian's line in The Matrix
[17:31] <jml> there's a thought.
[17:32] <jml> although I doubt that whatever the real Merovingians spoke sounded like that.
[17:32] <bigjools> "It's like wiping your arse with silk"
[17:33] <jml> An analogy for which I lack direct experience
[17:33] <jml> (since I've never sworn in French, of course)
[17:33] <bigjools> of course
[17:42]  * jml is filling in https://wiki.ubuntu.com/LaunchpadPermissions very very slowly
[17:48] <jml> !!@#@!
[17:49] <jml> ok. I am sick of surprises.
[17:53] <bigjools> nytol
[17:59] <jml> g'night all
[18:05] <jml> ugh.
[18:05] <jml> the build is failing again
[18:06] <jml> looks like a hangover from the previous test failure. forcing a new build after the failure ought to fix it.
[18:06] <jml> I know there are plenty of North Americans around to read this in a timely fashion, so I'm not going to send an email.
[19:53] <lifeless> jml: thank you
[20:04] <lifeless> sinzui: ping
[20:04] <sinzui> hi lifeless
[20:04] <lifeless> hiya
[20:04] <lifeless> you helped me get an oops for https://bugs.launchpad.net/launchpad/+bug/741092
[20:04] <_mup_> Bug #741092: Archive:+subscriptions times out with many subscribers <qa-needstesting> <timeout> <Launchpad itself:Fix Committed by lifeless> < https://launchpad.net/bugs/741092 >
[20:04] <sinzui> \o/
[20:04] <lifeless> could you perhaps help me qa it - see if it lets you view it cold, and after a few retries, on qastaging ?
[20:05] <lifeless> I haven't batched it
[20:05]  * sinzui types the url
[20:05] <lifeless> but in theory we can handle 7K in one shot; if it doesn't render even after a few refreshes we probably are running out of storm cache or something and will need to batch :(
[20:07] <sinzui> lifeless: I do not have permission to see that one, but I think the beta-font team has an equally pathological subscriber page. I will check production first
[20:09] <sinzui> ;/
[20:09] <lifeless> I found the design there a little confusing too, given it doesn't use teamparticipation, but presumably we need to permit new members to see automatically
[20:14] <sinzui> lifeless: I asked achuni to help since he reported the bug. The cold cache does timeout: OOPS-1914QS128 and the second try does too: OOPS-1914QS129
[20:17] <lifeless> sinzui: 'darn'
[20:17] <lifeless> sinzui: can you get him to try 6 times
[20:17] <lifeless> sinzui: (6 times because if its /just/ cold cache in the DB, we are only walking Person records 10 seconds worth at a time
[20:18] <lifeless> I'm syncing qastaging to look at the oops
[20:18] <sinzui> :) I explained the same to achuni
[20:18] <lifeless> sinzui: \o/
[20:22] <sinzui> lifeless: all oopes: oops ids are OOPS-1914QS133 to 139
[20:23] <LPCIBot> Project devel build #588: FAILURE in 5 hr 0 min: https://lpci.wedontsleep.org/job/devel/588/
[20:39] <lifeless> garh
[20:39] <lifeless> oops ids are being reused on qastaging
[20:39] <lifeless> garbo hourly is overwriting
[20:48] <lifeless> ok, I got a trace https://lp-oops.canonical.com/oops.py/?oopsid=1914QS133
[20:48] <lifeless> Statement Count: 1491 ><
[20:48] <lifeless> 1467 person lookups
[20:49]  * lifeless thinks the storm cache is too small
[20:51] <lifeless> no, thats not it, its clearly doing single loads
[20:51]  * lifeless failed 
[21:05] <lifeless> the traceback is really weird
[21:06] <lifeless> ids = set(map(attrgetter('subscriber_id'), result)) is triggering late evaluation. EWTF
[21:08] <lifeless> grraaaaaaa
[21:08] <lifeless> Module canonical.launchpad.webapp.authorization, line 218, in checkPermission
[21:08] <lifeless>     principal.account)
[21:13] <lifeless> sinzui: what do you think http://pastebin.com/70gVSwSt ?
[21:24] <sinzui> lifeless: we were doing the expensive check first?
[21:25] <lifeless> sinzui: we loaded the subscriptions
[21:25] <lifeless> then in the line I pasted - the attrgetter triggered permission check via the security proxy trying o read the subscriber_id
[21:26] <sinzui> :/
[21:26] <lifeless> the person hasn't been loaded at that point
[21:26] <lifeless> so it lazy evaluated the person
[21:26] <lifeless> found that the user wasn't the subscriber, found the user was archive admin, and we go around again
[21:27] <lifeless> I'm doing two changes
[21:27] <lifeless> one to change the order - common case will be the archive admin managing subscriptions
[21:27] <lifeless> and
[21:27] <lifeless> I think we should allow subscriber_id to any code
[21:27] <lifeless> I think http://pastebin.com/HTPWc93e will do it
[21:28] <sinzui> that is sensible
[21:28] <lifeless> but i don't know - because the _id is also in the interface
[21:28] <lifeless> will the lp.View declaration override the allow declaration ?
[21:30] <sinzui> lifeless: I do not have an issue with using the id. (I have some issues with is appearing in the ui/forms). I think the security.py change is good to land...
[21:30] <lifeless> sinzui: the id shouldn't appear in any ui/forms
[21:31] <LPCIBot> Project windmill build #120: STILL FAILING in 1 hr 8 min: https://lpci.wedontsleep.org/job/windmill/120/
[21:32] <sinzui> lifeless:  I do not think id will be exposed anywhere. Do we need a test to verify access to subscriber_id? Is it implicitly tested?
[21:32] <lifeless> sinzui: do you know if subscribers use this page as well ?
[21:33] <lifeless> sinzui: we know it works via the security proxy; the question is whether it avoids triggering a security check at all
[21:33] <lifeless> however its a bit of a stopgap
[21:33]  * sinzui checks the alternate path
[21:33] <lifeless> the basic problem is that when subscribers view this page we should /only/ load the subscribers subscription
[21:33] <lifeless> not all
[21:34] <lifeless> by which I mean, the security.py change fixes admins/archive owners
[21:34] <lifeless> but won't fix 'archive subscriber' viewing the page
[21:35] <lifeless> sinzui: so, I'll land the security.py thing
[21:35] <lifeless> we might not have subscribers view the page
[21:35] <lifeless> and if we do, we can do the root cause change rather than a bandaid
[21:35] <sinzui> lifeless: subscribers cannot access that page/view. They have a separate view
[21:36] <lifeless> sinzui: in which case, though its a little more expensive than idea, the change I've made is sufficient
[21:36] <lifeless> s/idea/ideal/
[21:37] <lifeless> sinzui: do you want to review, or should I self review?
[21:37] <sinzui> You have my r=,me
[21:37] <sinzui> I can mark it reviewed if you point me to an mp
[21:38] <lifeless> just filing it
[21:39] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-741092/+merge/55429
[21:42] <lifeless> sinzui: it has a diff now
[21:43] <sinzui> approved
[21:43] <lifeless> sinzui: wrong widget :P
[21:43] <sinzui> It will come
[21:43] <lifeless> man
[21:44] <lifeless> I really want callbacks
[21:44] <lifeless> would make our UI /so much better/
[21:48] <lifeless> now to see what my db patch is up to
[22:03] <lifeless> allenap: you've seen load_related, right ?
[22:07] <thumper> how do I find out what files get installed by a package?
[22:12] <wallyworld_> benji: thanks for the review. i run the tests using "bin/test" and assumed that would "Do The Right Thing"
[22:13] <wallyworld_> benji: are you saying it is using the wrong python doing that?
[22:13] <lifeless> benji: whats wrong with testtools as a dep ?
[22:13] <wallyworld_> benji: btw, i only ran the one new test i added and it did pass for me locally
[22:13] <benji> wallyworld_: it's more that it's the wrong Python to run the buildout with; you want a "clean" python, one without anything installed in it's site-packages
[22:14] <wallyworld_> benji: ah ok. in that case yes i was using the system python
[22:14] <wallyworld_> to do the buildout
[22:14] <benji> lifeless: nothing, it just looked accidental so I did the easiest thing that came to mind to get past the test failures
[22:15] <benji> wallyworld_: I wouldn't be surprised if the tests fail on the current trunk, if you can verify that for me then tomorrow I'll take a look at straightening them out
[22:17] <wallyworld_> benji: ok. you want me to do a fully clean build not using the system python?
[22:17] <benji> right; you can use a virtualenv to get a clean python
[22:18] <wallyworld_> benji: will do. thanks again.
[22:18] <benji> my pleasure
[22:18] <lifeless> benji: theres lots of test goodies in there; perhaps we should just add it as a dep to everything
[22:20] <benji> lifeless: that'd be fine with me; the problem was that it was an _undeclared_ dependency.  I don't have an opinion one way or the other on it being an actual dependency.  Well, I take that back; it wasn't really used for anything so I'd rather it wait until we actually want it to use it for something.  But that's a pretty low bar.
[22:22] <LPCIBot> Yippie, build fixed!
[22:22] <LPCIBot> Project db-devel build #505: FIXED in 5 hr 6 min: https://lpci.wedontsleep.org/job/db-devel/505/
[22:48]  * wallyworld_ starts a full windmill test run and runs off for a bit to see the accountant so the tax man doesn't come knocking
[22:54] <sinzui> I know how to fix the spastic mhonarc message encoding
[23:02] <sinzui> wgrant: can you see this https://code.launchpad.net/~sinzui/launchpad/mailing-list-archiving-0/+merge/55435
[23:02] <sinzui> wgrant: mumble?
[23:03] <wgrant> sinzui: I can.
[23:28] <thumper> wgrant: how do I find out what files get installed by a package?
[23:29] <lifeless> dpkg -L packagename
[23:30] <thumper> ta
[23:31] <allenap> lifeless: Yes, I saw load_related recently. Neat :) What about it?
[23:33] <lifeless> allenap: seems like the load recipe you mentioned in a recent review would also be generalisable
[23:33] <lifeless> :)
[23:35] <wgrant> sinzui: Are there docs on running a local mailman? I recall I did it 18 months ago, but don't recall how.
[23:35] <allenap> lifeless: I included a way to do it with load_related() too. Or are you saying there's scope for another helper? In that case the code turned out to be more concise just using load(), but maybe I missed a trick.
[23:36] <lifeless> allenap: I think theres room to improve our tooling furhter
[23:36] <lifeless> allenap: not quite sure how, its just a feeling
[23:38] <allenap> lifeless: Yeah, I feel the same way.
[23:38] <allenap> Right, properly off to bed now.
[23:39] <wgrant> Night allenap.
[23:58]  * StevenK assumes it is no longer Tuesday or that gmb is actively reviewing. Pointers about both assumptions welcome.