[00:00] wgrant: and bug 739051 [00:00] <_mup_> Bug #739051: Product:+index timeouts < https://launchpad.net/bugs/739051 > [00:00] Blah, yes, but that should die. [00:00] wgrant: you asked if I was sure. [00:01] wgrant: 4% or so of our timeouts-by-pageid involve this table :) [00:07] wgrant: Still not clear how the new DSP should look [00:17] Oh good, I don't even have to update the topic. [00:17] Because 10 bugs have been filed, and 10 fixed. [00:17] But deployment is still a mess? :-) [00:17] Until we have no complaints for two hours, yes. [00:18] I can't land branches! Does that count? [00:18] :( [00:19] we can freeze landings to devel if needed [00:24] lifeless: is there anything I can currently watch to see if if new packages have landed in a ppa? [00:25] mtaylor: getPublishedSources and getPublishedBinaries [00:26] mtaylor: e.g. see bzr-builder 's --watch option [00:26] mtaylor: which may still be in a branch if james_w hasn't merged it yet. [00:26] sweet [01:37] I need a buildout hand [01:37] I have a project which I want to build its own interpreter for, I don't want it importable by lp. [01:38] how do I tell lp's buildout to -literally- shell out and buildout that subproject rather than recursing into it and its dependencies [01:38] lifeless: Probably the 'develop' option. [01:39] nope [01:39] Really? [01:39] Huh. [01:39] that symlinks the develop egg in [01:39] so that you can import the thing [01:39] without it having a versioned egg [02:13] lifeless: please tell me you're building a stackless interpreter because you need to [02:14] mtaylor: I'm not building one; sorry. [02:14] lifeless: DAMN [02:14] mtaylor: I hear that $language de jour will save the world. [02:15] wgrant: look at the tip of lp:python-gpgfixtures [02:15] lifeless: yup [02:15] Uhoh, plural? This can't be good. [02:16] lifeless: LP needs to be able to upload too. [02:18] wgrant: you might think that. [02:18] We generate keys. [02:18] And upload them to keyservers. [02:18] That needs testing. [02:19] indeed [02:20] I will not reduce test coverage with what I do. [02:22] wgrant: the implications of that statement may terrify you. And rightly so. [02:22] I am wondering what your plan is. [02:28] ah [02:28] there is one use of /pks/add in the test suite [02:28] Not just the test suite. [02:28] Production code does it too. [02:29] I know prod does it [02:29] it looked untested to me [02:29] Ah [02:29] see the commented out calls to generateSigningKey, fo rinstance. [02:30] Ahaha lovely. [02:30] and the other ones that say 'no key is generated cause we reuse here...' [02:31] There is one more though, which appears to genuinely exist. [02:31] but we don't test the error path. [02:31] so I'll need to add enough glue to pull out the key fingerprint and id and shove them in /keys. [02:31] but most of our code can diaf [02:36] Yep. [02:36] The difficult bit is working out the fingerprint, as you say. [02:40] there is a gpg parser on pypi [02:40] but I'll just use a gnupghomefixture for the lifetime of the request + a mutex to avoid os.environ race conditions [03:12] Project db-devel build #822: STILL FAILING in 1 min 53 sec: https://lpci.wedontsleep.org/job/db-devel/822/ [03:12] lifeless: I haz made patch. simple one-liner with no recipe does what I want now [03:13] lifeless: thanks for pointing out the obvious :) [03:16] Project db-devel build #823: STILL FAILING in 0.57 sec: https://lpci.wedontsleep.org/job/db-devel/823/ === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 237 - 0:[#######=]:256 [03:19] OH! I can land stuff now? [03:19] 3 hours, OOPS levels around normal, no complaints. [03:19] Go wild. [03:33] * StevenK lands 3 branches in 80 seconds. [03:33] buildbot gave up waiting after the third and started, strangely enough [04:03] 35 of our 305 tables are unused or only used by a small bit of unimportant code :( [04:14] Can we remove some of them? [04:15] Yeah. [04:16] Once we have fastdowntime I'll do it. [04:16] ALthough some of them have ancient data that we should archive. [04:16] Probably. [04:27] wgrant: Which tables are you looking at? [04:30] stub: http://paste.ubuntu.com/674281/ [04:30] stub: The top list is either entirely unused or only used by person merges. [04:30] The bottom three are used by things that shouldn't exist any more. [04:30] And each require deleting like 10 lines of code. [04:30] Didn't we utterly remove bounties? [04:31] The tables have never been purged [04:32] wgrant: Nothing there requires archiving. [04:32] (that hasn't already been archived, like shipit) [04:32] k [04:32] Once we have fastdowntime I will do the honours. [04:33] Drop foreign keys in one downtime, then remove the merge code, then do away with the tables. [04:33] wgrant: staticdiff has a *lot* of code associated with it [04:33] Grar. [04:33] Except we can't actually drop them with slony, can we? [04:33] StevenK: It's mostly unused. [04:33] StevenK: It was for review diffs, but they are dead now. [04:33] StevenK: It is the hardest of the list to remove, though, right. [04:34] wgrant: I was told by lifeless to not remove stuff related to that. [04:34] StevenK: That was incrementaldiff. [04:34] They aren't related? [04:34] If they aren't, what's the difference? [04:34] There are three types of MP diffs. [04:35] staticdiffs (review diffs), previewdiffs (preview diffs), and incrementaldiffs (incremental diffs) [04:35] incrementaldiffs are incremental. [04:35] staticdiffs were from back when you had to run mad to update MP diffs. LP would generate the staticdiff, and it would never change. [04:35] previewdiffs were what mad generated -- they could update. [04:35] Can you expand 'mad' ? [04:35] But now LP updates previewdiffs itself, and I don't believe staticdiffs are used. [04:35] mad = Merge Analysis Daemon [04:35] Oh [04:36] lp:mad [04:36] Sounds hideous [04:36] Yes. [04:36] Right, so I will look at ripping out the use of staticdiff [04:36] No, it's not in scope :) [04:37] launchpad_dogfood=# SELECT MAX(date_created) FROM branchmergeproposal WHERE review_diff IS NOT NULL; [04:37] max [04:37] ---------------------------- [04:37] 2010-01-27 08:16:31.060926 [04:37] I'm either blocked on FDT for SPPH-SPN denorm, or working out how to hell to reimplement DSP? [04:37] StevenK: Have you got the garbo jobs and indices sorted out? [04:37] For denorm? Yes [04:37] fdt is probably not going to happen this week. [04:37] I can't do anything at all with them until the patch lands [04:38] The branches are up, thanks to the magic of sync-pipeline [04:38] StevenK: Ah, I saw they were still in Next and assumed they weren't implemented. [04:38] wgrant: Yes, because I don't want to utterly take over Coding single-handedly when I'm blocked by FDT [04:39] wgrant: So, I can either remove staticdiff, or we can argue about DSP. [04:39] Personally, the former sounds a little more fun [04:39] We need to sort out DSP in the next week or so. [04:40] Except I'm on holidays next week ... [04:40] Ah, forgot that bit. [04:40] And so is Curtis. [04:40] Yes. [04:40] So I guess we won't be pushing the in-progress stuff too hard. [04:40] wgrant: So, which of the two do you to tackle? [04:41] wgrant: Do you want to have a friendly argument about DSP? [04:41] StevenK: Do you have a list of uses of DSP and DSPID? [04:41] I do [04:42] The only thing that's stopped me writing stuff up is a name for the new hotness [04:42] Hm, pad.ubuntu.com looks sadly UDS-specific. [04:43] -> tea [04:49] StevenK: http://pad.ubuntu.com/cWLmLex2tc [04:49] StevenK: Where a caller is listed for something, that is all of the callers. [04:49] (apart from the maintenance methods, which are moving onto the classes themselves in ec2 now) [04:50] oooh, we deployed. Nice :) [04:50] wgrant: I don't have a list of callers, but I suspect they are numerable [04:50] nigelb: Yup. Your code works, too! [04:51] wgrant: Added my notes [04:51] \o/ [04:52] nigelb: Got any more 4 digit bug fixes? [04:53] wgrant: If they're relatively easy to fix, I'm interested. [04:54] That doesn't sound like an adequate sense of adventure. [04:54] THat's making sure I can fix them in a lifetime. [04:55] Let me see if I can get into top-5 list in this week. [04:55] 5 more bugs to fix to get there. [04:55] Just two more merges, IIRC. [04:55] 1 more will bring you to equal 5th place. [04:55] ooh. 2 more. Interesting. [04:56] Sadly, 127 more are required to bring you to 1st place. [04:56] Yeah. [04:56] That's hard, but not undoable. [04:56] A fix per week for 2+ years. [04:59] hmm, what needs to be done for bug 50661 [04:59] <_mup_> Bug #50661: Magic words in comments (e.g. bug NNNN) should be documented

< https://launchpad.net/bugs/50661 > [04:59] I could just take stuff documented in one of the doctests that I boke. [05:00] *broke [05:01] wgrant: Where's this list of top-5? [05:01] https://dev.launchpad.net/Contributions [05:01] Ah, nice [05:05] woah, bug 811028 is easy? Lies! [05:05] <_mup_> Bug #811028: Time frame 'a moment ago' incorrectly concatenated to 'on' < https://launchpad.net/bugs/811028 > [05:06] Bleh [05:06] sinzui fixed something like that for questions in the last deployment. [05:06] Yeah, looks like I missed a spot [05:07] That's registration of product etc, isn't it? [05:07] It says bug. [05:07] But bugs are fine. [05:07] I think... [05:07] bah. [05:07] Oh, it uses fmt:displaydate. [05:07] And bugs say 'Reported by ...' [05:08] def displaydate(self): [05:08] delta = abs(self._now() - self._datetime) [05:08] if delta > timedelta(1, 0, 0): [05:08] # far in the past or future, display the date [05:08] return 'on ' + self.date() [05:08] return self.approximatedate() [05:08] Right [05:09] stub: are you actually here, or just caffeinating ? [05:09] I'm here [05:09] approximatedate() returns a moment ago for =< 10 [05:09] care to have a brief catchup now ? [05:09] wgrant: that means its fixed, right? [05:09] lifeless: sure [05:09] * stub fires up skype [05:10] nigelb: Not necessarily. [05:10] Should test... [05:10] I hope qastaging doesn't timeout for me again. [05:10] bwahaha [05:10] well, in the first 5 tries. [05:10] Reported by William Grant a moment ago [05:10] hah [05:11] Looks fine to me [05:11] Registered a moment ago by William Grant [05:11] Projects are fine too. [05:11] what else? blueprints and answers? [05:12] Uploads? [05:12] commits ? [05:12] Asked by William Grant a moment ago [05:12] Questions are fine now too. [05:12] wgrant: add support pushed [05:12] Registered by William Grant on a moment ago [05:12] lifeless: I seriously doubt commits make use of approximatedate [05:12] Blueprints are bad. [05:13] Text? [05:13] wgrant: supports more keys which I think real keyservers do too :P [05:13] ah, blueprints are broken, yes. [05:13] Can haz paste? [05:13] wgrant: in general, or in principle? [05:13] nigelb: Want to clarify that in the bug and fix? [05:13] lifeless: Heh. [05:13] In this case [05:13] Registered by Nigel Babu on a moment ago [05:13] wgrant: yup, fixing in a bit. [05:13] That might be what the reporter actually meant [05:14] I presume so. [05:14] It's the only object I've found that uses that dext. [05:14] Possibly, blueprints is the unloved child. [05:14] Projects use "Registered", but put the name at the end. [05:14] I should careful that I don't end up owning it with 5 of my 6 fixes in blueprint I think. [05:15] Bwahah [05:16] *be careful [05:23] wgrant is going to not like me. [05:23] % bzr di | diffstat -s [05:23] 14 files changed, 11 insertions(+), 335 deletions(-) [05:23] did you remove more tests? [05:23] StevenK: StaticDiff? [05:24] wgrant: Yah [05:24] StevenK: What did you do with the review_diff fkey on BMP? a few things still seemed to use it. [05:24] I'm not sure they used it for anything. [05:24] But they used it. [05:24] I binned it [05:26] Ah, it was just constructors. [05:26] Easier than I expected. [05:27] I have missed stuff, though [05:28] Right [05:28] Now to fish around in sampledata [05:29] Hrm, need to learn to read configure.zcml better. [05:30] nigelb: Don't. Your eyes will bleed. [05:30] wgrant: The only I can't work out what to do with is lib/lp/codehosting/tests/test_jobs.py [05:30] StevenK: I'm submitting MP's to LP, one should assume that they've already started bleeding. [05:31] nigelb: But XML means the wounds can not be cauterised [05:31] heh [05:37] so, I see tal:content="context/bug/datecreated/fmt:displaydate" [05:38] where do I look to see the code behind it? [05:38] StevenK: Erm. It looks like RevisionMailJobs use StaticDiff, but apparently they don't stay around in the DB... [05:38] or rather, how do I figure out where to look. [05:38] In fact, they never hit the DB. [05:38] wgrant: That's disappointing [05:38] nigelb: lib/lp/app/browser/tales.py [05:39] ooh, thanks [05:40] StevenK: Nothing has been added to the table since January last year, as expected. But it looks like RevisionMailJobs should be calling StaticDiff.acquire, which instantiates and returns a StaticDiff. [05:40] And StaticDiff inherits SQLBase, so that should automatically add it to the DB. [05:41] Perhaps the commit gets dropped [05:41] The sequence should still be incremented, though, right? [05:41] Unless it gets reset on DF. [05:44] wgrant: So I wonder if I we should fix RMJ to use previewdiff? [05:45] * nigelb is gettign "chmod: changing permissions of `/var/tmp/bazaar.launchpad.dev/rewrite.log': Operation not permitted" [05:46] nigelb: Apache creates that sometimes, so it ends up owned by root or www-data. [05:46] nigelb: sudo chown it. [05:46] wgrant: chown to my user or apache user? [05:47] nigelb: Yours. [05:47] ah [05:48] It even calls transaction.commit()! WTF [05:48] StevenK: Yes. [05:48] It makes very little sense. [05:49] Instantiating a StaticDiff must not add it to the store. [05:49] (the commit is necessary because the diff is stored in the librarian) [05:49] Ah, I bet that is the issue [05:49] It isn't added to the store [05:50] SQLBase does that automatically, normally. [05:50] >>> from lp.code.model.diff import StaticDiff [05:50] >>> StaticDiff() [05:50] [05:50] >>> transaction.commit() [05:50] Traceback (most recent call last): [05:50] IntegrityError: null value in column "from_revision_id" violates not-null constraint [05:53] gah, I broke it. I can't read tal. [05:53] Ah. No wonder I can't find it. My branch removes it. [05:54] can someone spare a few minutes to explain how this works? I think I have no clue what I'm doing. [05:55] nigelb: Pastebin your diff [05:55] nigelb: bzr di | utilites/paste # if your friend [05:55] StevenK: I don't have a diff [05:55] because I don't know what to diff on. [05:55] :( [05:56] * nigelb pastbin's broken diff [05:56] http://pastebin.ubuntu.com/674302/ [05:57] I brain-deadly pasted what works for answers. Which I now realize is wrong. [05:57] Erm, that was from answers? [05:58] yeah [05:58] + tal:attributes="title context/bug/datecreated/fmt:datetime" [05:58] bug? [05:58] baaah [05:58] I wonder if sinzui stole that from Bugs. [05:58] no [05:58] my bad [05:58] messed up paste :| [05:59] StevenK: Ha ha ha. [05:59] oh lol [05:59] Now I have on 10 minutes ago [05:59] wgrant: Hm? [06:00] Okay, so current diff http://pastebin.ubuntu.com/674304/ [06:01] StevenK: http://paste.ubuntu.com/ [06:01] Er. [06:01] Bah, wants a name. [06:01] Nasty creature. [06:01] http://paste.ubuntu.com/674305/ [06:01] Note that it ran a REVISIONS_ADDED_MAIL job, not a REVISION_MAIL job. [06:01] I wonder if REVISION_MAIL is unused. [06:02] Code seems to be full of dead code. [06:03] Oh dear. [06:03] StevenK: lib/lp/codehosting/scanner/email.py +55 [06:04] StevenK: If it's not an initial scan, it uses REVISIONS_ADDED_MAIL [06:04] StevenK: If it is an initial scan, it uses REVISION_MAIL, but disables diff generation. [06:04] Haha [06:04] So I think all that code is actually unused. [06:04] But grepping harder. [06:05] All the callsites I can see set perform_diff=False. [06:05] And we've seen there have been none in the DB for more than 18 months. [06:05] I think we can be fairly confident it's dead, but we should check with Aaron. [06:05] wgrant: http://pastebin.ubuntu.com/674308/ [06:05] StevenK: Care to remove perform_diff from RevisionMailJob, delete that remaining test, and send it off to ec2 test? [06:06] Note that we can't drop the table yet. [06:06] But do revoke perms. [06:07] So, test_jobs.py can be deleted. [06:08] 17 files changed, 12 insertions(+), 415 deletions(-) [06:08] All the revno/diff handling from RevisionMailJob can go. [06:08] Even revno? [06:08] The only revnos passed in are 'removed' and 'initial'. The int stuff can go. [06:09] So drop the if/else in create(), drop the perform_diff argument, and delete most of getMailer. [06:09] Already done [06:09] Just passing None as diff_text instead. [06:09] Done that too [06:09] Does that leave BranchDiffJob unused/ [06:10] StevenK: got a chance to look at the diff? [06:10] Seems to. [06:10] No, it is still RMJ's parent, and it calls .getMetadata [06:10] StevenK: RMJ has no cause to use it any more. [06:11] Oh, the update can probably DIAF too [06:11] StevenK: Since it's not generating diffs, it doesn't need the metadata. [06:11] It has no revno any more, remember. [06:11] """A Job that calculates the a diff related to a Branch.""" [06:11] ok, so what's the difference beteen tal:attributes and tal:content? [06:11] * StevenK smirks [06:12] Heh. [06:12] Not any more it don't. [06:12] nigelb: tal:content sets the value inside the element. attributes sets attributes on it. [06:12] wgrant: """A Job that sends a mail for an initial scan of a Branch.""" [06:12] StevenK: Or scans where revisions are removed. [06:13] hah. Nailed it! [06:13] ... for a scan of a Branch [06:13] * nigelb wonders how many tests he broke [06:13] nigelb: will turn into William Grant [06:13] wgrant: AH! [06:14] okay, now I understand what the code in answerws is doing [06:14] Except for this bit [06:14] tal:replace replaces the element entirely. [06:14] tal:attributes="title context/datecreated/fmt:datetime" [06:14] tal:content="context/datecreated/fmt:displaydate">2005-10-05 [06:14] So '' just turns into 'William Grant' [06:14] what is that 2005-10-05 doing there? [06:14] nigelb: That's just a default value. If, say, someone views the template in a web browser directly. [06:15] tal:content replaces any existing content. [06:15] ah, that gets replaced by tal:content [06:15] ok, I understand this beast better. [06:15] StevenK: Up to -500 yet? [06:16] Also, if I do a "bin/test lp.blueprints" does it run doctests as well? [06:16] nigelb: It probably will, yes [06:16] nigelb: Probably. [06:16] nigelb: Some are a bit special. [06:16] But I think it should get all the blueprint ones. [06:16] I'll let the special ones be caught by ec2 land. [06:17] wgrant: No, sadly. [06:17] StevenK: Pick random lines of code to delete until you meet the quota. [06:17] Bwahah [06:17] heh [06:17] rm -r blueprints will get you a long way and make wgrant a happy man. [06:30] 19 files changed, 31 insertions(+), 483 deletions(-) [06:30] But the diff is 1,000 lines [06:30] find 7 more lines to delete? [06:31] That's only 490! FAIL [06:31] bah, 17 more! [06:31] Math fail [06:31] ok, not test failure in blueprints. [06:31] * nigelb proposes MP [06:33] steven@liquified:~/launchpad/lp-branches/no-more-staticdiff% bzr log -r -1.. | tail -n 1 [06:33] Consign StaticDiff and all that supported it to a watery grave. [06:34] bzr log --line -r-1 [06:34] hm, why does make lint error out... [06:34] bzr: ERROR: unknown command "pipes" [06:34]