=== BradB|London [~bradb@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === debonzi [~debonzi@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad === cprov_ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === lalo [~lalo@201.10.27.117] has joined #launchpad === lalo [~lalo@201.10.27.117] has left #launchpad [] === lalo [~lalo@201.10.27.117] has joined #launchpad === lalo adds more to his reputation for being weird by switching to dvorak === cprov__ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === jdub [~jdub@home.waugh.id.au] has joined #launchpad [02:11] ahar [02:11] launchpad dudes [02:11] now i have the megaphone [02:11] muhahaha [02:11] https://www.warthogs.hbd.com/Launchpad/Testers [02:11] just added a page for potential launchpad testers [02:12] give it love [02:12] and stuff === jdub blows kisses === jdub [~jdub@home.waugh.id.au] has left #launchpad [] === kiko [~kiko@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad [02:23] boing [02:23] stub? === cprov_ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === kiko_ [~kiko@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad === lalo [~lalo@201.10.27.117] has joined #launchpad === lalo [~lalo@201.10.27.117] has left #launchpad [] === lalo [~lalo@201.10.27.117] has joined #launchpad === lalo [~lalo@201.10.27.117] has left #launchpad [] === lalo [~lalo@201.10.27.117] has joined #launchpad === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad === lalo [~lalo@200.96.104.254] has joined #launchpad [09:56] any idea what creates a ,,star-merge directory, and whether i can safely delete it? === BradB|London [~bradb@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad === debonzi [~debonzi@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad === BradB|London is now known as BradB === cprov [~cprov@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad === kiko [~kiko@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [10:50] hey lunchpad [10:50] stub, around? [11:01] sabdfl: AIUI, any files beginning with ,, are working files [11:01] so, if the star merge is done, then you won't need it any more [11:01] tnx === elmo_ [~james@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [11:28] kiko: Yo [11:28] kiko: Re your indexes [11:28] and the gpg thingies [11:28] :) [11:29] those indexes are gold dude [11:29] I'm trying to work out if two of them are necessary (which I will get to when my mail folder opens...) [11:30] I spent a lot of time in explain, I am quite sure they are [11:30] these fields are not unique [11:31] so they are not indexed by default [11:31] given we join on them (for instance sourcepackage.sourcepackagename) a lot, seq scan shows its ugly head everywhere [11:31] I can paste in some killer queries if you still need convincing [11:31] The index on binarypackage (binarypackagename) should not be necessary, as the unique (binarypackagename, version) index should be used. [11:32] it isn't [11:32] The index on sourcepackageupload(sourcepackagerelease) should not be used if you are also matching on distrorelease (which you might not be) [11:32] hmmm [11:32] it was also necessary [11:32] well [11:32] here's how I proceeded [11:32] timed query, explained it [11:33] added index [11:33] timed query, explained it [11:33] repeat [11:33] Did you run a vacuume analyze, explain, create index, vacuume analyze, explain? [11:33] obviously the second query benefits from cached data [11:33] yes, but I wasn't too interested in the time, more in the explain output === stub can't spell [11:34] this set of indexes removed all the seq scans but one in both the queries that were biting us [11:34] Ok. If you did the analyze before trusting the explain output I'll add them too. [11:34] interestingly enough vaccum analyze didn't change the explain output (iirc) [11:35] It is more a paranoia check - I think a goal for postgres would be to make it totally unnecessary ;) [11:35] stub, there was something else I wanted to ask you [11:35] yeah, definitely [11:35] we used a view to make the package list render in non-geological time in the end [11:35] It might be that it is more expensive to use the combined index, and postgres decided a seq scan was more efficient. [11:36] maybe [11:36] I am not concerned with the time it takes to insert a new package [11:36] because that is not the bottleneck in that task (we need to open and process the package and that makes me sleepy) [11:36] anyway, about the view [11:36] the trick is that is allows me to "pre-query" for all the attributes on the object I wanted [11:37] I was doing a query on sourcepackage and then when displaying their names *each* package resulted in a query for the sourcepackagename.name [11:37] which means that queries scaled according to O(n) which is evil [11:37] the view allows us to prefetch all the information and just use it [11:38] what's your view on this, and how do you think we should deal with this on the sqlobject level [11:39] Hmm... sqlobject should cache the sourcepackagename instance [11:39] it does [11:39] but not the sourcepackagename's name [11:39] ah, I see what you mean [11:39] well [11:40] it *could* add fields to the join output to make sure it has all the data it might need to display all possible traverses from that object [11:40] but I think that would require hinting [11:41] By caching I mean 'assert id(SourcepackageName.get(1)) == id(SourcepackageName.get(1))' succeeds [11:42] doesn't it do that? [11:42] I thought we cached instances properly in sqlobj [11:43] the issue is pre-filling their dicts based on joins they participated in :) [11:44] So why is getting the sourcepackage's name causing extra database hits? [11:44] because sourcepackagename's instance's dict doesn't prefill [11:44] we just have its id [11:44] not the "name" attribute [11:44] does that make sense? [11:45] (sortof like a "ghosted" zodb persistent object if you are familiar with that) [11:46] so [11:46] print sourcepackagename.get(1).__dict__ just contains {"id": 2232} === kiko cries [11:49] gina needs dbschema [11:49] oic what you are getting at now :-) [11:50] I personally would rather destroy the sourcepackagename table and move the field back into sourcepackage [11:51] It doesn't gain us anything, and just makes queries hairier. [11:51] (ditto binarypackagename) [11:51] that will get you in trouble with sabdfl [11:52] stub: normalisation tradeoff? may be worth it [11:53] There isn't really a normalisation tradeoff - there is only one column of data in the sourcepackagename table so you don't gain anything. [11:54] well [11:54] it avoids duplicating the string in sourcepackage/binarypackage [11:54] update sourcepackagename set name='newname' where name='oldname' vs. update sourcepackage set name='newname' where name='oldname'. [11:54] that's about it though. [11:54] It is a bigger waste of space to maintain the extra table, so that is a furfy. [11:55] thank god someone knows what a furfy is [11:55] I just don't think I can spell it :-) [11:55] (or is that right?) [11:56] I had never even heard the word but I assume I know what it is [11:56] anyway, if this bothers us enough I'll ask you to kill it [11:56] however, it doesn't invalidate the sql view discussion because hah [11:57] there are other tables we want to get data from :-/ [11:57] just let me know how many views, the queries and how often they need to be updated (I suspect they need to be 'live' and maintained with triggers) [11:59] Or maybe it would be better to work out how to 'prefetch' all the data. eg. If you did SourcepackageName.select('1==1'), it might suck the entire table in. [12:02] Mmm.... sweet potatoe and tofu daal with couscous.... [12:07] I think prefetch could be done easily by providing an extraFields argument to query() but you didn't hear that from me :) [12:07] the views are just regular create view views -- they are fast enough as it is. [12:07] the only reason we're using them is to make sqlobject back [12:07] ok. sounds like a plan. [12:08] sqlobject happy, sorry. [12:08] I wonder if you can flag a SQLObject subclass as 'readonly'... Hmmm... [12:09] I guess it is called 'documentation' [12:11] sabdfl: Can I revert sourcepackagename and binarypackagename? Can I do it now or wait until after the migration work? [12:16] kiko: I'll want to give the views a prefix so people don't confuse them with tables. Lowercase 'v'? 'SoyuzView'? 'SView' ? [12:23] stub: wait until after the migration [12:24] I'd rather we suffixed them just with View -- I don't think they are application-specific [12:27] 'morning [12:28] The work on re-arranging the source code will be starting very soon. [12:31] Who has outstanding merges? [12:31] Looks like a merge from stub is being processed atm. [12:32] oooh. I don't merge my stuff into rf ever since I crashed it a few days ago. === lalo goes submit a merge [12:32] lalo: I sent an email to the list yesterday [12:32] lalo: I was very clear that all merges must be submitted by 0900 UTC [12:32] SteveA: I read it [12:33] and... [12:33] well [12:33] tell me why seven people here should be delayed by however long it takes your merge to go through [12:33] sitting around doing nothing productive? [12:34] if you need to begin, go on - I'll sort out my conflicts later [12:34] ok. don't merge. we'll start as soon as the pqm queue is empty [12:34] lalo: that's not good teamwork === kiko tap taps for pqm [12:36] when I said I was going to submit, I didn't consider that it is already more than 9UTC; and I would have realized that and given up submitting, if you didn't warn me [12:37] apt-get install grandfatherclock [12:37] so, sorry for thinking out loud :-P I'm still waking up [12:39] (as a matter of policy I never commit anything *or* submit merges till I'm sure I'm safely awake) === lalo goes read mail === carlos [~carlos@69.Red-80-33-181.pooles.rima-tde.net] has joined #launchpad [12:41] 'morning carlos [12:42] morning [12:42] New bug 2068 for Launchpad/Launchpad: Wishlist: view PQM's pending queue [12:42] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2068 [01:02] "dilys" ?? [01:04] that's daf's bugzilla-watching bot [01:04] oh [01:05] I thought it was his new nick ;) [01:08] daf: ping [01:09] SteveA: pong [01:16] SteveA: could we work with rocketfuel or should we wait for the changes you are doing ? === kiko [~kiko@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [01:18] hey daf dude [01:18] can you roll the database on at rosetta.wh? [01:19] "roll"? :) [01:19] well every second page generates a programming error traceback [01:19] so that's bad publicity for soyuz [01:19] you want me to reset it? [01:20] you're missing the "name" changes [01:20] https://rosetta.warthogs.hbd.com/++skin++Debug/soyuz/distros/ubuntu [01:20] fmi check the uri above [01:20] done [01:21] invalid gateway? [01:21] or bad gateway? [01:21] interesting [01:21] didn't run [01:22] daf, kick that lunchpad man [01:22] IOError: [Errno 2] No such file or directory: '/home/daf/launchpad-devel/launchpad/lib/canonical/doap/sql.zcml' [01:23] spiv: was that you? :) [01:24] it was sabdfl [01:24] so we need to wait now === kiko grumbles [01:25] sabdfl: you broke Launchpad [01:25] sabdfl: naughty! [01:25] daf, maybe just touch the file or somthing? [01:25] I'll try it [01:26] nope, doesn't work === kiko cries crocodile tears [01:28] ah, got it running [01:28] ah dude you are da man [01:28] let me run my crapola tester again [01:30] "crapola tester"? [01:30] is this the wget -r testing thing? [01:30] a traceback checker through wget [01:31] yes [01:31] groovy [01:31] you make me decide it's going to be called crapola [01:31] that explains the request rate [01:31] it's as slow as molasses right now though [01:31] I'll put it on chinstrap or something [01:32] kiko: when you're done, you need to write an input thingy for dilys to show results :) [01:32] wow that would rock [01:32] yeah [01:32] i'll email dilys and you parse the output? [01:32] exactly [01:33] wow [01:33] yeah [01:33] well, you email me and procmail catches it and feeds it to dilys [01:33] "page foo just busted, latest checkin was by stub the nasty penguin " [01:33] daf: how has sabdfl broken launchpad? [01:34] SteveA: by adding an include in ZCML for a file which doesn't exist [01:34] SteveA: AFAICT [01:34] ok, that will be fixed soon [01:34] daf: did all the rosetta test failures get fixed? [01:34] we talked about this a week or so ago [01:35] daf: you should hook it to pqm [01:35] I'm thinking lunchpad [01:35] we will be making all launchpad checkins run all tests [01:35] the unittest are working [01:36] all unit tests are passing? [01:36] we have a functional test failing (hope will be fixed today) [01:36] daf: what'd i do? === ..[topic/#launchpad:SteveA] : Source code re-arrangement in progress. No merges into rocketfuel until further notice! [01:37] sabdfl: the missed file [01:37] daf, sorry, it exists in my tree, forgot to add it [01:37] don't worry about it, it will be fixed when we merge stuff soon [01:37] sabdfl: no worries [01:37] shit [01:37] It was passing yesterday [01:37] 5 errors today? [01:37] sabdfl: by the way, I think "tla tree-lint" would have complained about the file if you didn't add it [01:38] carlos: have you modified your database? [01:38] carlos: they're affected by that [01:38] we need an "I want to commit" script [01:38] it should run tree-lint [01:38] and run tests [01:38] yes, it would... does it also give a useful exit code? [01:38] daf: the unittest should not depend on that, right? [01:38] the unit test should not depend on any zcml [01:38] sabdfl: I think the --strict option gives you that === carlos should leave to have lunch so will be back in time for the daily meeting [01:39] later [01:40] do we need to have the broken symlinks in lib/? [01:40] daf: yes [01:41] or, you can install the extra software [01:41] so that the symlinks will not be broken [01:41] why aren't the symlinks created when the extra software is installed? [01:41] if you remove these symlinks then lifeless will complain at you [01:41] and then add them back [01:41] please ask lifeless about them :-) [01:41] will do :) [01:42] having tree-lint perpetually complaining is like having tests that perpetually fail [01:42] people get used to it and don't fix it [01:44] daf: I'm kinda busy, but please file a bug for this! [01:45] good idea [01:53] daf: We've got some problems coming from the rosetta tests. I ran 81 launchpad tests, 5 failures, 8 errors. These seem to be mostly shallow, e.g. test_TranslatePOemplate_mungeMessageID has a few "NameError: name 't' is not defined" which appears to be because of the t = TranslatePOTemplate(context, request) assignment failing. [01:54] BradB: interesting [01:55] It's possible that it's only on my machine. It's possible that it's a Real Problem though. [01:55] I'll try and duplicate it [01:55] ok, thanks [01:55] how are you running the tests? [01:57] lalo: wow, canonical.rosetta.ftests.test_poexport.POExportTestCase is really intense [01:58] lalo: it's been running at ~90% CPU for a minute or two [01:58] daf: import is worse :-P [01:58] !! [01:59] these ftests suck [01:59] can they be improved? [01:59] BradB: it's a real problem [02:00] it's the kind of thing that is a nightmare to test automatically [02:00] daf: python test.py -u [02:00] I've been beating them in a branch, but nothing concrete came out of it yet [02:01] BradB: works for me [02:01] lalo: is it the size of the test data? [02:01] there's always some greater priority coming up [02:01] daf: not here [02:01] daf: Maybe you have something carlos and I don't. === carlos recreating the db [02:02] BradB: or you have something I don't :) [02:03] daf: I don't have any change that is not already in rocketfuel [02:03] no, it's probably, to begin with, that it's still on autocommit [02:03] autocommit? [02:03] oh, right [02:03] is that easy to fix? [02:03] no :-) [02:04] same problem after the db refresh [02:04] carlos: which test is failing for you? [02:04] Failure in example: t._mungeMessageID(u'foo\nbar', [] ) [02:04] but the *proper* fix is already in my work list [02:04] FAIL: test_TranslatePOemplate_mungeMessageID [02:05] hmm, I think it should be TranslatePOTemplate... [02:05] not POemplate [02:05] :-P [02:05] carlos: dude, that's a unit test -- it doesn't touch the DB at all :) [02:05] daf: I know, I told you that already :-P [02:05] no, Rocketfuel is not missing any changes from me [02:05] daf: btw, the functional test fails also with the bug I told you yesterday that has msgid_plural == msgid [02:06] carlos: hmmm [02:08] carlos: what happens if you run "tla missing -s daf@canonical.com--2004/launchpad--devel--0" in your tree? [02:09] none missing [02:09] sigh [02:10] I'm executing make fullcheck inside rosetta directory [02:10] I think my patch 451 changed the tests [02:10] sorry, rocketfuel@canonical.com/launchpad--devel--0--patch-451 [02:11] I changed the tests somedays ago [02:11] for the line wrap fix [02:12] but that's unrelated to this bug, or are you talking about other change? [02:12] another change [02:12] when I added the c-format hilighting [02:12] I added a new test and changed some existing ones [02:13] carlos@frodo ~/Work/launchpad $ tla missing -s rocketfuel@canonical.com/launchpad--devel--0 [02:13] gpg: Good signature from "Patch Queue Manager (Canonical.com arch-pqm) " [02:13] gpg: Good signature from "Patch Queue Manager (Canonical.com arch-pqm) " [02:13] patch-422 [02:13] get bug browsing by source package under way [02:13] patch-462 [02:13] Stub files for canonical.launchpad [02:13] is that normal? [02:14] carlos: so I changed then in patch-451 and you changed them in patch-453 [02:14] I don't have patch-453, so perhaps that broke them [02:15] I'll star-merge and try again [02:15] should I get the patch-422 by hand? === carlos is confused [02:15] ask an arch person [02:15] lifeless? [02:17] carlos: it's 22:15 in Sydney [02:17] [lifeless] idle 00:00:02, signon: Tue Sep 28 02:05:33 [02:17] I suppose he's online :-P [02:18] carlos: no, but I haven't time to look at it right now. just ignore it [02:18] ok [02:19] okay, the tests fail now [02:19] daf: so it's a problem with my patch? [02:19] I think so [02:20] I changed only the line wrap :-? [02:20] lalo: could you look at it? [02:20] please [02:20] no [02:20] it was the statistics that broke it [02:20] the changes to browser.py [02:20] daf: ok, you are talking about the unittest [02:21] yes [02:21] that makes sense, I forgot to upgrade the tests for the browser.py [02:21] what about the functional test? [02:21] daf: I will fix it now [02:21] that should not have been affected [02:21] carlos: I'm already fixing it :) [02:21] ok [02:21] thanks [02:22] btw, I will look at it to know how works those tests [02:22] sure [02:22] the problem is this: [02:22] since the unit tests can't depend on the database, they use dummy objects [02:22] DummyPOFile doesn't have the statistics attributes, so the browser.py code that tries to access them fails [02:23] oh, you are using stub like objects [02:23] ok, I get the idea [02:25] --- orig/lib/canonical/rosetta/tests/test_browser.py [02:25] +++ mod/lib/canonical/rosetta/tests/test_browser.py [02:25] @@ -56,6 +56,12 @@ [02:25] class DummyPOFile: [02:25] pluralForms = 4 [02:25] [02:25] + def __init__(self, template): [02:25] + self.poTemplate = template [02:25] + [02:25] + def translatedCount(self): [02:25] + return 3 [02:25] + [02:25] [02:25] class DummyMessageID: [02:25] msgid = "foo" [02:25] @@ -83,7 +89,7 @@ [02:25] self.language_code = language_code [02:25] [02:25] if language_code in ('ja', 'es'): [02:25] - return DummyPOFile() [02:25] + return DummyPOFile(self) [02:25] else: [02:25] raise KeyError [02:25] ^^^ the fix [02:26] by the way, the unit tests should test the statistics code [02:26] daf: Now that you talk about the statistics, we should talk about the policy we will follow to get/update that information... [02:26] well [02:27] if we update the statistics [02:27] 1) when we import [02:27] 2) when we get new translations through the web [02:27] then they should always be up to date [02:27] agreed? [02:27] yes [02:27] lalo is working on (1) [02:27] but, when will be updated "through the web"? [02:27] with every commit? [02:28] yes [02:28] sorry, submit [02:28] yes [02:28] but 2 is only +1/-1, right? [02:28] lalo: not really [02:28] you can update more than one string [02:28] door [02:28] I think our limit is 5 now [02:29] it doesn't involve actually calling pofile.updateStatistics()? [02:30] because that method is awfully slow, and I don't think that is fixable [02:31] lalo: we could get the value and increment or decrement it as needed [02:31] but we could get some race conditions [02:31] ugh [02:32] we could forget that for now [02:32] :-) [02:32] so don't worry [02:35] lalo, daf: could you confirm me that the functional tests fails for you? [02:36] I do [02:36] ok [02:37] thanks [02:46] can we avoid race conditions by having atomic database queries? [02:46] okay, my unit test merge has gone in [02:47] BradB: a star-merge should fix it [02:49] daf: what does the channel title say? [02:50] daf: and what do you mean "atomic database queries?" we're using transactions. [02:50] SteveA: sorry [02:51] SteveA: does changing lib/canonical/rosetta/tests/test_browser.py affect the rearranging? [02:51] just don't touch it in RF until further notice [02:51] SteveA: I'm thinking of ways of accessing the database that avoid race conditions [02:52] what race conditions [02:52] daf: the problem is not the access to one field [02:52] daf: is that you could increment the count from the web interface [02:52] daf: please express what you are saying in terms of multiple transactions [02:53] and at the same time import a new .po file so the cache is out of sync [02:54] I was just thinking of concurrent accesses through the web [02:54]