| === BradB|London [~bradb@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === debonzi [~debonzi@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad | ||
| === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad | ||
| === cprov_ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === lalo [~lalo@201.10.27.117] has joined #launchpad | ||
| === lalo [~lalo@201.10.27.117] has left #launchpad [] | ||
| === lalo [~lalo@201.10.27.117] has joined #launchpad | ||
| === lalo adds more to his reputation for being weird by switching to dvorak | ||
| === cprov__ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === jdub [~jdub@home.waugh.id.au] has joined #launchpad | ||
| jdub | ahar | 02:11 |
|---|---|---|
| jdub | launchpad dudes | 02:11 |
| jdub | now i have the megaphone | 02:11 |
| jdub | muhahaha | 02:11 |
| jdub | https://www.warthogs.hbd.com/Launchpad/Testers | 02:11 |
| jdub | just added a page for potential launchpad testers | 02:11 |
| jdub | give it love | 02:12 |
| jdub | and stuff | 02:12 |
| === jdub blows kisses | ||
| === jdub [~jdub@home.waugh.id.au] has left #launchpad [] | ||
| === kiko [~kiko@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| kiko | boing | 02:23 |
| kiko | stub? | 02:23 |
| === cprov_ [~cprov@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === kiko_ [~kiko@224-21-161-212.DSL.ONCOLT.COM] has joined #launchpad | ||
| === lalo [~lalo@201.10.27.117] has joined #launchpad | ||
| === lalo [~lalo@201.10.27.117] has left #launchpad [] | ||
| === lalo [~lalo@201.10.27.117] has joined #launchpad | ||
| === lalo [~lalo@201.10.27.117] has left #launchpad [] | ||
| === lalo [~lalo@201.10.27.117] has joined #launchpad | ||
| === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad | ||
| === lalo [~lalo@200.96.104.254] has joined #launchpad | ||
| sabdfl | any idea what creates a ,,star-merge directory, and whether i can safely delete it? | 09:56 |
| === BradB|London [~bradb@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| === debonzi [~debonzi@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| === BradB|London is now known as BradB | ||
| === cprov [~cprov@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| === kiko [~kiko@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| kiko | hey lunchpad | 10:50 |
| kiko | stub, around? | 10:50 |
| SteveA | sabdfl: AIUI, any files beginning with ,, are working files | 11:01 |
| SteveA | so, if the star merge is done, then you won't need it any more | 11:01 |
| sabdfl | tnx | 11:01 |
| === elmo_ [~james@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| stub | kiko: Yo | 11:28 |
| stub | kiko: Re your indexes | 11:28 |
| kiko | and the gpg thingies | 11:28 |
| kiko | :) | 11:28 |
| kiko | those indexes are gold dude | 11:29 |
| stub | I'm trying to work out if two of them are necessary (which I will get to when my mail folder opens...) | 11:29 |
| kiko | I spent a lot of time in explain, I am quite sure they are | 11:30 |
| kiko | these fields are not unique | 11:30 |
| kiko | so they are not indexed by default | 11:31 |
| kiko | given we join on them (for instance sourcepackage.sourcepackagename) a lot, seq scan shows its ugly head everywhere | 11:31 |
| kiko | I can paste in some killer queries if you still need convincing | 11:31 |
| stub | The index on binarypackage (binarypackagename) should not be necessary, as the unique (binarypackagename, version) index should be used. | 11:31 |
| kiko | it isn't | 11:32 |
| stub | The index on sourcepackageupload(sourcepackagerelease) should not be used if you are also matching on distrorelease (which you might not be) | 11:32 |
| kiko | hmmm | 11:32 |
| kiko | it was also necessary | 11:32 |
| kiko | well | 11:32 |
| kiko | here's how I proceeded | 11:32 |
| kiko | timed query, explained it | 11:32 |
| kiko | added index | 11:33 |
| kiko | timed query, explained it | 11:33 |
| kiko | repeat | 11:33 |
| stub | Did you run a vacuume analyze, explain, create index, vacuume analyze, explain? | 11:33 |
| kiko | obviously the second query benefits from cached data | 11:33 |
| kiko | yes, but I wasn't too interested in the time, more in the explain output | 11:33 |
| === stub can't spell | ||
| kiko | this set of indexes removed all the seq scans but one in both the queries that were biting us | 11:34 |
| stub | Ok. If you did the analyze before trusting the explain output I'll add them too. | 11:34 |
| kiko | interestingly enough vaccum analyze didn't change the explain output (iirc) | 11:34 |
| stub | It is more a paranoia check - I think a goal for postgres would be to make it totally unnecessary ;) | 11:35 |
| kiko | stub, there was something else I wanted to ask you | 11:35 |
| kiko | yeah, definitely | 11:35 |
| kiko | we used a view to make the package list render in non-geological time in the end | 11:35 |
| stub | It might be that it is more expensive to use the combined index, and postgres decided a seq scan was more efficient. | 11:35 |
| kiko | maybe | 11:36 |
| kiko | I am not concerned with the time it takes to insert a new package | 11:36 |
| kiko | because that is not the bottleneck in that task (we need to open and process the package and that makes me sleepy) | 11:36 |
| kiko | anyway, about the view | 11:36 |
| kiko | the trick is that is allows me to "pre-query" for all the attributes on the object I wanted | 11:36 |
| kiko | I was doing a query on sourcepackage and then when displaying their names *each* package resulted in a query for the sourcepackagename.name | 11:37 |
| kiko | which means that queries scaled according to O(n) which is evil | 11:37 |
| kiko | the view allows us to prefetch all the information and just use it | 11:37 |
| kiko | what's your view on this, and how do you think we should deal with this on the sqlobject level | 11:38 |
| stub | Hmm... sqlobject should cache the sourcepackagename instance | 11:39 |
| kiko | it does | 11:39 |
| kiko | but not the sourcepackagename's name | 11:39 |
| kiko | ah, I see what you mean | 11:39 |
| kiko | well | 11:39 |
| kiko | it *could* add fields to the join output to make sure it has all the data it might need to display all possible traverses from that object | 11:40 |
| kiko | but I think that would require hinting | 11:40 |
| stub | By caching I mean 'assert id(SourcepackageName.get(1)) == id(SourcepackageName.get(1))' succeeds | 11:41 |
| kiko | doesn't it do that? | 11:42 |
| kiko | I thought we cached instances properly in sqlobj | 11:42 |
| kiko | the issue is pre-filling their dicts based on joins they participated in :) | 11:43 |
| stub | So why is getting the sourcepackage's name causing extra database hits? | 11:44 |
| kiko | because sourcepackagename's instance's dict doesn't prefill | 11:44 |
| kiko | we just have its id | 11:44 |
| kiko | not the "name" attribute | 11:44 |
| kiko | does that make sense? | 11:44 |
| kiko | (sortof like a "ghosted" zodb persistent object if you are familiar with that) | 11:45 |
| kiko | so | 11:46 |
| kiko | print sourcepackagename.get(1).__dict__ just contains {"id": 2232} | 11:46 |
| === kiko cries | ||
| kiko | gina needs dbschema | 11:49 |
| stub | oic what you are getting at now :-) | 11:49 |
| stub | I personally would rather destroy the sourcepackagename table and move the field back into sourcepackage | 11:50 |
| stub | It doesn't gain us anything, and just makes queries hairier. | 11:51 |
| stub | (ditto binarypackagename) | 11:51 |
| kiko | that will get you in trouble with sabdfl | 11:51 |
| sabdfl | stub: normalisation tradeoff? may be worth it | 11:52 |
| stub | There isn't really a normalisation tradeoff - there is only one column of data in the sourcepackagename table so you don't gain anything. | 11:53 |
| kiko | well | 11:54 |
| kiko | it avoids duplicating the string in sourcepackage/binarypackage | 11:54 |
| stub | update sourcepackagename set name='newname' where name='oldname' vs. update sourcepackage set name='newname' where name='oldname'. | 11:54 |
| kiko | that's about it though. | 11:54 |
| stub | It is a bigger waste of space to maintain the extra table, so that is a furfy. | 11:54 |
| kiko | thank god someone knows what a furfy is | 11:55 |
| stub | I just don't think I can spell it :-) | 11:55 |
| stub | (or is that right?) | 11:55 |
| kiko | I had never even heard the word but I assume I know what it is | 11:56 |
| kiko | anyway, if this bothers us enough I'll ask you to kill it | 11:56 |
| kiko | however, it doesn't invalidate the sql view discussion because hah | 11:56 |
| kiko | there are other tables we want to get data from :-/ | 11:57 |
| stub | just let me know how many views, the queries and how often they need to be updated (I suspect they need to be 'live' and maintained with triggers) | 11:57 |
| stub | Or maybe it would be better to work out how to 'prefetch' all the data. eg. If you did SourcepackageName.select('1==1'), it might suck the entire table in. | 11:59 |
| stub | Mmm.... sweet potatoe and tofu daal with couscous.... | 12:02 |
| kiko | I think prefetch could be done easily by providing an extraFields argument to query() but you didn't hear that from me :) | 12:07 |
| kiko | the views are just regular create view views -- they are fast enough as it is. | 12:07 |
| kiko | the only reason we're using them is to make sqlobject back | 12:07 |
| stub | ok. sounds like a plan. | 12:07 |
| kiko | sqlobject happy, sorry. | 12:08 |
| stub | I wonder if you can flag a SQLObject subclass as 'readonly'... Hmmm... | 12:08 |
| stub | I guess it is called 'documentation' | 12:09 |
| stub | sabdfl: Can I revert sourcepackagename and binarypackagename? Can I do it now or wait until after the migration work? | 12:11 |
| stub | kiko: I'll want to give the views a prefix so people don't confuse them with tables. Lowercase 'v'? 'SoyuzView'? 'SView' ? | 12:16 |
| spiv | stub: wait until after the migration | 12:23 |
| kiko | I'd rather we suffixed them just with View -- I don't think they are application-specific | 12:24 |
| lalo | 'morning | 12:27 |
| SteveA | The work on re-arranging the source code will be starting very soon. | 12:28 |
| SteveA | Who has outstanding merges? | 12:31 |
| spiv | Looks like a merge from stub is being processed atm. | 12:31 |
| lalo | oooh. I don't merge my stuff into rf ever since I crashed it a few days ago. | 12:32 |
| === lalo goes submit a merge | ||
| SteveA | lalo: I sent an email to the list yesterday | 12:32 |
| SteveA | lalo: I was very clear that all merges must be submitted by 0900 UTC | 12:32 |
| lalo | SteveA: I read it | 12:32 |
| SteveA | and... | 12:33 |
| lalo | well | 12:33 |
| SteveA | tell me why seven people here should be delayed by however long it takes your merge to go through | 12:33 |
| SteveA | sitting around doing nothing productive? | 12:33 |
| lalo | if you need to begin, go on - I'll sort out my conflicts later | 12:34 |
| SteveA | ok. don't merge. we'll start as soon as the pqm queue is empty | 12:34 |
| sabdfl | lalo: that's not good teamwork | 12:34 |
| === kiko tap taps for pqm | ||
| lalo | when I said I was going to submit, I didn't consider that it is already more than 9UTC; and I would have realized that and given up submitting, if you didn't warn me | 12:36 |
| SteveA | apt-get install grandfatherclock | 12:37 |
| lalo | so, sorry for thinking out loud :-P I'm still waking up | 12:37 |
| lalo | (as a matter of policy I never commit anything *or* submit merges till I'm sure I'm safely awake) | 12:39 |
| === lalo goes read mail | ||
| === carlos [~carlos@69.Red-80-33-181.pooles.rima-tde.net] has joined #launchpad | ||
| lalo | 'morning carlos | 12:41 |
| carlos | morning | 12:42 |
| dilys | New bug 2068 for Launchpad/Launchpad: Wishlist: view PQM's pending queue | 12:42 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2068 | 12:42 |
| elmo_ | "dilys" ?? | 01:02 |
| lalo | that's daf's bugzilla-watching bot | 01:04 |
| elmo_ | oh | 01:04 |
| elmo_ | I thought it was his new nick ;) | 01:05 |
| SteveA | daf: ping | 01:08 |
| daf | SteveA: pong | 01:09 |
| carlos | SteveA: could we work with rocketfuel or should we wait for the changes you are doing ? | 01:16 |
| === kiko [~kiko@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| kiko | hey daf dude | 01:18 |
| kiko | can you roll the database on at rosetta.wh? | 01:18 |
| daf | "roll"? :) | 01:19 |
| kiko | well every second page generates a programming error traceback | 01:19 |
| kiko | so that's bad publicity for soyuz | 01:19 |
| daf | you want me to reset it? | 01:19 |
| kiko | you're missing the "name" changes | 01:20 |
| kiko | https://rosetta.warthogs.hbd.com/++skin++Debug/soyuz/distros/ubuntu | 01:20 |
| kiko | fmi check the uri above | 01:20 |
| daf | done | 01:20 |
| kiko | invalid gateway? | 01:21 |
| kiko | or bad gateway? | 01:21 |
| daf | interesting | 01:21 |
| daf | didn't run | 01:21 |
| kiko | daf, kick that lunchpad man | 01:22 |
| daf | IOError: [Errno 2] No such file or directory: '/home/daf/launchpad-devel/launchpad/lib/canonical/doap/sql.zcml' | 01:22 |
| daf | spiv: was that you? :) | 01:23 |
| kiko | it was sabdfl | 01:24 |
| kiko | so we need to wait now | 01:24 |
| === kiko grumbles | ||
| daf | sabdfl: you broke Launchpad | 01:25 |
| daf | sabdfl: naughty! | 01:25 |
| kiko | daf, maybe just touch the file or somthing? | 01:25 |
| daf | I'll try it | 01:25 |
| daf | nope, doesn't work | 01:26 |
| === kiko cries crocodile tears | ||
| daf | ah, got it running | 01:28 |
| kiko | ah dude you are da man | 01:28 |
| kiko | let me run my crapola tester again | 01:28 |
| daf | "crapola tester"? | 01:30 |
| daf | is this the wget -r testing thing? | 01:30 |
| kiko | a traceback checker through wget | 01:30 |
| kiko | yes | 01:31 |
| daf | groovy | 01:31 |
| kiko | you make me decide it's going to be called crapola | 01:31 |
| daf | that explains the request rate | 01:31 |
| kiko | it's as slow as molasses right now though | 01:31 |
| kiko | I'll put it on chinstrap or something | 01:31 |
| daf | kiko: when you're done, you need to write an input thingy for dilys to show results :) | 01:32 |
| kiko | wow that would rock | 01:32 |
| kiko | yeah | 01:32 |
| kiko | i'll email dilys and you parse the output? | 01:32 |
| daf | exactly | 01:32 |
| kiko | wow | 01:33 |
| kiko | yeah | 01:33 |
| daf | well, you email me and procmail catches it and feeds it to dilys | 01:33 |
| kiko | "page foo just busted, latest checkin was by stub the nasty penguin " | 01:33 |
| SteveA | daf: how has sabdfl broken launchpad? | 01:33 |
| daf | SteveA: by adding an include in ZCML for a file which doesn't exist | 01:34 |
| daf | SteveA: AFAICT | 01:34 |
| SteveA | ok, that will be fixed soon | 01:34 |
| SteveA | daf: did all the rosetta test failures get fixed? | 01:34 |
| SteveA | we talked about this a week or so ago | 01:34 |
| lalo | daf: you should hook it to pqm | 01:35 |
| kiko | I'm thinking lunchpad | 01:35 |
| SteveA | we will be making all launchpad checkins run all tests | 01:35 |
| carlos | the unittest are working | 01:35 |
| SteveA | all unit tests are passing? | 01:36 |
| carlos | we have a functional test failing (hope will be fixed today) | 01:36 |
| sabdfl | daf: what'd i do? | 01:36 |
| === ..[topic/#launchpad:SteveA] : Source code re-arrangement in progress. No merges into rocketfuel until further notice! | ||
| SteveA | sabdfl: the missed file | 01:37 |
| sabdfl | daf, sorry, it exists in my tree, forgot to add it | 01:37 |
| SteveA | don't worry about it, it will be fixed when we merge stuff soon | 01:37 |
| daf | sabdfl: no worries | 01:37 |
| carlos | shit | 01:37 |
| carlos | It was passing yesterday | 01:37 |
| carlos | 5 errors today? | 01:37 |
| daf | sabdfl: by the way, I think "tla tree-lint" would have complained about the file if you didn't add it | 01:37 |
| daf | carlos: have you modified your database? | 01:38 |
| daf | carlos: they're affected by that | 01:38 |
| SteveA | we need an "I want to commit" script | 01:38 |
| SteveA | it should run tree-lint | 01:38 |
| SteveA | and run tests | 01:38 |
| sabdfl | yes, it would... does it also give a useful exit code? | 01:38 |
| carlos | daf: the unittest should not depend on that, right? | 01:38 |
| SteveA | the unit test should not depend on any zcml | 01:38 |
| daf | sabdfl: I think the --strict option gives you that | 01:38 |
| === carlos should leave to have lunch so will be back in time for the daily meeting | ||
| carlos | later | 01:39 |
| daf | do we need to have the broken symlinks in lib/? | 01:40 |
| SteveA | daf: yes | 01:40 |
| SteveA | or, you can install the extra software | 01:41 |
| SteveA | so that the symlinks will not be broken | 01:41 |
| daf | why aren't the symlinks created when the extra software is installed? | 01:41 |
| SteveA | if you remove these symlinks then lifeless will complain at you | 01:41 |
| SteveA | and then add them back | 01:41 |
| SteveA | please ask lifeless about them :-) | 01:41 |
| daf | will do :) | 01:41 |
| daf | having tree-lint perpetually complaining is like having tests that perpetually fail | 01:42 |
| daf | people get used to it and don't fix it | 01:42 |
| SteveA | daf: I'm kinda busy, but please file a bug for this! | 01:44 |
| daf | good idea | 01:45 |
| BradB | daf: We've got some problems coming from the rosetta tests. I ran 81 launchpad tests, 5 failures, 8 errors. These seem to be mostly shallow, e.g. test_TranslatePOemplate_mungeMessageID has a few "NameError: name 't' is not defined" which appears to be because of the t = TranslatePOTemplate(context, request) assignment failing. | 01:53 |
| daf | BradB: interesting | 01:54 |
| BradB | It's possible that it's only on my machine. It's possible that it's a Real Problem though. | 01:55 |
| daf | I'll try and duplicate it | 01:55 |
| BradB | ok, thanks | 01:55 |
| daf | how are you running the tests? | 01:55 |
| daf | lalo: wow, canonical.rosetta.ftests.test_poexport.POExportTestCase is really intense | 01:57 |
| daf | lalo: it's been running at ~90% CPU for a minute or two | 01:58 |
| lalo | daf: import is worse :-P | 01:58 |
| daf | !! | 01:58 |
| lalo | these ftests suck | 01:59 |
| daf | can they be improved? | 01:59 |
| carlos | BradB: it's a real problem | 01:59 |
| lalo | it's the kind of thing that is a nightmare to test automatically | 02:00 |
| BradB | daf: python test.py -u | 02:00 |
| lalo | I've been beating them in a branch, but nothing concrete came out of it yet | 02:00 |
| daf | BradB: works for me | 02:01 |
| daf | lalo: is it the size of the test data? | 02:01 |
| lalo | there's always some greater priority coming up | 02:01 |
| carlos | daf: not here | 02:01 |
| BradB | daf: Maybe you have something carlos and I don't. | 02:01 |
| === carlos recreating the db | ||
| daf | BradB: or you have something I don't :) | 02:02 |
| carlos | daf: I don't have any change that is not already in rocketfuel | 02:03 |
| lalo | no, it's probably, to begin with, that it's still on autocommit | 02:03 |
| daf | autocommit? | 02:03 |
| daf | oh, right | 02:03 |
| daf | is that easy to fix? | 02:03 |
| lalo | no :-) | 02:03 |
| carlos | same problem after the db refresh | 02:04 |
| daf | carlos: which test is failing for you? | 02:04 |
| carlos | Failure in example: t._mungeMessageID(u'foo\nbar', [] ) | 02:04 |
| lalo | but the *proper* fix is already in my work list | 02:04 |
| carlos | FAIL: test_TranslatePOemplate_mungeMessageID | 02:04 |
| carlos | hmm, I think it should be TranslatePOTemplate... | 02:05 |
| carlos | not POemplate | 02:05 |
| carlos | :-P | 02:05 |
| daf | carlos: dude, that's a unit test -- it doesn't touch the DB at all :) | 02:05 |
| carlos | daf: I know, I told you that already :-P | 02:05 |
| daf | no, Rocketfuel is not missing any changes from me | 02:05 |
| carlos | daf: btw, the functional test fails also with the bug I told you yesterday that has msgid_plural == msgid | 02:05 |
| daf | carlos: hmmm | 02:06 |
| daf | carlos: what happens if you run "tla missing -s daf@canonical.com--2004/launchpad--devel--0" in your tree? | 02:08 |
| carlos | none missing | 02:09 |
| daf | sigh | 02:09 |
| carlos | I'm executing make fullcheck inside rosetta directory | 02:10 |
| daf | I think my patch 451 changed the tests | 02:10 |
| daf | sorry, rocketfuel@canonical.com/launchpad--devel--0--patch-451 | 02:10 |
| carlos | I changed the tests somedays ago | 02:11 |
| carlos | for the line wrap fix | 02:11 |
| carlos | but that's unrelated to this bug, or are you talking about other change? | 02:12 |
| daf | another change | 02:12 |
| daf | when I added the c-format hilighting | 02:12 |
| daf | I added a new test and changed some existing ones | 02:12 |
| carlos | carlos@frodo ~/Work/launchpad $ tla missing -s rocketfuel@canonical.com/launchpad--devel--0 | 02:13 |
| carlos | gpg: Good signature from "Patch Queue Manager (Canonical.com arch-pqm) <pqm@canonical.com>" | 02:13 |
| carlos | gpg: Good signature from "Patch Queue Manager (Canonical.com arch-pqm) <pqm@canonical.com>" | 02:13 |
| carlos | patch-422 | 02:13 |
| carlos | get bug browsing by source package under way | 02:13 |
| carlos | patch-462 | 02:13 |
| carlos | Stub files for canonical.launchpad | 02:13 |
| carlos | is that normal? | 02:13 |
| daf | carlos: so I changed then in patch-451 and you changed them in patch-453 | 02:14 |
| daf | I don't have patch-453, so perhaps that broke them | 02:14 |
| daf | I'll star-merge and try again | 02:15 |
| carlos | should I get the patch-422 by hand? | 02:15 |
| === carlos is confused | ||
| daf | ask an arch person | 02:15 |
| carlos | lifeless? | 02:15 |
| daf | carlos: it's 22:15 in Sydney | 02:17 |
| carlos | [lifeless] idle 00:00:02, signon: Tue Sep 28 02:05:33 | 02:17 |
| carlos | I suppose he's online :-P | 02:17 |
| lifeless | carlos: no, but I haven't time to look at it right now. just ignore it | 02:18 |
| carlos | ok | 02:18 |
| daf | okay, the tests fail now | 02:19 |
| carlos | daf: so it's a problem with my patch? | 02:19 |
| daf | I think so | 02:19 |
| carlos | I changed only the line wrap :-? | 02:20 |
| carlos | lalo: could you look at it? | 02:20 |
| carlos | please | 02:20 |
| daf | no | 02:20 |
| daf | it was the statistics that broke it | 02:20 |
| daf | the changes to browser.py | 02:20 |
| carlos | daf: ok, you are talking about the unittest | 02:20 |
| daf | yes | 02:21 |
| carlos | that makes sense, I forgot to upgrade the tests for the browser.py | 02:21 |
| carlos | what about the functional test? | 02:21 |
| carlos | daf: I will fix it now | 02:21 |
| daf | that should not have been affected | 02:21 |
| daf | carlos: I'm already fixing it :) | 02:21 |
| carlos | ok | 02:21 |
| carlos | thanks | 02:21 |
| carlos | btw, I will look at it to know how works those tests | 02:22 |
| daf | sure | 02:22 |
| daf | the problem is this: | 02:22 |
| daf | since the unit tests can't depend on the database, they use dummy objects | 02:22 |
| daf | DummyPOFile doesn't have the statistics attributes, so the browser.py code that tries to access them fails | 02:22 |
| carlos | oh, you are using stub like objects | 02:23 |
| carlos | ok, I get the idea | 02:23 |
| daf | --- orig/lib/canonical/rosetta/tests/test_browser.py | 02:25 |
| daf | +++ mod/lib/canonical/rosetta/tests/test_browser.py | 02:25 |
| daf | @@ -56,6 +56,12 @@ | 02:25 |
| daf | class DummyPOFile: | 02:25 |
| daf | pluralForms = 4 | 02:25 |
| daf | 02:25 | |
| daf | + def __init__(self, template): | 02:25 |
| daf | + self.poTemplate = template | 02:25 |
| daf | + | 02:25 |
| daf | + def translatedCount(self): | 02:25 |
| daf | + return 3 | 02:25 |
| daf | + | 02:25 |
| daf | 02:25 | |
| daf | class DummyMessageID: | 02:25 |
| daf | msgid = "foo" | 02:25 |
| daf | @@ -83,7 +89,7 @@ | 02:25 |
| daf | self.language_code = language_code | 02:25 |
| daf | 02:25 | |
| daf | if language_code in ('ja', 'es'): | 02:25 |
| daf | - return DummyPOFile() | 02:25 |
| daf | + return DummyPOFile(self) | 02:25 |
| daf | else: | 02:25 |
| daf | raise KeyError | 02:25 |
| daf | ^^^ the fix | 02:25 |
| daf | by the way, the unit tests should test the statistics code | 02:26 |
| carlos | daf: Now that you talk about the statistics, we should talk about the policy we will follow to get/update that information... | 02:26 |
| daf | well | 02:26 |
| daf | if we update the statistics | 02:27 |
| daf | 1) when we import | 02:27 |
| daf | 2) when we get new translations through the web | 02:27 |
| daf | then they should always be up to date | 02:27 |
| daf | agreed? | 02:27 |
| carlos | yes | 02:27 |
| daf | lalo is working on (1) | 02:27 |
| carlos | but, when will be updated "through the web"? | 02:27 |
| carlos | with every commit? | 02:27 |
| daf | yes | 02:28 |
| carlos | sorry, submit | 02:28 |
| daf | yes | 02:28 |
| lalo | but 2 is only +1/-1, right? | 02:28 |
| carlos | lalo: not really | 02:28 |
| carlos | you can update more than one string | 02:28 |
| daf | door | 02:28 |
| carlos | I think our limit is 5 now | 02:28 |
| lalo | it doesn't involve actually calling pofile.updateStatistics()? | 02:29 |
| lalo | because that method is awfully slow, and I don't think that is fixable | 02:30 |
| carlos | lalo: we could get the value and increment or decrement it as needed | 02:31 |
| carlos | but we could get some race conditions | 02:31 |
| lalo | ugh | 02:31 |
| carlos | we could forget that for now | 02:32 |
| carlos | :-) | 02:32 |
| carlos | so don't worry | 02:32 |
| carlos | lalo, daf: could you confirm me that the functional tests fails for you? | 02:35 |
| lalo | I do | 02:36 |
| carlos | ok | 02:36 |
| carlos | thanks | 02:37 |
| daf | can we avoid race conditions by having atomic database queries? | 02:46 |
| daf | okay, my unit test merge has gone in | 02:46 |
| daf | BradB: a star-merge should fix it | 02:47 |
| SteveA | daf: what does the channel title say? | 02:49 |
| SteveA | daf: and what do you mean "atomic database queries?" we're using transactions. | 02:50 |
| daf | SteveA: sorry | 02:50 |
| daf | SteveA: does changing lib/canonical/rosetta/tests/test_browser.py affect the rearranging? | 02:51 |
| SteveA | just don't touch it in RF until further notice | 02:51 |
| daf | SteveA: I'm thinking of ways of accessing the database that avoid race conditions | 02:51 |
| SteveA | what race conditions | 02:52 |
| carlos | daf: the problem is not the access to one field | 02:52 |
| carlos | daf: is that you could increment the count from the web interface | 02:52 |
| SteveA | daf: please express what you are saying in terms of multiple transactions | 02:52 |
| carlos | and at the same time import a new .po file so the cache is out of sync | 02:53 |
| daf | I was just thinking of concurrent accesses through the web | 02:54 |
| carlos | daf: that's easy to fix | 02:54 |
| carlos | with a "select for update" or things like that | 02:54 |
| SteveA | daf: each is in a separate transaction | 02:55 |
| carlos | you block the field, you can read it and write into it later and will be blocked for other transactions | 02:55 |
| SteveA | daf: this is fundamental to web applications | 02:55 |
| SteveA | daf: do not try to "fix" this, unless you can explain to me clearly what the problem is | 02:55 |
| daf | I won't try to fix it unless I'm sure there's a problem | 02:56 |
| daf | and I'm not sure at the moment | 02:56 |
| daf | I thinking of a situation like this: | 02:56 |
| daf | A: x = (SELECT foo FROM bar WHERE baz); | 02:57 |
| daf | B: y = (SELECT foo FROM bar WHERE baz); | 02:57 |
| daf | A: UPDATE foo SET bar=(x + i) WHERE baz; | 02:57 |
| daf | B: UPDATE foo SET bar=(y + j) WHERE baz; | 02:57 |
| lalo | to this point, I believed the plan was to update the stats from cron or something like that | 02:58 |
| daf | lalo: I'd like to avoid out-of-date stats if possible | 02:58 |
| lalo | I mean - on import AND from cron | 02:58 |
| daf | I suspect that caching the statistics may have been a premature optimisation | 02:58 |
| carlos | daf: UPDATE foo SET bar =(bar +i) WHERE baz; should work | 02:58 |
| daf | carlos: I think that would avoid the proble, | 02:59 |
| daf | problem | 02:59 |
| carlos | http://www.postgresql.org/docs/7.4/static/dml-update.html | 02:59 |
| carlos | lalo: yes, but from the interface, is interesting to know the real statistics when you are translationg | 03:00 |
| carlos | /s/translationg/translating/ | 03:00 |
| daf | not only "interesting", but useful | 03:00 |
| daf | if the statistics say that there are 5 untranslated messages when there are actually 0, you're going to wonder where they are | 03:01 |
| carlos | yeah | 03:01 |
| lalo | maybe we should either uncache the stats, or cache them more? | 03:02 |
| daf | more? | 03:02 |
| lalo | option 1. leave everything as is | 03:03 |
| lalo | 2. uncache | 03:03 |
| carlos | daf: http://www.postgresql.org/docs/7.4/static/transaction-iso.html | 03:03 |
| carlos | we should use the cache | 03:03 |
| daf | lalo: by 2, you mean always calculate | 03:03 |
| daf | lalo: right? | 03:03 |
| carlos | that way any query to see status will be fast | 03:03 |
| lalo | yes | 03:03 |
| daf | carlos: but we don't know how fast it is not to use the cache | 03:04 |
| daf | carlos: premature optimisation | 03:04 |
| carlos | and we try to update it always that it could change (an import or translation update from rosetta) | 03:04 |
| daf | carlos: (the root of all evil) | 03:04 |
| lalo | 3. add a "state" enum field to POMessageSet | 03:04 |
| carlos | daf: well, I can tell you that it will not scale, as lalo says, the current rutine to update the cached values takes some time | 03:05 |
| lalo | 4. add that field *and* remove *count from POFile | 03:05 |
| daf | carlos: have we tried optimising it? | 03:05 |
| carlos | and that should be executed every time we get a request... | 03:05 |
| daf | lalo: how would this new field work? | 03:05 |
| carlos | daf: <daf> carlos: premature optimisation <daf> carlos: (the root of all evil) | 03:05 |
| carlos | :-P | 03:05 |
| === carlos hides | ||
| daf | carlos: if it's fast enough, it doesn't matter if it gets executed each time | 03:06 |
| lalo | (missing, current, updated, in_rosetta, obsolete) | 03:06 |
| daf | since we already have the cache, let's just try keeping it up to date as much as we can | 03:06 |
| carlos | daf: sure, but it will be executed with every page request, I don't think it will scale... | 03:07 |
| lalo | so currentCount = select count(*) from pomessageset where state = current; | 03:07 |
| carlos | lalo: I don't see the "state" enum field, I mean I don't understand it :-) | 03:07 |
| daf | lalo: ah, so the point of this new field is to speed up the statistic calculations? | 03:08 |
| lalo | yes | 03:08 |
| daf | well, I suppose it could be useful in other ways too | 03:08 |
| lalo | what I meant by "cache more" :-) | 03:08 |
| === kiko [~kiko@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| carlos | hhmm, well, I think we have something like that already | 03:09 |
| carlos | if iscomplete == TRUE | 03:09 |
| lalo | do we? | 03:09 |
| carlos | it's translated | 03:09 |
| kiko | hey guys | 03:09 |
| lalo | carlos: that's orthogonal | 03:10 |
| carlos | so if iscomplete == TRUE && primemsgid = pot.primemsgid && pot.sequence > 0 | 03:10 |
| carlos | it's the same | 03:10 |
| carlos | po.iscomplete == TRUE && po.primemsgid = pot.primemsgid && pot.sequence > 0 | 03:10 |
| carlos | kiko: hey | 03:11 |
| === daf goes to make food | ||
| kiko | hey big c | 03:25 |
| dilys | Bug 2066 resolved: Error if I add a language that has a label from sampledata to my languages | 03:34 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2066 | 03:34 |
| === lifeless_ [~robertc@dsl-66.7.240.220.rns01-kent-syd.dsl.comindico.com.au] has joined #launchpad | ||
| === lifeless_ is now known as lifeless | ||
| kiko | lifeless, we're blowing up the repo :) | 04:23 |
| === ..[topic/#launchpad:SteveA] : Source code re-arrangement pretty much done. | ||
| SteveA | the major changes are now in rocketfuel | 04:30 |
| SteveA | please merge from RF, and start fixing up things you find are broken in the apps that you're responsible for. | 04:30 |
| SteveA | it is likely that the zcml files will be pointing to the wrong places for the SQL object classes | 04:30 |
| SteveA | All templates are now in lib/canonical/launchpad/templates | 04:31 |
| SteveA | All interfaces that we own are now in lib/canonical/launchpad/interfaces.py | 04:31 |
| SteveA | All SQL classes are now in lib/canonical/launchpad/database.py | 04:31 |
| SteveA | (or, will be once various other merges go through) | 04:32 |
| carlos | so is it better if we wait some minutes until pqm finish all merges? | 04:33 |
| kiko | carlos, I'd wait. it's not even running currently | 04:33 |
| carlos | daf, lalo: Are you going to fix Rosetta? | 04:43 |
| === carlos is ready to do it now, if I don't see anything against it I will start in 5 minutes | ||
| lalo | I suppose :-) | 04:44 |
| carlos | lalo: well, we should do it only one time :-P | 04:44 |
| lalo | ok, do it if you wish | 04:44 |
| lalo | I'm doing sth else | 04:45 |
| carlos | ok | 04:45 |
| carlos | What's imark.py, ikiko.py, etc...? | 04:56 |
| kiko | carlos, temparary crap that is to be consolidated | 04:58 |
| === carlos waits more time | ||
| carlos | :-P | 04:59 |
| SteveA | daf: ping | 05:05 |
| daf | pong | 05:06 |
| SteveA | there's some sql classes in rosetta/sql.py that need moving | 05:07 |
| SteveA | they need moving into lib/canonical/launchpad/database.py | 05:07 |
| SteveA | can you, or one of your team, do this? (and fix up the consequences) ? | 05:07 |
| daf | ok | 05:08 |
| lalo | is POFile one of these? | 05:12 |
| lalo | if it is, I ask to be responsible for doing this, as I can do it in the same process as merging my conflicts (which won't even be actual conflicts in this case) | 05:15 |
| kiko | just do it fast | 05:16 |
| SteveA | daf: and, like, do it now? ;-) | 05:18 |
| daf | lalo: why is it diffcult to get the functional tests running without autocommit? | 05:18 |
| lalo | daf: no relation | 05:18 |
| daf | lalo: I know there's no relation, but I'd like to know | 05:19 |
| daf | lalo: will you take repsonsibility for moving stuff out of rosetta/sql.py? | 05:19 |
| daf | SteveA: what's a vocabulary? | 05:20 |
| kiko | no idea eiter | 05:20 |
| lalo | daf: there is no relation between ftests being hard and transactions; I never said there is | 05:21 |
| SteveA | daf: stub is using them to make widgets for malone | 05:21 |
| lalo | there is, however, relation between ftests being SLOW and transactions | 05:22 |
| SteveA | a vocabulary is an interface to a queriable list of things | 05:22 |
| daf | lalo: ok, I got the impression that it would be difficult earlier | 05:22 |
| daf | lalo: if it's easy to make the functional tests use transactions, can you do it? | 05:23 |
| daf | SteveA: ok | 05:23 |
| lalo | it is difficult too, for completely unrelated reasons :-) | 05:23 |
| lalo | yes, that's what I'm doing | 05:24 |
| daf | oh, it *is* difficult | 05:24 |
| lalo | making ftests - and everything zopeless at once - use transactions | 05:24 |
| daf | that's what I was asking :) | 05:25 |
| daf | why is it difficult? | 05:25 |
| lalo | but now I'll interrupt that to integrate stuff | 05:25 |
| lalo | aha, I understand your question now :-) we had a precedence mismatch | 05:26 |
| kiko | lalo, dude, land these changes :) | 05:27 |
| lalo | I had understood "daf: lalo: why is it diffcult to (get the functional tests running) without autocommit?" and you meant "daf: lalo: why is it diffcult to get the functional tests (running without autocommit)?" | 05:28 |
| lalo | kiko: merging in | 05:28 |
| kiko | thanx | 05:29 |
| lalo | daf: it's not, per se. But we agreed in the list that, rather than doing that, I'd make zopeless itself use transactions. | 05:30 |
| lalo | which is what I'm working on. | 05:31 |
| daf | ok, and that's difficult? | 05:34 |
| lalo | in a 1..10 scale, 4 or 5 | 05:35 |
| daf | ok | 05:36 |
| daf | that's fine | 05:36 |
| lalo | SteveA: should I stick them into database.py for great justice, or use a temp drosetta (or dlalo) module like you guys did? | 05:41 |
| carlos | daf: I have the import script running with curl, thanks for the idea, works perfectly | 05:41 |
| daf | carlos: nice | 05:42 |
| lalo | there is the problem of integrating RosettaProduct with the canonical [:-)] Product class | 05:42 |
| kiko | yes | 05:43 |
| kiko | dont worry stuff it into ilalo or whatever | 05:43 |
| lalo | should I touch "your" code and do that? | 05:43 |
| kiko | and we'll sort it out (steveA says at least ;) | 05:43 |
| kiko | nah, keep rosettaproduct around but in ifoo.py | 05:43 |
| kiko | and we'll have to fix up the dupes after that | 05:43 |
| lalo | ok | 05:44 |
| daf | looks like I'll be offline for a while -- does anyone need something from me urgently before that happens? | 05:44 |
| lalo | and rename it to Product so that it conflicts, or leave it RosettaProduct so that it's easy to find? | 05:45 |
| daf | leave it for now, I think | 05:46 |
| carlos | daf: nothing from Spain :-P | 05:46 |
| daf | carlos: ok | 05:46 |
| kiko | lalo, leave it | 05:46 |
| daf | London? Brazil? | 05:46 |
| kiko | brazil wants a pillow | 05:46 |
| lalo | daf: I'm fine :-) | 05:47 |
| SteveA | daf: how long for? | 05:48 |
| daf | SteveA: no more than an hour or two, I think | 05:49 |
| SteveA | ok, see you later | 05:50 |
| lalo | SteveA: I assume helper classes like RosettaLanguages are going along | 06:03 |
| kiko | lalo: only SQLBase-inheriting stuff for now, I think | 06:05 |
| kiko | lalo, and hold off from mergeing anything else until this mess is sorted out | 06:05 |
| lalo | hmm. where do I leave RosettaLanguages etc then? | 06:07 |
| kiko | what sort of class is it? | 06:08 |
| lalo | set-of-all-foo | 06:08 |
| lalo | an utility to fetch Foo's | 06:09 |
| kiko | it will probably go as well, but it can be later | 06:09 |
| kiko | I'm going to suggest we have multiple database modules at least | 06:09 |
| lalo | well, it'd be easier if it went all at once :-/ | 06:12 |
| lalo | otherwise some integration work gets done twice | 06:12 |
| kiko | have you actually seen how hard the tree has been whacked? | 06:13 |
| lalo | yes :-) | 06:16 |
| lalo | precisely, I'd rather only have to recover from sth like that once ;-) | 06:17 |
| kiko | moving a couple of classes later shouldn't be hard, in comparison | 06:23 |
| lalo | what did other subprojects do with similar classes? I notice most sql.py's are gone | 06:25 |
| lalo | hmm | 06:28 |
| lalo | I can't test my changes, because yours don't actually import | 06:29 |
| SteveA | I think RosettaLanguages should go in database.py | 06:32 |
| lalo | thanks | 06:32 |
| lalo | do you want me to commit the important part (actually moving the classes) so that you guys can resume work, while I chase the orphan dependencies? | 06:35 |
| lalo | that part is done, I can commit right now | 06:35 |
| SteveA | lalo: don't check in now | 06:37 |
| SteveA | update from rf and see what mark has moved | 06:37 |
| lalo | ok | 06:37 |
| === justdave [~dave@24.247.63.44.gha.mi.chartermi.net] has joined #launchpad | ||
| -dmwaters_(dmwaters@dmwaters-gentoo.staff.freenode)- {global notice} Hi all! You guys can blame this netsplit on me.:) I restarted the firewall on that server, and didn't realize it wasn't running to start with, so it dropped all active connections.... sorry about that.... Thank you for your patience, and thank you for using freenode! | 06:45 | |
| lalo | I'm up to date now | 06:48 |
| kiko | now the *real* trouble begins | 07:03 |
| carlos | kiko: I have already troubles with the local changes I did this morning... | 07:04 |
| carlos | so, I'm not scared :-P | 07:04 |
| kiko | well | 07:04 |
| lalo | SteveA: what about stuff that are not classes, like our pet ugly hack personFromPrincipal? | 07:10 |
| === lalo ready to commit | ||
| SteveA | how is that a hack? | 07:13 |
| SteveA | ;-) | 07:13 |
| SteveA | that should live in lp for now | 07:13 |
| SteveA | it is an application component | 07:13 |
| SteveA | or, an application function | 07:13 |
| SteveA | or adapter | 07:14 |
| SteveA | or whatever | 07:14 |
| lalo | well, I'll leave it with the rest of the stuff then | 07:14 |
| lalo | you can sort out a "helpers.py" module later if you wish :-) | 07:15 |
| === lalo merges AGAIN before submitting his stuff | ||
| === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad | ||
| dilys | New bug 2069 for Launchpad/Launchpad: lib/canonical/database/configure.zcml should go to hell | 07:35 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2069 | 07:35 |
| spiv | lalo: Um, you didn't update launchpad.database.__init__ to import dlalo ;) | 07:37 |
| lalo | oops. I lost that when I used "undo" before updating from RF | 07:38 |
| lalo | I restored only dlalo from the undo :-) | 07:39 |
| dilys | New bug 2070 for Launchpad/Soyuz: Update the Wiki Documentation | 07:40 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2070 | 07:40 |
| lalo | I'll take a few hours break, then when you guys are done beating this stuff till it imports, I'll beat Rosetta till it runs again -- ok? | 07:41 |
| lalo | ("runs" defined as "accessible from the web, tests pass, scripts run") | 07:42 |
| === justdave [~dave@24.247.63.44.gha.mi.chartermi.net] has left #launchpad ["Poof"] | ||
| lalo | I believe daf and carlos will soon be past their working hours anyway | 07:43 |
| === justdave [~dave@24.247.63.44.gha.mi.chartermi.net] has joined #launchpad | ||
| carlos | lalo: but I will be online if you need anything from me | 07:44 |
| lalo | thx | 07:44 |
| carlos | I have still about two hours of work | 07:44 |
| lalo | well | 07:45 |
| lalo | if you want to merge from RF and start fixing orphan imports and references, have fun :-) | 07:46 |
| carlos | but did they finished all changes? | 07:47 |
| === carlos has some conflicts waiting for the end of the merge to be able to fix it | ||
| carlos | Working on a unrelated part, a script that does not needs anything from launchpad | 07:47 |
| lalo | fix the mess or leave it for me, either way it's fine with me. Now I'm leaving, bbl. | 07:51 |
| carlos | lalo: later | 07:54 |
| dilys | New bug 2073 for Launchpad/Soyuz: James Troup(elmo) as Soyuz Tester/Guide | 08:03 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2073 | 08:03 |
| dilys | New bug 2074 for Launchpad/Soyuz: Create the Postgresql Views for Soyuz App | 08:08 |
| dilys | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2074 | 08:08 |
| cprov | dilys: sorry, why are you in CC by default ? | 08:11 |
| carlos | cprov: he wants to have full control | 08:12 |
| carlos | cprov: he's THE big brother!! | 08:12 |
| cprov | carlos: aha | 08:13 |
| cprov | dilys: anyway, sorry for the spams :) | 08:14 |
| dilys | cprov: no problem | 08:18 |
| daf | carlos: it's "she", not "he" :) | 08:19 |
| carlos | daf: dilys is a girl name? | 08:20 |
| carlos | dilys: sorry :-P | 08:20 |
| daf | carlos: yep | 08:20 |
| carlos | daf: man, that sounds like she's your girlfriend :-P | 08:20 |
| daf | carlos: are you jealous? | 08:21 |
| carlos | kiko, cprov, SteveA: Are the rocketfuel changes over? | 08:21 |
| carlos | daf: not really :-P | 08:21 |
| === carlos is really loves python, with a 90 lines script I'm downloading and uncompressing all sources that are using cdbs, and I'm sure there should be a way to reduce the line numbers | ||
| cprov | carlos: not yet ... | 08:24 |
| carlos | well, please, forget the "is" in "carlos is really" | 08:24 |
| carlos | cprov: ok | 08:24 |
| kiko | carlos, no, the world is falling on my head | 08:39 |
| carlos | kiko: go go go | 08:40 |
| === lalo lends kiko a helmet | ||
| kiko | now it's raining scissors | 08:57 |
| === justdave [~dave@24.247.63.44.gha.mi.chartermi.net] has joined #launchpad | ||
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| === lal1 [~lalo@200-102-074-223.paemt7014.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| === lal1 [~lalo@200-102-074-223.paemt7014.dsl.brasiltelecom.net.br] has left #launchpad [] | ||
| !lilo:*! Hi all. Please welcome sysfault's #philosophy channel....if you are interested in such things, please stop by and look in! | 11:31 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!