| === stub [~zen@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| === stub [~zen@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| === stu1 [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| === kiko [~kiko@200-206-134-238.async.com.br] has joined #launchpad | ||
| === stu1 is now known as stub | ||
| === SteveA [~steve@adsl-213-190-44-43.takas.lt] has joined #launchpad | ||
| === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| === sabdfl [~mark@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| lifeless | stub: around | 10:46 |
|---|---|---|
| lifeless | ? | 10:46 |
| sabdfl | morning everyone | 10:55 |
| sabdfl | time for a little launchpad love | 10:55 |
| stub | lifeless: pong | 10:55 |
| sabdfl | daf, rosetta.warthogs.hbd.com seems borked in malone, db errors everywhere | 11:02 |
| daf | sabdfl: interesting | 11:04 |
| sabdfl | daf: can you see the cause? | 11:04 |
| daf | looks like it might be related to some work spiv was doing yesterday | 11:05 |
| daf | ProgrammingError: ERROR: current transaction is aborted, commands ignored until end of transaction block | 11:05 |
| sabdfl | also, earlier on it was asking me for a username and password, which should i use on that server? | 11:05 |
| daf | your username is your email address, assuming there is one for you in the sample data | 11:05 |
| daf | the password is also set in the sample data | 11:06 |
| daf | let me check... | 11:06 |
| daf | looks like you have an email address, but no password | 11:07 |
| stub | daf: I've seen that error before - if you get a database exception (such as you violated a key), the connection gets in a weird state and the server needs to be restarted. | 11:07 |
| daf | you can use test@canonical.com, password "test" | 11:07 |
| sabdfl | daf: thanks | 11:08 |
| stub | daf: I don't know if it sqlos, SQLObject or Zope3 yet - I'll be chasing it soon unless someone beats me to it. | 11:10 |
| daf | stub: ok, I don't think spiv's up yet, so i'ts probably up for grabs :) | 11:10 |
| daf | stub: do you know about the changes spiv made? | 11:11 |
| stub | He patched the publish routine didn't he? I don't know the reason he needed to | 11:11 |
| stub | I don't think it is related to the ProgrammingError exception though - that is an older bug. | 11:12 |
| daf | right, the problem we were seeing was that Zope was seeing stale DB data | 11:12 |
| daf | so spiv added an explicit transaction abort | 11:13 |
| daf | which apparently fixed the problem | 11:13 |
| stub | Was this stale data being retrieved from sqlobject/sqlos? | 11:13 |
| daf | not sure | 11:14 |
| daf | well, I assume it was | 11:14 |
| daf | since I'm pretty sure it wasn't being retrieved from anywhere else | 11:14 |
| daf | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=2005 is related, but I don't think it's the change in question | 11:15 |
| stub | I thought I'd fixed something similar with sqlos (by destroying the connection pool in the publish method, which sounds a similar fix to what spiv needed to do. Maybe my patch has been lost?) | 11:15 |
| daf | it's possible | 11:15 |
| daf | (that the patch was lost) | 11:15 |
| daf | can you check? | 11:15 |
| stub | Its still there at the top of beforeTraversal in publication.py | 11:17 |
| stub | I'll suck down spiv's changes as soon as I have my new arch archive online to see what he needed to add. | 11:19 |
| stub | (I think this was discussed on the launchpad@ list while I was 'away' | 11:19 |
| daf | well, the change I'm thinking of was something spiv did yesterday | 11:20 |
| daf | actually, forget that | 11:21 |
| daf | the change I'm thinking of *was* the patch at #2005 | 11:21 |
| daf | and it hasn't been merged to rocketfuel | 11:21 |
| stub | Hmm... actually... the ProgrammingError should be impossible if my fix is being invoked, as the connection object should not be reused in different requests. I suspect my fix is not being called :-( | 11:21 |
| daf | it's only been applied to the alpha server as a workaround | 11:21 |
| daf | I can test that | 11:22 |
| daf | where's the fix, exactly? | 11:23 |
| daf | (I can't find a method called "publish") | 11:28 |
| stub | Just above spiv's patch - lib/canonical/publication.py in beforeTraverse | 11:29 |
| daf | hmm, now that I've restarted the server the problem doesn't seem to occur | 11:32 |
| daf | ok, now I'm seeing "AttributeError: 'NoneType' object has no attribute 'id' | 11:34 |
| daf | " | 11:34 |
| === carlos [~carlos@67.Red-81-32-123.pooles.rima-tde.net] has joined #launchpad | ||
| daf | hrm, things are being weird | 11:40 |
| daf | hi carlos | 11:40 |
| carlos | morning | 11:40 |
| carlos | daf: so we launch today Rosetta? | 11:44 |
| daf | that's the plan | 11:45 |
| lulu | daf: are you removing things like: | 11:46 |
| lulu | Join translation team, Start transaltion effort in https://rosetta.ubuntulinux.org/rosetta/search | 11:46 |
| lulu | gives an error | 11:46 |
| lulu | to reproduce: I searched for firefox | 11:47 |
| daf | lulu: oops | 11:47 |
| daf | I'd forgotten about the "advanced" search page | 11:47 |
| lulu | in advanced search, search results we ahve some dummy data still | 11:47 |
| lulu | ok - could you sort that before go live? | 11:47 |
| daf | yep -- best thing is to disable it for now | 11:47 |
| daf | I'll file a bug about implementing it | 11:48 |
| lulu | daf: perhaps do a check of all pages of functionality you are launching to double check they are all working. | 11:48 |
| daf | lulu: right | 11:50 |
| daf | lulu: thanks for finding that | 11:50 |
| lulu | daf: no problem. The about page needs some info - I'm finding some for you... | 11:51 |
| lulu | daf: have you written the announcement yet? | 11:51 |
| lulu | daf: we also need rosetta@ubuntulinux.org mailing list set up | 11:53 |
| carlos | lulu: Will we create accounts into rosetta to anyone that ask for it? | 11:56 |
| carlos | I'm talking about the alpha | 11:56 |
| lulu | daf: oh - I see you have rosetta-testers for now - that should do the trick and shows it's still an Alpha. | 11:56 |
| stub | daf: Yes - the problem appears after you have violated a database constraint somehow, and stays around until you restart (as restarting the server resets all the database connections). | 11:59 |
| === carlos goes to have breakfast | ||
| lulu | carlos: why? do you want to restrict it only to those who are on your testing list? | 12:00 |
| ddaa | spiv: did I mention to you how I hate twisted? | 12:08 |
| === lalo [~lalo@200.102.200.234] has joined #launchpad | ||
| ddaa | spiv: I need your help for a quick crash course to twisted-proof subprocess handling. | 12:20 |
| spiv | ddaa: Ok... what do you need to know? | 12:21 |
| spiv | Or what problem are you having? | 12:21 |
| ddaa | Something totally weird like being unable to fdopen a fd we just created with pipe. | 12:22 |
| ddaa | We decided to use the twisted process handling when the environment has been twistedized. | 12:22 |
| ddaa | So as to avoid all the grief. Twisted assumes that everything going on is using it's process and thread stuff, so we'd rather make it happy3 | 12:23 |
| spiv | ddaa: Are you passing the right mode to fdopen? | 12:23 |
| lulu | carlos: Rosetta Alpha - people won't be able to create their own accounts. They will on the Beta when everything is integrated. | 12:24 |
| ddaa | Yes I'm passing the right mode. The problematic code is _working_ it only breaks on production in a twisted env... | 12:24 |
| spiv | (i.e. os.fdopen(os.pipe()[1] ) will fail, but os.fdopen(os.pipe()[1] , 'w') will work) | 12:24 |
| spiv | Ok, that's weird. :) | 12:24 |
| spiv | Where's the code? | 12:24 |
| ddaa | We had too much of that insanity. | 12:25 |
| spiv | (I've never heard of something like this before) | 12:25 |
| ddaa | It's just a bad idea to do things non-twistedly when twisted has pissed all over the place. | 12:25 |
| ddaa | Well, it's happening with some pretty ugly stuff... ya know the thread-that-reads-on-a-bloking-pipe | 12:26 |
| ddaa | But it would be simply least risky to just do things the twisted way. | 12:26 |
| ddaa | So | 12:27 |
| ddaa | 1. I would like to know what it is in twisted that do plenty of things with signal handlers, thread processing whatever which might cause grief. | 12:27 |
| ddaa | 2. | 12:27 |
| spiv | Twisted doesn't nothing that should affect threads. | 12:28 |
| ddaa | 2. How can I detect that the environment has been twistized? So I could use twisted process handling in that case, and only in that case. | 12:28 |
| spiv | (There are some things in Twisted and libraries for Twisted that spawn threads, and manage their own thread pools etc, but that's the extent of it) | 12:29 |
| ddaa | Okay. I take your word for it. | 12:29 |
| ddaa | Apparently, twisted _do_ things to the environment which cause my ugly subprocess handling to die in horrible ways. | 12:30 |
| spiv | Signal handling happens at reactor.run() time. | 12:30 |
| spiv | (unless you pass installSignalHandlers=0 to it) | 12:30 |
| spiv | It installs handlers for: | 12:30 |
| ddaa | our problem occurs within | 12:31 |
| ddaa | twisted/internet/threads.py:37:_putResultInDeferred | 12:31 |
| === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad | ||
| spiv | SIGINT (unless there's one already), SIGTERM (always), SIGBREAK (if it exists), and SIGCHLD (ditto) | 12:31 |
| ddaa | is that within reactor.run()? | 12:31 |
| spiv | Yes, that won't happen until the reactor has already started. | 12:32 |
| spiv | Oh, actually... | 12:32 |
| spiv | You could call that at any time. What's the error you get at that line. | 12:32 |
| spiv | ? | 12:32 |
| ddaa | Nothing at that line | 12:32 |
| ddaa | The error happens much lower in the stack in pyarch process handling code. | 12:33 |
| spiv | Hmmm.. | 12:33 |
| spiv | So, if that's the first call to that, that may cause a call to reactor._initThreadPool()... | 12:33 |
| ddaa | When I do things with the evil-blocking-pipe | 12:33 |
| spiv | What's the last line of the traceback, and the error? | 12:34 |
| ddaa | The pb is that this was the highest visible line in the backtrace lifeless gave me. | 12:34 |
| spiv | Ok, but what's the lowest? :) | 12:35 |
| ddaa | exceptions.OSError, [Errno 9] Bad file descriptor | 12:35 |
| ddaa | self.readfile = os.fdopen(read_end, 'r', 0) | 12:35 |
| ddaa | The line just before that is: | 12:35 |
| ddaa | read_end, self.write_end = os.pipe() | 12:35 |
| ddaa | Dude, I create a pipe, fdopen it and it barfs! | 12:36 |
| ddaa | I got another backtrace when closing self.readfile | 12:36 |
| spiv | What's the value of read_end? Perhaps you've run out of fds in that process? | 12:37 |
| spiv | The reactor does the exact same thing internally for it's "waker". | 12:37 |
| ddaa | do not have the value of read_end... | 12:38 |
| spiv | (See _UnixWaker in twisted/internet/default.py if you're curious) | 12:38 |
| ddaa | not going there unarmed... | 12:38 |
| ddaa | The problem here _might_ be a fd leak. | 12:39 |
| spiv | Hmm, although that should raise an OSError at the os.pipe call. | 12:39 |
| spiv | (I just tested) | 12:39 |
| ddaa | But then it would not explain the error on close. | 12:39 |
| ddaa | So, at this point, I just want to avoid doing things twisted is apparently not prepared for. | 12:40 |
| carlos | lulu: yes, that's why I was asking because I thought that, Alpha -> only accounts we created, Beta -> everyone | 12:40 |
| ddaa | We already had quite a lot of grief previously (remember at Oxford?) | 12:40 |
| ddaa | And know it breaking prod. | 12:41 |
| spiv | Well, I honestly don't know of anything in Twisted that interferes with this stuff. | 12:41 |
| spiv | In fact, Twisted internally relies on doing the exact same thing. | 12:41 |
| ddaa | So, please, just help me to figure out when the environment has been touched by twisted. | 12:41 |
| ddaa | So that the code would use twisted when run in a twisted program, and would use something else in other cases. | 12:42 |
| spiv | So unless there's some sort of bizarre race in your threads where another thread somehow can be closing the wrong fd, which seems unlikely, I don't know what's causing that problem (I don't know how to reproduce that behaviour even if I'm trying to...) | 12:42 |
| spiv | Oh, is this code running with my evil hack from Oxford? | 12:42 |
| ddaa | No. Your evil hack was only something for the test suite, and it did not work anyway. | 12:43 |
| spiv | Ok :) | 12:43 |
| ddaa | i separated the test suite in a twisted part and a pyarch part. | 12:43 |
| ddaa | I do not want to know what's the cause of the problem. | 12:44 |
| ddaa | I just want to use twisted when twisted is being used already. | 12:44 |
| ddaa | See what I mean? | 12:44 |
| spiv | Yes, I'm just looking that up | 12:44 |
| spiv | The .running attribute of the reactor can tell you if the reactor is running. | 12:45 |
| ddaa | Once I have that bit, i can dig up the "how to spawn coprocesses" part myself. | 12:45 |
| ddaa | How can I find the reactor? | 12:45 |
| spiv | from twisted.internet import reactor | 12:45 |
| ddaa | It's a module-global object? | 12:46 |
| spiv | Yes. | 12:46 |
| ddaa | Thx | 12:46 |
| spiv | A known design flaw, but convenient. | 12:46 |
| ddaa | Good for my purposes. | 12:46 |
| ddaa | I'll try reproduce the pb and do the twisted things, then. Thanks. | 12:47 |
| spiv | You're welcome. | 12:48 |
| spiv | I'd love to know what the problem is, it's very odd :) | 12:48 |
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| carlos | lalo: hey! | 01:06 |
| lalo | :-( | 01:06 |
| === lalo replaced the motherboard and stuff remains borked | ||
| lalo | that leaves two possibilities; either it's my cpu (that would suck, as it's not in warranty), or the hardware is now fine but the libc is borked due to having been compiled on faulty ram | 01:08 |
| lalo | meanwhile, gossip and lostirc to the rescue | 01:09 |
| daf | hi lalo | 01:12 |
| daf | good to have you on IRC again | 01:12 |
| lalo | hello | 01:13 |
| === cprov [~cprov@200.158.100.251] has joined #launchpad | ||
| lifeless | sqlos questions... | 01:22 |
| lifeless | how do I say 'NOW()' to sqlos | 01:22 |
| lifeless | and how do I say '1 day' to an interval field. | 01:22 |
| daf | lifeless: 'NOW' for 'NOW()' | 01:22 |
| lifeless | this is in an assigmnet: | 01:22 |
| lifeless | self._sync.processingapproved=NOW | 01:22 |
| lifeless | ? | 01:22 |
| daf | use the string 'NOW' | 01:23 |
| lifeless | oh, the string. | 01:23 |
| === SteveA goes to get lunch | ||
| === mdz [~mdz@69-167-148-207.vnnyca.adelphia.net] has joined #launchpad | ||
| carlos | stub: do you think we should add indexes also for foreign key or with inner joins is enough? (talking about improving the speed) | 01:27 |
| spiv | lifeless: A quick google suggests 'import sqlobject.sqlbuilder; sqlobject.sqlbuilder.SQLConstant("INTERVAL 1 DAY")' | 01:30 |
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| lifeless | spiv , oh ewww. so much for domain & data separation | 01:31 |
| spiv | There might be a nicer way using mxDateTime's DateTimeDelta or something like that. | 01:32 |
| spiv | Or even datetime.timedelta(1) | 01:33 |
| cprov | spiv, stub: hi guys ! | 01:34 |
| lalo | I seem to remember me and Daf wasting a good half hour in Oxford trying to find what kind of datetime objects sqlo uses | 01:34 |
| lalo | I think it was mx | 01:34 |
| spiv | lalo: It has "converters", so in theory it can support multiple kinds. | 01:34 |
| spiv | Glancing at the source though, it doesn't have any sane support for intervals, though, just absolute dates :/ | 01:35 |
| lalo | in theory :-) I don't know whether these converters are recent enough to know about the datetime module | 01:35 |
| lalo | anyway, the reason we were looking was because we wanted to know what objects it would *return* | 01:35 |
| spiv | lalo: They are. | 01:36 |
| spiv | But not smart enough to know about datetime.timedetla. | 01:36 |
| spiv | datetime.timedelta, rather. | 01:36 |
| lalo | daf: fix, part 1 of 3, done; if we're importing a pofile we already know about (in the db), and the db row has pluralform info, we use that | 01:38 |
| lalo | part 2 (being done now) is, if we don't have that, try the languages table; part 3 is scream bloody murder if we don't have either | 01:39 |
| lifeless | spiv: so how do I protect a form so only I can set a specific field | 01:41 |
| carlos | lalo: you need to take care of a fourth case, if the default plural form is not correct for that .po file | 01:41 |
| lalo | carlos: what kind of care? | 01:42 |
| carlos | lalo: the error should say that error is that the plural forms used on that file are not the same that the header info | 01:42 |
| carlos | or the table info so we can fix it easily | 01:42 |
| lifeless | spiv: that interval 1 day thing isn't working | 01:42 |
| carlos | where table == Language table | 01:43 |
| spiv | lifeless: Try SQLConstant("INTERVAL '1 DAY'") | 01:43 |
| lalo | carlos: ouch. That would be... rather involved :-P | 01:44 |
| spiv | lifeless: I'm currently working on making SQLObject support timedelta, it doesn't seem too hard... | 01:44 |
| lifeless | still not setting it | 01:44 |
| spiv | Try just the literal string? :/ | 01:44 |
| lifeless | nope | 01:45 |
| spiv | lifeless: You mean protect a field in a form in zope? I'm not sure, SteveA would know though... | 01:46 |
| lalo | oooh, found a bug | 01:47 |
| lifeless | well, not web form, the business logic level. Sync.enable() can only be called by me | 01:47 |
| lalo | oooh, found *another* bug | 01:50 |
| spiv | lifeless: andrew.bennetts@canonical.com/sqlobject--interval-support--0 should allow you to use datetime.timedelta(1) | 01:52 |
| spiv | lifeless: Only very lightly tested so far, though. | 01:52 |
| lifeless | and 1 is one second or 1 day ? | 01:53 |
| spiv | 1 day. | 01:53 |
| spiv | args for timedelta are days, seconds, microseconds. | 01:53 |
| lifeless | ok, cool | 01:53 |
| daf | lalo: I have a couple more interesting errors that happened overnight | 01:53 |
| carlos | daf: I'm testing the indexs part, I think with 3 indexs will be enough for the import | 01:54 |
| daf | carlos: yeah? | 01:55 |
| daf | carlos: what are they? | 01:55 |
| carlos | CREATE INDEX pomsgidsighting_index_inlastrevision ON POMsgIDSighting (pomsgset, pomsgid, inlastrevision); | 01:55 |
| carlos | CREATE INDEX pomsgidsighting_index_pluralform ON POMsgIDSighting (pomsgset, pluralform); | 01:55 |
| carlos | CREATE INDEX pomsgset_index_primemsgid ON POMsgSet (potemplate, pofile, primemsgid); | 01:55 |
| carlos | that's based on postgresql logs | 01:55 |
| daf | cool | 01:55 |
| daf | do you have any information on how they affect performance? | 01:55 |
| carlos | daf: I'm doing some tests now | 01:56 |
| carlos | my first test yesterday save me one minute importing the .pot file | 01:56 |
| daf | you gave me a link to some Postgres doc -- can you paste that again? | 01:56 |
| lalo | daf: good thing all pofiles we have are imported, because the header we're storing in newPOFile was absolutely BS and it would have breaken the exporter | 01:56 |
| spiv | lifeless: Ooops, give me 2 minutes before testing that :) | 01:57 |
| lifeless | spiv: ah ha | 01:57 |
| lifeless | I'll get a drink then shall I | 01:57 |
| daf | lalo: don't we have a but open on that? | 01:57 |
| daf | lalo: s/but/bug/ | 01:57 |
| === daf awaits the inevitable Beavis and Butt-head joke | ||
| lalo | daf: do we? | 01:57 |
| spiv | lifeless: Ok, working version mirroring now :) | 01:58 |
| daf | I thought we ded | 01:58 |
| daf | er, did | 01:58 |
| lalo | dude, get some caffeine :-P | 01:58 |
| carlos | http://www.postgresql.org/docs/7.4/static/tutorial-join.html | 01:59 |
| daf | lalo: just made some Earl Grey :) | 01:59 |
| carlos | http://www.postgresql.org/docs/7.4/static/indexes.html | 01:59 |
| daf | carlos: a | 01:59 |
| daf | grrr | 01:59 |
| daf | carlos: thanks | 02:00 |
| carlos | daf: :-P | 02:00 |
| spiv | lifeless: My branch lets you set values, but don't expect to get sane values back from reading that field yet :/ | 02:01 |
| daf | spiv: aren't you the SQLObject maintainer already? | 02:01 |
| lalo | ugh. I need to recreate the database each time I want to test this, because zopeless doesn't abort :-/ | 02:02 |
| spiv | daf: Hah. | 02:03 |
| lifeless | spiv: its still coming across as null. | 02:03 |
| lifeless | omg. | 02:03 |
| lifeless | IntCol('frequency', dbName='syncinterval', default=None), | 02:03 |
| lifeless | # WARNING: syncinterval column type is "interval", not "integer" | 02:03 |
| lifeless | # WARNING: make sure the data is what buildbot expects | 02:03 |
| lifeless | I blame you. | 02:03 |
| lalo | cool, there is no case 2. The database will always have info (from an old version or from languages table) due to newPOFile() | 02:03 |
| spiv | lifeless: Oh, d'oh. | 02:04 |
| spiv | lifeless: Change that to DateTimeCol. | 02:04 |
| lalo | so either case 1 (the database knows the info) or 3 (it doesn't, scream loud) | 02:04 |
| spiv | Incidentally, there's been a new upstream release of SQLObject, finally. | 02:05 |
| daf | oh, 0.6? | 02:08 |
| lalo | is that good? :-) | 02:08 |
| spiv | daf: yeah. | 02:08 |
| spiv | Although we're basically running it anyway... we're using a recent SVN snapshot. | 02:08 |
| spiv | But it's a sign that maybe upstream will get around to doing something about my bug reports one day ;) | 02:09 |
| daf | since we're basically including the source code in RF, I don't see any pressing need for me to update my packages, then | 02:09 |
| lifeless | spiv: thank thee. | 02:10 |
| spiv | thou art welcome ;) | 02:12 |
| === carlos goes to have a shower | ||
| === debonzi [~debonzi@200.158.100.251] has joined #launchpad | ||
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| daf | stub: I haven't seen any cache deletions happening yet | 02:26 |
| daf | but "yet" is about 3 requests | 02:26 |
| sabdfl | debonzi, cprov: good work on making the soyuz tables visible | 02:26 |
| sabdfl | for malone testing, we need some more sample data of packages, do you guys have any scripts that can populate the test db with a bunch of source packages and versions? | 02:26 |
| stub | daf: You lost me. You mean you stuck a debug message in the beforeTraverse and that fix is not being called? | 02:29 |
| daf | stub: exactly | 02:29 |
| stub | SteveA: ping | 02:29 |
| daf | stub: well, it gets the key, but they key is not in the cache | 02:29 |
| daf | stub: SteveA said he was having lunch | 02:30 |
| stub | daf: Hmm... maybe I'm just using the wrong key then? Can you dump repr(the_connection_cache_mapping) to see what is in it, and more importantly, what the keys are? | 02:31 |
| stub | It was working when I nuked the entire dictionary, but that isn't thread safe. | 02:31 |
| daf | each request seems to have a different key | 02:31 |
| daf | (16386, u'launchpad') | 02:31 |
| daf | (32771, u'launchpad') | 02:32 |
| daf | etc. | 02:32 |
| cprov | sabdfl: tks, we still working to have better results til sprint . | 02:32 |
| daf | stub: oh, ok, I'm seeing a deletion now | 02:33 |
| daf | stub: the error seems to occur when the cache deletions do | 02:33 |
| daf | stub: emphasis on "seems" | 02:33 |
| daf | I'll try dumping the entire cache mapping | 02:34 |
| stub | daf: You will only see a deletion the second time that thread handles a request. I think there are 4 threads by default, so you might not see one until you have loaded 4 or 5 pages. | 02:34 |
| sabdfl | cprov: what's the best way for us to get package data in for a couple of thousand packages? | 02:35 |
| daf | stub: right, that explains what I've been seeing | 02:35 |
| === stub wonders if z3 can be run in single thread mode like z2's '-t' command line argument | ||
| === stub reboots again | ||
| daf | wurg | 02:37 |
| stub | I think I've got this wireless nutted so it will come up on a reboot :) | 02:37 |
| cprov | sabdfl: I hope spiv is working on a DEB importer or something related | 02:37 |
| cprov | sabdfl: a extra component to get information of deb packages and insert automagicly(?) on DB | 02:38 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| stub | :) | 02:40 |
| lifeless | stub: so can you approve some of those projectproduct things? | 02:41 |
| daf | ok, the cache deletions seem to be working after all | 02:41 |
| daf | spiv: spiv is doubting that deleting the adapter is rolling back the transaction | 02:42 |
| daf | gaar | 02:42 |
| daf | stub: that was supposed to be directed at you | 02:42 |
| === daf is having a bad finger day | ||
| stub | daf: We can explicitly abort the connection at that point if we want (before we nuke it from the cache? spiv's fix might be running to late?) | 02:44 |
| stub | lifeless: I can't spell, and I get to approve them?? | 02:44 |
| lifeless | stub: yes. the only thing *I* care about is the project & product name so I can enter them into launchpad. | 02:45 |
| lifeless | that will probably make mark unhappy though, as his beloved shortdesc & description (whats with that?!??) will still need to be fleshed out. | 02:46 |
| stub | In the name of the senate and people of rome... *thwack* | 02:46 |
| lifeless | stub: well.. I just /need/ to be able to put them into the system such that I can associate Sync's to products, and then run the syncs., | 02:48 |
| stub | Ssh... I'm reading the wiki page... | 02:48 |
| lifeless | stub: ok, sshing. I've emailed mark asking to be able to ignore the desc & shortdesc, but haven't heard back. | 02:50 |
| stub | lifeless: So this is where we have to get the project -> product mapping right. I'd assumed bison was part of the GNU project, but you think it is part of its own. Isn't this getting very political? | 02:51 |
| daf | stub: I was worried about this happening :( | 02:51 |
| lifeless | stub: not my problem. | 02:52 |
| lifeless | my problem is getting cvs converted to arch. | 02:52 |
| spiv | Does it really matter if the mapping isn't quite right? We can update them later, surely? | 02:53 |
| lifeless | Not meaning to be a 'boundary drawing arsehole' .. just already about 20 imports behind. | 02:53 |
| lifeless | spiv: given the choice, I'd 100% ignore the projects and products until someone has time to add them and reattach the syncs. | 02:53 |
| stub | I'm not sure I can do that. I don't understand what it means when a package has a package or product of "". I also don't know what half of the packages are, and those I do recognize I'm not sure what the project should be :-( | 03:01 |
| stub | Does the project name get encoded anywhere permanent in arch (such as the repository name) that we can't fix without rebuilding? | 03:03 |
| lifeless | stub: package comes from the debian package name | 03:07 |
| lifeless | project "" means 'same name as the package' | 03:08 |
| lifeless | product "" means 'same name as the package' | 03:08 |
| lifeless | the project name is used as the arch archive name (project@arch.ubuntu.com) except when I decide otherwise. | 03:08 |
| lifeless | and you cannot fix that: these are one-shot. | 03:08 |
| lifeless | except when there is a very very very good reason, and /no one/ accessing it. | 03:09 |
| lifeless | (accessing it - ever) | 03:09 |
| lifeless | stub: if you can't do it, then I need Mark to be told that: he was trying to delegate it out of being his problem... | 03:09 |
| stub | Can a product move between projects, ever? | 03:10 |
| lifeless | I don't know: there are no use cases for these object. | 03:11 |
| lifeless | daf may know as rosetta directly exposes these all the time. | 03:11 |
| lalo | it is not immutable | 03:12 |
| lifeless | AIUI every aspect of launchpad depended on these objects. | 03:12 |
| stub | An example would be mod_python, which after being developed for a few years became an Apache project. | 03:12 |
| stub | Where nothing changed except their web server and their logos | 03:12 |
| lifeless | stub: again though: this isn't terribly relevant: we want to know what they should be to perform the imports /today/. | 03:12 |
| lifeless | if someone changes location in the arch world, they just tag across. | 03:12 |
| lifeless | that will be fine:: once we get arch available to the project. | 03:13 |
| stub | Cool. So the project looks quite changable - we just need to implement that if it becomes a problem. | 03:13 |
| lifeless | thus my "ignore them all and just do the imports' wish.. | 03:13 |
| daf | stub: "ever", sure | 03:14 |
| === lalo cries | ||
| === stub had also assumed that product names would be changable too | ||
| lalo | done recompiling the libc, and stuff remain broken :-( | 03:15 |
| daf | ouch | 03:15 |
| daf | that leaves the CPU, then? | 03:15 |
| lifeless | stub: I presume its all mutable. this is just hte initialdata load. | 03:16 |
| stub | Right. So what is all the fuss about then? | 03:16 |
| lalo | I'll try downgrading it | 03:16 |
| === stub ducks | ||
| lalo | (the libc, not the cpu :-P) | 03:16 |
| stub | lifeless: I'll go through and approve the ones I recognize, and reply-all to your email saying if they need approval we really should get a package maintainer or someone with the right knowledge to become involved. | 03:18 |
| lalo | ah well, firefox 1.0pre runs | 03:19 |
| lalo | YAY, gaim runs too! :-D my system is fixed | 03:20 |
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| daf | so it was a software problem all along? | 03:21 |
| lalo | no - a combo :-) | 03:22 |
| lalo | the libc was broken because it had been compiled on faulty ram | 03:22 |
| lalo | I've seen that before | 03:22 |
| daf | and you wonder why we say that Gentoo is crack :) | 03:22 |
| lalo | what if a debian maintainer gets faulty ram? :-) | 03:23 |
| lalo | at least I got bummered due to my own mistake, not someone else's | 03:24 |
| daf | only teasing :) | 03:24 |
| lalo | ooh, firefox1 knows about gnome and gconf | 03:25 |
| lalo | it just told me "it's not set as my default browser" :-P I never thought I'd see this message outside windows | 03:25 |
| daf | stub: ProgrammingError: ERROR: column sourcepackage.name does not exist | 03:42 |
| stub | Yer - I know that one. | 03:42 |
| spiv | stub: I'd be interested in your thoughts on my latest patches for #2005. | 03:43 |
| stub | daf: If you want to test Malone, click on the products link on the front page and use that (or the bug list). Sourcepackage browsing is buggered atm. | 03:43 |
| daf | they seem to be working fine | 03:43 |
| daf | (the patches) | 03:44 |
| stub | spiv: What is the thread issue being discussed in the comments of the SQLObject patch (seems to be about returning a list rather than an iterator from the database) | 03:47 |
| spiv | I'm not sure, exactly :) | 03:48 |
| spiv | I think the concern is that SQLObject can have multiple active cursors for a connection alive under some circumstances. | 03:48 |
| spiv | So if we have lazy iterator in one thread, and another releases the connection, then the lazy iterator will break. | 03:49 |
| spiv | Hence the listification. | 03:49 |
| spiv | Perhaps a safer change to SQLObject would've been to wrap the result in iter(...), rather than strip the list(...). | 03:49 |
| stub | spiv: So IZopeSQLConnection(newconn()).transaction() registeres the connection with the transaction system, starting the database transaction at the same time? | 03:49 |
| spiv | I think so :) | 03:50 |
| stub | spiv: The connections shouldn't be shared between threads. It might be that the cursor is being reused? | 03:51 |
| spiv | My basic thinking was that if we use the Transaction object that SQLObject can give us, that's a better fit for the model Zope expects. | 03:51 |
| spiv | (i.e. a web request gets its own transaction) | 03:51 |
| === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
| stub | So the sqlos lazy updating stuff is now useless? | 03:52 |
| stub | (as in unnecessary) | 03:52 |
| === ddaa [~david@nemesis.xlii.org] has joined #launchpad | ||
| spiv | I suppose so, except as an optimisation maybe. | 03:53 |
| spiv | (to avoid unnecessary round trips to the DB) | 03:53 |
| lifeless | stub, spiv - enterprise patterns applies here :) | 03:53 |
| lifeless | APE probably has it already too... | 03:53 |
| stub | I don't know too much about the SQLObject level stuff, and haven't dealt with that layer. | 03:54 |
| spiv | stub: Well, you haven't pointed out any reason why it's obviously stupid, which was my main concern ;) | 03:58 |
| daf | I'd be interested to hear what Steve thinks | 03:59 |
| stub | spiv: Seems fine actually | 04:03 |
| SteveA | hi stub | 04:10 |
| stub | Morning | 04:11 |
| SteveA | is there still a problem with transactions etc? | 04:15 |
| stub | Yes - to do with database connection ending up in a 'you are not in a transaction so I'm going to just raise exceptions' state, which shouldn't be happening since we are supposed to be creating a new database connection every transaction (the last time we fixed sqlobject/sqlos). | 04:18 |
| stub | Sounds like spiv fixed it though - I havn't tested myself yet. | 04:18 |
| stub | Ooh... just got an announcement that SQLObject 0.6 has been released | 04:19 |
| stub | from whats new: "n 0.5.3: some small bug fixes, and an important fix when iterating over selects in threaded environments" <-- spiv | 04:20 |
| spiv | stub: Hmm! | 04:39 |
| SteveA | can we have a launchpad meeting tomorrow? | 04:49 |
| daf | SteveA: I don't see why not | 04:50 |
| spiv | stub: Hmm, that change is the one my patch backed out | 04:50 |
| carlos | SteveA: it's ok for me | 04:51 |
| spiv | stub: But current 0.6 SVN has an iter(...) around that value. | 04:51 |
| spiv | stub: So we probably should update our SVN snapshot to the 0.6 release. | 04:51 |
| carlos | daf: with the indexes I get sometimes 11 minutes, other times 5 minutes... :-? | 04:51 |
| stub | Just gimme the time | 04:51 |
| daf | carlos: running VACUUM ANALYZE might affect it | 04:51 |
| daf | carlos: it might be worth trying EXPLAIN also | 04:52 |
| stub | spiv: Yes. I've lost track if we have local changes to ours though :-( | 04:52 |
| daf | carlos: I've been profiling the importer, and I should have some data soon | 04:52 |
| carlos | EXPLAIN with what? | 04:52 |
| SteveA | stub: what's the earliest time after 1200UTC that works for you? | 04:53 |
| carlos | daf: I mean, I'm not quering anything, is the script | 04:54 |
| spiv | stub: We do :( | 04:55 |
| stub | As early as possible - 1200UTC is 10PM here. | 04:55 |
| daf | carlos: sure, but VACUUM ANALYZE will update the indexes | 04:55 |
| carlos | daf: I know | 04:55 |
| lalo | now my system is too stable. I should install gnome 2.8. }:-) | 04:56 |
| carlos | daf: BTW seems like now it's more stable, a reimport takes always 5 minutes | 04:57 |
| carlos | and before the index it took about 10 minutes | 04:57 |
| carlos | I will do a last test before send the request to merge those database changes | 04:57 |
| carlos | to be sure we are improving anything | 04:57 |
| carlos | lalo: apt-get install warty :-P | 04:58 |
| SteveA | lalo: can you make any earlier than 1200 UTC? | 05:00 |
| lalo | SteveA: sure | 05:00 |
| daf | carlos: nice! | 05:00 |
| === kiko [~kiko@200-206-134-238.async.com.br] has joined #launchpad | ||
| daf | carlos: reimporting was getting progressively slower on the server | 05:00 |
| SteveA | lalo: suggest a time? | 05:01 |
| lalo | I've been 9-to-5 London time, except yesterday that I had to go out, and I intend to continue with this time schedule till the end of the month | 05:01 |
| kiko | morning guys | 05:01 |
| lalo | it's just less trouble to me | 05:01 |
| lalo | hello kiko | 05:01 |
| SteveA | kiko, cprov, debonzi: how much earlier than 1200 UTC is good for you for a launchpad meeting? | 05:01 |
| kiko | SteveA, 11UTC is the bare minimum | 05:02 |
| kiko | SteveA, what day? | 05:02 |
| daf | kiko: tomorrow | 05:02 |
| daf | carlos: have you looked at http://www.postgresql.org/docs/7.4/static/performance-tips.html? | 05:02 |
| SteveA | kiko: is 11.30 UTC okay? | 05:03 |
| kiko | yes, definitely. | 05:03 |
| SteveA | ok. I'll mail that out | 05:03 |
| carlos | lalo: cool | 05:03 |
| carlos | daf: same here, but I think the VACUUM command should improve it | 05:04 |
| carlos | daf: no, I will look at it now | 05:04 |
| lalo | carlos: can you verify that my latest merge closed #1979? | 05:19 |
| lalo | not urgent by any means | 05:20 |
| carlos | lalo: ok, added to the queue of tasks in my mind | 05:20 |
| carlos | as soon as I finish with the speed tests I will look at it | 05:20 |
| lalo | great, thanks | 05:20 |
| lalo | if I'm not around when you do, pls mark the bug "verified" | 05:21 |
| carlos | sure | 05:21 |
| daf | SteveA: could you take a look at #1907 and #1908? | 05:22 |
| daf | SteveA: they shouldn't take long | 05:22 |
| lalo | daf: what about zopeless... will we leave it be or what? | 05:22 |
| daf | lalo: what do you mean? | 05:23 |
| lalo | zopeless seems to be unsuitable for anything but unit tests (and even for that it would be nice if it aborted the transaction) | 05:23 |
| daf | why is it unsuitable? | 05:23 |
| lalo | 1: no lazyUpdate, therefore slower; 2: no transaction hooks, so it's impossible to abort (as I told you a few days ago) | 05:24 |
| lalo | (2) means if you get an exception in the middle of import, you're left with bogus data in your db | 05:24 |
| lalo | the solution to both is one and the same - do proper transaction hooking. The question is whether someone has time to do it; if they don't, either it's up to us, or we find a way for the script to run that doesn't involve zopeless | 05:26 |
| lalo | (zopeless-less? :-P) | 05:26 |
| spiv | lalo: I'll take a look at that right now. | 05:27 |
| lalo | spiv: thanks | 05:27 |
| daf | I thought there was some transactions going on somewhere even if we were running Zopeless | 05:28 |
| lalo | I don't know how well you know zope's transaction machinery; if you get lost, feel free to bug me, I'm reasonably intimate with it | 05:28 |
| lalo | daf: the zope transaction machinery is suitably poked, and I *think* sqlobject starts a database transaction too; but the two things don't talk to each other | 05:29 |
| SteveA | it would be nice to have a "zopeless" that isn't zopeless -- that uses a subset of zcml files, but does set up the whole environment | 05:29 |
| spiv | daf: the initZopeless doesn't do anything to set up transactions, and doesn't currently expose any way to get at them. | 05:29 |
| daf | spiv: ok | 05:30 |
| spiv | So, as lalo points out, you can't say "abort this transaction" in the Zopeless environ atm. | 05:30 |
| daf | spiv: we are using transaction.get_transaction -- is that a no-op? | 05:30 |
| lalo | SteveA: yeah. everything but zodb and publisher. | 05:30 |
| spiv | Yeah, that's Zope's transaction stuff. | 05:30 |
| lalo | daf: yes | 05:30 |
| SteveA | lalo: excluding browser too | 05:30 |
| daf | lovely | 05:30 |
| spiv | Well, actually, it's sort-of exposed. | 05:31 |
| daf | wouldn't it be enough to expose some of the SQLObject transactionality stuff? | 05:31 |
| daf | or does the Zope stuff give us something extra? | 05:31 |
| spiv | daf: That's what I'm looking at :) | 05:31 |
| lalo | daf: it's easy to hook stuff to the zope transaction machinery | 05:31 |
| spiv | (it'd be nice, but harder, to use the zope transaction machinery...) | 05:31 |
| lalo | it doesn't give you a lot of extra, just a nicer API :-) | 05:31 |
| daf | well, nice APIs are... nice | 05:32 |
| daf | :) | 05:32 |
| spiv | We haven't even got that right when running Zope yet ;) | 05:32 |
| daf | :D | 05:32 |
| lalo | true | 05:32 |
| SteveA | spiv: are you going to improve zopeless? | 05:32 |
| lalo | well, whatever works. :-) | 05:32 |
| spiv | SteveA: By the bare minimum necessary to support aborting the current transaction. | 05:32 |
| spiv | (unless it looks like a ridiculous amount of work) | 05:33 |
| lalo | I'd prefer to improve it enough to allow lazyUpdate to work | 05:33 |
| SteveA | I suggest you set up the minimal utilities and adapters necessary to make it use plug into the zope transaction machinery | 05:33 |
| lalo | which would probably involve hook to zope's transaction | 05:33 |
| SteveA | and then add a sys.exit() hook that aborts any outstanding transaction | 05:33 |
| spiv | lalo: What's not working about that? Just that it never finishes? | 05:34 |
| spiv | (and so never commits the lazy updates?) | 05:34 |
| daf | carlos: ok, I have some data on which sql.py calls are taking the most time | 05:34 |
| lalo | spiv: it is not working, as a whole :-) it was so broken that we disabled it completely | 05:35 |
| carlos | daf: could you add them to the bug report? | 05:35 |
| carlos | hmm | 05:35 |
| lalo | there is a bug with the detailed info, I can look it up to you | 05:35 |
| spiv | lalo: Yes please :) | 05:35 |
| carlos | daf: yes, attach it to bugzilla and I will try to improve them | 05:35 |
| daf | carlos: sure | 05:36 |
| lalo | https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=1935 | 05:36 |
| carlos | daf: thanks | 05:37 |
| lalo | lots of zen from Steve; and me, playing the role of Bill Gates, saying that "we probably won't need it anyway" :-P | 05:37 |
| lalo | but, well, premature optimization yadda yadda. | 05:38 |
| daf | carlos: at a glance, RosettaPOMessageSet.getMessageIDSighting, RosettaPOFile.__getitem__, RosettaPOMessageSet.pluralForms, RosettaPOMessageSet.templateMessageSet are by far the most expensive | 05:39 |
| SteveA | daf: do you mean that they are the most expensive run once, or most expensive cumulatively ? | 05:39 |
| daf | so the columns used in those queries are prime targets for optimisation | 05:39 |
| daf | SteveA: cumulativelyt | 05:39 |
| SteveA | which ones are called most often? | 05:39 |
| daf | SteveA: number of calls * time per call | 05:40 |
| carlos | daf: the indexes I'm adding will improve some of them | 05:40 |
| daf | carlos: right | 05:40 |
| spiv | SteveA: glancing over daf's shoulder, they're being called thousands of times, and taking 0.5 or even 1.0 second per call (cumulatively). | 05:40 |
| daf | SteveA: I don't have that data to hand; the hotshot stats printer is *very* slow | 05:41 |
| daf | SteveA: I could generate it, though | 05:41 |
| daf | SteveA: of the sql.py calls, it seems that getMessageIDSighting is called the most | 05:41 |
| SteveA | this is the point at which we may want to introduce a specific method or two | 05:42 |
| daf | SteveA: on the order of 3000 times in an import, compared to about 2000 for the other worst offenders | 05:42 |
| SteveA | that does a bunch of work quickly at the SQL level, | 05:42 |
| SteveA | trading the clarity and flexibility of objects with "one specific call" | 05:42 |
| daf | no, that's wrong | 05:42 |
| daf | getMessageIDSighting is getting called almost exactly twice as much as the others | 05:43 |
| SteveA | also, if we have a method that returns an essentially immutable object, we may choose to cache that | 05:43 |
| SteveA | why are we using a "1" for plural_form ? | 05:46 |
| SteveA | what is the meaning of the "1" ? | 05:46 |
| SteveA | or the "0" | 05:46 |
| daf | where? | 05:46 |
| lalo | 0 is invalid | 05:46 |
| daf | lalo: invalid where? | 05:47 |
| lalo | 1 means there is only one form | 05:47 |
| SteveA | I just grepped to see where getMessageIDSighting is used | 05:47 |
| lalo | daf: POFile.pluralForms | 05:47 |
| lalo | 1 is for, eg, Japanese, Turkish, Korean | 05:48 |
| SteveA | is 1 a number in this case? | 05:48 |
| daf | lalo: no, I think Steve is talking about MessageIDSighting.plural_form | 05:48 |
| lalo | yes | 05:48 |
| SteveA | or is it symbolic? | 05:48 |
| SteveA | msgset.getMessageIDSighting(0, allowOld=True).dateLastSeen = "NOW" | 05:48 |
| SteveA | old_plural = self._msgset.getMessageIDSighting(1) | 05:48 |
| lalo | ah, ok | 05:48 |
| SteveA | I find the 1 and 0 in there opaque | 05:48 |
| lalo | 0 and 1 are numbers in this case; indexes | 05:48 |
| SteveA | ok | 05:49 |
| SteveA | then that's fine | 05:49 |
| SteveA | but, I'm a little confused as to why I see only 1 and 0 | 05:49 |
| lalo | as messageIDs are supposed to be English, then 0 is the singular and 1 is the plural | 05:49 |
| SteveA | that makes them seem rather symbolic | 05:49 |
| SteveA | then why don't we say SINGULAR and PLURAL ? | 05:49 |
| daf | SteveA: this stems from the design of the database | 05:50 |
| lalo | we could | 05:50 |
| lalo | but ENGLISH_SINGULAR and ENGLISH_PLURAL rather | 05:50 |
| lalo | or something like that | 05:50 |
| daf | SteveA: Mark wanted us to cope with the theoretical possibility of the plural form of a message ID sighting being > 1 | 05:50 |
| carlos | daf: ok, the indexes improve the import in my machine, about 1 minute less | 05:50 |
| carlos | I will send now the database inclusion | 05:50 |
| spiv | daf: How does that conflict with using symbolic constants in the code? | 05:50 |
| daf | spiv: it doesn't | 05:50 |
| SteveA | daf: the app's code attaches meaning to 1 and 0 | 05:51 |
| spiv | Ok :) | 05:51 |
| daf | spiv: that only explains why you only see 0 and 1 | 05:51 |
| SteveA | 1 and 0 are hard-coded | 05:51 |
| lalo | you ppl want me to make the change? | 05:51 |
| SteveA | the app is hard-coded for english message ids | 05:51 |
| spiv | Right. Just making sure I understand :) | 05:51 |
| SteveA | so, I think the app's understanding on 0 and 1 should be hard-coded too | 05:51 |
| daf | SteveA: well, not strictly English | 05:51 |
| daf | "Germanic" would be a closer approximation | 05:52 |
| SteveA | work out a good, simple, name for the 0 and 1, and let's use that name | 05:52 |
| SteveA | or, those names, rather | 05:52 |
| spiv | daf: "normal" or "standard" :P | 05:52 |
| daf | SINGULAR and PLURAL would be reasonable names | 05:52 |
| SteveA | and perhaps these should be in a dbschema | 05:52 |
| daf | perhaps | 05:52 |
| lalo | the app is hard-coded for english message ids as a phase 1 decision; since gettext assumes msgids have only two forms (actually gettext assumes the msgids are English), and phase 1 is gettext only, therefore the app is hardcoded for English | 05:52 |
| daf | right | 05:53 |
| SteveA | that's fine. but let's not be half-hearted with the hard-coding | 05:53 |
| SteveA | because that's just confusing | 05:53 |
| lalo | ok | 05:53 |
| daf | but if we're targeting a system other than gettext, the number of plural forms is going to be the least of our worries | 05:53 |
| lalo | daf: are you sure you're comfortable with SINGULAR and PLURAL? I'll add these | 05:54 |
| daf | lalo: I'm sure | 05:54 |
| SteveA | why is the allowOld argument not in the interface? | 05:54 |
| SteveA | actually, I'm looking at old code, so I should update before asking further | 05:54 |
| lalo | ok | 05:54 |
| daf | SteveA: perhaps the interface is wrong | 05:54 |
| SteveA | perhaps. perhaps allowOld is used only as a private API. but that would be rather odd. | 05:55 |
| daf | it's not private | 05:55 |
| SteveA | Don't do this: | 05:57 |
| SteveA | raise UnknownUserError, \ | 05:57 |
| SteveA | "Tried to create objects but have no Person to associate it with" | 05:57 |
| SteveA | Instantiate the error, instead | 05:57 |
| SteveA | raise UnknownUserError("Tried to create objects but have no Person to associate it with") | 05:57 |
| SteveA | then, you don't need to use a \ | 05:57 |
| spiv | lalo: I just realised that I don't need to make any changes to SQLBase to support transactions. | 05:57 |
| spiv | lalo: The trick is to not throw away the connection you pass to initZopeless. | 05:58 |
| SteveA | daf: I'm looking at pofile_adapters.py | 05:58 |
| SteveA | try: | 05:58 |
| SteveA | return self._msgset.getTranslationSighting(index, allowOld=True).poTranslation.translation | 05:58 |
| SteveA | except KeyError: | 05:58 |
| lalo | spiv: hmm? | 05:58 |
| SteveA | it would be better to have the try:except: around exactly the thing that can raise a KeyError | 05:58 |
| SteveA | there is too much between the try:except | 05:58 |
| SteveA | _marker = [] | 05:59 |
| SteveA | better to use _marker = object() | 05:59 |
| daf | SteveA: lalo owns this code | 05:59 |
| SteveA | you also own the code | 05:59 |
| spiv | lalo: You could even do "conn = connectionForURI(...); trans = conn.transaction(); SQLBase.initZopeless(trans); .... ; trans.rollback()" | 05:59 |
| daf | SteveA: yes, transitively | 05:59 |
| daf | SteveA: lalo is the direct owner | 05:59 |
| lalo | spiv: ok | 05:59 |
| lalo | I'll try that | 06:00 |
| daf | SteveA: I completely agree with what you're saying | 06:00 |
| SteveA | great | 06:01 |
| SteveA | lalo, daf: why is there a bare except in pofile_adapters.TemplateImporter.__call__ ? | 06:02 |
| SteveA | there *must* be specific exceptions that you expect | 06:02 |
| lalo | no | 06:03 |
| SteveA | same for POFileImporter | 06:03 |
| SteveA | I don't want to see MemoryError caught | 06:03 |
| spiv | Or KeyboardInterrupt, or ConflictError... | 06:03 |
| SteveA | the only time when you don't know the specific exceptions that you expect is when you're interacting with a badly designed API | 06:04 |
| SteveA | so, code that interacts with the db api has this problem | 06:04 |
| SteveA | but, this shows flaws in the db api and how it exposes exceptions | 06:04 |
| lalo | I don't feel like arguing - I have decided a few hours ago that this exception masking thing is wrong, so instead of fixing it I'll remove it. Later. | 06:05 |
| SteveA | this isn't an "argument". I'm suprised at some of the code I'm coming across, and I want to find out why it is the way it is, and how it can be improved. | 06:05 |
| lalo | I know | 06:06 |
| lalo | it would be an argument, if I tried to defend the reasons why I used a bare except :-) | 06:06 |
| lalo | but before I even do, I concede, it's wrong. | 06:06 |
| SteveA | is it something that can be fixed in the pofile_adapters module, or does it need to be fixed in the code that that module is using? | 06:07 |
| lalo | the idea was that when you raise a POInvalidInputError you get the line number in the PO file where the exception was raised; so this would supposedly make it easier for us to debug stuff. But it's too open-ended, and it has caused as much troubled as it solved, so it's scheduled to die. | 06:07 |
| SteveA | is there a bugzilla bug for it? | 06:07 |
| lalo | which it? removing the exception masking? no, it was just something in my head I wanted to discuss it with the team first. | 06:08 |
| SteveA | ok. | 06:09 |
| SteveA | exception conversion is fine, and good, by the way | 06:09 |
| SteveA | it is natural to convert a KeyError into a ParseError for example | 06:09 |
| lalo | yes, been there... but in this case it has caused problems | 06:12 |
| lalo | do you think it should stay? if it stays, then I should make a list of all exceptions that could possibly be raised and catch only those | 06:13 |
| SteveA | another approach is that taken by page templates, using a __traceback_info__ variable | 06:13 |
| SteveA | that can contain extra information about where the error occured, in this case in the page template | 06:13 |
| SteveA | I agree that is should catch specific exceptions if it stays | 06:13 |
| lalo | yup, but that's not python standard, it's a zope extension and therefore won't be there if zopeless | 06:13 |
| SteveA | this is an adapter, and in rosetta | 06:14 |
| lalo | yes, and it runs zopeless a lot of the time :-) | 06:14 |
| lalo | it's the backbone of the importer script | 06:15 |
| carlos | lalo: I'm looking at your patch now | 06:15 |
| SteveA | yes, that doesn't stop importing the code that knows how to get the traceback info | 06:15 |
| lalo | hmm. could be done. | 06:16 |
| lalo | I could look into that - in zope2 it's not something you can simply import, it's a tangle of publisher exception handling hooking. If the zope3 equivalent is importable, I may use it. | 06:17 |
| spiv | Well, it's "just" a simple matter of grovelling through the locals of the frames that you find the traceback... ;) | 06:17 |
| stub | Whats wrong with DB-API exceptions? | 06:19 |
| SteveA | look in zope/exceptions/exceptionformatter.py | 06:19 |
| SteveA | that has classes which format exceptions, using __traceback_supplement__ as needed | 06:19 |
| lalo | ok | 06:20 |
| daf | SteveA: perhaps we should do a systematic code review? | 06:21 |
| carlos | daf: we need a "login" link | 06:21 |
| daf | carlos: we have one on the front apge | 06:21 |
| carlos | daf: oohh, I didn't know it O:-) | 06:21 |
| SteveA | daf: that would be a good thing | 06:22 |
| carlos | daf: but we should remove it if we are already logged in :-P | 06:22 |
| lalo | the whole exception handling department of pofile and pofile_adapters is quite klunky :-/ are you two ok if I take a few hours to refactor it as much as necessary? | 06:23 |
| SteveA | stub: iirc, there was a need for bare excepts because it was not clear exactly what errors the DBAPI could produce, and which of these were legitimate "database problem" errors | 06:23 |
| SteveA | daf: is that a good use of lalo's time right now, given the other tasks to do in rosetta? | 06:23 |
| lalo | right now I don't even give a reliable line number... the correct message would be "syntax error somewhere around line 520 or somesuch" | 06:24 |
| stub | All legitimate database errors should be subclasses of thedbmodule.DatabaseError or theconnectionobject.DatabaseError. Anything else is a bug or something that shouldn't be caught. | 06:26 |
| SteveA | I'm still trying to track down who was complaining to me about this | 06:27 |
| SteveA | then again, perhaps I misremember | 06:28 |
| SteveA | or it was a different API, not the DB API | 06:28 |
| carlos | lalo: ok, the export works, but with bugs | 06:28 |
| carlos | lalo: carlos@frodo ~ $ msgfmt --statistics -o /dev/null es.po | 06:28 |
| carlos | es.po:281:10: parse error | 06:28 |
| carlos | es.po:287:10: parse error | 06:28 |
| carlos | es.po:293:10: parse error | 06:28 |
| carlos | es.po:299:10: parse error | 06:28 |
| carlos | es.po:311:10: parse error | 06:28 |
| carlos | es.po:316:10: parse error | 06:28 |
| carlos | es.po:4437:10: parse error | 06:28 |
| carlos | es.po:4443:10: parse error | 06:28 |
| carlos | es.po:6151:10: parse error | 06:28 |
| lalo | interesting | 06:28 |
| carlos | msgfmt: found 9 fatal errors | 06:28 |
| carlos | lalo: I will attach the es.po to the bugreport | 06:29 |
| lalo | ok | 06:29 |
| carlos | I translated only one string | 06:29 |
| carlos | and nothing more | 06:29 |
| === SteveA pings daf | ||
| daf | SteveA: sorry, I was workraving | 06:30 |
| SteveA | I wonder if a method on a po message set that returns all sightings would help, or one that loads all sightings at once, in a single select | 06:30 |
| daf | that's a thought | 06:31 |
| SteveA | daf: no problem, just wanted you to have a checkpoint in the chatlog. I'd like you to make a decision about whether lalo should work on cleaning up the import code's use of exceptins, or whether other tasks have a greater need right now | 06:31 |
| daf | SteveA: lalo has one bug assigned to him (#1944) | 06:35 |
| daf | SteveA: there are a number of unassigned open bugs | 06:35 |
| SteveA | so, let's file a bug for "clean up exception handling in po parsing / import", come up with a tangible deliverable for it | 06:39 |
| SteveA | and then you and lalo decide which tasks should be next in his queue | 06:39 |
| daf | well, let's make a clear statement of what's problematic with the current exception handling | 06:40 |
| lalo | 1. it doesn't provide reliable, useful line numbers in all cases | 06:41 |
| lalo | 2. it masks some stuff it shouldn't mask, which sometimes makes it harder to debug rosetta code | 06:41 |
| daf | okay, can we put those two points in a bug report? | 06:44 |
| lalo | sure | 06:44 |
| daf | thanks | 06:44 |
| SteveA | thanks lalo, that's a very clear summary | 06:54 |
| lalo | thanks | 06:54 |
| daf | carlos: | 07:03 |
| daf | lalo: | 07:03 |
| daf | would now be a good time to have a team meeting for you? | 07:03 |
| carlos | daf: sure | 07:03 |
| lalo | I'd prefer tomorrow morning | 07:04 |
| lalo | my brain is already starting to shut down | 07:04 |
| daf | perhaps a very quick one now, and a more comprehensive one tomorrow morning? | 07:04 |
| carlos | ok | 07:04 |
| lalo | ok | 07:04 |
| daf | ok, let's set a time limit of 10 mintues | 07:05 |
| daf | so, we'll aim to finish by 17:15 UTC | 07:05 |
| daf | what have we been working on today? | 07:06 |
| daf | carlos: would you like to start? | 07:06 |
| carlos | daf: I was working on the indexes | 07:07 |
| carlos | doing some benchmarks | 07:07 |
| daf | ok, and you produced some proposed changes to the database as the result of that, right? | 07:07 |
| carlos | and trying different options | 07:07 |
| carlos | until I sent a proposal | 07:07 |
| carlos | yes | 07:07 |
| carlos | I'm working now | 07:07 |
| carlos | on a more evalorated report | 07:07 |
| carlos | as you and Steve asked | 07:07 |
| daf | great! | 07:08 |
| carlos | also I tested a patch from lalo | 07:08 |
| daf | for a bug? | 07:08 |
| carlos | and sent the impressions about it | 07:08 |
| carlos | daf: yes | 07:08 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| carlos | #1979 | 07:08 |
| daf | ok, good | 07:08 |
| daf | I've also been working on performance issues | 07:09 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has left #launchpad [] | ||
| daf | I did some profiling of the import code and posted the results to Bugzilla | 07:09 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| daf | I've also been making final preparations for the Rosetta Alpha launch | 07:09 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has left #launchpad [] | ||
| daf | doing the mailing list subscriptions | 07:09 |
| daf | writing the announcement email | 07:09 |
| daf | also, debugging problems with transactions | 07:10 |
| === stub [~stub@dsl-246.248.240.220.dsl.comindico.com.au] has joined #launchpad | ||
| daf | trying different patches on the development server | 07:10 |
| daf | lalo: how about you? | 07:10 |
| daf | (4 minutes left) | 07:11 |
| carlos | :-P | 07:11 |
| lalo | I was working on the plural form bug, what was its number again | 07:11 |
| lalo | 2017 | 07:11 |
| lalo | the fix for it "accidentally" fixed 1979 too | 07:12 |
| daf | nice :) | 07:12 |
| daf | this is good news, because it means we can export PO files from Rosetta | 07:12 |
| SteveA | when will rosetta alpha announcements be going out? | 07:12 |
| lalo | not really :-) carlos found another error | 07:12 |
| daf | SteveA: I've started writing the announcement | 07:13 |
| lalo | but the fix is simple | 07:13 |
| SteveA | yes, but... ? | 07:13 |
| SteveA | (that was for daf) | 07:13 |
| daf | lalo: and that error is in Bugzilla? | 07:13 |
| carlos | daf: yes | 07:13 |
| lalo | I'm filing it now | 07:13 |
| daf | SteveA: should it be reviewed before it's sent? | 07:13 |
| carlos | as an anex to the 1979 one | 07:13 |
| lalo | then looking at where Steve wanted those constants, and looking a bit at the exception handling | 07:13 |
| SteveA | daf: I didn't ask whether the announcements were being written | 07:13 |
| lalo | I'm pretty sure I did two other things today :-/ let me check my logs | 07:13 |
| daf | ok, before we finish, let's think of things we need to discuss tomorrow | 07:14 |
| daf | SteveA: the announcement can go out as soon as the announcement text has been finished | 07:14 |
| SteveA | ah, okay! | 07:14 |
| lalo | daf: 2022. Please decide if it blocks 1915, or 1965, or neither | 07:15 |
| daf | ok, time's up | 07:15 |
| SteveA | when is tomorrow's meeting? I want to put it in my diary | 07:15 |
| daf | how about half an hour before the Launchpad one? | 07:16 |
| daf | lalo: will do | 07:16 |
| carlos | lalo: I suppose we could move it to the beta | 07:16 |
| carlos | daf: hmm, perhaps it's better an hour before | 07:17 |
| lalo | it's a rather simple fix | 07:17 |
| carlos | that way we could handle any delay | 07:17 |
| lalo | about 15min work | 07:17 |
| daf | let's have a half-hour meeting before the Launchpad one | 07:17 |
| daf | if there's anything we need to discuss for longer, we can schedule another meeting later in the day | 07:18 |
| SteveA | let's allow a break between the meetings | 07:18 |
| daf | SteveA: good idea | 07:18 |
| carlos | lalo: we launch the alpha today or tomorrow, if you fixed it before the alpha release, great, if not, it makes no sense to block 1915 :-) | 07:18 |
| daf | let's say 45 minutes before the Launchpad one | 07:19 |
| carlos | daf: ok | 07:19 |
| daf | and allow a 15 minute break between meetings | 07:19 |
| daf | so: 11:15 UTC? | 07:19 |
| daf | no, the Launchpad meeting is at 11:30, right? | 07:19 |
| carlos | yes, 10:45 UTC | 07:20 |
| daf | great | 07:20 |
| daf | lalo: what's the estimated fix time for #2022? | 07:22 |
| lalo | daf: as I said, about 15m work | 07:23 |
| lalo | but I don't trust doing it now - I'm too tired, I'd probably introduce another bug ;-) | 07:23 |
| daf | let's pin it on #1915 since the timescale for fixing it is short | 07:24 |
| daf | i.e. tomorrow morning | 07:25 |
| === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has left #launchpad [] | ||
| carlos | So if all things goes well, tomorrow will be the "big" day :-) | 07:26 |
| lalo | carlos: what's the url to export a po? | 07:30 |
| carlos | lalo: it depends on the module | 07:31 |
| lalo | why the heck am I asking? :-P | 07:31 |
| carlos | http://localhost:8085/rosetta/gnome/gnome-applets/main-2.8/es/po | 07:31 |
| carlos | for example | 07:31 |
| lalo | ok, I seem to have fixed it | 07:57 |
| lalo | I'll go to bed, and when I wake up, I'll test the fix more thoroughly and merge it | 07:57 |
| lalo | good night all | 07:58 |
| lalo | carlos: I shouldn't even have asked - I was under the crackful assumption that the po export wasn't yet linked from the UI. It is not only linked, but rather easy to find. | 07:58 |
| carlos | lalo: :-) | 07:59 |
| carlos | lalo: night | 07:59 |
| lalo | for the record, I exported a po file, and passed it trough msgfmt successfully | 07:59 |
| carlos | lalo: I will review it later | 08:02 |
| lalo | hmm | 08:02 |
| lalo | prefs still seems to be kind of broken | 08:02 |
| lalo | I can remove languages, but not add | 08:08 |
| carlos | lalo: ? | 08:10 |
| carlos | lalo: ok, please don't add languages with countries (like pt_BR) | 08:10 |
| carlos | It's a know bug | 08:10 |
| lalo | ah | 08:10 |
| carlos | I need to fix the database info | 08:10 |
| carlos | or think about a way to improve that so it's automatically fixed with every exception raised. | 08:11 |
| lalo | interesting | 08:11 |
| lalo | hmm. I don't remember the sql syntax for update :-P | 08:20 |
| elmo | \h update in psql | 08:22 |
| lalo | oooh. thanks | 08:22 |
| lalo | daf: update language set pluralforms = 1, pluralexpression = '0' where englishname='Lojban'; | 08:23 |
| carlos | lalo: that language does not have plural forms? | 08:26 |
| lalo | no | 08:26 |
| carlos | ok | 08:26 |
| carlos | I will add it to our list | 08:26 |
| carlos | thanks :-D | 08:26 |
| lalo | np | 08:26 |
| lalo | I'm making a lojban translation of gnome-terminal as a futile exercise to get myself acquainted with the rosetta UI | 08:27 |
| daf | carlos: where are you adding new plural form data? | 08:45 |
| carlos | daf: to the file that feeds our script and also to the languages.sql file | 08:45 |
| carlos | scripts/plural-form-data | 08:46 |
| daf | great! | 08:46 |
| === lalo goes to bed | ||
| daf | lalo: night night :) | 08:49 |
| lalo | nite | 08:50 |
| daf | lalo: well spotted on #2023 | 08:50 |
| carlos | lalo: night | 08:52 |
| carlos | daf: any easy way to create a launchpad_test2 database with the current setup? | 08:57 |
| daf | make -C /home/daf/launchpad-test/database/schema DBNAME=launchpad_test2 | 09:10 |
| carlos | daf: I was playing with DBNAME=launchpad_test make | 09:10 |
| carlos | but it did not work | 09:10 |
| carlos | daf: thanks | 09:10 |
| daf | yeah | 09:11 |
| daf | you have to use -e to Make if you want environement variables to be applied | 09:11 |
| daf | but it's better to specify them on the command line | 09:11 |
| carlos | ok | 09:12 |
| === cprov [~cprov@200.158.100.251] has joined #launchpad | ||
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| === carlos [~carlos@67.Red-81-32-123.pooles.rima-tde.net] has joined #launchpad | ||
| === SteveA [~steve@adsl-213-190-44-43.takas.lt] has joined #launchpad | ||
| === elmo [~james@83.216.141.215] has joined #launchpad | ||
| === carlos [~carlos@67.Red-81-32-123.pooles.rima-tde.net] has joined #launchpad | ||
| === SteveA [~steve@adsl-213-190-44-43.takas.lt] has joined #launchpad | ||
| === carlos [~carlos@67.Red-81-32-123.pooles.rima-tde.net] has joined #launchpad | ||
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| === elmo [~james@83.216.141.215] has joined #launchpad | ||
| === lal1 [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| !dmwaters:*! Hi all, we are having problems with one of our main rotation servers. I'm going to rehub it and see if we can fix it a bit | 09:52 | |
| === carlos [~carlos@67.Red-81-32-123.pooles.rima-tde.net] has joined #launchpad | ||
| === lalo [~lalo@200-102-070-253.paemt7003.dsl.brasiltelecom.net.br] has joined #launchpad | ||
| === elmo [~james@83.216.141.215] has joined #launchpad | ||
| !alindeman:*! Hi all .. we're having a bit of trouble with one of our main rotation servers. We've pulled it from rotation and rehubbed a bit. We'll continue to monitor the situation. Sorry for any inconvenience and thanks for using freenode | 09:55 | |
| === carlos goes to have dinner | ||
| === sabdfl [~mark@host217-37-231-28.in-addr.btopenworld.com] has left #launchpad [] | ||
| === sabdfl [~mark@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!