dpmgood morning all08:04
TLEwhois mdke12:28
TLEwhoops, juts didn't remember the name ;)12:28
dpmhi TLE12:33
TLEI'm writing the todo list for tomorrow12:34
dpmah, cool12:34
dpmI'm progressing on the database import performance improvements for the translation stats. I've thrown away sqlite and I'm using postgresql and raw sql queries instead of django. Postgresql has a nice COPY command especially designed to import data from CSV files (well, json with a bit of massaging in my case)12:35
dpmthat reduced the import from hours to just a few minutes, which is rather nice :)12:36
dpmnow I'm fighting with SQL, as I need to do another additional INSERT query after the import, which is either taking a bit long (on django 1.3) or failing (on django 1.1)12:37
dpmbut that's probably me needing to learn SQL12:37
TLEahh, sounds like good progress12:37
dpmyeah, but I wanted to have it ready before the UGJ this weekend, and I'm not sure I'll have the time :/12:38
dpmI'll figure it out, I'll just have to find a postgresql or sql expert to give me a hand :)12:39
TLEwhen you say manual INSERT, you mean directly with and not via Django?12:41
dpmTLE, exactly, I'm doing raw sql inserts, which really boosts performance in this particular case12:59
TLEbut then it must be a python sql thing that is failing and not django right13:00
dpmyeah, psycopg213:00
dpmthey are the underlying python DB bindings Django is using for Postgresql access13:01
TLEahhhh that away13:01
dpmthe other query below, on line 256 is the one that's failing on django 1.1 (or rather in whichever psycopg2 version django 1.1 uses). You can also see the original Django code commented out above13:04
dpmI still haven't figured out why, but it returns duplicate rows when it shouldn't13:05
TLEhmm, I can't spot anything, there is an extra , after the columns in 256, but I don't think it matters13:12
TLEwell, better find an apropriate SQL expert13:13
dpmyeah, but thanks for looking anyway!13:34

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!