[12:00] thanks a lot for your hard work recently [12:01] daf, if I have a variable inside a zpt called context/foo and lets supose it is a list. Is there a way to get in zpt the firt element without tal:repeate? [12:02] debonzi: I don't think so [12:02] debonzi: you have to resort to "python: context.foo[0] " [12:02] debonzi, Ok, thanks [12:02] debonzi: or otherwise add a method to the view class to get it [12:03] daf, Ill find a way.. thanks :) [12:03] de nada :) [12:08] daf: no problem. It's a pity that rosetta is not only yet :-( === elmo_ [~james@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [12:20] carlos: why does the personal information form submit to ViewPOFile? [12:20] daf: does it? [12:21] let me check, I don't remember it now [12:21] daf: it submits to TranslatorDashboard... [12:22] I implemented the code there [12:22] about why TranslatorDashboard... the template was pointing there, I did not changed it [12:25] daf: ? [12:26] class ViewPOFile: [12:26] ... [12:26] def editSubmit(self): [12:26] ... [12:26] that's not the personal information [12:26] that's for the plural forms editor [12:26] elif "SAVE-PERSONAL" in self.request.form: [12:26] carlos, GO SLEEP [12:27] kiko: I did all pending tasks, let me end the conversation :-) [12:27] kiko: he needs to answer this question first :) [12:27] and I will go to sleep [12:27] daf: if you have it indide the viewpo [12:28] you have a broken file [12:28] I have it inside TranslatorDashboard [12:28] ah :) [12:28] I think I have a broken file, then [12:28] I've split up TranslatorDashboard into TranslatorDashboard and ViewPreferences, by the way [12:29] so it seems something weird has happened [12:29] daf: Sorry, back... use Foo.set(field1=value1, field2=value2) to clump UPDATEs into a single query, rather than one per field. [12:29] (I think I mention this in the SQLObjectGuide?) [12:29] daf: yes :-) [12:29] spiv: aha, thanks [12:29] do you want the original code? [12:29] I think I can work it out using a launchpad checkout === carlos adds that to the bug report and promises kiko that goes to the bed [12:30] ok [12:30] debonzi: You might be able to do context/foo/0 === spiv catches up on scrollback :) [12:31] spiv: nope [12:31] spiv: it's equivalent to context.foo["0"] [12:31] Oh, right. [12:31] spiv: (sadly) [12:31] good. [12:33] ok, now it's time [12:33] good night!!! [12:34] daf: do we have a meeting tomorrow? [12:34] carlos: yes [12:34] 12:00UTC? [12:34] yeah [12:34] ok [12:35] will you be able to make it? [12:35] sure [12:35] I'm willing to make an exception in this case [12:35] I'm not able to sleep more than 10 hours at the same time [12:35] ok :) [12:35] see you === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad [12:42] sleeping sucks anyway. [12:43] haha! [12:43] To: daf@muse.19inch.net [12:43] Name: KDE [12:43] Description: [12:43] http://www.kde.org [12:44] (somebody requested that KDE be imported into Rosetta) [12:44] I suspect it was Dwayne :) [12:45] spiv: I think Carlos was really tired -- he mistook you for Stewart (https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=1973) [12:46] Hah. [12:46] he was up all of yesterday, all last night and all of today [12:47] as I say, sleep is overrated? [12:48] :) === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad === debonzi see u all === lifeless_ is now known as lifeless === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad === npmccallum [~npmccallu@69-162-252-7.ironoh.adelphia.net] has joined #launchpad === stub [~zen@c211-28-34-252.sunsh1.vic.optusnet.com.au] has joined #launchpad === doko [doko@dsl-082-083-141-180.arcor-ip.net] has joined #launchpad === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad === limi [~limi@159.80-202-72.nextgentel.com] has joined #launchpad [11:41] limi: morning! === stub [~zen@dialup-33.53.194.203.acc03-dryb-mel.comindico.com.au] has joined #launchpad [11:52] lulu: morning :) [11:56] saw that you felt in the deployment there are things you feel are outstanding. could you mail me the problems you found. If there are any further config changes we need done, I can get Upfront to do them. [11:56] spiv, elmo: any progress on getting the xml-rpc server running on rosetta.w.h.c ? === carlos [~carlos@69.Red-80-33-181.pooles.rima-tde.net] has joined #launchpad [11:58] morning [11:59] lulu: nothing is set up for deployment at all on the caching side [11:59] everything was set to not cache etc [12:00] yes - Upfront were not asked to do that. There was a decision taken that Squid would be used for cacheing for Launchpad tools and the webserver. In the meantime we are using Apache. Thanks for your help. [12:02] Plone still needs to be tweaked before deployment, nop matter what system you use [12:02] I think it is not sufficient to just turn on squid. You need to do some configuration in plone to allow squid to cache certain things. [12:06] limi, lulu: Will you add the announcement mail into the news section of www.canonical.com? [12:06] some people asks me for a link to the announcement so they can publish it in their news website [12:06] carlos: will do [12:07] ok [12:07] they can link to the wiki, I think [12:07] daf: is the announcement there? [12:07] yes, I think so [12:08] ok [12:08] http://wiki.ubuntulinux.org/WartyWarthog_2fAnnouncementEmail [12:08] daf: thank you [12:08] I'm going to have breakfast now and back to work :-) [12:10] eek, more _2f madness [12:11] :) === limi loves the irony of trying to use Wiki like a file system [12:14] daf: thanks- no point in having it in 2 places - need to consolidate what's on the wiki and what will be on web. [12:15] lulu: yep [12:15] lulu: the website looks great, by the way -- congratulations :) [12:19] daf: thanks Daf - thanks to everyone's hard work. Needs a slick design and loads more content but that will come! :o) [12:19] daf: looking forward to having the Rosetta tan up there! [12:20] tab! [12:23] me too! [12:28] daf: howzit going? [12:30] we're feature-complete for the Alpha release [12:30] we'll do a review of the website today [12:31] then, once we've imported some data, we're done [12:38] daf: do you need some of Limi's magic on the UI? [12:40] lulu: yeah, we have some bugs filed about that [12:41] https://bugzilla.warthogs.hbd.com/bugzilla/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailcc2=1&emailtype2=substring&email2=limi&b [12:41] (apologies for the horrible URL) [12:42] daf: that's ok! [12:45] wow, that's crazy :) === carlos checking the daf's patch [12:52] basically #1949 and #1950 are pretty simple UI enhancements [12:52] #1934 is more of a comprehensive UI review [12:53] good start for today. My archive is broken [12:53] grrrrrr [01:02] carlos: any idea why scripts/poimpot.py instantiates TemplateImporter/POFileImporter with a None person? [01:03] daf: I fixed it yesterday before leaving [01:03] no idea why it was that way, but it's wrong and breaks the import [01:03] daf: you should have the patch at rocketfuel [01:04] oh, ok [01:04] I thought I was up to date [01:04] I fixed it locally [01:04] it's an easy change [01:06] It's me or chinstrap connection is slooooow [01:06] ? [01:07] I don't think it's you [01:07] I couldn't even connect [01:07] looks like they were discussing it on #canonical [01:07] it means I can't log in to rosetta [01:09] ok [01:14] daf: elmo can log in - just a little slow [01:14] daf: can you log in now? === daf tries [01:15] yep, it's working now [01:15] daf: good - thanks [01:23] daf: your patch works [01:23] we have only an UPDATE instead of 4 [01:23] I thought it was 6 :) [01:23] but it's still slow (as you already know) [01:23] right, I feared as much :( [01:24] daf: well, It was yesterday when I detected it, I don't remember the exact number :-P [01:24] I was guessing it [01:25] daf: I don't think it's the database the bottleneck [01:25] as we know now :-) [01:26] I'm going to execute the import of the .po/.pot files now that lalo has fixed a bug I detected yesterday and at the same time I will try to profile the script [01:27] daf: or do you prefer to wait for the meeting in 30 minutes before start the profiling? [01:28] no, I don't think we need to wait [01:28] I think the bottleneck might still be in the database [01:28] perhaps the SELECTs we're executing are slow [01:28] I think Postgres has some tools for optimising queries [01:29] daf: the selects are too restrictive [01:29] well, forget that [01:29] if it's not an index [01:29] if it's not using an index it could be slow [01:31] is it using an index? [01:31] daf: btw... I'm executing the same queries by hand from psql and they are really fast [01:31] daf: no, that's why I told you that forget my words :-) [01:32] btw, the queries are fast, and we don't have lots of queries for every msgset [01:33] you think it is the code then? [01:33] yes [01:33] I suspect that === carlos in profile mode [01:34] daf: could you commit your patch? [01:34] althought it's still slow, I think it's better if we do it that way [01:38] I'm in the middle of something right now [01:38] feel free to apply and commit it yourself [01:39] ok === ddaa [~david@nemesis.xlii.org] has joined #launchpad === debonzi [~debonzi@200.158.100.251] has joined #launchpad === cprov [~cprov@200.158.100.251] has joined #launchpad [02:02] It is quite possible the database is a bottleneck - there has been no optimization done atm. If you identify columns that need indexes, let me know. [02:05] yes, I was wrong, top let's you see that postgres has more than a 80% of my CPU [02:10] lifeless, stub: I sent yesterday two changes to the DB schema, could you review them? [02:14] someone added patch-1-08 and conflicted with me :[ [02:14] I thought only the nazi was meant to add patches to the schema ? [02:14] lifeless: you are true [02:14] That was me, not letting go ;-) [02:14] hnmmm [02:14] lifeless: you are right :-P === lifeless larts stub === lifeless larts stub === lifeless larts stub === lifeless larts stub === lifeless is done now [02:15] The larting? I hope so === stub bleeds [02:15] :} [02:17] daf: meeting? [02:17] daf: or are we waiting for lalo? [02:19] carlos: I've no idea whether lalo will turn up [02:19] SteveA: around? [02:20] lifeless: is that URI syntax correct? [02:20] daf what? [02:20] ok, then if you want we could do the meeting. [02:20] lifeless: see my last email to the launchpad list [02:20] dunno. [02:20] daf: If you get it working, please make a canonical.lp.connect method - I don't want to remember the syntax if I can help it :-) [02:21] daf: The syntax is an SQLObject thing. [02:21] stub: well, dbhost is '' at the moment, so it will work [02:21] daf: not in production its not. [02:21] thats why dbhost is there. [02:21] lifeless: exactly [02:21] is //host/stuff [02:21] that's why I'm asking :) [02:21] spiv: is it postgres://host/dbname? [02:22] daf: Don't know :) [02:22] stub: how do we identify columns that need indexed? [02:22] * indexes [02:24] daf: the ones that are used often [02:24] to access data [02:25] how do we identify those? [02:25] looking at Postgres logs? [02:26] daf: no, looking at the queries [02:26] the WHERE clause [02:26] or looking at the logs if you prefer [02:26] the selectBy and things like that [02:26] from the sqlobjects [02:26] right [02:27] well, how often they're referred to in the code and how often they are queried in the database are two different things [02:28] daf: true, but the import script (for instance) is executed many times [02:28] sure [02:28] so we can improve those first, rosetta does not have problems (yet) [02:29] stub: a unique key is also an index right? [02:29] Yup [02:30] so we have already some "optimizations" [02:30] and it's still slow [02:30] daf: If a column appears in a WHERE clause (or in a SQLObject.select keyword argument) it probably needs an index [02:31] postgress://dbhost/dbname [02:31] lifeless: thanks [02:31] lifeless: where did you find it? [02:32] I figured it out when we setup launchpad in production on macquarie [02:33] daf: If you turn on postgresql logging of SQL commands and watch the file, it is often obvious what queries are chewing up the time [02:33] stub: ok, thanks -- we'll make a list and let you know [02:34] Oh..... indexes won't be used after you import a load of data until you ANALYZE. This might be the problem. [02:35] (cause PostgreSQL thinks the tables have only a few rows and it isn't worth using them - analyze updates these stats) [02:35] stub: yes, that could be the problem [02:35] stub: any way to improve it? [02:35] psql -d launchpad_test -c vacuum analyze [02:35] psql -d launchpad_test -c 'vacuum analyze' [02:36] stub: will it work while the database is being used (adding more rows?) [02:36] Yes [02:36] that will fix a part of the problem [02:36] Just slows things down a bit while it is running :-) [02:37] stub: any way to do it automatically or every X inserts? [02:37] because we are importing about 1000 or 2000 new rows [02:38] and we lookup that table with every addition [02:38] You generally run it from cron, or after you have made significant changes (% wise, not number of rows type things) [02:38] If you have 10,000 rows and add 1000, it won't make any difference. If you have 0 and add 1000, it will [02:39] stub: the field we use is a string and it's a unique key and should be indexed because the WHERE clause ask for it [02:39] so VACUUM ANALYZE will lead to an immediate improvement, or we need to define indexes and VACUUM ANALYZE? [02:39] If you know the query that is slow (eg. select foo from bar), run 'explain select foo from bar' in psql. You get some confusing output, but if you see Seq Scan it means it is slow (full table scan). [02:40] daf: the VACUUM sentence is to update the index [02:40] stub: ok [02:41] daf: In this case, probably immediate improvements since the indexes are likely already there (primary keys etc). [02:42] Please don't put any ANALYZE commands in your scripts, unless they are scripts that load sample data for development btw - prod will do this via cron [02:42] ok [02:43] stub: what happens if it's a big transaction with lots of inserts and queries against those previous inserts? [02:45] we are talking about 1500 INSERTS or UPDATES [02:45] carlos: On production, it probably won't make any difference unless you are changing a significant % of the rows in the table (say - 30%). [02:46] no, with every new import, the proportion will be smaller [02:47] So does the vacuum analyze speed things up in this case? [02:48] no, it's still slow [02:48] perhaps it helped [02:48] Do you know the query that is running slow? [02:48] but I'm still importing the same .po file with 1500 strings [02:48] stub: no [02:48] carlos: did you time it? [02:49] carlos: "time python poimport.py"... [02:49] daf: no, but I think it started about 45 minutes ago [02:49] ouch! [02:49] I thought we were talking seconds... not minutes! [02:50] I was thinking more along the lines of importing one file twice, once before the analyze, once after [02:50] stub: even hours :-) === carlos laughts because don't want to cry [02:51] something else is really screwy. Look at your postgresql log file to see what queries are being run [02:52] (preferably after turning that logging feature on...) [02:53] stub: I did and the queries executed by hand were fast === carlos testing a small .po file [02:53] real 0m44.759s [02:53] user 0m5.635s [02:53] sys 0m0.335s [02:54] a .po with 77 messages [02:54] Still worth a 'tail -f' on the log during a run - might just be qualtity of queries [02:56] and a reimport test of the same .po file: [02:56] real 1m26.643s [02:56] user 0m5.238s [02:56] sys 0m0.334s [02:57] stub: I will test the import of a .po file with only a msgid so I get all queries we do to import it [03:01] hmm [03:01] 2004-09-16 15:01:09 [6093] LOG: unexpected EOF on client connection [03:01] is it normal after the script ends? [03:01] should we execute anything to close the connection? [03:02] conection.close() is nice [03:03] carlos: is this with poimport.py [03:03] ? [03:04] daf: ye [03:04] yes [03:04] hmm, that should be closing the transaction === daf will be out for 10-15 minutes [03:07] carlos: shall we have the meeting at 13:30? [03:07] daf: could we move it for later then? [03:07] daf: I will have lunch at that time [03:07] 14:00? [03:08] I have a meeting with the Soyuz guys then [03:08] perhaps we could do both in parallel [03:08] daf: we could delay it more if you want [03:09] daf: I have work to do [03:09] so I'm not blocked for the meeting [03:09] ok [03:09] carlos: in future, can you attach big files to Bugzilla rather than pasting them? :) [03:09] ok [03:09] O:-) [03:09] ok, see you later :) [03:10] later === lulu [~lu@host217-37-231-28.in-addr.btopenworld.com] has joined #launchpad [03:50] daf: meeting in ten minutes ? [03:50] cprov: hi [03:51] cprov: might have to delay by 10 minutes or so [03:51] hi daf [03:52] hi Steve [03:52] so, some things are running really slowly in rosetta? [03:53] yes [03:53] specifically, the import [03:53] on my laptop, about 1.5 minutes to import a 512-message set PO file [03:53] the server is slow also [03:54] daf: no problem, I and debonzi are going to kiko's place, 10 minutes will be enough, see you later :), tks [03:54] we have an open bug about it, which has some discussion about the problem [03:54] cprov: ok [03:55] https://bugzilla.warthogs.hbd.com/bugzilla/show_bug.cgi?id=1973 === kiko [~kiko@200-206-134-238.async.com.br] has joined #launchpad [03:55] are we launching? [03:56] (or are we keeping radio silence) [03:57] kiko: I'm going to be a bit late [03:58] daf: you heartless man [03:59] daf, see u in some minutes [03:59] kiko, 10 min I'll be there === debonzi running [04:11] daf: 1.5 minutes for a 512-message set PO file is not too much time [04:11] we should improve it, yes, but it's not bad [04:12] for me it takes much more time [04:17] carlos: what is the usual range of sizes for a po file? === cprov [~cprov@200-206-134-238.async.com.br] has joined #launchpad === debonzi [~debonzi@200-206-134-238.async.com.br] has joined #launchpad [04:18] SteveA: I think we could say between 1500 - 2000 for non trivial applications [04:19] wait [04:19] SteveA: http://l10n-status.gnome.org/gnome-2.8/es/desktop/index.html [04:19] so [04:19] are we on? [04:20] seems like most gnome applications have less than 500 strings [04:20] but we have some with > 1500 and evolution with > 4000 [04:21] if it's a text or trivial graphical application < 500 is the normal, a complex graphical application > 1500 [04:23] daf: just say when you are ready ... [04:23] ok, back [04:24] ah, good. [04:24] carlos: but if you have 30 languages, then importing that one product takes 45 minutes [04:24] daf: that's perfect, I'm taking 45 minutes for only one .po file!!1 [04:25] carlos: really?! [04:25] daf: I'm going to do real test with time now [04:25] but I think soi [04:25] carlos: which PO file? [04:25] gnome-applets [04:25] kiko: ready when you are [04:26] so [04:26] cprov: can you paste in some urls for daf to check out where we want to fit the data? [04:26] kiko: yep, one minute [04:26] daf: some open questions first. [04:27] daf: first, can you update https://rosetta... [04:27] daf: do translations for a certain package have a "lifecycle"? [04:27] daf: the DB too, please :) [04:27] daf: i.e. do they start as open and then go through a review and closing process? [04:28] kiko: they have a lifecycle [04:28] daf: what states? [04:28] kiko: https://www.warthogs.hbd.com/RosettaTranslationStates [04:29] daf: aha [04:30] okay. [04:30] daf: what do you think makes sense to display from the soyuz side of things, related to: [04:30] - source packages [04:30] - people [04:31] - distribution [04:31] - distribution release [04:31] (assume distro and distro release are just a group of packages) [04:33] hmm [04:34] well, for source packages, distributions and releases, you could show translation statistics for a language [04:34] you could show a pan-language translation statistic, but I don't think that's very useful [04:34] for a person, not sure [04:34] what sort of information about people does Soyuz currently show? [04:35] cprov: server should be up now [04:35] daf: can we get this sort of statistic ready-made from rosetta? [04:35] kiko: there are methods for statistics in th Rosetta interfaces [04:35] kiko: but you will need to map source packages to projects [04:35] daf: also, is it interesting to show "open" or "pending" translations for a specific source packages? [04:35] daf: how would that be doable? [04:36] what do you mean by "open" and "pending"? [04:36] daf: just have a look, at first, on ...soyuz/distros/ubuntu [04:36] I'm not sure -- perhaps something that indicates quickly that a certain package is "in need of" translations in certain languages. [04:37] daf: I'm thinking of something that will entice the soyuz user to hop into rosetta. [04:38] daf: It is the Distribution Main Page, we want to show, by Translation link (+translation) a set of translations informations . what do you think we can show ? [04:38] cprov: I see it [04:38] kiko: hmm [04:39] kiko: We can map SourcePackages to Products via the Packaging table, btw. [04:39] daf: so, do you have nice ideas ? [04:40] if you have information about what languages a user speaks, then you can give them status of that package in those languages [04:40] spiv: ah, that is interesting indeed. [04:41] spiv: is that an easy query? In reality I want to get from a sourcepackage to a project. [04:41] daf: neat idea. but the user doesn't specify anywhere what language he speaks [04:42] daf: given a source package, for instance, can we provide a direct link that says "translate this package (using rosetta)"? [04:42] or help translate this package. [04:42] kiko: we have it from rosetta [04:42] kiko: Rosetta stores information about a user's languages [04:42] kiko: and you can quite easily get that information [04:42] daf: ah, cool. but is the user the same user as a soyuz user? : [04:42] kiko: I think so: select Project.id from Project where Project.id = Product.project and Product.id = Packaging.product and Packaging.sourcepackage = %d; [04:43] daf: https://rosetta.warthogs.hbd.com/rosetta/prefs [04:43] kiko: yes [04:43] spiv: and with the project ID we can grab translation data, I assume? [04:43] daf: cool. jotted down. [04:43] Ask daf :) [04:43]