/srv/irclogs.ubuntu.com/2006/09/20/#launchpad-meeting.txt

=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting
=== carlos [n=carlos@138.Red-81-39-35.dynamicIP.rima-tde.net] has joined #launchpad-meeting
=== danilos [n=danilo@cable-89-216-150-63.dynamic.sbb.co.yu] has joined #launchpad-meeting
carlosSteveA: please, ping us when you are ready03:11
carlosif you are busy to have the meeting now, tell us it too so we can have another meeting that we have pending03:13
carlosplease03:13
SteveAcarlos: hi03:14
SteveAI'm here03:14
carloshi03:14
SteveAtwo things03:14
SteveA1. I want to understand better the timestamp and manually entered data thing03:14
SteveA2. about the view refactoring, one point that came out of discussion with kiko and mark is that it is something to do if it will be a small job, and will improve things03:15
SteveAdon't spend time on it before 1.0 if it is a larger thing03:15
SteveAit isn't a goal in itself03:15
SteveAit is an idea tha tmight help work on TranslationReview03:15
SteveAso, dealing with (2) first03:16
SteveAwhat do you think?03:16
carloswell, the thing is that I could implement TranslationReview with current views, but adding some hacks to see, for instance, the copy button that the user clicked to copy a message in the text area to modify it03:17
carlosI don't think it would be much more ugly than what we have right now03:18
carlosso if you prefer to leave it for later, I can do it, but taking into account that on review time03:19
SteveAmaybe you should guesstimate it?03:19
SteveAhow long to do the refactoring in days?  how long to do TranslationReview after the refactoring?  how long to do TranslationReview before the refactoring?  how long to do the refactoring after TranslationReview?03:20
carlosbased on what kiko told me or what danilo and I talk ?03:20
SteveAtell me what you think03:21
SteveAafter all, you'll be doing the work03:21
carloswell, I would need to think about it before I can give you an estimation03:22
carlosI know the idea behind kiko's suggestion, but I didn't think about it yet in depth because the pending meetings that would change a bit the solution03:22
danilos(just as a sidenote, this is exactly the reasoning why I want to work on generalized TranslationImport stuff: it will take a 2-4 days to implement it, but adding OOo, KDE PO would be much shorter and cleaner)03:22
carlosI could give you that info at the end of today03:22
carlosis that ok?03:23
SteveAyes, that's fine.03:24
SteveAthat will give you some idea of whether to do this first or later03:24
carlosdanilos: well, in that case you are preventing the hack, we already have that hack in production, so it makes mucho more sense in your case03:25
SteveAso, about the exports and timestamps03:25
carloss/mucho/much/03:25
carlosok03:25
daniloscarlos: I know, just giving the reasoning behind it03:25
carlosSteveA: let me tell you what we have atm and how does it work, ok?03:25
SteveAwhat I understand is that there is this process that involves producing exports03:26
SteveAand when pitti says one is good enough, that latest one becomes the baseline03:26
carlosyeah03:26
SteveAwhere does the manually entered timestamp come into it?03:26
carlospitti tells me the timestamp of the tarball used for the baseline03:27
carlosand I ask Stuart to store it in our database03:27
SteveAis that the same data that is the most recent baseline export?03:27
carlosnot always03:28
SteveAthe tarball is made from the most recent baseline export?03:28
SteveAwhy might it not be?03:28
carlosbecause there is a small delay03:28
SteveAa small delay between what and what?03:28
carlosbetween when pitti gets a tarball, creates the .deb packages and notify me the timestamp03:28
carlosand we create a new export every day03:28
SteveAthe timestamp is based on data in the database03:28
SteveAit is purely within the database's data03:29
carlosno, the timestamp is based on the date when the export was done03:29
SteveAwhat happens to that data -- making a tarball or a deb -- doesn't affect the timestamp of when the export was produced03:29
carlosoh03:29
carlosI see what you mean03:29
carlossort of03:29
SteveAso, I cannot see a reason to put a timestamp *into* the database03:30
carlosit would work that way if our exports come from production 03:30
SteveAit is something that should only ever come out of the database03:30
carlosbut it's not the case03:30
SteveAwhat I can see going into the database is03:30
SteveAmarking which export is the one that is used03:30
carloswe use a read only mirror03:30
carloswe don't store the list of exports in our database03:30
SteveAoh03:30
carlosour datamodel doesn't allow it03:31
carlosis not like Ubuntu packages, we don't need that complexity03:31
SteveAok03:32
SteveAso, the way I'd approach that is03:32
SteveAin the exported data, add a timestamp + checksum of timestamp03:32
SteveAso there cannot be a typo03:32
SteveAbut, I'm more inclined to connect to the real database03:34
carlosme too03:34
SteveAand store that an export was produced03:34
SteveAand store the date of the latest translation used, or whatever is appropriate03:34
SteveAand give that an export-id03:34
SteveAso pitti can just say "this export-id is the baseline"03:34
carlosI guess we could use the link to the librarian as that 'export-id'03:35
carlosso we could have it for free adding a link to latest baseline langpack03:36
carlosSteveA: also, I was thinking on use the timestamp for latest translation in that export as the timestamp for the language pack so this problem wouldn't happen again03:37
SteveAright03:37
SteveAplease file a bug or two on these issues03:38
carlosthere is still another issue03:38
SteveAwhat's that?03:38
carlosas we generate daily language packs03:38
carloswe need to provide Martin Pitti a way to go to launchpad and note himself which language packs has been used as the base package03:39
SteveAhow does pitti get a particular language pack?03:40
SteveAdoes he get an email?03:40
SteveAor go to a page in launchpad?03:40
carlosatm, he fetchs it from people.ubuntu.com03:40
carlosonce we move to production, he will get an email03:40
=== danilo_ [n=danilo@cable-89-216-150-89.dynamic.sbb.co.yu] has joined #launchpad-meeting
carloswith a link to librarian03:40
SteveAhow does it get to people.ubuntu.com?03:40
carlosmy script in carbon pushes the tarball there once built03:41
SteveAI see03:41
SteveAso, if we had a database table for langpacks produced03:41
SteveAhe could see it in a UI03:41
SteveAand mark in that same UI if one has been used as the baseline for what03:41
carlosalthough Martin asked us for a fixed URL in librarian so he doesn't need to parse the email03:41
carlosSteveA: right03:41
SteveAor we could offer him xmlrpc 03:41
carlosI think that would be the right approach03:42
SteveAif he wants to automate it03:42
carlosyeah, that would be also a good way to do it03:42
SteveAok03:42
SteveAthat seems like a small spec to me03:42
SteveAcouple of paragraphs explaining the background and proposed solution03:42
SteveAso we can schedule it for post 1.003:42
carlosok03:43
carlosalso, I think that's the only missing part to move language packs to production03:43
SteveAok03:44
SteveAso we talked about...03:44
SteveA - having the export script write to the db03:44
SteveA  maybe in a special table03:44
SteveAor maybe using the librarian id03:44
carloswell, we still need a table03:44
SteveAso that there is a database-produced unique ID for a langpack that is generated03:45
carlosit's a one to many relation03:45
SteveAso, use the 'id' in the table rather than the librarian id perhaps03:45
carlosone distrorelease has n language pack exports03:45
carlosok03:45
SteveAok03:46
SteveAand then we want to record there whether it is used as the baseline03:46
SteveAor a baseline03:46
SteveAand the timestamp of the most recent translation03:46
carlosok03:46
carlosalso we have another kind of language packs, the ones that only have updates...03:46
carlosbut I don't think I should bore you with that03:47
carlosI will note that in the spec03:47
SteveAI know that they exist03:47
SteveAand the script that produces them can use this data to know what range of translations to include03:47
carlosright03:47
SteveAand then we also talked about a UI + maybe xmlrpc for pitti03:48
carlosalso, I'm thinking on adding something to launchpad that allows pitti to select whether we are going to generate updates or full exports03:48
SteveAto get the langpacks, see what langpacks are available, and mark the one he chooses as a baseline03:48
SteveAok03:48
carlosso he can decide to do a new full export03:48
SteveAso, this is something to discuss in person with pitti perhaps?03:48
SteveAall these things together03:49
carlosto reduce the packages with just updates (it already happened with dapper point release, but he had to do it manually)03:49
carlosI guess, let's try first phone call after we have a draft03:49
carlosand see whether that's needed03:50
carlosanyway, if it's post 1.0, we could talk about it in the allhands meeting03:50
SteveAwell, when are you moving the langpack production into production?03:51
carlosonce we have this new spec implemented03:51
SteveAok03:51
SteveAadn that's not a 1.0 goal03:52
SteveAas far as I'm aware03:52
carlosright, it's post 1.003:52
SteveAok, then I think that's settled03:52
SteveAwhat do you think danilos ?03:52
danilosI'm fine with it03:52
SteveAok, great.03:52
SteveAthanks for having this meeting.03:53
danilosand it has just moved to carbon03:53
danilosso performance should not be an issue in the near future03:53
SteveAwe must just be careful about those timestamps03:53
carloswell, the performance issues were already fixed even before moving it to carbon03:53
SteveAuntil using a better system03:53
SteveAmaybe add a checksum to the timestamp as an interim measure... ?03:53
SteveAor ensure that the mail sent to set it in the db03:54
carlosSteveA: don't worry, what I will try to do is to set as the timestamp latest modified translation03:54
SteveAis sent to pitti too03:54
carlosSteveA: that way we solve the mirror problem03:54
SteveAcarlos:  you still need to store that somewhere03:54
carlosSteveA: inside the tarball exported03:54
SteveAso, there's still a timestamp coming out of the database03:54
SteveAacross to pitti03:54
carloswe already have a timestamp.txt03:54
SteveAthen back to you03:54
SteveAand back into the database03:54
carlosright03:54
SteveAand that is error-prone03:55
carlosI will add also the checksum as you suggested to be completely sure that nothing was lost03:55
SteveAso, consider this03:55
carlosthose are more or less trivial tasks03:55
SteveAadd a checksum to timestamp.txt03:55
SteveAand write a small script to read a timestamp.txt, check checksum and set it in the databaes03:56
SteveAif that will take under 2 hrs, then I'd say do that03:56
SteveAif more, then it is too much work03:56
carloshmmm, why should I do latest part?03:56
SteveAwhat does that mean?03:56
carlosI still need to ask stuart to do the update sentence03:56
carlosI don't have direct access to production database03:56
SteveAyou can tell stuart to run that script03:56
SteveAor you can send stuart the timestamp.txt03:57
SteveAand tell stuart to run the script on it03:57
carlosI see03:57
carlosok03:57
SteveAthe point is to avoid having someone typing in a manual query03:57
carlosso we reduce the chance to introduce a typo03:57
SteveAcopied from some email or text file03:57
SteveAbut, it is worth doing this only if it will be quick03:57
carlosgood plan03:57
SteveAdon't bother if it will be more than 2 hours03:57
carlosok03:57
carlosI think it would fit in a 2 hours slot03:58
SteveAbecause it is just an interim thing03:58
SteveAuntil the better system is designed and implemented03:58
carlosso pitti needs to send me the timestamp.txt file and that's all03:58
SteveAyes03:58
carlosok03:58
SteveA>>> md5.new('timestamp').hexdigest()03:59
SteveA'd7e6d55ba379a13d08c25d15faf2a23b'03:59
carlosSteveA: yeah, I already used md5 module while debugging this problem04:00
carlosto know which files change between current language packs04:00
carlosand a full export I forced04:00
carlosso don't worry04:00
SteveAsomething like that04:01
SteveAdsfok, great04:01
SteveAdsfok, gre04:01
SteveAum04:01
SteveAlag04:01
SteveAgreat04:01
carloslag + garbage :-P04:03
carlosSteveA: btw, now that we talk about this04:04
carlosthere is another issue we should fix04:04
SteveAwhat's that?04:04
carlosrelated with Rosetta and dapper04:04
carlosand perhaps breezy04:04
carloshttps://launchpad.net/bugs/5822104:04
carlosSteveA: pkgstriptranslations was stripping and feeding Rosetta with translations04:05
carlosfor packages in the backports pocket04:05
carlosdanilos: please, pay attention too, because we need to solve this issue04:05
daniloscarlos: no problem, I am ;)04:06
carlosthat means that we have some .pot files imported in dapper04:06
carlosthat doesn't match with what dapper has in its official release04:06
SteveAdoes this continue to happen?04:06
SteveAor is it just that we have some bad data now?04:07
carlosno, 30 minutes ago, the buildds for backports has been fixed04:07
=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting
carlosbut we have some bad data 04:07
SteveAno to what?04:07
carlosthat we need to fix04:07
SteveAyou're saying that the problem is fixed04:07
SteveAbut the bad data remains?04:07
carlosright04:07
SteveAdoes rosetta get told the pocket by the buildds?04:08
carlosWell, I think we can know that from our datamodel 04:08
carlosand is a protection we should implement to prevent such breakages in the future04:08
SteveAright, please file a bug on that04:09
SteveAor, add a task to that bug04:09
carlosI did already04:09
carloslet me look for it04:09
SteveAnow, about the bad data.  what do we need to do?04:09
carloshttps://launchpad.net/products/rosetta/+bug/5822304:10
carlosI think that we need to get the right .pot files and reimport them again04:10
carlosthat should be enough04:10
SteveAhow do you identify what pot files are needed?04:11
carlosbecause the .po imported didn't change anything in Rosetta other than what came from upstream, so nothing was broken from the Ubuntu translator point of view (we just added some new translations from upstream)04:11
carloswell, that's the problem I don't know exactly how to solve04:12
carlosI was thinking on ask for a list of packages in the backports pocket04:12
carlosand filter out the ones without translations04:12
carlosso I get that list04:12
carlosbut I think this would be a manual process04:12
carlosin the other hand, I guess the amount of packages should be low because Dapper was released only 3 months ago04:13
SteveAwell...04:14
SteveAall those packages will also need to be rebuilt04:14
SteveAwith non-stripped translations04:14
carlosbreezy would be more complicated, but the amount of templates imported is quite low than dapper so I don't think the problem would be too bad (we didn't get a full import for Hoary or Breezy)04:14
carlosSteveA: right04:14
carlosbecause they are without translations atm04:15
SteveAare there buildd logs we can use?04:15
carlosI guess, but I could check with Martin the right solution for this, because he would need to get that list too 04:16
carlosto rebuild the packages04:17
SteveAif there are buildd logs or records of this, that would be probably better04:17
SteveAmaybe ask celso04:17
SteveAor infinity04:17
carlosthere are buildd logs and I think we can see some output from pkgstriptranslations script04:17
carlosso it should be more or less doable04:17
carlosat least we use them from time to time to debug some problems with .pot regeneration04:18
SteveAso, what is the plan?04:18
carlostalk with Martin, just in case he already got the list of packages to rebuild04:18
carlosif he doesn't have such list, talk with celso/infinity to see if they could provide the logs using any script (they are available from launchpad/librarian) so we don't need to use the web interface for every single package in backports04:20
carlosand get the list of packages with translations using those logs04:20
SteveAok.  let me know how it goes.04:21
carlosonce we get the list, I think the only chance we have is to upload again one by one the .pot files for those packages04:21
carlos(we have them available at people.ubuntu.com)04:21
SteveAis there any risk of nuking work that has been done since?04:21
carlosno, we don't nuck anything04:21
carlosthat work will appear as suggestions04:21
SteveAso, there is work to do :-(04:21
carloswhat we will do is to 'hide' them for dapper04:21
SteveAespecially without translation review04:21
carlosand show again some others that were removed when the backports one were imported04:22
carloshmm04:22
carlosnot really04:22
carlosI mean, they don't need to reactivate anything04:22
SteveAif there are translations that were made, which were confirmed, and which are now just suggestions04:22
SteveAthen that's a step backwards04:22
SteveAand work needs to be done confirming the suggestions04:22
carlosit's just a matter of setting some strings as obsolete and remove the obsolete mark from others04:23
carloswhat I mean as suggestions is that if that string that we are hidding in Dapper appears later in other distro release04:23
carlosit will appear as a suggestion04:23
SteveAwe need to find out how many packages are involved.04:23
SteveAok04:23
SteveAthat's for strings that are in the backport04:23
carlosso we will reuse that work later as part of our translation memory04:23
SteveAbut not in the one in main04:23
carloshmm04:24
carlosthe problem is that the backports have packages in main04:24
carlosoh, you mean with 'main' release?04:24
SteveAI'm concerned that after uploading the new POTs04:26
SteveAthat the state of translations in there will overrule work people have done since those POTs were produced04:26
carlosno04:26
carlosany translation done04:26
carloswill remain selected04:27
carloswhat we do is that, for instance04:27
carlosa backport for ktorrent includes a new string 'ktorrent rules'04:27
carlosonce we revert to previous .pot04:27
carlosthat string will not appear anymore in dapper's imports04:27
SteveAconsider this04:27
SteveAweek 1: translation done on ktorrent04:28
carlosjust because it doesn't belongs to dapper04:28
SteveAweek 2: dapper POT produced04:28
SteveAweek 3: more translation done on ktorrent04:28
SteveAweek 4: backport built04:28
SteveAweek 5: we fix problem by uploading the dapper POT04:28
SteveAhave we lost the work done in week 3?04:28
carlosno04:28
carloswe will be at that exact status04:29
carlosmy point was that04:29
carlosweek 4.5: translated something new from backport04:29
danilosafai get this, we might only have some additional translations which belong in backports, but these won't be used04:29
carlosin that case, those new translations will be hidden with the .pot change, nothing else04:30
danilosright, week 4.5 :)04:30
carlosdanilos: ;-)04:30
SteveAok04:30
SteveAthat was my concern.  if you're confident that's not an issue, then that's good04:30
carlosbut we don't remove them so they would appear as suggestions for Edgy if we publish the same version that the backport had04:30
carlosthat was my point04:31
carlossorry if that introduced some misunderstandings04:31
SteveAok04:32
SteveAI have a call now.  let me know how the discussion with various people go.04:32
SteveAthanks04:32
carlosSteveA: you are welcome04:33
carlosdanilos: let me have a 15 minutes break and then, we could start our next meeting. Is that ok for you?04:33
daniloscarlos: sure04:33
carlosso let's talk at 16:45 our time04:34
daniloscarlos: I'd want to drop by store as well, so I wonder if I should do that first as well?04:34
danilos(but 10mins is not enough for that ;)04:34
daniloscarlos: so how about 17h?04:34
carlosok04:34
danilosgreat04:34
carlos17h works for me04:34
carlossee you later04:35
daniloslater; SteveA, carlos, thanks for bringing these issues up, even if I didn't have much to say on them :)04:35
carlosdanilos: you are welcome ;-)04:35
carlosdanilos: at least you should know about those issues04:36
daniloscarlos: indeed04:36
=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting
=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting
=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting
carlosdanilos: hi, should we have the meeting here?05:03
daniloscarlos: sure05:03
daniloscarlos: so, lets start with TranslationImport thing05:03
danilosdid you have a chance to take a look at the very drafty-spec, and more importantly, to think about it?05:04
danilosmy idea is to create a simple interface which will provide all data we need, *without* any database stuff05:04
danilosthe reasoning is that most of the database stuff is repeated (as experienced developing xpi import)05:05
carlosyeah I saw your document05:05
carlosbut I'm not completely sure of how do you plan to do it...05:05
carlosI know the idea05:06
daniloswell, I haven't written anything about implementation05:06
carlosto have an object with a single file as an input05:06
carlosand n files as output05:06
danilosand that's why I want to discuss it with you and have a preimplementation call with a reviewer05:06
carlosI think that the basic idea is good enough to work on it05:07
danilosmy current idea is as written above: accept path/content in constructor, and produce something like a list of templates with all the needed data05:07
danilosspecifically, I would make TranslationImport.templates a dict keyed by potemplate name05:07
carlosbut I would like to know how do you plan to deal with the import queue (this changes it a lot)05:07
daniloswell, I'd go for minimal changes in import queue05:08
daniloswhen there is only a single POT/PO being imported, we'd have the same thing we have now05:08
carlossure05:08
carlosbut most powerful part of the import queue05:08
daniloswhen there are more than one, we fully expect imported file to list all the templates it wants to go into05:08
carlosare tarball imports from Ubuntu packages05:08
carloswith multiple .po and .pot files05:08
carlosand the guessing code to decide were should that be imported05:09
danilosindeed, and no reason to abandon that05:09
danilosI'll just move that to separate TranslationImport class05:09
danilosthe only problem I can see is that we'd have to approve/disapprove the entire tarball05:10
carlosso you will need to open tarballs every single time we scan all entries in the queue?05:10
carlosthat's not possible, we should be able to reject or block single files within a tarball just like we do right now05:11
daniloshum, if I want to provide more details, yes05:11
carlosso I guess some extra metadata would be needed05:11
carloshmmm05:11
daniloswell, the other, probably better option is to also allow separation as it's done currently05:11
danilosfor tarballs, that is05:12
carloswell, the code that guess were every entry should be imported needs to check the paths of every single file05:12
danilosso, eg. GSI files would present themselves as separate queue entries as well05:12
danilosand we can link to the same librarian file for all of the entries05:12
carlosso we would have more than one entry for a single GSI?05:12
danilosyeah, for different potemplates/languages05:13
carlosso we use 'path' field as the way to filter out the file from the tarball stored in the librarian?05:13
danilosthat's right05:13
carloshmm, I think I like that05:13
danilosthe only thing we need to watch out for is not to delete file in librarian until all references to it are cleared ;)05:14
carlosthat will not need too many changes in our current code05:14
carloslibrarian handles that automatically05:14
carlosno entries are removed until there are no more references to it05:14
danilosexactly, and we get pretty sofisticated management of even other things, we'd be able to move KDE langpack support to that, etc.05:14
carloshow's that?05:15
daniloswell, a single kde-l10n-<LANG> will appear as several PO files in the queue entry; i.e. the same way as tarballs work now, just per-language, not per-template05:16
danilosand, we'd be able to approve "subcomponents" of files (eg. approve single language from GSI file, which may contain a number of languages)05:17
carlosI see05:18
carloswhat I mean is that in this concrete case, kde-l18n handling will not change a lot05:18
carloswe will eat less disk space, but that's all05:18
daniloswell, it won't change at all05:18
carlosphone...05:19
danilosexcept that code will be cleaner, imho ;)05:19
carlosI'm back, sorry05:20
danilosno problem05:20
carloswell, I'm not completely sure whether it would be really cleaner....05:21
carlosI mean, we still need code to handle the tarball extract05:21
carloso well, the list of the tarball05:21
carloss/o/or/05:21
carlosand the disk space needed will be reduced a lot05:21
danilosindeed, but there won't be things like hardcoding all the checks in translation_import_queue.py05:21
carloshow's that?05:22
carlosI know that's true for OO and FF05:22
danilosi.e. we currently check if path.endswith('.po') or path.endswith('.pot')...05:22
danilosand completely special-case kde stuff with another function05:22
carlosright05:22
carlosbut there are other checks05:23
carlosthat cannot be moved the way you plan05:23
carlosfor instance05:23
danilosok, let me rephrase that: instead of "cleaner", I should have said "more generalized"05:23
carlosGNOME tarballs mix two different layouts05:23
carlosone for the application with a po/foo.pot and several .po files insde that directory05:23
carlosand another for documentation with something/foo.pot and then subdirectories for the .po files05:24
danilosyeah, but what's the problem with that?05:24
carlosanyway, even leaving the code to handle GNOME things, I agree now, it would be cleaner so we move KDE specific code to KDE tarballs05:24
carlosdanilos: it cannot be moved outside translation_import_queue05:25
carlosthat's all05:25
danilosI don't understand why not05:25
danilosafaics, I'd have TranslationImport.providesTemplates which would return a list of all the templates a file provides, and then that would reference all translations for those templates05:26
carlosHmmm, I see05:27
danilosand don't forget that we also have default_file_format to determine which importer to use05:28
carlosso you mean that you 'extract' a tarball only when you know exactly where its entries will be imported?05:28
carlosright, in fact I was thinking on default_file_format and it doesn't solve the problem with GNOME layout05:28
daniloswell, you list the filenames when you create import queue entries05:28
carlosI'm a bit lost because I see some holes in the process you describe05:29
carloscould you please describe step by step05:29
danilosactually, I don't really care about space-savings, so I may also extract them right away, just like you do with current tarball imports05:29
carloswhat would happen when we import, for instance, evolution 2.18 translations?05:29
danilosespecially if I am going to lose a lot of speed with that approach05:30
carlosso we get a tarball and the reference to the sourcepackage and distrorelease05:30
danilosTranslationImport detects there is only one template in there, and creates a single template, and a bunch of pofile import queue entries05:30
carloslet's see what do you have in mind, and then, decide whether we need to untar the entries or not (I don't think we need to untar it, just extract it when the .po file is actually used)05:31
carlosdanilos: doesn't it have doc + application .pot files?05:31
danilos(as for untarring, that can be handled independently of TranslationImport design)05:31
carlossure05:31
daniloswell, if it has both doc & application .pot files, then it will provide two entries for templates, along with a bunch of pofile entries for each of them05:32
danilosnow, you're probably thinking of automatic po file matching?05:32
danilosespecially because evolution POT file will probably need a rename05:33
carlosno, I just want to see the full process, not thinking yet on specific details05:33
carlosfrom what you just described to me05:33
carlosis the same thing we do right now05:33
danilosand we should establish the relation between template and pofiles once they are added to queue05:33
carlosget the tarball and fill the queue with all .pot and .po files included05:34
carlosok so until this step, the code would be the same, perhaps moving it to other parts of our tree, but same procedure05:34
daniloswell, yes, that's the point; it would work mostly the same for what we have already, yet allow easy extending to what we don't have (like Firefox, OOo, KDE PO, Zope...)05:34
danilosI don't really see much wrong in the current procedure, to be honest05:34
danilosexcept that I would move it outside the import queue code and generalize it05:35
carlosso you would have a kind of adaptor05:35
carlosfor tarballs that will do what we do atm05:36
carlosanother for .po and .pot files that do nothing05:36
carlos(if it's not a KDE PO or Zope one05:36
carlos)05:36
danilosthe only problem I see with the current implementation is that I need to modify like 5-6 places and add a couple of lines on each of them, and when I develop a new importer, I duplicate much of the code from poimport.py05:36
danilosthat's right05:36
carloswell, even if it's a KDE PO or Zope one, as we agreed the format change will be done on import time not as a .po file 05:36
carlosanother for OOo that will split the file in smaller chunks05:37
daniloswell, for KDE PO file, we probably need to descend from POParser only05:37
carlosetc05:37
carlosetc05:37
danilosthat's right, but without duplication of database object creation05:37
carlossure 05:37
carlosall them inheriting from a common class05:38
danilosfor Firefox, I ended up copying most of the import_po stuff, and just replacing relevant parts05:38
danilosthat's right05:38
carlosand we only write the method to do the split05:38
carlosok05:38
daniloswell, all of that would be part of TranslationImport interface, that was my idea05:38
danilosso, how do you feel about that?05:38
carlosI see an easy optimisation to reduce wasted disk space, but let's leave that for a latter optimisation05:39
carlosThat solves the problem with code duplication that you talk about05:39
carlosbut05:39
danilos?05:40
carlosI still don't see how do you plan to move the code from translation_import_queue to guess the POFile and POTemplate where an entry should be imported (we should start thinking on rename those table names...)05:40
carlosI see that when you extract an entry05:40
carlosyou can try to guess it and link it 05:41
carlosbut05:41
daniloswell, it's easy to have TranslationImport.template['evolution'] .guessed_template property05:41
carlosas you already pointed, when you need to do a translation domain change, the .pot file will not find a link05:41
carlossure05:41
danilosand translation import queue will use the same method as now: you can override it05:41
carlosand you call that before creating the entry in the queue, right?05:41
danilosthat's right05:42
carlosok05:42
carloslet's see it this other way05:42
carloslet's say you have a layout like gtk+05:42
danilosok05:42
carloswhere they use the package version as part of the path05:42
carlosso you have something like gtk+-2.10.3/po/gtk20.pot05:42
carlosand we have imported 2.10.205:43
carlosthe automatic matching will not work here05:43
daniloshum, I don't follow05:43
carlosso you will not be able to do that link05:43
carloswe had this problem with Edgy05:43
carlosto link the new .pot file05:43
danilosif we have imported 2.10.2, where do we get gtk+-2.10.3 from?05:43
carlosis the new one05:43
carlosthat we are handling05:44
danilosah, ok05:44
daniloswe have sourcepackage and distrorelease here, right?05:44
carloswe do TranslationImport.template['gtk20']  .... this has a problem, the translation domain is not always the same as the .pot filename so we look for pot files based on its path05:45
carlosyeah, you know sourcepackagename and distrorelease05:45
danilosthat will appear in the queue as a separate entry, just like it appears now05:45
danilosand gtk20-properties will as well05:46
danilosi.e. I am still not getting what you are aiming at05:46
carlosok, but without a link to a POTemplate or POFile, right?05:47
danilosthat's right05:47
carlosjust like we do right now05:47
carlosok05:47
carloswhat I do atm is05:47
carlosgo to Edgy's gtk20 template05:48
carlosand change the path05:48
carlosoptionally, I could link the .pot file with this POTemplate that I just fixed05:48
carlosok?05:48
danilosok, so you want us to automatize that as well?05:49
carlosno, it's not my point, we should do it, but it's not related to this discussion05:49
carlosnow, what happens with the .po files?05:49
carloswith current code, the .po files will be visited again and this time we will be able to link them with POFiles05:49
carlosbecause we find now a POTemplate in the same path05:49
daniloswell, that's the point I had above: we need to link them to this template queue entry once we create them as well05:50
carloshmm I see your point05:50
danilosso, instead of doing "path matching" in queue entry code, we'd do that while adding queue entries05:50
daniloswhich means that we might need another column in translationimportqueue05:50
danilosrather, translationimportqueueentry ;)05:51
carloslike template_entry?05:51
danilossomething like that, yes05:51
danilosand then we can just directly approve them once template gets imported05:51
carlosI see05:52
carloswould you note that we need a trigger or something to check05:52
carlosthat the entry pointed by template_entry should have its template_entry set to NULL ?05:52
danilossure, I'll summarize this entire discussion in the spec when we are done, so I'll note that as well05:53
carloseither that, or use an external table to represent this relationship so we don't need triggers05:53
carlosStuart should tell you the best solution05:53
carlosok05:53
carlosI agree more or less with this solution05:53
carlosbut let's talk about some corner cases05:54
carlosok?05:54
danilossure05:54
carlos(I think this one is easy)05:54
carloswe get a tarball with a set of .po files and no .pot files at all05:54
carlosthat link will not be done05:54
carloswhich is fine05:54
carlosnow, we get a new version of that tarball that includes the .pot file05:55
carlosyour code should be able to detect the duplicated .po files and update those rows05:55
carlosI think this corner case is not a big deal05:55
danilosthat's right05:55
carlosok, next one05:55
carloswe implement a new layout support 05:56
carlosfor something that we already got imported into the queue05:56
carlosso the .pot and .po files aren't linked05:56
danilosbullocks, DELETE those, and reimport ;)05:56
carlosthat's not possible, a reimport requires a new Ubuntu package release05:56
daniloswe can also try to handle that using the same librarian reference and path's05:57
carlosthe DELETE is not needed, a new import will update those entries05:57
carloswith that, you will need then to store references to the tarball05:57
danilosif we go without extracting, but if we extract everything, then the only thing we can work with are paths05:57
carlosinstead of the content of the concrete entry05:58
carlosok05:58
carlosso no extracting++05:58
carlosafter handling the queue05:58
carlosyou will need to revisit every entry in the queue grouped by its librarian reference05:58
carlosand try again to detect its potemplate and pofile05:58
danilosthat's right, and make sure that logic for linking those in TranslationImport is separated, so it can be run just on paths05:59
carloshmmm05:59
carlosI'm not sure about that last thing06:00
carlosI mean, you are interested only in librarian links06:00
daniloswell, how would we otherwise do the matching after the import?06:00
danilosso you think redoing the import would be better?06:00
carlosonce you do that, you can handle it the same way a new import is handled06:00
danilosok, sure, it's simpler06:00
carlosbecause you already have code to deal with already existing code06:01
carlosotherwise the complexity would be higher than needed, isn't it?06:01
carloswe are talking about opening a tarball and get the list of entries inside it06:01
carlosbut we are not fetching its content06:02
danilosthat's right, I agree06:02
carlosif you see it as a performance problem, we can do what you suggest06:02
danilosit shouldn't be too much of a problem, I believe06:03
carlosalso, to prevent long delays in the queue06:03
carlosI think we should note last time we checked a set of entries06:03
carlosso we try to guess the same entries once per day06:03
carlosnot sure If you understand what I mean06:04
danilossure06:04
carlosok, let me see if I find any other corner case based on what we found already....06:04
daniloswe also need to do the guessing iff we have a new template import entry06:05
carlosfor product imports?06:06
danilosfor tarballs and stuff06:07
carlosother than products06:07
carloswhat's the point for that?06:07
carlosproducts relay on manual imports06:07
carlosso we could get a potemplate and later a tarball with languages06:08
carloswell, the other way, first translations and later a template06:08
danilosnot sure I understand you06:09
carlosif we get a tarball without template06:09
danilosI meant that apart from checking entries only once a day, we also don't need to do that if there was no template06:09
carlosand later a new upload06:09
danilosah, right06:09
carlosoh, I see06:09
carlossorry, I misunderstood you :-P06:09
danilosno problem ;)06:10
carlosbut, you need to do it anyway06:10
danilosright06:10
carlosjust in case we already had a template importe06:10
carlosd06:10
danilosanyway, who do you suggest that I ask to be my pre-implementation call reviewer? :)06:10
carlosphone, sorry06:10
carlosok, back06:18
carloshmm06:18
carloswell, SI guess james or Bjorn would fit06:19
carlosgrr06:19
carlosweel, Steve usually suggests james or Bjorn for this kind of things06:19
carloss/weel/well/06:19
carlosI'm a bit dyslexic today...06:19
danilosok ;)06:20
danilosI'll probably ask Bjorn, since I haven't worked with him so far ;)06:20
carlosok06:22
carlosbut prepare an spec update with what we just talked06:22
carlosso he can read it before the call06:22
carlosok?06:22
carloslet's take another 10 minutes break and we could start with our other meeting about view restructuring 06:23
carlosok?06:23
daniloscarlos: anyway, we also planned to discuss your view restructuring work, right?06:24
danilossure06:25
carlos:-)06:25
carlosI see you have lag...06:25
carloslet's meet again at 18:35 local time, ok?06:26
danilosyeah, that's fine06:26
carlosso06:40
carlosdanilos: meeting time? (again...)06:40
daniloscarlos: yay ;)06:40
carlosdanilos: You would want to read the email from kiko06:41
carloshe sent it 31st August with the subject: "POFileTranslationView/POMsgSetView cleanup guide, was Re: Status of a few bugs"06:41
danilosok, sure06:42
danilosok, it's already marked as "important" in my mail folder ;)06:43
carlosdanilos: the important bits are the ones related with pomsgset.py06:43
carlos:-P06:43
danilosok, I've read the msgset.py bits06:46
daniloscarlos: ping ;)06:48
carlosok06:48
carlossorry, I wasn't taking care of this channel O:-)06:48
carlosso06:48
danilosweren't you talking about not using several view classes?06:48
carloswell06:49
carlosthe suggestion by kiko is not exactly that06:49
danilosand cleaning up _*_submissions is basically what bug 30602 was all about ;)06:49
carlosat the moment, the view for POMsgSet pages is used from POFile's view06:49
carlosand that's broken06:49
danilosah, ok06:50
danilosI see your point06:50
carlosbecause we need to know when are we using it from a POFile view or a POMsgset view directly06:50
carlosin this case06:50
carlosthe shared bits are moved to a different view06:50
danilosso, the plan is to use POMsgSetView from both?06:50
carlosyeah06:50
carlosbut without using that view as a zope one06:51
danilosok, sounds much better06:51
carlosso it's not linked without any web page06:51
carlosit just have information06:51
carlosthat the POFile and POMsgSet views will use06:51
danilosyeah, understood06:51
carlosI still think that POFileTranslationView and POMsgSetPageView would use a common class from where they would inherit 06:52
carlosbecause they would share a lot of code06:52
carlosbecause POFileTranslationView is for a set of messages and POMSgSetPageView is just for a single message06:53
carlosbut I guess we could leave it for later06:53
danilosYeah, I know06:53
danilosthe thing, as I see it, is that it would mostly be about template sharing06:54
carlosso POFileTranslationView and POMsgSetPageView will have just the needed bits to render the web page06:54
carloswell, we already have that done06:54
danilosi.e. it would be mostly the same template for processing data from POMsgSetView06:54
carloswe are already sharing the template06:54
carlosand that doesn't need any change06:54
carlosto go with this solution06:54
danilosok, so what code is useful for both, yet can't be moved to PoMsgSetView?06:55
carlosthe main problem were with navigation links, that we had to check whether we were being used from a POFile view or a POMsgSetView directly06:55
carlosto generate them06:55
danilosright, but I am wondering what would need sharing? navigation would be separate, so that's cool ;)06:56
carlostabindex generation, general statistics info (the one at the end of the form)06:56
carlosthe alternative language selector code06:57
carlosmore or less I think that's all, but anyway, we can leave it duplicated as it's atm and think on the inheritance later06:57
daniloshum, some of them might belong in other classes (like statistics being part of POFile, no?)06:57
danilossure, I don't see much use of inheritance right away06:58
carloscould be06:58
carlosbut don't worry, I don't want to handle that right now06:58
danilosok06:58
danilosso, is there anything specific you want to discuss?06:59
carloswe just need POMsgSetView to handle all information that is part of the small section of a message06:59
carloswhether you think this is a good thing to do  ;-)06:59
danilosah, ok :)06:59
carlosI think this solves some problems that our current model has06:59
carlosfor instance07:00
daniloswell, I believe it's a very good thing, it will make it even more clear for anyone delving into code in the future as well ;)07:00
danilosi.e. I know it wasn't very simple for me to track down all the relations and dependencies this way, with views being used from views, etc. :)07:01
carlosthe link problem07:01
carloswe currently solve that adding a flag to the POMsgSetView class to note if it comes from a POFile07:01
danilosand at the same time, you will be doing the _*_submissions() cleanup, probably reducing the number of queries as well ;)07:01
carlosto give one kind of links or others...07:01
carloswell07:01
carlosnot really07:01
danilosyeah, which is a kludge, agreed07:02
carlosor not sure...07:02
carlosI should not change anything but the restructuring07:02
carlosanything else should be deferred to another branch07:02
carlos(I don't mind to take care of that task anyway, but not as part of that branch)07:02
danilosok, maybe you won't, but then I will later on, and it will be simpler for me :)07:03
carlosyeah, that's the goal07:03
carlosso do you think this solution would require less time to fix that bug?07:03
carloscould you quantify it for me?, you don't need to be precise ;-)07:03
carlosbecause this would be another argument to do this now instead of post 1.007:04
carlos:-)07:04
daniloswell, it will probably drop from 3-6 days of active work to 2-4 days, not really sure07:05
danilosthe thing is that it requires some optimization work and profiling, which you never knows how much it will take (just remember our edgy migration work in london ;)07:05
carlosI know, but that's enough07:06
carlosI think I could do the restructuring in around 4 hours + test fixes + move from POST to GET07:06
carlosI guess that in 1 day and a half of work would get that done07:07
carlosI just need to move code around07:07
carlosand change the POST to be a GET07:07
carlos+ fix a lot of tests07:07
carlosdo you think this is something optimistic or realistic?07:08
danilosrealo-optimistic ;)07:08
carlosin fact, add half day to the mix to cover me from being a bit lazy...07:08
carlosso 2 days07:08
danilossure, sounds reasonable07:08
danilosso, I guess with what we win (nicer code, altogether maybe 1 day more work), I believe it's worthy it07:09
daniloss/worthy/worth/07:09
carloswell, 1 day more work only related with your bug...07:09
carloswith TranslationReview we have less extra work07:10
carlosbut is hard to me to estimate what would we save07:10
carlosI think that we would save nothing, just clear code07:10
carloss/clear/clearer/ 07:10
carloswhich could save more time in the future...07:11
danilosyeah, right07:11
danilosso, if we stick to our time estimates, I believe it's the way to go07:11
daniloswhat do you think?07:11
carlosI think so, yes07:11
carlosI'm going to write to Steve and the list about this07:12
carlosin fact07:12
carlosdue I'm not going to change anything there, I'm not sure whether another meeting with a reviewer would be necessary07:12
carlosthe bigger part of this is part of your bug...07:13
carlosand I'm not going to do it a this stage07:13
carlosbut later07:13
carloswhat do you think?07:13
danilosyeah, sounds reasonable07:13
carlosok, thanks07:14
carlosIs there anything else we should talk about?07:14
danilosand you need to be careful with the tests and GET/POST switch07:14
carlosI already did such change for the message filtering code, so don't worry, I felt the pain already....07:15
carloswe were doing POST for them until some months ago07:15
danilosok, great :)07:16
danilosI think that's all, enough meetings for today :)07:16
carlosyeah07:17
carlostoday was pretty intense with meetings...07:17
carlosI didn't wrote any code :-(07:17
carlosthanks for your input07:18
=== carlos -> out for today...
carlosdanilos: do you need anything from me?07:19
danilosno, that's all; enjoy your evening ;)07:19
carlossame for you07:20
danilosI am out myself, will be back later for some more action though :)07:20
carlosok07:20
carloscheers!07:20

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!