=== ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting === carlos [n=carlos@138.Red-81-39-35.dynamicIP.rima-tde.net] has joined #launchpad-meeting === danilos [n=danilo@cable-89-216-150-63.dynamic.sbb.co.yu] has joined #launchpad-meeting [03:11] SteveA: please, ping us when you are ready [03:13] if you are busy to have the meeting now, tell us it too so we can have another meeting that we have pending [03:13] please [03:14] carlos: hi [03:14] I'm here [03:14] hi [03:14] two things [03:14] 1. I want to understand better the timestamp and manually entered data thing [03:15] 2. about the view refactoring, one point that came out of discussion with kiko and mark is that it is something to do if it will be a small job, and will improve things [03:15] don't spend time on it before 1.0 if it is a larger thing [03:15] it isn't a goal in itself [03:15] it is an idea tha tmight help work on TranslationReview [03:16] so, dealing with (2) first [03:16] what do you think? [03:17] well, the thing is that I could implement TranslationReview with current views, but adding some hacks to see, for instance, the copy button that the user clicked to copy a message in the text area to modify it [03:18] I don't think it would be much more ugly than what we have right now [03:19] so if you prefer to leave it for later, I can do it, but taking into account that on review time [03:19] maybe you should guesstimate it? [03:20] how long to do the refactoring in days? how long to do TranslationReview after the refactoring? how long to do TranslationReview before the refactoring? how long to do the refactoring after TranslationReview? [03:20] based on what kiko told me or what danilo and I talk ? [03:21] tell me what you think [03:21] after all, you'll be doing the work [03:22] well, I would need to think about it before I can give you an estimation [03:22] I know the idea behind kiko's suggestion, but I didn't think about it yet in depth because the pending meetings that would change a bit the solution [03:22] (just as a sidenote, this is exactly the reasoning why I want to work on generalized TranslationImport stuff: it will take a 2-4 days to implement it, but adding OOo, KDE PO would be much shorter and cleaner) [03:22] I could give you that info at the end of today [03:23] is that ok? [03:24] yes, that's fine. [03:24] that will give you some idea of whether to do this first or later [03:25] danilos: well, in that case you are preventing the hack, we already have that hack in production, so it makes mucho more sense in your case [03:25] so, about the exports and timestamps [03:25] s/mucho/much/ [03:25] ok [03:25] carlos: I know, just giving the reasoning behind it [03:25] SteveA: let me tell you what we have atm and how does it work, ok? [03:26] what I understand is that there is this process that involves producing exports [03:26] and when pitti says one is good enough, that latest one becomes the baseline [03:26] yeah [03:26] where does the manually entered timestamp come into it? [03:27] pitti tells me the timestamp of the tarball used for the baseline [03:27] and I ask Stuart to store it in our database [03:27] is that the same data that is the most recent baseline export? [03:28] not always [03:28] the tarball is made from the most recent baseline export? [03:28] why might it not be? [03:28] because there is a small delay [03:28] a small delay between what and what? [03:28] between when pitti gets a tarball, creates the .deb packages and notify me the timestamp [03:28] and we create a new export every day [03:28] the timestamp is based on data in the database [03:29] it is purely within the database's data [03:29] no, the timestamp is based on the date when the export was done [03:29] what happens to that data -- making a tarball or a deb -- doesn't affect the timestamp of when the export was produced [03:29] oh [03:29] I see what you mean [03:29] sort of [03:30] so, I cannot see a reason to put a timestamp *into* the database [03:30] it would work that way if our exports come from production [03:30] it is something that should only ever come out of the database [03:30] but it's not the case [03:30] what I can see going into the database is [03:30] marking which export is the one that is used [03:30] we use a read only mirror [03:30] we don't store the list of exports in our database [03:30] oh [03:31] our datamodel doesn't allow it [03:31] is not like Ubuntu packages, we don't need that complexity [03:32] ok [03:32] so, the way I'd approach that is [03:32] in the exported data, add a timestamp + checksum of timestamp [03:32] so there cannot be a typo [03:34] but, I'm more inclined to connect to the real database [03:34] me too [03:34] and store that an export was produced [03:34] and store the date of the latest translation used, or whatever is appropriate [03:34] and give that an export-id [03:34] so pitti can just say "this export-id is the baseline" [03:35] I guess we could use the link to the librarian as that 'export-id' [03:36] so we could have it for free adding a link to latest baseline langpack [03:37] SteveA: also, I was thinking on use the timestamp for latest translation in that export as the timestamp for the language pack so this problem wouldn't happen again [03:37] right [03:38] please file a bug or two on these issues [03:38] there is still another issue [03:38] what's that? [03:38] as we generate daily language packs [03:39] we need to provide Martin Pitti a way to go to launchpad and note himself which language packs has been used as the base package [03:40] how does pitti get a particular language pack? [03:40] does he get an email? [03:40] or go to a page in launchpad? [03:40] atm, he fetchs it from people.ubuntu.com [03:40] once we move to production, he will get an email === danilo_ [n=danilo@cable-89-216-150-89.dynamic.sbb.co.yu] has joined #launchpad-meeting [03:40] with a link to librarian [03:40] how does it get to people.ubuntu.com? [03:41] my script in carbon pushes the tarball there once built [03:41] I see [03:41] so, if we had a database table for langpacks produced [03:41] he could see it in a UI [03:41] and mark in that same UI if one has been used as the baseline for what [03:41] although Martin asked us for a fixed URL in librarian so he doesn't need to parse the email [03:41] SteveA: right [03:41] or we could offer him xmlrpc [03:42] I think that would be the right approach [03:42] if he wants to automate it [03:42] yeah, that would be also a good way to do it [03:42] ok [03:42] that seems like a small spec to me [03:42] couple of paragraphs explaining the background and proposed solution [03:42] so we can schedule it for post 1.0 [03:43] ok [03:43] also, I think that's the only missing part to move language packs to production [03:44] ok [03:44] so we talked about... [03:44] - having the export script write to the db [03:44] maybe in a special table [03:44] or maybe using the librarian id [03:44] well, we still need a table [03:45] so that there is a database-produced unique ID for a langpack that is generated [03:45] it's a one to many relation [03:45] so, use the 'id' in the table rather than the librarian id perhaps [03:45] one distrorelease has n language pack exports [03:45] ok [03:46] ok [03:46] and then we want to record there whether it is used as the baseline [03:46] or a baseline [03:46] and the timestamp of the most recent translation [03:46] ok [03:46] also we have another kind of language packs, the ones that only have updates... [03:47] but I don't think I should bore you with that [03:47] I will note that in the spec [03:47] I know that they exist [03:47] and the script that produces them can use this data to know what range of translations to include [03:47] right [03:48] and then we also talked about a UI + maybe xmlrpc for pitti [03:48] also, I'm thinking on adding something to launchpad that allows pitti to select whether we are going to generate updates or full exports [03:48] to get the langpacks, see what langpacks are available, and mark the one he chooses as a baseline [03:48] ok [03:48] so he can decide to do a new full export [03:48] so, this is something to discuss in person with pitti perhaps? [03:49] all these things together [03:49] to reduce the packages with just updates (it already happened with dapper point release, but he had to do it manually) [03:49] I guess, let's try first phone call after we have a draft [03:50] and see whether that's needed [03:50] anyway, if it's post 1.0, we could talk about it in the allhands meeting [03:51] well, when are you moving the langpack production into production? [03:51] once we have this new spec implemented [03:51] ok [03:52] adn that's not a 1.0 goal [03:52] as far as I'm aware [03:52] right, it's post 1.0 [03:52] ok, then I think that's settled [03:52] what do you think danilos ? [03:52] I'm fine with it [03:52] ok, great. [03:53] thanks for having this meeting. [03:53] and it has just moved to carbon [03:53] so performance should not be an issue in the near future [03:53] we must just be careful about those timestamps [03:53] well, the performance issues were already fixed even before moving it to carbon [03:53] until using a better system [03:53] maybe add a checksum to the timestamp as an interim measure... ? [03:54] or ensure that the mail sent to set it in the db [03:54] SteveA: don't worry, what I will try to do is to set as the timestamp latest modified translation [03:54] is sent to pitti too [03:54] SteveA: that way we solve the mirror problem [03:54] carlos: you still need to store that somewhere [03:54] SteveA: inside the tarball exported [03:54] so, there's still a timestamp coming out of the database [03:54] across to pitti [03:54] we already have a timestamp.txt [03:54] then back to you [03:54] and back into the database [03:54] right [03:55] and that is error-prone [03:55] I will add also the checksum as you suggested to be completely sure that nothing was lost [03:55] so, consider this [03:55] those are more or less trivial tasks [03:55] add a checksum to timestamp.txt [03:56] and write a small script to read a timestamp.txt, check checksum and set it in the databaes [03:56] if that will take under 2 hrs, then I'd say do that [03:56] if more, then it is too much work [03:56] hmmm, why should I do latest part? [03:56] what does that mean? [03:56] I still need to ask stuart to do the update sentence [03:56] I don't have direct access to production database [03:56] you can tell stuart to run that script [03:57] or you can send stuart the timestamp.txt [03:57] and tell stuart to run the script on it [03:57] I see [03:57] ok [03:57] the point is to avoid having someone typing in a manual query [03:57] so we reduce the chance to introduce a typo [03:57] copied from some email or text file [03:57] but, it is worth doing this only if it will be quick [03:57] good plan [03:57] don't bother if it will be more than 2 hours [03:57] ok [03:58] I think it would fit in a 2 hours slot [03:58] because it is just an interim thing [03:58] until the better system is designed and implemented [03:58] so pitti needs to send me the timestamp.txt file and that's all [03:58] yes [03:58] ok [03:59] >>> md5.new('timestamp').hexdigest() [03:59] 'd7e6d55ba379a13d08c25d15faf2a23b' [04:00] SteveA: yeah, I already used md5 module while debugging this problem [04:00] to know which files change between current language packs [04:00] and a full export I forced [04:00] so don't worry [04:01] something like that [04:01] dsfok, great [04:01] dsfok, gre [04:01] um [04:01] lag [04:01] great [04:03] lag + garbage :-P [04:04] SteveA: btw, now that we talk about this [04:04] there is another issue we should fix [04:04] what's that? [04:04] related with Rosetta and dapper [04:04] and perhaps breezy [04:04] https://launchpad.net/bugs/58221 [04:05] SteveA: pkgstriptranslations was stripping and feeding Rosetta with translations [04:05] for packages in the backports pocket [04:05] danilos: please, pay attention too, because we need to solve this issue [04:06] carlos: no problem, I am ;) [04:06] that means that we have some .pot files imported in dapper [04:06] that doesn't match with what dapper has in its official release [04:06] does this continue to happen? [04:07] or is it just that we have some bad data now? [04:07] no, 30 minutes ago, the buildds for backports has been fixed === ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting [04:07] but we have some bad data [04:07] no to what? [04:07] that we need to fix [04:07] you're saying that the problem is fixed [04:07] but the bad data remains? [04:07] right [04:08] does rosetta get told the pocket by the buildds? [04:08] Well, I think we can know that from our datamodel [04:08] and is a protection we should implement to prevent such breakages in the future [04:09] right, please file a bug on that [04:09] or, add a task to that bug [04:09] I did already [04:09] let me look for it [04:09] now, about the bad data. what do we need to do? [04:10] https://launchpad.net/products/rosetta/+bug/58223 [04:10] I think that we need to get the right .pot files and reimport them again [04:10] that should be enough [04:11] how do you identify what pot files are needed? [04:11] because the .po imported didn't change anything in Rosetta other than what came from upstream, so nothing was broken from the Ubuntu translator point of view (we just added some new translations from upstream) [04:12] well, that's the problem I don't know exactly how to solve [04:12] I was thinking on ask for a list of packages in the backports pocket [04:12] and filter out the ones without translations [04:12] so I get that list [04:12] but I think this would be a manual process [04:13] in the other hand, I guess the amount of packages should be low because Dapper was released only 3 months ago [04:14] well... [04:14] all those packages will also need to be rebuilt [04:14] with non-stripped translations [04:14] breezy would be more complicated, but the amount of templates imported is quite low than dapper so I don't think the problem would be too bad (we didn't get a full import for Hoary or Breezy) [04:14] SteveA: right [04:15] because they are without translations atm [04:15] are there buildd logs we can use? [04:16] I guess, but I could check with Martin the right solution for this, because he would need to get that list too [04:17] to rebuild the packages [04:17] if there are buildd logs or records of this, that would be probably better [04:17] maybe ask celso [04:17] or infinity [04:17] there are buildd logs and I think we can see some output from pkgstriptranslations script [04:17] so it should be more or less doable [04:18] at least we use them from time to time to debug some problems with .pot regeneration [04:18] so, what is the plan? [04:18] talk with Martin, just in case he already got the list of packages to rebuild [04:20] if he doesn't have such list, talk with celso/infinity to see if they could provide the logs using any script (they are available from launchpad/librarian) so we don't need to use the web interface for every single package in backports [04:20] and get the list of packages with translations using those logs [04:21] ok. let me know how it goes. [04:21] once we get the list, I think the only chance we have is to upload again one by one the .pot files for those packages [04:21] (we have them available at people.ubuntu.com) [04:21] is there any risk of nuking work that has been done since? [04:21] no, we don't nuck anything [04:21] that work will appear as suggestions [04:21] so, there is work to do :-( [04:21] what we will do is to 'hide' them for dapper [04:21] especially without translation review [04:22] and show again some others that were removed when the backports one were imported [04:22] hmm [04:22] not really [04:22] I mean, they don't need to reactivate anything [04:22] if there are translations that were made, which were confirmed, and which are now just suggestions [04:22] then that's a step backwards [04:22] and work needs to be done confirming the suggestions [04:23] it's just a matter of setting some strings as obsolete and remove the obsolete mark from others [04:23] what I mean as suggestions is that if that string that we are hidding in Dapper appears later in other distro release [04:23] it will appear as a suggestion [04:23] we need to find out how many packages are involved. [04:23] ok [04:23] that's for strings that are in the backport [04:23] so we will reuse that work later as part of our translation memory [04:23] but not in the one in main [04:24] hmm [04:24] the problem is that the backports have packages in main [04:24] oh, you mean with 'main' release? [04:26] I'm concerned that after uploading the new POTs [04:26] that the state of translations in there will overrule work people have done since those POTs were produced [04:26] no [04:26] any translation done [04:27] will remain selected [04:27] what we do is that, for instance [04:27] a backport for ktorrent includes a new string 'ktorrent rules' [04:27] once we revert to previous .pot [04:27] that string will not appear anymore in dapper's imports [04:27] consider this [04:28] week 1: translation done on ktorrent [04:28] just because it doesn't belongs to dapper [04:28] week 2: dapper POT produced [04:28] week 3: more translation done on ktorrent [04:28] week 4: backport built [04:28] week 5: we fix problem by uploading the dapper POT [04:28] have we lost the work done in week 3? [04:28] no [04:29] we will be at that exact status [04:29] my point was that [04:29] week 4.5: translated something new from backport [04:29] afai get this, we might only have some additional translations which belong in backports, but these won't be used [04:30] in that case, those new translations will be hidden with the .pot change, nothing else [04:30] right, week 4.5 :) [04:30] danilos: ;-) [04:30] ok [04:30] that was my concern. if you're confident that's not an issue, then that's good [04:30] but we don't remove them so they would appear as suggestions for Edgy if we publish the same version that the backport had [04:31] that was my point [04:31] sorry if that introduced some misunderstandings [04:32] ok [04:32] I have a call now. let me know how the discussion with various people go. [04:32] thanks [04:33] SteveA: you are welcome [04:33] danilos: let me have a 15 minutes break and then, we could start our next meeting. Is that ok for you? [04:33] carlos: sure [04:34] so let's talk at 16:45 our time [04:34] carlos: I'd want to drop by store as well, so I wonder if I should do that first as well? [04:34] (but 10mins is not enough for that ;) [04:34] carlos: so how about 17h? [04:34] ok [04:34] great [04:34] 17h works for me [04:35] see you later [04:35] later; SteveA, carlos, thanks for bringing these issues up, even if I didn't have much to say on them :) [04:35] danilos: you are welcome ;-) [04:36] danilos: at least you should know about those issues [04:36] carlos: indeed === ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting === ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting === ddaa [n=ddaa@nor75-18-82-241-238-155.fbx.proxad.net] has joined #launchpad-meeting [05:03] danilos: hi, should we have the meeting here? [05:03] carlos: sure [05:03] carlos: so, lets start with TranslationImport thing [05:04] did you have a chance to take a look at the very drafty-spec, and more importantly, to think about it? [05:04] my idea is to create a simple interface which will provide all data we need, *without* any database stuff [05:05] the reasoning is that most of the database stuff is repeated (as experienced developing xpi import) [05:05] yeah I saw your document [05:05] but I'm not completely sure of how do you plan to do it... [05:06] I know the idea [05:06] well, I haven't written anything about implementation [05:06] to have an object with a single file as an input [05:06] and n files as output [05:06] and that's why I want to discuss it with you and have a preimplementation call with a reviewer [05:07] I think that the basic idea is good enough to work on it [05:07] my current idea is as written above: accept path/content in constructor, and produce something like a list of templates with all the needed data [05:07] specifically, I would make TranslationImport.templates a dict keyed by potemplate name [05:07] but I would like to know how do you plan to deal with the import queue (this changes it a lot) [05:08] well, I'd go for minimal changes in import queue [05:08] when there is only a single POT/PO being imported, we'd have the same thing we have now [05:08] sure [05:08] but most powerful part of the import queue [05:08] when there are more than one, we fully expect imported file to list all the templates it wants to go into [05:08] are tarball imports from Ubuntu packages [05:08] with multiple .po and .pot files [05:09] and the guessing code to decide were should that be imported [05:09] indeed, and no reason to abandon that [05:09] I'll just move that to separate TranslationImport class [05:10] the only problem I can see is that we'd have to approve/disapprove the entire tarball [05:10] so you will need to open tarballs every single time we scan all entries in the queue? [05:11] that's not possible, we should be able to reject or block single files within a tarball just like we do right now [05:11] hum, if I want to provide more details, yes [05:11] so I guess some extra metadata would be needed [05:11] hmmm [05:11] well, the other, probably better option is to also allow separation as it's done currently [05:12] for tarballs, that is [05:12] well, the code that guess were every entry should be imported needs to check the paths of every single file [05:12] so, eg. GSI files would present themselves as separate queue entries as well [05:12] and we can link to the same librarian file for all of the entries [05:12] so we would have more than one entry for a single GSI? [05:13] yeah, for different potemplates/languages [05:13] so we use 'path' field as the way to filter out the file from the tarball stored in the librarian? [05:13] that's right [05:13] hmm, I think I like that [05:14] the only thing we need to watch out for is not to delete file in librarian until all references to it are cleared ;) [05:14] that will not need too many changes in our current code [05:14] librarian handles that automatically [05:14] no entries are removed until there are no more references to it [05:14] exactly, and we get pretty sofisticated management of even other things, we'd be able to move KDE langpack support to that, etc. [05:15] how's that? [05:16] well, a single kde-l10n- will appear as several PO files in the queue entry; i.e. the same way as tarballs work now, just per-language, not per-template [05:17]