[01:21] a8 mt3 succeeded [01:22] got no mail though [01:23] successful builds are usually not mailed [01:24] oh, ok === Admiral_laptop [n=Freddy@st074039212101.monm.edu] has joined #ubuntu-mozillateam === Paddy_EIRE [n=patrick@89.240.245.243] has joined #ubuntu-mozillateam === Mertiki [n=zxz@modemcable023.30-56-74.mc.videotron.ca] has joined #ubuntu-mozillateam === DarkMageZ [n=richard@ppp121-44-75-228.lns10.syd6.internode.on.net] has joined #ubuntu-mozillateam === asac_ [n=asac@debian/developer/asac] has joined #ubuntu-mozillateam === rexbron [n=rexbron@62.6.158.161] has joined #ubuntu-mozillateam [01:51] heya asac [01:51] :) [02:33] bluekuja: hi ... all fine? === gnomefreak [n=gnomefre@ubuntu/member/gnomefreak] has joined #ubuntu-mozillateam === Chiselhuk [n=cheryl@ACBDA7D2.ipt.aol.com] has joined #ubuntu-mozillateam [03:12] Hi Can anyone here help me plse?? [03:13] Having trouble with Mozilla Firefox crashing continually in Ubuntu Gutsy Gibbon! [03:14] asac: yup :) [03:14] asac: I'm studying atm, and later I'll move to fix two agg's bugs [03:15] asac: I hope to have them fixed as soon as I can, NM waits me [04:01] hehe === carlos [n=carlos@canonical/launchpad/carlos] has joined #ubuntu-mozillateam [04:59] asac: please, ping me when you are back [05:04] carlos: ping [05:04] asac: hi [05:04] ok how can we get this going [05:05] asac: so, I'm starting implementing the export code to get translations from Launchpad and generate language packs for packages using Mozilla XPI file format [05:05] I did some investigation after our last chat [05:05] yes [05:05] and indeed, using .manifest files is the way to go [05:06] so how does it currently work? [05:06] do you get initial strings from upstream xpis? [05:06] I talked with pitti about how to extract those files on build time like we do for other translation formats and import automatically into Launchpad [05:06] the idea is, you upload a new package into Ubuntu's archive [05:06] new package == new firefox package ? [05:06] we extract the strings automatically and import them into Launchpad [05:07] ok [05:07] asac: new firefox or any other package that uses xpi files [05:07] the problem is that not all packages are using XPI files [05:07] right [05:08] and Martin thought is not a good idea to look for .manifest files + all the subdirectories inside that directory [05:08] because is move file format dependent information in a layer that shouldn't know about it [05:08] so he asked me to check with you whether we could use xpi files inside source packages [05:08] instead of having it uncompressed (like ubufox does) [05:09] ubufox currently has no translations [05:09] then, the package build will extract the xpi file content and do the usual things [05:09] its actually the same way firefox ships its en-us locale [05:09] yeah, that would need to be updated too [05:09] if possible [05:10] so we only need to look for .xpi files [05:10] and move the logic inside Launchpad and the packages that use it [05:10] hmm [05:10] the .xpi file doesn't need to be locale specific [05:11] we are able to discard the bits that are not related with translations inside Launchpad [05:13] hmm why do we start with source files at all? [05:13] why not look at the contents of the binaries? [05:13] because it's at that level where we get the information for Launchpad [05:13] the extraction process is triggered on build time [05:14] by the pkgstriptranslations script [05:14] thats hooked into some standard debhelper command? [05:14] yes [05:14] or how is that triggered? [05:14] which one? [05:14] I don't know so much details [05:15] let me see whether pitti would join us [05:15] yes [05:15] he's the one handling that part [05:15] thats probably a good idea === pitti [n=pitti@ubuntu/member/pitti] has joined #ubuntu-mozillateam [05:15] hello [05:15] ola pitti [05:16] 17:13 < asac> hmm why do we start with source files at all? [05:16] 17:13 < asac> why not look at the contents of the binaries? [05:16] 17:13 < carlos> because it's at that level where we get the information for Launchpad [05:16] 17:13 < carlos> the extraction process is triggered on build time [05:16] 17:14 < carlos> by the pkgstriptranslations script [05:16] 17:14 < asac> thats hooked into some standard debhelper command? [05:16] 17:14 < carlos> yes [05:16] 17:14 < asac> or how is that triggered? [05:16] 17:14 < asac> which one? [05:16] 17:14 < carlos> I don't know so much details [05:16] asac: dpkg-deb [05:17] asac: that kills *.mo from debs and also extracts po files, etc. from the source tree into a trasnaltion tarball for Rosetta [05:17] plus some custom stuff for OO.o and Mozilla (extract .xpi, etc.) [05:17] pitti: can we take a look at the binary results at that point? [05:17] so, just to set the topic, the goal is to extract the rigth set of files for Rosetta from the builds of firefox itself (en-US) and ubufox, right? [05:18] pitti: yes [05:18] asac: we need to mangle at build time, and use the source; we need to strip off .mo files, and the binaries do not have *.po which we need for Rosetta [05:18] asac: what's the point on doing it for binaries instead of source packages? === jeromeg [n=jerome@gra94-2-82-66-142-60.fbx.proxad.net] has joined #ubuntu-mozillateam [05:18] point is that almost no mozilla source tree has .xpi files ... and that is impossible to change [05:19] impossible == without changing the world ;) [05:19] asac: well, that's done *after* source build [05:19] ok [05:19] so it could be done after you build the xpi file in hte source build [05:19] so, which files do we actually need? [05:19] asac: we take that same approach to update all .pot files before extracting them [05:20] we need to balance simplicity of pkgbinarymangler with special knowledge about mozilla-like translations [05:20] pitti: we have to search the binary tree for .jar files and extract the entities from there [05:20] asac: that's something the Rosetta importer should do [05:20] asac: no, that's not done at that level [05:20] at build time, the most complex operation possible is "find some files and copy them into the tarball" [05:20] if that grabs a few files too much, it doesn't hurt [05:21] it shouldn't be too many to avoid getting too big tarballs, of course [05:21] asac: then, Launchpad gets that tarball and process it to extract the translations [05:21] but that happens after the build and publishing process ends so we don't interfere with buildd [05:23] ok please look at: http://paste.ubuntu.com/603/ [05:23] from my position, what would be quite useful is to add a rule to the packages that use Mozilla l10n infrastructure to generate the .xpi files we need [05:23] and execute that rule before the pkgstriptranslations === carlos looks [05:23] usually you would have to take all .jar files that have a locale reference in their manifest [05:24] asac: that's why I talk about xpi, if you zip all those files inside an .xpi file, Launchpad will do that research [05:24] after the package is built [05:24] thats not possible ... really [05:24] and pitti's script will only need to know what to extract, because it finds the .xpi file [05:25] hmm [05:25] how many source packages are we talking about? just firefox and ubufox? [05:25] i don't understand why you can deal with .xpi while you can't with arbitrary chrome jars [05:25] pitti: in the end about all mozilla related packages [05:25] for two packages it makes sense to keep the special logic in their debian/rules, since *.manifest files are by no way mozilla specific [05:25] pitti: firefox and ubufox are good corner cases [05:25] if both work, most others should work too [05:25] .jar files are mozilla specific [05:26] so if we just extract all .jar files, would that work? [05:26] pitti: no, I need the manifest file too [05:26] pitti: also ubufox doesn't use jar [05:26] meh [05:26] pitti: the problem is that you have to generate a proper .manifest file during export ... look at /usr/share/firefox/chrome/en-US.manifest [05:26] it has the files uncompressed [05:26] . o O { gettext FTW! } [05:26] the only way this can be done right is too parse all .manifest files [05:26] and from there go on. [05:26] that's not something we should ever do for all packages in pkgbinarymangler [05:27] pitti: what about the approach I told you about? [05:27] extract all *.manifest files and any file in subdirectories where that file is [05:27] but [05:27] there are different ways to reference locations inside that ... jar: is one option, but there might as well be just directories ... like you can see in ubufox chrome.manifest [05:27] restrict it to a well known list of packages [05:28] asac: I know about that, but we don't want to add that logic to the extract process [05:28] but leave it inside Launchpad [05:28] so we don't need to maintain different code in different places but just in a single place [05:28] so, what about: [05:28] carlos: without looking at .manifest you won't be able to export them in a usable fashion. [05:28] if there are any .jar files, extract those, plus all *.manifest [05:29] that's simple enough for pkgbinarymangler [05:29] asac: I want that .manifest file so I will be able to generate the right export, don't worry but at Launchpad level [05:29] pitti: that leaves ubufox outside, it has .manifest files but it lacks .jar files [05:30] pitti: https://pastebin.canonical.com/461/ [05:31] carlos: can you psate to paste.ubuntu.com ? [05:31] i don't know the pass ;) [05:31] seriously, we cannot code 10 special cases into pkgbinarymangler [05:31] special cases should be handled by the particular source packages [05:31] asac: it's just the content of the ubufox source package [05:31] pitti: its not a special case ... its just: look at .manifest ... and use the method listed there [05:32] in the forth column: if there is jar:... you can find the right .jar file and extract that ... if its just a directory, you would just use that [05:32] asac: it's even more simple, extract everything in the directory where .manifest is and we will handle the parsing later [05:32] pitti: ^^^ [05:32] carlos: maybe that would work [05:33] but export needs to be smart then [05:33] if you have concerns for packages using different manifest files, let's lock that extraction to a well known list of sourcepackages [05:33] carlos: we need an additional condition when to do it [05:33] asac: we use the import to generate the export [05:33] *.manifest is way too generic [05:33] i would prefer to teach the pkgbinarymangler how to deal with: [05:33] locale necko en-US jar:en-US.jar!/locale/en-US/necko/ [05:33] and [05:33] locale ubufox en-US locale/en-US/ [05:33] that will cover all cases i guess [05:34] asac: really, that's useless at that stage [05:34] carlos: why? you would get the right list on what you want to import into launchpad [05:35] asac: because that adds more complexity and we can handle the filtering on Launchpad side where we need to know about .manifest content anyway [05:35] ok [05:35] pitti: combined with a well know list of sourcepackages shouldn't be a problem... [05:35] but you could at least filter out not needed .jar files/directories [05:36] right, we need something dead simple and 100% robust in the binary mangler [05:36] asac: right, that would generate smaller tarballs, although it's not a big problem, we could even get the source tar.gz imported into Launchpad, we ignore the files that are not interesting for us [05:37] so if it adds extra complexity, let's not do that [05:37] asac: what's the list of sourcepackages that use Mozilla like translations? Do you know that list? [05:38] ok ... to summarize: we use the binary results present at the end of build [05:38] asac: just to clarify, that's not the binary packages, but the final version of the built package [05:38] and unzip all .jar files that are in a directory that with at least one chrome.manifest? [05:39] asac: no need to unzip them === carlos is confused now... [05:39] i don't understand the difference ... for me its just debian/firefox or debian/ubufox at the end of build [05:39] no need to unzip? [05:39] no, Launchpad will inspect them [05:39] based on the manifest files [05:40] ok, so why wouldn't ubufox work at the moment? [05:40] i understood that launchpad would choke on that if we don't do anything [05:41] if pitti agrees on extract directory and contents that have a *.manifest file inside it for a given list of packages (so we only do that for packages that are really using this kind of translations) [05:41] ubufox will work without changes [05:42] as I said, that will work if you give me an additional conditions how to detect a mozilla-style package [05:42] pitti: I'm talking about giving you a fixed list of sourcepackages [05:42] no need to move gigabytes of OpenOffice stuff over, just because that happens to have a manifest file, too [05:42] pitti: isn't that enough? [05:42] carlos: ugly [05:42] pitti: how about looking for locale lines in .manifest? [05:42] it won't allow us to easily add stuff to that list [05:42] if they exist its likely a mozilla-style package [05:43] you can even test if the locale line has 4 columns [05:43] isnt' there a typical directory name, or something like that? [05:43] not really [05:45] pitti: extensions have a install.rdf file [05:46] ah, now that's something [05:46] asac: does main firefox package have it too? [05:46] no [05:46] just extensions [05:46] for moz apps we can only test if the .manifest file is in the chrome/ directory [05:46] asac: In fact, I think is more reliable to look for chrome.manifest for extensions, isn't that the name always? (except for xul applications) [05:47] so either you find a install.rdf ... or one or more manifest files in a chrome directory ... is that good enough pitti ? [05:47] carlos: i think pitti feares that there will be other packages shipping .manifest files [05:47] asac: well, he fears of looking for *.manufest [05:47] which is why i suggested to also introspect those .manifest files and search for lines that start with ^locale and have 4 columns [05:48] chrome.manifest should be enough [05:48] carlos: thats not given [05:48] ok [05:48] look at dpkg -L firefox [05:48] asac: yeah, .../chrome/.../*.manifest sounds good to me [05:48] pitti: its always directly in the chrome directory [05:48] ah [05:48] so chrome/*.manifest [05:48] so, a test for chrome/*.manifest [05:48] asac: that's why I said that xul applications are an exception, it's only valid for extensions ;-) [05:49] pitti: yes ... and if you have a install.rdf you have an extension package [05:49] asac: and fire firefox itself? [05:49] ah, ok [05:49] so "-e /chrome/*.manifest || -e ./install.rdf" [05:50] pitti: so that will give me that directory + all its content? [05:50] in this case I'll extract all *.manifest files and all files in their directories [05:50] (including subdirs, of course) [05:50] pitti: yes, but thats all only valid for the binary tree [05:50] (just to repeat) [05:51] that's quite a lot of non-trivial code for binarymangler, but bearable [05:51] in source tree there is no chance to find something common [05:51] if it cannot be simplified, I'm ok with that [05:51] asac: we have that, if the extract script is run before cleaning the build directory [05:51] ok [05:51] asac: oh, all rules need to be for the root directory of the built source tree [05:51] pitti: that means you don't need to look for .xpi files [05:52] ok [05:52] ok, so we had half task clear ;-) [05:53] asac: I need to talk with you about deploy translations from Launchpad [05:53] pitti: if you are busy you don't need to care about this part [05:54] asac: Is there any way to override mozilla translations without overwrite what you are installing right now? [05:54] for instance, If I want to ship a Spanish translation update for Firefox, is there a way to do it without overwriting /usr/lib/firefox/extensions/langpack-es-ES@firefox.mozilla.org/ directory content? [05:55] carlos: well ... should be [05:55] just /usr/lib/firefox/extensions/langpack-es-ES-rosetta@firefox.mozilla.org/ [05:55] alright; so the rule from above makes everyone happy? [05:55] and fill in previously not tranlsate strings [05:55] pitti: yes [05:55] carlos: did you already file a bug about this this morning? [05:55] pitti: no, not yet [05:55] ok [05:56] I'll create one [05:56] asac: yeah, I was thinking about using a Launchpad specific ID, although I would like to be able to change also existing translations in the other directory [05:56] asac: but I didn't find how Firefox looks for the translation directory [05:57] pitti: thank you [05:58] carlos: what do we want to achieve? why don't we try to export complete translations from rosetta? [05:58] will rosetta have the complete translations? if not, how does rosetta know the strings that need to be translated? [06:00] yes, we will have complete translations (or at least we should) [06:00] but we want to ship them inside language packs [06:00] so we cannot overwrite files from another package [06:00] like mozilla-firefox-locale-es-es === cwong1 [i=chatzill@nat/intel/x-1a244317143b9fb9] has joined #ubuntu-mozillateam [06:01] and we still need them built to get updates from upstream [06:01] hmm [06:01] asac: if it's a single file (like .mo ones for gettext applications) is quite trivial to remove them from the binary packages, but in this case, is not so trivial... [06:01] ok, so we need two packages anyway? [06:02] asac: yeah, you keep current ones, although the updates will be included in the standard existing language packs ones [06:02] will rosetta automatically package a locale package for me? [06:02] yes, that's the idea [06:02] or will i have to include translations that are somehow dumped from it [06:02] you need to keep packaging whatever is in upstream, though [06:02] no, you don't need to care about Rosetta [06:03] that's why I need to know how to override translations from upstream [06:04] carlos: i don't think you can reliably override ... would be more or less at random. so lets just fill in the gaps [06:04] is that enough? [06:04] e.g. only export entities that don't exist? [06:05] that prevents us to fix broken translations [06:05] and will be a bit confusing [06:06] maybe we could do something like we did with glibc to check first in language pack translations and then fall back into the standard translations... [06:06] but in Firefox [06:06] how about just removing all .jars in binarymanagler from language packs? [06:07] so all translations will live in our -rosetta packages? [06:07] e.g. mozilla-locale-de will be more or less empty ... and depend on mozilla-locale-de-rosetta [06:07] asac: will not that break having manifest files pointing to a non existing file? [06:08] carlos: since manifests need to be in -rosetta as well, we should drop them as well [06:08] e.g. rm *.jar *.manifest :) [06:08] pitti: we need you here again... [06:08] that's done in pitti's land so he's the one to answer that question [06:09] asac: btw, what's /var/lib/mozilla-firefox/extensions.d directory for ? [06:09] thats old ... you can ignore it [06:09] seems like you are able to give a priority to the extensions... [06:09] ok [06:10] no its not used anymore [06:10] ok [06:11] asac: how does Firefox know about which language pack use? where is noted that? [06:11] does it iterates over all extensions available on /usr/lib/firefox/extensions and your home directory? [06:12] carlos: it loads all extensins that way ... right [06:12] then it resolves all locale/ uris with the locale found in OS environment [06:12] so that's why you say it's random? [06:12] more or less yes. [06:12] it depends on which directory picks first? [06:12] ok [06:12] there are other source of randonmenss as well [06:13] that sounds sooo broken [06:13] for instance stuff gets indexed in hash maps ... so order gets lost [06:13] I see [06:13] welll overwriting locales isn't really a strong usecase :) [06:13] maybe for us, but not for others [06:13] re [06:14] https://bugs.edge.launchpad.net/ubuntu/+source/pkgbinarymangler/+bug/149017 [06:14] Launchpad bug 149017 in pkgbinarymangler "extract Mozilla-ish translations" [Undecided,New] [06:14] can you please scrutinize the description again and give your ack in the bug report? [06:14] pitti: we need to figure a way to remove Mozilla translation files like we do with .mo files [06:14] pitti: sure... [06:14] carlos: oh, why? [06:14] carlos: do the extensions ship their own translations? [06:14] pitti: because there is no way to override upstream ones installed [06:15] pitti: yeah, and also, language packs from upstream need to be still built and uploaded to Ubuntu's archive so we are able to keep Launchpad update [06:15] updated [06:15] btw, this rule does not really work with https://pastebin.canonical.com/461/ [06:16] well, it would just copy the entire source and build tree [06:17] pitti: if that bug description covers this: [06:17] /usr/share/firefox/chrome/en-US.jar [06:17] /usr/share/firefox/chrome/en-US.manifest [06:17] yes, that's it (I'm not sure I'm understanding it completely) [06:17] pitti: which is ok [06:18] pitti: Launchpad will discard the files that are not related with localisation [06:18] carlos: it won't cover /usr [06:18] it will pick up ./debian/foo/usr/share/firefox/chrome/en-US.{jar,manifest}, though [06:18] pitti: yeah, sorry, /debian/foo/usr/... [06:20] pitti: its best to remove all translations from locale packages if we want to support translation fixes from rosetta [06:20] i don't feel so good about that ... i would prefer just to fill the gaps and if we have improvements send them upstream [06:21] regularly [06:21] asac: my only concern about removals is that, if we have a package like ubufox including 'es-ES' or 'de' translations, how would we handle that? [06:22] asac: that's not the way it works for other applications and will be confusing (although we ask translators to do what you just said, send changes and new translations to upstream) [06:22] carlos: you have to import initial entities anyway ... regardless if there is en-US or something else [06:23] asac: I'm talking about content removal [06:23] ah ok [06:23] the import into Launchpad is completely clear, don't worry [06:23] carlos: languages included in application/extension just cannot be changed through rosetta i guess [06:23] we won't remove en-US from firefox as well [06:24] asac: right, en-US should stay in all cases [06:24] pitti: are you happy with the removal policy? [06:25] yes, so lets say: all translations that are in the _main_ packages have to stay [06:25] so only pure locale packages are stripped [06:25] pitti: it would mean to remove the manifest and .jar files for all extensions (the ones that have install.rdf) [06:25] (for all pure locale extensions only though) [06:26] asac: I will document that corner case and see whether we could figure a way to solve that later, once the initial support is in place [06:26] I did not follow scrollback, sorry; reading (thousand things today, crazy) [06:26] carlos: right [06:26] carlos: so, please give me a rule which pkgbinarymangler shuold exercise (which files, etc0 [06:26] I don't see any rule above, which files to remove [06:27] pitti: so you only remove files (chrome.manifest and *.jar) from packages that have install.rdf inside it [06:27] carlos: for ubufox i would prefer to only use rosetta locales (except for en-US) ... so thats not such a corner case. [06:27] asac: however, will not that break other extensions that are not translation ones? [06:27] asac: talking about the rule I gave to pitti [06:27] carlos: yes ... only remove files from _pure_ locale packages [06:27] asac: so how do we detect that? [06:27] never ever remove something for other packges [06:28] source package name should include -locale-all :) [06:28] pitti: is that enough to you? [06:28] that's fine for me (except for gutsy, of course) [06:28] and what shall I remove? [06:28] pitti: sure, this is only possible for Hardy [06:29] pitti: chrome.manifest and *.jar [06:29] asac: correct? ^^^ [06:29] so I should change the gutsy pkgbinarymangler to extract the translations, and the first hardy one to remove them, right? [06:29] pitti: if source package is -locale-all ... remove all *.jar and *.manifest [06:29] but I still don't see the purpose here [06:29] pitti: oh and install.rdf :) [06:29] wouldn't that make the -locale-all packages essentially empty? [06:29] pitti: actually remove all files taht are not directories [06:30] pitti: yeah, I guess is like kde-i18n one [06:30] why would we need to ship them still? [06:30] hm [06:30] pitti: those locale packages need to be completely empty and in turn depend on $pkgname-rosetta [06:30] so source should be in main, binaries in universe [06:30] pitti: because we have to build them in order to import upstream entities [06:30] hm, I thought we would ship them in the normal language-packs [06:30] pitti: yeah, that's what asacmeans with $pkgname-rosetta [06:30] pitti: we need a source to automatically import upstream entities [06:31] however _if_ we want to option to replace already translated entities we have to wipe everything from them and depend on the -rosetta package [06:32] pitti: do i need to rephrase? [06:32] right [06:33] in the long run it would probably be better to import those straight into rosetta (kde-i18n and upstream mozilla translations) [06:33] but ok for now [06:33] pitti: well its for hardy anyway ... so lets just import them straight [06:33] that sounds more reasonable [06:33] pitti: do you mean using an external process to fetch them automatically? [06:34] asac: we don't have such process available yet, so it's not possible [06:34] carlos: we can already upload po files and tarballs into Rosetta in the UI, right? [06:34] I mean, automatically [06:34] that can certainly happen with a script or wget call, etc. [06:34] carlos: well its just a small step to automize it ... and even if we can't doing it manually from time to time is good enough as well i guess [06:35] asac: believe me, you will get bored of that [06:35] I'm doing it manually for debian-installer [06:35] carlos: i won't do it :) [06:35] asac: that's why you suggest it ;-) [06:35] hehe [06:35] but pitti says it ... you can setup a script for that [06:36] pitti: I would prefer to keep doing it this way and expand our XML-RPC to do that automatic process you talk about (we have a spec to offer such XML-RPC api for products) [06:36] ok ... so we have a procedure to extract translatable strings during build ... and then we can fill rosetta by manually (scripted) importing upstream translations regularly [06:36] carlos: ok [06:36] is that correct? [06:36] asac: kind of, yes [06:37] ok, I think I have everything clear (or kind of and I could start working now) [06:37] pitti: don't deploy this yet, though [06:38] pitti: I will need to add support to Launchpad first or we will get untranslated Firefox/Firebird [06:38] ok ggreat [06:38] pitti: also, remember of doing the removal only for main packages [06:39] asac: I guess the output I should provide is the exact content of packages like mozilla-firefox-locale-es-es, right? [06:39]