[09:20] -ChanServ(ChanServ@services.)- You do not have channel operator access to [#canonical-ops] 
[12:15] <carlos> hi
[12:15] <SteveA> hi
[12:15] <SteveA> hi
[12:15] <danilos> hi
[12:16] <SteveA> so, stub and I were just talking about the downtime required to run the rosetta updates
[12:16] <danilos> yeah
[12:16] <carlos> SteveA: do we have info about when is launchpad less used ?
[12:16] <SteveA> stub tells me that with the current code, we're talking about many hours' downtime
[12:16] <SteveA> I have a few comments about this that we should consider
[12:17] <SteveA> 1. we don't want to take down all of launchpad for N hours in order to do rosetta updates
[12:17] <carlos> between 2 and 4 hours
[12:17] <SteveA> 2. is it not possible to do the updates incrementally?
[12:17] <SteveA> about 1,
[12:17] <stub> 2-4 hours per run, or 2-4 hours for both runs?
[12:17] <carlos> stub: per run
[12:18] <SteveA> if we need to do lots of rosetta database work, we can take down just rosetta,
[12:18] <carlos> that's on staging
[12:18] <carlos> and it depends on the load of the server, because sometimes is done in 2 hours and other times in 3 hours and a half
[12:18] <SteveA> by changing the authorization adapters to deny all rosetta edits
[12:18] <SteveA> and changing the "permission denied" page to say about rosetta being under maintenance
[12:18] <SteveA> so, that's one way of managing this
[12:18] <SteveA> so, the rest of launchpad (shipit, bugs, etc.) can stay running
[12:19] <danilos> I think that might be the best way out
[12:19] <SteveA> but
[12:19] <carlos> yeah, at least for this initial run
[12:19] <carlos> next time, for edgy + 1
[12:19] <carlos> will be rosetta and soyuz
[12:19] <SteveA> let's talk about why this needs to be done all in one go
[12:19] <danilos> on the other hand, it will take longer that way, no?
[12:19] <SteveA> why can't we run it a little at a time
[12:19] <SteveA> with pauses in between?
[12:20] <SteveA> for example, one package at a time
[12:20] <stub> I think Steve means one package at a time with commits in between
[12:20] <carlos> hmm
[12:20] <danilos> well, it was designed in such a way that it can be run that way, since it also supports incremental updates; carlos, isn't that so?
[12:20] <carlos> yes, it is
[12:20] <carlos> not per package
[12:20] <carlos> but per table
[12:20] <carlos> doing it per package will require to add some extra code
[12:20] <carlos> the problem of doing it per table
[12:21] <carlos> is that at some point people will start seeing some inconsistent data
[12:21] <carlos> but I don't think they could actually break anything
[12:21] <danilos> well, then the per-package seems easier and safer
[12:21] <carlos> we don't point to edgy translations so the amount of people looking at that data would be really small
[12:22] <carlos> danilos: but also much more slow
[12:22] <SteveA> that's okay
[12:22] <SteveA> it's okay for it to take a week
[12:22] <stub> yup
[12:22] <SteveA> if it means no interruptions in service
[12:22] <danilos> well, but it will lock only a package at a time
[12:22] <carlos> SteveA: but it will only be useful for this two runs we need now
[12:22] <danilos> and each will be locked for like a really small amount of time
[12:22] <SteveA> if one package is locked for 30 minutes, once
[12:22] <SteveA> then that's fine
[12:22] <carlos> edgy + 1 will not need that approach, right?
[12:22] <stub> It will lock the tables it is inserting into as normal, but the locks will be for a much shorter period and probably unnoticible.
[12:23] <SteveA> 3 minutes is even better than 30
[12:23] <danilos> yeah, I think it will be far less than 30 minutes
[12:23] <SteveA> but this is a much better approach for our service overall
[12:23] <stub> If it takes 30 mins to do the inserts for one package, or even 1 minute, then I don't know if this approach is doable on the live system.
[12:23] <carlos> oo.org would take more than 5 minutes...
[12:23] <carlos> it's a monster...
[12:24] <SteveA> so, we could do the large packages in a maintenance period
[12:24] <SteveA> we need to find out how long an average-sized package will take
[12:24] <carlos> SteveA: hmmm
[12:24] <SteveA> otherwise, look for a smaller granularity of commit still
[12:24] <carlos> I really prefer if we close rosetta
[12:25] <carlos> this change is going to delay edgy translations again
[12:25] <carlos> and I'm already getting lots of complains about it
[12:25] <carlos> the changes we are talking about would take more than a couple of days between code changes, testing on staging, code reviews and landing on production
[12:25] <carlos> + the time to deploy it
[12:26] <carlos> so that would be end of next week in the best case
[12:27] <danilos> well, this is actually true, so it all depends on the timing
[12:27] <SteveA> stu and i agree that we can take rosetta down
[12:27] <SteveA> if you guys can write some security code that
[12:28] <SteveA> has a launchpad.conf option
[12:28] <SteveA> so that when this option is set
[12:28] <SteveA> permission is denied to everything except for launchpad.Admins
[12:29] <carlos> even for the pages that are read only ?
[12:29] <SteveA> and to either
[12:29] <carlos> I mean, the ones that doesn't need to be logged in
[12:29] <SteveA> 1. raise a custom subclass of ForbiddenAttribute or whatever
[12:30] <SteveA> or 2. customize the forbidden oops page (and authorization required, or whatever)
[12:30] <SteveA> we need to do this for every page that allows writing to rosetta
[12:30] <danilos> so, carlos, what do you think about this?
[12:30] <SteveA> we'll test on staging
[12:30] <SteveA> by locking the rosetta tables
[12:30] <carlos> if it's just for teh pages that allow writing
[12:31] <SteveA> and checking that we can't lock up launchpad
[12:31] <SteveA> just for edit pages
[12:31] <carlos> the first part is quite easy
[12:31] <SteveA> I mean, we could do this in the forms instead
[12:31] <SteveA> instead of the security code
[12:31] <carlos> the second part... is the first time I do it so I'm not sure which option is better
[12:31] <SteveA> if there are only a few rosetta forms that allow updating rosetta tables
[12:31] <SteveA> that might be easier
[12:32] <carlos> there should be 4 or 5 different forms
[12:32] <SteveA> raise a RosettaDownForMaintenance error
[12:32] <danilos> well, we also need to block imports, no?
[12:32] <SteveA> yes
[12:32] <SteveA> and do an error page for RosettaDownForMaintenance
[12:32] <carlos> danilos: that's just a matter of disable the cron job
[12:32] <SteveA> we'll be making the tables read only to the launchpad user anyway
[12:32] <SteveA> so the database-level stuff will be okay
[12:32] <SteveA> but we want to give nice errors and not OOPSes
[12:33] <danilos> ok, just taking notes of everything
[12:33] <carlos> ok
[12:35] <danilos> so carlos
[12:35] <danilos> what do you want us to do? rosetta downtime, right?
[12:36] <SteveA> just to check something
[12:36] <SteveA> when we were in london together
[12:36] <SteveA> we talked about updating the stuff into a temporary table
[12:36] <SteveA> and then in a separate transaction, or separate translactions, moving them across to the real tables
[12:36] <SteveA> is that the approach you've taken?
[12:37] <danilos> mostly, I think carlos put the temp table creation into the same transaction (so it doesn't get out of sync)
[12:37] <SteveA> um
[12:37] <SteveA> that kinda obviates the point
[12:38] <carlos> danilos: yes, I would prefer to do the downtime
[12:38] <danilos> temp table is reading only, so it can probably be moved out without too much problems
[12:38] <danilos> carlos: ok, then lets do that as painlessly as possible
[12:39] <danilos> carlos: btw, didn't we get any improvements with temp tables? do we have indexes in there?
[12:39] <stub> So build all the inserts into tables. Then, a few rows at a time, insert some data into the production tables, delete from the temporary store, and commit.
[12:39] <carlos> danilos: indexes helped a bit
[12:39] <carlos> but not too much
[12:40] <carlos> SteveA, stub: I guess that it could make us to lose some translations for the migration (anyone added after the temporary table is created)
[12:41] <stub> carlos: That would have happened anyway, as the migration script would be using a snapshot of the database at the time the migration was started
[12:41] <carlos> stub: well, if rosetta is closed
[12:41] <danilos> stub: it gets a bit more complicated because of references and us updating table-by-table
[12:41] <carlos> no new translations will be added, right?
[12:43] <carlos> also, we should block any import from poimport script for edgy or as soon as the potemplate table is filled, entries from the queue will start to be importing
[12:43] <carlos> giving us conflicts with the temporary table
[12:47] <carlos> What problem do you see If we close Rosetta ?
[12:48] <carlos> I think that's doable and in the future, that feature would be interesting  in case of a performance problem like the ones we had in the past so the rest of launchpad is not also affected
[12:50] <danilos> well, we want to go for that anyway
[12:53] <carlos> In that case, I think that closing rosetta two days
[12:53] <carlos> one per migration
[12:53] <carlos> is not a big problem, if we choose the right time when users are not doing many translations
[12:56] <SteveA> stu and i are discussing various things...
[12:59] <carlos> ok
[01:02] <SteveA> ok
[01:02] <SteveA> here's the plan
[01:03] <SteveA> stu will try the update code on a clone of production, on the production server
[01:03] <SteveA> turning off indexes and shit like that
[01:04] <SteveA> we'll get an accurate estimate of the amount of downtime
[01:04] <danilos> ok
[01:04] <SteveA> meanwhile
[01:04] <SteveA> we can assume it will take at least 2 hrs
[01:04] <carlos> SteveA: should I start working on the 'disable Rosetta' feature ?
[01:05] <SteveA> so you guys need to add a switch to launchpad.conf
[01:05] <carlos> danilos: I think I should work on it as I'm just switching tasks and you already have a lot of open things
[01:05] <SteveA> that causes rosetta edit views etc. to raise RosettaDownForMaintenance when the view is viewed
[01:05] <SteveA> as well as edited
[01:05] <SteveA> and an error view for RosettaDownForMaintenance
[01:06] <SteveA> and we need to test this on staging, by stu turning off write access to rosetta tables
[01:06] <carlos> sounds good to me
[01:07] <carlos> stub: the script should be run with '-d ubuntu -r dapper' the first time and the next one with '-d ubuntu -r edgy'
[01:07] <danilos> carlos: sure, be my guest :)
[01:07] <carlos> the first one will migrate translations from breezy to dapper and the next one will copy all translations from dapper to edgy
[04:30] <flacoste> hi SteveA
[04:30] <SteveA> hi salgado 
[04:30] <SteveA> hi flacoste 
[04:30] <salgado> hey!
[04:30] <flacoste> i think the spec you were looking for was https://launchpad.canonical.com/LaunchpadI18n
[04:30] <flacoste> actually, that we should have looked for
[04:30] <SteveA> https://launchpad.net/products/launchpad-support-tracker/+spec/localized-support
[04:30] <SteveA> https://launchpad.canonical.com/LaunchpadI18n
[04:31] <SteveA> first, let's talk about the overall point of LocalizeSupportRequests
[04:31] <salgado> I pasted that on #canonical, didn't I?
[04:31] <salgado> okay
[04:31] <SteveA>     *
[04:31] <SteveA>       Grandma Lucille, who doesn't speak English, was navigating on the web until she reached a flash page, whose content she can't see because the flash plugin is not installed. She'll then click on "Help -> Get Help Online ..." and make a support request in Launchpad.
[04:31] <SteveA>     *
[04:31] <SteveA>       Malba Tahan, a very nice guy who knows a lot about Ubuntu is willing to share his knowledge by answering some support requests. He goes to the Support facet of Ubuntu and look for open requests. He sees a list of requests written in one of his preferred languages, pick up a randon one in Arabic and answers it.
[04:31] <SteveA> 
[04:31] <SteveA> these are use-cases from the spec
[04:32] <SteveA> so, we have requirements around identifying which support requests are written in a particular language
[04:32] <SteveA> and of allowing people to file support requests in their own language
[04:32] <SteveA> is that right, as overall goals?
[04:32] <salgado> right
[04:32] <SteveA> so, the support tracker needs to be able to track various things about what language the request is made in
[04:32] <flacoste> exactly
[04:32] <SteveA> and that's self-contained in the support tracker application
[04:33] <SteveA> I totally trust you guys to do a good job there
[04:33] <flacoste> that was the plan there
[04:33] <SteveA> what I'm concerned about is how this touches other parts of launchpad
[04:33] <salgado> what would these various things about the language be?
[04:33] <SteveA> and that a good plan for the support tracker may cause problems elsewhere
[04:33] <flacoste> the spec also covers what we needed form the launchpad infrastructure
[04:34] <SteveA> first of all, some overall concerns
[04:34] <SteveA>  - if we localize a page element that appears on the support pages, but also elsewhere
[04:35] <SteveA>    then we get partially localized pages in other parts of the application
[04:35] <flacoste> (that was covered indirectly in the spec)
[04:36] <salgado> was it?
[04:36] <flacoste> workaround) Translate strings for support. Display a message when going to untranslated section of the site.Mark template as translated or not. Force English when using an untranslated template.
[04:36] <SteveA> and then, all the unresolved issues in https://launchpad.canonical.com/LaunchpadI18n
[04:38] <SteveA> as far as managing risk goes...
[04:39] <SteveA> i think we should split the support tracker goals up into two specs
[04:39] <SteveA> 1. making the support tracker support multiple languages of support issues, but with its UI remaining untranslated
[04:40] <SteveA> 2. translating the launchpad UI, with the first parts to be translated being the support-tracker and the login workflow
[04:40] <SteveA> to make 1. a 1.0 goal, but not 2.
[04:40] <SteveA> because 2. touches so much stuff, and puts an unexpected burden on the infrastructure work
[04:41] <SteveA> that doesn't mean we can't get to 2. for 1.0, but it means that we aren't committing to it
[04:41] <SteveA> what do you all think?
[04:41] <salgado> I think that'd be quite awful, usability-wise, but I can understand it
[04:43] <flacoste> i agree that posting a question in my native language while the UI is in English doesn't look too good, but I can understand the idea of doing this two-steps
[04:43] <salgado> okay, now we need to convince mark of that, right?
[04:43] <SteveA> I'd really plan this as three specs
[04:43] <SteveA>  - make the support tracker work in multiple languages for the support requests, inc. data model changes etc.
[04:44] <stub> Last time we were looking at this, it seemed like the i18n would be postponed indefinitely. To participate in open source development, you needed to have at least basic English skills. I'm unsure if other parts of Launchpad will ever want to be localized, and it will be unfortunate if there is fallout in the rest of Launchpad to support localized support request tracking.
[04:44] <SteveA>  - make launchpad support internationalization
[04:44] <SteveA>  - make the support tracker and login / registration system internationalized and localized 
[04:44] <SteveA> we have a dependency relationship between these specs
[04:44] <SteveA> two of the specs are in the domain of logins and support tracking
[04:44] <SteveA> one is in the domain of infrastrcuture
[04:45] <SteveA> now, in terms of planning 1.0 features
[04:45] <SteveA> as leader of the infrastructure team, I'm being asked to support a new 1.0 requirement
[04:45] <SteveA> so we need to look at resources for that
[04:45] <stub> The spec mentions searching in particular. Our search infrastructure does not support searching in non-English languages. It might work by accident in many cases, but there will be glitches we cannot solve without features being implemented upstream.
[04:46] <SteveA> now, I can imagine a work-around of sorts
[04:46] <SteveA> that is, to ignore the login / signup parts
[04:46] <SteveA> becuase people manage to log in and sign up to get shipit CDs todasy
[04:46] <SteveA> so i expect they *can* do so for support requests
[04:46] <flacoste> stub: from a developer point of view you are right, but not from an Ubuntu user point of view
[04:47] <SteveA> and to localize just the body of support pages
[04:47] <SteveA> using the language the user is interacting with the support system
[04:47] <flacoste> SteveA: what do you think of the 'Mark template as l10n idea'?
[04:47] <flacoste> to prevent half-balked localization in the other part of the system
[04:47] <SteveA> I think that's an interesting idea to solve the problem of having a rendered page consistently translated or not
[04:48] <SteveA> that's just one part of the whole infrastructural issue
[04:48] <flacoste> indeed
[04:48] <SteveA> what I would ask for next is this:
[04:48] <SteveA>  - split up the specs into the three I mentioned
[04:48] <SteveA>  - take into account the existing spec about i18n for the infrastructural aspects
[04:49] <SteveA>  - write into that spec the minimum necessary to deploy an internationalized / localized support tracker
[04:49] <SteveA> then we can come back and look closely at what we end up with
[04:49] <SteveA> and decide if this is do-able for 1.0
[04:51] <salgado> sounds good to me
[04:51] <salgado> I can do that
[04:51] <stub> Cool
[04:51] <SteveA> ok
[04:52] <flacoste> flacoste: side note: in your infrastructure spec you forgot to mention how to handle string interpolation
[04:52] <SteveA> I think the support-only first part can be done immediately
[04:52] <SteveA> I mean, not blocked
[04:52] <stub> The search issue I mentioned should be listed on the spec. We may need to contract upstream if searching in different languages is particularly bad. 
[04:53] <SteveA> we should have an idea of what languages we're targetting too
[04:53] <stub> I suspect the search results will get worse the further the language gets from the standard roman alphabet.
[04:53] <flacoste> stub: there is UTF-8 support in 8.1 or 8.2 i think
[04:53] <stub> flacoste: tsearch2 has utf8 support, but it still is language dependant for stemming etc.
[04:53] <stub> (in 8.2, which doesn't exist yet, or CVS head)
[04:54] <flacoste> stub: indeed, but there is a 8.1 backport of the utf8 stuff i think
[04:54] <stub> And all our helpers force queries to ascii only, so that will all need to be rewritten.
[04:54] <stub> (because of the existing tsearch2 limitation)
[04:55] <flacoste> stub: ideally, we would also have to add support to the fti to index tickets based on their language
[04:55] <SteveA> my gut says we really should try to internationalize launchpad within the next 6 weeks
[04:55] <SteveA> um
[04:55] <SteveA> i mean
[04:55] <SteveA> my gut says we really should not try to internationalize launchpad within the next 6 weeks
[04:56] <flacoste> SteveA: yes, a list of supported language would be nice - who can decide on that?
[04:56] <SteveA> I can see it being a lot of rushed effort
[04:56] <SteveA> with flaky results because it is rushed
[04:56] <flacoste> stub: (OT: i have another fti question for you after the meeting)
[04:56] <SteveA> stu may have time to help on this
[04:57] <SteveA> but we need to look into this
[05:02] <SteveA> thanks salgado and flacoste !
[05:02] <SteveA> salgado: please tell me when we should look at the specs again
[05:02] <flacoste> To summarize:
[05:02] <flacoste> - salgado will merge the infrastructure part of LocalizedSupportRequest into LaunchpadI18n
[05:02] <flacoste> - salgado will move login workflow related changes to a separate spec
[05:02] <flacoste> - Infrastructure will review these specs to see if LaunchpadI18n and LaunchpadLoginL10n can be targetted for 1.0
[05:02] <flacoste> - add string interpolation example to LaunchpadI18n (flacoste)
[05:02] <flacoste> Other unassigned issues:
[05:02] <flacoste> - need to add search issue to LaunchpadI18n
[05:02] <flacoste> - determine list of languages to target
[05:03] <flacoste> should I mail the list with this?
[05:03] <SteveA> also
[05:03] <SteveA> we should have one spec for LocalizedSupportRequest
[05:03] <SteveA> and one spec for LocalizedSupportTracker
[05:03] <salgado> yeah, it will be kept, I assumed
[05:03] <SteveA> so that we can represent the dependencies well
[05:03] <salgado> ah, right
[05:04] <SteveA> thanks for summarizing flacoste 
[05:04] <flacoste> my pleasure
[05:04] <flacoste> should I mail this summary of our discussion to the list (after including SteveA update)
[05:04] <flacoste> ?
[05:04] <SteveA> mailing the launchpad list will be good.  it needs a bit of an introduction as well, of course
[05:04] <flacoste> no problem
[05:04] <SteveA> where my points for that are
[05:05] <SteveA> that it is good to make the support tracker localized
[05:05] <SteveA> it is in a way a special part of launchpad
[05:05] <SteveA> in that it is used by end-users who don't necessarily speak english well
[05:05] <SteveA> in a way this is like shipit
[05:05] <SteveA> it is unlike rosetta (translators always speak english)
[05:06] <SteveA> and unlike malone
[05:06] <SteveA> and unlike the spec tracker
[05:06] <SteveA> (programmers always speak englilsh)
[05:06] <SteveA> I generalize a little
[05:06] <SteveA> but these are generally true
[05:06] <SteveA> we must take seriously the task of internationalizing launchpad, because it has an effect on various areas of what we do
[05:06] <SteveA> including
[05:07] <SteveA>  - code reviews (reviewers need to know what to check for and understand the mechanisms that will be appearing throughout the code)
[05:07] <SteveA>  - test running (what locale / language do we run tests in ? )
[05:07] <SteveA>  - rollouts (how do we get PO files / generate POT files for doing new rollouts? )
[05:08] <SteveA>  - web request infrastructure (is the standard stuff in zope3 good enough for our purposes)
[05:08] <SteveA>  - cookie / database language preferences (new UI, new database work, new infrastructure)
[05:08] <SteveA>  - searching infrastructure (maybe doesn't cope with languages well)
[05:09] <SteveA>  - checking the current stuff about internationalizing python in zope, zcml, page templates
[05:10] <SteveA>  - having consistently translated pages (maybe using francis' ideas)
[05:10] <SteveA>    (but that implies changes infrastructure to support it)
[05:11] <SteveA> 
[05:11] <SteveA> there may be a middle-ground
[05:11] <SteveA> of just internationalizing certain page, like the support tracker pages
[05:11] <SteveA> and not worrying about having a full experience for support users
[05:11] <SteveA> oh, also
[05:11] <SteveA>  - internationalizing email we send out
[05:11] <SteveA>     like for the sign-up stuff
[05:12] <flacoste> and the ticket notifications
[05:12] <SteveA> right
[05:12] <SteveA> we should have a standard way of doing that
[05:12] <SteveA> and really, that's a spec in itself
[05:12] <SteveA> "internationalized email"
[05:12] <SteveA> as a management thing, we need to decide about doing all this in a planned and well-thought-out way
[05:13] <SteveA> or rushing it to completion over the next 6 weeks, fixing issues as we come across them
[05:13] <flacoste> (am i  to understand that 1.0 is scheduled for in 6 weeks?)
[05:13] <SteveA> and dealing with the fall-out later.   I'm keen on avoiding a rush.
[05:13] <SteveA> something like that
[05:13] <SteveA> 1 oct or so i think
[05:14] <SteveA> and people have vacations too
[05:14] <SteveA> 
[05:14] <SteveA> I think that's all
[05:14] <SteveA> any comments ?
[05:15] <flacoste> not from me
[05:16] <sivang> so make sure generilzation is made to include those languages for good support just as the latin ones.
[05:16] <sivang> (for example, emails need be RTL, input fields and the whole ballgame)
[05:16] <SteveA> depends if RTL languages are in the list of ones we're focusing on
[05:17] <sivang> true
[05:18] <sivang> probably better start with the latin ones, which would incur less fallout and more quick fixable issues I suppose , and only then advance further.
[05:24] <flacoste> SteveA: i added a MessageId interpolation example to LaunchpadI18n
[05:24] <SteveA> great
[05:25] <flacoste> we have a bug with notification currently for those strings, its bug 54987
[05:25] <flacoste> the bug report include a fix, I can update the tests and submit the fix for review if you care, otherwise it can wait
[05:28] <SteveA> stub should look at that
[05:28] <SteveA> I'll assign it to him
[05:29] <flacoste> great