#launchpad-meeting 2006-08-07
<spiv> ddaa: meeting time?
<ddaa> spiv: yes
<mpool> here?
<ddaa> SteveA: ping
<spiv> BazaarMeetingAgenda says here.
<ddaa> lifeless: ping
<lifeless> gnip
<SteveA> hi
<ddaa> jamesh should arrive soon
<ddaa> Meeting starts
<ddaa> Agenda
<ddaa>  * roll call
<ddaa>  * production status
<ddaa>  * Smart server
<ddaa>  * SFTP advertising
<ddaa>  * vcs-import knits
<ddaa>  * supermirror branch browser
<ddaa>  * private branches
<ddaa>  * dyson
<ddaa>  * empty hosted branches
<ddaa>  * Python import
<ddaa>  * critical bugs
<ddaa>  * pending sysadmin tasks
<ddaa>  * any other business?
<ddaa> -----
<ddaa> * Roll call
<ddaa> I'm obviously here
<mpool> here
<ddaa> and all participants too, I see
<ddaa>  * Production status
<ddaa> Nothing new.
<ddaa> https://launchpad.net/products/launchpad-bazaar/+bug/53825
<ddaa> Still spamming launchpad-error-reports. No short term fix expected.
<ddaa> What was the output of the launchpad-infra meeting about oops-from-scripts?
<lifeless> jamesh: ?
<jamesh> lifeless: yeah?
<lifeless> ^^ do you know :)
<jamesh> ddaa: we haven't discussed it yet
<ddaa> Okay, happy to leave it that way, eventually kiko will grow tired of the spam, but we are not there yet.
<jamesh> I'll be discussing some oops stuff with matsubara today, but I don't know if we'll get to oops-from-scripts
<ddaa>  * Smart server
<ddaa> mpool, spiv: report?
<mpool> no substantial progress this week
<ddaa> did not ask last week
<mpool> spiv, we should meet this week to pair on it, then i'd like you to do more, if possible
<spiv> mpool: tomorrow?
<lifeless> works for me
<mpool> ddaa: the status is that it's still trying to get all branch/repo tests to pass
<mpool> tomorrow is ok with me
<ddaa> Okay, I'll ask again next week.
<ddaa>  * SFTP advertising
<ddaa> jamesh posted draft on launchpad@, I commented, and expect a second draft.
<ddaa> I would like some feedback from the other guys (mpool, lifeless, spiv) if you have the time. Anybody wants to make a commitment?
<jamesh> I'll post a second draft once I've addressed the problems ddaa mentioned
<lifeless> no committment, sorry.I've followed the email discussion and it sounds like it will be good
<ddaa> mpool: spiv: ?
<spiv> I'm happy to read over the second draft.  I didn't pay close attention to the first one as I was trying to get through the flood of mail that built up during my week of leave :)
<ddaa>  * vcs-imports knits
<ddaa> Scream if you want that done this month, as I have enough things to keep me busy until leave (17th to 25th this month).
<mpool> draft looked reasonable
<mpool> ddaa: what are you working on til then?
<lifeless> It would be nice to do it this month. Depending.
<ddaa> mpool: I'll tell later, it's on the agenda :)
<ddaa> Okay, talk again about it at the end of the meeting.
<lifeless> I think weave branches are now into bad publicity area. 
<lifeless> right, later
<ddaa>  * Supermirror branch browser
<ddaa> spiv: what's your delivery estimate? (you promised one a couple of weeks ago).
<ddaa> spiv: you look a bit overloaded with bzr stuff...
<spiv> ddaa: Yeah, I've been switching to focusing heavily on smart server for that moment, so I don't intend to work on the branch browser until that's done.
<ddaa> Okay, if nobody disagrees, I'll drop the branch browser from the agenda until further notice.
<ddaa> Apparently, nobody disagrees.
<ddaa>  * Private branches
<ddaa> lifeless to reply to review comments. Unless somebody disagree, I'll drop that from the agenda.
<lifeless> fine
<ddaa>  * Dyson
<ddaa> Progress on that appears stalled. Unless somebody disagree, I'll drop that from the agenda.
<lifeless> please keep it
<lifeless> jamesh: that is currently with you AIUI
<ddaa> Okay. Will keep nagging people about Dyson.
<lifeless> jamesh: is that your understanding?
<SteveA> The branch browser is important to me for dealing with bad "trivial" landings on launchpad development.
<ddaa> mh?
<SteveA> If spiv can't do it over the next 2 weeks, I'd like to get someone else to do it
<jamesh> lifeless: I got the input from Mark and Scott about it
<spiv> SteveA: The task of setting it up internally is considerably smaller than the larger task.
<lifeless> SteveA: we can deploy a branch browser for launchpad separately
<ddaa> SteveA: isn't that something requiring private branches first?
<jamesh> lifeless: I'm not exactly sure how we want to proceed
<lifeless> SteveA: thats trivial
<spiv> ddaa: not involving launchpad at all.
<lifeless> jamesh: ok, after this meeting, lets talk in #launchpad and get you unblocked.
<jamesh> lifeless: I probably won't have time to work on it this week either way.
<lifeless> SteveA: I'm completley happy for you to get someone else to do the lp browser instance. If you have someone who can do one for the supermirror too, that would be great.
<SteveA> if it is trivial, can ew get it done this week?
<ddaa> spiv: SteveA: okay but that meeting item was about the launchpad branch browser. Having a non-launchpad one for rocketfuel is an infra-team thing AIUI.
<SteveA> ok
<SteveA> ddaa: "Dyson"
<SteveA> I forget what that is again
<lifeless> SteveA: I will get a core instance up and running and email rt for admin foo, after that its up to you
<lifeless> SteveA: upstream release tarball scanner
<SteveA> I asked last meeting for more descriptive names for "Dyson" and "Roomba"
<SteveA> and mpool seconded
<ddaa> SteveA: creating product releases automatically off tarballs found on the internet.
<spiv> ddaa: Well, having the internal one was a stepping stone to the larger one.
<lifeless> yes, dyson was scotts invention. roomba as mine
<ddaa> roomba -> soon-eradicated-importd-autotester
<lifeless> spiv: overengineering IMO. one sec
<SteveA> jamesh has offered to do the launchpad browser
<SteveA> that issue is closed now, thanks
<ddaa> dyson -> tarball-spider ?
<SteveA> importd-autotester adn tarball-spider
<SteveA> fine
<ddaa> Moving on.
<ddaa>  * Empty hosted branches
<ddaa> It occured to me that fixing https://launchpad.net/products/bzr/+bug/30576 (bzr push to an extant non-branch directory fails horribly)
<ddaa> was only half of the fix. The other half would be fixing 
<ddaa> https://launchpad.net/products/launchpad-bazaar/+bug/51130 (cannot use +admin on a branch I own)
<ddaa> and https://launchpad.net/products/launchpad-bazaar/+bug/34540 (cannot delete a branch),
<ddaa> I plan to fix 51130 before leave.
<ddaa> That said, I'll drop this item from the agenda.
<ddaa>  * Python import
<ddaa> bad news
<ddaa> Keeps failing in mid-course with a pysvn._pysvn.ClientError on PROPFIND "Could not resolve hostname `svn.python.org': Host not found (http://svn.python.org)".
<ddaa> It's not clear what's the cause of the problem, but I guess fixing it would involve:
<ddaa> - either import from svn dump (in importd)
<ddaa> - or replacing uses of pysvn by python-subversion in the critical sections (in cscvs), I expect that would give us better control on network usage.
<ddaa> I would like to have this sort of change done _after_ removing Arch support from the corresponding component.
<ddaa> In other words, we're not going to have it this month unless it's made a top priority.
<mpool> i'd be a bit unhappy to have it deferred so long
<ddaa> In still other words, our svn import stuff sucks badly.
<mpool> i think you should consider it a key test case
<ddaa> *sigh* test case fails, cscvs is not robust, old news
<ddaa> Though, fixing that in importd could be done without having to deal with Arch
<mpool> lifeless: what's your opinion?
<lifeless> work on concrete user visible goals
<lifeless> if there is interest in python in bzr-svn copy, then work on making it work
<ddaa> since that would be part of the "set up source tree for cscvs" code, which is target-independent
<lifeless> I think diagnosing the cause is important too
<lifeless> python-subversion was a PITA to work with 18 months ago
<lifeless> if it has improved, great.
<mpool> ddaa: so it always failes with "host not found"? 
<mpool> how odd
<ddaa> at random points in the conversion
<mpool> ok, we should take this to mail...
<ddaa> not at the start
<ddaa> want me to write email about it?
<ddaa> okay, will do
<ddaa>  * Critical bugs
<ddaa> No concrete change.
<ddaa> - Cannot set branch associated to a product series. https://launchpad.net/products/launchpad-bazaar/+bug/31308
<ddaa> lifeless to write spec
<ddaa> - renaming project, product or series breaks vcs imports. https://launchpad.net/products/launchpad-bazaar/+bug/37897
<ddaa> Have design and done preimpl call. My current top priority. Still expecting lifeless to brain dump about uploading/downloading cscvs source trees on remote servers.
<ddaa> - cannot use +admin on a branch I own. https://launchpad.net/products/launchpad-bazaar/+bug/51130
<ddaa> I plan to fix that before leave as part of ongoing plan to have fun improving user-visible stuff.
<ddaa> mpool: that answers your earlier question about my current coding focus
<ddaa> lifeless: please give me that braindump, I value your input on that design.
<lifeless> ok
<mpool> ddaa: ok
<mpool> (can i retitle that? bug 51130)
<ddaa> yeah, the title has gone wrong
<mpool> done
<ddaa> the expected fix is to delete +admin and move its contents to +edit
<ddaa> unless I change my mind
<lifeless> SteveA: browser is up and running
<ddaa> * Pending sysadmin tasks
<ddaa> * Any other business?
<SteveA> lifeless: cool!  URL please
<lifeless> on sodium, http://localhost:8088/ 
<SteveA> so, RT needed to expose externally
<lifeless> this is a minimal config, we can add multiple branches, etc in the future.
<spiv> lifeless: can't you run it as a cgi out of your homedir?
<lifeless> I'd like you to RT it
<lifeless> spiv: use case - trivial - done. fancy-pants-high-performance can wait for the supermirror code alterations
<mpool> just to check something from before - jamesh will be working on the branch viewer?
<ddaa> * vcs-imports knits, should that be set as a top priority for that month?
<spiv> lifeless: running as a cgi is hardly a performance booster ;)  But ok.
<jamesh> was going to work on the one for launchpad's branches (which lifeless seems to have up and running).  Not sure about the supermirror one
<lifeless> SteveA: I'd like you to RT it please, choose where you would like it to exposed etc. I've put it up immediately because that fulfils your requirements immediately.
<SteveA> lifeless: sure.  i'll do that. ta
<lifeless> np
<lifeless> we aim to please
<lifeless> its running as the PQM user. at this point if sodium is rebooted it will go away.
<lifeless> we can refactor the implementation later to be better as needed.
<jamesh> lifeless: so that's just the main rocketfuel branch.  I think we want other branches too (I suppose that can be done later)
<ddaa> jamesh: looks like the one for rocketfuel was grabbed by lifeless :)
<lifeless> well, I did mean it was trivial.
<ddaa> Focus, please.
<lifeless> SteveA: please file bugs on the launchpad-development-infrastructure product with things you want changed in the browser - branches to show etc.
* lifeless winds that up
<ddaa> So, SteveA request was granted, take deployment details on after meeting.
<lifeless> knits - IMO its a priority
<ddaa> Is that more important than fixing 51130?
<mpool> converting existing weave branches to knits?
<mpool> why is it difficult?
<lifeless> theres 500 to do
<lifeless> each has to be done in two places
<ddaa> some of them stupidly large
<ddaa> do not want to do that manually
<ddaa> so need to be implemented as transition automation in importd
<ddaa> so, not difficult per see, just a significant coding task
<ddaa> Waiting for opinion on whether I should drop one of my current tasks for that, or on somebody to provide extra workforce.
<lifeless> I'd like to get it done and out of the way
<lifeless> mpool: your thoughts?
<mpool> well, i can't judge how much work it woul dbe
<mpool> i mean, even if there are many of them surely it should be just a couple dozen lines of shell at most?
<ddaa> mpool: we do not want shell
<mpool> but i would not want someone actually using the weave branches - they will be awfully slow over http
<ddaa> we want that in importd because it's a shared resource issue and format upgrade is a recurrent task
<lifeless> mpool: unless you shut down the entire system while you do it you need to lock/unlock the branch being converted at the scheduler level
<lifeless> mpool: you also need urls from launchpad
<ddaa> and doing that sort of thing in shell is a LARGE pita because it needs to be distributed on many systems.
<mpool> i'm not actually suggesting shell, just saying that what actually runs is conceptually small
<mpool> so i wonder if there is any lazier way to do it?
<ddaa> BTW, fixing 51130 would make it easy to temporarily shut down importd-autotest with importd-production so we get maximal horsepower for the conversion.
<mpool> e.g. i'd rather shut down imports for 24-48 hours than have david spend a week writing this
<ddaa> s/shut down/merge
<ddaa> mpool: who do you think would spend 48 setting up, testing, and babysitting the upgrade hack?
<lifeless> see I dont understand why its more than 2 lines of python
<lifeless> but thats a different discussion. lets have that post meeting
<lifeless> mpool: lets talk priorities now, not costs
<ddaa> does the spec I wrote look like more than two lines of python?
<lifeless> lets talk after the meeting
<ddaa> Okay, so issue still open.
<lifeless> the discussion right now should be priorities IMO
<mpool> ok, so
<ddaa> And meeting time is over.
<lifeless> mpool: I think eliminating weaves is importnt
<mpool> after the meeting, dda, robert, mbp to talk about ddaa's suffering
<ddaa> ta, it's not whining, it's good use of programmer's time
<ddaa> Meeting over
<ddaa> I'm out for a small break.
<mpool> back in 10 say?
<ddaa> mpool: I'm back
<mpool> back
<ddaa> https://launchpad.canonical.com/VcsImportsKnitsUpgrade
<ddaa> There appears to be a single import in STOPPED mode, and I'm not sure it should not be syncing instead
<ddaa> so we can drop the provisions for STOPPED in that spec
<mpool> what does that mean?
<mpool> i mean, what does "stopped" mean?
<ddaa> it means "was once in sync, is published, but is no longer updated"
<ddaa> mainly so we do not get distracted by errors of things that we know will fail for reasons we cannot fix (like the repository went away)
<ddaa> that comes from there https://launchpad.canonical.com/SourceSourceRefactoring
<ddaa> look at the bottom of the page
<ddaa> also look for ImportStatus in canonical/lp/dbschema.py
<mpool> i see
<mpool> and the public server pulls from the internal http server
<ddaa> yes
<mpool> but if the branch already exists on the public server, how will the upgrade propagate?
<mpool> pull does not do an upgrade
<mpool> iirc
<ddaa> https://launchpad.canonical.com/VcsImportsKnitsUpgrade
<lifeless> I will comment, but I have to finish a meeting first
<mpool> oh i see, right
<mpool> it's not doing a regular pull
<ddaa> if an upgrade is needed, importd will 1. pull 2. upgrade locally 3. recursive delete remote branch 4. push before running cscvs
<mpool> "
<mpool> The branch-puller preserves changes in branch format, so format changes on the vcs-imports internal publication server will be propagated to  http://bazaar.launchpad.net."
<ddaa> mpool: that issue has been addressed already
<mpool> right but how does the branch-puller do that?
<ddaa> if the branch puller sees that the source and the target have different formats (in any way), it will delete the target and do a branch from scratch.
<mpool> ok
<mpool> so the branch will just disappear from b.lp.n until the upgrade is pulled across?
<mpool> i guess that's ok
<ddaa> yes
<ddaa> maybe there's some special provision to reduce the window of loss of service, but it's not really important
<spiv> After all, one of the nice things about distributed version control is that temporary outages aren't totally disruptive.
<ddaa> so, it might very well be that the required changes to importd are small
<ddaa> but I'm not fluent in bzrlib
<ddaa> and there tends to be annoying integration issues in importd
<ddaa> so it's not a low risk task
<ddaa> I'd say medium risk
<mpool> i don't think the interruption is a big deal, i just wondered
<mpool> one other question: presumably all of this is backed up?
<ddaa> There's also the fact that the conversion is expected to be a big deal processing wise
<mpool> so if something goes wrong and it starts deleting branches...
<mpool> ?
<ddaa> mh...
<ddaa> dunno, I expect it is backed up... but access to the repositories are quite restricted
<ddaa> and it's a bit hard to go _that_ wrong
<ddaa> so, big deal processing wisu
<ddaa> definitely not something you want to run on a single system
<mpool> i think it'd be sensible to check before starting
<mpool> the spec looks reasonable to me
<ddaa> I do end-to-end tests with the whole system on my workstation before deployinh disruptive code like that
<ddaa> There's usually a minor tweak or two needed when deploying things, but there was never a major disaster.
<mpool> cool
<ddaa> So, you want multiple systems running the stuff
<ddaa> that turns a small one-off hack into a relatively large mess
<ddaa> and format conversion is not one-off
<ddaa> it's going to happen again in the future
<mpool> yeah, it makes sense
<mpool> so, essentially we are trying to prioritise the upgrade spec against
<mpool> - unjamming the python import
<mpool> - cleanups of arch code
<mpool> what else?
<ddaa> - critical fixes I'm working on now
<mpool> ok
<ddaa> I particular, I think it makes sense to do the upgrade thing after fixing https://launchpad.net/products/launchpad-bazaar/+bug/37897
<mpool> so it looks like the upgrade mechanism is largely independent of other things - e.g. removing arch dependencies won't make it much easier, or vice versa?
<ddaa> because that will allow use to handle the load by putting all four importd systems (including the ones currently running autotest) temporarily in importd-production
<ddaa> I'm happy to proritise that before arch cleanups.
<ddaa> since it can be put into the bzr-specific code that gets the target branch
<ddaa> might not be the optimal way to organise the code, but that would avoid messing up the pattern of arch/bzr testing
<mpool> ok
<mpool> well, let's see what robert says
* ddaa realises that putting all importd in production will require some admin operation to set up ssh keys and db access, but that should not be a big deal
<ddaa> lifeless can give support to the relevent requests
<ddaa> And it's something I'll need eventually, since I want to remove the need for importd-autotest altoghether (we can now to that since we no longer have to deal with arch namespace issues)
<lifeless> ok
<lifeless> I'm back
<lifeless> where are we up tp?
<ddaa> need your input or prioritizing knits upgrade
<lifeless> ok
<lifeless> so I was going to ask
<lifeless> isn't the upgrade as simple as invoking bzr upgrade via python *always* on every branch ?
<lifeless> invoke it on both the local branch and the remote one, before pulling etc etc
<ddaa> hu
<ddaa> remote upgrade?
<ddaa> it's on LAN, but still...
<ddaa> if we put it down to that, it will be indeed just a couple of lines of code
<ddaa> since we currently download the branch anew for every sync
<ddaa> but really do not like the idea
<ddaa> for example, how will "bzr upgrade" behave if the first upgrade is interrupted (process killed, loss of connection), and we try to run a second one?
<ddaa> BTW, need to make a bug importd not breaking sftp locks
<ddaa> it should be able to do that
<lifeless> bzr upgrade takes a backup copy of data. It marks the branch as being in-upgrade
<lifeless> so it will need manual recovery- but this is IMO better than the failure mode of the upgrade you proposed
<lifeless> where it will have no data remotely, and no data locally (because it will be trying to pull anew)
<ddaa> gn?
<ddaa> remote data would only be deleted after a successful local upgrade
<ddaa> oh right
<lifeless> if we download the branch anew each time, my suggestion is just 'invoked bzr upgrade on the remote branch before pulling it'
* ddaa wrote that spec before reading about crash-only software
<lifeless> which should be trivial to implement, and result in automatic upgrades as we change the defaults
<ddaa> What does importd need to do to recover from an interrupted upgrade?
<lifeless> well, there are two failure types
<lifeless> a) bzr bugs
<lifeless> b) network glitch
<ddaa> c) kernel upgrade
<lifeless> for b), its - rm -rf the .bzr dir, and move .bzr.backup to .bzr
<lifeless> c) counts as b)
<ddaa> anh other variants of c, like importd upgrade
<ddaa> yeah
<lifeless> for a), you need to diagnose it and work with the bzr devs to get the upgrade to work
<ddaa> how can you automatically detect b?
<lifeless> IMO you should treat them all as a), because theres no magic way to tell what caused the failure
<ddaa> you realise that "needs manual diagnostic" for import pretty much means "will not be fixed"
<lifeless> is jamesh's log analysis stuff rolled out for importd ?
<ddaa> I ran it once
<lifeless> if so, then this should be marked clearly in that report
<ddaa> but not again, because there were just too many failures to deal with
<lifeless> and as any bug in this stuff is of critical importance to bzr, I dont agree that needs manual diagnostic means 'will not be fixed' 
<ddaa> and there was no easy way to classify them
<mpool> ok, well
<mpool> i'd like to call it a night...
<lifeless> jamesh: is there something that could be easily done to make flagging failed upgrades stand out hugely in your log analyser for importd ?
<lifeless> mpool: what time for spiv and I tomorrow ?
<mpool> presume you're coming here?
<mpool> after 10?
<lifeless> I presume that too
<lifeless> k
<lifeless> spiv: ^?
<ddaa> lifeless: does bzrlib gives a specific exception when trying to upgrade a branch that failed upgrade already (and is thus marked upgrade-in-progress)?
<lifeless> ddaa: yes 
<ddaa> okay, that should be easy to grep for
<lifeless> ddaa: UnknownFormat
<mpool> what effect does an ascii del have on a spiv? i hope it doesn't kill him :-)
<lifeless> ddaa: or ~ that - check bzrlib/errors.py ;)
<lifeless> ddaa: ok, does this sound like a workable plan then ?
<ddaa> Sounds workable.
<ddaa> Then _when_ ?
<lifeless> ok, cool.
<lifeless> yesterday :)
<ddaa> ASAP, or when we can put all our cpus on it?
<lifeless> IMO ASAP. upgrade will chew through over a couple of days
<lifeless> no need for massive parallisation
<lifeless> it wont block all imports indefinately, because each import will block just long enough to upgrade, then do a pull from CVS
<ddaa> yeah
<lifeless> ok, gnight then
<ddaa> once we get in the tail of the exponential distribution, (the 5 biggest branches), the other imports should be allowed to sync again
<ddaa> lifeless: BTW
<ddaa> jamesh appears to busy to review my last importd-bzr-native
<lifeless> ddaa: not the case
<ddaa> okay
<lifeless> ddaa: was discussed in the review meeting, see irc logs or lp-reviews
<lifeless> gnight
<ddaa> nigh
<ddaa> night
* ddaa -> lunch
<spiv> mpool, lifeless: soon after 10 sounds good to me.
#launchpad-meeting 2006-08-10
* Starting logfile irclogs/launchpad-meeting.log
-ChanServ(ChanServ@services.)- You do not have channel operator access to [#canonical-ops] 
* #canonical-ops is desynced from zelazny.freenode.net at 09:21am
<carlos> hi
<SteveA> hi
<SteveA> hi
<danilos> hi
<SteveA> so, stub and I were just talking about the downtime required to run the rosetta updates
<danilos> yeah
<carlos> SteveA: do we have info about when is launchpad less used ?
<SteveA> stub tells me that with the current code, we're talking about many hours' downtime
<SteveA> I have a few comments about this that we should consider
<SteveA> 1. we don't want to take down all of launchpad for N hours in order to do rosetta updates
<carlos> between 2 and 4 hours
<SteveA> 2. is it not possible to do the updates incrementally?
<SteveA> about 1,
<stub> 2-4 hours per run, or 2-4 hours for both runs?
<carlos> stub: per run
<SteveA> if we need to do lots of rosetta database work, we can take down just rosetta,
<carlos> that's on staging
<carlos> and it depends on the load of the server, because sometimes is done in 2 hours and other times in 3 hours and a half
<SteveA> by changing the authorization adapters to deny all rosetta edits
<SteveA> and changing the "permission denied" page to say about rosetta being under maintenance
<SteveA> so, that's one way of managing this
<SteveA> so, the rest of launchpad (shipit, bugs, etc.) can stay running
<danilos> I think that might be the best way out
<SteveA> but
<carlos> yeah, at least for this initial run
<carlos> next time, for edgy + 1
<carlos> will be rosetta and soyuz
<SteveA> let's talk about why this needs to be done all in one go
<danilos> on the other hand, it will take longer that way, no?
<SteveA> why can't we run it a little at a time
<SteveA> with pauses in between?
<SteveA> for example, one package at a time
<stub> I think Steve means one package at a time with commits in between
<carlos> hmm
<danilos> well, it was designed in such a way that it can be run that way, since it also supports incremental updates; carlos, isn't that so?
<carlos> yes, it is
<carlos> not per package
<carlos> but per table
<carlos> doing it per package will require to add some extra code
<carlos> the problem of doing it per table
<carlos> is that at some point people will start seeing some inconsistent data
<carlos> but I don't think they could actually break anything
<danilos> well, then the per-package seems easier and safer
<carlos> we don't point to edgy translations so the amount of people looking at that data would be really small
<carlos> danilos: but also much more slow
<SteveA> that's okay
<SteveA> it's okay for it to take a week
<stub> yup
<SteveA> if it means no interruptions in service
<danilos> well, but it will lock only a package at a time
<carlos> SteveA: but it will only be useful for this two runs we need now
<danilos> and each will be locked for like a really small amount of time
<SteveA> if one package is locked for 30 minutes, once
<SteveA> then that's fine
<carlos> edgy + 1 will not need that approach, right?
<stub> It will lock the tables it is inserting into as normal, but the locks will be for a much shorter period and probably unnoticible.
<SteveA> 3 minutes is even better than 30
<danilos> yeah, I think it will be far less than 30 minutes
<SteveA> but this is a much better approach for our service overall
<stub> If it takes 30 mins to do the inserts for one package, or even 1 minute, then I don't know if this approach is doable on the live system.
<carlos> oo.org would take more than 5 minutes...
<carlos> it's a monster...
<SteveA> so, we could do the large packages in a maintenance period
<SteveA> we need to find out how long an average-sized package will take
<carlos> SteveA: hmmm
<SteveA> otherwise, look for a smaller granularity of commit still
<carlos> I really prefer if we close rosetta
<carlos> this change is going to delay edgy translations again
<carlos> and I'm already getting lots of complains about it
<carlos> the changes we are talking about would take more than a couple of days between code changes, testing on staging, code reviews and landing on production
<carlos> + the time to deploy it
<carlos> so that would be end of next week in the best case
<danilos> well, this is actually true, so it all depends on the timing
<SteveA> stu and i agree that we can take rosetta down
<SteveA> if you guys can write some security code that
<SteveA> has a launchpad.conf option
<SteveA> so that when this option is set
<SteveA> permission is denied to everything except for launchpad.Admins
<carlos> even for the pages that are read only ?
<SteveA> and to either
<carlos> I mean, the ones that doesn't need to be logged in
<SteveA> 1. raise a custom subclass of ForbiddenAttribute or whatever
<SteveA> or 2. customize the forbidden oops page (and authorization required, or whatever)
<SteveA> we need to do this for every page that allows writing to rosetta
<danilos> so, carlos, what do you think about this?
<SteveA> we'll test on staging
<SteveA> by locking the rosetta tables
<carlos> if it's just for teh pages that allow writing
<SteveA> and checking that we can't lock up launchpad
<SteveA> just for edit pages
<carlos> the first part is quite easy
<SteveA> I mean, we could do this in the forms instead
<SteveA> instead of the security code
<carlos> the second part... is the first time I do it so I'm not sure which option is better
<SteveA> if there are only a few rosetta forms that allow updating rosetta tables
<SteveA> that might be easier
<carlos> there should be 4 or 5 different forms
<SteveA> raise a RosettaDownForMaintenance error
<danilos> well, we also need to block imports, no?
<SteveA> yes
<SteveA> and do an error page for RosettaDownForMaintenance
<carlos> danilos: that's just a matter of disable the cron job
<SteveA> we'll be making the tables read only to the launchpad user anyway
<SteveA> so the database-level stuff will be okay
<SteveA> but we want to give nice errors and not OOPSes
<danilos> ok, just taking notes of everything
<carlos> ok
<danilos> so carlos
<danilos> what do you want us to do? rosetta downtime, right?
<SteveA> just to check something
<SteveA> when we were in london together
<SteveA> we talked about updating the stuff into a temporary table
<SteveA> and then in a separate transaction, or separate translactions, moving them across to the real tables
<SteveA> is that the approach you've taken?
<danilos> mostly, I think carlos put the temp table creation into the same transaction (so it doesn't get out of sync)
<SteveA> um
<SteveA> that kinda obviates the point
<carlos> danilos: yes, I would prefer to do the downtime
<danilos> temp table is reading only, so it can probably be moved out without too much problems
<danilos> carlos: ok, then lets do that as painlessly as possible
<danilos> carlos: btw, didn't we get any improvements with temp tables? do we have indexes in there?
<stub> So build all the inserts into tables. Then, a few rows at a time, insert some data into the production tables, delete from the temporary store, and commit.
<carlos> danilos: indexes helped a bit
<carlos> but not too much
<carlos> SteveA, stub: I guess that it could make us to lose some translations for the migration (anyone added after the temporary table is created)
<stub> carlos: That would have happened anyway, as the migration script would be using a snapshot of the database at the time the migration was started
<carlos> stub: well, if rosetta is closed
<danilos> stub: it gets a bit more complicated because of references and us updating table-by-table
<carlos> no new translations will be added, right?
<carlos> also, we should block any import from poimport script for edgy or as soon as the potemplate table is filled, entries from the queue will start to be importing
<carlos> giving us conflicts with the temporary table
<carlos> What problem do you see If we close Rosetta ?
<carlos> I think that's doable and in the future, that feature would be interesting  in case of a performance problem like the ones we had in the past so the rest of launchpad is not also affected
<danilos> well, we want to go for that anyway
<carlos> In that case, I think that closing rosetta two days
<carlos> one per migration
<carlos> is not a big problem, if we choose the right time when users are not doing many translations
<SteveA> stu and i are discussing various things...
<carlos> ok
<SteveA> ok
<SteveA> here's the plan
<SteveA> stu will try the update code on a clone of production, on the production server
<SteveA> turning off indexes and shit like that
<SteveA> we'll get an accurate estimate of the amount of downtime
<danilos> ok
<SteveA> meanwhile
<SteveA> we can assume it will take at least 2 hrs
<carlos> SteveA: should I start working on the 'disable Rosetta' feature ?
<SteveA> so you guys need to add a switch to launchpad.conf
<carlos> danilos: I think I should work on it as I'm just switching tasks and you already have a lot of open things
<SteveA> that causes rosetta edit views etc. to raise RosettaDownForMaintenance when the view is viewed
<SteveA> as well as edited
<SteveA> and an error view for RosettaDownForMaintenance
<SteveA> and we need to test this on staging, by stu turning off write access to rosetta tables
<carlos> sounds good to me
<carlos> stub: the script should be run with '-d ubuntu -r dapper' the first time and the next one with '-d ubuntu -r edgy'
<danilos> carlos: sure, be my guest :)
<carlos> the first one will migrate translations from breezy to dapper and the next one will copy all translations from dapper to edgy
<flacoste> hi SteveA
<SteveA> hi salgado 
<SteveA> hi flacoste 
<salgado> hey!
<flacoste> i think the spec you were looking for was https://launchpad.canonical.com/LaunchpadI18n
<flacoste> actually, that we should have looked for
* SteveA waits for stub
<SteveA> https://launchpad.net/products/launchpad-support-tracker/+spec/localized-support
<SteveA> https://launchpad.canonical.com/LaunchpadI18n
<SteveA> first, let's talk about the overall point of LocalizeSupportRequests
<salgado> I pasted that on #canonical, didn't I?
<salgado> okay
<SteveA>     *
<SteveA>       Grandma Lucille, who doesn't speak English, was navigating on the web until she reached a flash page, whose content she can't see because the flash plugin is not installed. She'll then click on "Help -> Get Help Online ..." and make a support request in Launchpad.
<SteveA>     *
<SteveA>       Malba Tahan, a very nice guy who knows a lot about Ubuntu is willing to share his knowledge by answering some support requests. He goes to the Support facet of Ubuntu and look for open requests. He sees a list of requests written in one of his preferred languages, pick up a randon one in Arabic and answers it.
<SteveA> 
<SteveA> these are use-cases from the spec
<SteveA> so, we have requirements around identifying which support requests are written in a particular language
<SteveA> and of allowing people to file support requests in their own language
<SteveA> is that right, as overall goals?
<salgado> right
<SteveA> so, the support tracker needs to be able to track various things about what language the request is made in
<flacoste> exactly
<SteveA> and that's self-contained in the support tracker application
<SteveA> I totally trust you guys to do a good job there
<flacoste> that was the plan there
<SteveA> what I'm concerned about is how this touches other parts of launchpad
<salgado> what would these various things about the language be?
<SteveA> and that a good plan for the support tracker may cause problems elsewhere
<flacoste> the spec also covers what we needed form the launchpad infrastructure
<SteveA> first of all, some overall concerns
<SteveA>  - if we localize a page element that appears on the support pages, but also elsewhere
<SteveA>    then we get partially localized pages in other parts of the application
<flacoste> (that was covered indirectly in the spec)
<salgado> was it?
<flacoste> workaround) Translate strings for support. Display a message when going to untranslated section of the site.Mark template as translated or not. Force English when using an untranslated template.
<SteveA> and then, all the unresolved issues in https://launchpad.canonical.com/LaunchpadI18n
<SteveA> as far as managing risk goes...
<SteveA> i think we should split the support tracker goals up into two specs
<SteveA> 1. making the support tracker support multiple languages of support issues, but with its UI remaining untranslated
<SteveA> 2. translating the launchpad UI, with the first parts to be translated being the support-tracker and the login workflow
<SteveA> to make 1. a 1.0 goal, but not 2.
<SteveA> because 2. touches so much stuff, and puts an unexpected burden on the infrastructure work
<SteveA> that doesn't mean we can't get to 2. for 1.0, but it means that we aren't committing to it
<SteveA> what do you all think?
<salgado> I think that'd be quite awful, usability-wise, but I can understand it
<flacoste> i agree that posting a question in my native language while the UI is in English doesn't look too good, but I can understand the idea of doing this two-steps
<salgado> okay, now we need to convince mark of that, right?
<SteveA> I'd really plan this as three specs
<SteveA>  - make the support tracker work in multiple languages for the support requests, inc. data model changes etc.
<stub> Last time we were looking at this, it seemed like the i18n would be postponed indefinitely. To participate in open source development, you needed to have at least basic English skills. I'm unsure if other parts of Launchpad will ever want to be localized, and it will be unfortunate if there is fallout in the rest of Launchpad to support localized support request tracking.
<SteveA>  - make launchpad support internationalization
<SteveA>  - make the support tracker and login / registration system internationalized and localized 
<SteveA> we have a dependency relationship between these specs
<SteveA> two of the specs are in the domain of logins and support tracking
<SteveA> one is in the domain of infrastrcuture
<SteveA> now, in terms of planning 1.0 features
<SteveA> as leader of the infrastructure team, I'm being asked to support a new 1.0 requirement
<SteveA> so we need to look at resources for that
<stub> The spec mentions searching in particular. Our search infrastructure does not support searching in non-English languages. It might work by accident in many cases, but there will be glitches we cannot solve without features being implemented upstream.
<SteveA> now, I can imagine a work-around of sorts
<SteveA> that is, to ignore the login / signup parts
<SteveA> becuase people manage to log in and sign up to get shipit CDs todasy
<SteveA> so i expect they *can* do so for support requests
<flacoste> stub: from a developer point of view you are right, but not from an Ubuntu user point of view
<SteveA> and to localize just the body of support pages
<SteveA> using the language the user is interacting with the support system
<flacoste> SteveA: what do you think of the 'Mark template as l10n idea'?
<flacoste> to prevent half-balked localization in the other part of the system
<SteveA> I think that's an interesting idea to solve the problem of having a rendered page consistently translated or not
<SteveA> that's just one part of the whole infrastructural issue
<flacoste> indeed
<SteveA> what I would ask for next is this:
<SteveA>  - split up the specs into the three I mentioned
<SteveA>  - take into account the existing spec about i18n for the infrastructural aspects
<SteveA>  - write into that spec the minimum necessary to deploy an internationalized / localized support tracker
<SteveA> then we can come back and look closely at what we end up with
<SteveA> and decide if this is do-able for 1.0
<salgado> sounds good to me
<salgado> I can do that
<stub> Cool
<SteveA> ok
<flacoste> flacoste: side note: in your infrastructure spec you forgot to mention how to handle string interpolation
<SteveA> I think the support-only first part can be done immediately
<SteveA> I mean, not blocked
<stub> The search issue I mentioned should be listed on the spec. We may need to contract upstream if searching in different languages is particularly bad. 
<SteveA> we should have an idea of what languages we're targetting too
<stub> I suspect the search results will get worse the further the language gets from the standard roman alphabet.
<flacoste> stub: there is UTF-8 support in 8.1 or 8.2 i think
<stub> flacoste: tsearch2 has utf8 support, but it still is language dependant for stemming etc.
<stub> (in 8.2, which doesn't exist yet, or CVS head)
<flacoste> stub: indeed, but there is a 8.1 backport of the utf8 stuff i think
<stub> And all our helpers force queries to ascii only, so that will all need to be rewritten.
<stub> (because of the existing tsearch2 limitation)
<flacoste> stub: ideally, we would also have to add support to the fti to index tickets based on their language
<SteveA> my gut says we really should try to internationalize launchpad within the next 6 weeks
<SteveA> um
<SteveA> i mean
<SteveA> my gut says we really should not try to internationalize launchpad within the next 6 weeks
<flacoste> SteveA: yes, a list of supported language would be nice - who can decide on that?
<SteveA> I can see it being a lot of rushed effort
<SteveA> with flaky results because it is rushed
<flacoste> stub: (OT: i have another fti question for you after the meeting)
<SteveA> stu may have time to help on this
<SteveA> but we need to look into this
<SteveA> thanks salgado and flacoste !
<SteveA> salgado: please tell me when we should look at the specs again
<flacoste> To summarize:
<flacoste> - salgado will merge the infrastructure part of LocalizedSupportRequest into LaunchpadI18n
<flacoste> - salgado will move login workflow related changes to a separate spec
<flacoste> - Infrastructure will review these specs to see if LaunchpadI18n and LaunchpadLoginL10n can be targetted for 1.0
<flacoste> - add string interpolation example to LaunchpadI18n (flacoste)
<flacoste> Other unassigned issues:
<flacoste> - need to add search issue to LaunchpadI18n
<flacoste> - determine list of languages to target
<flacoste> should I mail the list with this?
<SteveA> also
<SteveA> we should have one spec for LocalizedSupportRequest
<SteveA> and one spec for LocalizedSupportTracker
<salgado> yeah, it will be kept, I assumed
<SteveA> so that we can represent the dependencies well
<salgado> ah, right
<SteveA> thanks for summarizing flacoste 
<flacoste> my pleasure
<flacoste> should I mail this summary of our discussion to the list (after including SteveA update)
<flacoste> ?
<SteveA> mailing the launchpad list will be good.  it needs a bit of an introduction as well, of course
<flacoste> no problem
<SteveA> where my points for that are
<SteveA> that it is good to make the support tracker localized
<SteveA> it is in a way a special part of launchpad
<SteveA> in that it is used by end-users who don't necessarily speak english well
<SteveA> in a way this is like shipit
<SteveA> it is unlike rosetta (translators always speak english)
<SteveA> and unlike malone
<SteveA> and unlike the spec tracker
<SteveA> (programmers always speak englilsh)
<SteveA> I generalize a little
<SteveA> but these are generally true
<SteveA> we must take seriously the task of internationalizing launchpad, because it has an effect on various areas of what we do
<SteveA> including
<SteveA>  - code reviews (reviewers need to know what to check for and understand the mechanisms that will be appearing throughout the code)
<SteveA>  - test running (what locale / language do we run tests in ? )
<SteveA>  - rollouts (how do we get PO files / generate POT files for doing new rollouts? )
<SteveA>  - web request infrastructure (is the standard stuff in zope3 good enough for our purposes)
<SteveA>  - cookie / database language preferences (new UI, new database work, new infrastructure)
<SteveA>  - searching infrastructure (maybe doesn't cope with languages well)
<SteveA>  - checking the current stuff about internationalizing python in zope, zcml, page templates
* flacoste will put these points in his message
<SteveA>  - having consistently translated pages (maybe using francis' ideas)
<SteveA>    (but that implies changes infrastructure to support it)
<SteveA> 
<SteveA> there may be a middle-ground
<SteveA> of just internationalizing certain page, like the support tracker pages
<SteveA> and not worrying about having a full experience for support users
<SteveA> oh, also
<SteveA>  - internationalizing email we send out
<SteveA>     like for the sign-up stuff
<flacoste> and the ticket notifications
<SteveA> right
<SteveA> we should have a standard way of doing that
* flacoste isn't sure if we covered that in the spec
<SteveA> and really, that's a spec in itself
<SteveA> "internationalized email"
<SteveA> as a management thing, we need to decide about doing all this in a planned and well-thought-out way
<SteveA> or rushing it to completion over the next 6 weeks, fixing issues as we come across them
<flacoste> (am i  to understand that 1.0 is scheduled for in 6 weeks?)
<SteveA> and dealing with the fall-out later.   I'm keen on avoiding a rush.
<SteveA> something like that
<SteveA> 1 oct or so i think
<SteveA> and people have vacations too
<SteveA> 
<SteveA> I think that's all
<SteveA> any comments ?
<flacoste> not from me
* sivang expects lots of RTL issues for RTL languags.
<sivang> so make sure generilzation is made to include those languages for good support just as the latin ones.
<sivang> (for example, emails need be RTL, input fields and the whole ballgame)
<SteveA> depends if RTL languages are in the list of ones we're focusing on
<sivang> true
<sivang> probably better start with the latin ones, which would incur less fallout and more quick fixable issues I suppose , and only then advance further.
<flacoste> SteveA: i added a MessageId interpolation example to LaunchpadI18n
<SteveA> great
<flacoste> we have a bug with notification currently for those strings, its bug 54987
<flacoste> the bug report include a fix, I can update the tests and submit the fix for review if you care, otherwise it can wait
<SteveA> stub should look at that
<SteveA> I'll assign it to him
<flacoste> great
#launchpad-meeting 2006-08-11
<SteveA> hi
<SteveA> so, what's the news about the rosetta update plans?
<carlos> SteveA: the code to do the migration is on rocketfuel already
<danilos> hi
<carlos> so I guess Stuart could start testing it on a production mirror as we talked yesterday
<SteveA> ok
<SteveA> I just spoke with stuart
<SteveA> he can do it toay
<carlos> SteveA: today, I'm going to write a spec about Rosetta in read only mode and start its implementation
<SteveA> today
<SteveA> ok
<SteveA> tell me when the spec is ready, and I'll review it
<carlos> ok
<danilos> sure
<carlos> SteveA: about the spec for Edgy translations
<danilos> btw, do you want a complete spec for edgy opening as well, or a stub on +specs marked critical is enough?
<carlos> I guess just a small document describing what we are going to do is enough, right?
<SteveA> a braindump saying what we're doing
<SteveA> that there's a script that does (short description)
<SteveA>  - test on carbon to see how long it takes
<SteveA>  - write add-on to allow us to put rosetta into r/o mode
<SteveA>  - test r/o mode on staging
<SteveA>  - annonce r/o downtime (where??)
<SteveA>  - include text of the announcement
<danilos> ah, so it will also be documentation
<SteveA>  - plan when to do it
<SteveA> what do you think of that?
<carlos> SteveA: it's fine for me
<danilos> sounds fine
<danilos> we might also want to document the approach we took in actually migrating
<danilos> just the big picture, of course
<SteveA> sure
<carlos> danilos: I will write an initial document for it and ask you to review/complete it. Is that ok for you?
<carlos> that way we don't miss anything
<danilos> carlos: yeah, of course
<danilos> carlos: and be sure to make a dependency on your planned rosetta-ro spec
<carlos> indeed
<carlos> SteveA: should we talk about anything else? or can we move to do it?
<SteveA> is there anything else that you discussed?
<SteveA> stuart is thinking of coming to visit you sometime
<SteveA> so you can work together on the rosetta data model
<carlos> sure
<carlos> I have a guests room
<carlos> and If they don't mind, danilo and stuart could share the room (two beds, don't worry ;-)
<carlos> SteveA: I don't think we discussed anything else
<danilos> :)
<danilos> sure
<SteveA> ok, great
<SteveA> let me know how it does
<SteveA> um, goes
<carlos> sure
<ddaa> crap
<ddaa> importd-bzr-upgrade blows up
<ddaa> https://chinstrap.ubuntu.com/~dsilvers/paste/fileURhZlT.html
<ddaa> anybody has a clue to what causes that problem?
<ddaa> mh...
<ddaa> oops
<ddaa> wrong chan
#launchpad-meeting 2008-08-06
<barry> #startmeeting
<MootBot> Meeting started at 09:04. The chair is barry.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<barry> hello everybody and welcome to this week's ameu reviewer's meeting.  who's here today?
<sinzui> me
<gmb> me
<bac> me
<allenap> me
<salgado> me
<intellectronica> me
<barry> EdwinGru EdwinGrub EdwinGrubb EdwinGrubbs is here 4 times i guess :)
<intellectronica> barry: BjornT is away on holiday
<EdwinGru> me
<barry> intellectronica: thanks
<barry> bigjools: ping
<bigjools> hello!
<bigjools> sorry
<barry> no worries!  hi
<barry> danilos: ping
<cprov-out> me
<barry> anybody else i'm missing?
<barry> cprov-out: hi
<bac> jtv?
<barry> he only shows up sometimes because of his weird timezone
<barry> sometimes he's in asiapac
<bac> ah, ok.
<cprov> barry: hi there
<barry> i think we're just missing abentley but i pinged him in #lp-code (tho he's marked away)
<barry> okay, let's get started!
<barry> [TOPIC] agenda
<MootBot> New Topic:  agenda
<barry>  * Roll call
<barry>  * Next meeting
<barry>  * Action items
<barry>  * Queue status
<barry>  * Mentoring update
<barry>  * Review process
<barry>    * dogfooding launchpad? (barry)
<barry>    * too many branches? (barry)
<barry> [TOPIC] next meeting
<MootBot> New Topic:  next meeting
<barry> week += 1
<barry> anybody know of sprints or other events that week?
<barry> cool
<barry> [TOPIC] action items
<MootBot> New Topic:  action items
<barry>  * intellectronica to write up guidelines on check_permission in the wiki and email the ml for additional input
<intellectronica> sorry, one more week passed without me doing that
<intellectronica> please let's carry this over to next week
<barry> intellectronica: will do
<barry>  * barry to ping jamesh and stub on the testsuite2 branch
<barry> done, but i have not heard back yet
<barry>  * barry to update PreMergeReviews
<barry> not done, and now i have to go back and try to remember what this was about ;)
<barry> vacations tend to blank my mind
<barry>  * intellectronica to start a list on the wiki of devs/available platforms
<intellectronica> see above :(
<barry> k
<barry>  * intellectronica to start the ball rolling on an email to the ml re: multiple browsers/platforms
<barry> same? :)
<intellectronica> yup
<barry> k
<barry> [TOPIC] queue status
<MootBot> New Topic:  queue status
<sinzui> I see I got a branch while I was away
<barry> i'm a bit concern that we're having trouble keeping up with reviewer branches.  there's 9 in the GQ
<intellectronica> we had lots of sprints lately, as well as the release
<sinzui> I think sprinting really undermines our reviewing
<bac> i think i was only able to get two off the GQ yesterday.
<bigjools> I get the feeling that OCR does the bulk of reviewing now, which is unfair on the OCRs
<barry> sinzui, intellectronica agreed
<barry> bac: i'm sorry that i didn't help matters :/
<gmb> bigjools: You mean the GQ reviewing?
<gmb> Or just in general?
<bigjools> non-GQ
<gmb> But that's what they're there for isn't it?
<bigjools> I mean that for reviewers not on call, they don't get much to do
<gmb> Ah, right.
<bigjools> s/not/never/
<barry> bigjools: my concern is that ocr reviews are consuming a full day so gc branches tend to get neglected.  is that a fair observation?
<bigjools> just an observation....
<bigjools> I concur
<bac> barry: yes
<gmb> barry, bigjools: Why don't we do an allocation run of the GQ today to get it cleared and mail the list. Better that we get all hands on deck to avoid the building.
<gmb> Buildup, even.
<bigjools> +1
<sinzui> I have had opposite days
<barry> gmb: +1.  allenap and cprov, ping me when you're done for the day and i'll do an assignment cull
<allenap> barry: Okay.
<gmb> sinzui: Yes, but you meke up for it by being the Friday Wk3 guy ;)
<cprov> barry: k
<sinzui> and saturday
<bigjools> here's an idea, can we make the review-submit output put the number of lines of diff in the PR stanza?
<bigjools> it would make it easier to distribute GQ items based on LOC workload
<barry> bigjools: that's a good idea.  i'd think that should be fairly easy
 * gmb remembers when the server side tool was written to do exactly that... Oh well.
<barry> bigjools: for this allocation run, i'll try to use pending-reviews to even things out
<bigjools> gmb: server-side tool?
<barry> i just want to get a feel for your thoughts: is the current backlog a symptom of process breakdown, or just a temporary situation caused by sprints, releases, and vacations?
<gmb> bigjools: At AllHands mwh and I hacked a client-server setup for review management together. The server side bit never got used though because of the (then) upcoming features in LP.
<gmb> barry: I *think* it's a result of all the sprints.
<bigjools> gmb: ah ok
<gmb> For example, the bugs sprint last week took 4 reviewers (well, 3 + a team lead) out of play.
<bigjools> barry: I think it's both - because OCRs are responsible for allocating the GQ at the end of the shift, when there's no OCR that process sufferes
<gmb> And the foundations sprint was bigger in terms of effect.
<barry> gmb: foundations was similarly occupied two weeks ago (and sinzui and i were on vacation last week)
<barry> bigjools: yes.  i also think there's more tension lately between ocr and gc
<bac> bigjools: i thought we agreed that the OCR would *not* allocate the GQ at the end of the shift anymore.
<barry> i'm going to think about that some this week, but if you have any comments, email or irc them to me
<sinzui> bac: We did agree
 * sinzui may have gotten his branch because it is JS.
<bigjools> ah, so who's doing it now?
<bigjools> my memory sucks
<barry> bigjools: the plan was to consume gq branches during lulls in ocr, but there have been any such lulls
<barry> bigjools: so the gq isn't getting consumed (or not enough to keep up with demand)
<barry> at least lately
<bac> barry: do the asiapac guys stay pretty busy with on call reviews?  are they working on the GQ otherwise?  just wondering...
<barry> bac: i'm not really sure, but that's a good question to ask them
<bigjools> I feel that the GQ should be allocated daily to ensure that OCRs don't do everything
<sinzui> I don't care for allocation. User who put something in the GQ are still obligated to see that their branch is reviewed.
<bac> we need to encourage the reviewers who aren't OCR to periodically take a branch, too.  generally those people are the team leads, who are incredibly busy, but a branch or two a week would help
 * bac admits they may be doing so already and i just haven't paid attention
<bigjools> sinzui: how do you chase a review if it's in GQ?
<barry> i don't much like allocation either, but i really don't want ocrs or really any reviewer to get burned out or feel compelled to review more than their fair share
<barry> i remember kiko saying at all-hands, if we have too many branches than can reasonably be reviewed, then that's a different problem that has to be solved
<bigjools> perhaps we need all reviewers to do OCR then?
<barry> (or something like that)
<sinzui> bigjools: I note that cpov is pretty good at finding a reviewer for his branch if it has not moved from the GQ in 24 hours. Maybe he know I can't say no.
<bigjools> :)
<barry> :)
<cprov> :)
<bigjools> if the dev has to always find a reviewer like that then the GQ is redundant
<barry> i'll just note that it's probably difficult for jtv to request an ocr
<bac> some developers throw stuff on the GQ and then ignore it.  i did one yesterday that was 10 days old or so
<sinzui> it is, I often pull his branch out for review because I want to see that old translations code purged.
<barry> bac: yes, you're right
<bac> barry: that's not really true.  jtv is in an odd time zone but he works really odd hours.  i do OCR for him all the time.
<barry> bac: maybe it's just mondays then :)
<barry> [ACTION] barry to do a gq allocation at the end of today
<MootBot> ACTION received:  barry to do a gq allocation at the end of today
<barry> let's move on...
<barry> [TOPIC] mentoring update
<MootBot> New Topic:  mentoring update
<cprov> wise decision.
<barry> any comments from mentors or mentorees?
<barry> i had a good conversation with abentley about his reviewing, though i guess i have to ping him about this meeting :)
<barry> he's just starting really
<bigjools> I'd like to commend cprov for his quality reviews
<barry> cprov: yay!
<cprov> bigjools: oh, thank you so much.
<barry> i think we currently have only 2 mentored reviewers.  do we have the bandwidth (and candidates) for maybe one more?
<barry> anyway, if you have suggestions, send them my way
<barry> [TOPIC] review process
<MootBot> New Topic:  review process
<barry>    * dogfooding launchpad? (barry)
<barry> so i talked to thumper before vacation about using merge proposals in lp for our reviews.  we all want to kill PR, so i'm wondering, even with the missing features (e.g. diff) is anybody up for an experiment to use lp to manage our reviews?
 * bigjools is up
<sinzui> me is
<gmb> +1
<bac> +1
<cprov> +1
<barry> i think the one thing you'll have to do is do the diff on the client side, but that shouldn't be too hard
<bigjools> a lot of us do that anyway
<barry> bigjools: good point
 * bac generates his own diff
<gmb> That's only a problem with dependant branches.
<bigjools> anything to drive a stake into PR and pour garlic and holy water over it is fine with me
<barry> i think we also need to ask a few devs to use lp instead of review-submit
<barry> but we don't want everyone to rush to lp yet :)
<barry> gmb: right.  so let's cut child branches out of the experiment for now
<bigjools> how easy is it?  are there any demos/instructions?
<barry> perhaps what we can do is invite each dev to submit one non-dependent merge proposal, and then have a few of us use it when we're ocr
<barry> bigjools: i've used it for public, hosted code, but not for private, remote branches
<barry> bigjools: but that's part of the experiment i guess ;)
<bigjools> barry: are you volunteering to write a howto then? :)
<barry> bigjools: yep, i'm willing to figure it out and write it up
<bigjools> you da man!
<barry> [ACTION] barry to figure out dogfooding lp and writing it up
<MootBot> ACTION received:  barry to figure out dogfooding lp and writing it up
<barry> then i suggest we go with the 6 volunteer reviewers (including myself) for a few weeks and see how it goes
<barry> sound good?
<bigjools> I will help you
<barry> (with the invite for devs to submit one standalone branch to lp)
<barry> bigjools: awesome
<barry> cool.  we have 5 minutes left.  i think we've basically talked about the other item i had on the list.  so i'll open the floor up to any other comments
<barry> if not, then we're done!
<barry> #endmeeting
<MootBot> Meeting finished at 09:45.
<barry> thanks everyone!
<bigjools> cheers!
#launchpad-meeting 2008-08-07
<matsubara> moo
<al-maisan> :-)
<Rinchen> ah what the hey
<Rinchen> #startmeeting
<Rinchen> Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development.
<MootBot> Meeting started at 13:03. The chair is Rinchen.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<Rinchen> cluck
<sinzui> thunk
<bigjools> moo
<Ursinha> me
<barry> squeek
<Rinchen> oh yeah
<Rinchen> [TOPIC] Roll Call
<MootBot> New Topic:  Roll Call
<mrevell> me
<Rinchen> cluck
<allenap> me
<adeuring> me
<bac> oink
<matsubara> me
<mars> me
<barry> me
<EdwinGrubbs> me
<sinzui> thunk
<Ursinha> me
<matsubara> the launchpad farm
<bigjools> baa
<salgado> me
<Rinchen> releases team is here
<Rinchen> even though we're excused!
<Rinchen> Translations won't be here
<al-maisan> me
<Rinchen> bjorn and francis are excused
<stub> me
<Rinchen> mthaddon is also excused
<thumper> me
<Rinchen> barry, can you report on foundations for me
<stub> jtv and danilo are on a sprint
<barry> Rinchen: sure
<intellectronica> me
<Rinchen> intellectronica, can you check on the bugs team please
<salgado> foundations is here too
<Rinchen> rockstar ping?
<barry> leonardr: ping
<rockstar> me
<Rinchen> ok code is here
<mars> leonardr, meeting?
<leonardr> me
<abentley> me
<bigjools> cprov: ping
<salgado> apart from leonardr, that is. :)
<salgado> no we're complete
<barry> foundations is here
<Rinchen> soyuz?
<salgado> now
<Rinchen> I saw al-maisan.  cprov? bigjools ?
<bigjools> one missing, Celsooooo?
<thumper> code is here
<Rinchen> thanks
<Rinchen> [TOPIC] Agenda
<MootBot> New Topic:  Agenda
<cprov> me
<Rinchen> you'll notice that mrevell is no longer reporting on his sections by design
<Rinchen> these will now be brought up on demand
<Rinchen>  * Next meeting
<Rinchen>  * Actions from last meeting
<Rinchen>  * Oops report (matsubara/ursinha)
<Rinchen>  * Critical Bugs (Rinchen)
<Rinchen>  * Bug tags
<Rinchen>  * Operations report (mthaddon/herb/spm)
<Rinchen>  * DBA report (stub)
<Rinchen>  * Sysadmin requests (Rinchen)
<Rinchen>  * New packages required (salgado)
<Rinchen> * user affecting issue (intellectronica)
<Rinchen> [TOPIC] Next meeting
<MootBot> New Topic:  Next meeting
<Rinchen> next week same time
<matsubara> I won't be here
<Rinchen> k, thanks
<Rinchen> anyone else
<Rinchen> Francis will as well
<Rinchen> [AGREED] same time same place on the 14th.  flacoste and matsubara are excused
<MootBot> AGREED received:  same time same place on the 14th.  flacoste and matsubara are excused
<Rinchen> [TOPIC] Actions from last meeting
<MootBot> New Topic:  Actions from last meeting
<Rinchen> there are none recorded
<Rinchen> [TOPIC] Oops report (matsubara/ursinha)
<MootBot> New Topic:  Oops report (matsubara/ursinha)
<Ursinha> hi all
 * Ursinha waves
<Ursinha> 5 bugs for today: 174480, 252695, 251569, 255816 and 246591
<Rinchen> (rockstar, bring your boat - flash floods here today)
<Ursinha> Someone of foundations is needed to take a look at bug 255816. Anyone?
<ubottu> Launchpad bug 255816 in launchpad "ComponentLookupError accessing /manage" [Undecided,New] https://launchpad.net/bugs/255816
<Ursinha> two bugs for salgado: bug 252695 and bug 251569, how are these?
<ubottu> Launchpad bug 252695 in launchpad "+peoplelist taking too long to render and timing out" [Undecided,In progress] https://launchpad.net/bugs/252695
<ubottu> Launchpad bug 251569 in launchpad "UnicodeEncodeError when trying to search in launchpad/people/+index" [Undecided,Confirmed] https://launchpad.net/bugs/251569
<barry> i can look at 255816
<Ursinha> barry, nice, thanks
<salgado> Ursinha, 252695 is on PQM
<salgado> Ursinha, didn't have a chance to look at the other
<Ursinha> salgado, when you have the time, please :)
<matsubara> barry, notice that the fix might be similar to whatever we did to fix bug 54944
<ubottu> Launchpad bug 54944 in launchpad "ComponentLookupError accessing ++skin++ page" [Low,Invalid] https://launchpad.net/bugs/54944
<barry> matsubara: thanks
<Ursinha> ok
<Ursinha> bug 246591, thumper, have you checked this out with mwhudson?
<ubottu> Ursinha: Bug 246591 on http://launchpad.net/bugs/246591 is private
<Ursinha> grrr
<thumper> Ursinha: he hasn't been online since we chatted last night
<Ursinha> thumper, oh
<thumper> Ursinha: I'll chase this morning
<Ursinha> thumper, ok, thanks
<Ursinha> the last one
<Ursinha> intellectronica, have news on bug 174480?
<ubottu> Launchpad bug 174480 in blueprint "Person's +roadmap page contains blueprints they're not assigned to" [Undecided,New] https://launchpad.net/bugs/174480
<intellectronica> Ursinha: sorry, not yet
<Ursinha> intellectronica, would be appreciated too :)
<Rinchen> [AGREED] barry to look at 255816, salgado has fix for 252695 in PQM, salgado to look at 251569, thumper to poke mwh about 246591, intellectronica to take on 174480.  Thanks guys!
<MootBot> AGREED received:  barry to look at 255816, salgado has fix for 252695 in PQM, salgado to look at 251569, thumper to poke mwh about 246591, intellectronica to take on 174480.  Thanks guys!
<Ursinha> that's all folks
<Ursinha> thanks!
<matsubara> thanks all!
<Rinchen> [TOPIC] Critical Bugs (Rinchen)
<MootBot> New Topic:  Critical Bugs (Rinchen)
<Rinchen> [LINK]https://bugs.edge.launchpad.net/launchpad/+bug/251869
<Rinchen> needs an owner
<Rinchen> [LINK]https://bugs.edge.launchpad.net/malone/+bug/253970
<Rinchen> Edwin, how is this progessing
<MootBot> LINK received: https://bugs.edge.launchpad.net/launchpad/+bug/251869
<MootBot> LINK received: https://bugs.edge.launchpad.net/malone/+bug/253970
<Rinchen> and I've already spoken to jtv about
<Rinchen> [LINK] https://bugs.edge.launchpad.net/rosetta/+bug/255758
<ubottu> Launchpad bug 251869 in launchpad ""not allowed here" while merging to accounts" [Critical,Confirmed]
<MootBot> LINK received:  https://bugs.edge.launchpad.net/rosetta/+bug/255758
<ubottu> Rinchen: Error: This bug is private
<Rinchen> Starting next week, QA will be taking over this section! Yippee :-)
<ubottu> MootBot: Error: This bug is private
<ubottu> Rinchen: Error: This bug is private
<ubottu> MootBot: Error: This bug is private
<Rinchen> EdwinGrubbs, ^^
<Rinchen> salgado, I know you have a lot on your plate. Is there someone who could look at 251869 though?
<Rinchen> it's marked as CRITICAL
<Rinchen> I'm not so sure it is
<salgado> that's exactly what I was going to say
<salgado> I don't agree it's critical
<EdwinGrubbs> Rinchen: It has already been cherry picked. Sorry I forgot to mark the ticket Fix Released.
<salgado> it has prevented a couple users from merging their accounts, and in both cases a LP admin was able to do the merge for them
<Rinchen> EdwinGrubbs, excellent. Please include the RF number in there if it is not already.
<Rinchen> ok salgado I've lowered that one
<Rinchen> ok, that's it from me.
<Rinchen> Anything else before I move on?
<Rinchen> [TOPIC] Bug Tags
<MootBot> New Topic:  Bug Tags
<Rinchen> we have one
<Rinchen> https://help.launchpad.net/TaggingLaunchpadBugs
<Rinchen> two actually
<Rinchen> first up, mrevell - tour tag
<mrevell> Hello. We have bugs filed against the launchpad-documentation project that cover several things - e.g. user guide, tour, and so on.
<mrevell> It would be handy to be able to group together anything relating to the tour as we then hand the changes off to a specific developer/outside designer.
<intellectronica> i think the tour should be a product
<sinzui> +1
<Rinchen> sinzui, on the tag or the product suggestion?
<mrevell> intellectronica: Why would a product be more suitable?
<mpt> It doesn't have separate source code
<sinzui> We don't need a tag, the tour is autonomous from launchpad apps
<intellectronica> mrevell: because it's not a subclass of bugs of any one lp product. it's a pretty self contained piece of functionality
<Rinchen> I think mrevell's point is that it already has a product, called launchpad-documentation
<sinzui> it is /more/ separate than answers or bus
<Rinchen> but there are other items that use that product as well
<intellectronica> that's not the same thing
<Rinchen> and he'd like to have a way of keeping them separate
<Rinchen> is that right mrevell ?
<intellectronica> the tour isn't really documentation, it's more marketing
<sinzui> and we update and review the tour differently form code
<mrevell> Yes Rinchen. I'm thinking of also suggesting "inline-help" and "user-guide" tags in the future.
<Rinchen> ok, let's not get ahead of ourselves :-)
<mpt> Ideally, where a change to Launchpad code affects what the tour should say, the same branch should update that part of the tour at the same time.
<mrevell> intellectronica: It's docu,mentation for prospective users :)
<intellectronica> inline-help and user-guide are documentation bugs, i agree
<Rinchen> thumper, do you have any comments?
<thumper> about a product for the tour?
<matsubara> I think we won't have that many bugs on the tour anyway, so that doesn't justify having its own product
<Rinchen> thumper, yes...or the tag.
<thumper> I don't think the current lp projects should be projects
<thumper> so I'm -1 on a product, +1 on a tag
<matsubara> I'm +1 on the tag, -1 on the product as well (but I'm biased :-)
<thumper> sorry for mixing products and projects there
<Rinchen> mrevell, do we really have a lot of bugs that would require a tag (let alone a product)?
<intellectronica> thumper: i don't look forward to a day when i'll have to dig out the relevant bugs from a long list of all lp bugs
<Rinchen> you didn't include any examples which is why I ask
<mpt> intellectronica, so fix the bugs that make tagging unnecessarily difficult :-)
<intellectronica> :)
<matsubara> +1 mpt
<mrevell> Rinchen: The tour bugs come when there's a major tour update. So, there were a few bugs after the most recent roll-out - but it is a *few*, ten maybe fifteen at most.
<mpt> The longer those bugs go unfixed, the more people are tempted to do silly things like splitting the same bunch of source code into separate "projects"
<mrevell> Rinchen: But they come at one time.
<mrevell> I suggest a tag so that it's easy for me to look at what needs fixing on the tour and then I can go to our external designer and get a quote.
<Rinchen> ok
<intellectronica> while i'd prefer a tag, i think that mrevell should ultimately decide on this one, since he's managing this project, and he will benefit (or suffer) from the change the most
<Rinchen> given that the amount of bugs are not very high and it currently fits well within it's existing product, I don't think a new product is the right thing in this particular instance
<Rinchen> I also do not think a tag would be warranted but I have "been there, done that" when dealing with the tour and the provider so I'm +1 on a tag for it's existing product
<Rinchen> intellectronica, well, he did. He wants a tag :-)
<thumper> JFDI
<mrevell> thumper: my thoughts exactly :)
<allenap> +1
<Rinchen> ok. so carried
<mrevell> thanks all for your input
<Rinchen> [AGREED] tour tag for the launchpad-documentation product approved. mrevell to update the tags page
<MootBot> AGREED received:  tour tag for the launchpad-documentation product approved. mrevell to update the tags page
<Rinchen> ok, for tag #2...
<Rinchen> matsubara, rca
<matsubara> one of the things we decided to do during the sprint is to have a root cause analysis for critical bugs, and we would use a bug report to hold information about the RCA. so the tag is basically to help us group those together. I'm ok on re-proposing this tag once we have the process in place and a initial RCA bug as an example.
<barry> it's an interesting idea, but can we pick something other than an abbreviation?
<thumper> barry: rooted?
<Rinchen> so the team leads spoke about RCAs on last week's call
<Rinchen> and the TL's agreed we should pilot this
<Rinchen> and the QA group is going to do this for Critical bugs (only)
<barry> thumper: sounds like pwnd :)
<barry> thumper: maybe root-cause?
<Rinchen> the process needs to be updated a bit so...stay tuned and we'll have that out to everyone
<thumper> Rinchen: I like root-cause better than rca too
<matsubara> barry, fine by me.
<abentley> thumper: Yeah, I guess Root Cause Analysis has a different flavour in some dialects.
<thumper> abentley: :p
<Rinchen> Any objections?
<Rinchen> ok. thanks
<thumper> Rinchen: only to rca, not the concept
<Rinchen> [AGREED] root-cause tag approved. matsubara to update the tags page.
<MootBot> AGREED received:  root-cause tag approved. matsubara to update the tags page.
<Rinchen> [TOPIC] Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  Operations report (mthaddon/herb/spm)
 * Rinchen pokes herb 
<Rinchen> ok
<Rinchen> well
<Rinchen> we'll move along then
<Rinchen> [TOPIC] DBA report (stub)
<MootBot> New Topic:  DBA report (stub)
<stub> The trunk will be open for database landings tomorrow, and the remaining db patch reviews done assuming this conjunctivitis doesn't get bad.
<stub> One landing will be the branch that gives us master and slave stores to play with.
<stub> You can explicitly use the slave store to speed things up when your code doesn't need to write to the db and doesn't mind if the data is slightly lagging. I'm sure we can all think of many examples that will benefit from using this.
<stub> You can explicitly use the master store when you need to make modifications or the data absolutely has to be up to date.
<stub> Existing code all uses the default store, which is set by the publisher based on the type of request - read only requests will get the slave store as their default, which should offload at least 30% of our load to the replica databases (when they are live).
<stub> oot.
<Rinchen> This is great news stub
<Rinchen> any questions for stub?
<barry> stub: what's the name of the slave store?
<thumper> stub: any idea what the delay will be like getting updates from the master to the slave(s)?
<stub> getUtility(IStoreSelector).get(LPMAIN_STORE, SLAVE_FLAVOR)
<barry> stub: thx
<stub> thumper: 2 seconds to minutes - we won't know until we see it under real load.
 * thumper nods
<stub> (which we will do before switching the load balancing on)
<Rinchen> stub, running this on demo or staging ahead of time and asking the lp community to test this would be a brilliant idea
 * thumper wonders about post redirects
<stub> Yes, but that still isn't real load. It is a hint, but not a true picture.
<thumper> if we have a post that goes to writable store, which does a redirect post saving to a get, will that go to a copy that doesn't have the update?
<sinzui> stub, does the master/slave negotiation for requests know about @safeget?
<stub> thumper: The current load balancing implementation will keep connections using the master store for 5 minutes after a POST request
<thumper> cool
<stub> sinzui: No, because I don't know about it either.
<Rinchen> any more questions for stub?
<stub> sinzui: There is at least one hole in the algo at the moment, so it wouldn't work if we switched it on today (webservice requests all need the master at the moment). This branch is landing the infrastructure though, and enough for testing on demo and staging.
<stub> sinzui: @safeget Sounds like something else the load balancing needs to be aware of as well.
<sinzui> I meant @safe_action that we use with get requests.
<sinzui> I was wondering if those submits will be using master when I think they should always use the slave
<stub> sinzui: It depends if we can make an inteligent decision that early in the publication.
<Rinchen> thanks stub.  I'm going to move on....
<Rinchen> in the interest of time
<Rinchen> [TOPIC] Sysadmin requests (Rinchen)
<Rinchen> Is anyone blocked on an RT or have any that are becoming urgent?
<MootBot> New Topic:  Sysadmin requests (Rinchen)
<thumper> Rinchen: there is one
<thumper> Rinchen: but I don't know the number
<bigjools> Rinchen: I can't remember the numbers right now but I have a couple for private PPAs
<Rinchen> last week it was  31340, 31296, and 31389
<thumper> Rinchen: it was about moving a staging code import box to production environment
<Rinchen> ok, if you guys could email those to me please I'll see what I can do
 * thumper nods
<Rinchen> sprinting this week so it's been rather hard for me to chase those
<Rinchen> thanks
<Rinchen> [TOPIC] New packages required (salgado)
<MootBot> New Topic:  New packages required (salgado)
<salgado> anything for me this week?
<Rinchen> ok, thanks salgado
<Rinchen> so, let's slow things down a little  ;-)
<Rinchen> [TOPIC] user affecting issue (intellectronica)
<MootBot> New Topic:  user affecting issue (intellectronica)
 * Rinchen pokes intellectronica 
<intellectronica> in discussions with the ubuntu community, it has come to my attention that one issue that makes life very difficult for them, is launchpad being slower than they think is usable
<intellectronica> ouch
<intellectronica> it occured to me, that we often don't know that pages become unusably slow until they start timing out
<Rinchen> we here this from folks in China as well.  We have experimented with non-ssl connections but it has not improved anything greatly
<intellectronica> (or until users report bugs)
<intellectronica> i think it's db queries in most cases, not the transport
<stub> We are still chasing hard timeouts. Ideally these would be 0 and we are chasing soft timeouts instead. I think many of us tend to think of soft timeouts as normal or maybe early warnings rather than problems that need fixing.
<intellectronica> so i have no idea how to tackle this, but would like to invite you all to think about this and maybe start a discussion on what we can do to track the speed of requests and improve things before they become serious problems
<matsubara> yeah, we bring those soft time outs up from time to time, but they are postponed as not yet critical
<thumper> I know on the code side, a usability point is the number of clicks / edits needed
<thumper> so it is more of a workflow thing
<thumper> rather than a webapp speed thing
<intellectronica> matsubara: yeah, we can think of those as usability issues, rather than critical bugs
<intellectronica> thumper: no, i think they are talking about page load times, not interaction
<thumper> intellectronica: oh, ok
<abentley> thumper: Rationally, yes.  Emotionally, high latency feels depressing.
<mpt> I have read that people who complain about slow page load times think that a site has become much faster when the design has improved but the speed has not
 * thumper is used to slowish pages from NZ
<sinzui> translations were hoping for AJAX to make it's page feel faster.
<stub> I suspect slow pages are those bugs that will never be looked at given our current priorities. Maybe now 2.0 is out we can look at polishing what we have over driving new features?
<mars> intellectronica, there was a little discussion about pulling the page times from the Apache logs.  That could give you some hard data.
<intellectronica> mpt: well, they were talking about this yesterday quite a lot, so obviously this hasn't affected their perception :)
<mpt> obviously what hasn't?
<intellectronica> mars: that, i think, is a really good thing to do. especially if it's done periodically and graphed
<intellectronica> mpt: obviously the new design hasn't changed the user's perception of speed
<Rinchen> the problem with load times it that it doesn't account for delay incurred over the wire.
<Rinchen> pages load for me in the USA about 10 times slower than in London
<mpt> intellectronica, I wouldn't expect it to, the 2.0 changes are tiny usability-wise
<intellectronica> Rinchen: sure, but given that there's little we can do about the transport, why worry about it?
<mars> yep.  I don't remember if there is a way to pull the full page load time though - not just the HTML, but scripts, images, etc. in aggregate.
<Rinchen> intellectronica, because I think a good portion of the issue is geographically induced
<intellectronica> mpt: indeed. and i agree that in-page interaction would improve things /a lot/
<Rinchen> we still serve the same amount, but depending on where you are, it may be fast (London) or slow (China)
<intellectronica> Rinchen: so maybe we should aim at researching that too?
<mars> intellectronica, btw, response times could also become a big issue when we add more AJAX.
<intellectronica> mars: how so?
<Rinchen> I've had some initial conversations with upper management about that in fact.  But yes, we should definitely look into ways of improving that
<mars> sure, it's AJAX, it's snappy, but that doesn't help if the XHR takes 1000ms to run :(
<beuno> mars, ajax will reduce the amount of unnecessary queries you do. I do agree slow ajax is even worst
<beuno> (and hi everyone)
<intellectronica> mars: well, you won't be waiting on the entire page, as you do now
<mars> intellectronica, yes, true
<Rinchen> My advice at the moment is that everyone here consider this topic for a bit and bring up ideas to your team leads (and also intellectronica).
<bigjools> Some of our pages carry a lot of data which can cause a lot of queries.  When optimising for fewer queries it's still easy to not improve the speed because the code becomes the slow part, so we need to be mindful of both code speed and number of queries.
<Rinchen> We have a TL meeting coming up in a few weeks. It would be an interesting time to discuss it
<mrevell> It's worse when an AJAX request goes slowly because you can't easily reload the page to start again.
<abentley> intellectronica: Sometimes JavaScript itself can take a long time to load.
<intellectronica> abentley: you mean js runtime? that depends on the client, but we should never reach the point where this can be an issue
<Rinchen> thank intellectronica for bringing this up.  I happen to think it's very important
<abentley> intellectronica: No, I mean the JavaScript code for a particular web site.
<Rinchen> I'd like to table the JS discussions until later.  Anything else for intellectronica before we close?
<Rinchen> ok then.
<Rinchen> Thank you all for attending this week's Launchpad Developer Meeting. See the channel topic for the location of the logs.
<Rinchen> #endmeeting
<MootBot> Meeting finished at 13:58.
<intellectronica> thanks Rinchen
<allenap> Cheerio everyone.
<mrevell> thanks Rinchen
<thumper> ta
<al-maisan> bye!
#launchpad-meeting 2009-08-05
<bac> #startmeeting
<MootBot> Meeting started at 09:00. The chair is bac.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<bac> hello everyone and welcome to the AMEU reviewer's meeting.
<bac> who is here today?  team leads will you help me round up your folks who may be tardy?
<gary_poster> me, me me!
<bigjools> me
<rockstar> ni!
<noodles775> me
<henninge> ich
<leonardr> me
<kfogel> hey
<sinzui> me
<bigjools> soyuz non-vacationing/non-seconded team members all correct and present SAH
<abentley> me
<bac> sinzui, EdwinGrubbs, salgado: registry team ping
<rockstar> Code's here.
<bac> sorry sinzu
<bac> i
<sinzui> o
<sinzui> u
<salgado> me!
<EdwinGrubbs> me
<bac> thanks bigjools
<allenap> me
<mars> me
<bac> henninge: any more bug folks?
<bigjools> bac: o7
<bac> will flacoste be joining us?
<flacoste> i am
<flacoste> me
<henninge> bac: you mean translations? danilos?
<bac> welcome back flacoste!
<bac> henninge: yeah, that's what i meant.  sorry.
<bac> allenap: any of your cohorts around?
<allenap> bac: I've just pinged them all... let's see..
<jtv> me
<intellectronica> me
<danilos> ok, ok, me too
<danilos> :)
<bac> welcome special guest kfogel
<bac> [TOPIC] agenda
<MootBot> New Topic:  agenda
 * jtv applauds kfogel
<bac> * Roll call
<bac> * Apologies: None
<bac> * Action items
<bac> * Mentoring update (leondardr/rockstar)
<bac> * Peanut gallery (anything not on the agenda)
<kfogel> thanks bac, jtv
<bac> [TOPIC] * Action items
<bac> * Karl to update wiki about sponsoring contributions and take RFC to the mailing list.
<MootBot> New Topic:  * Action items
<kfogel> So, kfogel would like to know if that item is done or not.
<kfogel> I wasn't sure what the original spec was; I improvised.  Thoughts, folks?
<bac> Thanks kfogel for taking care of that item.  I read it and it looks good.
<kfogel> bac: thank you.
<salgado> where is it?
<bac> Did anyone else get a chance to read karl's updates?
<intellectronica> no, url?
<kfogel> (most of the material was there already, actually -- barry and I put it there long ago)
 * kfogel finds urls...
<bac> https://dev.launchpad.net/FixBugs
<kfogel> okay, three:
<kfogel> https://dev.launchpad.net/Patches
<kfogel> https://dev.launchpad.net/FixBugs
<kfogel> https://dev.launchpad.net/Reviews
<kfogel> they interlink
<bac> kfogel: are these all new pages or just additions to existing ones?
<kfogel> the first two are directly linked from the top nav box on the home page
<kfogel> bac: FixBugs is a new page
<kfogel> bac: I updated Patches (and renamed it from PatchSubmission); I don't think I tweaked Reviews much.
<bac> i think this is a good start.
<intellectronica> i think we should avoid using the word "patch", because it might confuse people that are familiar with diff/patch but not with dvcs
<bac> the process and our description on the wiki will evolve but i appreciate karl writing the first cut.
<kfogel> I'd like to know what's missing; my email to the list didn't get much reaction AFAICT.
<kfogel> Happy to add more (also happy for people to edit directly, of course!).
<bac> intellectronica: do you have a suggested wording?
<intellectronica> bac: branch
<kfogel> intellectronica: I didn't quite understand: how will "patch" confuse people in that situation?
<intellectronica> or changeset, if you want to talk about it in the abstract
<kfogel> "change" ?
<intellectronica> kfogel: they might think that we want them to produce a patch file, rather than a bzr branch
<abentley> So I thought the idea was that when you wanted to fix a bug, for example, you found a sponsor and they guided you through the process, starting with some facimile of a pre-implementation call.
<kfogel> intellectronica: ah.  One way might be to simply explain at the top of the Patches page that the end result is a bzr branch, not an actual patch file.
<kfogel> I think "patch" might still be the term of art for changes.
<intellectronica> kfogel: yeah, i guess just stressing that might be enough
<kfogel> abentley: if that's the idea, I'd like to suggest that we be more lightweight.  It's certainly okay to do that, but people should also feel free to do more lightweight process...
<EdwinGrubbs> The name "Patches" doesn't explain whether they can get patches on that page or learn how to submit patches.
<bac> EdwinGrubbs: yeah, i think i prefer the old name PatchSubmission
<kfogel> EdwinGrubbs, bac: I can revert, easy enough.
<abentley> I try to refer to submitting a "change", rather than a patch.  Even that's broken, though, because we deal in branches, and branches represent a state, not a change.
<kfogel> so ACTION items so far: rename back to PatchSubmission; explain at top of page that 'patch' means 'branch'.  I do think sticking with the accepted term for patches would be a good idea.  It's what everyone uses, even when they're not trading actual patch files.
<intellectronica> abentley: +1 i think change or changeset are good terms which are much less ambiguous
<abentley> intellectronica: bzr doesn't have changesets, so I don't think that's a good term to use.
<jtv> "improvement"?
<intellectronica> abentley: we could call it revision, which is what bzr actually has, no?
<abentley> intellectronica: But multiple revisions are often submitted.
<kfogel> intellectronica: changeset is a technical term in VC; also, having tried to use it with lots of people, I find I often have to explain it.  Many people already know "patch"; those few who don't, once they understand it, will be permanently enabled across the software ecosystem.
<intellectronica> anyway, this turning into a theological discussion. sorry for starting. continue on the list?
<kfogel> Let's reinforce the existing term, not try to promote a new term.  (This is an old debate and an old effort, and it hasn't succeeded in dethroning "patch" yet, IMHO.)
<kfogel> heh
<deryck> I'm not sure people really get tripped up over thinking "patch" literally means a patch.  The generic use of the term seems common to me.
<bac> intellectronica: yes, good idea.
<kfogel> okay, let's go with the two ACTION items above.  Any others so far?
<bac> thanks again to karl.  please follow up to his email if you have further discussion.
<bac> [TOPIC] * Mentoring update (leondardr/rockstar)
<MootBot> New Topic:  * Mentoring update (leondardr/rockstar)
<bac> rockstar: you have the floor
<rockstar> leonardr has done an amazing job as a manatee since day one. His reviews are insightful and exploratory. I'm happy to announce his graduation.
<leonardr> yay
<rockstar> Congratulations leonardr
<bac> whee
<bac> congratulations leonard.
<noodles775> congrats leonardr !
<bac> with his graduation we are mentat-free, i believe.
<kfogel> leonardr++
<bac> have we reached the goal of every developer being a reviewer?
<rockstar> I believe so.
<bac> excellent.  i think that plan was hatched in our MIT sprint.  congratulations everyone.
<flacoste> and barry isn't here to join the fun!
<flacoste> congratulations leonard
<bac> [TOPIC] * Peanut gallery (anything not on the agenda)
<MootBot> New Topic:  * Peanut gallery (anything not on the agenda)
<bac> does anyone have another topic to discuss?
<bac> thanks for coming everyone.  please do read the new wiki pages in detail and offer feedback.
<bac> #endmeeting
<MootBot> Meeting finished at 09:21.
<abentley> bac: thank you!
<jtv> thank you bac!
<mars> thanks bac
#launchpad-meeting 2009-08-06
<Ursinha> me?
<sinzui> you
<sinzui> us
<intellectronica> Ursinha: no, not you
<sinzui> them
<Ursinha> ah
<Ursinha> :(
<matsubara> sorry
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<Ursinha> roll call,roll call
<matsubara> my firefox died
<sinzui> me
 * stub belches
<matsubara> hang on a second please
<henninge> me
<Ursinha> poor matsubara
 * jml eavesdrops
 * bigjools wafts stub's belch away
<rockstar> ni
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<matsubara> [TOPIC] Roll Call
<MootBot> New Topic:  Roll Call
<Ursinha> meeee
<bigjools> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
<stub> me
<henninge> me again
<intellectronica> i
<matsubara> flacoste, hi
<flacoste> me
<matsubara> herb, hi
<herb> me
<matsubara> ok, everyone here.
<matsubara> [TOPIC] Agenda
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (stub)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * matsubara to chase rockstar about failure on updatebranches script
<matsubara>  * stub to give a try on bug 354593 with mars help if needed
<matsubara>  * stub to fix bug 310818
<matsubara>  * mars to take a look at OOPS-1307J16
<matsubara>  * Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606
<matsubara>  * mars and stub to discuss the Disconnection and OperationalErrors after the meeting
<jml> me
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<ubottu> Launchpad bug 310818 in launchpad-foundations "Oops report does not always log timed-out query" [High,In progress] https://launchpad.net/bugs/310818
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<Ursinha> yay, jml
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<matsubara> I suck, I didn't chase rockstar about the updatebranches script failures
<rockstar> matsubara, I thought we agreed that mwhudson would be better to chase on it.
<jml> matsubara, got a URL for the failure?
<matsubara> otoh, the script is not failing anymore...
<rockstar> matsubara, I know mwhudson was looking at it on his Tuesday.
<rockstar> jml would be good to ask as well.
<matsubara> rockstar, all right. I'll talk to jml and mwhudson later on today
<matsubara> [action] * matsubara to chase mwhudson/jml about failure on updatebranches script
<MootBot> ACTION received:  * matsubara to chase mwhudson/jml about failure on updatebranches script
<rockstar> matsubara, jml is here right now. :)
<matsubara> jml, I'll get you an url for the scripts after the meeting. I need to trawl my emails to find it
<jml> matsubara, ok. thanks.
<matsubara> stub, how's 354593 fix coming along?
<flacoste> why is this High again?
<matsubara> I wonder if mars had time to look over OOPS-1307J16
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<matsubara> flacoste, do you know ^?
<flacoste> hmm, i put it as such
<flacoste> any reason it should be?
<matsubara> flacoste, according to the bug history you made it high :-)
<flacoste> debranding of the SSO is a U1/ISD affair anyway
<stub> matsubara: Slow. I need to discuss with people how to actually do it - maybe next week on the sprint if I get time.
 * sinzui agrees with flacoste
<flacoste> stub: i think we should try to get stu and James to do it :-)
<flacoste> especially, stu, it would be a test good case for transfer knowledge
<stub> Anything that means I don't have to work out how ZPT macros works is fine by me.
<flacoste> +1
<matsubara> Ursinha, what's up with  "Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606"?
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<Ursinha> matsubara, the ExpatErrors were being discussed by mars and gary
<gary_poster> matsubara: that;s now registry.  it actualy is a legitimate oops
<matsubara> [action] stub to delegate bug 354593 to ISD
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<MootBot> ACTION received:  stub to delegate bug 354593 to ISD
<gary_poster> it indicates a problem with mailman integration
<sinzui> I will ask barry to look into bug 403606
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<matsubara> stub, You recently fixed a DisconnectionError bug. was it related to the errors you discussed with mars? that action item is now done?
<matsubara> thanks sinzui and gary_poster
<gary_poster> :-)
<stub> matsubara: I landed code to log OOPS reports on DisconnectionError before retrying the request. Is that what you mean?
<matsubara> stub, I mean: "* mars and stub to discuss the Disconnection and OperationalErrors after the meeting"
<Ursinha> stub, is that what caused the TransactionRollbackError oopses?
<stub> We discussed. I don't recall much about the conversation though :)
<matsubara> :-)
<stub> Ursinha: That fix was, yes. I've got another branch that turns the volume down so we don't log the TransactionCommitError's
<matsubara> [action] sinzui to ask barry to fix bug 403606
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<MootBot> ACTION received:  sinzui to ask barry to fix bug 403606
<Ursinha> stub, good, I filed bug 409907 for that
<ubottu> Launchpad bug 409907 in launchpad-foundations "TransactionRollbackErrors may prevent us to detect real issues" [Undecided,New] https://launchpad.net/bugs/409907
<matsubara> Ursinha, is there a bug for OOPS-1307J16?
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<Ursinha> matsubara, not that I opened one, because we needed to know what was going on over there
<Ursinha> to open the bug
<Ursinha> so mars was going to investigate that
<Ursinha> I don't recall having those anymore
<matsubara> [action] ursinha to chase mars about OOPS-1307J16 and file a bug about it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<MootBot> ACTION received:  ursinha to chase mars about OOPS-1307J16 and file a bug about it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<matsubara> I think that's all for last meeting's action items
<matsubara> thanks everyone
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> there are two issues to discuss
<Ursinha> one was about bug 409907, that I already mentioned to stub and it's being handled
<ubottu> Launchpad bug 409907 in launchpad-foundations "TransactionRollbackErrors may prevent us to detect real issues" [Undecided,New] https://launchpad.net/bugs/409907
<Ursinha> the other is about the select replication_lag() timeouts we're having
<Ursinha> mthaddon also reported problems that we don't know if are related to that
<Ursinha> I don't know if there's much to be discussed at this point, because it seems we need to fix oops reports first to be able to see the real problem here
<Ursinha> is that correct stub:
<Ursinha> ?
<matsubara> should we request a CP for the branch that fixes the oops log?
<intellectronica> given that we're skipping a release, that's probably a good idea
<Ursinha> flacoste, I've spoken with jtv yesterday about those,and he also said that was unlikely to be his changes fault (possible but unlikely)
<stub> I landed code today that should tell us more about if the timeout is actually occuring due to blocking on the database, or elsewhere.
<Ursinha> s/his/translations/
<Ursinha> stub, should we request a CP?
<flacoste> yeah, i really think a CP is a good idea
<Ursinha> (please please)
<matsubara> [action] stub to request CP for his branch that fixes oops logging
<MootBot> ACTION received:  stub to request CP for his branch that fixes oops logging
<Ursinha> cool
<Ursinha> we have two critical bugs, already fix committed
<Ursinha> so, good
<matsubara> cool
<Ursinha> about the failing scripts
<matsubara> we had some scripts failing this week
<matsubara> nightly, productreleasefinder and garbo-hourly
<matsubara> and rosetta-poimport too
<matsubara> nightly was already addressed by jtv
<Ursinha> matsubara, productreleasefinder isn't expected to fail anymore? sinzui?
<matsubara> as a rosetta script was taking too much time and jtv will remove it from nightly and add a cronjob for it
<sinzui> Ursinha: no, but the errors is see are not failures...the script was not run
<matsubara> stub, do you know why garbo-hourly is failing?
<stub> Its failing?
<sinzui> matsubara: many scripts are not running because of one log process
<matsubara> henninge, rosetta-poimport failed on the 5th. can you investigate and reply to the list?
<sinzui> s/log/long/
<Ursinha> matsubara, it's not being run, it seems
<henninge> matsubara: sure, I will.
<matsubara> stub, I got a few emails: "Scripts failed to run: loganberry:garbo-hourly"
<sinzui> Ursinha: matsubara there is some traffic about this. spm reported the long running prcess a weeks ago. I has asked why the prf had not run
<matsubara> and no replies to the list, so I'm asking here
<matsubara> thanks henninge
<Ursinha> matsubara, actually stub repklied
<Ursinha> *replied
<stub> Oh - there were some blocked runs because the rosetta export-to-branch script was running in a 5 hour long transaction
<stub> So the script blocks because it doesn't want to make anything worse.
<matsubara> [action] henninge to investigate rosetta-poimport script failure on the Aug 5th and report back to the list
<MootBot> ACTION received:  henninge to investigate rosetta-poimport script failure on the Aug 5th and report back to the list
<Ursinha> so I guess it's ok
<Ursinha> that's all for this section
<Ursinha> from me
<Ursinha> thanks everyone
<Ursinha> !
<Ursinha> you can move on matsubara
<matsubara> all right. thanks everyone
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> 2009-07-31 - Rolled out r8323 to bzrsyncd
<herb> 2009-08-05 - Cherry picks for code imports, lpnet* and the script server.
<herb> Our monitoring system has been timing out in connecting to the app servers more often this week. Admittedly its timeout is set lower than the OOPS timeout. But we've also been noticing higher load on the app servers as well. This was discussed by Ursinha during the oops/critical bugs/broken scripts section.
<herb> There's currently 1 cherry pick and 1 database query awaiting (dis)approval.
<herb> The LOSAs currently have 14 bugs marked high and triaged. Only 1 of which is assigned to someone and targeted for a release. We would be grateful if we saw some movement on these.
<herb> We're currently running with a single slave in preparation for the sprint next week.
<mthaddon> also wanted to check that there should be a cherry pick request for the cowboyed storm change to lpnet9 and lpnet10 (per the production status wiki page)
<flacoste> cowboyed storm change?
<mthaddon> flacoste: https://pastebin.canonical.com/20503/ under eggs/storm-0.14salgado_storm_launchpad_288_308-py2.4-linux-i686.egg
<flacoste> mthaddon, herb: i'll look at the LPS to approve/decline
<flacoste> right
<flacoste> mthaddon: the cherry pick would simply be to update that dependency
<matsubara> herb, do you keep that list of 14 bugs somewhere? in a wiki page or have a tag to group them?
<herb> matsubara: bugs.launchpad.net/~canonical-losas
<mthaddon> flacoste: well in any case, the CP that was requested (and performed) yesterday overwrote it, so it needs to be formalised so other CPs don't overwrite it again
<flacoste> sinzui: can salgado makes an appropriate CP request?
<sinzui> Yes
<flacoste> it's simply a new upload to download-cache with a versions.cfg change
<matsubara> sinzui, flacoste, intellectronica, rockstar: Could you take a look at herb's bug list (bugs.launchpad.net/~canonical-losas) and see what your teams can do about the high ones in the short term?
<flacoste> ok
<herb> clearly we're not looking for all of them to be fixed by the next meeting (though that would be great ;)
<herb> just mostly would like to know they're staying on the right radars and are being worked on as appropriate.
<matsubara> cool
<matsubara> anything else for herb?
<intellectronica> herb: so, basically, these are mostly bugs which will make life easier for you when fixed?
<sinzui> bug 348722 should become invalid when we update all pmt teams to become true private teams
<ubottu> Launchpad bug 348722 in launchpad-code "Set default branch visibility to "forbidden" if any team set to 'Private'" [High,Triaged] https://launchpad.net/bugs/348722
<herb> intellectronica: some of them are geniune operational issues, some of them are quality of life issues for the LOSAs
<sinzui> There should be no private-membership teams at the start of week 1
<intellectronica> cool, sure, we'll take a look and see if there's any low hanging fruit
<sinzui> barry will be working with the losas on August 11 to fix bug 325962
<ubottu> Launchpad bug 325962 in launchpad-registry "lp-mailman startup is blocking on a pid file in the wrong directory" [High,Triaged] https://launchpad.net/bugs/325962
<herb> sinzui: that was the one that was assgned and targetted at a release.
<sinzui> herb, many times
<herb> assigned even
<herb> heh
<matsubara> all right. I think that's it
<sinzui> herb it failed my rules that bug is not high if it is not worked on by all parties in 3 months
<herb> thanks
<matsubara> thanks herb and everyone
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> We set off some alerts when the poimport script and PostgreSQL decided that lots of disk space should be used. We see some smaller spikes, which is just PG using disk to store intermediary results, but this time it was large enough to set of the alarms.
<stub> We have seen this once before, and in neither case have we been able to repeat it. My best hypothesis is the planner statistics triggering a really bad query plan, so I'll bump the planner statistic sample size on the production dbs in case this stops future occurances.
<matsubara> henninge, maybe the last rosetta-poimport failure was related to that ^
<henninge> matsubara: I believe we already know what it was about and it may be related to that.
<henninge> matsubara: I'll talk to the guys.
<matsubara> henninge, cool. thanks
<matsubara> stub, anything else?
<stub> Not that I can think of
<matsubara> all right. thank you stub
<matsubara> I guess that's all for today
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
<matsubara> #endmeeting
<MootBot> Meeting finished at 10:44.
<herb> thanks everyone
<Ursinha> right on time
<matsubara> :-)
<Ursinha> thanks guys
#launchpad-meeting 2010-08-11
<bac> #startmeeting
<MootBot> Meeting started at 09:01. The chair is bac.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<bac> hi who is here?
<abentley> me
<adeuring> me
<deryck> me
<bac> sinzui just wandered off
<bac> danilos:  ping
<danilos> me
<bac> gmb, EdwinGrubbs, leonardr, jelmer ping
<leonardr> me
<bac> bigjools:  yo
<EdwinGrubbs> me
<henninge> me, too
<bigjools> me ish
<bac> flacoste, noodles775, mars: ping
<noodles775> here
<mars> me
<jelmer> me
<sinzui> me
<bac> [topic] agenda
<MootBot> New Topic:  agenda
<bac>  * Roll call
<bac>  * Agenda
<bac>  * Outstanding actions
<bac>  * New topics
<gmb> me
<bac>   * Format import lists so they merge well and are clear to review. RobertCollins
<bac>   * Call for new reviewers.
<gmb> Sorry
<bac>   * Removing enums and errors out of interfaces.  TimPenhey
<bac>   * Policy decisions: Reviewers Meeting or mailing list?  RobertCollins
<bac>  * Peanut gallery
<bac> [topic] outstanding issue
<MootBot> New Topic:  outstanding issue
<bac> there is just the one, coop vs shared vs inter.  that discussion has been happening on the mailing list so i don't see the need to bring it up here too.  so i won't.
<bac> [topic] Format import lists so they merge well and are clear to review. RobertCollins
<MootBot> New Topic:  Format import lists so they merge well and are clear to review. RobertCollins
<bac> lifeless brought this up on the list but i was quite confused by the discussion
<bigjools> this is the issue that came up a year (?) ago
<bac> we discussed it in the asiapac meeting last week and it turns out, he is advocating for the one-line-per-item in lists (as we do now) plus extending that to imports
<bac> which we discussed and rejected a year ago as bigjools said
<bac> robert and the code team are still very much in favor in this change.
<abentley> bac, true, dat.
 * henninge doesn't mind either way
<henninge> Actually, I see both formatings in our code.
<henninge> formats, formations, formatations ???
<henninge> ;-)
<bigjools> I'm not sure if I care which way it is, but I do care that the files are all the same
<gmb> bigjools, +1
<bac> bigjools +1
<henninge> Ouch, that is a lot of work.
<gmb> That's the biggest bugbear.
<flacoste> me
<bigjools> surely we can write a tool to reformat them?
<bac> if we do decide to make the change it'll be a big, mechanical branch.
<sinzui> we may have such a tool
<bigjools> JFDI
<bac> people who do big apocalyptic branches report seeing lots of merge conflicts over imports...i personally don't see it often
<sinzui> the migrater has a script that rewrite imports. I think we need to update its formatter
<bac> bigjools:  before we JFDI we have to decide if that's what we really want.
<bac> let's vote then.  are there any strong arguments for or against first?
<bigjools> if we have a tool, we can try it out and revert it if we hate it
<flacoste> i don't like it
<flacoste> but only esthetics argument
<henninge> Readability is surely on the one-liner side.
<bigjools> flacoste: what you gain on the editor you lose in the diff.
<henninge> with lists being alphabetical!
<abentley> bac, the arguments pro are fewer conflicts and clearer patches.
<flacoste> yep
<abentley> bac, clearer patches because only the thing added/removed will show, not the whole import block for that module.
<bac> would a single import stay on one line but two or more become a list?
<henninge> bac: I'd like that.
<danilos> we should just fix the very long "one-line" imports as an alternative solution (i.e. be more specific about what you are importing from)
<abentley> bac, that would be fine by me.
<abentley> danilos, I'm not sure what you mean by 'very long "one-line" imports'
<flacoste> i think he means multi-line import
<danilos> abentley, well, single-statement imports with lots of things being imported
<flacoste> from canonical.launchpad.interfaces import A, B, C, D... over 3 lines
<flacoste> if you import for more specific location most of them will fit on one line
<flacoste> but it doesn't solve the conflict problem
<flacoste> if you need to add a 3rd symbol
<flacoste> or a second one
<flacoste> or the readability of the patch
<danilos> right, it does reduce the chances of it happening though (you are going to get conflicts with the proposal to split them up as well)
<abentley> flacoste, I haven't noticed a lot of too-general imports lately, but maybe I'm not looking for them.
<flacoste> we don't add them
<bac> ok, let's vote
<sinzui> There are still a lot from c.l
<bac> [vote] Change our coding standards to require one line per item for imports
<MootBot> Please vote on:  Change our coding standards to require one line per item for imports.
<MootBot> Public votes can be registered by saying +1/-1/+0 in the channel, private votes by messaging the channel followed by +1/-1/+0  to MootBot
<MootBot> E.g. /msg MootBot +1 #launchpad-meeting
<danilos> -1
<MootBot> -1 received from danilos. 0 for, 1 against. 0 have abstained. Count is now -1
<abentley> +1
<MootBot> +1 received from abentley. 1 for, 1 against. 0 have abstained. Count is now 0
<bigjools> +1 because I fancy a change :)
<MootBot> +1 received from bigjools. 2 for, 1 against. 0 have abstained. Count is now 1
<adeuring> -1
<MootBot> -1 received from adeuring. 2 for, 2 against. 0 have abstained. Count is now 0
<gmb> +1
<MootBot> +1 received from gmb. 3 for, 2 against. 0 have abstained. Count is now 1
<henninge> +1 for readability
<MootBot> +1 received from henninge. 4 for, 2 against. 0 have abstained. Count is now 2
<noodles775> +0
<MootBot> Abstention received from noodles775. 4 for, 2 against. 1 have abstained. Count is now 2
<sinzui> +1
<MootBot> +1 received from sinzui. 5 for, 2 against. 1 have abstained. Count is now 3
<mars> +0
<MootBot> Abstention received from mars. 5 for, 2 against. 2 have abstained. Count is now 3
<bac> -0
 * bigjools spies a fence-sitter
<danilos> :)
<henninge> bac: -0 doesn't work
<flacoste> -1
<MootBot> -1 received from flacoste. 5 for, 3 against. 2 have abstained. Count is now 2
<bac> -1
<MootBot> -1 received from bac. 5 for, 4 against. 2 have abstained. Count is now 1
<jelmer> +1
<MootBot> +1 received from jelmer. 6 for, 4 against. 2 have abstained. Count is now 2
<mars> fascinating :)
<danilos> (this seems like the most interesting vote so far :)
<bac> voting to close in ten seconds
<deryck> +1
<MootBot> +1 received from deryck. 7 for, 4 against. 2 have abstained. Count is now 3
<EdwinGrubbs> -1
<MootBot> -1 received from EdwinGrubbs. 7 for, 5 against. 2 have abstained. Count is now 2
<gmb> Democracy in action, folks.
<bac> #endvote
<bac> [endvote]
<MootBot> Final result is 7 for, 5 against. 2 abstained. Total: 2
<henninge> plus the code team and Rob
<bac> ok, so the voting in AsiaPac will be perfunctory.  the motion will carry as they all favor it.
<danilos> they haven't had the chance to listen to our wonderfully founded arguments, though :)
<bac> danilos:  yeah,i don't think that's going to sway them
<bac> :)
<bac> i'll seek an asiapac volunteer to update the wikis
<danilos> bac, perhaps some beating or bribing then? :) anyway, thanks all for the vote
<danilos> s/vote/voting/
<bac> [topic] call for new reviewers
<MootBot> New Topic:  call for new reviewers
<bac> Any team leads want to nominate new mentats?
<bac> bigjools:  ???
 * bigjools got distracted
 * bigjools is nominating SteveK
<bac> you have mentor?
<bac> hey bigjools, how about tim?
<bigjools> yes, thumper agreed to mentor him
<bac> i'll send an email to steve and invite him to the asiapac meeting
<bigjools> thanks bac
<bac> thanks bigjools for nominating him
<bigjools> my pleasure :)
<bac> [topic] Removing enums and errors out of interfaces.  TimPenhey
<MootBot> New Topic:  Removing enums and errors out of interfaces.  TimPenhey
<bac> tim sent out email about this topic last week but there was no discussion.  he asked me to bring it up here
<bigjools> +1 million to that
<bac> +1
<flacoste> yep, that's a JFDI case
<abentley> +1
<gmb> A thousand times +1
<danilos> +1 here as well :)
<jelmer> +1 too. Just to be sure, that means moving them to the model code?
<sinzui> +1
<bac> so create lp/app/enums.py and errors.py
<bac> err, lp/<yourapphere>/enums.py i mean
<bac> [topic] Policy decisions: Reviewers Meeting or mailing list?  RobertCollins
<MootBot> New Topic:  Policy decisions: Reviewers Meeting or mailing list?  RobertCollins
<henninge> meeting!
<bac> lifeless brought this up last week and wanted to discuss it here
<bac> his thought is having this meeting make policy as one of its goals drags out decisions
<bac> ML discussions can lead to quicker decision making, he argues
<bac> thoughts?
<flacoste> meeting are good for a sync point
<flacoste> making policy decision on mailing list is confusing
<abentley> Having a meeting focuses discussion.  It may increase latency, but the actual decision-making is faster, and the results are clearer.
<henninge> I don't agree. This meeting is prove of that re the decission on import formating.
<flacoste> well, not confusing
<flacoste> but requires art
<flacoste> it's often not clear when the decision is reached
<danilos> I strongly disagree with his assertion that meetings lead to slower decision making; if anything, they lead to faster decision making imho
<bac> i think we get more opinions in this meeting and it is easier to catch confusion and eliminate them
<henninge> danilos: +1
<flacoste> i think his impression might come from the AsiaPac/Ameu split
<henninge> Also, it is the only place left were we regularly meet as developers and discuss developing - sort of.
<flacoste> where the bullk of the devs are in Ameu
<abentley> danilos, agreed.  ML threads tend to wander, and don't necessarily reach a conclusion.
<bigjools> decisions in meetings, discussions about decisions on the ML, unless there's an obvious early consensus on the ML.
<danilos> I
<deryck> I favor the meeting, too.  Seems quicker to me, and you know when it's over. :-)
<danilos> I did suggest him to move to Europe, was what I was trying to say :)
<bac> henninge:  that is true and it makes this meeting sometimes be broader than the original intent.  i'm not saying that is bad, though.
<bac> bigjools:  +1
<danilos> bigjools, +1, and it seems most others who voice their opinion are in agreement
<danilos> however, flacoste does point a valid thing: how do we involve our dear AsiaPac developers in the decision making better
<abentley> danilos, hold their meeting before ours? :-)
<bigjools> if it's a simple vote we just tally votes
<danilos> bigjools, I seriously believe that the discussions we have are useful, so just tallying the votes is not always a true measure imho
<bac> well, i try to bring the items from here to their meeting.  and votes continue there.  really it's the best we can do.
<danilos> (i.e. I've been swayed before to a different opinion)
<flacoste> like we did today for a couple of items
<danilos> bac, right
<bac> good feedback.  thanks.
<danilos> anyway, I think we should keep the format, but look into other opportunities to improve it
<bigjools> danilos: that's why I said long discussions need to be on the ML
<bac> [topic] other items?
<MootBot> New Topic:  other items?
<bac> i guess not.
<danilos> bigjools, sometimes they are short discussions ;)
<bac> thanks for coming
<danilos> thanks all
<bac> #endmeeting
<MootBot> Meeting finished at 09:33.
<bigjools> thanks bac
<bigjools> danilos: short is fine for meetings, long needs to go to ML.  Simple :)
<danilos> bac, sinzui: btw, is it worthwhile to do import lines migration per-app? we've got a big branch in progress and would like to avoid having conflicts until that lands
<bac> danilos:  that is a good idea
<bac> danilos:  we didn't really talk about the D part of JFDI.
<bac> it is more tech-debt
<danilos> bac, I can imagine sinzui already hacking away at a script to do it, which is why I am saying that :)
<bac> danilos:  he is not.  his fingers are idle at the moment.
<danilos> bac, hopefully we don't end up with JFI :)
<bac> that's not to say he isn't thinking about it...
<danilos> bac, heh, thanks :)
<sinzui> danilos, I have my own agenda of removing imports from c.l and any tool that make it easier for me to do will probably help you.
<sinzui> danilos, I am hacking on codehosting at the moment. rosetta will be next week maybe
<danilos> sinzui, rosetta should be completely glob-import free, so I expect it to be a much easier job :) anyway, thanks for working on that
<sinzui> yes, code is clean too. answers and blueprints had a lot but, they were small apps
<sinzui> I dread the registry-soyuz apps
<bigjools> danilos: JFI - lol
<lifeless> oh hai?
<thumper> hey
<thumper> bac: are we doing this now?
<thumper> bac: or do you want to wait for StevenK?
<sinzui> bac: is trying to talk
<bac> thumper:  i cannot tonight
<bac> #startmeeting
<MootBot> Meeting started at 16:31. The chair is bac.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<bac> i'm having connectivity problems too
<bac> thumper, mwhudson, rockstar, lifeless :ping
<rockstar> hi
<mwhudson> hi
<thumper> hi
<bac> mwhudson:  i didn't know if you wanted to show up for these.  you're welcome, of course!
<mwhudson> bac: o
<mwhudson> i'm still doing launchpad reviews occasionally, so i think it makes sense
<bac> ok, great
<lifeless> yeah, Ii'm here
<lifeless> also in a meeting with emo :P
<lifeless> elmo
<bac> so we had a lively meeting this morning.
<thumper> elmo the emo?
<thumper> that is an interesting picture
<mwhudson> :)
<bac> we discussed the one-import-per-line issue
<bac> and voted
<bac> so let's vote here and make it official
<bac> [vote] Change our coding standards to require one line per item for imports
<MootBot> Please vote on:  Change our coding standards to require one line per item for imports.
<MootBot> Public votes can be registered by saying +1/-1/+0 in the channel, private votes by messaging the channel followed by +1/-1/+0  to MootBot
<MootBot> E.g. /msg MootBot +1 #launchpad-meeting
<mwhudson> bac: one line per item means "like bzr" ?
<bac> mwhudson:  yep
<rockstar> +1
<MootBot> +1 received from rockstar. 1 for, 0 against. 0 have abstained. Count is now 1
<thumper> benji had an interesting email
<mwhudson> +1
<MootBot> +1 received from mwhudson. 2 for, 0 against. 0 have abstained. Count is now 2
<thumper> +1
<MootBot> +1 received from thumper. 3 for, 0 against. 0 have abstained. Count is now 3
<bac> i wonder how lifeless would vote...
<bac> [endvote]
<MootBot> Final result is 3 for, 0 against. 0 abstained. Total: 3
<rockstar> Wait, is wgrant around?  Maybe he should have had a say too...
<bac> total:  10 for,5 against, 2 abstained
<bac> rockstar:  you don't need him
<rockstar> Nice.
<bac> so we decided it makes sense to make the change one app at a time
<lifeless> hmm +1
<rockstar> It really wasn't about need, but more that he contributes as well.
<bac> total:  11 for, 5 against, 2 abstain
<wgrant> +1, anyway.
<bac> 12 for
<bac> now you're all just rubbing it in
<thumper> I'll start with code
<rockstar> bac, wait, the chad!  It made my vote unreadable!  :)
<bac> yeah, it's another tech-debt issue.  so update your code as you have time.
<rockstar> thumper, of course code is going to be the first to make the change...  :)
<bac> thanks for championing it lifeless and thumper
<mwhudson> hooray for a definitively coloured bikeshed
<lifeless> Pink
<bac> \o/
<lifeless> btw
<lifeless> coop
<bac> gah, not yet
<lifeless> :)(
<lifeless> that was :)
<bac> we also got the good news that bigjools has asked stevek to become a reviewer and that thumper will be mentoring
 * thumper ndos
<thumper> nods
<bac> as mentioned earlier it probably makes sense for him to go to the ameu meeting
<thumper> ENOCAFFEINE
<bac> if not, we may need to move the time here to a few hours later
<thumper> bac: when is that?
<bac> er, 14Z
<thumper> bac: because AM of the AMEU meeting meens he won't be up
<thumper> that is a midnight meeting for him
<bac> yuk
<thumper> or 1 am
<thumper> not really practical
<bac> ok.  let me discuss with him
<bac> if we moved this meeting two hours later it would be his 930a and my 730p
<bac> not a problem, usually, as long as i don't forget
<thumper> that might be better...
<mwhudson> fine for me, obviously...
<bac> thumper:  i brought up your idea about moving enums and errors and it was met with universal praise.
<thumper> cool
<thumper> I'm happy with that
<bac> lifeless:  we also discussed your idea of moving policy decisions to the mailing list rather than discussion in the reviewers meeting
<bac> the feeling was that the meeting allows for decisions to be clarified and decided much more quickly than on mailing lists, where things tend to drag on and get muddled
<rockstar> Yes.
<lifeless> sure, I did mention it wasn't baked :)
<lifeless> also btw, policy decisions is overly broard
<lifeless> *broad*
<rockstar> There's value in IRC meetings, methinks.
<bac> i think we get more participation "in person" than on the ML
<lifeless> there are policy decisions about doing revisions
<lifeless> s/revisions/reviews/
<bac> lifeless:  sorry if i misrepresented your idea
<lifeless> and there are policy decisions about what the code should look like
<lifeless> I was only speculating about the second set
<lifeless> bac: I don't know if you did or didn't; I'm just being clear here-and-now
<rockstar> lifeless, well, yeah, that stuff contains bike shedding in both places, so it's better on the mailing list where I can ignore it if I want.
<lifeless> anyhow
<lifeless> like I said - unbaked; I will bring it up if I start to feel its baked
<lifeless> but please do think about how we can:
<bac> in reality, discussions on IRC need to be limited to about ten minutes.  so if the topic has not been introduced before, some topics can be decided quickly while others will need to go to the ML
<lifeless>  - bring in more community folk (not just wgrant, there are others :P) into code & design discussions
<lifeless>  - get things that are easy to happen without waiting for a meeting
<lifeless> bac: asgreed
<bac> lifeless:  thanks for bringing up the idea.  you have fresh eyes and can see where we've become stodgy
 * wgrant likes IRC for discussions, since it means you can't keep things on the iinternal list :P
<lifeless> wgrant: FTR I only use the internal list for actually confidential discussions, or if someone starts one there.
<mwhudson> wgrant: we have internal channels too, you know :-)
<bac> wgrant:  we've tried to be better about that
<wgrant> mwhudson: Shh.
<bac> any other topics for today?
<lifeless> ponies
<lifeless> lamont wants one
<bac> what happened to woody?
<thumper> send him a picture of the storm trooper my little pony
<lifeless> he moved on, I don't remember the details
<bac> we're sprinting this week and the conference room came with a 3' inflatable tux as a mascot
<lifeless> \o/
<lifeless> where are you?
<bac> my town
<lifeless> noice
<bac> carolina del norte
<bac> carrboro, specifically
<bac> ok, so if we've devolved into pony discussions i think we should call it a meeting
<bac> thanks for participating
<mwhudson> +1
<mwhudson> thanks bac
<thumper> thanks
<bac> i'll let you know if we have a time change
<bac> #endmeeting
<MootBot> Meeting finished at 16:53.
#launchpad-meeting 2010-08-13
<gmb> Baa
<gmb> Hmm. Entirely the wrong channel.
#launchpad-meeting 2017-08-13
<pjdc> igermelon#    sh int TenGigabitEthernet3/1
<pjdc> TenGigabitEthernet3/1 is down, line protocol is down (notconnect)
<pjdc> wrong window
