[14:58] <matsubara> ping MootBot
[15:00] <matsubara> #startmeeting
[15:00] <MootBot> Meeting started at 09:00. The chair is matsubara.
[15:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:00] <matsubara> [TOPIC] Roll Call
[15:00] <MootBot> New Topic:  Roll Call
[15:00] <herb> me
[15:00] <jtv> me
[15:00] <sinzui> me
[15:00] <Ursinha> me
[15:00] <rockstar> me
[15:01] <jtv> I'm standing in for henninge today, since he's on sprint.
[15:01] <matsubara> thanks jtv
[15:01] <danilos> me
[15:01] <BjornT> me
[15:02] <matsubara> so, if any of you can't make the meeting next week, please coordinate with another teammate to cover for you and add a notice in the Apologies section in the MeetingAgenda page.
[15:02] <matsubara> flacoste: ping
[15:02] <intellectronica> me
[15:03] <matsubara> bigjools: ping
[15:03] <bigjools> me
[15:03] <flacoste> me
[15:04] <matsubara> ok, let's move on, stub can join later
[15:04] <matsubara> [TOPIC] Agenda
[15:04] <MootBot> New Topic:  Agenda
[15:04] <matsubara>  * Actions from last meeting
[15:04] <matsubara>  * Oops report & Critical Bugs
[15:04] <matsubara>  * Operations report (mthaddon/herb/spm)
[15:04] <matsubara>  * DBA report (DBA contact)
[15:04] <matsubara> [TOPIC] * Actions from last meeting
[15:04] <MootBot> New Topic:  * Actions from last meeting
[15:04] <matsubara>  * bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one - there was, bug 280925
[15:04] <matsubara>  * intellectronica to work on bug 279561
[15:04] <matsubara>  * rockstar to check OOPS-1125CEMAIL1
[15:04] <matsubara>  * bac to take a look at OOPS-1125A165 - bac filed bug 322792
[15:04] <matsubara>  * Ursinha to check with kiko if any other rollouts will happen this week
[15:05] <Ursinha> holy crap
[15:05] <Ursinha> that bug is timing out
[15:05] <Ursinha> or what?
[15:06] <Ursinha> anyway
[15:06] <matsubara> intellectronica: any news about the api bug?
[15:06] <Ursinha> my items were done
[15:06] <matsubara> rockstar: what's up about that oops?
[15:06] <intellectronica> matsubara: sorry, no news yet
[15:06] <Ursinha> matsubara, he landed a fix for that
[15:06] <matsubara> Ursinha: who's he?
[15:06] <matsubara> :-)
[15:06] <Ursinha> matsubara, rockstar :)
[15:06] <Ursinha> sorry
[15:06] <rockstar> matsubara, I got an RC in for that one.
[15:07] <matsubara> ok, I remember that one.
[15:07] <matsubara> so I guess the only pending one is the api bug which is a mistery to everyone
[15:08] <matsubara> the good news is that intellectronica found out another thing about that bug that might lead to its root cause
[15:08] <matsubara> intellectronica: thanks for keeping us posted in the report
[15:08] <matsubara> let's move on
[15:08] <matsubara> [TOPIC] * Oops report & Critical Bugs
[15:08] <MootBot> New Topic:  * Oops report & Critical Bugs
[15:09] <matsubara> so in today's oops section I'd like to talk about the timeout bugs you guys are working for the LPW
[15:09] <matsubara> https://dev.launchpad.net/PerformanceWeeks/February2009
[15:10] <matsubara> I'm going to review all the landings related to LPW work and add to that page.
[15:10] <matsubara> so if you wanna help, point me to revision numbers on RF related to that work
[15:10] <jtv> matsubara: bug 324264 is now Fix Committed.
[15:11] <sinzui> matsubara: r7705 for Bug: 325321
[15:11] <Ursinha> jtv, great
[15:11] <matsubara> intellectronica, BjornT : any news about the bug number 1 time out?
[15:11] <sinzui> matsubara: EdwinGrubbs will land his branch today
[15:11] <danilos> work on bug 302798 is in progress in different ways (there's a commit from 323something which disabled external suggestions to give us a better idea on how stuff is working, and henning is working on removal of obsolete translations which will reduce our DB size by ~33%)
[15:12] <BjornT> matsubara: me, intellectronica, and allenap are working on it
[15:12] <jtv> danilos: hey, I was putting that paragraph together!
[15:12] <BjornT> matsubara: allenap is looking at reducing the time it take to render the comments, possibly by not showing all by default
[15:13] <danilos> jtv: ok, I'll let you handle it all from now on :)
[15:13] <BjornT> matsubara: intellectronica is working on loading the subscribers portlet in a different request
[15:13] <danilos> matsubara: you can have full trust in jtv as far as PW is concerned :)
[15:13] <jtv> *cough*
[15:13] <BjornT> matsubara: and i'm working on optimizing code, based off profiling information
[15:13] <matsubara> danilos: I do! he's been very helpful with the status updates
[15:14] <BjornT> matsubara: also, intellectronica's inital tests on dogfood were successful, reducing the time quite a lot :)
[15:14] <matsubara> great
[15:14] <flacoste> matsubara: i'm working on bug 316881 (which isn't an OOPS per-se, but related to performance anyhow)
[15:15] <matsubara> stub: there's an email from jono asking for some help with the +project-cloud oops, so if you could help out there, would be awesome
[15:15] <matsubara> thanks flacoste, i'll add it to the page
[15:15] <stub> I've replied on the bug. Not sure if I'm helpful though.
[15:15] <matsubara> stub: cool. thank you
[15:16] <matsubara> bigjools: news in soyuz. how about the one muharem is taking care of?
[15:16] <matsubara> s/./?/
[15:16] <bigjools> matsubara: it's not going so well unfortunately
[15:16] <bigjools> I don't expect any progress this week
[15:16] <matsubara> [action] matsubara to add 316881 to foundations section in LPW wiki page
[15:16] <MootBot> ACTION received:  matsubara to add 316881 to foundations section in LPW wiki page
[15:17] <matsubara> bigjools: why is that?
[15:17] <bigjools> matsubara: the first attempt to fix it failed.  he's also been at the distro sprint this week
[15:18] <matsubara> bigjools: oh right. well, you guys are excused since the whole team is sprinting and you already landed 2(?) timeout fixes :-)
[15:18] <bigjools> matsubara: thanks :)
[15:18] <matsubara> I guess that's it from me. Ursinha, anything else?
[15:19] <Ursinha> matsubara, no, the pending oops for soyuz I already talked with bigjools
[15:19] <bigjools> yeah, get edge updating again and we'll see how it went
[15:19] <matsubara> great. thanks everyone!
[15:19] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[15:19] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[15:19] <herb> - 2009-01-30 - Friday we updated lpnet, edge and the scripts servers to to r7676.
[15:19] <herb> - 2009-02-04 - Yesterday we updated codebrowse to r43
[15:20] <herb> - I feel like I'm starting to sound like a broken record... But we're still being bothered daily, often multiple times, by bug #156453 and bug #118625 which seem to be related.
[15:20] <herb> - We also continue to run into issues associated with bug #260171
[15:20] <flacoste> herb: i herd mwhudson was working on loggerhead performance this week
[15:20] <flacoste> as part of LPW
[15:20] <matsubara> yes!
[15:20] <sinzui> herb: There is a sprint in the planning to fix that too
[15:20] <Ursinha> yes, he is
[15:20] <matsubara> mwhudson is indeed working on that
[15:21] <flacoste> herb: and well-placed sources also told me that a sprint is being organized to fix those damn issues
[15:21] <flacoste> herb: so there is hope!
[15:21] <herb> excellent. a fix would be huge for the LOSAs
[15:21] <rockstar> herb, mwhudson is on the verge of insanity tracking loggerhead issues down.
[15:22] <herb> thanks for all the work on that. it is much appreciated.
[15:22] <matsubara> thanks herb
[15:22] <matsubara> [TOPIC] * DBA report (stub)
[15:22] <MootBot> New Topic:  * DBA report (stub)
[15:22] <stub> The production dbs seem to be ticking away nicely.
[15:22] <stub> Staging db updates are being disabled by the losas for some testing, so expect a drop in timeout OOPSES.
[15:22] <stub> I've had some db patches for this cycle come through already, which is great.
[15:24] <stub> 3
[15:24] <stub> 2
[15:24] <stub> 1
[15:24] <matsubara> all right. thanks stub
[15:24] <flacoste> what about the staging issues?
[15:25] <matsubara> herb: ^
[15:25] <flacoste> staging isn't available at the moment
[15:25] <herb> it's restoring
[15:25] <flacoste> but we didn't understand why it failed?
[15:25] <flacoste> or did we?
[15:25] <matsubara> herb: how long will it take to restore?
[15:25] <flacoste> and it's just that gmb is done with the testing and we can resume normal staging updates?
[15:25] <herb> matsubara: should be back up within the next couple of hours.
[15:26] <Ursinha> herb, about 10 hours ago I was talking with spm and he was unsuccessful to put staging on again
[15:27] <herb> flacoste: upgrade.py failed because there will still connections open to the staging db.  This left the DB in an indeterminate state, so we had to go through the full restore process with a new copy of the staging DB.
[15:27] <gmb> flacoste: I'm not done yet.
[15:27] <gmb> flacoste: I need staging to be up for that
[15:27] <flacoste> so this means that we are still screwed?
[15:28] <stub> This is all useful information for read only launchpad btw.
[15:28] <flacoste> we either need to fix upgrade.py to work without a db restore
[15:28] <flacoste> or to turn off staging ugprades for the duration of gmb's test
[15:28] <herb> flacoste: upgrade.py works fine
[15:29] <herb> and we'll need to turn off upgrades while gmb's testing, per stubs note above.
[15:29] <flacoste> why were there still connections open?
[15:29] <stub> upgrade.py cannot work when there are active db connections, as upgrade requires exclusive locks on all the replicated db tables.
[15:29] <herb> flacoste: the rollout process shuts down the app servers, but there are still potentially cron scripts running, etc.
[15:30] <flacoste> why usually is this not a problem then?
[15:31] <stub> Because the restore, replicate and upgrade process is done as a fresh db. Once it is finished, it is swapped into place.
[15:31] <flacoste> ah right!
[15:31]  * bigjools has to run, will catch scrollback later if you need anything from me
[15:32] <flacoste> so when I said fix upgrade.py, i should have said 'fix rollout process'
[15:32] <flacoste> so I guess it's best to simply disbale upgrade for now
[15:32] <flacoste> i also heard that gmb might do his tests elsewhere
[15:32] <flacoste> so that might becomes a moot issue
[15:32] <flacoste> but like stub said, very instructive for read-only launchpad
[15:33] <gmb> flacoste: Well, that's an embryonic idea right now. I still need a staging / demo machine for the forseeable future.
[15:34] <matsubara> I want to make an action item for the fix rollout process thing
[15:34] <matsubara> not sure who would be responsible for that
[15:34] <matsubara> losas?
[15:35] <herb> whoa
[15:35] <flacoste> with the help of stub probably
[15:35] <matsubara> [action] losas and stub to fix rollout process to avoid the staging restore problems
[15:35] <MootBot> ACTION received:  losas and stub to fix rollout process to avoid the staging restore problems
[15:35] <herb> there isn't anything inherently wrong with the rollout process.
[15:35] <herb> but ok
[15:36] <matsubara> thanks everyone
[15:36] <matsubara> anythign else before I close?
[15:36] <flacoste> well, it's currently not reliable if we don't do a DB restore it seems
[15:36] <flacoste> nope
[15:36] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:36] <matsubara> #endmeeting
[15:36] <MootBot> Meeting finished at 09:36.
[15:36] <Ursinha> thanks matsubara