[15:57] <matsubara> #startmeeting
[15:57] <MootBot> Meeting started at 09:57. The chair is matsubara.
[15:57] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:58] <matsubara> hmm it seems that my clock is a bit early
[15:58] <intellectronica> matsubara: isn't it in 1h?
[15:58] <rockstar> No, 2 min.
[15:58] <matsubara> 1500 UTC
[15:59]  * rockstar preemptively me-s
[15:59] <matsubara> ok, I can wait another 2 min :-)
[16:00]  * matsubara uses the time to actually fix ntp on this computer
[16:00] <rockstar> NOW!
[16:00] <matsubara> ok, thanks rockstar
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <rockstar> :)
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> so, who's here today?
[16:01] <matsubara> Ursinha, Rinchen ?
[16:01] <Rinchen> matsubara, me and mrevell are here but we're on a call
[16:01] <Ursinha> me
[16:01] <Rinchen> we'll be here if you need us
[16:01] <intellectronica> me
[16:01] <matsubara> Rinchen: okie
[16:01] <matsubara> so, bugs and code are here
[16:01] <matsubara> cprov: ?
[16:01] <matsubara> herb: ?
[16:01] <cprov> me
[16:02]  * mpt waves
[16:02] <bigjools> me (kinda)
[16:02] <Ursinha> soyuz is here
[16:02] <flacoste> me
[16:02] <matsubara> hi mpt!
[16:02] <danilos> me
[16:02] <matsubara> foundations is here
[16:02] <Ursinha> translations is here
[16:02] <matsubara> sinzu1: hi
[16:02] <sinzui> hi
[16:02] <matsubara> ok, registry is here.
[16:02] <sinzui> me
[16:03] <matsubara> flacoste: are you proxying for stub? or stub is here?
[16:03] <flacoste> stub?
[16:03] <stub> ?
[16:03] <flacoste> he' s here :-)
[16:03] <matsubara> ok, so DBA is here :-)
[16:03] <Ursinha> cool
[16:03] <matsubara> we're missing someone from losas then
[16:03] <matsubara> they can join us later, let's move on
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:04] <matsubara>  * Next meeting
[16:04] <matsubara>  * Actions from last meeting
[16:04] <matsubara>  * Oops report & Critical Bugs
[16:04] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:04] <matsubara>  * DBA report (DBA contact)
[16:04] <matsubara> [TOPIC] * Next meeting
[16:04] <MootBot> New Topic:  * Next meeting
[16:04] <matsubara> so, same time next week?
[16:04] <herb> me
[16:04] <danilos> yes
[16:04] <danilos> :)
[16:04] <matsubara> hi herb
[16:05] <matsubara> all right then
[16:05] <matsubara> [TOPIC] * Actions from last meeting
[16:05] <MootBot> New Topic:  * Actions from last meeting
[16:05] <matsubara> no actions from last meeting
[16:05] <matsubara> [TOPIC] * Oops report & Critical Bugs
[16:05] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:05] <matsubara> Ursinha: stage is yours
[16:05] <Ursinha> bug 276950
[16:05] <Ursinha> matsubara filed this one yesterday
[16:05] <Ursinha> right?
[16:06] <bigjools> BjornT is looking at that right now
[16:06] <Ursinha> cool
[16:06] <Ursinha> bug 277129
[16:06] <Ursinha> i just filed it, can anybody take a look>
[16:06] <Ursinha> ?
[16:06] <Ursinha> flacoste, I just talked with salgado about that
[16:07] <Ursinha> he's not aware of that, even having one OOPS authenticated as him
[16:07] <flacoste> and what was his assessment?
[16:07] <salgado> I don't remember seeing that OOPS
[16:07] <Ursinha> he never saw that
[16:08] <Ursinha> anyway, stub said that it's a timeout rendering page problem
[16:08] <Ursinha> as I wrote on the bug
[16:08] <salgado> it doesn't look like that to me
[16:08] <flacoste> hmm, not sure of that looking at the traceback
[16:08] <Ursinha> it's been happening a few times these days and with different users
[16:09] <flacoste> we don't have time to look into this week, but we can next week
[16:09] <Ursinha> flacoste, ok, if you want to give you two cents on what that would be, go ahead
[16:10] <Ursinha> flacoste, and ok, thanks
[16:10] <matsubara> [ACTION] foundations to look up bug 277129
[16:10] <MootBot> ACTION received:  foundations to look up bug 277129
[16:10] <Ursinha> the last bug is the critical of translations, bug 273489
[16:10] <Ursinha> danilos, when the script finished running?
[16:10] <danilos> Ursinha: that was a few days ago, we are basically handling the manual stuff right now
[16:11] <Ursinha> danilos, ok
[16:11] <danilos> Ursinha: that one might even be 'Fix released' so far, I'll make sure to check with jtv
[16:11] <Ursinha> danilos, ok, thanks
[16:11] <Ursinha> matsubara, you can move on
[16:11] <Ursinha> thanks guys
[16:11] <matsubara> thanks Ursinha
[16:11] <matsubara> thanks everyone.
[16:11] <matsubara> we have another critical bug which is already fix committed
[16:12] <matsubara> so, I won't bring it up again
[16:12] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[16:12] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:12] <herb> * 2008-09-25: Cherry pick r7080 to the FTP master, updating cron.
[16:12] <herb> * 2008-09-26: Cherry pick r7087 to the scripts server.
[16:12] <herb> * 2008-09-29: Cherry pick zope r46 to lpnet*
[16:12] <herb> * We still have app servers dying periodically. mwhudson provided some additional information to gather when core dumps are found.
[16:12] <herb> * Codebrowse was unresponse and needed to be restarted a couple of times this week.
[16:13] <herb> flacoste: have you been able to look through the backtraces?  anything of value there?
[16:13] <herb> re: app servers dying.
[16:13] <flacoste> nope
[16:13] <flacoste> well, nothing that I saw
[16:13] <herb> ok
[16:13] <herb> so are we at a dead end?
[16:13] <flacoste> i was very impressed by the kind of output you can get though
[16:14] <flacoste> when you know what you are doing
[16:14] <herb> heh
[16:14] <flacoste> and i was waiting to have at least another to see a pattern
[16:14] <herb> ok
[16:14] <herb> Well that's it from the losas unless there are any questions for us.
[16:15] <matsubara> ok. thanks herb
[16:15] <herb> thanks
[16:15] <matsubara> [TOPIC] * DBA report (stub)
[16:15] <MootBot> New Topic:  * DBA report (stub)
[16:15] <stub> Nothing thrilling happening with the production systems - business as usual.
[16:15] <stub> DB patch review call with Mark scheduled for tomorrow, although that might not be relevant to this forum now.
[16:15] <stub> demo.launchpad.net running with a replicated database backend just fine. The updated db maintenance scripts will be landing when the db freeze is lifted on launchpad/devel, so Friday or Monday. devs can run replicated if they want easily - we may want to give thought to making this the default.
[16:16] <stub> ping?
[16:16] <matsubara> stub: sounds like a good idea to run it replicated locally. is it going to give a big hit in the performance?
[16:16] <flacoste> yes
[16:16] <danilos> stub: we hear you
[16:16] <flacoste> stub: we discussed rolling out replicated yesterday
[16:16] <flacoste> we weren't sure it was a good idea
[16:17] <flacoste> one question came though is do we have to take LP down to set-up the replication?
[16:17] <stub> I don't think we would want to take the hit running tests against a replicated environment.
[16:17] <stub> The appservers and systems will only need to be down for a few minutes.
[16:18] <stub> It will take three hours to build the replica, but that can happen live. The appservers of course shouldn't be making use of the replica as it won't have data on it at that point.
[16:18] <stub> Once the replica is built, you change the appserver configs and bounce them.
[16:19] <flacoste> ok, that makes sense
[16:19] <stub> I should add a master-only flag or something to make that easy, as it will probably be standard rollout procedure.
[16:19] <flacoste> so it was deemed safer to not roll-out replication to staging next week
[16:20] <flacoste> well, run staging replicated
[16:20] <flacoste> and turn it on week-0
[16:20] <flacoste> if we need the performance boost
[16:20] <flacoste> we can turn it on in production after sufficient staging QA
[16:20] <flacoste> iow, we don't need to wait for the roll-out to turn it on in production
[16:20] <stub> I have some concern about how a replicated staging will affect a replicated production, given staging runs on the same server slated as being the lpmain slave and eventually the authdb master.
[16:21] <stub> But we can deal with that at the time - I think it is better to run the risk of losing staging for a while than purchase some very expensive hardware just in case.
[16:22] <stub> The concern being that the staging rebuild process will be going on for 4 hours of every day (if we continue with daily db builds) - that is 4 hours of a single core being hammered. I don't think we will really know how that impacts things in reality until we try it though.
[16:24] <stub> Sorry - 4.5 hours every day ;)
[16:24] <danilos> stub: so it means chokecherry will serve as the main readonly slave as well?
[16:25] <danilos> (once it all goes up, of course)
[16:25] <matsubara> there are proposals to change staging db update to once a week
[16:25] <stub> Yes. We only have two servers with suitable hardware - hackberry (current db server) and chokecherry.
[16:25] <danilos> ok, thanks for clarifying stub
[16:26] <stub> It also impacts our disaster recovery - instead of having a hot spare, we need scale back to a single server if one explodes.
[16:27] <stub> So we need to watch our load - at some point we will have too much load to run on a single box and we need to think about DR before then.
[16:27] <matsubara> DR?
[16:27] <stub> disaster recovery
[16:27] <rockstar> Disaster Recovery
[16:28] <matsubara> ok
[16:28] <matsubara> i'll add that to the disaster scenarios page, if that's still up to date
[16:28] <matsubara> anything else stub?
[16:28] <stub> nah - I'm just rambling.
[16:29] <matsubara> thanks for bringing up your concerns
[16:29] <matsubara> I have one last thing to ask you all before closing the meeting
[16:30] <matsubara> I've noticed that the number of "New" bugs are climbing, so please, keep an eye on the daily triage backlog. Ursinha and I can help each team, but you need to approach us.
[16:30] <matsubara> please QA contacts, pass that on to your TLs
[16:31] <matsubara> so I think that's all
[16:31] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:31] <matsubara> #endmeeting
[16:31] <MootBot> Meeting finished at 10:31.
[16:31] <Ursinha> thanks matsubara
[16:31] <rockstar> Thanks matsubara