=== mrevell is now known as mrevell-lunch
=== bac_afk is now known as bac
=== salgado-afk is now known as salgado
=== mrevell-lunch is now known as mrevell
MootBotMeeting started at 09:57. The chair is matsubara.15:57
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]15:57
matsubarahmm it seems that my clock is a bit early15:58
intellectronicamatsubara: isn't it in 1h?15:58
rockstarNo, 2 min.15:58
matsubara1500 UTC15:58
* rockstar preemptively me-s15:59
matsubaraok, I can wait another 2 min :-)15:59
* matsubara uses the time to actually fix ntp on this computer16:00
matsubaraok, thanks rockstar16:00
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.16:00
matsubara[TOPIC] Roll Call16:00
MootBotNew Topic:  Roll Call16:00
matsubaraso, who's here today?16:00
matsubaraUrsinha, Rinchen ?16:01
Rinchenmatsubara, me and mrevell are here but we're on a call16:01
Rinchenwe'll be here if you need us16:01
matsubaraRinchen: okie16:01
matsubaraso, bugs and code are here16:01
matsubaracprov: ?16:01
matsubaraherb: ?16:01
* mpt waves16:02
bigjoolsme (kinda)16:02
Ursinhasoyuz is here16:02
matsubarahi mpt!16:02
matsubarafoundations is here16:02
Ursinhatranslations is here16:02
matsubarasinzu1: hi16:02
=== sinzu1 is now known as sinzui
matsubaraok, registry is here.16:02
matsubaraflacoste: are you proxying for stub? or stub is here?16:03
flacostehe' s here :-)16:03
matsubaraok, so DBA is here :-)16:03
matsubarawe're missing someone from losas then16:03
matsubarathey can join us later, let's move on16:03
matsubara[TOPIC] Agenda16:03
MootBotNew Topic:  Agenda16:03
matsubara * Next meeting16:04
matsubara * Actions from last meeting16:04
matsubara * Oops report & Critical Bugs16:04
matsubara * Operations report (mthaddon/herb/spm)16:04
matsubara * DBA report (DBA contact)16:04
matsubara[TOPIC] * Next meeting16:04
MootBotNew Topic:  * Next meeting16:04
matsubaraso, same time next week?16:04
matsubarahi herb16:04
matsubaraall right then16:05
matsubara[TOPIC] * Actions from last meeting16:05
MootBotNew Topic:  * Actions from last meeting16:05
matsubarano actions from last meeting16:05
matsubara[TOPIC] * Oops report & Critical Bugs16:05
MootBotNew Topic:  * Oops report & Critical Bugs16:05
matsubaraUrsinha: stage is yours16:05
Ursinhabug 27695016:05
ubottuLaunchpad bug 276950 in soyuz "Timeout accepting more than 6 packages in queue page" [Undecided,New] https://launchpad.net/bugs/27695016:05
Ursinhamatsubara filed this one yesterday16:05
bigjoolsBjornT is looking at that right now16:06
Ursinhabug 27712916:06
ubottuLaunchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [Undecided,New] https://launchpad.net/bugs/27712916:06
Ursinhai just filed it, can anybody take a look>16:06
Ursinhaflacoste, I just talked with salgado about that16:06
Ursinhahe's not aware of that, even having one OOPS authenticated as him16:07
flacosteand what was his assessment?16:07
salgadoI don't remember seeing that OOPS16:07
Ursinhahe never saw that16:07
Ursinhaanyway, stub said that it's a timeout rendering page problem16:08
Ursinhaas I wrote on the bug16:08
salgadoit doesn't look like that to me16:08
flacostehmm, not sure of that looking at the traceback16:08
Ursinhait's been happening a few times these days and with different users16:08
flacostewe don't have time to look into this week, but we can next week16:09
Ursinhaflacoste, ok, if you want to give you two cents on what that would be, go ahead16:09
Ursinhaflacoste, and ok, thanks16:10
matsubara[ACTION] foundations to look up bug 27712916:10
MootBotACTION received:  foundations to look up bug 27712916:10
ubottuLaunchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [High,Triaged] https://launchpad.net/bugs/27712916:10
Ursinhathe last bug is the critical of translations, bug 27348916:10
ubottuLaunchpad bug 273489 in rosetta "Remaining Intrepid template approvals" [Critical,Fix committed] https://launchpad.net/bugs/27348916:10
Ursinhadanilos, when the script finished running?16:10
danilosUrsinha: that was a few days ago, we are basically handling the manual stuff right now16:10
Ursinhadanilos, ok16:11
danilosUrsinha: that one might even be 'Fix released' so far, I'll make sure to check with jtv16:11
Ursinhadanilos, ok, thanks16:11
Ursinhamatsubara, you can move on16:11
Ursinhathanks guys16:11
matsubarathanks Ursinha16:11
matsubarathanks everyone.16:11
matsubarawe have another critical bug which is already fix committed16:11
matsubaraso, I won't bring it up again16:12
matsubara[TOPIC] * Operations report (mthaddon/herb/spm)16:12
MootBotNew Topic:  * Operations report (mthaddon/herb/spm)16:12
herb* 2008-09-25: Cherry pick r7080 to the FTP master, updating cron.16:12
herb* 2008-09-26: Cherry pick r7087 to the scripts server.16:12
herb* 2008-09-29: Cherry pick zope r46 to lpnet*16:12
herb* We still have app servers dying periodically. mwhudson provided some additional information to gather when core dumps are found.16:12
herb* Codebrowse was unresponse and needed to be restarted a couple of times this week.16:12
herbflacoste: have you been able to look through the backtraces?  anything of value there?16:13
herbre: app servers dying.16:13
flacostewell, nothing that I saw16:13
herbso are we at a dead end?16:13
flacostei was very impressed by the kind of output you can get though16:13
flacostewhen you know what you are doing16:14
flacosteand i was waiting to have at least another to see a pattern16:14
herbWell that's it from the losas unless there are any questions for us.16:14
matsubaraok. thanks herb16:15
matsubara[TOPIC] * DBA report (stub)16:15
MootBotNew Topic:  * DBA report (stub)16:15
stubNothing thrilling happening with the production systems - business as usual.16:15
stubDB patch review call with Mark scheduled for tomorrow, although that might not be relevant to this forum now.16:15
stubdemo.launchpad.net running with a replicated database backend just fine. The updated db maintenance scripts will be landing when the db freeze is lifted on launchpad/devel, so Friday or Monday. devs can run replicated if they want easily - we may want to give thought to making this the default.16:15
matsubarastub: sounds like a good idea to run it replicated locally. is it going to give a big hit in the performance?16:16
danilosstub: we hear you16:16
flacostestub: we discussed rolling out replicated yesterday16:16
flacostewe weren't sure it was a good idea16:16
flacosteone question came though is do we have to take LP down to set-up the replication?16:17
stubI don't think we would want to take the hit running tests against a replicated environment.16:17
stubThe appservers and systems will only need to be down for a few minutes.16:17
stubIt will take three hours to build the replica, but that can happen live. The appservers of course shouldn't be making use of the replica as it won't have data on it at that point.16:18
stubOnce the replica is built, you change the appserver configs and bounce them.16:18
flacosteok, that makes sense16:19
stubI should add a master-only flag or something to make that easy, as it will probably be standard rollout procedure.16:19
flacosteso it was deemed safer to not roll-out replication to staging next week16:19
flacostewell, run staging replicated16:20
flacosteand turn it on week-016:20
flacosteif we need the performance boost16:20
flacostewe can turn it on in production after sufficient staging QA16:20
flacosteiow, we don't need to wait for the roll-out to turn it on in production16:20
stubI have some concern about how a replicated staging will affect a replicated production, given staging runs on the same server slated as being the lpmain slave and eventually the authdb master.16:20
stubBut we can deal with that at the time - I think it is better to run the risk of losing staging for a while than purchase some very expensive hardware just in case.16:21
stubThe concern being that the staging rebuild process will be going on for 4 hours of every day (if we continue with daily db builds) - that is 4 hours of a single core being hammered. I don't think we will really know how that impacts things in reality until we try it though.16:22
stubSorry - 4.5 hours every day ;)16:24
danilosstub: so it means chokecherry will serve as the main readonly slave as well?16:24
danilos(once it all goes up, of course)16:25
matsubarathere are proposals to change staging db update to once a week16:25
stubYes. We only have two servers with suitable hardware - hackberry (current db server) and chokecherry.16:25
danilosok, thanks for clarifying stub16:25
stubIt also impacts our disaster recovery - instead of having a hot spare, we need scale back to a single server if one explodes.16:26
stubSo we need to watch our load - at some point we will have too much load to run on a single box and we need to think about DR before then.16:27
stubdisaster recovery16:27
rockstarDisaster Recovery16:27
matsubarai'll add that to the disaster scenarios page, if that's still up to date16:28
matsubaraanything else stub?16:28
stubnah - I'm just rambling.16:28
matsubarathanks for bringing up your concerns16:29
matsubaraI have one last thing to ask you all before closing the meeting16:29
matsubaraI've noticed that the number of "New" bugs are climbing, so please, keep an eye on the daily triage backlog. Ursinha and I can help each team, but you need to approach us.16:30
matsubaraplease QA contacts, pass that on to your TLs16:30
matsubaraso I think that's all16:31
matsubaraThank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.16:31
MootBotMeeting finished at 10:31.16:31
Ursinhathanks matsubara16:31
rockstarThanks matsubara16:31
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
=== rockstar` is now known as rockstar
=== salgado is now known as salgado-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!