[14:08]  * Ursinha looks
[14:08]  * Ursinha hugs MootBot 
[14:08] <Ursinha> you're here!
[14:13] <mrevell> :)
[15:02] <Ursinha> #startmeeting
[15:02] <Ursinha> Welcome to this week's Launchpad Production Meeting.  For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:02] <Ursinha> [TOPIC] Roll Call
[15:02] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[15:02] <MootBot> Meeting started at 09:02. The chair is Ursinha.
[15:02] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:02] <MootBot> New Topic:  Roll Call
[15:02] <Ursinha> hi MootBot
[15:02] <Ursinha> me
[15:02] <herb> me
[15:02] <bac> me
[15:02] <intellectronica> me
[15:03] <gary_poster> me
[15:03] <Ursinha> rockstar, ping
[15:03] <cprov> me
[15:03] <Ursinha> henninge is mia
[15:03] <rockstar> me (barely)
[15:03] <Ursinha> :)
[15:03] <Ursinha> well, no issues for translations, so no much of a problem
[15:03] <Ursinha> alright
[15:04] <Ursinha> [TOPIC] Agenda
[15:04] <Ursinha>  * Actions from last meeting
[15:04] <Ursinha>  * Oops report & Critical Bugs-
[15:04] <Ursinha>  * Operations report (mthaddon/herb/spm)
[15:04] <Ursinha>  * DBA report (DBA contact)
[15:04] <MootBot> New Topic:  Agenda
[15:04] <Ursinha> [TOPIC] * Actions from last meeting
[15:04] <Ursinha>  1) matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
[15:04] <Ursinha>  2) TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week (26th to 30th Jan)
[15:04] <Ursinha>  3) matsubara to file bugs for OOPS-1116EC265 (bug 320293), OOPS-1113EB6 (bug 276144)
[15:04] <MootBot> New Topic:  * Actions from last meeting
[15:04] <Ursinha> 1 - I did that. From now on to discuss meeting time changes just add it to the list of meeting topics, on MeetingAgenda page
[15:04] <Ursinha> 2 - okay as well
[15:04] <Ursinha> next!
[15:05] <Ursinha> [TOPIC] * Oops report & Critical Bugs
[15:05] <MootBot> New Topic:  * Oops report & Critical Bugs
[15:05] <Ursinha> three oopses and one bug
[15:05] <Ursinha> the oopses
[15:05] <Ursinha> MissingMergeDirective: OOPS-1125CEMAIL1
[15:05] <Ursinha> this is yours rockstar?
[15:05]  * Ursinha kicks ubottu 
[15:05] <rockstar> Ursinha, I can look at it
[15:06] <Ursinha> rockstar, thanks
[15:06] <Ursinha> bac, to registry: OOPS-1125A165, OOPS-1125A1096 - this second one is the same in the email sinzui sent to barry
[15:06] <henninge> me!
[15:07] <Ursinha> hi henninge!
[15:07] <henninge> Ursinha: hi! :)
[15:07] <bac> Ursinha: barry is working on the second one
[15:07] <henninge> sorry for the delay ...
[15:07] <Ursinha> bac, do you know if there's an open bug for that?
[15:08] <Ursinha> intellectronica, bug 279561, that's affecting people more frequently lately
[15:08]  * intellectronica shoegazing
[15:08] <bac> Ursinha: i don't.  i'll check with barry when he returns from afk
[15:09] <intellectronica> sorry, it's still a mystery
[15:09] <intellectronica> i'll go back to working on it now
[15:09] <Ursinha> intellectronica, do you have time to deal with that? it's causing a lot of oopses and annoying people
[15:09] <Ursinha> bac, thanks
[15:09] <intellectronica> Ursinha: i do now
[15:10] <Ursinha> [action] bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
[15:10] <MootBot> ACTION received:  bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
[15:10] <Ursinha> bac, about the other one, do you have any clues?
[15:10] <Ursinha> intellectronica, thanks
[15:10] <Ursinha> [action] intellectronica to work on bug 279561
[15:11] <MootBot> ACTION received:  intellectronica to work on bug 279561
[15:11] <Ursinha> [action] rockstar to check OOPS-1125CEMAIL1
[15:11] <MootBot> ACTION received:  rockstar to check OOPS-1125CEMAIL1
[15:11] <Ursinha> any other issues you may want to point?
[15:11] <bac> Ursinha: no clue ATM.  will check it out
[15:12] <Ursinha> thanks bac
[15:12] <Ursinha> I appreciate that
[15:12] <Ursinha> [action] bac to take a look at OOPS-1125A165
[15:12] <MootBot> ACTION received:  bac to take a look at OOPS-1125A165
[15:12] <Ursinha> well, moving on
[15:12] <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
[15:12] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[15:12] <herb> * 2008-01-28 - Yesterday - We rolled out 2.2.1. Total downtime was approximately 35 minutes.
[15:12] <herb> * Even after the rollout, we continue to have problems as a result of bug #156453 and bug #118625. I cannont emphasize enough how much of a pain point this is becoming for the LOSAs. More than a quarter of the incidents that we've had to deal with in the last year have been related to these two bugs.
[15:12] <herb> * We had a couple of incidents in the last week relating to bug #260171.
[15:12] <herb> * Do we expect to roll out any additional patches this week? If so, when?
[15:13] <Ursinha> herb, I think at least one, I'll talk to kiko and confirm that to you
[15:13] <herb> ok
[15:14] <herb> Ursinha: do you suspect it will be rolled out today or tomorrow?
[15:14] <henninge> Ursinha: He approved one of mine, too.
[15:14] <Ursinha> [action] Ursinha to check with kiko if any other rollouts will happen this week
[15:14] <MootBot> ACTION received:  Ursinha to check with kiko if any other rollouts will happen this week
[15:14] <herb> I can wait, I was just curious if you have a best guess.
[15:14] <Ursinha> herb, guess tomorrow, but it depends on if we're having more issues to be fixed
[15:15] <herb> ok
[15:15] <herb> thanks
[15:15] <Ursinha> henninge, cool
[15:15] <Ursinha> thanks herb
[15:15] <Ursinha> I took note of the "pain to the LOSAs" about that two bugs
[15:16] <herb> Ursinha: thanks. much appreciated
[15:16] <Ursinha> well, next
[15:16] <Ursinha> [TOPIC] * DBA report (DBA contact)
[15:16] <Ursinha> stub sent one email about it:
[15:16] <Ursinha> Looks like the upgrade went smoothly and nothing exciting is happening. Yay.
[15:16] <Ursinha> Although neither db server is particularly heavily loaded on average, we do see spikes sometimes. Also, the slave is still more loaded than production. As I don't expect us to get a second slave in the near future, I'm thinking we need to look into some better load balancing to move some of the slave load back to the master.
[15:16] <MootBot> New Topic:  * DBA report (DBA contact)
[15:16] <Ursinha> Config changes on one or two of the appserver instances to use the master as the slave would work in the short term. Long term I think we could be better with a load balancer like pgbouncer or pgpool (1 or 2).
[15:16] <Ursinha> that's all from stub
[15:16] <Ursinha> anything else?
[15:16] <Ursinha> anybody wants to point or discuss something?
[15:17] <Ursinha> 5
[15:17] <Ursinha> 4
[15:17] <Ursinha> 3
[15:17] <Ursinha> 2
[15:17] <Ursinha> 1
[15:17] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:17] <Ursinha> #endmeeting
[15:17] <MootBot> Meeting finished at 09:17.
[15:17] <Ursinha> the fastest eva
[15:17] <gary_poster> Thanks Ursinha :-)
[15:17] <Ursinha> thank you all
[15:17] <Ursinha> gary_poster, :)
[15:17] <cprov> Ursinha: thanks!