=== ursula_ is now known as Ursinha
=== mrevell is now known as mrevell-lunch
=== mrevell-lunch is now known as mrevell
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:01] <flacoste> me
[16:01] <matsubara> hi francis
[16:01] <Ursinha> me
[16:01] <matsubara> rockstar, bigjools, henninge intellectronica sinzui: hi
[16:01] <sinzui> me
[16:01] <bigjools> meeee
[16:01] <intellectronica> me
[16:01] <henninge> me
[16:02] <matsubara> hi stub
[16:03] <stub> yo
[16:03] <matsubara> herb:
[16:03] <matsubara> ok, herb can join in later. let's move on
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs
[16:03] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:03] <matsubara>  * DBA report (stub)
[16:03] <matsubara> [TOPIC] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:03] <matsubara>     * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
[16:03] <matsubara>     * ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses"
[16:03] <matsubara>       * filed bug 360846
[16:03] <matsubara>     * intellectronica to talk to gmb about bug 269538
[16:04] <matsubara> ok, Ursinha done hers
[16:04] <sinzui> wow I do suck
[16:04] <matsubara> [action] sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
[16:04] <MootBot> ACTION received:  sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
[16:04] <sinzui> matsubara: I will do this email IMMEDIATELY after this meeting.
[16:04] <matsubara> thanks sinzui
[16:04]  * sinzui starts the subect line now
[16:04] <intellectronica> matsubara: sorry, i didn't. will try to do that later today
[16:05] <matsubara> intellectronica: cool. thanks. shall I keep the action item for the next meeting?
[16:05] <rockstar> me
[16:05] <intellectronica> matsubara: yes, please
[16:05] <matsubara> [action] intellectronica to talk to gmb about bug 269538
[16:05] <MootBot> ACTION received:  intellectronica to talk to gmb about bug 269538
[16:06] <matsubara> ok, let's move on. thanks sinzui and intellectronica
[16:06] <matsubara> hi rockstar
[16:06] <rockstar> hi
[16:06] <matsubara> [TOPIC] * Oops report & Critical Bugs
[16:06] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:07] <matsubara> Ursinha: take the stage please
[16:07] <Ursinha> I have only one bug for foundations, bug 357307
[16:07]  * Ursinha kicks the bot
[16:07] <Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/357307
[16:08] <Ursinha> flacoste, ^
[16:08] <flacoste> yep, looking at this now
[16:10] <flacoste> Ursinha: what's the priority on this?
[16:10] <matsubara> I have one for sinzui: bug 358332
[16:10] <flacoste> sinzui: do you think salgado could look into that token bug?
[16:10] <matsubara> curtis, do you agree with the importance I set for https://bugs.edge.launchpad.net/launchpad-registry/+bug/358332 ?
[16:11] <Ursinha> flacoste, we're had 25 yesterday on lpnet
[16:11] <Ursinha> about this average a day, oopses count
[16:11] <sinzui> Well I do not agree with Medium, I do not ever use that status
[16:12] <matsubara> sinzui: can you reset the importance to a more meaningful value then?
[16:12] <sinzui> flacoste: Would you like to take salgado to your team then ;)
[16:12] <Ursinha> haha
[16:13] <flacoste> sinzui: you know my opinion on that one :-)
[16:13] <sinzui> matsubara: Since I have committed to working on series issues, I will try to land a fix for bug 358332 this release
[16:13] <flacoste> sinzui: and I'll take that as a yes
[16:14] <matsubara> thank you sinzui
[16:14] <sinzui> Yes, I will ask salgado to look at the bug 357307
[16:15] <matsubara> [action] sinzui to take bug 358332
[16:15] <MootBot> ACTION received:  sinzui to take bug 358332
[16:15] <matsubara> [action] sinzui to ask salgado to fix bug 357307
[16:15] <MootBot> ACTION received:  sinzui to ask salgado to fix bug 357307
[16:16] <matsubara> All critical bugs are fix committed
[16:17] <matsubara> so, I think that's all. anything else Ursinha ?
[16:17] <Ursinha> no matsubara
[16:17] <Ursinha> thanks all
[16:17] <matsubara> ok, thanks everyone
[16:17] <matsubara> I'm going to skip herb's section since he's not here yet.
[16:17] <mthaddon> matsubara: I'm here
[16:17] <matsubara> oh
[16:17] <matsubara> hi mthaddon
[16:17] <mthaddon> matsubara: herb's off
[16:18] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[16:18] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:18] <mthaddon>  - Preparations for new rollout procedure - need to confirm a fair amount of stuff with stub before the next rollout
[16:18] <mthaddon>  - PostgreSQL upgrade yesterday meant some downtime
[16:18] <mthaddon>  - Need to decide on whether we're switching the master to wildcherry and if so when
[16:18] <mthaddon> I think that's it unless there are any questions
[16:18] <intellectronica> mthaddon: i have a question
[16:18] <matsubara> what's this new rollout procedure?
[16:18] <mthaddon> matsubara: SSO related
[16:18] <mthaddon> intellectronica: sure
[16:19] <intellectronica> mthaddon: yesterday morning (UTC) forster needed a restart because it was overwhelmed. Spads helped me with that. do you know what happened?
[16:19] <mthaddon> matsubara: involves splitting out a slave from as standalone so we can continue to serve SSO
[16:19] <mthaddon> intellectronica: I don't - we've had some issues with forster over the past few weeks
[16:19] <matsubara> mthaddon: cool.
[16:20] <mthaddon> intellectronica: i.e. this isn't the first time it's needed a kicking - I'll see if I can look into it a little more
[16:20] <intellectronica> mthaddon: ok, as long as you're not surprised. sorry i forgot to mention this yesterday
[16:21] <matsubara> anything else for mthaddon?
[16:21] <mthaddon> intellectronica: we're moving the incoming email parsing script off there, so hopefully that'll reduce load a little, although I'd be surprised if it was related to that
[16:21] <flacoste> mthaddon: any idea on the surge in buildbot failures yesterday?
[16:21] <mthaddon> flacoste: I was off yesterday so I wasn't aware of that, no
[16:21] <flacoste> mthaddon: it looks it was restarted, i assume by spm since gary nor i did it
[16:22] <intellectronica> mthaddon: if it's not the cause, then at least it won't stop working when whatever is the cause starts going wild :)
[16:22] <flacoste> mthaddon: ok, i'll check with spm later
[16:22] <mthaddon> intellectronica: absolutely :)
[16:22] <matsubara> flacoste: maybe it was related to the unexpected downtime we had yesterday
[16:23] <matsubara> flacoste: all over the DC
[16:23] <flacoste> matsubara: none of this lives in the DC yet
[16:23] <matsubara> well, but the code buildbot is pulling does :-)
[16:24] <matsubara> anyway, spm can clarify that later.
[16:24] <matsubara> thanks everyone
[16:24] <matsubara> [TOPIC] * DBA report (stub)
[16:24] <MootBot> New Topic:  * DBA report (stub)
[16:24] <stub> I have been on public holidays or ill for the last week, but things still seem to be running which is nice.
[16:24] <stub> We got a replication lag spike the other night - all the systems coped well while it was cleared. Possibly the rosetta imports being busier than usual. Possibly the rosetta imports where just delayed by something else. I don't think anyone has traced what processes where running at that time yet from the script activity records or the output on launchpad-errors.
[16:24] <stub> We are going to have to be more agressive at blocking scripts when the system is under load though. I might be able to retrofit this to all our existing scripts by blocking after a commit or rollback if the system is loaded using similar rules to the garbage collector (no transactions open longer than 30 mins and replication lag < 30 seconds).
[16:24] <stub> Yes, we should switch the master to wildcherry before the next rollout or during the next rollout if we don't mind a few minutes downtime on the SSO server.
[16:24] <stub> I need to go over rollout procedures with the losas once I know how much of read-only-launchpad lands this cycle.
[16:24] <stub> EOM
[16:25] <mthaddon> stub: when do you expect to know how much of read-only-launchpad lands this cycle?
[16:26] <stub> I don't know yet ;) What would be a good deadline?
[16:27] <mthaddon> stub: I think we'd need to discuss things at the latest by the end of week 3, so sometime before then?
[16:27] <mthaddon> in terms of rollout stuff, I mean
[16:28] <stub> Sure. I was thinking early next week to fix the plan so that works.
[16:28] <matsubara> anything else for stub?
[16:29] <matsubara> or anything else before I close?
[16:29] <matsubara> thanks stub
[16:29] <mthaddon> stub: cool, thx
[16:29] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:29] <matsubara> #endmeeting
[16:29] <MootBot> Meeting finished at 10:29.
=== matsubara is now known as matsubara-lunch
=== matsubara-lunch is now known as matsubara
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
=== salgado is now known as salgado-brb
=== salgado-brb is now known as salgado