| === ursula_ is now known as Ursinha | ||
| === mrevell is now known as mrevell-lunch | ||
| === mrevell-lunch is now known as mrevell | ||
| matsubara | #startmeeting | 16:00 |
|---|---|---|
| MootBot | Meeting started at 10:00. The chair is matsubara. | 16:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 16:00 |
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 16:00 |
| matsubara | [TOPIC] Roll Call | 16:00 |
| MootBot | New Topic: Roll Call | 16:00 |
| matsubara | Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! | 16:00 |
| flacoste | me | 16:01 |
| matsubara | hi francis | 16:01 |
| Ursinha | me | 16:01 |
| matsubara | rockstar, bigjools, henninge intellectronica sinzui: hi | 16:01 |
| sinzui | me | 16:01 |
| bigjools | meeee | 16:01 |
| intellectronica | me | 16:01 |
| henninge | me | 16:01 |
| matsubara | hi stub | 16:02 |
| stub | yo | 16:03 |
| matsubara | herb: | 16:03 |
| matsubara | ok, herb can join in later. let's move on | 16:03 |
| matsubara | [TOPIC] Agenda | 16:03 |
| MootBot | New Topic: Agenda | 16:03 |
| matsubara | * Actions from last meeting | 16:03 |
| matsubara | * Oops report & Critical Bugs | 16:03 |
| matsubara | * Operations report (mthaddon/herb/spm) | 16:03 |
| matsubara | * DBA report (stub) | 16:03 |
| matsubara | [TOPIC] * Actions from last meeting | 16:03 |
| MootBot | New Topic: * Actions from last meeting | 16:03 |
| matsubara | * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint) | 16:03 |
| matsubara | * ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses" | 16:03 |
| matsubara | * filed bug 360846 | 16:03 |
| matsubara | * intellectronica to talk to gmb about bug 269538 | 16:03 |
| matsubara | ok, Ursinha done hers | 16:04 |
| sinzui | wow I do suck | 16:04 |
| matsubara | [action] sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint) | 16:04 |
| MootBot | ACTION received: sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint) | 16:04 |
| sinzui | matsubara: I will do this email IMMEDIATELY after this meeting. | 16:04 |
| matsubara | thanks sinzui | 16:04 |
| * sinzui starts the subect line now | 16:04 | |
| intellectronica | matsubara: sorry, i didn't. will try to do that later today | 16:04 |
| matsubara | intellectronica: cool. thanks. shall I keep the action item for the next meeting? | 16:05 |
| rockstar | me | 16:05 |
| intellectronica | matsubara: yes, please | 16:05 |
| matsubara | [action] intellectronica to talk to gmb about bug 269538 | 16:05 |
| MootBot | ACTION received: intellectronica to talk to gmb about bug 269538 | 16:05 |
| matsubara | ok, let's move on. thanks sinzui and intellectronica | 16:06 |
| matsubara | hi rockstar | 16:06 |
| rockstar | hi | 16:06 |
| matsubara | [TOPIC] * Oops report & Critical Bugs | 16:06 |
| MootBot | New Topic: * Oops report & Critical Bugs | 16:06 |
| matsubara | Ursinha: take the stage please | 16:07 |
| Ursinha | I have only one bug for foundations, bug 357307 | 16:07 |
| * Ursinha kicks the bot | 16:07 | |
| Ursinha | https://bugs.edge.launchpad.net/launchpad-foundations/+bug/357307 | 16:07 |
| Ursinha | flacoste, ^ | 16:08 |
| flacoste | yep, looking at this now | 16:08 |
| flacoste | Ursinha: what's the priority on this? | 16:10 |
| matsubara | I have one for sinzui: bug 358332 | 16:10 |
| flacoste | sinzui: do you think salgado could look into that token bug? | 16:10 |
| matsubara | curtis, do you agree with the importance I set for https://bugs.edge.launchpad.net/launchpad-registry/+bug/358332 ? | 16:10 |
| Ursinha | flacoste, we're had 25 yesterday on lpnet | 16:11 |
| Ursinha | about this average a day, oopses count | 16:11 |
| sinzui | Well I do not agree with Medium, I do not ever use that status | 16:11 |
| matsubara | sinzui: can you reset the importance to a more meaningful value then? | 16:12 |
| sinzui | flacoste: Would you like to take salgado to your team then ;) | 16:12 |
| Ursinha | haha | 16:12 |
| flacoste | sinzui: you know my opinion on that one :-) | 16:13 |
| sinzui | matsubara: Since I have committed to working on series issues, I will try to land a fix for bug 358332 this release | 16:13 |
| flacoste | sinzui: and I'll take that as a yes | 16:13 |
| matsubara | thank you sinzui | 16:14 |
| sinzui | Yes, I will ask salgado to look at the bug 357307 | 16:14 |
| matsubara | [action] sinzui to take bug 358332 | 16:15 |
| MootBot | ACTION received: sinzui to take bug 358332 | 16:15 |
| matsubara | [action] sinzui to ask salgado to fix bug 357307 | 16:15 |
| MootBot | ACTION received: sinzui to ask salgado to fix bug 357307 | 16:15 |
| matsubara | All critical bugs are fix committed | 16:16 |
| matsubara | so, I think that's all. anything else Ursinha ? | 16:17 |
| Ursinha | no matsubara | 16:17 |
| Ursinha | thanks all | 16:17 |
| matsubara | ok, thanks everyone | 16:17 |
| matsubara | I'm going to skip herb's section since he's not here yet. | 16:17 |
| mthaddon | matsubara: I'm here | 16:17 |
| matsubara | oh | 16:17 |
| matsubara | hi mthaddon | 16:17 |
| mthaddon | matsubara: herb's off | 16:17 |
| matsubara | [TOPIC] * Operations report (mthaddon/herb/spm) | 16:18 |
| MootBot | New Topic: * Operations report (mthaddon/herb/spm) | 16:18 |
| mthaddon | - Preparations for new rollout procedure - need to confirm a fair amount of stuff with stub before the next rollout | 16:18 |
| mthaddon | - PostgreSQL upgrade yesterday meant some downtime | 16:18 |
| mthaddon | - Need to decide on whether we're switching the master to wildcherry and if so when | 16:18 |
| mthaddon | I think that's it unless there are any questions | 16:18 |
| intellectronica | mthaddon: i have a question | 16:18 |
| matsubara | what's this new rollout procedure? | 16:18 |
| mthaddon | matsubara: SSO related | 16:18 |
| mthaddon | intellectronica: sure | 16:18 |
| intellectronica | mthaddon: yesterday morning (UTC) forster needed a restart because it was overwhelmed. Spads helped me with that. do you know what happened? | 16:19 |
| mthaddon | matsubara: involves splitting out a slave from as standalone so we can continue to serve SSO | 16:19 |
| mthaddon | intellectronica: I don't - we've had some issues with forster over the past few weeks | 16:19 |
| matsubara | mthaddon: cool. | 16:19 |
| mthaddon | intellectronica: i.e. this isn't the first time it's needed a kicking - I'll see if I can look into it a little more | 16:20 |
| intellectronica | mthaddon: ok, as long as you're not surprised. sorry i forgot to mention this yesterday | 16:20 |
| matsubara | anything else for mthaddon? | 16:21 |
| mthaddon | intellectronica: we're moving the incoming email parsing script off there, so hopefully that'll reduce load a little, although I'd be surprised if it was related to that | 16:21 |
| flacoste | mthaddon: any idea on the surge in buildbot failures yesterday? | 16:21 |
| mthaddon | flacoste: I was off yesterday so I wasn't aware of that, no | 16:21 |
| flacoste | mthaddon: it looks it was restarted, i assume by spm since gary nor i did it | 16:21 |
| intellectronica | mthaddon: if it's not the cause, then at least it won't stop working when whatever is the cause starts going wild :) | 16:22 |
| flacoste | mthaddon: ok, i'll check with spm later | 16:22 |
| mthaddon | intellectronica: absolutely :) | 16:22 |
| matsubara | flacoste: maybe it was related to the unexpected downtime we had yesterday | 16:22 |
| matsubara | flacoste: all over the DC | 16:23 |
| flacoste | matsubara: none of this lives in the DC yet | 16:23 |
| matsubara | well, but the code buildbot is pulling does :-) | 16:23 |
| matsubara | anyway, spm can clarify that later. | 16:24 |
| matsubara | thanks everyone | 16:24 |
| matsubara | [TOPIC] * DBA report (stub) | 16:24 |
| MootBot | New Topic: * DBA report (stub) | 16:24 |
| stub | I have been on public holidays or ill for the last week, but things still seem to be running which is nice. | 16:24 |
| stub | We got a replication lag spike the other night - all the systems coped well while it was cleared. Possibly the rosetta imports being busier than usual. Possibly the rosetta imports where just delayed by something else. I don't think anyone has traced what processes where running at that time yet from the script activity records or the output on launchpad-errors. | 16:24 |
| stub | We are going to have to be more agressive at blocking scripts when the system is under load though. I might be able to retrofit this to all our existing scripts by blocking after a commit or rollback if the system is loaded using similar rules to the garbage collector (no transactions open longer than 30 mins and replication lag < 30 seconds). | 16:24 |
| stub | Yes, we should switch the master to wildcherry before the next rollout or during the next rollout if we don't mind a few minutes downtime on the SSO server. | 16:24 |
| stub | I need to go over rollout procedures with the losas once I know how much of read-only-launchpad lands this cycle. | 16:24 |
| stub | EOM | 16:24 |
| mthaddon | stub: when do you expect to know how much of read-only-launchpad lands this cycle? | 16:25 |
| stub | I don't know yet ;) What would be a good deadline? | 16:26 |
| mthaddon | stub: I think we'd need to discuss things at the latest by the end of week 3, so sometime before then? | 16:27 |
| mthaddon | in terms of rollout stuff, I mean | 16:27 |
| stub | Sure. I was thinking early next week to fix the plan so that works. | 16:28 |
| matsubara | anything else for stub? | 16:28 |
| matsubara | or anything else before I close? | 16:29 |
| matsubara | thanks stub | 16:29 |
| mthaddon | stub: cool, thx | 16:29 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. | 16:29 |
| matsubara | #endmeeting | 16:29 |
| MootBot | Meeting finished at 10:29. | 16:29 |
| === matsubara is now known as matsubara-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| === salgado is now known as salgado-lunch | ||
| === salgado-lunch is now known as salgado | ||
| === salgado is now known as salgado-brb | ||
| === salgado-brb is now known as salgado | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!