[15:57] <Ursinha> MootBot, hi hi -.-
[15:59] <sinzui> Wake up MootBot
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <gary-sprint> me
[16:00] <gary-sprint> :-D
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <sinzui> me
[16:00] <gary-sprint> uh, me
[16:00] <Ursinha> me
[16:00] <matsubara> rockstar, Chex, bigjools, allenap: hi
[16:01] <allenap> me
[16:01] <matsubara> apologies from stub
[16:02] <rockstar> me
[16:02] <mthaddon> me
[16:03] <bigjools> me
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:03] <mthaddon> matsubara: I'm sitting in for Chex this meeting as he's working on U1 stuff
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:03] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:03] <matsubara>  * DBA report (stub)
[16:03] <matsubara>  * Proposed items
[16:03] <matsubara> thanks mthaddon
[16:03] <matsubara> [TOPIC] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:03] <matsubara> * matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas
[16:03] <matsubara> * Chex to check app server logs and apache logs to see if it can shed any light in the high load issue.
[16:03] <matsubara> * adeuring to check with gmb about checkwatches failure
[16:03] <matsubara> * danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance
[16:03] <matsubara> * bigjools to investigate update-cache failure and reply back to the list
[16:04] <danilos> matsubara: the bug tells you what it was :)
[16:04] <danilos> oh, I forgot to 'me' myself
[16:04] <matsubara> I'll finish up my action today
[16:04] <matsubara> thanks danilos
[16:04] <bigjools> I chatted to curtis and as far as wecan tell it was caused by something else holding a transaction/table open
[16:05] <bigjools> not much I can do
[16:06] <matsubara> gmb replied to checkwatches failure email. it was a hung process which was killed and service resumed
[16:06] <sinzui> Since the PRF ran the following days, I believe it was a long running process that worried our watching proc
[16:06] <matsubara> bigjools, thanks for checking. I don't see new emails from that script failing so I take it's working normally
[16:06] <bigjools> yep
[16:07] <matsubara> mthaddon, any luck investigating the high loading issue?
[16:07] <matsubara> s/loading/load/
[16:08] <mthaddon> matsubara: I wasn't aware that was something we were following up on - not sure what the latest is, but I guess part of it plays into the new SplitIt stuff
[16:08] <mthaddon> i.e. we've just brought a whole bunch of new servers online so we need to see what effect this has on the overall load of the system
[16:09] <matsubara> all right. I'll take that item off the list and if high load shows up in the graphs we can pursue further
[16:09] <mthaddon> k
[16:09] <matsubara> thanks all, moving on
[16:09] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:09] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:10] <Ursinha> it'sme
[16:10] <Ursinha> gary_poster, bug 331990, can we CP it?
[16:10]  * sinzui stares at Ursinha
[16:10] <Ursinha> s/gary_poster/gary-sprint/
[16:10] <gary-sprint> Ursinha: I do not have CP-foo.
[16:12] <Ursinha> allenap, can we have a fix for bug 438802 and maybe CP it?
[16:12] <matsubara> gary-sprint, is this a matter of updating the lazr.restful lib used by lpnet?
[16:12] <Ursinha> allenap, also, we have bug 438985, it's in progress but without activity for a some time
[16:12] <Ursinha> allenap, and bug 458180, that's BugTask index timeouts
[16:12] <Ursinha> sinzui, I've filed bug 458169 and bug 458189, the timeouts on Milestone and DistroSeries index pages
[16:12] <Ursinha> rockstar, can we have a fix for bug 442981?
[16:12] <gary-sprint> Ursinha: maybe I misunderstood.  are you asking for CP-blessing or for CP-shepherding?  If the latter, sure, we can shepherd.
[16:13] <Ursinha> gary-sprint, shepherding
[16:13] <sinzui> Ursinha: I replied that I beleive they are dups of 455812
[16:13] <sinzui> brad is already working on it
[16:13] <Ursinha> bug 455812
[16:13] <Ursinha> hmmm
[16:13] <gary-sprint> matsubara: not sure, will ask leonardr.
[16:13] <Ursinha> sinzui,I'll mark it as a dupe then, thanks
[16:13] <sinzui> not yet
[16:13] <rockstar> Ursinha, the fix for that bug is closing it as a duplicate of bug 457541
[16:13] <Ursinha> oh
[16:14] <rockstar> Ursinha, also, that bug is Fix Released.
[16:14] <matsubara> [action] gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
[16:14] <MootBot> ACTION received:  gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
[16:14] <gary-sprint> +1
[16:14] <Ursinha> rockstar,it still happens, how come?
[16:14] <rockstar> Ursinha, so yes, you may have it before it was asked.
[16:14] <sinzui> I have assign the distroseries +index to edwin. I think EdwinGrubbs and bac will find this is the same problem
[16:15] <sinzui> The oopses of the two new bugs look the the oopses I have been tracking in the older bug
[16:15] <rockstar> Ursinha, doesn't oops for me.
[16:15] <Ursinha> rockstar, so the summaries are lying :)
[16:16] <rockstar> Ursinha, does this url oops for you? https://edge.launchpad.net/launchpad-project/+activereviews
[16:16] <Ursinha> rockstar, well, it's loading... I'll keep my eye on it and if needed reopen it, right?
[16:16] <Ursinha> rockstar, thanks
[16:17] <Ursinha> allenap, hi :)
[16:17] <allenap> Ursinha: I talk to deryck about getting bug 438802 fixed, and gmb about bug 438985.
[16:17] <Ursinha> allenap, thanks
[16:17] <allenap> Ursinha: Bug 458180 is a perennial problem.
[16:17] <Ursinha> allenap, I see the main offender is bug #1
[16:17] <Ursinha> yes, *sigh*
[16:17] <allenap> Ursinha: Yeah, it always is :)
[16:18] <rockstar> Ursinha, yes, but I can't see how it'd get reopened.  It was bad data, we fixededed the database records.
[16:18] <danilos> Ursinha: and you just made it worse with a reference now :)
[16:18] <Ursinha> danilos, yes, just to prove my point :P
[16:18] <matsubara> allenap, it still happens in other bugs too as per jono's email to launchpad-dev about ubuttu timing out
[16:18] <Ursinha> allenap, there are some oopses not caused by #1
[16:19] <Ursinha> thanks allenap
[16:19] <allenap> matsubara: Okay, as someone said, perhaps it's the +text interface.
[16:19] <Ursinha> gary-sprint, the "buildbot failure in Launchpad on jscheck", is it severe?
[16:19] <matsubara> allenap, I briefly trawled the summaries and there are a some sofr timeouts on +text, but soft timeouts shouldn't be affecting ubottu
[16:19] <allenap> matsubara, Ursinha: We need to do something more drastic to get the bug page quicker I think. Caching, etc, and that's coming alone. We've done a lot of the other things we can think of, but I'll discuss it with the team.
[16:20] <allenap> matsubara: That's interesting.
[16:20] <allenap> s/alone/along/
[16:21] <Ursinha> I see some emails from francis and rockstar talking about it, is there something that can or needs to be done?
[16:21] <matsubara> allenap, perhaps those timeouts are not being logged as OOPSes? similar to 500 we see eventually from apache
[16:21] <Ursinha> gary-sprint, ^
[16:21] <matsubara> to the 500 errors I mean
[16:21] <rockstar> Ursinha, gary-sprint, it is my belief that windmill sucks.
[16:21] <gary-sprint> Ursinha: it does not appear to be a problem in the basic buildbot setup at the moment.    There are failures in the tests.  This doesn't seem to be a foundations issue AFAICT.  Björn may very well be able to help when he returns
[16:22] <allenap> matsubara: Okay, I'm not sure what you mean, but we can talk about it after the meeting.
[16:22] <Ursinha> right gary-sprint, thanks for the info
[16:22] <matsubara> allenap, sure thing. I'll find the bug I'm referring to
[16:22] <allenap> matsubara: Thanks.
[16:22] <matsubara> [action] allenap and matsubara to talk about the timeouts on bug pages
[16:22] <MootBot> ACTION received:  allenap and matsubara to talk about the timeouts on bug pages
[16:23] <Ursinha> right, I'm done here
[16:23] <gary-sprint> rockstar: that's probably a given.  The more interesting question is whether it sucks worse than the alternatives.  My impression is no, but a champion could fight for an alternate view,
[16:23] <gary-sprint> .
[16:23] <Ursinha> allenap, is it possible to ask for a cp for bug 438802 when it's fixed?
[16:23] <rockstar> gary-sprint, sadly, there is no better alternative. Windmill sucks less than anything else out there.
[16:23] <gary-sprint> :-)
[16:23] <allenap> Ursinha: Sure.
[16:24] <Ursinha> anmar was having problems yesterday with bugs with chinese chars, I think it's worth doing a CP
[16:25] <Ursinha> thanks allenap
[16:25] <allenap> Ursinha: np, thank you :)
[16:26] <Ursinha> :)
[16:26] <matsubara> ok, two fix committed critical bugs
[16:26] <matsubara> rockstar, we had some failures on the update_preview_diffs script
[16:26] <matsubara> on the 19th
[16:28] <rockstar> matsubara, yeah, we're currently in the process of fixing the various oopses that script creates.
[16:28] <matsubara> rockstar, ok. can you give me the bug numbers after the meeting?
[16:29] <rockstar> matsubara, there are many.
[16:29] <matsubara> gladly we have an oops tag to filter those :-)
[16:29] <matsubara> rockstar, I'll ping you after the meeting
[16:30] <Ursinha> thanks everyone
[16:30] <matsubara> I think that's all for this section. thanks everyone
[16:30] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:30] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:30] <mthaddon> SplitIt is the big one this week - now complete with exception of Auth DB split.
[16:30] <mthaddon> New App servers brought online after haproxy throttlng of connections, we're watching how things are progressing
[16:30] <mthaddon> A number of CPs done this week
[16:30] <mthaddon> Is everyone clear on the new CP process?
[16:30] <mthaddon> Shipit now managed by ISD, and CPs to be approved by nigelp
[16:30] <mthaddon> Some app servers dying, loggerhead dying, poppy died once - is there a process for reviewing the Incident Log?
[16:30] <mthaddon> That's about it
[16:31] <matsubara> mthaddon, last I heard Francis was the one to champion the Incident Log process.
[16:32] <mthaddon> matsubara: basically we want to be sure someone's reviewing it to look for operational trends in production
[16:32] <bigjools> did he mention making trvial wiki edits for codebounce so we don't get email for those?
[16:32] <matsubara> ideally we won't need that codebounce all the time :-)
[16:32] <Ursinha> matsubara, +1 :)
[16:33] <mthaddon> bigjools: if we have to get alerts and go through the whole restart, edit wiki nightmare, you can put up with a few wiki edit notifications :)
[16:33]  * matsubara looks at rockstar 
[16:33] <bigjools> mthaddon: well, no I don't :)
[16:33] <danilos> mthaddon: the concern is that we may learn to ignore it unless we can filter stuff out *we* can't do anything about
[16:33] <rockstar> mthaddon, I'm subscribed and get the pleasure of seeing every time you restart loggerhead.
[16:33] <matsubara> any news about the codebrowser dying all the time?
[16:33] <bigjools> what danilos said
[16:33] <danilos> mthaddon: specifically, translations or soyuz team can't help much with codebrowse restarts
[16:34] <rockstar> matsubara, we are bringing someone on to look into the codebrowse issue.  That's all I know.  We certainly don't have the bandwidth currently to do it.
[16:34] <mthaddon> fwiw, I usually do trivial that one - I guess maybe the other losas don't - will mention it
[16:34] <danilos> bigjools: just as an example, and these are very, very common
[16:34] <sinzui> I believe there is a plan to but people to work on loggerhead
[16:34] <danilos> mthaddon: in general, anything else shoudn't be a trivial edit, and codebrowse should, that would help old men like bigjools deal with their email :)
[16:34] <bigjools> ha
[16:35] <danilos> sinzui: yeah, as flacoste mentioned today, I think we are having a contract that starts today or tomorrow
[16:35] <matsubara> that's great news
[16:35] <danilos> mthaddon: but, do note that most team leads are subscribed to LPIncidentLog, and if one isn't, feel free to poke them about it
[16:36] <mthaddon> danilos: k, thx
[16:37] <matsubara> that's all mthaddon ?
[16:37] <mthaddon> yep
[16:37] <matsubara> all right. thanks everyone
[16:38] <matsubara> [TOPIC] * DBA report (stub)
[16:38] <MootBot> New Topic:  * DBA report (stub)
[16:38] <matsubara> stub is on vacation and looks like the db is fine
[16:38] <matsubara> AFAICT
[16:38] <matsubara> so let's move on.
[16:39] <matsubara> [action] matsubara to talk to stub about the DBA report when he gets back
[16:39] <MootBot> ACTION received:  matsubara to talk to stub about the DBA report when he gets back
[16:39] <matsubara> [TOPIC] * Proposed items
[16:39] <MootBot> New Topic:  * Proposed items
[16:39] <matsubara> no new proposed items
[16:40] <matsubara> and I think that's all for today
[16:40] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[16:40] <matsubara> #endmeeting
[16:40] <MootBot> Meeting finished at 10:40.
[16:40] <Ursinha> thanks my dearest colleagues