=== mthaddon` is now known as mthaddon === danilo-afk is now known as danilos === mrevell is now known as mrevell-lunch === salgado-afk is now known as salgado === mrevell-lunch is now known as mrevell === gary_poster is now known as gary-sprint [15:57] MootBot, hi hi -.- [15:59] Wake up MootBot [16:00] #startmeeting [16:00] Meeting started at 10:00. The chair is matsubara. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] me [16:00] :-D [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] [TOPIC] Roll Call [16:00] New Topic: Roll Call [16:00] me [16:00] uh, me [16:00] me [16:00] rockstar, Chex, bigjools, allenap: hi [16:01] me [16:01] apologies from stub [16:02] me [16:02] me [16:03] me [16:03] [TOPIC] Agenda [16:03] New Topic: Agenda [16:03] matsubara: I'm sitting in for Chex this meeting as he's working on U1 stuff [16:03] * Actions from last meeting [16:03] * Oops report & Critical Bugs & Broken scripts [16:03] * Operations report (mthaddon/Chex/spm/mbarnett) [16:03] * DBA report (stub) [16:03] * Proposed items [16:03] thanks mthaddon [16:03] [TOPIC] * Actions from last meeting [16:03] New Topic: * Actions from last meeting [16:03] * matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas [16:03] * Chex to check app server logs and apache logs to see if it can shed any light in the high load issue. [16:03] * adeuring to check with gmb about checkwatches failure [16:03] * danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance [16:03] Launchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,Fix released] https://launchpad.net/bugs/438039 [16:03] * bigjools to investigate update-cache failure and reply back to the list [16:04] matsubara: the bug tells you what it was :) [16:04] oh, I forgot to 'me' myself [16:04] I'll finish up my action today [16:04] thanks danilos [16:04] I chatted to curtis and as far as wecan tell it was caused by something else holding a transaction/table open [16:05] not much I can do [16:06] gmb replied to checkwatches failure email. it was a hung process which was killed and service resumed [16:06] Since the PRF ran the following days, I believe it was a long running process that worried our watching proc [16:06] bigjools, thanks for checking. I don't see new emails from that script failing so I take it's working normally [16:06] yep [16:07] mthaddon, any luck investigating the high loading issue? [16:07] s/loading/load/ [16:08] matsubara: I wasn't aware that was something we were following up on - not sure what the latest is, but I guess part of it plays into the new SplitIt stuff [16:08] i.e. we've just brought a whole bunch of new servers online so we need to see what effect this has on the overall load of the system [16:09] all right. I'll take that item off the list and if high load shows up in the graphs we can pursue further [16:09] k [16:09] thanks all, moving on [16:09] [TOPIC] * Oops report & Critical Bugs & Broken scripts [16:09] New Topic: * Oops report & Critical Bugs & Broken scripts [16:10] it'sme [16:10] gary_poster, bug 331990, can we CP it? [16:10] * sinzui stares at Ursinha [16:10] Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 [16:10] s/gary_poster/gary-sprint/ [16:10] Ursinha: I do not have CP-foo. [16:12] allenap, can we have a fix for bug 438802 and maybe CP it? [16:12] gary-sprint, is this a matter of updating the lazr.restful lib used by lpnet? [16:12] allenap, also, we have bug 438985, it's in progress but without activity for a some time [16:12] Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 [16:12] allenap, and bug 458180, that's BugTask index timeouts [16:12] Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985 [16:12] Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180 [16:12] sinzui, I've filed bug 458169 and bug 458189, the timeouts on Milestone and DistroSeries index pages [16:12] rockstar, can we have a fix for bug 442981? [16:12] Launchpad bug 458169 in launchpad-registry "Distroseries:+index page timing out" [High,Triaged] https://launchpad.net/bugs/458169 [16:12] Launchpad bug 458189 in launchpad-registry "Milestone:+index pages timing out" [Undecided,New] https://launchpad.net/bugs/458189 [16:12] Launchpad bug 442981 in launchpad-code "launchpad-project/+activereviews is OOPSing with TypeError (dup-of: 457541)" [High,Triaged] https://launchpad.net/bugs/442981 [16:12] Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541 [16:12] Ursinha: maybe I misunderstood. are you asking for CP-blessing or for CP-shepherding? If the latter, sure, we can shepherd. [16:13] gary-sprint, shepherding [16:13] Ursinha: I replied that I beleive they are dups of 455812 [16:13] brad is already working on it [16:13] bug 455812 [16:13] Launchpad bug 455812 in launchpad-registry "distroseries milestone timeout" [High,Triaged] https://launchpad.net/bugs/455812 [16:13] hmmm [16:13] matsubara: not sure, will ask leonardr. [16:13] sinzui,I'll mark it as a dupe then, thanks [16:13] not yet [16:13] Ursinha, the fix for that bug is closing it as a duplicate of bug 457541 [16:13] Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541 [16:13] oh [16:14] Ursinha, also, that bug is Fix Released. [16:14] [action] gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990 [16:14] ACTION received: gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990 [16:14] Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 [16:14] +1 [16:14] rockstar,it still happens, how come? [16:14] Ursinha, so yes, you may have it before it was asked. [16:14] I have assign the distroseries +index to edwin. I think EdwinGrubbs and bac will find this is the same problem [16:15] The oopses of the two new bugs look the the oopses I have been tracking in the older bug [16:15] Ursinha, doesn't oops for me. [16:15] rockstar, so the summaries are lying :) [16:16] Ursinha, does this url oops for you? https://edge.launchpad.net/launchpad-project/+activereviews [16:16] rockstar, well, it's loading... I'll keep my eye on it and if needed reopen it, right? [16:16] rockstar, thanks [16:17] allenap, hi :) [16:17] Ursinha: I talk to deryck about getting bug 438802 fixed, and gmb about bug 438985. [16:17] Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 [16:17] Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985 [16:17] allenap, thanks [16:17] Ursinha: Bug 458180 is a perennial problem. [16:17] Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180 [16:17] allenap, I see the main offender is bug #1 [16:17] https://bugs.launchpad.net/ubuntu/+bug/1 (Timeout) [16:17] yes, *sigh* [16:17] Ursinha: Yeah, it always is :) [16:18] Ursinha, yes, but I can't see how it'd get reopened. It was bad data, we fixededed the database records. [16:18] Ursinha: and you just made it worse with a reference now :) [16:18] danilos, yes, just to prove my point :P [16:18] allenap, it still happens in other bugs too as per jono's email to launchpad-dev about ubuttu timing out [16:18] allenap, there are some oopses not caused by #1 [16:19] thanks allenap [16:19] matsubara: Okay, as someone said, perhaps it's the +text interface. [16:19] gary-sprint, the "buildbot failure in Launchpad on jscheck", is it severe? [16:19] allenap, I briefly trawled the summaries and there are a some sofr timeouts on +text, but soft timeouts shouldn't be affecting ubottu [16:19] matsubara, Ursinha: We need to do something more drastic to get the bug page quicker I think. Caching, etc, and that's coming alone. We've done a lot of the other things we can think of, but I'll discuss it with the team. [16:20] matsubara: That's interesting. [16:20] s/alone/along/ [16:21] I see some emails from francis and rockstar talking about it, is there something that can or needs to be done? [16:21] allenap, perhaps those timeouts are not being logged as OOPSes? similar to 500 we see eventually from apache [16:21] gary-sprint, ^ [16:21] to the 500 errors I mean [16:21] Ursinha, gary-sprint, it is my belief that windmill sucks. [16:21] Ursinha: it does not appear to be a problem in the basic buildbot setup at the moment. There are failures in the tests. This doesn't seem to be a foundations issue AFAICT. Björn may very well be able to help when he returns [16:22] matsubara: Okay, I'm not sure what you mean, but we can talk about it after the meeting. [16:22] right gary-sprint, thanks for the info [16:22] allenap, sure thing. I'll find the bug I'm referring to [16:22] matsubara: Thanks. [16:22] [action] allenap and matsubara to talk about the timeouts on bug pages [16:22] ACTION received: allenap and matsubara to talk about the timeouts on bug pages [16:23] right, I'm done here [16:23] rockstar: that's probably a given. The more interesting question is whether it sucks worse than the alternatives. My impression is no, but a champion could fight for an alternate view, [16:23] . [16:23] allenap, is it possible to ask for a cp for bug 438802 when it's fixed? [16:23] Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 [16:23] gary-sprint, sadly, there is no better alternative. Windmill sucks less than anything else out there. === salgado is now known as salgado-lunch [16:23] :-) [16:23] Ursinha: Sure. [16:24] anmar was having problems yesterday with bugs with chinese chars, I think it's worth doing a CP [16:25] thanks allenap [16:25] Ursinha: np, thank you :) [16:26] :) [16:26] ok, two fix committed critical bugs [16:26] rockstar, we had some failures on the update_preview_diffs script [16:26] on the 19th [16:28] matsubara, yeah, we're currently in the process of fixing the various oopses that script creates. [16:28] rockstar, ok. can you give me the bug numbers after the meeting? [16:29] matsubara, there are many. [16:29] gladly we have an oops tag to filter those :-) [16:29] rockstar, I'll ping you after the meeting [16:30] thanks everyone [16:30] I think that's all for this section. thanks everyone [16:30] [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) [16:30] New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) [16:30] SplitIt is the big one this week - now complete with exception of Auth DB split. [16:30] New App servers brought online after haproxy throttlng of connections, we're watching how things are progressing [16:30] A number of CPs done this week [16:30] Is everyone clear on the new CP process? [16:30] Shipit now managed by ISD, and CPs to be approved by nigelp [16:30] Some app servers dying, loggerhead dying, poppy died once - is there a process for reviewing the Incident Log? [16:30] That's about it [16:31] mthaddon, last I heard Francis was the one to champion the Incident Log process. [16:32] matsubara: basically we want to be sure someone's reviewing it to look for operational trends in production [16:32] did he mention making trvial wiki edits for codebounce so we don't get email for those? [16:32] ideally we won't need that codebounce all the time :-) [16:32] matsubara, +1 :) [16:33] bigjools: if we have to get alerts and go through the whole restart, edit wiki nightmare, you can put up with a few wiki edit notifications :) [16:33] * matsubara looks at rockstar [16:33] mthaddon: well, no I don't :) [16:33] mthaddon: the concern is that we may learn to ignore it unless we can filter stuff out *we* can't do anything about [16:33] mthaddon, I'm subscribed and get the pleasure of seeing every time you restart loggerhead. [16:33] any news about the codebrowser dying all the time? [16:33] what danilos said [16:33] mthaddon: specifically, translations or soyuz team can't help much with codebrowse restarts [16:34] matsubara, we are bringing someone on to look into the codebrowse issue. That's all I know. We certainly don't have the bandwidth currently to do it. [16:34] fwiw, I usually do trivial that one - I guess maybe the other losas don't - will mention it [16:34] bigjools: just as an example, and these are very, very common [16:34] I believe there is a plan to but people to work on loggerhead [16:34] mthaddon: in general, anything else shoudn't be a trivial edit, and codebrowse should, that would help old men like bigjools deal with their email :) [16:34] ha [16:35] sinzui: yeah, as flacoste mentioned today, I think we are having a contract that starts today or tomorrow [16:35] that's great news [16:35] mthaddon: but, do note that most team leads are subscribed to LPIncidentLog, and if one isn't, feel free to poke them about it [16:36] danilos: k, thx [16:37] that's all mthaddon ? [16:37] yep [16:37] all right. thanks everyone [16:38] [TOPIC] * DBA report (stub) [16:38] New Topic: * DBA report (stub) [16:38] stub is on vacation and looks like the db is fine [16:38] AFAICT [16:38] so let's move on. [16:39] [action] matsubara to talk to stub about the DBA report when he gets back [16:39] ACTION received: matsubara to talk to stub about the DBA report when he gets back [16:39] [TOPIC] * Proposed items [16:39] New Topic: * Proposed items [16:39] no new proposed items [16:40] and I think that's all for today [16:40] Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. [16:40] #endmeeting [16:40] Meeting finished at 10:40. [16:40] thanks my dearest colleagues === matsubara is now known as matsubara-lunch === salgado-lunch is now known as salgado === EdwinGrubbs is now known as Edwin-lunch === danilos is now known as danilo-afk === matsubara-lunch is now known as matsubara === Edwin-lunch is now known as EdwinGrubbs === salgado is now known as salgado-afk === matsubara is now known as matsubara-afk