=== thumper is now known as thumper-afk === thumper-afk is now known as thumper === bigjools is now known as bigjools-afk === bigjools-afk is now known as bigjools === cprov is now known as cprov-lunch [15:59] * Ursinha looks at MootBot [15:59] oh, he's here [16:00] #startmeeting [16:00] Meeting started at 10:00. The chair is matsubara. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] [TOPIC] Roll Call [16:00] New Topic: Roll Call [16:00] me [16:00] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [16:00] me [16:00] ni! [16:00] me [16:01] rockstar: that's plural, 'mi' is singular. [16:01] me [16:01] me [16:01] Chex, hi [16:01] gary_poster, hi [16:01] henninge, :) Making a Monty Python reference. [16:01] me [16:01] thanks [16:01] oh, don't know that one. [16:01] apologies from stub [16:01] rockstar: Mine was an Esperanto reference [16:01] [TOPIC] Agenda [16:01] New Topic: Agenda [16:02] * Actions from last meeting [16:02] * Oops report & Critical Bugs & Broken scripts [16:02] * Operations report (mthaddon/Chex/spm/mbarnett) [16:02] * DBA report (stub) [16:02] * Proposed items [16:02] [TOPIC] * Actions from last meeting [16:02] New Topic: * Actions from last meeting [16:02] * barry to continue debug on bug 403606 [16:02] * intellectronica to take a look or find someone to take a look on bug 408738 [16:02] * matsubara to chase ubottu maintainer to fix regexp used to identify oopses(i.e. oops-tools shouldn't trigger the bot) [16:02] * filed bug 420032 and talked to jussi01 on #ubuntu-bots about it. [16:02] Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 [16:02] Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738 [16:02] Launchpad bug 420032 in ubuntu-bots "don't trigger OOPS URL when 'oops-tools' is mentioned" [Undecided,New] https://launchpad.net/bugs/420032 [16:02] matsubara: hi [16:02] ubottu, oops-tools [16:02] Sorry, I don't know anything about oops-tools [16:02] oops-tools [16:03] cool. looks like that one was fixed :-) [16:03] \o/ [16:03] heh [16:03] ubottu, good boy [16:03] Sorry, I don't know anything about good boy [16:03] lol [16:03] matsubara: it's on the radar, but we haven't yet had the chance to fix it. as you can imagine, v3.0 UI is keeping everyone very busy [16:03] sinzui, any news about 403606? [16:04] intellectronica, right. shall I keep it in the action items list? [16:04] no [16:04] matsubara: yes, please [16:04] I ask my team to only work on UI [16:04] intellectronica, thanks [16:04] thanks intellectronica and sinzui [16:05] [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff [16:05] Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 [16:05] ACTION received: * barry to continue debug on bug 403606 after finishing 3.0 UI stuff [16:05] matsubara: Since there was nothing obviously wrong, we cannot judge the scope of the issue [16:05] [action] * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff [16:05] Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738 [16:05] ACTION received: * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff [16:06] sinzui, ok, let's keep on the radar though. that's the most frequent oops with have [16:06] right Ursinha ? [16:06] right [16:06] ok, thanks [16:06] let's move on [16:06] [TOPIC] * Oops report & Critical Bugs & Broken scripts [16:06] New Topic: * Oops report & Critical Bugs & Broken scripts [16:06] that's me [16:06] Ursinha, stage is yours [16:06] bugs 4-08738 and 4-03606 already mentioned and discussed [16:06] I have bug 422560 for answers, but rockstar told me yesterday he's on it, right rockstar? could you update the bug, pretty please? [16:06] Launchpad bug 422560 in launchpad-answers ""Create a FAQ" link fails - Oops! error page" [High,Triaged] https://launchpad.net/bugs/422560 [16:06] gary_poster, spm reported this one yesterday: bug 422960, seems to be foundations, what do you say? [16:06] Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960 [16:07] Ursinha, yup, getting ready to submit for review. [16:07] ...looking... [16:07] rockstar, cool, can you add your precious comments to the bug report? :) [16:07] I asked some info from spm on that foundations bug Ursinha [16:08] matsubara, I see [16:08] Ursinha, yes ma'am [16:08] thanks rockstar :) [16:08] ah [16:08] rockstar, bug 401631 is the one about +branches timeout, it's committed on edge but I see lots of timeouts on lpnet daily [16:08] Ursinha, how about: 497 ValueError: Invalid boundary in multipart form: '' ? [16:08] maybe worth a cherrypick? [16:08] Launchpad bug 401631 in launchpad-code "Repeated timeouts when viewing profile page on edge" [High,Fix committed] https://launchpad.net/bugs/401631 [16:08] Ursinha, I'll say yes at least on a first take. We'll at least do some investigation. [16:08] Ursinha, I'll chat with thumper about it. [16:09] and either take it or figure out who else should take it :-) [16:09] so it's ours now, anyway [16:09] thanks gary_poster [16:09] Ursinha, I'm pretty sure the timeouts on person +branches was CP already [16:10] matsubara, I still see timeouts [16:10] Ursinha, it was CP on the 1s. did the CP at least alleviated the problem? [16:11] Ursinha, and how about the 497 ValueErrors on edge? [16:11] matsubara, wait man :) [16:12] it seems the problem is better now [16:12] I am [16:12] I'll keep watching the next days [16:12] about the value error [16:12] I was going to ask rockstar if that's a code problem [16:12] rockstar, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1341EA197 [16:12] https://lp-oops.canonical.com/oops.py/?oopsid=1341EA197 [16:12] [action] ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed [16:12] ACTION received: ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed [16:13] Ursinha, huh, that's a weird and informationless oops. [16:13] rockstar, agreed about the informationless [16:13] [action] ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it [16:13] https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107 [16:13] ACTION received: ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it [16:13] https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107 [16:14] my only hint was the path [16:14] I'll file a bug [16:14] thanks Ursinha and rockstar [16:14] ok [16:14] we have one critical bug, already fix committed [16:14] about the failing script, rosetta-poimport, henninge, do you know what's causing that to fail? [16:15] Ursinha, isn't that the one that danilo replied to? [16:15] Ursinha: yes, it's got to do with sending emails. [16:15] s/danilo/jtv and danilo/ [16:15] matsubara, yes, but I don't seem to see a conclusion there [16:15] Ursinha: jtv found the bug and the fix is on the way in [16:15] henninge, awesome :) thanks! [16:16] cool [16:16] Ursinha: we are going to CP it [16:16] henninge, great, will jtv ask for a CP? [16:16] yes [16:16] I believe so, at least. [16:16] henninge, that's good [16:16] [action] matsubara to chase jtv about CP to fix rosetta-poimport script failure [16:16] ACTION received: matsubara to chase jtv about CP to fix rosetta-poimport script failure [16:16] thakn you :) [16:16] *thank [16:17] thanks all [16:17] matsubara, go ahead [16:17] hmm actually I think we should swap some action items Ursinha :-) [16:17] but I'll talk about it after the meeting [16:17] thanks Ursinha and everyone [16:17] [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) [16:17] New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) [16:18] Chex, ? [16:18] sorry, i think i might have been distracting him with a question. [16:19] sorry [16:19] here is our report for the week: [16:19] losas.. :) [16:20] Increasing number of 5XX errors in apache pretty much entirely from msn [16:20] bot on translations pages [16:20] Bug about missing OOPSes - bug 422960 [16:20] Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960 [16:20] Codebrowse is still needing restarting pretty regularly per the Incident Log [16:21] and thats all from us, unless there are any questions? [16:21] Chex, we (as in foundations team) will investigate the 500s. I don't think we should be logging OOPS for those as they don't seem to be hitting the app server. It looks like an issue with the load balancer or apache [16:21] Chex, I've looked into that. There is more to look at, but I think we have some hints on wtf is going on. [16:21] * mbarnett cheers [16:22] Chex, I'll follow up with you all on the 5XX later today. [16:22] great, thanks for the response/feedback [16:23] Chex, would be awesome if you could add a request that generated one of the 500 to the bug report [16:23] +1 [16:23] [action] chex to trawl logs and add request that caused 500 error to bug 422960 [16:23] Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960 [16:23] ACTION received: chex to trawl logs and add request that caused 500 error to bug 422960 [16:25] matsubara: ok, I will look into that for you [16:25] thanks Chex [16:25] matsubara: welcome [16:25] let's move on [16:25] [TOPIC] * DBA report (stub) [16:25] New Topic: * DBA report (stub) [16:25] so stuart is not around. I'll email him about the dba report [16:25] [action] matsubara to email stub about dba report [16:25] ACTION received: matsubara to email stub about dba report [16:25] [TOPIC] * Proposed items [16:25] New Topic: * Proposed items [16:26] no proposed items [16:26] so I think that's all for today [16:26] anything else before I close? [16:26] nothing here [16:27] ok, so be it [16:27] Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. [16:27] #endmeeting [16:27] Meeting finished at 10:27. [16:27] thanks matsubara and Ursinha! [16:27] thanks all! === matsubara is now known as matsubara-lunch === salgado is now known as salgado-lunch === gary_poster is now known as gary-lunch === cprov-lunch is now known as cprov === matsubara-lunch is now known as matsubara === salgado-lunch is now known as salgado === gary-lunch is now known as gary_poster === salgado is now known as salgado-afk === matsubara is now known as matsubara-afk === cprov is now known as cprov-afk === thumper is now known as thumper-afk