=== noodles785 is now known as noodles775 === Ursinha-lunch is now known as Ursinha === mrevell is now known as mrevell-lunch === salgado is now known as salgado-afk === mrevell-lunch is now known as mrevell === Chex_ is now known as Chex [15:59] * Ursinha pokes MootBot [16:00] #startmeeting [16:00] Meeting started at 10:00. The chair is matsubara. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] [TOPIC] Roll Call [16:00] New Topic: Roll Call [16:00] me [16:00] me [16:00] me me [16:00] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [16:01] sinzui, Chex, rockstar, bigjools, allenap, danilos: hi [16:01] \o [16:01] me [16:02] hi [16:02] me [16:02] hello [16:03] me [16:03] ok, everyone is here. [16:03] [TOPIC] Agenda [16:03] New Topic: Agenda [16:03] * Actions from last meeting [16:03] * Oops report & Critical Bugs & Broken scripts [16:03] * Operations report (mthaddon/Chex/spm/mbarnett) [16:03] * DBA report (stub) [16:03] * Proposed items [16:03] [TOPIC] * Actions from last meeting [16:03] New Topic: * Actions from last meeting [16:03] * matsubara to discuss with foundations bug 403281 and set importance on it [16:03] * discussed with Gary and it's now triaged and an importance set. [16:03] * rockstar to triage bug 531687 [16:03] * matsubara to discuss bug 531965 with foundations as well [16:03] Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Low,Fix committed] https://launchpad.net/bugs/403281 [16:03] * discussed with Gary and it's now triaged and an importance set. [16:03] Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [High,Triaged] https://launchpad.net/bugs/531687 [16:03] * rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts [16:03] Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Low,Invalid] https://launchpad.net/bugs/531965 [16:03] * matsubara, abentley is already figuring out what's going on with those scripts. [16:03] * rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it [16:03] * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section [16:03] I didn't add the count of untriaged bugs per team to the oops section yet [16:04] [action] * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section [16:04] ACTION received: * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section [16:04] yes sir [16:04] rockstar, any news about the OOM error? did you get back to losas about it? [16:05] I see that 531687 was properly triaged and an importance set. thanks for that rockstar [16:05] matsubara, others on my team are looking into it. We think it's probably just another scalability issue. [16:05] cool. thanks rockstar [16:06] [TOPIC] * Oops report & Critical Bugs & Broken scripts [16:06] New Topic: * Oops report & Critical Bugs & Broken scripts [16:06] I have 3 bugs for today, 2 of them were already dealt with [16:06] bug 535071 still need some foundations love [16:06] Launchpad bug 535071 in launchpad-foundations "Better error handling when librarian is offline" [Undecided,New] https://launchpad.net/bugs/535071 [16:06] gary_poster, can you take a look ^? [16:07] fwiw, the other two are bugs 535078, 535672 [16:07] Launchpad bug 535078 in launchpad-foundations "InvalidURIError raised accessing an API url" [High,Triaged] https://launchpad.net/bugs/535078 [16:07] Launchpad bug 535672 in malone "TransactionRollbackError using the api to find users affected with dupes" [Undecided,Won't fix] https://launchpad.net/bugs/535672 [16:07] looking (how did I miss this? :-/ ) [16:08] thanks gary_poster [16:08] we have 4 critical bugs, 3 fix committed and 1 in progress. [16:09] /o\ [16:09] for the failing scripts [16:09] we have two failing frequently: productreleasefinder and oops-pruner [16:09] https://lp-oops.canonical.com/oops.py/?oopsid=pruner [16:09] heh [16:10] ubottu: there, there [16:10] I replied to one of the oops pruner failure email but didn't get a reply yet. [16:10] Chex, could you take a look in the oops pruner script logs and check why it's failing? [16:10] matsubara: yes sure, I can look at that [16:10] Chex, please reply to one of the failures emails with the traceback or file a bug [16:11] [action] Chex to verify why oops-pruner is failing and reply to the failure email with more information. [16:11] https://lp-oops.canonical.com/oops.py/?oopsid=pruner [16:11] ACTION received: Chex to verify why oops-pruner is failing and reply to the failure email with more information. [16:11] https://lp-oops.canonical.com/oops.py/?oopsid=pruner [16:11] sinzui, how about the prf script? [16:11] been failing on a daily basis for the past week [16:11] It is always running 2 hours late [16:11] It always runs, just late [16:12] I think we need to consider changing the threshold, or changing the order [16:12] sinzui, so we should adjust the monitoring script to take that into account, shouldn't we? [16:12] [action] sinzui to work with losas to set the increase threshold for the product release finder monitoring script. [16:12] ACTION received: sinzui to work with losas to set the increase threshold for the product release finder monitoring script. [16:13] I think that's all for today's oops/scripts/critical section. thanks everyone [16:13] [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) [16:13] New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) [16:14] hello everyone, here is this weeks report: [16:14] - We are in the process of working on migrating to using SSO run by ISD branded for LP (open-ID single signon) in [16:14] Launchpad. [16:14] - The LP DBs on one of the slave DB servers may need to be rebuilt/repacked as we're getting short on [16:14] space on that server. [16:14] - LP incidents of note: [16:14] ; LP Cherry-picks: CP to crowberry/codehost; revno 9082 [16:14] ; There was a buildbot error that stopped merging from db-devel to db-stable for a few days, which has [16:14] been fixed. [16:15] and thats all for us. Any questions/comments?? [16:16] Chex, I take you're working with stub to have the db rebuilding done? [16:17] oh, and the buildbot merge error seems to be happening again, so this still needs to be investigated. [16:17] matsubara: yes I believe we will work with stub on the db rebuild. [16:17] rockstar, can you take that? that's a build engineer thing, right? [16:19] matsubara, I'm not build engineer anymore. [16:20] rockstar, oh, who's the build engineer for this cycle? [16:20] matsubara, I don't think there is one. [16:20] I guess it'll fallback to foundations then. gary_poster can you work with losas re: the buildbot issue above? [16:21] matsubara: yes [16:21] cool. thanks. [16:21] [TOPIC] * DBA report (stub) [16:21] New Topic: * DBA report (stub) [16:21] sorry, forgot to thank Chex for his report. thank you Chex! [16:22] dba report was sent to the list. any question please follow up there [16:22] [TOPIC] * Proposed items [16:22] New Topic: * Proposed items [16:22] no new proposed items [16:22] and I guess that's all for today folks. anything else before I close? [16:23] matsubara: no problem! [16:23] Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. [16:23] #endmeeting [16:23] Meeting finished at 10:23. [16:24] matsubara: thanks japanese [16:26] Ursinha: *chinese* [16:26] :) [16:26] bigjools: lol [16:27] last I hear, I was vietnamese. Must be the tan :-) [16:27] heard [16:27] vietnamese [16:27] the tan [16:27] rofl === gary_poster is now known as gary-lunch === Ursinha is now known as Ursinha-lunch === salgado-afk is now known as salgado === gary-lunch is now known as gary_poster === matsubara is now known as matsubara-afk === salgado is now known as salgado-afk