=== Peng_ is now known as Peng__ === Peng is now known as Peng_ === Peng__ is now known as Peng === noodles775_ is now known as noodles775 === _thumper_ is now known as thumper === matsubara-afk is now known as matsubara === matsubara is now known as matsubara-afk === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [14:38] MootBot, hi [15:01] #startmeeting [15:01] Meeting started at 10:01. The chair is Ursinha. [15:01] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [15:01] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [15:01] [TOPIC] Roll Call [15:01] New Topic: Roll Call [15:01] apologies from matsubara, he's unwell today [15:01] me [15:01] me [15:01] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [15:01] me! [15:01] salut [15:02] * Ursinha is happy, karmic koala is out :) [15:02] rockstar, hi [15:02] ,e [15:02] me [15:02] hi noodles775, are you replacing bigjools today? [15:03] stub, rockstar are missing [15:03] bigjools as well [15:03] lets move on, they should join us soon [15:03] (or not :P) [15:03] [TOPIC] Agenda [15:03] New Topic: Agenda [15:03] * Actions from last meeting [15:03] * Oops report & Critical Bugs & Broken scripts [15:03] * Operations report (mthaddon/Chex/spm/mbarnett) [15:03] ni [15:03] * DBA report (stub) [15:03] me [15:03] * Proposed items [15:04] danilos, go rest man, I can do that for us :) [15:04] Ursinha, ok, thanks! [15:04] [TOPIC] * Actions from last meeting [15:04] New Topic: * Actions from last meeting [15:04] * gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990 [15:04] Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 [15:04] * allenap and matsubara to talk about the timeouts on bug pages [15:04] * matsubara to talk to stub about the DBA report when he gets back [15:04] allenap, did you? :) [15:04] gary_poster, ^ [15:05] mine: I've been trying to land the pertinent branch for several days, encountering complications and my own mistakes. I'm trying today too. [15:05] thanks gary_poster [15:05] Ursinha: A bit :) There was some discussion about the +index page on the list, but, iirc, it didn't come to any conclusions other than "yes, we'd like to fix it". [15:05] [action] gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990 [15:06] ACTION received: gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990 [15:06] Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 [15:06] allenap, ok, right [15:07] allenap, BugTask:+index is not the main offender, though it's in the top ten :P [15:07] allenap, I'll talk with matsubara to ask him if he took notes somewhere [15:08] [action] ask matsubara about his outstanding items from last meeting [15:08] ACTION received: ask matsubara about his outstanding items from last meeting [15:08] hi stub :) [15:08] Ursinha: Surely that's the top offender from bugs though? [15:08] allenap, I see DistroSeries:+index [15:09] Ursinha: That's Soyuz then :) Not going near that :) [15:09] allenap, hehe [15:09] it's already being fixed by registry, iirc [15:09] We landed a fix for DistroSeries:+index but the change is not on edge yet [15:09] ha! [15:09] :) [15:09] [TOPIC] * Oops report & Critical Bugs & Broken scripts [15:09] New Topic: * Oops report & Critical Bugs & Broken scripts [15:09] Cool. Apologies to Soyuz for tarring them with the Registry brush. [15:10] allenap, about bug https://bugs.edge.launchpad.net/malone/+bug/453203, is there something we can do? [15:10] sinzui, is https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396EA55 related to https://bugs.edge.launchpad.net/launchpad-registry/+bug/462742? If not I'll be happy to file a bug [15:10] Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New] [15:10] https://lp-oops.canonical.com/oops.py/?oopsid=1396EA55 [15:10] I've already asked henninge to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891 [15:10] Launchpad bug 462742 in launchpad-registry "OOPS getting a project index page as an anonymous user" [High,In progress] [15:10] Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] [15:10] [action] Ursinha to ask translations to triage bug 462891 [15:10] ACTION received: Ursinha to ask translations to triage bug 462891 [15:10] Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891 [15:10] Ursinha: that fix was landed by intellectronica yesterday [15:11] sinzui, you told me that yesterday, but is that oops related? [15:11] Ursinha: yes it is the bug subscription link [15:12] sinzui: actually, this is oopsing on trying to access preferredemails/email. [15:12] Ursinha: sorry. I am wonrg [15:12] Ursinha: I opened a task against apport in Ubuntu. I can follow up with pitti, see if he has ideas. Also, if I have time I'll try to find a way to get the apport data when we next see this error. [15:12] oops [15:12] * Ursinha reverts the change in the bug === noodles775_ is now known as noodles775 [15:13] allenap, that would be much appreciated, thanks [15:13] [action] allenap to follow up with pitti about bug 453203 [15:13] ACTION received: allenap to follow up with pitti about bug 453203 [15:13] Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New] https://launchpad.net/bugs/453203 [15:14] sinzui, want me to file a bug? [15:15] I will [15:15] okay then [15:15] thanks sinzui [15:15] and allenap and henninge [15:16] henninge, you can triage bug 462891 if you have a moment :) [15:16] Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891 [15:16] well [15:16] all three critical bugs are already fix committed, good [15:16] the jscheck failure is being handled by BjornT, good [15:16] Ursinha: I am on it. [15:16] henninge, thank you :) ice_cream++ [15:17] moving on [15:17] [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) [15:17] New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) [15:17] hi Chex [15:17] :) [15:17] hello [15:17] here is our report for this week [15:17] Ursinha: yeah! [15:17] - LP Ship-it progress in time for Ubuntu release-day: [15:17] ; New LP app-servers are running [15:17] ; New LP front-end web servers are running [15:17] ; New LP Auth DB server has been switched to and is running [15:17] ; We have a spare LP server available in case of needed extra performance in high loads. [15:17] - LP Incidents of note: [15:17] ; CP 8536 to lpnet,bzrsyncd [15:17] ; cowboy config patch to switch auth db master to sweetcherry [15:17] ; CP 8539 to bzrsyncd, crowberry [15:18] does anyone have any questions?? [15:18] hackberry going boom might be an incident of note. [15:18] Chex, nice to hear we're ready to the ubuntu frenesy :) [15:19] stub: oh ok, neglected to put that on the list [15:19] *frenzy [15:20] * stub frenzies [15:20] lol [15:20] stub, you can comment on that in your section, that is now :) [15:20] any questions for Chex? [15:20] Ursinha: that is an interesting oops, the create_milestone error only affects an anonymous user on edge. [15:21] sinzui, why is it interesting? [15:21] How often is someone anonymous on edge? [15:22] sinzui, sure! [15:22] indeed [15:22] following a link, maybe? [15:22] this is indeed weird [15:23] well, lets move on just for the sake of the meeting [15:23] [TOPIC] * DBA report (stub) [15:23] New Topic: * DBA report (stub) [15:23] We lost one of the slave databases earlier this week (~05:20 UTC Tuesday). The server load went through the roof. PostgreSQL and shells where still responding despite the load peaking at 94. Processes were not responding to signals. [15:23] It turned out one of the drive arrays had hung, so writes were blocking. I'm not sure if this had end user visible results, as the queries made to the slave a read only and likely would have been successfully served from RAM. [15:23] The slave was removed from replication as it could not be immediately investigated by IS (not important enough to wake anyone). It was added back into replication the next day. [15:23] Staging database updates are currently disabled, as fallout from [15:23] moving the authdb replication set master to a new database. A fix for Bug #461800 is up for review which is part of this. I still need to work more on the restoration scripts after this lands. [15:23] Launchpad bug 461800 in launchpad-foundations "new-slave.py no longer works" [High,In progress] https://launchpad.net/bugs/461800 [15:23] The new database dedicated to the login service (launchpad_prod_4, lpmain slave and authdb master) is running fine. With this and the new appservers we should hopefully weather release day. [15:23] oot. [15:24] stub, fixing 461800 will make possible to reenable database updates on staging? [15:25] Its part of it, not the whole picture [15:25] That allows us to build the slave. I still need to build the original db from a pair of dumps instead of a single dump. [15:25] I wonder how and if disabled database updates can affect QA [15:26] Generally not adversely. [15:26] right [15:27] stub, do you have an estimated time to finish that? [15:28] I want to have them working again by Monday. It depends if anything explodes from the release I guess. [15:28] stub, right, good [15:28] thanks [15:28] does anyone have something for stub? [15:28] all right [15:29] Chex, a question, do you know when will PQM close? [15:29] perhaps the release manage knows it better, noodles775, hi :) [15:29] *manager [15:30] Ursinha, Chex: according to the docs, 00:00 UTC on Monday. [15:30] noodles775, on Monday, right [15:30] thanks :) [15:30] Ursinha: ok, good to know then [15:31] [TOPIC] * Proposed items [15:31] New Topic: * Proposed items [15:31] I have something [15:31] more of a reminder :) [15:31] don't forget about QAing your stuff, it's now or never for next release! :) Needing help, ping the QA-person-in-charge and we'll be glad to help [15:31] does anyone else have something? [15:31] Quite a few teams with > 15 items waiting for QA: [15:31] http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html [15:31] LINK received: http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html === salgado is now known as salgado-afk [15:33] noodles775, translations doesn't count, we're trying to use lp as QA tool, tagging bugs as qa-needstesting, qa-ok and so on [15:33] I'll send one email to the list explaining that today [15:33] Ursinha: great, thanks. [15:33] [action] Ursinha to send one email to lp list explaining the qa-tags experiment [15:33] ACTION received: Ursinha to send one email to lp list explaining the qa-tags experiment [15:33] thanks for the reminder noodles775 :) [15:33] right, anyone else? [15:34] good! [15:34] :-) [15:34] Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. [15:34] #endmeeting [15:34] Meeting finished at 10:34. [15:34] thanks everyone [15:34] thanks Ursinha [15:34] you all rock [15:34] gary_poster, :) === salgado-afk is now known as salgado === rockstar is now known as rockstar-afk === salgado is now known as salgado-afk === rockstar-afk is now known as rockstar