[14:38] <Ursinha> MootBot, hi
[15:01] <Ursinha> #startmeeting
[15:01] <MootBot> Meeting started at 10:01. The chair is Ursinha.
[15:01] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:01] <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:01] <Ursinha> [TOPIC] Roll Call
[15:01] <MootBot> New Topic:  Roll Call
[15:01] <Ursinha> apologies from matsubara, he's unwell today
[15:01] <allenap> me
[15:01] <sinzui> me
[15:01] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[15:01] <Ursinha> me!
[15:01] <Chex> salut
[15:02]  * Ursinha is happy, karmic koala is out :)
[15:02] <Ursinha> rockstar, hi
[15:02] <gary_poster> ,e
[15:02] <gary_poster> me
[15:02] <Ursinha> hi noodles775, are you replacing bigjools today?
[15:03] <Ursinha> stub, rockstar are missing
[15:03] <Ursinha> bigjools as well
[15:03] <Ursinha> lets move on, they should join us soon
[15:03] <Ursinha> (or not :P)
[15:03] <Ursinha> [TOPIC] Agenda
[15:03] <MootBot> New Topic:  Agenda
[15:03] <Ursinha>  * Actions from last meeting
[15:03] <Ursinha>  * Oops report & Critical Bugs & Broken scripts
[15:03] <Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
[15:03] <rockstar> ni
[15:03] <Ursinha>  * DBA report (stub)
[15:03] <danilos> me
[15:03] <Ursinha>  * Proposed items
[15:04] <Ursinha> danilos, go rest man, I can do that for us :)
[15:04] <danilos> Ursinha, ok, thanks!
[15:04] <Ursinha> [TOPIC] * Actions from last meeting
[15:04] <MootBot> New Topic:  * Actions from last meeting
[15:04] <Ursinha>  * gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
[15:04] <Ursinha>  * allenap and matsubara to talk about the timeouts on bug pages
[15:04] <Ursinha>  * matsubara to talk to stub about the DBA report when he gets back
[15:04] <Ursinha> allenap, did you? :)
[15:04] <Ursinha> gary_poster, ^
[15:05] <gary_poster> mine: I've been trying to land the pertinent branch for several days, encountering complications and my own mistakes.  I'm trying today too.
[15:05] <Ursinha> thanks gary_poster
[15:05] <allenap> Ursinha: A bit :) There was some discussion about the +index page on the list, but, iirc, it didn't come to any conclusions other than "yes, we'd like to fix it".
[15:05] <Ursinha> [action] gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
[15:06] <MootBot> ACTION received:  gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
[15:06] <Ursinha> allenap, ok, right
[15:07] <Ursinha> allenap, BugTask:+index is not the main offender, though it's in the top ten :P
[15:07] <Ursinha> allenap, I'll talk with matsubara to ask him if he took notes somewhere
[15:08] <Ursinha> [action] ask matsubara about his outstanding items from last meeting
[15:08] <MootBot> ACTION received:  ask matsubara about his outstanding items from last meeting
[15:08] <Ursinha> hi stub :)
[15:08] <allenap> Ursinha: Surely that's the top offender from bugs though?
[15:08] <Ursinha> allenap, I see DistroSeries:+index
[15:09] <allenap> Ursinha: That's Soyuz then :) Not going near that :)
[15:09] <Ursinha> allenap, hehe
[15:09] <Ursinha> it's already being fixed by registry, iirc
[15:09] <sinzui> We landed a fix for DistroSeries:+index but the change is not on edge yet
[15:09] <Ursinha> ha!
[15:09] <Ursinha> :)
[15:09] <Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[15:09] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[15:09] <allenap> Cool. Apologies to Soyuz for tarring them with the Registry brush.
[15:10] <Ursinha> allenap, about bug https://bugs.edge.launchpad.net/malone/+bug/453203, is there something we can do?
[15:10] <Ursinha> sinzui, is https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396EA55 related to https://bugs.edge.launchpad.net/launchpad-registry/+bug/462742? If not I'll be happy to file a bug
[15:10] <Ursinha> I've already asked henninge to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891
[15:10] <Ursinha> [action] Ursinha to ask translations to triage bug 462891
[15:10] <MootBot> ACTION received:  Ursinha to ask translations to triage bug 462891
[15:10] <sinzui> Ursinha: that fix was landed by intellectronica yesterday
[15:11] <Ursinha> sinzui, you told me that yesterday, but is that oops related?
[15:11] <sinzui> Ursinha: yes it is the bug subscription link
[15:12] <henninge> sinzui: actually, this is oopsing on trying to access preferredemails/email.
[15:12] <sinzui> Ursinha: sorry. I am wonrg
[15:12] <allenap> Ursinha: I opened a task against apport in Ubuntu. I can follow up with pitti, see if he has ideas. Also, if I have time I'll try to find a way to get the apport data when we next see this error.
[15:12] <Ursinha> oops
[15:12]  * Ursinha reverts the change in the bug
[15:13] <Ursinha> allenap, that would be much appreciated, thanks
[15:13] <Ursinha> [action] allenap to follow up with pitti about bug 453203
[15:13] <MootBot> ACTION received:  allenap to follow up with pitti about bug 453203
[15:14] <Ursinha> sinzui, want me to file a bug?
[15:15] <sinzui> I will
[15:15] <Ursinha> okay then
[15:15] <Ursinha> thanks sinzui
[15:15] <Ursinha> and allenap and henninge
[15:16] <Ursinha> henninge, you can triage bug 462891 if you have a moment :)
[15:16] <Ursinha> well
[15:16] <Ursinha> all three critical bugs are already fix committed, good
[15:16] <Ursinha> the jscheck failure is being handled by BjornT, good
[15:16] <henninge> Ursinha: I am on it.
[15:16] <Ursinha> henninge, thank you  :) ice_cream++
[15:17] <Ursinha> moving on
[15:17] <Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[15:17] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[15:17] <Ursinha> hi Chex
[15:17] <Ursinha> :)
[15:17] <Chex> hello
[15:17] <Chex> here is our report for this week
[15:17] <henninge> Ursinha: yeah!
[15:17] <Chex> - LP Ship-it progress in time for Ubuntu release-day:
[15:17] <Chex>         ; New LP app-servers are running
[15:17] <Chex>         ; New LP front-end web servers are running
[15:17] <Chex>         ; New LP Auth DB server has been switched to and is running
[15:17] <Chex>         ; We have a spare LP server available in case of needed extra performance in high loads.
[15:17] <Chex> - LP Incidents of note:
[15:17] <Chex>         ; CP 8536 to lpnet,bzrsyncd
[15:17] <Chex>         ; cowboy config patch to switch auth db master to sweetcherry
[15:17] <Chex>         ; CP 8539 to bzrsyncd, crowberry
[15:18] <Chex> does anyone have any questions??
[15:18] <stub> hackberry going boom might be an incident of note.
[15:18] <Ursinha> Chex, nice to hear we're ready to the ubuntu frenesy :)
[15:19] <Chex> stub: oh ok, neglected to put that on the list
[15:19] <Ursinha> *frenzy
[15:20]  * stub frenzies
[15:20] <Ursinha> lol
[15:20] <Ursinha> stub, you can comment on that in your section, that is now :)
[15:20] <Ursinha> any questions for Chex?
[15:20] <sinzui> Ursinha: that is an interesting oops, the create_milestone error only affects an anonymous user on edge.
[15:21] <Ursinha> sinzui, why is it interesting?
[15:21] <sinzui> How often is someone anonymous on edge?
[15:22] <Ursinha> sinzui, sure!
[15:22] <Ursinha> indeed
[15:22] <Ursinha> following a link, maybe?
[15:22] <Ursinha> this is indeed weird
[15:23] <Ursinha> well, lets move on just for the sake of the meeting
[15:23] <Ursinha> [TOPIC] * DBA report (stub)
[15:23] <MootBot> New Topic:  * DBA report (stub)
[15:23] <stub> We lost one of the slave databases earlier this week (~05:20 UTC Tuesday). The server load went through the roof. PostgreSQL and shells where still responding despite the load peaking at 94. Processes were not responding to signals.
[15:23] <stub> It turned out one of the drive arrays had hung, so writes were blocking. I'm not sure if this had end user visible results, as the queries made to the slave a read only and likely would have been successfully served from RAM.
[15:23] <stub> The slave was removed from replication as it could not be immediately investigated by IS (not important enough to wake anyone). It was added back into replication the next day.
[15:23] <stub> Staging database updates are currently disabled, as fallout from
[15:23] <stub> moving the authdb replication set master to a new database. A fix for Bug #461800 is up for review which is part of this. I still need to work more on the restoration scripts after this lands.
[15:23] <stub> The new database dedicated to the login service (launchpad_prod_4, lpmain slave and authdb master) is running fine. With this and the new appservers we should hopefully weather release day.
[15:23] <stub> oot.
[15:24] <Ursinha> stub, fixing 461800 will make possible to reenable database updates on staging?
[15:25] <stub> Its part of it, not the whole picture
[15:25] <stub> That allows us to build the slave. I still need to build the original db from a pair of dumps instead of a single dump.
[15:25] <Ursinha> I wonder how and if disabled database updates can affect QA
[15:26] <stub> Generally not adversely.
[15:26] <Ursinha> right
[15:27] <Ursinha> stub, do you have an estimated time to finish that?
[15:28] <stub> I want to have them working again by Monday. It depends if anything explodes from the release I guess.
[15:28] <Ursinha> stub, right, good
[15:28] <Ursinha> thanks
[15:28] <Ursinha> does anyone have something for stub?
[15:28] <Ursinha> all right
[15:29] <Ursinha> Chex, a question, do you know when will PQM close?
[15:29] <Ursinha> perhaps the release manage knows it better, noodles775, hi :)
[15:29] <Ursinha> *manager
[15:30] <noodles775> Ursinha, Chex: according to the docs, 00:00 UTC on Monday.
[15:30] <Ursinha> noodles775, on Monday, right
[15:30] <Ursinha> thanks :)
[15:30] <Chex> Ursinha: ok, good to know then
[15:31] <Ursinha> [TOPIC] * Proposed items
[15:31] <MootBot> New Topic:  * Proposed items
[15:31] <Ursinha> I have something
[15:31] <Ursinha> more of a reminder :)
[15:31] <Ursinha> don't forget about QAing your stuff, it's now or never for next release! :) Needing help, ping the QA-person-in-charge and we'll be glad to help
[15:31] <Ursinha> does anyone else have something?
[15:31] <noodles775> Quite a few teams with > 15 items waiting for QA:
[15:31] <noodles775> http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
[15:31] <MootBot> LINK received:  http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
[15:33] <Ursinha> noodles775, translations doesn't count, we're trying to use lp as QA tool, tagging bugs as qa-needstesting, qa-ok and so on
[15:33] <Ursinha> I'll send one email to the list explaining that today
[15:33] <noodles775> Ursinha: great, thanks.
[15:33] <Ursinha> [action] Ursinha to send one email to lp list explaining the qa-tags experiment
[15:33] <MootBot> ACTION received:  Ursinha to send one email to lp list explaining the qa-tags experiment
[15:33] <Ursinha> thanks for the reminder noodles775 :)
[15:33] <Ursinha> right, anyone else?
[15:34] <Ursinha> good!
[15:34] <gary_poster> :-)
[15:34] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[15:34] <Ursinha> #endmeeting
[15:34] <MootBot> Meeting finished at 10:34.
[15:34] <Ursinha> thanks everyone
[15:34] <gary_poster> thanks Ursinha
[15:34] <Ursinha> you all rock
[15:34] <Ursinha> gary_poster, :)