=== mrevell is now known as mrevell-lunch
=== mrevell-lunch is now known as mrevell
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:00] <Ursinha> me!
[16:00] <allenap> me
[16:00] <henninge> me
[16:01] <al-maisan> me
[16:01] <matsubara> Chex, rockstar: hi
[16:02] <matsubara> bac, can you stand in for sinzui?
[16:02] <Chex> matsubara: hello
[16:02] <bac> matsubara:  gladly
[16:02] <bac> me
[16:02] <Ursinha> :)
[16:02] <matsubara> thanks bac
[16:02] <jtv> me
[16:02] <Ursinha> oh, a special guest :P
[16:02] <matsubara> I'm standing in for gary
[16:02] <matsubara> hi jtv
[16:02] <jtv> hi
[16:02] <matsubara> we have 3/4 of the translations team around
[16:02] <matsubara> pretty cool
[16:03] <jtv> with all the oopses going on, we thought we'd look in just in case
[16:03]  * Ursinha hi-fives henninge and jtv 
[16:03]  * jtv hi-fives back
[16:03] <matsubara> ok, let's get this started
[16:03] <Ursinha> yay
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:03]  * henninge ducks
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:03] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:03] <matsubara>  * DBA report (stub)
[16:03] <matsubara>  * Proposed items
[16:03] <matsubara> [TOPIC] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:04] <matsubara> * add brian to the list of attendees in the MeetingAgenda page
[16:04] <matsubara>     * done
[16:04] <matsubara> * allenap to dig the master bug of OOPS-1474EA771
[16:04] <matsubara>     * done. it's #508302
[16:04] <matsubara> * rockstar to take a look in OOPS-1480CMP1
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
[16:04] <matsubara> * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
[16:04] <matsubara> * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
[16:04] <matsubara> * matsubara to talk to salgado about More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
[16:04] <matsubara> * matsubara to talk to stub or gary about InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
[16:04] <matsubara>     * https://launchpad.net/bugs/514267
[16:04] <Ursinha> oh dear
[16:04] <matsubara> * matsubara to fix bug link on OOPS-1474EA771 to point to bug 508302
[16:04] <ubottu> Ubuntu bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged]
[16:04] <matsubara>     * done
[16:04] <matsubara> * sinzui to investigate failure on send-person-notifications and reply to the list with his findings
[16:04] <ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,Fix committed] https://launchpad.net/bugs/508302
[16:04] <matsubara>     * https://bugs.edge.launchpad.net/launchpad-registry/+bug/514149
[16:04] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
[16:04] <matsubara> * rockstar to raise the importance of ensuring sufficient monitoring of this part (i.e. branch updates emails failing to be delivered) of code-hosting by thumper
[16:04] <ubottu> Ubuntu bug 514149 in launchpad-registry "send-person-notifications.py is broken" [High,Fix released]
[16:04] <matsubara> * Gary talk about "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
[16:04] <matsubara> * matsubara to update oops-tools to know about the new lp appserver
[16:04] <matsubara>     * done.
[16:04] <matsubara> hmm ok, rockstar is not around
[16:04] <matsubara> I haven't talked to leonard nor emailed team about those oopses
[16:05] <matsubara> [action] * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
[16:05] <MootBot> ACTION received:  * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
[16:05] <matsubara> [action] * rockstar to take a look in OOPS-1480CMP1
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
[16:05] <MootBot> ACTION received:  * rockstar to take a look in OOPS-1480CMP1
[16:05] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
[16:06] <matsubara> I guess that's it
[16:06] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:06] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:06] <Ursinha> okay, so
[16:06] <Ursinha> the alarming bug is bug 514267
[16:06] <ubottu> Launchpad bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged] https://launchpad.net/bugs/514267
[16:07] <Ursinha> apparently it's the reason of about 6k InternalError oopses only today in lpnet
[16:08] <matsubara> stub is on it, discussion ongoing in the internal channel
[16:08] <Ursinha> matsubara: okay, notes in that bug are welcome :)
[16:08] <Ursinha> matsubara: thanks
[16:08] <matsubara> feel free to add them Ursinha
[16:08] <Ursinha> matsubara: I don't know what's being discussed :)
[16:09] <Ursinha> anyway, I guess this is most critical thing
[16:10] <Ursinha> we have six critical bugs, two in progress
[16:10] <matsubara> no other oopses worth mentioning?
[16:10] <Ursinha> matsubara: not that already have bugs/are being worked on, afaik
[16:10] <matsubara> ok
[16:11] <Ursinha> matsubara: bug 511567 is triaged and marked to 10.01
[16:11] <ubottu> Launchpad bug 511567 in launchpad-foundations "Can't remove authorised oauth tokens" [Critical,Triaged] https://launchpad.net/bugs/511567
[16:11] <Ursinha> is that really critical?
[16:11] <matsubara> 10.01 should be closed by now, so I guess it should be 10.02
[16:12] <Ursinha> matsubara: right, but is that really critical?
[16:12] <matsubara> don't know I asked in the bug report to Curtis who set it as critical
[16:12] <Ursinha> bug 514461 is code, in progress, have to check with rockstar how's that
[16:12] <ubottu> Launchpad bug 514461 in launchpad-code "scan_branches.py screams and no one is listening" [Critical,In progress] https://launchpad.net/bugs/514461
[16:13] <Ursinha> [action] Ursinha to talk to rockstar about bug 514461
[16:13] <Ursinha> [action] Ursinha to talk to mwhudson about bug 513412
[16:13] <ubottu> Bug 513412 on http://launchpad.net/bugs/513412 is private
[16:14] <Ursinha> and, finally, bug 515473 is assigned to curtis and in progress
[16:14] <ubottu> Launchpad bug 515473 in soyuz "ppa description contains smal links" [Critical,In progress] https://launchpad.net/bugs/515473
[16:14] <Ursinha> bac: do you know the progress on that one?
[16:15] <bac> i do not
[16:15] <Ursinha> bac: can you ask him, if possible, please? :)
[16:15] <bac> looking into it
[16:15] <Ursinha> bac: thanks
[16:16] <Ursinha> bug 508025 is only waiting for a test commit so it can be closed
[16:16] <ubottu> Launchpad bug 508025 in lpbuildbot "Need production-devel builder" [Critical,In progress] https://launchpad.net/bugs/508025
[16:17] <Ursinha> end of critical bugs
[16:17] <Ursinha> now for something completely different
[16:17] <Ursinha> script process-pending-packagediffs has been failing lately, is that soyuz? al-maisan?
[16:18] <al-maisan> al-maisan: I have a branch that fixes these failure
=== henninge_ is now known as henninge
[16:18] <al-maisan> it has been reviewed and I am revising it in accordance with the review comments
[16:19] <Ursinha> that's great al-maisan, I'll reply in the failure mail
[16:19] <Ursinha> al-maisan: thanks for the info
[16:19] <Ursinha> matsubara: you can move on
[16:19] <al-maisan> you are welcome
[16:19] <matsubara> thanks Ursinha
[16:19] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:19] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:20] <rockstar> me
[16:20] <Ursinha> rockstar: good morning :)
[16:20] <Chex> hi everyone
[16:20] <rockstar> Ursinha, :)
[16:20] <Chex> here is the LOSA report for this week:
[16:20] <Chex> - LP Outage Today @1400UTC: LP is back running again after a outage; there was an issue with the new
[16:21] <Chex>         DB replication lag check not working properly. We are working to see why this happened and
[16:21] <Chex>         resolve.
[16:21] <Chex> - LP incidents of note:
[16:21] <Chex>         ; LP Cherry-picks: rolled out 8940 to Lpnet on 29-Jan
[16:21] <Chex> Does anyone have any comments or questions?
[16:22] <matsubara> thanks Chex
[16:22] <matsubara> [TOPIC] * DBA report (stub)
[16:22] <MootBot> New Topic:  * DBA report (stub)
[16:22] <matsubara> I'll ask stub to send the report to the list
[16:23] <matsubara> [action] matsubara to ask stub to send the dba report to the list
[16:23] <MootBot> ACTION received:  matsubara to ask stub to send the dba report to the list
[16:23] <matsubara> [TOPIC] * Proposed items
[16:23] <MootBot> New Topic:  * Proposed items
[16:23] <matsubara> no proposed items
[16:23] <matsubara> ok, since we have some time
[16:23] <matsubara> * rockstar to take a look in OOPS-1480CMP1
[16:23] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
[16:24] <matsubara> rockstar, that one is in the actions from last meeting topic for quite some week now
[16:24] <matsubara> did you have time to look into that?
[16:26] <rockstar> matsubara, I'll take a look.
[16:28] <Ursinha> thanks rockstar
[16:28] <matsubara> rockstar, I thought you were looking and would report back here...
[16:28] <matsubara> shall I close the meeting then?
[16:29] <rockstar> matsubara, that's the first I've seen that oops.  I'm not sure how I missed it from other meetings.
[16:29] <matsubara> by missing the meeting? :-)
[16:30] <matsubara> rockstar, anyway, thanks for looking into that. if it helps I'll file a place holder bug and you can fill in the details once you know more about it
[16:30] <matsubara> thanks everyone
[16:30] <rockstar> matsubara, sure.
[16:30] <Ursinha> thanks matsubara
[16:30] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[16:30] <matsubara> #endmeeting
[16:30] <MootBot> Meeting finished at 10:30.
[16:32] <bac> Ursinha:  sinzui reports bug 515473 has been reviewed and will be on ec2 for landing shortly
[16:32] <ubottu> Launchpad bug 515473 in soyuz "ppa description contains spam links" [Critical,In progress] https://launchpad.net/bugs/515473
[16:32] <Ursinha> thanks a lot bac
[16:34] <Ursinha> bac: do you know which branch fixes that bug? it's not linked
[16:34]  * bac looks
[16:35] <matsubara> rockstar, filed https://bugs.edge.launchpad.net/launchpad-code/+bug/517126
[16:35] <ubottu> Ubuntu bug 517126 in launchpad-code "BzrCheckError raised creating a merge proposal" [Undecided,New]
[16:35] <bac> i have linked it to the bug now, ursinha
[16:35] <Ursinha> thaaanks bac :)
[16:43] <Ursinha> rockstar: are you working on bug 514461?
[16:43] <ubottu> Launchpad bug 514461 in launchpad-code "scan_branches.py screams and no one is listening" [Critical,In progress] https://launchpad.net/bugs/514461
[16:44] <rockstar> Ursinha, yes, I have been trying to get it landed for a week.  If it got into production-stable, then I'm going to have it CP'd today.
[16:45] <Ursinha> rockstar: cool! can you tell the world about it in the bug by adding a comment? :)
[16:45] <rockstar> Ursinha, sure.
[16:45] <Ursinha> rockstar: thanks
=== matsubara is now known as matsubara-afk
=== salgado is now known as salgado-afk