=== ctrlsoft is now known as jelmer
=== matsubara is now known as matsubara-lunch
=== salgado is now known as salgado-lunch
=== matsubara-lunch is now known as matsubara
[17:00] <matsubara> #startmeeting
[17:00] <MootBot> Meeting started at 11:00. The chair is matsubara.
[17:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[17:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[17:00] <matsubara> [TOPIC] Roll Call
[17:00] <MootBot> New Topic:  Roll Call
[17:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[17:00] <sinzui> me
[17:00] <matsubara> Chex, gary_poster, bigjools, allenap, rockstar, danilos: Hi
[17:01] <gary_poster> me
[17:01] <bigjools> me sort of
[17:01] <danilos> me
[17:01] <Chex> me
[17:01] <allenap> me
[17:01] <matsubara> [TOPIC] Agenda
[17:01] <MootBot> New Topic:  Agenda
[17:02] <matsubara>  * Actions from last meeting
[17:02] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[17:02] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[17:02] <matsubara>  * DBA report (stub)
[17:02] <matsubara>  * QA stats of the week
[17:02] <matsubara>  * Proposed items
[17:02] <matsubara> [TOPIC] * Actions from last meeting
[17:02] <MootBot> New Topic:  * Actions from last meeting
[17:02] <matsubara>  * gary_poster and sinzui to discuss about the issues between master and slave that could be causing oopses
[17:02] <matsubara>  * allenap to follow up with deryck about bug 230106
[17:02] <matsubara>  * gary_poster to ask stub to investigate bug 556245
[17:02] <matsubara>  * Ursinha to send an email to stub asking for the dba report
[17:02] <ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
[17:02] <ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
[17:02] <matsubara>    * Done
[17:02] <sinzui> matsubara, done
[17:02] <matsubara> thanks sinzui
[17:03] <gary_poster> matsubara: we have progress, in that meliae is installed thanks to salgado.  We're still at the diagnosis stage.
[17:03] <matsubara> thanks gary_poster
[17:03] <gary_poster> we need to get the change CP'd.
[17:03] <matsubara> hwo about 556245?
[17:03] <allenap> matsubara: I suck, I'm sorry I didn't follow that up except to comment on the bug.
[17:04] <matsubara> allenap, any news about 230106
[17:04] <matsubara> ?
[17:04] <allenap> matsubara: See suckage above ^.
[17:04] <matsubara> hmm ok. shall I keep the action item?
[17:04] <allenap> matsubara: Yes please.
[17:04] <matsubara> [action] * allenap to follow up with deryck about bug 230106
[17:04] <MootBot> ACTION received:  * allenap to follow up with deryck about bug 230106
[17:04] <ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
[17:05] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[17:05] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[17:06] <matsubara> I have two oops bugs: https://bugs.edge.launchpad.net/launchpad-registry/+bug/568459 which sinzui already duped (Thanks dude!)
[17:06] <ubottu> Ubuntu bug 568459 in launchpad-registry "LocationError raised browsing code of conduct page (dup-of: 555734)" [Undecided,New]
[17:06] <ubottu> Ubuntu bug 555734 in launchpad-registry "oops signing CoC if keys mutate" [High,Fix committed]
[17:06] <matsubara> gary_poster, and this one for foundations: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/568456
[17:06] <ubottu> Ubuntu bug 568456 in launchpad-foundations "GpgmeError raised importing public gpg key" [Undecided,New]
[17:06] <gary_poster> ack, matsubara thanks
[17:07] <matsubara> thanks gary_poster
[17:07] <sinzui> Is this fixed in our tree now
[17:07]  * sinzui does not see how lucid issue could be affecting hardy production
[17:08] <sinzui> but gpgme is a lucid/lp issue that I believe was fixed by salgado
[17:08] <matsubara> we had a couple of script failures, which spm already took care of
[17:08] <matsubara> we have no critical bugs open
[17:09] <matsubara> [action] gary_poster to triage bug 568456
[17:09] <MootBot> ACTION received:  gary_poster to triage bug 568456
[17:09] <ubottu> Launchpad bug 568456 in launchpad-foundations "GpgmeError raised importing public gpg key" [Undecided,New] https://launchpad.net/bugs/568456
[17:09] <matsubara> let's move on, thanks everyone
[17:09] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[17:09] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[17:09] <Chex> hi everyone
[17:10] <Chex> here is the LOSA report for this week, just reporting on incidents this week:
[17:10] <Chex> - LP incidents of note:
[17:10] <Chex>         : 18-Apr:  swap problems on lpnet servers.  restarts on: lpnets 1,2,11,12,13,14,15 services; Bug: 531071
[17:10] <ubottu> Launchpad bug 531071 in launchpad-foundations "lpnet app servers are leaking memory again" [High,Triaged] https://launchpad.net/bugs/531071
[17:10] <Chex>         : 20-Apr: Checkwatches had hung, killed. Bug: 567088
[17:10] <ubottu> Launchpad bug 567088 in malone "checkwatches keeps hanging at irregular intervals" [Undecided,New] https://launchpad.net/bugs/567088
[17:10] <Chex>         : 21-Apr: crowberry h/w error, restarted the machine. OK for now, following up on the hardware issues.
[17:10] <Chex>         : 21-Apr: mizuho apparently OOMK'd, rebooted; necessitating full LP restart; Bug 556245
[17:10] <ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
[17:10] <Chex> does anyone have any questions or comments on these?
[17:12] <gary_poster> Chex, are the LOSAs familiar with the steps necessary to use the meliae hooks?  I know salgado has been trying to communicate those via the right channels.
[17:12] <gary_poster> (this is pertinent to the memory leaks in librarian and LP)
[17:12] <matsubara> allenap, can you bring the checkwatches issue with your team?
[17:12] <matsubara> allenap, nm, I just noticed that you commented on the bug already, thanks! :-)
[17:13] <Chex> gary_poster: I was working with maliae breifly with the U1 team earlier this week, but I am unaware of the process for LP..
[17:13] <gary_poster> Chex, could you get back to me or salgado with the best way to communicate these changes to the LOSAs?  I know he has updated the wiki pages with instructions
[17:14] <gary_poster> (the LOSA wiki pages)
[17:14] <Chex> gary_poster: sorry, I see it now, thru his email to the LOSA list, I will look at that and make sure the rest of the team is aware of that, also.
[17:14] <gary_poster> Chex, thank you very much
[17:14] <matsubara> [action] Chex to work with gary_poster or salgado to sort out how to use the  meliae hooks to debug librarian and LP memory leaks
[17:14] <MootBot> ACTION received:  Chex to work with gary_poster or salgado to sort out how to use the  meliae hooks to debug librarian and LP memory leaks
[17:15] <Chex> gary_poster: that is the correct way to communicate those, but hilighting these issues to us in these meetings helps, too.
[17:15] <gary_poster> cool :-)
[17:15] <matsubara> thanks Chex and gary_poster
[17:15] <matsubara> [TOPIC] * DBA report (stub)
[17:15] <MootBot> New Topic:  * DBA report (stub)
[17:15] <matsubara> I'll ask stub to send the report to the list
[17:15] <matsubara> [action] matsubara to ask stub for the DBA report
[17:15] <MootBot> ACTION received:  matsubara to ask stub for the DBA report
[17:15] <matsubara> [TOPIC] * QA stats of the week
[17:15] <MootBot> New Topic:  * QA stats of the week
[17:15] <matsubara> I don't have stats for this week. :-(
[17:15] <matsubara> sorry, I'll add them later to the MeetingAgenda page. need to find the script which generates it
[17:16] <matsubara> [action] matsubara to update qa stats in the MeetingAgenda page
[17:16] <MootBot> ACTION received:  matsubara to update qa stats in the MeetingAgenda page
[17:16]  * gary_poster is happy because they would show shameful stats :-)  ...working on them now...
[17:16] <matsubara> hehe
[17:16] <matsubara> [TOPIC] * Proposed items
[17:16] <MootBot> New Topic:  * Proposed items
[17:16] <sinzui> gary_poster, I think everyone will see soyuz first
[17:16] <matsubara> no new proposed items
[17:16] <gary_poster> ok
[17:16] <matsubara> sinzui, soyuz cleared their untriaged queue last week
[17:17]  * sinzui is shocked
[17:17] <matsubara> anything else before I close?
[17:17] <bigjools> feh
[17:17] <matsubara> :-)
[17:18] <matsubara> worst offenders are now foundations: launchpadlib and oops-tools
[17:18] <matsubara> actually no, oops-tools has 0 new untriaged. I cleared that a few weeks ago
[17:19] <matsubara> well, I guess that's it for today
[17:19] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda  for the logs.
[17:19] <matsubara> #endmeeting
[17:19] <MootBot> Meeting finished at 11:19.
[17:19] <gary_poster> thank you!
=== gary_poster is now known as gary-lunch
=== salgado-lunch is now known as salgado
=== gary-lunch is now known as gary_poster
=== EdwinGrubbs is now known as Edwin-lunch
=== salgado is now known as salgado-afk
=== mwhudson_ is now known as mwhudson