=== jelmer_ is now known as jelmer
[01:08] <bac> wgrant: i misspoke.
[01:08] <bac> bac@duckworth:~$ cat /etc/lsb-release
[01:08] <bac> DISTRIB_ID=Ubuntu
[01:08] <bac> DISTRIB_RELEASE=8.04
[01:08] <bac> DISTRIB_CODENAME=hardy
[01:08] <bac> DISTRIB_DESCRIPTION="Ubuntu 8.04.4 LTS"
[01:10] <wgrant> Phew.
=== jelmer_ is now known as jelmer
=== noodles785 is now known as noodles775
=== salgado_ is now known as salgado
=== salgado is now known as salgado-lunch
[16:00] <matsubara> #startmeeting
[16:00] <bigjools> me
[16:00] <MootBot> Meeting started at 11:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:01] <matsubara> chex, gary_poster, rockstar, bigjools, allenap: hi
[16:01] <gary_poster> me, sorry
[16:01]  * rockstar are here
[16:01] <bigjools> I meed earlier
[16:01] <matsubara> sorry bigjools, missed that
[16:01] <matsubara> apologies from Translations. they're sprinting
[16:02] <bigjools> easily missed I was premature :)
[16:02] <chex> here
[16:03] <allenap> me
[16:05] <matsubara> hi sinzui
[16:05] <sinzui> me
[16:05] <matsubara> ok, everyone is here
[16:05] <matsubara> [TOPIC] Agenda
[16:05] <MootBot> New Topic:  Agenda
[16:05] <matsubara>  * Actions from last meeting
[16:05] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:05] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:05] <matsubara>  * DBA report (stub)
[16:05] <matsubara>  * Proposed items
[16:05] <matsubara> [TOPIC] * Actions from last meeting
[16:05] <MootBot> New Topic:  * Actions from last meeting
[16:06] <matsubara>  * Ursinha to send email to stub about the dba report
[16:06] <matsubara>  * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
[16:06] <matsubara>  * matsubara to talk to gary_poster about bug 535071
[16:06] <matsubara>    * Done.
[16:06] <matsubara>  * Ursinha to send email to TLs about QA summaries
[16:06] <ubottu> Launchpad bug 535071 in launchpad-foundations "Better error handling when librarian is offline" [High,Triaged] https://launchpad.net/bugs/535071
[16:06] <matsubara>  * matsubara to triage open oops-tools bugs
[16:06] <matsubara>  * Ursinha to triage open lp-qa-tools bugs
[16:06] <matsubara> I did some triage for oops-tools bugs, still some to look at
[16:06] <Ursinha> matsubara: done the 1st and the 3rd, not the 2nd
[16:07] <sinzui> matsubara: I have instructions from spm regarding how to change the window or schedule. I will try to land a change next week
[16:07] <matsubara> [action] * Ursinha to send email to TLs about QA summaries
[16:07] <MootBot> ACTION received:  * Ursinha to send email to TLs about QA summaries
[16:07] <matsubara> [action]  * matsubara to triage open oops-tools bugs
[16:07] <MootBot> ACTION received:   * matsubara to triage open oops-tools bugs
[16:07] <matsubara> thanks sinzui and Ursinha
[16:08] <Ursinha> np matsubara
[16:08] <matsubara> sinzui, should I add the action item again or can I consider it as done?
[16:08] <sinzui> matsubara: yes. pease do
[16:09] <matsubara> [action]  * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
[16:09] <MootBot> ACTION received:   * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
[16:09] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:09] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:10] <matsubara> We're still seeing https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1541O1653 which seems to be a regression of bug 527985
[16:10] <ubottu> Launchpad bug 527985 in launchpad-foundations "backing out of openid dance creates oops" [High,Fix released] https://launchpad.net/bugs/527985
[16:10] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
[16:10] <matsubara> another one that seems to be a regression: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1545E465 --> bug 408738
[16:10] <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Fix released] https://launchpad.net/bugs/408738
[16:10] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:10] <matsubara> and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1544A949 regression of bug 508302
[16:10] <ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,Fix released] https://launchpad.net/bugs/508302
[16:10] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
[16:11] <matsubara> sinzui, timeout retrieving vouchers: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1543EA744, I remember a bug about that but couldn't find it
[16:11] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1543EA744
[16:11] <matsubara> not sure if it's the same thing
[16:11] <sinzui> matsubara: that was brad and myself
[16:11] <sinzui> matsubara: ISD is still making changes to services we depend on
[16:11] <matsubara> gary_poster, can you look at foundations one ^?
[16:12] <sinzui> matsubara: at this time we are fortunate that only bac and myself are affected
[16:12] <gary_poster> looking
[16:12] <matsubara> allenap, can you take a look at the two bugs above?
[16:12]  * allenap looks
[16:13] <bac> sinzui: argh
[16:13] <matsubara> sinzui, so should we just ignore those oopses for now? is there a bug that we could link to that oops report?
[16:13]  * sinzui read the oopses and knows what bac is up to during his work hours
[16:14] <sinzui> matsubara: I have not had a problem in the last two days.
[16:14] <sinzui> matsubara: If it persists I will report a bug on the correct ISD project,
[16:14] <bac> sinzui, matsubara : i see that user's project has had a voucher applied so perhaps it was spurious.  have we heard from him?
[16:15] <matsubara> bac, nope, I caught that in the oops summary
[16:16] <sinzui> matsubara: bac: this was a networking issue with saleforce. The reason you could not find the bug is that ISD retargeted the other bugs to their porjects
[16:16] <allenap> matsubara: I'll file a new bug for OOPS-1545E465; the linked bug is not right.
[16:16] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:16] <matsubara> sinzui, looks like the oops is triggerable by regular users as well
[16:16] <sinzui> matsubara: anyone using +vouchers is a regular user
[16:16] <allenap> matsubara: I'll talk to deryck about OOPS-1544A949.
[16:16] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
[16:17] <matsubara> [action] allenap to discuss with deryck about OOPS-1544A949 and file a bug for OOPS-1545E465.
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
[16:17] <MootBot> ACTION received:  allenap to discuss with deryck about OOPS-1544A949 and file a bug for OOPS-1545E465.
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:17] <sinzui> matsubara:  the oops is not in launchpad, it is salesforce. We started talking to ISD last month when vouchers broke
[16:17] <matsubara> [action] matsubara to fix linked bug for OOPS-1545E465
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:17] <MootBot> ACTION received:  matsubara to fix linked bug for OOPS-1545E465
[16:17] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
[16:17] <gary_poster> matsubara: I'll be discussing my oops with salgado and filing a new bug or something
[16:18] <matsubara> allenap, gary_poster, thanks
[16:18] <gary_poster> (s/or something/or reopening the old one)
[16:18] <matsubara> [action] gary_poster to discuss OOPS-1541O1653 with salgado and file a bug for it.
[16:18] <MootBot> ACTION received:  gary_poster to discuss OOPS-1541O1653 with salgado and file a bug for it.
[16:18] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
[16:18] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
[16:19] <matsubara> sinzui, in any case, lp should degrade gracefully when salesforce is offline or timing out
[16:19] <sinzui> DUDE! I know that
[16:20] <sinzui> This is not the channel to discuss ISD's changes to systems that affect launchpad and landscape
[16:21] <matsubara> ok lebowski, I know that you know that. just making sure we're in the same page. I'm more interested in the changes that we can make to avoid the oops from happening (i.e. degrade gracefully)
[16:21] <gary_poster> lol
[16:22] <sinzui> I favor removing salesforce. It is not my decision. We can evaluate our dependencies on outsiders when we start NewCo
[16:26] <matsubara> sinzui, rather than removing salesforce, can't we show a page saying that LP couldn't talk to salesforce while the duderino was browsing +vouchers? much like as your proposed solution to bug 267852?
[16:26] <ubottu> Launchpad bug 267852 in launchpad-foundations "google search timeouts are not usable" [Low,Triaged] https://launchpad.net/bugs/267852
[16:27] <sinzui> matsubara: because it is an oops. there is nothing to show nor any action to take. I could not license Canonical projects for an hour
[16:27] <bigjools> sigh - s/note/not/
[16:27] <bigjools> oops
[16:29] <matsubara> sinzui, makes sense then. I'll let you know if I see more of these OOPS and then you can escalate with ISD. thanks sinzui
[16:29] <matsubara> we have 5 critical bugs
[16:30] <matsubara> 3 fix committed, one in progress and one confirmed
[16:30] <matsubara> the one confirmed is assigned to me and I'll take care of it today
[16:31] <matsubara> the only broken script is flag-expired-memberships, which stub will take care of
[16:31] <matsubara> thanks everyone, let's move on
[16:31] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:31] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:32]  * matsubara hands the mic to chex 
[16:33] <chex> hello everyone
[16:33] <chex> here is the LOSA report for this week:
[16:33] <chex> - New ImportD server 'pear' has been setup this week and is processing jobs now.
[16:33] <chex> - PQM is closing at 2300 UTC Friday for LP roll-out next week.
[16:33] <chex> - SSO DB Migration project away from LP DB is delayed due to some issues, does anyone know
[16:33] <chex>         where we are placed with this now?
[16:33] <chex> - LP incidents of note: ; LP Cherry-picks:
[16:33] <chex>         22-Mar: CP 9095 to lpnet*, & codehost
[16:33] <chex>         24-Mar: CP 9097 to loganberry/checkwatches and 9098 to shipit*
[16:33] <chex> and thats the report for this week, any comments/questions??
[16:34] <matsubara> gary_poster, do you know about the SSO DB migration ^?
[16:34] <gary_poster> chex, matsubara, yes
[16:34] <gary_poster> chex, losas were cc'd on relevant emails
[16:35] <gary_poster> summary: um...
[16:35] <gary_poster> other Canonical openid apps that relied on launchpad team membership for privileges were going to break.
[16:36] <gary_poster> We have to handle that.
[16:36] <gary_poster> A plan is now in place to do so
[16:36] <gary_poster> It will involve some DB changes that stub is working on, and some changes for ISD
[16:37] <chex> gary_poster: ok, great, sounds good then, thanks
[16:37] <gary_poster> We regard splitting the by no later than two weeks from now as an absolute necessity, because of various pressures such as HD space and other excitement, and hope to have it addressed sooner (beginning of week after next)
[16:40] <matsubara> thanks gary_poster, chex
[16:40] <matsubara> [TOPIC] * DBA report (stub)
[16:40] <MootBot> New Topic:  * DBA report (stub)
[16:41] <matsubara> I'll email stub for the dba report
[16:41] <matsubara> [action] matsubara to email stub asking the dba report
[16:41] <MootBot> ACTION received:  matsubara to email stub asking the dba report
[16:41] <matsubara> [topic] QA Stats of the Week
[16:41] <MootBot> New Topic:  QA Stats of the Week
[16:41] <matsubara> Soyuz (soyuz): 70
[16:41] <matsubara> launchpadlib  (launchpadlib): 25
[16:41] <matsubara> OOPS Tools (oops-tools): 21
[16:41] <matsubara> Launchpad Developer Utilities (lp-dev-utils): 9
[16:41] <matsubara> Launchpad Bugs (malone): 8
[16:41] <matsubara> Launchpad Development Wiki Moin theme (launchpad-dev-moin-theme): 5
[16:41] <matsubara> Launchpad Auto Build System (launchpad-buildd): 5
[16:41] <matsubara> Launchpad Buildbot Configuration (lpbuildbot): 2
[16:42] <matsubara> those are the launchpad projects with untriaged bugs (new state)
[16:42] <bigjools> I'd like to point out that most of the soyuz ones are >2 years old
[16:42] <matsubara> close them all or mark them as triage low :-)
[16:42] <matsubara> triaged, I mean
[16:42] <bigjools> I do so when I have time
[16:43] <matsubara> thanks bigjools
[16:43] <bigjools> I spend more time triaging new ones
[16:43] <matsubara> cool
[16:43] <matsubara> Also please take a look at https://wiki.canonical.com/Launchpad/QATeam/OrphanedCommits/10.03-db-devel and https://wiki.canonical.com/Launchpad/QATeam/OrphanedCommits/10.03-devel
[16:43] <sinzui> bigjools: if you have ignored the issue for more than 6 months, it is 99% of the time low/triaged. the remaining 1% are security issues we should have fixed
[16:44] <matsubara> those are the landings for the 10.03 cycle which are not tracked in the http://people.canonical.com/~lpqateam/test-plan-report-10.03.html burndown
[16:44] <bigjools> sinzui: many of them are simply not bugs any more, in fact
[16:44] <sinzui> bigjools: yep
[16:44]  * sinzui closed 1/3 of the bugs as fixed that were supposed to go to launchpad-web
[16:45] <matsubara> [TOPIC] * Proposed items
[16:45] <MootBot> New Topic:  * Proposed items
[16:45] <matsubara> there's no new proposed items
[16:45] <matsubara> anything else before I close?
[16:45] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda  for the logs.
[16:46] <matsubara> #endmeeting
[16:46] <MootBot> Meeting finished at 11:46.
[16:46] <bigjools> thanks matsubara
=== Ursinha is now known as Ursinha-lunch
=== salgado-lunch is now known as salgado
=== Ursinha-lunch is now known as Ursinha
=== Ursinha is now known as Ursinha-sprintin
=== Ursinha-sprintin is now known as Ursinha-busy
=== salgado is now known as salgado-afk
=== Peng___ is now known as Peng_