[15:00] <matsubara> #startmeeting
[15:00] <MootBot> Meeting started at 09:00. The chair is matsubara.
[15:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:00] <matsubara> [TOPIC] Roll Call
[15:00] <MootBot> New Topic:  Roll Call
[15:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[15:00] <noodles775> moi
[15:00] <matsubara> apologies from stub and Ursinha
[15:00] <allenap> me
[15:00] <matsubara> and gary too
[15:01] <matsubara> rockstar, Chex, sinzui: hi
[15:01] <matsubara> danilos, h
[15:01] <matsubara> danilos, hi
[15:01] <sinzui> matsubara: It is a national holiday for the US members
[15:01] <danilos> me
[15:01] <danilos> oh, my clock is delayed still
[15:02] <matsubara> sinzui, right, I guess rockstar is not joining then. aren't you taking the holiday?
[15:02] <noodles775> matsubara: ah, that's why bigjools isn't around... I forgot.
[15:02] <noodles775> He took a day holiday as everyone else in the states was on holidays.
[15:02] <sinzui> matsubara: I am but I happen to be here and I can answer questions
[15:02] <matsubara> noodles775, I see. thanks for covering for him :-)
[15:03] <matsubara> today's meeting should be quick
[15:03] <matsubara> let's move on
[15:03] <matsubara> [TOPIC] Agenda
[15:03] <MootBot> New Topic:  Agenda
[15:03] <matsubara>  * Actions from last meeting
[15:03] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[15:03] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[15:03] <matsubara>  * DBA report (stub)
[15:03] <matsubara>  * Proposed items
[15:03] <matsubara> [TOPIC] * Actions from last meeting
[15:03] <MootBot> New Topic:  * Actions from last meeting
[15:03] <matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
[15:03] <matsubara> * matsubara to chase someone from code team about bug 480000
[15:03] <matsubara> * matsubara to chase code people about code script failures (create-merge-proposals, branch puller and update branches)
[15:03] <matsubara> * matsubara to ask someone from code about bug 485318
[15:03] <matsubara> * Chex to talk to thumper about the multiple git import failures on the importd
[15:04] <matsubara> I didn't talk to code people about all the issues.
[15:04] <matsubara> [action] * Ursinha to send one email to lp list explaining the qa-tags experiment
[15:04] <MootBot> ACTION received:  * Ursinha to send one email to lp list explaining the qa-tags experiment
[15:04] <matsubara> [action] * matsubara to chase someone from code team about bug 480000
[15:04] <MootBot> ACTION received:  * matsubara to chase someone from code team about bug 480000
[15:04] <matsubara> [action] * matsubara to chase code people about code script failures (create-merge-proposals, branch puller and update branches)
[15:04] <MootBot> ACTION received:  * matsubara to chase code people about code script failures (create-merge-proposals, branch puller and update branches)
[15:04] <matsubara> [action] * matsubara to ask someone from code about bug 485318
[15:04] <MootBot> ACTION received:  * matsubara to ask someone from code about bug 485318
[15:05] <matsubara> Chex, did you talk to thumper about the git import failures?
[15:06] <Chex> matsubara: hi, I pinged him in my morning today, but we are time-zone challenged, sorry I didnt get back to him sooner on that
[15:07] <matsubara> [action] Chex to follow up with thumper about the multiple git import failures on the importd
[15:07] <MootBot> ACTION received:  Chex to follow up with thumper about the multiple git import failures on the importd
[15:08] <matsubara> thanks Chex
[15:08] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[15:08] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[15:08] <matsubara> one bug for sinzui: https://bugs.edge.launchpad.net/launchpad-registry/+bug/488762
[15:08] <matsubara> which you already triaged
[15:09] <matsubara> :-)
[15:09] <sinzui> of course
[15:09] <matsubara> thanks sinzui
[15:09] <matsubara> and another one for danilos:  https://bugs.edge.launchpad.net/rosetta/+bug/488765
[15:09] <matsubara> and another OOPS-1420ED1047 for danilos which I haven't filed a bug for yet
[15:09] <danilos> matsubara, looking
[15:09] <matsubara> danilos, can you take a look at those and triage accordingly please?
[15:10] <danilos> matsubara, sure, thanks
[15:10] <matsubara> [action] matsubara to file a bug for OOPS-1420ED1047
[15:10] <MootBot> ACTION received:  matsubara to file a bug for OOPS-1420ED1047
[15:11] <matsubara> danilos, I'll let you know once I have a bug for that one
[15:11] <matsubara> thanks danilos and sinzui
[15:11] <matsubara> we have 2 critical bugs, one is fix committed and another one is in progress
[15:11] <matsubara> so all good
[15:11] <danilos> matsubara, I think I've seen that one already (https://lp-oops.canonical.com/oops.py/?oopsid=1420ED1047)
[15:11] <matsubara> we had 2 script failures since last week
[15:12] <matsubara> one in checkwatches which gmb already took care of
[15:12] <matsubara> and another one in distributionmirror-prober
[15:12] <matsubara> sinzui, does that one ^ belongs to registry?
[15:13] <danilos> matsubara, yes, that's bug 484368
[15:13] <matsubara> danilos, thanks
[15:14] <sinzui> matsubara: I will take the mirror prober
[15:14] <matsubara> [action] sinzui to investigate failure on the mirror prober (The script 'distributionmirror-prober' didn't run on 'loganberry' between 2009-11-23 06:07:04 and 2009-11-23 12:07:04 (last seen 2009-11-23 04:54:10.444057))
[15:14] <MootBot> ACTION received:  sinzui to investigate failure on the mirror prober (The script 'distributionmirror-prober' didn't run on 'loganberry' between 2009-11-23 06:07:04 and 2009-11-23 12:07:04 (last seen 2009-11-23 04:54:10.444057))
[15:14] <matsubara> thanks sinzui
[15:15] <matsubara> I think that's all for this section
[15:15] <matsubara> let's move on
[15:15] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[15:15] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[15:16] <Chex> hello everyone, quick report today
[15:16] <Chex> - Python 2.5 upgrade: Update on how testing is going on Python 2.5?
[15:16] <Chex> - next LP rollout: Scheduled for Wed 02 Dec, 2009 at 2200 UTC.
[15:17] <Chex> - LP incidents of note:
[15:17] <Chex>         ; CP 8653 rolled out to loganberry and bzrsyncd
[15:17] <Chex>         ; continued Codebrowse restarts
[15:17] <Chex> and thats our report for today. Any questions??
[15:19] <matsubara> Chex, I'm not sure about the python2.5 upgrade status. That's a question for Gary, but he's on holiday today. do you want me to chase that for you?
[15:20] <Chex> matsubara: yes that would be great if you could, please
[15:20] <matsubara> [action] matsubar to ask gary about python2.5 update and get back to losas
[15:20] <MootBot> ACTION received:  matsubar to ask gary about python2.5 update and get back to losas
[15:21] <matsubara> I can't type my own name :-(
[15:21] <matsubara> thanks Chex!
[15:21] <mthaddon> matsufail
[15:21] <matsubara> hehe
[15:22] <matsubara> Chex, btw, there's some progress regarding the codebrowser restarts. Max seems to have identified some bugs and likely some code will land soon
[15:23] <matsubara> Chex, mthaddon: there is an email thread going on in the list about it if you are interested in the details
[15:23] <matsubara> let's move on
[15:23] <matsubara> [TOPIC] * DBA report (stub)
[15:23] <MootBot> New Topic:  * DBA report (stub)
[15:23] <matsubara> relaying for stub
[15:23] <matsubara> I've got the first part of LP standalone DB working (build, configure and control a launchpad specific PostgreSQL).
[15:23] <matsubara> In DB land we have some more graphs detailing database load for scripts - no surprises so far.
[15:23] <matsubara> sanitized db generation is still underway. Last run was 5 days before I killed it and optimized the slow bit.
[15:23] <matsubara> Staging restorations were slow due to a very inefficient restore procedure (we had to change due to production replication setup changes). This process has been optimized and a restoration currently underway.
[15:24] <mthaddon> matsubara: nothing stands out from the DB user CPU usage graphs?
[15:24] <mthaddon> matsubara: we're seeing a lot of load increase on wildcherry and would really like to get to the bottom of it
[15:25] <matsubara> mthaddon, I haven't seen the graphs, I'm relaying the report stub sent to me. I'll pass your question on to him and ask stub to contact you. is that ok?
[15:25] <mthaddon> ah right
[15:26] <mthaddon> yeah, of course
[15:26] <matsubara> [action] matsubara to ask stub to contact losas about load increase on wildcherry
[15:26] <MootBot> ACTION received:  matsubara to ask stub to contact losas about load increase on wildcherry
[15:27] <matsubara> ok, last one
[15:27] <matsubara> [TOPIC] * Proposed items
[15:27] <matsubara> there are no proposed items
[15:27] <MootBot> New Topic:  * Proposed items
[15:27] <matsubara> which brings this meeting to an end, unless someone have any questions?
[15:29] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[15:29] <matsubara> #endmeeting
[15:29] <MootBot> Meeting finished at 09:29.