[15:59]  * Ursinha looks at MootBot
[15:59] <Ursinha> oh, he's here
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <sinzui> me
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:00] <henninge> me
[16:00] <rockstar> ni!
[16:00] <bigjools> me
[16:01] <henninge> rockstar: that's plural, 'mi' is singular.
[16:01] <Ursinha> me
[16:01] <intellectronica> me
[16:01] <matsubara> Chex, hi
[16:01] <matsubara> gary_poster, hi
[16:01] <rockstar> henninge, :)  Making a Monty Python reference.
[16:01] <gary_poster> me
[16:01] <gary_poster> thanks
[16:01] <henninge> oh, don't know that one.
[16:01] <matsubara> apologies from stub
[16:01] <henninge> rockstar: Mine was an Esperanto reference
[16:01] <matsubara> [TOPIC] Agenda
[16:01] <MootBot> New Topic:  Agenda
[16:02] <matsubara>  * Actions from last meeting
[16:02] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:02] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:02] <matsubara>  * DBA report (stub)
[16:02] <matsubara>  * Proposed items
[16:02] <matsubara> [TOPIC] * Actions from last meeting
[16:02] <MootBot> New Topic:  * Actions from last meeting
[16:02] <matsubara>     * barry to continue debug on bug 403606
[16:02] <matsubara>     * intellectronica to take a look or find someone to take a look on bug 408738
[16:02] <matsubara>     * matsubara to chase ubottu maintainer to fix regexp used to identify oopses(i.e. oops-tools shouldn't trigger the bot)
[16:02] <matsubara>         * filed bug 420032 and talked to jussi01 on #ubuntu-bots about it.
[16:02] <Chex> matsubara: hi
[16:02] <matsubara> ubottu, oops-tools
[16:02] <matsubara> oops-tools
[16:03] <matsubara> cool. looks like that one was fixed :-)
[16:03] <Ursinha> \o/
[16:03] <gary_poster> heh
[16:03] <Ursinha> ubottu, good boy
[16:03] <gary_poster> lol
[16:03] <intellectronica> matsubara: it's on the radar, but we haven't yet had the chance to fix it. as you can imagine, v3.0 UI is keeping everyone very busy
[16:03] <matsubara> sinzui, any news about 403606?
[16:04] <matsubara> intellectronica, right. shall I keep it in the action items list?
[16:04] <sinzui> no
[16:04] <intellectronica> matsubara: yes, please
[16:04] <sinzui> I ask my team to only work on UI
[16:04] <Ursinha> intellectronica, thanks
[16:04] <matsubara> thanks intellectronica and sinzui
[16:05] <matsubara> [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
[16:05] <MootBot> ACTION received:  * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
[16:05] <sinzui> matsubara: Since there was nothing obviously wrong, we cannot judge the scope of the issue
[16:05] <matsubara> [action]  * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
[16:05] <MootBot> ACTION received:   * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
[16:06] <matsubara> sinzui, ok, let's keep on the radar though. that's the most frequent oops with have
[16:06] <matsubara> right Ursinha ?
[16:06] <Ursinha> right
[16:06] <matsubara> ok, thanks
[16:06] <matsubara> let's move on
[16:06] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:06] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:06] <Ursinha> that's me
[16:06] <matsubara> Ursinha, stage is yours
[16:06] <Ursinha> bugs 4-08738 and 4-03606 already mentioned and discussed
[16:06] <Ursinha> I have bug 422560 for answers, but rockstar told me yesterday he's on it, right rockstar? could you update the bug, pretty please?
[16:06] <Ursinha> gary_poster, spm reported this one yesterday: bug 422960, seems to be foundations, what do you say?
[16:07] <rockstar> Ursinha, yup, getting ready to submit for review.
[16:07] <gary_poster> ...looking...
[16:07] <Ursinha> rockstar, cool, can you add your precious comments to the bug report? :)
[16:07] <matsubara> I asked some info from spm on that foundations bug Ursinha
[16:08] <Ursinha> matsubara, I see
[16:08] <rockstar> Ursinha, yes ma'am
[16:08] <Ursinha> thanks rockstar :)
[16:08] <Ursinha> ah
[16:08] <Ursinha> rockstar, bug 401631 is the one about +branches timeout, it's committed on edge but I see lots of timeouts on lpnet daily
[16:08] <matsubara> Ursinha, how about: 497 ValueError: Invalid boundary in multipart form: '' ?
[16:08] <Ursinha> maybe worth a cherrypick?
[16:08] <gary_poster> Ursinha, I'll say yes at least on a first take.  We'll at least do some investigation.
[16:08] <rockstar> Ursinha, I'll chat with thumper about it.
[16:09] <gary_poster> and either take it or figure out who else should take it :-)
[16:09] <gary_poster> so it's ours now, anyway
[16:09] <Ursinha> thanks gary_poster
[16:09] <matsubara> Ursinha, I'm pretty sure the timeouts on person +branches was CP already
[16:10] <Ursinha> matsubara, I still see timeouts
[16:10] <matsubara> Ursinha, it was CP on the 1s. did the CP at least alleviated the problem?
[16:11] <matsubara> Ursinha, and how about the 497 ValueErrors on edge?
[16:11] <Ursinha> matsubara, wait man :)
[16:12] <Ursinha> it seems the problem is better now
[16:12] <matsubara> I am
[16:12] <Ursinha> I'll keep watching the next days
[16:12] <Ursinha> about the value error
[16:12] <Ursinha> I was going to ask rockstar if that's a code problem
[16:12] <Ursinha> rockstar, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1341EA197
[16:12] <matsubara> [action] ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
[16:12] <MootBot> ACTION received:  ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
[16:13] <rockstar> Ursinha, huh, that's a weird and informationless oops.
[16:13] <Ursinha> rockstar, agreed about the informationless
[16:13] <matsubara> [action] ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
[16:13] <MootBot> ACTION received:  ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
[16:14] <Ursinha> my only hint was the path
[16:14] <Ursinha> I'll file a bug
[16:14] <matsubara> thanks Ursinha and rockstar
[16:14] <Ursinha> ok
[16:14] <Ursinha> we have one critical bug, already fix committed
[16:14] <Ursinha> about the failing script, rosetta-poimport, henninge, do you know what's causing that to fail?
[16:15] <matsubara> Ursinha, isn't that the one that danilo replied to?
[16:15] <henninge>  Ursinha: yes, it's got to do with sending emails.
[16:15] <matsubara> s/danilo/jtv and danilo/
[16:15] <Ursinha> matsubara, yes, but I don't seem to see a conclusion there
[16:15] <henninge> Ursinha: jtv found the bug and the fix is on the way in
[16:15] <Ursinha> henninge, awesome :) thanks!
[16:16] <matsubara> cool
[16:16] <henninge> Ursinha: we are going to CP it
[16:16] <Ursinha> henninge, great, will jtv ask for a CP?
[16:16] <henninge> yes
[16:16] <henninge> I believe so, at least.
[16:16] <Ursinha> henninge, that's good
[16:16] <matsubara> [action] matsubara to chase jtv about CP to fix rosetta-poimport script failure
[16:16] <MootBot> ACTION received:  matsubara to chase jtv about CP to fix rosetta-poimport script failure
[16:16] <Ursinha> thakn you :)
[16:16] <Ursinha> *thank
[16:17] <Ursinha> thanks all
[16:17] <Ursinha> matsubara, go ahead
[16:17] <matsubara> hmm actually I think we should swap some action items Ursinha :-)
[16:17] <matsubara> but I'll talk about it after the meeting
[16:17] <matsubara> thanks Ursinha and everyone
[16:17] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:17] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:18] <matsubara> Chex, ?
[16:18] <mbarnett> sorry, i think i might have been distracting him with a question.
[16:19] <Chex> sorry
[16:19] <Chex> here is our report for the week:
[16:19] <Ursinha> losas.. :)
[16:20] <Chex> Increasing number of 5XX errors in apache pretty much entirely from msn
[16:20] <Chex> bot on translations pages
[16:20] <Chex> Bug about missing OOPSes - bug 422960
[16:20] <Chex> Codebrowse is still needing restarting pretty regularly per the Incident Log
[16:21] <Chex> and thats all from us, unless there are any questions?
[16:21] <matsubara> Chex, we (as in foundations team) will investigate the 500s. I don't think we should be logging OOPS for those as they don't seem to be hitting the app server. It looks like an issue with the load balancer or apache
[16:21] <rockstar> Chex, I've looked into that.  There is more to look at, but I think we have some hints on wtf is going on.
[16:21]  * mbarnett cheers 
[16:22] <gary_poster> Chex, I'll follow up with you all on the 5XX later today.
[16:22] <Chex> great, thanks for the response/feedback
[16:23] <matsubara> Chex, would be awesome if you could add a request that generated one of the 500 to the bug report
[16:23] <gary_poster> +1
[16:23] <matsubara> [action] chex to trawl logs and add request that caused 500 error to bug 422960
[16:23] <MootBot> ACTION received:  chex to trawl logs and add request that caused 500 error to bug 422960
[16:25] <Chex> matsubara: ok, I will look into that for you
[16:25] <matsubara> thanks Chex
[16:25] <Chex> matsubara: welcome
[16:25] <matsubara> let's move on
[16:25] <matsubara> [TOPIC] * DBA report (stub)
[16:25] <MootBot> New Topic:  * DBA report (stub)
[16:25] <matsubara> so stuart is not around. I'll email him about the dba report
[16:25] <matsubara> [action] matsubara to email stub about dba report
[16:25] <MootBot> ACTION received:  matsubara to email stub about dba report
[16:25] <matsubara> [TOPIC] * Proposed items
[16:25] <MootBot> New Topic:  * Proposed items
[16:26] <matsubara> no proposed items
[16:26] <matsubara> so I think that's all for today
[16:26] <matsubara> anything else before I close?
[16:26] <Ursinha> nothing here
[16:27] <matsubara> ok, so be it
[16:27] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[16:27] <matsubara> #endmeeting
[16:27] <MootBot> Meeting finished at 10:27.
[16:27] <gary_poster> thanks matsubara and Ursinha!
[16:27] <Ursinha> thanks all!