[17:00] <matsubara> moo
[17:00] <sinzui> cluck
[17:00] <gary_poster> hee-haw
[17:01] <danilos> meow
[17:01] <Ursinha> hello
[17:01] <Ursinha> #startmeeting
[17:01] <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[17:01] <MootBot> Meeting started at 11:01. The chair is Ursinha.
[17:01] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[17:01] <Ursinha> [TOPIC] Roll Call
[17:01] <MootBot> New Topic:  Roll Call
[17:01] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[17:01] <sinzui> me
[17:01] <Ursinha> meeee
[17:01] <gary_poster> smee
[17:01] <matsubara> me
[17:02] <Ursinha> rockstar, Chex, hello :)
[17:03] <Ursinha> I see no soyuz people :(
[17:03] <Ursinha> I know we'll have no bugs people here, so ok
[17:03] <Ursinha> well adeuring is here, but not sure if it's for our meeting :)
[17:03] <Ursinha> I'll move on and expect people to show up
[17:03] <Ursinha> [TOPIC] Agenda
[17:03] <Ursinha>  * Actions from last meeting
[17:03] <Ursinha>  * Oops report & Critical Bugs & Broken scripts
[17:03] <Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
[17:03] <Ursinha>  * DBA report (stub)
[17:03] <MootBot> New Topic:  Agenda
[17:03] <Ursinha>  * QA stats of the week
[17:03] <Ursinha>  * Proposed items
[17:03] <adeuring> Ursinha: yes, that's why I'm here, but I have another (mumble) meeting right now.
[17:04] <Ursinha> [TOPIC] * Actions from last meeting
[17:04] <Ursinha>  * DONE: matsubara to chase someone from Bugs about https://bugs.edge.launchpad.net/malone/+bug/583385
[17:04] <MootBot> New Topic:  * Actions from last meeting
[17:04] <Ursinha> cool, all done
[17:04] <Ursinha> adeuring, thanks :)
[17:04] <Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[17:04] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[17:04] <Ursinha> main offenders are bug 504124 (spam processing bug?) and bug 580181
[17:04] <Ursinha> I see the latter should be worked by noodles as soon as he got rid of more urgent problems
[17:04]  * rockstar is here...  had computer problems
[17:04]  * Ursinha hugs rockstar 
[17:04] <Chex> Ursinha: All us LOSAs are unavailable this week as we are sprinting in Montreal.
[17:05] <Ursinha> Chex, oh, that's bad, LOSAs' report is so useful
[17:05]  * Ursinha thinks
[17:05] <sinzui> the process-mail.py bug is a spam attack. Do we want to suppress oopses when the header is malformed?
[17:05] <Ursinha> this meeting is crippled :/
[17:05] <Ursinha> sinzui, good question
[17:05] <Ursinha> gary_poster, we're having annoying high oops counts of 504124, maybe we should do something about it now the counting seems to be raising?
[17:05] <Ursinha> gary_poster, even what sinzui suggested
[17:06] <matsubara> sinzui, I think we should not log an oops in that case as you suggested in the bug report
[17:06] <sinzui> Ursinha, gary_poster. The problem is not the app, but the data so we should gracefully recover I think
[17:06] <matsubara> with a try:except when the code is processing the To: header
[17:06] <Ursinha> that's my opinion as well
[17:06] <gary_poster> Ursinha: yeah, I had that one open already.  If it is really bothering us then let's do it.  Agree with sinzui's approach
[17:06] <Ursinha> cool
[17:07] <Ursinha> gary_poster, should I update the bug report with that?
[17:07] <sinzui> well, that is now a trivial fix I think
[17:07] <gary_poster> Ursinha: I'll handle it, thank you :-)
[17:07] <Ursinha> thanks gary_poster :)
[17:07] <Ursinha> nice
[17:07] <Ursinha> thanks all :)
[17:07] <Ursinha> I'd need a soyuz people here now
[17:07] <Ursinha> I'll go with the code one then
[17:08] <Ursinha> rockstar, so, not sure if code team knows about it: OOPS-1607EA658
[17:08] <sinzui> I think the soyuz bug has a deeper root cause...
[17:08] <sinzui> I looked at it last month and could not see what was happening. I think it goes back to changes made in Jan/Feb
[17:08] <Ursinha> sinzui, might have, this was "fixed" previously and now shows two different oopses
[17:08] <rockstar> Ursinha, so, I didn't know about it, but this area still hasn't been tested much...  :/
[17:09] <rockstar> Ursinha, it won't be released.
[17:09] <Ursinha> rockstar, oh, right
[17:09] <Ursinha> rockstar, was that QAed on edge? I'm pointing the oops as it might be a consequence of QAing the feature
[17:09] <rockstar> Ursinha, well, so no, we can't QA on edge much because we're blocked on two RTs...
[17:10] <Ursinha> rockstar, I see :/ do you know if the blockage is something that might last for long?
[17:11] <rockstar> Ursinha, well, the RT was filed as high priority more than a week ago, so your guess is as good as mine.
[17:11] <Ursinha> rockstar, maybe we should poke some people? :)
[17:11]  * Ursinha cries for a losa
[17:11] <gary_poster> rockstar, Ursinha, is this ...
[17:11] <rockstar> gary_poster!
[17:12] <gary_poster> 39402?
[17:12] <rockstar> Ursinha, gary_poster is the one who's been chasing for us.
[17:12] <gary_poster> RT 39402?
[17:12] <Ursinha> ah, I see
[17:12] <rockstar> gary_poster, yes, that's it.
[17:12] <gary_poster> Ursinha, rockstar, mthaddon has said that he will look into it today
[17:12] <rockstar> gary_poster, ta
[17:12] <gary_poster> or to not misrepresent: "member:gary: I'll try and take a look at the latter one (the first one is marked resolved) sometime today" :-)
[17:13] <Ursinha> cool!
[17:13] <Ursinha> thanks gary_poster and rockstar :)
[17:13] <gary_poster> np
[17:13] <Ursinha> rockstar, I'll keep my eyes on that oopses and let you know if something shows up
[17:13] <Ursinha> thanks
[17:14] <danilos> rockstar, my suggestion is to also talk to bigjools and lamont to make sure your request is similar to what is being done for "unsupported" packages already
[17:14] <danilos> rockstar, about that RT at least
[17:14] <Ursinha> [action] talk to bigjools about the 300 timeouts in the DistroSeries:+index pages everyday, is that expected/known? e.g.: OOPS-1607C1282
[17:14] <MootBot> ACTION received:  talk to bigjools about the 300 timeouts in the DistroSeries:+index pages everyday, is that expected/known? e.g.: OOPS-1607C1282
[17:14] <matsubara> Ursinha, isn't that the one bigjools replied to recently?
[17:14] <matsubara> the one that is making an xmlrpc request to another server but is being blocked by a firewall
[17:14] <Ursinha> matsubara, not sure if that's the same, I guess that one was about builders
[17:15] <sinzui> Ursinha, That is my oops
[17:15] <matsubara> ok, just wondering
[17:15] <Ursinha> matsubara, thanks for that :)
[17:15] <Ursinha> sinzui, the DistroSeries:+index?
[17:15] <danilos> Ursinha, yeah, that sounds like registry
[17:15] <Ursinha> if so, even better, you're here :)
[17:15] <sinzui> the distroseries-portlet-packaging  is the root cause. I have a branch that add3 3 hours caching to it
[17:15] <danilos> and fwiw, storm taking too long to serialize stuff
[17:15] <Ursinha> [action] forget last action
[17:15] <MootBot> ACTION received:  forget last action
[17:16] <sinzui> Edwin's definitve SQL fix will not land until next release
[17:16] <gary_poster> Ursinha, so you know, thumper alerted me last night that this soft timeout is causing timeouts (hard, I assume) on the branch merge proposal page: https://lp-oops.canonical.com/oops.py/?oopsid=1607EA56 .  Therefore I need to make a bug and look into it.
[17:16] <gary_poster> (Which sucks because pretty much none of know librarian code, but oh well :-) )
[17:16] <Ursinha> gary_poster, thanks for that
[17:16] <Ursinha> sinzui, do you have the bug number?
[17:17] <sinzui> My oops is caused by a monster query. bug 535430 is the root cause
[17:17] <Ursinha> sinzui, right
[17:17] <sinzui> ^ this is really a model problem that manifests itself on three pages
[17:18] <Ursinha> sinzui, which ones?
[17:19] <sinzui> distroseries +index, +needs-packaging +packaging-links  (distroseries-portlet-packaging) plus two full pages
[17:20] <Ursinha> thank you
[17:20]  * Ursinha takes notes
[17:20] <Ursinha> thanks everyone
[17:20] <sinzui> ^ +packaging, not +packaging-links
[17:20]  * Ursinha fixes her notes
[17:21] <sinzui> https://edge.launchpad.net/ubuntu/maverick shows that the issue centers on the Upstream packaging data
[17:22] <Ursinha> sinzui, so your fix will "fix" only part of the problem
[17:22] <sinzui> yes. timeouts will be reduced by 50% to 80%
[17:22] <sinzui> That is still disappointing
[17:22] <Ursinha> that's something
[17:23] <Ursinha> I guess the model problem won't be fixed (if fixed) anytime soon..
[17:23] <sinzui> The fix will be too complicated for a CP
[17:24] <Ursinha> sinzui, do you think that will be fixed soon?
[17:24] <sinzui> It will be fixed in week 1 on staging
[17:24] <Ursinha> oh, cool
[17:24] <Ursinha> better than I imagined
[17:25] <Ursinha> thanks sinzui
[17:25] <Ursinha> I might move on, otherwise the meeting will last forever :)
[17:25] <Ursinha> we have no critical bugs! \o/
[17:25] <gary_poster> \o/
[17:26] <Ursinha> rockstar, upgrade_branches is still failing
[17:26] <Ursinha> gary_poster, :)
[17:26] <rockstar> Ursinha, yeah, I'll look into it.  I have some other P1 bugs that need to be fixed first.
[17:26] <Ursinha> rockstar, sure. Thanks for that. Do you know if there are any bugs opened for that? I can do that!
[17:27] <rockstar> Ursinha, can you please open one?
[17:27] <Ursinha> rockstar, sure :)
[17:27] <Ursinha> rockstar, I'll do that and let you know. Thanks
[17:27] <rockstar> Ursinha, thanks.
[17:28] <Ursinha> [action] Ursinha to open a bug for the failing upgrade_branches script
[17:28] <MootBot> ACTION received:  Ursinha to open a bug for the failing upgrade_branches script
[17:28] <Ursinha> cool, moving on
[17:28] <Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[17:28] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[17:28] <Ursinha> losas are sprinting, so no donut for us
[17:28] <Ursinha> :/
[17:29] <Ursinha> Chex, may I ask you to send any important news to the list, when you have time? please? :)
[17:29]  * Ursinha values that report
[17:29] <Ursinha> it's more a wish than anything else, as you're sprinting
[17:29] <Ursinha> :)
[17:29] <Ursinha> moving on!
[17:29] <Ursinha> [TOPIC] * DBA report (stub)
[17:29] <MootBot> New Topic:  * DBA report (stub)
[17:30] <Ursinha> Email asking for the report sent!
[17:30] <Ursinha> [TOPIC] * QA stats of the week
[17:30] <MootBot> New Topic:  * QA stats of the week
[17:30] <Ursinha> untested stuff goes this way:
[17:30] <Ursinha> Code: 7
[17:30] <Ursinha> Foundations: 3
[17:30] <Ursinha> Registry: 4
[17:30] <Ursinha> Bugs: 3
[17:30] <Ursinha> Translations: 7
[17:30] <Ursinha> Soyuz: 3
[17:30] <Ursinha> Strategy: 0
[17:31] <Ursinha> I see we're kind of stable re. the number of untested items every week, which is good :)
[17:31] <Ursinha> right
[17:31] <Ursinha> about the untriaged bugs
[17:31] <Ursinha> I've sent one email to lp list asking your opinions about all projects we should take care of
[17:32] <Ursinha> I don't think pasting the numbers here will be useful now, I'd like you to reply that thread so we know what to do about those
[17:32] <Ursinha> pretty please?
[17:32] <danilos> stats for untested stuff seem incorrect for translations :)
[17:32] <Ursinha> danilos, it was like that 30 mins ago
[17:32] <danilos> Ursinha, I doubt it, maybe something like 1h or more ago :)
[17:32] <gary_poster> heh
[17:33]  * gary_poster is huungry :-D
[17:33] <Ursinha> danilos, how come? give me a paper bag, please
[17:33] <Ursinha> ok ok
[17:33] <Ursinha> danilos, you're right, I see only three now
[17:33] <danilos> Ursinha, oh, the graph is probably slow to update, don't worry about it
[17:34] <Ursinha> danilos, thanks :)
[17:34] <Ursinha> so, please, answer the thread!
[17:34] <Ursinha> :_)
[17:34] <Ursinha> [TOPIC] * Proposed items
[17:34] <Ursinha> None from me.
[17:34] <MootBot> New Topic:  * Proposed items
[17:34] <Ursinha> anyone wants to say something?
[17:34] <Ursinha> okay :)
[17:34] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[17:34] <gary_poster> I'm preparing an email to Ursinha and matsubara about orphan branches
[17:34] <Ursinha> #endmeeting
[17:34] <MootBot> Meeting finished at 11:34.
[17:34] <gary_poster> bye
[17:35] <gary_poster> thank you
[17:35] <Ursinha> gary_poster, branches or commits?
[17:35] <Ursinha> oh, nevermind, I can read the thread later :P
[17:35] <gary-lunch> Ursinha: commits, from perspective of qa tagging :-)
[17:35] <Ursinha> gary-lunch, :)
[17:35] <Ursinha> thanks everyone, and bye :)
[17:35] <danilos> cheers
[17:36] <matsubara> thanks Ursinha