[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:00] <herb`> me
[16:00] <sinzui> me
[16:01] <rockstar> me
[16:01] <matsubara> Ursinha: intellectronica, flacoste: hi
[16:01] <bigjools> me
[16:01] <Ursinha> hi, me
[16:01] <intellectronica> me
[16:02] <matsubara> stub is excused today. he already sent the dba report to the list
[16:02] <flacoste> me
[16:02] <matsubara> henninge: hi
[16:02] <henninge> me
[16:02] <henninge> hi mars
[16:02] <henninge> hi matsubara,  I meant
[16:03] <matsubara> ok, everyone here.
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs
[16:03] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:03] <matsubara>  * DBA report (stub)
[16:03] <matsubara> [TOPIC] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:03] <matsubara>     * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
[16:03] <matsubara>     * intellectronica to talk to gmb about bug 269538
[16:03] <matsubara>     * sinzui to take bug 358332
[16:03] <matsubara>     * sinzui to ask salgado to fix bug 357307
[16:04] <sinzui> matsubara: done and discussed further at the teamleads meeting
[16:04] <matsubara> sinzui: good. thanks. what was the outcome?
[16:05] <intellectronica> matsubara: i talked to gmb and we agreed that the bug is now fix released
[16:06] <sinzui> matsubara: for the time being, foundations and registry need to do better triage. We want to assign people/teams to these projects in the future to do small development and regular maitenance
[16:07] <matsubara> all right. the only pending item is #358332 which wasn't fixed for 2.2.4 but it's triaged and set to high. I guess that's going to be fixed for 2.2.5 then?
[16:07] <sinzui> matsubara: I'm pushing bug 358332 to 2.5 and I will take it as a part of my series work
[16:07] <matsubara> sinzui: perfect
[16:07] <matsubara> thanks intellectronica
[16:07] <matsubara> thanks sinzui
[16:07] <matsubara> let's move on
[16:07] <matsubara> [TOPIC] * Oops report & Critical Bugs
[16:07] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:07] <matsubara> go ahead Ursinha
[16:07]  * Ursinha waves
[16:07] <Ursinha> we have three bugs today, two high and one annoying
[16:07] <Ursinha> the two high are already being handled: bug 351641 (salgado) and bug 369858 (henninge)
[16:07] <Ursinha> the annoying is bug 357316, that's bug's team - could someone do something/set the importance on this one, intellectronica?
[16:08] <Ursinha> thanks ubottu
[16:08] <intellectronica> Ursinha: sure, will check
[16:08]  * matsubara wants an annoying status
[16:08] <Ursinha> thanks intellectronica
[16:08]  * Ursinha wants too
[16:08] <Ursinha> :)
[16:08] <rockstar> matsubara: aren't all bugs annoying?
[16:08] <Ursinha> rockstar, that's also true
[16:09] <Ursinha> that's all from me from the oopses-after-the-rollout land
[16:10] <Ursinha> matsubara, want to point anything?
[16:10] <matsubara> we have one in progress critical bug, which flacoste is investigating
[16:10] <matsubara> related to the shipit + openid thing
[16:11] <matsubara> bug 368767
[16:11] <matsubara> but that's all
[16:11] <matsubara> let's move on
[16:11] <matsubara> thanks everyone
[16:11] <flacoste> matsubara: actullay, it seems solved
[16:11] <herb`> shipit is up and running, btw.
[16:11] <flacoste> matsubara: i'll confirm and update the status
[16:11] <Ursinha> matsubara, we didn't have more of those (except some 5xx for a period of one minute after the rollout,m about 10)
[16:11] <matsubara> flacoste: that's great news.
[16:11] <matsubara> flacoste: thanks!
[16:12] <herb`> Ursinha: we might have had a burst of 5xxs just before the meeting started.  I haven't had a chance to investigate yet.  but the service seems to be operating fine at the moment.
[16:12] <matsubara> as usual, we'll be checking the OOPS reports all day long and bring up any issues we find to the teams
[16:13] <Ursinha> herb, I'll check right after the meeting, thanks for the heads up
[16:13] <matsubara> thanks again
[16:13] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[16:13] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:13] <matsubara> there you go herb
[16:13] <herb> 2009-04-23 - The spike in traffic to shipit after the jaunty rollout caused some service disruptions to launchpad. We're still working to isolate the root cause, but have some good ideas and think we have a solid work around.
[16:13] <herb> 2009-04-24 - Cherry pick of r8270 and r8273 to lpnet* and the internal xmlrpc server.
[16:13] <herb> 2009-04-29 - Rolled out 2.2.4 to production. Process went relatively smoothly. Because of a bit of short-sightedness read-only wasn't available for this rollout but will be for the next.
[16:13] <herb> Keeping it on the radar: bug 156453 and bug 118625 continue to be pain points for the LOSAs
[16:13] <herb> With Tom on holiday today and tomorrow, if there is going to be a reroll the code *must* be available by 2030 UTC tomorrow (or it gets pushed to Monday). I would like to be able to have at least an hour post-reroll to make sure there aren't any problems before I EOD.
[16:14] <herb> I already mentioned the 5xx alarm I saw before the meeting. and the fact that shipit is back on as of yesterday around 1900 UTC.
[16:15] <matsubara> herb: most likely that we'll have a re-roll today
[16:16] <matsubara> I'll confirm with kiko and keep you posted
[16:16] <herb> matsubara: there is a config change that needs to land
[16:16] <herb> matsubara: I don't have a checkout at the moment.  Can I piggyback a diff on someone else's commit?
[16:17] <matsubara> herb: I think it's ok. bac is working on a fix for a timeout. it might be a good candidate
[16:17] <herb> matsubara: I will sync up with him.
[16:17] <matsubara> in any case we can sort that out after the meeting. thanks for bringing it up
[16:17] <rockstar> herb: hopefully, the loggerhead stuff will start dissipating with the new code.
[16:17] <herb> rockstar: woohoo!
[16:18] <matsubara> cool
[16:18] <matsubara> anything else for herb?
[16:19] <matsubara> [action] matsubara to add to the CRB page herb's config change
[16:19] <MootBot> ACTION received:  matsubara to add to the CRB page herb's config change
[16:19] <matsubara> thanks herb
[16:19] <herb> thanks matsubara
[16:19] <matsubara> [TOPIC] * DBA report (stub)
[16:19] <MootBot> New Topic:  * DBA report (stub)
[16:19] <matsubara> so stub is excused today. he sent the dba report to the list
[16:19] <matsubara> if you have questions, follow up there.
[16:19] <matsubara> that's all
[16:20] <matsubara> anything else before I close the meeting?
[16:20] <matsubara> all right. Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:20] <matsubara> a good job on this release. went really smooth!
[16:21] <matsubara> #endmeeting
[16:21] <MootBot> Meeting finished at 10:21.
[16:21] <Ursinha> thanks all
[16:22] <intellectronica> thanks Ursinha, matsubara
[16:26] <flacoste> matsubara: no kidding!
[16:26] <flacoste> smooth rollout!