[15:00] <matsubara> #startmeeting
[15:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:00] <Ursinha> erm
[15:00] <matsubara> [TOPIC] Roll Call
[15:00] <herb> me
[15:00] <Ursinha> me
[15:00] <mrevell> me
[15:00] <sinzui> me
[15:00] <rockstar> we
[15:00] <matsubara> so who's here today?
[15:00] <sinzui> I'm still here
[15:00] <Ursinha> everyone except the bot
[15:00] <rockstar> me too
[15:00] <matsubara> lag
[15:01] <matsubara> flacoste, bigjools: ?
[15:01] <bigjools> me
[15:01] <flacoste> me
[15:01] <henninge> me
[15:02] <intellectronica> me
[15:02] <matsubara> all right. everyone is here
[15:02] <matsubara> [TOPIC] Agenda
[15:03] <matsubara>  * Next meeting
[15:03] <matsubara>  * Actions from last meeting
[15:03] <matsubara>  * Oops report & Critical Bugs
[15:03] <matsubara>  * Operations report (mthaddon/herb/spm)
[15:03] <matsubara>  * DBA report (DBA contact)
[15:03] <matsubara> [TOPIC] * Next meeting
[15:03] <matsubara> so next meeting, same time next week?
[15:03] <Ursinha> +1
[15:03] <matsubara> how about removing that section and adding it again only when we won't be having the meeting?
[15:03] <rockstar> This part of the meeting should only be here if there's something that might make it not at the same time.
[15:03] <rockstar> :)
[15:03] <matsubara> :-)
[15:04] <flacoste> i'll need to send somebody
[15:04] <matsubara> [action] matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
[15:04] <flacoste> as I might not be able to attend (TL meeting)
[15:05] <bigjools> me too
[15:05] <flacoste> same for bigjools and sinzui
[15:05] <sinzui> me too
[15:05] <matsubara> same here, I'll be attending the TL meeting so Ursinha will likely run it
[15:05] <Ursinha> yes
[15:05] <henninge> so there is a reason to discuss the next meeting at each meeting?
[15:06] <henninge> -> planned attendance
[15:07] <matsubara> [action] TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week
[15:07] <matsubara> indeed henninge
[15:07] <matsubara> so, moving on
[15:07] <matsubara> [TOPIC] * Actions from last meeting
[15:07] <matsubara>  * matsubara and Ursinha to QA some of registry team items
[15:08] <matsubara> ok, we did that.
[15:08] <sinzui> Thanks very much
[15:08] <matsubara> so, if any of the teams need a hand testing stuff, talk to us!
[15:08] <matsubara> sinzui, my pleasure :-)
[15:09] <matsubara>  * Oops report & Critical Bugs
[15:09] <matsubara> Today oops report is about bugs 319173, 318894, OOPS-1116EC265, OOPS-1113EB6
[15:09] <matsubara> so the first one, which I'll move to -registry I need to try to reproduce on windows
[15:10] <flacoste> herb: can you check that buildbot-poll script is working? buildbot has blessed a lot of revisions that weren't pushed to stable
[15:10] <matsubara> Im working with the reporter to find out the cause and will add more info once I have it
[15:10] <sinzui> matsubara: I think we limit the upload to tar.gz, changelof, and .txt
[15:10] <matsubara> sinzui, who in your team should be assigned to fix that one?
[15:10] <herb> flacoste: will check after the meeting, yes
[15:10] <sinzui> matsubara: is it a bug?
[15:10] <matsubara> sinzui, it works on firefox for linux. it doesn't matter the file extension
[15:11] <matsubara> sinzui, the thing is, according to the reporter, it oopses when he tries to upload while using FF on windows
[15:11] <sinzui> matsubara: I'll give it to my favorite bug assassin--salgado
[15:11] <matsubara> sinzui, ok, once I have more info I'll coordinate with salgado. thanks sinzui
[15:12] <matsubara> the other two oopses are for lp-bzr team
[15:12] <matsubara> [action] matsubara to file bugs for OOPS-1116EC265, OOPS-1113EB6
[15:12] <matsubara> rockstar, can you help me out with those two oopses after the meeting? need to find a way to reproduce them
[15:12] <rockstar> matsubara, okay.
[15:13] <matsubara> flacoste, news about the -foundations one?
[15:13] <flacoste> matsubara: looking into it, but unless it is happening a lot, i don't see it as a high priority
[15:13] <matsubara> flacoste, barry's been bitten a lot by that one, IIRC
[15:14] <flacoste> that's what you get for using proprietary email applications!
[15:14] <matsubara> flacoste, shall I close it as invalid with that as the rationale? :-)
[15:15] <flacoste> i'll take care of it :-)
[15:15] <matsubara> flacoste, thanks!
[15:15] <matsubara> for critical bugs, we have 3. all of them fix committed.
[15:15] <matsubara> good job everyone!
[15:15] <matsubara> moving on
[15:15] <matsubara>  * Operations report (mthaddon/herb/spm)
[15:16] <herb> 2009-01-16 - Friday - We cherry picked r7586 to lpnet* to fix bug #315103
[15:16] <herb> 2009-01-20 - Tuesday - We cherry picked r7588 to lpnet* to fix bug #317241
[15:16] <herb> Bug #156453 and #118625 continue to be a problem requiring us to restart codebrowse at least daily.
[15:16] <herb> As noted by mthaddon and spm via email to the list, we've been seeing higher than normal DB load this week. We also experienced some issues with app servers taking too long to respond to requests (anywhere from 15-30 seconds). We suspect the two are related in some way but don't yet have any hard evidence to back this up.
[15:16] <Ursinha> hm, this explains the timeouts
[15:17] <matsubara> linkchecker was disabled last friday, but that didn't seem to make any difference on the staging DB load
[15:18] <herb> well. that's it from us unless there are questions.
[15:19] <matsubara> rockstar, how's the codebrowser bug going?
[15:19] <stub> The staging rebuild is the killer - it chews up an entire CPU and swaps important information out of RAM for its run, which is about 8 hours of the day IIRC.
[15:19] <rockstar> matsubara, we know no more this week than we did last week.
[15:19] <rockstar> jml and poolie are at the TL sprint early to work on it.
[15:19] <matsubara> rockstar, :-(
[15:19] <matsubara> that's great news
[15:20] <matsubara> stub, any way to optimize that?
[15:20] <stub> It already is optimized.
[15:20] <stub> We would have to start dropping features, which makes it less useful.
[15:21] <flacoste> or get a new box
[15:21] <flacoste> kiko is on it
[15:21] <flacoste> gpg: Erreur de CRC; E7341E - DC3E73
[15:21] <flacoste> gpg: aucune signature trouvée
[15:21] <flacoste> gpg: impossible de vérifier la signature.
[15:21] <flacoste> ignore those gpg thing
[15:22] <matsubara> hehe that's barry's broken signature :-)
[15:22] <matsubara> all right. thanks stub, flacoste, herb and rockstar
[15:22] <matsubara> moving on
[15:22] <matsubara> [TOPIC] * DBA report (DBA contact)
[15:22] <stub> Staging rebuilds are getting dangerously close to filling up their disk partition again. Avoiding this looming problem involves moving staging to new hardware, or bringing down the databases on Chokecherry while we repartition the disks.
[15:22] <stub> We have been getting some load spikes again, causing the losas to rewire the appservers to distribute db load a little more evenly. In the past, these have been bots request spamming. I don't think anyone has identified the trigger this time yet. My money is on a bot crawling an expensive part of the site, or the load of the staging rebuild causing trouble, or a combination of the two.
[15:22] <stub> I've suggested shutting down staging and the staging rebuild next time it happens to reduce the possible variables.
[15:22] <stub> DB patches for this cycle have all landed except for one of mine still being worked on (severing relationships between the future authentication replication set and the main db), and a late request from Salgado (caching the total download count on LibraryFileAlias, which I'll probably defer until next cycle rather than make a hasty decision).
[15:24] <matsubara> all right, questions for stub?
[15:26] <matsubara> thanks stub
[15:26] <matsubara> what do you think of officially make this meeting 30 min rather than 45?
[15:27] <Ursinha> matsubara, we in average use 40-45 mins
[15:27] <Ursinha> do you think it's needed?
[15:27] <Ursinha> to reduce the time, that is
[15:27] <matsubara> Ursinha, do we? I thought we used less than 30.
[15:27] <matsubara> but let's keep it 40-45 then
[15:27] <Ursinha> AFAIR
[15:28] <matsubara> anyway, thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:28] <matsubara> #endmeeting
[15:28] <stub> 20% slack time is good.
[15:28] <Ursinha> :)
[15:28] <Ursinha> thanks matsubara