=== mrevell-afk is now known as mrevell
=== salgado-afk is now known as salgado
=== mrevell is now known as mrevell-lunch
=== mrevell-lunch is now known as mrevell
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.15:00
matsubara[TOPIC] Roll Call15:00
matsubaraso who's here today?15:00
sinzuiI'm still here15:00
Ursinhaeveryone except the bot15:00
rockstarme too15:00
matsubaraflacoste, bigjools: ?15:01
matsubaraall right. everyone is here15:02
matsubara[TOPIC] Agenda15:02
matsubara * Next meeting15:03
matsubara * Actions from last meeting15:03
matsubara * Oops report & Critical Bugs15:03
matsubara * Operations report (mthaddon/herb/spm)15:03
matsubara * DBA report (DBA contact)15:03
matsubara[TOPIC] * Next meeting15:03
matsubaraso next meeting, same time next week?15:03
matsubarahow about removing that section and adding it again only when we won't be having the meeting?15:03
rockstarThis part of the meeting should only be here if there's something that might make it not at the same time.15:03
flacostei'll need to send somebody15:04
matsubara[action] matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time15:04
flacosteas I might not be able to attend (TL meeting)15:04
bigjoolsme too15:05
flacostesame for bigjools and sinzui15:05
sinzuime too15:05
matsubarasame here, I'll be attending the TL meeting so Ursinha will likely run it15:05
henningeso there is a reason to discuss the next meeting at each meeting?15:05
henninge-> planned attendance15:06
matsubara[action] TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week15:07
matsubaraindeed henninge15:07
matsubaraso, moving on15:07
matsubara[TOPIC] * Actions from last meeting15:07
matsubara * matsubara and Ursinha to QA some of registry team items15:07
matsubaraok, we did that.15:08
sinzuiThanks very much15:08
matsubaraso, if any of the teams need a hand testing stuff, talk to us!15:08
matsubarasinzui, my pleasure :-)15:08
matsubara * Oops report & Critical Bugs15:09
matsubaraToday oops report is about bugs 319173, 318894, OOPS-1116EC265, OOPS-1113EB615:09
ubottuLaunchpad bug 319173 in launchpad "zip files cannot be uploaded to a release" [Undecided,Incomplete] https://launchpad.net/bugs/31917315:09
ubottuLaunchpad bug 318894 in launchpad-foundations "oops when responding to merge proposal review via email" [Undecided,New] https://launchpad.net/bugs/31889415:09
matsubaraso the first one, which I'll move to -registry I need to try to reproduce on windows15:09
flacosteherb: can you check that buildbot-poll script is working? buildbot has blessed a lot of revisions that weren't pushed to stable15:10
matsubaraIm working with the reporter to find out the cause and will add more info once I have it15:10
sinzuimatsubara: I think we limit the upload to tar.gz, changelof, and .txt15:10
matsubarasinzui, who in your team should be assigned to fix that one?15:10
herbflacoste: will check after the meeting, yes15:10
sinzuimatsubara: is it a bug?15:10
matsubarasinzui, it works on firefox for linux. it doesn't matter the file extension15:10
matsubarasinzui, the thing is, according to the reporter, it oopses when he tries to upload while using FF on windows15:11
sinzuimatsubara: I'll give it to my favorite bug assassin--salgado15:11
matsubarasinzui, ok, once I have more info I'll coordinate with salgado. thanks sinzui15:11
matsubarathe other two oopses are for lp-bzr team15:12
matsubara[action] matsubara to file bugs for OOPS-1116EC265, OOPS-1113EB615:12
matsubararockstar, can you help me out with those two oopses after the meeting? need to find a way to reproduce them15:12
rockstarmatsubara, okay.15:12
matsubaraflacoste, news about the -foundations one?15:13
flacostematsubara: looking into it, but unless it is happening a lot, i don't see it as a high priority15:13
matsubaraflacoste, barry's been bitten a lot by that one, IIRC15:13
flacostethat's what you get for using proprietary email applications!15:14
matsubaraflacoste, shall I close it as invalid with that as the rationale? :-)15:14
flacostei'll take care of it :-)15:15
matsubaraflacoste, thanks!15:15
matsubarafor critical bugs, we have 3. all of them fix committed.15:15
matsubaragood job everyone!15:15
matsubaramoving on15:15
matsubara * Operations report (mthaddon/herb/spm)15:15
herb2009-01-16 - Friday - We cherry picked r7586 to lpnet* to fix bug #31510315:16
ubottuLaunchpad bug 315103 in malone "When consuming a token, Launchpad tries to update already-consumed tokens" [High,Fix committed] https://launchpad.net/bugs/31510315:16
herb2009-01-20 - Tuesday - We cherry picked r7588 to lpnet* to fix bug #31724115:16
ubottuLaunchpad bug 317241 in launchpad-foundations "Creating a launchpad account in the process of an OpenID authentication returns a failure to the RP" [Critical,Fix committed] https://launchpad.net/bugs/31724115:16
herbBug #156453 and #118625 continue to be a problem requiring us to restart codebrowse at least daily.15:16
ubottuLaunchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/15645315:16
herbAs noted by mthaddon and spm via email to the list, we've been seeing higher than normal DB load this week. We also experienced some issues with app servers taking too long to respond to requests (anywhere from 15-30 seconds). We suspect the two are related in some way but don't yet have any hard evidence to back this up.15:16
ubottuLaunchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/11862515:16
Ursinhahm, this explains the timeouts15:16
matsubaralinkchecker was disabled last friday, but that didn't seem to make any difference on the staging DB load15:17
herbwell. that's it from us unless there are questions.15:18
matsubararockstar, how's the codebrowser bug going?15:19
stubThe staging rebuild is the killer - it chews up an entire CPU and swaps important information out of RAM for its run, which is about 8 hours of the day IIRC.15:19
rockstarmatsubara, we know no more this week than we did last week.15:19
rockstarjml and poolie are at the TL sprint early to work on it.15:19
matsubararockstar, :-(15:19
matsubarathat's great news15:19
matsubarastub, any way to optimize that?15:20
stubIt already is optimized.15:20
stubWe would have to start dropping features, which makes it less useful.15:20
flacosteor get a new box15:21
flacostekiko is on it15:21
flacostegpg: Erreur de CRC; E7341E - DC3E7315:21
flacostegpg: aucune signature trouvée15:21
flacostegpg: impossible de vérifier la signature.15:21
flacosteignore those gpg thing15:21
matsubarahehe that's barry's broken signature :-)15:22
matsubaraall right. thanks stub, flacoste, herb and rockstar15:22
matsubaramoving on15:22
matsubara[TOPIC] * DBA report (DBA contact)15:22
stubStaging rebuilds are getting dangerously close to filling up their disk partition again. Avoiding this looming problem involves moving staging to new hardware, or bringing down the databases on Chokecherry while we repartition the disks.15:22
stubWe have been getting some load spikes again, causing the losas to rewire the appservers to distribute db load a little more evenly. In the past, these have been bots request spamming. I don't think anyone has identified the trigger this time yet. My money is on a bot crawling an expensive part of the site, or the load of the staging rebuild causing trouble, or a combination of the two.15:22
stubI've suggested shutting down staging and the staging rebuild next time it happens to reduce the possible variables.15:22
stubDB patches for this cycle have all landed except for one of mine still being worked on (severing relationships between the future authentication replication set and the main db), and a late request from Salgado (caching the total download count on LibraryFileAlias, which I'll probably defer until next cycle rather than make a hasty decision).15:22
matsubaraall right, questions for stub?15:24
matsubarathanks stub15:26
matsubarawhat do you think of officially make this meeting 30 min rather than 45?15:26
Ursinhamatsubara, we in average use 40-45 mins15:27
Ursinhado you think it's needed?15:27
Ursinhato reduce the time, that is15:27
matsubaraUrsinha, do we? I thought we used less than 30.15:27
matsubarabut let's keep it 40-45 then15:27
matsubaraanyway, thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.15:28
stub20% slack time is good.15:28
Ursinhathanks matsubara15:28
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
=== mrevell is now known as mrevell-afk
=== salgado is now known as salgado-afk
=== Ursinha is now known as Ursinha-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!