=== salgado-afk is now known as salgado === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [15:00] #startmeeting [15:00] Meeting started at 09:00. The chair is matsubara. [15:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [15:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [15:00] [TOPIC] Roll Call [15:00] New Topic: Roll Call [15:00] me [15:00] me [15:00] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [15:00] me [15:00] me [15:00] me [15:01] sinzui, flacoste? [15:01] me [15:01] me [15:01] bigjools: ? [15:01] meeeeeeee [15:01] m,e [15:01] me [15:01] all right, everyone is here [15:01] [TOPIC] Agenda [15:01] New Topic: Agenda [15:01] * sinzui hates windows that steal focus [15:01] * Next meeting [15:01] * Actions from last meeting [15:01] * Oops report & Critical Bugs [15:01] * QA pending items [15:01] * Operations report (mthaddon/herb/spm) [15:01] * DBA report (DBA contact) [15:01] [TOPIC] * Next meeting [15:01] New Topic: * Next meeting [15:02] same time next week? [15:02] I'll be on vacation, so Ursinha will take over [15:02] * bigjools is out for 3 weeks [15:02] o/ [15:02] [action] matsubara to add himself and bigjools to the apologies list [15:02] ACTION received: matsubara to add himself and bigjools to the apologies list [15:03] bigjools: can you arrange for someone in soyuz to cover for you? [15:03] matsubara: yep [15:03] bigjools: thanks [15:03] [TOPIC] * Actions from last meeting [15:03] New Topic: * Actions from last meeting [15:03] * matsubara email stub asking for the DBA report [15:03] * matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324 [15:03] * rockstar to talk with code team about their pending QA items [15:03] * henninge to talk with translation team about their pending QA items [15:03] * Ursinha to talk with bigjools about one account on dogfood machine to do some testing [15:03] Error: Launchpad bug 2 could not be found [15:03] * rockstar to handle bug 271879 today (12-04) [15:03] * Ursinha to watch timeouts in the next days to see if they continue to happen this often [15:03] Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324 [15:03] * rockstar to manage having something reported on bug 118625 and bug 156453 [15:03] * Ursinha to watch for strange OOPSes on edge in the next days [15:03] Bug 271879 on http://launchpad.net/bugs/271879 is private [15:03] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:03] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 [15:03] oh yeah [15:03] I've opened a RT to get access to mawson [15:04] matsubara, first item is talked about everyday now. [15:04] I did both of mine and gmb added the query to the the LPPS page effectively hiding the problematic bug [15:04] matsubara, great [15:04] I've watched for the oopses and nothing that bizarre came up [15:05] matsubara, the fix for the timeout bug is approved, but we need to do some query testing. [15:05] thanks rockstar. Is the daily nagging helping QA testing? [15:05] matsubara, my team doesn't like nagging, but they're getting to it. [15:05] Ursinha: anything unusual about the strange OOPS and timeouts? [15:06] matsubara, nothing that's not reported and being taken care of [15:06] henninge: how's the daily nagging about pending QA items going? is it helping your team? [15:07] matsubara: Well, currently all items have been tested that could have been. [15:07] matsubara: danilo is at UDS right now and jtv doesn't have anything else to test. [15:07] great. thanks henninge [15:08] matsubara: we'll keep reminding ourselves ... [15:08] cool [15:08] moving on [15:08] [TOPIC] * Oops report & Critical Bugs [15:08] New Topic: * Oops report & Critical Bugs [15:08] okay [15:08] Ursinha: do you want me to do it? [15:08] all right, go ahead [15:09] matsubara, just saw sinzui was talking to salgado about bug 304113 [15:09] Launchpad bug 304113 in launchpad-registry "Milestone:+index page timing out" [High,Triaged] https://launchpad.net/bugs/304113 [15:09] oops, this wasn't for you matsubara, sorry :) [15:09] Ursinha: we will start working on it in a few hours [15:09] sinzui, great, thanks [15:10] henninge, can you ask someone on translations to set priority and importance on bug 302798, please? [15:10] Launchpad bug 302798 in rosetta "Timeout on +translate page" [Undecided,New] https://launchpad.net/bugs/302798 [15:11] Ursinha: yes [15:11] henninge, thank you :) [15:11] priority and importance ? [15:11] erm [15:11] we have only importance :-) [15:11] status and importance [15:11] status and importance, sorry :) [15:12] Ursinha: I knew what you meant ... ;) [15:12] anything else Ursinha ? [15:12] rockstar, you [15:12] Me?! [15:12] rockstar, do you know anything about bug 305367 [15:12] Bug 305367 on http://launchpad.net/bugs/305367 is private [15:12] ? [15:12] grrr [15:12] it's a critical one [15:12] the only one we have [15:12] it's really critical? [15:13] Ursinha, I don't think they are critical. [15:13] activereviews is for your own code. [15:13] rockstar, stub marked it critical [15:14] I'm working on the pages to generate a list of reviews you need to vote on. [15:14] rockstar, right. [15:14] I'm going to bring it down from critical, anh talk with stub. [15:15] I marked it critical because I couldn't see a work around and might lose things. Feel free to point me at a work around :) [15:15] rockstar, thank you very much [15:15] stub, no work around. [15:15] stub, I think your people should be contacting you to tell you about them. [15:15] I'm done after that [15:15] That should be the workaround right now. [15:16] rockstar: Not really an option with the db review windows - I'll use a wiki page next cycle. [15:16] stub, next cycle you'll have the page you're looking for. I'm working on it now. [15:17] But activereviews is for seeing the code you've proposed for review and its status. [15:17] Ok. [15:17] all right, moving on. [15:17] thanks everyone [15:17] thanks guys [15:17] [TOPIC] * QA pending items [15:17] New Topic: * QA pending items [15:18] right [15:18] so we have a lot of items to be tested [15:18] We have too many still, and the ones that are getting done aren't getting recorded. [15:18] :/ [15:19] why not? [15:19] rockstar, this is a problem [15:19] Ursinha, I'll talk with thumper about it. [15:19] rockstar, you'll lost yourselves with that [15:19] rockstar, okay. I'm on my late shift today, will talk to thumper as weel [15:19] *well [15:20] rockstar: what's the issue about recording what's been tested? === salgado is now known as salgado-lunch [15:20] matsubara, no idea. I know some of these things have been tested. [15:21] rockstar: right, so it's just a matter of updating the wiki page or something else? [15:21] matsubara, wiki page. [15:22] I see. if the process to update the test plan page is too troublesome we need to find a better way. suggestions welcome. [15:22] matsubara, also, this week has been abnormally busy with the ec2 stuff. It might be that recording it isn't at the front of their minds. [15:22] matsubara, I need to think about that. [15:22] fwiw there's quite a lot of talk about using launchpad itself to track qa. if anyone has good ideas please let them be known [15:23] rockstar, thanks. [15:23] intellectronica, indeed [15:23] intellectronica, yea, I was going to think about that. [15:23] The Ubuntu guys might be happy to have qa handled in Lanuchpad. [15:23] also, Launchpad. [15:23] so i hear, yes [15:24] any suggestions/ideas are very welcome [15:24] that's probably something that will happen in the future, until we get to the pojnt we can use LP to track QA, we should be updating the wiki page [15:24] of course [15:24] I know that's far from perfect but don't let that stop the QA. [15:25] one thing you could do is email Ursinha and I all items tested and we'll take care of updating the test plan [15:25] and I'm going to repeat myself again, but, please, ping us if we can do something [15:25] like that matsubara said [15:26] Ursinha: halp [15:26] until we find a better way to deal with this [15:26] thanks [15:26] [TOPIC] * Operations report (mthaddon/herb/spm) [15:26] New Topic: * Operations report (mthaddon/herb/spm) [15:26] bigjools, as soon as a have access to mawson, will be glad too. but if there's something I could do now, let me know [15:26] * 2008-12-10 - Cherry picked r7386 enabling replication to production and cherry picked r7419 to edge. [15:26] s/as a/as I/ [15:26] * Bugs #118625 and #156453 continue to be a problem despite a patch and the addition of the pyrex package. [15:26] * Had an issue with the branch scanner where a user's subscription to a particularly large branch was causing the branchscanner to hang. See bug #242807 [15:26] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:26] * Had to kill off a bunch of old processes on the code hosting server. This seems to be happening a bit more often than once a week now. See bug #260171 [15:26] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 [15:26] Launchpad bug 242807 in ubuntu "my ipod nano is not showing up when plugged in" [Low,Invalid] https://launchpad.net/bugs/242807 [15:27] Bug 260171 on http://launchpad.net/bugs/260171 is private [15:27] hmm [15:27] ipod nano? [15:27] hehe [15:27] lol [15:27] 242807 doesn't seem to correlate. [15:27] something tells me I typoed at least one of those. [15:27] should be bug #252807 [15:27] Bug 252807 on http://launchpad.net/bugs/252807 is private [15:29] herb, there are many folken at UDS right now trying to figure out why loggerhead sucks. [15:29] herb: btw, did you find out why staging is down? [15:29] It's disappointing that it's still having issues. [15:29] hmm it's not down anymore, so why staging was down? :-) [15:30] matsubara: the old lp process didn't exit and so the new process couldn't bind to the port after the restore. [15:30] herb, iirc it was the second time this week [15:30] herb: ok. thanks for chasing [15:30] the other time happened for the same reason? [15:30] Ursinha: 3rd or 4th time in the last 10 days. [15:30] Ursinha: different reasons the last couple of times. it had been hanging in the middle of startup. [15:30] hm, right. [15:31] herb, was it the librarian? [15:31] rockstar: running. but didn't really look at it in too much detail. [15:31] herb: if it happens again, could you file a bug about it please and let me know? [15:31] herb, I've noticed the librarian doesn't close cleanly on my dev machine. [15:32] matsubara: sure. I can't provide much detail on the problems I've seen over and over though. stracing the startup just shows a wait call. [15:32] :( [15:33] thanks herb [15:33] [TOPIC] * DBA report (DBA contact) [15:33] New Topic: * DBA report (DBA contact) [15:33] The losas altered the production appserver configs so they start talking to the production replica where possible. This is working well... possibly too well. [15:33] Most of the load is now on the slave, with the production box pretty much idling. We can push some of the [15:33] load back to the main box by reverting a few of the appserver configs, but we should probably leave it be until we have the reverse web proxy caching anonymous requests to see how this affects the load distribution. [15:33] I will be running the script to open Jaunty translations with Jeroen tomorrow. This should be the last time it is a heavy weight operation, as translation inheritence is scheduled to be implemented before the next opening. No ETAs on the run time completion yet - I don't fully trust them given the extra overhead we have now on inserts thanks to replication. [15:33] DB patches for this cycle are all either landed or rejected, except for the per language translation policies link with is blocked on discussions upstairs. [15:33] (^-^) [15:34] stub: did Celso's get accepted or rejected? [15:35] Which one? [15:35] the licensing meta-data one [15:35] Oh... licencing sourcepackagereleae. Rejected. Needs mucho discussion. [15:35] blah [15:36] all right, anything else for stub? [15:36] thanks stub [15:36] anything else before I close? [15:36] 5 [15:36] 4 [15:36] 3 [15:36] 2 [15:36] 1 [15:36] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [15:37] lol [15:37] #endmeeting [15:37] Meeting finished at 09:37. [15:37] thanks matsubara [15:37] thanks matsubara === salgado-lunch is now known as salgado === bac_uds is now known as bac === salgado is now known as salgado-afk