[15:00] <matsubara> #startmeeting
[15:00] <MootBot> Meeting started at 09:00. The chair is matsubara.
[15:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:00] <matsubara> [TOPIC] Roll Call
[15:00] <MootBot> New Topic:  Roll Call
[15:00] <henninge> me
[15:00] <intellectronica> me
[15:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[15:00] <herb> me
[15:00] <Ursinha> me
[15:00] <stub> me
[15:01] <matsubara> sinzui, flacoste?
[15:01] <rockstar> me
[15:01] <flacoste> me
[15:01] <matsubara> bigjools: ?
[15:01] <bigjools> meeeeeeee
[15:01] <sinzui> m,e
[15:01] <sinzui> me
[15:01] <matsubara> all right, everyone is here
[15:01] <matsubara> [TOPIC] Agenda
[15:01] <MootBot> New Topic:  Agenda
[15:01]  * sinzui hates windows that steal focus
[15:01] <matsubara>  * Next meeting
[15:01] <matsubara>  * Actions from last meeting
[15:01] <matsubara>  * Oops report & Critical Bugs
[15:01] <matsubara>  * QA pending items
[15:01] <matsubara>  * Operations report (mthaddon/herb/spm)
[15:01] <matsubara>  * DBA report (DBA contact)
[15:01] <matsubara> [TOPIC] * Next meeting
[15:01] <MootBot> New Topic:  * Next meeting
[15:02] <matsubara> same time next week?
[15:02] <matsubara> I'll be on vacation, so Ursinha will take over
[15:02]  * bigjools is out for 3 weeks
[15:02] <Ursinha> o/
[15:02] <matsubara> [action] matsubara to add himself and bigjools to the apologies list
[15:02] <MootBot> ACTION received:  matsubara to add himself and bigjools to the apologies list
[15:03] <matsubara> bigjools: can you arrange for someone in soyuz to cover for you?
[15:03] <bigjools> matsubara: yep
[15:03] <matsubara> bigjools: thanks
[15:03] <matsubara> [TOPIC] * Actions from last meeting
[15:03] <MootBot> New Topic:  * Actions from last meeting
[15:03] <matsubara>  * matsubara email stub asking for the DBA report
[15:03] <matsubara>  * matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324
[15:03] <matsubara>  * rockstar to talk with code team about their pending QA items
[15:03] <matsubara>  * henninge  to talk with translation team about their pending QA items
[15:03] <matsubara>  * Ursinha to talk with bigjools about one account on dogfood machine to do some testing
[15:03] <matsubara>  * rockstar to handle bug 271879 today (12-04)
[15:03] <matsubara>  * Ursinha to watch timeouts in the next days to see if they continue to happen this often
[15:03] <matsubara>  * rockstar to manage having something reported on bug 118625 and bug 156453
[15:03] <matsubara>  * Ursinha to watch for strange OOPSes on edge in the next days
[15:03] <Ursinha> oh yeah
[15:03] <Ursinha> I've opened a RT to get access to mawson
[15:04] <rockstar> matsubara, first item is talked about everyday now.
[15:04] <matsubara> I did both of mine and gmb added the query to the the LPPS page effectively hiding the problematic bug
[15:04] <Ursinha> matsubara, great
[15:04] <Ursinha> I've watched for the oopses and nothing that bizarre came up
[15:05] <rockstar> matsubara, the fix for the timeout bug is approved, but we need to do some query testing.
[15:05] <matsubara> thanks rockstar. Is the daily nagging helping QA testing?
[15:05] <rockstar> matsubara, my team doesn't like nagging, but they're getting to it.
[15:05] <matsubara> Ursinha: anything unusual about the strange OOPS and timeouts?
[15:06] <Ursinha> matsubara, nothing that's not reported and being taken care of
[15:06] <matsubara> henninge: how's the daily nagging about pending QA items going? is it helping your team?
[15:07] <henninge> matsubara: Well, currently all items have been tested that could have been.
[15:07] <henninge> matsubara: danilo is at UDS right now and jtv doesn't have anything else to test.
[15:07] <matsubara> great. thanks henninge
[15:08] <henninge> matsubara: we'll keep reminding ourselves ...
[15:08] <matsubara> cool
[15:08] <matsubara> moving on
[15:08] <matsubara> [TOPIC] * Oops report & Critical Bugs
[15:08] <MootBot> New Topic:  * Oops report & Critical Bugs
[15:08] <Ursinha> okay
[15:08] <matsubara> Ursinha: do you want me to do it?
[15:08] <matsubara> all right, go ahead
[15:09] <Ursinha> matsubara, just saw sinzui was talking to salgado about bug 304113
[15:09] <Ursinha> oops, this wasn't for you matsubara, sorry :)
[15:09] <sinzui> Ursinha: we will start working on it in a few hours
[15:09] <Ursinha> sinzui, great, thanks
[15:10] <Ursinha> henninge, can you ask someone on translations to set priority and importance on bug 302798, please?
[15:11] <henninge> Ursinha: yes
[15:11] <Ursinha> henninge, thank you :)
[15:11] <matsubara> priority and importance ?
[15:11] <Ursinha> erm
[15:11] <matsubara> we have only importance :-)
[15:11] <henninge> status and importance
[15:11] <Ursinha> status and importance, sorry :)
[15:12] <henninge> Ursinha: I knew what you meant ... ;)
[15:12] <matsubara> anything else Ursinha ?
[15:12] <Ursinha> rockstar, you
[15:12] <rockstar> Me?!
[15:12] <Ursinha> rockstar, do you know anything about bug 305367
[15:12] <Ursinha> ?
[15:12] <Ursinha> grrr
[15:12] <Ursinha> it's a critical one
[15:12] <Ursinha> the only one we have
[15:12] <Ursinha> it's really critical?
[15:13] <rockstar> Ursinha, I don't think they are critical.
[15:13] <rockstar> activereviews is for your own code.
[15:13] <Ursinha> rockstar, stub marked it critical
[15:14] <rockstar> I'm working on the pages to generate a list of reviews you need to vote on.
[15:14] <Ursinha> rockstar, right.
[15:14] <rockstar> I'm going to bring it down from critical, anh talk with stub.
[15:15] <stub> I marked it critical because I couldn't see a work around and might lose things. Feel free to point me at a work around :)
[15:15] <Ursinha> rockstar, thank you very much
[15:15] <rockstar> stub, no work around.
[15:15] <rockstar> stub, I think your people should be contacting you to tell you about them.
[15:15] <Ursinha> I'm done after that
[15:15] <rockstar> That should be the workaround right now.
[15:16] <stub> rockstar: Not really an option with the db review windows - I'll use a wiki page next cycle.
[15:16] <rockstar> stub, next cycle you'll have the page you're looking for.  I'm working on it now.
[15:17] <rockstar> But activereviews is for seeing the code you've proposed for review and its status.
[15:17] <stub> Ok.
[15:17] <matsubara> all right, moving on.
[15:17] <matsubara> thanks everyone
[15:17] <Ursinha> thanks guys
[15:17] <matsubara> [TOPIC] * QA pending items
[15:17] <MootBot> New Topic:  * QA pending items
[15:18] <Ursinha> right
[15:18] <Ursinha> so we have a lot of items to be tested
[15:18] <rockstar> We have too many still, and the ones that are getting done aren't getting recorded.
[15:18] <rockstar> :/
[15:19] <matsubara> why not?
[15:19] <Ursinha> rockstar, this is a problem
[15:19] <rockstar> Ursinha, I'll talk with thumper about it.
[15:19] <Ursinha> rockstar, you'll lost yourselves with that
[15:19] <Ursinha> rockstar, okay. I'm on my late shift today, will talk to thumper as weel
[15:19] <Ursinha> *well
[15:20] <matsubara> rockstar: what's the issue about recording what's been tested?
[15:20] <rockstar> matsubara, no idea.  I know some of these things have been tested.
[15:21] <matsubara> rockstar: right, so it's just a matter of updating the wiki page or something else?
[15:21] <rockstar> matsubara, wiki page.
[15:22] <matsubara> I see. if the process to update the test plan page is too troublesome we need to find a better way. suggestions welcome.
[15:22] <rockstar> matsubara, also, this week has been abnormally busy with the ec2 stuff.  It might be that recording it isn't at the front of their minds.
[15:22] <rockstar> matsubara, I need to think about that.
[15:22] <intellectronica> fwiw there's quite a lot of talk about using launchpad itself to track qa. if anyone has good ideas please let them be known
[15:23] <Ursinha> rockstar, thanks.
[15:23] <Ursinha> intellectronica, indeed
[15:23] <rockstar> intellectronica, yea, I was going to think about that.
[15:23] <rockstar> The Ubuntu guys might be happy to have qa handled in Lanuchpad.
[15:23] <rockstar> also, Launchpad.
[15:23] <intellectronica> so i hear, yes
[15:24] <Ursinha> any suggestions/ideas are very welcome
[15:24] <matsubara> that's probably something that will happen in the future, until we get to the pojnt we can use LP to track QA, we should be updating the wiki page
[15:24] <intellectronica> of course
[15:24] <matsubara> I know that's far from perfect but don't let that stop the QA.
[15:25] <matsubara> one thing you could do is email Ursinha and I all items tested and we'll take care of updating the test plan
[15:25] <Ursinha> and I'm going to repeat myself again, but, please, ping us if we can do something
[15:25] <Ursinha> like that matsubara said
[15:26] <bigjools> Ursinha: halp
[15:26] <matsubara> until we find a better way to deal with this
[15:26] <matsubara> thanks
[15:26] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[15:26] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[15:26] <Ursinha> bigjools, as soon as a have access to mawson, will be glad too. but if there's something I could do now, let me know
[15:26] <herb> * 2008-12-10 - Cherry picked r7386 enabling replication to production and cherry picked r7419 to edge.
[15:26] <Ursinha> s/as a/as I/
[15:26] <herb> * Bugs #118625 and #156453 continue to be a problem despite a patch and the addition of the pyrex package.
[15:26] <herb> * Had an issue with the branch scanner where a user's subscription to a particularly large branch was causing the branchscanner to hang. See bug #242807
[15:26] <herb> * Had to kill off a bunch of old processes on the code hosting server. This seems to be happening a bit more often than once a week now. See bug #260171
[15:27] <herb> hmm
[15:27] <matsubara> ipod nano?
[15:27] <matsubara> hehe
[15:27] <Ursinha> lol
[15:27] <rockstar> 242807 doesn't seem to correlate.
[15:27] <herb> something tells me I typoed at least one of those.
[15:27] <herb> should be bug #252807
[15:29] <rockstar> herb, there are many folken at UDS right now trying to figure out why loggerhead sucks.
[15:29] <matsubara> herb: btw, did you find out why staging is down?
[15:29] <rockstar> It's disappointing that it's still having issues.
[15:29] <matsubara> hmm it's not down anymore, so why staging was down? :-)
[15:30] <herb> matsubara: the old lp process didn't exit and so the new process couldn't bind to the port after the restore.
[15:30] <Ursinha> herb, iirc it was the second time this week
[15:30] <matsubara> herb: ok. thanks for chasing
[15:30] <Ursinha> the other time happened for the same reason?
[15:30] <herb> Ursinha: 3rd or 4th time in the last 10 days.
[15:30] <herb> Ursinha: different reasons the last couple of times. it had been hanging in the middle of startup.
[15:30] <Ursinha> hm, right.
[15:31] <rockstar> herb, was it the librarian?
[15:31] <herb> rockstar: running.  but didn't really look at it in too much detail.
[15:31] <matsubara> herb: if it happens again, could you file a bug about it please and let me know?
[15:31] <rockstar> herb, I've noticed the librarian doesn't close cleanly on my dev machine.
[15:32] <herb> matsubara: sure. I can't provide much detail on the problems I've seen over and over though.  stracing the startup just shows a wait call.
[15:32] <herb> :(
[15:33] <matsubara> thanks herb
[15:33] <matsubara> [TOPIC] * DBA report (DBA contact)
[15:33] <MootBot> New Topic:  * DBA report (DBA contact)
[15:33] <stub> The losas altered the production appserver configs so they start talking to the production replica where possible. This is working well... possibly too well.
[15:33] <stub> Most of the load is now on the slave, with the production box pretty much idling. We can push some of the
[15:33] <stub> load back to the main box by reverting a few of the appserver configs, but we should probably leave it be until we have the reverse web proxy caching anonymous requests to see how this affects the load distribution.
[15:33] <stub> I will be running the script to open Jaunty translations with Jeroen tomorrow. This should be the last time it is a heavy weight operation, as translation inheritence is scheduled to be implemented before the next opening. No ETAs on the run time completion yet - I don't fully trust them given the extra overhead we have now on inserts thanks to replication.
[15:33] <stub> DB patches for this cycle are all either landed or rejected, except for the per language translation policies link with is blocked on discussions upstairs.
[15:33] <stub> (^-^)
[15:34] <bigjools> stub: did Celso's get accepted or rejected?
[15:35] <stub> Which one?
[15:35] <bigjools> the licensing meta-data one
[15:35] <stub> Oh... licencing sourcepackagereleae. Rejected. Needs mucho discussion.
[15:35] <bigjools> blah
[15:36] <matsubara> all right, anything else for stub?
[15:36] <matsubara> thanks stub
[15:36] <matsubara> anything else before I close?
[15:36] <matsubara> 5
[15:36] <matsubara> 4
[15:36] <matsubara> 3
[15:36] <matsubara> 2
[15:36] <matsubara> 1
[15:36] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:37] <Ursinha> lol
[15:37] <matsubara> #endmeeting
[15:37] <MootBot> Meeting finished at 09:37.
[15:37] <Ursinha> thanks matsubara
[15:37] <intellectronica> thanks matsubara