[14:59] <Ursinha> that time of the week is coming
[14:59] <intellectronica> it's weekend already?
[15:00] <Ursinha> intellectronica, almost :)
[15:00] <Ursinha> it's show time
[15:00] <intellectronica> maaan
[15:00] <Ursinha> #startmeeting
[15:00] <MootBot> Meeting started at 09:00. The chair is Ursinha.
[15:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:00] <sinzui> me
[15:00] <rockstar> me
[15:00] <Ursinha> haha
[15:01] <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:01] <Ursinha> [TOPIC] Roll Call
[15:01] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[15:01] <MootBot> New Topic:  Roll Call
[15:01] <Ursinha> now you can say mee
[15:01] <Ursinha> me
[15:01] <rockstar> me (again)
[15:01] <sinzui> me
[15:01] <intellectronica> meaux
[15:01] <Ursinha> erm
[15:01] <flacoste> me
[15:01] <bigjools> me
[15:01] <Ursinha> who's missing
[15:02] <Ursinha> a losa?
[15:02] <intellectronica> stub?
[15:02] <Ursinha> herb, ping
[15:02] <matsubara_> I'm here
[15:02] <herb> me
[15:02] <matsubara_> me
[15:02] <Ursinha> stub is not here, it seems
[15:02] <Ursinha> flacoste, do you know anything about him?
[15:02] <flacoste> he's on leave
[15:03] <Ursinha> flacoste, oh, okay
[15:03] <intellectronica> oh, so what's with db schema reviews?
[15:03] <matsubara_> [action] email stub asking for the DBA report
[15:03] <Ursinha> [action] email stub asking for the DBA report
[15:03] <MootBot> ACTION received:  email stub asking for the DBA report
[15:03] <rockstar> matsubara_, you have no Priesthood with Mootbot!
[15:03] <Ursinha> well, flacoste, can you be stub's ears today?
[15:03] <matsubara_> yeah, not today :-)
[15:03] <Ursinha> eyes, in case
[15:04] <Ursinha> well
[15:04] <Ursinha> [TOPIC] Agenda
[15:04] <Ursinha>  * Next meeting
[15:04] <Ursinha>  * Actions from last meeting
[15:04] <Ursinha>  * Oops report & Critical Bugs
[15:04] <Ursinha>  * QA pending items
[15:04] <Ursinha>  * Operations report (mthaddon/herb/spm)
[15:04] <MootBot> New Topic:  Agenda
[15:04] <Ursinha>  * DBA report (DBA contact)
[15:04] <Ursinha> [TOPIC] * Next meeting
[15:04] <MootBot> New Topic:  * Next meeting
[15:04] <Ursinha> okay for you?
[15:04] <Ursinha> same time and yada yada?
[15:04] <matsubara_> fine by me
[15:04] <Ursinha> hi henninge :)
[15:04] <henninge> me!
[15:04] <matsubara_> hi henninge
[15:05] <Ursinha> cool
[15:05] <flacoste> Ursinha: yes, i'll proxy stuff to stub
[15:05] <intellectronica> some ppl may be in UDS? (not me)
[15:05] <Ursinha> so that's fine
[15:05] <Ursinha> flacoste, thanks
[15:05] <henninge> hi!
[15:05] <matsubara_> ok, any of the QA contacts will be attending UDS?
[15:05] <flacoste> not me
[15:05] <flacoste> not stub
[15:05] <bigjools> nope
[15:05] <henninge> not me, but the rest of my team
[15:06] <rockstar> no
[15:06] <Ursinha> henninge, but you're the QA contact, so whatever
[15:06] <Ursinha> :)
[15:06] <henninge> :)
[15:06] <Ursinha> ok, fine
[15:06] <matsubara_> sinzui: ?
[15:06] <sinzui> No I will not
[15:06] <Ursinha> [TOPIC] * Actions from last meeting
[15:06] <MootBot> New Topic:  * Actions from last meeting
[15:06] <matsubara_> ok, so same time next week
[15:06] <Ursinha> * gmb to add sql to hide bug 297388 as a fix for bug 300324
[15:06] <Ursinha> * abentley to talk to code team about bug 301595
[15:06] <Ursinha> * matsubara email losas to request operations report
[15:06] <Ursinha> * matsubara email stub to request dba report
[15:06] <Ursinha> uh-oh
[15:06] <Ursinha> damn bug
[15:07] <matsubara_> I did my two items and losas/stub emailed their reports to the list
[15:07] <Ursinha> matsubara_, nice, thanks
[15:07] <Ursinha> abentley must have talked with code people, that bug is fix committed
[15:07] <Ursinha> and intellectronica, do you know about bug 300324?
[15:07] <Ursinha> it's still triaged
[15:07] <intellectronica> i don't
[15:08]  * intellectronica looking
[15:08] <Ursinha> well
[15:08] <intellectronica> the comment says it all
[15:08] <matsubara_> Ursinha, intellectronica: I'll chase that one. it's a simple SQL that needs approval to hide #297388
[15:09] <intellectronica> these are quite exceptional, so for now we're happy to deal with them on an individual basis
[15:09] <Ursinha> but it will be fixed, that's the point :)
[15:09] <Ursinha> thanks matsubara_
[15:09] <matsubara_> [action] matsubara to talk with gmb and add sql to hide bug 297388 as a fix for bug 300324
[15:09] <intellectronica> we already have the db infrastructure for dealing with this in a better way, but it's not the highest priority to finish that
[15:09] <Ursinha> [action] matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324
[15:09] <MootBot> ACTION received:  matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324
[15:09] <Ursinha> okay
[15:10] <Ursinha> [TOPIC] * QA pending items
[15:10] <MootBot> New Topic:  * QA pending items
[15:10] <Ursinha> well, I can see that there are lots of items to be tested on your testplans
[15:10] <Ursinha> are you having problems on testing them?
[15:10] <Ursinha> I
[15:11] <Ursinha> my fear is that we start to test it too late
[15:11] <matsubara_> [action] ursinha to add QA pending items topic to the MeetingPage agenda
[15:11] <Ursinha> matsubara_, it's done already
[15:11] <intellectronica> some of the landings on our team are not really testable (some of the hwdb stuff). i'll see that the rest are tested promptly
[15:11] <matsubara_> Ursinha: all right. thanks
[15:11] <sinzui> Ursinha: Registry has been finding lots of bugs, so some of the items we are working on will not move to OK until we close the blocking bugs
[15:11] <Ursinha> so, rockstar, can you ask your team what can be done about that, please?
[15:12] <Ursinha> sinzui, that's okay, you have lots of tested items
[15:12] <rockstar> Ursinha, yup, making a note now.
[15:12] <Ursinha> rockstar, thanks
[15:12] <Ursinha> [action] rockstar to talk with code team about their pending QA items
[15:12] <MootBot> ACTION received:  rockstar to talk with code team about their pending QA items
[15:12] <Ursinha> okay
[15:12] <Ursinha> henninge, can you do the same too, please?
[15:13] <henninge> will do
[15:13] <Ursinha> thanks henninge
[15:13] <Ursinha> [action] henninge  to talk with translation team about their pending QA items
[15:13] <MootBot> ACTION received:  henninge  to talk with translation team about their pending QA items
[15:13] <Ursinha> bigjools, what about you?
[15:14] <Ursinha> souyz has a lot of items
[15:14] <bigjools> we've been QA-ing fine
[15:14] <bigjools> but they're arriving thick and fast again
[15:14]  * bigjools will prod people
[15:14] <Ursinha> bigjools, it's true
[15:14] <Ursinha> how are you doing QA?
[15:15] <Ursinha> at the end of the week, on demand or what?
[15:15] <bigjools> when time allows after the branch lands
[15:15] <Ursinha> flacoste, sinzui, henninge, rockstar ^
[15:15] <Ursinha> intellectronica, ^
[15:15] <flacoste> like bigjools
[15:15] <Ursinha> bigjools, flacoste okay
[15:15] <sinzui> Ursinha: we are QAing a day after each landing
[15:15] <flacoste> ideally the day after it lands
[15:15] <henninge> on demand
[15:15] <intellectronica> we do it when we can
[15:15] <Ursinha> sinzui, awesome
[15:15] <flacoste> but unless i prompt people it doesn't happen
[15:16] <intellectronica> and try to remind each other that it's important to do
[15:16] <bigjools> flacoste: right, same here
[15:16] <flacoste> but this is really because we don't land that many QA-able changes
[15:16] <Ursinha> bigjools, flacoste, this is not quite that good
[15:16] <flacoste> most of work recently was on non front-end stuff
[15:16] <Ursinha> flacoste, I see you have only two items in there
[15:16] <bigjools> Ursinha: it has to be balanced with other important stuff
[15:16] <Ursinha> so I understand
[15:16] <intellectronica> sinzui's sounds like the sanest. i will suggest that we follow the same rule
[15:17] <bigjools> so it may go 2/3 days and then we'll clear them
[15:17] <Ursinha> bigjools, I understand, but as it's important too, I'd like to ask you please, please, to ask us help
[15:17] <Ursinha> if we can test it for you, we'll do
[15:17] <bigjools> Ursinha: ok!  please test all our items ;)
[15:17] <sinzui> intellectronica: of course that assume staging *is* updating.
[15:17] <Ursinha> bigjools, ha ha
[15:17] <intellectronica> sinzui: if we all adopt this rule, there will be many people on the case. sometimes staging doesn't update on purpose, but in most cases it just needs someone to care enough
[15:17] <Ursinha> sinzui, yes, this is a major problem
[15:17] <Ursinha> in my opinion
[15:17] <bigjools> Ursinha: you might need an account on the dogfood machine to do some of them
[15:18] <Ursinha> bigjools, please, can we talk a bit about that later?
[15:18] <bigjools> Ursinha: sure
[15:18] <Ursinha> [action] Ursinha to talk with bigjools about one account on dogfood machine to do some testing
[15:18] <MootBot> ACTION received:  Ursinha to talk with bigjools about one account on dogfood machine to do some testing
[15:18] <Ursinha> great
[15:19] <Ursinha> intellectronica, we're always watching if staging is updating
[15:19] <Ursinha> recently, at lease
[15:19] <Ursinha> least
[15:19] <intellectronica> cool
[15:19] <Ursinha> okay, so if there's something we can do, please poke us
[15:20] <Ursinha> so my beloved section of this meeting
[15:20] <Ursinha> [TOPIC] * Oops report & Critical Bugs
[15:20] <MootBot> New Topic:  * Oops report & Critical Bugs
[15:20] <Ursinha> well
[15:20] <Ursinha> I was going to mention the bug matsubara will look, that is the bug missing comment 0 one, ok
[15:21] <Ursinha> bug 271879
[15:21]  * Ursinha looks at rockstar 
[15:21] <Ursinha> crap
[15:21] <Ursinha> rockstar, that is a timeout in +code-index page
[15:21] <Ursinha> still happening, yesterday we have a lot of timeout oopses, btw
[15:21] <Ursinha> not only in +code-index
[15:22] <rockstar> Ursinha, that might be on my list of things to do today, if the network will let me see the bug.
[15:22] <Ursinha> flacoste, do you think it can be related with the DB load problem?
[15:22] <Ursinha> rockstar, okay, thanks
[15:22] <Ursinha> [action] rockstar to handle bug 271879 today (12-04)
[15:22] <MootBot> ACTION received:  rockstar to handle bug 271879 today (12-04)
[15:22] <flacoste> the +code-index?
[15:22] <flacoste> it might
[15:22] <flacoste> but if it continues, probably not
[15:22] <Ursinha> flacoste, all the timeouts
[15:23] <rockstar> Yea, I got that oops yesterday, opened my big mouth, and got work to do as a reward.  :)
[15:23] <Ursinha> I mean, we had a lot of timeouts yesterday
[15:23] <flacoste> yeah, DB load increase the likelyhood of any timeout to happen
[15:23] <rockstar> Regardless, the query needs love.
[15:23] <Ursinha> for instance, +bugs-index page had 998 timeouts
[15:24] <Ursinha> on lpnet
[15:24] <Ursinha> will keep one eye on it
[15:24] <rockstar> Damn
[15:24] <Ursinha> well, will keep both, better
[15:24] <Ursinha> [action] Ursinha to watch timeouts in the next days to see if they continue to happen this often
[15:24] <MootBot> ACTION received:  Ursinha to watch timeouts in the next days to see if they continue to happen this often
[15:24] <Ursinha> cool
[15:24] <Ursinha> so this is all in the oops land
[15:25] <Ursinha> matsubara is faster copying and pasting stuff around
[15:25] <Ursinha> :)
[15:25] <Ursinha> so there's only that critical bug
[15:26] <Ursinha> bug 269538, that will never be closed
[15:26] <Ursinha> matsubara, do you remember that from last meeting?
[15:26] <matsubara> Ursinha: gmb is chasing that with max
[15:26] <Ursinha> matsubara, okay
[15:26] <Ursinha> thanks
[15:26] <Ursinha> so the section is over
[15:26] <Ursinha> moving on
[15:27] <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
[15:27] <herb> Quiet week except for yesterday when the abusive user was slowing the system down.
[15:27] <Ursinha> hi herb
[15:27] <herb> - We're still rebooting codebrowse often as a result of bugs #156453 and #118625. I know it's being worked on, but there hasn't been an update to the bug reports in about a month.
[15:27] <herb> - Bug #301720 continues to be a problem requiring almost daily intervention from the LOSAs. We haven't seen an update on this one in the past week.
[15:27] <herb> hi
[15:27] <Ursinha> rockstar, beuno and you are looking at the loggerhead bug, right?
[15:27] <rockstar> Ursinha, yes, it's being discussed on list right now.
[15:28] <Ursinha> bug 301720
[15:28] <rockstar> There will probably be some work at UDS.
[15:28] <Ursinha> rockstar, nice to hear
[15:28] <Ursinha> about bug 301720, anybody?
[15:29] <herb> I think sinzui was working on it.
[15:29] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[15:29] <Ursinha> sinzui, can you do an update on that bug, please?
[15:29] <Ursinha> MootBot, you are slow, man
[15:29] <sinzui> Ursinha: it is in PQM now
[15:30] <Ursinha> sinzui, awesome :)
[15:30] <Ursinha> okay
[15:30] <Ursinha> anything else herb?
[15:30] <herb> nothing from me. thanks.
[15:30] <Ursinha> okay, rockstar, can you manage having something reported on those bugs, pretty please?
[15:31] <rockstar> Ursinha, doing that now.
[15:31] <Ursinha> rockstar, thanks!
[15:31] <Ursinha> [action] rockstar to manage having something reported on bug 118625 and bug 156453
[15:31] <MootBot> ACTION received:  rockstar to manage having something reported on bug 118625 and bug 156453
[15:31] <Ursinha> okay
[15:31] <Ursinha> moving on again
[15:31] <Ursinha> [TOPIC] * DBA report (DBA contact)
[15:31] <MootBot> New Topic:  * DBA report (DBA contact)
[15:31] <Ursinha> stub, that is
[15:31] <Ursinha> that's not here
[15:32] <Ursinha> and matsubara will mail him about his report
[15:32] <Ursinha> so
[15:32] <Ursinha> anyone have something to declare before I end all this?
[15:32] <flacoste> i have
[15:32] <flacoste> edge should run replicated as of tomorrow
[15:32] <flacoste> so watch out for any strange OOPS
[15:32] <Ursinha> flacoste, sure!
[15:33] <Ursinha> [action] Ursinha to watch for strange OOPSes on edge in the next days
[15:33] <MootBot> ACTION received:  Ursinha to watch for strange OOPSes on edge in the next days
[15:33] <Ursinha> anything else?
[15:33]  * Ursinha looks at the clock
[15:33] <Ursinha> okay
[15:33] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:33] <Ursinha> #endmeeting
[15:33] <MootBot> Meeting finished at 09:33.
[15:34] <Ursinha> thanks guys
[15:34] <Ursinha> see you next week!
[15:34] <matsubara> thanks Ursinha
[15:34] <intellectronica> thanks Ursinha
[15:35]  * rockstar goes to get dressed and shovel snow.