=== ursula__ is now known as Ursinha === salgado-afk is now known as salgado === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [14:59] that time of the week is coming [14:59] it's weekend already? [15:00] intellectronica, almost :) [15:00] it's show time [15:00] maaan [15:00] #startmeeting [15:00] Meeting started at 09:00. The chair is Ursinha. [15:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [15:00] me [15:00] me [15:00] haha [15:01] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [15:01] [TOPIC] Roll Call [15:01] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [15:01] New Topic: Roll Call [15:01] now you can say mee [15:01] me [15:01] me (again) [15:01] me [15:01] meaux [15:01] erm [15:01] me [15:01] me [15:01] who's missing [15:02] a losa? [15:02] stub? [15:02] herb, ping [15:02] I'm here [15:02] me [15:02] me [15:02] stub is not here, it seems [15:02] flacoste, do you know anything about him? [15:02] he's on leave [15:03] flacoste, oh, okay [15:03] oh, so what's with db schema reviews? [15:03] [action] email stub asking for the DBA report [15:03] [action] email stub asking for the DBA report [15:03] ACTION received: email stub asking for the DBA report [15:03] matsubara_, you have no Priesthood with Mootbot! [15:03] well, flacoste, can you be stub's ears today? [15:03] yeah, not today :-) [15:03] eyes, in case [15:04] well [15:04] [TOPIC] Agenda [15:04] * Next meeting [15:04] * Actions from last meeting [15:04] * Oops report & Critical Bugs [15:04] * QA pending items [15:04] * Operations report (mthaddon/herb/spm) [15:04] New Topic: Agenda [15:04] * DBA report (DBA contact) [15:04] [TOPIC] * Next meeting [15:04] New Topic: * Next meeting [15:04] okay for you? [15:04] same time and yada yada? [15:04] fine by me [15:04] hi henninge :) [15:04] me! [15:04] hi henninge [15:05] cool [15:05] Ursinha: yes, i'll proxy stuff to stub [15:05] some ppl may be in UDS? (not me) [15:05] so that's fine [15:05] flacoste, thanks [15:05] hi! [15:05] ok, any of the QA contacts will be attending UDS? [15:05] not me [15:05] not stub [15:05] nope [15:05] not me, but the rest of my team [15:06] no [15:06] henninge, but you're the QA contact, so whatever [15:06] :) [15:06] :) [15:06] ok, fine [15:06] sinzui: ? [15:06] No I will not [15:06] [TOPIC] * Actions from last meeting [15:06] New Topic: * Actions from last meeting [15:06] ok, so same time next week [15:06] * gmb to add sql to hide bug 297388 as a fix for bug 300324 [15:06] * abentley to talk to code team about bug 301595 [15:06] * matsubara email losas to request operations report [15:06] * matsubara email stub to request dba report [15:06] Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text) [15:06] Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324 [15:06] Launchpad bug 301595 in launchpad-bazaar "OOPS when deleting a branch with revisions" [High,Fix committed] https://launchpad.net/bugs/301595 [15:06] uh-oh [15:06] damn bug [15:07] I did my two items and losas/stub emailed their reports to the list [15:07] matsubara_, nice, thanks [15:07] abentley must have talked with code people, that bug is fix committed [15:07] and intellectronica, do you know about bug 300324? [15:07] it's still triaged [15:07] i don't [15:08] * intellectronica looking [15:08] well [15:08] the comment says it all [15:08] Ursinha, intellectronica: I'll chase that one. it's a simple SQL that needs approval to hide #297388 [15:09] these are quite exceptional, so for now we're happy to deal with them on an individual basis [15:09] but it will be fixed, that's the point :) [15:09] thanks matsubara_ [15:09] [action] matsubara to talk with gmb and add sql to hide bug 297388 as a fix for bug 300324 [15:09] we already have the db infrastructure for dealing with this in a better way, but it's not the highest priority to finish that [15:09] Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text) [15:09] Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324 [15:09] [action] matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324 [15:09] ACTION received: matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324 [15:09] Error: Launchpad bug 2 could not be found [15:09] Error: Launchpad bug 2 could not be found [15:09] okay [15:10] [TOPIC] * QA pending items [15:10] New Topic: * QA pending items [15:10] well, I can see that there are lots of items to be tested on your testplans [15:10] are you having problems on testing them? [15:10] I [15:11] my fear is that we start to test it too late [15:11] [action] ursinha to add QA pending items topic to the MeetingPage agenda === salgado is now known as salgado-lunch [15:11] matsubara_, it's done already [15:11] some of the landings on our team are not really testable (some of the hwdb stuff). i'll see that the rest are tested promptly [15:11] Ursinha: all right. thanks [15:11] Ursinha: Registry has been finding lots of bugs, so some of the items we are working on will not move to OK until we close the blocking bugs [15:11] so, rockstar, can you ask your team what can be done about that, please? [15:12] sinzui, that's okay, you have lots of tested items [15:12] Ursinha, yup, making a note now. [15:12] rockstar, thanks [15:12] [action] rockstar to talk with code team about their pending QA items [15:12] ACTION received: rockstar to talk with code team about their pending QA items [15:12] okay [15:12] henninge, can you do the same too, please? [15:13] will do [15:13] thanks henninge [15:13] [action] henninge to talk with translation team about their pending QA items [15:13] ACTION received: henninge to talk with translation team about their pending QA items [15:13] bigjools, what about you? [15:14] souyz has a lot of items [15:14] we've been QA-ing fine [15:14] but they're arriving thick and fast again [15:14] * bigjools will prod people [15:14] bigjools, it's true [15:14] how are you doing QA? [15:15] at the end of the week, on demand or what? [15:15] when time allows after the branch lands [15:15] flacoste, sinzui, henninge, rockstar ^ [15:15] intellectronica, ^ [15:15] like bigjools [15:15] bigjools, flacoste okay [15:15] Ursinha: we are QAing a day after each landing [15:15] ideally the day after it lands [15:15] on demand [15:15] we do it when we can [15:15] sinzui, awesome [15:15] but unless i prompt people it doesn't happen [15:16] and try to remind each other that it's important to do [15:16] flacoste: right, same here [15:16] but this is really because we don't land that many QA-able changes [15:16] bigjools, flacoste, this is not quite that good [15:16] most of work recently was on non front-end stuff [15:16] flacoste, I see you have only two items in there [15:16] Ursinha: it has to be balanced with other important stuff [15:16] so I understand [15:16] sinzui's sounds like the sanest. i will suggest that we follow the same rule [15:17] so it may go 2/3 days and then we'll clear them [15:17] bigjools, I understand, but as it's important too, I'd like to ask you please, please, to ask us help [15:17] if we can test it for you, we'll do [15:17] Ursinha: ok! please test all our items ;) [15:17] intellectronica: of course that assume staging *is* updating. [15:17] bigjools, ha ha [15:17] sinzui: if we all adopt this rule, there will be many people on the case. sometimes staging doesn't update on purpose, but in most cases it just needs someone to care enough [15:17] sinzui, yes, this is a major problem [15:17] in my opinion [15:17] Ursinha: you might need an account on the dogfood machine to do some of them [15:18] bigjools, please, can we talk a bit about that later? [15:18] Ursinha: sure [15:18] [action] Ursinha to talk with bigjools about one account on dogfood machine to do some testing [15:18] ACTION received: Ursinha to talk with bigjools about one account on dogfood machine to do some testing [15:18] great [15:19] intellectronica, we're always watching if staging is updating [15:19] recently, at lease [15:19] least [15:19] cool [15:19] okay, so if there's something we can do, please poke us [15:20] so my beloved section of this meeting [15:20] [TOPIC] * Oops report & Critical Bugs [15:20] New Topic: * Oops report & Critical Bugs [15:20] well [15:20] I was going to mention the bug matsubara will look, that is the bug missing comment 0 one, ok [15:21] bug 271879 [15:21] Bug 271879 on http://launchpad.net/bugs/271879 is private [15:21] * Ursinha looks at rockstar [15:21] crap [15:21] rockstar, that is a timeout in +code-index page [15:21] still happening, yesterday we have a lot of timeout oopses, btw [15:21] not only in +code-index [15:22] Ursinha, that might be on my list of things to do today, if the network will let me see the bug. [15:22] flacoste, do you think it can be related with the DB load problem? [15:22] rockstar, okay, thanks [15:22] [action] rockstar to handle bug 271879 today (12-04) [15:22] ACTION received: rockstar to handle bug 271879 today (12-04) [15:22] Bug 271879 on http://launchpad.net/bugs/271879 is private [15:22] Bug 271879 on http://launchpad.net/bugs/271879 is private [15:22] the +code-index? [15:22] it might [15:22] but if it continues, probably not [15:22] flacoste, all the timeouts [15:23] Yea, I got that oops yesterday, opened my big mouth, and got work to do as a reward. :) [15:23] I mean, we had a lot of timeouts yesterday [15:23] yeah, DB load increase the likelyhood of any timeout to happen [15:23] Regardless, the query needs love. [15:23] for instance, +bugs-index page had 998 timeouts [15:24] on lpnet [15:24] will keep one eye on it [15:24] Damn [15:24] well, will keep both, better [15:24] [action] Ursinha to watch timeouts in the next days to see if they continue to happen this often [15:24] ACTION received: Ursinha to watch timeouts in the next days to see if they continue to happen this often [15:24] cool [15:24] so this is all in the oops land [15:25] matsubara is faster copying and pasting stuff around [15:25] :) [15:25] so there's only that critical bug [15:26] bug 269538, that will never be closed [15:26] Launchpad bug 269538 in bugzilla-launchpad/bugzilla-3.2 "Compliation error in plugin when authenticating" [Critical,Fix committed] https://launchpad.net/bugs/269538 [15:26] matsubara, do you remember that from last meeting? [15:26] Ursinha: gmb is chasing that with max [15:26] matsubara, okay [15:26] thanks [15:26] so the section is over [15:26] moving on [15:27] [TOPIC] * Operations report (mthaddon/herb/spm) [15:27] Quiet week except for yesterday when the abusive user was slowing the system down. [15:27] hi herb [15:27] - We're still rebooting codebrowse often as a result of bugs #156453 and #118625. I know it's being worked on, but there hasn't been an update to the bug reports in about a month. [15:27] - Bug #301720 continues to be a problem requiring almost daily intervention from the LOSAs. We haven't seen an update on this one in the past week. [15:27] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 [15:27] hi [15:27] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:27] Bug 301720 on http://launchpad.net/bugs/301720 is private [15:27] rockstar, beuno and you are looking at the loggerhead bug, right? [15:27] Ursinha, yes, it's being discussed on list right now. [15:28] bug 301720 [15:28] Bug 301720 on http://launchpad.net/bugs/301720 is private [15:28] There will probably be some work at UDS. [15:28] rockstar, nice to hear [15:28] about bug 301720, anybody? [15:28] Bug 301720 on http://launchpad.net/bugs/301720 is private [15:29] I think sinzui was working on it. [15:29] New Topic: * Operations report (mthaddon/herb/spm) [15:29] sinzui, can you do an update on that bug, please? [15:29] MootBot, you are slow, man [15:29] Ursinha: it is in PQM now [15:30] sinzui, awesome :) [15:30] okay [15:30] anything else herb? [15:30] nothing from me. thanks. [15:30] okay, rockstar, can you manage having something reported on those bugs, pretty please? [15:31] Ursinha, doing that now. [15:31] rockstar, thanks! [15:31] [action] rockstar to manage having something reported on bug 118625 and bug 156453 [15:31] ACTION received: rockstar to manage having something reported on bug 118625 and bug 156453 [15:31] okay [15:31] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:31] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 [15:31] moving on again [15:31] [TOPIC] * DBA report (DBA contact) [15:31] New Topic: * DBA report (DBA contact) [15:31] stub, that is [15:31] that's not here [15:32] and matsubara will mail him about his report [15:32] so [15:32] anyone have something to declare before I end all this? [15:32] i have [15:32] edge should run replicated as of tomorrow [15:32] so watch out for any strange OOPS [15:32] flacoste, sure! [15:33] [action] Ursinha to watch for strange OOPSes on edge in the next days [15:33] ACTION received: Ursinha to watch for strange OOPSes on edge in the next days [15:33] anything else? [15:33] * Ursinha looks at the clock [15:33] okay [15:33] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [15:33] #endmeeting [15:33] Meeting finished at 09:33. [15:34] thanks guys [15:34] see you next week! [15:34] thanks Ursinha [15:34] thanks Ursinha [15:35] * rockstar goes to get dressed and shovel snow. === matsubara is now known as matsubara-lunch === salgado-lunch is now known as salgado === matsubara-lunch is now known as matsubara === salgado is now known as salgado-brb === thumper_laptop is now known as thumper === salgado-brb is now known as salgado-afk === bac is now known as bac_afk === bac_afk is now known as bac