[14:08] * Ursinha looks [14:08] * Ursinha hugs MootBot [14:08] you're here! [14:13] :) [15:02] #startmeeting [15:02] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [15:02] [TOPIC] Roll Call [15:02] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [15:02] Meeting started at 09:02. The chair is Ursinha. [15:02] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [15:02] New Topic: Roll Call [15:02] hi MootBot [15:02] me [15:02] me [15:02] me [15:02] me [15:03] me [15:03] rockstar, ping [15:03] me [15:03] henninge is mia [15:03] me (barely) [15:03] :) [15:03] well, no issues for translations, so no much of a problem [15:03] alright [15:04] [TOPIC] Agenda [15:04] * Actions from last meeting [15:04] * Oops report & Critical Bugs- [15:04] * Operations report (mthaddon/herb/spm) [15:04] * DBA report (DBA contact) [15:04] New Topic: Agenda [15:04] [TOPIC] * Actions from last meeting [15:04] 1) matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time [15:04] 2) TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week (26th to 30th Jan) [15:04] 3) matsubara to file bugs for OOPS-1116EC265 (bug 320293), OOPS-1113EB6 (bug 276144) [15:04] New Topic: * Actions from last meeting [15:04] Launchpad bug 320293 in launchpad-bazaar "OOPS traversing to branch on person page" [Medium,Confirmed] https://launchpad.net/bugs/320293 [15:04] Launchpad bug 276144 in launchpad-bazaar "OOPS when trying to delete stacked-on branches." [Medium,Triaged] https://launchpad.net/bugs/276144 [15:04] 1 - I did that. From now on to discuss meeting time changes just add it to the list of meeting topics, on MeetingAgenda page [15:04] 2 - okay as well [15:04] next! [15:05] [TOPIC] * Oops report & Critical Bugs [15:05] New Topic: * Oops report & Critical Bugs [15:05] three oopses and one bug [15:05] the oopses [15:05] MissingMergeDirective: OOPS-1125CEMAIL1 [15:05] this is yours rockstar? [15:05] * Ursinha kicks ubottu [15:05] Ursinha, I can look at it [15:06] rockstar, thanks [15:06] bac, to registry: OOPS-1125A165, OOPS-1125A1096 - this second one is the same in the email sinzui sent to barry [15:06] https://devpad.canonical.com/~jamesh/oops.cgi/1125A165 [15:06] https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096 [15:06] me! [15:07] hi henninge! [15:07] Ursinha: hi! :) [15:07] Ursinha: barry is working on the second one [15:07] sorry for the delay ... [15:07] bac, do you know if there's an open bug for that? [15:08] intellectronica, bug 279561, that's affecting people more frequently lately [15:08] Launchpad bug 279561 in malone "No url for when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561 [15:08] * intellectronica shoegazing [15:08] Ursinha: i don't. i'll check with barry when he returns from afk [15:09] sorry, it's still a mystery [15:09] i'll go back to working on it now [15:09] intellectronica, do you have time to deal with that? it's causing a lot of oopses and annoying people [15:09] bac, thanks [15:09] Ursinha: i do now [15:10] [action] bac to check with barry if there's an open bug for OOPS-1125A1096, if not, Ursinha to file one [15:10] https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096 [15:10] ACTION received: bac to check with barry if there's an open bug for OOPS-1125A1096, if not, Ursinha to file one [15:10] https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096 [15:10] bac, about the other one, do you have any clues? [15:10] intellectronica, thanks [15:10] [action] intellectronica to work on bug 279561 [15:11] ACTION received: intellectronica to work on bug 279561 [15:11] Launchpad bug 279561 in malone "No url for when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561 [15:11] [action] rockstar to check OOPS-1125CEMAIL1 [15:11] ACTION received: rockstar to check OOPS-1125CEMAIL1 [15:11] any other issues you may want to point? [15:11] Ursinha: no clue ATM. will check it out [15:12] thanks bac [15:12] I appreciate that [15:12] [action] bac to take a look at OOPS-1125A165 [15:12] ACTION received: bac to take a look at OOPS-1125A165 [15:12] https://devpad.canonical.com/~jamesh/oops.cgi/1125A165 [15:12] well, moving on [15:12] https://devpad.canonical.com/~jamesh/oops.cgi/1125A165 [15:12] [TOPIC] * Operations report (mthaddon/herb/spm) [15:12] New Topic: * Operations report (mthaddon/herb/spm) [15:12] * 2008-01-28 - Yesterday - We rolled out 2.2.1. Total downtime was approximately 35 minutes. [15:12] * Even after the rollout, we continue to have problems as a result of bug #156453 and bug #118625. I cannont emphasize enough how much of a pain point this is becoming for the LOSAs. More than a quarter of the incidents that we've had to deal with in the last year have been related to these two bugs. [15:12] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 [15:12] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:12] * We had a couple of incidents in the last week relating to bug #260171. [15:12] Bug 260171 on http://launchpad.net/bugs/260171 is private [15:12] * Do we expect to roll out any additional patches this week? If so, when? [15:13] herb, I think at least one, I'll talk to kiko and confirm that to you [15:13] ok [15:14] Ursinha: do you suspect it will be rolled out today or tomorrow? [15:14] Ursinha: He approved one of mine, too. [15:14] [action] Ursinha to check with kiko if any other rollouts will happen this week [15:14] ACTION received: Ursinha to check with kiko if any other rollouts will happen this week [15:14] I can wait, I was just curious if you have a best guess. [15:14] herb, guess tomorrow, but it depends on if we're having more issues to be fixed [15:15] ok [15:15] thanks [15:15] henninge, cool [15:15] thanks herb [15:15] I took note of the "pain to the LOSAs" about that two bugs [15:16] Ursinha: thanks. much appreciated [15:16] well, next [15:16] [TOPIC] * DBA report (DBA contact) [15:16] stub sent one email about it: [15:16] Looks like the upgrade went smoothly and nothing exciting is happening. Yay. [15:16] Although neither db server is particularly heavily loaded on average, we do see spikes sometimes. Also, the slave is still more loaded than production. As I don't expect us to get a second slave in the near future, I'm thinking we need to look into some better load balancing to move some of the slave load back to the master. [15:16] New Topic: * DBA report (DBA contact) [15:16] Config changes on one or two of the appserver instances to use the master as the slave would work in the short term. Long term I think we could be better with a load balancer like pgbouncer or pgpool (1 or 2). [15:16] that's all from stub [15:16] anything else? [15:16] anybody wants to point or discuss something? [15:17] 5 [15:17] 4 [15:17] 3 [15:17] 2 [15:17] 1 [15:17] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [15:17] #endmeeting [15:17] Meeting finished at 09:17. [15:17] the fastest eva [15:17] Thanks Ursinha :-) [15:17] thank you all [15:17] gary_poster, :) [15:17] Ursinha: thanks! === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado === Ursinha is now known as Jorjao === Jorjao is now known as Ursinha === EdwinGrubbs is now known as Edwin-lunch === salgado is now known as salgado-afk === bac is now known as bac_afk === bac_afk is now known as bac