=== noodles775 is now known as noodles-afk === salgado-afk is now known as salgado === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell === popey_ is now known as popey [15:00] moi [15:00] me [15:00] me === henninge_ is now known as henninge [15:00] erm [15:00] #startmeeting [15:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [15:00] Meeting started at 10:00. The chair is Ursinha. [15:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [15:01] [TOPIC] Roll Call [15:01] New Topic: Roll Call [15:01] me [15:01] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [15:01] me [15:01] me [15:01] me [15:01] me :) [15:01] me [15:01] herb: hi [15:01] I wish someone not on the dev team would "me" [15:01] rockstar: me too [15:01] anyway :) [15:01] rockstar, Ursinha: you can get that, but it requires /a lot/ of herding [15:02] herb: are you there by chance? :) [15:02] intellectronica: indeed [15:02] so here is stub [15:02] only herb missing [15:03] me [15:03] sorry [15:03] :) [15:03] hi herb! [15:03] moving on :) [15:03] hi [15:03] [TOPIC] Agenda * Actions from last meeting * Oops report & Critical Bugs * Operations report (mthaddon/herb/spm) * DBA report (stub) [15:03] New Topic: Agenda * Actions from last meeting * Oops report & Critical Bugs * Operations report (mthaddon/herb/spm) * DBA report (stub) [15:03] argh [15:04] [TOPIC] * Actions from last meeting [15:04] • sinzui to request CP for bugs 341085 and 341117 [15:04] • ursinha to talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 - done [15:04] New Topic: * Actions from last meeting [15:04] Bug 341085 on http://launchpad.net/bugs/341085 is private [15:04] Launchpad bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Low,Triaged] https://launchpad.net/bugs/341766 [15:04] • ursinha to debug 340058 and write the results back to the report - done [15:04] • bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-) [15:04] my items are done [15:04] done [15:04] Ursinha: Done [15:04] awesome! [15:04] next [15:04] [TOPIC] * Oops report & Critical Bugs [15:04] intellectronica, https://bugs.edge.launchpad.net/malone/+bug/329908 [15:04] New Topic: * Oops report & Critical Bugs [15:04] rockstar, as we previously talked, https://bugs.edge.launchpad.net/launchpad-bazaar/+bug/328302 [15:04] Ubuntu bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] [15:04] Error: This bug is private [15:04] is it? [15:04] * Ursinha looks again [15:04] * rockstar doesn't think that's the right bug... [15:05] Ursinha: no news about this one yet :( [15:05] Ursinha, yeah, it's private because it has query info [15:05] bug 328302 [15:05] Bug 328302 on http://launchpad.net/bugs/328302 is private [15:05] it's this one [15:06] rockstar: I told you about that a few mins ago :) [15:06] Ursinha, yeah, and it looks like it's In Progress. [15:06] intellectronica: that bug is happening like everyday [15:06] rockstar: I brought it here because you didn't comment on that [15:06] rockstar: at -code, that is [15:07] Ursinha, I have nothing to comment on. Looks like thumper is working on it. [15:07] rockstar: so as I presumed, you know as much about that as I do [15:07] Ursinha, unfortunately, due to illness and travelling, I've been a bit out of touch. [15:07] Ursinha: shit. ok, i'll try to tackle it soon [15:08] rockstar: no problem :) just brought to attention to see if you had some info that I (neither the bug report) would have [15:08] intellectronica: thanks for that [15:08] [action] intellectronica to make efforts to take a look at bug 329908 [15:08] Launchpad bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] https://launchpad.net/bugs/329908 [15:08] ACTION received: intellectronica to make efforts to take a look at bug 329908 [15:08] :) [15:09] :) [15:10] about the code bug, I'll catch up with thumper later, if the status doesn't change [15:10] thanks intellectronica and rockstar [15:10] Ursinha, he commented and said he had a fix for it. [15:10] Takes the query down to 3s instead of 17s. [15:10] rockstar: I saw that. [15:10] thanks [15:10] moving on [15:11] all critical bugs are fix committed, yay! [15:11] that's all for me in oops/bugs section [15:11] next [15:11] [TOPIC] * Operations report (mthaddon/herb/spm) [15:11] New Topic: * Operations report (mthaddon/herb/spm) [15:11] 2009-03-20 - CP r7866 and r7900 to production. [15:11] Sounding like a broken record, bug 156453 and bug 118625 continue to be a problem. [15:11] Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453 [15:11] Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 [15:11] There have been some problems with staging updates this week. Seem to be mostly DB related and resolved now. There is a staging update running now. [15:11] sinzui has a couple of cherry pick requests pending approval. [15:11] Other than that it's been a quiet week. Unless there are any questions, that's it from the LOSAs unless there are questions. [15:11] herb, we may actually have an answer for you next week! :) [15:11] * sinzui will speak with kiko. [15:12] rockstar: that is exciting news [15:12] sinzui: those are the ones of the actions from last meeting? [15:12] I'm not certain those items are worthy of a CP [15:12] herb, mwhudson and I are going to be hacking together on this soon. [15:12] [action] sinzui to talk to kiko about pending cp requests [15:12] ACTION received: sinzui to talk to kiko about pending cp requests [15:12] hence those bugs were high [15:13] sinzui: hm, I see [15:13] sinzui: are you going to sort out with kiko? [15:14] I'll get his opinion [15:14] rockstar: that's cool [15:14] right sinzui, thanks [15:14] anything else for herb? [15:14] that sounds a no :) [15:15] thanks herb [15:15] thanks Ursinha [15:15] moving on [15:15] [TOPIC] * DBA report (stub) [15:15] New Topic: * DBA report (stub) [15:15] If you see 'database in recovery mode', it means part of PostgreSQL has crashed. We have seen it this week on staging and twice with buildbot. We don't know the trigger yet. PostgreSQL will recover, but our code might not. When it happened on staging, it looks like the staging appserver didn't reconnect fully and needed to be manually restarted. [15:15] If anyone has statement logging on and triggers the bug, I'l like your log files. [15:15] I'll abuse an ec2 instance tomorrow in an attempt to trigger the bug and try to get a pointer to the trigger. [15:15] A new PostgreSQL update came through today which contains some crash fixes. With luck, we will upgrade and the problem will never be seen again. [15:15] All the bugs I can find in the updated database maintenance tools have been fixed, so staging updates should work fine now (barring the PostgreSQL crash). [15:15] Some bigish DB changes are part of the next rollout. We are now maintaining two slaves so the maintenance tools needed to be updated to cope. We have also split our single replication set into two now as part of keeping the SSO service running during Launchpad outages. With luck, nobody will notice and things will continue to run smoothly. [15:16] I'm done [15:16] hm. [15:16] anyone has questions for stub? [15:17] stub: that happened only on staging, right? [15:17] Crash on staging and twice on ec2 with buildbot [15:17] this explain "some" weird oopses [15:18] hm, thanks stub [15:18] anyone else wants to say something? [15:18] 3 [15:18] 2 [15:18] 1 [15:18] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [15:18] #endmeeting [15:18] Meeting finished at 10:18. [15:19] that was fast [15:19] thanks everyone [15:19] ciao [15:19] see you next week [15:19] bigjools: falou [15:19] :P [15:19] Ursinha: what did you just call me? [15:19] bigjools: lol [15:19] you're responsible for at least half of the pt_br swearing I know :) [15:19] bigjools: that's an informal way to say bye [15:20] lol! [15:20] you shouldn't say that in public, I'm a lady [15:20] ha-ha [15:20] * bigjools grins [15:20] bigjools: falou is the same as see ya [15:20] matsubara: you're late man [15:20] :P [15:20] hi there [15:20] kidding :) [15:20] Ursinha:right! [15:20] matsubara: the meeting has just finished :) [15:21] well, I told you yesterday I might not show up [15:21] Ursinha: is it time for some swearing aimed at matsubara? :) [15:21] matsubara: < Ursinha > kidding :) [15:21] gladly you're a superstart woman and took care of everything [15:21] matsubara: lol [15:21] bigjools: he's a nice guy, don't do that [15:21] bigjools: i know you've been saving some of your portuguese to use in special occasions like this [15:21] lol [15:22] you guys taught me all I know [15:23] bigjools: go QA! [15:24] :P [15:24] qa team is awesome :P [15:24] :-) [15:24] va se f... [15:24] haha [15:24] hahahahaha! [15:24] rofl [15:25] * bigjools cranks up Metallica and goes back to ninja coding mode [15:26] metallica rocks [15:26] they are touring the UK [15:26] I missed them :( [15:28] bigjools: there are rumours that they could come to brazil this year [15:28] that would be supa-awesome [15:28] for you! [15:28] lol [15:29] bigjools: why did you miss them? [15:29] Ursinha: because I get worked like a camel my kiko [15:29] by* [15:29] ohh [15:29] that's bad [15:29] :) [15:29] :) [15:30] well, I didn't keep up with the news, that's all [15:34] bigjools: man, that would hurt if it was me :/ [15:34] I already cry when I miss a *cover* band [15:34] imagine missing the original band [15:35] * Ursinha realizes she's not helping [15:35] yep [15:35] thanks for your support :) [15:35] lol [15:35] sorry === cprov is now known as cprov-lunch === matsubara is now known as matsubara-lunch === salgado is now known as salgado-lunch === cprov-lunch is now known as cprov === statik` is now known as statik === matsubara-lunch is now known as matsubara === salgado-lunch is now known as salgado === salgado is now known as salgado-afk === Ursinha_ is now known as Ursinha === matsubara is now known as matsubara-dinner