/srv/irclogs.ubuntu.com/2009/05/14/#launchpad-meeting.txt

jmlsorry guys.00:17
* jml is back.00:18
=== herb__ is now known as herb
=== salgado-afk is now known as salgado
=== salgado is now known as salgado-brb
=== salgado-brb is now known as salgado
=== salgado_ is now known as salgado
Ursinhahenninge, :)15:50
henningeUrsinha: Hi :)15:50
rockstarHi16:02
Ursinha#startmeeting16:02
UrsinhaWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.16:02
Ursinha[TOPIC] Roll Call16:02
UrsinhaNot on the Launchpad Dev team? Welcome! Come "me" with the rest of us!16:02
MootBotMeeting started at 10:02. The chair is Ursinha.16:02
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]16:02
MootBotNew Topic:  Roll Call16:02
bigjoolsme16:02
Ursinhame16:05
henningeich16:05
sinzuime16:05
rockstarme16:05
flacosteme16:05
herbme16:05
Ursinhaherb, intellectronica, hi16:05
Ursinha:)16:05
Ursinhawho's missing?16:05
Ursinhastub is missing, but he can join later16:05
Ursinhaintellectronica is missing too16:05
Ursinhalet's move on16:05
Ursinha[TOPIC] Agenda16:05
Ursinha* Actions from last meeting16:05
Ursinha* Oops report & Critical Bugs16:05
Ursinha* Operations report (mthaddon/herb/spm)16:05
Ursinha* DBA report (stub)16:05
MootBotNew Topic:  Agenda16:05
Ursinha[TOPIC] * Actions from last meeting16:05
Ursinha* Ursinha to talk to intellectronica about bug 35731616:05
Ursinha* Ursinha to talk to henninge about bug 30244916:05
Ursinha* rockstar to confirm that bzr fix for bug 360791 was applied to LP's bzr tree.16:05
MootBotNew Topic:  * Actions from last meeting16:05
Ursinha* cprov to request CP of fix for bug 37051316:05
ubottuLaunchpad bug 357316 in malone "hwdb +submit failing with KeyError OOPS" [Undecided,Triaged] https://launchpad.net/bugs/35731616:06
ubottuLaunchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/30244916:06
UrsinhaI suck and failed mine16:06
ubottuLaunchpad bug 360791 in bzr/1.14 "bzr pull/branch shows "Error received from smart server: ('NoSuchRevision',)"" [Critical,In progress] https://launchpad.net/bugs/36079116:06
ubottuLaunchpad bug 370513 in soyuz "failure to accept PPA uploads" [Critical,Fix committed] https://launchpad.net/bugs/37051316:06
Ursinha[action] Ursinha to talk to intellectronica about bug 35731616:06
MootBotACTION received:  Ursinha to talk to intellectronica about bug 35731616:06
Ursinhahenninge, hi :)16:06
henningeI think danilo was onto that ...16:06
rockstarI don't know if the fix has been cherry picked into production...16:06
Ursinhahenninge, can you just confirm that, please? it's set as medium, do we use that status?16:06
henningeUrsinha: in rosetta we do ;)16:08
Ursinharockstar, hm, can you check that too?16:08
rockstarCode team does, for things of medium importance.16:08
bigjoolsas does Soyuz16:08
herbrockstar: it was cherry picked on 2009-05-0916:08
UrsinhaI rarely see medium statuses :) that's why I'm asking16:08
Ursinhathanks herb16:08
rockstarherb: cool, thanks.16:08
Ursinha[action] henninge to check with danilo the status of bug 30244916:08
MootBotACTION received:  henninge to check with danilo the status of bug 30244916:08
ubottuLaunchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/30244916:08
Ursinhacool16:08
Ursinhamoving on then16:08
Ursinha[TOPIC] * Oops report & Critical Bugs16:08
Ursinhathere's only one worth mentioning, that is the one causing the InterfaceError oopses, we're still having lots and lots of occurrences (bugs 374909 and 376207), seems to be worked on by jamesh, is that correct flacoste?16:08
MootBotNew Topic:  * Oops report & Critical Bugs16:08
ubottuLaunchpad bug 374909 in storm "InterfaceError: connection already closed should be converted into DisconnectionError" [High,Triaged] https://launchpad.net/bugs/37490916:08
ubottuLaunchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/37620716:08
flacosteso, jamesh is working on 37490916:11
flacosteand the other one also16:11
Ursinharight16:11
flacostebut stuart has an easy fix for the later, that I'll likely asked to be cherrypicked16:11
flacostewe can deploy jamesh proper fix during next roll-out16:11
Ursinhaflacoste, hm, good16:11
Ursinhaflacoste, about the cp, when do you think it can be done?16:11
flacostei didn't look at the branch16:11
herb374909 is still cropping up from time to time, btw. though it's much gooder(tm) than it was last week.16:11
flacostebut once i approved it, as soon as the LOSA can take care of it16:11
Ursinhaflacoste, right. okay16:12
flacostei don't think i'll ask a C-P of the INterfaceError16:13
flacoste(storm fix being worked on by jamesh)16:14
herbflacoste: why?16:14
flacostewell, because that's not a root-cause16:14
flacosteso with the other fix in place, we shouldn't see it16:14
herbok16:14
flacostethat fix is more prophylactic16:14
Ursinhaflacoste, who's investigating the root cause?16:14
flacosteif you are talking about why we are getting disconnection errors in the first place16:17
flacostenobody, really16:17
flacostebut we have a fix for the places where we should be trapping those16:17
flacosteand recovering16:17
Ursinhayes, the disconnection errors16:17
flacostewe have no ideas at why it's happening16:17
flacostethere is nothing in the DB logs16:17
flacosteabout them16:17
Ursinhathis is creepy16:17
Ursinhathe fix you say you have16:17
Ursinhais inside the fixes for one of those bugs?16:18
flacosteyes16:18
Ursinhagreat16:18
Ursinhaso, you'll ask a cp for the second bug16:18
flacosteexactly16:18
Ursinha[action] flacoste to ask a cp for fix for bug 376207 after reviewing it16:20
MootBotACTION received:  flacoste to ask a cp for fix for bug 376207 after reviewing it16:20
ubottuLaunchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/37620716:20
Ursinhaawesome16:20
Ursinhawe have one critical bug, being worked on16:20
Ursinhaso, unless anyone has anything to point, moving to the next section!16:20
Ursinhagood16:20
Ursinha[TOPIC] * Operations report (mthaddon/herb/spm)16:20
MootBotNew Topic:  * Operations report (mthaddon/herb/spm)16:20
herb2009-05-07 - Cherry pick r8906 to the scripts server and r122 of storm to lpnet* & edge*16:20
herb2009-05-09 - Cherry pick r4006 of bzr to the codehosting server and r123 of storm to lpnet* & edge*16:20
herb2009-05-09 - Cherry pick r8348, r8312 to the PPA server and r8376 to lpnet*16:20
herb2009-05-10 - mailman didn't have access the necessary access to the DB server, but it was only noticed after restarting for log rotation. mailing lists were unavailable for approximately 7 hours.16:20
herbWe still seem to be encountering bug #156453 and bug #118626, but the situation is much improved since the rollout.16:20
herbflacoste: cprov requested a rollout of the current production tree to cesium. Apparently there was a critical fix that was included in while in RC, but we didn't re-roll to cesium. Can you (dis)approve that?16:23
ubottuLaunchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/15645316:23
ubottuLaunchpad bug 118626 in bzr-email "plugin documentation does not make interaction with checkouts clear" [Medium,Confirmed] https://launchpad.net/bugs/11862616:23
bigjoolsherb: it's already approved by kiko16:23
flacosteherb: i think kiko did? but otherwise, i can look into it16:23
flacosteright16:23
herbbigjools: missed it.  thanks16:23
Ursinhaany other questions to herb?16:23
Ursinhaokay16:23
Ursinha[TOPIC] * DBA report (stub)16:23
MootBotNew Topic:  * DBA report (stub)16:23
rockstarherb: I was under the impression that the loggerhead stuff was WAY better than "much improved"16:23
flacostestub sent it to the list16:23
herbrockstar: we're still restarting a couple of times a week. which is much improved over a couple of times a day.16:23
herbrockstar: order of magnitude.16:23
Ursinhaflacoste, to lp list? I can't seem to find it16:23
rockstarherb: okay.  Is it memory restarts, or hanging restarts16:23
flacosteThe ex-master database (launchpad_prod on hackberry) is bloated and16:23
flacostestarted exceeding its free space map settings. Nothing really to worry16:23
flacosteabout - it might cause bloat to spiral but I suspect not in this case.16:23
flacosteThe losas can bounce it after shutting down the systems using it as a16:23
flacosteslave, and I've suggested using it as the standalone replica for16:23
flacosteread-only mode launchpad during the rollout because we then rebuild it16:23
rockstar(The memory restarts shouldn't be happening anymore)16:23
flacosteafterwards and it will be all nice and freshly packed.16:23
flacosteNothing major do do with database patches this cycle. rockstar's16:23
flacostebugbranch and specbranch column pruning needs to be cleared with Mark16:23
flacostestill.16:23
Ursinhathanks flacoste16:23
rockstarflacoste: yeah, and there are other branches dependent on that one.16:24
herbrockstar: The memory situation seems ok (ie. not death spiraling).  seems to be hangs at this point.16:24
herbrockstar: we still see it ~1.5GB resident, but doesn't seem to grow beyond that.16:25
rockstarherb: well, "death spiraling" to be is different than the memory issues.16:25
rockstarherb: it's going to be *kinda* memory intensive just because of what it's serving.16:26
herbrockstar: understood16:26
herbrockstar: as I said, much improved. 1.2 - 1.5G is much better than 3.7G16:26
rockstarherb: and we know the cause of the hangs, we just don't know how to fix it.16:26
herbrockstar: good news, bad news, eh?16:26
rockstarherb: yeah, something like that.16:26
Ursinhaokay. anyone else want to say something?16:27
Ursinha516:27
Ursinha416:27
Ursinha316:28
Ursinha216:28
Ursinha116:28
UrsinhaThank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.16:28
Ursinha#endmeeting16:28
MootBotMeeting finished at 10:27.16:28
Ursinhathanks all16:28
henningethanks Ursinha ;)16:28
Ursinhahenninge, :)16:28
=== salgado is now known as salgado-lunch
=== Ursinha is now known as Marvin_
=== Marvin_ is now known as Ursinha
=== Ursinha is now known as Marvin_
=== Marvin_ is now known as Ursinha
=== salgado-lunch is now known as salgado
=== salgado_ is now known as salgado
=== salgado is now known as salgado-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!