/srv/irclogs.ubuntu.com/2009/08/27/#launchpad-meeting.txt

barrymwhudson: heh00:00
* barry laughs when he hears of people complaining about a few hundred messages a day00:00
* jml thinks...00:00
jmlany feedback from non-Canonical contributors about the review process?00:01
* barry looks at wgrant00:01
mwhudsoni guess i can take the opportunity to do a little survey00:01
wgrantbarry: Working well for me.00:01
barryi'll just say again that it is /fantastic/ to see yours and others from the community landing branches00:02
wgrantIt's great to finally be able to!00:02
mwhudsonas build engineer for the next cycle, what's causing you, the developer, the most pain?00:03
wgrantMy only slight issue is that sponsors are often unclear on how to do things.00:03
wgrantIs this not documented?00:03
wgrantmwhudson: 'build engineer'?00:03
jmlwasn't that announcement on launchpad-dev!00:03
wgrantNo. We don't find anything out.00:03
barrywgrant: can you elaborate?00:03
mwhudsonbah00:03
mwhudsonand the wiki page is internal00:04
* jml frowns00:04
mwhudsonstill learning at this sort of thing it seems :)00:04
barryyep.  old habits die hard, but we're really trying to move more and more resources, communications, etc. into the public square00:04
wgrantbarry: Well, some of my sponsors have had trouble working out how to ec2test my branches. I'm not sure why.00:04
wgrantWhat was this email meant to contain?00:05
barrymwhudson: we really should find a way for non-Canonicals to run ec200:05
jmlwe're going to have a build engineer position00:05
wgrantbarry: +inf00:05
jmlit's a rotating postiition00:05
mwhudson"Each development cycle, a different engineer will be taken off his normal development duty, and have has main focus for the cycle the development and well being of the build system. "00:05
jmlthe build engineer works on making developers lives more pleasant, essentially00:05
wgrantAha. I see.00:06
mwhudsoni'm the first one, starting officially on monday00:06
jmlmwhudson, in answer to your question00:06
mwhudsoni'm probably going to tackle some buildbot annoyances first00:06
mwhudsonjml: you sent me an email about this that i haven't read yet :)00:06
wgrantmwhudson: Bonus points for making it public.00:06
mwhudsonwgrant: yeah, that should be possible00:06
jmlmwhudson, my email wasn't so much about my pain points, actually :)00:07
jmlmwhudson, it takes way too long to land branches; buildbot should only send emails that require action; test failures in devel break ec2test runs00:08
barryjml, mwhudson what is the main hold up to allowing the public to run ec2?  is it the payment issue?00:08
mwhudsonbarry: i imagine so00:08
mwhudsonalso ec2test is still private i think, but that will change soon (?)00:08
jmlmwhudson, not a pain point for me, but one I've had to watch: it takes too long to get set up with a dev env00:09
barrysurely we can find a way to subsidize the ec2 costs.  it's gotta be trivial in the scheme of things.  i usually feel guilty expensing it every quarter00:09
jmlmwhudson, the main meta issue is that it's really hard to see what the status of various build-related projects are (e.g. buildbot going public, ec2test going public, etc)00:09
rockstarbarry, when we have a UEC set up in the data center, it might be easier.  *shrug*00:10
barryrockstar: good point00:10
mwhudsonjml: i think A build engineer, though probably not me, should focus on getting buildbot running the test suite in parallel across several instances00:10
jmlmwhudson, I very strongly agree.00:10
rockstarmwhudson, abentley and I had a discussion about test layers today, and whether or not we're really using the layers properly.00:11
rockstarI find I spend a lot of time setting up and tearing down the test harnesses just to run one test (that usually runs in shorter time than set up or tear down.00:11
mwhudsonrockstar: i would claim that using layers correctly is close to impossible, but maybe that's a bit bigoted00:11
barryrockstar: i can almost bet we're not.  the layers are horribly named and it is not intuitive to know what layer a test should go in00:12
mwhudsonrockstar: there's also the terrible way we start daemons in tests, that doesn't help00:12
rockstarbarry, yeah, GoogleServiceLayer and LibrarianLayer were two we identified as not really necessary for most of our tests.00:12
jmlswitching to testresources would help here00:12
mwhudson+1 jml00:12
jmlI'm a little terrified about the work involved00:12
rockstarjml, +100:12
barryrockstar: and they take a long time to start00:12
jmlbut maybe if we actually explored what needed to happen, it would be less terrifying00:13
rockstarbarry, yeah, which is basically what I spent much of yesterday doing.00:13
wgrantlibrarian takes forever!00:13
jmlwgrant, indeed.00:13
rockstarwgrant, this morning it took 15.391 seconds to start up, and 7.3 seconds to tear down the librarian.00:13
jmlmwhudson, so, that's not really a survey, since it's just me :)00:13
wgrantrockstar: It's about twice as fast here, most of the time.00:14
rockstarmwhudson, really, if I had a pony request, it'd be a graph for how long the test suite is taking, and maybe some mini graphs of different sections of the test.00:14
mwhudsonfor all that, step 1 is still going to be force build on buildbot00:14
mwhudsonrockstar: that's a good idea00:14
* jml agrees -- it's in an email I sent :)00:14
rockstarjml, dammit.  You and your organization...00:15
rockstarDo we have a place to aggregate all of these ideas?00:15
jmlexactly!00:15
rockstarjml, did you send an email about that too?00:16
jmlI was about to say, step 0 should really be tracking this.00:16
barryrockstar: like a wiki? <wink>00:16
jmlrockstar, it was in the same email as the graph stuff :)00:16
rockstarbarry, wikis are TERRIBLE issue trackers.00:16
jmlbarry, or, umm, a bug tracker.00:16
jmlbarry, I've got this good one I can sell you00:16
rockstarI know these really great guys who spend a lot of time writing a bug tracker...00:16
mwhudsonstep -1: move BE wiki pages to dev.lp.net00:16
barry:)00:16
jmlmwhudson, \o/00:17
mwhudsonstep 0: a tag in launchpad-project, i guess00:17
rockstarMaybe a product in launchpad-project.00:17
wgrantWouldn't a separate product be much better?00:17
jmlrockstar, wgrant, maaybe00:17
jmlI don't think that the way Launchpad uses products is the way Launchpad wants others to use products though00:18
* jml makes a note about that00:18
wgrantNo.00:18
wgrantBut this one is different.00:18
jmlwgrant, yeah, I think so too, but I'm not sure how it's different.00:19
wgrantjml: It's not part of the codebase.00:19
jmlwgrant, some of the bugs are in the code base00:19
wgrantMost of the other products are.00:19
wgrantHmm.00:19
jmlwgrant, but a subproduct would fit there, I guess00:19
* mwhudson writes another thing down, "make sure buildbot config is in a branch or two on launchpad"00:20
jmlthe secret, proprietary project with all of our developer utilities might well belong as a public sub-project of Launchpad00:20
barryjml: +100:20
wgrantjml: lp-dev-utils, or is there something else?00:20
jmlwgrant, lp-dev-utils, yes.00:20
jmlwgrant, that is, I can neither confirm nor deny the existence of said project...00:21
wgrantHeh.00:21
* wgrant whines about the product/project/project/projectgroup confusion.00:21
jmlpatches accepted :)00:21
barrywgrant: beuno just today told me i could jfdi (rename those things :)00:22
jmleither that, or sabotage barry's bass so he gets some more free time00:22
barryjml: that is so cruel00:22
wgrantHeh.00:22
jmlwgrant, all you need to do is add an extra string -- it'll baffle him00:22
jmlbarry, <300:22
barryjml: ouch :)00:23
barrydoes that mean we're done here?00:23
jmlI think so.00:23
barrywgrant, rockstar, mwhudson anything else?00:23
* wgrant can think of nothing more to whinge about.00:23
barry500:24
barry400:24
barry300:24
barry200:24
barry100:24
jmlThunderbirds are go!00:24
barry#endmeeting00:24
MootBotMeeting finished at 18:24.00:24
mwhudsoncheers all00:24
barrythanks guys!  see you back at the farm00:24
jmlbarry, thanks!00:24
lifelessjml: subunit has been a huge help for me in getting statistics00:42
lifelessFWIW00:42
jmllifeless, I'll bet.00:43
jmlthere's probably an interesting paper in 'making a test suite fast again'00:43
lifelessI just submitted a merge for bzr to make its testresults python2.7 shiny and more reusable00:43
lifelessjml: and ./bzr selftest selftest is down to 3.3 seconds on my laptop [by the reporter, so not including test loading time. Thats a TOFIX]00:44
jmllifeless, why are you only measuring the time for 'selftest' tests?00:44
lifelessjml: its a brain food spike00:45
lifelessjml: 'make this little bit of the test suite as lean and mean as possible'00:45
lifelessit was ~ 60 seconds when I started00:45
jmllifeless, makes sense.00:45
lifelessso far I've reduced overall test load time by 10%00:47
lifelessand identified that our base TestCase is about 2000 times more expensive than unittest.TestCase00:48
jmlheh heh00:48
lifelessso theres still lots of fat to go00:48
lifelessthis bit comes first, because fixing other areas will mean changing this bit and that should be a pleasure not a pain00:49
jml*nod*00:51
=== gary-out is now known as gary_poster
=== Ursinha is now known as Ursinha-afk
=== mrevell is now known as mrevell-lunch
=== matsubara-afk is now known as matsubara
=== mrevell-lunch is now known as mrevell
=== cprov-afk is now known as cprov
matsubara#startmeeting16:00
MootBotMeeting started at 10:00. The chair is matsubara.16:00
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]16:00
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.16:00
matsubara[TOPIC] Roll Call16:00
MootBotNew Topic:  Roll Call16:00
matsubaraNot on the Launchpad Dev team? Welcome! Come "me" with the rest of us!16:00
Ursinha-afkme16:00
gary_posterme16:00
intellectronicame16:00
barryme16:00
=== Ursinha-afk is now known as Ursinha
rockstarni!16:00
bigjoolsme16:00
intellectronicaUrsinha: and i thought you're doing your telepathic typing trick again...16:01
Ursinhaintellectronica, hehe16:01
matsubaraChex, hi16:01
Chexmatsubara: hello16:01
matsubaraok, so we're missing stub and henning/danilo16:02
danilosme16:02
matsubarahi danilos16:02
matsubara:-)16:02
danilosmatsubara: no you are not :)16:02
matsubaraso, from now on, you'll be attending the prod meeting, right?16:02
danilosmatsubara: well, not necessarily, let's see :)16:03
matsubaraok, let's move on then16:03
danilosmatsubara: but soon I will be replacing henninge as our QA contact16:03
matsubara[TOPIC] Agenda16:03
MootBotNew Topic:  Agenda16:03
matsubara * Actions from last meeting16:03
matsubara * Oops report & Critical Bugs & Broken scripts16:03
matsubara * Operations report (mthaddon/Chex/spm/mbarnett)16:03
matsubara * DBA report (stub)16:03
matsubara * Proposed items16:03
matsubaradanilos, ok16:03
matsubara[TOPIC] * Actions from last meeting16:03
MootBotNew Topic:  * Actions from last meeting16:03
matsubara  * cprov to follow up on bug 41374916:03
matsubara  * stub to investigate garbo-hourly failure after spm adjusted script checking to 12h16:03
matsubara  * sinzui to poke barry about ExpatError OOPSes (bug 403606)16:03
matsubara  * danilos, Ursinha and matsubara to discuss oops summaries split per team16:03
ubottuBug 413749 on http://launchpad.net/bugs/413749 is private16:03
ubottuLaunchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/40360616:03
danilosmatsubara: we kind of did that, flacoste is involved as well, and it's all on track I'd say :)16:04
matsubaraok, I know that stub replied to the garbo-hourly failure email, so I think that's done16:04
matsubaradanilos, right. if you have more suggestions let me know16:04
danilossinzui is on unavailable I believe16:04
matsubarabarry, hi!16:04
* barry has some thoughts about the expat errors16:04
barrymatsubara: hi!16:05
matsubarabarry, what's up with those expat errors?16:05
cprovmatsubara: I did reply on 413749 as well, needs discussion, though.16:05
matsubarathanks cprov16:05
Ursinhamatsubara, the one with the expaterrors I'd bring in the oops section16:06
barrymatsubara: well, some more investigation might be warranted, but they are intermittent and i've always chalked them up to problems with xmlrpc.  i believe they can be safely ignored though because the system will "catch up" the next time mailman talks to xmlrpc server.16:06
Ursinhabarry, here's the problem, they happen every day16:06
barrymatsubara: ultimately, we should get rid of the xmlrpc stuff.  it was a solution to a problem we no longer have (namely gpl pollution between mailman and launchpad)16:06
matsubaraif it's safe to ignore them, can we add some code to catch the exception and not log the oops?16:06
matsubarahi stub16:07
Ursinhayou said in the bug report that we have only a handful a week, but I see them everyday, and their numbers are the highest count16:07
barryUrsinha: i get many oops reports for mailman that are clean.  am i looking at the wrong reports?16:07
Ursinhabarry, I think so :)16:07
Ursinhalook at the lpnet reports16:07
barryUrsinha: meaning the daily emails16:07
barryhmm.  okay.  i see them only rarely in the mailman daily email.  i will double check next time i see the report, but i think ignoring the error is the right thing to do16:08
Ursinhabarry, ok. it floods the summary, and that's annoying16:09
barryUrsinha: gotcha!  i hadn't realized that.  i'll do something about it then16:09
gary_posterWhen I discussed hiding the problems, given the frequency, Francis suggested that we not hide them but investigate a bit.  Maybe a small timebox would be appropriate?16:10
Ursinhathaaaanks muchly barry!16:10
gary_postereh, nm16:10
gary_poster:-)16:10
Ursinhagary_poster, :)16:10
barrygary_poster: yes, we can do that.  it will take losa intervention and some cowboy debugging, but it's probably worth it16:10
gary_posterok cool16:11
matsubara[action] barry to continue debug on bug 40360616:11
MootBotACTION received:  barry to continue debug on bug 40360616:11
ubottuLaunchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/40360616:11
matsubarathanks barry, gary_poster, Ursinha16:11
Ursinhathanks16:11
matsubaralet's move on16:11
matsubara[TOPIC] * Oops report & Critical Bugs & Broken scripts16:11
MootBotNew Topic:  * Oops report & Critical Bugs & Broken scripts16:11
Ursinhaok16:11
matsubaraUrsinha, take the stage please16:12
Ursinhaso the ExpatErrors were already discussed here16:12
UrsinhaI have only one bug16:12
Ursinhaintellectronica, bug 40873816:12
ubottuLaunchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/40873816:12
Ursinhathat's another one that happens pretty often16:12
intellectronicareally? if so then we should at least recover gracefully from it16:13
Ursinhaintellectronica, I see those everyday as well16:13
intellectronicai mean in addition to figuring out why we have records like those16:13
UrsinhaI agree16:14
Ursinhaintellectronica, could you take a look or poke someone to do so, please?16:14
intellectronicayes yes16:15
Ursinhathanks intellectronica :)16:15
Ursinhamatsubara, [action] intellectronica to take a look or find someone to take a look on bug 40873816:15
ubottuLaunchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/40873816:15
matsubara[action] intellectronica to take a look or find someone to take a look on bug 40873816:16
MootBotACTION received:  intellectronica to take a look or find someone to take a look on bug 40873816:16
Ursinhathanks :)16:16
matsubarathanks Ursinha and intellectronica16:16
matsubarawe have 2 critical bugs16:16
Ursinhacritical bugs are fix committed, and the process-pending-packagediffs script failure was already explained by cprov16:16
Ursinhaso, we're done!16:16
matsubaraboth fix committed16:16
Ursinhayou can move on, matsubara16:16
Ursinhathanks guys!16:16
matsubarathanks everyone16:16
matsubara[TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)16:16
MootBotNew Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)16:16
Chexhello everyone :16:17
Ursinhahi Chex :)16:17
Chexthe report for this week:16:17
ChexLots more cherry picks this week16:17
ChexMore and more problems with codebrowse (per the Incident Log)16:17
ChexNew losas getting up to speed (Chex and mbarnett)16:17
ChexTesting tarmac for oops-tools project - making progress but not quite there yet16:17
ubottuhttps://lp-oops.canonical.com/oops.py/?oopsid=tools16:17
ChexWill begin process of implementing recommendations from SplitIt Sprint before too long16:17
ChexOne of our top priority bugs is bug 416990 - others per https://bugs.launchpad.net/~canonical-losas16:17
ChexThat's it unless there are any questions16:17
ubottuLaunchpad bug 416990 in bzr "Memory usage of codehosting processes is excessive" [High,Confirmed] https://launchpad.net/bugs/41699016:17
matsubaraChex, re: tarmac setup for oops-tools, I'll submit a merge proposal to tarmac today with the remaining fixes needed.16:19
ubottuhttps://lp-oops.canonical.com/oops.py/?oopsid=tools16:19
matsubaraChex, I'll coordinate with mthaddon later on16:19
mthaddoncool16:19
Chexmatsubara: ok, great16:20
matsubararockstar, can you poke people in your team about the codehosting memory issue?16:20
danilosmatsubara: is it worth fixing ubottu not to choke on oops-tools mention? ;)16:20
ubottuhttps://lp-oops.canonical.com/oops.py/?oopsid=tools16:20
Ursinhalol16:21
matsubaraI can ask the maintainer16:21
danilosmatsubara: thanks :)16:21
matsubara[action] matsubara to chase ubottu maintainer to fix regexp used to identify oopses16:21
MootBotACTION received:  matsubara to chase ubottu maintainer to fix regexp used to identify oopses16:21
matsubaralet's move on16:22
matsubarathanks Chex16:22
matsubara[TOPIC] * DBA report (stub)16:22
MootBotNew Topic:  * DBA report (stub)16:22
stubThe long running transaction killer is back set to 3 hours, thanks to the Rosetta team sorting out the language pack export issue quickly.16:22
stuboot16:22
Ursinhathat was fast16:22
matsubarahehe16:23
bigjoolslike mrevell, it was short16:23
matsubarathanks stub and rosetta people :-)16:23
matsubaralol16:23
Ursinhalol16:23
matsubara[TOPIC] * Proposed items16:23
MootBotNew Topic:  * Proposed items16:23
matsubarawe have no proposed items16:23
mrevellhaha bigjools16:23
matsubaraanything else before I close?16:23
matsubarahi mrevell16:23
matsubaraThank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.16:24
mrevellmatsubara: nothing from me, just responding to bigjools... :)16:24
matsubara#endmeeting16:24
MootBotMeeting finished at 10:24.16:24
Ursinhathanks everyone!16:24
bigjoolsmrevell: I apologise profusely, please don't hurt me16:25
mrevellbigjools: Ah, ok, just this once16:27
danilosthanks matsubara, all16:27
Ursinhaoops-thisisnotright16:28
ubottuhttps://lp-oops.canonical.com/oops.py/?oopsid=thisisnotright16:28
danilosUrsinha: come on, come, don't be too hard on it16:28
Ursinha:P16:29
bigjoolsoops-I-did-it-again16:47
ubottuhttps://lp-oops.canonical.com/oops.py/?oopsid=I16:47
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
=== matsubara is now known as matsubara-lunch
=== matsubara-lunch is now known as matsubara
=== Ursinha is now known as Ursinha-afk
=== EdwinGrubbs is now known as Edwin-lunch
=== Ursinha-afk is now known as Ursinha
=== salgado is now known as salgado-afk
=== Edwin-lunch is now known as EdwinGrubbs
=== cprov is now known as cprov-afk
=== matsubara is now known as matsubara-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!