[00:00] <barry> mwhudson: heh
[00:00]  * barry laughs when he hears of people complaining about a few hundred messages a day
[00:00]  * jml thinks...
[00:01] <jml> any feedback from non-Canonical contributors about the review process?
[00:01]  * barry looks at wgrant
[00:01] <mwhudson> i guess i can take the opportunity to do a little survey
[00:01] <wgrant> barry: Working well for me.
[00:02] <barry> i'll just say again that it is /fantastic/ to see yours and others from the community landing branches
[00:02] <wgrant> It's great to finally be able to!
[00:03] <mwhudson> as build engineer for the next cycle, what's causing you, the developer, the most pain?
[00:03] <wgrant> My only slight issue is that sponsors are often unclear on how to do things.
[00:03] <wgrant> Is this not documented?
[00:03] <wgrant> mwhudson: 'build engineer'?
[00:03] <jml> wasn't that announcement on launchpad-dev!
[00:03] <wgrant> No. We don't find anything out.
[00:03] <barry> wgrant: can you elaborate?
[00:03] <mwhudson> bah
[00:04] <mwhudson> and the wiki page is internal
[00:04]  * jml frowns
[00:04] <mwhudson> still learning at this sort of thing it seems :)
[00:04] <barry> yep.  old habits die hard, but we're really trying to move more and more resources, communications, etc. into the public square
[00:04] <wgrant> barry: Well, some of my sponsors have had trouble working out how to ec2test my branches. I'm not sure why.
[00:05] <wgrant> What was this email meant to contain?
[00:05] <barry> mwhudson: we really should find a way for non-Canonicals to run ec2
[00:05] <jml> we're going to have a build engineer position
[00:05] <wgrant> barry: +inf
[00:05] <jml> it's a rotating postiition
[00:05] <mwhudson> "Each development cycle, a different engineer will be taken off his normal development duty, and have has main focus for the cycle the development and well being of the build system. "
[00:05] <jml> the build engineer works on making developers lives more pleasant, essentially
[00:06] <wgrant> Aha. I see.
[00:06] <mwhudson> i'm the first one, starting officially on monday
[00:06] <jml> mwhudson, in answer to your question
[00:06] <mwhudson> i'm probably going to tackle some buildbot annoyances first
[00:06] <mwhudson> jml: you sent me an email about this that i haven't read yet :)
[00:06] <wgrant> mwhudson: Bonus points for making it public.
[00:06] <mwhudson> wgrant: yeah, that should be possible
[00:07] <jml> mwhudson, my email wasn't so much about my pain points, actually :)
[00:08] <jml> mwhudson, it takes way too long to land branches; buildbot should only send emails that require action; test failures in devel break ec2test runs
[00:08] <barry> jml, mwhudson what is the main hold up to allowing the public to run ec2?  is it the payment issue?
[00:08] <mwhudson> barry: i imagine so
[00:08] <mwhudson> also ec2test is still private i think, but that will change soon (?)
[00:09] <jml> mwhudson, not a pain point for me, but one I've had to watch: it takes too long to get set up with a dev env
[00:09] <barry> surely we can find a way to subsidize the ec2 costs.  it's gotta be trivial in the scheme of things.  i usually feel guilty expensing it every quarter
[00:09] <jml> mwhudson, the main meta issue is that it's really hard to see what the status of various build-related projects are (e.g. buildbot going public, ec2test going public, etc)
[00:10] <rockstar> barry, when we have a UEC set up in the data center, it might be easier.  *shrug*
[00:10] <barry> rockstar: good point
[00:10] <mwhudson> jml: i think A build engineer, though probably not me, should focus on getting buildbot running the test suite in parallel across several instances
[00:10] <jml> mwhudson, I very strongly agree.
[00:11] <rockstar> mwhudson, abentley and I had a discussion about test layers today, and whether or not we're really using the layers properly.
[00:11] <rockstar> I find I spend a lot of time setting up and tearing down the test harnesses just to run one test (that usually runs in shorter time than set up or tear down.
[00:11] <mwhudson> rockstar: i would claim that using layers correctly is close to impossible, but maybe that's a bit bigoted
[00:12] <barry> rockstar: i can almost bet we're not.  the layers are horribly named and it is not intuitive to know what layer a test should go in
[00:12] <mwhudson> rockstar: there's also the terrible way we start daemons in tests, that doesn't help
[00:12] <rockstar> barry, yeah, GoogleServiceLayer and LibrarianLayer were two we identified as not really necessary for most of our tests.
[00:12] <jml> switching to testresources would help here
[00:12] <mwhudson> +1 jml
[00:12] <jml> I'm a little terrified about the work involved
[00:12] <rockstar> jml, +1
[00:12] <barry> rockstar: and they take a long time to start
[00:13] <jml> but maybe if we actually explored what needed to happen, it would be less terrifying
[00:13] <rockstar> barry, yeah, which is basically what I spent much of yesterday doing.
[00:13] <wgrant> librarian takes forever!
[00:13] <jml> wgrant, indeed.
[00:13] <rockstar> wgrant, this morning it took 15.391 seconds to start up, and 7.3 seconds to tear down the librarian.
[00:13] <jml> mwhudson, so, that's not really a survey, since it's just me :)
[00:14] <wgrant> rockstar: It's about twice as fast here, most of the time.
[00:14] <rockstar> mwhudson, really, if I had a pony request, it'd be a graph for how long the test suite is taking, and maybe some mini graphs of different sections of the test.
[00:14] <mwhudson> for all that, step 1 is still going to be force build on buildbot
[00:14] <mwhudson> rockstar: that's a good idea
[00:14]  * jml agrees -- it's in an email I sent :)
[00:15] <rockstar> jml, dammit.  You and your organization...
[00:15] <rockstar> Do we have a place to aggregate all of these ideas?
[00:15] <jml> exactly!
[00:16] <rockstar> jml, did you send an email about that too?
[00:16] <jml> I was about to say, step 0 should really be tracking this.
[00:16] <barry> rockstar: like a wiki? <wink>
[00:16] <jml> rockstar, it was in the same email as the graph stuff :)
[00:16] <rockstar> barry, wikis are TERRIBLE issue trackers.
[00:16] <jml> barry, or, umm, a bug tracker.
[00:16] <jml> barry, I've got this good one I can sell you
[00:16] <rockstar> I know these really great guys who spend a lot of time writing a bug tracker...
[00:16] <mwhudson> step -1: move BE wiki pages to dev.lp.net
[00:16] <barry> :)
[00:17] <jml> mwhudson, \o/
[00:17] <mwhudson> step 0: a tag in launchpad-project, i guess
[00:17] <rockstar> Maybe a product in launchpad-project.
[00:17] <wgrant> Wouldn't a separate product be much better?
[00:17] <jml> rockstar, wgrant, maaybe
[00:18] <jml> I don't think that the way Launchpad uses products is the way Launchpad wants others to use products though
[00:18]  * jml makes a note about that
[00:18] <wgrant> No.
[00:18] <wgrant> But this one is different.
[00:19] <jml> wgrant, yeah, I think so too, but I'm not sure how it's different.
[00:19] <wgrant> jml: It's not part of the codebase.
[00:19] <jml> wgrant, some of the bugs are in the code base
[00:19] <wgrant> Most of the other products are.
[00:19] <wgrant> Hmm.
[00:19] <jml> wgrant, but a subproduct would fit there, I guess
[00:20]  * mwhudson writes another thing down, "make sure buildbot config is in a branch or two on launchpad"
[00:20] <jml> the secret, proprietary project with all of our developer utilities might well belong as a public sub-project of Launchpad
[00:20] <barry> jml: +1
[00:20] <wgrant> jml: lp-dev-utils, or is there something else?
[00:20] <jml> wgrant, lp-dev-utils, yes.
[00:21] <jml> wgrant, that is, I can neither confirm nor deny the existence of said project...
[00:21] <wgrant> Heh.
[00:21]  * wgrant whines about the product/project/project/projectgroup confusion.
[00:21] <jml> patches accepted :)
[00:22] <barry> wgrant: beuno just today told me i could jfdi (rename those things :)
[00:22] <jml> either that, or sabotage barry's bass so he gets some more free time
[00:22] <barry> jml: that is so cruel
[00:22] <wgrant> Heh.
[00:22] <jml> wgrant, all you need to do is add an extra string -- it'll baffle him
[00:22] <jml> barry, <3
[00:23] <barry> jml: ouch :)
[00:23] <barry> does that mean we're done here?
[00:23] <jml> I think so.
[00:23] <barry> wgrant, rockstar, mwhudson anything else?
[00:23]  * wgrant can think of nothing more to whinge about.
[00:24] <barry> 5
[00:24] <barry> 4
[00:24] <barry> 3
[00:24] <barry> 2
[00:24] <barry> 1
[00:24] <jml> Thunderbirds are go!
[00:24] <barry> #endmeeting
[00:24] <MootBot> Meeting finished at 18:24.
[00:24] <mwhudson> cheers all
[00:24] <barry> thanks guys!  see you back at the farm
[00:24] <jml> barry, thanks!
[00:42] <lifeless> jml: subunit has been a huge help for me in getting statistics
[00:42] <lifeless> FWIW
[00:43] <jml> lifeless, I'll bet.
[00:43] <jml> there's probably an interesting paper in 'making a test suite fast again'
[00:43] <lifeless> I just submitted a merge for bzr to make its testresults python2.7 shiny and more reusable
[00:44] <lifeless> jml: and ./bzr selftest selftest is down to 3.3 seconds on my laptop [by the reporter, so not including test loading time. Thats a TOFIX]
[00:44] <jml> lifeless, why are you only measuring the time for 'selftest' tests?
[00:45] <lifeless> jml: its a brain food spike
[00:45] <lifeless> jml: 'make this little bit of the test suite as lean and mean as possible'
[00:45] <lifeless> it was ~ 60 seconds when I started
[00:45] <jml> lifeless, makes sense.
[00:47] <lifeless> so far I've reduced overall test load time by 10%
[00:48] <lifeless> and identified that our base TestCase is about 2000 times more expensive than unittest.TestCase
[00:48] <jml> heh heh
[00:48] <lifeless> so theres still lots of fat to go
[00:49] <lifeless> this bit comes first, because fixing other areas will mean changing this bit and that should be a pleasure not a pain
[00:51] <jml> *nod*
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:00] <Ursinha-afk> me
[16:00] <gary_poster> me
[16:00] <intellectronica> me
[16:00] <barry> me
[16:00] <rockstar> ni!
[16:00] <bigjools> me
[16:01] <intellectronica> Ursinha: and i thought you're doing your telepathic typing trick again...
[16:01] <Ursinha> intellectronica, hehe
[16:01] <matsubara> Chex, hi
[16:01] <Chex> matsubara: hello
[16:02] <matsubara> ok, so we're missing stub and henning/danilo
[16:02] <danilos> me
[16:02] <matsubara> hi danilos
[16:02] <matsubara> :-)
[16:02] <danilos> matsubara: no you are not :)
[16:02] <matsubara> so, from now on, you'll be attending the prod meeting, right?
[16:03] <danilos> matsubara: well, not necessarily, let's see :)
[16:03] <matsubara> ok, let's move on then
[16:03] <danilos> matsubara: but soon I will be replacing henninge as our QA contact
[16:03] <matsubara> [TOPIC] Agenda
[16:03] <MootBot> New Topic:  Agenda
[16:03] <matsubara>  * Actions from last meeting
[16:03] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:03] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:03] <matsubara>  * DBA report (stub)
[16:03] <matsubara>  * Proposed items
[16:03] <matsubara> danilos, ok
[16:03] <matsubara> [TOPIC] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:03] <matsubara>   * cprov to follow up on bug 413749
[16:03] <matsubara>   * stub to investigate garbo-hourly failure after spm adjusted script checking to 12h
[16:03] <matsubara>   * sinzui to poke barry about ExpatError OOPSes (bug 403606)
[16:03] <matsubara>   * danilos, Ursinha and matsubara to discuss oops summaries split per team
[16:04] <danilos> matsubara: we kind of did that, flacoste is involved as well, and it's all on track I'd say :)
[16:04] <matsubara> ok, I know that stub replied to the garbo-hourly failure email, so I think that's done
[16:04] <matsubara> danilos, right. if you have more suggestions let me know
[16:04] <danilos> sinzui is on unavailable I believe
[16:04] <matsubara> barry, hi!
[16:04]  * barry has some thoughts about the expat errors
[16:05] <barry> matsubara: hi!
[16:05] <matsubara> barry, what's up with those expat errors?
[16:05] <cprov> matsubara: I did reply on 413749 as well, needs discussion, though.
[16:05] <matsubara> thanks cprov
[16:06] <Ursinha> matsubara, the one with the expaterrors I'd bring in the oops section
[16:06] <barry> matsubara: well, some more investigation might be warranted, but they are intermittent and i've always chalked them up to problems with xmlrpc.  i believe they can be safely ignored though because the system will "catch up" the next time mailman talks to xmlrpc server.
[16:06] <Ursinha> barry, here's the problem, they happen every day
[16:06] <barry> matsubara: ultimately, we should get rid of the xmlrpc stuff.  it was a solution to a problem we no longer have (namely gpl pollution between mailman and launchpad)
[16:06] <matsubara> if it's safe to ignore them, can we add some code to catch the exception and not log the oops?
[16:07] <matsubara> hi stub
[16:07] <Ursinha> you said in the bug report that we have only a handful a week, but I see them everyday, and their numbers are the highest count
[16:07] <barry> Ursinha: i get many oops reports for mailman that are clean.  am i looking at the wrong reports?
[16:07] <Ursinha> barry, I think so :)
[16:07] <Ursinha> look at the lpnet reports
[16:07] <barry> Ursinha: meaning the daily emails
[16:08] <barry> hmm.  okay.  i see them only rarely in the mailman daily email.  i will double check next time i see the report, but i think ignoring the error is the right thing to do
[16:09] <Ursinha> barry, ok. it floods the summary, and that's annoying
[16:09] <barry> Ursinha: gotcha!  i hadn't realized that.  i'll do something about it then
[16:10] <gary_poster> When I discussed hiding the problems, given the frequency, Francis suggested that we not hide them but investigate a bit.  Maybe a small timebox would be appropriate?
[16:10] <Ursinha> thaaaanks muchly barry!
[16:10] <gary_poster> eh, nm
[16:10] <gary_poster> :-)
[16:10] <Ursinha> gary_poster, :)
[16:10] <barry> gary_poster: yes, we can do that.  it will take losa intervention and some cowboy debugging, but it's probably worth it
[16:11] <gary_poster> ok cool
[16:11] <matsubara> [action] barry to continue debug on bug 403606
[16:11] <MootBot> ACTION received:  barry to continue debug on bug 403606
[16:11] <matsubara> thanks barry, gary_poster, Ursinha
[16:11] <Ursinha> thanks
[16:11] <matsubara> let's move on
[16:11] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:11] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:11] <Ursinha> ok
[16:12] <matsubara> Ursinha, take the stage please
[16:12] <Ursinha> so the ExpatErrors were already discussed here
[16:12] <Ursinha> I have only one bug
[16:12] <Ursinha> intellectronica, bug 408738
[16:12] <Ursinha> that's another one that happens pretty often
[16:13] <intellectronica> really? if so then we should at least recover gracefully from it
[16:13] <Ursinha> intellectronica, I see those everyday as well
[16:13] <intellectronica> i mean in addition to figuring out why we have records like those
[16:14] <Ursinha> I agree
[16:14] <Ursinha> intellectronica, could you take a look or poke someone to do so, please?
[16:15] <intellectronica> yes yes
[16:15] <Ursinha> thanks intellectronica :)
[16:15] <Ursinha> matsubara, [action] intellectronica to take a look or find someone to take a look on bug 408738
[16:16] <matsubara> [action] intellectronica to take a look or find someone to take a look on bug 408738
[16:16] <MootBot> ACTION received:  intellectronica to take a look or find someone to take a look on bug 408738
[16:16] <Ursinha> thanks :)
[16:16] <matsubara> thanks Ursinha and intellectronica
[16:16] <matsubara> we have 2 critical bugs
[16:16] <Ursinha> critical bugs are fix committed, and the process-pending-packagediffs script failure was already explained by cprov
[16:16] <Ursinha> so, we're done!
[16:16] <matsubara> both fix committed
[16:16] <Ursinha> you can move on, matsubara
[16:16] <Ursinha> thanks guys!
[16:16] <matsubara> thanks everyone
[16:16] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:16] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:17] <Chex> hello everyone :
[16:17] <Ursinha> hi Chex :)
[16:17] <Chex> the report for this week:
[16:17] <Chex> Lots more cherry picks this week
[16:17] <Chex> More and more problems with codebrowse (per the Incident Log)
[16:17] <Chex> New losas getting up to speed (Chex and mbarnett)
[16:17] <Chex> Testing tarmac for oops-tools project - making progress but not quite there yet
[16:17] <Chex> Will begin process of implementing recommendations from SplitIt Sprint before too long
[16:17] <Chex> One of our top priority bugs is bug 416990 - others per https://bugs.launchpad.net/~canonical-losas
[16:17] <Chex> That's it unless there are any questions
[16:19] <matsubara> Chex, re: tarmac setup for oops-tools, I'll submit a merge proposal to tarmac today with the remaining fixes needed.
[16:19] <matsubara> Chex, I'll coordinate with mthaddon later on
[16:19] <mthaddon> cool
[16:20] <Chex> matsubara: ok, great
[16:20] <matsubara> rockstar, can you poke people in your team about the codehosting memory issue?
[16:20] <danilos> matsubara: is it worth fixing ubottu not to choke on oops-tools mention? ;)
[16:21] <Ursinha> lol
[16:21] <matsubara> I can ask the maintainer
[16:21] <danilos> matsubara: thanks :)
[16:21] <matsubara> [action] matsubara to chase ubottu maintainer to fix regexp used to identify oopses
[16:21] <MootBot> ACTION received:  matsubara to chase ubottu maintainer to fix regexp used to identify oopses
[16:22] <matsubara> let's move on
[16:22] <matsubara> thanks Chex
[16:22] <matsubara> [TOPIC] * DBA report (stub)
[16:22] <MootBot> New Topic:  * DBA report (stub)
[16:22] <stub> The long running transaction killer is back set to 3 hours, thanks to the Rosetta team sorting out the language pack export issue quickly.
[16:22] <stub> oot
[16:22] <Ursinha> that was fast
[16:23] <matsubara> hehe
[16:23] <bigjools> like mrevell, it was short
[16:23] <matsubara> thanks stub and rosetta people :-)
[16:23] <matsubara> lol
[16:23] <Ursinha> lol
[16:23] <matsubara> [TOPIC] * Proposed items
[16:23] <MootBot> New Topic:  * Proposed items
[16:23] <matsubara> we have no proposed items
[16:23] <mrevell> haha bigjools
[16:23] <matsubara> anything else before I close?
[16:23] <matsubara> hi mrevell
[16:24] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[16:24] <mrevell> matsubara: nothing from me, just responding to bigjools... :)
[16:24] <matsubara> #endmeeting
[16:24] <MootBot> Meeting finished at 10:24.
[16:24] <Ursinha> thanks everyone!
[16:25] <bigjools> mrevell: I apologise profusely, please don't hurt me
[16:27] <mrevell> bigjools: Ah, ok, just this once
[16:27] <danilos> thanks matsubara, all
[16:28] <Ursinha> oops-thisisnotright
[16:28] <danilos> Ursinha: come on, come, don't be too hard on it
[16:29] <Ursinha> :P
[16:47] <bigjools> oops-I-did-it-again