[00:00] mwhudson: heh [00:00] * barry laughs when he hears of people complaining about a few hundred messages a day [00:00] * jml thinks... [00:01] any feedback from non-Canonical contributors about the review process? [00:01] * barry looks at wgrant [00:01] i guess i can take the opportunity to do a little survey [00:01] barry: Working well for me. [00:02] i'll just say again that it is /fantastic/ to see yours and others from the community landing branches [00:02] It's great to finally be able to! [00:03] as build engineer for the next cycle, what's causing you, the developer, the most pain? [00:03] My only slight issue is that sponsors are often unclear on how to do things. [00:03] Is this not documented? [00:03] mwhudson: 'build engineer'? [00:03] wasn't that announcement on launchpad-dev! [00:03] No. We don't find anything out. [00:03] wgrant: can you elaborate? [00:03] bah [00:04] and the wiki page is internal [00:04] * jml frowns [00:04] still learning at this sort of thing it seems :) [00:04] yep. old habits die hard, but we're really trying to move more and more resources, communications, etc. into the public square [00:04] barry: Well, some of my sponsors have had trouble working out how to ec2test my branches. I'm not sure why. [00:05] What was this email meant to contain? [00:05] mwhudson: we really should find a way for non-Canonicals to run ec2 [00:05] we're going to have a build engineer position [00:05] barry: +inf [00:05] it's a rotating postiition [00:05] "Each development cycle, a different engineer will be taken off his normal development duty, and have has main focus for the cycle the development and well being of the build system. " [00:05] the build engineer works on making developers lives more pleasant, essentially [00:06] Aha. I see. [00:06] i'm the first one, starting officially on monday [00:06] mwhudson, in answer to your question [00:06] i'm probably going to tackle some buildbot annoyances first [00:06] jml: you sent me an email about this that i haven't read yet :) [00:06] mwhudson: Bonus points for making it public. [00:06] wgrant: yeah, that should be possible [00:07] mwhudson, my email wasn't so much about my pain points, actually :) [00:08] mwhudson, it takes way too long to land branches; buildbot should only send emails that require action; test failures in devel break ec2test runs [00:08] jml, mwhudson what is the main hold up to allowing the public to run ec2? is it the payment issue? [00:08] barry: i imagine so [00:08] also ec2test is still private i think, but that will change soon (?) [00:09] mwhudson, not a pain point for me, but one I've had to watch: it takes too long to get set up with a dev env [00:09] surely we can find a way to subsidize the ec2 costs. it's gotta be trivial in the scheme of things. i usually feel guilty expensing it every quarter [00:09] mwhudson, the main meta issue is that it's really hard to see what the status of various build-related projects are (e.g. buildbot going public, ec2test going public, etc) [00:10] barry, when we have a UEC set up in the data center, it might be easier. *shrug* [00:10] rockstar: good point [00:10] jml: i think A build engineer, though probably not me, should focus on getting buildbot running the test suite in parallel across several instances [00:10] mwhudson, I very strongly agree. [00:11] mwhudson, abentley and I had a discussion about test layers today, and whether or not we're really using the layers properly. [00:11] I find I spend a lot of time setting up and tearing down the test harnesses just to run one test (that usually runs in shorter time than set up or tear down. [00:11] rockstar: i would claim that using layers correctly is close to impossible, but maybe that's a bit bigoted [00:12] rockstar: i can almost bet we're not. the layers are horribly named and it is not intuitive to know what layer a test should go in [00:12] rockstar: there's also the terrible way we start daemons in tests, that doesn't help [00:12] barry, yeah, GoogleServiceLayer and LibrarianLayer were two we identified as not really necessary for most of our tests. [00:12] switching to testresources would help here [00:12] +1 jml [00:12] I'm a little terrified about the work involved [00:12] jml, +1 [00:12] rockstar: and they take a long time to start [00:13] but maybe if we actually explored what needed to happen, it would be less terrifying [00:13] barry, yeah, which is basically what I spent much of yesterday doing. [00:13] librarian takes forever! [00:13] wgrant, indeed. [00:13] wgrant, this morning it took 15.391 seconds to start up, and 7.3 seconds to tear down the librarian. [00:13] mwhudson, so, that's not really a survey, since it's just me :) [00:14] rockstar: It's about twice as fast here, most of the time. [00:14] mwhudson, really, if I had a pony request, it'd be a graph for how long the test suite is taking, and maybe some mini graphs of different sections of the test. [00:14] for all that, step 1 is still going to be force build on buildbot [00:14] rockstar: that's a good idea [00:14] * jml agrees -- it's in an email I sent :) [00:15] jml, dammit. You and your organization... [00:15] Do we have a place to aggregate all of these ideas? [00:15] exactly! [00:16] jml, did you send an email about that too? [00:16] I was about to say, step 0 should really be tracking this. [00:16] rockstar: like a wiki? [00:16] rockstar, it was in the same email as the graph stuff :) [00:16] barry, wikis are TERRIBLE issue trackers. [00:16] barry, or, umm, a bug tracker. [00:16] barry, I've got this good one I can sell you [00:16] I know these really great guys who spend a lot of time writing a bug tracker... [00:16] step -1: move BE wiki pages to dev.lp.net [00:16] :) [00:17] mwhudson, \o/ [00:17] step 0: a tag in launchpad-project, i guess [00:17] Maybe a product in launchpad-project. [00:17] Wouldn't a separate product be much better? [00:17] rockstar, wgrant, maaybe [00:18] I don't think that the way Launchpad uses products is the way Launchpad wants others to use products though [00:18] * jml makes a note about that [00:18] No. [00:18] But this one is different. [00:19] wgrant, yeah, I think so too, but I'm not sure how it's different. [00:19] jml: It's not part of the codebase. [00:19] wgrant, some of the bugs are in the code base [00:19] Most of the other products are. [00:19] Hmm. [00:19] wgrant, but a subproduct would fit there, I guess [00:20] * mwhudson writes another thing down, "make sure buildbot config is in a branch or two on launchpad" [00:20] the secret, proprietary project with all of our developer utilities might well belong as a public sub-project of Launchpad [00:20] jml: +1 [00:20] jml: lp-dev-utils, or is there something else? [00:20] wgrant, lp-dev-utils, yes. [00:21] wgrant, that is, I can neither confirm nor deny the existence of said project... [00:21] Heh. [00:21] * wgrant whines about the product/project/project/projectgroup confusion. [00:21] patches accepted :) [00:22] wgrant: beuno just today told me i could jfdi (rename those things :) [00:22] either that, or sabotage barry's bass so he gets some more free time [00:22] jml: that is so cruel [00:22] Heh. [00:22] wgrant, all you need to do is add an extra string -- it'll baffle him [00:22] barry, <3 [00:23] jml: ouch :) [00:23] does that mean we're done here? [00:23] I think so. [00:23] wgrant, rockstar, mwhudson anything else? [00:23] * wgrant can think of nothing more to whinge about. [00:24] 5 [00:24] 4 [00:24] 3 [00:24] 2 [00:24] 1 [00:24] Thunderbirds are go! [00:24] #endmeeting [00:24] Meeting finished at 18:24. [00:24] cheers all [00:24] thanks guys! see you back at the farm [00:24] barry, thanks! [00:42] jml: subunit has been a huge help for me in getting statistics [00:42] FWIW [00:43] lifeless, I'll bet. [00:43] there's probably an interesting paper in 'making a test suite fast again' [00:43] I just submitted a merge for bzr to make its testresults python2.7 shiny and more reusable [00:44] jml: and ./bzr selftest selftest is down to 3.3 seconds on my laptop [by the reporter, so not including test loading time. Thats a TOFIX] [00:44] lifeless, why are you only measuring the time for 'selftest' tests? [00:45] jml: its a brain food spike [00:45] jml: 'make this little bit of the test suite as lean and mean as possible' [00:45] it was ~ 60 seconds when I started [00:45] lifeless, makes sense. [00:47] so far I've reduced overall test load time by 10% [00:48] and identified that our base TestCase is about 2000 times more expensive than unittest.TestCase [00:48] heh heh [00:48] so theres still lots of fat to go [00:49] this bit comes first, because fixing other areas will mean changing this bit and that should be a pleasure not a pain [00:51] *nod* === gary-out is now known as gary_poster === Ursinha is now known as Ursinha-afk === mrevell is now known as mrevell-lunch === matsubara-afk is now known as matsubara === mrevell-lunch is now known as mrevell === cprov-afk is now known as cprov [16:00] #startmeeting [16:00] Meeting started at 10:00. The chair is matsubara. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] [TOPIC] Roll Call [16:00] New Topic: Roll Call [16:00] Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! [16:00] me [16:00] me [16:00] me [16:00] me === Ursinha-afk is now known as Ursinha [16:00] ni! [16:00] me [16:01] Ursinha: and i thought you're doing your telepathic typing trick again... [16:01] intellectronica, hehe [16:01] Chex, hi [16:01] matsubara: hello [16:02] ok, so we're missing stub and henning/danilo [16:02] me [16:02] hi danilos [16:02] :-) [16:02] matsubara: no you are not :) [16:02] so, from now on, you'll be attending the prod meeting, right? [16:03] matsubara: well, not necessarily, let's see :) [16:03] ok, let's move on then [16:03] matsubara: but soon I will be replacing henninge as our QA contact [16:03] [TOPIC] Agenda [16:03] New Topic: Agenda [16:03] * Actions from last meeting [16:03] * Oops report & Critical Bugs & Broken scripts [16:03] * Operations report (mthaddon/Chex/spm/mbarnett) [16:03] * DBA report (stub) [16:03] * Proposed items [16:03] danilos, ok [16:03] [TOPIC] * Actions from last meeting [16:03] New Topic: * Actions from last meeting [16:03] * cprov to follow up on bug 413749 [16:03] * stub to investigate garbo-hourly failure after spm adjusted script checking to 12h [16:03] * sinzui to poke barry about ExpatError OOPSes (bug 403606) [16:03] * danilos, Ursinha and matsubara to discuss oops summaries split per team [16:03] Bug 413749 on http://launchpad.net/bugs/413749 is private [16:03] Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 [16:04] matsubara: we kind of did that, flacoste is involved as well, and it's all on track I'd say :) [16:04] ok, I know that stub replied to the garbo-hourly failure email, so I think that's done [16:04] danilos, right. if you have more suggestions let me know [16:04] sinzui is on unavailable I believe [16:04] barry, hi! [16:04] * barry has some thoughts about the expat errors [16:05] matsubara: hi! [16:05] barry, what's up with those expat errors? [16:05] matsubara: I did reply on 413749 as well, needs discussion, though. [16:05] thanks cprov [16:06] matsubara, the one with the expaterrors I'd bring in the oops section [16:06] matsubara: well, some more investigation might be warranted, but they are intermittent and i've always chalked them up to problems with xmlrpc. i believe they can be safely ignored though because the system will "catch up" the next time mailman talks to xmlrpc server. [16:06] barry, here's the problem, they happen every day [16:06] matsubara: ultimately, we should get rid of the xmlrpc stuff. it was a solution to a problem we no longer have (namely gpl pollution between mailman and launchpad) [16:06] if it's safe to ignore them, can we add some code to catch the exception and not log the oops? [16:07] hi stub [16:07] you said in the bug report that we have only a handful a week, but I see them everyday, and their numbers are the highest count [16:07] Ursinha: i get many oops reports for mailman that are clean. am i looking at the wrong reports? [16:07] barry, I think so :) [16:07] look at the lpnet reports [16:07] Ursinha: meaning the daily emails [16:08] hmm. okay. i see them only rarely in the mailman daily email. i will double check next time i see the report, but i think ignoring the error is the right thing to do [16:09] barry, ok. it floods the summary, and that's annoying [16:09] Ursinha: gotcha! i hadn't realized that. i'll do something about it then [16:10] When I discussed hiding the problems, given the frequency, Francis suggested that we not hide them but investigate a bit. Maybe a small timebox would be appropriate? [16:10] thaaaanks muchly barry! [16:10] eh, nm [16:10] :-) [16:10] gary_poster, :) [16:10] gary_poster: yes, we can do that. it will take losa intervention and some cowboy debugging, but it's probably worth it [16:11] ok cool [16:11] [action] barry to continue debug on bug 403606 [16:11] ACTION received: barry to continue debug on bug 403606 [16:11] Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 [16:11] thanks barry, gary_poster, Ursinha [16:11] thanks [16:11] let's move on [16:11] [TOPIC] * Oops report & Critical Bugs & Broken scripts [16:11] New Topic: * Oops report & Critical Bugs & Broken scripts [16:11] ok [16:12] Ursinha, take the stage please [16:12] so the ExpatErrors were already discussed here [16:12] I have only one bug [16:12] intellectronica, bug 408738 [16:12] Launchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/408738 [16:12] that's another one that happens pretty often [16:13] really? if so then we should at least recover gracefully from it [16:13] intellectronica, I see those everyday as well [16:13] i mean in addition to figuring out why we have records like those [16:14] I agree [16:14] intellectronica, could you take a look or poke someone to do so, please? [16:15] yes yes [16:15] thanks intellectronica :) [16:15] matsubara, [action] intellectronica to take a look or find someone to take a look on bug 408738 [16:15] Launchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/408738 [16:16] [action] intellectronica to take a look or find someone to take a look on bug 408738 [16:16] ACTION received: intellectronica to take a look or find someone to take a look on bug 408738 [16:16] thanks :) [16:16] thanks Ursinha and intellectronica [16:16] we have 2 critical bugs [16:16] critical bugs are fix committed, and the process-pending-packagediffs script failure was already explained by cprov [16:16] so, we're done! [16:16] both fix committed [16:16] you can move on, matsubara [16:16] thanks guys! [16:16] thanks everyone [16:16] [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) [16:16] New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) [16:17] hello everyone : [16:17] hi Chex :) [16:17] the report for this week: [16:17] Lots more cherry picks this week [16:17] More and more problems with codebrowse (per the Incident Log) [16:17] New losas getting up to speed (Chex and mbarnett) [16:17] Testing tarmac for oops-tools project - making progress but not quite there yet [16:17] https://lp-oops.canonical.com/oops.py/?oopsid=tools [16:17] Will begin process of implementing recommendations from SplitIt Sprint before too long [16:17] One of our top priority bugs is bug 416990 - others per https://bugs.launchpad.net/~canonical-losas [16:17] That's it unless there are any questions [16:17] Launchpad bug 416990 in bzr "Memory usage of codehosting processes is excessive" [High,Confirmed] https://launchpad.net/bugs/416990 [16:19] Chex, re: tarmac setup for oops-tools, I'll submit a merge proposal to tarmac today with the remaining fixes needed. [16:19] https://lp-oops.canonical.com/oops.py/?oopsid=tools [16:19] Chex, I'll coordinate with mthaddon later on [16:19] cool [16:20] matsubara: ok, great [16:20] rockstar, can you poke people in your team about the codehosting memory issue? [16:20] matsubara: is it worth fixing ubottu not to choke on oops-tools mention? ;) [16:20] https://lp-oops.canonical.com/oops.py/?oopsid=tools [16:21] lol [16:21] I can ask the maintainer [16:21] matsubara: thanks :) [16:21] [action] matsubara to chase ubottu maintainer to fix regexp used to identify oopses [16:21] ACTION received: matsubara to chase ubottu maintainer to fix regexp used to identify oopses [16:22] let's move on [16:22] thanks Chex [16:22] [TOPIC] * DBA report (stub) [16:22] New Topic: * DBA report (stub) [16:22] The long running transaction killer is back set to 3 hours, thanks to the Rosetta team sorting out the language pack export issue quickly. [16:22] oot [16:22] that was fast [16:23] hehe [16:23] like mrevell, it was short [16:23] thanks stub and rosetta people :-) [16:23] lol [16:23] lol [16:23] [TOPIC] * Proposed items [16:23] New Topic: * Proposed items [16:23] we have no proposed items [16:23] haha bigjools [16:23] anything else before I close? [16:23] hi mrevell [16:24] Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. [16:24] matsubara: nothing from me, just responding to bigjools... :) [16:24] #endmeeting [16:24] Meeting finished at 10:24. [16:24] thanks everyone! [16:25] mrevell: I apologise profusely, please don't hurt me [16:27] bigjools: Ah, ok, just this once [16:27] thanks matsubara, all [16:28] oops-thisisnotright [16:28] https://lp-oops.canonical.com/oops.py/?oopsid=thisisnotright [16:28] Ursinha: come on, come, don't be too hard on it [16:29] :P [16:47] oops-I-did-it-again [16:47] https://lp-oops.canonical.com/oops.py/?oopsid=I === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado === matsubara is now known as matsubara-lunch === matsubara-lunch is now known as matsubara === Ursinha is now known as Ursinha-afk === EdwinGrubbs is now known as Edwin-lunch === Ursinha-afk is now known as Ursinha === salgado is now known as salgado-afk === Edwin-lunch is now known as EdwinGrubbs === cprov is now known as cprov-afk === matsubara is now known as matsubara-afk