| barry | mwhudson: heh | 00:00 |
|---|---|---|
| * barry laughs when he hears of people complaining about a few hundred messages a day | 00:00 | |
| * jml thinks... | 00:00 | |
| jml | any feedback from non-Canonical contributors about the review process? | 00:01 |
| * barry looks at wgrant | 00:01 | |
| mwhudson | i guess i can take the opportunity to do a little survey | 00:01 |
| wgrant | barry: Working well for me. | 00:01 |
| barry | i'll just say again that it is /fantastic/ to see yours and others from the community landing branches | 00:02 |
| wgrant | It's great to finally be able to! | 00:02 |
| mwhudson | as build engineer for the next cycle, what's causing you, the developer, the most pain? | 00:03 |
| wgrant | My only slight issue is that sponsors are often unclear on how to do things. | 00:03 |
| wgrant | Is this not documented? | 00:03 |
| wgrant | mwhudson: 'build engineer'? | 00:03 |
| jml | wasn't that announcement on launchpad-dev! | 00:03 |
| wgrant | No. We don't find anything out. | 00:03 |
| barry | wgrant: can you elaborate? | 00:03 |
| mwhudson | bah | 00:03 |
| mwhudson | and the wiki page is internal | 00:04 |
| * jml frowns | 00:04 | |
| mwhudson | still learning at this sort of thing it seems :) | 00:04 |
| barry | yep. old habits die hard, but we're really trying to move more and more resources, communications, etc. into the public square | 00:04 |
| wgrant | barry: Well, some of my sponsors have had trouble working out how to ec2test my branches. I'm not sure why. | 00:04 |
| wgrant | What was this email meant to contain? | 00:05 |
| barry | mwhudson: we really should find a way for non-Canonicals to run ec2 | 00:05 |
| jml | we're going to have a build engineer position | 00:05 |
| wgrant | barry: +inf | 00:05 |
| jml | it's a rotating postiition | 00:05 |
| mwhudson | "Each development cycle, a different engineer will be taken off his normal development duty, and have has main focus for the cycle the development and well being of the build system. " | 00:05 |
| jml | the build engineer works on making developers lives more pleasant, essentially | 00:05 |
| wgrant | Aha. I see. | 00:06 |
| mwhudson | i'm the first one, starting officially on monday | 00:06 |
| jml | mwhudson, in answer to your question | 00:06 |
| mwhudson | i'm probably going to tackle some buildbot annoyances first | 00:06 |
| mwhudson | jml: you sent me an email about this that i haven't read yet :) | 00:06 |
| wgrant | mwhudson: Bonus points for making it public. | 00:06 |
| mwhudson | wgrant: yeah, that should be possible | 00:06 |
| jml | mwhudson, my email wasn't so much about my pain points, actually :) | 00:07 |
| jml | mwhudson, it takes way too long to land branches; buildbot should only send emails that require action; test failures in devel break ec2test runs | 00:08 |
| barry | jml, mwhudson what is the main hold up to allowing the public to run ec2? is it the payment issue? | 00:08 |
| mwhudson | barry: i imagine so | 00:08 |
| mwhudson | also ec2test is still private i think, but that will change soon (?) | 00:08 |
| jml | mwhudson, not a pain point for me, but one I've had to watch: it takes too long to get set up with a dev env | 00:09 |
| barry | surely we can find a way to subsidize the ec2 costs. it's gotta be trivial in the scheme of things. i usually feel guilty expensing it every quarter | 00:09 |
| jml | mwhudson, the main meta issue is that it's really hard to see what the status of various build-related projects are (e.g. buildbot going public, ec2test going public, etc) | 00:09 |
| rockstar | barry, when we have a UEC set up in the data center, it might be easier. *shrug* | 00:10 |
| barry | rockstar: good point | 00:10 |
| mwhudson | jml: i think A build engineer, though probably not me, should focus on getting buildbot running the test suite in parallel across several instances | 00:10 |
| jml | mwhudson, I very strongly agree. | 00:10 |
| rockstar | mwhudson, abentley and I had a discussion about test layers today, and whether or not we're really using the layers properly. | 00:11 |
| rockstar | I find I spend a lot of time setting up and tearing down the test harnesses just to run one test (that usually runs in shorter time than set up or tear down. | 00:11 |
| mwhudson | rockstar: i would claim that using layers correctly is close to impossible, but maybe that's a bit bigoted | 00:11 |
| barry | rockstar: i can almost bet we're not. the layers are horribly named and it is not intuitive to know what layer a test should go in | 00:12 |
| mwhudson | rockstar: there's also the terrible way we start daemons in tests, that doesn't help | 00:12 |
| rockstar | barry, yeah, GoogleServiceLayer and LibrarianLayer were two we identified as not really necessary for most of our tests. | 00:12 |
| jml | switching to testresources would help here | 00:12 |
| mwhudson | +1 jml | 00:12 |
| jml | I'm a little terrified about the work involved | 00:12 |
| rockstar | jml, +1 | 00:12 |
| barry | rockstar: and they take a long time to start | 00:12 |
| jml | but maybe if we actually explored what needed to happen, it would be less terrifying | 00:13 |
| rockstar | barry, yeah, which is basically what I spent much of yesterday doing. | 00:13 |
| wgrant | librarian takes forever! | 00:13 |
| jml | wgrant, indeed. | 00:13 |
| rockstar | wgrant, this morning it took 15.391 seconds to start up, and 7.3 seconds to tear down the librarian. | 00:13 |
| jml | mwhudson, so, that's not really a survey, since it's just me :) | 00:13 |
| wgrant | rockstar: It's about twice as fast here, most of the time. | 00:14 |
| rockstar | mwhudson, really, if I had a pony request, it'd be a graph for how long the test suite is taking, and maybe some mini graphs of different sections of the test. | 00:14 |
| mwhudson | for all that, step 1 is still going to be force build on buildbot | 00:14 |
| mwhudson | rockstar: that's a good idea | 00:14 |
| * jml agrees -- it's in an email I sent :) | 00:14 | |
| rockstar | jml, dammit. You and your organization... | 00:15 |
| rockstar | Do we have a place to aggregate all of these ideas? | 00:15 |
| jml | exactly! | 00:15 |
| rockstar | jml, did you send an email about that too? | 00:16 |
| jml | I was about to say, step 0 should really be tracking this. | 00:16 |
| barry | rockstar: like a wiki? <wink> | 00:16 |
| jml | rockstar, it was in the same email as the graph stuff :) | 00:16 |
| rockstar | barry, wikis are TERRIBLE issue trackers. | 00:16 |
| jml | barry, or, umm, a bug tracker. | 00:16 |
| jml | barry, I've got this good one I can sell you | 00:16 |
| rockstar | I know these really great guys who spend a lot of time writing a bug tracker... | 00:16 |
| mwhudson | step -1: move BE wiki pages to dev.lp.net | 00:16 |
| barry | :) | 00:16 |
| jml | mwhudson, \o/ | 00:17 |
| mwhudson | step 0: a tag in launchpad-project, i guess | 00:17 |
| rockstar | Maybe a product in launchpad-project. | 00:17 |
| wgrant | Wouldn't a separate product be much better? | 00:17 |
| jml | rockstar, wgrant, maaybe | 00:17 |
| jml | I don't think that the way Launchpad uses products is the way Launchpad wants others to use products though | 00:18 |
| * jml makes a note about that | 00:18 | |
| wgrant | No. | 00:18 |
| wgrant | But this one is different. | 00:18 |
| jml | wgrant, yeah, I think so too, but I'm not sure how it's different. | 00:19 |
| wgrant | jml: It's not part of the codebase. | 00:19 |
| jml | wgrant, some of the bugs are in the code base | 00:19 |
| wgrant | Most of the other products are. | 00:19 |
| wgrant | Hmm. | 00:19 |
| jml | wgrant, but a subproduct would fit there, I guess | 00:19 |
| * mwhudson writes another thing down, "make sure buildbot config is in a branch or two on launchpad" | 00:20 | |
| jml | the secret, proprietary project with all of our developer utilities might well belong as a public sub-project of Launchpad | 00:20 |
| barry | jml: +1 | 00:20 |
| wgrant | jml: lp-dev-utils, or is there something else? | 00:20 |
| jml | wgrant, lp-dev-utils, yes. | 00:20 |
| jml | wgrant, that is, I can neither confirm nor deny the existence of said project... | 00:21 |
| wgrant | Heh. | 00:21 |
| * wgrant whines about the product/project/project/projectgroup confusion. | 00:21 | |
| jml | patches accepted :) | 00:21 |
| barry | wgrant: beuno just today told me i could jfdi (rename those things :) | 00:22 |
| jml | either that, or sabotage barry's bass so he gets some more free time | 00:22 |
| barry | jml: that is so cruel | 00:22 |
| wgrant | Heh. | 00:22 |
| jml | wgrant, all you need to do is add an extra string -- it'll baffle him | 00:22 |
| jml | barry, <3 | 00:22 |
| barry | jml: ouch :) | 00:23 |
| barry | does that mean we're done here? | 00:23 |
| jml | I think so. | 00:23 |
| barry | wgrant, rockstar, mwhudson anything else? | 00:23 |
| * wgrant can think of nothing more to whinge about. | 00:23 | |
| barry | 5 | 00:24 |
| barry | 4 | 00:24 |
| barry | 3 | 00:24 |
| barry | 2 | 00:24 |
| barry | 1 | 00:24 |
| jml | Thunderbirds are go! | 00:24 |
| barry | #endmeeting | 00:24 |
| MootBot | Meeting finished at 18:24. | 00:24 |
| mwhudson | cheers all | 00:24 |
| barry | thanks guys! see you back at the farm | 00:24 |
| jml | barry, thanks! | 00:24 |
| lifeless | jml: subunit has been a huge help for me in getting statistics | 00:42 |
| lifeless | FWIW | 00:42 |
| jml | lifeless, I'll bet. | 00:43 |
| jml | there's probably an interesting paper in 'making a test suite fast again' | 00:43 |
| lifeless | I just submitted a merge for bzr to make its testresults python2.7 shiny and more reusable | 00:43 |
| lifeless | jml: and ./bzr selftest selftest is down to 3.3 seconds on my laptop [by the reporter, so not including test loading time. Thats a TOFIX] | 00:44 |
| jml | lifeless, why are you only measuring the time for 'selftest' tests? | 00:44 |
| lifeless | jml: its a brain food spike | 00:45 |
| lifeless | jml: 'make this little bit of the test suite as lean and mean as possible' | 00:45 |
| lifeless | it was ~ 60 seconds when I started | 00:45 |
| jml | lifeless, makes sense. | 00:45 |
| lifeless | so far I've reduced overall test load time by 10% | 00:47 |
| lifeless | and identified that our base TestCase is about 2000 times more expensive than unittest.TestCase | 00:48 |
| jml | heh heh | 00:48 |
| lifeless | so theres still lots of fat to go | 00:48 |
| lifeless | this bit comes first, because fixing other areas will mean changing this bit and that should be a pleasure not a pain | 00:49 |
| jml | *nod* | 00:51 |
| === gary-out is now known as gary_poster | ||
| === Ursinha is now known as Ursinha-afk | ||
| === mrevell is now known as mrevell-lunch | ||
| === matsubara-afk is now known as matsubara | ||
| === mrevell-lunch is now known as mrevell | ||
| === cprov-afk is now known as cprov | ||
| matsubara | #startmeeting | 16:00 |
| MootBot | Meeting started at 10:00. The chair is matsubara. | 16:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 16:00 |
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 16:00 |
| matsubara | [TOPIC] Roll Call | 16:00 |
| MootBot | New Topic: Roll Call | 16:00 |
| matsubara | Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! | 16:00 |
| Ursinha-afk | me | 16:00 |
| gary_poster | me | 16:00 |
| intellectronica | me | 16:00 |
| barry | me | 16:00 |
| === Ursinha-afk is now known as Ursinha | ||
| rockstar | ni! | 16:00 |
| bigjools | me | 16:00 |
| intellectronica | Ursinha: and i thought you're doing your telepathic typing trick again... | 16:01 |
| Ursinha | intellectronica, hehe | 16:01 |
| matsubara | Chex, hi | 16:01 |
| Chex | matsubara: hello | 16:01 |
| matsubara | ok, so we're missing stub and henning/danilo | 16:02 |
| danilos | me | 16:02 |
| matsubara | hi danilos | 16:02 |
| matsubara | :-) | 16:02 |
| danilos | matsubara: no you are not :) | 16:02 |
| matsubara | so, from now on, you'll be attending the prod meeting, right? | 16:02 |
| danilos | matsubara: well, not necessarily, let's see :) | 16:03 |
| matsubara | ok, let's move on then | 16:03 |
| danilos | matsubara: but soon I will be replacing henninge as our QA contact | 16:03 |
| matsubara | [TOPIC] Agenda | 16:03 |
| MootBot | New Topic: Agenda | 16:03 |
| matsubara | * Actions from last meeting | 16:03 |
| matsubara | * Oops report & Critical Bugs & Broken scripts | 16:03 |
| matsubara | * Operations report (mthaddon/Chex/spm/mbarnett) | 16:03 |
| matsubara | * DBA report (stub) | 16:03 |
| matsubara | * Proposed items | 16:03 |
| matsubara | danilos, ok | 16:03 |
| matsubara | [TOPIC] * Actions from last meeting | 16:03 |
| MootBot | New Topic: * Actions from last meeting | 16:03 |
| matsubara | * cprov to follow up on bug 413749 | 16:03 |
| matsubara | * stub to investigate garbo-hourly failure after spm adjusted script checking to 12h | 16:03 |
| matsubara | * sinzui to poke barry about ExpatError OOPSes (bug 403606) | 16:03 |
| matsubara | * danilos, Ursinha and matsubara to discuss oops summaries split per team | 16:03 |
| ubottu | Bug 413749 on http://launchpad.net/bugs/413749 is private | 16:03 |
| ubottu | Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 | 16:03 |
| danilos | matsubara: we kind of did that, flacoste is involved as well, and it's all on track I'd say :) | 16:04 |
| matsubara | ok, I know that stub replied to the garbo-hourly failure email, so I think that's done | 16:04 |
| matsubara | danilos, right. if you have more suggestions let me know | 16:04 |
| danilos | sinzui is on unavailable I believe | 16:04 |
| matsubara | barry, hi! | 16:04 |
| * barry has some thoughts about the expat errors | 16:04 | |
| barry | matsubara: hi! | 16:05 |
| matsubara | barry, what's up with those expat errors? | 16:05 |
| cprov | matsubara: I did reply on 413749 as well, needs discussion, though. | 16:05 |
| matsubara | thanks cprov | 16:05 |
| Ursinha | matsubara, the one with the expaterrors I'd bring in the oops section | 16:06 |
| barry | matsubara: well, some more investigation might be warranted, but they are intermittent and i've always chalked them up to problems with xmlrpc. i believe they can be safely ignored though because the system will "catch up" the next time mailman talks to xmlrpc server. | 16:06 |
| Ursinha | barry, here's the problem, they happen every day | 16:06 |
| barry | matsubara: ultimately, we should get rid of the xmlrpc stuff. it was a solution to a problem we no longer have (namely gpl pollution between mailman and launchpad) | 16:06 |
| matsubara | if it's safe to ignore them, can we add some code to catch the exception and not log the oops? | 16:06 |
| matsubara | hi stub | 16:07 |
| Ursinha | you said in the bug report that we have only a handful a week, but I see them everyday, and their numbers are the highest count | 16:07 |
| barry | Ursinha: i get many oops reports for mailman that are clean. am i looking at the wrong reports? | 16:07 |
| Ursinha | barry, I think so :) | 16:07 |
| Ursinha | look at the lpnet reports | 16:07 |
| barry | Ursinha: meaning the daily emails | 16:07 |
| barry | hmm. okay. i see them only rarely in the mailman daily email. i will double check next time i see the report, but i think ignoring the error is the right thing to do | 16:08 |
| Ursinha | barry, ok. it floods the summary, and that's annoying | 16:09 |
| barry | Ursinha: gotcha! i hadn't realized that. i'll do something about it then | 16:09 |
| gary_poster | When I discussed hiding the problems, given the frequency, Francis suggested that we not hide them but investigate a bit. Maybe a small timebox would be appropriate? | 16:10 |
| Ursinha | thaaaanks muchly barry! | 16:10 |
| gary_poster | eh, nm | 16:10 |
| gary_poster | :-) | 16:10 |
| Ursinha | gary_poster, :) | 16:10 |
| barry | gary_poster: yes, we can do that. it will take losa intervention and some cowboy debugging, but it's probably worth it | 16:10 |
| gary_poster | ok cool | 16:11 |
| matsubara | [action] barry to continue debug on bug 403606 | 16:11 |
| MootBot | ACTION received: barry to continue debug on bug 403606 | 16:11 |
| ubottu | Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606 | 16:11 |
| matsubara | thanks barry, gary_poster, Ursinha | 16:11 |
| Ursinha | thanks | 16:11 |
| matsubara | let's move on | 16:11 |
| matsubara | [TOPIC] * Oops report & Critical Bugs & Broken scripts | 16:11 |
| MootBot | New Topic: * Oops report & Critical Bugs & Broken scripts | 16:11 |
| Ursinha | ok | 16:11 |
| matsubara | Ursinha, take the stage please | 16:12 |
| Ursinha | so the ExpatErrors were already discussed here | 16:12 |
| Ursinha | I have only one bug | 16:12 |
| Ursinha | intellectronica, bug 408738 | 16:12 |
| ubottu | Launchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/408738 | 16:12 |
| Ursinha | that's another one that happens pretty often | 16:12 |
| intellectronica | really? if so then we should at least recover gracefully from it | 16:13 |
| Ursinha | intellectronica, I see those everyday as well | 16:13 |
| intellectronica | i mean in addition to figuring out why we have records like those | 16:13 |
| Ursinha | I agree | 16:14 |
| Ursinha | intellectronica, could you take a look or poke someone to do so, please? | 16:14 |
| intellectronica | yes yes | 16:15 |
| Ursinha | thanks intellectronica :) | 16:15 |
| Ursinha | matsubara, [action] intellectronica to take a look or find someone to take a look on bug 408738 | 16:15 |
| ubottu | Launchpad bug 408738 in malone "OOPS when rendering bug activity" [Undecided,Triaged] https://launchpad.net/bugs/408738 | 16:15 |
| matsubara | [action] intellectronica to take a look or find someone to take a look on bug 408738 | 16:16 |
| MootBot | ACTION received: intellectronica to take a look or find someone to take a look on bug 408738 | 16:16 |
| Ursinha | thanks :) | 16:16 |
| matsubara | thanks Ursinha and intellectronica | 16:16 |
| matsubara | we have 2 critical bugs | 16:16 |
| Ursinha | critical bugs are fix committed, and the process-pending-packagediffs script failure was already explained by cprov | 16:16 |
| Ursinha | so, we're done! | 16:16 |
| matsubara | both fix committed | 16:16 |
| Ursinha | you can move on, matsubara | 16:16 |
| Ursinha | thanks guys! | 16:16 |
| matsubara | thanks everyone | 16:16 |
| matsubara | [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) | 16:16 |
| MootBot | New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) | 16:16 |
| Chex | hello everyone : | 16:17 |
| Ursinha | hi Chex :) | 16:17 |
| Chex | the report for this week: | 16:17 |
| Chex | Lots more cherry picks this week | 16:17 |
| Chex | More and more problems with codebrowse (per the Incident Log) | 16:17 |
| Chex | New losas getting up to speed (Chex and mbarnett) | 16:17 |
| Chex | Testing tarmac for oops-tools project - making progress but not quite there yet | 16:17 |
| ubottu | https://lp-oops.canonical.com/oops.py/?oopsid=tools | 16:17 |
| Chex | Will begin process of implementing recommendations from SplitIt Sprint before too long | 16:17 |
| Chex | One of our top priority bugs is bug 416990 - others per https://bugs.launchpad.net/~canonical-losas | 16:17 |
| Chex | That's it unless there are any questions | 16:17 |
| ubottu | Launchpad bug 416990 in bzr "Memory usage of codehosting processes is excessive" [High,Confirmed] https://launchpad.net/bugs/416990 | 16:17 |
| matsubara | Chex, re: tarmac setup for oops-tools, I'll submit a merge proposal to tarmac today with the remaining fixes needed. | 16:19 |
| ubottu | https://lp-oops.canonical.com/oops.py/?oopsid=tools | 16:19 |
| matsubara | Chex, I'll coordinate with mthaddon later on | 16:19 |
| mthaddon | cool | 16:19 |
| Chex | matsubara: ok, great | 16:20 |
| matsubara | rockstar, can you poke people in your team about the codehosting memory issue? | 16:20 |
| danilos | matsubara: is it worth fixing ubottu not to choke on oops-tools mention? ;) | 16:20 |
| ubottu | https://lp-oops.canonical.com/oops.py/?oopsid=tools | 16:20 |
| Ursinha | lol | 16:21 |
| matsubara | I can ask the maintainer | 16:21 |
| danilos | matsubara: thanks :) | 16:21 |
| matsubara | [action] matsubara to chase ubottu maintainer to fix regexp used to identify oopses | 16:21 |
| MootBot | ACTION received: matsubara to chase ubottu maintainer to fix regexp used to identify oopses | 16:21 |
| matsubara | let's move on | 16:22 |
| matsubara | thanks Chex | 16:22 |
| matsubara | [TOPIC] * DBA report (stub) | 16:22 |
| MootBot | New Topic: * DBA report (stub) | 16:22 |
| stub | The long running transaction killer is back set to 3 hours, thanks to the Rosetta team sorting out the language pack export issue quickly. | 16:22 |
| stub | oot | 16:22 |
| Ursinha | that was fast | 16:22 |
| matsubara | hehe | 16:23 |
| bigjools | like mrevell, it was short | 16:23 |
| matsubara | thanks stub and rosetta people :-) | 16:23 |
| matsubara | lol | 16:23 |
| Ursinha | lol | 16:23 |
| matsubara | [TOPIC] * Proposed items | 16:23 |
| MootBot | New Topic: * Proposed items | 16:23 |
| matsubara | we have no proposed items | 16:23 |
| mrevell | haha bigjools | 16:23 |
| matsubara | anything else before I close? | 16:23 |
| matsubara | hi mrevell | 16:23 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. | 16:24 |
| mrevell | matsubara: nothing from me, just responding to bigjools... :) | 16:24 |
| matsubara | #endmeeting | 16:24 |
| MootBot | Meeting finished at 10:24. | 16:24 |
| Ursinha | thanks everyone! | 16:24 |
| bigjools | mrevell: I apologise profusely, please don't hurt me | 16:25 |
| mrevell | bigjools: Ah, ok, just this once | 16:27 |
| danilos | thanks matsubara, all | 16:27 |
| Ursinha | oops-thisisnotright | 16:28 |
| ubottu | https://lp-oops.canonical.com/oops.py/?oopsid=thisisnotright | 16:28 |
| danilos | Ursinha: come on, come, don't be too hard on it | 16:28 |
| Ursinha | :P | 16:29 |
| bigjools | oops-I-did-it-again | 16:47 |
| ubottu | https://lp-oops.canonical.com/oops.py/?oopsid=I | 16:47 |
| === salgado is now known as salgado-lunch | ||
| === salgado-lunch is now known as salgado | ||
| === matsubara is now known as matsubara-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| === Ursinha is now known as Ursinha-afk | ||
| === EdwinGrubbs is now known as Edwin-lunch | ||
| === Ursinha-afk is now known as Ursinha | ||
| === salgado is now known as salgado-afk | ||
| === Edwin-lunch is now known as EdwinGrubbs | ||
| === cprov is now known as cprov-afk | ||
| === matsubara is now known as matsubara-afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!