[00:22] <jml> lalalalala
[00:23] <thumper> note to self: webservice glob in pages
[00:23] <barry> #startmeeting
[00:23] <MootBot> Meeting started at 18:23. The chair is barry.
[00:23] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[00:23] <barry> yada yada yada who's here today yada yada
[00:24] <thumper> me
[00:24] <jml> Present and accounted for!
[00:24] <barry> mwhudson: ping!
[00:24] <barry> [TOPIC] agenda
[00:24] <MootBot> New Topic:  agenda
[00:24] <mwhudson> hi
[00:25] <barry>  * Roll call
[00:25] <barry>  * Action items
[00:25] <barry>  * Mentoring update
[00:25] <barry>  * Peanut gallery (anything not on the agenda)
[00:25] <barry> [TOPIC] action items
[00:25] <MootBot> New Topic:  action items
[00:25] <barry>  * abentley to experiment on storm base class
[00:25] <barry> so, i actually semi-reviewed a branch of his that looks pretty good. stub also semi-reviewed it
[00:25] <barry> i think actually jml is the official reviewer
[00:25] <jml> yeah.
[00:26] <jml> I'll look at that today
[00:26] <barry> jml: great! because i think you'll have some useful insights
[00:26] <barry> see what you think of stub and my comments
[00:26] <jml> I am a man of constant sorrow.
[00:26] <jml> and occasional insight.
[00:27] <barry> ah what a great movie
[00:27] <barry>  * barry to add `field_id` to coding guideline
[00:27] <barry> not done, but no disagreement from ameu
[00:28] <barry>  * barry to add `pretty()` functions to reviewers docs
[00:28] <barry>  * flacoste to work on API reviewer cheat sheet
[00:28] <barry> not done, we suck
[00:28] <barry> [TOPIC] mentoring update
[00:28] <MootBot> New Topic:  mentoring update
[00:28] <barry> so jml how goes it with stub?
[00:29] <jml> almost no action between last week and now.
[00:29] <barry> jml: shall we go another week?
[00:29] <jml> barry: yeah
[00:29] <barry> excellent
[00:29] <jml> barry: the stormsugar review will be a good opportunity
[00:29] <barry> jml: great
[00:30] <barry> anything else on mentoring?
[00:30] <mwhudson> how many devs are not reviewers yet?
[00:30] <barry> mwhudson: that's a great question
[00:31] <jml> nope
[00:32] <barry> noodles, leonardr, hennige
[00:32] <barry> maybe that's it
[00:32] <mwhudson> cool
[00:32] <barry> all three could definitely join the ranks of mentats
[00:32] <mwhudson> suprised leonard isn't on the way
[00:32] <sinzui> leonardr: had no interest I believe
[00:33] <barry> mwhudson: i think he's resisted it for a long time because he dabbles mostly in apiland
[00:33] <jml> how many people are not OCRs?
[00:33] <barry> jml: team leads and intellectronica
[00:33] <mwhudson> i'm not sure people should be allowed to opt out :)
[00:33] <barry> team leads have to i think
[00:33] <mwhudson> (of being a reviewer, not necessarily ocr)
[00:33] <barry> intellectronica: is on sabbatical but he should come back!
[00:34] <barry> thumper: can be an exception
[00:34] <intellectronica> barry: i'm happy to come back this cycle
[00:34] <barry> intellectronica !
[00:34] <barry> intellectronica: excellent!
[00:35] <barry> intellectronica: monday euro is open
[00:35] <barry> but we can also move people around if you want
[00:35] <sinzui> team leads cover review of their team. I do about 4 a week
[00:35] <intellectronica> monday euro it is, then!
[00:35] <sinzui> but today I did 7
[00:35] <barry> sinzui: so every day is ocr for you? :)
[00:35] <barry> intellectronica: done
[00:36] <jml> barry: in asiapac, every day is ocr day.
[00:36] <barry> jml: yay?
[00:36] <jml> barry: hire more aussies :)
[00:37] <barry> jml: if it were up to me, i'd hire 10.  i love aussies
[00:37] <jml> *hugs*

[00:37] <jml> anyway...
[00:37] <barry> anyway...
[00:37] <mwhudson> moving on?
[00:37] <barry> i will try to find mentors for those guys
[00:37] <barry> [TOPIC] peanut gallery
[00:37] <MootBot> New Topic:  peanut gallery
[00:38] <jml> mwhudson: I guess the Tasman is not choppy enough to block your mind-reading powers :)
[00:38] <barry> do you guys have anything?
[00:38] <jml> thumper has a thing.
[00:38]  * thumper does
[00:38] <thumper> the webservice in pagetest globs should die a horrible death
[00:38] <thumper> it defaults to salgado
[00:38] <thumper> who is an admin user
[00:38] <thumper> also it defaults to change everything
[00:38] <thumper> it doesn't really allow testing of security at all through the pagetests
[00:38] <mwhudson> +1 +1 +1
[00:38] <thumper> we shouldn't use it
[00:38] <thumper> ever
[00:39] <thumper> there is a function called webservice_for_person
[00:39] <thumper> that takes a person object
[00:39] <thumper> we should use that more
[00:39] <thumper> this caused me considerable pain this week
[00:39] <barry> thumper: should we change webservice globs to not use an admin?
[00:39] <thumper> barry: at the very least
[00:39] <thumper> barry: we shouldn't use sample data
[00:39] <barry> how much breaks if we do that?
[00:39] <thumper> barry: and the webservice glob does
[00:40] <thumper> barry: so my preference would to be remove it all together
[00:40] <barry> that was my next suggestion :)
[00:40] <thumper> die sample data die!!!
[00:40] <barry> thumper: is there an open bug on this?  my inclination would be to assign such a bug to leonardr and/or foundations
[00:40] <thumper> no, not yet
[00:41] <barry> can you open one, or at least email the ml?
[00:41] <barry> fwiw, +1 btw :)
[00:41] <thumper> ok
[00:41] <barry> [ACTION] thumper to report about webserver glob as admin problem
[00:41] <MootBot> ACTION received:  thumper to report about webserver glob as admin problem
[00:41] <barry> excellent, thanks
[00:41] <thumper> next
[00:41] <thumper> is
[00:42] <thumper>  http://code.launchpad.net/launchpad/+activereviews
[00:42] <thumper> two main changes here
[00:42] <thumper> tables explaining what to do
[00:42] <thumper> or at least what you can do
[00:42] <thumper> and also the titles of the votes is the people who voted/reviewed
[00:42] <thumper> just an fyi
[00:43] <jml> One thing from me.
[00:43] <mwhudson> thumper: it's not totally obvious to me what "Completed reviews"/"Other reviews" means
[00:43]  * jml waits though.
[00:43] <barry> mwhudson: i had the same confusion
[00:43] <thumper> mwhudson: I'm open to suggestions
[00:43] <thumper> it was beuno's idea
[00:43] <barry> thumper: [ui=barry]
[00:43] <barry> dang
[00:44] <jml> heh heh
[00:44] <jml> so, my one thing. it's quick
[00:44] <barry> thumper: but i really do like the new page
[00:44] <barry> jml: go4it
[00:44] <mwhudson> well, i guess it exacerbated by the fact that i'm not in ~launchpad-reviewers
[00:44] <jml> I came across some code that called _foo methods on objects defined in a different module
[00:44] <jml> please, please no one ever do that
[00:45] <mwhudson> thumper: i'll let you know if i had any better ideas :)
[00:45] <jml> given that we control both ends of the stick, just rename the method to 'foo' if you have to do that.
[00:45] <jml> it's not like we care about internal API compat.
[00:45] <barry> jml: +1
[00:46] <thumper> add to reviewers checklist?
[00:46] <jml> (as well as edit the reviewers checklist to make it short and punchy)
[00:46] <jml> ((we talked about that last week))
[00:46] <barry> yes, yes, yes ;)
[00:46] <jml> barry: <3
[00:47] <barry> one of these days i really will garden those pages
[00:47] <barry> but i think i'll wait until the nags head sprint when i'm good and sloshed
[00:47] <barry> anybody else have anything?
[00:47] <jml> nope
[00:47] <thumper> i'm done
[00:47] <barry> just one or two quick ones from the ameu meeting then...
[00:48]  * mwhudson done too
[00:48] <barry> bigjools is going to experiment on helper functions to make circular import backpatching less disgusting
[00:48] <barry> we talked about a few things there, see the irc log for details (which i'll send out tomorrow <wink>)
[00:48] <thumper> there is that file in c.l.interfaces
[00:48]  * thumper goes to look
[00:48] <barry> thumper: yep
[00:49] <barry> thumper: we know about it :)
[00:49] <thumper> oh, ok
[00:49] <barry> next
[00:49] <thumper> only code-team items in there though
[00:49] <jml> Ich habe ein Bärenhunger
[00:49] <barry> everyone needs to bribe gary so that he buildouts launchpad in his spare time
[00:49] <thumper> _schema_circular_imports
[00:50] <barry> but anyway.  must abide hunger, so that's it from me
[00:50] <mwhudson> ok
[00:50] <barry> let's call it
[00:50] <mwhudson> thanks barry
[00:50] <barry> #endmeeting
[00:50] <MootBot> Meeting finished at 18:50.
[00:50] <barry> thanks guys!
[00:50] <jml> barry: thanks.
[00:50] <thumper> ta
[15:00] <matsubara> #startmeeting
[15:00] <MootBot> Meeting started at 09:00. The chair is matsubara.
[15:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[15:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[15:00] <matsubara> [TOPIC] Roll Call
[15:00] <MootBot> New Topic:  Roll Call
[15:00] <bigjools> me
[15:00] <Ursinha> me
[15:00] <sinzui> me
[15:00] <henninge> ich
[15:00] <herb> me
[15:00] <flacoste> me
[15:01] <gmb> me
[15:01] <matsubara> stub is missing.
[15:02] <matsubara> well, let's continue. stub can join later
[15:02] <gmb> Presumed hairy.
[15:02] <matsubara> [TOPIC] Agenda
[15:02] <MootBot> New Topic:  Agenda
[15:02] <matsubara>  * Actions from last meeting
[15:02] <matsubara>  * Oops report & Critical Bugs
[15:02] <matsubara>  * Operations report (mthaddon/herb/spm)
[15:02] <matsubara>  * DBA report (DBA contact)
[15:02] <matsubara>  * Next Launchpad Performance Week
[15:02] <Ursinha> traditionally he'll join in a few moments
[15:02] <matsubara> :-)
[15:02] <matsubara> [TOPIC] * Actions from last meeting
[15:02] <MootBot> New Topic:  * Actions from last meeting
[15:02] <matsubara> * stub to investigate the fix to avoid staging restore problems
[15:02] <matsubara>  * rockstar to fix https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138CEMAIL12
[15:02] <matsubara>  * sinzui to have someone to fix bug 237722
[15:02] <matsubara>  * Ursinha to file bugs to https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138EC328 - bug 329917, and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138B2582 - bug 329908
[15:02] <matsubara>  * intellectronica to investigate https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1137F2893
[15:03] <Ursinha> mine is fine, filed bugs are there
[15:03] <sinzui> matsubara: Not done, but it was discussed
[15:03] <matsubara> sinzui, thanks. I see that's triaged as low priority so it's ok
[15:03]  * sinzui will pick a victim to close the bug
[15:04] <Ursinha> :)
[15:04] <sinzui> matsubara: the person who discovers the problem has the power to fix it. He is often the person who put the teams in the situation
[15:05] <sinzui> matsubara: so we believe the right answer is to inform the user what he is doing before he gets the warning he has done something impossible
[15:05] <Ursinha> :)
[15:05] <matsubara> sinzui, inform the user or just block the user from doing it?
[15:05] <matsubara> hi stub
[15:06] <matsubara>  * stub to investigate the fix to avoid staging restore problems
[15:06] <matsubara> any news about this?
[15:06] <matsubara> [action] matsubara to chase rockstar about a fix for OOPS-1138CEMAIL12
[15:06] <MootBot> ACTION received:  matsubara to chase rockstar about a fix for OOPS-1138CEMAIL12
[15:07] <stub> No news
[15:07] <sinzui> matsubara: both actually. Explain that launchpad has removed a pending membership because between x in y. If this is wrong the user should remove the member ship he just approved and switch propose the team again
[15:08] <matsubara> sinzui, right
[15:08] <matsubara> gmb, any news about OOPS-1137F2893?
[15:09] <gmb> matsubara: Not as far as I know; will confirm with intellectronica.
[15:09] <Ursinha> matsubara, iirc flacoste was discussing that with intellectronica
[15:11] <gmb> intellectronica's sprinting atm so let's move on and I'll get back to you when I know more.
[15:11] <matsubara> Ursinha, gmb: ok thanks
[15:11] <matsubara> [TOPIC] * Oops report & Critical Bugs
[15:11] <MootBot> New Topic:  * Oops report & Critical Bugs
[15:11] <Ursinha> all right
[15:11] <Ursinha> one bug for bugs, two OOPSes for soyuz, one OOPS for foundations
[15:11] <Ursinha> for bugs: bug 329908
[15:11] <bigjools> go soyuz
[15:11] <flacoste> matsubara: yes, we discussed a solution, i don't know the status of the fix though
[15:12] <Ursinha> gmb, that one is happening quite frequently
[15:12] <Ursinha> on a daily basis, but only a few
[15:12] <matsubara> flacoste, okie. thanks
[15:12] <gmb> Ursinha: Hmm. Okay.
[15:13] <gmb> Ursinha: That sounds - at first glance - like a replication problem again.
[15:13] <gmb> But that's just a stab in the dark
[15:13] <Ursinha> gmb, I didn't stick to this idea because it's happening like every day
[15:13] <Ursinha> do you think it's possible to be a replication problem even considering this?
[15:14] <gmb> Ursinha: Well, last time something like that happened it was  a replication problem, but that was fixed last cycle.
[15:14] <gmb> Ursinha: Frankly, at this point, I've no idea. I'll look into it.
[15:14] <Ursinha> gmb, I remember that
[15:14] <Ursinha> thanks gmb
[15:14]  * Ursinha looks at bigjools 
[15:14] <Ursinha> bigjools,  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
[15:15] <bigjools> ok
[15:15] <Ursinha> what do you say?
[15:15] <bigjools> first one was caused by cprov
[15:15] <bigjools> I will check with him
[15:16] <cprov> I did it on purpose
[15:16] <bigjools> is there a bug filed?
[15:16] <Ursinha> bigjools, I didn't file one, but can do that
[15:16] <stub> Librarian OOPSes are not caused by replication - the Librarian only talks to the master database.
[15:17] <bigjools> second one looks nasty, we'll look into it
[15:17] <Ursinha> gmb, ^
[15:17] <cprov> bigjools: the second oops need investigation
[15:17] <gmb> stub, Ursinha: Curse. Oh well, I'll have to do some work then.
[15:17] <bigjools> Ursinha: can you file bugs on those for me please
[15:17] <Ursinha> matsubara, [action] cprov and bigjools to investigate OOPS-1145EA14
[15:17] <Ursinha> bigjools, sure
[15:17] <bigjools> thanks muchly
[15:17] <Ursinha> matsubara, [action] Ursinha to file bugs for https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
[15:18] <matsubara> [action] cprov and bigjools to investigate OOPS-1145EA14
[15:18] <MootBot> ACTION received:  cprov and bigjools to investigate OOPS-1145EA14
[15:18] <matsubara>  [action] Ursinha to file bugs for https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
[15:18] <Ursinha> bigjools, np :)
[15:18] <Ursinha> the last one for foundations:  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1146XMLP1
[15:18] <Ursinha> flacoste, ^
[15:18] <stub> gmb: We have seen similar 'impossible' situations before, and have failed to work it out. No requests to that URL should be made until the transaction that created it commits, and as soon as it commits not founds shouldn't happen.
[15:19] <gmb> stub: Hmm. Okay. Thanks.
[15:19] <flacoste> Ursinha: that's not a bug
[15:19]  * gmb wonders about the possibility of getting a `sleep(10)` into apport...
[15:19] <flacoste> Ursinha: that's something posting non-XMLRPC request to the xmlrpc server
[15:19] <flacoste> Ursinha: so find what is doing that and stop it from doing it :-)
[15:20] <flacoste> Ursinha: probably a Mailman bad request for some reason (restart?)
[15:20] <stub> gmb: It might help. I suspect commit is returning before the commit really has finished to make things look fast (suspect because everything else seems impossible)
[15:20] <Ursinha> flacoste, hm, right
[15:21] <Ursinha> flacoste, I'll keep watching if we have more of those
[15:21] <gmb> stub: This is Launchpad; we eat impossible things for breakfast.
[15:21] <flacoste> yeah, if it happens often there is probably either a bug in Mailman or a deployment issue
[15:22] <Ursinha> flacoste, fair enough
[15:22] <Ursinha> that's fine
[15:22] <Ursinha> all for me
[15:22] <matsubara> gmb, that could be the new slogan :-)
[15:22] <Ursinha> lol
[15:22] <matsubara> thanks Ursinha
[15:22] <Ursinha> thanks everyone
[15:22] <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
[15:22] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[15:23] <herb> It's been a quiet week. No cherry picks, no major service affecting problems.
[15:23] <herb> Just to make sure it's staying on everyone's radar, we continue to have daily problems related to bug #156453 and bug #118625.
[15:23] <herb> Further we contine to have problems every couple of days related to bug #260171.
[15:23] <herb> We need to come up with a plan on how to handle Bug #327423 and bug #327455. Just throwing more RAM at the problem isn't likely to be a workable solution. We're happy to do anything we can do to help reduce the impact of these bugs, but we clearly need some input from the devs.
[15:23] <herb> Thats all from us, unless there are questions.
[15:24] <matsubara> sinzui, can you raise the importance of #327423?
[15:24] <sinzui> I can
[15:24] <matsubara> henninge, can you comment and triage accordingly #327455?
[15:25] <henninge> I know that danilo and jtv worked on that and are now able to produce language packs again.
[15:25] <henninge> I don't know how they did that, though.. ;)
[15:25] <herb> henninge: I think they had to use the DB server to make it work though.
[15:25] <henninge> I will poke danilo to commtnt
[15:25] <matsubara> thanks henninge
[15:26] <henninge> herb: ok, so it is not really solved, then?
[15:26] <herb> henninge: to the best of my knowledge, no.
[15:27] <henninge> our team communication is not at its best this week as 2/3 are sprinting heavily ... :-/
[15:27] <herb> henninge: or at least not in a permanent way.
[15:27] <matsubara> all right. thanks herb
[15:28] <herb> thanks
[15:28] <matsubara> [TOPIC] * DBA report (DBA contact)
[15:28] <MootBot> New Topic:  * DBA report (DBA contact)
[15:28] <stub> The production systems are now using the standard startup/shutdown script to control the slony daemons. The wiki documentation has been updated. LOSAs should have sudo access to run the /etc/init.d script.
[15:28] <stub> I'll be investigating database connection load balancers to make best use of the new hardware. pg_pool, pg_pool 2 and pgbouncer are all options I'm aware of to distribute slave connections evenly across the slave backends.
[15:28] <stub> fini.
[15:28] <matsubara> questions for stub?
[15:29] <matsubara> ok, let's move on
[15:29] <matsubara> [TOPIC] * Next Launchpad Performance Week
[15:29] <MootBot> New Topic:  * Next Launchpad Performance Week
[15:29] <matsubara> I updated https://dev.launchpad.net/PerformanceWeeks/April2009
[15:30] <matsubara> with a few page ids that we could tackle for the next performance week
[15:30] <matsubara> if you have more page ids/oopses that you want to work on in the next LPW, please add to the list
[15:31] <matsubara> that's all from me
[15:31] <matsubara> anything else before I close?
[15:31] <Ursinha> nope
[15:31] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[15:31] <matsubara> #endmeeting
[15:31] <MootBot> Meeting finished at 09:31.
[15:32] <Ursinha> thanks matsubara
[15:32] <henninge> Ursinha, matsubara: thanks! :)
[15:32] <Ursinha> henninge, np :)