[19:00] <Rinchen> #startmeeting
[19:00] <Rinchen> Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development.
[19:00] <Rinchen> hmm
[19:00]  * Rinchen pokes MootBot 
[19:00] <Rinchen> mootbot is dead
[19:00] <mrevell> long live mootbot
[19:00] <mrevell> me
[19:00] <mrevell> oops
[19:00] <Rinchen> Well, Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development.
[19:00] <sinzui> or MootBot is very drunk again
[19:01] <Rinchen> [TOPIC] Roll Call
[19:01] <Rinchen> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[19:01] <Ursinha> haha
[19:01] <Rinchen> me!
[19:01] <mrevell> me
[19:01] <bigjools> me
[19:01] <gmb> me (what timing!)
[19:01] <cprov> me
[19:01] <salgado> me
[19:01] <beuno> me
[19:01] <flacoste> me
[19:01] <Ursinha> me
[19:01] <stub> me
[19:01] <mthaddon> me
[19:01] <thumper> me
[19:01] <EdwinGrubbs> me
[19:01] <intellectronica> me
[19:01] <sinzui> me
[19:01] <adeuring> me
[19:01] <mars> me
[19:01] <BjornT> me
[19:01] <mwhudson> me
[19:01] <flacoste> barry is missing this meeting
[19:01] <Rinchen> as will abently
[19:01] <Ursinha> where is matsubara?
[19:01] <al-maisan> me
[19:01] <allenap> me
[19:02] <BjornT> bugs team is here
[19:02] <bigjools> soyuz here
[19:02] <Rinchen> rockstar, ping
[19:02] <rockstar> me
[19:02] <flacoste> bac, leonardr: ping
[19:02] <Rinchen> Releases is missing Diogo.. we're trying to find him
[19:03] <matsubara> me
[19:03] <rockstar> Found him!
[19:03] <Ursinha> haha
[19:03] <Rinchen> :-)
[19:03] <Ursinha> :)
[19:03] <bac> me
[19:03] <matsubara> sorry
[19:03] <Rinchen> ok Releases is here
[19:03] <matsubara> I was grabbing something to eat
[19:03] <Rinchen> danilos, here?
[19:03] <Rinchen> jtv, here?
[19:03] <jtv> Rinchen: yes, just got in
[19:04] <Rinchen> ok, we'll go ahead then since we'll mostly here
[19:04] <leonardr> me
[19:04] <kiko> me
[19:04] <kiko> I'm here, I'm here!
[19:04] <jtv> who?
[19:04] <Rinchen> [TOPIC] Agenda
[19:04] <Rinchen>  * Next meeting
[19:04] <Rinchen>  * Actions from last meeting
[19:04] <Rinchen>  * Oops report & Critical Bugs (matsubara/ursinha)
[19:04] <Rinchen>  * Operations report (mthaddon/herb/spm)
[19:04] <Rinchen>  * DBA report (stub)
[19:04] <Rinchen>  * Sysadmin requests (Rinchen)
[19:04] <Rinchen>  * New packages required (salgado
[19:04] <Rinchen>  * Next meeting
[19:04] <Rinchen> hm
[19:04] <Rinchen> Next meeting
[19:05] <Rinchen> kiko, will we have a mtg next week?
[19:05] <mrevell> Rinchen: I'm away next Thursday.
[19:05] <kiko> I think we should if there's somebody that would like to run it
[19:05] <mthaddon> herb will be away next Thursday
[19:05] <gmb> So will I.
[19:05] <Rinchen> The team leads will be sprinting
[19:05] <sinzui> I may not be here next week
[19:06] <rockstar> Maybe it would be better to ask who will be here?
[19:06] <flacoste> Rinchen: Foundations is complete (apart barry which is excused)
[19:06] <Rinchen> I propose skipping next week and starting on the 4th
[19:06] <Ursinha> rockstar, +1 :)
[19:06] <kiko> Ursinha, matsubara: why don't you guys run the meeting?
[19:06] <mwhudson> (i am unlikely to be here but then you don't expect me anyway)
[19:06] <Ursinha> Ursinha, i can try that, matsubara, what do you think?
[19:06] <matsubara> kiko, you mean next week's meeting or all meetings from now on?
[19:06] <mthaddon> matsubara, well volunteered!
[19:06] <Ursinha> talking to myself is not good
[19:07] <kiko> next week? :)
[19:07] <matsubara> kiko, sure thing
[19:07] <Ursinha> cool
[19:07] <kiko> cool
[19:07] <Rinchen> ok, then, thanks.
[19:08] <Rinchen> we will have a meeting next with with several apologies
[19:08] <Rinchen> Actions from last meeting
[19:08] <Rinchen> We had two
[19:08] <Rinchen>   * mwhudson to get with mthaddon to see if we can super-charge gdb to give us better diagnostic info on the staging
[19:08] <Rinchen>   * Joey to review RT ticket Priorities in RT to ensure they are accurate.
[19:08] <mwhudson> so we set some stuff up, but didn't collect any data aiui
[19:08] <mthaddon> I think the first one ended up being herb, but I know there was some work done on that
[19:09] <Rinchen> I did a once through on the RTs so they are in order as far as I can see for those who gave priorities. I did do some cleanup
[19:09] <mwhudson> (because now we're looking at it staging so closely it's resolutely refusing to crash)
[19:09] <kiko> herb_, mwhudson: really? I am a bit surprised because I think linkchecker didn't finish running and I thought that was the cause
[19:09] <mthaddon> staging has crashed numerous times over the past week
[19:10] <mthaddon> although to be fair, the last time seems to have been on the 18th
[19:10] <mwhudson> mthaddon: oh, did my gdb voodoo work then?
[19:10] <mthaddon> mwhudson, not sure - as I say, it's herb that's been working on this
[19:10] <mwhudson> because the last time i got spm to look, my stuff had gone walkabout and it was the SIGPIPE thing again
[19:11] <mwhudson> (which might have been on the 18th, indeed)
[19:12] <kiko> mthaddon, herb_: we /really/ need to be able to run staging interactively.
[19:12] <mthaddon> kiko, we've had this discussion before - I'm still getting you an answer on that one
[19:12] <kiko> mthaddon, not me as in we. we as in OSAs :)
[19:13] <Rinchen> mthaddon, herb_, spm - can you please maintain vigilance on staging and when gdb kicks in to contact one of the LP devs?
[19:13] <mthaddon> oh okay :)
[19:13] <kiko> mthaddon, or is that the same problem?
[19:13] <mthaddon> no, I don't think that's the same problem
[19:13] <kiko> I mean, I consider myself an OSA. just a particurly underprivileged one. :)
[19:13] <mthaddon> I'll check with herb about the gdb issue when he's back
[19:14] <kiko> mthaddon, sanks
[19:14] <Rinchen> ok, so we'll keep this action item open but replace mwhudson with "LP devs"
[19:14] <mthaddon> yah
[19:14] <kiko> Rinchen, meaning nobody's responsible?
[19:14] <Rinchen> kiko, meaning the OSAs are responsible
[19:14] <kiko> I want somebody to own this problem and to actually want to fix it
[19:15] <Rinchen> and to find whomever they can when it happens
[19:15] <Rinchen> regardless of the hour
[19:15] <kiko> is that the right direction for this vector, mthaddon?
[19:15] <mthaddon> sure
[19:16] <Rinchen> thanks... moving on then...
[19:16] <Rinchen>  * Oops report & Critical Bugs (matsubara/ursinha)
[19:16] <Ursinha> alright
[19:16] <Ursinha> two new bugs, one question for thumper
[19:16] <Ursinha> for translations, bug 259433, who can take a look at it, jtv or danilos? A candidate to RC, i think
[19:16] <thumper> Ursinha: shoot
[19:17] <Ursinha> thumper, about that two items in the test plan that were hanging (stacked branches, i guess), what was the solution to them in the end?
[19:17] <jtv> Ursinha: I'll have a look, thanks
[19:17] <matsubara> that one is quite weird, I initially thought was a hacked url, but it OOPSes even with no parameters
[19:17] <Ursinha> jtv, thanks
[19:17] <thumper> Ursinha: it's bad, very bad
[19:18]  * thumper is unhappy about the first bad item
[19:18] <Ursinha> thumper, go ahead :)
[19:18] <thumper> Ursinha: bad as in broken, not working, fubared
[19:18] <Ursinha> holy cow
[19:18] <matsubara> thumper, hmm did you disabled the feature for now?
[19:18] <mwhudson> no, that was something else
[19:18] <thumper> as much as we can, yes
[19:19] <mwhudson> no stacking will happen by default, which limits the carnage
[19:19] <matsubara> thumper, mwhudson: cool. can we expect a fix ready for the second rollout?
[19:19] <Ursinha> thumper, too bad that it will take forever to fix?
[19:20] <mwhudson> matsubara: when is the second rollout?
[19:20] <thumper> I'm not sure that we know exactly what the problem is yet
[19:20] <kiko> mwhudson, when would you like it?
[19:20] <matsubara> when kiko/joey says so
[19:20] <mwhudson> matsubara: but i guess the answer is going to be "maybe", the problems aren't well understood yet :/
[19:20] <mwhudson> (too many moving parts)
[19:21] <matsubara> all right. keep us posted on that one. I'll add that to the CurrentRolloutBlockers page
[19:21] <Ursinha> so we can't really expect them as items to be RC
[19:21] <Ursinha> thanks matsubara
[19:21] <kiko> that sucks
[19:21] <Ursinha> ok, moving on
[19:21] <mwhudson> i guess it's what we'll be working on today
[19:21] <mwhudson> kiko: yep
[19:22] <kiko> mwhudson, can I help somehow?
[19:22] <mwhudson> kiko: if we can think of a way, we'll let you know, but i guess it's a me, thumper, spm job mostly
[19:23] <kiko> mwhudson, okay, I'm your man if you need some effort and energy to hack away with it
[19:23] <Ursinha> cool
[19:23] <Ursinha> may i move on?
[19:23] <matsubara> go ahead
[19:24] <Ursinha> for foundations, i guess, bug 260140 - an annoying timeout. Who can take that?
[19:24] <Rinchen> go4it
[19:24] <matsubara> bac or EdwinGrubbs ^?
[19:24] <flacoste> Ursinha: that's for thumper's team, i reassigned
[19:24] <thumper> that is probably me
[19:24]  * mwhudson points at thumper 
[19:24] <Ursinha> flacoste, thanks
[19:24] <flacoste> 'retargeted' is the proper term, thumper will assign
[19:24] <thumper> although it is purely DB query optimisation
[19:25] <Ursinha> thumper, i thought so
[19:25] <Ursinha> is that hard to fix?
[19:25] <thumper> so stub is probably most able to understand it
[19:25] <Ursinha> thumper, i'll assign him the bug
[19:25] <kiko> thumper, maybe somebody on your team could have a call with him later today?
[19:26] <Ursinha> and then you could talk to him about it
[19:26] <thumper> kiko: sure, I'll try to get hold of him later
[19:26] <Ursinha> ok, fine
[19:26] <Ursinha> i'm done here
[19:26] <matsubara> for the critical bugs session
[19:26] <Ursinha> thanks guys
[19:26] <kiko> thanks Ursinha
[19:27] <flacoste> kiko, thumper: after this meeting is usually not the best time to speak to stub, it's pretty late for him
[19:27] <matsubara> we have 5 critical bugs, all of them Fix Committed.
[19:27] <Ursinha> matsubara, the same last week
[19:27] <matsubara> so, good job guys!
[19:27] <Rinchen> thanks Ursinha and matsubara
[19:27] <matsubara> that's all from me. thanks
[19:27] <kiko> flacoste, I meant tomorrow morning?
[19:27] <flacoste> kiko: right, that's better
[19:27] <Rinchen>  * Operations report (mthaddon/herb/spm)
[19:28] <mthaddon> 2.1.8 rollout:
[19:28] <mthaddon> • default LPCONFIG has mailman build - now fixed by explicitly calling LPCONFIG for all make builds
[19:28] <mthaddon> • vostok connecting to internal xmlrpc server - temp fix in place, will be figuring out infrastructure issues later
[19:28] <mthaddon> Continuing issues with app servers dying and leaving a stale pidfile(average 2-3 per day) - gdb testing on staging since 19th, hasn't yet crashed since then
[19:28] <mthaddon> Codebrowse has needed a few restarts over the past week
[19:28] <mthaddon> Codehosting needed a restart too (31 lp-serve processes of 32MB each, some up to 19 days old)
[19:28] <mthaddon> Rollout time should be 22:00 UTC, not 00:00 UTC, right? If so, the blog will need updating
[19:28] <mthaddon> That's it from herb, spm and me unless there are any questions
[19:28] <thumper> flacoste: I'll talk when I see him turn up later
[19:28] <mwhudson> mthaddon: any movement on the internal xml-rpc thing yet?
[19:28] <Rinchen> mthaddon, do we have logs for the codehosting restart?
[19:28] <mthaddon> mwhudson, not yet - will be following up post-meeting
[19:29] <mwhudson> mthaddon: ok
[19:29] <mthaddon> Rinchen, same as usual logs, yeah
[19:29] <Rinchen> Do we have a bug for it?  That seems rather...unusual.
[19:29] <Rinchen> (the event)
[19:29] <mthaddon> Rinchen, no bug as yet - can create one
[19:30] <Rinchen> mthaddon, please and pass it to Ursinha, matsubara, and thumper  for investigation
[19:30] <mthaddon> will do
[19:30] <Rinchen> thank you sir
[19:30] <mthaddon> Rinchen, can you confirm on the rollout time question as well?
[19:31] <Rinchen> kiko, 2nd code update tomorrow at 22?
[19:31] <Rinchen> er
[19:31] <mthaddon> Rinchen, I was meaning for future rollouts (2.1.9, etc.)
[19:31] <Rinchen> that won't work will it
[19:31] <kiko> Rinchen, or tonight?
[19:31] <Rinchen> yeah, should be tonight
[19:32] <Rinchen> mthaddon, yeah, let's set them for 22:00 and we'll try to be aggressive for getting PQM's last job on by 20:00
[19:32] <Rinchen> mrevell, ^^
[19:32] <mrevell> noted, thanks Rinchen
[19:32] <mthaddon> Rinchen, that'd be great
[19:32] <Rinchen> 22:00 seems to be working better for us than 00:00 ....  just seems to go smoother with less stress
[19:32] <Rinchen> and I get to go home before 9pm :-)
[19:32] <mthaddon> Rinchen, absolutely - esp if we need SA help
[19:32] <mrevell> mthaddon: Still want to allow a two hour window in announcements?
[19:33] <mthaddon> mrevell, I think it's best to leave it as that - gives us padding - maybe "up to 2 hours"?
[19:33] <Rinchen> +1 on "up to 2 hours"
[19:33] <mrevell> mthaddon: Yes, makes sense. Great, thanks.
[19:33] <Rinchen> Anything else for the LOSAs?
[19:33] <Rinchen> ok,
[19:33] <Rinchen>  * DBA report (stub)
[19:33] <stub> Nothing thrilling happening I can think of.
[19:34] <Rinchen> yay! you're still awake
[19:34] <stub> Not for long...
[19:34] <Rinchen> I don't have anything for stub today.  Does anyone else?
[19:35] <Ursinha> Rinchen, only the bug we've just talked about
[19:35] <Rinchen> right.
[19:35] <Ursinha> assigned to him already
[19:35] <Rinchen> And that'll be taken care of after stub rises from the dead tomorrow
[19:35] <Rinchen> ok, thanks stub.  nighty night
[19:35] <Rinchen>  * Sysadmin requests (Rinchen)
[19:35] <Rinchen> Is anyone blocked on an RT or have any that are becoming urgent?
[19:35] <Rinchen> I've made some progress with IS this week
[19:36] <Rinchen> on some higher priority tickets
[19:36] <Rinchen> The two tickets I'm currently still tracking are
[19:36] <allenap> I can't merge until RT 31486 is in, but it's not urgent.
[19:36] <Rinchen> #31131 AutoReply: Jackass needs to access the Launchpad restricted librarian
[19:36] <Rinchen> #31296: convert staging code import machine for production use, set up VMs on remaining staging box
[19:37] <Rinchen> ok allenap, I'll have a look at that and set a priority and date for it
[19:37] <Rinchen> anyone else?
[19:37] <mwhudson> Rinchen: the codeimport one is partially in my court at the moment
[19:37] <allenap> Rinchen: Thanks :)
[19:38] <Rinchen> k, I'll keep it on my radar mwhudson
[19:38] <mwhudson> (we need to make some code changes to reduce disk space usage)
[19:38] <Rinchen> ok, thanks
[19:38] <Rinchen>  * New packages required (salgado)
[19:38] <salgado> anything for me?
[19:39] <Rinchen> salgado, Jamesh's request?
[19:39] <salgado> copying psycopg2 to lpdebs?
[19:39] <salgado> (already did that one)
[19:39] <salgado> is there another?
[19:40] <Rinchen> no but I needed that one to remove deb http://ppa.launchpad.net/jamesh/ubuntu hardy main from the newlaunchpadder start up list
[19:40] <Rinchen> I'll go remove that and email everyone to remove that from apt
[19:40] <Rinchen> thanks salgado
[19:41] <Rinchen> and with that, I come to the end of the agenda!
[19:41] <Ursinha> yay
[19:41] <Rinchen> Anything anyone wants to add before I close?
[19:41] <Rinchen> ok then!
[19:41] <Rinchen> Thank you all for attending this week's Launchpad Developer Meeting. See the channel topic for the location of the logs.
[19:42] <Rinchen> except there won't be any logs for today from mootbot :-(
[19:42] <mwhudson> thanks Rinchen
[19:42] <mrevell> thanks rinchen, everyone
[19:42] <al-maisan> bye!
[19:43]  * Ursinha waves goodbye
[19:46] <mohbana_> hi