[18:57]  * Rinchen sobs for mootbot
[18:58] <schwuk> be brave Rinchen
[18:58] <kiko> WTF is mootbot
[18:58] <gmb> It's been eaten by a grue.
[18:58] <sinzui> kiko: It was Rinchen's horse. We shot it when it broke its leg.
[18:59] <schwuk> it is no more
[18:59] <gmb> It is an ex bot
[18:59] <Rinchen> it sunk with seveas
[18:59] <sinzui> It has ceased to be
[18:59] <schwuk> it has shuffled of this mortal coil
[18:59] <Rinchen> there's a meeting coming up to talk about the future of all the bots
[18:59] <Rinchen> like ubotu
[18:59] <bigjools> 'e 'as gone to meet 'is maker
[18:59] <kiko> Rinchen, really?
[18:59] <kiko> dinnaknothat
[18:59] <sinzui> It is an ex-bot
[19:00] <Rinchen> well, mootbot is offline. Dennis is still around. :-)
[19:00] <Rinchen> just not on irc
[19:00] <schwuk> sinzui: gmb beat you to it
[19:00] <mpt> I could pretend to be MootBot
[19:00] <Rinchen> scribes team has had a few offers but they are not sure what to do because mootbot is eggdrop code.  They were talking about building it into ubotu
[19:00] <sinzui> damn
[19:01] <sinzui> Let's move on to the cheese shop sketch then
[19:01] <mpt> (just without the semi-permanent-record part)
[19:01] <Rinchen> eggdrop makes swiss cheese look like high-density concrete
[19:01] <kiko> go!
[19:01] <Rinchen> Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development.
[19:01] <gmb> sinzui: I'll fetch my bourzouki, hang on.
[19:01] <Rinchen> Roll Call
[19:01] <Rinchen> me!
[19:01] <mrevell> me
[19:01] <mpt> me
[19:01] <bigjools> me
[19:01] <gmb> mě
[19:01] <flacoste> me
[19:01] <mars> me
[19:01] <intellectronica> me
[19:01] <bac> me
[19:01] <sinzui> me
[19:01] <salgado> me
[19:01] <adeuring> me
[19:01] <thumper> me
[19:01] <SteveA> me
[19:01] <stub> ,e
[19:01] <barry> me
[19:01] <matsubara> me
[19:01] <leonardr> me
[19:01] <stub> me
[19:01] <schwuk> me
[19:01] <herb> me
[19:01] <kiko> me
[19:01] <jt1> me
[19:02] <danilos> me
[19:02] <statik> me
[19:02] <EdwinGrubbs> me
[19:02] <Rinchen> jt1 = jtv in disguise
[19:02] <Rinchen> Releases is here
[19:02] <Rinchen> foundations, bugs, etc?
[19:02] <jtv> Rinchen: no, that was an impostor.  Here I am.
[19:02] <sinzui> not a very good disguise it seems
[19:02] <statik> lpcomm is here
[19:03] <cprov> me
[19:03] <Rinchen> ok there is soyuz
[19:03] <al-maisan> me
[19:03] <bigjools> soyuz was here ages ago :)
[19:03] <Rinchen> allenap_, BjornT ?
[19:03] <flacoste> Rinchen: foundations all there
[19:03] <Rinchen> and I see thumper...ok, let's go
[19:03] <Rinchen> Agenda
[19:03] <BjornT> me
[19:03] <Rinchen>  * Next meeting
[19:03] <Rinchen>  * Actions from last meeting
[19:03] <Rinchen>  * Oops report (Matsubara)
[19:03] <Rinchen>  * Critical Bugs (Rinchen)
[19:03] <Rinchen>  * Bug tags
[19:03] <Rinchen>  * Operations report (mthaddon/herb)
[19:03] <Rinchen>  * DBA report (stub)
[19:03] <Rinchen>  * Sysadmin requests (Rinchen)
[19:03] <Rinchen>  * New packages required (salgado)
[19:04] <Rinchen>  * A top user-affecting issue (mrevell)
[19:04] <Rinchen>  * Doc Team report (mrevell)
[19:04] <Rinchen> Next meeting
[19:04] <Rinchen> well, there was not much consensus on the rotation option email
[19:04] <Rinchen> Kiko, SteveA -  you're still against two meetings, correct?
[19:05] <kiko> I'm not against anything per se, but we lose a lot by having separate meetings.
[19:05] <allenap_> me, sorry I'm late
[19:05] <kiko> so I find the drawbacks kinda high
[19:05] <SteveA> I'm against split meetings
[19:05] <Rinchen> that leaves us to pick one, or do thumper's 6 hour incrementing meeting
[19:05] <SteveA> I'm fine with having two meetings, provided they're approximately one week apart
[19:05] <Rinchen> lol
[19:05] <kiko> incrementing meetings are a bit scary in that everybody gets confused
[19:05] <thumper> SteveA: :)
[19:05] <thumper> kiko: people are smart enough
[19:05] <thumper> and we have a team calendar
[19:06] <kiko> thumper, speaking from experience on the Ubuntu side, not really.
[19:06] <gmb> Also, we set the next meeting date in the meeting...
[19:06] <kiko> so I'm kinda -1 on rotating meetings
[19:06] <kiko> but +1 on two meeting times alternating
[19:06] <kiko> sounds like nobody liked the times there, though
[19:06] <kiko> could we propose other times?
[19:06] <kiko> I will be at both meetings
[19:06] <thumper> midnight UTC is good for me :)
[19:07] <kiko> worksforme
[19:07] <jtv> gah
[19:07] <Rinchen> I can go back and look for other slots
[19:07] <Rinchen> the 11am UTC one is the one with the most chance more highest attendance
[19:07] <kiko> anyway, if no decision comes in through email this week, same time next week.
[19:07] <kiko> what time is that for jtv?
[19:07] <Rinchen> ok, I'll set up everything for 18:00 UTC for next week and we can change it
[19:07] <kiko> moving on
[19:08] <mpt> kiko, 1am-ish
[19:08] <jtv> kiko: 07:00, or 6 hours before my working day
[19:08] <SteveA> jtv is in thailand.  surely he can get drugs to make the current meeting time palettable
[19:08]  * mpt doesn't know what he's talking about
[19:08] <jtv> SteveA: to me, yes.  To you?  Funny maybe, but...
[19:08] <intellectronica> Rinchen: if so then i'm afraid i'll have to miss next week. sorry!
[19:08] <Rinchen> right then, moving on
[19:08] <Rinchen> oh, thanks intellectronica
[19:08] <Rinchen> Actions from last meeting
[19:08] <Rinchen> none
[19:08] <Rinchen> Oops report (Matsubara)
[19:09] <matsubara> Today's oops report is about bugs 228305, OOPS-855EB78, 228307.
[19:09] <matsubara> intellectronica, can you take 228305. I think it's related to the email interface. It needs further investigation.
[19:09] <matsubara> cprov, wasn't OOPS-855EB78 RCFIXED? Or is that a new one?
[19:09] <kiko> jtv, 7pm?
[19:09] <intellectronica> matsubara: sure, i'll investigate and fix as appropriate
[19:09] <jtv> kiko: midnight UTC is 7am for me.
[19:09] <matsubara> I've noticed some timeouts in +storeblob and +hwdb/+submit pages. I've asked about it to flacoste last Friday. More eyes on the issue would be appreciated(#228307)
[19:09] <kiko> jtv, it's 11am utc.
[19:09] <cprov> matsubara: bigjools fixed it last friday, IIRC
[19:10] <flacoste> my hypothesis was some libarian connection dalys
[19:10] <jtv> kiko: 11am utc is just dandy for me.
[19:10] <matsubara> cprov: the oops is from sunday, on edge
[19:10] <flacoste> but that's just a wild guess
[19:10] <kiko> matsubara, so hmmm something else is amiss?
[19:10] <cprov> matsubara: uhmm ... let me check it again then
[19:11] <SteveA> do we record times taken talking to the HTTP librarian?
[19:11] <SteveA> if not, we should do
[19:11] <SteveA> and these should be available in OOPS reports
[19:11] <SteveA> as a general principle, any time we do something that talks to another process or another computer, we should record the time taken
[19:11] <cprov> matsubara: no, that's a new oops, needs a bug.
[19:12] <matsubara> cprov: I'll file one after the meeting. thanks for checking
[19:12] <kiko> SteveA, for the upload? we do not.
[19:12] <matsubara> SteveA: I don't think we do.
[19:12] <cprov> matsubara: np, thank you
[19:12] <matsubara> SteveA: I mean, I don't think we do the logging
[19:12] <SteveA> that's too bad.  it would help us out in knowing what's going on.
[19:13] <flacoste> librarian logging sucks also, but that's probably another issue
[19:14] <Rinchen> matsubara, anything else?
[19:14] <matsubara> Rinchen: that's all from me.
[19:14] <Rinchen> thanks
[19:14] <Rinchen> Critical Bugs (Rinchen)
[19:14] <matsubara> thanks all. I'll update the bug
[19:14] <Rinchen> Memory issue.   flacoste, can we reopen the bug since it's still a problem, or create a new one? How is the current investigation going?
[19:15] <SteveA> matsubara: would you file a bug on the librarian client that it should record times for HTTP calls
[19:15] <flacoste> Rinchen: file a new one
[19:15] <matsubara> SteveA: sure. doing it now
[19:15] <flacoste> and SteveA thinks it might be related to batch size
[19:15] <SteveA> thanks
[19:15] <flacoste> i'm going to put a hard limit on it
[19:15] <flacoste> 300
[19:15] <stub> (watch out for the exception handler that uploads the exception to the librarian - might end up in a loop)
[19:15] <SteveA> Rinchen: earlier, tom, jtv and I did some experiments on staging
[19:15] <kiko> I am okay with that. would be even happier with 500 but anyway.. :)
[19:15] <flacoste> i'm going to file a bug about the batch size issue
[19:15] <jtv> flacoste: I just discussed the issue with sinzui as well, so
[19:15] <Rinchen> thanks flacoste
[19:16] <jtv> flacoste: can we get together and compare notes?
[19:16] <Rinchen> https://bugs.edge.launchpad.net/rosetta/+bug/224617
[19:16] <Rinchen> jtv - is this really critical or simply high?
[19:16] <jtv> flacoste: (I have a bug for the Translations side)
[19:16] <SteveA> and we could see a large-ish memory increase for translation pages with large batch sizes
[19:16] <jtv> Rinchen: that's critical to the Firefox people.
[19:17] <jtv> Rinchen: the fix landed today, and I'm negotiating a CP
[19:17] <Rinchen> jtv, k, thanks.
[19:17] <kiko> Rinchen, jtv: I'm going to chat with asac; meanwhile jtv will test on staging.
[19:17] <Rinchen> https://bugs.edge.launchpad.net/launchpad/+bug/224623
[19:17] <Rinchen> DB load.  stub, mthaddon - this hasn't been worked on. Therefore I submit that it's not a critical bug but rather high. Do you agree?
[19:17] <Rinchen> :-)
[19:17] <kiko> agreed.
[19:17] <flacoste> jtv: sure
[19:17] <flacoste> jtv: grab me after meeting
[19:17] <jtv> flacoste: not now though; it's deep night and I've been at it for 15 hours straight
[19:18] <Rinchen> herb, if you could pass that on
[19:18] <herb> will do
[19:18] <flacoste> jtv: ok, i'll probably fix it later today though
[19:18] <Rinchen> I'll go ahead and lower this. If it happens again, herb if you could get Tom to update the details in the bug report please.  thanks
[19:18] <stub> I'll flag Bug 224623 as incomplete - there is no way to progress the bug report.
[19:18] <jtv> flacoste: okay, then we'll have to do it right after
[19:18] <herb> Rinchen: got it.
[19:18] <SteveA> Rinchen: we should adapt some code that gustavo wrote for landscape
[19:18] <Rinchen> ok, thanks stub
[19:19] <SteveA> Rinchen: to limit the connections that are accepted from the network into the webapp
[19:19] <SteveA> Rinchen: based on the size of the queue of connections waiting for app threads.
[19:19] <Rinchen> Sounds like a job for spec circumstances or foundations.
[19:19] <SteveA> Rinchen: that will keep launchpad responsive even in situations that look like the one described by tom in the bug report
[19:19] <SteveA> it's a foundations job, for post 2.0
[19:20] <SteveA> which we can look at moving forward if it happens again
[19:20] <Rinchen> ok, I'll add that to the laundry list. :-)
[19:20] <Rinchen> That's it from me.
[19:20] <Rinchen> Operations report (mthaddon/herb)
[19:20] <herb> Highlights from the last week:
[19:20] <herb>  - 05/02: Production re-roll.
[19:20] <herb>  - 05/05: lpnet6 (running with the debug config) stopped responding and was restarted.
[19:20] <herb>  - 05/06: lpnet4 died and was restarted.
[19:20] <herb>  - 05/06: Cherry pick r6222 to lpnet*
[19:20] <herb>  - 05/07: Restarted librarian after a 13 minute outage.
[19:20] <herb> New edge server running hardy should be up and in the rotation starting tomorrow.
[19:20] <herb> No update from us on the memory issue. The debug instance (lpnet6) doesn't seem to be growing in the same way that the non-debug instances are. The typical lpnet process seems to grow to 800-1000MB RSS (3x-5x it's initial RSS) within a few hours and stay there idefinitely.
[19:20] <herb> That's it from Tom and me unless there are any questions.
[19:21] <stub> So we should fix the memory issue by running in debug mode? :-/
[19:21] <SteveA> herb: did you get anyone to look at lp6 when it stopped responding?
[19:21] <kiko> what's the overhead in running debug mode?
[19:21] <SteveA> kiko: lots
[19:21] <SteveA> kiko: we run single threaded
[19:22] <SteveA> and it doesn't fix the memory issues
[19:22] <herb> SteveA: Tom handled it.  so I don't know.  I know he copied the ref* files over in case there was something of value in there.
[19:22] <jtv> "Less damage per second"
[19:22] <SteveA> it makes them less likely to occur, because the server handles fewer requests
[19:22] <flacoste> maybe 800-1000Mb is what LP needs as working memory to process our requests that perform badly
[19:22] <herb> 1/16th the number of requests.
[19:22] <SteveA> herb: if an app server becomes unresponsive and it's only one, please take it out of rotation in the load balancer
[19:22] <SteveA> and then find someone ifrastructural to look into it
[19:23] <SteveA> if two go down in the same way, then restart the second one, leaving the first one hung
[19:23] <SteveA> please add this to the LOSA operational manual, or whatever :-)
[19:23] <herb> SteveA: ok.
[19:23] <SteveA> we can usually afford to lose one app server
[19:24] <Rinchen> anything else for herb?
[19:24] <herb> SteveA: when you say someone infrastructural, who should we be looking for?
[19:24] <SteveA> and it's good to have the opportunity to diagnose it
[19:24] <SteveA> a member of the foundations team
[19:24] <SteveA> or SC
[19:24] <SteveA> or me
[19:24] <herb> SteveA: ok
[19:24] <Rinchen> herb, you can ping steve, kiko, or I on -code if it happens and we can help you find someone
[19:24] <herb> Rinchen: sounds good.
[19:24] <Rinchen> thanks herb
[19:24] <Rinchen> off we go....
[19:24] <Rinchen> DBA report (stub)
[19:25] <stub> The production DB server is being upgraded to hardy as soon as IS can schedule it. This involves downtime.
[19:25] <stub> Hopefully we get the PQM box running Hardy at around the same time. This means we can switch to PG 8.3 for development.
[19:25] <stub> If things go to plan, we can upgrade production to PG 8.3 later with minimal downtime. I need to test using Slony-I to perform the migration with real data on Carbon once it has been Hardified.
[19:25] <stub> Devs will need to switch to PG 8.3 when PQM switches. You are welcome to switch earlier if you want. The docs are already updated on the Wiki.
[19:25] <stub> Had a good discussion on the person/auth split with jamesh and refactored the model again. The diagram on the wiki page has been updated (and no longer matches the text).
[19:25] <stub> OOT.
[19:26] <Rinchen> Better watch out kiko, I'm going to hardify you
[19:26] <thumper> oot?
[19:26] <Rinchen> lol
[19:26] <stub> over and out
[19:26] <SteveA> out of time
[19:26] <thumper> ah
[19:26] <kiko> Rinchen, I'm on hardy!!
[19:26] <SteveA> our obstinate technologist
[19:26] <Rinchen> jtv, put down that GTA list right now and step away from the computer
[19:26] <jtv> Rinchen: it's GTF
[19:26] <SteveA> our other thailander
[19:26] <Rinchen> Thanks stub
[19:26] <jtv> Rinchen: and you're too late: it's already uploading
[19:27] <Rinchen> Sysadmin requests (Rinchen)
[19:27] <Rinchen> Is anyone blocked on an RT or have any that are becoming urgent?
[19:27] <mrevell> our original Thailander?
[19:27] <Rinchen> and bigjools, I didn't look, are you done?
[19:27] <SteveA> mrevell: nice
[19:27] <bigjools> Rinchen: yes!  \o/
[19:27] <Rinchen> yippee
[19:28] <bigjools> Rinchen: having said that I need to talk to flacoste about the restricted librarian rollout, which will need another RT
[19:28] <flacoste> i suck
[19:28] <flacoste> sorry, completely forgot that one
[19:28] <kiko> jesus guys
[19:28] <Rinchen> Pass it my way and I'll do the priority magic on it
[19:28] <kiko> this was due last CENTURY
[19:29] <Rinchen> I'll get SteveA to do a rain dance too
[19:29] <bigjools> flacoste: are you free after the meeting sometime?
[19:29] <Rinchen> kiko, yeah, bigjools ppa fought the Hardy release and Hardy won
[19:30] <Rinchen> New packages required (salgado)
[19:30] <flacoste> bigjools: tomorrow might be better, i have three persons already in line :-(
[19:30] <salgado> anything to add to launchpad-dependencies this week?
[19:30] <Rinchen> beer?
[19:30] <SteveA> so...
[19:30] <kiko> not that I can think of
[19:30] <SteveA> if launchpad-dependencies were managed in /sourcecode
[19:30] <kiko> thanks salgado!
[19:30] <SteveA> then we could have a check on 'make run'
[19:30] <SteveA> that warned if launchpad-dependencies is not of the appropriate version for this LP tree
[19:31] <flacoste> SteveA: we can have a check even if it's not managed in sourcecode
[19:31] <SteveA> sorry to bring up this old chestnut, but I miss xmas
[19:31] <SteveA> flacoste: it means we need to record the version number somewhere, and keep it up to date
[19:31] <SteveA> flacoste: rather than just work off what's in /sourcecode
[19:32] <Rinchen> SteveA, how about you file a bug for that and tell flacoste about it?
[19:32] <kiko> SteveA, well, depends if sourcecode is linked out of somewhere or not
[19:32] <kiko> i.e. if it is shared
[19:32] <flacoste> Won't fix
[19:32] <Rinchen> we won't lose your chestnut then
[19:33] <Rinchen> flacoste, well, you can negotiate that with your boss ;-)
[19:34] <Rinchen> going once
[19:34] <flacoste> we need more value out of it to offset the additional process cost
[19:34] <Rinchen> ok, sounds like an offline discussion is in order.  ACTION: Steve and Francis to chat about dependencies and make check
[19:35] <SteveA> thanks, but it's fine
[19:35] <Rinchen> A top user-affecting issue (mrevell)
[19:35] <mrevell> howdy
[19:35] <SteveA> I will go with flacoste's judgement on this issue.  I'm sure we'll revisit it later, once we move more of launchpad code into launchpad.
[19:35] <mrevell> A common theme over the past week - although we haven't been inundated with requests - has been requests to either report spam or edit existing bug comments.
[19:35] <mrevell> As we've discussed this in meetings before, I'd be interested in hearing from other people who have either dealt with or seen an interesting user-affecting issue.
[19:36] <flacoste> SteveA: actually, once servers are upgraded to hardy, the cost will lessen, so we should revisit at that time
[19:36] <intellectronica> one thing i encountered is users wanting to edit or delete their own comments
[19:36] <SteveA> we need to have "add a 'report this comment' link on comments" and the same for other user-submitted content
[19:36] <intellectronica> for example, because they disclosed information they would rather not, accidentally
[19:36] <SteveA> and that should be on the post-2.0 list
[19:37] <SteveA> services like facebook do this well
[19:37] <mrevell> intellectronica: Yes, I've seen that come up too. There's a problem of editing history there, though, isn't there?
[19:37] <stub> Why is that a problem?
[19:37] <intellectronica> "other user-submitted content"? almost all of it is...
[19:37] <intellectronica> mrevell: what do you mean by "editing history"?
[19:37] <kiko> intellectronica, changing what I said in the past.
[19:37] <mpt> mrevell, not if it's within five minutes or before anyone else comments.
[19:37] <mrevell> intellectronica: Someone could edit their comment to cast themselves in a better light, perhaps.
[19:38] <mrevell> mpt: Hmm, fair point.
[19:38] <mpt> (5 minutes being when the mail notification goes out)
[19:38] <Rinchen> There is a general topic of spam handling that I've already added on behalf of the OSAs
[19:38] <stub> Why is that a problem? We are writing a bug tracker and things - not a banking system.
[19:38] <Rinchen> on the post-2.0 list
[19:38] <mrevell> Do we have a bug report? I didn't see one.
[19:38] <mpt> We have bug reports on both those issues
[19:38] <mpt> spam is bug 45419
[19:38] <SteveA> I think we should always keep the comments
[19:39] <mpt> I don't remember the editing one offhand
[19:39] <mrevell> mpt: Yeah, I saw the spam but not the edit history. I'll look again, thanks.
[19:39] <SteveA> and hide them, and have a link saying "N comments removed"
[19:39] <SteveA> openness is important
[19:39] <mpt> bug 80895
[19:39] <mrevell> thanks mpt
[19:39] <mpt> np
[19:40] <mrevell> As we have a plan, I'm done.
[19:40] <kiko> yep
[19:40] <Rinchen> Doc Team report (mrevell)
[19:40] <kiko> it's post 2.0 unfo
[19:40] <intellectronica> mpt: i'm not sure 5 minutes is enough. i realise that after that we've sent an email anyway, but i suspect that in some cases users will still want to remove a comment
[19:40] <BjornT> SteveA: a db patch was landed last cycle, which will allow us to hide comment. we just need to implement it.
[19:40] <mrevell> I'd like to give the doc team members a list of bite-sized items that would be outside the scope of our normal docs work. If you have ideas for such work, that involves your part of LP, please let me know.
[19:40] <mrevell> On a docs-related note: Rinchen, Statik and I are recording a new episode of the Launchpad podcast next week. All ideas, such as suggestions of CC-licensed music for the theme and a name are welcome :)
[19:40] <mpt> intellectronica, then they can put themselves into the "Inappropriate" queue just like anyone marking spam
[19:40] <mpt> and be moderated accordingly
[19:41] <SteveA> mrevell: a new episode?
[19:41] <SteveA> mrevell: we have episodes? what's the RSS feed?  I'll add it to my reader
[19:41] <intellectronica> mpt: maybe you're right. anyway, there's a bug, so let's continue there
[19:41] <mpt> sure
[19:41] <jtv> 4-minute warning...
[19:41] <gmb> SteveA: I think there was one too many adjectives in mrevell's sentence.
[19:41] <mrevell> SteveA: Well, let's say episode 1.0, to the version 0 that I posted last year. The feed is...
[19:42] <mrevell> http://news.launchpad.net/category/podcast/feed
[19:42] <SteveA> thanks
[19:42] <mrevell> Thanks Rinchen
[19:42] <Rinchen> Thanks.
[19:42] <Rinchen> Thank you all for attending this week's Launchpad Developer Meeting.
[19:42] <Rinchen> The end!
[19:42] <thumper> thanks Rinchen
[19:43] <SteveA> thanks Rinchen !
[19:43] <jtv> thanks Rinchen!
[19:43]  * Rinchen laments the loss of Mootbot.
[19:43] <SteveA> mootbot, I mourns it
[19:43] <Rinchen> I'll just have to get the scissors out and cut up this log.
[19:43] <kiko> woo
[19:43] <Rinchen> matsubara, pass me the glue eh?
[19:44] <Rinchen> gmb, can I use your scissors? Yours seem awfully sharp.
[19:44] <intellectronica> did anybody say "glue"?
[19:44]  * intellectronica sniffles 
[19:44] <gmb> Rinchen: Alas, I only use scalpels.
[19:45] <Rinchen> gmb, ok, I'll check with sinzui then.  Last I knew his where pink
[19:45] <Rinchen> or maybe they were pinking shears....