/srv/irclogs.ubuntu.com/2008/05/08/#launchpad-meeting.txt

=== cprov is now known as cprov-afk
=== cprov-afk is now known as cprov
=== mrevell is now known as mrevell-lunch
=== salgado-afk is now known as salgado
=== mrevell-lunch is now known as mrevell
=== EdwinGrub is now known as EdwinGrubbs
=== salgado is now known as salgado-lunch
=== salgado-lunch is now known as salgado
* Rinchen sobs for mootbot18:57
schwukbe brave Rinchen18:58
kikoWTF is mootbot18:58
gmbIt's been eaten by a grue.18:58
sinzuikiko: It was Rinchen's horse. We shot it when it broke its leg.18:58
schwukit is no more18:59
gmbIt is an ex bot18:59
Rinchenit sunk with seveas18:59
sinzuiIt has ceased to be18:59
schwukit has shuffled of this mortal coil18:59
Rinchenthere's a meeting coming up to talk about the future of all the bots18:59
Rinchenlike ubotu18:59
bigjools'e 'as gone to meet 'is maker18:59
kikoRinchen, really?18:59
kikodinnaknothat18:59
sinzuiIt is an ex-bot18:59
Rinchenwell, mootbot is offline. Dennis is still around. :-)19:00
Rinchenjust not on irc19:00
schwuksinzui: gmb beat you to it19:00
=== EdwinGrub is now known as EdwinGrubbs
mptI could pretend to be MootBot19:00
Rinchenscribes team has had a few offers but they are not sure what to do because mootbot is eggdrop code.  They were talking about building it into ubotu19:00
sinzuidamn19:00
sinzuiLet's move on to the cheese shop sketch then19:01
mpt(just without the semi-permanent-record part)19:01
Rincheneggdrop makes swiss cheese look like high-density concrete19:01
kikogo!19:01
RinchenWelcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development.19:01
gmbsinzui: I'll fetch my bourzouki, hang on.19:01
RinchenRoll Call19:01
Rinchenme!19:01
mrevellme19:01
mptme19:01
bigjoolsme19:01
gmbmě19:01
flacosteme19:01
marsme19:01
intellectronicame19:01
bacme19:01
sinzuime19:01
salgadome19:01
adeuringme19:01
thumperme19:01
SteveAme19:01
stub,e19:01
barryme19:01
matsubarame19:01
leonardrme19:01
stubme19:01
schwukme19:01
herbme19:01
kikome19:01
jt1me19:01
danilosme19:02
statikme19:02
EdwinGrubbsme19:02
Rinchenjt1 = jtv in disguise19:02
RinchenReleases is here19:02
=== jt1 is now known as jtv
Rinchenfoundations, bugs, etc?19:02
jtvRinchen: no, that was an impostor.  Here I am.19:02
sinzuinot a very good disguise it seems19:02
statiklpcomm is here19:02
cprovme19:03
Rinchenok there is soyuz19:03
al-maisanme19:03
bigjoolssoyuz was here ages ago :)19:03
Rinchenallenap_, BjornT ?19:03
flacosteRinchen: foundations all there19:03
Rinchenand I see thumper...ok, let's go19:03
RinchenAgenda19:03
BjornTme19:03
Rinchen * Next meeting19:03
Rinchen * Actions from last meeting19:03
Rinchen * Oops report (Matsubara)19:03
Rinchen * Critical Bugs (Rinchen)19:03
Rinchen * Bug tags19:03
Rinchen * Operations report (mthaddon/herb)19:03
Rinchen * DBA report (stub)19:03
Rinchen * Sysadmin requests (Rinchen)19:03
Rinchen * New packages required (salgado)19:03
Rinchen * A top user-affecting issue (mrevell)19:04
Rinchen * Doc Team report (mrevell)19:04
RinchenNext meeting19:04
Rinchenwell, there was not much consensus on the rotation option email19:04
RinchenKiko, SteveA -  you're still against two meetings, correct?19:04
kikoI'm not against anything per se, but we lose a lot by having separate meetings.19:05
allenap_me, sorry I'm late19:05
kikoso I find the drawbacks kinda high19:05
SteveAI'm against split meetings19:05
Rinchenthat leaves us to pick one, or do thumper's 6 hour incrementing meeting19:05
SteveAI'm fine with having two meetings, provided they're approximately one week apart19:05
Rinchenlol19:05
kikoincrementing meetings are a bit scary in that everybody gets confused19:05
thumperSteveA: :)19:05
thumperkiko: people are smart enough19:05
thumperand we have a team calendar19:05
kikothumper, speaking from experience on the Ubuntu side, not really.19:06
gmbAlso, we set the next meeting date in the meeting...19:06
kikoso I'm kinda -1 on rotating meetings19:06
kikobut +1 on two meeting times alternating19:06
kikosounds like nobody liked the times there, though19:06
kikocould we propose other times?19:06
kikoI will be at both meetings19:06
thumpermidnight UTC is good for me :)19:06
kikoworksforme19:07
jtvgah19:07
RinchenI can go back and look for other slots19:07
Rinchenthe 11am UTC one is the one with the most chance more highest attendance19:07
kikoanyway, if no decision comes in through email this week, same time next week.19:07
kikowhat time is that for jtv?19:07
Rinchenok, I'll set up everything for 18:00 UTC for next week and we can change it19:07
kikomoving on19:07
mptkiko, 1am-ish19:08
jtvkiko: 07:00, or 6 hours before my working day19:08
SteveAjtv is in thailand.  surely he can get drugs to make the current meeting time palettable19:08
* mpt doesn't know what he's talking about19:08
jtvSteveA: to me, yes.  To you?  Funny maybe, but...19:08
intellectronicaRinchen: if so then i'm afraid i'll have to miss next week. sorry!19:08
Rinchenright then, moving on19:08
Rinchenoh, thanks intellectronica19:08
RinchenActions from last meeting19:08
Rinchennone19:08
RinchenOops report (Matsubara)19:08
matsubaraToday's oops report is about bugs 228305, OOPS-855EB78, 228307.19:09
ubottuLaunchpad bug 228305 in malone "OOPS accessing contextless bug url" [High,New] https://launchpad.net/bugs/22830519:09
matsubaraintellectronica, can you take 228305. I think it's related to the email interface. It needs further investigation.19:09
matsubaracprov, wasn't OOPS-855EB78 RCFIXED? Or is that a new one?19:09
kikojtv, 7pm?19:09
intellectronicamatsubara: sure, i'll investigate and fix as appropriate19:09
jtvkiko: midnight UTC is 7am for me.19:09
matsubaraI've noticed some timeouts in +storeblob and +hwdb/+submit pages. I've asked about it to flacoste last Friday. More eyes on the issue would be appreciated(#228307)19:09
kikojtv, it's 11am utc.19:09
cprovmatsubara: bigjools fixed it last friday, IIRC19:09
flacostemy hypothesis was some libarian connection dalys19:10
jtvkiko: 11am utc is just dandy for me.19:10
matsubaracprov: the oops is from sunday, on edge19:10
flacostebut that's just a wild guess19:10
kikomatsubara, so hmmm something else is amiss?19:10
cprovmatsubara: uhmm ... let me check it again then19:10
SteveAdo we record times taken talking to the HTTP librarian?19:11
SteveAif not, we should do19:11
SteveAand these should be available in OOPS reports19:11
SteveAas a general principle, any time we do something that talks to another process or another computer, we should record the time taken19:11
cprovmatsubara: no, that's a new oops, needs a bug.19:11
matsubaracprov: I'll file one after the meeting. thanks for checking19:12
kikoSteveA, for the upload? we do not.19:12
matsubaraSteveA: I don't think we do.19:12
cprovmatsubara: np, thank you19:12
matsubaraSteveA: I mean, I don't think we do the logging19:12
SteveAthat's too bad.  it would help us out in knowing what's going on.19:12
flacostelibrarian logging sucks also, but that's probably another issue19:13
Rinchenmatsubara, anything else?19:14
matsubaraRinchen: that's all from me.19:14
Rinchenthanks19:14
RinchenCritical Bugs (Rinchen)19:14
matsubarathanks all. I'll update the bug19:14
RinchenMemory issue.   flacoste, can we reopen the bug since it's still a problem, or create a new one? How is the current investigation going?19:14
SteveAmatsubara: would you file a bug on the librarian client that it should record times for HTTP calls19:15
flacosteRinchen: file a new one19:15
matsubaraSteveA: sure. doing it now19:15
flacosteand SteveA thinks it might be related to batch size19:15
SteveAthanks19:15
flacostei'm going to put a hard limit on it19:15
flacoste30019:15
stub(watch out for the exception handler that uploads the exception to the librarian - might end up in a loop)19:15
SteveARinchen: earlier, tom, jtv and I did some experiments on staging19:15
kikoI am okay with that. would be even happier with 500 but anyway.. :)19:15
flacostei'm going to file a bug about the batch size issue19:15
jtvflacoste: I just discussed the issue with sinzui as well, so19:15
Rinchenthanks flacoste19:15
jtvflacoste: can we get together and compare notes?19:16
Rinchenhttps://bugs.edge.launchpad.net/rosetta/+bug/22461719:16
Rinchenjtv - is this really critical or simply high?19:16
ubottuLaunchpad bug 224617 in rosetta "XPI import stumbles over malformed or email-less contributor entries." [Critical,In progress]19:16
jtvflacoste: (I have a bug for the Translations side)19:16
SteveAand we could see a large-ish memory increase for translation pages with large batch sizes19:16
jtvRinchen: that's critical to the Firefox people.19:16
jtvRinchen: the fix landed today, and I'm negotiating a CP19:17
Rinchenjtv, k, thanks.19:17
kikoRinchen, jtv: I'm going to chat with asac; meanwhile jtv will test on staging.19:17
Rinchenhttps://bugs.edge.launchpad.net/launchpad/+bug/22462319:17
RinchenDB load.  stub, mthaddon - this hasn't been worked on. Therefore I submit that it's not a critical bug but rather high. Do you agree?19:17
ubottuRinchen: Error: This bug is private19:17
Rinchen:-)19:17
kikoagreed.19:17
flacostejtv: sure19:17
flacostejtv: grab me after meeting19:17
jtvflacoste: not now though; it's deep night and I've been at it for 15 hours straight19:17
Rinchenherb, if you could pass that on19:18
herbwill do19:18
flacostejtv: ok, i'll probably fix it later today though19:18
RinchenI'll go ahead and lower this. If it happens again, herb if you could get Tom to update the details in the bug report please.  thanks19:18
stubI'll flag Bug 224623 as incomplete - there is no way to progress the bug report.19:18
jtvflacoste: okay, then we'll have to do it right after19:18
herbRinchen: got it.19:18
SteveARinchen: we should adapt some code that gustavo wrote for landscape19:18
ubottustub: Bug 224623 on http://launchpad.net/bugs/224623 is private19:18
Rinchenok, thanks stub19:18
SteveARinchen: to limit the connections that are accepted from the network into the webapp19:19
SteveARinchen: based on the size of the queue of connections waiting for app threads.19:19
RinchenSounds like a job for spec circumstances or foundations.19:19
SteveARinchen: that will keep launchpad responsive even in situations that look like the one described by tom in the bug report19:19
SteveAit's a foundations job, for post 2.019:19
SteveAwhich we can look at moving forward if it happens again19:20
Rinchenok, I'll add that to the laundry list. :-)19:20
RinchenThat's it from me.19:20
RinchenOperations report (mthaddon/herb)19:20
herbHighlights from the last week:19:20
herb - 05/02: Production re-roll.19:20
herb - 05/05: lpnet6 (running with the debug config) stopped responding and was restarted.19:20
herb - 05/06: lpnet4 died and was restarted.19:20
herb - 05/06: Cherry pick r6222 to lpnet*19:20
herb - 05/07: Restarted librarian after a 13 minute outage.19:20
herbNew edge server running hardy should be up and in the rotation starting tomorrow.19:20
herbNo update from us on the memory issue. The debug instance (lpnet6) doesn't seem to be growing in the same way that the non-debug instances are. The typical lpnet process seems to grow to 800-1000MB RSS (3x-5x it's initial RSS) within a few hours and stay there idefinitely.19:20
herbThat's it from Tom and me unless there are any questions.19:20
stubSo we should fix the memory issue by running in debug mode? :-/19:21
SteveAherb: did you get anyone to look at lp6 when it stopped responding?19:21
kikowhat's the overhead in running debug mode?19:21
SteveAkiko: lots19:21
SteveAkiko: we run single threaded19:21
SteveAand it doesn't fix the memory issues19:22
herbSteveA: Tom handled it.  so I don't know.  I know he copied the ref* files over in case there was something of value in there.19:22
jtv"Less damage per second"19:22
SteveAit makes them less likely to occur, because the server handles fewer requests19:22
flacostemaybe 800-1000Mb is what LP needs as working memory to process our requests that perform badly19:22
herb1/16th the number of requests.19:22
SteveAherb: if an app server becomes unresponsive and it's only one, please take it out of rotation in the load balancer19:22
SteveAand then find someone ifrastructural to look into it19:22
SteveAif two go down in the same way, then restart the second one, leaving the first one hung19:23
SteveAplease add this to the LOSA operational manual, or whatever :-)19:23
herbSteveA: ok.19:23
SteveAwe can usually afford to lose one app server19:23
Rinchenanything else for herb?19:24
herbSteveA: when you say someone infrastructural, who should we be looking for?19:24
SteveAand it's good to have the opportunity to diagnose it19:24
SteveAa member of the foundations team19:24
SteveAor SC19:24
SteveAor me19:24
herbSteveA: ok19:24
Rinchenherb, you can ping steve, kiko, or I on -code if it happens and we can help you find someone19:24
herbRinchen: sounds good.19:24
Rinchenthanks herb19:24
Rinchenoff we go....19:24
RinchenDBA report (stub)19:24
stubThe production DB server is being upgraded to hardy as soon as IS can schedule it. This involves downtime.19:25
stubHopefully we get the PQM box running Hardy at around the same time. This means we can switch to PG 8.3 for development.19:25
stubIf things go to plan, we can upgrade production to PG 8.3 later with minimal downtime. I need to test using Slony-I to perform the migration with real data on Carbon once it has been Hardified.19:25
stubDevs will need to switch to PG 8.3 when PQM switches. You are welcome to switch earlier if you want. The docs are already updated on the Wiki.19:25
stubHad a good discussion on the person/auth split with jamesh and refactored the model again. The diagram on the wiki page has been updated (and no longer matches the text).19:25
stubOOT.19:25
RinchenBetter watch out kiko, I'm going to hardify you19:26
thumperoot?19:26
Rinchenlol19:26
stubover and out19:26
SteveAout of time19:26
thumperah19:26
kikoRinchen, I'm on hardy!!19:26
SteveAour obstinate technologist19:26
Rinchenjtv, put down that GTA list right now and step away from the computer19:26
jtvRinchen: it's GTF19:26
SteveAour other thailander19:26
RinchenThanks stub19:26
jtvRinchen: and you're too late: it's already uploading19:26
RinchenSysadmin requests (Rinchen)19:27
RinchenIs anyone blocked on an RT or have any that are becoming urgent?19:27
mrevellour original Thailander?19:27
Rinchenand bigjools, I didn't look, are you done?19:27
SteveAmrevell: nice19:27
bigjoolsRinchen: yes!  \o/19:27
Rinchenyippee19:27
bigjoolsRinchen: having said that I need to talk to flacoste about the restricted librarian rollout, which will need another RT19:28
flacostei suck19:28
flacostesorry, completely forgot that one19:28
kikojesus guys19:28
RinchenPass it my way and I'll do the priority magic on it19:28
kikothis was due last CENTURY19:28
RinchenI'll get SteveA to do a rain dance too19:29
bigjoolsflacoste: are you free after the meeting sometime?19:29
Rinchenkiko, yeah, bigjools ppa fought the Hardy release and Hardy won19:29
RinchenNew packages required (salgado)19:30
flacostebigjools: tomorrow might be better, i have three persons already in line :-(19:30
salgadoanything to add to launchpad-dependencies this week?19:30
Rinchenbeer?19:30
SteveAso...19:30
kikonot that I can think of19:30
SteveAif launchpad-dependencies were managed in /sourcecode19:30
kikothanks salgado!19:30
SteveAthen we could have a check on 'make run'19:30
SteveAthat warned if launchpad-dependencies is not of the appropriate version for this LP tree19:30
flacosteSteveA: we can have a check even if it's not managed in sourcecode19:31
SteveAsorry to bring up this old chestnut, but I miss xmas19:31
SteveAflacoste: it means we need to record the version number somewhere, and keep it up to date19:31
SteveAflacoste: rather than just work off what's in /sourcecode19:31
RinchenSteveA, how about you file a bug for that and tell flacoste about it?19:32
kikoSteveA, well, depends if sourcecode is linked out of somewhere or not19:32
kikoi.e. if it is shared19:32
flacosteWon't fix19:32
Rinchenwe won't lose your chestnut then19:32
Rinchenflacoste, well, you can negotiate that with your boss ;-)19:33
Rinchengoing once19:34
flacostewe need more value out of it to offset the additional process cost19:34
Rinchenok, sounds like an offline discussion is in order.  ACTION: Steve and Francis to chat about dependencies and make check19:34
SteveAthanks, but it's fine19:35
RinchenA top user-affecting issue (mrevell)19:35
mrevellhowdy19:35
SteveAI will go with flacoste's judgement on this issue.  I'm sure we'll revisit it later, once we move more of launchpad code into launchpad.19:35
mrevellA common theme over the past week - although we haven't been inundated with requests - has been requests to either report spam or edit existing bug comments.19:35
mrevellAs we've discussed this in meetings before, I'd be interested in hearing from other people who have either dealt with or seen an interesting user-affecting issue.19:35
flacosteSteveA: actually, once servers are upgraded to hardy, the cost will lessen, so we should revisit at that time19:36
intellectronicaone thing i encountered is users wanting to edit or delete their own comments19:36
SteveAwe need to have "add a 'report this comment' link on comments" and the same for other user-submitted content19:36
intellectronicafor example, because they disclosed information they would rather not, accidentally19:36
SteveAand that should be on the post-2.0 list19:36
SteveAservices like facebook do this well19:37
mrevellintellectronica: Yes, I've seen that come up too. There's a problem of editing history there, though, isn't there?19:37
stubWhy is that a problem?19:37
intellectronica"other user-submitted content"? almost all of it is...19:37
intellectronicamrevell: what do you mean by "editing history"?19:37
kikointellectronica, changing what I said in the past.19:37
mptmrevell, not if it's within five minutes or before anyone else comments.19:37
mrevellintellectronica: Someone could edit their comment to cast themselves in a better light, perhaps.19:37
mrevellmpt: Hmm, fair point.19:38
mpt(5 minutes being when the mail notification goes out)19:38
RinchenThere is a general topic of spam handling that I've already added on behalf of the OSAs19:38
stubWhy is that a problem? We are writing a bug tracker and things - not a banking system.19:38
Rinchenon the post-2.0 list19:38
mrevellDo we have a bug report? I didn't see one.19:38
mptWe have bug reports on both those issues19:38
mptspam is bug 4541919:38
ubottuLaunchpad bug 45419 in launchpad "Launchpad needs a way of easily flagging spam" [High,Confirmed] https://launchpad.net/bugs/4541919:38
SteveAI think we should always keep the comments19:38
mptI don't remember the editing one offhand19:39
mrevellmpt: Yeah, I saw the spam but not the edit history. I'll look again, thanks.19:39
SteveAand hide them, and have a link saying "N comments removed"19:39
SteveAopenness is important19:39
mptbug 8089519:39
ubottuLaunchpad bug 80895 in malone "Give people five minutes to edit/delete their comment" [Undecided,Confirmed] https://launchpad.net/bugs/8089519:39
mrevellthanks mpt19:39
mptnp19:39
mrevellAs we have a plan, I'm done.19:40
kikoyep19:40
RinchenDoc Team report (mrevell)19:40
kikoit's post 2.0 unfo19:40
intellectronicampt: i'm not sure 5 minutes is enough. i realise that after that we've sent an email anyway, but i suspect that in some cases users will still want to remove a comment19:40
BjornTSteveA: a db patch was landed last cycle, which will allow us to hide comment. we just need to implement it.19:40
mrevellI'd like to give the doc team members a list of bite-sized items that would be outside the scope of our normal docs work. If you have ideas for such work, that involves your part of LP, please let me know.19:40
mrevellOn a docs-related note: Rinchen, Statik and I are recording a new episode of the Launchpad podcast next week. All ideas, such as suggestions of CC-licensed music for the theme and a name are welcome :)19:40
mptintellectronica, then they can put themselves into the "Inappropriate" queue just like anyone marking spam19:40
mptand be moderated accordingly19:40
SteveAmrevell: a new episode?19:41
SteveAmrevell: we have episodes? what's the RSS feed?  I'll add it to my reader19:41
intellectronicampt: maybe you're right. anyway, there's a bug, so let's continue there19:41
mptsure19:41
jtv4-minute warning...19:41
gmbSteveA: I think there was one too many adjectives in mrevell's sentence.19:41
mrevellSteveA: Well, let's say episode 1.0, to the version 0 that I posted last year. The feed is...19:41
mrevellhttp://news.launchpad.net/category/podcast/feed19:42
SteveAthanks19:42
mrevellThanks Rinchen19:42
RinchenThanks.19:42
RinchenThank you all for attending this week's Launchpad Developer Meeting.19:42
RinchenThe end!19:42
thumperthanks Rinchen19:42
SteveAthanks Rinchen !19:43
jtvthanks Rinchen!19:43
* Rinchen laments the loss of Mootbot.19:43
SteveAmootbot, I mourns it19:43
RinchenI'll just have to get the scissors out and cut up this log.19:43
kikowoo19:43
Rinchenmatsubara, pass me the glue eh?19:43
Rinchengmb, can I use your scissors? Yours seem awfully sharp.19:44
intellectronicadid anybody say "glue"?19:44
* intellectronica sniffles 19:44
gmbRinchen: Alas, I only use scalpels.19:44
Rinchengmb, ok, I'll check with sinzui then.  Last I knew his where pink19:45
Rinchenor maybe they were pinking shears....19:45
=== salgado is now known as salgado-brb
=== salgado-brb is now known as salgado
=== salgado is now known as salgado-brb
=== EdwinGrub is now known as EdwinGrubbs

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!