=== Ursinha is now known as Ursinha-afk === kiko is now known as kiko-afk === kiko-afk is now known as kiko-zzz === salgado-afk is now known as salgado === Ursinha-afk is now known as Ursinha === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell === kiko__ is now known as kiko === salgado_ is now known as salgado === salgado is now known as salgado-lunch === salgado-lunch is now known as salgado === kiko is now known as kiko-phony [18:57] * Rinchen prays to the buggy intel driver not to crash during the meeting. [18:59] * Rinchen continues to pray to the buggy intel driver not to crash during the meeting. [18:59] Rinchen, use irssi :P [19:00] if you're talking about the video one.. [19:00] that won't help with my external display crashes :-( [19:00] #startmeeting [19:00] Welcome to this week's Launchpad development meeting. For the next 45 minutes or so, we'll be coordinating Launchpad development. [19:00] Meeting started at 13:03. The chair is Rinchen. [19:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [19:00] [TOPIC] Roll Call [19:00] New Topic: Roll Call [19:00] moo [19:00] moo [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] me [19:00] barry, sinzui are on leave === gmb_ is now known as gmb [19:00] me [19:00] apologies from gavin, barry, curtis.... [19:01] me [19:01] mpt? [19:01] me [19:01] EdwinGrubbs, mars, leonardr: ping [19:01] me [19:01] flacoste, moo? [19:01] mars: moo doesn't count ;-) [19:01] flacoste, sorry, couldn't help myself :) [19:01] me [19:01] me [19:01] next week is "cluck" [19:01] ผม [19:01] Foundations is here [19:01] thanks [19:02] Rinchen: i'll miss next two meetings [19:02] me [19:02] me [19:02] thumper won't make it today [19:02] me [19:02] yo [19:02] me [19:02] Bugs is here [19:02] me [19:02] ok, releases is here [19:02] code is sprinting so they won't be here [19:02] jtv, danilo coming? [19:03] and no stub. :-( I even emailed him [19:03] Rinchen: not sure, forgot about the time so only just woke up again [19:03] k [19:03] [TOPIC] Agenda [19:03] New Topic: Agenda [19:03] * Next meeting [19:03] * Actions from last meeting [19:03] * Oops report (Matsubara) [19:03] * Critical Bugs (Rinchen) [19:03] * Bug tags [19:03] * Operations report (mthaddon/herb/spm) [19:03] * DBA report (stub) [19:03] * Sysadmin requests (Rinchen) [19:03] * New packages required (salgado) [19:03] * A top user-affecting issue (mrevell) [19:03] * Doc Team report (mrevell) [19:03] plus [19:03] Introduce Reinhard Tartler as the new MOTU liaison - mrevell [19:03] # [19:03] Store OOPSes for Unauthorized exceptions - salgado [19:03] # [19:04] Bug triage & the triage status - kiko [19:04] librarian upload concerns - kiko [19:04] cool [19:04] [19:04] action packed mtg [19:04] heh [19:04] Rinchen: Damn, sorry, I thought the MOTU-liaison thing was off the agenda. that's old. [19:04] no worries, we can skip that one [19:04] [TOPIC] Next meeting [19:04] New Topic: Next meeting [19:04] Aug 7th. Same time, same channel. [19:04] i'll miss next two meetings [19:05] I'll be sprinting :) [19:05] I'll probably be here, but I won't be taking part [19:05] Translations'll be sprinting too [19:05] me too [19:05] goodbye mpt [19:05] mpt: thanks for all the fish [19:05] i'll miss the next meeting [19:05] ok, releases team will try to be there :-) [19:06] ok. Kiko you don't want to skip next week do you? :-) [19:06] nooooo [19:06] I'll be here [19:06] [AGREED] Aug 7th. Same time, same channel. [19:06] AGREED received: Aug 7th. Same time, same channel. [19:06] k [19:07] [TOPIC] Actions from last meeting [19:07] New Topic: Actions from last meeting [19:07] nothing was entered from last week [19:07] stub! [19:07] so I'll assume it was just salgado to file a bug about the store OOPSes :-) [19:07] me [19:07] and I haven't done that [19:07] will do today [19:07] thanks salgado [19:08] [TOPIC] Oops report (Matsubara) [19:08] New Topic: Oops report (Matsubara) [19:08] and I will add Ursinha there too [19:08] Rinchen, thanks [19:08] Ursinha is handling that :-) [19:08] ok [19:08] i have some timeout bugs here [19:08] bug 252695 [19:08] Launchpad bug 252695 in launchpad "+peoplelist taking too long to render and timing out" [Undecided,New] https://launchpad.net/bugs/252695 [19:09] can anybody please take a look at it? guess it's foundations one [19:09] I wonder, is there any value in retaining +peoplelist? [19:09] salgado can you take care of it sometimes while i'm away? [19:09] Rinchen: crawlers [19:10] flacoste, sure thing [19:10] googlebot ftw. ok [19:10] ok, next one, specworkload bug, bug 124205 [19:10] Rinchen: didn't you read, cuil is the next big thing ;-) [19:10] Launchpad bug 124205 in blueprint "timeout in specworkload page" [Medium,In progress] https://launchpad.net/bugs/124205 [19:10] bac is handling that one [19:10] flacoste, haha [19:11] flacoste, I don't think that's really a reason [19:11] Ursinha: in review [19:11] flacoste: aren't all the interesting people linked to anyway? [19:11] flacoste, because if someone's worth indexing, they'll be linked to from plenty of other pages in LP [19:11] BjornT: good point [19:11] bac, ok, thanks [19:11] salgado: acceptable fix is to remove the page ;-) [19:11] ln -s +peoplelist ~rinchen and be done with it ;-) [19:11] consider it done, then [19:12] Ursinha, any other bugs for today? [19:12] yes [19:12] bug 252709, the roadmap page [19:12] Launchpad bug 252709 in blueprint "+roadmap pages taking too much time to render and timing out" [Undecided,New] https://launchpad.net/bugs/252709 [19:12] another page that we could get rid of [19:12] it's timing out a lot, and it was suggested in bug 174480 to get rid of it [19:12] Launchpad bug 174480 in blueprint "Person's +roadmap page contains blueprints they're not assigned to" [Undecided,New] https://launchpad.net/bugs/174480 [19:12] +roadmap, a wonderful idea that we never expanded on [19:12] again? [19:13] at this juncture I'm +1 to remove +roadmap [19:13] it is not providing the value we had intended originally [19:13] +1, since it's quite hard to optimise, and not a very useful or popular page [19:13] err, I didn't say "me", sorry [19:14] BjornT, can intellectronica do the honours there? [19:14] Rinchen: sure [19:15] thanks BjornT and intellectronica [19:15] ok [19:15] the next [19:15] ok, one for bugs now, i suppose: bug 252674 [19:15] Ursinha: Bug 252674 on http://launchpad.net/bugs/252674 is private [19:15] it's a bug on malone, too many timeouts because of large queries [19:15] Rinchen, intellectronica: you sure the mysql guys aren't using +roadmap? [19:16] kiko-phony: i can find out before killing it, just to make sure [19:16] intellectronica, I can help you with the email on that. Ping me post meeting! [19:16] Ursinha: has that been happening lately? it looked like an intermittent problem [19:17] Ursinha: i.e, usually those queries are fast to run [19:17] BjornT, the timeouts are strange, the oopses show low sql AND non-sql times [19:18] it's been happening quite frequently [19:18] intellectronica, mtaylor is the guy to ask, on #launchpad [19:18] Ursinha: right, and it should be because the query takes a really long time. do you have an example of such an oops for today/yesterday? [19:18] several in the bug report [19:19] oh, not on hands now, sorry [19:19] Ursinha: ok. i'm asking since all the oops in the bug report are around the same time [19:19] BjornT, found one: https://devpad.canonical.com/~matsubara/oops.cgi/2008-07-31/F2030 [19:21] Ursinha: ok, thanks. i'm going to have to check with the losas to get more information [19:21] ok, thanks BjornT [19:21] the last one [19:21] bug 251569 [19:21] Launchpad bug 251569 in launchpad "UnicodeEncodeError when trying to search in launchpad/people/+index" [Undecided,Confirmed] https://launchpad.net/bugs/251569 [19:21] guess i've mentioned that one with salgado [19:22] salgado, sorry, didn't have the time to look at it closely [19:22] can you do that for me? [19:22] Ursinha, it might be better to reping the foundations guys on monday or tuesday once they've done the API announcement [19:22] Ursinha, yes, I can have a look at it next week [19:22] Ursinha, right now they are scrambling to get that done [19:22] [AGREED] salgado to handle 252695, bac to handle 124205, intellectronica to handle 174480, bjorn to talk to losas about 252674, salgado to handle 251569 [19:22] kiko-phony, ok [19:22] AGREED received: salgado to handle 252695, bac to handle 124205, intellectronica to handle 174480, bjorn to talk to losas about 252674, salgado to handle 251569 [19:22] Ursinha, and they are likely to forget everything they're saying they will do by then :) [19:22] kiko-phony, ok, thanks :) [19:23] anything else Ursinha ? [19:23] Rinchen, guess that's all folks [19:23] thanks all [19:23] thanks Ursinha [19:23] [TOPIC] Critical Bugs (Rinchen) [19:23] https://bugs.edge.launchpad.net/launchpad/+bug/253217 [19:23] needs an owner [19:23] New Topic: Critical Bugs (Rinchen) [19:23] Rinchen: Error: This bug is private [19:23] flacoste, please update the status on [19:23] https://bugs.edge.launchpad.net/launchpad/+bug/252673 [19:23] Rinchen: Error: This bug is private [19:23] flacoste, and also milestone if appropriate please [19:24] it looks like leonardr has been working on 253217 [19:24] Rinchen: leonardr has been working on a fix for the first one [19:24] duh [19:24] Rinchen: i updated both of them [19:24] Thanks [19:24] i'm responding to salgado's review [19:24] it's almost ready to go in [19:25] that's it from me. Thanks for resolving the other criticals on the list earlier. [19:25] [TOPIC] Bug tags [19:25] New Topic: Bug tags [19:25] we have one from matsubara [19:25] https://help.launchpad.net/TaggingLaunchpadBugs [19:25] pqm-analyzer [19:25] This is for the atomic pqm analyzer that kiko wrote that we're trying to get setup in the DC [19:25] flacoste, you did? :) [19:25] I'm not fussed about that tag, but I want something to group bugs related to the kiko's APA tool [19:26] matsubara, does it have bugs in it?? [19:26] kiko-phony: refresh [19:26] flacoste, woo! thanks :) [19:26] why not have a product for it? [19:26] +1 [19:26] less bugs for me to triage :-) [19:26] intellectronica's on the ball [19:26] There are two questions that I've been debating. A new product and having PQM manage the code [19:26] new product, not sure about PQM management [19:27] until it's deployed, not sure it's worth it [19:27] it's right now two files.. :) no tests! [19:27] at this point I was trying to be flexible :-) [19:27] not like there were any tests to run yet anyway [19:27] PQM != flexible [19:27] ken [19:27] I meant, that's why I didn't do either :-) [19:28] ok. so no tag and create a new product. fine by me. [19:28] Any objections to creating a new product and not a tag? [19:28] DO IT === kiko-phony is now known as kiko [19:28] * matsubara wonders if I should do the same with oops-tools [19:28] matsubara, probably [19:29] * Ursinha agrees on doing the same to oops-tools [19:29] [AGREED] create new product of the APA instead of using a tag. [19:29] AGREED received: create new product of the APA instead of using a tag. [19:29] matsubara, can you please update the tag page? [19:29] sure Rinchen. thanks everyone [19:29] oops-tools should be a tag since it is part of Launchapd, but you knew I would say that :) [19:30] [TOPIC] Operations report (mthaddon/herb/spm) [19:30] New Topic: Operations report (mthaddon/herb/spm) [19:30] Early morning Tuesday (2008-07-29) we rolled out 2.0 at r6771. [19:30] Late in the day Tuesday (2008-07-29) we re-rolled 2.0 at r6773 to catch some important bug fixes. [19:30] All week we've been fighting the dreaded "app server dying" bug. Bug 247227. [19:30] herb: Bug 247227 on http://launchpad.net/bugs/247227 is private [19:30] Other than that it's been a quiet week. That's it from mthaddon, spm and me forthis week. [19:30] herb, has anyone been looking at that with you? [19:31] kiko: not really, no. [19:31] herb, okay. how much is this being a problem for production? [19:31] * kiko #@@! [19:32] kiko: if you take a look at the bug it shows how many times we've had to deal with a downed app server. happens almost daily. [19:32] and we have seen it multiple times in a day. [19:32] herb, can we do something to figure out what it's doing when it dies I wonder [19:32] flacoste, stub: any ideas of possible instrumenting we could do? [19:33] For it to die like that, something is messing up in C code. And I'm utterly useless at that stuff. [19:33] python dying is kind of out of my compentcy [19:33] kiko: we need barry for that [19:33] well, he might have an idea [19:33] herb, we still have yet to see this on edge? [19:33] or lifeless [19:34] or second jamesh back for a bit ;) [19:34] Rinchen: correct. so far just staging and production. [19:34] herb, flacoste, stub: well, hang on. are we saying it dies in a horrible SEGV or similar? [19:34] kiko: yes [19:34] herb, and not an OOM? [19:34] kiko: not an oom [19:34] herb, can't we run one instance inside gdb? [19:35] I suspect we could. [19:35] it would be a bit slower but since it's so reliable it'd eventually crash.. or at least we'd know that we could move everything to gdb and fix the problem [19:35] we could address that by using staging [19:35] herb, do you want assistance doing this? I could add a make target I guess [19:35] that would then not impact production users [19:35] good point joey [19:35] well, it's not reliable in the sense that we know which app server will die in any particular day. :) [19:35] herb, how often does it die on staging? [19:35] I find it very odd that we don't see this on edge. What is different about edge over staging and lpnet? [19:35] The load for a start [19:36] we've seen it die a couple of times on staging in the last couple of weeks. [19:36] staging isn't very loaded, is it? [19:36] linkchecker [19:36] which hasn't run properly for the past weeks [19:36] i.e. look at the staging oops reports [19:36] But it has run? [19:36] matsubara, Ursinha - please look into the linkchecker issue [19:36] it's running, yes. [19:36] not completely [19:36] the OOPS reports we get are very sparse [19:37] it's not hitting any bug pages, for instance [19:37] Well... it won't if it crashes the staging servers ;) [19:37] stub, it doesn't crash them every day though :-( [19:37] anyway [19:37] there's only one instance running. I disable the translations.staging.launchpad.net one as that one was causing OOM on devpad. [19:37] we can fire up staging under gdb and pound it. we're good at loading up staging :-) [19:37] disabled, I mean [19:37] herb, do you want assistance doing this gdb thing? I could add a make target I guess [19:37] matsubara, it's not hitting any bugs pages, regardless. [19:38] matsubara, or if it is, then we need to lower the soft timeout threshold [19:38] because we're not getting any oopses [19:38] kiko: I can do it. would prefer some direction to make sure I'm doing it right. [19:38] and very few 404s [19:38] kiko: I'll look into it [19:38] herb, okay, I'll test on my local box after the meeting and then ping you. [19:38] kiko: cool. thanks. [19:38] Thanks gents + lady [19:38] anything else for herb ? [19:38] matsubara, the 404s are the hint to me. we normally get hundreds. we now get tens [19:39] [TOPIC] DBA report (stub) [19:39] New Topic: DBA report (stub) [19:39] After the earlier cleanups, the Session database had way too much empty space so I trashed the session tables. This logged everyone out but meant no downtime needed to cleanthings up and we are back within our free space map settings. [19:39] I've got more disk space on Chokecherry. Yay. I will be abusing demo.launchpad.net in earnest - any objections speak now. [19:39] There is a branch due to land as soon as I fix a few failing tests and get Edwins approval that changes things so we have four Storm Stores instead of the current single store. The load balancing algorithms between the launchpad master and slave stores will be tested on demo.launchpad.net. The other two stores - authdb master and slave - will not be active when the branch lands. [19:39] The envisaged roadmap is to test load balancing between a single master and slave while finishing of the scripts needed to maintain a replicated production database environment. The initial rollout may have a single replica set or two replica sets (lpmain and authdb), although either model the appserver will be accessing everything through a single master/slave pair of Stores (ie. authdb master and authdb slave will remain unused). [19:39] And that is all my incoherent ramblings for now. [19:40] stub, what are you abusing demo for? [19:41] ah, I see [19:41] stub, do you have a general timeframe when we can move past the single pair of stores? [19:41] Seeing how Launchpad works with a real data set and talking to both a master and slave replica [19:41] gotcha, yeah, your final paragraph made it very clear [19:41] stub, this is awesome news, I'm thrilled to see this progress [19:41] yeah me too. I'm excited for this all to land and be turned up [19:42] Rinchen: Not long I think. Next cycle? [19:42] Great! I'll keep the champagne on ice then. :-) [19:42] anything else for stub? [19:42] thanks stub [19:42] yes [19:42] The trick with the authdb master and slaves becoming active is updating the code to actually use them - that is all the screens on login.launchpad.net for a start I think [19:43] stub, what do you make of the librarian timeouts we're getting? [19:43] We are getting Librarian timeouts? [19:43] stub, when people upload larger files, it pretty consistently dies and gives us a 503 [19:43] it's only on uploads -- downloads work reliably [19:43] (well, I hope -- we don't have OOPSes there) [19:43] stub: something like this https://pastebin.canonical.com/7785/ [19:43] I'd say there is an exception in the Librarian log file that needs to be looked at. [19:44] herb, but there isn't, is there? [19:44] nope [19:44] the librarian log file is less than good. [19:44] Bad gateway is from Apache [19:44] and doesn't seem to be logging anything related to uploads. [19:45] So Apache is not passing the huge request through to the Librarian. [19:45] stub, only sometimes, it works! [19:45] hmph [19:45] there is an apport bug reported with a 503 as well and I suspect the cause of that is related to what we're seeing. Both are large uploads [19:45] this affects us with apport and file uploads [19:45] for file uploads I'd propose using sftp:// [19:45] but for apport.. [19:46] stub, any clue where we could start looking if it is indeed an apache issue? [19:46] 503 or 502? The pastbin is a 502. There might be two bugs if you are getting two different codes. [19:47] good point [19:47] Check the Apache logs to see if there is anything useful in there for that request [19:47] 502 sorry [19:47] thanks [19:47] https://bugs.edge.launchpad.net/apport/+bug/114962 [19:47] Launchpad bug 114962 in apport "502 bad gateway while automated crash reporting happening" [Undecided,Confirmed] [19:48] Otherwise, not sure. We might have to instrument the Librarian to log all the responses it sends out or stick in some sort of a sniffer in the middle. [19:48] stub, okay, I'll do some googling. [19:48] Rinchen, what's the apport bug id, have it handy? [19:48] My guess would be the Librarian is taking too long to store the file and give a response, and Apache is giving up waiting. [19:48] kiko, in fact I do. :-) read up a few lines [19:49] There will be some Apache tuning knobs for those timeouts we can poke [19:49] thanks Rinchen [19:49] anything else for stub? [19:50] thanks stub [19:50] [TOPIC] Sysadmin requests (Rinchen) [19:50] Is anyone blocked on an RT or have any that are becoming urgent? I know about 31340, 31296, and 31389. [19:50] New Topic: Sysadmin requests (Rinchen) [19:50] ok then [19:50] [TOPIC] New packages required (salgado) [19:50] New Topic: New packages required (salgado) [19:51] anything for me this week? [19:51] please say NO [19:51] or don't say anything. :) [19:51] Rinchen, I'm done. :) [19:51] :-) [19:51] salgado, well [19:51] salgado, wanted to check with you [19:51] I saw that we placed a request for xsltproc this week [19:51] yesterday, to be precise [19:51] salgado, was this brought up last meeting? [19:52] no, it wasn't [19:52] should it have been? [19:52] it should, but we didn't know about it at that time, I think [19:52] flacoste, when did we decide to use xsltproc? [19:52] I just want to understand if this is something we just discovered, or some piece of the process that's busted [19:52] I first heard about it being a dependency yesterday [19:53] we discussed it some time ago, and then toyed with it last week [19:53] because IS are understandably unhappy about having to do last-minute package installs on all our servers [19:53] but it was only this week that it was clear it becomes a build-time dependancy [19:53] i could work around it for a while [19:53] commit the file to bzr and only generate it on demand [19:53] flacoste, okay. in the future, if you "play around with it" then request it -- the lead time is important [19:53] until that is deployed [19:54] flacoste, that might have to be done -- depends on what Rinchen tells us [19:54] I'll know more tomorrow morning. [19:54] flacoste, it's better to request and then not need it than to request too late, ok? [19:54] kiko: let's do it [19:54] no need for IS to be more pissed-off at us [19:54] kiko: got it [19:54] thanks [19:54] guys please [19:54] lead time on package installations [19:55] no fucking around or Rinchen and I get in big trouble with the man [19:55] over and out [19:55] Rinchen: i'll work around it [19:55]