[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <sinzui> me
[16:00] <matsubara> so, who's here today?
[16:00] <cprov> me
[16:01] <rockstar> me
[16:01] <Rinchen> me
[16:01] <bigjools> me
[16:01] <matsubara> intellectronica, herb, danilos, flacoste: ?
[16:01] <danilos> me
[16:01] <intellectronica> me
[16:01] <flacoste> me
[16:01] <herb> me
[16:02] <matsubara> Apologies from Ursula, she had to run some errands
[16:02] <flacoste> him
[16:02] <matsubara> hi stub
[16:02] <stub> hi
[16:02] <matsubara> ok, everyone is here.
[16:02] <matsubara> [TOPIC] Agenda
[16:02] <MootBot> New Topic:  Agenda
[16:02] <matsubara>  * Next meeting
[16:02] <matsubara>  * Actions from last meeting
[16:02] <matsubara>  * Oops report & Critical Bugs
[16:02] <matsubara>  * Operations report (mthaddon/herb/spm)
[16:02] <matsubara>  * DBA report (DBA contact)
[16:03] <matsubara> [TOPIC] Next meeting
[16:03] <MootBot> New Topic:  Next meeting
[16:03] <matsubara> same time next week?
[16:03] <sinzui> yes
[16:03] <matsubara> so be it.
[16:03] <matsubara> [topic] * Actions from last meeting
[16:03] <MootBot> New Topic:  * Actions from last meeting
[16:04] <matsubara> flacoste: * foundations to look up bug 277129
[16:04] <flacoste> we have a fix for it
[16:04] <flacoste> that landed
[16:04] <matsubara> well, seems to be done :-)
[16:04] <flacoste> yesterday
[16:04] <flacoste> well
[16:04] <flacoste> we don't know
[16:04] <flacoste> if the OOPS disappears, it worked
[16:04] <matsubara> thanks flacoste
[16:04] <flacoste> otherwise its back to the drawing board
[16:04] <matsubara> I'll ask Ursinha to keep an eye on those OOPSes
[16:04] <matsubara> and comment on the bug.
[16:04] <matsubara> thanks for handling it
[16:04] <flacoste> (only on edge for the moment)
[16:05] <matsubara> all right
[16:05] <matsubara> [topic] * Oops report & Critical Bugs
[16:05] <MootBot> New Topic:  * Oops report & Critical Bugs
[16:05] <matsubara> Today's oops report is about bug 279561
[16:05] <intellectronica> this one is really weird, and i'm still working on diagnosing it
[16:05] <matsubara> intellectronica: hi tom, can you set importance on that one? Ursinha told me that one is happening quite frequently
[16:05] <intellectronica> the good news is that it's quite rare. only a few messages are affected by it
[16:06] <intellectronica> if it happens frequently it's only because both myself and thekorn have been hitting it a lot for testing
[16:06] <intellectronica> it only happens when accessing comments via the api, and only very few comments
[16:07] <flacoste> intellectronica: might be related to the fact that there are several "kind" of messages
[16:07] <intellectronica> shall we say importance=high? i don't think it deserves critical
[16:07] <sinzui> intellectronica: is the comment from a user with a hidden email address?
[16:07] <intellectronica> flacoste: tell me more
[16:07] <matsubara> right. I'll ask Ursinha to keep an eye on users other than you and the korn triggering it
[16:07] <flacoste> intellectronica: and that the canonical url is only registered for a couple of those
[16:07] <flacoste> intellectronica: well, Message, BugComment, BugCommentWithDecorations (that last one is not named like that)
[16:08] <matsubara> intellectronica: high seems to be the right importance
[16:08] <flacoste> intellectronica: i think canonical url is only registered for BugComment, if you return a Message in some case, it will probably fail
[16:08] <matsubara> interesting
[16:08] <intellectronica> this is always bug comments, i can't think why we'd return anything that's not a bug comment there
[16:09] <intellectronica> sinzui: don't know. i'll check, but i find it hard to imagine that this would be the problem, since the user's email is never involved in getting the canonical url for a comment
[16:10] <flacoste> intellectronica: i'm talking about the class of the object
[16:10] <sinzui> intellectronica: understood
[16:10] <flacoste> intellectronica: and this isn't consistent in the bug api
[16:10] <flacoste> intellectronica: something you get Messages, something BugMessage, sometime BugComment
[16:11] <intellectronica> flacoste: but doesn't the canonical url adapter rely on the interface, rather than the class?
[16:11] <intellectronica> they all implement IMessage, for which there is an adapter
[16:11] <flacoste> intellectronica: yes, but they provide different interfaces, especially IMessage
[16:11] <flacoste> there is a canonical url for IMessage?
[16:11] <flacoste> i wouldn't think so
[16:11] <intellectronica> yes, there is
[16:11] <flacoste> ok, i shut up then :-)
[16:11] <flacoste> i have no clue!
[16:12] <intellectronica> matsubara: anyway, i'm setting the importance to high and continue investigating
[16:12] <matsubara> [ACTION] intellectronica to continue diagnosing bug 279561
[16:12] <MootBot> ACTION received:  intellectronica to continue diagnosing bug 279561
[16:12] <matsubara> thanks intellectronica
[16:12] <matsubara> moving on
[16:12] <matsubara> we have a single critical bug
[16:12] <matsubara> which is fix committed
[16:13] <matsubara> so, it's all fine
[16:13] <matsubara> thanks all
[16:13] <matsubara> [topic] * Operations report (mthaddon/herb/spm)
[16:13] <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
[16:13] <herb> 2008-10-07 - Migrated codehosting to a new, much more powerful server. There was a few hours of downtime for code as we did the migration.
[16:13] <herb> Had a couple of instances where codebrowse became unresponsive and needed to be restarted.
[16:13] <herb> Had a few instances of app servers core dumping. Seems niemeyer and kiko made some progress on isolating the problem. Where do we stand on fixing it?
[16:14] <herb> BTW, is there a bug for the app servers core dumping?
[16:14] <flacoste> i don't know the answer to both of these questions :-/
[16:14] <matsubara> herb: yes, I think
[16:14]  * herb hopes so.
[16:15] <herb> so no update on a fix?
[16:15] <matsubara> I think there's one filed by mthaddon a long time ago. I can't find it now.
[16:15] <herb> if not, that's it from the LOSAs.
[16:15] <mthaddon> I'll see if I can find it
[16:15] <flacoste> actually
[16:16] <matsubara> haven't heard from kiko about a fix herb
[16:16] <Rinchen> the one Tom filed was for stale pidfiles
[16:16] <Rinchen> not for core dumps
[16:16] <flacoste> the last i heard was that barry was going to look into writing a dedicated parser for apport email
[16:16] <flacoste> and his quick strategy didn't work
[16:16] <flacoste> sinzui: anything else from barry?
[16:16] <mthaddon> Rinchen, true, yeah
[16:16] <sinzui> Nothing else
[16:16] <matsubara> bug 273618 perhaps?
[16:16] <flacoste> sinzui: is he still working on a fix?
[16:16] <sinzui> barry: solution failed outright
[16:16] <Rinchen> I'm beginning to to grow increasingly uncomfortable with the core dumps and would like to see appropriate debugging attention applied to those please.
[16:16] <flacoste> i know he's on leave today
[16:17] <danilos> fyi, forster is, last I heard, pretty CPU starved, especially when language packs are being built, so if that might be related, take it into consideration
[16:17] <flacoste> Rinchen: we could cap the size of accepted apport bug fix
[16:17] <sinzui> He is not working on a fix; he was hoping that kiko could find a work around.
[16:17] <flacoste> Rinchen: while we work on a fix
[16:17] <flacoste> danilos: no, it's really a memory problem when parsing big apport upload
[16:18] <danilos> flacoste: ok
[16:18] <flacoste> danilos: an innefenciy in the python email module
[16:18] <flacoste> wow, that was badly spelled!
[16:19] <Rinchen> flacoste, I guess either way the apport attachments will not get posted so perhaps we should.  We should let distro know about this as I'm sure that there will be no feedback other than "error" when the user looks at apport.
[16:19] <flacoste> which is probably what they are getting right now :-)
[16:19] <flacoste> intellectronica: could the bugs team patch this??
[16:20] <intellectronica> flacoste: technically, i think it's doable. i have to think and discuss more whether this is the appropriate solution
[16:21] <flacoste> intellectronica: it's a workaround to prevent us core-dumping
[16:21] <flacoste> the proper fix is to have a better parser that doesn't crash
[16:21] <danilos> fwiw, apport eats all of my memory when trying to get core dumps for big programs (like evolution or epiphany), and then just dies as well, so maybe that's why there are not too many reports about this problem
[16:22] <danilos> i.e. this needs to be closely coordinated with ubuntu guys
[16:22] <intellectronica> flacoste: i really don't know enough about the problem
[16:22] <flacoste> ok, i'll take it to Bjorn then
[16:22] <matsubara> [ACTION] intellectronica to discuss app server core dump bug with Bugs team and try to find a quick workaround
[16:22] <MootBot> ACTION received:  intellectronica to discuss app server core dump bug with Bugs team and try to find a quick workaround
[16:22] <intellectronica> i do know that the ubuntu guys rely on the apport attachments, so we should consider anything that might affect them
[16:23] <matsubara> ok, anything else on that matter?
[16:23] <sinzui> intellectronica: barry was of the opinion that we should use a out-of-proc tool to read the data from the filesystem
[16:24] <herb> matsubara: not from over here.
[16:24] <herb> thanks for the update everyone
[16:24] <matsubara> [ACTION] matsubara to find or file a new bug for the core dump issue and add info discussed today there, subscribe niemeyer, kiko and someone from ubuntu
[16:24] <MootBot> ACTION received:  matsubara to find or file a new bug for the core dump issue and add info discussed today there, subscribe niemeyer, kiko and someone from ubuntu
[16:24] <matsubara> thanks all
[16:25] <matsubara> moving on
[16:25] <matsubara> [topic] * DBA report (stub)
[16:25] <MootBot> New Topic:  * DBA report (stub)
[16:25] <stub> Nothing happening in DB land.
[16:25] <stub> :-)
[16:26] <matsubara> thanks stub
[16:26] <matsubara> so that's it for today, anyone have anything else before I close up?
[16:26] <rockstar> thanks matsubara
[16:27] <flacoste> matsubara: subscribe Bjorn to the bug also
[16:27] <Rinchen> thanks matsubara
[16:27] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
[16:27] <matsubara> #endmeeting
[16:27] <MootBot> Meeting finished at 10:27.
[16:27] <matsubara> flacoste: will do. thanks for the reminder