=== daniloff is now known as danilos
=== mrevell is now known as mrevell-lunch
=== EdwinGrubbs is now known as Edwin-afk
=== salgado is now known as salgado-lunch
[16:00] <matsubara> #startmeeting
[16:00] <MootBot> Meeting started at 10:00. The chair is matsubara.
[16:00] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[16:00] <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
[16:00] <matsubara> [TOPIC] Roll Call
[16:00] <MootBot> New Topic:  Roll Call
[16:00] <allenap> me
[16:00] <mrjazzcat> me
[16:00] <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[16:00] <matsubara> Chex, rockstar, bigjools, sinzui, danilos: hi
[16:00] <matsubara> Ursinha, hi
[16:00] <rockstar> me
[16:01] <bigjools> hello
[16:01] <Ursinha> me
[16:01] <Ursinha> hi
[16:01] <danilos> hi
[16:01] <matsubara> apologies from Gary, he's in a meeting and I'll be the foundations contact for today
[16:01] <Chex> me
[16:01] <matsubara> ok, everyone is here
[16:01] <matsubara> [TOPIC] Agenda
[16:01] <MootBot> New Topic:  Agenda
[16:01] <matsubara>  * Actions from last meeting
[16:01] <matsubara>  * Oops report & Critical Bugs & Broken scripts
[16:01] <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:01] <matsubara>  * DBA report (stub)
[16:01] <matsubara>  * Proposed items
[16:01] <matsubara> [TOPIC] * Actions from last meeting
[16:01] <MootBot> New Topic:  * Actions from last meeting
[16:02] <matsubara>     * rockstar to investigate failure on update_branches script and reply to the email.
[16:02] <matsubara> no new failures since last week, so I guess it sorted out by itself
[16:02]  * rockstar nods
[16:03] <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[16:03] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[16:03] <sinzui> me
[16:03] <Ursinha> it's me
[16:03] <Ursinha> so
[16:03]  * sinzui did not see the rollcall
[16:03] <Ursinha> three issues to point
[16:04] <Ursinha> matsubara: bug 403281 seems an old bug, and I saw a bunch of oopses yesterday caused by this problem, could you set the importance, please?
[16:04] <ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Undecided,Triaged] https://launchpad.net/bugs/403281
[16:04] <Ursinha> matsubara: also, I've filed bug 531695, but not sure if that's foundations
[16:04] <ubottu> Launchpad bug 531695 in libgraph-perl "libgraph-perl v0.81 is not compatible with libheap-perl v0.80" [Undecided,New] https://launchpad.net/bugs/531695
[16:04] <Ursinha> it seems a rollout issue
[16:04] <Ursinha> hmm, that's not right
[16:04] <matsubara> hmm
[16:04] <matsubara> hehe
[16:04] <Ursinha> bug 531965
[16:04] <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
[16:04] <Ursinha> oops :)
[16:05] <Ursinha> matsubara: what do you say?
[16:05] <matsubara> [action] matsubara to discuss with foundations bug 403281 and set importance on it
[16:05] <MootBot> ACTION received:  matsubara to discuss with foundations bug 403281 and set importance on it
[16:05] <ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Undecided,Triaged] https://launchpad.net/bugs/403281
[16:05] <Ursinha> cool
[16:05] <sinzui> I think bug 531965 is a soyuz issue
[16:05] <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
[16:06] <Ursinha> sinzui: but that isn't oopsing anymore
[16:06] <bigjools> no it's not a soyuz issue
[16:06] <matsubara> there's also bug 403281 which is either Code or foundations
[16:06] <matsubara> errr
[16:06] <matsubara> wrong bug number
[16:06] <Ursinha> sinzui: all occurrences I could find happened during the rollout
[16:06] <matsubara> I meant bug 531687
[16:06] <ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [Undecided,New] https://launchpad.net/bugs/531687
[16:06] <matsubara> rockstar, can you take a look and triage accordingly ^?
[16:07] <rockstar> matsubara, sure
[16:07] <rockstar> matsubara, is this part of the meeting really necessary?
[16:07] <Ursinha> allenap: I see occurrences of https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1524C2016, but not sure this is a known issue
[16:07] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1524C2016
[16:07] <matsubara> [action] rockstar to triage bug 531687
[16:07] <MootBot> ACTION received:  rockstar to triage bug 531687
[16:07] <ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [Undecided,New] https://launchpad.net/bugs/531687
[16:08] <allenap> Ursinha: Looking.
[16:08] <rockstar> We should (and do) keep up with triaging bugs.  I'm not sure that we need to bring it up in a meeting.
[16:08] <matsubara> rockstar, I think this section of the meeting is the main reason for this meeting :-)
[16:08] <Ursinha> rockstar: specially after rollouts
[16:08] <matsubara> may I ask why do you think it's not important?
[16:08] <Ursinha> :)
[16:08] <rockstar> matsubara, because you asked me to do something that someone from our team is going to do at some point today anyway.
[16:09] <rockstar> matsubara, I think the ops report is the most helpful part of this meeting.
[16:09] <matsubara> yeah, that part is very useful as well, I agree
[16:09] <matsubara> we bring up the OOPS that we think are important so it won't fall through the cracks
[16:10] <rockstar> matsubara, do we need to gather everyone together to do that?
[16:10] <Ursinha> rockstar: but I recall you suggesting us to file bugs for oopses before coming to the meeting, so why exactly you think the oops section is useful?
[16:10] <rockstar> Ursinha, ops wasn't a typo.  :)  Operations report
[16:10] <allenap> Ursinha: That reminds me of a known issue. I don't have a bug number for it yet. If it is the same issue then it's proven very difficult to reproduce.
[16:11] <matsubara> rockstar, well, we already gather everyone together for this meeting. discussing oops issues is just one side of it, along with scripts failing and critical issues
[16:11] <Ursinha> allenap: that reminds me of a known issue as well, but I couldn't find the bug
[16:12] <matsubara> the meeting was evolving to the point we're today. from our experience a lot of things end up falling through the cracks if the QA team don't bring them up
[16:12] <matsubara> but I'm all ears for suggestion to improve the format of the meeting that would be more useful to you
[16:12] <matsubara> and by you, I mean all of you
[16:12] <matsubara> :-)
[16:13] <matsubara> not only rockstar
[16:13] <rockstar> matsubara, but I ask if we need a meeting to bring them up.  Can't you just bring them up as they happen?
[16:13] <danilos> rockstar, I think OOPS section is necessary until we get to a point where every team is on top of their OOPSes *without* matsubara and Ursinha
[16:13] <matsubara> +1 danilos
[16:13] <Ursinha> rockstar: I think it's nice to have a main contact to express our concerns :)
[16:14] <rockstar> danilos, I agree that matsubara and Ursinha are a HUGE lifesaver.  I just wonder if it needs to be part of the meeting.
[16:14] <Ursinha> I agree with danilos
[16:14] <rockstar> Ursinha, oh yeah, I'm happy to be the main contact for QA questions.
[16:14] <danilos> rockstar, let's discuss that at the end of the meeting; Ursinha, matsubara, anyone else interested, let's move that discussion after the regular agenda
[16:14] <bigjools> I quite value the input from matsubara and Ursinha for the oopses
[16:15]  * rockstar sighs
[16:15] <matsubara> as danilos pointed out, once the teams are taking care of OOPSes without our help, then I think this section will slowly fade away
[16:15] <rockstar> bigjools, I don't disagree with you.
[16:15] <sinzui> I think we have a lot of unowned or shared apps that do not have team
[16:15] <bigjools> rockstar: I wasn't referring to anything you'd said actually
[16:16] <rockstar> bigjools, okay.  :)
[16:16] <danilos> let's go on and come back to this topic later, pleaaseee :)
[16:16] <sinzui> I am tempted to fix a bug in malone, launchapd-answers, and blueprints because I am tired of reading spam attacks to find my bugs, but who would know about these attacks if we were not collectively seeing all oopses
[16:17] <matsubara> all right, so Ursinha, anything else? (did I get all the action items right, specially the ones for foundations?)
[16:17] <Ursinha> matsubara: nothing else in the oops section
[16:17] <matsubara> [action] matsubara to discuss bug 531965 with foundations as well
[16:17] <MootBot> ACTION received:  matsubara to discuss bug 531965 with foundations as well
[16:17] <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
[16:18] <matsubara> scripts failing: mpcreationjobs and sendbranchmail just failed
[16:18] <Ursinha> there are two critical bugs, in progress
[16:19] <matsubara> rockstar, can you take care of those and reply to the list? is this something that could be caused by the rollout?
[16:20] <allenap> Ursinha: I can't find out any more about that OOPS now, so I'll investigate later.
[16:20] <Ursinha> thanks allenap :)
[16:20] <rockstar> matsubara, sure
[16:21] <matsubara> [action] rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
[16:21] <MootBot> ACTION received:  rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
[16:21] <Ursinha> bigjools: will you need a rc to fix bug 530566?
[16:21] <ubottu> Launchpad bug 530566 in soyuz "cron.germinate still believes lpia is active" [Critical,In progress] https://launchpad.net/bugs/530566
[16:21] <bigjools> Ursinha: I have one but thanks for reminding me to land that branch :)
[16:21] <Ursinha> bigjools: :)
[16:22] <Ursinha> matsubara: there's bug 530354 too that seems approved as well
[16:22] <ubottu> Launchpad bug 530354 in launchpad-foundations "wadl generation is broken after multiversion code has landed" [Critical,In progress] https://launchpad.net/bugs/530354
[16:22] <rockstar> matsubara, abentley is already figuring out what's going on with those scripts.
[16:22] <matsubara> thank you rockstar
[16:22] <Ursinha> matsubara: it says "Merged", so that is Fix Committed or even Fix Released? could you check, please?
[16:23] <matsubara> Ursinha, if it says merged, it's fix committed. I think it's this one: # r9069 [release-critical=thumper][r=gary][ui=none] Cache a static WADL file for every version of the web service.
[16:23] <matsubara> probably my script that closes it didn't get to it yet :-)
[16:23] <Ursinha> matsubara: hehe
[16:23] <Ursinha> okay
[16:24] <matsubara> let's move on then
[16:24] <Ursinha> thanks everyone
[16:24] <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[16:24] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[16:24] <matsubara> Chex, the stage is yours
[16:25] <mbarnett> take it and run!
[16:25] <Chex> hi everyone
[16:26] <Chex> Here is the LOSA report for this week:
[16:26] <Chex> - LP rollout 10.02 took 70 mins longer than scheduled. Some related bugs were:
[16:26] <Chex>         ; Need more visibility into progress of upgrade.py/fti.py/security.py bug 531833
[16:26] <ubottu> Launchpad bug 531833 in launchpad-foundations "Need more visibility into the progress of upgrade.py/fti.py/security.py" [High,Triaged] https://launchpad.net/bugs/531833
[16:26] <Chex>         ; Read-only seemed to leave lots of "idle" and "select waiting" connections to the master and
[16:26] <Chex>                 slave DBs which possibly blocked the DB upgrade bug 531834
[16:26] <ubottu> Launchpad bug 531834 in launchpad-foundations "When switching to read-only mode, we're left with lots of "idle/select waiting" connections to the DBs which may be blocking the schema upgrade process" [High,Triaged] https://launchpad.net/bugs/531834
[16:26] <Chex> - LP incidents of note:
[16:26] <Chex>         ; Crowberry Out-Of-Memory error: https://pastebin.canonical.com/28737/ 2010-03-04 12:30 UTC
[16:26] <Chex>           Does anyone have any insight into this issue??
[16:26] <Chex> and thats the report. anyone have questions or comments?
[16:27] <matsubara> crowberry is code hosting, right? rockstar ^
[16:27] <rockstar> matsubara, I'm not sure.  I never remember the names of our boxes...
[16:28] <matsubara> the pastebin indicates it's a code hosting machine
[16:29] <matsubara> [action] rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
[16:29] <MootBot> ACTION received:  rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
[16:29] <Chex> sorry, crowberry is codehosting, yes.
[16:30] <matsubara> ok, I think we can move on then.
[16:30] <matsubara> thanks Chex
[16:31] <Chex> matsubara: sure
[16:32] <matsubara> [TOPIC] * DBA report (stub)
[16:32] <MootBot> New Topic:  * DBA report (stub)
[16:32] <matsubara> stub will send the report to the list. I'll remind him.
[16:32] <matsubara> [action] matsubara to remind stub about dba report
[16:32] <allenap> matsubara: I think he sent it already.
[16:32] <MootBot> ACTION received:  matsubara to remind stub about dba report
[16:32] <matsubara> oops
[16:32] <matsubara> yes, indeed he did. thanks allenap
[16:32] <matsubara> [action] cancel last action item about dba report :-)
[16:32] <MootBot> ACTION received:  cancel last action item about dba report :-)
[16:33] <matsubara> [TOPIC] * Proposed items
[16:33] <MootBot> New Topic:  * Proposed items
[16:33] <matsubara> rockstar, so would you like to discuss a better format for the oops section?
[16:34] <matsubara> seems to me we have a agreement that the oops section is generally useful and once the teams start to take care of their oopses that section will be unecessary
[16:34] <danilos> matsubara, we don't have agreement that it will be unnecessary :)
[16:35] <danilos> matsubara, sinzui has raised an interesting point
[16:35] <danilos> matsubara, there's a whole bunch of problems which don't belong in any particular team domain
[16:35] <danilos> matsubara, and then there are also those which appear only for one or two teams, but might be global
[16:35] <matsubara> right, so we'll continue to bring up those during this meeting
[16:35] <danilos> I am hoping we'll get to a point where OOPS section will be about those
[16:36] <matsubara> yep. agreed.
[16:36] <rockstar> I think we're pretty good at saying "this doesn't look like a <app> issue.  Maybe ask <other-app-dev>"
[16:37] <danilos> and prerequisite for not reporting any OOPSes in a meeting is that both teams handle them themselves and that teams are good at bug triage
[16:37] <rockstar> If there's a bug filed on an oops, it'll get triaged.  That should be a team's responsibility, and if the team isn't getting to it, then we need to evaluate that.
[16:37] <rockstar> Code has been going out of our way to make sure we get bugs triaged.
[16:37] <danilos> rockstar, well, we are still very far off zero OOPSes in normal circumstances
[16:38] <Ursinha> danilos: +1
[16:38] <rockstar> danilos, this is true, but that's not the issue.
[16:38] <danilos> rockstar, right, I'd say we have a pretty good situation in translations (rarely more than 10 untriaged bugs), but does it hold for all teams?
[16:38] <danilos> rockstar, how about launchpad-foundations which is our hold-all bag?
[16:38] <rockstar> If there's an oops that Ursinha or matsubara sees, they file a bug (they are good at this).  At that point, they should assume the ball has been passed.
[16:39] <rockstar> danilos, how is launchpad-foundations the "hold-all" bag?  If they have bugs that belong to code, they're hurting foundations and code by not re-assigning them.
[16:39] <danilos> rockstar, sharing knowledge is useful; when I've seen some OOPSes related to authdb split in translations, I pinged bigjools because I suspected it might bite soyuz as well
[16:40] <rockstar> danilos, yeah, but that doesn't need to happen in the meeting.
[16:40] <danilos> rockstar, they have a gazillion bugs which are generic LP problems but not necessarily part of infrastructure, like UI features and such
[16:40] <sinzui> One problem that I am thinking of was caused by our switch to storm and each application has to make their own fix. This is not a foundations issue, it is an oops that should be targeted to 3 projects
[16:40] <danilos> sinzui, +1
[16:40] <rockstar> sinzui, exactly.
[16:41] <danilos> what I am saying is that we do have problems in some teams that need to be fixed before we can even consider dropping OOPS section from the meeting
[16:41] <rockstar> Make no mistake.  matsubara and Ursinha are invaluable to Launchpad.  I just don't think they need to duplicate their efforts because I can't triage bugs.
[16:42] <matsubara> danilos, +1 on the share knowledge usefulness. that's why I think this meeting (and the oops section) is still useful. we bring everyone together and talk about those issues. ideally this would all happen in the bug tracker, but reality is that sometimes some bugs end up falling through the cracks.
[16:42] <Ursinha> matsubara: +1
[16:42] <matsubara> and that's where Ursinha and I could help by nagging you guys endlessly :-)
[16:42] <rockstar> So let's keep the oops section, but not bring up untriaged bugs.
[16:42] <Ursinha> yes hehe
[16:42] <danilos> rockstar, I agree, but we first need to make sure "you" are triaging your bugs; care to do the evaluation of all LP subprojects and come back with data that either confirms or rejects my feeling of what it is?
[16:43] <rockstar> danilos, should QA have to babysit us to triage our bugs?  That doesn't seem fair.
[16:43] <Ursinha> rockstar: I don't know if that's ideal now; sometimes we dig old triaged bugs (or partially triaged) that need to be taken care now
[16:43] <danilos> rockstar, no, they shouldn't; can you guarantee all teams will have a very low number of untriaged bugs by next Thursday?
[16:43] <danilos> Ursinha, rockstar is not complaining about those
[16:44] <danilos> or at least not strongly :)
[16:44] <rockstar> danilos, I can't guarantee that, but maybe it's something we should start pushing harder.
[16:44] <Ursinha> danilos: I'm saying that mentioning bugs isn't useless, triaged or untriaged, because afaik me and matsubara only bring up that ones that need to be taken care of
[16:45] <danilos> rockstar, what they do today is to raise importance of OOPS bugs in this meeting: sometimes they'll be untriaged, sometimes triaged
[16:45] <Ursinha> danilos: agreeing with you
[16:45] <danilos> rockstar, I agree we should start pushing harder, let's have untriaged bugs part of the OOPS section until we feel confident we've done a good push
[16:46] <matsubara> danilos, so you're asking some kinda of count of untriaged bugs per team in the oops section? we could do that easily, I think.
[16:47] <danilos> rockstar, we have 218 'New' bugs on launchpad-project today
[16:47] <rockstar> Alright.  It's obvious that we're at an impass.  I'll bring up my personal feelings with my line manager, and see what comes of it.
[16:47] <rockstar> I guess leave the meeting as it is.
[16:47] <matsubara> danilos, or bring up those untriaged, but until we get those to manageable number, bringing up all of them during the meeting will be difficult and take a lot of time
[16:47] <danilos> rockstar, now I have to look up 'impass' in a dictionary :)
[16:47] <rockstar> danilos, deadlock.
[16:48] <danilos> rockstar, ok, thanks
[16:48] <sinzui> suck to be soyuz it seems
[16:49] <danilos> rockstar, matsubara: having a per-team count of untriaged bugs would be a good start
[16:49] <sinzui> and I see one of the new bugs claims to have a patch!
[16:49] <matsubara> [action] matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
[16:49] <MootBot> ACTION received:  matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
[16:49] <sinzui> launchpad-foundations has a bug with a patch. I think they are hoping that jml will take the bug away from them
[16:49] <jml> what
[16:50] <jml> (gotta ask me first)
[16:50] <sinzui> jml: https://bugs.edge.launchpad.net/launchpad-foundations/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&fie
[16:50]  * sinzui hates bug urls
[16:50] <danilos> rockstar, btw, I'd be very happy if you get this moving in any way you can, so thanks for raising it!
[16:51] <danilos> sinzui, we should integrate shorturl service into LP and generate/reuse those URLs with every bugs page :)
[16:52] <sinzui> danilos: what? implement saved searches for users and teams?
[16:52] <danilos> sinzui, kind of :)
[16:52] <danilos> anyway, matsubara, I believe the topic is settled
[16:53] <sinzui> We certainly would get a lot of cheers from users and even OEM
[16:53] <matsubara> danilos, thanks for the feedback on this. you too rockstar and sinzui
[16:53] <matsubara> anything else before I close?
[16:54] <jml> sinzui, me too
[16:54] <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[16:54] <jml> sinzui, anyway, I'm busy and not really paying attention. If you want me to do something, please email me something asking about it :)
[16:54] <Ursinha> sinzui: from me too :P
[16:54] <matsubara> #endmeeting
[16:54] <MootBot> Meeting finished at 10:54.
[16:54] <Ursinha> matsubara: thanks
[16:55] <Ursinha> thanks all
[16:55] <bigjools> gnargh
[16:55] <sinzui> jml: I was only surprised to see bugs in the new state with patches, then I remembers there is one marked UI, we are planning to move thos bugs to another project
=== salgado-lunch is now known as salgado
[17:06] <danilos> sinzui, the count of bugs with patches is totally broken for 'rosetta' project
[17:07] <danilos> sinzui, it tells me "8 bugs with patches" and when I click it, it's two bugs both in 'fix released' (with multiple bugtasks, but all of them are fix released)
[17:23] <sinzui> danilos: I think the bugs team need to fix that. I wonder it they skipped the default rule of ignore fixed bugs
=== danilos is now known as daniloff
=== gary_poster is now known as gary-lunch
[18:25] <Ursinha> daniloff: sinzui, hm, I just filed a bug for that
=== Edwin-afk is now known as EdwinGrubbs
=== gary-lunch is now known as gary_poster
=== matsubara is now known as matsubara-afk
=== salgado is now known as salgado-afk