=== mthaddon` is now known as mthaddon
=== danilo-afk is now known as danilos
=== mrevell is now known as mrevell-lunch
=== salgado-afk is now known as salgado
=== mrevell-lunch is now known as mrevell
=== gary_poster is now known as gary-sprint
UrsinhaMootBot, hi hi -.-15:57
sinzuiWake up MootBot15:59
MootBotMeeting started at 10:00. The chair is matsubara.16:00
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]16:00
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.16:00
matsubara[TOPIC] Roll Call16:00
MootBotNew Topic:  Roll Call16:00
gary-sprintuh, me16:00
matsubararockstar, Chex, bigjools, allenap: hi16:00
matsubaraapologies from stub16:01
matsubara[TOPIC] Agenda16:03
MootBotNew Topic:  Agenda16:03
mthaddonmatsubara: I'm sitting in for Chex this meeting as he's working on U1 stuff16:03
matsubara * Actions from last meeting16:03
matsubara * Oops report & Critical Bugs & Broken scripts16:03
matsubara * Operations report (mthaddon/Chex/spm/mbarnett)16:03
matsubara * DBA report (stub)16:03
matsubara * Proposed items16:03
matsubarathanks mthaddon16:03
matsubara[TOPIC] * Actions from last meeting16:03
MootBotNew Topic:  * Actions from last meeting16:03
matsubara* matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas16:03
matsubara* Chex to check app server logs and apache logs to see if it can shed any light in the high load issue.16:03
matsubara* adeuring to check with gmb about checkwatches failure16:03
matsubara* danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance16:03
ubottuLaunchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,Fix released] https://launchpad.net/bugs/43803916:03
matsubara* bigjools to investigate update-cache failure and reply back to the list16:03
danilosmatsubara: the bug tells you what it was :)16:04
danilosoh, I forgot to 'me' myself16:04
matsubaraI'll finish up my action today16:04
matsubarathanks danilos16:04
bigjoolsI chatted to curtis and as far as wecan tell it was caused by something else holding a transaction/table open16:04
bigjoolsnot much I can do16:05
matsubaragmb replied to checkwatches failure email. it was a hung process which was killed and service resumed16:06
sinzuiSince the PRF ran the following days, I believe it was a long running process that worried our watching proc16:06
matsubarabigjools, thanks for checking. I don't see new emails from that script failing so I take it's working normally16:06
matsubaramthaddon, any luck investigating the high loading issue?16:07
mthaddonmatsubara: I wasn't aware that was something we were following up on - not sure what the latest is, but I guess part of it plays into the new SplitIt stuff16:08
mthaddoni.e. we've just brought a whole bunch of new servers online so we need to see what effect this has on the overall load of the system16:08
matsubaraall right. I'll take that item off the list and if high load shows up in the graphs we can pursue further16:09
matsubarathanks all, moving on16:09
matsubara[TOPIC] * Oops report & Critical Bugs & Broken scripts16:09
MootBotNew Topic:  * Oops report & Critical Bugs & Broken scripts16:09
Ursinhagary_poster, bug 331990, can we CP it?16:10
* sinzui stares at Ursinha16:10
ubottuLaunchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/33199016:10
gary-sprintUrsinha: I do not have CP-foo.16:10
Ursinhaallenap, can we have a fix for bug 438802 and maybe CP it?16:12
matsubaragary-sprint, is this a matter of updating the lazr.restful lib used by lpnet?16:12
Ursinhaallenap, also, we have bug 438985, it's in progress but without activity for a some time16:12
ubottuLaunchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/43880216:12
Ursinhaallenap, and bug 458180, that's BugTask index timeouts16:12
ubottuLaunchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/43898516:12
ubottuLaunchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/45818016:12
Ursinhasinzui, I've filed bug 458169 and bug 458189, the timeouts on Milestone and DistroSeries index pages16:12
Ursinharockstar, can we have a fix for bug 442981?16:12
ubottuLaunchpad bug 458169 in launchpad-registry "Distroseries:+index page timing out" [High,Triaged] https://launchpad.net/bugs/45816916:12
ubottuLaunchpad bug 458189 in launchpad-registry "Milestone:+index pages timing out" [Undecided,New] https://launchpad.net/bugs/45818916:12
ubottuLaunchpad bug 442981 in launchpad-code "launchpad-project/+activereviews is OOPSing with TypeError (dup-of: 457541)" [High,Triaged] https://launchpad.net/bugs/44298116:12
ubottuLaunchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/45754116:12
gary-sprintUrsinha: maybe I misunderstood.  are you asking for CP-blessing or for CP-shepherding?  If the latter, sure, we can shepherd.16:12
Ursinhagary-sprint, shepherding16:13
sinzuiUrsinha: I replied that I beleive they are dups of 45581216:13
sinzuibrad is already working on it16:13
Ursinhabug 45581216:13
ubottuLaunchpad bug 455812 in launchpad-registry "distroseries milestone timeout" [High,Triaged] https://launchpad.net/bugs/45581216:13
gary-sprintmatsubara: not sure, will ask leonardr.16:13
Ursinhasinzui,I'll mark it as a dupe then, thanks16:13
sinzuinot yet16:13
rockstarUrsinha, the fix for that bug is closing it as a duplicate of bug 45754116:13
ubottuLaunchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/45754116:13
rockstarUrsinha, also, that bug is Fix Released.16:14
matsubara[action] gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 33199016:14
MootBotACTION received:  gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 33199016:14
ubottuLaunchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/33199016:14
Ursinharockstar,it still happens, how come?16:14
rockstarUrsinha, so yes, you may have it before it was asked.16:14
sinzuiI have assign the distroseries +index to edwin. I think EdwinGrubbs and bac will find this is the same problem16:14
sinzuiThe oopses of the two new bugs look the the oopses I have been tracking in the older bug16:15
rockstarUrsinha, doesn't oops for me.16:15
Ursinharockstar, so the summaries are lying :)16:15
rockstarUrsinha, does this url oops for you? https://edge.launchpad.net/launchpad-project/+activereviews16:16
Ursinharockstar, well, it's loading... I'll keep my eye on it and if needed reopen it, right?16:16
Ursinharockstar, thanks16:16
Ursinhaallenap, hi :)16:17
allenapUrsinha: I talk to deryck about getting bug 438802 fixed, and gmb about bug 438985.16:17
ubottuLaunchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/43880216:17
ubottuLaunchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/43898516:17
Ursinhaallenap, thanks16:17
allenapUrsinha: Bug 458180 is a perennial problem.16:17
ubottuLaunchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/45818016:17
Ursinhaallenap, I see the main offender is bug #116:17
ubottuhttps://bugs.launchpad.net/ubuntu/+bug/1 (Timeout)16:17
Ursinhayes, *sigh*16:17
allenapUrsinha: Yeah, it always is :)16:17
rockstarUrsinha, yes, but I can't see how it'd get reopened.  It was bad data, we fixededed the database records.16:18
danilosUrsinha: and you just made it worse with a reference now :)16:18
Ursinhadanilos, yes, just to prove my point :P16:18
matsubaraallenap, it still happens in other bugs too as per jono's email to launchpad-dev about ubuttu timing out16:18
Ursinhaallenap, there are some oopses not caused by #116:18
Ursinhathanks allenap16:19
allenapmatsubara: Okay, as someone said, perhaps it's the +text interface.16:19
Ursinhagary-sprint, the "buildbot failure in Launchpad on jscheck", is it severe?16:19
matsubaraallenap, I briefly trawled the summaries and there are a some sofr timeouts on +text, but soft timeouts shouldn't be affecting ubottu16:19
allenapmatsubara, Ursinha: We need to do something more drastic to get the bug page quicker I think. Caching, etc, and that's coming alone. We've done a lot of the other things we can think of, but I'll discuss it with the team.16:19
allenapmatsubara: That's interesting.16:20
UrsinhaI see some emails from francis and rockstar talking about it, is there something that can or needs to be done?16:21
matsubaraallenap, perhaps those timeouts are not being logged as OOPSes? similar to 500 we see eventually from apache16:21
Ursinhagary-sprint, ^16:21
matsubarato the 500 errors I mean16:21
rockstarUrsinha, gary-sprint, it is my belief that windmill sucks.16:21
gary-sprintUrsinha: it does not appear to be a problem in the basic buildbot setup at the moment.    There are failures in the tests.  This doesn't seem to be a foundations issue AFAICT.  Bj√∂rn may very well be able to help when he returns16:21
allenapmatsubara: Okay, I'm not sure what you mean, but we can talk about it after the meeting.16:22
Ursinharight gary-sprint, thanks for the info16:22
matsubaraallenap, sure thing. I'll find the bug I'm referring to16:22
allenapmatsubara: Thanks.16:22
matsubara[action] allenap and matsubara to talk about the timeouts on bug pages16:22
MootBotACTION received:  allenap and matsubara to talk about the timeouts on bug pages16:22
Ursinharight, I'm done here16:23
gary-sprintrockstar: that's probably a given.  The more interesting question is whether it sucks worse than the alternatives.  My impression is no, but a champion could fight for an alternate view,16:23
Ursinhaallenap, is it possible to ask for a cp for bug 438802 when it's fixed?16:23
ubottuLaunchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/43880216:23
rockstargary-sprint, sadly, there is no better alternative. Windmill sucks less than anything else out there.16:23
=== salgado is now known as salgado-lunch
allenapUrsinha: Sure.16:23
Ursinhaanmar was having problems yesterday with bugs with chinese chars, I think it's worth doing a CP16:24
Ursinhathanks allenap16:25
allenapUrsinha: np, thank you :)16:25
matsubaraok, two fix committed critical bugs16:26
matsubararockstar, we had some failures on the update_preview_diffs script16:26
matsubaraon the 19th16:26
rockstarmatsubara, yeah, we're currently in the process of fixing the various oopses that script creates.16:28
matsubararockstar, ok. can you give me the bug numbers after the meeting?16:28
rockstarmatsubara, there are many.16:29
matsubaragladly we have an oops tag to filter those :-)16:29
matsubararockstar, I'll ping you after the meeting16:29
Ursinhathanks everyone16:30
matsubaraI think that's all for this section. thanks everyone16:30
matsubara[TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)16:30
MootBotNew Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)16:30
mthaddonSplitIt is the big one this week - now complete with exception of Auth DB split.16:30
mthaddonNew App servers brought online after haproxy throttlng of connections, we're watching how things are progressing16:30
mthaddonA number of CPs done this week16:30
mthaddonIs everyone clear on the new CP process?16:30
mthaddonShipit now managed by ISD, and CPs to be approved by nigelp16:30
mthaddonSome app servers dying, loggerhead dying, poppy died once - is there a process for reviewing the Incident Log?16:30
mthaddonThat's about it16:30
matsubaramthaddon, last I heard Francis was the one to champion the Incident Log process.16:31
mthaddonmatsubara: basically we want to be sure someone's reviewing it to look for operational trends in production16:32
bigjoolsdid he mention making trvial wiki edits for codebounce so we don't get email for those?16:32
matsubaraideally we won't need that codebounce all the time :-)16:32
Ursinhamatsubara, +1 :)16:32
mthaddonbigjools: if we have to get alerts and go through the whole restart, edit wiki nightmare, you can put up with a few wiki edit notifications :)16:33
* matsubara looks at rockstar 16:33
bigjoolsmthaddon: well, no I don't :)16:33
danilosmthaddon: the concern is that we may learn to ignore it unless we can filter stuff out *we* can't do anything about16:33
rockstarmthaddon, I'm subscribed and get the pleasure of seeing every time you restart loggerhead.16:33
matsubaraany news about the codebrowser dying all the time?16:33
bigjoolswhat danilos said16:33
danilosmthaddon: specifically, translations or soyuz team can't help much with codebrowse restarts16:33
rockstarmatsubara, we are bringing someone on to look into the codebrowse issue.  That's all I know.  We certainly don't have the bandwidth currently to do it.16:34
mthaddonfwiw, I usually do trivial that one - I guess maybe the other losas don't - will mention it16:34
danilosbigjools: just as an example, and these are very, very common16:34
sinzuiI believe there is a plan to but people to work on loggerhead16:34
danilosmthaddon: in general, anything else shoudn't be a trivial edit, and codebrowse should, that would help old men like bigjools deal with their email :)16:34
danilossinzui: yeah, as flacoste mentioned today, I think we are having a contract that starts today or tomorrow16:35
matsubarathat's great news16:35
danilosmthaddon: but, do note that most team leads are subscribed to LPIncidentLog, and if one isn't, feel free to poke them about it16:35
mthaddondanilos: k, thx16:36
matsubarathat's all mthaddon ?16:37
matsubaraall right. thanks everyone16:37
matsubara[TOPIC] * DBA report (stub)16:38
MootBotNew Topic:  * DBA report (stub)16:38
matsubarastub is on vacation and looks like the db is fine16:38
matsubaraso let's move on.16:38
matsubara[action] matsubara to talk to stub about the DBA report when he gets back16:39
MootBotACTION received:  matsubara to talk to stub about the DBA report when he gets back16:39
matsubara[TOPIC] * Proposed items16:39
MootBotNew Topic:  * Proposed items16:39
matsubarano new proposed items16:39
matsubaraand I think that's all for today16:40
matsubaraThank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.16:40
MootBotMeeting finished at 10:40.16:40
Ursinhathanks my dearest colleagues16:40
=== matsubara is now known as matsubara-lunch
=== salgado-lunch is now known as salgado
=== EdwinGrubbs is now known as Edwin-lunch
=== danilos is now known as danilo-afk
=== matsubara-lunch is now known as matsubara
=== Edwin-lunch is now known as EdwinGrubbs
=== salgado is now known as salgado-afk
=== matsubara is now known as matsubara-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!