| === mthaddon` is now known as mthaddon | ||
| === danilo-afk is now known as danilos | ||
| === mrevell is now known as mrevell-lunch | ||
| === salgado-afk is now known as salgado | ||
| === mrevell-lunch is now known as mrevell | ||
| === gary_poster is now known as gary-sprint | ||
| Ursinha | MootBot, hi hi -.- | 15:57 |
|---|---|---|
| sinzui | Wake up MootBot | 15:59 |
| matsubara | #startmeeting | 16:00 |
| MootBot | Meeting started at 10:00. The chair is matsubara. | 16:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 16:00 |
| gary-sprint | me | 16:00 |
| gary-sprint | :-D | 16:00 |
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 16:00 |
| matsubara | [TOPIC] Roll Call | 16:00 |
| MootBot | New Topic: Roll Call | 16:00 |
| sinzui | me | 16:00 |
| gary-sprint | uh, me | 16:00 |
| Ursinha | me | 16:00 |
| matsubara | rockstar, Chex, bigjools, allenap: hi | 16:00 |
| allenap | me | 16:01 |
| matsubara | apologies from stub | 16:01 |
| rockstar | me | 16:02 |
| mthaddon | me | 16:02 |
| bigjools | me | 16:03 |
| matsubara | [TOPIC] Agenda | 16:03 |
| MootBot | New Topic: Agenda | 16:03 |
| mthaddon | matsubara: I'm sitting in for Chex this meeting as he's working on U1 stuff | 16:03 |
| matsubara | * Actions from last meeting | 16:03 |
| matsubara | * Oops report & Critical Bugs & Broken scripts | 16:03 |
| matsubara | * Operations report (mthaddon/Chex/spm/mbarnett) | 16:03 |
| matsubara | * DBA report (stub) | 16:03 |
| matsubara | * Proposed items | 16:03 |
| matsubara | thanks mthaddon | 16:03 |
| matsubara | [TOPIC] * Actions from last meeting | 16:03 |
| MootBot | New Topic: * Actions from last meeting | 16:03 |
| matsubara | * matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas | 16:03 |
| matsubara | * Chex to check app server logs and apache logs to see if it can shed any light in the high load issue. | 16:03 |
| matsubara | * adeuring to check with gmb about checkwatches failure | 16:03 |
| matsubara | * danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance | 16:03 |
| ubottu | Launchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,Fix released] https://launchpad.net/bugs/438039 | 16:03 |
| matsubara | * bigjools to investigate update-cache failure and reply back to the list | 16:03 |
| danilos | matsubara: the bug tells you what it was :) | 16:04 |
| danilos | oh, I forgot to 'me' myself | 16:04 |
| matsubara | I'll finish up my action today | 16:04 |
| matsubara | thanks danilos | 16:04 |
| bigjools | I chatted to curtis and as far as wecan tell it was caused by something else holding a transaction/table open | 16:04 |
| bigjools | not much I can do | 16:05 |
| matsubara | gmb replied to checkwatches failure email. it was a hung process which was killed and service resumed | 16:06 |
| sinzui | Since the PRF ran the following days, I believe it was a long running process that worried our watching proc | 16:06 |
| matsubara | bigjools, thanks for checking. I don't see new emails from that script failing so I take it's working normally | 16:06 |
| bigjools | yep | 16:06 |
| matsubara | mthaddon, any luck investigating the high loading issue? | 16:07 |
| matsubara | s/loading/load/ | 16:07 |
| mthaddon | matsubara: I wasn't aware that was something we were following up on - not sure what the latest is, but I guess part of it plays into the new SplitIt stuff | 16:08 |
| mthaddon | i.e. we've just brought a whole bunch of new servers online so we need to see what effect this has on the overall load of the system | 16:08 |
| matsubara | all right. I'll take that item off the list and if high load shows up in the graphs we can pursue further | 16:09 |
| mthaddon | k | 16:09 |
| matsubara | thanks all, moving on | 16:09 |
| matsubara | [TOPIC] * Oops report & Critical Bugs & Broken scripts | 16:09 |
| MootBot | New Topic: * Oops report & Critical Bugs & Broken scripts | 16:09 |
| Ursinha | it'sme | 16:10 |
| Ursinha | gary_poster, bug 331990, can we CP it? | 16:10 |
| * sinzui stares at Ursinha | 16:10 | |
| ubottu | Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 | 16:10 |
| Ursinha | s/gary_poster/gary-sprint/ | 16:10 |
| gary-sprint | Ursinha: I do not have CP-foo. | 16:10 |
| Ursinha | allenap, can we have a fix for bug 438802 and maybe CP it? | 16:12 |
| matsubara | gary-sprint, is this a matter of updating the lazr.restful lib used by lpnet? | 16:12 |
| Ursinha | allenap, also, we have bug 438985, it's in progress but without activity for a some time | 16:12 |
| ubottu | Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 | 16:12 |
| Ursinha | allenap, and bug 458180, that's BugTask index timeouts | 16:12 |
| ubottu | Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985 | 16:12 |
| ubottu | Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180 | 16:12 |
| Ursinha | sinzui, I've filed bug 458169 and bug 458189, the timeouts on Milestone and DistroSeries index pages | 16:12 |
| Ursinha | rockstar, can we have a fix for bug 442981? | 16:12 |
| ubottu | Launchpad bug 458169 in launchpad-registry "Distroseries:+index page timing out" [High,Triaged] https://launchpad.net/bugs/458169 | 16:12 |
| ubottu | Launchpad bug 458189 in launchpad-registry "Milestone:+index pages timing out" [Undecided,New] https://launchpad.net/bugs/458189 | 16:12 |
| ubottu | Launchpad bug 442981 in launchpad-code "launchpad-project/+activereviews is OOPSing with TypeError (dup-of: 457541)" [High,Triaged] https://launchpad.net/bugs/442981 | 16:12 |
| ubottu | Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541 | 16:12 |
| gary-sprint | Ursinha: maybe I misunderstood. are you asking for CP-blessing or for CP-shepherding? If the latter, sure, we can shepherd. | 16:12 |
| Ursinha | gary-sprint, shepherding | 16:13 |
| sinzui | Ursinha: I replied that I beleive they are dups of 455812 | 16:13 |
| sinzui | brad is already working on it | 16:13 |
| Ursinha | bug 455812 | 16:13 |
| ubottu | Launchpad bug 455812 in launchpad-registry "distroseries milestone timeout" [High,Triaged] https://launchpad.net/bugs/455812 | 16:13 |
| Ursinha | hmmm | 16:13 |
| gary-sprint | matsubara: not sure, will ask leonardr. | 16:13 |
| Ursinha | sinzui,I'll mark it as a dupe then, thanks | 16:13 |
| sinzui | not yet | 16:13 |
| rockstar | Ursinha, the fix for that bug is closing it as a duplicate of bug 457541 | 16:13 |
| ubottu | Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541 | 16:13 |
| Ursinha | oh | 16:13 |
| rockstar | Ursinha, also, that bug is Fix Released. | 16:14 |
| matsubara | [action] gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990 | 16:14 |
| MootBot | ACTION received: gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990 | 16:14 |
| ubottu | Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990 | 16:14 |
| gary-sprint | +1 | 16:14 |
| Ursinha | rockstar,it still happens, how come? | 16:14 |
| rockstar | Ursinha, so yes, you may have it before it was asked. | 16:14 |
| sinzui | I have assign the distroseries +index to edwin. I think EdwinGrubbs and bac will find this is the same problem | 16:14 |
| sinzui | The oopses of the two new bugs look the the oopses I have been tracking in the older bug | 16:15 |
| rockstar | Ursinha, doesn't oops for me. | 16:15 |
| Ursinha | rockstar, so the summaries are lying :) | 16:15 |
| rockstar | Ursinha, does this url oops for you? https://edge.launchpad.net/launchpad-project/+activereviews | 16:16 |
| Ursinha | rockstar, well, it's loading... I'll keep my eye on it and if needed reopen it, right? | 16:16 |
| Ursinha | rockstar, thanks | 16:16 |
| Ursinha | allenap, hi :) | 16:17 |
| allenap | Ursinha: I talk to deryck about getting bug 438802 fixed, and gmb about bug 438985. | 16:17 |
| ubottu | Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 | 16:17 |
| ubottu | Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985 | 16:17 |
| Ursinha | allenap, thanks | 16:17 |
| allenap | Ursinha: Bug 458180 is a perennial problem. | 16:17 |
| ubottu | Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180 | 16:17 |
| Ursinha | allenap, I see the main offender is bug #1 | 16:17 |
| ubottu | https://bugs.launchpad.net/ubuntu/+bug/1 (Timeout) | 16:17 |
| Ursinha | yes, *sigh* | 16:17 |
| allenap | Ursinha: Yeah, it always is :) | 16:17 |
| rockstar | Ursinha, yes, but I can't see how it'd get reopened. It was bad data, we fixededed the database records. | 16:18 |
| danilos | Ursinha: and you just made it worse with a reference now :) | 16:18 |
| Ursinha | danilos, yes, just to prove my point :P | 16:18 |
| matsubara | allenap, it still happens in other bugs too as per jono's email to launchpad-dev about ubuttu timing out | 16:18 |
| Ursinha | allenap, there are some oopses not caused by #1 | 16:18 |
| Ursinha | thanks allenap | 16:19 |
| allenap | matsubara: Okay, as someone said, perhaps it's the +text interface. | 16:19 |
| Ursinha | gary-sprint, the "buildbot failure in Launchpad on jscheck", is it severe? | 16:19 |
| matsubara | allenap, I briefly trawled the summaries and there are a some sofr timeouts on +text, but soft timeouts shouldn't be affecting ubottu | 16:19 |
| allenap | matsubara, Ursinha: We need to do something more drastic to get the bug page quicker I think. Caching, etc, and that's coming alone. We've done a lot of the other things we can think of, but I'll discuss it with the team. | 16:19 |
| allenap | matsubara: That's interesting. | 16:20 |
| allenap | s/alone/along/ | 16:20 |
| Ursinha | I see some emails from francis and rockstar talking about it, is there something that can or needs to be done? | 16:21 |
| matsubara | allenap, perhaps those timeouts are not being logged as OOPSes? similar to 500 we see eventually from apache | 16:21 |
| Ursinha | gary-sprint, ^ | 16:21 |
| matsubara | to the 500 errors I mean | 16:21 |
| rockstar | Ursinha, gary-sprint, it is my belief that windmill sucks. | 16:21 |
| gary-sprint | Ursinha: it does not appear to be a problem in the basic buildbot setup at the moment. There are failures in the tests. This doesn't seem to be a foundations issue AFAICT. Björn may very well be able to help when he returns | 16:21 |
| allenap | matsubara: Okay, I'm not sure what you mean, but we can talk about it after the meeting. | 16:22 |
| Ursinha | right gary-sprint, thanks for the info | 16:22 |
| matsubara | allenap, sure thing. I'll find the bug I'm referring to | 16:22 |
| allenap | matsubara: Thanks. | 16:22 |
| matsubara | [action] allenap and matsubara to talk about the timeouts on bug pages | 16:22 |
| MootBot | ACTION received: allenap and matsubara to talk about the timeouts on bug pages | 16:22 |
| Ursinha | right, I'm done here | 16:23 |
| gary-sprint | rockstar: that's probably a given. The more interesting question is whether it sucks worse than the alternatives. My impression is no, but a champion could fight for an alternate view, | 16:23 |
| gary-sprint | . | 16:23 |
| Ursinha | allenap, is it possible to ask for a cp for bug 438802 when it's fixed? | 16:23 |
| ubottu | Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802 | 16:23 |
| rockstar | gary-sprint, sadly, there is no better alternative. Windmill sucks less than anything else out there. | 16:23 |
| === salgado is now known as salgado-lunch | ||
| gary-sprint | :-) | 16:23 |
| allenap | Ursinha: Sure. | 16:23 |
| Ursinha | anmar was having problems yesterday with bugs with chinese chars, I think it's worth doing a CP | 16:24 |
| Ursinha | thanks allenap | 16:25 |
| allenap | Ursinha: np, thank you :) | 16:25 |
| Ursinha | :) | 16:26 |
| matsubara | ok, two fix committed critical bugs | 16:26 |
| matsubara | rockstar, we had some failures on the update_preview_diffs script | 16:26 |
| matsubara | on the 19th | 16:26 |
| rockstar | matsubara, yeah, we're currently in the process of fixing the various oopses that script creates. | 16:28 |
| matsubara | rockstar, ok. can you give me the bug numbers after the meeting? | 16:28 |
| rockstar | matsubara, there are many. | 16:29 |
| matsubara | gladly we have an oops tag to filter those :-) | 16:29 |
| matsubara | rockstar, I'll ping you after the meeting | 16:29 |
| Ursinha | thanks everyone | 16:30 |
| matsubara | I think that's all for this section. thanks everyone | 16:30 |
| matsubara | [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett) | 16:30 |
| MootBot | New Topic: * Operations report (mthaddon/Chex/spm/mbarnett) | 16:30 |
| mthaddon | SplitIt is the big one this week - now complete with exception of Auth DB split. | 16:30 |
| mthaddon | New App servers brought online after haproxy throttlng of connections, we're watching how things are progressing | 16:30 |
| mthaddon | A number of CPs done this week | 16:30 |
| mthaddon | Is everyone clear on the new CP process? | 16:30 |
| mthaddon | Shipit now managed by ISD, and CPs to be approved by nigelp | 16:30 |
| mthaddon | Some app servers dying, loggerhead dying, poppy died once - is there a process for reviewing the Incident Log? | 16:30 |
| mthaddon | That's about it | 16:30 |
| matsubara | mthaddon, last I heard Francis was the one to champion the Incident Log process. | 16:31 |
| mthaddon | matsubara: basically we want to be sure someone's reviewing it to look for operational trends in production | 16:32 |
| bigjools | did he mention making trvial wiki edits for codebounce so we don't get email for those? | 16:32 |
| matsubara | ideally we won't need that codebounce all the time :-) | 16:32 |
| Ursinha | matsubara, +1 :) | 16:32 |
| mthaddon | bigjools: if we have to get alerts and go through the whole restart, edit wiki nightmare, you can put up with a few wiki edit notifications :) | 16:33 |
| * matsubara looks at rockstar | 16:33 | |
| bigjools | mthaddon: well, no I don't :) | 16:33 |
| danilos | mthaddon: the concern is that we may learn to ignore it unless we can filter stuff out *we* can't do anything about | 16:33 |
| rockstar | mthaddon, I'm subscribed and get the pleasure of seeing every time you restart loggerhead. | 16:33 |
| matsubara | any news about the codebrowser dying all the time? | 16:33 |
| bigjools | what danilos said | 16:33 |
| danilos | mthaddon: specifically, translations or soyuz team can't help much with codebrowse restarts | 16:33 |
| rockstar | matsubara, we are bringing someone on to look into the codebrowse issue. That's all I know. We certainly don't have the bandwidth currently to do it. | 16:34 |
| mthaddon | fwiw, I usually do trivial that one - I guess maybe the other losas don't - will mention it | 16:34 |
| danilos | bigjools: just as an example, and these are very, very common | 16:34 |
| sinzui | I believe there is a plan to but people to work on loggerhead | 16:34 |
| danilos | mthaddon: in general, anything else shoudn't be a trivial edit, and codebrowse should, that would help old men like bigjools deal with their email :) | 16:34 |
| bigjools | ha | 16:34 |
| danilos | sinzui: yeah, as flacoste mentioned today, I think we are having a contract that starts today or tomorrow | 16:35 |
| matsubara | that's great news | 16:35 |
| danilos | mthaddon: but, do note that most team leads are subscribed to LPIncidentLog, and if one isn't, feel free to poke them about it | 16:35 |
| mthaddon | danilos: k, thx | 16:36 |
| matsubara | that's all mthaddon ? | 16:37 |
| mthaddon | yep | 16:37 |
| matsubara | all right. thanks everyone | 16:37 |
| matsubara | [TOPIC] * DBA report (stub) | 16:38 |
| MootBot | New Topic: * DBA report (stub) | 16:38 |
| matsubara | stub is on vacation and looks like the db is fine | 16:38 |
| matsubara | AFAICT | 16:38 |
| matsubara | so let's move on. | 16:38 |
| matsubara | [action] matsubara to talk to stub about the DBA report when he gets back | 16:39 |
| MootBot | ACTION received: matsubara to talk to stub about the DBA report when he gets back | 16:39 |
| matsubara | [TOPIC] * Proposed items | 16:39 |
| MootBot | New Topic: * Proposed items | 16:39 |
| matsubara | no new proposed items | 16:39 |
| matsubara | and I think that's all for today | 16:40 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. | 16:40 |
| matsubara | #endmeeting | 16:40 |
| MootBot | Meeting finished at 10:40. | 16:40 |
| Ursinha | thanks my dearest colleagues | 16:40 |
| === matsubara is now known as matsubara-lunch | ||
| === salgado-lunch is now known as salgado | ||
| === EdwinGrubbs is now known as Edwin-lunch | ||
| === danilos is now known as danilo-afk | ||
| === matsubara-lunch is now known as matsubara | ||
| === Edwin-lunch is now known as EdwinGrubbs | ||
| === salgado is now known as salgado-afk | ||
| === matsubara is now known as matsubara-afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!