| === mrevell is now known as mrevell-lunch | ||
| === salgado is now known as salgado-brb | ||
| === mrevell-lunch is now known as mrevell | ||
| === salgado-brb is now known as salgado | ||
| matsubara | ping MootBot | 14:58 |
|---|---|---|
| matsubara | #startmeeting | 15:00 |
| MootBot | Meeting started at 09:00. The chair is matsubara. | 15:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 15:00 |
| === salgado is now known as salgado-lunch | ||
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 15:00 |
| matsubara | [TOPIC] Roll Call | 15:00 |
| MootBot | New Topic: Roll Call | 15:00 |
| herb | me | 15:00 |
| jtv | me | 15:00 |
| sinzui | me | 15:00 |
| Ursinha | me | 15:00 |
| rockstar | me | 15:00 |
| jtv | I'm standing in for henninge today, since he's on sprint. | 15:01 |
| matsubara | thanks jtv | 15:01 |
| danilos | me | 15:01 |
| BjornT | me | 15:01 |
| matsubara | so, if any of you can't make the meeting next week, please coordinate with another teammate to cover for you and add a notice in the Apologies section in the MeetingAgenda page. | 15:02 |
| matsubara | flacoste: ping | 15:02 |
| intellectronica | me | 15:02 |
| matsubara | bigjools: ping | 15:03 |
| bigjools | me | 15:03 |
| flacoste | me | 15:03 |
| matsubara | ok, let's move on, stub can join later | 15:04 |
| matsubara | [TOPIC] Agenda | 15:04 |
| MootBot | New Topic: Agenda | 15:04 |
| matsubara | * Actions from last meeting | 15:04 |
| matsubara | * Oops report & Critical Bugs | 15:04 |
| matsubara | * Operations report (mthaddon/herb/spm) | 15:04 |
| matsubara | * DBA report (DBA contact) | 15:04 |
| matsubara | [TOPIC] * Actions from last meeting | 15:04 |
| MootBot | New Topic: * Actions from last meeting | 15:04 |
| matsubara | * bac to check with barry if there's an open bug for OOPS-1125A1096, if not, Ursinha to file one - there was, bug 280925 | 15:04 |
| matsubara | * intellectronica to work on bug 279561 | 15:04 |
| matsubara | * rockstar to check OOPS-1125CEMAIL1 | 15:04 |
| matsubara | * bac to take a look at OOPS-1125A165 - bac filed bug 322792 | 15:04 |
| matsubara | * Ursinha to check with kiko if any other rollouts will happen this week | 15:04 |
| ubottu | Launchpad bug 280925 in launchpad-registry "Project overview page shows obsolete series" [Low,Fix released] https://launchpad.net/bugs/280925 | 15:04 |
| ubottu | https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096 | 15:05 |
| ubottu | Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/279561/+text) | 15:05 |
| ubottu | Launchpad bug 322792 in launchpad-bazaar "Attempting traversal past an unknown object causes an OOPS" [Undecided,Fix released] https://launchpad.net/bugs/322792 | 15:05 |
| ubottu | https://devpad.canonical.com/~jamesh/oops.cgi/1125A165 | 15:05 |
| Ursinha | holy crap | 15:05 |
| Ursinha | that bug is timing out | 15:05 |
| Ursinha | or what? | 15:05 |
| Ursinha | anyway | 15:06 |
| matsubara | intellectronica: any news about the api bug? | 15:06 |
| Ursinha | my items were done | 15:06 |
| matsubara | rockstar: what's up about that oops? | 15:06 |
| intellectronica | matsubara: sorry, no news yet | 15:06 |
| Ursinha | matsubara, he landed a fix for that | 15:06 |
| matsubara | Ursinha: who's he? | 15:06 |
| matsubara | :-) | 15:06 |
| Ursinha | matsubara, rockstar :) | 15:06 |
| Ursinha | sorry | 15:06 |
| rockstar | matsubara, I got an RC in for that one. | 15:06 |
| matsubara | ok, I remember that one. | 15:07 |
| matsubara | so I guess the only pending one is the api bug which is a mistery to everyone | 15:07 |
| matsubara | the good news is that intellectronica found out another thing about that bug that might lead to its root cause | 15:08 |
| matsubara | intellectronica: thanks for keeping us posted in the report | 15:08 |
| matsubara | let's move on | 15:08 |
| matsubara | [TOPIC] * Oops report & Critical Bugs | 15:08 |
| MootBot | New Topic: * Oops report & Critical Bugs | 15:08 |
| matsubara | so in today's oops section I'd like to talk about the timeout bugs you guys are working for the LPW | 15:09 |
| matsubara | https://dev.launchpad.net/PerformanceWeeks/February2009 | 15:09 |
| matsubara | I'm going to review all the landings related to LPW work and add to that page. | 15:10 |
| matsubara | so if you wanna help, point me to revision numbers on RF related to that work | 15:10 |
| jtv | matsubara: bug 324264 is now Fix Committed. | 15:10 |
| ubottu | Launchpad bug 324264 in rosetta "Speed up +translations" [High,Fix committed] https://launchpad.net/bugs/324264 | 15:10 |
| sinzui | matsubara: r7705 for Bug: 325321 | 15:11 |
| Ursinha | jtv, great | 15:11 |
| matsubara | intellectronica, BjornT : any news about the bug number 1 time out? | 15:11 |
| sinzui | matsubara: EdwinGrubbs will land his branch today | 15:11 |
| danilos | work on bug 302798 is in progress in different ways (there's a commit from 323something which disabled external suggestions to give us a better idea on how stuff is working, and henning is working on removal of obsolete translations which will reduce our DB size by ~33%) | 15:11 |
| ubottu | Launchpad bug 302798 in rosetta "Timeout on +translate page" [High,Triaged] https://launchpad.net/bugs/302798 | 15:11 |
| BjornT | matsubara: me, intellectronica, and allenap are working on it | 15:12 |
| jtv | danilos: hey, I was putting that paragraph together! | 15:12 |
| BjornT | matsubara: allenap is looking at reducing the time it take to render the comments, possibly by not showing all by default | 15:12 |
| danilos | jtv: ok, I'll let you handle it all from now on :) | 15:13 |
| BjornT | matsubara: intellectronica is working on loading the subscribers portlet in a different request | 15:13 |
| danilos | matsubara: you can have full trust in jtv as far as PW is concerned :) | 15:13 |
| jtv | *cough* | 15:13 |
| BjornT | matsubara: and i'm working on optimizing code, based off profiling information | 15:13 |
| matsubara | danilos: I do! he's been very helpful with the status updates | 15:13 |
| BjornT | matsubara: also, intellectronica's inital tests on dogfood were successful, reducing the time quite a lot :) | 15:14 |
| matsubara | great | 15:14 |
| flacoste | matsubara: i'm working on bug 316881 (which isn't an OOPS per-se, but related to performance anyhow) | 15:14 |
| ubottu | Launchpad bug 316881 in launchpad "Page headers not suitable for HTTP caching" [High,In progress] https://launchpad.net/bugs/316881 | 15:14 |
| matsubara | stub: there's an email from jono asking for some help with the +project-cloud oops, so if you could help out there, would be awesome | 15:15 |
| matsubara | thanks flacoste, i'll add it to the page | 15:15 |
| stub | I've replied on the bug. Not sure if I'm helpful though. | 15:15 |
| matsubara | stub: cool. thank you | 15:15 |
| matsubara | bigjools: news in soyuz. how about the one muharem is taking care of? | 15:16 |
| matsubara | s/./?/ | 15:16 |
| bigjools | matsubara: it's not going so well unfortunately | 15:16 |
| bigjools | I don't expect any progress this week | 15:16 |
| matsubara | [action] matsubara to add 316881 to foundations section in LPW wiki page | 15:16 |
| MootBot | ACTION received: matsubara to add 316881 to foundations section in LPW wiki page | 15:16 |
| matsubara | bigjools: why is that? | 15:17 |
| bigjools | matsubara: the first attempt to fix it failed. he's also been at the distro sprint this week | 15:17 |
| matsubara | bigjools: oh right. well, you guys are excused since the whole team is sprinting and you already landed 2(?) timeout fixes :-) | 15:18 |
| bigjools | matsubara: thanks :) | 15:18 |
| matsubara | I guess that's it from me. Ursinha, anything else? | 15:18 |
| Ursinha | matsubara, no, the pending oops for soyuz I already talked with bigjools | 15:19 |
| bigjools | yeah, get edge updating again and we'll see how it went | 15:19 |
| matsubara | great. thanks everyone! | 15:19 |
| matsubara | [TOPIC] * Operations report (mthaddon/herb/spm) | 15:19 |
| MootBot | New Topic: * Operations report (mthaddon/herb/spm) | 15:19 |
| herb | - 2009-01-30 - Friday we updated lpnet, edge and the scripts servers to to r7676. | 15:19 |
| herb | - 2009-02-04 - Yesterday we updated codebrowse to r43 | 15:19 |
| herb | - I feel like I'm starting to sound like a broken record... But we're still being bothered daily, often multiple times, by bug #156453 and bug #118625 which seem to be related. | 15:20 |
| ubottu | Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453 | 15:20 |
| herb | - We also continue to run into issues associated with bug #260171 | 15:20 |
| ubottu | Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 | 15:20 |
| ubottu | Bug 260171 on http://launchpad.net/bugs/260171 is private | 15:20 |
| flacoste | herb: i herd mwhudson was working on loggerhead performance this week | 15:20 |
| flacoste | as part of LPW | 15:20 |
| matsubara | yes! | 15:20 |
| sinzui | herb: There is a sprint in the planning to fix that too | 15:20 |
| Ursinha | yes, he is | 15:20 |
| matsubara | mwhudson is indeed working on that | 15:20 |
| flacoste | herb: and well-placed sources also told me that a sprint is being organized to fix those damn issues | 15:21 |
| flacoste | herb: so there is hope! | 15:21 |
| herb | excellent. a fix would be huge for the LOSAs | 15:21 |
| rockstar | herb, mwhudson is on the verge of insanity tracking loggerhead issues down. | 15:21 |
| herb | thanks for all the work on that. it is much appreciated. | 15:22 |
| matsubara | thanks herb | 15:22 |
| matsubara | [TOPIC] * DBA report (stub) | 15:22 |
| MootBot | New Topic: * DBA report (stub) | 15:22 |
| stub | The production dbs seem to be ticking away nicely. | 15:22 |
| stub | Staging db updates are being disabled by the losas for some testing, so expect a drop in timeout OOPSES. | 15:22 |
| stub | I've had some db patches for this cycle come through already, which is great. | 15:22 |
| stub | 3 | 15:24 |
| stub | 2 | 15:24 |
| stub | 1 | 15:24 |
| matsubara | all right. thanks stub | 15:24 |
| flacoste | what about the staging issues? | 15:24 |
| matsubara | herb: ^ | 15:25 |
| flacoste | staging isn't available at the moment | 15:25 |
| herb | it's restoring | 15:25 |
| flacoste | but we didn't understand why it failed? | 15:25 |
| flacoste | or did we? | 15:25 |
| matsubara | herb: how long will it take to restore? | 15:25 |
| flacoste | and it's just that gmb is done with the testing and we can resume normal staging updates? | 15:25 |
| herb | matsubara: should be back up within the next couple of hours. | 15:25 |
| Ursinha | herb, about 10 hours ago I was talking with spm and he was unsuccessful to put staging on again | 15:26 |
| herb | flacoste: upgrade.py failed because there will still connections open to the staging db. This left the DB in an indeterminate state, so we had to go through the full restore process with a new copy of the staging DB. | 15:27 |
| gmb | flacoste: I'm not done yet. | 15:27 |
| gmb | flacoste: I need staging to be up for that | 15:27 |
| flacoste | so this means that we are still screwed? | 15:27 |
| stub | This is all useful information for read only launchpad btw. | 15:28 |
| flacoste | we either need to fix upgrade.py to work without a db restore | 15:28 |
| flacoste | or to turn off staging ugprades for the duration of gmb's test | 15:28 |
| herb | flacoste: upgrade.py works fine | 15:28 |
| herb | and we'll need to turn off upgrades while gmb's testing, per stubs note above. | 15:29 |
| flacoste | why were there still connections open? | 15:29 |
| stub | upgrade.py cannot work when there are active db connections, as upgrade requires exclusive locks on all the replicated db tables. | 15:29 |
| herb | flacoste: the rollout process shuts down the app servers, but there are still potentially cron scripts running, etc. | 15:29 |
| flacoste | why usually is this not a problem then? | 15:30 |
| stub | Because the restore, replicate and upgrade process is done as a fresh db. Once it is finished, it is swapped into place. | 15:31 |
| flacoste | ah right! | 15:31 |
| * bigjools has to run, will catch scrollback later if you need anything from me | 15:31 | |
| flacoste | so when I said fix upgrade.py, i should have said 'fix rollout process' | 15:32 |
| flacoste | so I guess it's best to simply disbale upgrade for now | 15:32 |
| flacoste | i also heard that gmb might do his tests elsewhere | 15:32 |
| flacoste | so that might becomes a moot issue | 15:32 |
| flacoste | but like stub said, very instructive for read-only launchpad | 15:32 |
| gmb | flacoste: Well, that's an embryonic idea right now. I still need a staging / demo machine for the forseeable future. | 15:33 |
| matsubara | I want to make an action item for the fix rollout process thing | 15:34 |
| matsubara | not sure who would be responsible for that | 15:34 |
| matsubara | losas? | 15:34 |
| herb | whoa | 15:35 |
| flacoste | with the help of stub probably | 15:35 |
| matsubara | [action] losas and stub to fix rollout process to avoid the staging restore problems | 15:35 |
| MootBot | ACTION received: losas and stub to fix rollout process to avoid the staging restore problems | 15:35 |
| herb | there isn't anything inherently wrong with the rollout process. | 15:35 |
| herb | but ok | 15:35 |
| matsubara | thanks everyone | 15:36 |
| matsubara | anythign else before I close? | 15:36 |
| flacoste | well, it's currently not reliable if we don't do a DB restore it seems | 15:36 |
| flacoste | nope | 15:36 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. | 15:36 |
| matsubara | #endmeeting | 15:36 |
| MootBot | Meeting finished at 09:36. | 15:36 |
| Ursinha | thanks matsubara | 15:36 |
| === matsubara is now known as matsubara-lunch | ||
| === salgado-lunch is now known as salgado | ||
| === matsubara-lunch is now known as matsubara | ||
| === bac is now known as bac_lunch | ||
| === abentley1 is now known as abentley | ||
| === bac_lunch is now known as bac | ||
| === salgado is now known as salgado-afk | ||
| === mthaddon_ is now known as mthaddon | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!