| === afk is now known as mthaddon | ||
| === afk is now known as mthaddon | ||
| === Edwin is now known as Guest18207 | ||
| === Edwin is now known as Guest90684 | ||
| === _thumper_ is now known as thumper | ||
| === mthaddon is now known as afk | ||
| === salgado-afk is now known as salgado | ||
| danilos | Ursinha: ok, ok, I hear you | 16:00 |
|---|---|---|
| Ursinha | #startmeeting | 16:00 |
| Ursinha | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 16:00 |
| Ursinha | [TOPIC] Roll Call | 16:00 |
| Ursinha | Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! | 16:00 |
| Ursinha | me | 16:00 |
| MootBot | Meeting started at 10:00. The chair is Ursinha. | 16:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 16:00 |
| MootBot | New Topic: Roll Call | 16:00 |
| bigjools | me | 16:00 |
| sinzui | me | 16:00 |
| herb | me | 16:00 |
| danilos | me | 16:00 |
| stub | me | 16:00 |
| Ursinha | intellectronica, rockstar, hi | 16:00 |
| intellectronica | me | 16:01 |
| rockstar | me | 16:01 |
| Ursinha | hey! | 16:01 |
| Ursinha | so we're all here | 16:01 |
| Ursinha | apologies from matsubara, he's not here today | 16:01 |
| Ursinha | [TOPIC] Agenda | 16:01 |
| Ursinha | * Actions from last meeting | 16:01 |
| Ursinha | * Oops report & Critical Bugs | 16:01 |
| Ursinha | * Operations report (mthaddon/herb/spm) | 16:01 |
| Ursinha | * DBA report (stub) | 16:01 |
| MootBot | New Topic: Agenda | 16:01 |
| Ursinha | [TOPIC] * Actions from last meeting | 16:02 |
| Ursinha | * matsubara to check with Ursula about last week's meeting action itens - Items were done | 16:02 |
| Ursinha | * flacoste to approve CP requests | 16:02 |
| MootBot | New Topic: * Actions from last meeting | 16:02 |
| Ursinha | my items were okay | 16:02 |
| stub | and apologies from Francis | 16:02 |
| Ursinha | hmm, I missed flacoste | 16:02 |
| Ursinha | stub, so he's not coming today.. | 16:02 |
| Ursinha | I'll have to check with him later then | 16:03 |
| stub | I'm designated victim | 16:03 |
| Ursinha | oh :) | 16:03 |
| Ursinha | stub, so, do you know if he did that? | 16:03 |
| stub | Nope :) | 16:03 |
| === afk is now known as mthaddon | ||
| danilos | there were a bunch of CPs done afaik | 16:03 |
| herb | Ursinha: he did. they've been rolled out. | 16:03 |
| Ursinha | herb, awesome | 16:04 |
| stub | No outstanding CP requests | 16:04 |
| Ursinha | danilos, I'm seeing them in lp list | 16:04 |
| Ursinha | allright | 16:04 |
| Ursinha | thanks guys | 16:04 |
| Ursinha | moving on | 16:04 |
| Ursinha | [TOPIC] * Oops report & Critical Bugs | 16:04 |
| Ursinha | the bugs I had I already discussed with people | 16:04 |
| Ursinha | I don't think I need to mention them here again :) | 16:04 |
| Ursinha | all critical bugs are Fix Committed, so we're okay | 16:04 |
| Ursinha | unless someone have something to say here, let's move on | 16:04 |
| MootBot | New Topic: * Oops report & Critical Bugs | 16:04 |
| danilos | well, there's one that's not | 16:04 |
| Ursinha | danilos, which one? | 16:05 |
| danilos | pytz installation, let me look up the bug number | 16:05 |
| danilos | bug 388825 | 16:05 |
| ubottu | Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825 | 16:05 |
| sinzui | That will fix a bug in the registry | 16:05 |
| danilos | Ursinha: we need help from LOSAs to resolve that, but there's more to it than that | 16:05 |
| danilos | i.e. how do we prevent this from happening | 16:05 |
| Ursinha | danilos, let me look, I checked half an hour ago and it wasn't there :) | 16:06 |
| Ursinha | danilos, hm, I see | 16:06 |
| danilos | Ursinha: stub marked it as critical 4 hours ago :) | 16:06 |
| sinzui | I think bug 155334 will be fixed with the pytz landing | 16:06 |
| ubottu | Launchpad bug 155334 in pytz "In select timezone section, "Kolkata" is shown as "Calcutta"" [Undecided,Fix released] https://launchpad.net/bugs/155334 | 16:06 |
| Ursinha | danilos, so refresh fail :) | 16:06 |
| Ursinha | Undecided and Fix released | 16:07 |
| Ursinha | is that correct? | 16:07 |
| stub | pytz is updated on edge, so production is now seeing timezones it can't cope with. Fail. | 16:07 |
| Ursinha | stub, argh, I understand the problem | 16:07 |
| Ursinha | danilos, so you need a losa to help you with that | 16:08 |
| * sinzui moves Kolkata for a moment | 16:08 | |
| danilos | Ursinha: well, I need losa to try to fix it, we are totally not in power of doing anything about it | 16:08 |
| Ursinha | danilos, okay | 16:08 |
| danilos | Ursinha: this is about date time formatter in TAL which breaks | 16:09 |
| stub | I thought I had assigned it to Tom already? | 16:09 |
| herb | stub, danilos: we can discuss it after the meeting. | 16:09 |
| Ursinha | herb, can you help with that? | 16:09 |
| Ursinha | oh, awesome | 16:09 |
| Ursinha | :) | 16:09 |
| danilos | herb: sure, thanks | 16:09 |
| Ursinha | thanks herb | 16:09 |
| sinzui | stub: danilos Who do I give credit to for fixing pytz? | 16:09 |
| Ursinha | [action] danilos to discuss with herb a fix for bug 388825 | 16:10 |
| danilos | stub: can you join us in that discussion as well? (you seem to know more about it than I do :) | 16:10 |
| MootBot | ACTION received: danilos to discuss with herb a fix for bug 388825 | 16:10 |
| ubottu | Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825 | 16:10 |
| herb | danilos, stub: I think we can keep it pretty short so stub can get offline for a few hours. | 16:10 |
| danilos | herb: sure | 16:11 |
| Ursinha | okay | 16:11 |
| Ursinha | anything else here? | 16:11 |
| Ursinha | moving on then | 16:12 |
| Ursinha | [TOPIC] * Operations report (mthaddon/herb/spm) | 16:12 |
| MootBot | New Topic: * Operations report (mthaddon/herb/spm) | 16:12 |
| herb | 2009-06-12: Cherry picked r8488, r8508 and r8527 to most (all?) of soyuz. | 16:12 |
| herb | 2009-06-16: Cherry picked r8608 to lpnet* and the librarian. | 16:12 |
| herb | 2009-06-17: Cherry picked r8644, r8647 and bzr 1.16rc1 to just about everything. I think we only skipped soyuz. | 16:12 |
| herb | Since the CP yesterday we've had to restart codebrowse a couple of times. Apparently it's taking a while to restart (10 minutes) because it's trying to fetch wadllib and timing out before starting. I don't have much more detail than that unfortunately. | 16:12 |
| Ursinha | rockstar, are you aware of that? | 16:13 |
| rockstar | Ursinha, I indeed was, although not involved, so cannot provide support. | 16:13 |
| Ursinha | rockstar, okay, can you ask who could help then? :) | 16:14 |
| Ursinha | herb, hmm, I guess we don't have a bug for that, do we? | 16:15 |
| herb | Ursinha: I don't know. I saw it in scrollback when I started this morning. I haven't experienced the problem and haven't had a chance to check with spm or mthaddon to figure out if a bug was filed. | 16:16 |
| rockstar | Ursinha, I think everyone knows what's going on. spm, jml, and mwhudson are quite the think tank. | 16:16 |
| rockstar | Everyone == "the people who really can do something about it" | 16:16 |
| Ursinha | rockstar, ok, I'll talk to matsubara and make him aware of that, as he'll be doing the late shift today | 16:16 |
| Ursinha | thanks rockstar and herb | 16:17 |
| mthaddon | yeah, I'm not aware of a bug report - happened after my EOD so I heard a little about it but not the full details | 16:17 |
| Ursinha | mthaddon, I'll catch up with spm later, thanks! | 16:17 |
| Ursinha | anything else for herb | 16:18 |
| Ursinha | ? | 16:18 |
| Ursinha | [action] Ursinha to talk to matsubara and spm later about the codebrowse taking 10 mins to restart | 16:19 |
| MootBot | ACTION received: Ursinha to talk to matsubara and spm later about the codebrowse taking 10 mins to restart | 16:19 |
| Ursinha | moving on then | 16:19 |
| Ursinha | [TOPIC] * DBA report (stub) | 16:19 |
| MootBot | New Topic: * DBA report (stub) | 16:19 |
| danilos | yeah, entire LOSA team was a great help for QA last week, my thanks :) | 16:19 |
| stub | Simulated replication lag has been turned off on staging as it was triggering the lag detection we use to make batch jobs play nicely, blocking them from running. Replication settings on staging now match production on the Launchpad trunk. I can reset things manually again if further testing is needed and there is difficulty updating the staging code base at the moment. | 16:19 |
| stub | The Storm update has landed, and been reverted. There are issues with buildout and buildbot. Perhaps buildout is running a script without PYTHONPATH set correctly, perhaps it is a catch-22 where the environment setup process invokes scripts that require Storm which isn't found because the environment isn't setup yet. Bug #388825 reassigned to Francis as I think he is the only one around at the moment with access to investigate further? | 16:19 |
| stub | autovacuum settings have been made less agressive on production and staging. Its still more agressive than the default. I'll keep an eyeball on it to see if it needs tweaking up or down further. | 16:19 |
| ubottu | Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825 | 16:19 |
| stub | The autovacuum change was prompted by seeing hung autovacuum jobs on staging (again) and reports of testrun failures from possibly hung vacuum processes (again). Vacuum is much nicer under PG 8.4, so I suspect we will live with it or work around it for a few months as getting anything useful for upstream will be a pain. | 16:19 |
| stub | Erm... wrong bug. | 16:19 |
| Ursinha | :) | 16:19 |
| stub | Bug #352965 | 16:20 |
| ubottu | Launchpad bug 352965 in launchpad-foundations "Update Storm" [Medium,In progress] https://launchpad.net/bugs/352965 | 16:20 |
| stub | PG 8.4 RC1 is out now, yay. | 16:20 |
| danilos | stub: are we doing any testing with PG 8.4? is it worth starting it? | 16:21 |
| stub | Not much point until release - we only do pretty standard stuff, so unless we run it on production we are unlikely to find anything other will not find too. | 16:22 |
| * herb suspects it's unlikely we'll move to 8.4 until the next LTS, unless it's going to be backported to hardy. | 16:22 | |
| herb | of course I can be outvoted on that. | 16:23 |
| stub | It needs to be backported to hardy or 8.3 supported under LTS anyway or upgrades will be difficult. | 16:24 |
| stub | Anyway - no need to make that call yet. | 16:24 |
| danilos | yeah, let's move on; Ursinha? :) | 16:25 |
| Ursinha | danilos, I'm seeing people discuss about that | 16:25 |
| Ursinha | because it's important, but as stub said, no need to make that call yet | 16:25 |
| Ursinha | moving on then, per danilos request :) | 16:25 |
| danilos | perhaps another question first :) | 16:26 |
| Ursinha | next topic is the end of this meeting | 16:26 |
| danilos | stub: what's going to happen with storm upgrade? post 2.2.6 or will you try to figure it out? | 16:26 |
| stub | It is out of my hands - needs people with buildbot access to investigate further. | 16:27 |
| danilos | stub: ok, thanks | 16:27 |
| stub | The branch is ready and waiting to land (again) | 16:27 |
| Ursinha | stub, who could help you with that? | 16:27 |
| danilos | stub: perhaps we can test it on staging today even if it's not landed? | 16:27 |
| Ursinha | testing, that is | 16:28 |
| danilos | stub: I'd hate to have a change that big land (I am referring to storm upgrade itself) without proper QA for 2.2.6 | 16:29 |
| Ursinha | +1 | 16:29 |
| stub | Ursinha: I've assigned the bug to Francis. As far as I'm aware, only Francis and Gary have the required buildbot access to look further. If other people also have access and the debugging skills, they can look at it too. | 16:29 |
| stub | There is no reason we can't test it on staging if we want. | 16:30 |
| Ursinha | stub, good. | 16:30 |
| stub | Someone can make another attempt at switching from buildout back to keeping storm in sourcecode if they want - I just tried and failed, but it is new to me too. | 16:30 |
| Ursinha | stub, I see, I'll talk to flacoste about that later today | 16:31 |
| Ursinha | anything else here? | 16:32 |
| Ursinha | [action] talk to flacoste about buildbot and storm updating for testing when he's available today | 16:33 |
| MootBot | ACTION received: talk to flacoste about buildbot and storm updating for testing when he's available today | 16:33 |
| Ursinha | okay | 16:33 |
| Ursinha | let's finish this | 16:33 |
| Ursinha | Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. | 16:33 |
| Ursinha | #endmeeting | 16:33 |
| MootBot | Meeting finished at 10:33. | 16:33 |
| Ursinha | thanks guys | 16:33 |
| danilos | Ursinha, all, thanks | 16:33 |
| herb | thanks Ursinha | 16:33 |
| === salgado is now known as salgado-lunch | ||
| === bac` is now known as bac | ||
| === salgado-lunch is now known as salgado | ||
| === mwhudson_ is now known as mwhudson | ||
| === Edwin is now known as Guest48257 | ||
| === mthaddon is now known as afk | ||
| === afk is now known as mthaddon | ||
| === salgado is now known as salgado-afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!