| === salgado-afk is now known as salgado | ||
| === ursula is now known as Ursinha | ||
| matsubara | #startmeeting | 15:00 |
|---|---|---|
| MootBot | Meeting started at 10:00. The chair is matsubara. | 15:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 15:00 |
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 15:00 |
| matsubara | [TOPIC] Roll Call | 15:00 |
| MootBot | New Topic: Roll Call | 15:00 |
| rockstar | me | 15:00 |
| danilos | me | 15:00 |
| Ursinha | me | 15:00 |
| stub | me | 15:01 |
| gary_poster | me | 15:01 |
| gary_poster | (for flacoste) | 15:01 |
| matsubara | hi gary_poster | 15:01 |
| gary_poster | hi :-) | 15:01 |
| matsubara | herb, | 15:01 |
| herb | me | 15:01 |
| matsubara | bigjools, hi | 15:01 |
| matsubara | sinzui, hi | 15:02 |
| sinzui | me | 15:02 |
| matsubara | ok, apologies from the Bugs team, as they're sprinting | 15:02 |
| matsubara | eveyrone else is here. | 15:02 |
| bigjools | me | 15:02 |
| matsubara | [TOPIC] Agenda | 15:02 |
| MootBot | New Topic: Agenda | 15:02 |
| matsubara | * Actions from last meeting | 15:02 |
| matsubara | * Oops report & Critical Bugs | 15:02 |
| matsubara | * Operations report (mthaddon/herb/spm) | 15:02 |
| matsubara | * DBA report (stub) | 15:02 |
| matsubara | [TOPIC] * Actions from last meeting | 15:02 |
| MootBot | New Topic: * Actions from last meeting | 15:02 |
| matsubara | * Ursinha to file a bug about OOPS-1159EA69 and coordinate a fix with rockstar | 15:03 |
| matsubara | * https://bugs.edge.launchpad.net/launchpad-bazaar/+bug/345415 | 15:03 |
| matsubara | * Ursinha to file a bug about OOPS-1166F631 and assign to intellectronica | 15:03 |
| matsubara | * https://launchpad.net/bugs/341766 | 15:03 |
| matsubara | * sinzui to request CP for bugs 341085 and 341117 | 15:03 |
| ubottu | https://devpad.canonical.com/~jamesh/oops.cgi/1166F631 | 15:03 |
| ubottu | Ubuntu bug 345415 in launchpad-bazaar "NoSuchDistroSeries OOPS in resolve_lp_path" [Undecided,New] | 15:03 |
| ubottu | Bug 341085 on http://launchpad.net/bugs/341085 is private | 15:03 |
| ubottu | Ubuntu bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Undecided,Won't fix] | 15:03 |
| matsubara | ok, Ursinha did her stuff | 15:03 |
| * sinzui hangs head in shame | 15:03 | |
| matsubara | sinzui, did you request those CPs? | 15:03 |
| sinzui | matsubara: I will do that within the hour | 15:03 |
| matsubara | thanks sinzui | 15:03 |
| matsubara | [TOPIC] * Oops report & Critical Bugs | 15:04 |
| MootBot | New Topic: * Oops report & Critical Bugs | 15:04 |
| matsubara | Ursinha, go ahead | 15:04 |
| Ursinha | have only one bug, sinzui: bug 340058 | 15:04 |
| ubottu | Launchpad bug 340058 in launchpad-registry "Sometimes people/team pages don't load" [Undecided,New] https://launchpad.net/bugs/340058 | 15:04 |
| Ursinha | I'm experiencing that sometimes, it's annoying | 15:04 |
| Ursinha | the other bug I have is for bugs: [action] talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 | 15:05 |
| ubottu | Launchpad bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Undecided,Won't fix] https://launchpad.net/bugs/341766 | 15:06 |
| sinzui | Ursinha: do you browse with maps on or off? | 15:06 |
| matsubara | [action] talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 | 15:06 |
| MootBot | ACTION received: talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 | 15:06 |
| Ursinha | sinzui: on | 15:06 |
| Ursinha | sinzui: but it's random | 15:06 |
| sinzui | Ursinha: I have experienced that once when I discovered I had bad connectivity to google. | 15:06 |
| Ursinha | sometimes it happens | 15:06 |
| Ursinha | hmmm | 15:06 |
| matsubara | I couldn't reproduce that one | 15:07 |
| danilos | sinzui, Ursinha: that should be easy to test, if it means no connectivity to google | 15:07 |
| sinzui | well it was half a day. I wrote hide google maps to keep working | 15:07 |
| matsubara | I wonder if it might be an firefox extension thing | 15:07 |
| danilos | just adding 127.0.0.1 whatever-google.com to /etc/hosts could emulate it :) | 15:08 |
| gary_poster | heh, good idea :-) | 15:08 |
| matsubara | and try it with a clean firefox profile | 15:08 |
| Ursinha | sinzui: I can do this testing and report on the bug | 15:09 |
| matsubara | [action] ursinha to debug 340058 and results back to the report | 15:09 |
| MootBot | ACTION received: ursinha to debug 340058 and results back to the report | 15:09 |
| Ursinha | awesome | 15:09 |
| Ursinha | no critical bugs | 15:10 |
| Ursinha | so that's it | 15:10 |
| sinzui | Ursinha: thanks. please do. There is not enough information for me to say this is a bug | 15:10 |
| matsubara | ok, thanks Ursinha, sinzui | 15:10 |
| matsubara | moving on | 15:10 |
| matsubara | [TOPIC] * Operations report (mthaddon/herb/spm) | 15:10 |
| MootBot | New Topic: * Operations report (mthaddon/herb/spm) | 15:10 |
| herb | 2009-03-13 - Cherry picked r7939 to the codehosting server as a fix for bug #260171. | 15:10 |
| ubottu | Bug 260171 on http://launchpad.net/bugs/260171 is private | 15:10 |
| herb | 2009-03-14 - The librarian box hung and didn't come back cleanly when power cycled. The librarian service was unavailable for about an hour and a half. | 15:10 |
| herb | 2009-03-17 - Cherry picked r7942 to the scripts server as a fix for bug #337658. | 15:10 |
| ubottu | Launchpad bug 337658 in rosetta "Disable success notifications for published uploads" [High,Fix released] https://launchpad.net/bugs/337658 | 15:10 |
| herb | Bug #156453 and bug #118625 continue to be a problem for us. | 15:10 |
| ubottu | Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453 | 15:10 |
| herb | Bug #45419 continues to be a pain point for the LOSAs. | 15:10 |
| ubottu | Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625 | 15:11 |
| ubottu | Launchpad bug 45419 in launchpad-foundations "Launchpad needs a way of easily flagging spam" [High,Triaged] https://launchpad.net/bugs/45419 | 15:11 |
| herb | There are currently 2 requested cherry picks pending approval/rejection, and have been that way for more than a week. Can we get those approved or rejected? | 15:11 |
| danilos | I'd also mention that staging is under a special regime, for anyone who hasn't seen my email | 15:11 |
| herb | danilos: good point. thanks | 15:12 |
| gary_poster | Is it appropriate to ask now about the status of [rt.admin.canonical.com #32927] launchpad buildbot master in data center ? | 15:12 |
| mthaddon | gary_poster: it's in the "setting up the EC2 account" stage | 15:12 |
| gary_poster | mthaddon: cool, thanks | 15:12 |
| herb | gary_poster: you can ask. I don't have any input for you though. I'll can look into it for you. | 15:12 |
| danilos | matsubara: who's the soyuz QA contact? | 15:12 |
| matsubara | bigjools, ^ | 15:12 |
| * bigjools waves | 15:12 | |
| gary_poster | herb: cool | 15:12 |
| herb | or mthaddon can tell you where we stand. :p | 15:12 |
| gary_poster | :-) | 15:13 |
| danilos | bigjools: the two cherrypick requests are from cprov, just fyi :) | 15:13 |
| === salgado is now known as salgado-afk | ||
| matsubara | bigjools, kiko/flacoste needs to approve those. can you sort it out with one of them? | 15:14 |
| bigjools | matsubara: talk to me after the meeting about this please | 15:15 |
| herb | I think that's it from us unless there are questions. | 15:15 |
| matsubara | [action] bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-) | 15:16 |
| MootBot | ACTION received: bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-) | 15:16 |
| matsubara | bigjools, sure | 15:16 |
| matsubara | rockstar, do you have any news about the LH thing? | 15:17 |
| rockstar | matsubara, nothing definitive. | 15:17 |
| matsubara | rockstar, still chasing the root cause? | 15:17 |
| rockstar | matsubara, basically. It's voodoo at this point. | 15:18 |
| matsubara | :-( | 15:18 |
| matsubara | well, nothing much we can do here then... | 15:18 |
| matsubara | let's move on. thanks herb and everyone else | 15:18 |
| matsubara | [TOPIC] * DBA report (stub) | 15:19 |
| MootBot | New Topic: * DBA report (stub) | 15:19 |
| stub | The new slave is online and ticking away happily. Its lonely though - LOSAs should configure some of the app servers to talk to it soon. Batch jobs should consider making use of slave databases as we now have spare capacity there. | 15:19 |
| stub | It will be interesting to see how the new 16 core box compares to the existing 8 core boxes. Not many people get to play with DB setups in the same league, so we are in somewhat uncharted territory. Previous anecdotal evidence had the twin core Opterons outperforming the quad core Xeons, but that was a while ago and it is a moving target. | 15:19 |
| stub | We still need a real load balancer to distribute connections to the slaves. What we have is fine though until the batch jobs start making heavy use of the slaves. | 15:19 |
| stub | oot. | 15:19 |
| matsubara | all right. questions for stub? | 15:20 |
| matsubara | thanks stub | 15:20 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. | 15:21 |
| matsubara | #endmeeting | 15:21 |
| MootBot | Meeting finished at 10:21. | 15:21 |
| Ursinha | thanks matsubara and MootBot | 15:21 |
| gary_poster | thanks | 15:21 |
| rockstar | matsubara, I had a point to bring up about the format. | 15:21 |
| matsubara | rockstar, sorry, what format? | 15:21 |
| rockstar | Oops Report / Critical Bugs should be condensed down to Critical bug status | 15:22 |
| mthaddon | rockstar: what do you mean by "it's voodoo at this point"? is it not reproducible locally (setup an instance with a large branch history and browse it)? | 15:22 |
| danilos | matsubara: thanks | 15:22 |
| rockstar | Often, when OOPS reports are referenced, I haven't seen them before, and so I'm not prepared really to make a full comment. | 15:22 |
| rockstar | mthaddon, I mean "We shook this part of loggerhead, and fixed its performance. Did it work? Do I need to make a sacrifice to Apollo?" | 15:23 |
| mthaddon | rockstar: but presumably you're still able to reproduce the problem locally? | 15:23 |
| matsubara | rockstar, right. I asked Ursinha to bring up only bugs instead oopses, that way we keep the discussion in the report | 15:24 |
| rockstar | matsubara, also, we should know about critical bugs before the meeting, so that we have something to report. | 15:24 |
| matsubara | and when we deal with newly filed bug reports re: unseen oopses, the meeting section would be a topic to remind you to investigate that report further | 15:24 |
| rockstar | mthaddon, nominally. We can see the memory climbing, but we don't really get the "hang" which I'm presuming is a separate bug. | 15:25 |
| rockstar | mthaddon, the "hang" is probably specific to the way Launchpad serves branches, so we'll probably need to do some sleuthing there. | 15:25 |
| mthaddon | rockstar: so it continues to respond quickly for you? The "hang" is basically just "it takes more than 20 seconds to respond and our proxy connection times out | 15:25 |
| mthaddon | s/out/out"/ | 15:26 |
| matsubara | rockstar, we kinda expect you to know about critical bugs under your teams responsibility | 15:26 |
| rockstar | mthaddon, but mwhudson has landed quite a few branches to make loggerhead load only what's used when it's used. | 15:26 |
| mthaddon | rockstar: I'll chat with mwhudson when he comes online later - I haven't asked him specifically about what he can and can't reproduce - thx | 15:27 |
| matsubara | rockstar, so basically, what I'm saying is that the QA contact should check the critical bugs for their team before the meeting. | 15:27 |
| rockstar | matsubara, except that this is the first thing I do in the morning. | 15:28 |
| rockstar | :/ | 15:28 |
| matsubara | rockstar, hmm that's a fair point. | 15:28 |
| rockstar | matsubara, but basically, I think it should be treated more like an action | 15:28 |
| rockstar | "rockstar to report on critical bug 123456" | 15:29 |
| ubottu | Launchpad bug 123456 in xine-lib "podcast crashes amarok" [Undecided,Fix released] https://launchpad.net/bugs/123456 | 15:29 |
| matsubara | I'd suggest check those criticals the day before then | 15:29 |
| rockstar | ubottu, sofa king we todd did | 15:29 |
| ubottu | Error: I am only a bot, please don't think I'm intelligent :) | 15:29 |
| matsubara | rockstar, and for new things that you're not prepared for, I think what you're suggesting is perfectly fine | 15:29 |
| rockstar | matsubara, okay, great. | 15:29 |
| matsubara | rockstar, thanks for the suggestions. | 15:30 |
| === matsubara is now known as matsubara-lunch | ||
| === salgado-afk is now known as salgado | ||
| === salgado is now known as salgado-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| === salgado-lunch is now known as salgado | ||
| === salgado is now known as salgado-afk | ||
| === ursula is now known as Ursinha | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!