| === salgado-afk is now known as salgado | ||
| danilos | me | 16:00 |
|---|---|---|
| matsubara | #startmeeting | 16:00 |
| MootBot | Meeting started at 10:00. The chair is matsubara. | 16:00 |
| MootBot | Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] | 16:00 |
| matsubara | Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. | 16:00 |
| matsubara | [TOPIC] Roll Call | 16:00 |
| MootBot | New Topic: Roll Call | 16:00 |
| sinzui | me | 16:00 |
| matsubara | who's here today? | 16:01 |
| stub | yo | 16:01 |
| danilos | me | 16:01 |
| flacoste | me | 16:01 |
| matsubara | bigjools, intellectronica, rockstar, Ursinha ? | 16:01 |
| bigjools | meeeee | 16:01 |
| Ursinha | me | 16:01 |
| matsubara | hmm I think rockstar is flying to London, isn't he? | 16:02 |
| flacoste | yep | 16:02 |
| matsubara | ok, let's move on without Code and Bugs | 16:02 |
| matsubara | [TOPIC] Agenda | 16:02 |
| MootBot | New Topic: Agenda | 16:02 |
| matsubara | * Next meeting | 16:03 |
| matsubara | * Actions from last meeting | 16:03 |
| matsubara | * Oops report & Critical Bugs | 16:03 |
| matsubara | * Operations report (mthaddon/herb/spm) | 16:03 |
| matsubara | * DBA report (DBA contact) | 16:03 |
| matsubara | * QA Stats of the week | 16:03 |
| matsubara | [topic] next meeting | 16:03 |
| MootBot | New Topic: next meeting | 16:03 |
| matsubara | so, no meeting next week? | 16:03 |
| Ursinha | matsubara, will be live! | 16:03 |
| danilos | or live meeting? | 16:03 |
| mpt | We'll do it live! | 16:03 |
| bigjools | oh man | 16:03 |
| bigjools | our secret identities will be blown | 16:04 |
| matsubara | forgot the LOSA | 16:04 |
| stub | You mean I have to wear pants?? | 16:04 |
| matsubara | herb: around? | 16:04 |
| herb | me | 16:04 |
| Ursinha | lol | 16:04 |
| danilos | stub: that's never been necessary for live meetings | 16:04 |
| matsubara | ok, will any of the LOSAs attending the Epic? | 16:04 |
| * sinzui will still be in his jim-jams | 16:04 | |
| danilos | matsubara: I don't think the timezone suits them :) | 16:04 |
| herb | matsubara: I don't think so, no. | 16:04 |
| danilos | herb: really? | 16:04 |
| herb | danilos: I know I'm not. | 16:05 |
| matsubara | ok, we can have the meeting there then and ask one of the LOSAs to join us | 16:05 |
| herb | I'm fairly certain spm isn't. That only leaves mthaddon and he hasn't mentioned anything to me about it. | 16:05 |
| matsubara | i guess he's not going by his reply to mwhudson_ climbing email | 16:06 |
| flacoste | mthaddon isn't coming | 16:06 |
| flacoste | matsubara: right | 16:06 |
| matsubara | we'll sort it out there and let you know herb | 16:06 |
| matsubara | [topic] * Actions from last meeting | 16:07 |
| MootBot | New Topic: * Actions from last meeting | 16:07 |
| matsubara | * Ursinha to keep an eye on OOPSes regarding bug 277129, to confirm that the bug is really fixed. | 16:07 |
| ubottu | Launchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [High,Fix committed] https://launchpad.net/bugs/277129 | 16:07 |
| flacoste | actually | 16:07 |
| flacoste | that fix was only deployed to staging | 16:07 |
| Ursinha | yes | 16:07 |
| flacoste | so that watch was meaningless | 16:07 |
| Ursinha | yes again | 16:07 |
| flacoste | i can say now that it is fix | 16:07 |
| flacoste | fixed | 16:07 |
| flacoste | but that we have a fall-out for it :-) | 16:07 |
| flacoste | the DoomedTransaction OOPS you saw this morning | 16:08 |
| matsubara | the other two items are for intellectronica | 16:08 |
| flacoste | and that gary is working on | 16:08 |
| matsubara | and another item for me, which i did (matsubara to find or file a new bug for the core dump issue and add info discussed today there, subscribe niemeyer, kiko and someone from ubuntu | 16:09 |
| matsubara | ) | 16:09 |
| matsubara | thanks flacoste | 16:09 |
| flacoste | good progress has been made on that one | 16:09 |
| flacoste | it was in reviewed last time i checked | 16:09 |
| matsubara | great | 16:10 |
| matsubara | I guess I'm not receiving notifications for it | 16:10 |
| matsubara | maybe I forgot to subscribe myself | 16:11 |
| matsubara | anyway, moving on | 16:11 |
| flacoste | i,m not sure there was progress report on the bug | 16:11 |
| matsubara | flacoste: hmm, I'll update it after the meeting then. | 16:11 |
| matsubara | [action] matsubara update core dump bug with progress made last week | 16:12 |
| MootBot | ACTION received: matsubara update core dump bug with progress made last week | 16:12 |
| matsubara | thanks | 16:12 |
| matsubara | [topic] * Oops report & Critical Bugs | 16:12 |
| MootBot | New Topic: * Oops report & Critical Bugs | 16:12 |
| herb | flacoste, matsubara what was the bug number for that so I can subscribe myself? | 16:12 |
| matsubara | Ursinha: floor is yours | 16:12 |
| matsubara | herb: let me look | 16:12 |
| Ursinha | okay | 16:12 |
| Ursinha | 7 bugs, not sure if it'll be pointless but will list all them anyway | 16:13 |
| Ursinha | 3 for foundations: bug 284458, 284418, 284403 | 16:13 |
| ubottu | Launchpad bug 284458 in launchpad-foundations "API NotFoundErrors should be converted to NotFound" [Undecided,New] https://launchpad.net/bugs/284458 | 16:13 |
| ubottu | Launchpad bug 284418 in launchpad-foundations "oops reports not displaying the whole traceback" [High,Triaged] https://launchpad.net/bugs/284418 | 16:13 |
| ubottu | Launchpad bug 284403 in launchpad-foundations "DoomedTransaction OOPS in +specworkload page" [High,Triaged] https://launchpad.net/bugs/284403 | 16:13 |
| Ursinha | 1 for shipit: 284448, salgado already told that it's only caused when users handcraft URLs, need to set priority on it | 16:13 |
| Ursinha | 1 for translations: 284429 | 16:13 |
| Ursinha | 1 for soyuz: 264319 | 16:13 |
| Ursinha | oops | 16:13 |
| Ursinha | bug 284448 | 16:13 |
| ubottu | Launchpad bug 284448 in shipit "Custom 404 page for shipit" [Undecided,New] https://launchpad.net/bugs/284448 | 16:13 |
| Ursinha | bug 284429 | 16:13 |
| ubottu | Launchpad bug 284429 in rosetta "NotOneError raised while uploading a translation file" [Undecided,New] https://launchpad.net/bugs/284429 | 16:13 |
| Ursinha | bug 264319 | 16:14 |
| ubottu | Launchpad bug 264319 in soyuz "Broken link in breadcrumbs for source packages" [High,Triaged] https://launchpad.net/bugs/264319 | 16:14 |
| Ursinha | 1 for bugs: bug 186262 | 16:14 |
| ubottu | Launchpad bug 186262 in malone "+filebug pages time out" [High,Confirmed] https://launchpad.net/bugs/186262 | 16:14 |
| bigjools | I'll take 264319 | 16:14 |
| matsubara | flacoste: can you comment on 284458? | 16:14 |
| Ursinha | bigjools, thanks | 16:14 |
| danilos | Ursinha: we'll look into 284429 | 16:15 |
| matsubara | the other two from foundations are being diagnosed/addressed already | 16:15 |
| Ursinha | danilos, thanks | 16:15 |
| flacoste | matsubara: that's a bug for SOyuz | 16:15 |
| danilos | Ursinha: jtv was looking at it just before he left | 16:15 |
| flacoste | it's not API code | 16:15 |
| flacoste | it's soyuz code invoked through the API | 16:15 |
| bigjools | flacoste: is it just a case of returning a different exception? | 16:15 |
| flacoste | yep | 16:15 |
| flacoste | bigjools: NotFound | 16:15 |
| bigjools | ok, easy | 16:15 |
| matsubara | flacoste: right, I thought the NotFoundError was a generic thing | 16:16 |
| bigjools | I got that one too Ursinha | 16:16 |
| flacoste | not in this case | 16:16 |
| matsubara | all right, thanks flacoste | 16:16 |
| Ursinha | bigjools, ok, I'll take not | 16:16 |
| Ursinha | bigjools, do you want me to assign it to you? | 16:16 |
| bigjools | Ursinha: I'll do it right now when I target it | 16:16 |
| flacoste | 284418 is something to fix when we deploy | 16:16 |
| flacoste | i assigned it to canonical-losas | 16:16 |
| Ursinha | bigjools, ok, thanks | 16:16 |
| flacoste | i think Tom is the one who would fix it | 16:16 |
| matsubara | herb: bug 95822 | 16:17 |
| ubottu | Launchpad bug 95822 in apport "Malone connection generates an "Internal Server Error" on large file attachments" [Medium,Confirmed] https://launchpad.net/bugs/95822 | 16:17 |
| herb | matsubara: thanks. | 16:17 |
| flacoste | (we need to remove the .pyc after renaming the directory and before starting the app server) | 16:17 |
| flacoste | gary is handling 284403 | 16:17 |
| flacoste | and that's the fall-out from the fix to the timeout-related exception | 16:18 |
| matsubara | I guess that's all Ursinha ? | 16:18 |
| Ursinha | salgado, what about bug 284448? | 16:19 |
| ubottu | Launchpad bug 284448 in shipit "Custom 404 page for shipit" [Undecided,Triaged] https://launchpad.net/bugs/284448 | 16:19 |
| sinzui | I'm not certain about the priority | 16:19 |
| matsubara | Ursinha: salgado already said it's low priority and he's not supposed to attend this meeting :-) | 16:19 |
| Ursinha | matsubara, old habit :) I'll set the bug to low | 16:19 |
| Ursinha | sinzui, why? | 16:20 |
| sinzui | It interferes with bookmarks. We should be redirecting in the example given. | 16:20 |
| sinzui | The 404 page is needed, but the users who have used shipit.edubuntu should know they can access their information at shipit.ubuntu | 16:21 |
| Ursinha | sinzui, can you set the proper priority on that, please? | 16:21 |
| sinzui | I'm not certain how many users the problem really affect to know whether we should be redirecting | 16:22 |
| sinzui | Ursinha: I think seeing the frequency of oopses help me to know the priority | 16:22 |
| Ursinha | sinzui, so we should keep it low and see if it happens more often? | 16:22 |
| matsubara | Ursinha: bring it up again if the OOPS count for that one increases too much | 16:22 |
| Ursinha | ok, I'll feed the bug with more oopses if they happen | 16:23 |
| Ursinha | and bring it here | 16:23 |
| Ursinha | there's only the bugs' bug left | 16:23 |
| sinzui | Ursinha: I would definitely make it high if this happens almost every day...but I think in this case the problem is that the user need to get to shipit.ubuntu | 16:23 |
| matsubara | sinzui: so far we had like 3 OOPSes, so doesn't seem to happen often | 16:23 |
| matsubara | intellectronica: around? | 16:24 |
| Ursinha | intellectronica, bug 186262 | 16:24 |
| intellectronica | matsubara: yes | 16:24 |
| Ursinha | intellectronica, hi :) | 16:24 |
| ubottu | Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/186262/+text) | 16:24 |
| Ursinha | oh my | 16:24 |
| Ursinha | intellectronica, https://launchpad.net/bugs/186262 | 16:24 |
| ubottu | Launchpad bug 186262 in malone "+filebug pages time out" [High,Confirmed] | 16:24 |
| intellectronica | ouch | 16:25 |
| intellectronica | wow, this is quite old, is it a comeback? | 16:25 |
| matsubara | intellectronica: yeah, looks like. mthaddon tried to file a bug recently and got a timeout | 16:26 |
| sinzui | Is it really High if the bug is old? | 16:26 |
| Ursinha | intellectronica, yes, it seems | 16:26 |
| Ursinha | I've seen it lately | 16:26 |
| Ursinha | the oopses, that is | 16:26 |
| intellectronica | Ursinha: ok, assigning to myself and will investigate | 16:26 |
| mthaddon | matsubara, I know someone else got it recently too (a non-canonical employee) | 16:26 |
| Ursinha | intellectronica, ok, thanks | 16:26 |
| Ursinha | nothing else from me, then | 16:27 |
| intellectronica | Ursinha: is this something that happens consistently? | 16:27 |
| Ursinha | intellectronica, I was able to reproduce | 16:27 |
| matsubara | intellectronica: yes, depending on what you use as the bug summary | 16:27 |
| intellectronica | anyway, we can discuss later | 16:27 |
| matsubara | and it's workaroundable | 16:27 |
| matsubara | so, I guess that's why people are not complaining so loudly :-) | 16:27 |
| matsubara | thanks for taking it intellectronica | 16:27 |
| matsubara | we have only 1 critical bug which is fix committed | 16:28 |
| Ursinha | no critical bugs, also, besides one fix committed | 16:28 |
| matsubara | actually should be fix released by now | 16:28 |
| Ursinha | thanks guys, you can move on matsubara | 16:29 |
| matsubara | thanks everyone | 16:29 |
| matsubara | [topic] * Operations report (mthaddon/herb/spm) | 16:29 |
| MootBot | New Topic: * Operations report (mthaddon/herb/spm) | 16:29 |
| herb | * Monday (2008-10-13): Did a cherry pick to lpnet* to update the shipit pages for intrepid. | 16:29 |
| herb | * Last night (2008-10-16): Rolled out 2.1.10 | 16:29 |
| herb | * Over the last week we've had to restart codebrowse 7 times because it was unresponsive. | 16:29 |
| matsubara | very well executed rollout, btw. Congrats! 21 min downtime only | 16:30 |
| herb | I'll look into whether or not there are any bugs filed for the codebrowse issue. if not I'll take care of that after the meething. | 16:30 |
| matsubara | [action] herb to find or file a bug about codebrowse being unresponsive and needing restarts | 16:30 |
| MootBot | ACTION received: herb to find or file a bug about codebrowse being unresponsive and needing restarts | 16:30 |
| herb | matsubara: lifeless and mthaddon worked out a change in procedure. really seemed to help | 16:30 |
| mthaddon | we've actually had a 21 minutes rollout before (1.2.4), but I definitely agree the new rollout method is much better and quicker) | 16:31 |
| matsubara | herb: indeed. the only downside is that oops reports are less informative now, but flacoste will email you guys about it | 16:32 |
| flacoste | that's done | 16:32 |
| matsubara | cool | 16:32 |
| matsubara | anything else for herb? | 16:32 |
| flacoste | are we having a re-roll? | 16:32 |
| herb | I haven't heard anything about that. | 16:32 |
| herb | doesn't mean we aren't | 16:32 |
| flacoste | matsubara, Ursinha? | 16:32 |
| flacoste | what's your feeling? | 16:32 |
| flacoste | personnally, candidate for it would only be for the DoomedTransaction problem | 16:33 |
| mthaddon | OOPS reports less informative? | 16:33 |
| matsubara | flacoste: it'd be nice to have the oops thing fixed and I guess we don't need to re-roll for that, right? | 16:33 |
| flacoste | matsubara: just restart | 16:33 |
| mthaddon | is this because we make build in another directory, or unrelated? | 16:33 |
| matsubara | the other issues so far don't look that critical | 16:33 |
| flacoste | mthaddon: it is, i sent you an email about it | 16:33 |
| matsubara | in any case we'll be checking the oops summaries the whole day, and if anything comes up, we'll let you know | 16:33 |
| flacoste | DoomedTransaction problem is the ghost of the old timeout problem | 16:34 |
| flacoste | so would happen as often as it | 16:34 |
| mthaddon | flacoste, ok, see the email here | 16:34 |
| flacoste | we could cherry-pick the fix | 16:34 |
| flacoste | or re-roll | 16:34 |
| Ursinha | flacoste, well, so far I guess that only a CP would be enough | 16:34 |
| flacoste | since we are still in r-c | 16:35 |
| flacoste | mode | 16:35 |
| flacoste | i think it's easier for the LOSA to do a re-roll than a CP | 16:35 |
| flacoste | mthaddon, herb: is that right? | 16:35 |
| sinzui | flacoste: the old timeout problem as in bug #244957? | 16:35 |
| ubottu | Bug 244957 on http://launchpad.net/bugs/244957 is private | 16:35 |
| herb | it's easier on pqm. :) | 16:35 |
| herb | if we re-roll we don't need to run the test suite on the release+patch. | 16:35 |
| flacoste | matsubara: no as in bug 277129 | 16:35 |
| ubottu | Launchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [High,Fix committed] https://launchpad.net/bugs/277129 | 16:35 |
| matsubara | sinzui: ^ | 16:36 |
| flacoste | herb, mthaddon: what time would be good for a re-roll today? | 16:36 |
| mthaddon | flacoste, any time is fine - re-roll to where? | 16:37 |
| flacoste | app-servers | 16:37 |
| mthaddon | flacoste, yeah, anytime is fine | 16:37 |
| stub | After the fix exists might be best though | 16:37 |
| flacoste | :-) | 16:37 |
| mthaddon | flacoste, sooner the better, but really no difference | 16:37 |
| mthaddon | and what he said :) | 16:37 |
| flacoste | so unless anything else comes up, we can target a re-roll after gary's land his fix | 16:38 |
| matsubara | deal | 16:38 |
| flacoste | which should happen later this afternoon | 16:38 |
| matsubara | so, anything else for LOSAs? | 16:39 |
| matsubara | thanks herb, mthaddon | 16:39 |
| matsubara | moving on | 16:39 |
| matsubara | [topic] * DBA report (DBA contact) | 16:39 |
| MootBot | New Topic: * DBA report (DBA contact) | 16:39 |
| stub | Load on the db seems to have decreased a little after the rollout, although it is too early to tell for sure. Hopefully this is a sign of good work and not broken systems :) | 16:39 |
| stub | The replicated staging db rebuild work has been done and awaiting review. I don't see any problems getting this landed and switched on Monday, time in London permitting. | 16:39 |
| stub | Also in London, we need to find time with Mark to discuss some of the db design that was bounced this cycle (bzr Job system, milestone/productrelease linking). | 16:39 |
| stub | fini. | 16:39 |
| matsubara | all right. any questions for stub? | 16:40 |
| danilos | what are you wearing? | 16:40 |
| danilos | uhm, wrong channel | 16:40 |
| danilos | ;) | 16:40 |
| matsubara | heh | 16:40 |
| matsubara | thanks stub | 16:41 |
| matsubara | [topic] * QA Stats of the week | 16:41 |
| MootBot | New Topic: * QA Stats of the week | 16:41 |
| Ursinha | :) | 16:41 |
| matsubara | so this is a new section, and I have no stats for this week | 16:41 |
| matsubara | but the idea is to bring up some number about triage and needstesting items from the current week and last week | 16:41 |
| matsubara | kinda show how we're progressing during the cycle | 16:42 |
| matsubara | what you guys think? keep, bag, change? | 16:42 |
| danilos | sounds nice, not sure how you are going to present it live :) | 16:42 |
| danilos | maybe using a drawing board? | 16:42 |
| Ursinha | danilos, never heard of ascii-art? :) | 16:42 |
| matsubara | that's all from me. | 16:43 |
| danilos | Ursinha: ascii-art is going to be a long stretch during Epic, though I'd like to see him try :) | 16:43 |
| matsubara | danilos: I'll come up with a way :-) | 16:43 |
| matsubara | or probably postpone it to the next IRC meeting :-) | 16:43 |
| danilos | ok, thanks | 16:43 |
| matsubara | ok, anything else before I close? | 16:43 |
| Ursinha | not from me | 16:44 |
| matsubara | Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. | 16:44 |
| matsubara | #endmeeting | 16:44 |
| MootBot | Meeting finished at 10:44. | 16:44 |
| danilos | thank you as well, bye bye :) | 16:44 |
| matsubara | just in time | 16:44 |
| === cprov is now known as cprov-lunch | ||
| === salgado is now known as salgado-lunch | ||
| === cprov-lunch is now known as cprov | ||
| === salgado-lunch is now known as salgado | ||
| === matsubara is now known as matsubara-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| === thumper_laptop is now known as thumper | ||
| === thumper_laptop is now known as thumper | ||
| === salgado is now known as salgado-afk | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!