=== salgado-afk is now known as salgado [16:00] me [16:00] #startmeeting [16:00] Meeting started at 10:00. The chair is matsubara. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] [TOPIC] Roll Call [16:00] New Topic: Roll Call [16:00] me [16:01] who's here today? [16:01] yo [16:01] me [16:01] me [16:01] bigjools, intellectronica, rockstar, Ursinha ? [16:01] meeeee [16:01] me [16:02] hmm I think rockstar is flying to London, isn't he? [16:02] yep [16:02] ok, let's move on without Code and Bugs [16:02] [TOPIC] Agenda [16:02] New Topic: Agenda [16:03] * Next meeting [16:03] * Actions from last meeting [16:03] * Oops report & Critical Bugs [16:03] * Operations report (mthaddon/herb/spm) [16:03] * DBA report (DBA contact) [16:03] * QA Stats of the week [16:03] [topic] next meeting [16:03] New Topic: next meeting [16:03] so, no meeting next week? [16:03] matsubara, will be live! [16:03] or live meeting? [16:03] We'll do it live! [16:03] oh man [16:04] our secret identities will be blown [16:04] forgot the LOSA [16:04] You mean I have to wear pants?? [16:04] herb: around? [16:04] me [16:04] lol [16:04] stub: that's never been necessary for live meetings [16:04] ok, will any of the LOSAs attending the Epic? [16:04] * sinzui will still be in his jim-jams [16:04] matsubara: I don't think the timezone suits them :) [16:04] matsubara: I don't think so, no. [16:04] herb: really? [16:05] danilos: I know I'm not. [16:05] ok, we can have the meeting there then and ask one of the LOSAs to join us [16:05] I'm fairly certain spm isn't. That only leaves mthaddon and he hasn't mentioned anything to me about it. [16:06] i guess he's not going by his reply to mwhudson_ climbing email [16:06] mthaddon isn't coming [16:06] matsubara: right [16:06] we'll sort it out there and let you know herb [16:07] [topic] * Actions from last meeting [16:07] New Topic: * Actions from last meeting [16:07] * Ursinha to keep an eye on OOPSes regarding bug 277129, to confirm that the bug is really fixed. [16:07] Launchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [High,Fix committed] https://launchpad.net/bugs/277129 [16:07] actually [16:07] that fix was only deployed to staging [16:07] yes [16:07] so that watch was meaningless [16:07] yes again [16:07] i can say now that it is fix [16:07] fixed [16:07] but that we have a fall-out for it :-) [16:08] the DoomedTransaction OOPS you saw this morning [16:08] the other two items are for intellectronica [16:08] and that gary is working on [16:09] and another item for me, which i did (matsubara to find or file a new bug for the core dump issue and add info discussed today there, subscribe niemeyer, kiko and someone from ubuntu [16:09] ) [16:09] thanks flacoste [16:09] good progress has been made on that one [16:09] it was in reviewed last time i checked [16:10] great [16:10] I guess I'm not receiving notifications for it [16:11] maybe I forgot to subscribe myself [16:11] anyway, moving on [16:11] i,m not sure there was progress report on the bug [16:11] flacoste: hmm, I'll update it after the meeting then. [16:12] [action] matsubara update core dump bug with progress made last week [16:12] ACTION received: matsubara update core dump bug with progress made last week [16:12] thanks [16:12] [topic] * Oops report & Critical Bugs [16:12] New Topic: * Oops report & Critical Bugs [16:12] flacoste, matsubara what was the bug number for that so I can subscribe myself? [16:12] Ursinha: floor is yours [16:12] herb: let me look [16:12] okay [16:13] 7 bugs, not sure if it'll be pointless but will list all them anyway [16:13] 3 for foundations: bug 284458, 284418, 284403 [16:13] Launchpad bug 284458 in launchpad-foundations "API NotFoundErrors should be converted to NotFound" [Undecided,New] https://launchpad.net/bugs/284458 [16:13] Launchpad bug 284418 in launchpad-foundations "oops reports not displaying the whole traceback" [High,Triaged] https://launchpad.net/bugs/284418 [16:13] Launchpad bug 284403 in launchpad-foundations "DoomedTransaction OOPS in +specworkload page" [High,Triaged] https://launchpad.net/bugs/284403 [16:13] 1 for shipit: 284448, salgado already told that it's only caused when users handcraft URLs, need to set priority on it [16:13] 1 for translations: 284429 [16:13] 1 for soyuz: 264319 [16:13] oops [16:13] bug 284448 [16:13] Launchpad bug 284448 in shipit "Custom 404 page for shipit" [Undecided,New] https://launchpad.net/bugs/284448 [16:13] bug 284429 [16:13] Launchpad bug 284429 in rosetta "NotOneError raised while uploading a translation file" [Undecided,New] https://launchpad.net/bugs/284429 [16:14] bug 264319 [16:14] Launchpad bug 264319 in soyuz "Broken link in breadcrumbs for source packages" [High,Triaged] https://launchpad.net/bugs/264319 [16:14] 1 for bugs: bug 186262 [16:14] Launchpad bug 186262 in malone "+filebug pages time out" [High,Confirmed] https://launchpad.net/bugs/186262 [16:14] I'll take 264319 [16:14] flacoste: can you comment on 284458? [16:14] bigjools, thanks [16:15] Ursinha: we'll look into 284429 [16:15] the other two from foundations are being diagnosed/addressed already [16:15] danilos, thanks [16:15] matsubara: that's a bug for SOyuz [16:15] Ursinha: jtv was looking at it just before he left [16:15] it's not API code [16:15] it's soyuz code invoked through the API [16:15] flacoste: is it just a case of returning a different exception? [16:15] yep [16:15] bigjools: NotFound [16:15] ok, easy [16:16] flacoste: right, I thought the NotFoundError was a generic thing [16:16] I got that one too Ursinha [16:16] not in this case [16:16] all right, thanks flacoste [16:16] bigjools, ok, I'll take not [16:16] bigjools, do you want me to assign it to you? [16:16] Ursinha: I'll do it right now when I target it [16:16] 284418 is something to fix when we deploy [16:16] i assigned it to canonical-losas [16:16] bigjools, ok, thanks [16:16] i think Tom is the one who would fix it [16:17] herb: bug 95822 [16:17] Launchpad bug 95822 in apport "Malone connection generates an "Internal Server Error" on large file attachments" [Medium,Confirmed] https://launchpad.net/bugs/95822 [16:17] matsubara: thanks. [16:17] (we need to remove the .pyc after renaming the directory and before starting the app server) [16:17] gary is handling 284403 [16:18] and that's the fall-out from the fix to the timeout-related exception [16:18] I guess that's all Ursinha ? [16:19] salgado, what about bug 284448? [16:19] Launchpad bug 284448 in shipit "Custom 404 page for shipit" [Undecided,Triaged] https://launchpad.net/bugs/284448 [16:19] I'm not certain about the priority [16:19] Ursinha: salgado already said it's low priority and he's not supposed to attend this meeting :-) [16:19] matsubara, old habit :) I'll set the bug to low [16:20] sinzui, why? [16:20] It interferes with bookmarks. We should be redirecting in the example given. [16:21] The 404 page is needed, but the users who have used shipit.edubuntu should know they can access their information at shipit.ubuntu [16:21] sinzui, can you set the proper priority on that, please? [16:22] I'm not certain how many users the problem really affect to know whether we should be redirecting [16:22] Ursinha: I think seeing the frequency of oopses help me to know the priority [16:22] sinzui, so we should keep it low and see if it happens more often? [16:22] Ursinha: bring it up again if the OOPS count for that one increases too much [16:23] ok, I'll feed the bug with more oopses if they happen [16:23] and bring it here [16:23] there's only the bugs' bug left [16:23] Ursinha: I would definitely make it high if this happens almost every day...but I think in this case the problem is that the user need to get to shipit.ubuntu [16:23] sinzui: so far we had like 3 OOPSes, so doesn't seem to happen often [16:24] intellectronica: around? [16:24] intellectronica, bug 186262 [16:24] matsubara: yes [16:24] intellectronica, hi :) [16:24] Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/186262/+text) [16:24] oh my [16:24] intellectronica, https://launchpad.net/bugs/186262 [16:24] Launchpad bug 186262 in malone "+filebug pages time out" [High,Confirmed] [16:25] ouch [16:25] wow, this is quite old, is it a comeback? [16:26] intellectronica: yeah, looks like. mthaddon tried to file a bug recently and got a timeout [16:26] Is it really High if the bug is old? [16:26] intellectronica, yes, it seems [16:26] I've seen it lately [16:26] the oopses, that is [16:26] Ursinha: ok, assigning to myself and will investigate [16:26] matsubara, I know someone else got it recently too (a non-canonical employee) [16:26] intellectronica, ok, thanks [16:27] nothing else from me, then [16:27] Ursinha: is this something that happens consistently? [16:27] intellectronica, I was able to reproduce [16:27] intellectronica: yes, depending on what you use as the bug summary [16:27] anyway, we can discuss later [16:27] and it's workaroundable [16:27] so, I guess that's why people are not complaining so loudly :-) [16:27] thanks for taking it intellectronica [16:28] we have only 1 critical bug which is fix committed [16:28] no critical bugs, also, besides one fix committed [16:28] actually should be fix released by now [16:29] thanks guys, you can move on matsubara [16:29] thanks everyone [16:29] [topic] * Operations report (mthaddon/herb/spm) [16:29] New Topic: * Operations report (mthaddon/herb/spm) [16:29] * Monday (2008-10-13): Did a cherry pick to lpnet* to update the shipit pages for intrepid. [16:29] * Last night (2008-10-16): Rolled out 2.1.10 [16:29] * Over the last week we've had to restart codebrowse 7 times because it was unresponsive. [16:30] very well executed rollout, btw. Congrats! 21 min downtime only [16:30] I'll look into whether or not there are any bugs filed for the codebrowse issue. if not I'll take care of that after the meething. [16:30] [action] herb to find or file a bug about codebrowse being unresponsive and needing restarts [16:30] ACTION received: herb to find or file a bug about codebrowse being unresponsive and needing restarts [16:30] matsubara: lifeless and mthaddon worked out a change in procedure. really seemed to help [16:31] we've actually had a 21 minutes rollout before (1.2.4), but I definitely agree the new rollout method is much better and quicker) [16:32] herb: indeed. the only downside is that oops reports are less informative now, but flacoste will email you guys about it [16:32] that's done [16:32] cool [16:32] anything else for herb? [16:32] are we having a re-roll? [16:32] I haven't heard anything about that. [16:32] doesn't mean we aren't [16:32] matsubara, Ursinha? [16:32] what's your feeling? [16:33] personnally, candidate for it would only be for the DoomedTransaction problem [16:33] OOPS reports less informative? [16:33] flacoste: it'd be nice to have the oops thing fixed and I guess we don't need to re-roll for that, right? [16:33] matsubara: just restart [16:33] is this because we make build in another directory, or unrelated? [16:33] the other issues so far don't look that critical [16:33] mthaddon: it is, i sent you an email about it [16:33] in any case we'll be checking the oops summaries the whole day, and if anything comes up, we'll let you know [16:34] DoomedTransaction problem is the ghost of the old timeout problem [16:34] so would happen as often as it [16:34] flacoste, ok, see the email here [16:34] we could cherry-pick the fix [16:34] or re-roll [16:34] flacoste, well, so far I guess that only a CP would be enough [16:35] since we are still in r-c [16:35] mode [16:35] i think it's easier for the LOSA to do a re-roll than a CP [16:35] mthaddon, herb: is that right? [16:35] flacoste: the old timeout problem as in bug #244957? [16:35] Bug 244957 on http://launchpad.net/bugs/244957 is private [16:35] it's easier on pqm. :) [16:35] if we re-roll we don't need to run the test suite on the release+patch. [16:35] matsubara: no as in bug 277129 [16:35] Launchpad bug 277129 in launchpad-foundations "OOPS when attempting to render a timeout page" [High,Fix committed] https://launchpad.net/bugs/277129 [16:36] sinzui: ^ [16:36] herb, mthaddon: what time would be good for a re-roll today? [16:37] flacoste, any time is fine - re-roll to where? [16:37] app-servers [16:37] flacoste, yeah, anytime is fine [16:37] After the fix exists might be best though [16:37] :-) [16:37] flacoste, sooner the better, but really no difference [16:37] and what he said :) [16:38] so unless anything else comes up, we can target a re-roll after gary's land his fix [16:38] deal [16:38] which should happen later this afternoon [16:39] so, anything else for LOSAs? [16:39] thanks herb, mthaddon [16:39] moving on [16:39] [topic] * DBA report (DBA contact) [16:39] New Topic: * DBA report (DBA contact) [16:39] Load on the db seems to have decreased a little after the rollout, although it is too early to tell for sure. Hopefully this is a sign of good work and not broken systems :) [16:39] The replicated staging db rebuild work has been done and awaiting review. I don't see any problems getting this landed and switched on Monday, time in London permitting. [16:39] Also in London, we need to find time with Mark to discuss some of the db design that was bounced this cycle (bzr Job system, milestone/productrelease linking). [16:39] fini. [16:40] all right. any questions for stub? [16:40] what are you wearing? [16:40] uhm, wrong channel [16:40] ;) [16:40] heh [16:41] thanks stub [16:41] [topic] * QA Stats of the week [16:41] New Topic: * QA Stats of the week [16:41] :) [16:41] so this is a new section, and I have no stats for this week [16:41] but the idea is to bring up some number about triage and needstesting items from the current week and last week [16:42] kinda show how we're progressing during the cycle [16:42] what you guys think? keep, bag, change? [16:42] sounds nice, not sure how you are going to present it live :) [16:42] maybe using a drawing board? [16:42] danilos, never heard of ascii-art? :) [16:43] that's all from me. [16:43] Ursinha: ascii-art is going to be a long stretch during Epic, though I'd like to see him try :) [16:43] danilos: I'll come up with a way :-) [16:43] or probably postpone it to the next IRC meeting :-) [16:43] ok, thanks [16:43] ok, anything else before I close? [16:44] not from me [16:44] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [16:44] #endmeeting [16:44] Meeting finished at 10:44. [16:44] thank you as well, bye bye :) [16:44] just in time === cprov is now known as cprov-lunch === salgado is now known as salgado-lunch === cprov-lunch is now known as cprov === salgado-lunch is now known as salgado === matsubara is now known as matsubara-lunch === matsubara-lunch is now known as matsubara === thumper_laptop is now known as thumper === thumper_laptop is now known as thumper === salgado is now known as salgado-afk