=== salgado-afk is now known as salgado === EdwinGrubbs_ is now known as EdwinGrubbs [16:00] #startmeeting [16:00] Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. [16:00] Meeting started at 10:00. The chair is Ursinha. [16:00] Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE] [16:00] [TOPIC] Roll Call [16:00] Apologies from matsubara, he's on leave today [16:00] New Topic: Roll Call [16:01] me [16:01] me [16:01] me [16:01] me [16:01] of course [16:01] rockstar, hi [16:01] me [16:01] me [16:01] ni! [16:01] hahaha [16:01] henninge, hi! [16:01] hi Ursinha! [16:01] mi [16:01] moo [16:02] so we're all here [16:02] [TOPIC] Agenda [16:02] * Actions from last meeting [16:02] * Oops report & Critical Bugs & Broken scripts [16:02] * Operations report (mthaddon/herb/spm) [16:02] * DBA report (stub) [16:02] New Topic: Agenda [16:02] [TOPIC] * Actions from last meeting [16:02] all items are okay, moving on [16:02] New Topic: * Actions from last meeting [16:03] [TOPIC] * Oops report & Critical Bugs & Broken scripts [16:03] we're also now going to talk about the scripts that are broken in this section [16:03] we want to make sure that these errors are tracked and solved [16:03] New Topic: * Oops report & Critical Bugs & Broken scripts [16:03] I have five bugs, one critical bug and two broken scripts [16:03] * OOPS report [16:03] sinzui, I have bug 396585 and bug 396383, maybe they can be fixed together? [16:03] flacoste, bug 397224 and bug 373837, this last one is targeted to 2.2.6 [16:03] since, we had some DisconnectionError oopses yesterday, I decided to mention the bug here [16:03] rockstar, could you set the importance on bug 397220, please? [16:03] Launchpad bug 396585 in launchpad-registry "/projects/+review-licenses OOPSes with subscription date" [Low,Triaged] https://launchpad.net/bugs/396585 [16:03] Launchpad bug 396383 in launchpad-foundations "date widget could be smarter, giving a date in format DDMMYYYY OOPSes" [Low,Triaged] https://launchpad.net/bugs/396383 [16:03] Launchpad bug 397224 in launchpad-foundations "TraversalError in person page accessing openid_identity_url" [Undecided,New] https://launchpad.net/bugs/397224 [16:03] Launchpad bug 373837 in launchpad-foundations "DisconnectionError should log a soft OOPS" [High,Triaged] https://launchpad.net/bugs/373837 [16:03] Launchpad bug 397220 in launchpad-code "Better validation for project field in new code import form" [Undecided,Triaged] https://launchpad.net/bugs/397220 [16:04] thanks ubottu [16:05] the ones for sinzui matsubara asked me to mention [16:05] Ursinha: I am not sure about either of these [16:06] Ursinha: the issue is somewhat deep. A foundational widget and the fact that we stupidly design schemes to use datetime stamps that no human will ever enter [16:07] Ursinha: the DisconnectionError soft oops, i'll move to 2.2.7 and see if we have time to fix it, the other one, i don't understand [16:07] Ursinha: i'll have a look at it and triage [16:07] Ursinha: there are a few bugs open about the datetime stamps. I think all of those should be worked on. It is database work. I do not want to engage in that level of work for the next 3 months [16:07] sinzui, right [16:07] flacoste, thanks [16:08] [action] flacoste to take a look and triage bug 373837 [16:08] Launchpad bug 373837 in launchpad-foundations "DisconnectionError should log a soft OOPS" [High,Triaged] https://launchpad.net/bugs/373837 [16:08] ACTION received: flacoste to take a look and triage bug 373837 [16:08] Ursinha: The Launchpad date widget could be smarter. (which is why I changed the title) Note that the page that error came from has a date picker and states the expected format [16:09] sinzui, indeed [16:09] flacoste, and about the other one, the TraversalError one? [16:09] Ursinha: that's the one i'm going to look and triage [16:10] flacoste, oh, I thought that would be the other one [16:10] Ursinha: the other one, I'm moving milestone and we'll see if we have time to fix it [16:10] Ursinha: so I think the oops is rightly classified as low. I'll look into fixing the date widget for another bug though, There are use cases where the default is the current date. The widget should show that [16:10] Ursinha: any particular reason to bring it here? [16:10] [action] flacoste to take a look and triage bug 397224 and will move bug 373837 to 2.2.7 and see if it's possible to fix [16:10] ACTION received: flacoste to take a look and triage bug 397224 and will move bug 373837 to 2.2.7 and see if it's possible to fix [16:10] Launchpad bug 397224 in launchpad-foundations "TraversalError in person page accessing openid_identity_url" [Undecided,New] https://launchpad.net/bugs/397224 [16:10] Launchpad bug 373837 in launchpad-foundations "DisconnectionError should log a soft OOPS" [High,Triaged] https://launchpad.net/bugs/373837 [16:11] flacoste, we had some DisconnectionErrors yesterday and I saw this bug was targeted to an old milestone [16:11] well, that's unrelated :-) [16:11] and it's a high bug [16:11] well, foundations have a lot of High bugs unfortunately :-( [16:12] flacoste, I know :( [16:12] you need a High High :) [16:12] it's called Critical [16:12] no, it's somewhere between! [16:12] no, no [16:12] rockstar, hi :) can you set the importance in that bug, please? [16:12] bigjools: there is critical, assigned High bugs and unassigned High bugs [16:12] I agree with flacoste [16:13] Ursinha: so the DisconnetionError the 373837 bug are talking about are the ojnes we recover from [16:13] Ursinha: when you see them on the OOPS report, we didn't recover from it [16:13] flacoste, hm, I see [16:14] the ones we didn't recover, do we know why? [16:14] OOPS? [16:15] (i didn't look) [16:15] flacoste, fetching one [16:15] flacoste, OOPS-1285J112 [16:15] https://lp-oops.canonical.com/oops.py/?oopsid=1285J112 [16:16] * Ursinha pokes the hell out of rockstar [16:17] Ursinha, sorry. [16:17] Ursinha, I had done it when you asked. [16:17] sinzui, I'll make notes in the bugs then (if you didn't already - you're too fast :)) [16:17] rockstar, thanks man [16:17] Ursinha: that disconnection error is weird [16:17] Ursinha: please file a bug about it [16:17] don't go away, I have one critical bug for you [16:17] flacoste, I will [16:18] [action] Ursinha to file a bug about OOPS-1285J112 and tell flacoste [16:18] https://lp-oops.canonical.com/oops.py/?oopsid=1285J112 [16:18] ACTION received: Ursinha to file a bug about OOPS-1285J112 and tell flacoste [16:18] https://lp-oops.canonical.com/oops.py/?oopsid=1285J112 [16:18] right [16:18] now I have one critical bug, that's code's [16:18] Ursinha, is it the ACF one? [16:18] * Critical bugs [16:18] rockstar, bug 390563 is critical but has no activity since 06-29, is it really critical [16:18] tried to ask thumper yesterday about it but he was already gone, could you check this one with your team, please? [16:18] Launchpad bug 390563 in bzr "absent factory exception from smart server when streaming 2a stacked branches" [Critical,In progress] https://launchpad.net/bugs/390563 [16:19] I guess so [16:19] * Ursinha looks [16:19] after guessing what ACF means :) [16:19] Ursinha, flacoste: the datetime widget bug (while real) is not the bug in the oops as I read it. The restful marshaller expects a string, but the JSON serialized the numbers as numbers. [16:19] Ursinha, there have been multiple team discussions about it. [16:19] rockstar, is that bug really critical? [16:20] ^ We should probably add that as an expected exception to let standard error messaging handle the issue [16:21] sinzui, you may want to comment in the bug about it [16:21] Ursinha, so, yes, it's really critical, but we're dependent on bzr fixing it. [16:22] Ursinha, we may downgrade it for the sake of not misusing Critical. We're still discussing it. [16:22] rockstar, I'm going to write this in the bug, so anyone that reads the bug report will know what's going on [16:23] rockstar, can you update the bug as soon as you have news, pretty please? [16:23] Ursinha, will do. [16:23] thanks a lot rockstar [16:27] sinzui, do you want me to do something in that bug? [16:27] or maybe you want to discuss later [16:27] I just updated it. It is an easy fix that I will make today since I have the file open [16:27] sinzui, great, thanks a lot [16:28] * sinzui just needs to workout where the bug really belongs? launchpad-foundation or lazr.restful [16:28] leave it in foundations for now [16:29] let me just introduce the broken scripts we have, one seems to be yours sinzui [16:29] sinzui, productreleasefinder is having issues for quite some time, it's bug 394391 [16:29] could you take a look at this one? maybe change the importance [16:29] Launchpad bug 394391 in launchpad-registry "product-release-finder should make sane milestone names" [Low,Triaged] https://launchpad.net/bugs/394391 [16:29] also, I see that language-pack-exporter also failed to run, herb, henninge, do you know if that was expected? [16:30] I don't. [16:30] Ursinha: I know how to fix that, and 3 other bugs. I am working on that on my own time [16:30] I don't know either. [16:30] Ursinha: but it is a pretty long running script. [16:30] Ursinha: which run was it? when? [16:31] Ursinha: I'm not certain it is not more important since it have been broken for 4 years and no one reported the bug before me [16:31] It is not uncommon for the script to be interrupted. [16:31] henninge, quoting the email: between 2009-07-07 17:00:04 and 2009-07-08 13:00:04 [16:32] Ursinha: but in the big picture, fixing 99% of all product-release-finder errors is a big win for me so I have started a plan to fix the bugs [16:32] sinzui, I see [16:32] * henninge just realizes he doesn't know the current schedule for that script. [16:33] sinzui, well, I think the important is that it's being fixed, not abandoned [16:34] that's what I'm trying to achieve mentioning them [16:34] Ursinha: My team closed more than 40 low bugs last release. This is not forgotten there is a 30 point story attached to these PRF bugs [16:34] Ursinha: But really that script is monitored by the Ubuntu guys and they complain if they miss a language pack ... [16:34] gee [16:34] ArneGoetje to be exact. [16:35] henninge, monitored or maintained? [16:35] l-p-e runs every monday,tuesday,thursday friday [16:35] @2200 BST [16:35] mon,fri for jaunty [16:35] tue for intrepid [16:35] herb: ah, was gonna ask that [16:35] thur for hardy [16:36] herb, thanks for the info [16:36] * Ursinha took notes [16:36] Ursinha: so that is that was the intrepid run failing. Not critical. [16:36] the monday run finished at 0310 UTC on Tuesday [16:36] sinzui, thanks for that [16:36] henninge, right. if that keeps happening may I poke you? :) [16:37] and it does look like intrepid was the one the failed [16:37] Ursinha: you may :) [16:37] but it ran successfully the previous week. [16:37] * henninge looks forward to being poked by Ursinha because it will definietly happen ... [16:38] [action] Ursinha to watch the lang-pack-exporter script and poke henninge if it fails again [16:38] ACTION received: Ursinha to watch the lang-pack-exporter script and poke henninge if it fails again [16:38] well, let's move on [16:38] thank you all guys [16:38] [TOPIC] * Operations report (mthaddon/herb/spm) [16:38] New Topic: * Operations report (mthaddon/herb/spm) [16:38] 2009-07-02 - Cherry picked r8700 to the scripts server. [16:38] 2009-07-07 - Cherry picked r8766 to the scripts server. [16:38] 2009-07-07 - Cherry picked r8729 to the importd slaves. [16:38] Other than these it's been a quiet week. No service interruptions, a handful of minor incidents reported and no outstanding cherry pick or query requests. [16:39] herb, that's good to hear [16:39] does anyone have anything for herb? [16:39] hmm [16:40] moving on then :) [16:40] [TOPIC] * DBA report (stub) [16:40] New Topic: * DBA report (stub) [16:40] A few accounts had to be repaired, fallout from the dodgy data repair work performed last week. These gave me more clues to track down some unreported ones, which have also been fixed. Hopefully everything is correct now. [16:40] We will be testing the person pruner on staging, which should remove 90% of our Person records (which we can do because Account was split out of Person). When we run on production, it should have positive benefits throughout launchpad. The ValidPerson*Cache and TeamParticipation tables will shrink too, so three of our most common tables to join with will be much smaller. And lots of namespace will be freed up too. [16:40] Not much in the way of db patches this cycle. Should have reviews all sorted and things ready to land tomorrow. [16:40] oot. [16:41] stub, do you know if all accounts that were reporting problems on feedback@ were fixed? [16:41] All that have been forwarded to me have been fixed. [16:41] I don't personally monitor feedback@ [16:41] okay, I'll check if there's any outstanding one [16:42] thanks stub [16:42] [action] Ursinha to check if all accounts requesting fix on feedback@ were fixed [16:42] ACTION received: Ursinha to check if all accounts requesting fix on feedback@ were fixed [16:42] anything else for stub? [16:42] or [16:42] anything else at all? [16:43] 5 [16:43] 4 [16:43] 3 [16:43] 2 [16:43] 1 [16:43] Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. [16:43] #endmeeting [16:43] Meeting finished at 10:43. [16:43] thanks a lot guys [16:43] thanks Ursinha [16:45] Ursinha: Dammit, I had to go out for 1 minute and I missed the anything else bit! Rats :) [16:45] I'll email the ml/ [16:45] allenap, oops :) [16:45] Ursinha: Yep, my bad. [16:45] allenap, I should've poked you, sorry [16:46] Ursinha: No worries, I should have paid more attention. === salgado is now known as salgado-afk