=== Ursinha changed the topic of #launchpad-meeting to: Launchpad Meeting Grounds | Channel logs: http://irclogs.ubuntu.com/ | Weekly meetings: https://dev.launchpad.net/IRCMeetings | Lost in time? date -u
=== Ursinha is now known as Ursinha-afk
=== cprov is now known as cprov-zzz
=== Ursinha is now known as Ursinha-afk
=== ejat is now known as e-jat
=== cprov-zzz is now known as cprov
=== noodles775 is now known as noodles775-afk
=== Ursinha-afk is now known as Ursinha
=== noodles775-afk is now known as noodles775
henningeme ;)15:30
henningeoh, I did not miss it. Google Calendar *still* has the wrong time ...15:31
Ursinhahenninge, I thought google calendar was showing utc time15:37
henningeUrsinha: it does15:39
henningeUrsinha: But my copy of the meeting was at the wrong time ...15:40
henningeUrsinha: but there used to be some confusion about this in Google Calendar but it seems to be fixed now ... ;)15:40
Ursinhahenninge, :P15:41
MootBotMeeting started at 10:00. The chair is matsubara.16:00
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]16:00
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.16:00
matsubara[TOPIC] Roll Call16:00
MootBotNew Topic:  Roll Call16:00
matsubarahi stub16:01
matsubaralet's move on, tom berger can join in later16:02
matsubara[TOPIC] Agenda16:02
MootBotNew Topic:  Agenda16:02
matsubara * Actions from last meeting16:02
matsubara * Oops report & Critical Bugs & Broken scripts16:02
matsubara * Operations report (mthaddon/herb/spm)16:02
matsubara * DBA report (stub)16:02
matsubara[TOPIC] * Actions from last meeting16:02
MootBotNew Topic:  * Actions from last meeting16:02
matsubara * Ursinha to check if all accounts requesting fix on feedback@ were fixed16:02
matsubara * matsubara to chase rockstar about failure on updatebranches script16:02
Ursinhamatsubara, apparently all accounts were fixed16:03
matsubaraI talked to rockstar about the updatebranches script failures, he was debugging. I'll ask him again when he comes back from holidays16:03
matsubara[action] matsubara to chase rockstar about failure on updatebranches script16:03
MootBotACTION received:  matsubara to chase rockstar about failure on updatebranches script16:03
matsubarathanks Ursinha16:03
matsubara[TOPIC] * Oops report & Critical Bugs & Broken scripts16:03
MootBotNew Topic:  * Oops report & Critical Bugs & Broken scripts16:03
matsubaraUrsinha, please take the stage16:03
Ursinhamost of the "unusual" oopses we had were rollout glitches, also we had a few very low priority which I talked to people and filed bugs16:04
Ursinhawe have two critical bugs, after the rollout16:04
ubottuError: This bug is private16:05
ubottuUbuntu bug 403306 in launchpad-foundations "missing softlinks for shipit and login" [Critical,In progress]16:05
matsubaraUrsinha, the private one is in progress but not assigned?16:05
matsubarawho's taking care of it?16:05
UrsinhaI wasn't sure if that was ok for the librarian bug to be public, given the info in the report, if so I'll be happy to switch16:05
Ursinhamatsubara, apparently stub changed the status16:06
Ursinhastub, are you taking care of that?16:06
matsubaraI assigned that one to stub16:06
Ursinhaabout the missing softlinks, I know stub is taking care of that, but I have a question16:06
stubI have a branch awaiting RC16:06
Ursinhahow could we avoid that to happen?16:06
UrsinhaI see that the revision that caused that wasn't QAd - at least in the testplan that was NEEDSTESTING16:07
stubHowever, the production side of things needs to be done by the losas - editing the /etc/init.d script etc.16:07
matsubara[action] stub to get RC for branch that fixes bug 40328316:07
MootBotACTION received:  stub to get RC for branch that fixes bug 40328316:07
ubottuBug 403283 on http://launchpad.net/bugs/403283 is private16:07
ubottuBug 403283 on http://launchpad.net/bugs/403283 is private16:07
herbstub: is the branch listed in the bug? I'd like to take a look at it so we know what to expect.16:08
stubherb: it should be linked. You don't need that script of course, but it makes sense to keep it in the lp tree so devs can test if they broke it16:09
herbstub: cool. thanks. I'll take a look.16:09
matsubaraUrsinha, are we expecting some of those low importance oops fixes to land for the re-roll?16:09
marsstrange, the commit message doesn't mention having to touch those two links16:09
Ursinhamatsubara, nope16:09
mars        duplicate subscribers in the bug subscribers portlet even when16:10
mars        there is already an indirect subscription via a team membership."16:10
Ursinhamatsubara, not so far16:10
matsubaraUrsinha, isn't that what kiko asked yesterday?16:10
stubherb: And feel free to jump in and change and implement stuff if you want - lp-foundations doesn't have to own this stuff16:10
Ursinhamatsubara, nope16:10
=== bigjools-afk is now known as bigjools
stubmars: I think it was a cock up rather than a deliberate change16:11
Ursinhabut we have some outstanding oopses that keep happening16:11
Ursinhawho's on behalf of flacoste today?16:11
matsubaraUrsinha, yes, those are the ones I meant16:11
matsubaraUrsinha, mars16:11
marsstub, agreeded, but if so, did it just happen to slip past the dev, and the rc-reviewer?16:11
stubhttps://code.edge.launchpad.net/~stub/launchpad/pending-db-changes/+merge/9188 is also awaiting rc anyway. What I'm more interested in is how come the test suite still passes.16:12
marsUrsinha, I'm standing in for flacoste16:12
matsubarastub, why the branch that introduced the critical bug wasn't QA'd on staging?16:12
Ursinhamatsubara, I guess that was a bugs branch16:12
matsubaraoh, sorry. I thought it was a foundations issue16:12
marsstub, has the test suite changed to accommodate open sourcing in some way?  And thus removing those components from the suite?16:13
Ursinhamatsubara, the missing links are, but which introduced that not16:13
matsubaraUrsinha, ok.16:13
stubmars: I don't know.16:13
Ursinhamars, we have an oopses that happens almost everyday16:14
Ursinha*an oops16:14
Ursinhamars, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1300XMLP516:14
Ursinhamars, once I talked with flacoste about them, and he said that basically that wasn't our problem, iirc :)16:15
marsUrsinha, who would be the 'us' in "our problem"?16:15
Ursinhamars, foundations/launchpad16:15
Ursinhamars, i.e. not a bug16:16
Ursinhado you know what could we do to avoid those?16:16
abentleyUrsinha: is this the private or public XMLRPC server?16:16
marsUrsinha, do we have a procedure for dealing with such noise?  The feels like a 404: it is an error in the system state, but not a programmer error.16:16
UrsinhaI mean, we have a list of oopses that are growing every day because of hanging little things16:17
Ursinhamars, that's exactly my question :)16:17
marsUrsinha, for this specific issue, I would have to investigate.16:17
Ursinhamars, if we could deal differently those oopses, raising 404 instead of that oops, or even if we (me and matsubara) could move them to another section of the report16:17
matsubaraabentley, private one16:18
marsfor the larger issue of noise, well, maybe creating a new OOPS category or tag that can filter them out16:18
stubmatsubara: The branch was rc, so not much QA time. Even if it was qa'd, that doesn't mean anyone would have tested the login or shipit systems which should have been unaffected according to the commit message.16:18
abentleymatsubara: Well, it kinda is our problem, since we control both ends, then.16:18
marsUrsinha, not 404, but one of the other status codes, for "Ill-formed request"16:18
Ursinhamars, sorry, bad brain link :)16:19
abentleyI agree that bad input should not be treated as an OOPS16:19
matsubaraI don't like moving those oopses to another section.16:20
Ursinhamatsubara, me neither.16:20
abentleyBut it should be logged *somewhere*.16:20
matsubaraif the input is bad and we know it, let's not log an oops for it16:20
Ursinhamars, could you do some more investigation on this? I'll file a bug, can I?16:21
matsubarawe used to have a bug for ExpatErrors16:21
* Ursinha looks16:21
matsubarawas it fixed?16:21
marsUrsinha, certainly, please do16:21
Ursinhamars, ok, so I'll search for the bug, and file one if not able to find16:22
matsubara[action] ursinha do file bug for OOPS-1300XMLP516:22
MootBotACTION received:  ursinha do file bug for OOPS-1300XMLP516:22
Ursinhathanks matsubara16:23
matsubaraUrsinha, anything else?16:23
marsI think the procedure is to turn bad data from a "500: Internal Server Error" into a "400: Bad Request"16:23
Ursinhamatsubara, yes16:23
Ursinhacopy and paste fail16:23
Ursinhawe also have constant UnicodeDecodeErrors and the like16:24
Ursinhaeveryday, and I noticed that they're somehow growing16:24
Ursinhais the unicode problem really unsolvable?16:25
abentleyUrsinha: A bunch of them should disappear as of this rollout.16:25
Ursinhaabentley, right, but they seem to be everywhere. I'll keep one eye on them, and will report back next meeting16:26
Ursinhamatsubara, [action] Ursinha to ^16:26
Ursinhamars, this one is hanging for some time: bug 35459316:26
marsUrsinha, a unicode error in Python is solvable.  Are they coming from everywhere in the system?16:26
ubottuLaunchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/35459316:26
abentleyUrsinha: If you're looking for an ultimate solution, it's along the lines of "use Python 3.0".16:26
matsubara[action] Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting16:26
MootBotACTION received:  Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting16:26
Ursinhaabentley, :)16:26
Ursinhamars, I'll show you later, we have several oopses and different bugs for them16:27
Ursinha5 digit bugs :)16:27
Ursinhamars, can you take a look in that bug, please?16:28
Ursinhausing kiko's words: "if they happen more than once a week they are not low priority"16:29
Ursinhamatsubara, [action] mars to take a look at bug 35459316:30
ubottuLaunchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/35459316:30
marsok, so we know the problem, and we have an ideal solution16:30
matsubara[action] mars to take a look at bug 35459316:30
MootBotACTION received:  mars to take a look at bug 35459316:30
marsdid Francis mention a quick interim solution?16:30
marsstub, ^?16:30
stubinterim solution of what?16:31
Ursinhastub, bug 35459316:31
ubottuLaunchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/35459316:31
stubNo - I haven't discussed that with him.16:31
marsI'll have a look at the reports, split them into new bugs, try to eliminate the OOPSes16:32
Ursinhamars, can you make comments in the bug, if that applies, please?16:32
stubIt should be possible to register new exception views for that layer, no?16:32
marsthe branding is a related, but larger issue16:32
marsUrsinha, sure16:34
matsubaraUrsinha, anything else? we need to move on as we only have 11min16:34
Ursinhamatsubara, no, I'm done for today.16:34
matsubarathanks everyone16:34
Ursinhathanks mars, stub, abentley and herb16:34
matsubara[TOPIC] * Operations report (mthaddon/herb/spm)16:34
MootBotNew Topic:  * Operations report (mthaddon/herb/spm)16:34
herb2009-07-19 - We had another app server hang during log rotation. Fortunately it seems there has been some progress on bug #28730416:34
Ursinhaand all the others :)16:34
ubottuLaunchpad bug 287304 in launchpad-foundations "App Servers: Remove need for restart on logrotation" [High,Incomplete] https://launchpad.net/bugs/28730416:34
herb2009-07-21 - We open sourced! Congratulations to all the developers on a job well done. We saw an uptick in traffic to the codehosting system and, apparently, quite a few new project registrations. Overall the system handled the increased load well.16:34
herb2009-07-22 - Yesterday evening we rolled out 2.2.7. We had some hiccups again but we have bugs filed for those, as was discussed during the critical bugs section.16:34
herbThat's it from the LOSAs unless there are questions.16:34
matsubarathanks herb16:35
matsubara[TOPIC] * DBA report (stub)16:35
MootBotNew Topic:  * DBA report (stub)16:35
herbthanks matsubara16:35
stubDatabase update went painlessly, or at least I haven't heard any tales of pain and misery and nobody woke me up.16:36
stubWith 2.2.7, we can start pruning unwanted People. We just need to add '--experimental' to garbo-daily.py and maybe tweak the --abort-script argument if the default of 24 hours is too long.16:36
stubNothing else interesting happening.16:36
matsubarastub, are you taking care of enabling the pruning script? or have an RT for the losas to do it?16:37
mthaddonstub: yep, db updates were pretty quick and painless this time around16:37
stubI think salgado might have done that already. I was going to worry about it after reroll and the fires are all out.16:37
matsubara[action] matsubara to chase salgado about people pruning script16:39
MootBotACTION received:  matsubara to chase salgado about people pruning script16:39
matsubaraok, anything else for stub?16:39
matsubarathanks stub16:40
matsubarabefore I close, just want to let you know that I created the 2.2.8 milestone16:40
matsubaraduring yesterday's TLs meeting, the TL said it would be useful to coordinate work even if we're not doing a release for 2.2.816:41
matsubaraso please, since we're not doing the 2.2.8 release, don't let your QA itens otherwise will have a huge backlog for 2.2.916:42
matsubaras/itens/items slip/16:42
matsubaraI think that's all for today16:42
matsubaraThank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.16:42
MootBotMeeting finished at 10:43.16:43
herbthanks everyone16:43
Ursinhathanks matsubara16:43
Ursinhaand everyone16:43
abentleythanks matsubara.16:43
Ursinhamatsubara, you have to change the final message, the location at the logs aren't in topic anymore :)16:53
Ursinhamatsubara, "Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs."16:54
=== noodles775 is now known as noodles775-afk
matsubaraUrsinha, will update. thanks16:54
=== matsubara is now known as matsubara-lunch
=== salgado is now known as salgado-lunch
=== andrea-bs_ is now known as andrea-bs
=== ursula_ is now known as Ursinha
=== salgado-lunch is now known as salgado
=== matsubara-lunch is now known as matsubara
=== Ursinha is now known as Ursinha-lunch
=== cprov is now known as cprov-afk
=== Ursinha-lunch is now known as Ursinha
=== salgado is now known as salgado-afk
=== Edwin is now known as Guest73206
=== matsubara is now known as matsubara-dinner

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!