=== mwhudson_ is now known as mwhudson
=== mrevell is now known as mrevell-lunch
=== danilo_ is now known as danilos
=== mrevell-lunch is now known as mrevell
MootBotMeeting started at 09:00. The chair is matsubara.15:00
MootBotCommands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]15:00
matsubaraWelcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.15:00
matsubara[TOPIC] Roll Call15:00
MootBotNew Topic:  Roll Call15:00
matsubaraNot on the Launchpad Dev team? Welcome! Come "me" with the rest of us!15:00
=== citrus is now known as sinzui
matsubaraChex, rockstar, bigjools, danilos, allenap: hi15:01
Ursinhamatsubara, rockstar is sprinting, not sure he'll answer15:01
matsubaraapologies from stuart. he won't be around today and sent me the dba report15:01
bigjoolsUrsinha: you're talking two more languages now then?15:02
Ursinhabigjools, I pretend, and it's working :P15:02
matsubara[TOPIC] Agenda15:03
MootBotNew Topic:  Agenda15:03
matsubara * Actions from last meeting15:03
matsubara * Oops report & Critical Bugs & Broken scripts15:03
matsubara * Operations report (mthaddon/Chex/spm/mbarnett)15:03
matsubara * DBA report (stub)15:03
matsubara * Proposed items15:03
matsubara[TOPIC] * Actions from last meeting15:03
MootBotNew Topic:  * Actions from last meeting15:03
matsubara* Ursinha to send one email to lp list explaining the qa-tags experiment15:03
matsubara* allenap to follow up with pitti about bug 45320315:03
matsubara* matsubara to talk to francis about the IncidentLog QA section15:03
matsubara* matsubara to email the list about lib/lp/scripts/utilities/sanitizedb.py script15:03
ubottuLaunchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/45320315:03
matsubara    * email sent15:03
matsubarahi Chex15:03
Ursinhamatsubara, there's a note in that page about my item15:03
=== mbarnett` is now known as mbarnett
matsubaraI sent the email to the list about the sanitizeddb.py script. I don't think anybody replied15:03
matsubaraUrsinha, reloading :-)15:04
Ursinhamatsubara, I still have to send it, CHR week is kinda consuming :)15:04
bigjoolsmatsubara: I did!15:04
matsubara[action] * Ursinha to send one email to lp list explaining the qa-tags experiment15:04
MootBotACTION received:  * Ursinha to send one email to lp list explaining the qa-tags experiment15:04
allenapmatsubara: Re. bug 453203. It's an issue in apport, pitti is going to fix, and he says, from his perspective, that we don't need to work around in Launchpad. However, we may want to anyway.15:04
ubottuLaunchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/45320315:05
gary_postermatsubara: bigjools reviewed the santizedb script and stub has or is going to adjust based on his feedback.  soyuz was stub's biggest worry, so that's cool.15:05
matsubarasorry bigjools, I missed your reply. thanks!15:05
allenapmatsubara: However, any workaround should probably be removed once apport has been fixed.15:05
matsubaraI emailed Francis about the IncidentLog QA section and I'll help him out nagging the TL about completing the QA section15:06
matsubaraallenap, cool. thanks15:07
matsubaraI think that's all for this section15:07
matsubara[TOPIC] * Oops report & Critical Bugs & Broken scripts15:07
MootBotNew Topic:  * Oops report & Critical Bugs & Broken scripts15:07
matsubarago ahead little bear15:08
Ursinhasinzui, bug 48135615:08
ubottuLaunchpad bug 481356 in launchpad-registry "AttributeError OOPS when trying to subscribe to bugmail of a milestone" [Undecided,New] https://launchpad.net/bugs/48135615:08
Ursinhaallenap, bug 458180, we're having high timeout counts in lpnet and this one is the main one15:08
ubottuLaunchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/45818015:08
Ursinhaallenap, also, as this is my CHR week, I saw lots of complaints (and some hate mail) about bug 41865915:08
ubottuLaunchpad bug 418659 in malone "Reporting duplicate bugs leads to receiving notifications for every duplicate of the original bug" [Undecided,New] https://launchpad.net/bugs/41865915:08
Ursinhais there something we could do?15:08
sinzuiUrshina, that is again a bug link broken by intellectronica's bugmail-ui-fixes15:09
allenapUrsinha: Re. the timeout bug, gmb is doing some magic with subscriber lists to help alleviate this.15:09
Ursinhaallenap, hm, great15:09
sinzuiI am note certain who should fix it, the diff shows the that this line was removed:15:09
sinzuienabled = not IProjectMilestone.providedBy(self.context)15:09
Ursinhaallenap, can you ask him or comment in the bug, please?15:09
allenapUrsinha: I'm sure we can do something about the dupe email one; I'll ask the team.15:10
allenapUrsinha: Sure.15:10
Ursinhathanks a lot allenap15:10
Ursinhathat would be much appreciated by lots of users that report ubuntu bugs15:10
Ursinhasinzui, well15:12
Ursinhasinzui, doesn't seem to hard to fix15:12
sinzuiIt is not hard, every link the milestone menu has to have that line15:13
Ursinhasinzui, so, can you fix it? :)15:13
sinzuiThe alternate fix is to remove projectgroups since they break more things then they fix15:13
sinzuiUrsinha: The bugs team should fix it since it is their bug link.15:14
Ursinhasinzui, right then.. allenap?15:14
* allenap reads backwards15:15
allenapsinzui: Do you think the fix is simply to reinstate the line you've mentioned in the bug report?15:16
sinzuiallenap: no, because intellectronica had icon add/edit handling to it15:17
allenapsinzui: Okay, thanks.15:17
Ursinhathanks sinzui and allenap15:17
allenapUrsinha: I'll pass this to intellectronica. He's at lazr-js this week, UDS next, but perhaps he'll have time anyway :-/15:18
Ursinhathanks allenap15:18
sinzuiMy fix would involve removing projectgroups since they break more things then the fix. I would introduce pillar tags15:18
allenapsinzui: I'll pass that on.15:19
Ursinhawe have two critical bugs in progress, one is bug 475394, another is bug 47916915:19
ubottuLaunchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/47539415:19
ubottuBug 479169 on http://launchpad.net/bugs/479169 is private15:19
Ursinhathe bzr bug is code, so we'll have to poke rockstar or abentley to know15:20
Ursinhathe other is soyuz15:20
Ursinhaal-maisan is working on it, right bigjools?15:21
bigjoolsUrsinha: it's been rolled out15:21
Ursinhabigjools, cherrypicked, you mean?15:22
Ursinhabigjools, could you change it to fix released then, please? and add a comment saying that it was cped :)15:22
Ursinhamatsubara, [action] chase rockstar/abentley about bug 47539415:23
ubottuLaunchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/47539415:23
Ursinhathanks bigjools :)15:23
* rockstar just logged in15:23
matsubara[action] Ursinha to chase rockstar/abentley about bug 47539415:23
MootBotACTION received:  Ursinha to chase rockstar/abentley about bug 47539415:23
bigjoolsanytime :)15:23
Ursinhascript parse-librarian-apache-logs is failing for some time, is that a real failure?15:23
rockstarUrsinha, that's fixed now.15:23
Ursinharockstar, fixed and cherrypicked?15:23
rockstarUrsinha, it was cherrypicked to production, I'm not sure if the fix has been landed on trunk.15:24
rockstarUrsinha, the code team, with the exception of thumper, is at sprints this week.15:24
rockstarUrsinha, looks like the linked branch has been merged.15:25
rockstarI marked the bug as "Fix released"15:25
Ursinharockstar, good, could you do the same I've asked bigjools to do, please?15:25
Ursinhathanks rockstar :)15:25
matsubara[action] cancel previous action item15:25
MootBotACTION received:  cancel previous action item15:25
Ursinhafast boy15:25
Ursinhaabout the failing script, who's responsible for parse-librarian-apache-logs?15:25
matsubaragary_poster, ^15:26
matsubaraI think salgado is the right person to ask about it. we need to find why it's failing though15:26
gary_postermatsubara, Ursinha: I have no idea and have never heard of it, but am happy to acknowledge that it sounds vaguely like foundations :-)  I'll take it.15:27
sinzuiWas the script failing or just not running15:27
matsubaragary_poster, I think salgado wrote that script, IIRC15:27
Ursinhamy question is similar to sinzui's15:27
sinzuiThe script has definitely been running since the logs were synced15:27
gary_postermatsubara: ok.  If you are willing to go straight to him, +115:27
salgadoit's failing15:27
salgadolet me find the logs15:28
gary_postermaybe another lovely _pythonpath thing?15:28
sinzuisalgado: ok, we did see downloads counts get updated15:28
salgadolooks like an actual bug in the script15:28
salgadosinzui, right, the script parsed all the logs but some recent log lines seem to have broken it15:29
salgadoit might just be trying to parse an empty file15:29
salgadojust need to ask the losas to get a 'ls -al' of the files in the directory to which we point the script15:29
salgadomthaddon, can you do that?15:29
mthaddonsalgado: what's that?15:30
matsubarasalgado, in any case we'd need a bug reported to deal with that case right?15:30
salgadoyeah, definitely looks like an empty file15:30
salgadomatsubara, right15:30
Ursinhasalgado, want me to file a bug?15:30
salgadomthaddon, 'ls -al <dir-containing-librarian-log-files-to-be-parsed>'15:30
salgadoUrsinha, please.  I won't have time for filing/fixing it this week15:31
mthaddonsalgado: https://pastebin.canonical.com/24669/15:31
matsubara[action] matsubara to file a bug about the script that parses librarian logs failure15:31
MootBotACTION received:  matsubara to file a bug about the script that parses librarian logs failure15:31
Ursinhamatsubara, I'm doing it right now15:31
matsubara[action] cancel the previous action item15:31
MootBotACTION received:  cancel the previous action item15:31
matsubarathanks Ursinha15:31
matsubaraUrsinha, anything else for this section?15:32
salgadomthaddon, actually need 'ls -al /srv/launchpadlibrarian.net-logs/*/'15:32
Ursinhamatsubara, no, thanks15:33
matsubarathanks everyone.15:33
Ursinhasalgado, is that unrelated to bug 422552?15:33
ubottuLaunchpad bug 422552 in launchpad-foundations "Apache log parser failed to parse a line because it had too many values to unpack" [Undecided,Triaged] https://launchpad.net/bugs/42255215:33
matsubara[TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)15:33
MootBotNew Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)15:33
mthaddonsalgado: https://pastebin.canonical.com/24672/15:34
salgadoUrsinha, yes, unrelated15:34
ChexHello everyone15:34
Chexhere is our report for this week:15:34
Chex- New LOSA python script to hide Bugs & Questions comments:15:34
Chex        ; This is to replace direct DB editing. Chex is working on getting this setup and documented properly.15:34
Chex- QA Column of the incident log:15:35
Chex        ; This column is still not being updated by anyone; we would appreciate15:35
Chex          if people could look at and update this when possible.15:35
Chex- LP Incidents of note:15:35
Chex        ; Launchpad 3.1.10 major release rolled out on 06-Nov15:35
Chex        ; CP 8648 rolled out to loganberry-launchpad, loganberry-bzrsyncd,15:35
Chex          codehost, lpnet, edge, cesium, cocoplum, germanium15:35
Chex        ; CP 8649 of production-stable rolled out to cesium, germanium,15:35
Chex          loganberry-launchpad, loganberry-bzrsyncd, codehost15:35
matsubaraChex, I'm help Francis with the nagging about the QA column incident log15:35
Ursinha(salgado: bug 481375, could you triage it, please?)15:35
ubottuLaunchpad bug 481375 in launchpad-foundations "parse-librarian-apache-logs script failing trying to parse empty files" [Undecided,New] https://launchpad.net/bugs/48137515:35
Chexmatsubara: great, thanks a lot for that15:36
Chexand thats our report, any questions for us??15:36
matsubaraChex, basically the only problem there is the codebrowse issue which is the same bug as usual (https://bugs.edge.launchpad.net/launchpad-code/+bug/118625)15:36
ubottuLaunchpad bug 118625 in loggerhead "codebrowse sometimes hangs" [High,Triaged]15:36
matsubaraChex, and I have asked gary_poster to investigate the core dump output from the lpnet13 death recently.15:36
Chexmatsubara: ok, thats great, too.15:37
matsubaraonce more info is known about it, a bug will be reported and added to the column15:37
stubI suspect the incident log isn't going to get updated without some prodding as people are generally worrying about today rather than yesterday.15:37
matsubarastub, I'll be doing the prodding15:38
stubThats good - I can't even find the wiki page ;)15:39
bigjoolsit would be good if you made codebrowse changes trivial so I only get emailed about less-comment incidents15:39
bigjools(on the incident log wiki I mean)15:39
Chexstub: https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog15:39
matsubarastub, and once we have a bug filed about it, all the discussion can happen in the bug report, soothing losas worries15:39
matsubarastub, https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog15:39
bigjoolss/less-comment/less-common/   geez15:40
matsubaraI think that's all from Chex. thank you15:41
matsubara[TOPIC] * DBA report (stub)15:41
MootBotNew Topic:  * DBA report (stub)15:41
stubA second attempted run of the database sanitization script is underway on production data. We should see how far it gets this time in a few hours (it took over 6 hours to fail last time). This has been increasing load on chokecherry which hosts one of the production slave databases, but I doubt it is noticeable.15:41
stubA fix has landed and staging database restores have been reenabled. The current rebuild will fail because the fix hasn't got through to launchpad/stable, so we need to wait until tomorrow to see if things work out as hoped.15:41
stubLoad has been higher for a while. From eyeball testing, this load is from the appservers. I think this is more to do with increased traffic since we brought the new appservers online rather than code regressions. We may need to start identifying and optimizing heavy db load pages soon. memcache might help here once that branch can land.15:41
matsubarano questions for stub?15:44
matsubaraall right, thanks stub15:44
matsubara[TOPIC] * Proposed items15:44
MootBotNew Topic:  * Proposed items15:44
matsubarano new proposed items15:45
matsubaraThank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.15:45
MootBotMeeting finished at 09:45.15:45
danilosthank you15:45
=== matsubara is now known as matsubara-lunch
=== salgado_ is now known as salgado
=== matsubara-lunch is now known as matsubara
=== danilos is now known as danilo-afk
=== matsubara is now known as matsubara-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!