[00:07] sinzui: It is not an error if the DSP has no publishing_history [00:58] StevenK, Really? How can it be legitimate if it has not even gotten to the pending state. How do we ensure that https://launchpad.net/ubuntu/+source/html5-browser (ubuntu/html5-browser) is not accepted as an Ubuntu package? [00:58] My latest branch touches on that for bug targetting. [00:59] But only in terms of erroring, not restricting vocab results. [00:59] lifeless, I never want to see an open-team membership notification. I was tempted this weekend to hack a fix for team mail notification so that I can have a rule to delete them [00:59] I will see that in a few minutes then [01:00] sinzui: Hmm. You may be right. I will investigate today. [01:00] sinzui: https://bugs.launchpad.net/launchpad/+bug/815623 [01:00] <_mup_> Bug #815623: Mail notifications sent to team admins on joins / leaves to open teams < https://launchpad.net/bugs/815623 > [01:00] sinzui: I have a branch that nukes the notifications [01:00] sinzui: it has one wart [01:00] rs=me [01:00] But it looks like some poeple want the notification [01:00] sinzui: it squelches notification about folk that become / leave the admins of the team, which is still actually restricted; I think thats undesirable but not a Big Deal [01:01] lifeless, yes. that is a bad [01:01] sinzui: so I'd like your input on two things [01:01] sinzui: does the admin changes not being notified matter ? [01:01] sinzui: -and- [01:01] lifeless, this wart is one if underlying issue team notifications were not an easy fix. [01:02] sinzui: do we care about the folk that care about these spam messages ? [01:02] sinzui: I can fix it to retain admin notifications by checking the status field of the teammembership change, I suspect. [01:03] sinzui: but I don't know if we need to do that, or if we can file a high bug to come back and reinstate just those messages. [01:04] lifeless, I am not aware of a team admin that wanted the those notifications. I would not want to put a switch in. I think this class of issue is really structural subscriptions for team, which is a request of sorts in our bug tracker already [01:05] sinzui: so you support - squelch the notifications; when we do explicit subscriptions to team changes, then we can default a subscription for non-open team admins, default to no subscription for open team admins, and moderation etc will still be automatic ? [01:05] lifeless, yes [01:06] do you happen to know the bug # for subscriptions-to-team-changes ? [01:07] I can find it quickly [01:08] lifeless, bug 507515 [01:08] <_mup_> Bug #507515: want to subscribe to changes in a team

< https://launchpad.net/bugs/507515 > [01:08] thanks [01:08] lifeless, you may be fixing the root cause of bug 285414 [01:08] <_mup_> Bug #285414: There should be an option so that team admins don't receive notifications

< https://launchpad.net/bugs/285414 > [01:08] I will comment to this effect in the bug, file a bug about no admin notifications on open teams, and land my branch. [01:10] I think I am [01:11] lifeless: I just think landing your branch will lead to more bugs along the lines of "I used to be notified, now I'm not." [01:12] StevenK: which we'll dup onto bug 507515 [01:12] <_mup_> Bug #507515: want to subscribe to changes in a team

< https://launchpad.net/bugs/507515 > [01:12] Fairy nuff [01:26] sinzui: can admins make someone else an admin ? [01:27] No. [01:27] no. I can find the bug. [01:27] thats ok [01:27] it means that this notification change w.r.t. admins is ok [01:27] because there is also no action the admins can take [01:27] in fact, we shouldn't be notifying admins in that case, we should be notifying the owner. [01:27] They can demote others. But that's it. [01:30] sinzui: https://bugs.launchpad.net/launchpad/+bug/816196 [01:30] <_mup_> Bug #816196: new / deactivated admins of teams are not notified to the owner < https://launchpad.net/bugs/816196 > [01:30] sinzui: and https://bugs.launchpad.net/launchpad/+bug/815623/comments/8 [01:30] <_mup_> Bug #815623: Mail notifications sent to team admins on joins / leaves to open teams < https://launchpad.net/bugs/815623 > [01:31] thanks [01:39] sinzui: you might want to add tags to the new bug, I'm not sure what it should have [01:46] lifeless, done: this is my short list of team email issues: https://bugs.launchpad.net/launchpad/+bugs?field.tag=teams+email&field.tags_combinator=ALL [01:47] cool [01:52] sinzui: surely we've fixed https://bugs.launchpad.net/launchpad/+bug/120037 as a drive-by by now ? [01:52] <_mup_> Bug #120037: Action reported in email as done by owner rather than administrator

< https://launchpad.net/bugs/120037 > [01:53] lifeless: hi [01:53] hi poolie [01:53] why use a pretty printer at all, rather than just having recommendations for code style? [01:54] poolie: I thought I covered that earlier in the thread; because its easier for contributors. [01:55] hm maybe i misunderstood [01:55] i think bouncing changes because they're not pep8 clean raises the bar [01:56] lifeless, maybe. I thought there were one or two emails that were hard coded to admin [01:56] would using a pretty printer make things easier? [01:56] poolie: we could eliminate all code review formatting discussion [01:56] poolie: that seems easier to me. [01:57] if it happens at pqm post-commit, and if the contributor uses 200-character wide lines, reviewers will still complain [01:57] or tell them to run the formatter themselves [01:58] oh, i guess that was mentioned [02:03] poolie: are you against the use of a pretty printer ? [02:05] only slightly, because it seems to have some risk of breaking things and i don't think it gains very much [02:05] i am definitely in favor of making it easy to contribute, and in particular avoiding unnecessary rejections [02:08] lifeless, I am not sure that bug is fixed. There are rules to fallback to an historic membership.reviewed_by. [02:12] lifeless, NM. I think this issue was in TM.setStatus() and enforce its use now. I marked the bug released. [02:13] sinzui: \o/ cheap bug fixes ftw === almaisan-away is now known as al-maisan [04:02] does the API show hidden messages ? [04:12] i wonder how many hours a year the buildds spend "Building database of manual pages ..." [04:12] it's about 25% of the pbuilder run time on my machine... [04:19] wgrant: StevenK: either of you know why we do this - https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2032CD17#longstatements [04:19] the packagebuildjob query [04:20] buildfarmjob [04:20] I meant [04:20] lifeless: To show the failed/successful build counts, I guess. [04:21] wgrant: I'm unclear what you mean in https://bugs.launchpad.net/launchpad/+bug/711073/comments/3 [04:21] <_mup_> Bug #711073: Archive:+packages (Archive:+index) timeouts < https://launchpad.net/bugs/711073 > [04:21] are you saying 'the queries in this bug are fixed but others are a problem' [04:21] or [04:21] ??? [04:21] lifeless: +index and +packages were 99% < 5s. [04:22] Not fantastic, but not terrible. [04:22] ok [04:22] and bug 758258 is in-progress with no assignee ? [04:22] <_mup_> Bug #758258: buildfarmjob schema is inefficient for reporting < https://launchpad.net/bugs/758258 > [04:23] Fixed. [04:24] wgrant: did you land the new columns & garbo jobs? [04:24] No. Was going to do it soon before the DB deploy to minimise conflicts, but then got unexpectedly moved to Disclosure when Yellow was prematurely moved to maintenance. [04:25] There's only a couple of code changes needed in addition to the DB changes. [04:25] Most of them landed two months ago. [04:26] so, new world order, schema and code separate ;) [04:26] Yes. [04:27] I never planned to land them together. [04:27] bug https://bugs.launchpad.net/launchpad/+bug/816233 [04:27] <_mup_> Bug #816233: Archive:+packages timeouts in buildfarmjob status summaries < https://launchpad.net/bugs/816233 > [04:27] They did not need to be. [04:27] https://code.launchpad.net/~wgrant/launchpad/flatten-sprb [04:27] Actually flattens SPRB and BPB. [04:28] And mostly works. [04:28] * wgrant links it to the bug. [04:29] bah [04:29] edge needs to begone [04:29] Ah, already is linked. [04:29] Yes. [04:29] We should do that. [04:29] Or just hit people. [04:30] losa are chewing through things now [04:30] which reminds me [04:36] hmm [04:36] jtv: around ? I'm guessing too early. [04:39] ahha [04:40] stub: I've figured out one of our timeouts... due to a unique constraint. [04:40] Oh? [04:43] https://bugs.launchpad.net/launchpad/+bug/816235/comments/1 [04:43] <_mup_> Bug #816235: Bug:EntryResource:linkBranch timeouts < https://launchpad.net/bugs/816235 > [04:44] (I've added a second comment there now) [04:57] poolie: hi [04:57] hi hi [04:57] in bug 721166 yesterday [04:57] you pushed a branch [04:57] and linked it to the bug [04:57] <_mup_> Bug #721166: Tests sometimes fail on EC2 due to _LockWarner garbage

< https://launchpad.net/bugs/721166 > [04:57] the link timed out on you [04:58] did you retry it, or had you done a commit --fixes lp:721166 ? [04:58] istr getting a javascript error while establishing the link [04:58] with no details [04:59] it was OOPS-2032F24 [04:59] why, did it leave debris behind? [04:59] hell orobot? [04:59] I'm trying to confirm a theory behind the cause [04:59] if I'm right, we may have an explanation for a wide range of sporadic failures, and more data on why backend scripts with long (>1 second) transactions are bad) [05:00] the other possibility is that the scanner updates the branch row... that may be it [05:00] i actually wondered whether to mention it and then decided that "i got a js error with no oops" was boring [05:01] ok; can i do anything to help settle it? [05:03] I've got it sussed I think [05:03] https://bugs.launchpad.net/launchpad/+bug/816235 [05:03] <_mup_> Bug #816235: Bug:EntryResource:linkBranch timeouts due to branch scanner transaction length < https://launchpad.net/bugs/816235 > [05:03] it reminds me a bit of some other bug i filed [05:03] i forget the number [05:03] where what should have been a trivial action timed out after 9 seconds [05:06] yes [05:06] I've been of the opinion that backend scripts are a problem for a while [05:07] but 'contention' was never a very satisfying explanation [05:07] hm [05:07] i can't actually find that [05:07] and I can hardly expect to get a cultural change to something that isn't quite as easy on the basis of something that isn't satisfying :) [05:08] is there {waves hands} a way to run the branch scanner at a different isolation level or something? [05:08] wouldn't help [05:08] this is because we have foreign keys [05:10] the appserver commit cannot progress until it obtains a lock preventing anyone deleting or changing the id on the branch row. [05:11] pg does row locks, I don't think it does row,column locks [05:11] there are some design things we can do [05:12] putting status-tracking data in separate tables, for instance [05:12] but the biggest is short transactions [05:12] UbuntuSourcePackageNameWidget :( [05:12] Can I add a precommit hook forbidding any diffs matching [Uu]buntu? [05:13] no [05:13] you need a cultural change I think [05:13] or rather, yes, if you give me the db one for pqm at the same time ;) [05:13] Heh [05:13] anyhow, the things i saw, of should-be-tiny transactions timing out, would be consistent with that kind of long running lock holding transaction [05:14] perhaps also with other things like a thrashing db, but in that case i presume we'd notice [05:14] indeed [05:15] Anybody up for reviewing two ~100 line branches? [05:15] sure [05:15] https://code.launchpad.net/~wgrant/launchpad/transitionToTarget-validate_target/+merge/69189 and https://code.launchpad.net/~wgrant/launchpad/launchpadtargetwidget-prefix-respect/+merge/69187 [05:30] lifeless: another one for the 'long transactions are evil' bucket. [05:30] yes, exactly [05:30] that and DRI :P [05:33] lifeless: not sure why the branch scanner needs long transactions. It could probably run in autocommit mode even, provided it remembered to insert rows in earliest -> latest revision order. [05:34] It already partially commits sometimes. [05:34] Because if the initial scan fails it needs manual cleanup. [05:36] it probably updates the last_scanned field too early [05:36] or something [05:43] stub: how hard would it be to make the transaction reaper kill *updating* transactions over 2.5 seconds long ? [05:44] Goodbye bug modifications. [05:48] lifeless: We could kill transactions that have been trying to execute an UPDATE for longer than 2.5-3.0 seconds (0.5 seconds range, as the field we need is only updated every 500ms). Not sure if that would help anything though. [05:48] lifeless: We can't kill transactions that issued an update earlier in their transaction and might be blocking others. [05:49] stub: because idle in transaction might be a non-mutating request ? [05:49] yes [05:49] stub: we could introspect pg_locks ? [05:50] Better off doing all this with query timeouts. [05:50] yeah [05:50] was really just speculating [05:50] Less chance of blowing off your whole leg. [05:58] see -dev for some rambling about this :) [06:00] oh, i remember [06:01] lifeless: the thing i was thinking of was the failure of the ssh server to talk to the internal xmlrpc [06:01] xmlrpc was failing to look up a branch (iirc) [06:05] stub: francis is excited by our progress :) [06:05] ah, it's not a readonly query though [06:08] which bug ? [06:08] https://bugs.launchpad.net/bugs/294726 [06:08] <_mup_> Bug #294726: "ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch < https://launchpad.net/bugs/294726 > [06:08] i replied [06:09] especially #4 [06:14] lifeless: these are interesting cases because of course a person is quite likely to want to link a branch, or push it again, at the time it's being scanned [06:14] yup [06:15] Or create an MP... [06:15] That's where I see it most. [06:15] right [06:15] also these seem like fairly interesting things to put into a message queue or into a non sql database [06:15] though, probably that's not the best short term step [06:16] Launchpad has very few good short term steps :) [06:16] lifeless: Bug is the other table with similar issues to Branch. Want to move heat and heat_last_updated to a separate table, probably the other timestamps and caches too. [06:16] yup [06:17] (I agree, I haven't thought deeply about where or how. [06:17] e.g. should different fields be in separate tables [06:17] or a k:v table we get row locks on [06:17] or ... [06:17] i saw your comment reccently about "generally lp doesn't want ...options" [06:17] which i agree with in many ways [06:17] however it was in the context of notifications [06:18] and i do think a fb or g+ style "do/don't tell me about X action" per user would be ok, don't you? [06:18] perhaps [06:18] however, we have a bug asking for reasonably comprehensive subscriptions to teams [06:19] which is worth doing directly I think, rather than an awkward one off in a corner [06:19] what does "subscription to teams" mean? [06:19] telling you about changes to memberships? [06:19] and possibly other metadata === al-maisan is now known as almaisan-away [06:25] stub: 0.0.2 of pgbouncer is in the download cache for you [06:25] ta [06:26] stub: I have to run, but I'd love to know whats left to do a no-op downtime run ? [06:26] stub: I am guessing docs and landing last nights branch ? [06:26] lifeless: We need to push out configs to make everything go via pgbouncer. [06:27] lifeless: Apart from that, we can test now. Last nights branch is an improvement, but not necessary. And docs I prefer to write after I've confirmed everything works rather than having to adjust them for reality later :-) [06:27] \o/ [06:27] pgbouncer configs is the slow step [06:27] I've blogged about this, as promised [06:28] so users won't be shocked [06:28] Does this mean we need downtime for all downtime services in the next couple of days? [06:28] (librarian being the main problem) [06:29] ah the uploader ? yes we do [06:29] Think librarian actually reconnects happily - did on staging anyway when I ran this stuff. [06:29] We restarted it two weeks ago without hiccups, but still, something to consider. [06:30] stub: wgrant is referring to migrating it to use pgbouncer [06:30] stub: uploads to it are not load balanced at the moment [06:30] Ahh, yes. But outages will be minor (test connectivity first, then bounce). [06:31] Is the upload port going via haproxy? [06:31] Not yet. [06:31] Firewall rules are unclear. [06:32] it shouldn't be a problem being generous with access to the upload port. Not much you can do apart from try and fill up our disk. [06:32] There is no try [06:32] They *can* [06:33] uploads go via appservers [06:33] so an interrupted librarian results in an appserver OOPS [06:33] its not a big deal [06:34] Uh. [06:34] Yes, let's pretend that scripts don't exist :) [06:35] they can be queisced prior to the librarian bounce [06:35] True. [06:35] stub: sounds like the config update is the critical path thing then; if you need anything from me on it, let me know, otherwise its in your court [06:36] So, codehosting, ftpmaster, librarian, mailman are the downtime services I know of. [06:36] But mailman doesn't have DB access, I suppose. [06:57] hi lifeless—almost 9AM here now. What's up? [07:05] Good morning. [07:06] good morning rvba! [07:07] StevenK, wgrant: mind if I update dogfood and run generate-contents-files? [07:07] Please do. [07:12] rvba: dogfood conflict in package cloner. Were you running an experimental branch there? [07:21] danilos: Hi! ;) [07:25] morning, former teammates === wgrant changed the topic of #launchpad-dev to: Performance Tuesday | https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 238 - 0:[#######=]:256 [07:26] Hi jtv! [07:26] jtv: What do you think should be done to fix bug 504062? [07:26] <_mup_> Bug #504062: External suggestion "Used in" links to disabled template <404> < https://launchpad.net/bugs/504062 > [07:27] * jtv looks [07:27] jtv: Should non-admins not see the suggestion at all or just don't get a link to the template? [07:28] jtv: sorry, was afk ... I might know what the problem is ... [07:28] jtv: how can I take a look? (I don't have access to DF) [07:28] rvba: ah. Hang on, have to look at something else now. [07:28] spiv: re your question on bug 813349, people filed a couple of nit type bugs [07:28] <_mup_> Bug #813349: hard to get from mp to per-revision diffs

< https://launchpad.net/bugs/813349 > [07:28] like your link colour is wrong [07:28] personally i wouldn't block on them [07:29] henninge: istm the reason for showing suggestions from disabled templates is mostly obsolete thanks to sharing. [07:30] henninge: we could try getting some data — how often are those translations both different and useful? [07:30] If the answer is "rarely," then the solution is simple. [07:31] (Well, apart perhaps from lag between the time the template is disabled and the next update of the suggestive-templates cache). [07:32] rvba, here's the conflict: http://paste.ubuntu.com/652250/ [07:34] henninge: if it turns out we're getting real benefit from disabled templates, then we'd have to worry about disabling links. I even think Zope would do that for us if this were a permission error rather than a 404, but at a performance cost. [07:35] good morning [07:35] HI adeuring! [07:35] hi henninge! [07:36] jtv: It's Julian SQL optimisation from yesterday. It looks to me like he tried it manually on DF, then he got it properly landed because (I think) all the code that causes the conflict is there already. [07:36] jtv: Hm, I'd like to keep this simple ... [07:36] rvba: in that case, I should be able to revert & pull. That's great. [07:36] jtv: I think so yes. [07:37] henninge: I sympathize, but if it were truly simple maybe this bug would've been fixed by now. :) [07:37] jtv: Before I read this I would have expected translations from disabled templates to not show up [07:38] Yes, I thought they were excluded from the suggestive-templates cache. Have you been able to reproduce the problem? [07:38] jtv: on qastaging, yes. But I am not sure about the result being cached. [07:39] jtv: how long does stuff live in that cache? [07:39] poolie: hmm, it'd be nice to have those bugs linked to 813349 perhaps, but probably not important if they're not blockers [07:39] also, I never worked with the cache before. [07:40] henninge: the suggestive templates cache is rewritten in its entirely… I think daily at the moment, but may be much more frequently. We could afford to do it much more often than that. [07:40] poolie: my main concern is that the bug doesn't languish indefinitely as Fix Committed. [07:41] henninge: it's just a database table containing the ids of POTemplates that can be used to get suggestions from. And it does indeed exclude disabled templates. [07:41] (Just checked) [07:41] spiv so fortunately 'new bugs' should find them [07:41] henninge: so whenever a template is disabled, this problem can exist until the cache is rewritten. [07:41] ah! [07:41] henninge: (Also, I see it hasn't been updated for usage enums) [07:42] henninge: The bug description therefore would seem a little out of date. [07:42] yes [07:43] jtv: so the problem is really just about how to deal with an invalid cache. [07:43] Yes, it's cache invalidation — one of the two things that are hard. [07:43] ;) [07:44] We could afford to update the cache whenever anyone deactivates a template. [07:44] IIRC it takes only a fraction of a second to find all those templates; it's just that we were doing it multiple times for every translations page. [07:45] If the suggestions queries also join in the POTemplate, we could just add a redundant iscurrent check. [07:46] spiv if only there was automatic connection between related bugs :) [07:46] jtv: yes, those were the two things I was considering right now. [07:46] Alas, the query doesn't include POTemplate. [07:47] jtv: I just wasn't sure how expensive the joining would be. [07:47] And this is velly velly pelformance-snsitive. [07:47] yup [07:47] So best not add that join. [07:47] jtv: but removing that entry from the cache should be simple. [07:48] What's really weird by the way is that the code in _getExternalTranslationMessages seems to fetch POTMsgSet ids and call them "pots." [07:48] spiv, added [07:48] Maybe that's just someone who wasn't familiar with the conventional abbreviations. [07:49] henninge: it looks like someone's made a mess of that code. There's also an over-long line of SQL, with the keywords not capitalized. That smells of someone just trying something out and accidentally landing it as soon as it worked, without review. [07:49] poolie: thanks [07:50] henninge: ah no, it's lifeless code. [07:51] jtv: how is that different? [07:52] This is hard to say diplomatically. [07:52] In Launchpad we have traditionally focused on producing code that's easy to read. [07:52] Robert leans more towards keeping it easy to write. [07:52] ... [07:52] So I often struggle reading his code. [07:53] For that reason I don't think it saves any time, but opinion seems divided on the subject. [07:53] ... [07:53] Still, you would have expected a "make lint." [07:53] very much so [07:53] * spiv offers hennige a …, for variety [07:53] thanks spiv ;) [07:53] Also, try .. [07:54] jtv: how do you think I typed it? :) [07:54] spiv: I do apologize. :) I only checked Henning's. :) [07:54] Or perhaps I should say: ☺ [07:54] I ♥ [07:54] … [07:54] ⁺¹ [07:55] * henninge learnt something [07:55] "You've taken your first step into a larger world…" [07:55] ¹²³¼½ [07:56] I already know those … [07:56] s/know/knew/ [07:57] * spiv offers jtv <" [07:57] Oooh! Thank you! [07:57] You know what's going to happen now, don't you? [07:57] “Quote-mania.” [07:58] ¡Por supuesto! ;) [07:58] wøñđêřfüł [08:01] morning bigjools [08:02] morgen [08:02] bigjools: I'm just messing around on dogfood right now, so beware [08:02] jtv: eek :) [08:04] Ah-HAaah. No "idle in transaction" no more for generate-contents-files. [08:06] Deployments to cocoplum do take downtime, right? [08:06] Yes. We'll probably need one in the next day or so for config updates. [08:07] stub: Any idea on the timing of that? [08:07] wgrant: nup [08:07]