[00:07] <StevenK> sinzui: It is not an error if the DSP has no publishing_history
[00:58] <sinzui> StevenK, Really? How can it be legitimate if it has not even gotten to the pending state. How do we ensure that https://launchpad.net/ubuntu/+source/html5-browser (ubuntu/html5-browser) is not accepted as an Ubuntu package?
[00:58] <wgrant> My latest branch touches on that for bug targetting.
[00:59] <wgrant> But only in terms of erroring, not restricting vocab results.
[00:59] <sinzui> lifeless, I never want to see an open-team membership notification. I was tempted this weekend to hack a fix for team mail notification so that I can have a rule to delete them
[00:59] <sinzui> I will see that in a few minutes then
[01:00] <StevenK> sinzui: Hmm. You may be right. I will investigate today.
[01:00] <lifeless> sinzui: https://bugs.launchpad.net/launchpad/+bug/815623
[01:00] <_mup_> Bug #815623: Mail notifications sent to team admins on joins / leaves to open teams <Launchpad itself:Triaged> < https://launchpad.net/bugs/815623 >
[01:00] <lifeless> sinzui: I have a branch that nukes the notifications
[01:00] <lifeless> sinzui: it has one wart
[01:00] <sinzui> rs=me
[01:00] <StevenK> But it looks like some poeple want the notification
[01:00] <lifeless> sinzui: it squelches notification about folk that become / leave the admins of the team, which is still actually restricted; I think thats undesirable but not a Big Deal
[01:01] <sinzui> lifeless, yes. that is a bad
[01:01] <lifeless> sinzui: so I'd like your input on two things
[01:01] <lifeless> sinzui: does the admin changes not being notified matter ?
[01:01] <lifeless> sinzui: -and-
[01:01] <sinzui> lifeless, this wart is one if underlying issue team notifications were not an easy fix.
[01:02] <lifeless> sinzui: do we care about the folk that care about these spam messages ?
[01:02] <lifeless> sinzui: I can fix it to retain admin notifications by checking the status field of the teammembership change, I suspect.
[01:03] <lifeless> sinzui: but I don't know if we need to do that, or if we can file a high bug to come back and reinstate just those messages.
[01:04] <sinzui> lifeless, I am not aware of a team admin that wanted the those notifications. I would not want to put a switch in. I think this class of issue is really structural subscriptions for team, which is a request of sorts in our bug tracker already
[01:05] <lifeless> sinzui: so you support - squelch the notifications; when we do explicit subscriptions to team changes, then we can default a subscription for non-open team admins, default to no subscription for open team admins, and moderation etc will still be automatic ?
[01:05] <sinzui> lifeless, yes
[01:06] <lifeless> do you happen to know the bug # for subscriptions-to-team-changes ?
[01:07] <sinzui> I can find it quickly
[01:08] <sinzui> lifeless, bug 507515
[01:08] <_mup_> Bug #507515: want to subscribe to changes in a team <email> <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/507515 >
[01:08] <lifeless> thanks
[01:08] <sinzui> lifeless, you may be fixing the root cause of bug 285414
[01:08] <_mup_> Bug #285414: There should be an option so that team admins don't receive notifications <email> <feature> <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/285414 >
[01:08] <lifeless> I will comment to this effect in the bug, file a bug about no admin notifications on open teams, and land my branch.
[01:10] <lifeless> I think I am
[01:11] <StevenK> lifeless: I just think landing your branch will lead to more bugs along the lines of "I used to be notified, now I'm not."
[01:12] <lifeless> StevenK: which we'll dup onto bug 507515
[01:12] <_mup_> Bug #507515: want to subscribe to changes in a team <email> <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/507515 >
[01:12] <StevenK> Fairy nuff
[01:26] <lifeless> sinzui: can admins make someone else an admin ?
[01:27] <wgrant> No.
[01:27] <sinzui> no. I can find the bug.
[01:27] <lifeless> thats ok
[01:27] <lifeless> it means that this notification change w.r.t. admins is ok
[01:27] <lifeless> because there is also no action the admins can take
[01:27] <lifeless> in fact, we shouldn't be notifying admins in that case, we should be notifying the owner.
[01:27] <wgrant> They can demote others. But that's it.
[01:30] <lifeless> sinzui: https://bugs.launchpad.net/launchpad/+bug/816196
[01:30] <_mup_> Bug #816196: new / deactivated admins of teams are not notified to the owner <Launchpad itself:Triaged> < https://launchpad.net/bugs/816196 >
[01:30] <lifeless> sinzui: and https://bugs.launchpad.net/launchpad/+bug/815623/comments/8
[01:30] <_mup_> Bug #815623: Mail notifications sent to team admins on joins / leaves to open teams <Launchpad itself:Triaged> < https://launchpad.net/bugs/815623 >
[01:31] <sinzui> thanks
[01:39] <lifeless> sinzui: you might want to add tags to the new bug, I'm not sure what it should have
[01:46] <sinzui> lifeless, done: this is my short list of team email issues: https://bugs.launchpad.net/launchpad/+bugs?field.tag=teams+email&field.tags_combinator=ALL
[01:47] <lifeless> cool
[01:52] <lifeless> sinzui: surely we've fixed https://bugs.launchpad.net/launchpad/+bug/120037 as a drive-by by now ?
[01:52] <_mup_> Bug #120037: Action reported in email as done by owner rather than administrator <email> <lp-registry> <teams> <Launchpad itself:Triaged> < https://launchpad.net/bugs/120037 >
[01:53] <poolie> lifeless: hi
[01:53] <lifeless> hi poolie
[01:53] <poolie> why use a pretty printer at all, rather than just having recommendations for code style?
[01:54] <lifeless> poolie: I thought I covered that earlier in the thread; because its easier for contributors.
[01:55] <poolie> hm maybe i misunderstood
[01:55] <poolie> i think bouncing changes because they're not pep8 clean raises the bar
[01:56] <sinzui> lifeless, maybe. I thought there were one or two emails that were hard coded to admin
[01:56] <poolie> would using a pretty printer make things easier?
[01:56] <lifeless> poolie: we could eliminate all code review formatting discussion
[01:56] <lifeless> poolie: that seems easier to me.
[01:57] <poolie> if it happens at pqm post-commit, and if the contributor uses 200-character wide lines, reviewers will still complain
[01:57] <poolie> or tell them to run the formatter themselves
[01:58] <poolie> oh, i guess that was mentioned
[02:03] <lifeless> poolie: are you against the use of a pretty printer ?
[02:05] <poolie> only slightly, because it seems to have some risk of breaking things and i don't think it gains very much
[02:05] <poolie> i am definitely in favor of making it easy to contribute, and in particular avoiding unnecessary rejections
[02:08] <sinzui> lifeless, I am not sure that bug is fixed. There are rules to fallback to an historic membership.reviewed_by.
[02:12] <sinzui> lifeless, NM. I think this issue was in TM.setStatus() and enforce its use now. I marked the bug released.
[02:13] <lifeless> sinzui: \o/ cheap bug fixes ftw
[04:02] <lifeless> does the API show hidden messages ?
[04:12] <mwhudson> i wonder how many hours a year the buildds spend "Building database of manual pages ..."
[04:12] <mwhudson> it's about 25% of the pbuilder run time on my machine...
[04:19] <lifeless> wgrant: StevenK: either of you know why we do this - https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-2032CD17#longstatements
[04:19] <lifeless> the packagebuildjob query
[04:20] <lifeless> buildfarmjob
[04:20] <lifeless> I meant
[04:20] <wgrant> lifeless: To show the failed/successful build counts, I guess.
[04:21] <lifeless> wgrant: I'm unclear what you mean in https://bugs.launchpad.net/launchpad/+bug/711073/comments/3
[04:21] <_mup_> Bug #711073: Archive:+packages (Archive:+index) timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/711073 >
[04:21] <lifeless> are you saying 'the queries in this bug are fixed but others are a problem'
[04:21] <lifeless> or
[04:21] <lifeless> ???
[04:21] <wgrant> lifeless: +index and +packages were 99% < 5s.
[04:22] <wgrant> Not fantastic, but not terrible.
[04:22] <lifeless> ok
[04:22] <lifeless> and bug 758258 is in-progress with no assignee ?
[04:22] <_mup_> Bug #758258: buildfarmjob schema is inefficient for reporting <timeout> <Launchpad itself:In Progress> < https://launchpad.net/bugs/758258 >
[04:23] <wgrant> Fixed.
[04:24] <lifeless> wgrant: did you land the new columns & garbo jobs?
[04:24] <wgrant> No. Was going to do it soon before the DB deploy to minimise conflicts, but then got unexpectedly moved to Disclosure when Yellow was prematurely moved to maintenance.
[04:25] <wgrant> There's only a couple of code changes needed in addition to the DB changes.
[04:25] <wgrant> Most of them landed two months ago.
[04:26] <lifeless> so, new world order, schema and code separate ;)
[04:26] <wgrant> Yes.
[04:27] <wgrant> I never planned to land them together.
[04:27] <lifeless> bug https://bugs.launchpad.net/launchpad/+bug/816233
[04:27] <_mup_> Bug #816233: Archive:+packages timeouts in buildfarmjob status summaries <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/816233 >
[04:27] <wgrant> They did not need to be.
[04:27] <wgrant> https://code.launchpad.net/~wgrant/launchpad/flatten-sprb
[04:27] <wgrant> Actually flattens SPRB and BPB.
[04:28] <wgrant> And mostly works.
[04:28]  * wgrant links it to the bug.
[04:29] <lifeless> bah
[04:29] <lifeless> edge needs to begone
[04:29] <wgrant> Ah, already is linked.
[04:29] <wgrant> Yes.
[04:29] <wgrant> We should do that.
[04:29] <wgrant> Or just hit people.
[04:30] <lifeless> losa are chewing through things now
[04:30] <lifeless> which reminds me
[04:36] <lifeless> hmm
[04:36] <lifeless> jtv: around ? I'm guessing too early.
[04:39] <lifeless> ahha
[04:40] <lifeless> stub: I've figured out one of our timeouts... due to a unique constraint.
[04:40] <wgrant> Oh?
[04:43] <lifeless> https://bugs.launchpad.net/launchpad/+bug/816235/comments/1
[04:43] <_mup_> Bug #816235: Bug:EntryResource:linkBranch timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/816235 >
[04:44] <lifeless> (I've added a second comment there now)
[04:57] <lifeless> poolie: hi
[04:57] <poolie> hi hi
[04:57] <lifeless> in bug 721166 yesterday
[04:57] <lifeless> you pushed a branch
[04:57] <lifeless> and linked it to the bug
[04:57] <_mup_> Bug #721166: Tests sometimes fail on EC2 due to _LockWarner garbage <build-infrastructure> <spurious-test-failure> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/721166 >
[04:57] <lifeless> the link timed out on you
[04:58] <lifeless> did you retry it, or had you done a commit --fixes lp:721166 ?
[04:58] <poolie> istr getting a javascript error while establishing the link
[04:58] <poolie> with no details
[04:59] <lifeless> it was OOPS-2032F24
[04:59] <poolie> why, did it leave debris behind?
[04:59] <poolie> hell orobot?
[04:59] <lifeless> I'm trying to confirm a theory behind the cause
[04:59] <lifeless> if I'm right, we may have an explanation for a wide range of sporadic failures, and more data on why backend scripts with long (>1 second) transactions are bad)
[05:00] <lifeless> the other possibility is that the scanner updates the branch row... that may be it
[05:00] <poolie> i actually wondered whether to mention it and then decided that "i got a js error with no oops" was boring
[05:01] <poolie> ok; can i do anything to help settle it?
[05:03] <lifeless> I've got it sussed I think
[05:03] <lifeless> https://bugs.launchpad.net/launchpad/+bug/816235
[05:03] <_mup_> Bug #816235: Bug:EntryResource:linkBranch timeouts due to branch scanner transaction length <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/816235 >
[05:03] <poolie> it reminds me a bit of some other bug i filed
[05:03] <poolie> i forget the number
[05:03] <poolie> where what should have been a trivial action timed out after 9 seconds
[05:06] <lifeless> yes
[05:06] <lifeless> I've been of the opinion that backend scripts are a problem for a while
[05:07] <lifeless> but 'contention' was never a very satisfying explanation
[05:07] <poolie> hm
[05:07] <poolie> i can't actually find that
[05:07] <lifeless> and I can hardly expect to get a cultural change to something that isn't quite as easy on the basis of something that isn't satisfying :)
[05:08] <poolie> is there {waves hands} a way to run the branch scanner at a different isolation level or something?
[05:08] <lifeless> wouldn't help
[05:08] <lifeless> this is because we have foreign keys
[05:10] <lifeless> the appserver commit cannot progress until it obtains a lock preventing anyone deleting or changing the id on the branch row.
[05:11] <lifeless> pg does row locks, I don't think it does row,column locks
[05:11] <lifeless> there are some design things we can do
[05:12] <lifeless> putting status-tracking data in separate tables, for instance
[05:12] <lifeless> but the biggest is short transactions
[05:12] <wgrant> UbuntuSourcePackageNameWidget :(
[05:12] <wgrant> Can I add a precommit hook forbidding any diffs matching [Uu]buntu?
[05:13] <lifeless> no
[05:13] <lifeless> you need a cultural change I think
[05:13] <lifeless> or rather, yes, if you give me the db one for pqm at the same time ;)
[05:13] <wgrant> Heh
[05:13] <poolie> anyhow, the things i saw, of should-be-tiny transactions timing out, would be consistent with that kind of long running lock holding transaction
[05:14] <poolie> perhaps also with other things like a thrashing db, but in that case i presume we'd notice
[05:14] <lifeless> indeed
[05:15] <wgrant> Anybody up for reviewing two ~100 line branches?
[05:15] <lifeless> sure
[05:15] <wgrant> https://code.launchpad.net/~wgrant/launchpad/transitionToTarget-validate_target/+merge/69189 and https://code.launchpad.net/~wgrant/launchpad/launchpadtargetwidget-prefix-respect/+merge/69187
[05:30] <stub> lifeless: another one for the 'long transactions are evil' bucket.
[05:30] <lifeless> yes, exactly
[05:30] <lifeless> that and DRI :P
[05:33] <stub> lifeless: not sure why the branch scanner needs long transactions. It could probably run in autocommit mode even, provided it remembered to insert rows in earliest -> latest revision order.
[05:34] <wgrant> It already partially commits sometimes.
[05:34] <wgrant> Because if the initial scan fails it needs manual cleanup.
[05:36] <lifeless> it probably updates the last_scanned field too early
[05:36] <lifeless> or something
[05:43] <lifeless> stub: how hard would it be to make the transaction reaper kill *updating* transactions over 2.5 seconds long ?
[05:44] <wgrant> Goodbye bug modifications.
[05:48] <stub> lifeless: We could kill transactions that have been trying to execute an UPDATE for longer than 2.5-3.0 seconds (0.5 seconds range, as the field we need is only updated every 500ms). Not sure if that would help anything though.
[05:48] <stub> lifeless: We can't kill transactions that issued an update earlier in their transaction and might be blocking others.
[05:49] <lifeless> stub: because idle in transaction might be a non-mutating request ?
[05:49] <stub> yes
[05:49] <lifeless> stub: we could introspect pg_locks ? <evil face>
[05:50] <stub> Better off doing all this with query timeouts.
[05:50] <lifeless> yeah
[05:50] <lifeless> was really just speculating
[05:50] <stub> Less chance of blowing off your whole leg.
[05:58] <lifeless> see -dev for some rambling about this :)
[06:00] <poolie> oh, i remember
[06:01] <poolie> lifeless: the thing i was thinking of was the failure of the ssh server to talk to the internal xmlrpc
[06:01] <poolie> xmlrpc was failing to look up a branch (iirc)
[06:05] <lifeless> stub: francis is excited by our progress :)
[06:05] <poolie> ah, it's not a readonly query though
[06:08] <lifeless> which bug ?
[06:08] <poolie> https://bugs.launchpad.net/bugs/294726
[06:08] <_mup_> Bug #294726: "ProtocolError for xmlrpc.lp.internal:8097/branchfilesystem" when pushing a branch <lp-code> <Launchpad itself:Triaged> < https://launchpad.net/bugs/294726 >
[06:08] <poolie> i replied
[06:09] <poolie> especially #4
[06:14] <poolie> lifeless: these are interesting cases because of course a person is quite likely to want to link a branch, or push it again, at the time it's being scanned
[06:14] <lifeless> yup
[06:15] <wgrant> Or create an MP...
[06:15] <wgrant> That's where I see it most.
[06:15] <poolie> right
[06:15] <poolie> also these seem like fairly interesting things to put into a message queue or into a non sql database
[06:15] <poolie> though, probably that's not the best short term step
[06:16] <wgrant> Launchpad has very few good short term steps :)
[06:16] <stub> lifeless: Bug is the other table with similar issues to Branch. Want to move heat and heat_last_updated to a separate table, probably the other timestamps and caches too.
[06:16] <lifeless> yup
[06:17] <lifeless> (I agree, I haven't thought deeply about where or how.
[06:17] <lifeless> e.g. should different fields be in separate tables
[06:17] <lifeless> or a k:v table we get row locks on
[06:17] <lifeless> or ...
[06:17] <poolie> i saw your comment reccently about "generally lp doesn't want ...options"
[06:17] <poolie> which i agree with in many ways
[06:17] <poolie> however it was in the context of notifications
[06:18] <poolie> and i do think a fb or g+ style "do/don't tell me about X action" per user would be ok, don't you?
[06:18] <lifeless> perhaps
[06:18] <lifeless> however, we have a bug asking for reasonably comprehensive subscriptions to teams
[06:19] <lifeless> which is worth doing directly I think, rather than an awkward one off in a corner
[06:19] <poolie> what does "subscription to teams" mean?
[06:19] <poolie> telling you about changes to memberships?
[06:19] <lifeless> and possibly other metadata
[06:25] <lifeless> stub: 0.0.2 of pgbouncer is in the download cache for you
[06:25] <stub> ta
[06:26] <lifeless> stub: I have to run, but I'd love to know whats left to do a no-op downtime run ?
[06:26] <lifeless> stub: I am guessing docs and landing last nights branch ?
[06:26] <stub> lifeless: We need to push out configs to make everything go via pgbouncer.
[06:27] <stub> lifeless: Apart from that, we can test now. Last nights branch is an improvement, but not necessary. And docs I prefer to write after I've confirmed everything works rather than having to adjust them for reality later :-)
[06:27] <lifeless> \o/
[06:27] <stub> pgbouncer configs is the slow step
[06:27] <lifeless> I've blogged about this, as promised
[06:28] <lifeless> so users won't be shocked
[06:28] <wgrant> Does this mean we need downtime for all downtime services in the next couple of days?
[06:28] <wgrant> (librarian being the main problem)
[06:29] <lifeless> ah the uploader ? yes we do
[06:29] <stub> Think librarian actually reconnects happily - did on staging anyway when I ran this stuff.
[06:29] <wgrant> We restarted it two weeks ago without hiccups, but still, something to consider.
[06:30] <lifeless> stub: wgrant is referring to migrating it to use pgbouncer
[06:30] <lifeless> stub: uploads to it are not load balanced at the moment
[06:30] <stub> Ahh, yes. But outages will be minor (test connectivity first, then bounce).
[06:31] <stub> Is the upload port going via haproxy?
[06:31] <wgrant> Not yet.
[06:31] <wgrant> Firewall rules are unclear.
[06:32] <stub> it shouldn't be a problem being generous with access to the upload port. Not much you can do apart from try and fill up our disk.
[06:32] <StevenK> There is no try
[06:32] <StevenK> They *can*
[06:33] <lifeless> uploads go via appservers
[06:33] <lifeless> so an interrupted librarian results in an appserver OOPS
[06:33] <lifeless> its not a big deal
[06:34] <wgrant> Uh.
[06:34] <wgrant> Yes, let's pretend that scripts don't exist :)
[06:35] <lifeless> they can be queisced prior to the librarian bounce
[06:35] <wgrant> True.
[06:35] <lifeless> stub: sounds like the config update is the critical path thing then; if you need anything from me on it, let me know, otherwise its in your court
[06:36] <wgrant> So, codehosting, ftpmaster, librarian, mailman are the downtime services I know of.
[06:36] <wgrant> But mailman doesn't have DB access, I suppose.
[06:57] <jtv> hi lifeless—almost 9AM here now.  What's up?
[07:05] <rvba> Good morning.
[07:06] <jtv> good morning rvba!
[07:07] <jtv> StevenK, wgrant: mind if I update dogfood and run generate-contents-files?
[07:07] <wgrant> Please do.
[07:12] <jtv> rvba: dogfood conflict in package cloner.  Were you running an experimental branch there?
[07:21] <henninge> danilos: Hi! ;)
[07:25] <jtv> morning, former teammates
[07:26] <henninge> Hi jtv!
[07:26] <henninge> jtv: What do you think should be done to fix bug 504062?
[07:26] <_mup_> Bug #504062: External suggestion "Used in" links to disabled template <404> <lp-translations> <oops> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/504062 >
[07:27]  * jtv looks
[07:27] <henninge> jtv: Should non-admins not see the suggestion at all or just don't get a link to the template?
[07:28] <rvba> jtv: sorry, was afk ... I might know what the problem is ...
[07:28] <rvba> jtv: how can I take a look? (I don't have access to DF)
[07:28] <jtv> rvba: ah.  Hang on, have to look at something else now.
[07:28] <poolie> spiv: re your question on bug 813349, people filed a couple of nit type bugs
[07:28] <_mup_> Bug #813349: hard to get from mp to per-revision diffs <code-review> <javascript> <qa-ok> <ui> <Launchpad itself:Fix Committed by spiv> < https://launchpad.net/bugs/813349 >
[07:28] <poolie> like your link colour is wrong
[07:28] <poolie> personally i wouldn't block on them
[07:29] <jtv> henninge: istm the reason for showing suggestions from disabled templates is mostly obsolete thanks to sharing.
[07:30] <jtv> henninge: we could try getting some data — how often are those translations both different and useful?
[07:30] <jtv> If the answer is "rarely," then the solution is simple.
[07:31] <jtv> (Well, apart perhaps from lag between the time the template is disabled and the next update of the suggestive-templates cache).
[07:32] <jtv> rvba, here's the conflict: http://paste.ubuntu.com/652250/
[07:34] <jtv> henninge: if it turns out we're getting real benefit from disabled templates, then we'd have to worry about disabling links.  I even think Zope would do that for us if this were a permission error rather than a 404, but at a performance cost.
[07:35] <adeuring> good morning
[07:35] <henninge> HI adeuring!
[07:35] <adeuring> hi henninge!
[07:36] <rvba> jtv: It's Julian SQL optimisation from yesterday. It looks to me like he tried it manually on DF, then he got it properly landed because (I think) all the code that causes the conflict is there already.
[07:36] <henninge> jtv: Hm, I'd like to keep this simple ...
[07:36] <jtv> rvba: in that case, I should be able to revert & pull.  That's great.
[07:36] <rvba> jtv: I think so yes.
[07:37] <jtv> henninge: I sympathize, but if it were truly simple maybe this bug would've been fixed by now.  :)
[07:37] <henninge> jtv: Before I read this I would have expected translations from disabled templates to not show up
[07:38] <jtv> Yes, I thought they were excluded from the suggestive-templates cache.  Have you been able to reproduce the problem?
[07:38] <henninge> jtv: on qastaging, yes. But I am not sure about the result being cached.
[07:39] <henninge> jtv: how long does stuff live in that cache?
[07:39] <spiv> poolie: hmm, it'd be nice to have those bugs linked to 813349 perhaps, but probably not important if they're not blockers
[07:39] <henninge> also, I never worked with the cache before.
[07:40] <jtv> henninge: the suggestive templates cache is rewritten in its entirely… I think daily at the moment, but may be much more frequently.  We could afford to do it much more often than that.
[07:40] <spiv> poolie: my main concern is that the bug doesn't languish indefinitely as Fix Committed.
[07:41] <jtv> henninge: it's just a database table containing the ids of POTemplates that can be used to get suggestions from.  And it does indeed exclude disabled templates.
[07:41] <jtv> (Just checked)
[07:41] <poolie> spiv so fortunately 'new bugs' should find them
[07:41] <jtv> henninge: so whenever a template is disabled, this problem can exist until the cache is rewritten.
[07:41] <henninge> ah!
[07:41] <jtv> henninge: (Also, I see it hasn't been updated for usage enums)
[07:42] <jtv> henninge: The bug description therefore would seem a little out of date.
[07:42] <henninge> yes
[07:43] <henninge> jtv: so the problem is really just about how to deal with an invalid cache.
[07:43] <jtv> Yes, it's cache invalidation — one of the two things that are hard.
[07:43] <henninge> ;)
[07:44] <jtv> We could afford to update the cache whenever anyone deactivates a template.
[07:44] <jtv> IIRC it takes only a fraction of a second to find all those templates; it's just that we were doing it multiple times for every translations page.
[07:45] <jtv> If the suggestions queries also join in the POTemplate, we could just add a redundant iscurrent check.
[07:46] <poolie> spiv if only there was automatic connection between related bugs :)
[07:46] <henninge> jtv: yes, those were the two things I was considering right now.
[07:46] <jtv> Alas, the query doesn't include POTemplate.
[07:47] <henninge> jtv: I just wasn't sure how expensive the joining would be.
[07:47] <jtv> And this is velly velly pelformance-snsitive.
[07:47] <henninge> yup
[07:47] <jtv> So best not add that join.
[07:47] <henninge> jtv: but removing that entry from the cache should be simple.
[07:48] <jtv> What's really weird by the way is that the code in _getExternalTranslationMessages seems to fetch POTMsgSet ids and call them "pots."
[07:48] <poolie> spiv, added
[07:48] <jtv> Maybe that's just someone who wasn't familiar with the conventional abbreviations.
[07:49] <jtv> henninge: it looks like someone's made a mess of that code.  There's also an over-long line of SQL, with the keywords not capitalized.  That smells of someone just trying something out and accidentally landing it as soon as it worked, without review.
[07:49] <spiv> poolie: thanks
[07:50] <jtv> henninge: ah no, it's lifeless code.
[07:51] <henninge> jtv: how is that different?
[07:52] <jtv> This is hard to say diplomatically.
[07:52] <jtv> In Launchpad we have traditionally focused on producing code that's easy to read.
[07:52] <jtv> Robert leans more towards keeping it easy to write.
[07:52] <henninge> ...
[07:52] <jtv> So I often struggle reading his code.
[07:53] <jtv> For that reason I don't think it saves any time, but opinion seems divided on the subject.
[07:53] <henninge> ...
[07:53] <jtv> Still, you would have expected a "make lint."
[07:53] <henninge> very much so
[07:53]  * spiv offers hennige a …, for variety
[07:53] <henninge> thanks spiv ;)
[07:53] <jtv> Also, try <compose>..
[07:54] <spiv> jtv: how do you think I typed it? :)
[07:54] <jtv> spiv: I do apologize.  :)  I only checked Henning's.  :)
[07:54] <spiv> Or perhaps I should say: ☺
[07:54] <jtv> I ♥ <compose>
[07:54] <henninge> …
⁺¹
[07:55]  * henninge learnt something
[07:55] <jtv> "You've taken your first step into a larger world…"
[07:55] <henninge> ¹²³¼½
[07:56] <henninge> I already know those …
[07:56] <henninge> s/know/knew/
[07:57]  * spiv offers jtv <compose><"
[07:57] <jtv> Oooh!  Thank you!
[07:57] <jtv> You know what's going to happen now, don't you?
[07:57] <jtv> “Quote-mania.”
[07:58] <spiv> ¡Por supuesto! ;)
[07:58] <jtv> wøñđêřfüł
[08:01] <jtv> morning bigjools
[08:02] <bigjools> morgen
[08:02] <jtv> bigjools: I'm just messing around on dogfood right now, so beware
[08:02] <bigjools> jtv: eek :)
[08:04] <jtv> Ah-HAaah.  No "idle in transaction" no more for generate-contents-files.
[08:06] <jtv> Deployments to cocoplum do take downtime, right?
[08:06] <wgrant> Yes. We'll probably need one in the next day or so for config updates.
[08:07] <wgrant> stub: Any idea on the timing of that?
[08:07] <stub> wgrant: nup
[08:07] <stub> asap, but out of my hands now (apart from nagging and whining)
[08:11] <jtv> hloeung: about https://code.launchpad.net/~jtv/lp-production-configs/config-810989/+merge/68867 — shall we just merge & deploy it then?
[08:11] <mrevell> Hi
[08:12] <jtv> hi mrevell
[08:13] <jtv> wgrant: any idea whether staging and qastaging have the ubuntu-contents dir?
[08:13] <jtv> In other words, whether we ever run generate-contents-files on them (whether from cron or for testing)?
[08:13] <wgrant> jtv: They should not.
[08:14] <jtv> Then I'll not worry about it.
[08:14] <wgrant> AFAIK nobody has ever been foolish enough to run a publisher on them.
[08:17] <jtv> huwshimi: thanks for the icon by the way.  I'll just bung it into l/c/l/images and update the code to use it.
[08:17] <huwshimi> jtv: Great, no problems.
[08:22] <henninge> jtv: so, should I put the removal into the admin view code (which AFAIK is the only place where templates can be disabled) or in the model code (i.e. introduce a setter).
[08:23] <henninge> jtv: I'd prefer model code I think
[08:23] <jtv> Absolutely.
[08:23] <henninge> cool, great to see we agree …
[08:24] <jtv> At this point it may possibly be worth reviving the utility.
[08:26] <henninge> jtv: which utility?
[08:26] <jtv> I originally wrote a utility for this cache, but IIRC the reviewer thought it was too much.
[08:27] <jtv> Up to you to draw the line of where it deserves its own utility though.
[08:28] <henninge> jtv: oh yes, I noticed there is no model code for it.
[08:28] <jtv> Exactly.
[08:29] <jtv> If you want to add that, just be sure to define the template id as the primary key.
[08:29] <jtv> I don't think we really need it though.
[08:30] <henninge> I don't either ...
[08:33] <henninge> quick question
[08:33] <henninge> getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
[08:34] <henninge> is now equivalent to
[08:34] <henninge> IMasterStore(something)
[08:34] <henninge> right?
[08:34] <henninge> oh, I don't have "something" here ...
[08:40] <jtv> henninge: no, default flavor is IStore(something).  May be a class.
[08:40] <jtv> IMasterStore is the master flavor.
[08:40] <henninge> I see
[08:46] <jelmer> gmb: Thanks for the re-review and ec2 submission
[08:46] <gmb> np
[09:38] <lifeless> bigjools: you know how I said 10m was ok for a backend script the other day ?
[09:38] <bigjools> yeeaasssss?
[09:39] <lifeless> I was wrong; see my performance-tuesday mail from today :)
[09:39]  * bigjools reaches for razor blades
[09:39]  * bigjools is not sure whether to use them on himself or lifeless
[09:40] <lifeless> jtv: I pinged you when I was looking at some locking / concurrency/contention innards of postgresql
[09:40] <lifeless> jtv: for a teddy bear, but I sorted itout
[09:42] <wgrant> lifeless: Well, it should be OK for some things :)
[09:43] <bigjools> lifeless: I'd say that doing txns in less than 2.5 seconds is lala land
[09:43] <bigjools> unless the magic performance fairy visits the packagecopier
[09:44] <bigjools> and somehow we get divine intervention in the soyuz schema and all the code gets rewritten to avoid races and contentions
[09:44] <wgrant> Yes.
[09:45] <wgrant> Some sorts of transactions are OK.
[09:45] <jtv> Teddy bear..?
[09:53] <lifeless> jtv: someone to talk about a problem with
[09:56] <lifeless> bigjools: so, if we can't get to 2.5 seconds consistently, we can never get to 5 seconds reliably if there are any common things being changed.
[09:56] <lifeless> bigjools: its not something we can wave a wand to have happen, but its definitely doable.
[09:57] <bigjools> lifeless: I am OTP but see our conversation this time yesterday
[09:57] <bigjools> for all the scary reasons why this is hard
[09:57] <bigjools> tl;dr = consistency issues
[09:59] <lifeless> bigjools: sure, so the key question is how we can reduce the risk / find ways to assess it.
[09:59] <bigjools> yup
[09:59] <bigjools> we kinda ruled out using staging tables
[10:00] <bigjools> which means fixing the rest of the code.  And we're not sure what needs fixing.
[10:00] <lifeless> the first thing that occurs to me is divide and conquer:
[10:00] <lifeless> break the problem code (whatever it is) into a few categories:
[10:01] <lifeless>  - readonly code that we can tolerate stupidity in - things with ephemeral output like queuing decisions or web ui
[10:01] <lifeless>  - readonly code that must be accurate like archive publication and contents generation
[10:02] <lifeless>  - writing code (which we assume we cannot tolerate stupidity in)
[10:02] <lifeless> then, look for queries that reference distroseries - probably a few hundred tops that do that; toss them into each of those categories as much as possible, and things that are ok we can just move on.
[10:03] <bigjools> yeah, not saying it's impossible, just a tedious and error-prone job
[10:04] <bigjools> I'm interested in re-assessing our model to prevent this crap
[10:04] <bigjools> we need a discussion at the next T'Dome
[10:05] <bigjools> the basic issue is that the model doesn't prevent the inconsistency
[10:06] <lifeless> thats a pretty core ingredient :)
[10:07] <bigjools> lifeless: yes - there's another timebomb as well, I'll tell you about it in -ops
[10:22] <jtv> gmb: review?  https://code.launchpad.net/~jtv/launchpad/bug-800573/+merge/69240
[10:22] <gmb> jtv: Sure
[10:22] <jtv> thanks
[10:23] <gmb> 'Huw "Emo" Wilkins'? Nice :)
[10:26] <gmb> jtv: I've come to expect fairly weighty branches from you. Nothing like breaking stereotypes, eh? Approved.
[10:27] <jtv> gmb: a lot of that is just plain bad luck, I'm afraid.
[10:27] <jtv> Not all.
[10:27] <gmb> :)
[10:27] <jtv> Maybe you're just my reviewer of last resort, the one I end up with when everyone else has turned me down.
[10:39] <jtv> gmb: thanks by the way.  :)
[10:40] <gmb> np :)
[10:46] <jtv> bigjools, cjwatson: is the -commercial pocket completely and utterly gone now that dapper is eol?  Can we drop the publisher provisions (the commercial-compat.sh script)?
[10:47] <jtv> wgrant also said to talk to the releases team, IIRC — if so, what's the right communication channel for that?
[10:49] <cjwatson> hmm
[10:49] <cjwatson> yes, I think it is
[10:50]  * gmb lunches
[10:50] <wgrant> Dapper is officially EOL, but someone may be playing OEM-like tricks with it...
[10:50] <bigjools> jtv: talk to skaet
[10:51] <henninge> jtv: any idea why I might get this? ForbiddenAttribute: ('__call__', <lp.testing.fakemethod.FakeMethod object at 0xdc8d74c>)
[10:54] <cjwatson> jtv: you should also check with Brian Thomason (iamfuzz), since he does 99% of the work on partner
[10:54] <cjwatson> if he's no longer uploading anything to dapper-commercial, then it can die
[10:55] <cjwatson> I expect he'll in fact be more directly up to date on this than skaet will
[10:56] <cjwatson> bigjools: you asked me to prepare that staging PPA with changes intended for the Launchpad PPA (https://lists.launchpad.net/launchpad-dev/msg07686.html) - are you also able to review it?
[10:56] <bigjools> cjwatson: someone on the maintenance cycle should review it, I'm kinda busy.  Perhaps gmb?
[11:03] <jtv> henninge: fakemethod behind a proxy, I think.
[11:03] <jtv> bigjools: thanks
[11:03] <henninge> jtv: yes
[11:03] <henninge> jtv: I was patching a utility. bad idea
[11:04] <jtv> rather
[11:04] <jtv> cjwatson: thanks
[11:23] <jtv> henninge: I guess you're probably already cleaning this up, but just so we don't forget: bug 816366
[11:23] <_mup_> Bug #816366: Misleading code in POTMsgSet <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/816366 >
[11:26] <henninge> jtv: http://paste.ubuntu.com/652366/
[11:27] <jtv> henninge: great, thanks.  Phe!
[11:27] <jtv> Phew.
[11:38] <lifeless> night y'all
[12:05] <gmb> sinzui: Can you take a look at, and perhaps give a cogent response to https://answers.launchpad.net/launchpad/+question/165200?
[12:10] <gmb> henninge, danilos, jtv: Can one of you good fellers take a look at https://answers.launchpad.net/launchpad/+question/165699 please?
[12:13] <danilos> gmb, on it
[12:14] <gmb> Thanks
[12:21] <danilos> gmb, btw, can you take a look at https://code.launchpad.net/~danilo/launchpad/bug-800123/+merge/69264 when you find some time?
[12:21] <gmb> Sure
[12:30] <danilos> gmb, thanks
[12:55] <gmb> danilos: r=me
[12:55] <danilos> gmb, ta
[13:32] <deryck> adeuring, henninge-lunch -- sorry I'm late, coming to standup in a couple minutes.
[13:55] <jtv> gmb: still reviewing?  If so → https://code.launchpad.net/~jtv/launchpad/bug-814141/+merge/69285
[13:56] <gmb> jtv: Sure.
[13:56] <jtv> thx
[14:39] <deryck> bac, hi there.
[14:39] <bac> hi deryck
[14:39] <deryck> bac, I was going to do a nodowntime deploy and if the qa page is correct, your fix for bug 788685 is blocking other qa-ok revs....
[14:39] <_mup_> Bug #788685: Enable translating selected Ubuntu universe packages in Launchpad <escalated> <oem-services> <qa-needstesting> <Launchpad itself:Fix Committed by bac> <OEM Priority Project:Invalid> <Ubuntu Translations:Triaged> <pkgbinarymangler (Ubuntu):Fix Released by pitti> < https://launchpad.net/bugs/788685 >
[14:39] <deryck> bac, any chance of qa'ing that today?
[14:40] <bac> deryck: i'll try.  it isn't straightforward
[14:41] <deryck> bac, I assumed it wasn't.  do you need a former translations persons help?
[14:41] <bac> deryck: luckily i have one.  :)
[14:42] <deryck> bac, right, I knew danilos was on your team. :-)  I didn't know if he was still around to help. :)
[14:42] <deryck> was going to offer my translation guy :)
[14:43] <bac> deryck: i may take you up.  will have a go now.
[14:43] <bac> bigjools: ping
[14:43] <deryck> bac, ok, cool.  Thanks!
[14:43] <bigjools> bac: hello
[14:44] <bac> bigjools: i need to QA bug 788685.  danilos suggested yesterday we do it on dogfood and "ask soyuz people about re-uploading (re-publishing or something) existing ubuntu packages on dogfood"
[14:44] <_mup_> Bug #788685: Enable translating selected Ubuntu universe packages in Launchpad <escalated> <oem-services> <qa-needstesting> <Launchpad itself:Fix Committed by bac> <OEM Priority Project:Invalid> <Ubuntu Translations:Triaged> <pkgbinarymangler (Ubuntu):Fix Released by pitti> < https://launchpad.net/bugs/788685 >
[14:45] <bac> bigjools: would you have a moment or two and does that make sense to you?
[14:45] <bigjools> bac: you can upload to DF and I can run the upload processor
[14:46] <bigjools> you just need to "apt-get source" on an existing universe package and bump its version
[14:46] <bigjools> build the package then dput to DF
[14:46] <bac> bigjools: do i just ask a losa to deploy my branch onto DF?
[14:46] <bigjools> bac: no you ask me :)
[14:47] <bigjools> is it landed?
[14:47] <bac> bigjools: excellent
[14:47] <bac> bigjools: yeah, it is
[14:47] <bac> bigjools: r13510
[14:47] <bigjools> bac: is it on db-devel (we run DF off db-devel)
[14:47] <bac> no
[14:47] <bigjools> if not I'll merge the branch
[14:48] <bigjools> donde esta?
[14:49] <bac> bigjools:  lp:~bac/launchpad/bug-788685
[14:50] <bigjools> bac: ah it was already on db-devel it seems.
[14:50] <bac> oh righty
[14:50] <bigjools> bac: so go ahead and upload, ping me when it needs processing
[14:50] <bac> bigjools: about that...
[14:50] <bac> bigjools: i need to get a source that has translations, right?
[14:50] <bigjools> yes
[14:52] <bigjools> bac: and you need the XS-Ubuntu-Use-Langpack: yes header in debian/control
[14:52] <bac> yes, of course i do.  i was just thinking that
[14:52] <bigjools> and also we need to check that the DF buildds chroots have got the new pkgbinarymangler
[15:00] <sinzui> matsubara, mumble or skype?
[15:00] <matsubara> sinzui, mumble
[15:06] <henninge> gmb: Hi!
[15:06] <gmb> Hi henninge
[15:06] <henninge> gmb: Can you please do a review for me?
[15:06] <henninge> https://code.launchpad.net/~henninge/launchpad/bug-504062-external-suggestions/+merge/69296
[15:06] <gmb> Sure
[15:23] <gmb> henninge: Approved
[15:23] <henninge> gmb: thanks
[15:52] <henninge> gary_poster: Hi!
[15:52] <gary_poster> hey henninge :-) what's up?
[15:53] <henninge> gary_poster: I am looking at bug 481512
[15:53] <_mup_> Bug #481512: race condition when rotating logs <canonical-losa-lp> <escalated> <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/481512 >
[15:53] <gary_poster> henninge, cool
[15:54] <henninge> gary_poster: but I am not familiar with the logging system in LP.
[15:54] <henninge> gary_poster: I hoped that you could give me any input you have on that bug.
[15:54] <gary_poster> henninge, we send a signal which is supposed to rotate the logs.  I found some pertinent code recently; lemme see if I can find it.
[15:55] <henninge> ah
[15:56] <gary_poster> henninge, still looking, but lib/canonical/launchpad/webapp/sigusr2.py
[15:56]  * henninge looks
[15:58] <henninge> ok, basic stuff
[15:58] <henninge> there is a test case for it
[16:02] <gary_poster> henninge, I've looked for more--I thought I saw some custom log handlers but the only ones I can find now are for tests and for scripts (and a simple one that adds custom log levels).  So, I'm afraid ZConfig might be the place to look for that.
[16:02] <gary_poster> Author of that log rotate code in ZConfig is Fred Drake, but I don't think knowing that helps us any :-)
[16:02] <gary_poster> so that's it henninge
[16:02] <henninge> ;-)
[16:03] <henninge> gary_poster: thank, that's a start
[16:03] <henninge> s
[16:03] <gary_poster> cool henninge, thanks for looking at that one
[16:04] <henninge> gary_poster: it was escalated, so somebody has to do it ... ;-)
[16:07] <gary_poster> :-)
[16:46] <bac> deryck[lunch]: making progress
[16:57] <bigjools> good night all
[20:16] <sinzui> jcsackett, do you have time to mumble
[20:17] <jcsackett> sinzui: sure.
[20:18] <jcsackett> sinzui: i'm on mumble now.
[20:44] <LPCIBot> Project devel build #920: FAILURE in 5 hr 39 min: https://lpci.wedontsleep.org/job/devel/920/
[21:07] <lifeless> sinzui: hi
[21:17] <deryck> Later everyone.
[21:27] <sinzui> hi lifeless
[21:28] <lifeless> sinzui: hi; I've forwarded you a mailling list test failure
[21:28] <sinzui> I saw
[21:28] <lifeless> do we use the --notifications option ?
[21:28] <lifeless> and do we want it to override this quieter behaviour ?
[21:28] <sinzui> The list import is an interesting case. I think we can drop it though since
[21:29] <sinzui> lifeless, I really do not know about --notifications
[21:30] <sinzui> I was not certain what was going on in the archive email failure. Do we add users to teams when they are subscribed to archives?
[21:30] <lifeless> sinzui: the test made an open team
[21:30] <sinzui> I really do understand that case then
[21:30] <lifeless> sinzui: so it then purged the mails that adding someone to the team generated. (factory.makeTeam makes OPEN teams)
[21:31] <lifeless> sinzui: e.g. the archive failure was trivial and you can ignore it.
[21:31] <lifeless> sinzui: the comment above the failing line said (paraphrased) 'ignore the mails about joining the team'
[21:31] <sinzui> okay
[21:33] <lifeless> sinzui: so, I'm trying to change the test_mlists test to use a moderated team
[21:36] <allenap> Night all.
[21:37] <sinzui> lifeless, we have run The mailing list import script twice
[21:38] <sinzui> lifeless, it is run by a losa to migrate a list. it create and find users from the email address in the foreign list id
[21:38] <sinzui> lifeless, I think the emails are spam in this case. When the import is done, the team admin is going to inspect the  team subscription page and ask for the log of the import
[21:39] <sinzui> s/id/DB/
[21:39] <lifeless> sinzui: this fixes the test: http://paste.ubuntu.com/652682/
[21:39] <lifeless> sinzui: but if you want, I could rip out the --notifications feature entirely.
[21:40] <sinzui> lifeless, keep the test chnge
[21:40] <sinzui> change
[21:40] <sinzui> I do not think this script will be used until it can be used via the UI
[21:41] <lifeless> ok, so that pastebin is ok with you ?
[21:41] <sinzui> yep. +1
[21:41] <lifeless> I will land this change then.
[23:01] <sinzui> StevenK, mumble?
[23:05] <sinzui> StevenK, http://pastebin.ubuntu.com/652721/
[23:50] <wgrant> These "Job ran too long" scanner OOPSes really suck.
[23:53] <wgrant> sinzui: https://code.launchpad.net/~wgrant/launchpad/transitionToTarget-series-sourcepackage, if you have time.