[00:01] <lifeless> changes to shipit itself? via PQM I believe
[00:01] <wgrant> I mean reviewer-wise.
[00:01] <wgrant> Since it's ISD.
[00:02] <lifeless> dunno
[00:02] <lifeless> is there even a project/
[00:02] <wgrant> There is.
[00:02] <wgrant> That's where the branch lives.
[00:02] <jelmer_> wgrant: last time I changed it I just proposed a merge in Launchpad and then pinged somebody in ISD (Anthony Lenton I believe) about it.
[00:02] <jelmer_> wgrant: He reviewed and took care of landing my changes.
[00:02] <wgrant> :(
[00:02] <wgrant> Thanks.
[00:02] <jml> lifeless: I'd be interested in disagreeing with you about that some time.
[00:03] <jml> lifeless: but not now.
[00:10] <lifeless> jml: about what? [ambiguous that]
[00:10] <jml> lifeless: moving devscripts out of tree
[00:10] <jml> good night!
[00:11] <wgrant> lifeless: Also, you could export id in the interface rather than using rSP.
[00:11] <huwshimi> jml: Night
[00:11] <lifeless> jml: gnight
[00:11] <wgrant> Night jml.
[00:12] <lifeless> wgrant: err, the ids are exported
[00:12] <lifeless> wgrant: they get wrapped, because thats what security proxies do
[00:12] <lifeless> then sqlvalues goes boom
[00:12] <wgrant> lifeless: Huh? ints are not proxied.
[00:12] <wgrant> If it's on the interface it should be fine :/
[00:12] <lifeless> lists of ints are
[00:13] <wgrant> Oh, right, all_distro_archive_ids.
[00:21] <lifeless> oh man
[00:22] <lifeless> launchpad_dev=# SELECT COUNT(*) FROM BugTask JOIN Bug ON BugTask.bug = Bug.id WHERE Bug.id = BugTask.bug AND ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25)) AND BugTask.milestone = 14 AND Bug.duplicateof is NULL;
[00:22] <lifeless> 2
[00:22] <lifeless> launchpad_dev=# SELECT COUNT(distinct bugtask.bug) FROM BugTask JOIN Bug ON BugTask.bug = Bug.id WHERE Bug.id = BugTask.bug AND ((BugTask.status = 10) OR (BugTask.status = 15) OR (BugTask.status = 20) OR (BugTask.status = 21) OR (BugTask.status = 22) OR (BugTask.status = 25)) AND BugTask.milestone = 14 AND Bug.duplicateof is NULL;
[00:22] <lifeless> 1
[00:22] <lifeless> our production milestone bug counts are a little inflated
[00:23] <lifeless> give you one guess why
[00:24] <lifeless> I'm going to go beat my head against a wall for a few minutes
[00:26] <wgrant> Hmm.
[00:26] <wgrant> librariangc creates dozens of cursors.
[00:26] <wgrant> Can anyone see a good reason to keep that?
[00:27] <lifeless> I'd check with stub
[00:27] <wgrant> I can see no benefit over passing a single one around, instead of the connection that it currently passes.
[00:27] <wgrant> k
[00:28] <lifeless> anyone remember where the storm expr for Distinct is
[00:28] <wgrant> .config(distinct=True) isn't good enough?
[00:28] <wgrant> Ah, you want that second query?
[00:28] <lifeless> well
[00:28] <lifeless> I'm not going to fix the current for loop
[00:29] <lifeless> but countBugs also has the same defect
[00:29] <lifeless> it just isn't showing it in the current code path because its constrained to only consider series tasks
[00:29] <lifeless> the reason conjoined masters mess things up is because they have /two/ tasks
[00:29] <lifeless> rather than rendering as two tasks but actually having one.
[00:29] <wgrant> Right.
[00:38] <lifeless> sigh, we definitely need two queries.
[00:38] <lifeless> mmm
[00:38] <lifeless> unless, I group by - bingo
[00:39] <lifeless> mmm
[00:39] <lifeless> no
[00:39] <lifeless> our compiler just isn't good enough, and I think we'll get enough benefit for now with two queries
[01:10] <lifeless> right, thats another second or so off of bug searching in ubuntu
[01:10] <lifeless> + more accurate counts
[01:10] <wgrant> Excellent.
[01:10] <lifeless> [should be marginally cheaper too]
[01:10] <lifeless> we have laze evaluation of milestones too
[01:10] <lifeless> going to look at that in a minute
[01:14] <lifeless> I can has review number two? https://code.launchpad.net/~lifeless/launchpad/bug-717394/+merge/50541
[01:16] <lifeless> poolie: hi, I will be in syd evening of the 4th,5th and through early avo 6th.
[01:16] <StevenK> lifeless: \o/
[01:17] <poolie> hi lifeless
[01:17] <poolie> great! i'll be here too
[01:17] <wgrant> lifeless: milestones are targets now?
[01:18] <lifeless> wgrant: for bug search, yes
[01:18] <wgrant> :(
[01:18] <poolie> 98	-            group_on + (Count(BugTask.bugID),), [], None, params).result_set
[01:18] <poolie> 99	+            group_on + (SQL("COUNT(Distinct BugTask.bug)"),), [], None, params).result_set
[01:18] <poolie> it seems like a storm bug if that's any faster
[01:18] <poolie> but it is?
[01:19] <lifeless> poolie: how  would it be a storm bug?
[01:19] <lifeless> wgrant: I can take that out if it doesn't make sense
[01:20] <poolie> does it generate different sql? only becaues of the 'distinct' constraint?
[01:20] <wgrant> lifeless: It doesn't seem to be used and doesn't comply with the definition of 'bug target' everywhere else.
[01:20] <lifeless> wgrant: it just made sense at the time; I guess rather than setTarget a better name would be 'constrainBy'
[01:20] <wgrant> Right.
[01:20] <lifeless> wgrant: its used in browser/bugtarget.py
[01:21] <lifeless> poolie: yes, different sql
[01:21] <poolie> lifeless, i'm just curious what it does differently
[01:21] <lifeless> SELECT COUNT(*) FROM BugTask -> SELECT COUNT(distinct bugtask.bug)
[01:21] <lifeless> erm
[01:21] <lifeless> SELECT COUNT(bugtask.bug) FROM BugTask -> SELECT COUNT(distinct bugtask.bug)
[01:21] <lifeless> ^ thats the change
[01:22] <poolie> and that's faster in psql?
[01:23] <lifeless> slightly smaller intermediary table
[01:23] <lifeless> the big win is using the countBugs function rather than doing a loop in python
[01:23] <poolie> sure
[01:23] <poolie> there's no way to say 'distinct' in storm?
[01:23] <lifeless> doing a distinct forces a sort
[01:23] <lifeless> but so does grouping
[01:24] <lifeless> so in this case it just lets it return less data internally before the aggregation
[01:24] <lifeless> poolie: you can define an Expr for it
[01:24] <lifeless> but it likes to put () around things
[01:24] <lifeless> and COUNT((Distinct BugTask.bug)) isn't valid SQL
[01:24] <lifeless> s/it/storm/
[01:25] <wgrant> Storm will do distinct, but not in a group by.
[01:25] <wgrant> Oh.
[01:26] <wgrant> I misread.
[01:26] <wgrant> But still, no way to do that in Storm right now.
[01:26] <poolie> i wonder if lp should ban branch renames until more of the bugs related to it are fixed
[01:26] <poolie> eg https://answers.launchpad.net/launchpad/+question/146165
[01:27] <wgrant> Is there more than one bug?
[01:27] <poolie> renaming branches that are stacked upon causes troubel
[01:27] <wgrant> That is the only real bug that I know around renaming.
[01:27] <poolie> renaming branches causes confusing errors in a bunch of cases
[01:30] <lifeless> wgrant: IIRC recipes decided to not link to branches
[01:30] <lifeless> wgrant: because modelling recipes was too hard
[01:30] <lifeless> or something
[01:31] <wgrant> lifeless: False.
[01:31] <wgrant> See SourcePackageRecipeDataInstruction.branch
[01:32] <wgrant> I think we need a SourcePackageRecipeDataInstructionPartToken, or something like that.
[01:32] <lifeless> wgrant: interesting
[01:32] <lifeless> wgrant: so why did that guys branch start barfing? or is it a stacking issue
[01:32] <wgrant> lifeless: Stacking.
[01:32] <wgrant> So, I am all for banning renames of stacked-on branches.
[01:32] <wgrant> Because it causes a lot of trouble.
[01:33] <lifeless> I think thats appropriate
[01:33] <lifeless> but not a desirable constraint per se
[01:33] <wgrant> Sure.
[01:33] <wgrant> But it *will* break things.
[01:33] <wgrant> So we might as well ban it until it's fixed.
[01:33] <spiv> +1
[01:35]  * spiv blinks.  Isn't "product" a deprecated term?
[01:35] <wgrant> Yes.
[01:35] <wgrant> Is this on +newrecipe?
[01:35] <spiv> wgrant: no, on clicking the ajaxy bug task affects to reassign something from bzr to bzr-loom
[01:36] <wgrant> Ah.
[01:36] <spiv> Thanks; I'll file/find a bug.
[01:36] <wgrant> A new one was introduced recently (bug #714536)
[01:36] <_mup_> Bug #714536: Branch:+new-recipe related branches listing has formatting issues <recipe> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/714536 >
[01:36] <wgrant> I don't know a bug for yours.
[01:39] <thumper> spiv: yes
[01:39] <thumper> spiv: there is a bug already
[01:39] <thumper> spiv: bug 714534
[01:40] <_mup_> Bug #714534: Branch:+new-recipe refers to "product series" <recipe> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/714534 >
[01:42] <spiv> Just filed https://bugs.launchpad.net/launchpad/+bug/722412
[01:42] <_mup_> Bug #722412: Changing project of bug task prompts to change/search "product" <Launchpad itself:New> < https://launchpad.net/bugs/722412 >
[01:42] <spiv> thumper: hmm, that bug to my eyes doesn't seem obviously the same
[01:42] <thumper> spiv: ah... differrent one
[01:42] <spiv> thumper: unless it is meant as a global s/product/project/ bug?
[01:42] <thumper> spiv, no it wasn't
[01:43] <thumper> spiv: just my misunderstanding of your bug
[01:43] <wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/use-script-isolation-arg/+merge/50543
[01:44] <spiv> thumper: Heh, I'm guessing you've been working on recipes a lot lately, because I didn't mention them at all :)
[01:44] <thumper> spiv: perhaps it was wgrant mentioning it in passing interleaved with your messages :)
[01:46] <spiv> thumper: he also guessed recipes were involved... I'm detecting a pattern!
[01:46] <thumper> I fixed the bug page as part of some recipe work
[01:46] <thumper> but I didn't notice the product term used
[01:49]  * thumper is a little confused
[01:50] <thumper> lifeless: are you familiar with the DocTestMatches matcher?
[01:52] <thumper> lifeless: nm
[01:53] <lifeless> yes
[01:53] <lifeless> I wrotes it
[01:53] <thumper> the whole point of using the DocTestMatches was to have it normalize the whitespace.  A little frustrating that I have to pass in the flag
[01:54] <lifeless> when I wrote it I wasn't sure what sensible defaults would be
[01:54] <lifeless> if we were to put the flag on by default
[01:54] <lifeless> I think we'd need a way to turn it off too
[01:54] <lifeless> but we could change the default, or you could use functools.partial to make a version that has the flag supplied for all your tests
[01:55] <thumper> hmm...
[01:56] <wgrant> lifeless: Thanks.
[01:57] <lifeless> thumper: idioms that work in functional environments will work well with matches
[01:57]  * thumper off for school run
[01:57] <lifeless> thumper: its a very similar environment
[02:00] <lifeless> ok, that was too easy.
[02:03] <lifeless> if I put a method 'getPillar' on I*Series and I*[things that also have series variants]
[02:03] <lifeless> what would you expect it to do?
[02:03] <wgrant> Isn't there a .parent or similar to do that?
[02:04] <lifeless> ah, but IProduct.getPillar() -> self
[02:04] <lifeless> possibly I want IPillar(thing)
[02:04] <lifeless> do we have that already
[02:04] <lifeless> ?
[02:04] <wgrant> We have IRootContext or similar, which does that.
[02:05] <wgrant> But it's probably not a good idea to reuse it.
[02:05] <lifeless> another review plox https://code.launchpad.net/~lifeless/launchpad/bug-717394-2/+merge/50545
[02:05] <poolie> o/ spiv
[02:06]  * spiv waves
[02:11] <lifeless> wgrant: if you look at my new branch you can see it would be rather more pithy and clear
[02:11] <lifeless> wgrant: I don't know how often we do this
[02:15]  * lifeless headdesks
[02:15] <lifeless> lib/lp/registry/model/distribution.py
[02:15] <wgrant> lifeless: Sure, I would support an IPillar. Take stuff out IRootContext.
[02:15] <lifeless> line 724 - __getitem__
[02:15] <wgrant> Wow.
[02:16] <wgrant> OTOH it does save queries.
[02:16] <wgrant> Hm, except not.
[02:16] <wgrant> But Distribution.series is cached, so yes, it does save queries.
[02:16] <wgrant> Slightly.
[02:16] <poolie> huwshimi, does https://bugs.launchpad.net/launchpad/+bug/722419 ring any bells?
[02:16] <_mup_> Bug #722419: bug title edit buttons go under portlets, can't be clicked <bugs> <css> <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722419 >
[02:17] <lifeless> it triggers the query I'm seeing
[02:17] <huwshimi> poolie: I can reproduce it if that's what you mean?
[02:17] <lifeless> whether it saves queries or not is a little dicy at this point
[02:18] <lifeless> I'm not sure thats a regression, its been like that since ajaxification
[02:20] <huwshimi> poolie: Oh actually that's a dupe
[02:21] <poolie> huwshimi, i guess i was politely asking "i wonder if you broke it" :-)
[02:21] <wgrant> lifeless: I like Distribution._sort_key
[02:21] <poolie> not really a big deal
[02:22] <huwshimi> poolie: haha, ok. No I didn't. It's actually been like that for ages. It has to do with how big your browser window is
[02:24] <poolie> yes
[02:24] <lifeless> wgrant: facedesk
[02:24] <huwshimi> poolie: Have you changed your res or browser size recently?
[02:24] <huwshimi> poolie: Like in the last week :P
[02:24] <lifeless> wgrant: you'll love 722421
[02:25] <lifeless> bug 722421 that is
[02:25] <_mup_> Bug #722421: lib/lp/bugs/model/bug.py _known_viewers causes per-row lookups on public bugs in bug searches. <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722421 >
[02:26] <wgrant> It's also wrong.
[02:27] <lifeless> argh
[02:27] <lifeless> wgrant: no, its not
[02:27] <lifeless> bac broke the optimisaiton
[02:27] <wgrant> lifeless: Pillar owners cannot always see bugs...
[02:27] <lifeless> either way
[02:28] <lifeless> whether its correct to do so or not
[02:28] <lifeless> the defect is real
[02:28] <wgrant> Also some of the manipulations of that cached property seem to be a little screwed.
[02:28] <wgrant>             # The bugtask is unassigned, so clear the _known_viewer cached
[02:28] <wgrant>             # property for the bug.
[02:28] <wgrant>             get_property_cache(self.bug)._known_viewers = set()
[02:28] <lifeless> rev 12156.8.20
[02:28] <lifeless> the clearing is conservative
[02:28] <wgrant> That's not clearing it.
[02:28] <wgrant> That's *emptying* it.
[02:28] <lifeless> indeed
[02:28] <lifeless> so
[02:29] <lifeless> the branch to set assignee as a viewer as broken
[02:29] <lifeless> bug 702429
[02:29] <_mup_> Bug #702429: Pillar owners and private bug visibility <Launchpad itself:Triaged> < https://launchpad.net/bugs/702429 >
[02:30] <lifeless> wgrant: however the contract for it is that it can be an empty set and is added to
[02:30] <lifeless> wgrant: so del() would be equally glitchy
[02:31] <wgrant> Yes, but it would at least crash instead of being wrong.
[02:31] <lifeless> wgrant: its not wrong
[02:31] <lifeless> wgrant: if a user is not in the set the rest of the 'can see' code triggers and it will repopulate
[02:31] <wgrant> Ah, I see.
[02:32] <lifeless> this is fallout from folk still learning about how eager loading and late evaluation interact
[02:32] <lifeless> another 75 queries a page removed
[02:33] <wgrant> Assuming each pillar is distinct.
[02:33] <wgrant> Which will happen approximately never.
[02:34] <lifeless> what was also buggy
[02:34] <lifeless> was the missing TP lookup
[02:35] <lifeless> wgrant: uhm, no, it requeries the *bugtasks* per bug
[02:35] <lifeless> wgrant: so you don't need distint pillars, you just need multiple bugs
[02:35] <wgrant> Oh.
[02:35] <wgrant> Nice.
[02:35] <wgrant> I didn't actually read the bug past the summary, FWIW.
[02:36] <wgrant> I am looking to see if there is a sensible way to put tracebacks in the OOPS format.
[02:36] <wgrant> I suspect not.
[02:36] <lifeless> there isn't
[02:37] <lifeless> I'm kicking off a multi-team requirements list for oops tools
[02:37] <lifeless> give it a couple of days and I should have their needs and be able to write it up for public consumption
[02:37] <wgrant> Great.
[02:37] <wgrant> For now I guess I'll just write invalid OOPSes locally...
[02:40] <huwshimi> poolie: that bug is a dupe of #692291 which was reported in December. I'
[02:40] <_mup_> Bug #692291: Impossible to modify the title of the report <css> <javascript> <lp-bugs> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/692291 >
[02:41] <huwshimi> poolie: I'll mark yours as a dupe. Feel free to elevate the existing bug if you feel the need
[02:52] <poolie> huwshimi, i don't think i changed my screen resolution recently
[02:52] <poolie> hm, i did tweak fontconfig to use the ubuntu mono beta font
[02:52] <poolie> it's possible that changed the layout
[02:55] <poolie> my bug title was better :)
[02:55] <poolie> where do people get these things
[02:56] <huwshimi> poolie: Yes, your description was probably better too
[02:57] <huwshimi> poolie: welcome to the problem of using a combination of relative and fixed sizing :)
[02:59] <wgrant> lifeless: It looks like the problematic change was actually a little earlier... 12126.4.1
[02:59] <wgrant> I don't know why 12156.8.20 seems to have that change too...
[02:59] <huwshimi> poolie: Although the mono font doesn't look like it should have affected that
[02:59] <lifeless> it was meant to be reverted according to the bug bac filed
[02:59] <lifeless> I wonder if someone remerged it in accidentally
[03:00] <wgrant> THe diff in the MP shows it being changed, not restored or removed.
[03:00] <wgrant> Odd.
[03:00] <poolie> maybe i just noticed it today
[03:03] <wgrant> Oh yeah, github's branch browser is nice. We should totally have one that doesn't suck.
[03:06] <wgrant> lifeless: Hmm. The SQL statement value substitution has broken duplicate statement detection completely.
[03:06] <wgrant> That's a bit sad.
[03:06] <wgrant> I think.
[03:06] <wgrant> Or maybe this view just does lots of raw SQL.
[03:08] <lifeless> wgrant: python strings send to python-pgsql are u"foo" - this threw lpoops off for a few days
[03:08] <lifeless> but it should be fixed now
[03:08] <lifeless> whats ths oops
[03:09] <lifeless> ah
[03:09] <lifeless> milestones also trigger lazy evaluation on bug searches
[03:13] <wgrant> Huh.
[03:14] <wgrant> In OOPS-1878QS4 I see lots of queries for people who are message owners... but those comments came out of memcache.
[03:14] <lifeless> the bind variables stuff captures at our layer, not what the serialised form of real sql would be
[03:14] <lifeless> if yo usee what I mean
[03:15] <lifeless> wgrant: makes sense
[03:15] <wgrant> Why?
[03:15] <lifeless> wgrant: memcache is incompatible with eager loading
[03:15] <wgrant> Sure.
[03:15] <wgrant> But this is a separate query for each message.
[03:15] <lifeless> hmm
[03:15] <lifeless> yes
[03:15] <wgrant> Which means something is accessing message.owner.
[03:15] <lifeless> right
[03:15] <wgrant> Something other than the message view.
[03:15] <lifeless> permission checks
[03:15] <wgrant> On a message?
[03:15] <wgrant> Messages have no security.
[03:15] <lifeless> read get_visible_messages
[03:16] <wgrant> wut
[03:16]  * wgrant needs a sharper desk.
[03:16] <lifeless> now you see why I was like \o/ when I got shit working
[03:18] <wgrant> Are you burning that, or should I?
[03:18] <lifeless> wgrant: I took it out of the key path for performance
[03:19] <lifeless> wgrant: by ignoring it for range query purposes
[03:19] <lifeless> wgrant: we need to push it down to the db, delete things that we don't want, etc.
[03:19] <lifeless> wgrant: are you seeing VPC calls ?
[03:20] <lifeless> wgrant: if so, the getMessagesForView eager loading may be incorrect
[03:21] <wgrant> lifeless: There are also VPC calls, which I was about to look into.
[03:21] <wgrant> There are also another 60 person queries unaccounted for, even for anonymous requests.
[03:21] <wgrant> I'm trying to track those down.
[03:22] <lifeless> cool
[04:26] <lifeless> another review plox https://code.launchpad.net/~lifeless/launchpad/bug-717394-4/+merge/50548
[04:44]  * StevenK grumbles at tal
[04:45] <StevenK> It works for one case, but not the other? :-(
[04:49] <StevenK> thumper: Still around?
[04:55] <lifeless> hmm
[04:55] <lifeless> 5
[04:55] <lifeless> branches today
[04:56] <wgrant> Not bad.
[04:56] <lifeless> Should I do more ?
[04:56] <wgrant> Yes.
[04:56] <wgrant> Always.
[04:56] <lifeless> it is nice to be doing stuff
[05:00] <StevenK> lifeless: Can haz a little TAL help?
[05:00] <lifeless> sure
[05:01] <lifeless> how do you like it ?
[05:01] <StevenK> lifeless: I can pastebin a diff and a traceback?
[05:01] <lifeless> sure
[05:02] <lifeless> wgrant: ahha, I think I know why the subscriptions are needed to do bug pages
[05:02] <lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/webapp/menu.py", line 262, in _buildLink
[05:02] <lifeless>     linkdata = method()
[05:02] <lifeless>   File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/structuralsubscription.py", line 338, in subscribe
[05:03] <lifeless>     if sst.userHasBugSubscriptions(self.user):
[05:03] <StevenK> lifeless: Just had a thought, let me try something
[05:03] <wgrant> Hah.
[05:11] <StevenK> lifeless: Right, I have it working, but not how I want
[05:12] <lifeless> garh fail
[05:13] <lifeless> the vocabulary to change bug targets
[05:13] <lifeless> type 'launchpad' in there
[05:13] <StevenK> lifeless: http://pastebin.ubuntu.com/569920/
[05:13] <StevenK> lifeless: Looking at the page in my browser, I get: "Built by recipe generic-string548131 for Person-name548129" which is brilliant, but I'd like the recipe to be a link, and it isn't.
[05:13] <lifeless> StevenK: and what do you want it to do?
[05:14] <lifeless> you don't want tal:replace
[05:14] <StevenK> lifeless: And given this is first time ever touching tal, I suspect I'm missing something fundamental
[05:14] <lifeless> replace replaces a tag by something
[05:14] <lifeless> e.g. structure, a string or a context
[05:14] <lifeless> you want to set the href
[05:15] <lifeless> e.g.
[05:15] <lifeless> tal:attributes="href context/bug/duplicateof/fmt:url">bug
[05:15] <lifeless>              #<span tal:replace="context/bug/duplicateof/id">42</span></a>
[05:15] <lifeless> bah
[05:15] <lifeless> <a
[05:15] <lifeless> before that
[05:15] <StevenK> Right
[05:18] <StevenK> lifeless: Excellent, thanks
[05:18] <lifeless> cool
[05:18] <lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/722455 may entertain you, amidst more headbanging
[05:18] <_mup_> Bug #722455: bug task target selector widget cannot be used to select 'launchpad' <regression> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722455 >
[05:19] <wgrant> lifeless: Oh, that's even more awesome than last time I tried.
[05:20] <StevenK> lifeless: It does the same thing for 'launchpad-project' too
[05:20] <lifeless> win
[05:20] <lifeless> though launchpad-project can't have bugs, it would be nice to list all the elements of it there
[05:21] <StevenK> wgrant: Did you see https://bugs.launchpad.net/launchpad/+bug/722344 ?
[05:21] <_mup_> Bug #722344: PPA traversal assumes distribution is ubuntu <Launchpad itself:New> < https://launchpad.net/bugs/722344 >
[05:21] <wgrant> StevenK: Yes.
[05:21] <wgrant> Let's design URLs without the distribution in them! It's going to be awesome!
[05:21] <StevenK> And no wails of dispair. Awesome.
[05:22] <lifeless> ok
[05:22] <lifeless> why does bugtask need the main archive and archive admin person as json
[05:23] <wgrant> Hah.
[05:23] <lifeless> however
[05:23] <lifeless> I've now audited distro bug search
[05:23] <wgrant> I didn't realise we pracached the context as JSON.
[05:24] <lifeless> that might explain the what of it. Tho not the why
[05:32] <StevenK> lifeless: http://people.canonical.com/~stevenk/built-by-recipe.png
[05:33] <lifeless> StevenK: that looks pretty nice, but why is the word 'for' in ?blue?
[05:34] <StevenK> lifeless: The whole thing is one link to the recipe
[05:34] <lifeless> I wouldn't link the 'for' or person name then
[05:34] <StevenK> Right
[05:34] <lifeless> I think folk would expect the person to be a link to the person, if anything
[05:35] <StevenK> Okay, fixed
[05:35] <StevenK> lifeless: Just wanted your thoughts before I put together an MP and beg for a UI review
[05:36] <lifeless> StevenK: de nada
[05:36] <StevenK> (My first, surprisingly)
[05:36] <wgrant> UI review? Surely you jest.
[05:36] <StevenK> wgrant: I have your manager on the phone, he wants a word.
[05:37] <StevenK> :-P
[05:37] <wgrant> TALES, please stop swallowing every exception under the sun with LocationError...
[05:38] <StevenK> wgrant: Ah yes, I've been hitting that today too
[05:38] <wgrant> It even swallows NameErrors.
[05:38] <wgrant> That I did not expect.
[05:38] <StevenK> "Feature"
[05:39] <wgrant> I bet it would swallow SyntaxErrors if they weren't fatal to compilation...
[05:39] <StevenK> wgrant: Try it and raise one
[05:41] <lifeless> StevenK: whats the milestone for ?
[05:42] <StevenK> lifeless: Which milestone?
[05:42] <lifeless> the one you set on the launchpad selection bug
[05:42] <StevenK> If it was 722455 that was by accident
[05:43] <lifeless> indeed, it is
[05:43] <StevenK> wgrant already fixed
[05:43] <lifeless> well
[05:43] <lifeless> wgrant: why the milestone?
[05:43] <StevenK> Haha
[05:43] <wgrant> Assuming StevenK's assignment to the milestone indicated urgency, 11.04 was clearly a mistake.
[05:43] <wgrant> For that is some time away.
[05:43] <lifeless> ah
[05:44] <lifeless> so milestones aren't part of our what to work on workflow
[05:44] <StevenK> False assumption, sadly
[05:44] <lifeless> I really need something that will flash my browser window when an SSL handshake occurs
[05:45] <wgrant> The added latency isn't enough of a stab in the face?
[05:46] <lifeless> I'm in NZ
[05:47] <lifeless> and we have other queue points
[05:47] <lifeless> so it can be hard to be /sure/
[05:47] <lifeless> stub: good morning!
[05:47] <lifeless> stub: you're down as asiapac OCR for mondays.
[05:47] <stub> yo
[05:47] <stub> I am?
[05:47] <lifeless> stub: so I'm going to pounce on you now:
[05:47] <StevenK> Read as: "I have latency to everything, except if it's hosted in my house."
[05:47] <lifeless> apparently
[05:47] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-636158/+merge/50534
[05:48] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394/+merge/50541
[05:48] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394-2/+merge/50545
[05:48] <stub> I never entered the program, and in general spoke out against it. I've never been OCR :)
[05:48] <lifeless> https://code.launchpad.net/~lifeless/launchpad/bug-717394-4/+merge/50548
[05:48] <stub> Oh... OCR... different acronym. Yes
[05:48] <lifeless> stub: https://dev.launchpad.net/ReviewerSchedule is where I found this, if you need to change it
[05:48] <lifeless> stub: anyhow, I can has some reviews?
[05:48] <stub> Nah... just too early to deal with TLAs
[05:49] <stub> Yes, I am OCR :)
[05:51] <lifeless> cool
[05:53] <lifeless> I think I have ubuntu context bug searches down to a constant number - 40 queries (contrast with 118 on lpnet)
[06:03] <lifeless> hmm
[06:03] <lifeless> we need more qa
[06:07] <wgrant> Oh?
[06:07] <wgrant> We do, but do you have a specific example?
[06:08] <lifeless> we're only half qaed
[06:08] <lifeless> also
[06:08] <lifeless> can you check my deployment request for other gotchas in your feature flag changing patch
[06:09] <wgrant> Ah, I'm blocking on a LOSA for mine :(
[06:10] <wgrant> lifeless: Doesn't the value need to be ''?
[06:10] <lifeless> wgrant: it gets stripped
[06:10] <wgrant> Oh, handy.
[06:10] <lifeless> long as there is a trailing space it will be good
[06:10] <lifeless> I'm /pretty/ sure
[06:11] <wgrant> Let's see.
[06:11] <lifeless> flag, scope, priority_str, value = re.split('[ \t]+', line, 3)
[06:11] <wgrant> Indeed.
[06:12] <wgrant> So it all looks good.
[06:25] <lifeless> stub: btw, mthaddon says that apache can be told to use one haproxy as failover
[06:25] <lifeless> so we don't need to chain the haproxies at al
[06:25] <stub> yay.
[06:59] <lifeless> wgrant: ping
[07:03] <wgrant> lifeless: Hi.
[07:11] <lifeless> -> other
[07:15] <lifeless> oh I love getting ISE's from wiki.u.c
[08:15] <thumper> https://code.launchpad.net/~thumper/launchpad/choosing-recipe-name/+merge/50530 anyone... anyone...
[08:15] <thumper> not that I'll be around to respond...
[08:19] <wgrant> stub: Hi.
[08:19] <stub> yo
[08:20] <wgrant> stub: I started destroying Zopeless yesterday. I got rid of it from most scripts and got all passing their tests except librariangc.
[08:20] <wgrant> It passes around a connection internally and creates dozens of cursors from it.
[08:20] <wgrant> And its tests use this fact to maintain parallel transactions.
[08:21] <wgrant> This doesn't work so well when everything is using the global transaction manager.
[08:22] <stub> I don't recall doing that but I guess I must have :)
[08:22] <wgrant> I think I can sensibly collapse it into one cursor and transaction.
[08:22] <wgrant> But I was wondering if there was a reason I should not.
[08:22] <stub> Can we just steal the connection from Storm internals instead of Zopeless internals?
[08:23] <stub> If the tests pass, it is fine.
[08:23] <stub> Oh...
[08:25] <stub> wgrant: Each step looks like it commits at the end already. But changing this to use transaction.commit() and Storm's internal connection fails?
[08:25] <wgrant> The tests seem very crufty and are possibly already not testing what they should, since the transaction lifetimes are identical and implicit begin is on globally.
[08:26] <wgrant> stub: It's not that it fails. It's that I'm scared I'm missing a reason for this strangeness.
[08:28] <StevenK> wgrant: How big is that diff, out of interest?
[08:29] <wgrant> StevenK: The first step (which I'm about to re-EC2) removes use of specialised Zopeless features from scripts. It's 400 lines.
[08:30] <wgrant> The big diff is going to be in making the tests work on the non-ZTM.
[08:31] <wgrant> The second branch is renaming Zopeless mail and detaching it from ZTM. Then another branch introduces a new database configuration thing and shifts scripts onto that.
[08:31] <wgrant> Then I just need test helpers to replace switchDbUser.
[08:31] <wgrant> And that will be done.
[08:31] <wgrant> Because production code doesn't use it any more.
[08:31] <wgrant> *cough* buildd-manager
[08:32] <wgrant> I was surprised at how few test failures there were after replacing initZopeless and ZTM with 10 lines...
[08:33] <stub> wgrant: The whole process runs serially, so there won't be odd fallout. If it is wierd, it is because it is some of the oldest code in our codebase.
[08:33] <lifeless> stub: btw, for your interest - https://code.launchpad.net/~lifeless/launchpad/rabbit/+merge/50488 - failed on ec2, but the diagnostics were insufficient
[08:34] <stub> The test might be crufty, but they have good coverage.
[08:34] <wgrant> stub: Right, but some comments in the tests suggest that it is trying to check its transaction usage.
[08:34] <wgrant> But that is already false.
[08:34] <wgrant> So I'm not loosening them any further.
[08:35] <stub> wgrant: Which tests are these? I don't see it.
[08:36] <wgrant> stub: ftests/test_gc.py
[08:36] <wgrant> It does lots of explicit begins and aborts while saying "merge_duplicates should have committed"
[08:37] <wgrant> It may just be explaining that it is expected, or it may be explaining that the following lines verify that. I can't really see how, but I am going to be paranoid about something like this :)
[08:37] <stub> That is a comment why the following ztm dance is needed I think (I don't think that was me - I always use implicit begins)
[08:38] <wgrant> Right.
[08:38] <wgrant> Thanks.
[08:39] <stub> Other transaction stuff seems to be to deliberately break things, leaving disk and db inconsistent. Most of these have already been switched to using transaction.abort()
[08:50] <adeuring> good morning
[09:01] <lifeless> stub: thanks for the reviews
[09:14] <bigjools> morning
[09:17] <henninge> Hi wallyworld!
[09:17] <wallyworld> hi there
[09:17] <henninge> wallyworld: someting changed in your branch, the link to create a recipe is gone.
[09:17] <henninge> wallyworld: do you know where it went? I guess that is from something you merged?
[09:18] <wallyworld> henninge: the build now branch?
[09:18] <henninge> yes
[09:18] <henninge> I go to https://code.launchpad.dev/~vcs-imports/evolution/main
[09:18] <wallyworld> i'll check the code - it's possible. tim also checked in stuff which touched the same pt file
[09:19] <wallyworld> and there were conflicts
[09:19] <henninge> and it used to have a "Related source package recipes" section in r12389 but that is fone now.
[09:19] <wallyworld> so perhaps i made a mistake dealing with those
[09:19] <henninge> s/fone/gone/
[09:19] <wallyworld> give me 1/2 an hour or so - i've got to get the kids to bed :-)
[09:21] <mrevell> Hello
[09:22] <henninge> wallyworld: sure ;-)
[09:22] <henninge> Hi mrevell ;)
[09:54] <jml> good morning
[10:01] <henninge> Hi jtv!
[10:01] <henninge> jtv: how many more mornings at this time UTC?
[10:01] <jtv> hi henninge… just this week
[10:05] <wallyworld> henninge: i think i remember what happened - tim landed a branch which put all recipe stuff behind a feature flag
[10:05] <wallyworld> you need to do this: insert into FeatureFlag (flag, priority, scope, value) values ('code.recipes_enabled', 1, 'default', 'o
[10:06] <wallyworld> n')
[10:06] <henninge> wallyworld: I figured something like that ;-)
[10:06] <henninge> let me try
[10:06] <wallyworld> ok
[10:14] <henninge> wallyworld: looks perfect! r=me (ui) ;-)
[10:15] <wallyworld> henninge: \o/ thanks!
[10:31] <henninge> wallyworld: the MP is timing out for me atm.
[10:32] <henninge> I will add the approval later.
[10:32] <wallyworld> henninge: np. launchpad has been slooooow for me today too
[10:32] <bigjools> something is horribly borked in the DB
[10:32] <wallyworld> henninge: thanks for all the effort you put into reviewing it
[10:32] <wgrant> henninge: Which URL timed out?
[10:32] <wallyworld> bigjools: is borked the technical term for fucked?
[10:33] <bigjools> wallyworld: fucked *is* the technical term
[10:33] <wallyworld> :-)
[10:36] <jtv> it's spelled b0rked
[10:36] <jtv> henninge, danilos: did you see the problem with the wiserearth zh_cn stats?
[10:36] <henninge> nope
[10:36] <danilos> jtv, I didn't
[10:37] <jtv> Reporting 95.x% translated, whereas in reality it's just under half.
[10:37] <jtv> I wonder if maybe we're just adding up rosettacount and currentcount in the display.
[10:40] <jtv> Yes, we are.
[10:41] <henninge> wgrant: looks like any mp
[10:41] <jtv> danilos, henninge: rosettastats is still computing translatedCount as currentCount + rosettaCount.  Do you concur that that is wrong in the recife world?
[10:41] <jtv> Or is it the interpretation of currentCount that's wrong maybe?
[10:42] <danilos> jtv, currentCount should be translations that are active in both ubuntu and upstream
[10:42] <danilos> jtv, with that, it should still be correct
[10:43] <jtv> danilos: then it looks as if we're probably computing the wrong thing
[10:43] <danilos> jtv, calculation of currentCount might be wrong
[10:43] <jtv> we never updated the interface docstring for currentCount.  It basically says "current translations."  :(
[10:43] <jtv> Well, "current upstream."
[10:43] <danilos> jtv, oh, it was incorrect for ages then :)
[10:44] <jtv> No surprise with RS!
[10:44] <jtv> I mean RosettaStats, not Serbia.
[10:45] <danilos> jtv, yeah, basically, we need to do away with RosettaStats, as we all know :)
[10:45] <danilos> jtv, the mapping on ITranslatedLanguage.translation_statistics (or something) should clarify it all
[10:46] <danilos> jtv, TranslatedCount.recalculateCounts() transforms RosettaStats counts to useful counts, fwiw
[10:46] <danilos> jtv, RosettaStats meaning of "current" was always "synced with upstream"
[10:46] <jtv> Yes
[10:47] <jtv> Unfortunately the documentation remained confused all over the place
[10:47] <danilos> jtv, right; so, it's likely that TranslatedLanguage (uhm, not TranslatedCount :) needs to be updated to be side-aware
[10:47] <jtv> TranslatedLanguageMixin still takes about "imported" messages
[10:48] <danilos> jtv, and others
[10:48] <danilos> jtv, yeah, that seems to be the problem
[10:48] <wgrant> henninge: Thanks, that helped to track the issue down.
[10:49] <lifeless> stub: update to the query: http://bazaar.launchpad.net/~lifeless/launchpad/bug-636158/revision/12415
[10:49] <jtv> danilos: it's the actual pofile.currentcount that seems to be wrong.
[10:49] <jtv> so not a TL problem
[10:49] <danilos> jtv, :(
[10:49] <danilos> jtv, that means that updateStatistics is wrong
[10:49] <jtv> yes
[10:50] <danilos> jtv, current count is the only one not based on an actual POFile:+translate filter, iirc
[10:50] <jtv> danilos: that's a long time ago—we (I, I'm afraid) rewrote to compute basically everything in one query.
[10:51] <danilos> jtv, oh
[10:51] <danilos> jtv, well, then you probably know much more about it than I do :)
[10:51] <jtv> yar
[10:52] <jtv> danilos, one thing you do know better probably: do we still run the full stats update runs?
[10:52] <jtv> ISTR you stopped them at some point
[10:52] <jtv> And this is data that hasn't changed in a while…
[10:52] <danilos> jtv, yeah, the daily ones are not working because of DBLoopTuner craziness
[10:52] <danilos> jtv, but weekly ones run on Thu evenings or something
[10:52] <jtv> so what about the weekly ones?
[10:52] <jtv> :/
[10:53] <lifeless> stub: can you update your vote please (assuming the new query makes you happy ;))
[10:53] <jtv> danilos: the stats query selects…
[10:53] <jtv>                     (Other.id = Current.id) AS same_on_both_sides
[10:53] <jtv> so the intention is certainly right
[10:53] <danilos> jtv, you can always confirm that in
[10:53] <danilos> jtv, ...crontabs
[10:53] <jtv> danilos: yes, but having some painful wrist trouble, trying to minimize movement
[10:55] <danilos> jtv, you probably had hard time mapping to rosetta stats from sane values: I am pretty sure it's not even a bijection
[10:55] <jtv> danilos: why would it have to be?
[10:56] <danilos> jtv, because when it isn't, it means that in some cases you can't even get a correct value back
[10:59] <jtv> danilos: think I have it
[10:59] <jtv> rosettacount includes both "translated differently" and "translated this side only"
[10:59] <jtv> but no
[11:00] <jtv> that wouldn't affect currentcount
[11:13] <jtv> Damn, it's empty TMs.  :(
[11:13] <jtv> They're shared upstream/ubuntu, but "not translated on the other side"
[11:27] <wgrant> jml: Did you get a response to your PQM submission?
[11:27] <jml> wgrant: which one?
[11:27] <jml> wgrant: more-canonical-cleanups got rejected because of testfix
[11:27] <wgrant> Ahh, so we are testfix.
[11:27] <jml> with that new-fangled less-useful email that PQM sends these days.
[11:27] <wgrant> I keep getting no response to my shipit branches.
[11:27] <wgrant> Perhaps that is relevant.
[11:28] <wgrant> Forced.
[11:32] <bigjools> do we have a "right" way of removing the hundreds of old eggs/dependencies?
[11:35] <jml> bigjools: perhaps doc/buildout.txt has the answer?
[11:36] <bigjools> real documentation!
[11:37] <bigjools> unfortunately it doesn't say
[11:37] <bigjools> I suspect this is like the linux kernel package problem
[11:38] <bigjools> I have 21 of them installed here :/
[11:38] <jml> bigjools: yeah, doc/buildout.txt is one of the more useful technical docs in the tree
[11:38] <stub> lifeless: Not seeing the new diff in the mp :-(
[11:38] <wgrant> stub: mizuho was broked... it's possible the diff update job failed.
[11:38] <bigjools> jml: it is
[11:54] <wgrant> It turns out that PQM's response to an invisible branch is nothing.
[11:54] <wgrant> Excellent.
[11:57] <jml> it's not the most polished of tools.
[13:30] <danilo> allenap, hi, got a minute perhaps?
[13:46] <jtv> Reviewer wanted!  https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
[13:48] <jtv> Hmm strange error from bzr lp-propose:
[13:48] <jtv> ERROR: Too many attempts to read from the player!
[14:13] <henninge> adeuring: Hi
[14:14] <adeuring> moin henninge!
[14:14] <henninge> adeuring: wanna do a stand-up call?
[14:14] <adeuring> henninge: , sure let's do it. I have a small question about a factory method
[14:22] <jml> gary is not here :\
[14:29] <jml> hm.
[14:30] <jml> President's Day
[14:35] <beuno> gary's the president now?
[14:42] <danilos> :)
[14:42] <danilos> henninge, hey, I see you are OCR, and I am fighting a zope permissions problem
[14:43] <danilos> henninge, and I remember you knowing a lot about those :)
[14:50] <jtv> Still need a reviewer!  https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
[14:52] <jtv> henninge:            ^^^^^^^^^^   from when you were at lunch probably
[14:53] <henninge> jtv: oh, missed that.
[14:53] <jtv> henninge: you weren't connected at the time
[14:54] <henninge> jtv: I am working on another review but I can take yours next.
[14:54] <jtv> great, thanks
[14:54] <henninge> danilos: what is the question?
[15:01] <danilos> henninge, I figured it out, thanks :)
[15:33] <henninge> adeuring: Please have a look if this affects your current work in any way: https://code.launchpad.net/~jtv/launchpad/bug-722568/+merge/50593
[15:34] <adeuring> henninge: yeah,  Ithanks, I've seen it. jtv spotted an issue I hadn't yet
[15:34] <jtv> adeuring: you're also working on stats?
[15:34] <adeuring> jtv: bug 688130
[15:34] <_mup_> Bug #688130: Statistics clean-ups and extra tests <lp-translations> <upstream-translations-sharing> <Launchpad itself:In Progress by adeuring> < https://launchpad.net/bugs/688130 >
[15:35] <jtv> "Test for empty messages"… that's something my branch does
[15:36] <jtv> "Treat incomplete messages as untranslated" will be affected.
[15:36] <adeuring> right
[15:37] <danilos> henninge, will you be able to fit in https://code.launchpad.net/~danilo/launchpad/bug-722626/+merge/50618 as well soon? (it is very short branch)
[15:37] <henninge> danilos: No.
[15:37] <henninge> :-P
[15:37] <danilos> :(
[15:38] <henninge> danilos: let me look first ... ;-)
[15:38] <danilos> henninge, 98 lines of diff according to "wc -l", so, pleaaaseee
[15:39] <henninge> danilos: why is it depending on a branch that has no mp?
[15:39] <danilos> henninge, my overeager pipelining
[15:39] <danilos> henninge, nothing in that branch
[15:40] <henninge> ah!
[15:40] <henninge> danilos: I will look at it as soon as I am done with jtv's branch.
[15:40] <danilos> henninge, this bug was uncovered while trying to land a different branch
[15:40] <danilos> henninge, thanks
[15:44] <henninge> jtv: abentley recently added a "side" parameter to makePOFile, did you know that?
[15:45] <henninge> This way it is easier to create messages on the other side.
[15:45] <jtv> henninge: ah yes, I saw that… did I neglect to use it?
[15:45] <jtv> You mean, create an extra POFile and then create a current message in that?
[15:46] <henninge> jtv: exactly but I am not sure it is worth it
[15:47] <jtv> It does seem a bit roundabout.
[15:47] <henninge> jtv: remind me how COALESCE works, please.
[15:47]  * henninge goes rtfm
[15:52] <henninge> jtv: r=me
[15:52] <jtv> thanks!
[15:52] <jtv> heh… no proof-of-read?  :)
[15:52] <jtv> tsk
[16:10] <henninge> danilos: I am pretty sure that using check_permision in security.py is broken
[16:10] <henninge> because check_permission assuems a user and that might (theoretically I admit) not be the same as is requested by checkAuthenticated
[16:11] <henninge> so you might end up checking permissions for the wrong user ... theoretically.
[16:11]  * henninge wants to fix LP security so badly!
[16:13] <danilos> henninge, well, what would you propose to be used instead?
[16:13] <danilos> henninge, (not to mention that it's already used elsewhere in security.py)
[16:13] <henninge> yes, I feared you where going to ask that ...
[16:14] <henninge> That is no excuse! :-P
[16:14] <henninge> danilos: Maybe you cannot do anything different (easily) but you have to admit that it is broken.
[16:15] <henninge> checkAuthenticated asks for a specific user which is ignored.
[16:15] <danilos> henninge, heh, well, it's broken but it's convenient
[16:15]  * henninge looks closer
[16:16] <danilos> henninge, the right approach for my particular problem would be to check all target types (IProduct, IProductSeries, IDistribution, ...) and use their own checkers instead
[16:16] <danilos> henninge, and doing it that way is even uglier and even more broken imo
[16:17] <henninge> yes, I can see that
[16:33] <henninge> danilos: r=me
[16:34] <danilos> henninge, thanks
[16:37] <danilos> henninge, I did learn about IPersonRoles playing with security.py right now (by having to use user.person to pass in :)
[16:38] <danilos> henninge, also, only now I realize your point about check_permission not having a user parameter
[16:38] <danilos> henninge, I am a bit more worried right now
[16:39] <henninge> danilos: check_permission goes through to zope.security.checkPermission which in term uses the "current interaction"
[16:39] <henninge> if no other is passed in.
[16:39] <danilos> henninge, yeah, I realize that
[16:40] <henninge> I am not sure how likely it might be that "user" is differnt from the "principal" in the current interaction.
[16:46] <henninge> s/in term/in turn/ ... JFTR
[16:49] <bigjools> jml: if I wanted to force the ftp scenario in the test_poppy tests, how would I do that?
[16:50] <jml> bigjools: looking.
[16:50] <bigjools> jml: thanks.  the best I could think of was to include "ftp" as the test match but that doesn't exclude sftp... I was hoping there would be a magic matcher for scenarios
[16:52] <jml> my machine is slow today.
[16:52] <jml> 23s to start the database
[16:52] <jml> bigjools: you need to match the parenthesis
[16:53] <bigjools> ah ok so it is part of the test name
[16:53] <bigjools> thanks
[16:53] <jml> yeah.
[16:53] <bigjools> jml: you and StevenK did a great job of writing new tests for the older FTP stuff - I can just replace the setup phase - trivial - and run them!
[16:54] <jml> There are some subtleties there, tests have at least three things that could be called "name",
[16:54] <jml> bigjools: glad to hear it :)
[16:54] <bigjools> I just changed 2 lines to use a different fixture!
[16:56] <bigjools> jml: unfortunately I think the test runner hates parentheses :/
[16:57] <bigjools> no tests matched
[16:57] <jml> bigjools: try this: ./bin/test -cvv test_poppy '\(ftp\)'
[16:57] <bigjools> ah escape
[16:57] <jml> it's a regex.
[16:58] <bigjools> now I really do have problems
[17:04] <jml> bigjools: :)
[17:04] <bigjools> jml: one more thing - were you involved with the actual tests in that file?
[17:05] <bigjools> I'm trying to work out the rather opaque "self.assertEqual(os.stat(wanted_path).st_mode, 0102674)"
[17:05] <jml> bigjools: I *think* I was, but I don't remember.
[17:05] <bigjools> since it fails now :)
[17:05] <bigjools> but that is such a magic number I've no idea what it's trying to test :/
[17:06] <jml> bigjools: Don't know off the top of my head. I'd look up os.stat documentation for the constants and start making some educated guesses.
[17:06] <bigjools> yeah I was going to head there - just wondered if you could save time :)
[17:07] <jml> https://code.launchpad.net/~jml/launchpad/list-ec2-test-runs-721784/+merge/50537 might interest some folk
[17:07]  * jml has to dash off
[17:07] <jml> g'night all.
[17:07] <bigjools> nn jml
[17:07] <jml> be back later to talk to Aussies
[18:01] <lifeless> hi
[18:02] <jtv> hi lifeless!
[18:03] <jtv> say, would it be okay with you to set a much longer timeout for the approval form for translations import queue entries?  It's blocking queue review.
[18:03] <lifeless> whats the page id
[18:05] <jtv> TranslationImportQueueEntry:+index
[18:06] <lifeless> hmm, thats interesting
[18:06] <lifeless> according to the histogram on  https://devpad.canonical.com/~lpqateam/ppr/lpnet/latest-daily-pageids.html
[18:07] <lifeless> its not had any requests near the timeout
[18:07] <lifeless> though of course, thats probably showing the weekend
[18:07] <lifeless> how long do you think it needs?
[18:08] <jtv> This is a page only a handful of people can access, so you won't be seeing many timeouts—especially during the weekends.
[18:09] <jtv> I'm taking a wild stab at 30 seconds for now.  Terrible, I know.
[18:09] <lifeless> we can't set it to that
[18:09] <lifeless> (note, not 'I don't want to' : cannot)
[18:09] <jtv> Right, I noticed that—why not?
[18:10] <lifeless> haproxy will shutdown a request with a total in-dc queue time of 30 seconds
[18:10] <lifeless> so will apache
[18:10] <lifeless> ha proxy currently has 1second +- 0.5 seconds queuing on most requests afaict through adhoc measurement
[18:11] <lifeless> setting anything to > 28 seconds has a high probability of running into the 30 second time on either of apache or haproxy
[18:11] <jtv> Okay… so how about 25 seconds?
[18:11] <lifeless> that said, this page worked when we had a 15 second timeout, no ?
[18:11] <jtv> Not to sure of that.
[18:12] <jtv> You see, the work that this form does hasn't been getting done for a while.
[18:12] <jtv> We're in the process of fixing that.
[18:12] <lifeless> ah
[18:12] <lifeless> ok - add a request to LPS to set it to 20 seconds
[18:12] <lifeless> lets see if thats sufficient
[18:12] <lifeless> I'm +1 on that
[18:12] <jtv> We can also fix the code to be faster, but a bit of goodness will have to be sacrificed until we can work something out with rabbit or somesuch.
[18:12] <jtv> Thanks.
[18:13] <lifeless> looking at the oops
[18:13] <jtv> In any case it'll allow me to get _more_ done, and that's something.
[18:13] <lifeless> it has 600ms of query repeated 19 times
[18:13] <jtv> You'll see a lot of repetition in the oops; not something that can be batched very easily.
[18:13] <lifeless> the plan will be the key
[18:13] <bigjools> night all
[18:14] <jtv> night bigjools!
[18:14] <lifeless> if its getting into scans, doing a larger query may be no slower
[18:14] <jtv> It's not a matter of slower so much as of complexity.
[18:14] <jtv> This query, believe it or not, is already pretty well-optimized compared to what we used to have.
[18:14] <lifeless> you say in the bug that its statistics
[18:15] <jtv> It is, yes.
[18:15] <lifeless> I thought a background job did that already?
[18:15] <jtv> Yes, but it's slow.
[18:15] <jtv> But I am indeed thinking of just not calculating them in this scenario.
[18:15] <lifeless> does that matter for approval ?
[18:15] <jtv> After all, since the end user isn't hitting this form, there's no expectation of an interactive update.
[18:16] <jtv> There is also already an accepted situation where the template is empty until the import happens.
[18:16] <jtv> And *blink* when that happens, the statistics need to be recomputed anyway.
[18:16] <jtv> In fact they are, IIRC.
[18:17] <jtv> I don't see any point to calculating them at this particular point!
[18:19] <jtv> Why don't I just go fix that…
[18:20] <lifeless> cool
[18:20] <lifeless> btw
[18:20] <lifeless> when I add an additional language to the query
[18:20] <lifeless> it stays at 300ms
[18:20] <lifeless> with the same plan
[18:20] <lifeless> of course this is on qastaging, so may be useless for production prediction
[18:21] <lifeless> ...and I didn't finish the conversion. bah.
[18:21] <jtv> May well be different there, yes, though you're getting a similar time and that suggests it may be a similar plan.
[18:21] <jtv> Conversion?
[18:22] <lifeless> adding group by, and = -> IN ()
[18:24] <lifeless> jtv: I've added it to the bug, in the offchance that its helpful
[18:25] <jtv> thanks lifeless—though AFAICS we can just eliminate this use-case and then only template imports will need to do this calculation in bulk.
[18:25] <lifeless> I think thats better still :)
[18:26] <jtv> There may still be a few interactive cases though that may run several of these, so who knows.
[18:26] <lifeless> we can look at their performance when we get timeout protection on backend services
[18:26]  * jtv looks forward to that day in trepidation
[18:26] <lifeless> heh
[18:26] <lifeless> we need it - rogue backends regularly cause frontend timeouts at the moment
[18:27] <lifeless> even with tunable loop
[18:27] <jtv> And I beheld the sixth terrible horseman, and his shirt said "Internal Server Error"
[18:27] <lifeless> :(
[18:27] <lifeless> what site ?
[18:27] <jtv> Just a general observation.
[18:28] <jtv> lifeless: btw the postgres-R presentation at FOSDEM was impressive—but alas not production-ready yet.
[18:29] <lifeless> http://www.fosdem.org/2011/schedule/event/clustered_databases ?
[18:29] <jtv> Optimistic multi-master… it felt a lot like distributed 2PC
[18:29] <lifeless> was it taped?
[18:30] <jtv> Yes, and I think it was taped.  If you find a recording, you may recognize the voice of the heckler in the front.
[18:30] <lifeless> :)
[18:30] <lifeless> uhm
[18:30] <lifeless> does it partition as well?
[18:30] <lifeless> what we need for scaling is either much less bulk/historical data (gives us a lower coefficient per user-year)
[18:31] <jtv> I don't think it does that, no.
[18:31] <lifeless> or the ability to scale everything horizontal - lets us choose an good price-point per machine and add machines
[18:31] <jtv> Although if disk space isn't the issue, I suppose one could get similar effects by implementing some kind of affinity layer on top.
[18:32] <lifeless> scaling horizontally requires capping the local data per node in some fashion
[18:32] <jtv> Try to get locality for caches through clever load distribution.
[18:32] <lifeless> (e.g. a nfs backend that can be raided across many servers - meep!)
[18:32] <jtv> aaiggh!
[18:32] <elmo> ..
[18:32] <lifeless> now that I've made your eyes bleed
[18:32] <elmo> lifeless: did you just suggest nfs in the context of DB servers?
[18:33] <jtv> it was him!  it was him!
[18:33] <jtv> I didn't do anything!
[18:33] <lifeless> elmo: Not in any serious sense
[18:33] <jtv> Maybe with sqlite—I hear nfs is really good with locks
[18:33] <lifeless> that would be a win
[18:34] <jtv> Anyway, as long as we're blue-skying, we already have slaves, so I don't see a huge difference with multi-master when it comes to resource allocation.
[18:35] <lifeless> so -xc is referenced in the slides as doing partitioning
[18:35] <jtv> I was just saying it mightn't even matter so much how much is on each disk, as long as you could limit how much needs to be in memory on a given node.
[18:36] <jtv> Not limit actually—more like guide.
[18:36] <lifeless> jtv: our cluster isn't symmetric: the load isn't evenly balanced atm, though multimaster might help with that substantially, the master and slave machines are not equivalently sized
[18:40] <lifeless> elmo: so what are you guys sprinting on - puppet ?
[18:40] <jtv> lifeless: postgres-R has no problem with that, with the obvious exception (which someone insisted on lament^H^H^H^Hdiscussing at length) of increased lock contention.
[18:41] <elmo> lifeless: i'm not there, so I'm at best an idle spectator, but it's just our bi-annual sprint - puppet is one of the focuses this time though, yeah
[18:42] <lifeless> elmo: cool
[18:43] <lifeless> jtv: speaking of rabbit, I tried to land an incremental step towards it in the weekend but it failed on ec2; I haven't dug into why, yet.
[18:44] <jtv> lifeless: I wish I'd gotten around to looking into it…  but this is something that we've long, long wanted message queues for.
[18:44] <jtv> It'll let us lift these queries out of interactive transactions and be more aggressive about recalculating.
[18:45] <lifeless> you can use the regular task system today
[18:45] <lifeless> or put in some energy towards rabbit
[18:45] <lifeless> :)
[18:49] <jtv> I wonder if the existing task system would get us to the desired response time of "a few seconds" though.
[18:52] <lifeless> if you can't do it in a 13 second web request
[18:52] <lifeless> then a task / queue system won't do it either ;)
[19:12] <lifeless> james_w: what project are your soup matchers in ?
[19:13] <lifeless> ah found it
[19:40] <Ursinha> bug 711064
[19:40] <_mup_> Bug #711064: POFile:+translate timeouts <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/711064 >
[19:48] <lifeless> Ursinha: >
[19:48] <lifeless> Ursinha: ?
[19:49] <Ursinha> lifeless, seeking help of the bot to get the bug link
[19:52] <lifeless> ah :)
[20:09]  * thumper relocates to somewhere with coffee to hand
[20:09] <jtv> No OCR?
[20:09] <jtv> Reviewer wanted!  https://code.launchpad.net/~jtv/launchpad/bug-719267/+merge/50651
[20:10] <jtv> It's small.
[20:10] <jtv> Very small.
[20:27] <jtv> thanks once more lifeless!
[20:32] <lifeless> jtv: no worries
[20:32] <lifeless> jtv: hey
[20:32] <lifeless> do you know why +needs-packaging does pofile lookups ?
[20:32] <lifeless> bug 722794
[20:32] <_mup_> Bug #722794: DistroSeries:+needs-packaging timeouts <dba> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/722794 >
[20:33] <jtv> lifeless: priority computation IIRC
[20:33] <jtv> There's some kind of rating system that includes a wide variety of factors… sinzui will know the details.
[20:33] <jtv> As so often in line.
[20:33] <jtv> life.
[20:33] <lifeless> thanks
[21:01] <flacoste> lifeless: call?
[21:03] <flacoste> lifeless: https://lpstats.canonical.com/graphs/LPProjectCriticalBurndown/
[21:04] <flacoste> https://lpstats.canonical.com/graphs/LPProjectBugsClosed/
[21:04] <flacoste> https://lpstats.canonical.com/graphs/LPProjectCriticalFixed/
[21:04] <wallyworld> thumper: we having standup?
[21:04] <thumper> aye
[21:21] <StevenK> http://people.canonical.com/~stevenk/built-by-recipe.png
[21:32] <huwshimi> Morning
[21:32] <jml> huwshimi: hi
[21:33] <jml> huwshimi: will brb.
[21:33] <huwshimi> jml: Sure
[21:39]  * StevenK grumbles at the lack of UI reviewers in this timezone
[21:39] <StevenK> And no, I'm not volunteering. :-P
[21:39]  * jml in back
[21:42] <flacoste> lifeless: https://lpstats.canonical.com/graphs/PPR/20100222/20110222/
[21:43] <lifeless> 6312026
[21:43] <lifeless> 2.78
[21:47] <lifeless> flacoste: https://lpstats.canonical.com/graphs/PPR/20110208/20110215/
[21:54] <poolie> hi flacoste
[21:54] <flacoste> hi poolie
[21:54] <flacoste> poolie want to chat earlier?
[21:54] <poolie> lifeless, teddy, just thinking about flags for controlling dkim
[21:54] <poolie> i think a per-email-domain scope may be yagni for now
[21:54] <poolie> we can add one when there's a second thing controlled by email
[21:54] <poolie> does thta make sense to you?
[21:55] <poolie> flacoste, sure, give me a few minutes to get set up
[21:55] <lifeless> poolie: how would you do it today then?
[21:56] <poolie> just a flag that gives a space-separated list of trusted domains
[21:57] <poolie> just wondering if is this enlightened laziness or the regular kind
[21:57] <lifeless> it might be a bit awkward for dealing with concurrent updates
[21:57] <lifeless> a row-per-thing would be easier (once we stop regenerating all the rules)
[22:01] <lifeless> poolie: so I think a per email scope would be quite beneficial
[22:08] <poolie> ok
[22:21] <lifeless> jml: welcome back
[22:21] <lifeless> jml: looked at the testtools thing yet ?
[22:21] <lifeless> thumper: hi; did you know about soupmatchers?
[22:21] <thumper> lifeless: yes
[22:21] <lifeless> thumper: needed something different again ?
[22:21] <thumper> somewhat
[22:21]  * thumper has head down right now
[22:22] <jml> lifeless: no, going to after this call.
[22:22] <lifeless> thumper: I'd like us to not have generic matchers in the lp tree - they should be in testtools, or in soupmatchers etc - just to keep our code base focused
[22:22] <lifeless> thumper: gl with your head downing
[22:23] <thumper> I agree in principle
[22:27] <lifeless> (there is a but there, isn't there?)
[22:43] <jml> satisfice!
[22:46] <jml> lifeless: No, I haven't sadly. I don't really get time for much discretionary hacking on Mondays.
[22:46] <jml> lifeless: I'll probably be able to take a look tomorrow. Right now am too knackered to be of any use.
[22:47] <lifeless> jml: sleep well
[22:48] <wgrant> Night jml.
[22:49] <jml> g'night
[22:53] <lifeless> wgrant: hey
[22:54] <lifeless> wgrant: what timeout are you beating up on today?
[22:54] <wgrant> lifeless: Finishing tracking down all the Person queries.
[22:54] <wgrant> I've fixed the comment ones. But there are still lots from permission checks.
[22:55] <lifeless> bugtask:+index?
[22:55] <wgrant> Yes.
[22:55] <lifeless> cool
[22:55] <wgrant> Bug #1 has ~140 Person queries, which means 100 aren't from comments :(
[22:55] <_mup_> Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <metacity:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:I
[22:56] <lifeless> wgrant: activities perhaps
[22:56] <wgrant> Possibly.
[22:56] <wgrant> We'll find out soon.
[22:57] <wgrant> You?
[23:09] <lifeless> working on the test fixes ec2 found for me last night
[23:09] <lifeless> waiting for stub to discuss the bug search query change
[23:11] <wgrant> Aha
[23:19] <lifeless> oh
[23:19] <lifeless> and writing leps
[23:20] <wgrant> For?
[23:22] <lifeless> oops
[23:22] <lifeless> parallel testing
[23:24] <poolie> lifeless, so istm the incoming mail processor needs to be able to establish a flag scope from something like a context manager
[23:24] <poolie> which is eminently possible, but there is no infrastructure at the moment
[23:24] <lifeless> bug 717394 should get through ec2 this time
[23:24] <_mup_> Bug #717394: Distribution:+bugs timeouts <qa-ok> <timeout> <Launchpad itself:In Progress by lifeless> < https://launchpad.net/bugs/717394 >
[23:24] <poolie> all of them work from global state
[23:24] <poolie> i'm thinking something like
[23:24] <poolie> with feature_scopes(MailFromDomain(domain)): ....
[23:25] <lifeless> poolie: as I understand the caching, you'll want to push the existing controller completely to the side
[23:26] <lifeless> poolie: anyhow, it sounds fie
[23:26] <lifeless> *fine*
[23:27] <poolie> it does have to deal with caching
[23:27] <poolie> having seen some of the bugs people hit, i think having multiple controllers around may not be such a good idea
[23:56] <lifeless> wow
[23:56] <lifeless> massive quake
[23:56] <lifeless> internet is up
[23:56] <lifeless> phones & cell towers down
[23:57] <spiv> lifeless: hooray for internet