[00:17] <StevenK> So, you have a dastardly plan?
[00:17] <StevenK> wgrant: &
[00:17]  * StevenK grumbles at his fingers.
[00:25] <wgrant> StevenK: Yeah, testing it on DF
[00:38]  * StevenK resurrects the redux branch
[00:52] <wgrant> StevenK: I have an equivalent that is 1ms rather than 1000ms, but I'm not sure that it is truly equivalent yet.
[00:54] <StevenK> wgrant: That was the problem? The query was taking 1 second per notification?
[00:55] <wgrant> StevenK: Calculating the structural subscribers for a private bug took 600-1200ms
[00:55] <wgrant> StevenK: And because the notification code is crap, it ran that calculation several times for a single operation.
[00:55] <wgrant> I have a query which fixes the slowness
[00:55] <wgrant> And I might refactor it all later to remove the duplication
[00:56] <wgrant> Since four separate bugnotificationrecipient timeouts plagued my last week
[00:57] <StevenK> wgrant: Ah, so the real smoking gun is the change in model/structuralsubscription.py
[00:58] <wgrant> StevenK: I believe so
[00:58] <wgrant> StevenK: Though dupe/etc subs may be similarly afflicted, they weren't important in this case.
[00:58] <wgrant> (probably since private bugs with dupes are relatively rare)
[01:15] <wgrant> StevenK: I've added the full suite of APGs on DF, so performance testing there should be more accurate.
[01:22]  * StevenK stabs django for lying
[01:25] <lifeless> StevenK: oh?
[01:28] <StevenK> lifeless: Figured it out. The docs could be clearer about lots of things.
[01:29]  * lifeless is no wiser
[01:30] <StevenK> lifeless: Adding commands that manage.py can execute
[01:42] <wgrant> Oh
[01:42] <wgrant> EXPLAIN ANALYZE gets a bit confused when dealing with UNIONed subqueries
[01:43] <wgrant>            ->  Index Scan using teamparticipation_team_key on public.teamparticipation  (cost=0.00..30.16 rows=8 width=8) (actual time=0.016..0.173 rows=56 loops=5)
[01:43] <wgrant>                  Output: public.teamparticipation.id, public.teamparticipation.team, public.teamparticipation.person
[01:43] <wgrant>                  Index Cond: (public.teamparticipation.team = (2794))
[01:43] <wgrant> What it actually means is "Index Cond: (public.teamparticipation.team = [result of UNIONed subquery that happens to start with a literal 2794])" :(
[01:43] <StevenK> Hah, helpfl
[01:43] <StevenK> *helpful
[01:45] <lifeless> wgrant: really ?
[01:45] <lifeless> wgrant: can I see the full explain analyze + original query ?
[01:47] <wgrant> lifeless: https://pastebin.canonical.com/71501/
[01:47] <wgrant> Observe the last couple of lines of the plan
[01:48] <wgrant> And how they make no sense unless it actually means the full UNIONed result
[01:50] <wgrant> Note also that the output of the HashAgg is '(2794)' because of that
[01:52] <wgrant> (ignore the Hash Semi Join... I'm trying to work out what it's doing there)
[01:55] <lifeless> What happens if you replace the select constant as grantee
[01:55] <lifeless> with select grantee from (select constant as grantee)
[01:55] <lifeless> I wonder if its misplanning entirely
[01:55] <lifeless> -> worried
[01:56] <wgrant> Nah, it just optimises that away
[02:02] <wgrant> The plan seems correct apart from the hash semi join.
[02:02] <wgrant> Which duplicates the other subplan, which is largely identical and more efficient
[02:02] <wgrant> I don't understand why it thinks it also needs the hash semi join
[02:09] <lifeless> 11294 Time Outs
[02:09] <lifeless> 6415 /    0  Branch:EntryResource:getMergeProposals
[02:09] <lifeless> ^ regression methinks
[02:09] <wgrant> lifeless: That's the bug you triaged a week ago
[02:09] <lifeless> ah'
[02:09] <wgrant> (and yes)
[02:09] <lifeless> i'm getting old in my young age.
[02:12] <spm> you have a beard. bearded people are old. qed.
[02:12] <spm> a 9yo expert in old people told me that.
[02:12] <wgrant> spm: He forgot that already
[02:12] <wgrant> Due to his age.
[02:12] <spm> clearly
[02:13] <StevenK> spm: '9yo expert' is a tautology
[02:13] <spm> not from the 9yo's perspective.
[02:13] <spm> ^^ relativity
[02:13] <lifeless> thats special
[02:13] <wgrant> StevenK: So, mind if I take over the structsubs branch?
[02:13] <StevenK> wgrant: ... to do what?
[02:14] <wgrant> StevenK: Make it fast.
[02:14] <wgrant> StevenK: I mean the private structsubs branch
[02:14] <StevenK> wgrant: If you like
[02:14] <wgrant> Not the information type one
[02:14] <StevenK> The model code has landed already.
[02:15] <StevenK> The UI will happen tomorrow now that get_bug_tags() no longer exists
[02:15] <wgrant> Right
[02:15] <StevenK> I'm still trying to bend django to my will
[02:16] <StevenK> It acts utterly and completly stuffed with :memory: sqlite
[02:22] <lifeless> StevenK: :(
[02:22] <lifeless> StevenK: it possibly is dropping all the handles to it
[02:22] <lifeless> StevenK: if thats the case, use a random /tmp path for the db
[02:28] <StevenK> lifeless: Yeah, I'm getting to that point
[02:35] <StevenK> lifeless: I thought it was a process thing last Monday -- as in, if I could do syncdb and runserver in one process, it should be fine. It isn't.
[02:35] <lifeless> StevenK: sorry for making your life hard
[02:35] <lifeless> StevenK: /tmp will be the go
[02:37] <StevenK> lifeless: Yeah, I have two choices there. Hard code /tmp/auditor.sql in auditor's config, or forcibly import auditor's settings in the fixture and mktemp
[02:38] <lifeless> StevenK: you don't want a hard coded path :)
[02:38] <lifeless> StevenK: If I can make a (little awkward but tolerable I think) suggestion.
[02:38] <StevenK> Yes I do, it will be EASY
[02:38] <StevenK> As opposed to that last thing you suggested
[02:39] <lifeless> it will fail horribly when you have concurrency.
[02:39] <lifeless> so, in your settings.py, consult an environment variable for the path
[02:39] <lifeless> if its not set, use whatever path you were using before the :memory: experinment.
[02:39] <lifeless> in the fixture, set a unique path using TemporaryDirectory to get a throwawy subtree.
[02:40] <lifeless> and export it via EnvironmentVariable
[02:40] <lifeless> (both are fixtures from fixtures)
[02:41] <StevenK> The fixture already creates a temp directory for the logfile, so that should be easy
[02:41] <lifeless> cool
[02:48] <StevenK> lifeless: http://pastebin.ubuntu.com/1131833/
[02:50] <lifeless> cool
[02:50] <StevenK> Now to hack the fixture
[02:51] <wgrant> There'd better not be a default of a well-known path in /tmp, or I will be unhappy :)
[02:52] <StevenK> wgrant: BLEH
[02:52] <StevenK> Because debugging via connecting to the DB is for chumps or something.
[02:53] <lifeless> wgrant: there will be, but the fixture will override it
[02:53] <lifeless> wgrant: and prod deploys will use postgresql.
[02:54] <lifeless> wgrant: so suck it up white boy :)
[02:54] <wgrant> lifeless: The fact that /tmp exposes a "create item by name" interface is a historical mistake.
[02:55] <wgrant> So a path of /tmp/auditor.sql doesn't make sense
[02:55] <wgrant> As it's not possible by sensible means to create a file with that path
[02:57] <StevenK> wgrant: So I'm supposed to make use of tempfile.mkstemp and just guess?
[02:57] <wgrant> Guess?
[02:57] <StevenK> Even though the fixture is going to override it and it's just for people wanting to run auditor's tests locally?
[02:58] <wgrant> Is any callsite not going to override the default?
[02:58] <wgrant> The test suite surely has to
[02:58] <wgrant> If nothing uses the default, then the default doesn't need to be there.
[02:59] <StevenK> There are three testsuites here -- auditor's, auditorfixture's and LP's.
[02:59] <StevenK> The fixture will override the default, so it's fine for the fixture itself and LP.
[02:59] <lifeless> the risk is that someone will attack your system by having a symlink to something you care about at that path
[02:59] <lifeless> but they can't generate arbitrary contents
[02:59] <wgrant> It's not a significant risk here.
[03:00] <wgrant> But it's something that people will copy.
[03:00] <wgrant> And it provides no value here
[03:00] <lifeless> so put big warning messages around it.
[03:00] <wgrant> => it should not exist
[03:00] <lifeless> wgrant: whats your alternative ?
[03:00] <lifeless> wgrant: whenever you find yourself saying 'has no value' please provide an alternative, as an existence proof, for something functionally equivalent.
[03:00] <StevenK> wgrant: It provides value for people being able to run auditor's test suite
[03:01] <lifeless> wgrant: and yes, I realise the irony that I didn't do that for ORM discussion.
[03:01] <lifeless> wgrant: I should have.
[03:01] <StevenK> Without having to set environment variable
[03:01] <wgrant> lifeless: Well, in order to use the default, I'd have to copy auditor.sql to /tmp/auditor.sql before running auditor. It's surely just as easy to set AUDITOR_SQL before running auditor.
[03:01] <wgrant> StevenK: Can't the test suite poke it in other ways?
[03:01] <lifeless> wgrant: huh?!
[03:01] <wgrant> Having the test suite use a single well-known commonly-conflicted path is not a good idea.
[03:01] <StevenK> wgrant: Like I said before, which test suite?
[03:01] <lifeless> wgrant: tell you what, to get StevenK moving along.
[03:01] <StevenK> There are three.
[03:02] <wgrant> StevenK: The one that would use /tmp/auditor.sql
[03:02] <wgrant> Any test suite using that is non-concurrent
[03:02] <lifeless> wgrant: understand we're *improving* the situation, it *used* to be just /tmp/auditor.sql always.
[03:02] <lifeless> wgrant: so file a bug, and StevenK can look at it later.
[03:02] <StevenK> Actually, it used to be /home/steven/auditor.sql ... :-(
[03:02] <lifeless> StevenK: hah, same issue though.
[03:03] <StevenK> Right
[03:07] <wgrant> Ah, if it's already worse, then OK :)
[03:11] <StevenK> Although, thinking about it, the testsuite doesn't use that, so the default can probably die
[03:12] <lifeless> StevenK: one step at a time
[03:12] <lifeless> StevenK: you may be right, but why yak shave when you don't need to.
[04:55] <StevenK> wgrant: Are you winning at making private structsubs fast?
[05:14] <wgrant> StevenK: Yes.
[05:15] <wgrant> Trying to also win at not making the queries horrid
[07:53] <adeuring> good morning
[11:27] <rick_h_> jcsackett: ping when you get in, quick ? for you
[13:56] <jcsackett> rick_h_: ping.
[13:57] <rick_h_> jcsackett: pong
[13:57] <deryck> adeuring, here's my branch, if you don't mind:  https://code.launchpad.net/~deryck/launchpad/support-pape-max-auth-age/+merge/117781
[13:57] <rick_h_> jcsackett: nvm about earilier, don't think it'll work
[13:58] <adeuring> deryck: ok
[13:58] <jcsackett> rick_h_: ok.
[14:03] <jcsackett> rick_h_: hangout still planned for now?
[14:03] <rick_h_> jcsackett: yea, just hangouts not playing nice and starting up for me atm
[14:03] <rick_h_> 5th try to get it to load/start
[14:04] <deryck> rick_h_, are we using our url, or you starting a random one?
[14:04] <rick_h_> deryck: I'll try our url next I think
[14:04] <rick_h_> was trying to start a new one with no success
[14:04] <deryck> worked for me.
[14:21] <adeuring> deryck: r=me
[15:09] <rick_h_> jcsackett: so I'm going to go think for a bit around lunch. Do you want to get together for a call this afternoon and kind of run through some points?
[15:10] <jcsackett> rick_h_: sounds like a plan. have a particular time in mind?
[15:10] <rick_h_> jcsackett: shoot for 3:00?
[15:11] <jcsackett> sounds good.
[15:11] <rick_h_> cool, thanks
[15:21] <deryck> adeuring, thanks for the review
[15:53] <jcsackett> sinzui: ping.
[15:53] <sinzui> hello
[15:53] <jcsackett> sinzui: free to chat a bit?
[15:53] <sinzui> yes
[15:53] <jcsackett> fantastic.
[15:54] <jcsackett> sinzui: hangout invite sent.
[18:56] <rick_h_> jcsackett: up for hangout?
[18:56] <rick_h_> deryck: will want to put you on deck perhaps after this one as I try to get stuff talked over pre-EOD (heads up)
[18:56] <jcsackett> rick_h_: i'm free.
[18:57] <rick_h_> jcsackett: awesome
[18:57] <deryck> rick_h_, sure.
[18:57] <deryck> rick_h_, I'm done in an hour, since I started early and am at the beach.  Just to give you a heads up too ;)
[18:57] <rick_h_> deryck: ok, will try to go quick
[18:57] <jcsackett> rick_h_: we can keep ours short so you can talk to deryck.
[18:59] <jcsackett> rick_h_: can you send me the hangout invite? there are two of you in my G+ list and i'm not sure which to send the invite too. :P
[19:02] <lifeless> deryck: o/
[19:20] <nigelb> mwhudson: welcome back, post-new family member arrival :)
[19:21] <mwhudson> nigelb: thanks!
[19:23] <lifeless> mwhudson: wb
[19:23] <lifeless> mwhudson: everything go smoothly ?
[19:24] <mwhudson> lifeless: ended up with a c-section due to baby getting stuck on the way out, but aside from that everything is good
[19:24] <lifeless> mwhudson: woo (we did too)
[19:27] <mwhudson> ah right
[19:27] <mwhudson> it's a pretty intense 45 mins or so between them saying "we need to do a caesar" and getting it done :)
[19:27] <mwhudson> especially if everyone's been awake for 24 hours by that point
[19:28] <lifeless> mwhudson: I have /no idea/ how long it took
[19:28] <lifeless> tis all a blur
[19:28] <lifeless> I remember the operating room vividly
[19:28] <mwhudson> heh yeah
[19:28] <mwhudson> i remember being taken away and forcefully fed some carbs
[19:28] <lifeless> before, after or during ?
[19:29] <mwhudson> before the op
[19:29] <lifeless> interesting
[19:29] <lifeless> they didn't do that for me
[19:29] <lifeless> perhaps my weight dissuaded them :P
[19:31] <rick_h_> deryck: still got 10min?
[19:31] <deryck> rick_h_ sure
[19:31] <rick_h_> deryck: cool, normal hangout url?
[19:32] <lifeless> deryck: want to pick a time for a call?
[19:33] <deryck> lifeless, yeah.  let's do my Thursday afternoon, your Friday morning.  if that works for you.
[19:33] <lifeless> its clear
[19:58] <lifeless> sinzui: did my logic make sense w.r.t. mutually invisible teams in a project?
[20:04] <sinzui> lifeless: yes
[20:04] <sinzui> I asked Cody if they really use mutually invisible teams in a project?
[20:20] <lifeless> maxb: *headdesk* ** 100
[20:25] <maxb> indeed :-/
[22:59] <StevenK> wgrant: http://pastebin.ubuntu.com/1133382/ is the traceback
[23:11] <wgrant> StevenK: Something under related_projects is throwing an AttributeError, probably.
[23:12] <wgrant> You may want to call it manually
[23:12] <wgrant> in a test or harness
[23:12] <wgrant> And see what the actual error s
[23:12] <wgrant> is
[23:35] <StevenK> wgrant: Sorted it out. Did you want to review the MP I just put up?
[23:36] <wgrant> StevenK: Am doing so
[23:48] <wgrant> StevenK: Just a few things that need fixing.
[23:51] <StevenK> wgrant: Thanks. I've never quite figured out how IStoreSelector maps to I{,Master,Slave}Store
[23:52] <wgrant> StevenK: DEFAULT_FLAVOR == IStore
[23:52] <wgrant> MASTER_FLAVOR = IMasterStore
[23:52] <wgrant> SLAVE_FLAVOR = ISlaveStore
[23:52] <wgrant> Webapp code should very very very very very rarely use anything other than IStore
[23:54] <lifeless> cjwatson: nominations seem like overkill to me
[23:55] <lifeless> cjwatson: we had to cripple *who* could nominate because folk just nominated everything.
[23:55] <cjwatson> I'm aware of that
[23:55] <lifeless> good good
[23:56] <cjwatson> Nevertheless our defect analysts etc. need to be able to target bugs, but we don't want them to have full driver privs
[23:56] <cjwatson> note target not nominate
[23:56] <lifeless> so target is a superset of nominate.
[23:56] <cjwatson> I know
[23:56] <cjwatson> BugNomination.canApprove => target
[23:57] <wgrant> cjwatson: What undesirable privilege does being a series driver convey?
[23:57] <lifeless> I am confused; perhaps you can describe the things you /don't/ want the defect analysts to do, and how often they do them today.