[00:17] So, you have a dastardly plan? [00:17] wgrant: & [00:17] * StevenK grumbles at his fingers. [00:25] StevenK: Yeah, testing it on DF [00:38] * StevenK resurrects the redux branch [00:52] StevenK: I have an equivalent that is 1ms rather than 1000ms, but I'm not sure that it is truly equivalent yet. [00:54] wgrant: That was the problem? The query was taking 1 second per notification? [00:55] StevenK: Calculating the structural subscribers for a private bug took 600-1200ms [00:55] StevenK: And because the notification code is crap, it ran that calculation several times for a single operation. [00:55] I have a query which fixes the slowness [00:55] And I might refactor it all later to remove the duplication [00:56] Since four separate bugnotificationrecipient timeouts plagued my last week [00:57] wgrant: Ah, so the real smoking gun is the change in model/structuralsubscription.py [00:58] StevenK: I believe so [00:58] StevenK: Though dupe/etc subs may be similarly afflicted, they weren't important in this case. [00:58] (probably since private bugs with dupes are relatively rare) [01:15] StevenK: I've added the full suite of APGs on DF, so performance testing there should be more accurate. [01:22] * StevenK stabs django for lying [01:25] StevenK: oh? [01:28] lifeless: Figured it out. The docs could be clearer about lots of things. [01:29] * lifeless is no wiser [01:30] lifeless: Adding commands that manage.py can execute [01:42] Oh [01:42] EXPLAIN ANALYZE gets a bit confused when dealing with UNIONed subqueries [01:43] -> Index Scan using teamparticipation_team_key on public.teamparticipation (cost=0.00..30.16 rows=8 width=8) (actual time=0.016..0.173 rows=56 loops=5) [01:43] Output: public.teamparticipation.id, public.teamparticipation.team, public.teamparticipation.person [01:43] Index Cond: (public.teamparticipation.team = (2794)) [01:43] What it actually means is "Index Cond: (public.teamparticipation.team = [result of UNIONed subquery that happens to start with a literal 2794])" :( [01:43] Hah, helpfl [01:43] *helpful [01:45] wgrant: really ? [01:45] wgrant: can I see the full explain analyze + original query ? [01:47] lifeless: https://pastebin.canonical.com/71501/ [01:47] Observe the last couple of lines of the plan [01:48] And how they make no sense unless it actually means the full UNIONed result [01:50] Note also that the output of the HashAgg is '(2794)' because of that [01:52] (ignore the Hash Semi Join... I'm trying to work out what it's doing there) [01:55] What happens if you replace the select constant as grantee [01:55] with select grantee from (select constant as grantee) [01:55] I wonder if its misplanning entirely [01:55] -> worried [01:56] Nah, it just optimises that away [02:02] The plan seems correct apart from the hash semi join. [02:02] Which duplicates the other subplan, which is largely identical and more efficient [02:02] I don't understand why it thinks it also needs the hash semi join [02:09] 11294 Time Outs [02:09] 6415 / 0 Branch:EntryResource:getMergeProposals [02:09] ^ regression methinks [02:09] lifeless: That's the bug you triaged a week ago [02:09] ah' [02:09] (and yes) [02:09] i'm getting old in my young age. [02:12] you have a beard. bearded people are old. qed. [02:12] a 9yo expert in old people told me that. [02:12] spm: He forgot that already [02:12] Due to his age. [02:12] clearly [02:13] spm: '9yo expert' is a tautology [02:13] not from the 9yo's perspective. [02:13] ^^ relativity [02:13] thats special [02:13] StevenK: So, mind if I take over the structsubs branch? [02:13] wgrant: ... to do what? [02:14] StevenK: Make it fast. [02:14] StevenK: I mean the private structsubs branch [02:14] wgrant: If you like [02:14] Not the information type one [02:14] The model code has landed already. [02:15] The UI will happen tomorrow now that get_bug_tags() no longer exists [02:15] Right [02:15] I'm still trying to bend django to my will [02:16] It acts utterly and completly stuffed with :memory: sqlite [02:22] StevenK: :( [02:22] StevenK: it possibly is dropping all the handles to it [02:22] StevenK: if thats the case, use a random /tmp path for the db [02:28] lifeless: Yeah, I'm getting to that point [02:35] lifeless: I thought it was a process thing last Monday -- as in, if I could do syncdb and runserver in one process, it should be fine. It isn't. [02:35] StevenK: sorry for making your life hard [02:35] StevenK: /tmp will be the go [02:37] lifeless: Yeah, I have two choices there. Hard code /tmp/auditor.sql in auditor's config, or forcibly import auditor's settings in the fixture and mktemp [02:38] StevenK: you don't want a hard coded path :) [02:38] StevenK: If I can make a (little awkward but tolerable I think) suggestion. [02:38] Yes I do, it will be EASY [02:38] As opposed to that last thing you suggested [02:39] it will fail horribly when you have concurrency. [02:39] so, in your settings.py, consult an environment variable for the path [02:39] if its not set, use whatever path you were using before the :memory: experinment. [02:39] in the fixture, set a unique path using TemporaryDirectory to get a throwawy subtree. [02:40] and export it via EnvironmentVariable [02:40] (both are fixtures from fixtures) [02:41] The fixture already creates a temp directory for the logfile, so that should be easy [02:41] cool [02:48] lifeless: http://pastebin.ubuntu.com/1131833/ [02:50] cool [02:50] Now to hack the fixture [02:51] There'd better not be a default of a well-known path in /tmp, or I will be unhappy :) [02:52] wgrant: BLEH [02:52] Because debugging via connecting to the DB is for chumps or something. [02:53] wgrant: there will be, but the fixture will override it [02:53] wgrant: and prod deploys will use postgresql. [02:54] wgrant: so suck it up white boy :) [02:54] lifeless: The fact that /tmp exposes a "create item by name" interface is a historical mistake. [02:55] So a path of /tmp/auditor.sql doesn't make sense [02:55] As it's not possible by sensible means to create a file with that path [02:57] wgrant: So I'm supposed to make use of tempfile.mkstemp and just guess? [02:57] Guess? [02:57] Even though the fixture is going to override it and it's just for people wanting to run auditor's tests locally? [02:58] Is any callsite not going to override the default? [02:58] The test suite surely has to [02:58] If nothing uses the default, then the default doesn't need to be there. [02:59] There are three testsuites here -- auditor's, auditorfixture's and LP's. [02:59] The fixture will override the default, so it's fine for the fixture itself and LP. [02:59] the risk is that someone will attack your system by having a symlink to something you care about at that path [02:59] but they can't generate arbitrary contents [02:59] It's not a significant risk here. [03:00] But it's something that people will copy. [03:00] And it provides no value here [03:00] so put big warning messages around it. [03:00] => it should not exist [03:00] wgrant: whats your alternative ? [03:00] wgrant: whenever you find yourself saying 'has no value' please provide an alternative, as an existence proof, for something functionally equivalent. [03:00] wgrant: It provides value for people being able to run auditor's test suite [03:01] wgrant: and yes, I realise the irony that I didn't do that for ORM discussion. [03:01] wgrant: I should have. [03:01] Without having to set environment variable [03:01] lifeless: Well, in order to use the default, I'd have to copy auditor.sql to /tmp/auditor.sql before running auditor. It's surely just as easy to set AUDITOR_SQL before running auditor. [03:01] StevenK: Can't the test suite poke it in other ways? [03:01] wgrant: huh?! [03:01] Having the test suite use a single well-known commonly-conflicted path is not a good idea. [03:01] wgrant: Like I said before, which test suite? [03:01] wgrant: tell you what, to get StevenK moving along. [03:01] There are three. [03:02] StevenK: The one that would use /tmp/auditor.sql [03:02] Any test suite using that is non-concurrent [03:02] wgrant: understand we're *improving* the situation, it *used* to be just /tmp/auditor.sql always. [03:02] wgrant: so file a bug, and StevenK can look at it later. [03:02] Actually, it used to be /home/steven/auditor.sql ... :-( [03:02] StevenK: hah, same issue though. [03:03] Right [03:07] Ah, if it's already worse, then OK :) [03:11] Although, thinking about it, the testsuite doesn't use that, so the default can probably die [03:12] StevenK: one step at a time [03:12] StevenK: you may be right, but why yak shave when you don't need to. [04:55] wgrant: Are you winning at making private structsubs fast? [05:14] StevenK: Yes. [05:15] Trying to also win at not making the queries horrid [07:53] good morning === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [11:27] jcsackett: ping when you get in, quick ? for you === matsubara is now known as matsubara-lunch [13:56] rick_h_: ping. [13:57] jcsackett: pong [13:57] adeuring, here's my branch, if you don't mind: https://code.launchpad.net/~deryck/launchpad/support-pape-max-auth-age/+merge/117781 [13:57] jcsackett: nvm about earilier, don't think it'll work [13:58] deryck: ok [13:58] rick_h_: ok. [14:03] rick_h_: hangout still planned for now? [14:03] jcsackett: yea, just hangouts not playing nice and starting up for me atm [14:03] 5th try to get it to load/start [14:04] rick_h_, are we using our url, or you starting a random one? [14:04] deryck: I'll try our url next I think [14:04] was trying to start a new one with no success [14:04] worked for me. === matsubara-lunch is now known as matsubara [14:21] deryck: r=me [15:09] jcsackett: so I'm going to go think for a bit around lunch. Do you want to get together for a call this afternoon and kind of run through some points? [15:10] rick_h_: sounds like a plan. have a particular time in mind? [15:10] jcsackett: shoot for 3:00? [15:11] sounds good. [15:11] cool, thanks [15:21] adeuring, thanks for the review [15:53] sinzui: ping. [15:53] hello [15:53] sinzui: free to chat a bit? [15:53] yes [15:53] fantastic. [15:54] sinzui: hangout invite sent. === deryck is now known as deryck[lunch] === deryck[lunch] is now known as deryck [18:56] jcsackett: up for hangout? [18:56] deryck: will want to put you on deck perhaps after this one as I try to get stuff talked over pre-EOD (heads up) [18:56] rick_h_: i'm free. [18:57] jcsackett: awesome [18:57] rick_h_, sure. [18:57] rick_h_, I'm done in an hour, since I started early and am at the beach. Just to give you a heads up too ;) [18:57] deryck: ok, will try to go quick [18:57] rick_h_: we can keep ours short so you can talk to deryck. [18:59] rick_h_: can you send me the hangout invite? there are two of you in my G+ list and i'm not sure which to send the invite too. :P [19:02] deryck: o/ [19:20] mwhudson: welcome back, post-new family member arrival :) [19:21] nigelb: thanks! [19:23] mwhudson: wb [19:23] mwhudson: everything go smoothly ? [19:24] lifeless: ended up with a c-section due to baby getting stuck on the way out, but aside from that everything is good [19:24] mwhudson: woo (we did too) [19:27] ah right [19:27] it's a pretty intense 45 mins or so between them saying "we need to do a caesar" and getting it done :) [19:27] especially if everyone's been awake for 24 hours by that point [19:28] mwhudson: I have /no idea/ how long it took [19:28] tis all a blur [19:28] I remember the operating room vividly [19:28] heh yeah [19:28] i remember being taken away and forcefully fed some carbs [19:28] before, after or during ? [19:29] before the op [19:29] interesting [19:29] they didn't do that for me [19:29] perhaps my weight dissuaded them :P [19:31] deryck: still got 10min? [19:31] rick_h_ sure [19:31] deryck: cool, normal hangout url? [19:32] deryck: want to pick a time for a call? [19:33] lifeless, yeah. let's do my Thursday afternoon, your Friday morning. if that works for you. [19:33] its clear [19:58] sinzui: did my logic make sense w.r.t. mutually invisible teams in a project? [20:04] lifeless: yes [20:04] I asked Cody if they really use mutually invisible teams in a project? [20:20] maxb: *headdesk* ** 100 [20:25] indeed :-/ [22:59] wgrant: http://pastebin.ubuntu.com/1133382/ is the traceback [23:11] StevenK: Something under related_projects is throwing an AttributeError, probably. [23:12] You may want to call it manually [23:12] in a test or harness [23:12] And see what the actual error s [23:12] is [23:35] wgrant: Sorted it out. Did you want to review the MP I just put up? [23:36] StevenK: Am doing so [23:48] StevenK: Just a few things that need fixing. [23:51] wgrant: Thanks. I've never quite figured out how IStoreSelector maps to I{,Master,Slave}Store [23:52] StevenK: DEFAULT_FLAVOR == IStore [23:52] MASTER_FLAVOR = IMasterStore [23:52] SLAVE_FLAVOR = ISlaveStore [23:52] Webapp code should very very very very very rarely use anything other than IStore [23:54] cjwatson: nominations seem like overkill to me [23:55] cjwatson: we had to cripple *who* could nominate because folk just nominated everything. [23:55] I'm aware of that [23:55] good good [23:56] Nevertheless our defect analysts etc. need to be able to target bugs, but we don't want them to have full driver privs [23:56] note target not nominate [23:56] so target is a superset of nominate. [23:56] I know [23:56] BugNomination.canApprove => target [23:57] cjwatson: What undesirable privilege does being a series driver convey? [23:57] I am confused; perhaps you can describe the things you /don't/ want the defect analysts to do, and how often they do them today.