[00:06] I'm working on an app that authenticates users with lp/openid and I'd like to use their team memberships for context. I must be missing something because using OPENID_LAUNCHPAD_TEAMS_MAPPING_AUTO = True in conjunction w/ https://login.launchpad.net/ does not seem to do anything [00:06] Oh, and one important detail, I'm using django and the django-openid-auth library [00:11] timrc: I think you need to be whitelisted to query that via openid [00:11] to prevent harvesting; IMBW [00:11] lifeless: thanks! do you know who I would talk too? [00:11] timrc: anyhow, separately, don't use login.launchpad.net - login.ubuntu.com *is* login.launchpad.net [00:12] https://forms.canonical.com/lp-login-support/ [00:12] lifeless: right, thanks and thanks again [00:12] from the bottom of https://login.launchpad.net/ [00:12] bach [00:12] https://login.ubuntu.com/ [00:12] https://forms.canonical.com/sso-support/ <- that one [00:14] lifeless: great, thank you [00:14] jml: around? [00:40] lifeless - that SQL query [00:41] I wrote it out indented, was slightly perturbed to see lack of brackets between OR and AND clauses in the bug status selection portion [00:41] might be correct, but the operator ordering isn't totally clear [00:42] actually, on second reading, probably OK - I must avoid adding too many () for the sake of not having to remember operator ordering! [00:44] I trust the postgreql engine is smart enough not to execute the Auth lookup part of the query in the case where Bug.private = FALSE [00:44] EXISTS (SELECT BugSubscription.bug FROM BugSubscription, TeamParticipation [00:44] WHERE TeamParticipation.person = 882987 AND [00:44] BugSubscription.person = TeamParticipation.team AND [00:44] BugSubscription.bug = Bug.id) [00:44] could take a while if executed for every bug it has to consider in the list [00:46] thats a correlated subquery yes [00:48] I used to speak SQL at one point (A-Level computing) ;) [00:48] but not so much any more - its mostly read-only [00:49] I'm trying to run it on my local instance (although doesn't have such a big DB ;)) [00:49] hey if you've reformattted it to work [00:49] paste that to me in pastebin or something [00:50] not reformatted, just entered in psql [00:50] reindented it so I could read it [00:50] doesn't get any hits on my DB - so I suspect I didn't munge the product IDs properly to reflect my local instance [00:55] BugTask.bug NOT IN (SELECT bug FROM BugTag) AND [00:56] what about having NOT EXISTS (SELECT bug from BugTag where bug = Bug.id) [00:56] so you don't search the entire tag list for every bug in the DB for every bug we query? [00:57] or perhaps "BugTask.bug NOT IN (SELECT bug FROM BugTag WHERE bug = Bug.id)" [00:57] pcjc2: there were \\n in the thing I pasted [00:57] if you can paste me something ready to run [00:57] then I can do something interesting [00:57] ok - will try [00:57] I deleted the \\n and indented differenly, but never mind [00:58] LP ought to grow a "no wrap" mode for some comments IMO [01:00] http://ubuntu.pastebin.com/DVg1NzYm [01:00] try that [01:00] Just adds a "WHERE bug = Bug.id" to the tag sub-query [01:01] I'm curious what bit of code translates the tag query the user enters into that SQL though - other than the missing WHERE clause, it looks pretty efficient [01:02] (of course.. I'm not certain that adding the WHERE clause will help - it should in theory be able to do just as good a search without it, since it is filtered by that afterwards) [01:03] lifeless: hi [01:03] lifeless: do we have regex string matchers yet? [01:03] lifeless: just wondering if someone else has written them yet [01:04] thumper: in testtools trunk I think [01:04] :( [01:05] lifeless: are you able to run that SQL on the same DB as a real server? (If not, we need different owner IDs and product Ids) [01:06] Since the query is ~instant on my machine it probably needs testing on a large DB to reproduce the huge exec time [01:14] if that helps (not that I probably need to point you at this ;)), the SQL in question comes from lib/lp/bugs/model/bugtask.py ~ line 1369 [01:15] Sorry, line 1422 [01:15] exclude_clause = "SELECT bug FROM BugTag" [01:15] ok, 0.3 seconds [01:15] hmm [01:15] we don't have your data in prod yet [01:15] hang on [01:16] and 17 when executed before? [01:18] lifeless: https://pastebin.canonical.com/41738/ [01:19] 2nd run https://pastebin.canonical.com/41739/ [01:19] s[is this related to what I'm looking at? [01:19] spm: thanks; what does it return without the explain analyse - whats the found count ? [01:20] spm: ... (^___) [01:20] pcjc2: yes, its your query analyzed [01:20] don't have privs to look at that [01:20] o rows [01:20] ok, thats very interesting [01:20] that'd be 0. not o. [01:20] pcjc2: did you add braces perhaps ? [01:20] That comes from the fact it was a stupid query for our bug-set [01:21] ALL the bugs are tagged with something to determine which sourceforge tracker they came from [01:21] pcjc2: does it still oops if you try via the web UI ? [01:21] I only realised this when I tried to run it locally [01:21] * pcjc2 checks [01:22] Yes, it does [01:22] (Error ID: OOPS-1837O107) [01:22] Any chance I can see those pastebin postings? [01:23] ah yes [01:23] http://pastebin.com/732mWULr [01:26] most of the parts look like they were never executed [01:26] Is that on the production database? [01:28] yes [01:28] so the same thing the OOPs said took 17 seconds, executed in 0.369ms ?? [01:28] unless there was a transcription error [01:29] doesn't show me in the dump what product IDs were used [01:29] flacoste: I've removed the 'edge' series to avoid confusing folk [01:33] spm: again please - http://pastebin.com/raw.php?i=3S3RKCSY [01:34] spm: two by explain analyze and one for the result itself [01:36] pcjc2: that will be an hour or so [01:36] execution time, or just fitting the job in? [01:36] getting an spm timeslice [01:37] spm is a LOSA ? [01:39] yes [01:39] just for scale - what sort of size data-centre is all this running on? [01:40] data centres [01:40] The DB servers are amusing. [01:40] pcjc2: primary db server is 16 cores, 128GB of ram, db is ~250GB on disk [01:41] pcjc2: we have 2 active replicas with 8 cores eachm same ram, same db size on disk [01:41] I assumed some kind of DB cluster [01:41] so queries are load-balanced between all 3? [01:41] appservers direct read only queries to replicas when possible [01:41] writes go to the primary [01:42] was just about to ask that [01:42] does it gaurantee in-order processing? [01:43] no [01:43] so if you get a write to the primary, followed by a read at some point later (from the same app?), the write is going to be replicated and affect the read? [01:43] we so, uhm 4million page hits a day [01:43] if we serialised we'd be dead in the water [01:43] pcjc2: The DB is determined largely by the request type. [01:43] a write to the primary and then a read from a slave may read the old data [01:43] pcjc2: POSTs go to the master, most GETs to the slave. [01:43] pcjc2: So within a request it should be consistent. [01:44] Across requests, possibly not. [01:44] cool - this is computing on a scale larger than I ever knew before (writing MS Access databases ;)) [01:44] lifeless: that first analyze execute you posted the results for.. [01:45] was that the SQL query which LP executed, the one from the OOPS, or the modified one? [01:45] all the one you gave me [01:45] * pcjc2 checks i didn't screw it up [01:46] I've regenerated - its the new one I'm asking spm to run [01:46] I screwed it up... [01:46] The one I pasted you had product IDs replaced as I was running it on my local DB [01:47] I suspected it might [01:47] I saw the IDs ok in the new one you asked to execute [01:48] how does security work here - I presume at some level, it is because the LOSAs know who you are and won't ask them to execute a DROP *; [01:48] And presumably they scan the SQL and keep an eye for any nasty side-effects possible? [01:48] its because a) they know who I am and b) they will read the sql [01:48] I'm authed to freenode, etc. [01:49] I recall hearing the LOSAs don't like executing SQL much [01:49] (Or certainly - not tested and approved SQL) [01:49] thats nuanced [01:49] mutating sql we require a bug describing how we can avoid ever running the sql again [01:50] read only sql we only do as part of diagnosing faults [01:50] and every such request is an interrupt which reduces project work [01:50] so its moderately important to us not to interrupt them casually [01:50] so if there was some manual fixup required which could be done in a few SQL statements.. someone would have to file a bug and decide if LP needed to grow internal support for that feature etc.. [01:51] pcjc2: yes [01:51] pcjc2: and we'd have an incident report [01:51] our schema is extremely complexx [01:51] noted ;) [01:51] the chance of screwing something up on even simple sql is nontrivial [01:51] I've been reading bits of it for amusements sake [01:53] When I run ANALYSE EXECUTE here I get -> Index Scan using bugtag__bug__idx on bugtag (cost=0.00..8.27 rows=1 width=4) (actual time=0.003..0.003 rows=1 loops=267) for the tag lookup [01:53] that will be affected by the stats which affect th eplanner [01:53] also [01:53] I wouldn't obsess on the tag component [01:54] we need data [01:54] trye [01:54] But without the SELECT, there is rows=33445 loops=1 [01:54] So I'm guessing that it will show a difference in the query you analyse too [01:55] Is spm going to run ANALYSE EXECUTE on the original SQL LP emitted? [01:57] I'm going to have to call it a night - 2AM here - will read the bug report tomorrow to see if there was any development [01:57] (Please leave a comment if there is marked difference in analysis results with the tag part changed) [01:57] Btw.. in case you didn't hear it enough - Launchpad is AWESOME [01:57] thanks :) [01:58] Moving from Sourceforge to LP with our bugs has TOTALLY reinvigorated our two projects [01:58] Contributors are coming out of the woodwork with patches, participating in review, reworking their patches [01:58] On SF, patches went to die [01:58] thats awesome [01:58] We had to shut our -dev email list down to open membership due to Signal to noise ratio problems [01:59] LP bug comments have proved to be the -dev list we never had. Active, constructive discussion of the item in hand - far less OT [01:59] (And this is still only a couple of days into our migration and triage effort!) [02:00] I know SF is not a great shining example to compare against, but I've had several positive comments from users who far prefer LP as well now they've tried it [02:01] I wonder if we have a page for quotes ;) [02:08] lifeless: I've got a question if you've got a minute [02:10] Feel free to quote me if you want [02:10] Really -> bed now, night! [02:10] Project devel build (348): STILL FAILING in 3 hr 26 min: https://hudson.wedontsleep.org/job/devel/348/ [02:11] thumper: sure [02:11] lifeless: it is about the +build urls [02:11] lifeless: that bigjools replied to on launchpad-dev [02:12] lifeless: he mentions three points, but only one (IMO) is valid [02:12] lifeless: is it worth fighting or just add redirects? [02:12] whats the thread subject? [02:13] Url change for binary package builds [02:13] Project db-devel build (262): STILL FAILING in 3 hr 28 min: https://hudson.wedontsleep.org/job/db-devel/262/ [02:20] lifeless: are you reading or busy? [02:21] thumper: its dinner/drinks time, so multiplexing [02:21] :) [02:25] which one is valid ? [02:27] that people may keep ids around [02:28] my change altered ALL binary builds [02:28] not just ppa ones [02:28] so that is one of the three out [02:28] the second point I disagree with [02:28] the api stability argument may matter if the build objects were exported in '1.0 [02:28] ' [02:28] I talked with flacoste about it and he doesn't think it is an issue [02:29] he seemed to think that the url wasn't a big issue... [02:29] I think we need to decide if our apis are stable or not [02:29] changing the url space for existing 1.0 exported things is a big deal IMO [02:29] lifeless: in order to get this fixed probably best to just add redirects yes? [02:30] this is an oops issue for us [02:30] I think it probably is best overall, yes. [02:30] ok [02:30] it is more work [02:30] * thumper gets on it [02:30] it is [02:30] the work is complete as is [02:30] so yes, more work [02:30] OTOH the work makes the transition smoother during the deploy [02:30] * thumper shrugs [02:30] I'll get it done [02:31] cool [02:45] thumper: hey, do you think you could land mkanat's loggerhead version bump? [02:45] lifeless: where is it? [02:46] thumper: https://code.launchpad.net/~mkanat/launchpad/lp-loggerhead-update/+merge/45787 [02:47] lifeless: yep [02:47] awesome, thanks! [02:50] gah [02:50] can't lp-land it [02:50] as I don't have that branch [02:50] isn't there a setting to say "land that one" [02:51] if there isn't, there should be a bug asking for that [02:51] I'd be happy for such a bug to be triaged 'high' [02:54] lifeless: which plugin? [02:55] lp-submit I think [02:55] perhaps pqm-submit [02:58] lifeless: it is in the pqm plugin [03:00] lifeless: is devel open? [03:00] lifeless: topic says no [03:03] oh [03:03] hmm [03:04] we should be able to open again as soon as we're qa'd up past the db merge. [03:04] I guess that isn't in the process docs yet or something, or perhaps we aren't qa'd to that point yet. [03:38] spm: hi [03:38] lifeless: ho! [03:39] spm: did you see my analyze request ? [03:40] oh no. sorry. doing now. [03:42] spm: new post - http://pastebin.com/raw.php?i=5HBhT6R4 [03:43] ugh. 15+ seconds on getting the explain analyse [03:43] yes, thats why we're asking for it ;) [03:43] data isn't in staging yet [03:45] ew. https://pastebin.canonical.com/41741/ 1 minute just for the analyse. [03:45] spm: 2 plox, + the actual result [03:46] yah. 2nd runing now. [03:50] . wireless . at . hotels .. sucks [03:50] count [03:50] ------- [03:50] 9 [03:50] (1 row) [03:50] lifeless: ^^ [03:50] ??????? [03:51] what was the second analyze? [03:54] I didn't paste that? oops. one sec. [03:54] note that I can see [03:54] https://pastebin.canonical.com/41742/ [03:54] no I didn't meant to add it to the results above [03:54] didn't meant to... [03:58] around for a little? [04:02] spm: can you try [04:02] http://pastebin.com/LcA17msn [04:04] lifeless: https://pastebin.canonical.com/41743/ I like that version better [04:07] thanks spm [04:07] me too [04:29] Wow, KDE is stupid. [04:29] ... [04:29] I installed python-kde4 last night to try to debug the SIGCHLD thing. [04:29] I just logged out and back in, and now the test suite causes a KWallet prompt followed by a segfault. [04:29] \o/ [04:31] Clearly, KDE is for people who want a broken Launchpad. [04:31] It's probably not Launchpad's fault that there's a segfault. [04:31] That is entirely down to KDE :) [04:35] spm: Hi. [04:37] wgrant: yo! [04:38] yoyo [04:38] spm: Could you please run http://paste.ubuntu.com/552687/ on prod? [04:40] architecturetag | count [04:40] -----------------+------- [04:40] ia64 | 1982 [04:40] sparc | 1983 [04:40] (2 rows) [04:40] wgrant: ^^ [04:40] Damn. [04:40] OK, thanks :( [04:41] wgrant: pitti is having lots of fun with your VGA cable [04:42] StevenK: Oh? [04:42] spm: http://paste.ubuntu.com/552689/, if you could. [04:43] wgrant: Martin's laptop with Natty, 52" TV in the hotel room, do you need a calculator? [04:43] StevenK: Ah, of course, making Unity crash even more spectacularly than usual! [04:45] wgrant: Sadly, it didn't crash once. [04:45] wgrant: https://pastebin.canonical.com/41744/ [04:45] StevenK: It doesn't do anything *but* crash on my desktop. [04:46] spm: aaaaaaaaaaaaaaaaa [04:46] heh [04:47] wgrant: Can has bug with a backtrace? [04:47] wgrant: is bug 78845 still true? [04:47] <_mup_> Bug #78845: publication process should allow per-release holds < https://launchpad.net/bugs/78845 > [04:48] StevenK: Possibly. [04:48] StevenK: It's a bit hard. [04:48] Because fglrx is awful. [04:48] And likes to hang once Unity crashes. [04:48] Oh, ye gods. [04:48] well there is your bug [04:48] Although even vanilla Compiz crashes sometimes now. [04:48] It was stable in maverick :( [04:48] lifeless: I don't think that's ever been true. [04:49] And it's even less true now that security uploads don't go through the queue. [04:50] I suspect you're on your own, masochist [04:52] spm: https://pastebin.canonical.com/41745 :( [04:52] wgrant: https://pastebin.canonical.com/41746/ [04:53] * spm is an SQL bot [04:53] Today, you are. [04:53] spm: Thanks. [04:53] * wgrant goes off to cry in the Soyuz corner. [04:53] * StevenK cries at test_hasbuildrecords.py [04:54] Yes, let's use broken sampledata that's been changed by use of removeSecurityProxy() and THEN push it through code. Ja. That would be fun. [04:57] Oh, they're not actually in the primary archive. [04:57] Phew. [05:01] * StevenK grumbles at sourcepackage.py [05:01] # End of duplication (see XXX cprov 2006-09-25 above). [05:01] What XXX by cprov?! [05:02] StevenK: IIRC the original XXX is in one of the duplicates. [05:02] But not the other. [05:02] Must. Resist. Throwing. Laptop. [05:03] It's a ThinkPad, it'll survive. [05:03] If it's closed. [05:03] True. [05:06] so close it, throw, then re-open? [05:07] Hah [05:07] * StevenK rings Chubb first ... [05:27] (Pdb) print self.builds[0].build_farm_job.date_finished [05:27] None [05:27] Ah ha. Hello, smoking gun. [05:29] * wgrant pushes delayed copies off a cliff. [05:36] spm: http://paste.ubuntu.com/552703/ <- last one, but there'll be about 4000 rows returned. [05:36] do you need all 4k? [05:37] Ideally. [05:37] We have at least three variations of the bad data... I'm not sure how many more there are. [05:39] spm: Only give him 3,999. He'll never know ... [05:39] StevenK: you are kidding right? this is *wgrant* we're talking about. [05:40] Bwahaha [05:40] amusingly, it's 3969 rows [05:40] well, lines of output. [05:41] I was expected 3965 rows of data. [05:41] Which is one off. [05:41] Odd. [05:43] wgrant: https://devpad.canonical.com/~spm/wgrant.out [05:43] (3965 rows) [05:44] spm: Thanks. [06:09] Sounds like it's a good time to visit Brisbane. [06:09] brb [06:36] Is there a reason we don't have a PropertyCache-based memoization decorator yet? === almaisan` is now known as al-maisan === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [08:44] good morning [09:00] Good morning adeuring ! [09:00] Hi al-maisan! [09:01] how are things! [09:01] sad to see you leaving [09:01] s/!/?/ [09:01] adeuring: das ist der Lauf der Dinge :) [09:02] althings are fine here. Looking forward to the epic [09:03] nice :) when and where is the epic taking place? === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [09:11] morning === al-maisan is now known as almaisan-away === mrevell__ is now known as mrevell === mrevell_ is now known as mrevell === Ursinha-afk is now known as Ursinha === mrevell_ is now known as mrevel === Ursinha is now known as Ursinha-lunch [14:13] leonardr: what's the qa status of bug 686690? is it qa-able? [14:13] <_mup_> Bug #686690: 1.8.0 breaks login_with() API compat with existing credentials files, and forces keyrings

< https://launchpad.net/bugs/686690 > [14:14] jcsackett: yeah, it's qaed, let me just make sure the new version made it into launchpad [14:14] leonardr: ok. thanks. [14:16] jcsackett: yeah, looks good [14:17] leonardr: excellent. that's the last bug blocking the deploy-stable queue. [14:17] thanks, leonardr. [14:18] np === matsubara is now known as matsubara-lunch [15:00] jcsackett: do we have revision number? [15:00] flacoste: unless someone contacts me with something that needs RC and landing by midday today, r12177. [15:00] jcsackett: how about doing a nodowntime today of 12176? [15:01] flacoste: i'm comfortable with that. [15:01] jcsackett: can you make it happen? [15:02] we can't do nodowntime to codehosting yet can we? [15:02] mwhudson: i don't know. flacoste? ^^ [15:03] i thought we could nodowntime anything that didn't require db changes, but i could be quite wrong. [15:03] mwhudson: we can't no [15:04] ah well [15:04] mwhudson: codehosting isn't part of nodowntime yet, we have a RT in progress to fix that though [15:04] cool [15:04] jcsackett: soyuz FTP services, librarian and codehosting are not part of the nodowntime set [15:04] because deploying there actually causes downtime to the services [15:04] I had an idea about ftp services [15:04] we have RTs to fix codehosting and librarian [15:05] and ftp :) [15:05] soyuz FTP services will require some engineering [15:05] bigjools: i thought ftp was more than a RT? [15:05] flacoste: given that we never, ever change the ftp code, I filed an RT to have 2 deployment trees, one for FTP and one for everything else [15:05] flacoste: let me scan the deployment queue again then and make sure we don't have anything that couldn't be nodowntime then--i haven't been reviewing the queue with that in mind. [15:06] then we can roll out to germanium's "everything else" tree as part of the nodowntime set [15:06] jcsackett: most things [15:06] bigjools: that would be a nice incremental improvement! [15:06] bigjools: we can do an active-passive mode for ftp / sftp [15:07] flacoste: rt 43238 if you want to poke [15:07] bigjools: wgrant seems to think its harder than that though, and was going to speak to you about it [15:07] lifeless: I think it will be hard too [15:07] since we can't interrupt in-progress uploads [15:07] sure [15:07] same constraint we have on appservers [15:08] so needs a bit of engineering - my 2-tree solution is much nicer I think [15:08] I think its a good short term fix, I don't think its a good long term fix because we want to eliminate skew [15:08] the only problem after that is worrying about a SPOF [15:08] skew? [15:08] lifeless: most things? [15:08] there is no skew [15:08] the ftp code never ever changes [15:10] the losas monitor deployed versions [15:10] if we have two deploys on a machine [15:10]