[00:13] wow [00:13] some hair shows up when you dig [00:13] found a code path that disables prejoins for speed [00:13] (ok, fine so far) [00:13] then, in the inner code it was calling, there is an if block that uses Storm rather than sqlobject joins, and so the prejoins don't get disabled ... [00:26] Slow branch scanner makes me sad :( [00:26] latency, throughput or capacity? [00:27] * rockstar picks all three [00:27] Well, it took nearly four minutes from push to bug link. [00:27] wgrant, that's odd. I've never seen it take more than two. [00:28] rockstar: call? [00:29] thumper, I thought you'd never ask. [00:30] * rockstar repeatedly slaps skype and pulseaudio === rstat1-brb is now known as rstat1 [00:45] oh ffs [00:46] sqlobjectresultset is evil [00:53] why's that? [00:54] because it doesn't implement IResultSet [00:54] Isn't that the point? [00:54] so stuff like canonical.lib.components.decoratedresultset fail spectacularly when wrapping SQLORS [00:54] wgrant: well, we want to migrate [00:54] wgrant: yes [00:54] you'd think it would be...given the name [00:54] rstat1: yes [00:55] got anymore ideas as to why my openid doesn't work? [00:55] rstat1: I don't, sorry : I haven'ted poked at openid at all yet [00:56] ahh...well ok. [00:56] lifeless, hi [00:56] Ursinha: hi! [00:57] lifeless, so, are you willing to work on the rollback thing? :) [00:57] yes, I'd love to [00:57] I'm putting it in my queue right after fixing the fallout from trying to use DRS around a SQLRS [00:57] lifeless, cool [00:57] rstat1: What if you revert to the stock LP Apache config? [00:57] as its a different branch, I can work on it concurrently [00:58] Ursinha: remind me of the log [00:58] lifeless, mars worked today on buildoutifying qa-tagger [00:58] qa-rollback means 'it has been rolledout [00:58] I'm merging then you can branch it [00:58] wgrant: I could try that..One seccond [00:58] and thus qa-rollback means 'can deploy' [00:58] lifeless, yes, qa-rollback is, well, a rollback, so it's safe to land [00:59] I guess it's well described on a wiki page [00:59] let me find the link [00:59] Ursinha: so, its a one-liner I think. [00:59] one sec [00:59] lifeless, https://dev.launchpad.net/QAProcessContinuousRollouts [01:00] lifeless, tagger also has to identify which revision they're rolling back [01:07] lp:~lifeless/qa-tagger/qa-rollback [01:08] Ursinha: devs can manually tag that initially though, right? same as tagging qa-ok ? [01:08] well of course it'd work when I did the default config [01:08] lifeless, no, the script sets that [01:08] So its definitely an apache issue [01:08] Ursinha: does or should [01:08] lifeless, should [01:08] Ursinha: how should it ? [01:09] that's what you're supposed to change [01:09] lifeless, identifying a [rollback] tag in the commit message [01:09] ok, let me look at that stuff [01:11] Ursinha: how are you generating your test data for test_commit - [01:11] lifeless, in sample_data.tar.bz2 there's a branch [01:12] :( [01:12] you might like to migrate to self.make_branch_and_tree; its much more convenient IMO [01:13] Ursinha: how do you propose rollback will mark what commit its rolling back ? [01:13] lifeless, when a person submits a branch that is a rollback, it marks the commit message as such [01:13] I get that [01:13] that's adding [rollback] [01:14] ah [01:14] but how do you propose we determine *what* branch is being rolledback [01:14] lifeless, sorry [01:14] if you said [01:14] lifeless, by bug number [01:14] but the rollback doesn't have one itself, does it ? [01:14] we have two commits, A and B [01:14] A says 'bug X' [01:14] B says 'rollback' [01:14] according to the wiki page [01:14] if a branch/bug is stuck with qa-needstesting or qa-bad, and the next commit is a rollback, you allow that last stuck revision to be rolled out [01:15] we can look for tags in bug X when we're looking at commit A [01:15] Ursinha: it won't be the next commit though [01:15] Ursinha: it will be many commits - the pqm queue depth - later [01:15] lifeless, hm, right, so we would need to guarantee that all the others in between are ready to land [01:15] hm... [01:16] Ursinha: well thats a separate problem. [01:16] I recall discussing that on the epic [01:16] Ursinha: here's what I think [01:16] developers should land a rollback branch [01:16] and manually mark the bug as qa-rollback [01:16] the rollback commit *itself* gets treated as deployable [01:16] right [01:16] so if we have three commits [01:17] A - faulty [01:17] B - ok [01:17] C - rollback [01:17] when we process them, we see [01:17] A:-> bug X -> tagged rollback [01:17] B:-> bug Y -> tagged qa-ok [01:17] C:-> rollback [01:17] and we let all three past in this case [01:18] if C has landed but bug X is still tagged qa-bad, we assume that C was rolling back something else. [01:18] what do you think? [01:18] we can look at a fancy way to figure out what C rolled back later [01:18] reading [01:19] but right now we have a many branches with revnos that differ etc, so I'd be worried about adding a revno to the rollback message, or a bug. [01:19] OTOH we could add [rollback=X] [01:19] to mean 'rolls back bug X' [01:20] that was the idea, but indirectly, marking the commit as [rollback] and it having a bug linked [01:20] linking a bug to it would be hard today, as neither bzr nor lp support links other than 'fixes' [01:20] lifeless, so your idea is to not depend on the script to identify which bug it's rolling back? [01:20] Ursinha: as a first iteration [01:21] Ursinha: I want to achieve 'ready to switch to this layout' _asap_; and improve once we're ready. [01:21] lifeless, ok [01:21] spm: btw ping [01:21] so we have rollback and bad in the same bug, or only one of them? [01:21] tags, that is [01:22] Ursinha: I think for sanity we should only have one qa-STATE tag at a time [01:22] if its bad, we're blocked on a matching rollback [01:22] right [01:22] hmm [01:22] that was my understanding [01:22] actually - ah, got it. [01:22] so, I propose we say [rollback=REVNO-ON-DEVEl] [01:23] it is more work than I'd like [01:23] but if we don't do that, we'll deploy something broken, I guarantee it. [01:23] lifeless: yo [01:24] and when we set qa-bad we need a revno too [01:24] Ursinha: what time is it for you ? [01:24] lifeless, 9h24pm [01:24] spm: heya. Staging prod shema QA environment [01:24] Ursinha: ok, I'll write up a proposal for you [01:24] and mail it to you and mars. [01:24] lifeless, are you doing this right now? [01:25] I need to play on staging a bit and see if what I have in mind will work, and I don't want to keep you up late just waiting on me :) [01:25] lifeless: I'll need a little more context than that :-) [01:25] lifeless, because I'm merging two branches to trunk that would be better for you to write code on top of them [01:25] Ursinha: I'll deal [01:25] if you want me to work on em, land em now :) [01:25] lifeless, okidoki [01:25] spm: rt 40482 or something like that [01:27] spm: yes, thats the one [01:28] spm: we've missed you for a week, I wants to monopolise you now :) [01:28] .... [01:28] welcome back ? [01:28] it's good to be 'needed'; for values of 'need' :-) [01:31] :) [01:31] Ursinha: basically what I'm going to arrange is enough data flow that commits in devel that are messed up are persistently recorded as messed up [01:31] Ursinha: and commits that fix messed up ones record in their commit message that they do that [01:33] lifeless, right [01:34] Ursinha: is it ok with you if I just edit the wiki page and ask you, Diogo and mars to tell me if I messed up ? [01:34] lifeless, that's ok, we have history there :) [01:35] Ursinha: we'll need to change the lp-land stuff for rollback [01:35] to take a parameter [01:35] lifeless, I was expecting that [01:35] ok cool [01:35] lp-land and ec2 land [01:40] Ursinha: ok, updated the page: qa-rollback state is gone [01:40] Ursinha: please see if it makes sense :) [01:40] spm: ok, so jocularity aside: are we there yet ? :) [01:41] wgrant, lifeless: of course now I'm getting "Couldn't find a distribution for 'zope.publisher==3.12.0'" *sigh* [01:42] jam: update your download-cache. [01:42] Or just run rocketfuel-get, which does all that for you. [01:51] wgrant: well, through a mix of various voodoo incantations, it finally works. Thanks for your help [01:51] (rocketfuel-get isn't quite enough because it assumes you have an unpacked 'devel' working tree, etc. but it seemed to do what I needed) [02:24] lifeless, seems ok [02:24] thumper, around? [02:25] I needs a query run on staging: http://pastebin.ubuntu.com/479695/ [02:25] lifeless, maybe you can help? ^^ [02:28] spm, ^^ [02:30] rockstar: not without the magic word [02:30] spm, even on staging? [02:30] Oh, er, please? [02:30] hahaha [02:31] * rockstar is American, so forgets his manners often... [02:31] rockstar: actually, the magic word, so SWMBO tells me: is **NOW**!!. fwiw... [02:31] spm, oh, then I should have gone with my instinct. [02:31] :) [02:31] heh [02:31] rockstar: 0 rows [02:32] spm, boo... [02:33] Ursinha: thanks [02:33] Ursinha: nearly done [02:34] lifeless, I'm feeling ill, I'll talk to mars and matsubara-afk tomorrow to be sure I'm not missing anything :) [02:34] Ursinha: go sleep! [02:36] spm, can you please try http://pastebin.ubuntu.com/479698/ **NOW**!! [02:37] rockstar: whazzup? [02:37] rockstar: same; 0 rows [02:38] thumper, I just needed some SQL queries run. [02:38] spm, and how close to production is the staging db? [02:38] Doesn't staging have its build queue cleared during the restore? [02:39] wgrant, ooh... [02:39] wgrant: along with several other tables of 'do stuff', I'd hope so. [02:40] spm, can lifeless give the okay to run the first query on production? [02:40] I can [02:40] thumper should by the new policy in preference to me, as hes your tl [02:41] lifeless: I could but I'm multi-tasking with non-laptop tasks [02:41] lifeless, yeah, but thumper's got his hands full. :) [02:41] ok [02:41] rockstar: aiui, we only need approval to run modification queries; selects are fine. happy to be corrected. [02:42] for now its mods that must get approval and be logged as incidents with high pri bug about not repeating it [02:42] spm, could you run the first query on production then? [02:43] rockstar: see. there's that missing magic word again. tsk tsk tsk. [02:43] spm, **NOW**!! [02:43] rockstar: 86 rows; need a paste? [02:43] heh [02:43] spm, not yet. Lemme get a better query first, one that actually tells me more. [02:44] 85... <== it's going down. [02:44] spm, wait, what? [02:44] * rockstar is confused [02:44] wgrant, around? [02:46] spm, point of clarification: you ran http://pastebin.ubuntu.com/479695/ correct? [02:46] rockstar: Indeed. [02:46] Why wouldn't it be going down? [02:46] You're looking for unstarted jobs. [02:47] sometimes b-m builds stuff [02:47] lifeless, nice performance tuesday follow-up mail, especially the steps you went through while solving. I learned a thing or two from it. [02:47] mars: cool, thanks [02:47] lifeless: Occasionally. [02:47] of course, now I'm fighting all sorts of shitty aqlobject-half-transition pain [02:48] rockstar: hrm. no. was using the '698 paste, but doing so with 695. [02:48] lifeless: Well, you could always port the problematic callers to something that isn't ancient. [02:48] rockstar: and 78 rows now. [02:48] spm, okay, I'd be interested to know if that one changes. [02:48] (I suspect it won't) [02:50] wgrant: have a look at BugTaskSet.search() and its callers [02:50] wgrant: (e.g. yes, I want to) [02:51] lifeless: Looks Stormy to me already... [02:52] return SQLObjectResultSet [02:52] in what possible world is that storm ? :) [02:53] wgrant: its got some bits one way and some another - and its /callers/ are depending on some particular behaviours [02:53] wgrant: its nuts. [02:53] lifeless: Ah, true. [02:53] Missed that bit. [02:53] rockstar: 73 now... :-) [02:53] spm, oh crap... [02:54] rockstar: This must be the first time I've heard someone complain about buildd-manager being too fast. [02:54] rockstar: need to talk about it? [02:55] wgrant, I'm complaining about the fact that my queries are proving my hypothesis wrong, and I don't like square one. [02:55] spm: is staging up? [02:55] thumper, possibly. One more query. [02:55] thumper: end of an upgrade cycle I think [02:55] * thumper waits [02:56] thumper: I suspect not. looks like we're in the security.py part of a DB restore/setup [02:56] spm: ETA? [02:57] Oh wait, crap. [02:57] "RSN" [02:57] it's been going for around 1h 45 atm. so soonish. [02:57] thumper: are we there yet. "" [02:57] thumper, yes, I'd like to talk about it, if you have some time. [02:58] spm: I know RSN means NFI but soon [02:58] ish [02:58] rockstar: sure [02:58] thumper: damn. you're onto me.... [02:59] rockstar: What's going on? [02:59] wgrant, investimigating the cause of https://bugs.edge.launchpad.net/launchpad-code/+bug/618860 [03:00] rockstar: What's the traceback? [03:01] That suggests that either the actual or estimated start time are wrong. [03:01] s/wrong/None. [03:01] Neither of which your query will identify. [03:02] wgrant, http://pastebin.ubuntu.com/479708/ [03:03] rockstar: So you're searching for WAITING with a NULL date_started. Every waiting build should satisfy that. Don't you really want to look for non-WAITING with NULL date_started? [03:04] wgrant, *facepalm* [03:04] In fact, you probably want RUNNING. [03:04] spm, can you please run http://pastebin.ubuntu.com/479709/ on production **NOW**!! [03:04] sadness, /participants on staging still timesout [03:05] lifeless: staging may not be up atm... is doing a DB restore/setup [03:05] rockstar: woo. 3 rows. [03:06] spm, okay, great. Let's wait a bit, I'll tweak the query, and we'll run that again to see if it changes. [03:06] What's their status? 1? [03:06] spm: I know [03:06] spm: the frontend is back [03:06] whats the oops sync freq for staging ? [03:07] it was 3 min at one point... [03:07] sigh [03:07] 1691S10 [03:07] when you have a cycle. [03:08] 6 minutes [03:08] hmm, I probably asked at 5 minutes... [03:08] its there [03:08] thanks [03:08] SQL time: 10011 ms [03:08] -sql time: 73 ms [03:08] Total time: 10084 ms [03:08] Statement Count: 8 [03:09] \o/ select count(*) ... [03:09] OTOH 8 queries is fairly good [03:13] so yeah, its finished the updated, still timing out :P [03:13] probably we want something more sophisticated than decoratedresultset here [03:14] because - count(*) doesn't need all the left joins [03:14] (OTOH count(*) there is a bad idea anyway) [03:23] thumper: "Compare with another revision" in loggerhead doesn't seem to do what I was assuming it would do [03:23] thumper: that being - to compare this revision with another revision [03:23] mtaylor: loggerhead has issues [03:23] mtaylor: but it does work, but the ui for it is hard to understand [03:23] thumper: ok. :) [03:31] \o/ 5.8 seconds to do the count(*) :< [03:32] lifeless: that's not good [03:33] I'll paste you the query [03:34] mtaylor: its not entirely unexpected [03:34] we have some attributes that are 1:M and we're selecting the first-one in correlated subqueries for a team with 174 folk in it [03:35] denormalasing and caching the preferred email and default archive would save a batshit amount of time, I suspect [03:36] or [03:36] it may be bad indexes or something on some component [03:36] I'm pulling sections of the query out and its staying the same ;) [03:37] also, I bet sodium just shat itself [03:38] so, back to qa tagging [03:38] spm: please time http://pastebin.com/CV4wDPE1 on a prod slave === mordred_ is now known as mtaylor [03:40] lifeless: 8.5 seconds [03:41] spm: grah [03:41] spm: need to find out why [03:41] and 2 x 6 seconds on repeats. [03:41] when sodium comes back I'll pokey pokey there [03:42] lifeless: https://pastebin.canonical.com/35966/ fwiw.... [03:42] oh god I hate views [03:42] have I mentioned that? [03:43] can probably make this fractional-second without that view in there [03:44] i do str you mentioning a mild dislike of views a few weeks back.... [03:45] Which view is it today? [03:45] validpersoncache [03:45] Ah, ValidPersonCache? [03:45] which is a LIE [03:45] is that really a view? [03:45] a stinking lie [03:45] It's not a view in the dev schema. [03:45] Or. [03:45] Wait. [03:45] It *is* a view, not a cache. That's right. [03:46] * wgrant headdesks. [03:49] ! [03:49] heh [03:50] rockstar: How is it Fix Released? [03:51] It was broken just an hour ago... [03:51] mwhudson: how do you specify a layer for the create_initialized_view? [03:51] mwhudson: what form does the layer param take? [03:52] mwhudson: class? or string? === almaisan-away is now known as al-maisan [03:52] wgrant, that's the glory of running sql to fix the bug. :) [03:53] you can probably haul what the view is checking into the rest of the query? [03:53] rockstar: But isn't there still a bug that created the broken data? [03:53] wgrant, the corrupted records were from before we added the cancel build button, and we were cancelling builds with raw sql. [03:53] Ah, right. [03:54] Sorry, I'm just a little wary about people waving away buildfarm data corruption bugs without explanation. [03:54] thumper: um, probably class [03:54] That tends to come back to bite us and cause things to burn down. [03:55] thumper: i'd lean to passing a request that provided the appropriate layer though, i think [03:56] mwhudson: attempting to pass the CodeLayer through [03:56] lets see... [03:56] I've been a bit more strict on where the views live [03:56] now all the tests fail :) [03:56] wgrant, yeah, I'm not about to sweep buildfarm bugs under the road. [03:56] yep that works [03:56] thumper: haha [03:56] good though [03:56] * thumper ndos [03:56] * thumper nods even [03:59] SELECT COUNT(*) FROM Person JOIN TeamParticipation ON TeamParticipation.person = Person.id LEFT JOIN KarmaTotalCache ON KarmaTotalCache.person = Person.id LEFT JOIN PersonLocation ON PersonLocation.person = Person.id LEFT JOIN Archive ON Archive.owner = Person.id LEFT JOIN EmailAddress ON EmailAddress.person = Person.id LEFT JOIN ValidPersonCache ON ValidPersonCache.id = Person.id WHERE TeamParticipation.team = 238131 AND NOT (Te [03:59] bah [03:59] anyhow [03:59] sorry [03:59] 19ms without validpersoncache [03:59] :) [04:00] mtaylor: ^ [04:00] wgrant: ^ [04:00] LEFT JOIN ValidPersonCache ? [04:00] I thought you said without? [04:02] lifeless: I wonder how hard it is to drop that view. [04:02] lifeless: lovely === al-maisan is now known as almaisan-away [04:02] thumper: as I said, paste fail [04:02] thumper: thats not the query without it [04:03] without it, and with the logic put back in via account directly [04:03] its 32ms [04:03] LEFT JOIN account on person.account=account.id WHERE TeamParticipation.team = 238131 AND NOT (TeamParticipation.person = 238131) AND (Archive.id IS NULL OR Archive.id = (SELECT MIN(Archive.id) FROM Archive WHERE Archive.owner = Person.id) AND Archive.purpose = 2) AND (EmailAddress.status IS NULL OR (EmailAddress.status = 4 AND account.status=20)); [04:03] is the end of the updated query [04:03] lifeless: How does the view make the plan so bad? [04:03] Also, EmailAddress.status IS NULL? [04:03] dunno [04:03] WTF? [04:04] wgrant: teams [04:04] status is NOT NULL. [04:04] Ah, right. [04:04] Fair point. [04:04] But I'd rather you checked Person.owner explicitly. [04:04] so, this isn't *quite* right, we're losing 4 participants somewhere, but I know the general shape that needs changing [04:05] wgrant: if you want to do this follow on fix, I'd be delighted [04:06] oh, I know why, I need an ACCOUNT.status is NULL in there [04:06] wgrant: you have to be very careful otherwise you exclude rows you don't want to [04:06] wgrant: or generate a torturous query plan [04:07] wgrant: semantically though, status is NULL IFF no row was returned [04:07] thats the correct end bit : AND (account.status is NULL or account.status=20))); [04:08] so you'd need a pretty deep nested check on person.owner to do it your way. [04:08] lifeless: I know. But you don't want to check that there's no emailaddress; you want to check that it's a team. [04:09] Why would it need a deep check? [04:09] so [04:10] consider this: [04:10] AND (EmailAddress.status IS NULL OR (EmailAddress.status = 4 AND account.status=20)); [04:10] assuming we don't have data corruption [04:10] mmm [04:10] let me get back to you in a sec [04:11] you are suggesting [04:11] AND (person.teamowner IS NULL OR (EmailAddress.status = 4 AND account.status=20)); [04:11] right ? [04:11] to replace the whole email address and account status checks [04:12] hi folks [04:12] heya jtv [04:14] wgrant: the problem is that that generates 663 rows, not 174 [04:14] lifeless: What. [04:15] hi spm [04:15] Something's not doing what we think it is, clearly. [04:15] whoo-hoo, FakeLibrarian just landed [04:15] lifeless: What are the two complete queries? [04:15] And what is their purpose? [04:15]