[01:05] wgrant: you just have to poke don't you? [01:05] wgrant: messing with whiteboards [01:08] wgrant: what awas that oops you mentioned earlier ? [01:14] thumper: That was my "Ahem, oops.", yes. Forgot I wasn't on staging. [01:14] (also, I really shouldn't be able to write to that whiteboard at all) [01:15] lifeless: OOPS-1681S1580? [01:16] Hm. No (LO|G)SAs during release week. Great. [01:16] :) [01:16] there should be someone interruptable [01:16] but you are right [01:16] no one on hand right now [01:27] mwhudson: can you have member functions be function decorators? [01:28] * thumper thinks [01:28] thumper: what does that mean? [01:28] I'm trying to clean up some test code [01:28] but with a function decorator, I don't need the self.addCleanup I guess [01:28] thumper: self.addCleanup is generally a good thing [01:29] thumper: can you describe a bit more, or pastebin an example ? [01:29] sure [01:29] liefhttp://pastebin.ubuntu.com/475182/ [01:29] lifeless: http://pastebin.ubuntu.com/475182/ [01:30] stab stab stab opening [01:30] openind [01:30] lifeless: we repeatedly have the lines 4-9 in a number of tests [01:30] I wanted to have a simple test function decorator to do the config push and pop [01:31] you could [01:31] but I'd put it in a fixture [01:31] using installfixture [01:31] example somewhere? [01:32] class BuilderConfig: [01:32] def setUp(self): [01:32] bzr_builder_config = """ [01:32] ... [01:32] def tearDown(self): [01:32] does the fixture need a tearDown? [01:32] config.pop("bzr_builder_config") [01:32] self.installFixture(BuilderConfig()) [01:33] now a question would be, is it worth making a fixture, or just have a method? [01:34] fixtures can be used by different test classes [01:34] so yes, its worth it [01:37] wgrant: [01:37] Module lp.code.model.diff, line 73, in text [01:37] self.diff_text.open() [01:37] Module canonical.launchpad.database.librarian, line 110, in open [01:37] self._datafile = self.client.getFileByAlias(self.id, timeout) [01:38] Module canonical.librarian.client, line 349, in getFileByAlias [01:38] raise LookupError, aliasID [01:38] LookupError: 52704649 [01:42] lifeless: Yeah, that makes sense. Thanks. [02:18] \o/ cleanup part 2 done. [02:18] poolie: around ? Would like to land your flags branch - have you looked at the failures from friday, or should I do that ? [02:18] wgrant: what do we do with stuck builders? [02:18] what sort of privledges do you need? [02:18] take em out back and shoot em [02:19] thumper: isn't there a pushConfig on lp.testing.TestCase already? [02:19] i know i've written it somewhere... [02:19] mwhudson: you know, I think you might be right [02:23] lifeless: do you have a rifle? [02:32] WTF? [02:32] admins don't have launchpad.Special for IPerson? [02:38] thumper: It needs a lamont. [02:38] thumper: Which builder? [02:38] thumper: That's the point of .Special. Admins don't have it. [02:38] That is its entire purpose. [02:38] Why it's used, I don't know. [02:38] wgrant: see #launchpad for stuck builders [02:38] Only for SSH keys, AIUI. [02:38] wgrant: and account deactivation [02:38] Craaap, not terranova. [02:38] It's not meant to die. [02:39] Nothing that can be done without a GSA. [02:39] And lamont, normally. [02:50] wgrant: are the build jobs actually instantiated by the build master before dispatching to a builder? [02:50] wgrant: I'm wanting to oops on a recipe for a deactivated person before it gets to the actual builder [02:51] thumper: Yes -- the builder can't talk to the DB. [02:51] The master has to do everything. [02:51] ah ha [02:51] so the recipes are taking out the build master? [02:51] Yes. [02:51] -> the entire farm. [02:51] and it can't handle exceptions? [02:51] Anything outside lib/canonical/buildd runs on the master. [02:51] that's a bit poor [02:52] It is somewhat. [02:52] But it's not that bad. [02:52] ORLY? [02:52] wgrant: so where is the code in the buildmaster that is dying? [02:53] when the job is barfing on no email? [02:53] thumper: In recipebuilder.py, where I showed you before. Or do you mean what calls taht? [02:53] I've fixed the recipe owned by team with no email part [02:53] How have you fixed that? [02:53] now looking at the possibility of a person deactivating themselves after a recipe job has been created [02:54] I use the recipe registrant instead [02:54] and say email sent on behalf of the team [02:54] in that edge case [02:55] I want the job to blowup earlier though in the situation where we can't get a preferred email for the team or registrant [02:55] which means an oops [02:55] Erm. [02:55] Sent on behalf of the team? [02:55] which means that the code has to handle it [02:55] as the sender [02:56] The crash is not sending an email. [02:56] It is producing an email address to be used in the changelog. [02:57] brb [02:59] oh... [02:59] hmm... [02:59] best fix that then... [03:02] well, setting the author_name to "Eric the Viking (on behalf of Vikings team)" [03:02] still seems to make sense to me [03:02] for a build author [03:02] as it should be the team, but there is no email address set [03:03] although, I'm tempted to walk up the team.owner chain until we hit something with a preferred email [03:03] as we know we'll get someone eventually [03:03] as disabled accounts aren't part of teams [03:04] thumper: there is a failure mode there ;) [03:04] lifeless: which is? [03:04] thumper: two in fact - do we permit circular team ownership [03:04] a team with one disabled member? [03:04] team A owns team B owns team A [03:04] I'm pretty sure we have DAGs for owners [03:05] secondly, you may run up a team list until you find mark :) [03:05] lifeless: so your solution would be? [03:05] what you're doing [03:05] check that there is a DAG constraint is all [03:06] what I'm doing is to use the registrant of the recipe [03:06] an alternative would be to say 'source-recipe-builder@launchpad.net' [03:06] which could be a disabled person [03:06] I'm pretty sure that soyuz does something like that for the actual buildd [03:07] hmm... I reckon we use the recipe owner first, and if that isn't valid, use the @lp.net email [03:10] Back. [03:10] Using the owner or registrant seems wrong. [03:10] I'd construct some special email address. [03:10] Like Bugs does. [03:11] Lying about the email address is less bad than lying about the name. [03:12] s/owner/team owner/ [03:15] I think having a dedicated email to represent the things lp did makes sense [03:15] its a little odd what ppa signing keys do, which is a similar situation [03:15] How's what they do odd? [03:15] they claim to be me [03:15] They say 'Launchpad PPA for Somebody' [03:15] No email address. [03:15] hmmm [03:16] I remember a bug from somewhere [03:16] There is the great Bugs discussion. [03:16] But that seems to have been resolved fairly reasonably. [03:18] wow, I'm nearly at the end of my code queye [03:18] bwah [03:18] queue [03:18] Oh? [03:18] two branches to go [03:19] /participants timeouts [03:19] Ah, right. [03:19] and ffeature fflags [03:21] I've got a tagger that alerts about deployable revisions [03:22] we need to do something to make that releasable [03:23] Is the new system going to be open source? [03:24] yes [03:24] the sticking point here is a filein the history that maps emails <-> lp ids [03:24] which maybe indexing is a bit nuts [03:25] Hmm. [03:25] so we may need to restart the history with a blank file, and have the deployed version have that file. [03:25] or something. [03:26] It can't use the API? [03:26] ENOIDEA [03:26] anyhow, if we get it open [03:27] with a blank file [03:27] you can fix it to not need the file ;) [03:27] Heh. [03:28] [I don't know any of the history or design of it - I spent a day hacking and rearranging, so I know some of it, but that bit I didn't touch] [03:29] If it's anything like Launchpad, we probably don't want to know the history.... [03:34] well [03:50] wgrant: do you thing the build author should be a special LP email all the time? [03:51] thumper: No. [03:51] or just when the owner doesn't have an email set [03:51] thumper: Yes. [03:51] * thumper nods [03:51] Or possibly when it's private. [03:51] Like Bugs now does. [03:51] But that's not clear. [03:55] brb [03:56] * thumper afk for kid collection [03:59] jkakar: around ? I has a storm question on left outer joins in tuple-result sets [04:01] the question is, are NULL matches represented as None, or Undef, or something else ? [04:09] also [04:09] is there a result set decorator already existing ? [04:09] if not, any suggestions where I'd put it [04:14] lifeless: You mean DecoratedResultSet? [04:14] possibly [04:14] where is that [04:14] c.l.components.decoratedresultset [04:14] wgrant: I want a thing that will handle a tuple-query, answer count() and the like, but call into some adaption code to support first(), any() and iteration [04:15] plus copy config etce tc [04:15] I believe that does what you want. [04:15] ok cool [04:16] brilliant [04:16] thumper: ^ [04:17] thumper: your code query-tuning stuff that uses a list should use that [04:23] wgrant: yup, exactly what I needed [04:29] lifeless: why? [04:33] thumper: listification starts to perform badly when you exceed batch_size [04:33] I don't listify anything bigger than the batch [04:34] I may be misremembering [04:34] sec [04:35] thumper: garh, lost the reference; what was the thing you tuned ? [04:35] lifeless: I've tuned lots of things :) [04:35] heh [04:35] ok [04:35] I'm a tuning machine [04:35] so you had something that was doing potato queries [04:35] and you pointed me at it [04:36] it was, perhaps, an API [04:36] ah no [04:36] it was the thing you were created a cached decorator for [04:37] ah, linked_bugs is one example [04:38] it listifies, it would be better if it was built on DecoratedResultSet, though its also got the challenge that it does permission checks late [04:39] _getRevisionsSinceReviewStart is another candidate for pushing more work to the db [04:39] thumper: I'm just noting places we can do better - the code pages perform well, as you know :) [04:40] lifeless: not as well as I'd like [04:40] thumper: well, tomorrow is performance day! if you want we can pair on analysing/tuning [04:41] I've got two^Wthree urgent things to get landed [04:42] ah well [04:42] how does one do subqueries in storm ? [04:43] [want to get coc signing with a person query [04:45] also [04:45] can anyone confirm the Class.q.column syntax is unneeded now, with storm ? [04:46] lifeless: Select(...) [04:47] store.find(..., self.some_id.in.(Select(...))) [04:47] actually, it is 'is_in' [04:47] I think Class.q.column isn't needed [04:47] personally I never used it [04:48] ok, so thats a subquery as a condition [04:48] to use it as a result [04:48] store.find((Person, Select()), conditions) ? [04:55] select person.id, exists (select * from signedcodeofconduct where signedcodeofconduct.owner = person.id) as signed_coc from person <- what I want [05:07] Exists is a subquery I think [05:07] * thumper looks at some code [05:08] hmm... [05:15] right, nearly got it [05:15] just need to get the exists into the where clause [05:15] lifeless: are you wanting it in the where clause [05:15] or the select cols? [05:18] >>> print [(p[0].id, p[1]) for p in store.using((Person,)).find((Person, Alias(Exists(Select(SignedCodeOfConduct, SignedCodeOfConduct.owner == Person.id, SignedCodeOfConduct)), name="signed")), And(Person.id >= 16, Person.id <= 17))] [05:18] [(16, True), (17, False)] [05:18] now to make it pretty [05:19] thumper: select cols, I knew what I meant :P -sorry for the confusion I'm sure I engendered [05:19] lifeless: yeah, I just couldn't think of anything else... [05:19] thumper: this sounds interesting [05:19] is this for a graph or for a feature in lp? [05:20] poolie: hi, see up above re ff [05:20] hi lifeless, i saw the features [05:21] i spent friday afternoon trying to work out why my lp dev instance was failing, and then getting a chroot up [05:21] i'm going to have another go at it now [05:21] poolie: ok cool. [05:21] poolie: if I can help, please shout. [05:21] * lifeless goes back to bashing up storm stuff [05:21] lifeless: you can go find((Person.id, ...), ...) [05:22] lifeless: to stop getting all of the people columns returned too (and chewing up non-sql time) [05:22] so lifeless, what are you doing with coc signing? [05:25] thumper: we want all the people columns [05:25] poolie: improving the performance of a registry API which is the top OOPS at the moment [05:26] cool :) [05:29] thumper: thanks for the hint though - will keep it in the toolkit [05:29] poolie: this is is the go-back-and-do-single-query pathology I was talking about [05:44] down to 23 queries to generate the page for a team with 3 members >< [06:01] lifeless: What's trying to find CoC sigs? [06:02] /participants [06:02] Hm. [06:02] because is_ubuntu_coc_signer is a property [06:02] just don't say that out loud [06:02] Ahh. [06:02] wgrant: you can look at my 'registry' branch if you want to see the arc [06:03] That name concerns me. [06:03] thumper: What's the error in the #lp OOPS? [06:04] lifeless: how ludicrous of a feature request would it be to add "pastebin url" to the list of metadata info kept for a project? (you know, in there in the laundry list of all the other urls it keeps) [06:04] AssertionError: Wrong account! 2401541, 3228204 [06:04] wgrant: ^^ [06:04] mtaylor: patches considered [06:04] thumper: Ah, he merged his person, most probably. [06:04] lifeless: ok [06:04] I believe that's scheduled for the next release or two. [06:04] so, API question [06:04] mtaylor: add the description to the project [06:04] there is a property [06:04] 'archive' [06:05] which is an exported Reference [06:05] the API outputs a _link for it [06:05] but it - and this is the nuts bit - creates the object to do that [06:05] We should probably kill that (it's been deprecated for ages), but that's another story. [06:05] thumper: so - the pastebin I'm working with has a remote api... so I was thinking that a "bzr paste" command would be a nice way to spit a diff into a pastebin ... but it would be even cooler if bzr could look up _where_ to paste on the project via launchpadlib [06:06] wgrant: so, if its in 1.0, we're screwed ? [06:06] lifeless: Of course. It needs to know the name to get the canonical_url. [06:06] Well. [06:06] I think we should take API versioning with a grain of salt. [06:06] by needs you mean 'its a constant pattern wtf' [06:06] ? [06:06] thumper: pretty low on the list of things I want to get done -but I just wanted to see if was a waste of time to even put it on the todo list [06:06] :) [06:06] How else can the URL be created? [06:07] wgrant: we have the person, the url is a constant suffix to the person canonical url [06:07] wgrant: we don't need a DB lookup to do that [06:07] lifeless: Erm, it shouldn't be constant. [06:07] It should be /+archive/NAME [06:08] wgrant: which for the first archive is _always_ ppa [06:08] lifeless: No. [06:08] lifeless: It's merely the default. [06:08] it's been changeable for quite a while now. [06:08] wgrant: unless its been changed, the form that you might be thinking of to let you change it actually ignores it on the first ppa [06:09] It changed within the last year. [06:09] (And broke a few things in the process.) === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [06:54] wgrant: why does my branch name concern you? [06:54] lifeless: Well, it suggests that you have one branch per app. [06:54] Rather than one per piece of work. [06:55] wgrant: I keep my queue very shallow [06:55] I've done 2 separate landings from my registry branch so far [06:55] both targeted and small [06:57] True. [07:02] of course, the thirds a doozy [07:02] but its still one thing [07:12] whats ValidPersonCache all about ? [07:13] also, \o/ 14 [07:16] lifeless: I have no idea [07:16] lifeless: probably histerical raisins [07:16] lifeless: worth asking the list though [07:16] someone might have a better answer [07:18] lifeless: "A materialized view listing the Person.ids of all valid people (but not teams)." :P [07:19] It doesn't seem to be materialised. [07:19] But that might just be the dev schema. [07:19] I ran into it this morning. [07:19] And WTFed a bit. [07:23] spiv: thanks :P [07:24] \o/ 11 [07:28] spm: what tz is your sprint in/ [07:30] lifeless: Is that including the auth[nz] and traversal queries? [07:30] They number quite a few themselves. [07:30] wgrant: yes [07:31] wgrant: its now constant at 11 for 2 members or 3 members [07:32] Excellent. [07:32] I'd appreciate an eyeball on the branch, revno 11316 [07:32] I'm merging devel into it now to find out about conflicts [07:34] and excellent, no conflicts. [07:36] lifeless: re. r11309, conjunction is the default. You can just use a sequence. [07:38] you mean the conditions variable ? [07:38] Yes. [07:42] Also, why do you get a whole KarmaTotalCache object, then switch on its .karma_total? Why not COALESCE the column directly in the query? [07:42] Saves you an if and some object construction, which must be good. [07:43] However. [07:43] Adding all this caching at the model layer does not sit well with me. [07:43] wgrant: so, coalesce - yes thats possible. [07:44] OTOH the pain we're feeling is primarily driven by latency here [07:44] It is. [07:44] and vertical bouncing [07:44] But let's make the DB server work for its dinner. [07:45] I'd like to see how it works on staging before tuning further [07:45] it may be so fast we don't care [07:46] or it may be sql work thats needed, or python in which case absolutely focus on that [07:46] so, caching. [07:46] is in invalidation that concerns you [07:46] s/in/it/? [07:46] It is. [07:47] are model objects persisted across transactions ? [07:48] and does cachedproperty caches get cleared (the storm cache has some stuff done to it) [07:48] No. [07:48] I believe it gets cleared, yes. [07:48] if the objects are dumped after a transaction thats suffiicent [07:48] its only if they are reused that the cachedproperty stuff would have a *huge glaring hole* [07:49] Oh yes. [07:49] But I'm worried about what happens if this is used outside a webservice request. [07:49] so, do you have alternative approach [07:50] There's no alternative approach possible at the moment. [07:50] so, this has the following properties: [07:50] - the caching will happen automatically [07:50] - precaching will require explicit action [07:50] - only the all_members_prepopulated takes that explicit action [07:53] But calling that will taint people throughout the transaction. [07:53] yes [07:53] the first point is more worrying to me [07:53] which is that Person.is_valid_person; change EmailAddress table; Person.is_valid_person won't change [07:53] but! [07:54] its no worse than having to call store.flush() [07:54] which is also needed. [07:54] Essentially we have a missing concept in our DSL [07:54] which is 'the database is slow' [07:54] Erm, store.flush() won't do this AFAIK. [07:54] wgrant: its an analagous, overlapping problem. [07:54] if you make a person valid, and don't call store.flush, is_valid will continue to return False [07:55] without my patch [07:55] Will it? [07:55] yes [07:55] Doesn't it make a query? [07:55] because its a view [07:55] Ah. [07:55] But... no. [07:55] it will query [07:55] It will cause a flush. [07:55] I didn't think every query caused a flush [07:55] It will flush before it makes a query, so the view will update. [07:55] I believe it does, or everything would be pretty broken. [07:55] ok [07:56] yet-another-reason to have an explicit step for [07:56] please talk to the storage layer [07:56] rather than things-that-look-cheap. [07:57] It would be really nice if we could statically analyse the code and just say 'prejoin everything kthxbye' [07:57] yes [07:57] however [07:57] in the absence of that, I have a few ideas. [07:58]