[01:05] <thumper> wgrant: you just have to poke don't you?
[01:05] <thumper> wgrant: messing with whiteboards
[01:08] <lifeless> wgrant: what awas that oops you mentioned earlier ?
[01:14] <wgrant> thumper: That was my "Ahem, oops.", yes. Forgot I wasn't on staging.
[01:14] <wgrant> (also, I really shouldn't be able to write to that whiteboard at all)
[01:15] <wgrant> lifeless: OOPS-1681S1580?
[01:16] <wgrant> Hm. No (LO|G)SAs during release week. Great.
[01:16] <thumper> :)
[01:16] <thumper> there should be someone interruptable
[01:16] <thumper> but you are right
[01:16] <thumper> no one on hand right now
[01:27] <thumper> mwhudson: can you have member functions be function decorators?
[01:28]  * thumper thinks
[01:28] <mwhudson> thumper: what does that mean?
[01:28] <thumper> I'm trying to clean up some test code
[01:28] <thumper> but with a function decorator, I don't need the self.addCleanup I guess
[01:28] <lifeless> thumper: self.addCleanup is generally a good thing
[01:29] <lifeless> thumper: can you describe a bit more, or pastebin an example ?
[01:29] <thumper> sure
[01:29] <thumper> liefhttp://pastebin.ubuntu.com/475182/
[01:29] <thumper> lifeless: http://pastebin.ubuntu.com/475182/
[01:30] <lifeless> stab stab stab opening
[01:30] <lifeless> openind
[01:30] <thumper> lifeless: we repeatedly have the lines 4-9 in a number of tests
[01:30] <thumper> I wanted to have a simple test function decorator to do the config push and pop
[01:31] <lifeless> you could
[01:31] <lifeless> but I'd put it in a fixture
[01:31] <lifeless> using installfixture
[01:31] <thumper> example somewhere?
[01:32] <lifeless> class BuilderConfig:
[01:32] <lifeless>    def setUp(self):
[01:32] <lifeless>     bzr_builder_config = """
[01:32] <lifeless> ...
[01:32] <lifeless> def tearDown(self):
[01:32] <thumper> does the fixture need a tearDown?
[01:32] <lifeless>     config.pop("bzr_builder_config")
[01:32] <lifeless> self.installFixture(BuilderConfig())
[01:33] <thumper> now a question would be, is it worth making a fixture, or just have a method?
[01:34] <lifeless> fixtures can be used by different test classes
[01:34] <lifeless> so yes, its worth it
[01:37] <lifeless> wgrant:
[01:37] <lifeless> Module lp.code.model.diff, line 73, in text
[01:37] <lifeless>     self.diff_text.open()
[01:37] <lifeless>   Module canonical.launchpad.database.librarian, line 110, in open
[01:37] <lifeless>     self._datafile = self.client.getFileByAlias(self.id, timeout)
[01:38] <lifeless>   Module canonical.librarian.client, line 349, in getFileByAlias
[01:38] <lifeless>     raise LookupError, aliasID
[01:38] <lifeless> LookupError: 52704649
[01:42] <wgrant> lifeless: Yeah, that makes sense. Thanks.
[02:18] <lifeless> \o/ cleanup part 2 done.
[02:18] <lifeless> poolie: around ? Would like to land your flags branch - have you looked at the failures from friday, or should I do that ?
[02:18] <thumper> wgrant: what do we do with stuck builders?
[02:18] <thumper> what sort of privledges do you need?
[02:18] <lifeless> take em out back and shoot em
[02:19] <mwhudson> thumper: isn't there a pushConfig on lp.testing.TestCase already?
[02:19] <mwhudson> i know i've written it somewhere...
[02:19] <thumper> mwhudson: you know, I think you might be right
[02:23] <thumper> lifeless: do you have a rifle?
[02:32] <thumper> WTF?
[02:32] <thumper> admins don't have launchpad.Special for IPerson?
[02:38] <wgrant> thumper: It needs a lamont.
[02:38] <wgrant> thumper: Which builder?
[02:38] <wgrant> thumper: That's the point of .Special. Admins don't have it.
[02:38] <wgrant> That is its entire purpose.
[02:38] <wgrant> Why it's used, I don't know.
[02:38] <thumper> wgrant: see #launchpad for stuck builders
[02:38] <wgrant> Only for SSH keys, AIUI.
[02:38] <thumper> wgrant: and account deactivation
[02:38] <wgrant> Craaap, not terranova.
[02:38] <wgrant> It's not meant to die.
[02:39] <wgrant> Nothing that can be done without a GSA.
[02:39] <wgrant> And lamont, normally.
[02:50] <thumper> wgrant: are the build jobs actually instantiated by the build master before dispatching to a builder?
[02:50] <thumper> wgrant: I'm wanting to oops on a recipe for a deactivated person before it gets to the actual builder
[02:51] <wgrant> thumper: Yes -- the builder can't talk to the DB.
[02:51] <wgrant> The master has to do everything.
[02:51] <thumper> ah ha
[02:51] <thumper> so the recipes are taking out the build master?
[02:51] <wgrant> Yes.
[02:51] <wgrant> -> the entire farm.
[02:51] <thumper> and it can't handle exceptions?
[02:51] <wgrant> Anything outside lib/canonical/buildd runs on the master.
[02:51] <thumper> that's a bit poor
[02:52] <wgrant> It is somewhat.
[02:52] <wgrant> But it's not that bad.
[02:52] <thumper> ORLY?
[02:52] <thumper> wgrant: so where is the code in the buildmaster that is dying?
[02:53] <thumper> when the job is barfing on no email?
[02:53] <wgrant> thumper: In recipebuilder.py, where I showed you before. Or do you mean what calls taht?
[02:53] <thumper> I've fixed the recipe owned by team with no email part
[02:53] <wgrant> How have you fixed that?
[02:53] <thumper> now looking at the possibility of a person deactivating themselves after a recipe job has been created
[02:54] <thumper> I use the recipe registrant instead
[02:54] <thumper> and say email sent on behalf of the team
[02:54] <thumper> in that edge case
[02:55] <thumper> I want the job to blowup earlier though in the situation where we can't get a preferred email for the team or registrant
[02:55] <thumper> which means an oops
[02:55] <wgrant> Erm.
[02:55] <wgrant> Sent on behalf of the team?
[02:55] <thumper> which means that the code has to handle it
[02:55] <thumper> as the sender
[02:56] <wgrant> The crash is not sending an email.
[02:56] <wgrant> It is producing an email address to be used in the changelog.
[02:57] <wgrant> brb
[02:59] <thumper> oh...
[02:59] <thumper> hmm...
[02:59] <thumper> best fix that then...
[03:02] <thumper> well, setting the author_name to "Eric the Viking (on behalf of Vikings team)"
[03:02] <thumper> still seems to make sense to me
[03:02] <thumper> for a build author
[03:02] <thumper> as it should be the team, but there is no email address set
[03:03] <thumper> although, I'm tempted to walk up the team.owner chain until we hit something with a preferred email
[03:03] <thumper> as we know we'll get someone eventually
[03:03] <thumper> as disabled accounts aren't part of teams
[03:04] <lifeless> thumper: there is a failure mode there ;)
[03:04] <thumper> lifeless: which is?
[03:04] <lifeless> thumper: two in fact - do we permit circular team ownership
[03:04] <thumper> a team with one disabled member?
[03:04] <lifeless> team A owns team B owns team A
[03:04] <thumper> I'm pretty sure we have DAGs for owners
[03:05] <lifeless> secondly, you may run up a team list until you find mark :)
[03:05] <thumper> lifeless: so your solution would be?
[03:05] <lifeless> what you're doing
[03:05] <lifeless> check that there is a DAG constraint is all
[03:06] <thumper> what I'm doing is to use the registrant of the recipe
[03:06] <lifeless> an alternative would be to say 'source-recipe-builder@launchpad.net'
[03:06] <thumper> which could be a disabled person
[03:06] <lifeless> I'm pretty sure that soyuz does something like that for the actual buildd
[03:07] <thumper> hmm... I reckon we use the recipe owner first, and if that isn't valid, use the @lp.net email
[03:10] <wgrant> Back.
[03:10] <wgrant> Using the owner or registrant seems wrong.
[03:10] <wgrant> I'd construct some special email address.
[03:10] <wgrant> Like Bugs does.
[03:11] <wgrant> Lying about the email address is less bad than lying about the name.
[03:12] <wgrant> s/owner/team owner/
[03:15] <lifeless> I think having a dedicated email to represent the things lp did makes sense
[03:15] <lifeless> its a little odd what ppa signing keys do, which is a similar situation
[03:15] <wgrant> How's what they do odd?
[03:15] <lifeless> they claim to be me
[03:15] <wgrant> They say 'Launchpad PPA for Somebody'
[03:15] <wgrant> No email address.
[03:15] <lifeless> hmmm
[03:16] <lifeless> I remember a bug from somewhere
[03:16] <wgrant> There is the great Bugs discussion.
[03:16] <wgrant> But that seems to have been resolved fairly reasonably.
[03:18] <lifeless> wow, I'm nearly at the end of my code queye
[03:18] <lifeless> bwah
[03:18] <lifeless> queue
[03:18] <wgrant> Oh?
[03:18] <lifeless> two branches to go
[03:19] <lifeless>  /participants timeouts
[03:19] <wgrant> Ah, right.
[03:19] <lifeless> and ffeature fflags
[03:21] <lifeless> I've got a tagger that alerts about deployable revisions
[03:22] <lifeless> we need to do something to make that releasable
[03:23] <wgrant> Is the new system going to be open source?
[03:24] <lifeless> yes
[03:24] <lifeless> the sticking point here is a filein the history that maps emails <-> lp ids
[03:24] <lifeless> which maybe indexing is a bit nuts
[03:25] <wgrant> Hmm.
[03:25] <lifeless> so we may need to restart the history with a blank file, and have the deployed version have that file.
[03:25] <lifeless> or something.
[03:26] <wgrant> It can't use the API?
[03:26] <lifeless> ENOIDEA
[03:26] <lifeless> anyhow, if we get it open
[03:27] <lifeless> with a blank file
[03:27] <lifeless> you can fix it to not need the file ;)
[03:27] <wgrant> Heh.
[03:28] <lifeless> [I don't know any of the history or design of it - I spent a day hacking and rearranging, so I know some of it, but that bit I didn't touch]
[03:29] <wgrant> If it's anything like Launchpad, we probably don't want to know the history....
[03:34] <lifeless> well
[03:50] <thumper> wgrant: do you thing the build author should be a special LP email all the time?
[03:51] <wgrant> thumper: No.
[03:51] <thumper> or just when the owner doesn't have an email set
[03:51] <wgrant> thumper: Yes.
[03:51]  * thumper nods
[03:51] <wgrant> Or possibly when it's private.
[03:51] <wgrant> Like Bugs now does.
[03:51] <wgrant> But that's not clear.
[03:55] <wgrant> brb
[03:56]  * thumper afk for kid collection
[03:59] <lifeless> jkakar: around ? I has a storm question on left outer joins in tuple-result sets
[04:01] <lifeless> the question is, are NULL matches represented as None, or Undef, or something else ?
[04:09] <lifeless> also
[04:09] <lifeless> is there a result set decorator already existing ?
[04:09] <lifeless> if not, any suggestions where I'd put it
[04:14] <wgrant> lifeless: You mean DecoratedResultSet?
[04:14] <lifeless> possibly
[04:14] <lifeless> where is that
[04:14] <wgrant> c.l.components.decoratedresultset
[04:14] <lifeless> wgrant: I want a thing that will handle a tuple-query, answer count() and the like, but call into some adaption code to support first(), any() and iteration
[04:15] <lifeless> plus copy config etce tc
[04:15] <wgrant> I believe that does what you want.
[04:15] <lifeless> ok cool
[04:16] <lifeless> brilliant
[04:16] <lifeless> thumper: ^
[04:17] <lifeless> thumper: your code query-tuning stuff that uses a list should use that
[04:23] <lifeless> wgrant: yup, exactly what I needed
[04:29] <thumper> lifeless: why?
[04:33] <lifeless> thumper: listification starts to perform badly when you exceed batch_size
[04:33] <thumper> I don't listify anything bigger than the batch
[04:34] <lifeless> I may be misremembering
[04:34] <lifeless> sec
[04:35] <lifeless> thumper: garh, lost the reference; what was the thing you tuned ?
[04:35] <thumper> lifeless: I've tuned lots of things :)
[04:35] <lifeless> heh
[04:35] <lifeless> ok
[04:35] <thumper> I'm a tuning machine
[04:35] <lifeless> so you had something that was doing potato queries
[04:35] <lifeless> and you pointed me at it
[04:36] <lifeless> it was, perhaps, an API
[04:36] <lifeless> ah no
[04:36] <lifeless> it was the thing you were created a cached decorator for
[04:37] <lifeless> ah, linked_bugs is one example
[04:38] <lifeless> it listifies, it would be better if it was built on DecoratedResultSet, though its also got the challenge that it does permission checks late
[04:39] <lifeless> _getRevisionsSinceReviewStart is another candidate for pushing more work to the db
[04:39] <lifeless> thumper: I'm just noting places we can do better - the code pages perform well, as you know :)
[04:40] <thumper> lifeless: not as well as I'd like
[04:40] <lifeless> thumper: well, tomorrow is performance day! if you want we can pair on analysing/tuning
[04:41] <thumper> I've got two^Wthree urgent things to get landed
[04:42] <lifeless> ah well
[04:42] <lifeless> how does one do subqueries in storm ?
[04:43] <lifeless> [want to get coc signing with a person query
[04:45] <lifeless> also
[04:45] <lifeless> can anyone confirm the Class.q.column syntax is unneeded now, with storm ?
[04:46] <thumper> lifeless: Select(...)
[04:47] <thumper> store.find(..., self.some_id.in.(Select(...)))
[04:47] <thumper> actually, it is 'is_in'
[04:47] <thumper> I think Class.q.column isn't needed
[04:47] <thumper> personally I never used it
[04:48] <lifeless> ok, so thats a subquery as a condition
[04:48] <lifeless> to use it as a result
[04:48] <lifeless> store.find((Person, Select()), conditions) ?
[04:55] <lifeless> select person.id, exists (select * from signedcodeofconduct where signedcodeofconduct.owner = person.id) as signed_coc from person <- what I want
[05:07] <thumper> Exists is a subquery I think
[05:07]  * thumper looks at some code
[05:08] <thumper> hmm...
[05:15] <lifeless> right, nearly got it
[05:15] <lifeless> just need to get the exists into the where clause
[05:15] <thumper> lifeless: are you wanting it in the where clause
[05:15] <thumper> or the select cols?
[05:18] <lifeless> >>> print [(p[0].id, p[1]) for p in store.using((Person,)).find((Person, Alias(Exists(Select(SignedCodeOfConduct, SignedCodeOfConduct.owner == Person.id, SignedCodeOfConduct)), name="signed")), And(Person.id >= 16, Person.id <= 17))]
[05:18] <lifeless> [(16, True), (17, False)]
[05:18] <lifeless> now to make it pretty
[05:19] <lifeless> thumper: select cols, I knew what I meant :P -sorry for the confusion I'm sure I engendered
[05:19] <thumper> lifeless: yeah, I just couldn't think of anything else...
[05:19] <poolie> thumper: this sounds interesting
[05:19] <poolie> is this for a graph or for a feature in lp?
[05:20] <lifeless> poolie: hi, see up above re ff
[05:20] <poolie> hi lifeless, i saw the features
[05:21] <poolie> i spent friday afternoon trying to work out why my lp dev instance was failing, and then getting a chroot up
[05:21] <poolie> i'm going to have another go at it now
[05:21] <lifeless> poolie: ok cool.
[05:21] <lifeless> poolie: if I can help, please shout.
[05:21]  * lifeless goes back to bashing up storm stuff
[05:21] <thumper> lifeless: you can go find((Person.id, ...), ...)
[05:22] <thumper> lifeless: to stop getting all of the people columns returned too (and chewing up non-sql time)
[05:22] <poolie> so lifeless, what are you doing with coc signing?
[05:25] <lifeless> thumper: we want all the people columns
[05:25] <lifeless> poolie: improving the performance of a registry API which is the top OOPS at the moment
[05:26] <poolie> cool :)
[05:29] <lifeless> thumper: thanks for the hint though - will keep it in the toolkit
[05:29] <lifeless> poolie: this is is the go-back-and-do-single-query pathology I was talking about
[05:44] <lifeless> down to 23 queries to generate the page for a team with 3 members ><
[06:01] <wgrant> lifeless: What's trying to find CoC sigs?
[06:02] <lifeless>  /participants
[06:02] <wgrant> Hm.
[06:02] <lifeless> because is_ubuntu_coc_signer is a property
[06:02] <lifeless> just don't say that out loud
[06:02] <wgrant> Ahh.
[06:02] <lifeless> wgrant: you can look at my 'registry' branch if you want to see the arc
[06:03] <wgrant> That name concerns me.
[06:03] <wgrant> thumper: What's the error in the #lp OOPS?
[06:04] <mtaylor> lifeless: how ludicrous of a feature request would it be to add "pastebin url" to the list of metadata info kept for a project? (you know, in there in the laundry list of all the other urls it keeps)
[06:04] <thumper> AssertionError: Wrong account! 2401541, 3228204
[06:04] <thumper> wgrant: ^^
[06:04] <lifeless> mtaylor: patches considered
[06:04] <wgrant> thumper: Ah, he merged his person, most probably.
[06:04] <mtaylor> lifeless: ok
[06:04] <wgrant> I believe that's scheduled for the next release or two.
[06:04] <lifeless> so, API question
[06:04] <thumper> mtaylor: add the description to the project
[06:04] <lifeless> there is a property
[06:04] <lifeless> 'archive'
[06:05] <lifeless> which is an exported Reference
[06:05] <lifeless> the API outputs a _link for it
[06:05] <lifeless> but it - and this is the nuts bit - creates the object to do that
[06:05] <wgrant> We should probably kill that (it's been deprecated for ages), but that's another story.
[06:05] <mtaylor> thumper: so - the pastebin I'm working with has a remote api... so I was thinking that a "bzr paste" command would be a nice way to spit a diff into a pastebin ... but it would be even cooler if bzr could look up _where_ to paste on the project via launchpadlib
[06:06] <lifeless> wgrant: so, if its in 1.0, we're screwed ?
[06:06] <wgrant> lifeless: Of course. It needs to know the name to get the canonical_url.
[06:06] <wgrant> Well.
[06:06] <wgrant> I think we should take API versioning with a grain of salt.
[06:06] <lifeless> by needs you mean 'its a constant pattern wtf'
[06:06] <lifeless> ?
[06:06] <mtaylor> thumper: pretty low on the list of things I want to get done -but I just wanted to see if was a waste of time to even put it on the todo list
[06:06] <mtaylor>  :)
[06:06] <wgrant> How else can the URL be created?
[06:07] <lifeless> wgrant: we have the person, the url is a constant suffix to the person canonical url
[06:07] <lifeless> wgrant: we don't need a DB lookup to do that
[06:07] <wgrant> lifeless: Erm, it shouldn't be constant.
[06:07] <wgrant> It should be /+archive/NAME
[06:08] <lifeless> wgrant: which for the first archive is _always_ ppa
[06:08] <wgrant> lifeless: No.
[06:08] <wgrant> lifeless: It's merely the default.
[06:08] <wgrant> it's been changeable for quite a while now.
[06:08] <lifeless> wgrant: unless its been changed, the form that you might be thinking of to let you change it actually ignores it on the first ppa
[06:09] <wgrant> It changed within the last year.
[06:09] <wgrant> (And broke a few things in the process.)
[06:54] <lifeless> wgrant: why does my branch name concern you?
[06:54] <wgrant> lifeless: Well, it suggests that you have one branch per app.
[06:54] <wgrant> Rather than one per piece of work.
[06:55] <lifeless> wgrant: I keep my queue very shallow
[06:55] <lifeless> I've done 2 separate landings from my registry branch so far
[06:55] <lifeless> both targeted and small
[06:57] <wgrant> True.
[07:02] <lifeless> of course, the thirds a doozy
[07:02] <lifeless> but its still one thing
[07:12] <lifeless> whats ValidPersonCache all about ?
[07:13] <lifeless> also, \o/ 14
[07:16] <thumper> lifeless: I have no idea
[07:16] <thumper> lifeless: probably histerical raisins
[07:16] <thumper> lifeless: worth asking the list though
[07:16] <thumper> someone might have a better answer
[07:18] <spiv> lifeless: "A materialized view listing the Person.ids of all valid people (but not teams)." :P
[07:19] <wgrant> It doesn't seem to be materialised.
[07:19] <wgrant> But that might just be the dev schema.
[07:19] <wgrant> I ran into it this morning.
[07:19] <wgrant> And WTFed a bit.
[07:23] <lifeless> spiv: thanks :P
[07:24] <lifeless> \o/ 11
[07:28] <lifeless> spm: what tz is your sprint in/
[07:30] <wgrant> lifeless: Is that including the auth[nz] and traversal queries?
[07:30] <wgrant> They number quite a few themselves.
[07:30] <lifeless> wgrant: yes
[07:31] <lifeless> wgrant: its now constant at 11 for 2 members or 3 members
[07:32] <wgrant> Excellent.
[07:32] <lifeless> I'd appreciate an eyeball on the branch, revno 11316
[07:32] <lifeless> I'm merging devel into it now to find out about conflicts
[07:34] <lifeless> and excellent, no conflicts.
[07:36] <wgrant> lifeless: re. r11309, conjunction is the default. You can just use a sequence.
[07:38] <lifeless> you mean the conditions variable ?
[07:38] <wgrant> Yes.
[07:42] <wgrant> Also, why do you get a whole KarmaTotalCache object, then switch on its .karma_total? Why not COALESCE the column directly in the query?
[07:42] <wgrant> Saves you an if and some object construction, which must be good.
[07:43] <wgrant> However.
[07:43] <wgrant> Adding all this caching at the model layer does not sit well with me.
[07:43] <lifeless> wgrant: so, coalesce - yes thats possible.
[07:44] <lifeless> OTOH the pain we're feeling is primarily driven by latency here
[07:44] <wgrant> It is.
[07:44] <lifeless> and vertical bouncing
[07:44] <wgrant> But let's make the DB server work for its dinner.
[07:45] <lifeless> I'd like to see how it works on staging before tuning further
[07:45] <lifeless> it may be so fast we don't care
[07:46] <lifeless> or it may be sql work thats needed, or python in which case absolutely focus on that
[07:46] <lifeless> so, caching.
[07:46] <lifeless> is in invalidation that concerns you
[07:46] <lifeless> s/in/it/?
[07:46] <wgrant> It is.
[07:47] <lifeless> are model objects persisted across transactions ?
[07:48] <lifeless> and does cachedproperty caches get cleared (the storm cache has some stuff done to it)
[07:48] <wgrant> No.
[07:48] <wgrant> I believe it gets cleared, yes.
[07:48] <lifeless> if the objects are dumped after a transaction thats suffiicent
[07:48] <lifeless> its only if they are reused that the cachedproperty stuff would have a *huge glaring hole*
[07:49] <wgrant> Oh yes.
[07:49] <wgrant> But I'm worried about what happens if this is used outside a webservice request.
[07:49] <lifeless> so, do you have alternative approach
[07:50] <wgrant> There's no alternative approach possible at the moment.
[07:50] <lifeless> so, this has the following properties:
[07:50] <lifeless>  - the caching will happen automatically
[07:50] <lifeless>  - precaching will require explicit action
[07:50] <lifeless>  - only the all_members_prepopulated takes that explicit action
[07:53] <wgrant> But calling that will taint people throughout the transaction.
[07:53] <lifeless> yes
[07:53] <lifeless> the first point is more worrying to me
[07:53] <lifeless> which is that Person.is_valid_person; change EmailAddress table; Person.is_valid_person won't change
[07:53] <lifeless> but!
[07:54] <lifeless> its no worse than having to call store.flush()
[07:54] <lifeless> which is also needed.
[07:54] <lifeless> Essentially we have a missing concept in our DSL
[07:54] <lifeless> which is 'the database is slow'
[07:54] <wgrant> Erm, store.flush() won't do this AFAIK.
[07:54] <lifeless> wgrant: its an analagous, overlapping problem.
[07:54] <lifeless> if you make a person valid, and don't call store.flush, is_valid will continue to return False
[07:55] <lifeless> without my patch
[07:55] <wgrant> Will it?
[07:55] <lifeless> yes
[07:55] <wgrant> Doesn't it make a query?
[07:55] <lifeless> because its a view
[07:55] <wgrant> Ah.
[07:55] <wgrant> But... no.
[07:55] <lifeless> it will query
[07:55] <wgrant> It will cause a flush.
[07:55] <lifeless> I didn't think every query caused a flush
[07:55] <wgrant> It will flush before it makes a query, so the view will update.
[07:55] <wgrant> I believe it does, or everything would be pretty broken.
[07:55] <lifeless> ok
[07:56] <lifeless> yet-another-reason to have an explicit step for
[07:56] <lifeless> please talk to the storage layer
[07:56] <lifeless> rather than things-that-look-cheap.
[07:57] <wgrant> It would be really nice if we could statically analyse the code and just say 'prejoin everything kthxbye'
[07:57] <lifeless> yes
[07:57] <lifeless> however
[07:57] <lifeless> in the absence of that, I have a few ideas.
[07:58] <lifeless> replacing cachedproperty with static derived attributes is one - with an explicit talk-to-the-db api in there
[07:58] <lifeless> its on my list to write up a RFC about this
[07:58] <lifeless> I wanted to get this done, as concrete experience within the current code, before shooting my mouth off
[08:05] <lifeless> https://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/32067
[08:05] <lifeless> wgrant: also prejoining has exactly the same side effects as this, except its even more magic.
[08:06] <wgrant> lifeless: prejoining only caches references, and is perfectly safe.
[08:06] <lifeless> mmm
[08:06] <lifeless> I'll have to revisit it then
[08:06] <lifeless> it may have changed
[08:06] <wgrant> It requests the extra objects from the DB, then throws them away.
[08:06] <wgrant> But that gets them into Storm's cache.
[08:07] <wgrant> So when it looks at EmailAddress.personID and sees '1', it checks its cache and finds (Person, 1), and doesn't bother to grab the object again.
[08:07] <lifeless> storm, because of sql's definition, really doesn't get much mileage from the cache
[08:07] <wgrant> AIUI, at least.
[08:08] <lifeless> for collections defined on a column, yes
[08:08] <lifeless> but not for any of this stuff I was doing
[08:08] <wgrant> Right.
[08:08] <wgrant> it's safe, but limited.
[08:25] <lifeless> wgrant: so, please do review and criticise
[08:30] <lifeless> poolie: how goes it ?
[08:32] <poolie> nb, still fiddling with vms
[08:33] <lifeless> I might have a look at the failures then
[08:33] <lifeless> my API tuning arc is complete
[08:34] <poolie> ok
[08:35] <poolie> it looks like the base template i put the 'features' macro into is not actually used by all the tests
[08:36] <lifeless> ah
[08:36] <poolie> do you know any better place to define it?
[08:36] <lifeless> all the tests you wrote, or some other tests ?
[08:37] <poolie> other tests
[08:37] <poolie> not sure if it's all or only some subclass
[08:38] <lifeless> ok
[08:38] <lifeless> and what causes the failure then ?
[08:38] <lifeless> oh, you use the features/thing macro in a different .pt file which those others tests do use ?
[08:38] <lifeless> wgrant: do you perhaps have any thoughts here?
[08:38] <lifeless> or thumper, though I know its late for you
[08:39] <poolie> i have it in i think base-template.pt
[08:40] <poolie> i wondered if that would work :)
[08:40] <poolie> apparently not
[08:40] <poolie> perhaps it can get stuffed into the tal context by the publication code?
[08:40] <poolie> actually i'd like to try to fix this with you rather than have you just fix it
[08:40] <lifeless> it may need to be
[08:40] <lifeless> great
[08:40] <lifeless> I will just dive around the code
[08:40] <lifeless> and provide entertaining chatter
[08:42] <wgrant> lifeless: Not sure, sorry.
[08:45] <lifeless> poolie: so you put it in what, base-layout-macros ?
[08:46] <lifeless> poolie: also, did you merge-back my tweak ?
[08:48] <poolie> i did
[08:48] <lifeless> ah, simple pull - cool
[08:49] <lifeless> poolie: I was thinking of doing a skin like the private-page skin
[08:49] <lifeless> for showing to lp developers only
[08:49] <lifeless> red-when-50-queries
[08:49] <lifeless> with a big SLOW as a watermark
[08:50] <poolie> yes, that's what i was thinknig too
[08:50] <lifeless> \o/
[08:50] <lifeless> anyhow, this bug
[08:50] <poolie> i was trying to communicate that by vocie the other day :)
[08:50] <lifeless> no wonder the idea felt familiar ;)
[08:52] <lifeless> poolie: so perhaps try lib/lp/app/templates/base-layout-macros.py
[08:52] <lifeless> bah
[08:52] <lifeless> .pt
[08:53] <poolie> mm i thought i did
[08:53] <lifeless> :!bzr st -r ancestor:../db-devel
[08:53] <lifeless>   lib/lp/app/templates/base-layout.pt
[08:53] <lifeless> is all thats mentioned as a .pt file
[08:54] <lifeless> but it seems to be the parent of base-layout
[08:54] <poolie> wow doctest knockon failures suck :)
[08:55] <lifeless> and some portlets use base-layout-macros stuff directly
[08:55] <lifeless> I am of course totally guessing
[08:55] <poolie> seems like a reasonable guess
[08:55] <poolie> multicore machines are nice
[08:55] <StevenK> poolie: I've been dealing with those for the past week. I am now getting revenge and replacing the doctest with unit tests ...
[08:56] <wgrant> StevenK: Who is the victim?
[08:56] <poolie> it's building one vm and running loggerhead-setup in another and there's no perceptible slowdown for interactive use
[08:56] <poolie> probably would be if i did something disk-heavy
[08:56] <StevenK> wgrant: lib/lp/soyuz/doc/initialise-from-parent.txt
[08:56] <wgrant> Ah, that lovely one.
[09:02] <lifeless> poolie: so, lets use science on this thing
[09:02] <lifeless> this isn't part of the stack I've dug into
[09:02] <poolie> i have apport crash warnings all over my screen :(
[09:02] <lifeless> the first failure is oauth-pages
[09:02] <poolie> science may have to wait...
[09:03] <lifeless> oops!
[09:03] <lifeless> ok, well I'll journal here
[09:03] <poolie> i am certainly noticing less suggestions from the dupefinder :)
[09:03] <poolie> i guess it'll be up to ~ubuntu whether that's better or not
[09:03] <lifeless> poolie: if you find its dropping *relevant* dupes, shout at me
[09:04] <wgrant> It's still in one-term-missing mode, right?
[09:04] <poolie> well, for instance "gnome-settings-daemon crashed with SIGSEGV" shows nothing
[09:04] <poolie> which is a bit surprising
[09:04] <lifeless> on /ubuntu/ ?
[09:04] <poolie> finding dupes based on that string is a bit problematic anyhow
[09:04] <poolie> https://bugs.edge.launchpad.net/ubuntu/+source/gnome-settings-daemon/+filebug/.....
[09:05] <poolie> ah, my mistake
[09:05] <poolie> bad example
[09:05] <lifeless> how so ?
[09:06] <poolie> because that does find dupes
[09:06] <poolie> "update-alternatives crashed with SIGSEGV in __strcmp_sse2" in dpkg is a better one
[09:06] <poolie> also you can see why it worried me :)
[09:07] <lifeless> well, lets see what the oracle of prod says
[09:07] <lifeless> it says ...
[09:07] <lifeless> timeout
[09:09] <poolie> win
[09:10] <poolie> so at least the instructions in https://dev.launchpad.net/Running/VirtualMachine#Alternatively seem to work
[09:10] <poolie> which is nice
[09:13] <poolie> so when you say b-l-macros is the parent of b-l.pt
[09:14] <poolie> where is that expressed?
[09:14] <poolie> i can see it refers to some specfici macros
[09:16] <lifeless> I did a grep for b-l-macros
[09:16] <lifeless> and base-layout uses it
[09:16] <lifeless> as well as portlets
[09:17] <lifeless> lib/canonical/launchpad/webapp/tales.py provides a hint
[09:18] <lifeless> base = ViewPageTemplateFile('../../../lp/app/templates/base-layout.pt')
[09:18] <lifeless> so it does look like thats the core
[09:19] <lifeless> anyhow, going back to science
[09:19] <lifeless> oauth test failure
[09:19] <lifeless> PathExpr standard:u'request/features'>
[09:19] <lifeless> is whats blowing up
[09:19] <poolie> mm
[09:19] <poolie> i defined 'features' as a macro evaluating to that
[09:20] <poolie> so it has the request object
[09:20] <poolie> it looks like the request object probably doesn't have a features attribute
[09:20] <lifeless> raise LocationError(subject, name)
[09:21] <lifeless>        - __traceback_info__: (<canonical.launchpad.webapp.servers.LaunchpadTestRequest instance URL=http://127.0.0.1>, 'features', [])
[09:21] <lifeless> yes
[09:21] <poolie> perhaps the place i hooked in isn't reached by in-process requests for testing
[09:22] <lifeless> poolie: perhaps!
[09:23] <lifeless> so, you're hooked into start_request and end_request
[09:23] <lifeless> there are two possibilities that leap out
[09:23] <lifeless> three possibilities that leap out
[09:24] <lifeless> firstly, the test environment might not be raising startrequest properly
[09:24] <lifeless> its a new things I had to add, so this is possible
[09:25] <lifeless> secondly, perhaps the request object is being shanghaied or something
[09:26] <lifeless> lastly, perhaps doing a singleton and not indirecting via request would be better; I-don't-know-its-just-a-thought
[09:27] <lifeless> oh, also perhaps its got a proxy on it that makes your setattr work but getattr die a silent death
[09:27] <poolie> we have a singleton
[09:27] <lifeless> poolie: does the requests/features macro work for you in interactive, 'make run' style testing ?
[09:28]  * poolie makes schema, and fails
[09:28] <poolie> yes, it does
[09:28] <poolie> so we could check the singleton
[09:28] <poolie> it's the same object
[09:28] <lifeless> ok, thats good
[09:28] <poolie> it seemed slightly cleaner to go via 'request'
[09:28] <poolie> mm the fact it works interactively is why i think that
[09:28] <poolie> it's something about the request object used here
[09:29] <poolie> or the way things are set up
[09:30] <lifeless> so, setting stuff on request seems idiomatic enough, from what I'm seeing
[09:30] <poolie> mm it looked like it
[09:32] <lifeless> ok, and there is only one beforeTraversal in our code base
[09:32] <lifeless> so we should definitely be raising the signal
[09:32] <lifeless> that leaves possibly only being registered for the signal some of the time
[09:32] <lifeless> ah! layers
[09:32] <lifeless> what test layer do ftests run in...
[09:35] <poolie> mm, how would layers interact with this?
[09:35] <lifeless> different layers active more of the zope machinery
[09:36] <lifeless> I get the impression that the DB level stuff is available before all the security proxies (and I'm gyessing events) are available
[09:36] <poolie> oh, you think the "fire an event when a request starts" stuff may depend on a particular layer being running?
[09:36] <poolie> possible
[09:38] <lifeless> stub: ping
[09:38] <poolie> yay, vm lp is running
[09:42] <lifeless> another possibility
[09:42] <lifeless> is that oauth-pages is calling stuff under the covers
[09:42] <lifeless> and in fact, it appears to be doing that
[09:42] <lifeless> its invoking the view directly
[09:43] <poolie> right, because it's not every pagetest failing
[09:43] <lifeless> and webservice-error.txt is doing the same thing
[09:44] <lifeless> using a custom RequestExpired
[09:44] <lifeless> so
[09:45] <lifeless> I get the feeling that doing what the other similar things do - accessing the singleton directly - is a lot less work
[09:45] <poolie> some of these failing tests, like test_base_layout, are not even in my branch yet
[09:45] <poolie> only in devel
[09:45] <poolie> ok so that sounds good
[09:45] <poolie> i guess seeing if that fails will at least help us triangulate
[09:45] <lifeless> define features as modules/lp.services.features.xxx/get_singleton
[09:46] <lifeless> or whatever the helper-that-looksup-in-a-thread-and-sets-if-not-set is called
[09:46] <poolie> hm
[09:46] <poolie> but
[09:46] <poolie> if it's not set on the request, i wonder if it will be set at all?
[09:47] <poolie> i don't think it makes sense to have "just some random flags" set
[09:47] <poolie> iow it has to be created for a request
[09:47] <poolie> i don't think we want a ground-state value
[09:47] <lifeless> so
[09:47] <lifeless> a few thoughts
[09:48] <lifeless> the NullObject pattern may help - you can say 'here is a features thing' that does nothing
[09:48] <lifeless> this is really a tests-do-not-match-reality problem its nothing to do with reality
[09:48] <poolie> mm
[09:48] <lifeless> in reality, we'll only look things up in the context of the publication,
[09:49] <poolie> i think if the tests explicitly opt in to "i'm not in a request but i want to pretend"
[09:49] <poolie> that would be ok
[09:49] <lifeless> but I'd like to get the feature first and cleanup the testing second, because of the timeliness fo rthis
[09:49] <poolie> but there's some risk of hiding bugs
[09:49] <lifeless> well they are doing that, but its implicit
[09:49] <poolie> so i see in some of these cases that we have a LaunchpadTestRequest instance
[09:49] <poolie> maybe that's not created in the usual way
[09:49] <lifeless> as in, we can only systematically tell that they haven't used the publication by the fact that they haven't used the publication
[09:50] <poolie> if we wanted to give it a null object, that might be a reasonable place to put it
[09:50] <lifeless> you could, if you are particularly concerned, use the hook you have to replace the null object with a real one, and have folk coming in the back door get the real object
[09:50] <lifeless> OTOH
[09:50] <lifeless> we want features in cron scripts and so on too
[09:50] <lifeless> so 'request' is really too tight a scope anyhow
[09:50] <poolie> sure
[09:51] <poolie> i think basically, someone should have an opinion on what flags ought to be active
[09:51] <poolie> that could be publication, or cron framework
[09:51] <poolie> or a test saying "just notihng"
[09:51] <lifeless> Ideally, I completely agree.
[09:51] <lifeless> We could change the doctest loader to state this as a setUp for all doctests.
[09:52] <lifeless> but that has the risk of fallout across the board impacting things
[09:52] <poolie> :/ i should add the tribunal "hide tests containing string $x" fetaure :)
[09:53] <lifeless> so, I might sleep on this.
[09:53] <poolie> i think i will add a null object created by the test request
[09:53] <poolie> to start with
[09:53] <lifeless> I think we are running into an accomodation of the test suite to the performance constraint
[09:53] <poolie> perhaps just actually hardcoded to null
[09:53] <poolie> or more precisely getitem(x)->None
[09:53] <lifeless> and its going to cause a lot of friction like this
[09:53] <poolie> oh, which constraint?
[09:54] <lifeless> 'things are slow' -> developers avoid calling the full stack :)
[09:54] <lifeless> its also a stack thing - folk want to (reasonably) exercise the views directly
[09:54] <lifeless> but the views implicitly depend on the full publication happening
[09:54] <lifeless> (I can name other things like this that are only by-chance not causing headaches)
[09:54] <lifeless> such as profiling
[09:55] <lifeless> or the request summary information which is reset by the publisher.
[09:55] <lifeless> or the storm cache
[09:55] <poolie> heh
[09:55] <lifeless> so the thing that is unique about your code, is that you are indirecting via the request
[09:55] <lifeless> rathe than being sloppy and tolerating these tests getting silly things back
[09:56] <lifeless> This is why I'm encouraging you to be sloppy (but not careless) here
[09:58] <lifeless> anyhow, 9pm; time for a break, I'll be needing to chat to mars in a few hours
[09:58] <lifeless> avoir
[09:59] <poolie> let's try this: http://pastebin.ubuntu.com/475347/
[09:59] <wgrant> [A22
[09:59] <wgrant> Gah.
[09:59] <poolie> sure, see you later
[09:59] <stub> lifeless: pong
[10:00] <lifeless> stub: nevermind, sorry. 'unpong' :P
[10:00] <lifeless> poolie: I think thats a good start
[10:00] <lifeless> poolie: also RequestExpired will need it
[10:00] <lifeless> (note that it is RuntimeError :P)
[10:01] <poolie> hm
[10:01] <lifeless> (look at the webservice-error.txt file)
[10:02] <lifeless> poolie: as for the tests that are not in your branch and failing, merge db-stable to your branch, and you should get them all
[10:02] <poolie> ?
[10:02] <lifeless> or db-devel to your branch
[10:02] <poolie> no that's the same Request tiype
[10:02] <poolie> sure
[10:02] <poolie> is this now targeted at devel or db-devel?
[10:02] <poolie> i guess db-devel now merged to devel?
[10:02] <poolie> since my db changes?
[10:02] <lifeless> not yet, late this week
[10:02] <lifeless> class RequestExpired(RuntimeError):
[10:02] <lifeless>     """Request has timed out."""
[10:02] <lifeless>     implements(IRequestExpired)
[10:03] <lifeless> thats in adapter.py
[10:03] <lifeless> anyhow, gl, chat with you tomorrow
[10:04] <lifeless> by 'thats a good start' I meant your pastebin, in case it wasn't clear.
[10:04] <poolie> good night
[10:04] <poolie> understood
[10:04] <poolie> i don't see what i need to do with requestexpired
[10:04] <poolie> that test is using the same request class
[10:04] <lifeless> oh
[10:04] <lifeless> interesting
[10:04] <lifeless> ok, cool.
[10:07] <poolie> is it just me with launchpad tests failing in
[10:07] <poolie> with a deprecationwarning in bzr builder?
[10:08] <poolie> maybe update-sourcecode will fix it
[10:08] <bigjools> wgrant: evening
[10:09] <wgrant> bigjools: Hi.
[10:10] <bigjools> wgrant: what part of soyuz uses guessPackageNames>
[10:10] <bigjools> ?
[10:10] <wgrant> I have to disappear in a few minutes.
[10:10] <wgrant> bigjools: Nothing.
[10:10] <bigjools> ok
[10:10]  * bigjools wonders why it's a soyuz bug then :)
[10:10] <wgrant> bigjools: Someone said so. I don't recall who.
[10:10] <wgrant> And it is a Soyuzy query!
[10:10] <bigjools> curtis probably
[10:10] <bigjools> sigh
[10:10] <wgrant> It's mostly used by the bug target picker.
[10:10] <bigjools> I've no idea how to QA it
[10:11] <bigjools> which most likely means it's not a soyuz bug
[10:11] <wgrant> It's one of those things that's in the void, and nobody has touched for years.
[10:11] <wgrant> I will QA it when I return later this evening.
[10:11] <bigjools> ok
[10:11] <bigjools> cheers
[10:16] <poolie> night all
[10:16] <wgrant> [A22
[10:16] <wgrant> GAH.
[10:16] <wgrant> Why has this started happening?
[10:37] <bigjools> thumper: around?
[11:12] <thumper> bigjools: no
[11:12] <thumper> bigjools: you have 3 minutes
[11:12] <thumper> whazzup?
[11:12] <bigjools> reviewed your branch
[11:12] <bigjools> gotta dash myself
[11:12] <jml> hello
[11:13] <thumper> bigjools: I had taken lifeless's rc as ok and landed it already, the original diff was updated to use config.launchpad.no_reply_email
[11:14] <thumper> bigjools: if you like we can file another bug and look at it for the start of next cycle
[12:06] <lifeless> leonardr: hi
[12:06] <leonardr> hi lifeless
[12:07] <deryck> Morning, all
[12:07] <lifeless> morning deryck
[12:07] <lifeless> have a good weekend ?
[12:07] <wgrant> bigjools: Hi. Am I likely to be granted an RC for bug #615286?
[12:07] <_mup_> Bug #615286: DEPWAIT not recognized from build log <Launchpad Auto Build System:New> <https://launchpad.net/bugs/615286>
[12:07] <wgrant> Simple lp-buildd regex change.
[12:07] <deryck> lifeless, indeed.  Busy but relaxing.  Thanks!  And you?
[12:07] <bigjools> wgrant: no, it's not RC
[12:07] <bigjools> they can manually retry
[12:07] <lifeless> deryck: pretty good, house stuff + hacked on qa-tagger sunday
[12:08] <deryck> cool
[12:08] <lifeless> wgrant: cherrypick it IMO
[12:08] <lifeless> we're getting rid of the whole release thing this cycle, if everything comes out well
[12:08] <bigjools> lifeless: it's a buildd change, it's not affected by the release at all
[12:09] <bigjools> IS releases buildd changes
[12:09] <lifeless> bigjools: yah
[12:09] <bigjools> so it's not a CP :)
[12:09] <lifeless> well, in the sense of 'deploy without waiting a month'
[12:09] <lifeless> we dont have a good phrase for that atm
[12:10] <deryck> heh
[12:10] <spiv> "deploy now"
[12:13] <deryck> my "heh" was because I read lifeless as "deploy without *wasting* a month."  Clearly I need more coffee.
[12:13] <lifeless> heh
[12:13] <lifeless> nice misread there
[12:19] <lifeless> grah
[12:19] <lifeless> when we are moving to lucid ec2 test images ?
[12:21] <jml> after we move to lucid everything else?
[12:21]  * jml isn't sure.
[12:21] <lifeless> neither am I
[12:21] <lifeless> AIUI bb looks at lucid now for blessing
[12:21] <lifeless> so it would make sense to move the ec2 images to lucid
[12:21] <jml> agreed
[12:22] <lifeless> have I mentioned I fear and loath doctest failures
[12:22] <jml> lifeless, once or twice
[12:23] <jml> (also, "loathe" is a verb, "loath" is an adjective)
[12:23] <lifeless> +e
[12:23] <StevenK> lifeless: O hai.
[12:24] <StevenK> lifeless: You mentioned a transform that Storm could do for me, WRT _copy_lucille_config. Could I have an example?
[12:24] <lifeless> uhm
[12:24] <lifeless> show me your diff again :P
[12:25] <StevenK> lifeless: https://code.edge.launchpad.net/~stevenk/launchpad/move-ifp-from-idistroseries/+merge/31520
[12:25] <StevenK> lifeless: I didn't want to change the current code, but I have more work in the same area, and I could use what you were suggesting
[12:26] <lifeless> right
[12:26] <lifeless> you create a result set
[12:28] <lifeless> store.find(DistroSeries, DistroSeries.id == self.distroseries.id)
[12:28] <lifeless> then you call set on it
[12:30] <lifeless> pdr = Alias(DistroSeries,
[12:30] <lifeless> name='pdr')
[12:31] <lifeless> resultset.set(DistroSeries.lucilleconfig=Select(pdr.lucilleconfig, pdr.id==self.parentid))
[12:31] <lifeless> I think thats it - check the sql it emits of course
[12:32] <StevenK> lifeless: Right, but to do a copy like that in general, .set() is my friend?
[12:32] <lifeless> to do an UPDATE, .set() on a result set is your friend
[12:32] <lifeless> that 'copy' is actually a two-stage thing, it duplicates, then it changes
[12:33] <lifeless> or something like that - the one method is really 'trash the config using that one over there'
[12:34]  * bigjools actually thinks that SQL syntax is better for once
[12:34]  * lifeless doesn't really care
[12:34] <lifeless> with the registry stuff, doing it in raw sql would be worse I think
[12:35] <bigjools> probably, but why?
[12:35] <bigjools> Storm syntax is generally pretty unintelligible for me
[12:35] <lifeless> for the registry stuff I was doing, because it is combining 6-7 different clauses, constraints and tables into one aggregate query
[12:36] <lifeless> bigjools: I'm saying 'consider on a case by case basis' :)
[12:36] <bigjools> I completely agree :)
[12:36] <lifeless> storm is so tied to sql that if we wanted to do things differently we're no better-or-worse off
[12:46] <gmb> lifeless, Can we consider https://bugs.edge.launchpad.net/malone/+bug/607776 qa-ok?
[12:46] <_mup_> Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>
[12:46] <deryck> lifeless, I think we can mark bug 607776 as qa-ok.  Would you agree?
[12:46] <_mup_> Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>
[12:46] <deryck> heh
[12:46] <lifeless> jinx
[12:46] <lifeless> yes, absolutely.
[12:46] <lifeless> mea culpa
[12:46] <deryck> no worries.
[12:47] <gmb> DOne
[12:47] <deryck> gmb, I'll leave it to you to mark, since I stepped on you once. ;)
[12:47] <lifeless> I've been watching like a hawk for issue
[12:47] <lifeless> *issues*
[12:47] <gmb> deryck, Heh. I wonder what happens if two people try to make the same change. Probably the world ends or something.
[12:47] <deryck> me, too.  Seems solid to me.
[12:47] <deryck> the AJAX world ends
[12:47] <lifeless> its still possible we'll find an important common use case it doesn't handle well, but for now, its happy
[12:48] <deryck> gmb, I'm still working on all the ones for my dupe work.  Staging doesn't have the change yet, so I'm poking around bugs that have moved duplicates to make sure things look good.
[12:49] <gmb> Right
[12:50] <lifeless> _mup_: poke mars
[13:02] <deryck> Ok, bugs team is down to 4 bugs needing qa now.  One requires a chat with bdmurray.  I think gmb is aware and working on qa for bug 594211.
[13:02] <_mup_> Bug #594211: Add a RecipientReason class to bugnotificationrecipients.py <qa-needstesting> <story-better-bug-notification> <story-refactor-bugnotifications> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/594211>
[13:02] <deryck> The other two are me and need my changes on staging to qa.
[13:02] <deryck> bigjools, ^^
[13:03] <bigjools> deryck: rock n roll, thanks
[13:03] <gmb> deryck, bigjools That's qa-ok; marking it as such now.
[13:03] <deryck> np, sorry it took us so long this time.
[13:03] <deryck> gmb, excellent, thanks!
[13:04]  * gmb lunches
[13:10]  * bigjools -> lunch
[13:17] <lifeless> gmb: are you sure bug 606914 is fixed ?
[13:17] <_mup_> Bug #606914: Bug.initial_message fetches all of the bug's messages <qa-ok> <timeout> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/606914>
[13:18] <lifeless> gmb: unless there is a test checking the sql count, I suspect we'll find its still a problem after the rollout
[13:37] <lifeless> gnight all
[13:54] <jtv-afk> oh, yeah, nick.
[14:06] <gmb> lifeless, I'll check
[14:51] <gmb> lifeless, So, that particular bug - and the problem described in it (that initial_message was stupid) is fixed. It doesn't fix the timeout problem, but there are other bugs covering that (which may or may not need consolidating into one sensible report, or at least clarifying so that the distinctions between them become more obvious). I'm happy for this particular one to be qa-ok, though.
[16:16] <salgado> sinzui, do you know where's the code that allows traversal from a distro to a distroseries? I thought it'd be in DistributionNavigation but it's not
[16:18] <salgado> oh, it must come from GetitemNavigation,
[16:18] <salgado> indeed, that's it
[17:59] <cperrin88> Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?
[17:59] <cperrin88> the whole root wadl file
[18:10] <cperrin88> The sounds like no
[18:11] <mars> cperrin88, leonardr is working on that very problem
[18:11] <mars> cperrin88, IIRC older versions of wadllib grabbed the whole thing.  Newer versions cache it.
[18:11] <leonardr> mars, cperrin88, i was off irc during the crucial moment. can you recap?
[18:12] <mars> leonardr, <cperrin88> Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?
 the whole root wadl file
[18:13] <leonardr> cperrin88: yes, it will get the wadl file on first startup and about once a month afterwards (whenever launchpad is upgraded)
[18:14] <cperrin88> okay .... well ... I'm doind a Java port of the lib ...
[18:14] <leonardr> cperrin88: ok, you can implement the same behavior. the wadl file is served with cache-control headers and an etag
[18:14] <cperrin88> Yeas
[18:15] <leonardr> so you can know how long to cache it for, and once the cache expires, you can avoid getting the whole document unless launchpad has been upgraded
[18:15] <cperrin88> that's what I wanted to do
[18:15] <cperrin88> I just wanted to make sure that you really grab the whole thing
[18:15] <cperrin88> it's rather big
[18:15] <leonardr> cperrin88: if you send Accept-Encoding: gzip it's only about 100k
[18:16] <cperrin88> hey
[18:16] <cperrin88> your right
[18:16] <cperrin88> that's cool
[18:17] <cperrin88> without it's more than 10 times the size
[18:17] <leonardr> xml compresses very well
[18:17] <cperrin88> seems like .... that helps a lot
[18:17] <leonardr> so if you implement all these optimizations your client will be making 1 http request once a week and once a month that http request will result in a 100k download
[18:17] <cperrin88> I was concerned about mobile users
[18:18] <cperrin88> yep
[18:18] <leonardr> note that there is also a json version of the service root, which has the same rules, but it's much smaller--only about 1k uncompressed
[18:18] <cperrin88> The lib is a result of my development of a Launchpad android client
[18:19] <cperrin88> Yeah
[18:19] <cperrin88> I know
[18:20] <leonardr> ok
[18:20] <cperrin88> but I think I will handle it more like launchpadlib does now
[18:20] <cperrin88> thank you for your help
[18:20] <leonardr> sure
[18:31] <rockstar> benji, ping
[18:31] <benji> rockstar: hey
[18:32] <rockstar> benji, do you know much about View permissions?
[18:32] <benji> a bit, what's up?
[18:34] <cperrin88> leonardr: I think I found a small error in the WADL file. Two doc parts have invalied entitiys ... At least my parser says that
[18:35] <leonardr> cperrin: send me the doc output. we validate the wadl before deploying, but it's possible we have access to some external doc you don't
[18:35] <cperrin88> it's   `     (Apostrophe)
[18:36] <cperrin88> lemme see
[18:38] <cperrin88> I take it back
[18:38] <cperrin88> I guess it was my fault
[18:39] <leonardr> ok
[18:41] <benji___> rockstar: sorry, had some technical difficulties; will you repeat your question?
[18:41] <rockstar> benji, permissions on View don't care what the request type is, correct?  It's the same for GET and POST?
[18:42] <benji> rockstar: correct
[18:42] <benji> permissions don't assume "correct" use of HTTP verbs
[20:40] <lifeless> moin
[20:40] <lifeless> mars: hi
[20:42] <mars> Hi lifeless
[20:43] <mwhudson> morning
[20:44] <lifeless> mars: Ursinha: I just got up, so haven't trawled email yet; I'm going to look for review feedback now
[20:44] <mars> lifeless, Ursinha and I have both looked over your changes.  They look good - thanks for doing that.  She is just finishing her review now.
[20:44] <Ursinha> yes sir
[20:44] <lifeless> cool, I hope its helpful
[20:44] <mars> lifeless, very.  Good hooks for the parts we still need to write, too
[20:45] <Ursinha> lifeless, do you intend to write more code there?
[20:45] <lifeless> Ursinha: maybe :) - AFAICT theres enough there now that we could switch to the new workflow and iterate on polish ?
[20:46] <Ursinha> lifeless, and yes, that was pretty much what I thought to be a way to reuse qa-tagger structure, very helpful
[20:48] <mars> lifeless, Ursinha may have to double-check the qa-sort algorithm.  But there should be enough to start integration with whatever downstream component there will be.
[20:48] <Ursinha> lifeless, hm, not sure, I'm not that familiar with the whole changes to put that in prod now
[20:49] <mars> lifeless, assuming Ursinha thinks the script output is still sane for our current daily production use :)
[20:49] <lifeless> so
[20:50] <lifeless> if more work is needed to make this usable for the new merge workflow where we qa edge and request deployments from the losas at up-to revision granularity
[20:50] <lifeless> (usable, not 'perfect' :P)
[20:50] <mars> of course
[20:50] <lifeless> then - yes, I'll write more code, while you guys sleep tonight :)
[20:51] <lifeless> If, OTOH, its sufficient to get us started, then I've some other things to help along to get that workflow transition to happen
[20:52] <mars> Ursinha, would it be possible to run this tagger code beside the existing stable tagger code?  So we don't interrupt our production line?
[20:53] <mars> lifeless, Ursinha is in the process of reviewing the code to make sure that the already in-place QA process and tagger are not interrupted.
[20:53] <Ursinha> mars, well, yes, we can do that and tag bugs twice or change one of them a bit so we wouldn't have the two fighting over bugs
[20:54] <mars> Ursinha, perhaps we could *not* run the tagger part?  By adding a CLI switch or something?
[20:54] <Ursinha> lifeless, btw, waking up having code to review is nice
[20:54] <Ursinha> mars, that can be done
[20:54] <lifeless> mars: we don't need to run them in parallel so much as be sure that we have *enough* so that when we switch we can use it
[20:54] <lifeless> Ursinha: my pleasure
[20:54] <lifeless> mars: although in parallel with a smooth transition is nice, obviously
[20:54] <Ursinha> we still need to consider the rollback commits
[20:55] <lifeless> ok
[20:55] <Ursinha> not much of a big deal but needs a bit of code there
[20:55] <lifeless> they are tagged qa-rolledback or something ?
[20:55] <Ursinha> qa-rollback
[20:55] <mars> lifeless, well, I believe Ursinha's goal is not to get enough working, it is 'can be guaranteed to still work like it did on Friday' :)
[20:55] <Ursinha> pretty much :)
[20:55] <lifeless> does it mean 'needs to be rolled back', or 'it has been rolled back'
[20:55] <Ursinha> lifeless, it was rolled back
[20:55] <lifeless> kk
[20:56] <lifeless> so, if 'qa-ok' in tags or 'qa-rollback' in tags :)
[20:56] <lifeless> should be a 12-character fix :)
[20:56] <mars> + test
[20:56] <lifeless> of course
[20:56] <Ursinha> lifeless, https://dev.launchpad.net/QAProcessContinuousRollouts
[20:57] <Ursinha> lifeless, it also needs to add the tag, when appropriated
[20:57] <lifeless> mars: so the overall -goal- here is to change the way we work; keeping things unaltered is insufficient ;)
[20:57] <lifeless> Ursinha: ok, based on the new lp-land switches ?
[20:57] <Ursinha> the commit should have a [rollback] clause, same as [testfix]
[20:58] <Ursinha> so we know that if a new commit is a rollback, we know the bug and know the last revision that's blocked waiting for this rollback
[20:58] <lifeless> so the tagger would tag the old revisions bug as rollback
[20:58] <Ursinha> a rollback is a reversion of a qa-bad or qa-needstesting previous one
[20:58] <Ursinha> yes sir
[20:58] <Ursinha> and let it go
[20:59] <lifeless> righto
[20:59] <lifeless> so 'rollback' is 'unqaable' for the reporter, but still taggable for the tagger
[20:59] <Ursinha> lifeless, yes
[21:00] <lifeless> is that the total of the remaining must-have features ?
[21:00] <Ursinha> lifeless, why are you pinging me? :)
[21:00] <lifeless> bah, fat fingers
[21:00] <Ursinha> lifeless, unqaable: untestable, orphaned, testfix, buildbot, and now rollback
[21:00] <Ursinha> therefore blessed
[21:00] <lifeless> I was meaning to do 'TIME'
[21:00] <mars> lifeless, I understand.  I think Ursinha should make the call on when the new qa-tagger code can replace what is already running.  Parallel deployments is just an idea.  We don't have to take it.
[21:01] <lifeless> mars: I'm happy either way
[21:01] <Ursinha> mars, I like the idea of running them in parallel
[21:01] <lifeless> ok, phone time for me
[21:01] <lifeless> Ursinha: Its your 5pm now right?
[21:01] <Ursinha> lifeless, yes
[21:01] <lifeless> I'll look at adding rollback now
[21:02] <Ursinha> lifeless, cool
[21:02] <Ursinha> thanks
[21:02] <lifeless> if there is more than that needed, let me know please before you retire for the night.
[21:02] <Ursinha> sure
[21:03] <Ursinha> lifeless, that wiki page has details, maybe that helps a bit
[21:03] <mars> lifeless, btw, are you on the Kanban board, and can't edit?  Or do you not have an account at all?
[21:03] <lifeless> couldn't edit your board
[21:03] <mars> ok
[21:06] <mars> no admin privs - I can't change it.  Oh well.
[21:11] <lifeless> flacoste: https://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/32067
[21:12] <sinzui> lifeless, I started this. I got distracted by my sprint
[21:13] <lifeless> sinzui: sprinting is good
[21:13] <sinzui> lifeless, I forsee a few state issues in this branch
[21:13] <lifeless> sinzui: how do you feel if it lands without your review? or could you give it enough cycles to say 'thats ok we can tweak later' or 'lifeless, you need to look at x,y,z'
[21:14] <lifeless> sinzui: ok, can you roughly describe them ?
[21:14] <sinzui> I can deactivate a person and the page will still show active...I need to refresh my browser I think
[21:16] <sinzui> lifeless, Person.preferredemail knows when it is mutated to manually purge its _*_cached property
[21:18] <sinzui> lifeless, But i am speculating on these scenarios  for coc and archive
[21:20] <sinzui> lifeless, I am fine with this landing. We can fix issues as we discover them
[21:20] <lifeless> sinzui: thanks!
[21:21] <lifeless> flacoste: line 680
[21:27] <flacoste> lifeless: lp/registry/model/product.py:470
[21:37] <EdwinGrubbs> jelmer: ping
[21:38] <jelmer> EdwinGrubbs, pong
[21:40] <EdwinGrubbs> jelmer: it looks like you can set the upstream project differently on each distroseries? Is that a good idea.
[21:42] <jelmer> EdwinGrubbs: there are some (very rare) situations in which that is necessary
[21:42] <jelmer> EdwinGrubbs: e.g. I think that the chromium package used to be for the chromium bsu game but now is for the chromium browser
[21:42] <EdwinGrubbs> ok
[21:50] <mars> sinzui, does pocketlint not catch tab and indentation errors?  (check person.py line 3603-5)
[21:51] <mars> tabnanny.py yelled at me after finding that
[21:51] <mars> sinzui, lib/lp/registry/browser/person.py:3603
[21:52] <sinzui> mars, I think it yells twice in some cases
[21:53] <mars> I'm kind of weirded out by the interpreter not stopping on that.  I would figure the change of indentation type in the middle of a method would cause something bad to happen.
[21:56] <benji> mars: the interpreter equates tabs with eight spaces
[21:57] <sinzui> Someone landed tabs in registry code 6 weeks ago. jtv reported it and I added tab checking
[21:57] <sinzui> mars, yes, pocketlint hates the file
[21:57] <sinzui> mars, should it call bzr blame for those lines too?
[21:57] <sinzui> mars, actually, I see bac's emacs is set to correct tabs to spaces and his diff in review accidentally started removing the tabs :)
[21:57] <mars> sinzui, pocketlint on dev systems?  It ignores it on mine.
[21:58] <mars> sinzui, should we ditch tabnanny then?
[21:58] <mars> It would make test-on-merge faster
[21:59] <sinzui> hmm, maybe the version is wrong in developer-deps
[21:59] <sinzui> mars, this is what I see http://pastebin.ubuntu.com/475630/
[22:00] <mars> sinzui, exit code zero?
[22:01] <mars> sinzui, ok, PEBCAK here on the output - but the exit code is a bit odd.
[22:01] <mars> sinzui, so I'll remove tabnanny from test_on_merge.py then.  That is what lint is for, and that file has no need of the additional complexity.
[22:02] <sinzui> As I said above, developer-deps is behind
[22:02] <sinzui> I stopped publishing pocket-lint because hardy had build issues
[22:03] <sinzui> You can get the latest version from my ppa, or wait till next week (after the release) where I can push a version that will play in the test runner
[22:03] <mars> done - tabnanny in test_on_merge is no more
[22:03] <sinzui> lint also screams at pdbs
[22:04] <sinzui> maybe we want a test that checks the code for pdbs
[22:04] <mars> don't know - might be a good idea, but it would take a while to run.  If most people run lint, then it may not be necessary.
[22:05] <mars> lint catches far more things
[22:05] <mars> lint is already a 'best effort'
[22:13] <benji> leonardr: I started to write the MP for ~benji/lazr.restfulclient/total_size_link and realized that you might be the better one to write it because you did almost all the work :)
[22:13] <benji> oops, wrong chan
[22:20] <lifeless> sinzui: thanks for approving that; will land as soon as pqm opens :)
[22:27] <bdmurray> sinzui: I'm having an issue with return Link() in lib/lp/registry/browser/structuralsubscription.py and StructuralSubscriptionMenuMixin.  deryck thought I might check with you about it.
[22:28] <lifeless> leonardr: ping
[22:43] <mwhudson> rockstar: are you confusing TAL and ZCML in your mail to launchpad-dev?
[22:44] <rockstar> mwhudson, yes, because it is all ugly...
[23:05] <wgrant> mwhudson: ShipIt sadly uses TeamParticipation and Person.
[23:05] <wgrant> So it's not that easy.
[23:05] <mwhudson> wgrant: ah
[23:05] <mwhudson> oh well
[23:05] <lifeless> web services
[23:05] <wgrant> Right.
[23:05] <wgrant> The former can be done from SSO.
[23:05] <wgrant> And the latter is just for karma.
[23:05] <wgrant> So it's not *that* hard.
[23:06]  * lifeless hates code freeze
[23:06] <lifeless> do we have an unfrozen branch for folk to work on this week ?
[23:06]  * ajmitch hates seeing edge timeouts :)
[23:06] <wgrant> lifeless: No :(
[23:06] <lifeless> or do we all just pile up and conflict on friday?
[23:07] <wgrant> Precisely.
[23:07] <wgrant> I believe you have a fix for this, though :)
[23:07] <lifeless> ajmitch: whats the oops id ?
[23:07] <ajmitch> OOPS-1682EC4078
[23:07] <ajmitch> it was for a bug link that had comments=all
[23:07] <ajmitch> so no surprise that it'd take a little while
[23:07] <lifeless> well
[23:07] <lifeless> lets see
[23:08] <lifeless> Statement Count: 303
[23:08] <ajmitch> performance tuesday for you?
[23:08] <lifeless> thats going to be a large driver
[23:08] <lifeless> ajmitch: yup
[23:08] <lifeless> yeah, lots of repeated things
[23:09] <lifeless> 285.	6825	24ms	launchpad-main-slave	SELECT LibraryFileContent.datecreated, LibraryFileContent.filesize, LibraryFileContent.id, LibraryFileContent.md5, LibraryFileContent.sha1 FROM LibraryFileContent WHERE LibraryFileContent.id = %s LIMIT 1
[23:09] <lifeless> 286.	11541	46ms	launchpad-main-slave	SELECT LibraryFileAlias.content, LibraryFileAlias.date_created, LibraryFileAlias.expires, LibraryFileAlias.filename, LibraryFileAlias.hits, LibraryFileAlias.id, LibraryFileAlias.last_accessed, LibraryFileAlias.mimetype, LibraryFileAlias.restricted FROM LibraryFileAlias WHERE LibraryFileAlias.id = %s LIMIT 1
[23:09] <lifeless> note that there is 3 seconds of time between those two db calls - thats going to be python
[23:09] <lifeless> not 'lots of bytecode', rather 'some other thread screwed us'
[23:12] <lifeless> 120 subscription lookups
[23:12] <lifeless> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/131094/+index
[23:12] <_mup_> Bug #131094: Heavy Disk I/O harms desktop responsiveness <cft-2.6.27> <Linux:Invalid> <linux (Ubuntu):Confirmed for ubuntu-kernel-team> <linux-source-2.6.22 (Ubuntu):Won't Fix by ben-collins> <https://launchpad.net/bugs/131094>
[23:16] <poolie> spm: hey, awake yet?
[23:17] <wgrant> poolie: IS sprint...
[23:17] <james_w> hi poolie
[23:17] <james_w> hi wgrant
[23:17] <wgrant> Morning james_w.
[23:17] <poolie> hi james_w; thanks wgrant
[23:18] <lifeless> poolie: all the losas are in UTC+2 this week
[23:25] <lifeless> Ursinha: / matsubara-afk: how can I reset the bug infestation for an oops ?
[23:27] <Ursinha> lifeless, let me find the tomboy note that explains that
[23:28] <leonardr> lifeless, hi
[23:28] <lifeless> \o/
[23:28] <lifeless> leonardr: hi
[23:28] <lifeless> leonardr: uhm, uhm, I've forgotten my thread
[23:28] <Ursinha> lifeless, wait, do you want to know just to know or you just need something to be changed?
[23:28] <lifeless> Ursinha: I keep finding wrong ones when you guys are asleep
[23:28] <lifeless> so I want to know how to do it myself
[23:28] <Ursinha> lifeless, ah sure
[23:28] <leonardr> lifeless, just type it if you remember, i'll come back later
[23:28] <Ursinha> lifeless, I'll send one email
[23:29] <lifeless> Ursinha: thanks!
[23:29] <lifeless> leonardr: no worries!
[23:29] <Ursinha> no problem :)
[23:40] <james_w> wgrant: want to tell me where https://code.edge.launchpad.net/~james-w/launchpad/move-soyuz-test-publisher/+merge/32157 falls on a scale of 1 to heinous?
[23:41] <wgrant> That scrollbar scares me.
[23:41] <wgrant> And those are some serious docstrings.
[23:42] <james_w> yeah, I think that's where a lot of the vertical height comes from
[23:42] <mars> bzr question: I am using sandboxes with 'bzr switch', I have pushed my branch, bzr says 'Created new branch', but Launchpad insists "This branch has not been pushed to yet.".  What gives?
[23:43] <mars> On top of that, 'bzr push --overwrite' says "No new revisions to push"
[23:43] <wgrant> mars: LP may be being a little slow. Replication lag, maybe?
[23:44] <lifeless> mars: did the url it reported make sense to you?
[23:44] <mars> lifeless, which URL from which command?
[23:44] <lifeless> when you push, bzr tells you where it is pushing
[23:44] <lifeless> I think, or is that commit.
[23:44] <lifeless> anyhow, where di dyou push to ?
[23:45] <lifeless> precisely
[23:45] <mars> lifeless, ah, got it: incompatible repo formats
[23:45] <mars> lifeless, and bzr doesn't re-raise the error when I try again
[23:45] <wgrant> james_w: Hm, so test_publishing has a special STP derivative which uses sampledata?
[23:46] <mars> lifeless, how did you push your qa-tagger code without it barfing?
[23:46] <lifeless> mars: I didn't put it in a shared repo
[23:47] <lifeless> its odd that bzr would only whine once
[23:47] <mars> lifeless, ok - using 'upgrade this branch' on the UI is harmless? >:)
[23:47] <lifeless> mars: you need to coordinate with everyone
[23:47] <mars> yarg
[23:47] <lifeless> mars: e.g. with the losas because we roll this stuff out, unless qa have direct control over where its rolled out to and can do the upgrade on sodium-or-whatever
[23:48] <mars> lifeless, this is under qa's control
[23:48] <lifeless> so, coordinate with them :)
[23:48] <mars> lifeless, so I hit 'upgrade this branch', then what happens?  They try to pull new revisions and it blows up or something?
[23:49] <lifeless> well
[23:49] <lifeless> tell me what 'this' is bound to
[23:49] <lifeless> because I don't have the full picture
[23:49] <mars> https://code.edge.launchpad.net/~launchpad-qa/qa-tagger/devel
[23:49] <lifeless> ok
[23:49] <lifeless> so if you upgrade that
[23:49] <lifeless> then yes, everyone merging or pulling from it will blow up
[23:50] <lifeless> they then need to upgrade and can then pull/merge appropriately
[23:50] <mars> lifeless, will you still be able to land your unmerged work if I do this?
[23:50] <wgrant> james_w: So, looks heinous, but it's probably the right thing anyway.
[23:51] <lifeless> mars: yes, I just need to upgrade my branch too - as I said, you need to coordinate with everyone :)
[23:51] <wgrant> Until we can port everything away from the old one, which could be a while...
[23:51] <lifeless> mars: e.g. send a mail
[23:51] <mars> lifeless, ok, will do
[23:51]  * mars presses the Big Yellow Button
[23:51] <thumper> yellow?
[23:52] <wgrant> The drunken exclamation mark.
[23:52] <mars> thumper, to be honest, I think the +upgrade page needs some work :/
[23:52] <thumper> mars: probably
[23:53] <mars> I could just copy and paste this IRC conversation :)
[23:53] <mars> "What does this button do?"  "Well, this conversation describes it well: ..."
[23:55] <lifeless> hmm, I could have sworn there was a bug open for oops' miscategorising storm timeouts
[23:57] <mars> lifeless, so besides all the branch format shenanigans, I have some code that is built on top of yours.  You may want to merge it: lp:~mars/qa-tagger/lp-service-switch
[23:58] <mars> (not pushed yet, need the upgrade to finish first)
[23:58] <lifeless> what does it do /
[23:58] <lifeless> ?
[23:58] <lifeless> does anyone remember the oops-doesn't-include-storm-timeouts-as-sql-time bug ?
[23:58] <mars> lifeless, adds a --lp-service command-line switch.  That switch tells the script which lp instance to run against.  We want to use this for smoke-testing
[23:59] <mars> lifeless, failsafes to 'staging'.  Any developer should be able to check out the code and safely do a smoketest this way.