/srv/irclogs.ubuntu.com/2010/08/09/#launchpad-dev.txt

thumperwgrant: you just have to poke don't you?01:05
thumperwgrant: messing with whiteboards01:05
lifelesswgrant: what awas that oops you mentioned earlier ?01:08
wgrantthumper: That was my "Ahem, oops.", yes. Forgot I wasn't on staging.01:14
wgrant(also, I really shouldn't be able to write to that whiteboard at all)01:14
wgrantlifeless: OOPS-1681S1580?01:15
wgrantHm. No (LO|G)SAs during release week. Great.01:16
thumper:)01:16
thumperthere should be someone interruptable01:16
thumperbut you are right01:16
thumperno one on hand right now01:16
thumpermwhudson: can you have member functions be function decorators?01:27
* thumper thinks01:28
mwhudsonthumper: what does that mean?01:28
thumperI'm trying to clean up some test code01:28
thumperbut with a function decorator, I don't need the self.addCleanup I guess01:28
lifelessthumper: self.addCleanup is generally a good thing01:28
lifelessthumper: can you describe a bit more, or pastebin an example ?01:29
thumpersure01:29
thumperliefhttp://pastebin.ubuntu.com/475182/01:29
thumperlifeless: http://pastebin.ubuntu.com/475182/01:29
lifelessstab stab stab opening01:30
lifelessopenind01:30
thumperlifeless: we repeatedly have the lines 4-9 in a number of tests01:30
thumperI wanted to have a simple test function decorator to do the config push and pop01:30
lifelessyou could01:31
lifelessbut I'd put it in a fixture01:31
lifelessusing installfixture01:31
thumperexample somewhere?01:31
lifelessclass BuilderConfig:01:32
lifeless   def setUp(self):01:32
lifeless    bzr_builder_config = """01:32
lifeless...01:32
lifelessdef tearDown(self):01:32
thumperdoes the fixture need a tearDown?01:32
lifeless    config.pop("bzr_builder_config")01:32
lifelessself.installFixture(BuilderConfig())01:32
thumpernow a question would be, is it worth making a fixture, or just have a method?01:33
lifelessfixtures can be used by different test classes01:34
lifelessso yes, its worth it01:34
lifelesswgrant:01:37
lifelessModule lp.code.model.diff, line 73, in text01:37
lifeless    self.diff_text.open()01:37
lifeless  Module canonical.launchpad.database.librarian, line 110, in open01:37
lifeless    self._datafile = self.client.getFileByAlias(self.id, timeout)01:37
lifeless  Module canonical.librarian.client, line 349, in getFileByAlias01:38
lifeless    raise LookupError, aliasID01:38
lifelessLookupError: 5270464901:38
wgrantlifeless: Yeah, that makes sense. Thanks.01:42
lifeless\o/ cleanup part 2 done.02:18
lifelesspoolie: around ? Would like to land your flags branch - have you looked at the failures from friday, or should I do that ?02:18
thumperwgrant: what do we do with stuck builders?02:18
thumperwhat sort of privledges do you need?02:18
lifelesstake em out back and shoot em02:18
mwhudsonthumper: isn't there a pushConfig on lp.testing.TestCase already?02:19
mwhudsoni know i've written it somewhere...02:19
thumpermwhudson: you know, I think you might be right02:19
thumperlifeless: do you have a rifle?02:23
thumperWTF?02:32
thumperadmins don't have launchpad.Special for IPerson?02:32
wgrantthumper: It needs a lamont.02:38
wgrantthumper: Which builder?02:38
wgrantthumper: That's the point of .Special. Admins don't have it.02:38
wgrantThat is its entire purpose.02:38
wgrantWhy it's used, I don't know.02:38
thumperwgrant: see #launchpad for stuck builders02:38
wgrantOnly for SSH keys, AIUI.02:38
thumperwgrant: and account deactivation02:38
wgrantCraaap, not terranova.02:38
wgrantIt's not meant to die.02:38
wgrantNothing that can be done without a GSA.02:39
wgrantAnd lamont, normally.02:39
thumperwgrant: are the build jobs actually instantiated by the build master before dispatching to a builder?02:50
thumperwgrant: I'm wanting to oops on a recipe for a deactivated person before it gets to the actual builder02:50
wgrantthumper: Yes -- the builder can't talk to the DB.02:51
wgrantThe master has to do everything.02:51
thumperah ha02:51
thumperso the recipes are taking out the build master?02:51
wgrantYes.02:51
wgrant-> the entire farm.02:51
thumperand it can't handle exceptions?02:51
wgrantAnything outside lib/canonical/buildd runs on the master.02:51
thumperthat's a bit poor02:51
wgrantIt is somewhat.02:52
wgrantBut it's not that bad.02:52
thumperORLY?02:52
thumperwgrant: so where is the code in the buildmaster that is dying?02:52
thumperwhen the job is barfing on no email?02:53
wgrantthumper: In recipebuilder.py, where I showed you before. Or do you mean what calls taht?02:53
thumperI've fixed the recipe owned by team with no email part02:53
wgrantHow have you fixed that?02:53
thumpernow looking at the possibility of a person deactivating themselves after a recipe job has been created02:53
thumperI use the recipe registrant instead02:54
thumperand say email sent on behalf of the team02:54
thumperin that edge case02:54
thumperI want the job to blowup earlier though in the situation where we can't get a preferred email for the team or registrant02:55
thumperwhich means an oops02:55
wgrantErm.02:55
wgrantSent on behalf of the team?02:55
thumperwhich means that the code has to handle it02:55
thumperas the sender02:55
wgrantThe crash is not sending an email.02:56
wgrantIt is producing an email address to be used in the changelog.02:56
wgrantbrb02:57
thumperoh...02:59
thumperhmm...02:59
thumperbest fix that then...02:59
thumperwell, setting the author_name to "Eric the Viking (on behalf of Vikings team)"03:02
thumperstill seems to make sense to me03:02
thumperfor a build author03:02
thumperas it should be the team, but there is no email address set03:02
thumperalthough, I'm tempted to walk up the team.owner chain until we hit something with a preferred email03:03
thumperas we know we'll get someone eventually03:03
thumperas disabled accounts aren't part of teams03:03
lifelessthumper: there is a failure mode there ;)03:04
thumperlifeless: which is?03:04
lifelessthumper: two in fact - do we permit circular team ownership03:04
thumpera team with one disabled member?03:04
lifelessteam A owns team B owns team A03:04
thumperI'm pretty sure we have DAGs for owners03:04
lifelesssecondly, you may run up a team list until you find mark :)03:05
thumperlifeless: so your solution would be?03:05
lifelesswhat you're doing03:05
lifelesscheck that there is a DAG constraint is all03:05
thumperwhat I'm doing is to use the registrant of the recipe03:06
lifelessan alternative would be to say 'source-recipe-builder@launchpad.net'03:06
thumperwhich could be a disabled person03:06
lifelessI'm pretty sure that soyuz does something like that for the actual buildd03:06
thumperhmm... I reckon we use the recipe owner first, and if that isn't valid, use the @lp.net email03:07
wgrantBack.03:10
wgrantUsing the owner or registrant seems wrong.03:10
wgrantI'd construct some special email address.03:10
wgrantLike Bugs does.03:10
wgrantLying about the email address is less bad than lying about the name.03:11
wgrants/owner/team owner/03:12
lifelessI think having a dedicated email to represent the things lp did makes sense03:15
lifelessits a little odd what ppa signing keys do, which is a similar situation03:15
wgrantHow's what they do odd?03:15
lifelessthey claim to be me03:15
wgrantThey say 'Launchpad PPA for Somebody'03:15
wgrantNo email address.03:15
lifelesshmmm03:15
lifelessI remember a bug from somewhere03:16
wgrantThere is the great Bugs discussion.03:16
wgrantBut that seems to have been resolved fairly reasonably.03:16
lifelesswow, I'm nearly at the end of my code queye03:18
lifelessbwah03:18
lifelessqueue03:18
wgrantOh?03:18
lifelesstwo branches to go03:18
lifeless /participants timeouts03:19
wgrantAh, right.03:19
lifelessand ffeature fflags03:19
lifelessI've got a tagger that alerts about deployable revisions03:21
lifelesswe need to do something to make that releasable03:22
wgrantIs the new system going to be open source?03:23
lifelessyes03:24
lifelessthe sticking point here is a filein the history that maps emails <-> lp ids03:24
lifelesswhich maybe indexing is a bit nuts03:24
wgrantHmm.03:25
lifelessso we may need to restart the history with a blank file, and have the deployed version have that file.03:25
lifelessor something.03:25
wgrantIt can't use the API?03:26
lifelessENOIDEA03:26
lifelessanyhow, if we get it open03:26
lifelesswith a blank file03:27
lifelessyou can fix it to not need the file ;)03:27
wgrantHeh.03:27
lifeless[I don't know any of the history or design of it - I spent a day hacking and rearranging, so I know some of it, but that bit I didn't touch]03:28
wgrantIf it's anything like Launchpad, we probably don't want to know the history....03:29
lifelesswell03:34
thumperwgrant: do you thing the build author should be a special LP email all the time?03:50
wgrantthumper: No.03:51
thumperor just when the owner doesn't have an email set03:51
wgrantthumper: Yes.03:51
* thumper nods03:51
wgrantOr possibly when it's private.03:51
wgrantLike Bugs now does.03:51
wgrantBut that's not clear.03:51
wgrantbrb03:55
* thumper afk for kid collection03:56
lifelessjkakar: around ? I has a storm question on left outer joins in tuple-result sets03:59
lifelessthe question is, are NULL matches represented as None, or Undef, or something else ?04:01
lifelessalso04:09
lifelessis there a result set decorator already existing ?04:09
lifelessif not, any suggestions where I'd put it04:09
wgrantlifeless: You mean DecoratedResultSet?04:14
lifelesspossibly04:14
lifelesswhere is that04:14
wgrantc.l.components.decoratedresultset04:14
lifelesswgrant: I want a thing that will handle a tuple-query, answer count() and the like, but call into some adaption code to support first(), any() and iteration04:14
lifelessplus copy config etce tc04:15
wgrantI believe that does what you want.04:15
lifelessok cool04:15
lifelessbrilliant04:16
lifelessthumper: ^04:16
lifelessthumper: your code query-tuning stuff that uses a list should use that04:17
lifelesswgrant: yup, exactly what I needed04:23
thumperlifeless: why?04:29
lifelessthumper: listification starts to perform badly when you exceed batch_size04:33
thumperI don't listify anything bigger than the batch04:33
lifelessI may be misremembering04:34
lifelesssec04:34
lifelessthumper: garh, lost the reference; what was the thing you tuned ?04:35
thumperlifeless: I've tuned lots of things :)04:35
lifelessheh04:35
lifelessok04:35
thumperI'm a tuning machine04:35
lifelessso you had something that was doing potato queries04:35
lifelessand you pointed me at it04:35
lifelessit was, perhaps, an API04:36
lifelessah no04:36
lifelessit was the thing you were created a cached decorator for04:36
lifelessah, linked_bugs is one example04:37
lifelessit listifies, it would be better if it was built on DecoratedResultSet, though its also got the challenge that it does permission checks late04:38
lifeless_getRevisionsSinceReviewStart is another candidate for pushing more work to the db04:39
lifelessthumper: I'm just noting places we can do better - the code pages perform well, as you know :)04:39
thumperlifeless: not as well as I'd like04:40
lifelessthumper: well, tomorrow is performance day! if you want we can pair on analysing/tuning04:40
thumperI've got two^Wthree urgent things to get landed04:41
lifelessah well04:42
lifelesshow does one do subqueries in storm ?04:42
lifeless[want to get coc signing with a person query04:43
lifelessalso04:45
lifelesscan anyone confirm the Class.q.column syntax is unneeded now, with storm ?04:45
thumperlifeless: Select(...)04:46
thumperstore.find(..., self.some_id.in.(Select(...)))04:47
thumperactually, it is 'is_in'04:47
thumperI think Class.q.column isn't needed04:47
thumperpersonally I never used it04:47
lifelessok, so thats a subquery as a condition04:48
lifelessto use it as a result04:48
lifelessstore.find((Person, Select()), conditions) ?04:48
lifelessselect person.id, exists (select * from signedcodeofconduct where signedcodeofconduct.owner = person.id) as signed_coc from person <- what I want04:55
thumperExists is a subquery I think05:07
* thumper looks at some code05:07
thumperhmm...05:08
lifelessright, nearly got it05:15
lifelessjust need to get the exists into the where clause05:15
thumperlifeless: are you wanting it in the where clause05:15
thumperor the select cols?05:15
lifeless>>> print [(p[0].id, p[1]) for p in store.using((Person,)).find((Person, Alias(Exists(Select(SignedCodeOfConduct, SignedCodeOfConduct.owner == Person.id, SignedCodeOfConduct)), name="signed")), And(Person.id >= 16, Person.id <= 17))]05:18
lifeless[(16, True), (17, False)]05:18
lifelessnow to make it pretty05:18
lifelessthumper: select cols, I knew what I meant :P -sorry for the confusion I'm sure I engendered05:19
thumperlifeless: yeah, I just couldn't think of anything else...05:19
pooliethumper: this sounds interesting05:19
poolieis this for a graph or for a feature in lp?05:19
lifelesspoolie: hi, see up above re ff05:20
pooliehi lifeless, i saw the features05:20
pooliei spent friday afternoon trying to work out why my lp dev instance was failing, and then getting a chroot up05:21
pooliei'm going to have another go at it now05:21
lifelesspoolie: ok cool.05:21
lifelesspoolie: if I can help, please shout.05:21
* lifeless goes back to bashing up storm stuff05:21
thumperlifeless: you can go find((Person.id, ...), ...)05:21
thumperlifeless: to stop getting all of the people columns returned too (and chewing up non-sql time)05:22
poolieso lifeless, what are you doing with coc signing?05:22
lifelessthumper: we want all the people columns05:25
lifelesspoolie: improving the performance of a registry API which is the top OOPS at the moment05:25
pooliecool :)05:26
lifelessthumper: thanks for the hint though - will keep it in the toolkit05:29
lifelesspoolie: this is is the go-back-and-do-single-query pathology I was talking about05:29
lifelessdown to 23 queries to generate the page for a team with 3 members ><05:44
wgrantlifeless: What's trying to find CoC sigs?06:01
lifeless /participants06:02
wgrantHm.06:02
lifelessbecause is_ubuntu_coc_signer is a property06:02
lifelessjust don't say that out loud06:02
wgrantAhh.06:02
lifelesswgrant: you can look at my 'registry' branch if you want to see the arc06:02
wgrantThat name concerns me.06:03
wgrantthumper: What's the error in the #lp OOPS?06:03
mtaylorlifeless: how ludicrous of a feature request would it be to add "pastebin url" to the list of metadata info kept for a project? (you know, in there in the laundry list of all the other urls it keeps)06:04
thumperAssertionError: Wrong account! 2401541, 322820406:04
thumperwgrant: ^^06:04
lifelessmtaylor: patches considered06:04
wgrantthumper: Ah, he merged his person, most probably.06:04
mtaylorlifeless: ok06:04
wgrantI believe that's scheduled for the next release or two.06:04
lifelessso, API question06:04
thumpermtaylor: add the description to the project06:04
lifelessthere is a property06:04
lifeless'archive'06:04
lifelesswhich is an exported Reference06:05
lifelessthe API outputs a _link for it06:05
lifelessbut it - and this is the nuts bit - creates the object to do that06:05
wgrantWe should probably kill that (it's been deprecated for ages), but that's another story.06:05
mtaylorthumper: so - the pastebin I'm working with has a remote api... so I was thinking that a "bzr paste" command would be a nice way to spit a diff into a pastebin ... but it would be even cooler if bzr could look up _where_ to paste on the project via launchpadlib06:05
lifelesswgrant: so, if its in 1.0, we're screwed ?06:06
wgrantlifeless: Of course. It needs to know the name to get the canonical_url.06:06
wgrantWell.06:06
wgrantI think we should take API versioning with a grain of salt.06:06
lifelessby needs you mean 'its a constant pattern wtf'06:06
lifeless?06:06
mtaylorthumper: pretty low on the list of things I want to get done -but I just wanted to see if was a waste of time to even put it on the todo list06:06
mtaylor :)06:06
wgrantHow else can the URL be created?06:06
lifelesswgrant: we have the person, the url is a constant suffix to the person canonical url06:07
lifelesswgrant: we don't need a DB lookup to do that06:07
wgrantlifeless: Erm, it shouldn't be constant.06:07
wgrantIt should be /+archive/NAME06:07
lifelesswgrant: which for the first archive is _always_ ppa06:08
wgrantlifeless: No.06:08
wgrantlifeless: It's merely the default.06:08
wgrantit's been changeable for quite a while now.06:08
lifelesswgrant: unless its been changed, the form that you might be thinking of to let you change it actually ignores it on the first ppa06:08
wgrantIt changed within the last year.06:09
wgrant(And broke a few things in the process.)06:09
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
lifelesswgrant: why does my branch name concern you?06:54
wgrantlifeless: Well, it suggests that you have one branch per app.06:54
wgrantRather than one per piece of work.06:54
lifelesswgrant: I keep my queue very shallow06:55
lifelessI've done 2 separate landings from my registry branch so far06:55
lifelessboth targeted and small06:55
wgrantTrue.06:57
lifelessof course, the thirds a doozy07:02
lifelessbut its still one thing07:02
lifelesswhats ValidPersonCache all about ?07:12
lifelessalso, \o/ 1407:13
thumperlifeless: I have no idea07:16
thumperlifeless: probably histerical raisins07:16
thumperlifeless: worth asking the list though07:16
thumpersomeone might have a better answer07:16
spivlifeless: "A materialized view listing the Person.ids of all valid people (but not teams)." :P07:18
wgrantIt doesn't seem to be materialised.07:19
wgrantBut that might just be the dev schema.07:19
wgrantI ran into it this morning.07:19
wgrantAnd WTFed a bit.07:19
lifelessspiv: thanks :P07:23
lifeless\o/ 1107:24
lifelessspm: what tz is your sprint in/07:28
wgrantlifeless: Is that including the auth[nz] and traversal queries?07:30
wgrantThey number quite a few themselves.07:30
lifelesswgrant: yes07:30
lifelesswgrant: its now constant at 11 for 2 members or 3 members07:31
wgrantExcellent.07:32
lifelessI'd appreciate an eyeball on the branch, revno 1131607:32
lifelessI'm merging devel into it now to find out about conflicts07:32
lifelessand excellent, no conflicts.07:34
wgrantlifeless: re. r11309, conjunction is the default. You can just use a sequence.07:36
lifelessyou mean the conditions variable ?07:38
wgrantYes.07:38
wgrantAlso, why do you get a whole KarmaTotalCache object, then switch on its .karma_total? Why not COALESCE the column directly in the query?07:42
wgrantSaves you an if and some object construction, which must be good.07:42
wgrantHowever.07:43
wgrantAdding all this caching at the model layer does not sit well with me.07:43
lifelesswgrant: so, coalesce - yes thats possible.07:43
lifelessOTOH the pain we're feeling is primarily driven by latency here07:44
wgrantIt is.07:44
lifelessand vertical bouncing07:44
wgrantBut let's make the DB server work for its dinner.07:44
lifelessI'd like to see how it works on staging before tuning further07:45
lifelessit may be so fast we don't care07:45
lifelessor it may be sql work thats needed, or python in which case absolutely focus on that07:46
lifelessso, caching.07:46
lifelessis in invalidation that concerns you07:46
lifelesss/in/it/?07:46
wgrantIt is.07:46
lifelessare model objects persisted across transactions ?07:47
lifelessand does cachedproperty caches get cleared (the storm cache has some stuff done to it)07:48
wgrantNo.07:48
wgrantI believe it gets cleared, yes.07:48
lifelessif the objects are dumped after a transaction thats suffiicent07:48
lifelessits only if they are reused that the cachedproperty stuff would have a *huge glaring hole*07:48
wgrantOh yes.07:49
wgrantBut I'm worried about what happens if this is used outside a webservice request.07:49
lifelessso, do you have alternative approach07:49
wgrantThere's no alternative approach possible at the moment.07:50
lifelessso, this has the following properties:07:50
lifeless - the caching will happen automatically07:50
lifeless - precaching will require explicit action07:50
lifeless - only the all_members_prepopulated takes that explicit action07:50
wgrantBut calling that will taint people throughout the transaction.07:53
lifelessyes07:53
lifelessthe first point is more worrying to me07:53
lifelesswhich is that Person.is_valid_person; change EmailAddress table; Person.is_valid_person won't change07:53
lifelessbut!07:53
lifelessits no worse than having to call store.flush()07:54
lifelesswhich is also needed.07:54
lifelessEssentially we have a missing concept in our DSL07:54
lifelesswhich is 'the database is slow'07:54
wgrantErm, store.flush() won't do this AFAIK.07:54
lifelesswgrant: its an analagous, overlapping problem.07:54
lifelessif you make a person valid, and don't call store.flush, is_valid will continue to return False07:54
lifelesswithout my patch07:55
wgrantWill it?07:55
lifelessyes07:55
wgrantDoesn't it make a query?07:55
lifelessbecause its a view07:55
wgrantAh.07:55
wgrantBut... no.07:55
lifelessit will query07:55
wgrantIt will cause a flush.07:55
lifelessI didn't think every query caused a flush07:55
wgrantIt will flush before it makes a query, so the view will update.07:55
wgrantI believe it does, or everything would be pretty broken.07:55
lifelessok07:55
lifelessyet-another-reason to have an explicit step for07:56
lifelessplease talk to the storage layer07:56
lifelessrather than things-that-look-cheap.07:56
wgrantIt would be really nice if we could statically analyse the code and just say 'prejoin everything kthxbye'07:57
lifelessyes07:57
lifelesshowever07:57
lifelessin the absence of that, I have a few ideas.07:57
lifelessreplacing cachedproperty with static derived attributes is one - with an explicit talk-to-the-db api in there07:58
lifelessits on my list to write up a RFC about this07:58
lifelessI wanted to get this done, as concrete experience within the current code, before shooting my mouth off07:58
lifelesshttps://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/3206708:05
lifelesswgrant: also prejoining has exactly the same side effects as this, except its even more magic.08:05
wgrantlifeless: prejoining only caches references, and is perfectly safe.08:06
lifelessmmm08:06
lifelessI'll have to revisit it then08:06
lifelessit may have changed08:06
wgrantIt requests the extra objects from the DB, then throws them away.08:06
wgrantBut that gets them into Storm's cache.08:06
wgrantSo when it looks at EmailAddress.personID and sees '1', it checks its cache and finds (Person, 1), and doesn't bother to grab the object again.08:07
lifelessstorm, because of sql's definition, really doesn't get much mileage from the cache08:07
wgrantAIUI, at least.08:07
lifelessfor collections defined on a column, yes08:08
lifelessbut not for any of this stuff I was doing08:08
wgrantRight.08:08
wgrantit's safe, but limited.08:08
lifelesswgrant: so, please do review and criticise08:25
lifelesspoolie: how goes it ?08:30
poolienb, still fiddling with vms08:32
lifelessI might have a look at the failures then08:33
lifelessmy API tuning arc is complete08:33
poolieok08:34
poolieit looks like the base template i put the 'features' macro into is not actually used by all the tests08:35
lifelessah08:36
pooliedo you know any better place to define it?08:36
lifelessall the tests you wrote, or some other tests ?08:36
poolieother tests08:37
poolienot sure if it's all or only some subclass08:37
lifelessok08:38
lifelessand what causes the failure then ?08:38
lifelessoh, you use the features/thing macro in a different .pt file which those others tests do use ?08:38
lifelesswgrant: do you perhaps have any thoughts here?08:38
lifelessor thumper, though I know its late for you08:38
pooliei have it in i think base-template.pt08:39
pooliei wondered if that would work :)08:40
poolieapparently not08:40
poolieperhaps it can get stuffed into the tal context by the publication code?08:40
poolieactually i'd like to try to fix this with you rather than have you just fix it08:40
lifelessit may need to be08:40
lifelessgreat08:40
lifelessI will just dive around the code08:40
lifelessand provide entertaining chatter08:40
wgrantlifeless: Not sure, sorry.08:42
lifelesspoolie: so you put it in what, base-layout-macros ?08:45
lifelesspoolie: also, did you merge-back my tweak ?08:46
pooliei did08:48
lifelessah, simple pull - cool08:48
lifelesspoolie: I was thinking of doing a skin like the private-page skin08:49
lifelessfor showing to lp developers only08:49
lifelessred-when-50-queries08:49
lifelesswith a big SLOW as a watermark08:49
poolieyes, that's what i was thinknig too08:50
lifeless\o/08:50
lifelessanyhow, this bug08:50
pooliei was trying to communicate that by vocie the other day :)08:50
lifelessno wonder the idea felt familiar ;)08:50
lifelesspoolie: so perhaps try lib/lp/app/templates/base-layout-macros.py08:52
lifelessbah08:52
lifeless.pt08:52
pooliemm i thought i did08:53
lifeless:!bzr st -r ancestor:../db-devel08:53
lifeless  lib/lp/app/templates/base-layout.pt08:53
lifelessis all thats mentioned as a .pt file08:53
lifelessbut it seems to be the parent of base-layout08:54
pooliewow doctest knockon failures suck :)08:54
lifelessand some portlets use base-layout-macros stuff directly08:55
lifelessI am of course totally guessing08:55
poolieseems like a reasonable guess08:55
pooliemulticore machines are nice08:55
StevenKpoolie: I've been dealing with those for the past week. I am now getting revenge and replacing the doctest with unit tests ...08:55
wgrantStevenK: Who is the victim?08:56
poolieit's building one vm and running loggerhead-setup in another and there's no perceptible slowdown for interactive use08:56
poolieprobably would be if i did something disk-heavy08:56
StevenKwgrant: lib/lp/soyuz/doc/initialise-from-parent.txt08:56
wgrantAh, that lovely one.08:56
lifelesspoolie: so, lets use science on this thing09:02
lifelessthis isn't part of the stack I've dug into09:02
pooliei have apport crash warnings all over my screen :(09:02
lifelessthe first failure is oauth-pages09:02
pooliescience may have to wait...09:02
lifelessoops!09:03
lifelessok, well I'll journal here09:03
pooliei am certainly noticing less suggestions from the dupefinder :)09:03
pooliei guess it'll be up to ~ubuntu whether that's better or not09:03
lifelesspoolie: if you find its dropping *relevant* dupes, shout at me09:03
wgrantIt's still in one-term-missing mode, right?09:04
pooliewell, for instance "gnome-settings-daemon crashed with SIGSEGV" shows nothing09:04
pooliewhich is a bit surprising09:04
lifelesson /ubuntu/ ?09:04
pooliefinding dupes based on that string is a bit problematic anyhow09:04
pooliehttps://bugs.edge.launchpad.net/ubuntu/+source/gnome-settings-daemon/+filebug/.....09:04
poolieah, my mistake09:05
pooliebad example09:05
lifelesshow so ?09:05
pooliebecause that does find dupes09:06
poolie"update-alternatives crashed with SIGSEGV in __strcmp_sse2" in dpkg is a better one09:06
pooliealso you can see why it worried me :)09:06
lifelesswell, lets see what the oracle of prod says09:07
lifelessit says ...09:07
lifelesstimeout09:07
pooliewin09:09
poolieso at least the instructions in https://dev.launchpad.net/Running/VirtualMachine#Alternatively seem to work09:10
pooliewhich is nice09:10
poolieso when you say b-l-macros is the parent of b-l.pt09:13
pooliewhere is that expressed?09:14
pooliei can see it refers to some specfici macros09:14
lifelessI did a grep for b-l-macros09:16
lifelessand base-layout uses it09:16
lifelessas well as portlets09:16
lifelesslib/canonical/launchpad/webapp/tales.py provides a hint09:17
lifelessbase = ViewPageTemplateFile('../../../lp/app/templates/base-layout.pt')09:18
lifelessso it does look like thats the core09:18
lifelessanyhow, going back to science09:19
lifelessoauth test failure09:19
lifelessPathExpr standard:u'request/features'>09:19
lifelessis whats blowing up09:19
pooliemm09:19
pooliei defined 'features' as a macro evaluating to that09:19
poolieso it has the request object09:20
poolieit looks like the request object probably doesn't have a features attribute09:20
lifelessraise LocationError(subject, name)09:20
lifeless       - __traceback_info__: (<canonical.launchpad.webapp.servers.LaunchpadTestRequest instance URL=http://127.0.0.1>, 'features', [])09:21
lifelessyes09:21
poolieperhaps the place i hooked in isn't reached by in-process requests for testing09:21
lifelesspoolie: perhaps!09:22
lifelessso, you're hooked into start_request and end_request09:23
lifelessthere are two possibilities that leap out09:23
lifelessthree possibilities that leap out09:23
lifelessfirstly, the test environment might not be raising startrequest properly09:24
lifelessits a new things I had to add, so this is possible09:24
lifelesssecondly, perhaps the request object is being shanghaied or something09:25
lifelesslastly, perhaps doing a singleton and not indirecting via request would be better; I-don't-know-its-just-a-thought09:26
lifelessoh, also perhaps its got a proxy on it that makes your setattr work but getattr die a silent death09:27
pooliewe have a singleton09:27
lifelesspoolie: does the requests/features macro work for you in interactive, 'make run' style testing ?09:27
* poolie makes schema, and fails09:28
poolieyes, it does09:28
poolieso we could check the singleton09:28
poolieit's the same object09:28
lifelessok, thats good09:28
poolieit seemed slightly cleaner to go via 'request'09:28
pooliemm the fact it works interactively is why i think that09:28
poolieit's something about the request object used here09:28
poolieor the way things are set up09:29
lifelessso, setting stuff on request seems idiomatic enough, from what I'm seeing09:30
pooliemm it looked like it09:30
lifelessok, and there is only one beforeTraversal in our code base09:32
lifelessso we should definitely be raising the signal09:32
lifelessthat leaves possibly only being registered for the signal some of the time09:32
lifelessah! layers09:32
lifelesswhat test layer do ftests run in...09:32
pooliemm, how would layers interact with this?09:35
lifelessdifferent layers active more of the zope machinery09:35
lifelessI get the impression that the DB level stuff is available before all the security proxies (and I'm gyessing events) are available09:36
poolieoh, you think the "fire an event when a request starts" stuff may depend on a particular layer being running?09:36
pooliepossible09:36
lifelessstub: ping09:38
poolieyay, vm lp is running09:38
lifelessanother possibility09:42
lifelessis that oauth-pages is calling stuff under the covers09:42
lifelessand in fact, it appears to be doing that09:42
lifelessits invoking the view directly09:42
poolieright, because it's not every pagetest failing09:43
lifelessand webservice-error.txt is doing the same thing09:43
lifelessusing a custom RequestExpired09:44
lifelessso09:44
lifelessI get the feeling that doing what the other similar things do - accessing the singleton directly - is a lot less work09:45
pooliesome of these failing tests, like test_base_layout, are not even in my branch yet09:45
poolieonly in devel09:45
poolieok so that sounds good09:45
pooliei guess seeing if that fails will at least help us triangulate09:45
lifelessdefine features as modules/lp.services.features.xxx/get_singleton09:45
lifelessor whatever the helper-that-looksup-in-a-thread-and-sets-if-not-set is called09:46
pooliehm09:46
pooliebut09:46
poolieif it's not set on the request, i wonder if it will be set at all?09:46
pooliei don't think it makes sense to have "just some random flags" set09:47
poolieiow it has to be created for a request09:47
pooliei don't think we want a ground-state value09:47
lifelessso09:47
lifelessa few thoughts09:47
lifelessthe NullObject pattern may help - you can say 'here is a features thing' that does nothing09:48
lifelessthis is really a tests-do-not-match-reality problem its nothing to do with reality09:48
pooliemm09:48
lifelessin reality, we'll only look things up in the context of the publication,09:48
pooliei think if the tests explicitly opt in to "i'm not in a request but i want to pretend"09:49
pooliethat would be ok09:49
lifelessbut I'd like to get the feature first and cleanup the testing second, because of the timeliness fo rthis09:49
pooliebut there's some risk of hiding bugs09:49
lifelesswell they are doing that, but its implicit09:49
poolieso i see in some of these cases that we have a LaunchpadTestRequest instance09:49
pooliemaybe that's not created in the usual way09:49
lifelessas in, we can only systematically tell that they haven't used the publication by the fact that they haven't used the publication09:49
poolieif we wanted to give it a null object, that might be a reasonable place to put it09:50
lifelessyou could, if you are particularly concerned, use the hook you have to replace the null object with a real one, and have folk coming in the back door get the real object09:50
lifelessOTOH09:50
lifelesswe want features in cron scripts and so on too09:50
lifelessso 'request' is really too tight a scope anyhow09:50
pooliesure09:50
pooliei think basically, someone should have an opinion on what flags ought to be active09:51
pooliethat could be publication, or cron framework09:51
poolieor a test saying "just notihng"09:51
lifelessIdeally, I completely agree.09:51
lifelessWe could change the doctest loader to state this as a setUp for all doctests.09:51
lifelessbut that has the risk of fallout across the board impacting things09:52
poolie:/ i should add the tribunal "hide tests containing string $x" fetaure :)09:52
lifelessso, I might sleep on this.09:53
pooliei think i will add a null object created by the test request09:53
poolieto start with09:53
lifelessI think we are running into an accomodation of the test suite to the performance constraint09:53
poolieperhaps just actually hardcoded to null09:53
poolieor more precisely getitem(x)->None09:53
lifelessand its going to cause a lot of friction like this09:53
poolieoh, which constraint?09:53
lifeless'things are slow' -> developers avoid calling the full stack :)09:54
lifelessits also a stack thing - folk want to (reasonably) exercise the views directly09:54
lifelessbut the views implicitly depend on the full publication happening09:54
lifeless(I can name other things like this that are only by-chance not causing headaches)09:54
lifelesssuch as profiling09:54
lifelessor the request summary information which is reset by the publisher.09:55
lifelessor the storm cache09:55
poolieheh09:55
lifelessso the thing that is unique about your code, is that you are indirecting via the request09:55
lifelessrathe than being sloppy and tolerating these tests getting silly things back09:55
lifelessThis is why I'm encouraging you to be sloppy (but not careless) here09:56
lifelessanyhow, 9pm; time for a break, I'll be needing to chat to mars in a few hours09:58
lifelessavoir09:58
poolielet's try this: http://pastebin.ubuntu.com/475347/09:59
wgrant[A2209:59
wgrantGah.09:59
pooliesure, see you later09:59
stublifeless: pong09:59
lifelessstub: nevermind, sorry. 'unpong' :P10:00
lifelesspoolie: I think thats a good start10:00
lifelesspoolie: also RequestExpired will need it10:00
lifeless(note that it is RuntimeError :P)10:00
pooliehm10:01
lifeless(look at the webservice-error.txt file)10:01
lifelesspoolie: as for the tests that are not in your branch and failing, merge db-stable to your branch, and you should get them all10:02
poolie?10:02
lifelessor db-devel to your branch10:02
poolieno that's the same Request tiype10:02
pooliesure10:02
poolieis this now targeted at devel or db-devel?10:02
pooliei guess db-devel now merged to devel?10:02
pooliesince my db changes?10:02
lifelessnot yet, late this week10:02
lifelessclass RequestExpired(RuntimeError):10:02
lifeless    """Request has timed out."""10:02
lifeless    implements(IRequestExpired)10:02
lifelessthats in adapter.py10:03
lifelessanyhow, gl, chat with you tomorrow10:03
lifelessby 'thats a good start' I meant your pastebin, in case it wasn't clear.10:04
pooliegood night10:04
poolieunderstood10:04
pooliei don't see what i need to do with requestexpired10:04
pooliethat test is using the same request class10:04
lifelessoh10:04
lifelessinteresting10:04
lifelessok, cool.10:04
poolieis it just me with launchpad tests failing in10:07
pooliewith a deprecationwarning in bzr builder?10:07
pooliemaybe update-sourcecode will fix it10:08
bigjoolswgrant: evening10:08
wgrantbigjools: Hi.10:09
bigjoolswgrant: what part of soyuz uses guessPackageNames>10:10
bigjools?10:10
wgrantI have to disappear in a few minutes.10:10
wgrantbigjools: Nothing.10:10
bigjoolsok10:10
* bigjools wonders why it's a soyuz bug then :)10:10
wgrantbigjools: Someone said so. I don't recall who.10:10
wgrantAnd it is a Soyuzy query!10:10
bigjoolscurtis probably10:10
bigjoolssigh10:10
wgrantIt's mostly used by the bug target picker.10:10
bigjoolsI've no idea how to QA it10:10
bigjoolswhich most likely means it's not a soyuz bug10:11
wgrantIt's one of those things that's in the void, and nobody has touched for years.10:11
wgrantI will QA it when I return later this evening.10:11
bigjoolsok10:11
bigjoolscheers10:11
poolienight all10:16
wgrant[A2210:16
wgrantGAH.10:16
wgrantWhy has this started happening?10:16
=== jtv is now known as jtv-afk
bigjoolsthumper: around?10:37
thumperbigjools: no11:12
thumperbigjools: you have 3 minutes11:12
thumperwhazzup?11:12
bigjoolsreviewed your branch11:12
bigjoolsgotta dash myself11:12
jmlhello11:12
thumperbigjools: I had taken lifeless's rc as ok and landed it already, the original diff was updated to use config.launchpad.no_reply_email11:13
thumperbigjools: if you like we can file another bug and look at it for the start of next cycle11:14
lifelessleonardr: hi12:06
leonardrhi lifeless12:06
deryckMorning, all12:07
lifelessmorning deryck12:07
lifelesshave a good weekend ?12:07
wgrantbigjools: Hi. Am I likely to be granted an RC for bug #615286?12:07
_mup_Bug #615286: DEPWAIT not recognized from build log <Launchpad Auto Build System:New> <https://launchpad.net/bugs/615286>12:07
wgrantSimple lp-buildd regex change.12:07
derycklifeless, indeed.  Busy but relaxing.  Thanks!  And you?12:07
bigjoolswgrant: no, it's not RC12:07
bigjoolsthey can manually retry12:07
lifelessderyck: pretty good, house stuff + hacked on qa-tagger sunday12:07
deryckcool12:08
lifelesswgrant: cherrypick it IMO12:08
lifelesswe're getting rid of the whole release thing this cycle, if everything comes out well12:08
bigjoolslifeless: it's a buildd change, it's not affected by the release at all12:08
bigjoolsIS releases buildd changes12:09
lifelessbigjools: yah12:09
bigjoolsso it's not a CP :)12:09
lifelesswell, in the sense of 'deploy without waiting a month'12:09
lifelesswe dont have a good phrase for that atm12:09
deryckheh12:10
spiv"deploy now"12:10
deryckmy "heh" was because I read lifeless as "deploy without *wasting* a month."  Clearly I need more coffee.12:13
lifelessheh12:13
lifelessnice misread there12:13
lifelessgrah12:19
lifelesswhen we are moving to lucid ec2 test images ?12:19
jmlafter we move to lucid everything else?12:21
* jml isn't sure.12:21
lifelessneither am I12:21
lifelessAIUI bb looks at lucid now for blessing12:21
lifelessso it would make sense to move the ec2 images to lucid12:21
jmlagreed12:21
lifelesshave I mentioned I fear and loath doctest failures12:22
jmllifeless, once or twice12:22
jml(also, "loathe" is a verb, "loath" is an adjective)12:23
lifeless+e12:23
StevenKlifeless: O hai.12:23
StevenKlifeless: You mentioned a transform that Storm could do for me, WRT _copy_lucille_config. Could I have an example?12:24
lifelessuhm12:24
lifelessshow me your diff again :P12:24
StevenKlifeless: https://code.edge.launchpad.net/~stevenk/launchpad/move-ifp-from-idistroseries/+merge/3152012:25
StevenKlifeless: I didn't want to change the current code, but I have more work in the same area, and I could use what you were suggesting12:25
lifelessright12:26
lifelessyou create a result set12:26
lifelessstore.find(DistroSeries, DistroSeries.id == self.distroseries.id)12:28
lifelessthen you call set on it12:28
lifelesspdr = Alias(DistroSeries,12:30
lifelessname='pdr')12:30
lifelessresultset.set(DistroSeries.lucilleconfig=Select(pdr.lucilleconfig, pdr.id==self.parentid))12:31
lifelessI think thats it - check the sql it emits of course12:31
StevenKlifeless: Right, but to do a copy like that in general, .set() is my friend?12:32
lifelessto do an UPDATE, .set() on a result set is your friend12:32
lifelessthat 'copy' is actually a two-stage thing, it duplicates, then it changes12:32
lifelessor something like that - the one method is really 'trash the config using that one over there'12:33
* bigjools actually thinks that SQL syntax is better for once12:34
* lifeless doesn't really care12:34
lifelesswith the registry stuff, doing it in raw sql would be worse I think12:34
bigjoolsprobably, but why?12:35
bigjoolsStorm syntax is generally pretty unintelligible for me12:35
lifelessfor the registry stuff I was doing, because it is combining 6-7 different clauses, constraints and tables into one aggregate query12:35
lifelessbigjools: I'm saying 'consider on a case by case basis' :)12:36
bigjoolsI completely agree :)12:36
lifelessstorm is so tied to sql that if we wanted to do things differently we're no better-or-worse off12:36
gmblifeless, Can we consider https://bugs.edge.launchpad.net/malone/+bug/607776 qa-ok?12:46
_mup_Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>12:46
derycklifeless, I think we can mark bug 607776 as qa-ok.  Would you agree?12:46
_mup_Bug #607776: +filebug-show-similar FTI query timeouts <qa-needstesting> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/607776>12:46
deryckheh12:46
lifelessjinx12:46
lifelessyes, absolutely.12:46
lifelessmea culpa12:46
deryckno worries.12:46
gmbDOne12:47
deryckgmb, I'll leave it to you to mark, since I stepped on you once. ;)12:47
lifelessI've been watching like a hawk for issue12:47
lifeless*issues*12:47
gmbderyck, Heh. I wonder what happens if two people try to make the same change. Probably the world ends or something.12:47
deryckme, too.  Seems solid to me.12:47
deryckthe AJAX world ends12:47
lifelessits still possible we'll find an important common use case it doesn't handle well, but for now, its happy12:47
deryckgmb, I'm still working on all the ones for my dupe work.  Staging doesn't have the change yet, so I'm poking around bugs that have moved duplicates to make sure things look good.12:48
gmbRight12:49
lifeless_mup_: poke mars12:50
=== bigjools changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
deryckOk, bugs team is down to 4 bugs needing qa now.  One requires a chat with bdmurray.  I think gmb is aware and working on qa for bug 594211.13:02
_mup_Bug #594211: Add a RecipientReason class to bugnotificationrecipients.py <qa-needstesting> <story-better-bug-notification> <story-refactor-bugnotifications> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/594211>13:02
deryckThe other two are me and need my changes on staging to qa.13:02
deryckbigjools, ^^13:02
bigjoolsderyck: rock n roll, thanks13:03
gmbderyck, bigjools That's qa-ok; marking it as such now.13:03
derycknp, sorry it took us so long this time.13:03
deryckgmb, excellent, thanks!13:03
* gmb lunches13:04
* bigjools -> lunch13:10
lifelessgmb: are you sure bug 606914 is fixed ?13:17
_mup_Bug #606914: Bug.initial_message fetches all of the bug's messages <qa-ok> <timeout> <Launchpad Bugs:Fix Committed by gmb> <https://launchpad.net/bugs/606914>13:17
lifelessgmb: unless there is a test checking the sql count, I suspect we'll find its still a problem after the rollout13:18
=== Ursinha-afk is now known as Ursinha
lifelessgnight all13:37
=== matsubara-afk is now known as matsubara
jtv-afkoh, yeah, nick.13:54
=== jtv-afk is now known as jtv
gmblifeless, I'll check14:06
gmblifeless, So, that particular bug - and the problem described in it (that initial_message was stupid) is fixed. It doesn't fix the timeout problem, but there are other bugs covering that (which may or may not need consolidating into one sensible report, or at least clarifying so that the distinctions between them become more obvious). I'm happy for this particular one to be qa-ok, though.14:51
=== matsubara is now known as matsubara-lunch
=== deryck is now known as deryck[lunch]
=== Ursinha is now known as Ursinha-lunch
salgadosinzui, do you know where's the code that allows traversal from a distro to a distroseries? I thought it'd be in DistributionNavigation but it's not16:16
salgadooh, it must come from GetitemNavigation,16:18
salgadoindeed, that's it16:18
=== deryck[lunch] is now known as deryck
=== matsubara-lunch is now known as matsubara
=== Ursinha-lunch is now known as Ursinha
=== beuno is now known as beuno-lunch
cperrin88Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?17:59
cperrin88the whole root wadl file17:59
cperrin88The sounds like no18:10
marscperrin88, leonardr is working on that very problem18:11
marscperrin88, IIRC older versions of wadllib grabbed the whole thing.  Newer versions cache it.18:11
leonardrmars, cperrin88, i was off irc during the crucial moment. can you recap?18:11
marsleonardr, <cperrin88> Hey, is someone here familiar with how launchpadlib handels the WADL file. Does it download the whole file at first?18:12
mars<cperrin88> the whole root wadl file18:12
=== beuno-lunch is now known as beuno
leonardrcperrin88: yes, it will get the wadl file on first startup and about once a month afterwards (whenever launchpad is upgraded)18:13
cperrin88okay .... well ... I'm doind a Java port of the lib ...18:14
leonardrcperrin88: ok, you can implement the same behavior. the wadl file is served with cache-control headers and an etag18:14
cperrin88Yeas18:14
leonardrso you can know how long to cache it for, and once the cache expires, you can avoid getting the whole document unless launchpad has been upgraded18:15
cperrin88that's what I wanted to do18:15
cperrin88I just wanted to make sure that you really grab the whole thing18:15
cperrin88it's rather big18:15
leonardrcperrin88: if you send Accept-Encoding: gzip it's only about 100k18:15
cperrin88hey18:16
cperrin88your right18:16
cperrin88that's cool18:16
cperrin88without it's more than 10 times the size18:17
leonardrxml compresses very well18:17
cperrin88seems like .... that helps a lot18:17
leonardrso if you implement all these optimizations your client will be making 1 http request once a week and once a month that http request will result in a 100k download18:17
cperrin88I was concerned about mobile users18:17
cperrin88yep18:18
leonardrnote that there is also a json version of the service root, which has the same rules, but it's much smaller--only about 1k uncompressed18:18
cperrin88The lib is a result of my development of a Launchpad android client18:18
cperrin88Yeah18:19
cperrin88I know18:19
leonardrok18:20
cperrin88but I think I will handle it more like launchpadlib does now18:20
cperrin88thank you for your help18:20
leonardrsure18:20
rockstarbenji, ping18:31
benjirockstar: hey18:31
rockstarbenji, do you know much about View permissions?18:32
benjia bit, what's up?18:32
cperrin88leonardr: I think I found a small error in the WADL file. Two doc parts have invalied entitiys ... At least my parser says that18:34
leonardrcperrin: send me the doc output. we validate the wadl before deploying, but it's possible we have access to some external doc you don't18:35
cperrin88it's   `     (Apostrophe)18:35
cperrin88lemme see18:36
cperrin88I take it back18:38
cperrin88I guess it was my fault18:38
leonardrok18:39
benji___rockstar: sorry, had some technical difficulties; will you repeat your question?18:41
=== benji___ is now known as benji
rockstarbenji, permissions on View don't care what the request type is, correct?  It's the same for GET and POST?18:41
benjirockstar: correct18:42
benjipermissions don't assume "correct" use of HTTP verbs18:42
=== salgado is now known as salgado-afk
lifelessmoin20:40
lifelessmars: hi20:40
marsHi lifeless20:42
=== matsubara is now known as matsubara-afk
mwhudsonmorning20:43
lifelessmars: Ursinha: I just got up, so haven't trawled email yet; I'm going to look for review feedback now20:44
marslifeless, Ursinha and I have both looked over your changes.  They look good - thanks for doing that.  She is just finishing her review now.20:44
Ursinhayes sir20:44
lifelesscool, I hope its helpful20:44
marslifeless, very.  Good hooks for the parts we still need to write, too20:44
Ursinhalifeless, do you intend to write more code there?20:45
lifelessUrsinha: maybe :) - AFAICT theres enough there now that we could switch to the new workflow and iterate on polish ?20:45
Ursinhalifeless, and yes, that was pretty much what I thought to be a way to reuse qa-tagger structure, very helpful20:46
=== lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Performance Day! | Week 4 of 10.08, Release Manager: bigjools | PQM is release-critical | firefighting: - | buildbot/pqm has been switched to watching the *lucid* builders | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes
marslifeless, Ursinha may have to double-check the qa-sort algorithm.  But there should be enough to start integration with whatever downstream component there will be.20:48
Ursinhalifeless, hm, not sure, I'm not that familiar with the whole changes to put that in prod now20:48
marslifeless, assuming Ursinha thinks the script output is still sane for our current daily production use :)20:49
lifelessso20:49
lifelessif more work is needed to make this usable for the new merge workflow where we qa edge and request deployments from the losas at up-to revision granularity20:50
lifeless(usable, not 'perfect' :P)20:50
marsof course20:50
lifelessthen - yes, I'll write more code, while you guys sleep tonight :)20:50
lifelessIf, OTOH, its sufficient to get us started, then I've some other things to help along to get that workflow transition to happen20:51
marsUrsinha, would it be possible to run this tagger code beside the existing stable tagger code?  So we don't interrupt our production line?20:52
marslifeless, Ursinha is in the process of reviewing the code to make sure that the already in-place QA process and tagger are not interrupted.20:53
Ursinhamars, well, yes, we can do that and tag bugs twice or change one of them a bit so we wouldn't have the two fighting over bugs20:53
marsUrsinha, perhaps we could *not* run the tagger part?  By adding a CLI switch or something?20:54
Ursinhalifeless, btw, waking up having code to review is nice20:54
Ursinhamars, that can be done20:54
lifelessmars: we don't need to run them in parallel so much as be sure that we have *enough* so that when we switch we can use it20:54
lifelessUrsinha: my pleasure20:54
lifelessmars: although in parallel with a smooth transition is nice, obviously20:54
Ursinhawe still need to consider the rollback commits20:54
lifelessok20:55
Ursinhanot much of a big deal but needs a bit of code there20:55
lifelessthey are tagged qa-rolledback or something ?20:55
Ursinhaqa-rollback20:55
marslifeless, well, I believe Ursinha's goal is not to get enough working, it is 'can be guaranteed to still work like it did on Friday' :)20:55
Ursinhapretty much :)20:55
lifelessdoes it mean 'needs to be rolled back', or 'it has been rolled back'20:55
Ursinhalifeless, it was rolled back20:55
lifelesskk20:55
lifelessso, if 'qa-ok' in tags or 'qa-rollback' in tags :)20:56
lifelessshould be a 12-character fix :)20:56
mars+ test20:56
lifelessof course20:56
Ursinhalifeless, https://dev.launchpad.net/QAProcessContinuousRollouts20:56
Ursinhalifeless, it also needs to add the tag, when appropriated20:57
lifelessmars: so the overall -goal- here is to change the way we work; keeping things unaltered is insufficient ;)20:57
lifelessUrsinha: ok, based on the new lp-land switches ?20:57
Ursinhathe commit should have a [rollback] clause, same as [testfix]20:57
Ursinhaso we know that if a new commit is a rollback, we know the bug and know the last revision that's blocked waiting for this rollback20:58
lifelessso the tagger would tag the old revisions bug as rollback20:58
Ursinhaa rollback is a reversion of a qa-bad or qa-needstesting previous one20:58
Ursinhayes sir20:58
Ursinhaand let it go20:58
lifelessrighto20:59
lifelessso 'rollback' is 'unqaable' for the reporter, but still taggable for the tagger20:59
Ursinhalifeless, yes20:59
lifelessis that the total of the remaining must-have features ?21:00
Ursinhalifeless, why are you pinging me? :)21:00
lifelessbah, fat fingers21:00
Ursinhalifeless, unqaable: untestable, orphaned, testfix, buildbot, and now rollback21:00
Ursinhatherefore blessed21:00
lifelessI was meaning to do 'TIME'21:00
marslifeless, I understand.  I think Ursinha should make the call on when the new qa-tagger code can replace what is already running.  Parallel deployments is just an idea.  We don't have to take it.21:00
lifelessmars: I'm happy either way21:01
Ursinhamars, I like the idea of running them in parallel21:01
lifelessok, phone time for me21:01
lifelessUrsinha: Its your 5pm now right?21:01
Ursinhalifeless, yes21:01
lifelessI'll look at adding rollback now21:01
Ursinhalifeless, cool21:02
Ursinhathanks21:02
lifelessif there is more than that needed, let me know please before you retire for the night.21:02
Ursinhasure21:02
Ursinhalifeless, that wiki page has details, maybe that helps a bit21:03
marslifeless, btw, are you on the Kanban board, and can't edit?  Or do you not have an account at all?21:03
lifelesscouldn't edit your board21:03
marsok21:03
marsno admin privs - I can't change it.  Oh well.21:06
lifelessflacoste: https://code.edge.launchpad.net/~lifeless/launchpad/registry/+merge/3206721:11
sinzuilifeless, I started this. I got distracted by my sprint21:12
lifelesssinzui: sprinting is good21:13
sinzuilifeless, I forsee a few state issues in this branch21:13
lifelesssinzui: how do you feel if it lands without your review? or could you give it enough cycles to say 'thats ok we can tweak later' or 'lifeless, you need to look at x,y,z'21:13
lifelesssinzui: ok, can you roughly describe them ?21:14
sinzuiI can deactivate a person and the page will still show active...I need to refresh my browser I think21:14
sinzuilifeless, Person.preferredemail knows when it is mutated to manually purge its _*_cached property21:16
sinzuilifeless, But i am speculating on these scenarios  for coc and archive21:18
sinzuilifeless, I am fine with this landing. We can fix issues as we discover them21:20
lifelesssinzui: thanks!21:20
lifelessflacoste: line 68021:21
flacostelifeless: lp/registry/model/product.py:47021:27
EdwinGrubbsjelmer: ping21:37
jelmerEdwinGrubbs, pong21:38
EdwinGrubbsjelmer: it looks like you can set the upstream project differently on each distroseries? Is that a good idea.21:40
jelmerEdwinGrubbs: there are some (very rare) situations in which that is necessary21:42
jelmerEdwinGrubbs: e.g. I think that the chromium package used to be for the chromium bsu game but now is for the chromium browser21:42
EdwinGrubbsok21:42
marssinzui, does pocketlint not catch tab and indentation errors?  (check person.py line 3603-5)21:50
marstabnanny.py yelled at me after finding that21:51
marssinzui, lib/lp/registry/browser/person.py:360321:51
sinzuimars, I think it yells twice in some cases21:52
marsI'm kind of weirded out by the interpreter not stopping on that.  I would figure the change of indentation type in the middle of a method would cause something bad to happen.21:53
benjimars: the interpreter equates tabs with eight spaces21:56
sinzuiSomeone landed tabs in registry code 6 weeks ago. jtv reported it and I added tab checking21:57
sinzuimars, yes, pocketlint hates the file21:57
sinzuimars, should it call bzr blame for those lines too?21:57
sinzuimars, actually, I see bac's emacs is set to correct tabs to spaces and his diff in review accidentally started removing the tabs :)21:57
marssinzui, pocketlint on dev systems?  It ignores it on mine.21:57
marssinzui, should we ditch tabnanny then?21:58
marsIt would make test-on-merge faster21:58
sinzuihmm, maybe the version is wrong in developer-deps21:59
sinzuimars, this is what I see http://pastebin.ubuntu.com/475630/21:59
marssinzui, exit code zero?22:00
marssinzui, ok, PEBCAK here on the output - but the exit code is a bit odd.22:01
marssinzui, so I'll remove tabnanny from test_on_merge.py then.  That is what lint is for, and that file has no need of the additional complexity.22:01
sinzuiAs I said above, developer-deps is behind22:02
sinzuiI stopped publishing pocket-lint because hardy had build issues22:02
sinzuiYou can get the latest version from my ppa, or wait till next week (after the release) where I can push a version that will play in the test runner22:03
marsdone - tabnanny in test_on_merge is no more22:03
sinzuilint also screams at pdbs22:03
sinzuimaybe we want a test that checks the code for pdbs22:04
marsdon't know - might be a good idea, but it would take a while to run.  If most people run lint, then it may not be necessary.22:04
marslint catches far more things22:05
marslint is already a 'best effort'22:05
benjileonardr: I started to write the MP for ~benji/lazr.restfulclient/total_size_link and realized that you might be the better one to write it because you did almost all the work :)22:13
benjioops, wrong chan22:13
lifelesssinzui: thanks for approving that; will land as soon as pqm opens :)22:20
bdmurraysinzui: I'm having an issue with return Link() in lib/lp/registry/browser/structuralsubscription.py and StructuralSubscriptionMenuMixin.  deryck thought I might check with you about it.22:27
lifelessleonardr: ping22:28
mwhudsonrockstar: are you confusing TAL and ZCML in your mail to launchpad-dev?22:43
rockstarmwhudson, yes, because it is all ugly...22:44
=== Ursinha is now known as Ursinha-afk
wgrantmwhudson: ShipIt sadly uses TeamParticipation and Person.23:05
wgrantSo it's not that easy.23:05
mwhudsonwgrant: ah23:05
mwhudsonoh well23:05
lifelessweb services23:05
wgrantRight.23:05
wgrantThe former can be done from SSO.23:05
wgrantAnd the latter is just for karma.23:05
wgrantSo it's not *that* hard.23:05
* lifeless hates code freeze23:06
lifelessdo we have an unfrozen branch for folk to work on this week ?23:06
* ajmitch hates seeing edge timeouts :)23:06
wgrantlifeless: No :(23:06
lifelessor do we all just pile up and conflict on friday?23:06
wgrantPrecisely.23:07
wgrantI believe you have a fix for this, though :)23:07
lifelessajmitch: whats the oops id ?23:07
ajmitchOOPS-1682EC407823:07
ajmitchit was for a bug link that had comments=all23:07
ajmitchso no surprise that it'd take a little while23:07
lifelesswell23:07
lifelesslets see23:07
lifelessStatement Count: 30323:08
ajmitchperformance tuesday for you?23:08
lifelessthats going to be a large driver23:08
lifelessajmitch: yup23:08
lifelessyeah, lots of repeated things23:08
lifeless285.682524mslaunchpad-main-slaveSELECT LibraryFileContent.datecreated, LibraryFileContent.filesize, LibraryFileContent.id, LibraryFileContent.md5, LibraryFileContent.sha1 FROM LibraryFileContent WHERE LibraryFileContent.id = %s LIMIT 123:09
lifeless286.1154146mslaunchpad-main-slaveSELECT LibraryFileAlias.content, LibraryFileAlias.date_created, LibraryFileAlias.expires, LibraryFileAlias.filename, LibraryFileAlias.hits, LibraryFileAlias.id, LibraryFileAlias.last_accessed, LibraryFileAlias.mimetype, LibraryFileAlias.restricted FROM LibraryFileAlias WHERE LibraryFileAlias.id = %s LIMIT 123:09
lifelessnote that there is 3 seconds of time between those two db calls - thats going to be python23:09
lifelessnot 'lots of bytecode', rather 'some other thread screwed us'23:09
lifeless120 subscription lookups23:12
lifelesshttps://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/131094/+index23:12
_mup_Bug #131094: Heavy Disk I/O harms desktop responsiveness <cft-2.6.27> <Linux:Invalid> <linux (Ubuntu):Confirmed for ubuntu-kernel-team> <linux-source-2.6.22 (Ubuntu):Won't Fix by ben-collins> <https://launchpad.net/bugs/131094>23:12
pooliespm: hey, awake yet?23:16
wgrantpoolie: IS sprint...23:17
james_whi poolie23:17
james_whi wgrant23:17
wgrantMorning james_w.23:17
pooliehi james_w; thanks wgrant23:17
=== Ursinha-afk is now known as Ursinha
lifelesspoolie: all the losas are in UTC+2 this week23:18
lifelessUrsinha: / matsubara-afk: how can I reset the bug infestation for an oops ?23:25
Ursinhalifeless, let me find the tomboy note that explains that23:27
leonardrlifeless, hi23:28
lifeless\o/23:28
lifelessleonardr: hi23:28
lifelessleonardr: uhm, uhm, I've forgotten my thread23:28
Ursinhalifeless, wait, do you want to know just to know or you just need something to be changed?23:28
lifelessUrsinha: I keep finding wrong ones when you guys are asleep23:28
lifelessso I want to know how to do it myself23:28
Ursinhalifeless, ah sure23:28
leonardrlifeless, just type it if you remember, i'll come back later23:28
Ursinhalifeless, I'll send one email23:28
lifelessUrsinha: thanks!23:29
lifelessleonardr: no worries!23:29
Ursinhano problem :)23:29
james_wwgrant: want to tell me where https://code.edge.launchpad.net/~james-w/launchpad/move-soyuz-test-publisher/+merge/32157 falls on a scale of 1 to heinous?23:40
wgrantThat scrollbar scares me.23:41
wgrantAnd those are some serious docstrings.23:41
james_wyeah, I think that's where a lot of the vertical height comes from23:42
marsbzr question: I am using sandboxes with 'bzr switch', I have pushed my branch, bzr says 'Created new branch', but Launchpad insists "This branch has not been pushed to yet.".  What gives?23:42
marsOn top of that, 'bzr push --overwrite' says "No new revisions to push"23:43
wgrantmars: LP may be being a little slow. Replication lag, maybe?23:43
lifelessmars: did the url it reported make sense to you?23:44
marslifeless, which URL from which command?23:44
lifelesswhen you push, bzr tells you where it is pushing23:44
lifelessI think, or is that commit.23:44
lifelessanyhow, where di dyou push to ?23:44
lifelessprecisely23:45
marslifeless, ah, got it: incompatible repo formats23:45
marslifeless, and bzr doesn't re-raise the error when I try again23:45
wgrantjames_w: Hm, so test_publishing has a special STP derivative which uses sampledata?23:45
marslifeless, how did you push your qa-tagger code without it barfing?23:46
lifelessmars: I didn't put it in a shared repo23:46
lifelessits odd that bzr would only whine once23:47
marslifeless, ok - using 'upgrade this branch' on the UI is harmless? >:)23:47
lifelessmars: you need to coordinate with everyone23:47
marsyarg23:47
lifelessmars: e.g. with the losas because we roll this stuff out, unless qa have direct control over where its rolled out to and can do the upgrade on sodium-or-whatever23:47
marslifeless, this is under qa's control23:48
lifelessso, coordinate with them :)23:48
marslifeless, so I hit 'upgrade this branch', then what happens?  They try to pull new revisions and it blows up or something?23:48
lifelesswell23:49
lifelesstell me what 'this' is bound to23:49
lifelessbecause I don't have the full picture23:49
marshttps://code.edge.launchpad.net/~launchpad-qa/qa-tagger/devel23:49
lifelessok23:49
lifelessso if you upgrade that23:49
lifelessthen yes, everyone merging or pulling from it will blow up23:49
lifelessthey then need to upgrade and can then pull/merge appropriately23:50
marslifeless, will you still be able to land your unmerged work if I do this?23:50
wgrantjames_w: So, looks heinous, but it's probably the right thing anyway.23:50
lifelessmars: yes, I just need to upgrade my branch too - as I said, you need to coordinate with everyone :)23:51
wgrantUntil we can port everything away from the old one, which could be a while...23:51
lifelessmars: e.g. send a mail23:51
marslifeless, ok, will do23:51
* mars presses the Big Yellow Button23:51
thumperyellow?23:51
wgrantThe drunken exclamation mark.23:52
marsthumper, to be honest, I think the +upgrade page needs some work :/23:52
thumpermars: probably23:52
marsI could just copy and paste this IRC conversation :)23:53
mars"What does this button do?"  "Well, this conversation describes it well: ..."23:53
lifelesshmm, I could have sworn there was a bug open for oops' miscategorising storm timeouts23:55
marslifeless, so besides all the branch format shenanigans, I have some code that is built on top of yours.  You may want to merge it: lp:~mars/qa-tagger/lp-service-switch23:57
mars(not pushed yet, need the upgrade to finish first)23:58
lifelesswhat does it do /23:58
lifeless?23:58
lifelessdoes anyone remember the oops-doesn't-include-storm-timeouts-as-sql-time bug ?23:58
marslifeless, adds a --lp-service command-line switch.  That switch tells the script which lp instance to run against.  We want to use this for smoke-testing23:58
marslifeless, failsafes to 'staging'.  Any developer should be able to check out the code and safely do a smoketest this way.23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!