[00:14] wgrant: some already filed a bug http://youtrack.jetbrains.net/issue/PY-2741 [00:14] someone [00:20] actually, it seems like it pertains to code completion but it will be related [00:31] wallyworld_: this is the whole 'our packaging can only support one version at a time' issue. [00:33] lifeless: thanks. btw, a gentle reminder about https://code.launchpad.net/~xaav/loggerhead/export-tarball/+merge/66408 :-) [00:33] thanks ;P [00:37] an adaptor question - i think we should be able to specify a class as being an adaptor for more than one interface, wither via @adaptor(foo, bar) or in zcml for="foo bar"? it appears not since it doesn't work. and so i have to have one zcml adaptor element per interface, which the same adapter class/factory specfified. is there something i'm missing? [00:39] wallyworld_: I'm not sure... but it's probably not an adapter for more than one interface. [00:40] Those interfaces probably share a subinterface. [00:41] wgrant: in this case it is the same adapter implementation for more than one interface that i would like to specify. they do share a subinterface (in fact > 1) and the adaptor makes use of those >1 subinterfaces. you'd think it would be possible [00:42] in this case, i want to say "this specified class adapts both IDistribution and IProduct" [00:43] i can do it but need to have multiple adapter declarations with just a different for="xxx" which seems dumb [00:43] and i can't user @adapter() so am forced to use zcml [00:46] StevenK: QA! [00:47] wgrant: Yeah yeah :-P [01:01] lifeless: Can haz bazaar-experts team deleted? [01:01] StevenK: we've deployed everywhere since? [01:01] StevenK: including crannies like soyuz ? [01:02] I know cocobanana has been deployed to since, and one of germanium's trees is in NDT. Not sure about the other tree [01:10] poppy doesn't use bazaar-experts, so meh. [01:11] I'd be a little concerned if it did ... [01:48] Good morning [02:05] lifeless: Do you think the timeouts are regressions, or bloat? [02:05] We went from 300 to 1100 in a week. [02:06] Wheee [02:06] I suspect the subscription changes. [02:06] But that doesn't explain Person:EntryResource:searchTasks [02:06] I don't think. [02:08] wgrant: the soft counts last week would have been large if it was creeping bloatism [02:10] lifeless: So, can haz bazaar-experts deleted, or do you want to wait until every single tree doesn't mention the celebrity? [02:10] I'd like to wait [02:10] easy to do, reduces risk, doesn't slow anything down. [02:10] lifeless: Until when? :-) [02:11] If you say the heat death of the universe, I'm booking a flight to NZ :-P [02:11] no live trees with the celeb in them [02:12] The only services I can think of are germanium's poppy and probably forster [02:12] Oh, and the librarian? [02:17] likely, yes. [02:35] wallyworld_: hi [02:35] hello [02:35] wallyworld_: your affiliation work; it looks like its going to do lots of eager loading. [02:36] i put in tests for the query count (for bugtask) - it's only 4 quesies soince stuff is already loaded [02:36] per person [02:37] yes, that is true [02:37] s/eager loading/lazy loading/ [02:37] so i probably need to fix that [02:38] wallyworld_: anytime you end up writing adapters, and object orientated code that does data access, its probably going to involve late evaluation/lazy loading problems. [02:38] wallyworld_: the -only- case I know where such code is fine is when it happens /once/ on a page/http request as part of establishing the context (rather than potentially in inner loops etc) [02:39] i'm starting to see that. i still haven't got it hard wired in my brain that zope object navigation = queries [02:39] also inTeam() for instance. [02:39] userA.inTeam(teamT) [02:39] i'm used to popo/pojo type domain models [02:39] userB.inTeam(teamT) [02:40] will do 2 queries, even if userA, userB and teamT are all already loaded. [02:40] to access the join table? [02:40] yup [02:40] ah, well it's behind a feature flag at least. i'll do a branch to optimise. that's for pointing it out [02:42] except i think we turn on the feature flag recently, must check [02:42] picker_enhancements is on. [02:47] lifeless: the issue with the implementation is that the call to the affiliation adapter is done inside getPickerEntry(), which itself is an adapter method which is done per picker row [02:48] so there's potentially a bit of work to do to change all that [02:49] still, must be done [02:56] not quite so straightforward, since each item in a batch might not use the same picker entry adaptor [02:56] right [02:57] It's common to run into friction like this [03:05] yeah, so now picker entry adapters and HugeVocabularyJSONView are getting a bit of rework [03:49] lifeless: the current affiliation adaptor implmentation may not be so bad. the inTeam checks will be cached and the 4 queries i mentioned before are not per person i don't believe - eg once the pillar.bug_supervisor is fetched, it is not required to hit the db again for each person to check [03:51] actually, the inteam caching assertion is bogus [03:52] the current caching implementation is (team, member) specific [03:52] what you need is an interface for populating many such cache tuples at once [03:53] there's lots of other places where we do inTeam(xxx) or inTeam(yyy) eg canBeUnsubscribedBy() [03:53] and to call that with all the teams, for all the rows, and all the row person-or-team objects at once. [03:54] that is a solution but it severely breaks encapulation of the adapter implementation [03:55] yes [03:55] such encapasulation is harmful in our environment TBH [03:55] it may be easier to load the team members inside the adapter implementatiin and cache those [03:56] encapsulation is preserved and cachiing can still occur [03:56] is the adapter per row ? [03:56] the adapter is per row but the teams are static for a given adaper instance [03:57] to solve the problem you need to have set based interfaces. [03:57] anything that is object based or row based will have late evaluation symptoms [03:57] or load the team members the adapter knows it needs up front [03:57] no [03:58] that won't work because the adapter is per row [03:58] it will work - each row is checking for emebrship in fixed set of teams, known in advance [03:58] uhm [03:58] so if those team members are loaded and cached, no extra db access is needed [03:59] Bug #822500 [03:59] <_mup_> Bug #822500: select < https://launchpad.net/bugs/822500 > [03:59] wallyworld_: This doesn't fit with my understanding of the code. [03:59] Nice title [04:00] wallyworld_: calling personA.inTeam(teamT) will not seed the cache for calling personB.inTeam(teamT) [04:00] In fact, I found something odd last night -- https://code.launchpad.net/~aaacavazos/+recipe/launchpad-daily-1 [04:00] lifeless: yes, that's why i am saying we need to load and cache these relationships [04:01] do it once up front for the known, fixed teams [04:01] wallyworld_: which requires knowing all the personX's and all the teamY's, doesn't it ? [04:01] And there are seven recipes for stable ... [04:01] lifeless: a person (row) is affiliated with the context (eg bug task) if they are a member of (eg) the security contacts team or the bug supervisors team [04:02] so we can load and cache the team membership of those 2 teams [04:02] then as each row is checked, no extra db acces sis needed [04:02] i guess that doesn't account for indirect membership [04:03] but it could [04:03] wallyworld_: this seems both more work, and more code, than accumulating the set of (person, team) pairs we need an doing one call to check them all. [04:04] but it preserves encapulation and the inheritance model used but the adapter implenmentation [04:04] by more work I mean more runtime work. [04:05] You have to be sure what you're encapsulating; if you're encapsulting the wrong things its a pessimisation. [04:05] hmm. not sure. maybe. [04:06] what's being encapulated is the method to find out a person's affiliation with a given context [04:06] yes, (I've read the patches) [04:06] this is *exactly* the sort of thing I've had to rip out to fix performance issues up and down the LP stack. bugs. soyuz. questions. blueprints. [04:06] and as dicussed, it hooks into the existing workflow of one call per picker row [04:07] productseries has its own driver [04:07] a product with 15 series will have 15 series teams to check. [04:07] i don't think that sort of check is done currently [04:07] (if you honour them all) [04:08] if the context is a product, then series doesn't come into it [04:08] anyhow, I can see this isn't making sense to you [04:08] so we can drop it and be evidence based. [04:08] it is, i'm just not quite agreeing with the suggested approach (yet) [04:09] All I can say, is that everything I've seen over the last year predisposes me to consider *any* per-single-object interface flawed by design. [04:09] sure, but there's a balance between that and adhering to good oo design principals [04:10] not for code involved in batches/iteration/queries [04:11] yes, i can see the merit in that statement [04:11] (oh and for loops, which is implicit in iteration) [04:11] OO design hasn't scaled to non-main-memory datasets IMO [04:12] there's two ways of solving it - use set based interfaces or provide a caching implementation that prevents single calls from hitting the db [04:12] I agree there are two approaches, but the latter isn't one of the two - it doesn't work. [04:13] ie for a given use case, load the domain model known to be required to satisfy the request [04:13] (the second approach I'd layout is the object query language approach) [04:14] i've used that in combination with my second approach above [04:14] there are two disjoint problems with caching implementations [04:14] if you avoid one you get the other [04:15] the first problem is bog standard late evaluation [04:15] even if you cache each single call, when you iterate you make a new call for each person, and thats a cache miss [04:16] but if the domain model that is being reference for each call is the same... [04:16] the answer to that is (broadly) readahead: when you get two similar lookups your cache can decide to load everything related [04:16] wallyworld_: its not in your case, but we'll come back to that :) [04:16] the problem with readahead is that it blows out in size [04:17] e.g. readahead in the inTeam case would be to select person from teamparticipation where team=self.id, in inTeam [04:17] readahead is suboptimal - i normally don't like tyo use it except ib very narrow, specific circumstances [04:17] rather than looking up single members. [04:17] for LP, we have some very big teams: [04:17] select count(*) from teamparticipation group by team order by count(*) desc limit 20; [04:17] count [04:17] ------- [04:18] 18490 [04:18] (dropping to 855 at the 20th) [04:18] but still, those big teams, in a naive implementation, will be a bit of a headache. [04:19] (for instance, loading 18K rows from disk, if we had a cold cache situation, is ~ 40 seconds on its own) [04:19] what's that large team called? [04:20] that one is locoteams [04:20] 18490 | 2968414 | locoteams [04:20] 11026 | 2979678 | locoteams-approved [04:20] 3842 | 2388044 | launchpad-users [04:20] 2936 | 1121852 | ubuntu-us [04:22] wallyworld_: but I only brought that up to illustrate the solution-is-worse-than-the-problem nature of using on demand caching strategies in this sort of problem [04:23] given the size of the largest teams, i can see the issue. [04:24] although i do wonder whether such team would ever be a product owner or bug supervisor etc [04:24] i had the feeling such attributes would be individuals or small teams [04:24] for loco team issue tracking, sure ;) [04:25] fair enough. one of my underlying assumptions was that the team sizes involved in affiliation would be smallish [04:26] thus allowing other interfaces than a set based one [04:26] ubuntu-bugs has 416 people in it [04:26] openstack has 1300 [04:27] ubuntu-marketing 1400 [04:27] that's not too bad - caching 1300 id tuples [04:28] but anyway, that seems like suboptimal approach now [04:29] it also means that the adaptor based approach needs to be rethought i suspect [04:29] actually, maybe not [04:29] if you expose the things that need looking up [04:30] just the getAffiliation interface [04:30] a single call can seed the caches of the various teams with the results for all the queries that may be made [04:30] this is a set based interface [04:30] set based doesn't mean 'no abstraction' :) [04:30] I think we probably need to do away with adapters at some point. [04:31] It is always tempting to use them because they are nice and we use them already, but they make it awkward to setify. [04:31] i think i just need to change getAffiliationBadge(self, person) to getAffiliationBadges(self, people) [04:31] Oh, if the API is like that, that's not too bad. [04:31] or something like that, plus all the upstream changes for hooking it into the vocab [04:31] As long as there aren't many affiliation contexts. [04:32] well, atm, each batch of picker results is for the same context, so there's only one [04:33] lifeless: i think the TeamParticipation table is good for indirect memberships too, right? [04:33] Yes. [04:34] TeamParticipation includes indirect memberships. [04:34] right. makes it a bit easier. [04:34] wallyworld_: yes, I would encourage any implementation to start with refactoring inTeam() to have a single multi-lookup method it goes through. [04:34] lifeless: agreed. inTeam will likely not be used now [04:38] wallyworld_: cool; I suggest a single method so that we don't have duplication [04:39] lifeless: depending on how many usages of inTeam there are, I may optimise the affiliation stuff separately [04:39] i want to get that done asap without the baggage associated with an inTeam refactoring [04:40] across the entire product [04:40] wallyworld_: you shouldn't need to touch other callers of inTeam [04:40] they are either fine, or separate problems. [04:40] I'm just talking 'extract Method' [04:41] And leave an inTeam wrapper around the new method, for compatibility. [04:41] lifeless: oh, i see what you are saying i think. as if i would cut'n'paste code everywhere :-/ [04:41] wgrant: compat/convenience [04:45] wgrant: all the searchTasks timeouts are on one person [04:45] Aha. [04:45] I didn't look that far. [04:45] could well be patching [04:45] 276 https://api.launchpad.net/1.0/%7Eubuntu-security-sponsors (Person:EntryResource:searchTasks) [04:45] OOPS-2045A18, OOPS-2045A33, OOPS-2045A34, OOPS-2045A39, OOPS-2045A52 [04:45] 276 SELECT BugTask.assignee, BugTask.bug, BugTask.bugwatch, BugTask.date_assigned, BugTask.date_close ... = FALSE)) AS BugTask JOIN Bug ON BugTask.bug = Bug.id ORDER BY BugTask.id LIMIT $INT OFFSET $INT: [04:46] I suspect its the high offset issue [04:46] So adeuring will save the world. === danilo_ is now known as danilos === henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 247 - 0:[#######=]:256 [08:00] morning [08:22] good morning [08:24] Hi [09:47] wgrant or StevenK, care to talk about bug 820941 for a moment? [09:47] <_mup_> Bug #820941: Distro uploads always email XXXX_derivatives@packages.qa.debian.org < https://launchpad.net/bugs/820941 > [09:47] bigjools: New field on distribution to provide a template? [09:48] yeah that's my first reaction too [09:48] well, since it's not going to change it can be fixed, we just need to decide whether a distro is going to email or not [09:49] Nobody is ever going to want to email anybody except Debian? [09:50] there's currently zero requirement to do that [09:50] and I don't want to fix nonexistent bugs [09:50] We can almost freely remove a hard-coded Debian-specific string. [09:50] it should move to config [09:51] Why? [09:52] It's distribution-specific. [09:52] no, it's Debian specific [09:52] And not everybody wants to send stuff to Debian. [09:52] there is no requirement to do this for any other LP-hosted distro [09:52] They may want to send it elsewhere. [09:52] "may" [09:52] We need either add a bool or a string field. [09:53] but they don't, right now. I am not interested in fixing nonexistent bugs [09:53] If we add a bool, we get to keep the hardcodedness for another 5 years. [09:53] If we add a string field instead, we get an extra line of code and remove the hardcoded string. [09:54] The overhead of making it customisable is probably somewhere around two minutes. [09:54] loving your optimism [09:54] you're usually pessimistic ;)( [09:56] where's lifeless's wiki page on the DB change process? [09:57] At the moment, don't. [09:57] the usual place [09:57] There's https://dev.launchpad.net/PolicyAndProcess/DatabaseSchemaChangesProcess [09:57] But it will probably be at least two weeks before we can do it. [09:57] lifeless: that's not helpful :) [09:57] bigjools: :P I was looking fo rit [09:58] PolicyAndProcess isn't linked from the front page [09:59] lifeless: are we blocked on DB changes until it's implemeted? [10:02] bigjools: fastdowntime? not blocked no. *If* we reach the end of the monthly cycle without being ready to use it, we can manually schedule a 90 minute legacy downtime. [10:02] lifeless: ok thanks [10:12] lifeless: so that announcement about something being the last 90 minute downtime was optimistic? [10:14] jml: slightly [10:14] jml: its certainly the last as-regular-schedule [10:15] jml: I would consider us having to do another as a big deal. [10:15] stub: where do you keep your list of db patch numbers? I need one and I can't remember where it lives. :( [10:15] bigjools: its linked on that wiki page [10:15] lifeless: ok. fair enough. I sort of thought from skimming it that fast db rollouts were a done thing. [10:15] lp:~launchpad/+junk/dbpatches [10:16] but that was a reckless assumption. [10:16] jml: we're in pretty good shape [10:16] stub: ta [10:16] jml: code is written, staging is updating using it [10:21] lifeless: that's pretty good. [10:23] We really have no idea how long it will take on production, though :( [10:26] well [10:26] the first one will be nerve racking [10:26] and will change nothing. [10:26] Yes. [10:26] Stuart is investigating slony 1.2 as a possible major improvement [10:26] Well. [10:26] We *hope* it will change nothing. [10:27] wgrant: -har-, yes. [10:27] It's slony1 v2, not slony 1.2 :) [10:27] blah :P [10:27] (the product is named slony-i or slony1) [10:27] * lifeless dons the not-a-details-job hat [10:28] Heh [10:36] bigjools: I agree with William, make it a string [10:36] My attention span: |---| [10:36] Packaging learning curve: |-------------------------------------------------------------| [10:36] stub: What are you packaging? [10:37] Not Slony-I 2? [10:37] Will need a 2.0.7 package, but basic packaging has been on my todo list for 6 years now... [10:37] That's easy. [10:37] :-p [10:39] Assuming none of the patches need changing, apt-get source slony1-2; cd slony1-2-2.0.6; uupdate /path/to/new/tarball, dch -e and edit the changelog as fit, debuild -S -sa, dput ppa:what/ever ../slony1-2*_source.changes [10:39] Its arcane. Hunger is winning out over this introductory documentation atm. [10:42] StevenK: I am doing that. I just dislike only having one solution discussed. [10:43] bigjools: So, you can either kill it, or make it generic. [10:43] henninge: could I beg for a review? https://code.launchpad.net/~jtv/launchpad/bug-820796/+merge/70701 [10:43] Killing it requires a discussion with distro and Debian [10:43] Making it generic with a string is a my vote. [10:43] jtv: sure, you may beg ;) [10:43] Good, that's step 1. [10:44] "henninge, please, _please_ review my branch for me!" [10:44] That's step 2. [10:44] jtv: I will do that. [10:44] Thanks! That's step 3. [10:44] jtv: welcome [10:44] baby steps? ;-) [10:45] bigjools: There. Two solutions discussed. :-P [10:45] why is this diff taking so long ... Is it long? [10:45] henninge: nope—looks like the differ broke. :( [10:46] It's a very simple branch. [10:46] cool, I'll pull [10:46] branch, I mean [10:46] whatever [10:46] StevenK: you've been taking lessons in diplomacy from your wife [10:48] cjwatson, are you back? [10:49] He is back today [10:56] lifeless: fancy a review of a StormRangeFactory MP? https://code.launchpad.net/~adeuring/launchpad/bug-739052-4/+merge/70717 [11:04] mrevell, hi [11:09] stub: Any luck? [11:09] adeuring: conflicts [11:09] eh? [11:09] adeuring: [11:09] Diff against target: 1338 lines (+879/-25) 21 files modified (has conflicts) [11:09] Text conflict in lib/canonical/launchpad/webapp/batching.py [11:09] stub: With the evils of packaging. [11:09] etc [11:10] stub: db MP winging its way to you [11:10] lifeless: gahhh [11:10] wgrant: your cheat sheet + common sense + google got me as far as a debsign failure. Docs bored me. Still hungry. [11:11] stub: Heh. Probably just your name + email in debian/changelog not matching your OpenPGP identity exactly. [11:11] stub: You can use debsign's -k option to specify a different key ID. [11:12] Yer - need a (Work) in there [11:12] lifeless: devel merged; new revision pushed [11:12] wgrant: But that would involve me knowing what I'm doing and where to edit that :) [11:12] stub: Should be reasonably obvious in debian/changelog. [11:12] Although this is Debian-obvious, not classical obvious. [11:13] wgrant: I meant getting different options passed to debsign [11:13] stub: Should just be able to give it to debuild. [11:13] wgrant: debbuild done now. Now to learn about ppas and uploading [11:14] http://help.launchpad.net/Packaging/PPA/Uploading [11:17] wgrant: At what point to I get to specify 'lucid and above'? [11:17] stub: You have to upload to one, and copy to the rest later. [11:17] Normally one would upload to the earliest, as that is what it will build for. [11:17] Then the binaries and source will be copied to maverick/natty/oneiric. [11:18] AFAICT this should build fine on Lucid, but I guess we're about to find out. [11:18] stub: The upload series is specified on the top line in debian/changelog, in case you hadn't seen that. [11:19] Defaults to 'unstable'. [11:19] Which obviously won't work too well for Ubuntu. [11:19] ta [11:20] wgrant: version number 2.0.7-0~ppa1~lucid1 or 2.0.7-0~ppa1 ? [11:20] stub: That's one of the great unanswered questions of the universe. [11:20] stub: I'd probably use the latter. [11:21] I actually tend to use -0ppa1 [11:21] what I have atm. docs are trying to confuse me :) [11:21] Ahh... what I had earlier before something told me to put in a twiddly [11:23] stub: It depends. [11:23] We often backport stuff, in which case we'll normally use $(original version)~ppa1 [11:23] But -0ppa1 < -0ubuntu1 < -1, so it's cleaner and still works fine. [11:24] its in there now... lets see what lp does with it while I get fud [11:26] And building already. [11:27] jml: can we unsubscribe testtools-dev from the bug tracker ? [11:28] https://bugs.launchpad.net/testtools/+subscriptions [11:28] lifeless: you just want to be on the mailing list? [11:29] jml: I think having the mailing list get a copy of every bug change makes it useless for discussion [11:29] lifeless: oh huh. I'm not getting those mails. [11:30] jml: see the page I linked; you may need to look under 'other subscriptions' or something [11:30] lifeless: I'm not getting those mails. [11:30] jml: thats because they have the same message id [11:31] jml: and you are also subscribed directly [11:31] ah ok. [11:31] jml: and you are using a MUA that dedupes on message id. [11:31] https://lists.launchpad.net/testtools-dev/ [11:32] jml: its also getting merge proposals [11:32] jml: have you, by chance, combined a discussion-group and a permission-granting-team ? [11:32] sigh [11:32] no. [11:32] I uncombined it. [11:33] henninge: any questions about the review so far? I'm hoping to scoot off soon. [11:35] jtv: no, sorry, did not get that far yet. [11:36] henninge: in the meantime, I replaced the core of the Ubuntu distribution infrastructure and it performed its first-ever production run. I think I need to peddle out the tension on my bicycle soon. :-) [11:36] jtv: you mean, you pushed a new revisionß [11:36] jtv: but go ahead and peddle [11:37] s/ß/?/ [11:37] henninge: sort of — Tom rolled out the necessary cron change for me. And then we all sat around worrying until the script ran. [11:38] ah that [11:38] adeuring: reviewed; couple of tiny things [11:38] lifeless: thanks! [11:40] lifeless: >>> def f(a,b): [11:40] ... return a+b [11:40] ... [11:40] >>> reduce (f, 1, 2, 3) [11:40] Traceback (most recent call last): [11:40] File "", line 1, in [11:40] TypeError: reduce expected at most 3 arguments, got 4 [11:41] so, reduce(And, clauses) is right, I think [11:41] adeuring: huh? that seems unrelated ;) [11:41] lifeless: argh, sorry, got your comment wrong [11:42] And is a CompoundExpr [11:42] def __init__(self, *exprs): [11:42] self.exprs = exprs [11:42] right [11:44] jtv: no questions, r=me [11:44] thanks henninge! That's step… 4? [11:45] jtv: wasn't that 5 already? [11:45] Could be, could be. [12:13] stub: So, looks like slony1-2 all went well. === benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge, benji | Critical bugs: 247 - 0:[#######=]:256 [12:13] Need to copy it into natty so I can test with it [12:14] You should see a handy https://launchpad.net/~stub/+archive/launchpad/+copy-packages [12:14] Make sure to select to copy with binaries, and to the right archive. [12:14] Not rebuild? [12:15]