[00:01] wgrant: I've finally found the method that generates the query for bug 1001838 [00:01] <_mup_> Bug #1001838: Builder:+history timeouts due to terrible main query < https://launchpad.net/bugs/1001838 > [00:02] Would be nice if I could actually find a relevant OOPS that hasn't been pruned. [00:09] awesome, after upgrading to quantal I now have two X servers running [00:11] bigjools: What's the other one doing? [00:11] StevenK: Should be pretty easily reproducible on prod [00:11] StevenK: eg. try palmer's +history [00:11] Any of the busier builders should work [00:12] wgrant: one is gdm one is kdm ... [00:13] gdm or lightdm? [00:13] also: [00:13] The former would be pretty strange :) [00:13] $ dpkg -l [00:13] dpkg-query: error: parsing file '/var/lib/dpkg/status' near line 56897 package 'libao4:i386': [00:13] mixed non-coinstallable and coinstallable package instances present [00:13] Uhoh [00:14] yeah [00:35] Ah [00:35] Found the bugsummary bug [00:39] wgrant: Heh, except is palmer is gone [00:40] roseapple works [00:40] palmer's still accessible [00:40] Just probably not linked [00:41] hooker works too [00:41] Right, palmer is cursed [00:42] wgrant: Ah, so there are two real cases in the code -- your explain and index is for the case where user is None. [00:42] Indeed. [00:42] wgrant: And what's the bugsummary bug? [00:43] A typo in bug_summary_dec [00:43] - AND access_policy IS NOT DISTINCT FROM access_policy; [00:43] + AND access_policy IS NOT DISTINCT FROM $1.access_policy; [00:43] is the fix [00:43] Heh [00:43] See, this is why we should move to 9.2 [00:43] SQL functions can now use named args [00:43] Which would have made that more obvious :) [00:44] lifeless: So I expect us to be on 9.2 by this evening [01:02] lifeless: https://code.launchpad.net/~wgrant/launchpad/bug-1046713/+merge/123668 [01:02] Or leave it for stub if you so desire [01:13] wgrant: what you expect and what you get may differ [01:14] bigjools: around ? [01:14] lifeless: otp [01:14] bigjools: need info about where in BNE to stay [01:14] bigjools: so that I don't end up with an hour commute or something [01:14] wallyworld can probably help too [01:14] yeah he's better for that [01:15] lifeless: where are you meeting? [01:15] we dunno yet [01:15] there's a few good irish pubs inthe city [01:15] I need to book accom via btstravel [01:15] so this is going to be tight if I don't put the request in today. [01:16] if you can tell me where the meetings are, i can help with somewhere close by [01:16] I have a spare room if all else fails :) [01:16] (bring earplugs then) [01:16] haha [01:16] and i don't mean for the babies [01:17] comedian [01:17] i'm here till friday [01:21] boom tish [01:21] I thought it was Ng that was. [01:21] Hah [01:26] lifeless: :( [01:59] implicit type promotion sucks. [01:59] that is all [02:10] can has review plox : https://code.launchpad.net/~lifeless/python-oops-tools/bug-1048470/+merge/123672 [02:10] will unbreak neem [02:15] wgrant: StevenK: ^ [02:36] lifeless: python3 :) [02:36] wgrant: that would be nice [02:37] Anyway, r=me [02:37] Thanks [02:37] wgrant: but I'm missing 3 unicorns and a couple of fairy godmothers to make that happen [02:37] Well [02:37] Django's mostly ported now [02:40] wgrant: StevenK: should a distribution source package have an +expirable-bugs view? [02:40] bigjools: travel booked, but now we need to sort out accom :) [02:40] bigjools: [02:42] wallyworld: Doesn't really matter. [02:42] wallyworld: If it's easy then fix it, otherwise remove it [02:43] wgrant: well, _getTargetJoinAndClause only wants distro, product, distroseries, productseries. if i can convert a dsp to any of those, then it will work? [02:44] BugTaskSet._getTargetJoinAndClause [02:44] Ah [02:44] So it's only used by findExpirableBugTasks [02:45] I'd just add support for IDistributionSourcePackage and ISourcePackage in there [02:45] shouldn't be too difficult [02:46] ok, thanks. just wanted to check that it made sense to do that [02:46] It doesn't not make sense [02:46] But it might not be too useful [02:46] So if it looks difficult, don't bother [02:47] ok. i have no idea how often that view might be used, or if you need to url hack to see it [02:47] Not very, and probably [02:49] hmm. in that case i might leave it. no point fixing something not useful [03:48] bigjools: did we need to speak arch ? [04:42] StevenK: Have you tested COUNT(*) with that new query? [04:42] It will probably be dreadfully slow [04:42] ... You say after I close the gedit that had my notes in it [04:43] Oh, and after I cleaned up the index on DF [04:46] wgrant: Yeah, 8.5 seconds [05:00] StevenK: Right, this is why StormRangeFactory is important [05:03] I'm not sure how difficult the port will be [05:03] Probably not very [05:06] this is also something we could denorm [05:06] avoid the count(*) at all [05:06] wgrant: Based on my reading, it looks like BuilderHistoryView uses it already? [05:08] StevenK: Where? [05:09] self.batchnav = BatchNavigator( [05:09] builds, self.request, range_factory=self.range_factory(builds)) [05:09] Right [05:09] It uses a range factory [05:09] Probably not StormRangeFactory [05:09] range_factory = ListRangeFactory [05:09] It's likely that a subclass or two override it [05:10] Can we just tell BuilderHistoryView to override to SRF and move on? [05:10] Right, probably [05:10] Try :) [05:17] wgrant: TestBuilderHistoryView doesn't explode [05:17] See if it works in the UI :) [05:18] Cowboy my changes onto DF, or 'make run' see if it works? [05:19] Try it locally [05:19] Confirm that no COUNT(*) is executed [05:20] steven@undermined:~/launchpad/lp-branches/buildfarmjob-index% grep BuildFarmJob /var/log/postgresql/postgresql-9.1-main.log | grep -c COUNT [05:20] 0 [05:20] And yes, I have query logging on [05:20] :-) [05:21] Oh drat, I didn't use cat. [05:21] * StevenK peers at spm. [05:23] wgrant: ^ Does that look good to you, or shall I pastebin the first grep? [05:24] StevenK: I'd confirm in ++oops++ [05:25] Or the query log at the bottom of the page [05:29] * StevenK tries to rememeber where OOPSes go [05:30] Nowhere, if rabbitmq is running [05:30] So I'd use the thing at the bottom of the page [05:31] * StevenK turns on the FF for it [05:32] wgrant: No match for 'count(' [05:32] Lots of matches for count, but that's things like 'Account' and 'failure_count' [05:34] StevenK: And it shows up if you revert the range_factory change? [05:35] 1.099ms [05:35] SQL-main-slave: SELECT COUNT(*) FROM BuildFarmJob LEFT JOIN PackageBuild ON PackageBuild.build_farm_job = BuildFarmJob.id WHERE BuildFarmJob.builder = 1 [05:35] Excellent [05:35] Now I'd cowboy this onto DF and watch the fireworks [05:36] But it's hopefully fine [05:36] * StevenK recreates the index again. [05:45] wgrant: DF cowboyed [05:47] wgrant: "31 queries/external actions issues in 0.93 seconds" [05:48] Means we can probably kill the Builder:+history FF too [05:48] That's the plan [05:49] wgrant: https://code.launchpad.net/~stevenk/launchpad/builder-history-better-query/+merge/123676 [05:50] StevenK: Does anything else use the modified method? [05:50] eg. the API? [05:52] wgrant: There's one other callsite, and it isn't exported [05:52] And the method itself isn't either [05:52] What is the callsite? [05:52] IBuilder.getBuildRecords() [05:53] Isn't that the method that we're looking for callsites for? [05:53] No, the method I've been beating into submission is IBuildFormJobSet.getBuildsForBuilder [05:53] IBuildFarmJobSet, even [05:54] Right, but isn't that only used by getBuildRecords? [05:54] The view should call getBuildRecords [05:54] Nothing else should call getBuildsForBuilder directly, AFAIK [05:56] Right, doesn't look like anything else wants IBuilder.getBuildRecords() [05:56] Pretty much everything is interested in IDistroSeries.getBuildRecords() [05:59] wgrant: So, are you happy with everything, and I can get stuff rolling? [06:03] StevenK: Indeed, r=me with one comment [06:04] wgrant: I was going to rename extra_clauses to clauses to save a bit too [06:04] wgrant: http://pastebin.ubuntu.com/1198057/ [06:05] stub: https://code.launchpad.net/~stevenk/launchpad/buildfarmjob-index/+merge/123675 would love a review. [06:06] StevenK: Do we really need id in there? [06:07] Ooh [06:07] wgrant wrote that index, I've just been using it. [06:07] I think I just won my battle against the planner [06:07] That was months ago [06:07] It looks like a sacrifice to the test suite [06:07] It probably has id as a tiebreaker [06:07] I don't remember [06:08] Can a builder run multiple jobs in one build? [06:08] So we need the tie breaker in the real world? [06:08] probably not, but it won't really hurt [06:08] TBH, I'm inclined to leave it, I've been testing against this index all afternoon [06:09] StevenK: Do the queries have id in the order by at the moment? [06:09] And it makes Builder:+history performant for the first time in $A_LONG_TIME [06:10] it is an extra integer column in an index on 4million odd items, makes the index slower [06:10] They do and they will. [06:10] ok. [06:11] wgrant: Happy with the pastebin'd diff I posted? [06:11] Both builder and date_finished are nullable. [06:11] Should this index be partial? [06:12] stub: builder will be NULL when a job is waiting in the queue, and date_finished will be NULL until the job completes. [06:13] The grand majority of rows will have them both set. [06:13] Right. Just looking at prod, 300k rows have a NULL builder [06:13] So thinking a WHERE builder IS NOT NULL clause might be appropriate [06:13] Probably mostly the imported binaries [06:13] I would indeed go with WHERE builder IS NOT NULL [06:13] But I think date_finished IS NOT NULL would break things [06:14] Since we don't have that constraint in the query today [06:14] Yup [06:14] And it's conceivable that a builder has something without a date_finished, though that should only happen if something is terribly broken [06:14] If it's currently building [06:14] The reverse is the weird one [06:15] date_finished but no builder, but that would be if we removed builders from the db and wanted to keep historical records. [06:16] Possibly [06:16] Does it exist on prod today? [06:16] -CREATE INDEX buildfarmjob__builder__date_finished__id__idx ON BuildFarmJob(builder, date_finished DESC, id); [06:16] +CREATE INDEX buildfarmjob__builder__date_finished__id__idx ON BuildFarmJob(builder, date_finished DESC, id) WHERE builder IS NOT NULL; [06:16] stub: ^ [06:16] StevenK: yer [06:17] wgrant: 5081 [06:20] stub: The diff on the MP has updated if you want to rubber stamp [06:49] stub: I've also got a DB review for you. A nice complex DB function change: https://code.launchpad.net/~wgrant/launchpad/bug-1046713/+merge/123668 [06:49] stub: No +1 for me? :-( [06:49] Heh [06:54] eh? oh [06:55] StevenK: +1 [06:58] stub: Thanks, that's landed as r15931. You can apply it to qas/prod when you're able. [07:00] wgrant: Dare I ask where the typo was? [07:01] stub: Check the diff I linked in the description [07:01] oh, ta [07:03] Thanks [07:03] It's been through ec2 already [07:08] wgrant: Happy for me to apply that with StevenK's index? [07:08] stub: That would be lovely [07:08] Then I'll rebuild bugsummary once revisionkarma has been killed off and the backup is done. [07:11] wgrant, StevenK: Both applied to qastaging [07:13] stub: Prod is blocked due to backups? [07:13] I'm just checking [07:13] Yeah, another 4 hours or so until it can be applied [07:14] That's a little worrying, given the FDT window is in 3 hours. [07:14] Well, one of them. :-) [07:14] wgrant: Your patch is applied to prod [07:15] Ah, functions aren't blocked by the backups, but indicies are? [07:15] stub: Thanks [07:15] StevenK: Correct [07:15] Well, that's just terrible. [07:15] I shall have to deal. [07:15] But it makes sense :) [07:16] (the backup and revisionkarma should both finish in 2h±10min [07:16] We are going to keep having 10.5 hour logical dumps until we get the diskspace to store binary backups, so consider this next window pointless. [07:16] Nah, they finish in time [07:16] It's the midday AEST window that is unusable unless the backup fails [07:17] * StevenK goes to kill Geth [07:17] Well, some Geth [07:17] I've got a 6.5 hour backup transaction in process, and I could have sworn the last successful backup took 10.5 hours [07:18] stub: The backups run from 00:41 to somewhere between 09:15 and 09:20, usually [07:18] (then takes 20-30 minutes to copy to sourcherry, but by then it's out of the DB) === almaisan-away is now known as al-maisan [07:56] good morning === frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban | Firefighting: - | Critical bugs: ∞ [08:30] hmm [08:30] we need to do some releases [08:30] moring [08:30] *morning even [08:30] https://bugs.launchpad.net/launchpad-project/?field.status%3Alist=FIXCOMMITTED [08:31] hi czajkowski [08:31] hiya === rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban, rick_h | Firefighting: - | Critical bugs: ∞ === al-maisan is now known as almaisan-away [12:10] sinzui: beat you on this one today :P [12:10] okay [12:11] last week you beat me through one for jcsackett so need to get my revenge in [12:15] rick_h_: Caffeine overload there? :P [12:16] nigelb: working on it [12:17] Heh [12:17] hey jtv [12:22] I tried my hand at scratching an LP itch last night. bug 891862. any one care to have a quick look at it and discuss it with me? [12:22] <_mup_> Bug #891862: Changing the owner of a packageset requires SQL

< https://launchpad.net/bugs/891862 > [12:34] tumbleweed: You don't seem to have any tests [12:35] I am aware of that [12:35] there aren't any tests for any of that code, as it stands [12:35] and writing some is going to put me into a big positive LoC problem :/ [12:36] lib/lp/soyuz/tests/test_packageset.py [12:36] Yeah, I've tried that excuse in the past, it never works :) [12:37] oh, does that cover the API? [12:37] Oh and lib/lp/soyuz/stories/webservice/xx-packageset.txt [12:37] For the API [12:37] aah, doctests [12:37] doctest unfortunately, but it's workable if you hold your nose [12:37] that's why I couldn't find them [12:38] Also destroySelf needs a model test [12:38] yeah [12:38] (not totally sure I like that name but I'll leave it to a reviewer ...) [12:38] and I would say you should be exporting the new method on API version devel, not beta [12:40] destroySelf seems to be the name everywhere else [12:40] duh, re devel [12:41] do you know of anything that can some me some LoC points? [12:46] tumbleweed: https://dev.launchpad.net/PolicyAndProcess/MaintenanceCosts/HitList is the running list, I've not checked it out lately [12:49] * tumbleweed supposes converting those doctests to unittests may be a starting point [12:57] that's usually a worthwhile place to start although it can take a while === almaisan-away is now known as al-maisan [13:32] adeuring, ping for standup. [13:32] deryck: oopps, coming... [13:32] adeuring, np. === skaet_ is now known as skaet [13:54] deryck: abentley think I found it. It had a new information type that wasn't there before so that failed the verbose prettyprint match [13:54] thanks for peeking [13:54] rick_h_: np. === al-maisan is now known as almaisan-away [13:56] rick_h_, cool, glad to hear. === almaisan-away is now known as al-maisan [14:02] rick_h_: See, that's the thing. Doctests should show you where the match failed, not what the differences are. [14:04] abentley: right, blind by too much info [14:04] just glad it wasn't anything huge. Hate it when you change something and then things seemingly unrelated go boom [14:24] jam I appear to have enough network now to have a proper talk about maintenance [14:25] jam: I am free today, Thursday, and Friday [14:28] sinzui: love the new chart!! [14:28] thank you [14:29] deryck: I've added a card for "Add Add Product.getAllowedSpecificationInformationTypes and Product.getDefaultBugInformationType" [14:29] abentley, thanks! === al-maisan is now known as almaisan-away [15:33] rick_h_: could you pelase review this MP: https://code.launchpad.net/~adeuring/launchpad/use-access-grants-for-specifications-auth-check/+merge/123774 ? [15:33] sure thing adeuring [15:33] thanks! [15:44] adeuring: should an XXX or something be setup for the follow up branch work? [15:44] adeuring: I guess how are we tracking that there's follow up for the owner not getting a grant atm [15:45] cjwatson: added a few tests and started finding bugs... Do you think we should try and re-link packageset heirarchies across removal? or just abort the deletion if there are relationships? [15:45] rick_h_: I think I should add a card about the fact that several people with "special roles" do not have automatic access [15:45] adeuring: ok, cool [15:45] adeuring: r=me [15:45] rick_h_ thanks! [15:46] tumbleweed: err not sure :-) [15:47] or even cascade the deletion... [15:50] and a more general question: if I expose an API method for deletion. How do I not return a 404 on success? === deryck is now known as deryck[lunch] === frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: rick_h | Firefighting: - | Critical bugs: ∞ === deryck[lunch] is now known as deryck === matsubara is now known as matsubara-lunch [18:18] jcsackett: have time for a JS review? https://code.launchpad.net/~rharding/launchpad/productjs/+merge/123803 [18:19] rick_h_: sure. [18:19] jcsackett: thanks, more a start than a finish, but want to do it in stages [18:19] dig. [18:22] jam: I'm afraid I have to punt the lpsetup tarmac issue back to you guys. I wasn't able to figure much out yesterday. When rebooting the canonistac instance it did not come back up. [18:23] benji: technically it is up to purple, since they are now on maintenance, but since I'm the one with the delayed patch, I'll give it a look tomorrow. [18:23] I was able to log into the machine [18:23] so that worked [18:24] jam: oh, sorry, I thought you were on maint. [18:24] we just switched on Monday [18:24] so not too surprising you would think so [18:52] sinzui: free to chat? [18:52] sinzui: ignore that, posted from irssi history. === rick_h_ changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: ∞ [19:10] rick_h_: some comments on MP. [19:10] jcsackett: ty much looking [19:12] rick_h_: also, given the branch of mine just landed that you reviewed, lazr.* > lp.ui.* [19:12] jcsackett: yea, expected that to come along, just waiting on the merge [19:16] jcsackett: comments added [19:21] * jcsackett looks [19:23] rick_h_: thanks for answers, that all makes sense. r=me. [19:23] I've always wondered, how'd LP have a lazr namespace? [19:23] i.e. where'd it come from? [19:27] nigelb: so I believe the history was that lazr was supposed to be a set of reusable UI components [19:27] but it's really only used in LP [19:27] ah [19:28] the idea was that it could be used in other places in canonical? [19:28] right [19:28] nigelb: Actually, I think the hope was that it would be useful outside Canonical, too. [19:28] which is again my pipe dream, but we're just subsuming/taking over lazr into LP [19:28] ah === Ursinha_ is now known as Ursinha [19:40] rick_h_: I think its a good dream. I think we tried to run before we could walk. Lots of effort in planning for things that haven't eventuated, too little on solving the friction around consuming widgets from the store etc. [19:41] lifeless: yea, definitely. Not saying it's more than a dream to keep in mind as we do things [19:41] in my head at least :) [19:41] and it'll be easier with later YUI with Views [19:44] how do I make this API delete() function not return a 404 on success? http://bazaar.launchpad.net/~stefanor/launchpad/edit-packagesets/view/head:/lib/lp/soyuz/interfaces/packageset.py#L351 === matsubara-lunch is now known as matsubara [20:21] Is there a way to get what the link url would be for a resource without actually fetching it [20:21] I want to get the resource URL for a person [20:22] I want to pass this to another API call manually instead of the person object [20:22] as I might not be able to fetch the person object myself [20:22] I could easily do a hack but am wondering if there is a clean way to do this [20:27] sinzui, ^^ [20:28] Also, it doens't appear that the objects returned by getSharedArtifacts get transformed into the right launchpadlib objects [20:28] I manually construct them sometimes because I know the API is stable and cannot change [20:29] If you know the type, you can trust the URL will be the same given that you know the unique names that construct it [20:29] oh dear [20:29] cody-somerville, you are not getting IBranch and IBug? [20:33] Doens't appear so. In fact, for one example I have a person that has one artifact shared with them. What the API returns is a list that contains two lists. The first sub-list contain a dictionary for the single bug shared. The second sub-list is empty. [20:33] sinzui, The dictionary does contain a resource_type_link key. The value is https://api.launchpad.net/devel/#bug_task as expected. [20:34] ah [20:34] This probably relates to what stopped me from exporting getAffiliatedPillar() [20:34] python-lazr.restfulclient: 0.11.1-1build1; python-launchpadlib: 1.9.7-0ubuntu2.1 [20:35] sinzui, ^^ if that matters [20:35] It is not easy to return a heterogeneous set of objects [20:36] versions do not matter. The issue is Lp trying to adapt two types into one return type [20:36] report a bug. [20:37] I will discuss the issue the the squad. I think we have two cases that demonstrate that we need a solution that can be reused. [20:37] Will do. [20:38] sinzui, so the two lists within the lists. Is the second list intended to have branches? [20:38] yes [20:40] ah, I wonder if the rule is real method returns a tuple and Lp restful knows how to expand the tuple to the real types. That would allow me to send back distros, project groups, and projects from a singe method, and informs me of how to rewrite the query === gary_poster is now known as gary_poster|away [20:47] sinzui, FYI, I imagine one of the primary use cases will be to display same information that the web UI does on command line. Since the current implementation returns a bug_task instead of a bug, you have to fetch each individual bug object to get the information_type. [20:53] cody-somerville, agreed [20:54] The bug-bugtask division does not help anyone [20:54] Even Lp has to use both every time [20:54] Pillar and PillarName are examples of intermediary object that no callsite ever want tot get back [21:32] sinzui, I filed LP #1049374 [21:32] <_mup_> Bug #1049374: Resources returned by getSharedArtifacts not adapted to resource object type < https://launchpad.net/bugs/1049374 > [21:32] thanks [21:48] whats involved in doing a launchpad-buildd release ? [22:25] wgrant: OOPS-995da926c1f98feed6844957a3a94559 [23:00] sinzui, wgrant, wallyworld: http://www.somethingofthatilk.com/index.php?id=135 [23:50] wgrant: https://code.launchpad.net/~stevenk/launchpad/illegaltarget-if-pillar-is-new/+merge/123862 [23:52] StevenK: Can you rework the test to not use rSP? [23:52] StevenK: eg. set the bug to USERDATA, then transition the sharing policy afterwards [23:54] wgrant: I think it will still be needed for the validate_target call [23:54] Not if you log in [23:55] Wait a minute, that test is invalid [23:55] The sharing policy is PROPRIETARY_OR_PUBLIC [23:55] So it permits USERDATA [23:55] Bleh [23:56] This diff should fix that too