/srv/irclogs.ubuntu.com/2012/11/29/#launchpad-dev.txt

StevenKwgrant: r16321 is only a testfix or a testfix and a rollback?00:33
CyberJacobHi00:43
wgrantStevenK: Just a testfix00:55
wgrantsinzui: Hm, this same MP diff technique could/should probably be applied to the front page blog feed relatively easily, I guess00:58
StevenKreference = u'unique-from-factory-py-line3339-102754 binarypackage-102763'01:09
StevenKactual    = u'binarypackage-102763'01:09
StevenKSuccess01:09
* StevenK peers at the garbo query of doom01:28
CyberJacobHi Guys01:35
CyberJacobI'm trying to look up a user, but I keep getting a ComponentLookupError01:36
CyberJacobany ideas?01:36
StevenKWhat's the traceback?01:36
CyberJacobhttp://pastebin.ubuntu.com/1395890/01:36
wgrantCyberJacob: Where are you running that?01:37
wgrantLooks like bin/py, when you might want bin/harness01:38
wgrantbin/py just starts a Python interpreter, without setting up any of the infrastructure01:38
CyberJacoboh, that might be the problem then01:38
CyberJacobis there a binary for the infrastructure?01:38
StevenKbin/harness01:38
StevenKOr 'make harness'01:39
StevenKwgrant: I've added the joins for BPB, SPR and SPN, but I can't work out how to get it into the output of the SELECT array_agg01:40
wgrantHowso?01:40
StevenKwgrant: Everywhere I try and put sourcepackagename.name, postgres complains01:41
wgrantComplains?01:41
StevenKHINT:  No aggregate function matches the given name and argument types. Perhaps you misplaced ORDER BY; ORDER BY must appear after all regular arguments of the aggregate.01:42
wgrantWhat's your query?01:42
CyberJacobStephenK: works perfectly with bin/harness, thanks01:42
StevenKIt's a V, damn it!01:43
StevenKwgrant: http://pastebin.ubuntu.com/1395899/ is what I'm testing on DF with01:43
StevenKWith some variants01:43
wgrantStevenK: What does that mean?01:45
wgrantYou're trying to create an array of rows?01:45
StevenKwgrant: I'm trying to get 4749089 | debhelper hello-debhelper-df01:45
wgrantStevenK: Right, but you seem to be trying to create an array of [(spn, bpn)] there, which is a bit odd01:47
StevenKwgrant: I've been trying various things, that was the latest attempt01:47
wgrantStevenK: I'd just do a separate join to get the SPN01:47
wgrantBring it out in a separate array01:48
wgrantAs we do with the source name01:48
StevenKRight01:52
StevenKThat works nicely01:52
bigjoolsany regex experts want to do a maas pre-imp with me?01:59
wgrantA pre-imp for a regex... this does not bode well :)02:00
wgrantI'm about to have lunch but could talk after02:00
bigjoolswell it's more about how to deal with some postinst processing on a config02:01
StevenKwgrant: Right, that's the garbo and IPackageUpload.addBuild working with adding the SPR02:01
StevenKbigjools: In a word, "don't"02:01
bigjoolsheh02:01
bigjoolsI *know(02:01
StevenKIf you can possibly get away with a .d directory, do that02:01
bigjoolsbut we are stuck with it02:01
wgrantbigjools: What are you trying to do?02:02
StevenKIf it isn't your config file, it's against policy, too ...02:02
bigjoolsfixing someone else's crap^Wwork02:02
bigjoolsa dpkg reconfigure is supposed to take an input value and insert it in various configs02:02
StevenKwgrant: My vague plan was to have the garbo job done and out of the tree by Friday, too ...02:03
bigjoolsthis particular config may or may not have a commented line that we need to update02:03
bigjoolsit may also have a commented and a non commented line02:03
wgrantStevenK: I generally suggest getting all the work done before landing any of it (other than an obviously safe DB patch)02:04
wgrantIt's far easier to fix the garbo job than it is to fix several million rows02:05
CyberJacobIs there a way of starting the Salesforce (voucher) server locally?02:07
CyberJacobor is that a non-GPL part of launchpad?02:08
wgrantCyberJacob: It's not part of Launchpad at all. I've never seen the source, and I'm not sure how much of it we actually have (it's just a thin wrapper around SalesForce itself, AIUI)02:11
CyberJacobwhat about forcing `make run` to use the test proxy transport02:12
CyberJacobthat should get it talking properly02:12
wgrantWhat are you trying to do?02:12
wgrantIf you just want to extend the complimentary commercial subscription, we use an API script like http://bazaar.launchpad.net/~canonical-launchpad-branches/lp-dev-utils/trunk/view/head:/extend_subscription.py02:13
wgrantThere's little point going through the voucher system unless you need to02:13
CyberJacobthat's basically what I was looking for02:14
CyberJacob(I didn't realise SalesForce was a full piece of software, just looked like another module to me)02:14
wgrantThe SalesForce referred to here is the service provided by salesforce.com02:15
CyberJacobwhich explains why there's just an RPCXML connecter in LaunchPad02:15
wgrantYep02:16
CyberJacobwgrant: I'm getting a 404 error with that script...02:59
CyberJacobLooks like it's trying to load NotFound: Object: <lp.services.webapp.publisher.RootObject object at 0x41d7d50>, name: u'devel'03:00
CyberJacobany ideas?03:00
StevenKwgrant: So the garbo job and the population now sets source name as well, and I've changed IPackageUploadSet.getAll() to use searchable_names in a different branch. That's all of it, isn't it?03:01
wgrantCyberJacob: Have you told it to connect to the right URL?03:02
wgrantStevenK: How are you handling version?03:02
CyberJacobwgrant: I'm using ./extend_subscription.py -s "https://launchpad.dev/" -d 90 caveman03:02
wgrantCyberJacob: api.launchpad.dev, not just launchpad.dev03:02
StevenKwgrant: So far, I'm not.03:02
wgrantStevenK: You'd best03:02
wgrantCyberJacob: (also, for internal projects we normally just force it to extend for 10 years or so)03:03
CyberJacobwgrant: ah, perfect!03:03
StevenK40-1 turns into ADD COLUMN searchable_versions ?03:03
wgrantStevenK: It depends on whether there's a performance issue or not03:03
StevenKwgrant: With the existing queries?03:04
wgrantStevenK: With name denormed, and version not03:05
StevenKYeah03:05
wgrantOptimise things that need optimising03:05
wgrantDon't optimise things that don't :)03:05
StevenKThe garbo job on DF has 740k rows to go, but will need to start again03:06
StevenKAnyway, I'll add the trigram, and test a few queries03:06
StevenKCome on, DF, it's an index, not solving pi to 1 trillion places.03:12
wgrantIt'll take a while to build03:12
wgrantThey're longish string fields, and trigram indices aren't exactly efficient03:13
StevenKIt's been at it for 5 minutes already03:13
wgrantAnd there's millions of rows03:13
wgrantThis will be one of our largest non-FTI indices03:13
StevenKCREATE INDEX03:17
StevenKTime: 621691.915 ms03:17
StevenKwgrant: How can I check how large it is?03:17
wgrantIt's 656MB03:18
wgrantWhich places it just between the bug and bugtaskflat FTI indices, IIRC03:19
wgrantThe only indices likely to be bigger than those are message FTI, and branchrevision + *downloadcount indices03:19
wgrant:)03:19
StevenKSELECT max(char_length(searchable_names)) FROM packageupload; => 1190803:22
wgrantOuch03:23
wgrantI wonder if that's langpacks on a delayed copy03:23
StevenKI'm just trying to figure out how to check that03:24
StevenKNot quite03:26
StevenK... libghc6-vte-dev libghc6-vte-doc libghc6-vty-dev libghc6-vty-doc libghc6-vty-prof ...03:26
wgrantAh03:26
* StevenK looks for a good victim03:28
StevenKActually, that Haskell one looks good03:29
StevenKJust searching for a version is pathetically slow03:39
StevenKwgrant: The way this query is going, it is clearly searching all BPR and SPR rows03:42
wgrantWell, what does EXPLAIN say?03:43
StevenKIt hasn't finished yet03:43
wgrantwithout ANALYZE03:43
StevenK                     ->  Seq Scan on sourcepackagerelease  (cost=0.00..321829.18 rows=1627518 width=25)03:44
StevenK                                 ->  Seq Scan on packageuploadbuild  (cost=0.00..49832.41 rows=3137541 width=8)03:44
wgrantThat's less than ideal03:44
StevenK                     ->  Seq Scan on binarypackagerelease  (cost=0.00..1618381.64 rows=11488564 width=28)03:44
lifelessnice03:44
StevenKLet's look over what, 17 million rows?03:45
bigjoolssigh, how can linking a branch to a blueprint time out03:45
wgrantbigjools: When the branch row is locked by the scanner, usually03:45
bigjoolsit's an old landed branch03:45
wgrantWhat does the OOPS show?03:45
lifelessur face?03:46
lifeless</troll>03:46
bigjoolshaha03:46
bigjoolshttps://oops.canonical.com/oops/?oopsid=OOPS-cf507af414fb03b447b2658c8c964a7203:46
StevenKlifeless: Fail.03:46
StevenKlifeless: This is you, you never close that tag03:46
lifelessStevenK: closing it is itself a troll03:47
lifelessStevenK: thus YHBT03:47
wgrantbigjools: Ah, you probably typoed the branch name.03:47
wgrantYes03:47
wgrantYou gave the URL, not the name, so it did a search03:47
bigjoolsit has SPACES on the end03:47
wgrantand the search algorithm is insane03:47
bigjoolsthat is all03:47
bigjoolsWTAF03:47
wgrantBug #79383003:48
_mup_Bug #793830: Branch:+register-merge / Specification:+linkbranch time out due to substring matching many tables <critical-analysis> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793830 >03:48
bigjoolsyou gotta love the optimism in the code around user input03:48
wgrantWell03:48
wgrantIt'd be all good if the search mechanism wasn't completely stupid03:49
StevenKbigjools: I'm in the middle of a fixing a critical around IPackageUploadSet.getAll(). That code is just as good03:49
bigjoolstrailing spaces being very stupid03:49
StevenKLet's JOIN against SPR, SPN, BPB, PCJ and LFA and do a substring match03:49
StevenKOh, I'm forgetting BPR and BPN03:50
StevenKwgrant: So, I guess denorming version is a go?03:51
StevenKwgrant: Oh, and that trigram index isn't even fully populated03:52
wgrantStevenK: And you forgot PUS, PUB and PUC03:58
wgrantBut anyway03:58
wgrantStevenK: Right, denorming version would make sense here, but we probably don't care about substring matches on it03:58
wgrantWe support it today, but that's fairly insane and probably not useful03:58
StevenKI have a garbo query, too03:59
StevenKwgrant: http://pastebin.ubuntu.com/1396146/03:59
wgrantRight, but there's not much reason to use a string here04:00
wgrantWe don't care about substring matching04:00
StevenKWhat do you suggest?04:00
wgrantA plain array of strings with a GIN index04:00
wgrantMuch faster04:00
StevenKHmmm, don't know how to craft that one04:01
wgrantJust don't turn the array into a string04:01
StevenK 2873221 | {NULL,1:4}04:02
StevenKGiven we don't have mixed uploads, I don't think an array is useful04:02
wgrantBinaries can have different versions04:03
StevenKFor the same upload, for the same source, from the same BPB?04:05
wgrantStevenK: Yes04:09
wgrantIt's not hugely common, but some fairly significant packages like gcc do it04:09
StevenK 3857156 | {NULL,4.4.6-15ubuntu1,1:4.4.6-15ubuntu1}04:12
StevenKWell, damn04:12
StevenKwgrant: ADD COLUMN searchable_versions TEXT[]; ?04:19
wgrantStevenK: It's technically a debversion, but that's probably unimportant and possibly detrimental here04:20
StevenKRight04:20
StevenKI'm not sure I care enough to test both04:21
wgrantI should really try some multi-column GIN btf indices at some point.04:26
wgrantHm04:37
wgrantIt actually works pretty well04:37
wgrantPotentially useful for translations, too04:40
* wgrant plots04:40
StevenKwgrant: How can I declare searchable_versions as an array in the model code so I can avoid SQL('') everywhere in the garbo job?05:12
wgrantStevenK: A List, probably05:14
=== shadeslayer_ is now known as shadeslayer
=== Ursinha_ is now known as Ursinha
stubStevenK: Today's search column is an array, yesterday's wasn't. Is that a foot gun for one or t'other?05:53
StevenKNeither05:55
wgrant"TODO: Test IE compatibility. StuartBishop 20041118"06:15
stubhaha06:39
stubDoes it work in Ireland now?06:39
wgrantHeh06:39
czajkowskimorning folks07:35
adeuringgood morning08:41
StevenKstub: So, no, it isn't a fail. searchable_names is a string that will be substring matched via a GIN index. searchable_versions is an array that will not be substring matched, so it made sense to store it as an array08:45
stubStevenK: How does a package upload end up with multiple versions btw?09:07
StevenKstub: Some packages do it, like gcc.09:33
StevenKIt's rare, but does happen09:34
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== mgz_ is now known as mgz
=== dpm_ is now known as dpm
=== yofel_ is now known as yofel
deryckMorning, all.13:59
rick_h_morning14:00
cjwatsonstub: Binary package generation is controlled by the source package's debian/rules; by default it produces binaries with versions matching the source, but it's occasionally useful to use the -v option to dpkg-gencontrol (or its wrappers) to change that14:09
stubIt sounds very quantum.14:13
cjwatsonEntirely deterministic :)14:16
czajkowskideryck: hope you're feeling better14:22
deryckczajkowski, thanks! I am. Little worn down by being sick yesterday, but overall, much better.14:22
czajkowski:(14:22
czajkowskipleanty of tea!14:23
czajkowskihot tea now, none of that ice cold stuff14:23
stubAre Debian version numbers Turing complete?14:26
cjwatsonNo. :-P14:28
cjwatsonWell, except that they can be arbitrarily long so I suppose you could encode any program in them.14:29
cjwatsonSame as you could encode any program in YOUR MUM.14:29
* cjwatson runs14:30
stubI don't think my mum is Turing complete14:30
=== jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: jcsackett | Firefighting: - | Critical bugs: ~170
rick_h_abentley: pushed updated changes with tests to my MP. Did you want to go over it or should I ask jcsackett? https://code.launchpad.net/~rharding/launchpad/nonpublic_1052659/+merge/13647915:41
abentleyrick_h_: I'll look.15:41
rick_h_abentley: ty much15:41
cjwatsonhttp://www.rogerharford.com/content/img/blog-images/database-design-breakdown.png15:41
abentleyrick_h_: Looks great.  Thanks for making all those changes.15:44
abentleyrick_h_: r=me.15:45
rick_h_abentley: thanks for the review and the push to 'do the right thing' by adding in the tests15:45
rick_h_sometimes need a nudge :)15:45
abentleyrick_h_: np.15:45
=== matsubara is now known as matsubara-lunch
czajkowskirvba: is jtv on today ?16:17
abentleyrick_h_: What's the status of your rollback for _getTranslatables ?16:42
rick_h_abentley: done and landed16:43
rick_h_deployed as part of yesterday16:43
abentleyrick_h_: Shouldn't test_getTranslatables_filters_private_products no longer exist?  I still see that in stable and devel.16:45
sinzuijcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/fast-contact-via-web/+merge/13698616:45
rick_h_abentley: looking right now.16:47
jcsackettsinzui: yes.16:47
rick_h_abentley: that looks to have come from somewhere else. All I did was revert my branch in rev 1631316:49
rick_h_which was a rollback of rev 1629516:49
rick_h_neither of which hit test_product16:49
=== teknico_ is now known as teknico
=== deryck is now known as deryck[lunch]
abentleyrick_h_: Okay.  It looks like abel did something similar to what you did, and I thought it was the same thing.16:51
rvbaczajkowski: he was this morning, but he's gone now.16:52
czajkowskirvba: cheers16:54
adeuringjcsackett: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-1079116/+merge/136991 ?16:58
jcsackettadeuring: right after i'm done with sinzui's.16:58
adeuringjcsackett: thanks16:58
jcsackettsinzui: r=me. adeuring: looking at yours now.16:59
sinzuithank you jcsackett16:59
=== matsubara-lunch is now known as matsubara
jcsackettadeuring: r=me. thanks for addressing the possible performance point in the proposal. i agree with your logic there.17:14
adeuringjcsackett: thanks!17:14
=== Ursinha is now known as Ursinha-afk
=== deryck[lunch] is now known as deryck
=== Ursinha-afk is now known as Ursinha
rick_h_abentley: so in looking, the branch sharing policy method alone has a check for if branch_sharing_policy is embargoed_or_proprietary19:09
rick_h_       # If the branch sharing policy is EMBARGOED_OR_PROPRIETARY, then we19:09
rick_h_        # do not allow any other policies.19:09
rick_h_do you know what branches might be different than bugs/blueprints for in this way?19:10
abentleyrick_h_: So you're saying once the policy has been set to embargoed_or_proprietary, it can never be changed?19:11
rick_h_abentley: that's what the current code/comment seem to be doing19:11
abentleyrick_h_: Weird.19:11
rick_h_but because we never allowed the option for embaroed_or_proprietary to be displayed it was never chosen19:12
rick_h_so trying to think through a use case for this/issue19:12
abentleyrick_h_: This is in the browser code?19:12
rick_h_no, sharingservice getBranchSharingPolicies19:12
abentleyrick_h_: I don't know why that's there, and I had discussions with Curtis about changing policies and he never mentioned it.19:16
rick_h_abentley: ok, tracking down with our annotate suggestions19:16
rick_h_if I don't see any reason thinking of removing that and seeing what tests blow up19:16
abentleyrick_h_: It's from ian.booth@canonical.com-20120927011545-3l2bzqe7qv5q7py919:17
rick_h_abentley: yea, looking19:17
abentleyrick_h_: Here's the merge proposal: https://code.launchpad.net/~wallyworld/launchpad/embargoed-sharing-policy-ui-1055617/+merge/12658519:18
abentleyrick_h_: "My primary motivation for requesting this is that it would be most unfortunate if someone accidentally changed the branch privacy policy and was not aware that you had to set it via API and then was not able to change it back to the proper value."19:19
rick_h_ah ok, so this was part of keeping embargoed from the list19:19
abentleyrick_h_: (from the bug).19:19
rick_h_abentley: right19:20
rick_h_so they went the route of preventing one from accidentally changing it instead of adding the option in19:20
abentleyrick_h_: But you were already planning to add the option in, so I say drop the restriction.19:23
rick_h_abentley: right, sounds like a plan thanks19:23
sinzuijcsackett, Do you have time to review https://code.launchpad.net/~sinzui/launchpad/librarian-merge-proposal/+merge/13703920:32
jcsackettsinzui: yes.20:32
sinzui^ There is a mess in there where I fought import fascist20:32
jcsackettsinzui: line 20 of your diff. should that be "return self.preview_diff_text" or is that supposed to be a function call?20:35
rick_h_jcsackett: for your queue if you have time before EOD please https://code.launchpad.net/~rharding/launchpad/limit_sharing_infotype_1083761/+merge/13703820:39
sinzuijcsackett, I think I need a comment there to explain that the diff is being precached by the call20:41
* sinzui wrote it yesterday and boggled at the line with fresh eyes20:41
jcsackettsinzui: dig. a comment would be appreciated.20:42
jcsacketti sort of feared it might be a "do something by side effect" case.20:42
jcsackettsinzui: 87 looks like a typo. view = view = create_initialized_view20:43
sinzuioh yes.20:43
sinzuimy apologies. I did not read the diff properly before asking you to review it20:44
jcsackettsinzui: all good.20:48
jcsackettotherwise, r=me.20:48
sinzuijcsackett, thank you.20:49
sinzuiabentley, I am was looking at oops reports and see that you dominate all the oopses for bug 1074385. I don't see it affect anyone else. Any idea why you are special?20:56
_mup_Bug #1074385: Timeout from getMergeProposals <api> <branches> <privacy> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1074385 >20:56
abentleysinzui: No one else uses it?20:56
sinzuioh, you might be right.20:56
sinzuiI should write a script and verify it happens to me too. In any case, I think the queries need optimisation20:57
abentleysinzui: No need to write a script, just use bzr lp-find-proposal.20:57
sinzuinoted, thanks20:58
rick_h_jcsackett: going to be afk for a few. Let me know if there's any questions and such. Will check back in.20:58
abentleysinzui: Ideally, we'd change BranchMergeProposal.merged_revno to BranchMergeProposal.merged_revevision_id.21:00
sinzuiah, we are joining though BranchRevision. We need to rethink that table or purge the duplication in it Lp's branches are  os 75%of the table21:04
abentleysinzui: Yes, it's long been an intention of the Code team to purge that table.  The last plan involved using bzr-history-db, IIRC.21:06
sinzuiwgrant is on leave, be he did some analysis about removing the duplication.21:07
james_wjcsackett, would you take a look at https://code.launchpad.net/~james-w/python-oops-tools/recent-oopses/+merge/137055 please?21:38
jcsackettjames_w: certainly, but it will be a moment while i finish another review.21:38
james_wjcsackett, no problem21:38
jcsackettjames_w: looks good. r=me.21:50
james_wjcsackett, thanks, do you know if you can land it?21:50
rick_h_thanks for the review jcsackett21:51
jcsackettjames_w: i can tomorrow, i'm running low on time for it now. is that ok?21:51
james_wjcsackett, sure21:52
jcsackettcool. i'll make a note of it.21:52
james_wjcsackett, oh, seems there is tarmac, so it will land itself when I fix that test failure22:00
james_wthanks though22:00
jcsackettjames_w: ah, excellent.22:00
lifelessjames_w: cool22:12
james_whi lifeless22:13
lifelesshi :)22:13
=== jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: ~170
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!