[00:33] <StevenK> wgrant: r16321 is only a testfix or a testfix and a rollback?
[00:43] <CyberJacob> Hi
[00:55] <wgrant> StevenK: Just a testfix
[00:58] <wgrant> sinzui: Hm, this same MP diff technique could/should probably be applied to the front page blog feed relatively easily, I guess
[01:09] <StevenK> reference = u'unique-from-factory-py-line3339-102754 binarypackage-102763'
[01:09] <StevenK> actual    = u'binarypackage-102763'
[01:09] <StevenK> Success
[01:28]  * StevenK peers at the garbo query of doom
[01:35] <CyberJacob> Hi Guys
[01:36] <CyberJacob> I'm trying to look up a user, but I keep getting a ComponentLookupError
[01:36] <CyberJacob> any ideas?
[01:36] <StevenK> What's the traceback?
[01:36] <CyberJacob> http://pastebin.ubuntu.com/1395890/
[01:37] <wgrant> CyberJacob: Where are you running that?
[01:38] <wgrant> Looks like bin/py, when you might want bin/harness
[01:38] <wgrant> bin/py just starts a Python interpreter, without setting up any of the infrastructure
[01:38] <CyberJacob> oh, that might be the problem then
[01:38] <CyberJacob> is there a binary for the infrastructure?
[01:38] <StevenK> bin/harness
[01:39] <StevenK> Or 'make harness'
[01:40] <StevenK> wgrant: I've added the joins for BPB, SPR and SPN, but I can't work out how to get it into the output of the SELECT array_agg
[01:40] <wgrant> Howso?
[01:41] <StevenK> wgrant: Everywhere I try and put sourcepackagename.name, postgres complains
[01:41] <wgrant> Complains?
[01:42] <StevenK> HINT:  No aggregate function matches the given name and argument types. Perhaps you misplaced ORDER BY; ORDER BY must appear after all regular arguments of the aggregate.
[01:42] <wgrant> What's your query?
[01:42] <CyberJacob> StephenK: works perfectly with bin/harness, thanks
[01:43] <StevenK> It's a V, damn it!
[01:43] <StevenK> wgrant: http://pastebin.ubuntu.com/1395899/ is what I'm testing on DF with
[01:43] <StevenK> With some variants
[01:45] <wgrant> StevenK: What does that mean?
[01:45] <wgrant> You're trying to create an array of rows?
[01:45] <StevenK> wgrant: I'm trying to get 4749089 | debhelper hello-debhelper-df
[01:47] <wgrant> StevenK: Right, but you seem to be trying to create an array of [(spn, bpn)] there, which is a bit odd
[01:47] <StevenK> wgrant: I've been trying various things, that was the latest attempt
[01:47] <wgrant> StevenK: I'd just do a separate join to get the SPN
[01:48] <wgrant> Bring it out in a separate array
[01:48] <wgrant> As we do with the source name
[01:52] <StevenK> Right
[01:52] <StevenK> That works nicely
[01:59] <bigjools> any regex experts want to do a maas pre-imp with me?
[02:00] <wgrant> A pre-imp for a regex... this does not bode well :)
[02:00] <wgrant> I'm about to have lunch but could talk after
[02:01] <bigjools> well it's more about how to deal with some postinst processing on a config
[02:01] <StevenK> wgrant: Right, that's the garbo and IPackageUpload.addBuild working with adding the SPR
[02:01] <StevenK> bigjools: In a word, "don't"
[02:01] <bigjools> heh
[02:01] <bigjools> I *know(
[02:01] <StevenK> If you can possibly get away with a .d directory, do that
[02:01] <bigjools> but we are stuck with it
[02:02] <wgrant> bigjools: What are you trying to do?
[02:02] <StevenK> If it isn't your config file, it's against policy, too ...
[02:02] <bigjools> fixing someone else's crap^Wwork
[02:02] <bigjools> a dpkg reconfigure is supposed to take an input value and insert it in various configs
[02:03] <StevenK> wgrant: My vague plan was to have the garbo job done and out of the tree by Friday, too ...
[02:03] <bigjools> this particular config may or may not have a commented line that we need to update
[02:03] <bigjools> it may also have a commented and a non commented line
[02:04] <wgrant> StevenK: I generally suggest getting all the work done before landing any of it (other than an obviously safe DB patch)
[02:05] <wgrant> It's far easier to fix the garbo job than it is to fix several million rows
[02:07] <CyberJacob> Is there a way of starting the Salesforce (voucher) server locally?
[02:08] <CyberJacob> or is that a non-GPL part of launchpad?
[02:11] <wgrant> CyberJacob: It's not part of Launchpad at all. I've never seen the source, and I'm not sure how much of it we actually have (it's just a thin wrapper around SalesForce itself, AIUI)
[02:12] <CyberJacob> what about forcing `make run` to use the test proxy transport
[02:12] <CyberJacob> that should get it talking properly
[02:12] <wgrant> What are you trying to do?
[02:13] <wgrant> If you just want to extend the complimentary commercial subscription, we use an API script like http://bazaar.launchpad.net/~canonical-launchpad-branches/lp-dev-utils/trunk/view/head:/extend_subscription.py
[02:13] <wgrant> There's little point going through the voucher system unless you need to
[02:14] <CyberJacob> that's basically what I was looking for
[02:14] <CyberJacob> (I didn't realise SalesForce was a full piece of software, just looked like another module to me)
[02:15] <wgrant> The SalesForce referred to here is the service provided by salesforce.com
[02:15] <CyberJacob> which explains why there's just an RPCXML connecter in LaunchPad
[02:16] <wgrant> Yep
[02:59] <CyberJacob> wgrant: I'm getting a 404 error with that script...
[03:00] <CyberJacob> Looks like it's trying to load NotFound: Object: <lp.services.webapp.publisher.RootObject object at 0x41d7d50>, name: u'devel'
[03:00] <CyberJacob> any ideas?
[03:01] <StevenK> wgrant: So the garbo job and the population now sets source name as well, and I've changed IPackageUploadSet.getAll() to use searchable_names in a different branch. That's all of it, isn't it?
[03:02] <wgrant> CyberJacob: Have you told it to connect to the right URL?
[03:02] <wgrant> StevenK: How are you handling version?
[03:02] <CyberJacob> wgrant: I'm using ./extend_subscription.py -s "https://launchpad.dev/" -d 90 caveman
[03:02] <wgrant> CyberJacob: api.launchpad.dev, not just launchpad.dev
[03:02] <StevenK> wgrant: So far, I'm not.
[03:02] <wgrant> StevenK: You'd best
[03:03] <wgrant> CyberJacob: (also, for internal projects we normally just force it to extend for 10 years or so)
[03:03] <CyberJacob> wgrant: ah, perfect!
[03:03] <StevenK> 40-1 turns into ADD COLUMN searchable_versions ?
[03:03] <wgrant> StevenK: It depends on whether there's a performance issue or not
[03:04] <StevenK> wgrant: With the existing queries?
[03:05] <wgrant> StevenK: With name denormed, and version not
[03:05] <StevenK> Yeah
[03:05] <wgrant> Optimise things that need optimising
[03:05] <wgrant> Don't optimise things that don't :)
[03:06] <StevenK> The garbo job on DF has 740k rows to go, but will need to start again
[03:06] <StevenK> Anyway, I'll add the trigram, and test a few queries
[03:12] <StevenK> Come on, DF, it's an index, not solving pi to 1 trillion places.
[03:12] <wgrant> It'll take a while to build
[03:13] <wgrant> They're longish string fields, and trigram indices aren't exactly efficient
[03:13] <StevenK> It's been at it for 5 minutes already
[03:13] <wgrant> And there's millions of rows
[03:13] <wgrant> This will be one of our largest non-FTI indices
[03:17] <StevenK> CREATE INDEX
[03:17] <StevenK> Time: 621691.915 ms
[03:17] <StevenK> wgrant: How can I check how large it is?
[03:18] <wgrant> It's 656MB
[03:19] <wgrant> Which places it just between the bug and bugtaskflat FTI indices, IIRC
[03:19] <wgrant> The only indices likely to be bigger than those are message FTI, and branchrevision + *downloadcount indices
[03:19] <wgrant> :)
[03:22] <StevenK> SELECT max(char_length(searchable_names)) FROM packageupload; => 11908
[03:23] <wgrant> Ouch
[03:23] <wgrant> I wonder if that's langpacks on a delayed copy
[03:24] <StevenK> I'm just trying to figure out how to check that
[03:26] <StevenK> Not quite
[03:26] <StevenK> ... libghc6-vte-dev libghc6-vte-doc libghc6-vty-dev libghc6-vty-doc libghc6-vty-prof ...
[03:26] <wgrant> Ah
[03:28]  * StevenK looks for a good victim
[03:29] <StevenK> Actually, that Haskell one looks good
[03:39] <StevenK> Just searching for a version is pathetically slow
[03:42] <StevenK> wgrant: The way this query is going, it is clearly searching all BPR and SPR rows
[03:43] <wgrant> Well, what does EXPLAIN say?
[03:43] <StevenK> It hasn't finished yet
[03:43] <wgrant> without ANALYZE
[03:44] <StevenK>                      ->  Seq Scan on sourcepackagerelease  (cost=0.00..321829.18 rows=1627518 width=25)
[03:44] <StevenK>                                  ->  Seq Scan on packageuploadbuild  (cost=0.00..49832.41 rows=3137541 width=8)
[03:44] <wgrant> That's less than ideal
[03:44] <StevenK>                      ->  Seq Scan on binarypackagerelease  (cost=0.00..1618381.64 rows=11488564 width=28)
[03:44] <lifeless> nice
[03:45] <StevenK> Let's look over what, 17 million rows?
[03:45] <bigjools> sigh, how can linking a branch to a blueprint time out
[03:45] <wgrant> bigjools: When the branch row is locked by the scanner, usually
[03:45] <bigjools> it's an old landed branch
[03:45] <wgrant> What does the OOPS show?
[03:46] <lifeless> ur face?

[03:46] <bigjools> haha
[03:46] <bigjools> https://oops.canonical.com/oops/?oopsid=OOPS-cf507af414fb03b447b2658c8c964a72
[03:46] <StevenK> lifeless: Fail.
[03:46] <StevenK> lifeless: This is you, you never close that tag
[03:47] <lifeless> StevenK: closing it is itself a troll
[03:47] <lifeless> StevenK: thus YHBT
[03:47] <wgrant> bigjools: Ah, you probably typoed the branch name.
[03:47] <wgrant> Yes
[03:47] <wgrant> You gave the URL, not the name, so it did a search
[03:47] <bigjools> it has SPACES on the end
[03:47] <wgrant> and the search algorithm is insane
[03:47] <bigjools> that is all
[03:47] <bigjools> WTAF
[03:48] <wgrant> Bug #793830
[03:48] <_mup_> Bug #793830: Branch:+register-merge / Specification:+linkbranch time out due to substring matching many tables <critical-analysis> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/793830 >
[03:48] <bigjools> you gotta love the optimism in the code around user input
[03:48] <wgrant> Well
[03:49] <wgrant> It'd be all good if the search mechanism wasn't completely stupid
[03:49] <StevenK> bigjools: I'm in the middle of a fixing a critical around IPackageUploadSet.getAll(). That code is just as good
[03:49] <bigjools> trailing spaces being very stupid
[03:49] <StevenK> Let's JOIN against SPR, SPN, BPB, PCJ and LFA and do a substring match
[03:50] <StevenK> Oh, I'm forgetting BPR and BPN
[03:51] <StevenK> wgrant: So, I guess denorming version is a go?
[03:52] <StevenK> wgrant: Oh, and that trigram index isn't even fully populated
[03:58] <wgrant> StevenK: And you forgot PUS, PUB and PUC
[03:58] <wgrant> But anyway
[03:58] <wgrant> StevenK: Right, denorming version would make sense here, but we probably don't care about substring matches on it
[03:58] <wgrant> We support it today, but that's fairly insane and probably not useful
[03:59] <StevenK> I have a garbo query, too
[03:59] <StevenK> wgrant: http://pastebin.ubuntu.com/1396146/
[04:00] <wgrant> Right, but there's not much reason to use a string here
[04:00] <wgrant> We don't care about substring matching
[04:00] <StevenK> What do you suggest?
[04:00] <wgrant> A plain array of strings with a GIN index
[04:00] <wgrant> Much faster
[04:01] <StevenK> Hmmm, don't know how to craft that one
[04:01] <wgrant> Just don't turn the array into a string
[04:02] <StevenK>  2873221 | {NULL,1:4}
[04:02] <StevenK> Given we don't have mixed uploads, I don't think an array is useful
[04:03] <wgrant> Binaries can have different versions
[04:05] <StevenK> For the same upload, for the same source, from the same BPB?
[04:09] <wgrant> StevenK: Yes
[04:09] <wgrant> It's not hugely common, but some fairly significant packages like gcc do it
[04:12] <StevenK>  3857156 | {NULL,4.4.6-15ubuntu1,1:4.4.6-15ubuntu1}
[04:12] <StevenK> Well, damn
[04:19] <StevenK> wgrant: ADD COLUMN searchable_versions TEXT[]; ?
[04:20] <wgrant> StevenK: It's technically a debversion, but that's probably unimportant and possibly detrimental here
[04:20] <StevenK> Right
[04:21] <StevenK> I'm not sure I care enough to test both
[04:26] <wgrant> I should really try some multi-column GIN btf indices at some point.
[04:37] <wgrant> Hm
[04:37] <wgrant> It actually works pretty well
[04:40] <wgrant> Potentially useful for translations, too
[04:40]  * wgrant plots
[05:12] <StevenK> wgrant: How can I declare searchable_versions as an array in the model code so I can avoid SQL('') everywhere in the garbo job?
[05:14] <wgrant> StevenK: A List, probably
[05:53] <stub> StevenK: Today's search column is an array, yesterday's wasn't. Is that a foot gun for one or t'other?
[05:55] <StevenK> Neither
[06:15] <wgrant> "TODO: Test IE compatibility. StuartBishop 20041118"
[06:39] <stub> haha
[06:39] <stub> Does it work in Ireland now?
[06:39] <wgrant> Heh
[07:35] <czajkowski> morning folks
[08:41] <adeuring> good morning
[08:45] <StevenK> stub: So, no, it isn't a fail. searchable_names is a string that will be substring matched via a GIN index. searchable_versions is an array that will not be substring matched, so it made sense to store it as an array
[09:07] <stub> StevenK: How does a package upload end up with multiple versions btw?
[09:33] <StevenK> stub: Some packages do it, like gcc.
[09:34] <StevenK> It's rare, but does happen
[13:59] <deryck> Morning, all.
[14:00] <rick_h_> morning
[14:09] <cjwatson> stub: Binary package generation is controlled by the source package's debian/rules; by default it produces binaries with versions matching the source, but it's occasionally useful to use the -v option to dpkg-gencontrol (or its wrappers) to change that
[14:13] <stub> It sounds very quantum.
[14:16] <cjwatson> Entirely deterministic :)
[14:22] <czajkowski> deryck: hope you're feeling better
[14:22] <deryck> czajkowski, thanks! I am. Little worn down by being sick yesterday, but overall, much better.
[14:22] <czajkowski> :(
[14:23] <czajkowski> pleanty of tea!
[14:23] <czajkowski> hot tea now, none of that ice cold stuff
[14:26] <stub> Are Debian version numbers Turing complete?
[14:28] <cjwatson> No. :-P
[14:29] <cjwatson> Well, except that they can be arbitrarily long so I suppose you could encode any program in them.
[14:29] <cjwatson> Same as you could encode any program in YOUR MUM.
[14:30]  * cjwatson runs
[14:30] <stub> I don't think my mum is Turing complete
[15:41] <rick_h_> abentley: pushed updated changes with tests to my MP. Did you want to go over it or should I ask jcsackett? https://code.launchpad.net/~rharding/launchpad/nonpublic_1052659/+merge/136479
[15:41] <abentley> rick_h_: I'll look.
[15:41] <rick_h_> abentley: ty much
[15:41] <cjwatson> http://www.rogerharford.com/content/img/blog-images/database-design-breakdown.png
[15:44] <abentley> rick_h_: Looks great.  Thanks for making all those changes.
[15:45] <abentley> rick_h_: r=me.
[15:45] <rick_h_> abentley: thanks for the review and the push to 'do the right thing' by adding in the tests
[15:45] <rick_h_> sometimes need a nudge :)
[15:45] <abentley> rick_h_: np.
[16:17] <czajkowski> rvba: is jtv on today ?
[16:42] <abentley> rick_h_: What's the status of your rollback for _getTranslatables ?
[16:43] <rick_h_> abentley: done and landed
[16:43] <rick_h_> deployed as part of yesterday
[16:45] <abentley> rick_h_: Shouldn't test_getTranslatables_filters_private_products no longer exist?  I still see that in stable and devel.
[16:45] <sinzui> jcsackett, do you have time to review https://code.launchpad.net/~sinzui/launchpad/fast-contact-via-web/+merge/136986
[16:47] <rick_h_> abentley: looking right now.
[16:47] <jcsackett> sinzui: yes.
[16:49] <rick_h_> abentley: that looks to have come from somewhere else. All I did was revert my branch in rev 16313
[16:49] <rick_h_> which was a rollback of rev 16295
[16:49] <rick_h_> neither of which hit test_product
[16:51] <abentley> rick_h_: Okay.  It looks like abel did something similar to what you did, and I thought it was the same thing.
[16:52] <rvba> czajkowski: he was this morning, but he's gone now.
[16:54] <czajkowski> rvba: cheers
[16:58] <adeuring> jcsackett: could you please review this MP: https://code.launchpad.net/~adeuring/launchpad/bug-1079116/+merge/136991 ?
[16:58] <jcsackett> adeuring: right after i'm done with sinzui's.
[16:58] <adeuring> jcsackett: thanks
[16:59] <jcsackett> sinzui: r=me. adeuring: looking at yours now.
[16:59] <sinzui> thank you jcsackett
[17:14] <jcsackett> adeuring: r=me. thanks for addressing the possible performance point in the proposal. i agree with your logic there.
[17:14] <adeuring> jcsackett: thanks!
[19:09] <rick_h_> abentley: so in looking, the branch sharing policy method alone has a check for if branch_sharing_policy is embargoed_or_proprietary
[19:09] <rick_h_>        # If the branch sharing policy is EMBARGOED_OR_PROPRIETARY, then we
[19:09] <rick_h_>         # do not allow any other policies.
[19:10] <rick_h_> do you know what branches might be different than bugs/blueprints for in this way?
[19:11] <abentley> rick_h_: So you're saying once the policy has been set to embargoed_or_proprietary, it can never be changed?
[19:11] <rick_h_> abentley: that's what the current code/comment seem to be doing
[19:11] <abentley> rick_h_: Weird.
[19:12] <rick_h_> but because we never allowed the option for embaroed_or_proprietary to be displayed it was never chosen
[19:12] <rick_h_> so trying to think through a use case for this/issue
[19:12] <abentley> rick_h_: This is in the browser code?
[19:12] <rick_h_> no, sharingservice getBranchSharingPolicies
[19:16] <abentley> rick_h_: I don't know why that's there, and I had discussions with Curtis about changing policies and he never mentioned it.
[19:16] <rick_h_> abentley: ok, tracking down with our annotate suggestions
[19:16] <rick_h_> if I don't see any reason thinking of removing that and seeing what tests blow up
[19:17] <abentley> rick_h_: It's from ian.booth@canonical.com-20120927011545-3l2bzqe7qv5q7py9
[19:17] <rick_h_> abentley: yea, looking
[19:18] <abentley> rick_h_: Here's the merge proposal: https://code.launchpad.net/~wallyworld/launchpad/embargoed-sharing-policy-ui-1055617/+merge/126585
[19:19] <abentley> rick_h_: "My primary motivation for requesting this is that it would be most unfortunate if someone accidentally changed the branch privacy policy and was not aware that you had to set it via API and then was not able to change it back to the proper value."
[19:19] <rick_h_> ah ok, so this was part of keeping embargoed from the list
[19:19] <abentley> rick_h_: (from the bug).
[19:20] <rick_h_> abentley: right
[19:20] <rick_h_> so they went the route of preventing one from accidentally changing it instead of adding the option in
[19:23] <abentley> rick_h_: But you were already planning to add the option in, so I say drop the restriction.
[19:23] <rick_h_> abentley: right, sounds like a plan thanks
[20:32] <sinzui> jcsackett, Do you have time to review https://code.launchpad.net/~sinzui/launchpad/librarian-merge-proposal/+merge/137039
[20:32] <jcsackett> sinzui: yes.
[20:32] <sinzui> ^ There is a mess in there where I fought import fascist
[20:35] <jcsackett> sinzui: line 20 of your diff. should that be "return self.preview_diff_text" or is that supposed to be a function call?
[20:39] <rick_h_> jcsackett: for your queue if you have time before EOD please https://code.launchpad.net/~rharding/launchpad/limit_sharing_infotype_1083761/+merge/137038
[20:41] <sinzui> jcsackett, I think I need a comment there to explain that the diff is being precached by the call
[20:41]  * sinzui wrote it yesterday and boggled at the line with fresh eyes
[20:42] <jcsackett> sinzui: dig. a comment would be appreciated.
[20:42] <jcsackett> i sort of feared it might be a "do something by side effect" case.
[20:43] <jcsackett> sinzui: 87 looks like a typo. view = view = create_initialized_view
[20:43] <sinzui> oh yes.
[20:44] <sinzui> my apologies. I did not read the diff properly before asking you to review it
[20:48] <jcsackett> sinzui: all good.
[20:48] <jcsackett> otherwise, r=me.
[20:49] <sinzui> jcsackett, thank you.
[20:56] <sinzui> abentley, I am was looking at oops reports and see that you dominate all the oopses for bug 1074385. I don't see it affect anyone else. Any idea why you are special?
[20:56] <_mup_> Bug #1074385: Timeout from getMergeProposals <api> <branches> <privacy> <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1074385 >
[20:56] <abentley> sinzui: No one else uses it?
[20:56] <sinzui> oh, you might be right.
[20:57] <sinzui> I should write a script and verify it happens to me too. In any case, I think the queries need optimisation
[20:57] <abentley> sinzui: No need to write a script, just use bzr lp-find-proposal.
[20:58] <sinzui> noted, thanks
[20:58] <rick_h_> jcsackett: going to be afk for a few. Let me know if there's any questions and such. Will check back in.
[21:00] <abentley> sinzui: Ideally, we'd change BranchMergeProposal.merged_revno to BranchMergeProposal.merged_revevision_id.
[21:04] <sinzui> ah, we are joining though BranchRevision. We need to rethink that table or purge the duplication in it Lp's branches are  os 75%of the table
[21:06] <abentley> sinzui: Yes, it's long been an intention of the Code team to purge that table.  The last plan involved using bzr-history-db, IIRC.
[21:07] <sinzui> wgrant is on leave, be he did some analysis about removing the duplication.
[21:38] <james_w> jcsackett, would you take a look at https://code.launchpad.net/~james-w/python-oops-tools/recent-oopses/+merge/137055 please?
[21:38] <jcsackett> james_w: certainly, but it will be a moment while i finish another review.
[21:38] <james_w> jcsackett, no problem
[21:50] <jcsackett> james_w: looks good. r=me.
[21:50] <james_w> jcsackett, thanks, do you know if you can land it?
[21:51] <rick_h_> thanks for the review jcsackett
[21:51] <jcsackett> james_w: i can tomorrow, i'm running low on time for it now. is that ok?
[21:52] <james_w> jcsackett, sure
[21:52] <jcsackett> cool. i'll make a note of it.
[22:00] <james_w> jcsackett, oh, seems there is tarmac, so it will land itself when I fix that test failure
[22:00] <james_w> thanks though
[22:00] <jcsackett> james_w: ah, excellent.
[22:12] <lifeless> james_w: cool
[22:13] <james_w> hi lifeless
[22:13] <lifeless> hi :)