[00:23] thumper: just talked to amazon. they are sending out a replacement. they are charging me but will refund the money when i send back the broken one [00:23] wallyworld_: awesome news [00:24] thumper: yeah. they asked several times if it had been dropped or pressure put on the screen. but it lives in the amazon hard covered case.... [00:34] So much QA :/ [00:36] lifeless: Can we restore qastaging? [00:37] +1 [00:39] jelmer: Hi. [00:39] jelmer: http://staging.launchpadlibrarian.net/73288606/vcs-imports-pydoctor-trunk.log -- I don't think cscvs likes the autoupgrade stuff. [00:47] I will be back later; today is in 2 parts :) === wallyworld is now known as 92AADA0EO [00:59] wgrant: darn [01:00] wgrant: should be a trivial change to clean it up after upgrade [01:03] jelmer: Rather. [01:12] jelmer: Trivial enough that it's worth fixing rather than rolling it back? [01:12] wgrant, https://code.launchpad.net/~jelmer/launchpad/auto-upgrade-qafix/+merge/64771 [01:13] jelmer: Perfect, thanks. [01:13] I think we shall skip ec2 this time. [01:14] Hm. No wallyworld [01:15] I guess he did see the State of Origin, then ... [01:15] wgrant: are you landing it, or should I? [01:15] jelmer: I've approved it, and can land it if you want. [01:15] I realise it's late. [01:16] wgrant: if you can, that would be great [01:16] * jelmer gets some sleep [01:17] jelmer: Thanks for the fix. [01:17] Night! [01:17] flacoste: ping? [02:58] <92AADA0EO> StevenK: hi. did you ping me? [02:58] <92AADA0EO> stupid nick :-( [02:59] 92AADA0EO: Yes. To gloat. [03:00] <92AADA0EO> ns [03:01] StevenK: well, i bet you didn't even watch the game? [03:01] Right. [03:02] so nothing to gloat about :-P [03:02] But QLD lost, so that makes me happy. [03:02] but you don't care you said [03:03] Project devel build #809: STILL FAILING in 5 hr 51 min: https://lpci.wedontsleep.org/job/devel/809/ [03:06] Project windmill-devel build #232: STILL FAILING in 1 hr 7 min: https://lpci.wedontsleep.org/job/windmill-devel/232/ [04:48] wallyworld_: Hi. [04:48] hello [04:49] Have you seen bug #797820? [04:49] <_mup_> Bug #797820: searching for the last name makes still hard to find the person

< https://launchpad.net/bugs/797820 > [04:49] The name-prefix matches drown everything else out :/ [04:49] no, not yet [04:49] Even after I reduced the numbers. [04:49] let me have a quick read [04:49] I think we may need to take them right down :( [04:51] wgrant: so that bug report implies there are lots of names starting with "lange"? [04:51] wallyworld_: It doesn't say that, but when you try it it's clear. [04:51] * wallyworld_ tries it [04:52] https://pastebin.canonical.com/48548/ are the results of that search. [04:52] The second column is the rank. [04:53] jml's FTI match is 0.6, but the karma scales it up. [04:53] But not high enough. [04:53] (0.6 is about right for a single name match on a two-part name) [04:53] wgrant: i tried it on lp.net and yes there are lots of usernames starting with "lange". so i think we just de-prioritise name prefix matches [04:53] adjust the scaling factor [04:54] Adjust the scaling factor, or reduce the hardcoded ranks? [04:54] reduce the hard coded ranks [04:54] for name prefix [04:54] name prefix = 5, displayname prefix = 4, IRC nick match = 3, email prefix = 2 [04:54] That's where we are now. [04:55] where is exact name match in that list? [04:55] Exact name match is always 100 [04:55] There is nobody with username 'lange', so it doesn't show up here. [04:56] (yeah, having it hardcoded to 100 is bad, but it will work unless someone has 10^20 karma, in which case we are probably screwed anyway) [04:56] i don't like 5,4,3,2. i think ranks needs to be between 0,100 and sacling factor applied to fti to put it in the 0,100 range [04:56] Why 0-100 rather than 0-1? [04:57] so that we can make minor adjustments using whole number arithmatic [04:57] I wonder what happens if we just reduce them to a perfect FTI match... 1. [04:58] That would probably be good for full email address local part matches. [04:58] But not general prefix matches. [04:58] i think we need to discriminate between them though, not just make all = 1 [04:58] hence the 0-100 range [04:58] Probably, but how? [04:59] conceptually it's easier to think about adjustments in that range [04:59] Not for me, but OK :) [04:59] i mean whole number arithmetic is easier than decimals between 0..1 === Ursinha is now known as Ursinha-zzz [04:59] so we scale the fti result by 100 [04:59] Hardly :) [05:00] But this is not really crucial to the problem. [05:00] We don't know where to put these non-FTI matches. [05:00] That is the important bit. [05:00] sure, butno we have a 0-100 range to play in, we can experiment [05:00] but now [05:01] so start by reducing the weight of name prefix matches [05:01] but then again, if we scale fti by 100 maybe no adjustemtn needed [05:02] to the name prefix weighting if it's say 50 [05:02] since 0.6 * 100 = 60 > 50 [05:02] But is that what we want? [05:03] well, i think an exact match on lastname should be > name prefix match [05:03] too bad we can't weight based on how closely the name prefix matches [05:03] Indeed. [05:03] that would help a lot [05:04] but in any case, if someone types the full lastname and it matches, then surely that should place those results near the top? [05:04] above any name prefix matches [05:04] What if I'm not typing a full lastname, but a name prefix? [05:05] then you see that result below the display name matches :-) [05:05] i think it will be more common for people to type last name [05:05] if they search on lp id (name), then surely they will typr the whole id? [05:06] i would :-) [05:07] Possibly. [05:07] Perhaps we should remove the name prefix matching and see how it goes. [05:07] Not sure about displayname. [05:07] There are still 9 displayname match results above jml. [05:07] i just think we need to reduce the name prefix weight [05:08] But I wonder if FTI deals with them OK. [05:08] Yes, but reduce it to where? [05:08] wgrant: so imho, i think we should: scale fti by 100 and adjust the 5,4,3,2 to 50, 40, 35 or whatever [05:08] There are tonnes of FTI matches. [05:08] tonnes yes, but all with different rankings [05:08] Lots at the same rankings. [05:09] so if we scale correctly, they will fall somewhere in the range also occupied by those other matches [05:09] The visible non-jml FTI matches are 0.75990885..., and there are dozens of them. [05:09] What FTI match does jml get for 'lange'? [05:10] 0.6 [05:10] 0.607927 [05:11] winder why a full lastname match is lower than those other results [05:14] wallyworld_: lange stems to lang. [05:14] alexander-lang's displayname is Alexander Lang [05:14] So he gets a double lang match. [05:14] Project db-devel build #639: STILL FAILING in 5 hr 27 min: https://lpci.wedontsleep.org/job/db-devel/639/ [05:14] launchpad_dogfood=> SELECT name, fti, rank(fti, ftq('lange')) FROM person WHERE name IN ('jml', 'alexander-lang'); [05:14] name | fti | rank [05:14] ----------------+--------------------------------------------------+---------- [05:14] jml | 'jml':1A 'jonathan':2A 'lang':3A | 0.607927 [05:15] alexander-lang | 'alexand':2A,4A 'alexander-lang':1A 'lang':3A,5A | 0.759909 [05:15] why does it chop off the last e? [05:15] So, in this case, I don't think jml has to be *first*, I think he has to be in the top 10 [05:15] Stemming does not work well on names :) [05:15] StevenK: He should be first. [05:15] StevenK: He's the only one related to the project. [05:17] That's a point [05:17] wgrant: so perhaps we can do in memory affiliation sorting once the db query has been done [05:18] use the affiliation to fine tune the result [05:18] We can do that in the DB just as easily. [05:18] We have no choice. [05:18] We have to do it there. [05:18] Or we have to pull back EVERYTHING from the DB. [05:20] i didn't think the affiliation was easily queryable [05:21] and i mean we use affiliation to fine tune the results [05:21] "fine tune" is a rather broad term. [05:21] so we still limit the db query to 100 [05:22] and then "fine tune" the final order of those 100 records, for whatever mean of "fine tune" [05:22] meaning [05:22] What if there are 500 0.7 FTI matches? [05:22] We pull back the top 100 sorted by displayname. [05:22] And bump the affiliated people. [05:22] well we limit the results to 100 now don't we? [05:22] But the person I wanted was named Zack. [05:22] So he's number 499. [05:22] We do. But we do affiliation in the query. [05:22] All ordering is done in the query. [05:22] And we take the top 100. [05:23] you mean karma = affiliation? [05:23] i haven't seen the latest query [05:23] For the purposes of search, yes. [05:23] ORDER BY rank * COALESCE( (SELECT LOG(karmavalue) FROM KarmaCache WHERE person = Person.id AND product = 10294 AND category IS NULL AND karmavalue > 10), 1) DESC, Person.displayname, Person.name LIMIT 100 [05:23] i think that's perhaps a false assumption that karma == affiliation [05:23] Yes, but it was quick and it's pretty effective. [05:24] And it handles unofficial relationships well. [05:24] And it's logarithmic, so infrequent contributors aren't penalised too much. [05:24] And if you're searching for someone to assign them to a bugtask, that's probably been done before, so they have karma. [05:24] ok. so we have affiliation in the query. we "just" need to tweak the ranking factors :-) [05:25] but it will be hard [05:25] for every search we "fix", we will "break" another [05:25] Yes. [05:26] I question the utility of the prefix matches. [05:26] so i think so long as the intended person appears on the first page every time, that wil suffice [05:26] But perhaps there is a reason. [05:26] Yes. [05:26] The correct person will rarely be first. [05:26] Because exact name matches always win. [05:26] and I don't think that's negotiable. [05:26] yes [05:27] if we are unsure about prefix matches we just need to devalue them [05:27] but not exclude them [05:27] totally [05:27] Devaluation is pretty much exclusion, unless there are no FTI matches at all. [05:27] Mm. [05:27] Although I guess relevant prefix matches will be bumped up a bit. [05:28] surely bad fti matches will be < 0.3 or whatever [05:28] so prefix matches can be made to be higher than 0.5 or whatever [05:28] so that only close fti matches 'win" [05:28] "close" [05:28] :-) [05:28] We have to choose arbitrary constants :/ [05:29] yep [05:29] unless we can run the data through some statistical analysis [05:29] s/statistical/magical/ [05:30] well, if we had expected search terms and results, and a lot of those, then we could do something. but we don't [05:30] Even if we did, there will always be other cases. [05:30] outliers. sure. but that's unavoidable [05:31] What if the cases we test are the outliers? [05:31] :) [05:31] up to us to make sure they're not [05:32] i think there's wiggle room. desired person on first page seems reasonably doable hopefully. or very least, on page 2. [05:32] i'd like to be on page 3 personally :-) [05:37] wallyworld_: *blind* [05:39] lifeless: maybe we should do a naked geeks calendar for charity :-) i'll be Mr December. [05:39] And none of us will enjoy Christmas again [05:39] -ever- [05:40] * StevenK high fives lifeless [05:42] * wallyworld_ lols [05:56] jtv: Around yet? [05:56] far too round, thank you [05:56] Had to go make some travel preps etc. [05:56] jtv: Ah, hi! Bug #797151 needs QA, when you have time. [05:56] <_mup_> Bug #797151: Display DSD copy errors differently from comments

< https://launchpad.net/bugs/797151 > [05:56] thanks for bringing it to my attention. Won't take long. [05:57] mwhudson: How much effort is setting up importds for qastaging? [05:57] mwhudson: Is it just a matter of configuring the importd with a qastaging tree too? [06:01] wgrant: um, probably [06:01] the process is documented, would you believe [06:01] I don't appreciate your lies. [06:02] wgrant: https://wiki.canonical.com/InformationInfrastructure/OSA/LPHowTo/SetUpCodeImportSlave [06:02] I don't appreciate lies with evidence, either. [06:02] But OK. [06:02] most of those steps will have been done already for the staging importds, i don't know what modifications will be needed for qastaging [06:03] mwhudson: There's no master setup needed? It just does xmlrpc? [06:03] wgrant: yeah [06:03] Great, thanks. [06:03] the production importds connect to the database for oops-prune [06:04] Well, at the moment some of them don't. [06:04] And they cronspam about it :) [06:04] pear? it should now. [06:04] Indeed! It stopped a week ago. [06:04] This is good. [06:05] yah. just needed same rules to allow those connections thru and we were gold. [06:05] some* [06:10] Anyone mind if I sync some packages on staging's oneiric? wgrant, StevenK? [06:10] wgrant: do me a favour? [06:10] Oh darn, can't do it on staging — not privileged [06:10] wgrant: file bugs for any db access you run across that isn't from an appserver, db-stats or security/py/upgrade.py/fti.py. [06:11] lifeless: The publisher? :-) [06:11] lifeless: Um, you know we have dozens of scripts, right? :) [06:11] Like StevenK said. [06:11] loganberry [06:12] process-accepted, queue, change-overrides, process-death-row ... [06:12] jtv: StevenK or I can do it there, but I don't think the flag is enabled. [06:12] ahhh [06:12] nm I'll stick with df [06:14] wgrant, StevenK: mind if I restart the app server on df? [06:14] Go ahead [06:14] Doit. [06:15] thx [06:15] the +localpackagediffs is currently oopsing on df because it picks up my tal change on the fly, but needs a restart for the code change [06:18] wgrant: it's out of your way. [06:19] wallyworld_: Can't believe I missed the game last night [06:20] wgrant: did you Q/A my update to the queue.py script? Julian said it was broken. [06:21] jtv: It seemed to work for me. [06:21] jtv: I initially thought the code was wrong, but it seems OK. [06:22] Any idea whether you made it print out any package uploads with PCJs attached? [06:22] That's the part that was in question. [06:22] No, but they don't exist yet, so I don't care :) [06:23] They don't exist? [06:23] wgrant: yes, I just mean whenever you think of one. [06:24] wgrant: particularly ones where we're close to nuking all db access [06:24] jtv: PCJ PUs don't exist in any real environment yet, and they are not close to done. [06:24] wgrant: in the current code we create PUs with PCJs by creating a fresh PPCJ, and .run()ing it. [06:25] wgrant: that's no reason to deploy a branch that may or may not have spectacularly failed Q/A! [06:25] jtv: It's not going to affect production. [06:25] So it has not failed QA. [06:25] Production does not yet use PCJs. [06:25] Until these jobs exist, and suddenly it may break. [06:25] The whole point is to notice that _before_ it hits production. [06:26] There is a fuckload more QA to do of the whole thing before PCJs are going anywhere near production. [06:26] Once it's done. [06:26] And working. [06:26] Yes, but this could be an easy thing to miss at that point. [06:27] It's one of the more obvious things to test. [06:27] And it can be tested now. [06:27] It can be. [06:28] Anyway, I think I'm conflating qa-tagging with q/a [06:28] Probably. [06:29] It'd be really nice to have that process cleaned up. You're right that it can be safely deployed; I've still got the ingrained habit of qa-needstesting meaning what it did. [06:30] It needs fixing, yes. [06:30] deploy-needstesting / deploy-ok / deploy-bad [07:16] Project parallel-test build #42: STILL FAILING in 1 hr 3 min: https://lpci.wedontsleep.org/job/parallel-test/42/ [07:16] stub: O hai! [07:19] yo [07:19] stub: Mind looking at https://code.launchpad.net/~stevenk/launchpad/db-add-bprc/+merge/64783 for a db review when you have a sec? [07:20] k [07:23] StevenK: So we don't stuff the files in the Librarian because they already exist somewhere on disk, and we can find that from the filename and the bpr? [07:24] oic. The filename is the data, and we generate a file containing all the filenames [07:25] Right [07:25] The tablename makes me think 'This row represents the contents of a BinaryPackageRelease', which would be a binary blob or maybe the TOC of a tarball. [07:25] It will be many rows [07:26] Which is why the primary key is both columns [07:26] This is the TOC of a tarball. === almaisan-away is now known as al-maisan [07:27] Which answers my next question on why this is a separate table too (not that it would be a bad idea even if it was 1:1, just interested in reasoning) === al-maisan is now known as almaisan-away [07:27] StevenK: stub: denorm the filename [07:28] Hm? [07:28] theres only ~2M unique filenames in Ubuntu [07:28] Oh [07:28] sorry [07:28] I mean 'normalise the filename' [07:28] I like that idea even less [07:28] normalize [07:29] It will save lots of disk. [07:29] Hmm... its always a tough call that. It might save a little space, but will be slower and harder to query. [07:29] Probably a good idea. [07:29] But... [07:29] How much we looking at? [07:29] Makes it hard to search. [07:29] This table will be huge. [07:29] stub: MANY MANY LOTS [07:29] well [07:29] Probably >100m rows. [07:29] it changes the search dynamic [07:29] sometimes its better [07:29] sometimes its worse. [07:29] No way we can do it before we have more disk. [07:29] At this point, I only want the table there. I will look at populating it later [07:30] conflictchecker has this table [07:30] done normalised style [07:30] StevenK: Will these files already be in the librarian? [07:30] it can find conflicting filenames -extremely- fast [07:30] stub: no, they are the unpacked contents [07:31] stub: No, they won't. [07:31] StevenK: so, rather than liking or not liking the table [07:31] stub: It will be a list of the contents of the files in the librarian [07:31] StevenK: lets analyze. What are the queries you want to do on it? [07:32] lifeless: In pseudo-SQL: SELECT filename FROM bprc WHERE binarypackagerelease IN (lots); [07:32] StevenK: so for that, we'll be more efficient with two tables. [07:32] With some possible bulk loading [07:32] StevenK: because 8 bytes are << 50 bytes. [07:33] StevenK: the whole 2-4M unique filenames can page into memory, this can't happen with the table you have today because of the 100's of M of rows will be huge. [07:34] Although we don't care if 100's of millions of filenames fit - we only care if the index fits. [07:34] A id, filename table, and BPRC links between them? [07:34] * stub tries to recall if each node in the btree contains the text [07:35] stub: agreed, but the PK index will be large (because it has to have every filename hundreds or thousands of times) [07:35] stub: yes [07:35] bah [07:35] StevenK: yes [07:35] I doubt it is thousands [07:35] debian + ubuntu + ppa builds [07:35] dailies of some packages, with a couple of series [07:36] That's still only 500 to 600 [07:36] Just stuff it in Cassandra and stop bothering me before my coffee has kicked in. [07:36] StevenK: I'm sure we will have hundreds; we may have thousands for some. [07:36] StevenK: 600 * 2M = 1.2B :) I don't think 600 is the common case though. [07:36] stub: :) [07:37] StevenK: the point is, you aren't building a search schema, you are building an export schema. [07:38] lifeless: If it's two tables, it's much harder to populate, too. [07:38] not at all [07:38] For some values of 'much'... [07:38] I can no longer be done in one query [07:38] Er, It [07:39] StevenK: So technically, it could, but we don't want to go there [07:42] I suspect lifeless is right in that we want a separate table. Its 100,000,000 integers + 2,000,000 strings vs 100,000,000 strings. I could do real calculations if we knew the average filename length. [07:43] I also concur with lifeless. [07:43] FWIW [07:44] 38 characters, picking on yelp [07:44] average or max? [07:44] StevenK: average length? [07:45] I suspect they're going to be well over than 4 characters :) [07:45] That is the average [07:45] wow. [07:45] if you want, I can dig up my conflict checker credentials and tell you the histogram of sizes for all ubuntu uploads [07:45] So yeah, we would break even around an average length of 3 I think... [07:45] stub: These are absolute FS paths. [07:45] So it's 3, we have a problem [07:45] Does that mess up the 2,000,000 unique guestimate? [07:46] No. [07:46] But it's at least 3.2 million. [07:46] Suggestions for a filename table? If I use BinaryPackageReleaseFilename, wgrant will hurt me. [07:46] But it's roughly around that order of magnitude. [07:46] I like Contents. [07:46] But I won't murder over Filename :) [07:47] StevenK: so the filename table is just filenames [07:47] Oh. [07:47] Right, that table. [07:47] * stub checks up on toast tables [07:47] Well, paths, not filenames. [07:47] sure [07:48] so anyhow, it's an optimisation table not a semantic table really today; BinaryPaths ? PackagePaths? whatever [07:48] Project windmill-db-devel build #395: STILL FAILING in 1 hr 4 min: https://lpci.wedontsleep.org/job/windmill-db-devel/395/ [07:49] stub: funky test failure [07:49] lifeless, stub: http://pastebin.ubuntu.com/627812/ [07:49] InternedString? :) [07:50] Hm? [07:50] wgrant: diaf ? :> [07:50] lifeless: :( [07:50] I was only partially joking. [07:50] I know. Uhm.... needs thought. [07:51] BinaryPackageReleaseContentsFilename.... bleh [07:51] and I have a virus or someshite [07:51] stub: BPRCF makes me sad [07:51] Should I be concerned that I immediately wondered if you were running Windows, and didn't consider it might be a biological entity? [07:51] I've gone with BinaryPackagePaths [07:51] It makes me sad two. In hindsight, there would be common acronyms we use like SPR, BPR, SPN [07:52] But Code destroyed SPR. [07:52] This gives us BPP and BPRC [07:52] They overloaded it [07:58] StevenK: how about PRC ? [07:58] * lifeless just wants to get country codes into the codebase [07:59] Package is ambiguous. [08:05] yes, but it would be such a cool acronym [08:07] "Meh" :-P [08:07] lifeless, stub: Changes pushed, diff updated [08:32] stub: I don't see those tests [08:32] lifeless: eh? [08:32] you had 3 failures [08:32] only one of the tests matches a -t parameter to bin/test [08:37] stub: anyhow, lib/canonical/launchpad/doc/unicode_csv.txt passes for me [08:37]