/srv/irclogs.ubuntu.com/2012/12/13/#launchpad-dev.txt

=== slank is now known as slank_away
timrcIt is possible to get the Ubuntu series from the build record? I don't see an obvious way to do it :(01:34
wgrantIndeed, apparently the distroarchseries attribute is not exported01:37
timrcdoh01:41
timrccurrent_source_publication.distro_series I think could work01:41
StevenKwgrant: This reached out and slapped me yesterday evening:01:44
StevenK     def test_displayarchs_for_copy_job_is_sync(self):01:44
StevenK-        # For copy jobs, displayarchs is "source."01:44
StevenK+        # For copy jobs, displayarchs is "sync."01:44
StevenKwgrant: Can triage https://bugs.launchpad.net/launchpad/+bug/1089615 , by the way?01:46
_mup_Bug #1089615: Source builds fail for packages with "3.0 (quilt)" format and unapplied patches <Launchpad itself:New> < https://launchpad.net/bugs/1089615 >01:46
wgrantStevenK: https://rt.admin.canonical.com/Ticket/Display.html?id=46345 will hopefully fix that01:56
StevenKwgrant: How are bzr-builder and 3.0 (quilt) related?02:00
timrcHm, is the "Date finished" part of the description for 'date_first_dispatched' a copy-paste error? (see, https://launchpad.net/+apidoc/devel.html#build)02:01
wgrantStevenK: Because that's an SPRB02:03
wgrantSO it uses bzr-builder02:03
wgranttimrc: Yes02:03
timrcwgrant, Thanks... one last question... does the time between 'datecreated' and 'date_first_dispatched' represent the time the build spends waiting to build?02:14
wgranttimrc: Usually, yes. But for private PPAs it will also include the publication latency, as the build will not be dispatched until its source is published02:15
timrcor would it be more accurate to use the package_upload.date_created timestamp inconjunction with 'date_first_dispatched'?02:15
timrcah, that's a useful bit of info02:15
wgrantThe source package_upload.date_created and build.datecreated should usually be identical for PPA packages02:16
timrcwgrant, Okay, thanks02:17
StevenKwgrant: http://pastebin.ubuntu.com/1428915/02:51
StevenKwgrant: Have you seen buildbot's waterfall? It looked like it lost its mind02:54
=== statik_ is now known as statik
wgrantStevenK: What was wrong with it?03:00
wgrantprasé had some disk issues and was offline for a while overnight03:00
StevenKwgrant: The pastebin? I don't like the line "+        if kind in ([], [u'']):"03:01
wgrantI was speaking of the waterfall03:02
wgrantLooking at the diff now03:02
StevenKAh03:02
wgrantWhy does that line exist?03:02
wgrantIt should never be called with an empty string03:03
StevenKBecause u''.split(' ') == [u''] :-(03:03
wgrantAh, right03:03
StevenKwgrant: Yes, it was masking a bug03:03
wgrantkind has to be the worst name ever03:03
StevenKIf I don't split and we call addBuild() twice, then searchable_names becomes 'a c f g h j newname'03:04
StevenKwgrant: s/kind/existing/ ?03:05
wgrantYes03:05
wgrantSo, you would perhaps do well to simply exclude elements that evaluate to False03:05
StevenK[u''] != False03:06
StevenK>>> u'' == False03:06
StevenKFalse03:06
wgrantThis isn't JavaScript or PHP03:07
wgrantBy "evaluate to" I mean in the context of a boolean expression03:07
wgrantie. an if statement or clause of a list comprehension03:07
StevenK>>> bool(u'')03:07
StevenKFalse03:07
wgrantYes03:07
StevenKHmmm, can't we that using filter() or map() ?03:08
wgrantIf you want, but I'd just say [x for x in existing if x]03:08
wgrantMostly because Guido hates lambdas03:09
StevenKConstruct a list from those elements of iterable for which function returns true. iterable may be either a sequence, a container which supports iteration, or an iterator. If iterable is a string or a tuple, the result also has that type; otherwise it is always a list. If function is None, the identity function is assumed, that is, all elements of iterable that are false are removed.03:09
StevenKfilter(None, ...)03:09
wgrantOh, didn't know about that special case03:10
wgrantHow hideous03:10
StevenKDoes that mean I can't use it? :-)03:11
wgrantYou can03:11
StevenK    def _appendSearchables(self, existing, new):03:12
StevenK        return sorted(filter(None, set(existing) | set(new)))03:12
StevenKBUILDBOT03:13
StevenKwgrant: Or is that function now hideous? :-)03:15
wgrantThat's fine03:16
nigelbWhat's the link for the LXC setup?03:28
nigelbAh, nvm. Found it.03:29
StevenKnigelb: apt-get install lpsetup ; lp-setup install-lxc or so03:30
nigelboh03:30
nigelbthat's all?03:30
lifelessnigelb: not quite03:31
lifelessnigelb: need to add the lp ppa first I think03:31
nigelboh right.03:31
nigelbbut I was in the middle of doing it the all fashioned way from the LXC page.03:31
wgrantI use lp-setup on my laptop. It seems to work.03:32
nigelbshould I stop that and do this? or does lp-setup accomplish the same thing?03:32
wgrantThey're pretty much equivalent03:32
wgrantlp-setup just sets things up slightly unconventionally and in a more automated fashion03:32
wgrantThe instructions on Running/LXC are what I use on my main dev machine, so I know they work fine :)03:32
nigelbok :)03:33
StevenKwgrant: Hmm, I am stupid and thought contains_string for searchable_versions would work :-(03:37
wgrantOh03:39
wgrantYou landed a DB patch without working out whether you could actually use it? :)03:39
StevenKSurely postgres has a function for that03:39
StevenKAnd then we SQL() in it03:40
wgrant'foover' = ANY(searchable_versions) with a GIN index should work, but testing now03:41
wgrantNope, the ANY doesn't get indexed03:46
wgrantSELECT * FROM packageupload WHERE ARRAY['1.2.3'] <@ searchable_versions;03:46
wgrantworks, though03:46
wgrant(the index now exists on DF)03:46
StevenKAnd no tgrm03:59
wgrantThe trgm index has been building for a few minutes04:00
wgrantMust be nearly done04:00
wgrantThere we are04:02
wgrantAs expected the queries aren't exactly fast, but it's better than what we have now04:02
* wgrant tries a composite04:03
nigelbThe LXC is lucid.04:29
StevenKIntended behaviour04:30
wgrantHopefully only for another month or so, though :)04:30
nigelbwoo.04:31
nigelbSo, I have python-virtualenv and virtualenv wrapper.04:31
nigelbThey aren't in Lucid yet :)04:31
nigelbOh, I still need to let launchpad take over my machine. just that it'll be the LXC that it'll take over.04:33
wgrantRight04:37
wgrantHmm04:40
wgrantI wonder how big the biggest queue is04:40
wgrantOh, <200k items04:40
wgrantSo it may not be worth indexing either of the search columns at all04:43
wgrantAlthough the table won't be hot04:44
StevenKwgrant: Oh?04:48
wgrantOh05:00
wgrantThe tsearch2 prefix matching stuff from years ago actually got merged05:00
* wgrant experiments05:00
wgrantSo we may be able to ts2 this nicely rather than having to hack up a custom GIN operator05:03
wgrant(this is a shortcut that will let us quickly do prefix matching on each name, which is probably roughly as useful as the current exact_match behaviour, but much much faster)05:03
StevenKNeat05:05
wgrant(GIN indices are basically just btree indices which know how to do stuff like split an array up into multiple keys, so they can obviously internally be used for prefix searching on keys. But utilising that functionality through SQL can be a problem, and you sometimes have to end up defining your own GIN operator classes.)05:08
StevenKwgrant: So, do we want to deploy?05:10
StevenKIt will drop us below 150 total criticals05:11
wgrantIs the db-stable merge through?05:13
wgrantShould be by now, I guess05:13
StevenKYeah, it is05:14
wgrant Total runtime: 55.598 ms05:15
wgrantBut does it actually work...05:15
wgrantYes05:15
wgrantWell that makes things easy05:16
wgrantAs long as cjwatson agrees we can live with just prefix matching on each name, rather than full substring matching05:17
StevenKwgrant: I have a deployment request ready, if you have no objections?05:17
wgrantStevenK: doit05:18
wgrantOh05:18
wgrantIt was only added in 8.405:18
wgrantSo I don't have to feel like a complete idiot :)05:19
wgrantStevenK: SELECT * FROM packageupload WHERE (to_tsvector('simple', searchable_names) @@ to_tsquery('simple', 'ubiquity:*')) AND distroseries = 103 AND archive IN (1, 534) AND status = 3;05:19
wgrantIs a prefix match on ubiquity05:20
wgrantDrop the :* for exact match05:20
StevenKHmmm05:20
StevenKNeat05:20
wgrant(the 'simple' makes lexeme conversion a no-op, so no stemming and such)05:20
=== Ursinha is now known as Ursinha-afk
wgrantAre the columns fully populated on DF?05:20
StevenKNo05:21
wgrant:(05:21
wgrantStill, this should be rather fast05:21
StevenKI think some of the customs probably have NULL versions05:21
wgrantAh05:21
wgrantBut the names should be populated?05:21
StevenKNot fully05:22
wgrant:(05:22
StevenK~ half the table is done05:22
StevenKI was planning to sorched earth the two of them and start it again05:22
StevenK*scorched05:22
StevenKAnd probably hack out the timeout05:23
wgrantI think a composite index might be more successful here05:24
wgrantTrying05:24
wgrant(prefixing the trigram index with (archive, distroseries) was not effective, simply because the index has so many entries)05:24
wgrantHm, no, doesn't work05:27
wgrantHow odd05:27
wgrantStill, fast enough without it :)05:27
StevenKUsing ts2?05:28
wgrantYes05:30
wgrant    "temp_pu5" gin (to_tsvector('simple'::text, searchable_names))05:31
StevenKAnd just GIN on searchable_versions?05:31
wgrantYeah05:32
wgrant    "temp_pu" gin (searchable_versions)05:32
wgrantIts selectivity estimates are even pretty good05:33
wgranthttp://paste.ubuntu.com/1429076/05:34
StevenKrows=0, though?05:36
wgrantBitmapAnd nodes lie sometimes05:36
wgrantLook at the end of the plan05:36
wgrant Bitmap Heap Scan on packageupload  (cost=1062.97..1424.58 rows=4 width=135) (actual time=5.810..5.835 rows=7 loops=1)05:36
wgrantrows=705:37
StevenKYeah05:37
StevenKThat looks pretty awesome05:37
StevenKexact match drops it to 3.3ms05:38
wgrantWell, yeah05:39
wgrantOnce you include a version match it's going to be pretty much trivial05:39
wgrantA long enough search string on a trigram index can take a couple of seconds, but I'd be surprised if this ever exceeded 100ms05:39
=== almaisan-away is now known as al-maisan
StevenKwgrant: So shall I scorched earth searchable_* and garbo them up?05:42
StevenKWhich I think requires the indicies die05:42
wgrantThese indices shouldn't have a huge impact on performance, but it's probably still worth it to nuke them05:43
wgrantgarbo performance, that is05:43
wgrantThey only take about 5 minutes to regen afterwards, so feel free to delete temp_*05:43
StevenKwgrant: I was going to DROP COLUMN/ADD COLUMN because it's MUCH faster05:43
wgrantAh, right05:44
wgrantk05:44
wgrantHmm05:47
wgrantWe may actually want to skip to_tsquery and to_tsvector at this point05:47
wgrantAnd instead just cast directly to tsquery and tsvector05:47
wgrantSince we don't eg. want to separate lexemes by -05:47
StevenK2012-12-13 05:48:20 DEBUG2  [PopulatePackageUploadSearchables] Iteration 4 (size 26.8): 0.658 seconds05:48
wgrantHopefully it'll accelerate...05:49
StevenKIt will, it just takes a while05:49
wgrantJust be glad we're no longer doing that inline in searches :)05:49
StevenKHaha05:49
wgrantStevenK: Hm, it seems to be pretty slow06:01
wgrantOnly 15k done06:01
StevenKYeah06:01
StevenKIf last run is anything to go by, the first 800 iterations are very slow06:02
StevenKThen it speeds up massively06:02
wgrantO_o06:02
StevenK2012-12-13 06:04:04 DEBUG2  [PopulatePackageUploadSearchables] Iteration 264 (size 144.6): 4.541 seconds06:04
wgrantHm06:08
wgrantStill surprisingly awful06:08
wgrantBut I guess it'll get there ventually.06:08
wgrantOnly 4.6 million to go06:08
wgrantStevenK: Oh06:17
StevenKOh?06:17
wgrantShould be faster now06:18
wgrantBy a factor of several06:18
wgrantCan you confirm?06:18
StevenKIndeed, it just sped up06:18
StevenK2012-12-13 06:18:37 DEBUG2  [PopulatePackageUploadSearchables] Iteration 523 (size 1284.2): 4.860 seconds06:18
wgrantLovely06:18
wgrantThat's more like it06:18
StevenKAnd you did what?06:18
wgrantMagic06:18
wgrantaka. ANALYZE06:19
StevenKANALYZE packageupload ?06:19
wgrantThe stats for the nullness of searchable_* were bad06:19
wgrantYes06:19
StevenKOh06:19
StevenKRight06:19
wgrantSo it was doing a seqscan to find them06:19
StevenKBecause I've been bad06:19
wgrantRather than an index scan on id06:19
wgrantRight, that's a little bit faster06:20
StevenKRight. Happily, that will affect us not at all on prod.06:20
wgrantUm06:20
wgrantIt might06:20
wgrantBut a vacuum would probably have fixed it in an hour or so anyway06:20
StevenKIf we deploy 40-1 tonight, the earliest the garbo job will hit prod is tomorrow morning06:21
wgrantSure06:21
* StevenK waits for staging to finish building06:24
=== StevenK changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: <150
cjwatsonwgrant: hmm.  I've definitely used full non-prefix substring matching in the not too distant past.  I'm not certain whether all of those were purely workarounds for lack of SPN.09:03
wgrantI can't think of many circumstances in which it would actually be useful, assuming all the names are searchable09:05
cjwatsonMostly surprising because all other non-exact matches in LP (AFAIK) are substring, not prefix09:11
wgrantThere aren't very many non-exact matches in LP09:14
wgrant(and most of those that do exist should not :))09:14
wgrantThere's no real use-case for getPublishedSources(exact_match=False), for example09:14
wgrantFull substring matches are extremely expensive09:15
cjwatsonSo, we'd have to document it, and it's a tad surprising, but we can live without full substring matches given the other improvements09:17
wgrantMmm09:19
wgrantIt's not really surprising09:19
wgrantOnly because people didn't think at all before introducing substring matching in the first place09:19
cjwatsonIt's potentially surprising to people who've been using it09:20
cjwatsonWhich I don't think is often, but now and again09:20
wgrantRight09:20
cjwatsonMostly in "oh, I need to reject these three packages and they happen to be the only ones containing 'splotnib'" kinds of ways, probably09:20
wgrantHopefully everyone will now ask themselves "is this is any way not stupid" when designing their next webservice API :)09:21
wgrant'cause in any database context, substring matching is pretty much entirely impractical and not all that useful :(09:22
wgrantI dream that one day the argument can be removed from getPublishedSources and getPublishedBinaries, perhaps letting those API methods actually be possibly sustainable.09:28
jtvjam: got a moment for a maas-hardware-db question?11:15
jtvWant to know how you pull the lshw output out of the maas database.  Is it all in the maas codebase, or do you have something else somewhere?11:16
jtvOh, is it the special case for "01-lshw" that's in the metadataserver api.py?11:17
jtvallenap: think I found the answer to your question ^  :)11:17
jtvIt's in the diff.11:18
allenapjtv: Ta!11:30
mgzjtv: it's in maas, basically there's an api for getting the lshw output, which exists as the db is on the region controller but we want the cluster controllers to do the work11:33
jtvmgz: I guess it's the bit in the metadata API then, where the node calls "signal" and the API has a special-case check for "is the name of the file 01-lshw.out?"11:35
mgzthat's for putting it *into* the db11:36
jtvWe may be talking about different "the" dbs.11:59
* jtv resists the temptation to get pedantic about the definition of Database11:59
jtv(Would have been a cheap dig at mongo)12:00
=== Ursinha-afk is now known as Ursinha
=== yofel_ is now known as yofel
timrcsinzui, Hi, out of curiosity, do proprietary and embargoed projects hide membership details now as well?14:02
sinzuitimrc, team are not proprietary/embargoed14:04
sinzuitimrc, if you mean Private teams, no one can see them by default, if they are placed in a  public relationship like a bug subscription, only the Launchpad id, name, and icons are revealed14:05
sinzuiso members are never shown to non-private-team-members14:06
timrcsinzui, ok14:09
timrcsinzui, so if a project is set to proprietary or embargoed and the client attempting to access it is not authenticated or authorized will it 404?14:15
sinzuiIt will 404...it does not exist to non-shared users14:15
timrcsinzui, good to know, that's what I'd expect14:19
=== jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: jcsackett | Firefighting: - | Critical bugs: <150
=== jcsackett changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On-call reviewer: - | Firefighting: - | Critical bugs: <150
=== teknico_ is now known as teknico
=== slank_away is now known as slank
=== gary_poster is now known as gary_poster|away
=== gary_poster|away is now known as gary_poster
=== gary_poster is now known as gary_poster|away
=== gary_poster|away is now known as gary_poster
sinzuioh sweet, Lp will call focus() on <input type="hidden"/>16:50
jmlsinzui: :D17:23
=== deryck is now known as deryck[lunch]
timrcIt's known (or maybe it isn't) that our use of PPAs is a little unorthodox.  Specifically we really only use them to build packages.  When the package is built and published, it gets funneled into another repository which in turn is used to build Ubuntu images.19:39
timrcOne architectural problem with this design is that a user has the ability to upload or copy-rebuild the same package in multiple PPAs.  If they do this, it will cause a package collision "downstream", as the archive they're going to has a shared pool, and so only one copy of the package can exist there.19:40
timrcWhat would be nice, for us, at least, is if we could at least disable copy-rebuild.  That does not completely solve the problem, but it solves the most common cause for us.19:41
timrcDoes anyone have outright objections to doing this?19:41
=== deryck[lunch] is now known as deryck
=== ev_ is now known as ev
=== micahg_ is now known as micahg
lifelesssinzui: radical honesty in bugs? zomg :)21:59
sinzuiyes. the number is true22:00
lifelesssinzui: oh, I know. I'm just waiting for the slashdot now :)22:00
sinzuiI doubt the 90 days remaining to get the list to zero though. Maybe I am too pessimistic. I fixed 7 last week, so there are still 1-day bug fixes22:01
timrcsinzui, So, we have Bug sharing policy set to "Proprietary, can be public" for a project and they want to use 'also effect another project' option but can't... this seems odd if the bugs are configured to share the same way for both projects... what am I missing?22:12
timrchm actually the other project has the Bug sharing policy as "Proprietary" not "Proprietary, can be public"22:13
sinzuitimrc, I think you found the problem22:15
timrcsinzui, thanks and sorry for disturbing ya :)22:20
maxbtimrc: Re your PPA thing above.... surely this comes down to ensuring the users understand the tools?22:21
timrcmaxb, you'd think22:21
timrcmaxb, it's been an on going problem for us for the last 2 years... it turns out people don't like to read documentation and / or periodic PSA's on our lists are not enough22:22
timrcit is sort of a weird thing to consider when using a PPA, so I don't exactly blame people for doing it... also the default is to rebuild, so the opportunity for human error is greater22:23
maxbCopying packages between PPAs isn't usually something novice packagers do - what's different about your workflow that makes it more likely that people do it without the experience to understand the consequences?22:25
maxb(Though I do agree that the default is probably the wrong way around)22:25
sinzuiwgrant, StevenK, Do you want to deal with this: https://answers.launchpad.net/launchpad/+question/21666522:41
StevenKwgrant: SELECT regexp_matches(json_data, 'package_version": "([^"]+)') FROM packagecopyjob;23:04
StevenKSELECT regexp_matches(json_data, 'package_version": "([^"]+)"') FROM packagecopyjob;23:13
StevenKNow we can scare wallyworld_23:13
StevenKSELECT regexp_matches(json_data, '"package_version": "([^"]+)"') FROM packagecopyjob;23:13
wallyworld_i'm not scared yet23:13
StevenKwallyworld_: regex matches on a JSON field in a SQL statement? :-)23:14
wallyworld_is regexp matching new?23:14
StevenKNope23:14
StevenKWas around in 8.3, at least23:14
wallyworld_does lp use it much?23:14
StevenKNot at all, I think.23:15
wallyworld_but it does now :-)23:15
StevenKFor a short time. That is going into a garbo job.23:15
wgrantSELECT (regexp_matches(json_data, '"package_version": "([^"]+)"')::debversion[])[1] FROM packagecopyjob;23:24
wgrantStevenK: Indices done23:37
wgrantEXPLAIN ANALYZE SELECT * FROM packageupload WHERE (searchable_names::tsvector @@ 'ubiquity:*') AND distroseries = 103 AND archive = 1 AND status = 3 AND searchable_versions @> ARRAY['2.1.0'];23:37

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!