[00:16] cjwatson: That seems too easy. [00:17] Where does the other 30 seconds go, though? [00:17] Mars. [00:17] http://www.thirtysecondstomars.com/ <-- Proof. [00:20] Heh [09:09] wgrant: it spends a fair bit of time in _find_all_versions for each requirement (i.e. the "Collecting" steps) [09:16] I would like a way to say "don't use setuptools.file_finders no matter what" in LP's setup.py [09:17] because we care not about the sdist and having to patch setuptools-git is a ridiculous distraction [12:18] wgrant: But once the virtualenv is populated, it's 3.5 seconds to rerun, which I guess is actually more analogous to buildout's behaviour with a symlink to a populated eggs directory. [12:23] Oh, indeed, without the eggs symlink, buildout takes 155 seconds vs. the current figure I have for a full virtualenv build with pip of 52 seconds (my previous figure was badly measured, I missed a bit at the start) [12:24] So performance is not an issue any more and I just need to sort out the large pile of details [12:31] The pip figure is actually cheating because I had cached wheels; without those it's 164 seconds. But that's sufficiently close to the comparable buildout run that I don't care [12:35] I found a reasonably horrible but just about tolerable way to cut setuptools-git out of the loop in our setup.py [20:45] lifeless: How would I go about debugging 'pip install "zope.security[untrustedpython]"' not actually installing the untrustedpython extra in my in-progress pip conversion of LP? [20:45] (this is, hope you understand, several layers deep in a rabbit hole, but it's where I've got to thus far ...) [20:47] pdb ? [20:47] try pip install -v ... first [20:48] if that doesn't enlighten [20:48] show me a pastebin [20:48] http://paste.ubuntu.com/12144491/ [20:49] I was hoping for an answer other than pdb, but I guess I can give that a try ;-) [20:50] should be installing RestrictedPython, but isn't; works fine if I explicitly request RestrictedPython [20:51] zope.security's metadata.json looks right, contains {"requires": ["RestrictedPython"], "extra": "untrustedpython"} [20:52] so if this isn't just something that plain doesn't work yet in pip, I can certainly attack with pdb, just wanted to check first since that's likely to take me a while [20:52] I know you've been working on various corners of pip's requirement handling [20:52] I'm not at all sure that metadata.json is consuled [20:53] I'd need to check the embedded pkgresources - I think it may still be reading METADATA [20:53] METADATA has [20:53] Provides-Extra: untrustedpython [20:53] Requires-Dist: RestrictedPython; extra == 'untrustedpython' [20:54] so it works for me :) [20:54] Successfully installed RestrictedPython-3.6.0 zope.component-4.2.2 zope.event-4.0.3 zope.i18nmessageid-4.0.3 zope.interface-4.1.2 zope.location-4.0.3 zope.proxy-4.1.6 zope.schema-4.4.2 zope.security-4.0.3 zope.untrustedpython-4.0.0 [20:56] hmm, I wasn't using a constraints file [20:56] whats in your constraints file ? [20:56] zope.security 4.x goes through a different chain, but it does seem to work with a fresh virtualenv, let's see [20:57] http://paste.ubuntu.com/12144571/ [20:58] oh and also an initial run before that with http://paste.ubuntu.com/12144573/ as -r [20:58] cjwatson: are you capturing both stderr and stdout ? [20:58] yes [20:58] hmm [20:58] I'm not seeing 'collecting' [20:59] that was in an env that already had lp installed [20:59] -at all- [21:00] yeah, its the constraints file [21:00] workaround - put the [] in the constraints file [21:00] I'll file a bug now [21:01] [I wrote the constraints support for pip :), so was a very good choice to ask] [21:01] aha, the constraints implies a null set of extras? [21:01] no [21:01] its worse [21:01] brilliant, thanks - I thought you might be [21:01] pip doesn't merge together extras references from different places in the dependency graph [21:01] which is a subset of the 'does not do resolving at all' [21:02] problem [21:02] I'm using 7.1.0 FWIW but 7.1.1's changelog doesn't look different here [21:04] and confirmed that adding the [] in the constraints file fixes it, thanks [21:06] https://github.com/pypa/pip/issues/3046 [21:07] yw [21:08] now I wonder how to grep for what extras *should* be installed - I guess just compare with the buildout-installed package list [21:08] well, they'll be getting in the dependency list somehow [21:09] so I think a two-phase thing [21:10] phase 1, any extras you define [21:10] phase 2, interrogate the dist-requires of all the things you depend on, transitively [21:10] you could automate phase2 by runnign 'pip install' of each thing you directly dep on separately, and looking in the output: [21:11] perhaps [21:11] * lifeless shrugs ;) [21:14] I just did the stupid thing and did bin/buildout -vc buildout.cfg, mangled output into roughly compatible form, compared with pip list [21:14] some noise from buildout 1's semi-isolation vs. virtualenv --system-site-packages just allowing everything [21:15] auditing the final output is probably sufficient [21:15] you might end up there coincidentally and be missing a extra for the future [21:15] yeah, I really only have to do this once [21:15] true [21:15] but, we're going to get that bug fixed [21:15] so its a time limited window [21:15] once we're on virtualenv/pip, it's a lot easier to upgrade