[00:16] <wgrant> cjwatson: That seems too easy.
[00:17] <wgrant> Where does the other 30 seconds go, though?
[00:17] <Spads> Mars.
[00:17] <Spads> http://www.thirtysecondstomars.com/ <-- Proof.
[00:20] <wgrant> Heh
[09:09] <cjwatson> wgrant: it spends a fair bit of time in _find_all_versions for each requirement (i.e. the "Collecting" steps)
[09:16] <cjwatson> I would like a way to say "don't use setuptools.file_finders no matter what" in LP's setup.py
[09:17] <cjwatson> because we care not about the sdist and having to patch setuptools-git is a ridiculous distraction
[12:18] <cjwatson> wgrant: But once the virtualenv is populated, it's 3.5 seconds to rerun, which I guess is actually more analogous to buildout's behaviour with a symlink to a populated eggs directory.
[12:23] <cjwatson> Oh, indeed, without the eggs symlink, buildout takes 155 seconds vs. the current figure I have for a full virtualenv build with pip of 52 seconds (my previous figure was badly measured, I missed a bit at the start)
[12:24] <cjwatson> So performance is not an issue any more and I just need to sort out the large pile of details
[12:31] <cjwatson> The pip figure is actually cheating because I had cached wheels; without those it's 164 seconds.  But that's sufficiently close to the comparable buildout run that I don't care
[12:35] <cjwatson> I found a reasonably horrible but just about tolerable way to cut setuptools-git out of the loop in our setup.py
[20:45] <cjwatson> lifeless: How would I go about debugging 'pip install "zope.security[untrustedpython]"' not actually installing the untrustedpython extra in my in-progress pip conversion of LP?
[20:45] <cjwatson> (this is, hope you understand, several layers deep in a rabbit hole, but it's where I've got to thus far ...)
[20:47] <lifeless> pdb ?
[20:47] <lifeless> try pip install -v ... first
[20:48] <lifeless> if that doesn't enlighten
[20:48] <lifeless> show me a pastebin
[20:48] <cjwatson> http://paste.ubuntu.com/12144491/
[20:49] <cjwatson> I was hoping for an answer other than pdb, but I guess I can give that a try ;-)
[20:50] <cjwatson> should be installing RestrictedPython, but isn't; works fine if I explicitly request RestrictedPython
[20:51] <cjwatson> zope.security's metadata.json looks right, contains {"requires": ["RestrictedPython"], "extra": "untrustedpython"}
[20:52] <cjwatson> so if this isn't just something that plain doesn't work yet in pip, I can certainly attack with pdb, just wanted to check first since that's likely to take me a while
[20:52] <cjwatson> I know you've been working on various corners of pip's requirement handling
[20:52] <lifeless> I'm not at all sure that metadata.json is consuled
[20:53] <lifeless> I'd need to check the embedded pkgresources - I think it may still be reading METADATA
[20:53] <cjwatson> METADATA has
[20:53] <cjwatson> Provides-Extra: untrustedpython
[20:53] <cjwatson> Requires-Dist: RestrictedPython; extra == 'untrustedpython'
[20:54] <lifeless> so it works for me :)
[20:54] <lifeless> Successfully installed RestrictedPython-3.6.0 zope.component-4.2.2 zope.event-4.0.3 zope.i18nmessageid-4.0.3 zope.interface-4.1.2 zope.location-4.0.3 zope.proxy-4.1.6 zope.schema-4.4.2 zope.security-4.0.3 zope.untrustedpython-4.0.0
[20:56] <lifeless> hmm, I wasn't using a constraints file
[20:56] <lifeless> whats in your constraints file ?
[20:56] <cjwatson> zope.security 4.x goes through a different chain, but it does seem to work with a fresh virtualenv, let's see
[20:57] <cjwatson> http://paste.ubuntu.com/12144571/
[20:58] <cjwatson> oh and also an initial run before that with http://paste.ubuntu.com/12144573/ as -r
[20:58] <lifeless> cjwatson: are you capturing both stderr and stdout ?
[20:58] <cjwatson> yes
[20:58] <lifeless> hmm
[20:58] <lifeless> I'm not seeing 'collecting'
[20:59] <cjwatson> that was in an env that already had lp installed
[20:59] <lifeless> -at all-
[21:00] <lifeless> yeah, its the constraints file
[21:00] <lifeless> workaround - put the [] in the constraints file
[21:00] <lifeless> I'll file a bug now
[21:01] <lifeless> [I wrote the constraints support for pip :), so was a very good choice to ask]
[21:01] <cjwatson> aha, the constraints implies a null set of extras?
[21:01] <lifeless> no
[21:01] <lifeless> its worse
[21:01] <cjwatson> brilliant, thanks - I thought you might be
[21:01] <lifeless> pip doesn't merge together extras references from different places in the dependency graph
[21:01] <lifeless> which is a subset of the 'does not do resolving at all'
[21:02] <lifeless> problem
[21:02] <cjwatson> I'm using 7.1.0 FWIW but 7.1.1's changelog doesn't look different here
[21:04] <cjwatson> and confirmed that adding the [] in the constraints file fixes it, thanks
[21:06] <lifeless> https://github.com/pypa/pip/issues/3046
[21:07] <lifeless> yw
[21:08] <cjwatson> now I wonder how to grep for what extras *should* be installed - I guess just compare with the buildout-installed package list
[21:08] <lifeless> well, they'll be getting in the dependency list somehow
[21:09] <lifeless> so I think a two-phase thing
[21:10] <lifeless> phase 1, any extras you define
[21:10] <lifeless> phase 2, interrogate the dist-requires of all the things you depend on, transitively
[21:10] <lifeless> you could automate phase2 by runnign 'pip install' of each thing you directly dep on separately, and looking in the output:
[21:11] <lifeless> perhaps
[21:11]  * lifeless shrugs ;)
[21:14] <cjwatson> I just did the stupid thing and did bin/buildout -vc buildout.cfg, mangled output into roughly compatible form, compared with pip list
[21:14] <cjwatson> some noise from buildout 1's semi-isolation vs. virtualenv --system-site-packages just allowing everything
[21:15] <lifeless> auditing the final output is probably sufficient
[21:15] <lifeless> you might end up there coincidentally and be missing a extra for the future
[21:15] <cjwatson> yeah, I really only have to do this once
[21:15] <cjwatson> true
[21:15] <lifeless> but, we're going to get that bug fixed
[21:15] <lifeless> so its a time limited window
[21:15] <cjwatson> once we're on virtualenv/pip, it's a lot easier to upgrade