=== micahg_ is now known as micahg
cjwatsonwgrant: the setup_requires thing you mention in https://code.launchpad.net/~stub/launchpad/trivial/+merge/245822 - is that that zc.buildout just doesn't support it?02:21
cjwatsonI've been experimenting with some upgrades lately, and seem to be going round in circles.  ZTK 2.0a1 plus latest zc.buildout has trouble because site-packages isolation has gone away.  Twisted >= 14.0.0 has trouble because it ideally wants new pyOpenSSL and that conflicts with a system package, but even if I ignore that, it hits https://github.com/testing-cabal/testtools/pull/149.  Even once that or similar is merged, upgrading to ...02:25
cjwatson... current testtools requires new pbr, which requires the swiftclient upgrade, and so we're back to buildout ...02:25
cjwatsonAnd if we're ever to fix Twisted-based SSH servers to support OpenSSH 7.0 default config, we're going to have to upgrade Twisted at some point02:26
cjwatsonMaybe we have to bite the bullet and convert to virtualenv and pip, perhaps with multiple virtualenvs so that we can have some of them configured with system site-packages which we seem to need02:28
lifelesswhat site packages do you need ?03:05
wgrantlifeless: python-debian, python-apt, IIRC04:00
wgrantcjwatson: zc.buildout can't support it.04:01
wgrantI believe I have a workaround for pbr, but I would have to look up my notes.04:01
wgrantBut basically setup_requires in its current implementation is the worst idea ever, unfortunately.04:01
StevenKWorse than Zope 3?04:04
wgrantZope 3 is actually possible to use.04:04
lifelesswgrant: whats the pbr problem ?04:13
wgrantlifeless: setuptools has a PyPI fetish04:14
wgrant"oh there's a setup_requires that isn't installed when you import setup.py? i guess i'll just go and download it from PyPI and install it for you that sounds like a good idea doesn't it"04:14
wgrantIf you can't convince buildout to install pbr first, setuptools will try to install pbr itself.04:15
wgrantFirewalls (and good security practice) disagree with that.04:16
lifelessyou could write a recipe04:20
lifelesslike the one in the mock bug I was whinging about earlier04:20
cjwatsonwgrant: https://pip.pypa.io/en/latest/reference/pip_install.html#controlling-setup-requires might be helpful, perhaps?09:53
wgrantcjwatson: Yeah, .pydistutils.cfg works09:53
wgrantIt is rather unfortunate, however.09:53
wgrantcjwatson: If we were using pip, though, we could just convince it to install pbr beforehand.09:57
wgrantWe don't have that option with setuptools, unless we like add a special eggs-setup config key to the recipes we use.09:58
cjwatsonWe'd probably want .pydistutils.cfg anyway for robustness, but switching to pip is feeling increasingly like a good idea.09:59
cjwatsonNot a trivial amount of work though.10:00
wgrantIt's easier if we can kill python-apt entirely :P10:01
cjwatsonAre you sure that's the only one?10:02
cjwatsonAlso python-debian comes from eggs, not a system package10:02
wgrantIt's by no means the only one.10:02
wgrantBut it's the only really weird one I know of.10:03
wgrantWell, other than tickcount.10:03
cjwatsonIt's kind of a zero or any thing.10:04
lifelesswgrant: or add a recipe10:05
wgrantIt is.10:05
wgrantBut python-apt is the only really weird thing.10:05
wgrantlifeless: Indeed, it's looking tempting to use a custom buildout or z3c.recipe.scripts in the short term.10:05
cjwatsonNot z3c.recipe.scripts please, that doesn't work with buildout 2.10:06
cjwatsonAt least we need a way to migrate to something more like zc.recipe.egg10:06
wgrantI think we may only need that for the weird stuff we do with site-packages, but I forget.10:08
wgrantIt's been a while since I've looked at buildout in depth.10:09
cjwatsonYes, I believe z3c.recipe.scripts is just for site-packages mangling10:09
wgrantDoes anything stop us from using virtualenv+pip once we no longer need site-packages mangling?10:10
lifelesshttps://github.com/testing-cabal/mock/issues/310 may provide some inspiration for you10:10
wgrantWe use a consistent versions.cfg, so the isolation between txlongpoll etc. and LP is not critical.10:10
cjwatsonDoes anything stop us from using virtualenv+pip even while we do still need site-packages mangling?  We could use multiple virtualenvs.10:11
cjwatsonOne with --system-site-packages and one without.10:11
wgrantBut we can't install the new Twisted in a system-site-packages virtualenv, can we?10:11
wgrantSo there's not much point, other than as an hour-long migration strategy.10:11
wgrantI don't have an issue with it, I just don't see how it solves the problem.10:13
cjwatsonWe ... can't?  http://paste.ubuntu.com/12123638/10:14
wgrantcjwatson: What was the pyOpenSSL issue?10:15
wgrant"Twisted >= 14.0.0 has trouble because it ideally wants new pyOpenSSL and that conflicts with a system package"10:15
cjwatsonlifeless: I can see how that would be helpful, though the fact that it would have to be based on z3c.recipe.scripts for buildout 1 and zc.recipe.egg for buildout 2 makes migration rather entertaining10:15
cjwatsonwgrant: Stuff like http://paste.ubuntu.com/12123660/ which turns out to be because python-openssl 0.12-1ubuntu2.1 is installed10:17
cjwatsonAnd indeed is a dep of launchpad-dependencies10:17
cjwatsonThe scripts stanza in buildout.cfg uses system site-packages.10:18
wgrantSo that's why we can't install the new Twisted in a system-site-packages virtualenv.10:18
wgrantOr something.10:18
wgrantYour motivation for moving away from system-site-packages was the pyOpenSSL issue.10:18
wgrantI thought.10:18
cjwatsonIt doesn't work in a buildout 2 environment with no virtualenv at all, but it works fine in a system-site-packages virtualenv.10:19
wgrantAhh, I see.10:19
cjwatsonWell, I need libffi-dev or something for it to actually install here, but at least pip seems to resolve things fine.10:20
lifelesspyopenssl works fine in a pure virtualenv too10:20
lifelessthe pip cache can mitigate the compilation time10:20
cjwatsonIt would probably work with buildout 2 and appropriate virtualenvs, but I'm not sure doing it that way round is easier than just migrating to pip first10:20
wgrantYep, that might be reasonable.10:21
cjwatsonI mean, that seems like strictly greater work.10:21
cjwatsonAt least if we plan to use pip eventually.10:22
wgrantI was just confused as to why you provided the site-packages stuff as a motivation for not moving to zc.buildout 2.0, and then proposed switching to a system-site-packages virtualenv in its place.10:22
cjwatsonRight, it's the lack of any virtualenv-like facility (even buildout 1's limited isolation support) in buildout 2 that causes the problem.10:22
cjwatsonbuildout 2 says to use virtualenv if you need that.10:23
wgrantWell, it says to use virtualenv if you need isolation.10:23
wgrantBut you're suggesting using it even if we *don't* need isolation.10:23
wgrantWith evidence, which seems reasonable.10:23
cjwatsonOh, right, yes.10:23
wgrantIt just seemed totally insane without that critical piece of rationale.10:23
cjwatsonActually it must be a problem even in buildout 1, thinking about it.10:24
cjwatsonBecause my attempt to upgrade twisted didn't include my attempt to upgrade buildout.10:24
cjwatsonI believe that we currently have pyOpenSSL in versions.cfg but do not actually rely on buildout to install it.10:24
cjwatsonThe problem comes as soon as something depends on a newer version.  So we could alternatively backport a newer .deb, I suppose10:25
wgrantYeah, or we could just move to pip with system-site-packages since it seems to sidestep it.10:26
wgrantexclude-site-packages is simply a sensible default; I doubt txlongpoll/txpkgupload actually require it.10:26
wgrantSo we can probably get away with one.10:26
cjwatsonFair point10:27
cjwatsonSo the next worry is how to cope with all the magic on praseodymium10:29
cjwatsonI wonder if I have a copy of the relevant bits anywhere10:29
wgrantMost of that magic is actually on carob10:30
wgrantRecall that carob builds the tree, including all the eggs, then runs "make clean_buildout"10:31
wgrantI'm not sure carob even builds the tree itself, since we used to have i386 machines.10:31
wgrantNot sure *praseodymium* even builds the tree10:31
wgrantIIRC everything uses make targets, except for the weird config-manager bits.10:32
wgrantBut they don't directly affect this.10:32
cjwatsoncarob does a "make build_eggs"; what deals with syncing the eggs directory around?10:32
wgrantThe eggs directory is a subdirectory of the tree.10:33
cjwatsonSince that would presumably need to do the equivalent for a pip cache10:33
wgrantSo it just gets synced along with it.10:33
wgrant(this is why make sync-stable sometimes fails hilariously; the tree build on carob is by no means atomic)10:33
cjwatsonOh, right, it's only a symlink in development setups10:33
wgrantI forgot rf-setup did it that way.10:33
wgrantI don't.10:33
cjwatsonAnd AIUI our deployment-manager configs run "make compile" or equivalent on each target host, so in the new world order that would install stuff from the pip cache into an appropriate virtualenv.  And we would need something site.py-like to cause our scripts to stick themselves into the right virtualenv.10:37
lifelessI'd seriously consider building the virtualenv to go and then syncing it out rather than building on each host10:38
cjwatsonThe make targets I see that would now have silly names and need to have compatibility preserved at least for a transition period are build_eggs and clean_buildout10:38
cjwatsonlifeless: Right, make compile could do nothing if it was there10:38
wgrantlifeless: We could do that now, since the last i386 machines are dead10:39
wgrantBut that only happened a year ago :/10:39
cjwatsonWould it cause any problem in the case of heterogeneous Ubuntu releases?10:39
cjwatsonOr Python versions?10:39
cjwatson(Sure, everything's 2.7 now, but I don't want to put further roadblocks in the path of 3.x)10:40
wgrantI guess our eggs dir is sort of like a wheel cache.10:40
wgrantBut I'm not sure wheels can be deployed for everything yet, can they?10:40
lifelesswheels aren't perfect10:40
lifelessbut the cache and shipping it around is doable10:41
lifelesshowever my point is more about deploy reliability10:41
lifelessyou're deploying to enough machines10:41
cjwatsonI would be very happy to reduce the time taken by make compile during deployments, if nothing else.  But I don't want to leave landmines in the process.10:41
lifelessthat its worth the complexity of a build+distribute mechanism rather than distributed builds10:41
wgrantmake compile is mostly slow because half our machines are piles of crap.10:41
wgrantmake compile on a fast machine is pretty quick.10:42
lifelessas far as moving to new pythons goes10:42
cjwatsonI'm not at all arguing against build+distribute, just trying to make sure I understand the issues.10:42
lifelessvirtualenvs are not particularly portable10:42
lifelesseven minor cpython builds can break them10:42
cjwatsonNot all our deployment targets have the same paths, either.10:43
lifelessso the requirement to do build+distribute is that for each target machine you have the same python locally10:43
cjwatsonAnd the virtualenv docs caution against expecting path identity10:43
lifelessand yeah, their paths are embedded in the virtualenv10:43
lifelessbut a container can address both things easily10:44
cjwatsonThis is really sounding like a later step that we should make sure we don't design ourselves out of, rather than something to do in the first run; where I came in was trying to cut Gordian knots10:45
cjwatsonI mean, not dodging the issue or anything, just trying to avoid having to change All The Things at once10:46
lifelessI get it10:46
lifelessI'm just surfacing options10:46
lifelessrunning pip on a prod server with --no-index -f <path> should be reliable as long as you've neutered easy-install via .pydistutils.cfg10:47
wgrantYeah, I was hoping to avoid that.10:47
lifelessno, you must neuter it10:47
lifelessotherwise you'll be auditing forever10:47
wgrantThere was discussion a while ago about making pip pass options down so easy_install would be neatured10:47
wgrantbut I haven't seen whether that actually got done10:47
lifelessthere are no such options10:47
lifelessand pip can't do it until we have declarative metadata for setup_requires used by *every package including old releases*10:48
lifelesse.g. never10:48
lifelessthere is a route that might work10:48
lifelesswhich is to make it error out with the missing dep10:48
wgrantRight, easy_install had, or was going to grow, options for allow-hosts and find-links10:48
lifelessand have pip parse the output10:48
wgrantWhich would be respected.10:48
cjwatsonWe can put the pydistutils.cfg in the virtualenv, I think?10:49
cjwatsonWhich seems a little easier to arrange than either system-wide or user-wide configuration10:49
wgrantOh, can we?10:49
lifelessyou can always export a custom HOME10:50
cjwatsonOr even setup.cfg, according to the docs10:50
wgrantThat seems too easy.10:50
cjwatsonOr is that setup.cfg of the package being processed?10:50
lifelesssetup.cfg isn't accessible to you10:50
lifelessbecause yes10:50
cjwatsonRight, bah.10:50
wgrantlib/python2.7/distutils/distutils.cfg might work10:50
wgrantBut I doubt it.10:50
wgrantAgain, too easy.10:50
cjwatsonvirtualenv writes out a template file that claims it works.10:52
cjwatsonAlso /path/to/env/.pydistutils.cfg, according to the changelog for virtualenv 1.3.310:54
cjwatsonAllegedly, though I'm not seeing that in the code.10:55
cjwatsonOh, yes I do, in the source.  Couldn't find it in the installed virtualenv.py because it's helpfully zlib-compressed and base64-encoded.11:04
wgrantI don't know why people also don't xor things with "Beware of the Leopard" while they're at it.11:05
cjwatsonSo env/.pydistutils.cfg seems by far the easiest11:06
wgrantif it works, indeed that is amazing.11:07
cjwatsonwgrant: http://paste.ubuntu.com/12124302/  actually a success for .pydistutils.cfg!12:24
cjwatson(that "succeeds" if I remove .pydistutils.cfg, so correctly blocked.)12:24
wgrantI knew ~/.pydistutils.cfg worked, but if the virtualenv one works too then all the better.12:25
cjwatsonRight, this is the virtualenv one.12:26
cjwatsonOK, that can actually install everything - just needed a couple of wheel conversions and suchlike.  The next steps are figuring out how to handle sitecustomize and buildout-templates.12:45
wgrantcjwatson: Does "everything" include our dpkg deps?12:45
cjwatsonwgrant: They're accessible from virtualenv/bin/python, if that's what you mean12:46
wgrantEr of course12:47
wgrantSorry, neutron ate my brain.12:47
cjwatsonI'm not sure I'm seeing this alleged wheel caching working.  Maybe I need to install wheel first ...12:47
wgrantWhat're you using wheels for here?12:47
cjwatsonNothing yet, but installing the world from sdist doesn't appear to be very quick.12:48
cjwatson(Well, except for two packages that were previously installed from eggs.)12:48
cjwatsonI grabbed pytz's wheel from PyPI, and ran 'wheel convert' for meliae.12:49
wgrantI haven't actually investigated wheels in depth.12:49
wgrantI didn't think they were totally a thing yet, but it seems they are.12:49
cjwatsonhttp://paste.ubuntu.com/12124493/ is my output so far.12:55
wgrantVery slow, but functionality is half the battle.12:56
cjwatsonYeah.  Will shelve for a little while, don't want to accidentally spend the entire day on it13:01
blr_morning cjwatson, how was your trip?20:47
cjwatsonblr_: Legoland++20:50
blr_cjwatson: I've heard it is quite impressive :)20:51
=== blr_ is now known as blr
lifelesswgrant: wheels are very much a thing :)20:53
lifelesscjwatson: wgrant: I presume you both know about 'pip wheel' ?20:53
lifelessoh right, wheel convert is a totally different thing21:23
cjwatsonYeah, things I did not want to spend any time on whatsoever included figuring out how to make a binary distribution of meliae and have it match the existing egg21:24

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!