| === micahg_ is now known as micahg | ||
| cjwatson | wgrant: the setup_requires thing you mention in https://code.launchpad.net/~stub/launchpad/trivial/+merge/245822 - is that that zc.buildout just doesn't support it? | 02:21 |
|---|---|---|
| cjwatson | I've been experimenting with some upgrades lately, and seem to be going round in circles. ZTK 2.0a1 plus latest zc.buildout has trouble because site-packages isolation has gone away. Twisted >= 14.0.0 has trouble because it ideally wants new pyOpenSSL and that conflicts with a system package, but even if I ignore that, it hits https://github.com/testing-cabal/testtools/pull/149. Even once that or similar is merged, upgrading to ... | 02:25 |
| cjwatson | ... current testtools requires new pbr, which requires the swiftclient upgrade, and so we're back to buildout ... | 02:25 |
| cjwatson | And if we're ever to fix Twisted-based SSH servers to support OpenSSH 7.0 default config, we're going to have to upgrade Twisted at some point | 02:26 |
| cjwatson | Maybe we have to bite the bullet and convert to virtualenv and pip, perhaps with multiple virtualenvs so that we can have some of them configured with system site-packages which we seem to need | 02:28 |
| lifeless | what site packages do you need ? | 03:05 |
| wgrant | lifeless: python-debian, python-apt, IIRC | 04:00 |
| wgrant | cjwatson: zc.buildout can't support it. | 04:01 |
| wgrant | I believe I have a workaround for pbr, but I would have to look up my notes. | 04:01 |
| wgrant | But basically setup_requires in its current implementation is the worst idea ever, unfortunately. | 04:01 |
| StevenK | Worse than Zope 3? | 04:04 |
| wgrant | Substantially. | 04:04 |
| wgrant | Zope 3 is actually possible to use. | 04:04 |
| lifeless | wgrant: whats the pbr problem ? | 04:13 |
| wgrant | lifeless: setuptools has a PyPI fetish | 04:14 |
| wgrant | "oh there's a setup_requires that isn't installed when you import setup.py? i guess i'll just go and download it from PyPI and install it for you that sounds like a good idea doesn't it" | 04:14 |
| wgrant | If you can't convince buildout to install pbr first, setuptools will try to install pbr itself. | 04:15 |
| wgrant | Firewalls (and good security practice) disagree with that. | 04:16 |
| lifeless | you could write a recipe | 04:20 |
| lifeless | like the one in the mock bug I was whinging about earlier | 04:20 |
| cjwatson | wgrant: https://pip.pypa.io/en/latest/reference/pip_install.html#controlling-setup-requires might be helpful, perhaps? | 09:53 |
| wgrant | cjwatson: Yeah, .pydistutils.cfg works | 09:53 |
| wgrant | It is rather unfortunate, however. | 09:53 |
| wgrant | cjwatson: If we were using pip, though, we could just convince it to install pbr beforehand. | 09:57 |
| wgrant | We don't have that option with setuptools, unless we like add a special eggs-setup config key to the recipes we use. | 09:58 |
| wgrant | s/setuptools/buildout/ | 09:58 |
| cjwatson | We'd probably want .pydistutils.cfg anyway for robustness, but switching to pip is feeling increasingly like a good idea. | 09:59 |
| cjwatson | Not a trivial amount of work though. | 10:00 |
| wgrant | It's easier if we can kill python-apt entirely :P | 10:01 |
| cjwatson | Are you sure that's the only one? | 10:02 |
| cjwatson | Also python-debian comes from eggs, not a system package | 10:02 |
| wgrant | It's by no means the only one. | 10:02 |
| wgrant | But it's the only really weird one I know of. | 10:03 |
| wgrant | Well, other than tickcount. | 10:03 |
| cjwatson | It's kind of a zero or any thing. | 10:04 |
| lifeless | wgrant: or add a recipe | 10:05 |
| wgrant | It is. | 10:05 |
| wgrant | But python-apt is the only really weird thing. | 10:05 |
| wgrant | lifeless: Indeed, it's looking tempting to use a custom buildout or z3c.recipe.scripts in the short term. | 10:05 |
| cjwatson | Not z3c.recipe.scripts please, that doesn't work with buildout 2. | 10:06 |
| cjwatson | At least we need a way to migrate to something more like zc.recipe.egg | 10:06 |
| wgrant | I think we may only need that for the weird stuff we do with site-packages, but I forget. | 10:08 |
| wgrant | It's been a while since I've looked at buildout in depth. | 10:09 |
| cjwatson | Yes, I believe z3c.recipe.scripts is just for site-packages mangling | 10:09 |
| wgrant | Does anything stop us from using virtualenv+pip once we no longer need site-packages mangling? | 10:10 |
| lifeless | https://github.com/testing-cabal/mock/issues/310 may provide some inspiration for you | 10:10 |
| wgrant | We use a consistent versions.cfg, so the isolation between txlongpoll etc. and LP is not critical. | 10:10 |
| cjwatson | Does anything stop us from using virtualenv+pip even while we do still need site-packages mangling? We could use multiple virtualenvs. | 10:11 |
| cjwatson | One with --system-site-packages and one without. | 10:11 |
| wgrant | But we can't install the new Twisted in a system-site-packages virtualenv, can we? | 10:11 |
| wgrant | So there's not much point, other than as an hour-long migration strategy. | 10:11 |
| wgrant | I don't have an issue with it, I just don't see how it solves the problem. | 10:13 |
| cjwatson | We ... can't? http://paste.ubuntu.com/12123638/ | 10:14 |
| wgrant | cjwatson: What was the pyOpenSSL issue? | 10:15 |
| wgrant | "Twisted >= 14.0.0 has trouble because it ideally wants new pyOpenSSL and that conflicts with a system package" | 10:15 |
| cjwatson | lifeless: I can see how that would be helpful, though the fact that it would have to be based on z3c.recipe.scripts for buildout 1 and zc.recipe.egg for buildout 2 makes migration rather entertaining | 10:15 |
| cjwatson | wgrant: Stuff like http://paste.ubuntu.com/12123660/ which turns out to be because python-openssl 0.12-1ubuntu2.1 is installed | 10:17 |
| cjwatson | And indeed is a dep of launchpad-dependencies | 10:17 |
| cjwatson | The scripts stanza in buildout.cfg uses system site-packages. | 10:18 |
| wgrant | Right. | 10:18 |
| wgrant | So that's why we can't install the new Twisted in a system-site-packages virtualenv. | 10:18 |
| wgrant | Or something. | 10:18 |
| wgrant | Your motivation for moving away from system-site-packages was the pyOpenSSL issue. | 10:18 |
| wgrant | I thought. | 10:18 |
| cjwatson | It doesn't work in a buildout 2 environment with no virtualenv at all, but it works fine in a system-site-packages virtualenv. | 10:19 |
| wgrant | Ahh, I see. | 10:19 |
| cjwatson | Well, I need libffi-dev or something for it to actually install here, but at least pip seems to resolve things fine. | 10:20 |
| lifeless | pyopenssl works fine in a pure virtualenv too | 10:20 |
| lifeless | the pip cache can mitigate the compilation time | 10:20 |
| cjwatson | It would probably work with buildout 2 and appropriate virtualenvs, but I'm not sure doing it that way round is easier than just migrating to pip first | 10:20 |
| wgrant | Yep, that might be reasonable. | 10:21 |
| cjwatson | I mean, that seems like strictly greater work. | 10:21 |
| cjwatson | At least if we plan to use pip eventually. | 10:22 |
| wgrant | I was just confused as to why you provided the site-packages stuff as a motivation for not moving to zc.buildout 2.0, and then proposed switching to a system-site-packages virtualenv in its place. | 10:22 |
| cjwatson | Right, it's the lack of any virtualenv-like facility (even buildout 1's limited isolation support) in buildout 2 that causes the problem. | 10:22 |
| cjwatson | buildout 2 says to use virtualenv if you need that. | 10:23 |
| wgrant | Well, it says to use virtualenv if you need isolation. | 10:23 |
| wgrant | But you're suggesting using it even if we *don't* need isolation. | 10:23 |
| wgrant | With evidence, which seems reasonable. | 10:23 |
| cjwatson | Oh, right, yes. | 10:23 |
| wgrant | It just seemed totally insane without that critical piece of rationale. | 10:23 |
| cjwatson | Actually it must be a problem even in buildout 1, thinking about it. | 10:24 |
| cjwatson | Because my attempt to upgrade twisted didn't include my attempt to upgrade buildout. | 10:24 |
| cjwatson | I believe that we currently have pyOpenSSL in versions.cfg but do not actually rely on buildout to install it. | 10:24 |
| cjwatson | The problem comes as soon as something depends on a newer version. So we could alternatively backport a newer .deb, I suppose | 10:25 |
| wgrant | Yeah, or we could just move to pip with system-site-packages since it seems to sidestep it. | 10:26 |
| wgrant | exclude-site-packages is simply a sensible default; I doubt txlongpoll/txpkgupload actually require it. | 10:26 |
| wgrant | So we can probably get away with one. | 10:26 |
| cjwatson | Fair point | 10:27 |
| cjwatson | So the next worry is how to cope with all the magic on praseodymium | 10:29 |
| cjwatson | I wonder if I have a copy of the relevant bits anywhere | 10:29 |
| wgrant | Most of that magic is actually on carob | 10:30 |
| wgrant | Recall that carob builds the tree, including all the eggs, then runs "make clean_buildout" | 10:31 |
| wgrant | I'm not sure carob even builds the tree itself, since we used to have i386 machines. | 10:31 |
| wgrant | Er. | 10:31 |
| wgrant | Not sure *praseodymium* even builds the tree | 10:31 |
| wgrant | IIRC everything uses make targets, except for the weird config-manager bits. | 10:32 |
| wgrant | But they don't directly affect this. | 10:32 |
| cjwatson | carob does a "make build_eggs"; what deals with syncing the eggs directory around? | 10:32 |
| wgrant | The eggs directory is a subdirectory of the tree. | 10:33 |
| cjwatson | Since that would presumably need to do the equivalent for a pip cache | 10:33 |
| wgrant | So it just gets synced along with it. | 10:33 |
| wgrant | (this is why make sync-stable sometimes fails hilariously; the tree build on carob is by no means atomic) | 10:33 |
| cjwatson | Oh, right, it's only a symlink in development setups | 10:33 |
| wgrant | Oh | 10:33 |
| wgrant | I forgot rf-setup did it that way. | 10:33 |
| wgrant | I don't. | 10:33 |
| cjwatson | And AIUI our deployment-manager configs run "make compile" or equivalent on each target host, so in the new world order that would install stuff from the pip cache into an appropriate virtualenv. And we would need something site.py-like to cause our scripts to stick themselves into the right virtualenv. | 10:37 |
| lifeless | I'd seriously consider building the virtualenv to go and then syncing it out rather than building on each host | 10:38 |
| cjwatson | The make targets I see that would now have silly names and need to have compatibility preserved at least for a transition period are build_eggs and clean_buildout | 10:38 |
| cjwatson | lifeless: Right, make compile could do nothing if it was there | 10:38 |
| wgrant | lifeless: We could do that now, since the last i386 machines are dead | 10:39 |
| wgrant | But that only happened a year ago :/ | 10:39 |
| cjwatson | Would it cause any problem in the case of heterogeneous Ubuntu releases? | 10:39 |
| cjwatson | Or Python versions? | 10:39 |
| wgrant | Yeah | 10:40 |
| cjwatson | (Sure, everything's 2.7 now, but I don't want to put further roadblocks in the path of 3.x) | 10:40 |
| wgrant | I guess our eggs dir is sort of like a wheel cache. | 10:40 |
| wgrant | But I'm not sure wheels can be deployed for everything yet, can they? | 10:40 |
| lifeless | so | 10:40 |
| lifeless | wheels aren't perfect | 10:40 |
| lifeless | but the cache and shipping it around is doable | 10:41 |
| lifeless | however my point is more about deploy reliability | 10:41 |
| lifeless | you're deploying to enough machines | 10:41 |
| cjwatson | I would be very happy to reduce the time taken by make compile during deployments, if nothing else. But I don't want to leave landmines in the process. | 10:41 |
| lifeless | that its worth the complexity of a build+distribute mechanism rather than distributed builds | 10:41 |
| wgrant | make compile is mostly slow because half our machines are piles of crap. | 10:41 |
| wgrant | make compile on a fast machine is pretty quick. | 10:42 |
| lifeless | as far as moving to new pythons goes | 10:42 |
| cjwatson | I'm not at all arguing against build+distribute, just trying to make sure I understand the issues. | 10:42 |
| lifeless | virtualenvs are not particularly portable | 10:42 |
| lifeless | even minor cpython builds can break them | 10:42 |
| cjwatson | Not all our deployment targets have the same paths, either. | 10:43 |
| lifeless | so the requirement to do build+distribute is that for each target machine you have the same python locally | 10:43 |
| cjwatson | And the virtualenv docs caution against expecting path identity | 10:43 |
| lifeless | and yeah, their paths are embedded in the virtualenv | 10:43 |
| lifeless | but a container can address both things easily | 10:44 |
| cjwatson | This is really sounding like a later step that we should make sure we don't design ourselves out of, rather than something to do in the first run; where I came in was trying to cut Gordian knots | 10:45 |
| lifeless | sure | 10:45 |
| cjwatson | I mean, not dodging the issue or anything, just trying to avoid having to change All The Things at once | 10:46 |
| lifeless | I get it | 10:46 |
| lifeless | I'm just surfacing options | 10:46 |
| cjwatson | Yeah | 10:46 |
| lifeless | running pip on a prod server with --no-index -f <path> should be reliable as long as you've neutered easy-install via .pydistutils.cfg | 10:47 |
| wgrant | Yeah, I was hoping to avoid that. | 10:47 |
| lifeless | no, you must neuter it | 10:47 |
| lifeless | otherwise you'll be auditing forever | 10:47 |
| wgrant | There was discussion a while ago about making pip pass options down so easy_install would be neatured | 10:47 |
| wgrant | neutered | 10:47 |
| wgrant | but I haven't seen whether that actually got done | 10:47 |
| lifeless | there are no such options | 10:47 |
| lifeless | and pip can't do it until we have declarative metadata for setup_requires used by *every package including old releases* | 10:48 |
| lifeless | e.g. never | 10:48 |
| lifeless | there is a route that might work | 10:48 |
| lifeless | which is to make it error out with the missing dep | 10:48 |
| wgrant | Right, easy_install had, or was going to grow, options for allow-hosts and find-links | 10:48 |
| lifeless | and have pip parse the output | 10:48 |
| wgrant | Which would be respected. | 10:48 |
| cjwatson | We can put the pydistutils.cfg in the virtualenv, I think? | 10:49 |
| cjwatson | Which seems a little easier to arrange than either system-wide or user-wide configuration | 10:49 |
| wgrant | Oh, can we? | 10:49 |
| lifeless | you can always export a custom HOME | 10:50 |
| cjwatson | Or even setup.cfg, according to the docs | 10:50 |
| cjwatson | https://docs.python.org/2/install/index.html#distutils-configuration-files | 10:50 |
| wgrant | That seems too easy. | 10:50 |
| cjwatson | Or is that setup.cfg of the package being processed? | 10:50 |
| wgrant | Huh | 10:50 |
| lifeless | setup.cfg isn't accessible to you | 10:50 |
| lifeless | because yes | 10:50 |
| cjwatson | Right, bah. | 10:50 |
| wgrant | lib/python2.7/distutils/distutils.cfg might work | 10:50 |
| wgrant | Maybe | 10:50 |
| wgrant | But I doubt it. | 10:50 |
| wgrant | Again, too easy. | 10:50 |
| cjwatson | virtualenv writes out a template file that claims it works. | 10:52 |
| cjwatson | http://paste.ubuntu.com/12123833/ | 10:52 |
| wgrant | ! | 10:52 |
| cjwatson | Also /path/to/env/.pydistutils.cfg, according to the changelog for virtualenv 1.3.3 | 10:54 |
| cjwatson | Allegedly, though I'm not seeing that in the code. | 10:55 |
| cjwatson | Oh, yes I do, in the source. Couldn't find it in the installed virtualenv.py because it's helpfully zlib-compressed and base64-encoded. | 11:04 |
| wgrant | I don't know why people also don't xor things with "Beware of the Leopard" while they're at it. | 11:05 |
| cjwatson | So env/.pydistutils.cfg seems by far the easiest | 11:06 |
| wgrant | if it works, indeed that is amazing. | 11:07 |
| cjwatson | wgrant: http://paste.ubuntu.com/12124302/ actually a success for .pydistutils.cfg! | 12:24 |
| cjwatson | (that "succeeds" if I remove .pydistutils.cfg, so correctly blocked.) | 12:24 |
| wgrant | Aha | 12:25 |
| wgrant | Excellent. | 12:25 |
| wgrant | I knew ~/.pydistutils.cfg worked, but if the virtualenv one works too then all the better. | 12:25 |
| cjwatson | Right, this is the virtualenv one. | 12:26 |
| cjwatson | OK, that can actually install everything - just needed a couple of wheel conversions and suchlike. The next steps are figuring out how to handle sitecustomize and buildout-templates. | 12:45 |
| wgrant | cjwatson: Does "everything" include our dpkg deps? | 12:45 |
| cjwatson | wgrant: They're accessible from virtualenv/bin/python, if that's what you mean | 12:46 |
| wgrant | Er of course | 12:47 |
| wgrant | Sorry, neutron ate my brain. | 12:47 |
| cjwatson | I'm not sure I'm seeing this alleged wheel caching working. Maybe I need to install wheel first ... | 12:47 |
| wgrant | What're you using wheels for here? | 12:47 |
| cjwatson | Nothing yet, but installing the world from sdist doesn't appear to be very quick. | 12:48 |
| wgrant | Quite. | 12:48 |
| cjwatson | (Well, except for two packages that were previously installed from eggs.) | 12:48 |
| cjwatson | I grabbed pytz's wheel from PyPI, and ran 'wheel convert' for meliae. | 12:49 |
| wgrant | I haven't actually investigated wheels in depth. | 12:49 |
| wgrant | I didn't think they were totally a thing yet, but it seems they are. | 12:49 |
| cjwatson | http://paste.ubuntu.com/12124493/ is my output so far. | 12:55 |
| wgrant | Very slow, but functionality is half the battle. | 12:56 |
| cjwatson | Yeah. Will shelve for a little while, don't want to accidentally spend the entire day on it | 13:01 |
| blr_ | morning cjwatson, how was your trip? | 20:47 |
| cjwatson | blr_: Legoland++ | 20:50 |
| blr_ | cjwatson: I've heard it is quite impressive :) | 20:51 |
| === blr_ is now known as blr | ||
| lifeless | wgrant: wheels are very much a thing :) | 20:53 |
| lifeless | cjwatson: wgrant: I presume you both know about 'pip wheel' ? | 20:53 |
| cjwatson | Yes | 20:55 |
| lifeless | oh right, wheel convert is a totally different thing | 21:23 |
| cjwatson | Yeah, things I did not want to spend any time on whatsoever included figuring out how to make a binary distribution of meliae and have it match the existing egg | 21:24 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!