I have been doing some experimentation and preparation for trying to solve bug
#732703, i.e. fixing ensurepip in Python 3.4 and thus the pyvenv command.
Why this is tricky: ensurepip in the Python stdlib bundles two wheels, one for
setuptools and another for pip, and these wheels get installed into the
virtual environment. ensurepip does this to ease the installation process and
to only have to rely on known compatible versions of the libraries. Further,
pip itself bundles (a.k.a. vendorizes) several packages itself, such as
colorama, distlib, html5lib, requests, and six. pip does this to also rely on
known dependency versions, but also to prevent upgrades to e.g. requests
breaking pip itself. To make matters worse, requests also vendorizes packages
such as chardet, charade, and urllib3. It's a recursive mess.
All this vendorizing causes problems because it violates Debian policy. When
we package up e.g. requests for Debian, we *unvendorize* everything (through
quilt patches mostly) so that when requests imports urllib3, it does so from
the python*-urllib3 Debian package. It's then our job to ensure compatible
versions across the board, but that's what a distro does.
ensurepip and thus pyvenv is disabled in Debian's Python 3.4 because unwinding
all this vendorizing is a huge PITA. I think I have an approach that might be
acceptable, and could possibly work. I haven't fully implemented it, but some
experimentation makes me think it's not too horrible. I'd like to get your
feedback before spending much more time on it. If it seems generally
acceptable, I'll continue to work on the stack of things we'll need.
The idea generally is to create PEP 427 wheels out of all the dependencies
*from the packages in the archive at build time*. We add a new binary package
to the source package, something like python-<foo>-wheels, and these drop the
.whl files into /usr/share/python-wheels. Thus, most people can ignore the
-wheels packages entirely with no change to their code or packaging.
For a few things that need it, e.g. python3.4, we Build-Depend on the
appropriate -wheel binary packages and add a Built-Using field to d/control to
so indicate the package version we're depending on. Then we recursively
rebuild the ensurepip embedded .whl files from contents in
/usr/share/python-wheels. This guarantees that all the embedded stuff is at
least built from source that is in the archives.
From Python import's point of view, we'll still be vendorizing, but now we'll
be vendorizing wheels, and we could either copy the .whl file in place, or
Depends on the -wheel packages and symlink to the appropriate
/usr/share/python-wheels/*.whl file. We would have to modify some __init__.py
files to set up importing from the vendorized .whl. We *could* unpack the
wheel into place (it's just a zip file after all), but I rather like that
vendorized packages are self-contained. It seems cleaner.
PEP 427 does discourage direct import from wheels[1] but for the things we
care about immediately, the two conditions should not affect us (i.e. no C
extensions, and no bad use of __file__) so I think we could get away with it.
I'm open to other good arguments for or against using the wheels directly.
The __init__.py code we'd have to add would be pretty minimal:
-----snip snip-----
import os
here = os.path.dirname(__file__)
for filename in os.listdir(here):
if os.path.splitext(filename)[1] == '.whl':
__path__.append(os.path.join(here, filename))
-----snip snip-----
E.g. this would go in pip/_vendor/__init__.py and then when you `import
pip._vendor.six` it would come from pip/_vendor/six-1.6.1-py2.py3-none-any.whl
Generating the wheels during package build is pretty easy I think. You just
B-D on python*-wheel (which just got approved from NEW) and then if the
package uses setuptools, you just add something like:
python3 setup.py bdist_wheel -d \
debian/python-foo-wheels/usr/share/python-wheels
dh_installdirs -ppython-foo-wheels usr/share
after you've added the python-foo-wheels binary package. (Exact package
names, directory paths, etc. TBD, but these are what I've picked for my
experiments.)
It's still a lot of work to upload new packages (not all of which are team
maintained) and test out the full stack.
No doubt there are lots of things I'm missing, plenty of gotchas lurking, and
maybe the whole idea is plain ol' stupid. But maybe it'll be just good enough
to fix ensurepip and pyvenv.
Feedback welcome,
-Barry
[1] http://tinyurl.com/n3p5psl
Attachment:
signature.asc
Description: PGP signature