TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally bringing us up to the latest version. It's been a long slog, with many people helping out. I hope that all the hard work done to get us here means it will be much, much easier to track new pip, virtualenv, and pyvenv updates going forward. And now, details! pip requires the use of PEP 427 wheels, which are essentially zip files of importable, pure-Python, bilingual code, with a little bit of metadata. Put a wheel on sys.path and Python can import from it. pip and friends use wheels because it ensures that they can't break themselves when their dependent packages are updated. E.g. if pip depends on requests, and you `pip install --upgrade requests` it's possible the new version would be incompatible with pip. Then you'd have a broken pip on your hands. Remember that you can use pip to install pre-release packages too. So by putting the wheels on sys.path *before* anything else, even if you pip install an incompatible requests, it won't break pip itself. pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good versions, but this breaks Debian policy, so we have to unbundle these. Several upstreams use this technique, with requests and urllib3 as probably the most popular examples. In all these cases, we have to remove the bundled wheels and hope that the versions of their dependencies in Debian remain compatible. It's a chore, but that's our job as distro developers, and it's the best option given the conflicting requirements. Fortunately, we can create our own wheels and use them with pip and friends. Until now, we had to build the wheels at the time that the dependent packages where built, and we had to have a corresponding foo-whl binary package in the archives, because we could only use setup.py's bdist_wheel command (which comes with python{,3}-wheel) to create these. With 15 run time dependencies as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's pip has lagged so far behind upstream. (That, and a few ITPs for new dependencies. ;) Along comes dirtbike. https://github.com/paulproteus/dirtbike While it has been much discussed and attempted over the years, it would be really nice to have a tool that turned installed Python packages into wheels. I think we discussed such a tool for Debian at last year's (two years ago?) Debian BOF at Pycon. Anyway, Asheesh Laroia finally took hold of the, ahem, wheel, and wrote the initial implementation, which he called dirtbike. Since then I've been enhancing it to work with several oddball cases in the Debian archive, and now it's good enough to turn pip's dependencies into wheels. Yesterday I uploaded the last of the new pip dependencies (modulo any surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place. What pip now does during its own build process, is turn its installed dependencies into .whl files and puts them in the policy defined location. This allows us to eliminate all those scattered -whl binary packages, and just Build-Depend/Built-Using in pip to install the standard Python packages, rewheel them, and put them in a python-pip-whl package. Going forward, this will make life so much easier, because only python-pip needs to know anything about wheels, even if new dependencies are added. I have everything working locally. I need to update the master branch for pip, which involves many curse words hurled at git-dpm (and that's a post for another day!). I also have to update my local branch to 8.0.2, but I'm assuming there won't be any new surprises there. Eventually, I'll be using the same techique for virtualenv and pyvenv, but not right away. Huge thanks also go to Donald Stufft, who always is so helpful with various technical details, and works really hard to understand our needs and balance them with upstream's. Finally, we need to update Python policy to describe the new wheel policy. Fortunately, this is much better because it removes constraints on other packages, and reduces the number of -whl binary packages in the distro. Yes, I will update those packages that already ship -whls, or file bugs, but only once I'm sure everything is working correctly. Below is the policy diff, which I'll commit to the python-defaults bzr. As always, once everything's uploaded, please do let me know via this list or BTS, if you notice any problems with pip. Cheers, -Barry === modified file 'debian/python-policy.sgml' --- debian/python-policy.sgml 2016-01-24 06:01:19 +0000 +++ debian/python-policy.sgml 2016-01-29 16:49:59 +0000 @@ -458,13 +458,12 @@ <p> <url id="http://legacy.python.org/dev/peps/pep-0427/" name="PEP 427"> - defines a built-package format called "wheels", which is a zip - format archive containing Python code and a "dist-info" metadata + defines a built-package format called "wheels", a zip + archive containing Python code and a "dist-info" metadata directory, in a single file named with the .whl suffix. As zip files, wheels containing pure-Python can be put on sys.path and modules in the wheel can be imported directly by Python's "import" - statement. (Importing extension modules from wheels is not yet - supported as of Python 3.4.) + statement. </p><p> Except as described below, packages must not build or provide wheels. They are redundant to the established way of providing @@ -475,18 +474,20 @@ </p><p> A very limited set of wheel packages are available in the archive, but these support the narrow purpose of enabling - the <prgn>pip</prgn> tool, in a Debian policy compliant way. The - set of packages providing wheels for this purpose are (by source - package name): chardet, distlib, html5lib, python-colorama, - python-pip, python-setuptools, python-urllib3, requests, and six. + the <prgn>pip</prgn>, <prgn>virtualenv</prgn>, + and <prgn>pyvenv</prgn> tools in a Debian policy compliant way. + These packages build their own dependent wheels through the use of + the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed + Debian packages and turns them back into wheels. Only universal + wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are + supported. Since only the programs that require wheels need build + them, only they may provide <var>-whl</var> packages, + e.g. <package>python-pip-whl</package>. </p><p> - Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn> - are named with the <var>python-</var> prefix, and the <var>-whl</var> - suffix, e.g. <package>python-chardet-whl</package>. When these - binary packages are installed, their .whl files must be placed in - the /usr/share/python-wheels directory. Such wheels must be built - with the <tt>--universal</tt> flag so as to generate wheels - compatible with both Python 2 and Python 3. + When these binary packages are installed, .whl files must be placed + in the /usr/share/python-wheels directory. The location inside a + virtual environment will be rooted in the virtual environment, + instead of in /usr. </p> </sect> <sect id="package_names">
Attachment:
pgp0aHplKQMhn.pgp
Description: OpenPGP digital signature