Re: New, updated pip coming
I am confused. Here's my understanding of things...
- Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
- Virtualenv needs to install pip, wheel, setuptools when it makes a
new environment, and it does that by some oogly code
- venv might do the same thing, but I haven't check its actual implementation
- the pip-whl package was used to give virtualenv a thing to use to
install pip when making a virtualenv
If that aligns with your understanding, I'll put it down to the prose
you wrote rather than us actually having a different worldview; OTOH
if it doesn't, I'd like to find out where I'm wrong, so as to improve
my understanding.
With my understanding in mind - Can I just check something?
If a new requests package is built, uploaded to the archive and
apt-get installed on my machine, and I then do:
virtualenv test
. test/bin/activate
pip install foobar
^--- what version of requests will be used by this in-virtualenv
invocation of pip?
>From your description I expect it to be the version of requests that
was dpkg installed at the time the pip deb was built?
-Rob
On 30 January 2016 at 08:28, Barry Warsaw <barry@debian.org> wrote:
> TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
> bringing us up to the latest version.
>
> It's been a long slog, with many people helping out. I hope that all the hard
> work done to get us here means it will be much, much easier to track new pip,
> virtualenv, and pyvenv updates going forward. And now, details!
>
> pip requires the use of PEP 427 wheels, which are essentially zip files of
> importable, pure-Python, bilingual code, with a little bit of metadata. Put a
> wheel on sys.path and Python can import from it. pip and friends use wheels
> because it ensures that they can't break themselves when their dependent
> packages are updated. E.g. if pip depends on requests, and you `pip install
> --upgrade requests` it's possible the new version would be incompatible with
> pip. Then you'd have a broken pip on your hands. Remember that you can use
> pip to install pre-release packages too.
>
> So by putting the wheels on sys.path *before* anything else, even if you pip
> install an incompatible requests, it won't break pip itself.
>
> pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
> versions, but this breaks Debian policy, so we have to unbundle these.
> Several upstreams use this technique, with requests and urllib3 as probably
> the most popular examples. In all these cases, we have to remove the bundled
> wheels and hope that the versions of their dependencies in Debian remain
> compatible. It's a chore, but that's our job as distro developers, and it's
> the best option given the conflicting requirements.
>
> Fortunately, we can create our own wheels and use them with pip and friends.
> Until now, we had to build the wheels at the time that the dependent packages
> where built, and we had to have a corresponding foo-whl binary package in the
> archives, because we could only use setup.py's bdist_wheel command (which
> comes with python{,3}-wheel) to create these. With 15 run time dependencies
> as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
> pip has lagged so far behind upstream. (That, and a few ITPs for new
> dependencies. ;)
>
> Along comes dirtbike. https://github.com/paulproteus/dirtbike
>
> While it has been much discussed and attempted over the years, it would be
> really nice to have a tool that turned installed Python packages into wheels.
> I think we discussed such a tool for Debian at last year's (two years ago?)
> Debian BOF at Pycon. Anyway, Asheesh Laroia finally took hold of the, ahem,
> wheel, and wrote the initial implementation, which he called dirtbike. Since
> then I've been enhancing it to work with several oddball cases in the Debian
> archive, and now it's good enough to turn pip's dependencies into wheels.
>
> Yesterday I uploaded the last of the new pip dependencies (modulo any
> surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place. What
> pip now does during its own build process, is turn its installed dependencies
> into .whl files and puts them in the policy defined location. This allows us
> to eliminate all those scattered -whl binary packages, and just
> Build-Depend/Built-Using in pip to install the standard Python packages,
> rewheel them, and put them in a python-pip-whl package. Going forward, this
> will make life so much easier, because only python-pip needs to know anything
> about wheels, even if new dependencies are added.
>
> I have everything working locally. I need to update the master branch for
> pip, which involves many curse words hurled at git-dpm (and that's a post for
> another day!). I also have to update my local branch to 8.0.2, but I'm
> assuming there won't be any new surprises there.
>
> Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
> right away.
>
> Huge thanks also go to Donald Stufft, who always is so helpful with various
> technical details, and works really hard to understand our needs and balance
> them with upstream's.
>
> Finally, we need to update Python policy to describe the new wheel policy.
> Fortunately, this is much better because it removes constraints on other
> packages, and reduces the number of -whl binary packages in the distro. Yes,
> I will update those packages that already ship -whls, or file bugs, but only
> once I'm sure everything is working correctly.
>
> Below is the policy diff, which I'll commit to the python-defaults bzr.
>
> As always, once everything's uploaded, please do let me know via this list or
> BTS, if you notice any problems with pip.
>
> Cheers,
> -Barry
>
> === modified file 'debian/python-policy.sgml'
> --- debian/python-policy.sgml 2016-01-24 06:01:19 +0000
> +++ debian/python-policy.sgml 2016-01-29 16:49:59 +0000
> @@ -458,13 +458,12 @@
> <p>
> <url id="http://legacy.python.org/dev/peps/pep-0427/";
> name="PEP 427">
> - defines a built-package format called "wheels", which is a zip
> - format archive containing Python code and a "dist-info" metadata
> + defines a built-package format called "wheels", a zip
> + archive containing Python code and a "dist-info" metadata
> directory, in a single file named with the .whl suffix. As zip
> files, wheels containing pure-Python can be put on sys.path and
> modules in the wheel can be imported directly by Python's "import"
> - statement. (Importing extension modules from wheels is not yet
> - supported as of Python 3.4.)
> + statement.
> </p><p>
> Except as described below, packages must not build or provide
> wheels. They are redundant to the established way of providing
> @@ -475,18 +474,20 @@
> </p><p>
> A very limited set of wheel packages are available in the archive,
> but these support the narrow purpose of enabling
> - the <prgn>pip</prgn> tool, in a Debian policy compliant way. The
> - set of packages providing wheels for this purpose are (by source
> - package name): chardet, distlib, html5lib, python-colorama,
> - python-pip, python-setuptools, python-urllib3, requests, and six.
> + the <prgn>pip</prgn>, <prgn>virtualenv</prgn>,
> + and <prgn>pyvenv</prgn> tools in a Debian policy compliant way.
> + These packages build their own dependent wheels through the use of
> + the <prgn>dirtbike</prgn> "rewheeling" tool, which takes installed
> + Debian packages and turns them back into wheels. Only universal
> + wheels (i.e. pure-Python, Python 2 and 3 compatible packages) are
> + supported. Since only the programs that require wheels need build
> + them, only they may provide <var>-whl</var> packages,
> + e.g. <package>python-pip-whl</package>.
> </p><p>
> - Wheel packages supporting <prgn>pyvenv</prgn> and <prgn>pip</prgn>
> - are named with the <var>python-</var> prefix, and the <var>-whl</var>
> - suffix, e.g. <package>python-chardet-whl</package>. When these
> - binary packages are installed, their .whl files must be placed in
> - the /usr/share/python-wheels directory. Such wheels must be built
> - with the <tt>--universal</tt> flag so as to generate wheels
> - compatible with both Python 2 and Python 3.
> + When these binary packages are installed, .whl files must be placed
> + in the /usr/share/python-wheels directory. The location inside a
> + virtual environment will be rooted in the virtual environment,
> + instead of in /usr.
> </p>
> </sect>
> <sect id="package_names">
>
--
Robert Collins <rbtcollins@hpe.com>
Distinguished Technologist
HP Converged Cloud
Reply to: