Bug#990111: unblock: python-virtualenv/20.4.0+ds-2
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
Please unblock package python-virtualenv
python-virtualenv (20.4.0+ds-2) unstable; urgency=medium
.
* Patch: Fix --upgrade-embed-wheels.
* Replace the pkg_resources addition part of
debian_update_for_available_wheels.patch with include-pkg_resources.patch
which will only include pkg_resources when using Debian's bundled
setuptools wheel. (Closes: #976796)
[ Reason ]
The --upgrade-embed-wheels option was not working at all, it would
crash, if you attempted to use it. This was fixed upstream later in
20.4.x, so cherry-picked that trivial patch.
Relatedly, we got to the bottom of #976796, which was caused by upgraded
wheels, which would include pkg_resources in the setuptools wheel
(Debian splits it into its own binary package). This could cause a race
on unpacking, crashing.
[ Impact ]
If a user has an upgraded virtualenv wheel cache, then virtualenv
becomes unreliable, due to a race (two threads unpacking the target
files).
[ Tests ]
Manually tested 4 variants:
--seeder pip before and after --upgrade-embed-wheels
--seeder app-data and after --upgrade-embed-wheels
Autopkgtests verify that the basic functionality is unaffected.
[ Risks ]
The changes are relatively straightforward, and should improve
robustness.
[ Checklist ]
[x] all changes are documented in the d/changelog
[x] I reviewed all changes and I approve them
[x] attach debdiff against the package in testing
unblock python-virtualenv/20.4.0+ds-2
diff -Nru python-virtualenv-20.4.0+ds/debian/changelog python-virtualenv-20.4.0+ds/debian/changelog
--- python-virtualenv-20.4.0+ds/debian/changelog 2021-01-22 23:40:18.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/changelog 2021-06-20 17:31:30.000000000 -0400
@@ -1,3 +1,13 @@
+python-virtualenv (20.4.0+ds-2) unstable; urgency=medium
+
+ * Patch: Fix --upgrade-embed-wheels.
+ * Replace the pkg_resources addition part of
+ debian_update_for_available_wheels.patch with include-pkg_resources.patch
+ which will only include pkg_resources when using Debian's bundled
+ setuptools wheel. (Closes: #976796)
+
+ -- Stefano Rivera <stefanor@debian.org> Sun, 20 Jun 2021 17:31:30 -0400
+
python-virtualenv (20.4.0+ds-1) unstable; urgency=medium
* New upstream release.
diff -Nru python-virtualenv-20.4.0+ds/debian/patches/debian_update_for_available_wheels.patch python-virtualenv-20.4.0+ds/debian/patches/debian_update_for_available_wheels.patch
--- python-virtualenv-20.4.0+ds/debian/patches/debian_update_for_available_wheels.patch 2021-01-22 23:40:18.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/patches/debian_update_for_available_wheels.patch 2021-06-20 17:31:30.000000000 -0400
@@ -1,42 +1,17 @@
-From: Debian Python Modules Team
- <python-modules-team@lists.alioth.debian.org>
-Date: Sat, 21 Mar 2020 03:16:18 -0400
+From: Scott Kitterman <scott@kitterman.com>
+Date: Sun, 20 Jun 2021 13:49:30 -0400
Subject: Update base embed to include pip provided wheels for --no-download
Generate wheel lists and attributes for base install to match pip wheel
versions and add pkg_resources to the base install for no download.
-Author: Scott Kitterman <scott@kitterman.com>
Origin: vendor
Forwarded: not-needed
Last-Update: 2020-07-15
---
- src/virtualenv/seed/embed/base_embed.py | 7 ++++++-
src/virtualenv/seed/wheels/embed/__init__.py | 15 +++++++++++++++
- 2 files changed, 21 insertions(+), 1 deletion(-)
+ 1 file changed, 15 insertions(+)
-diff --git a/src/virtualenv/seed/embed/base_embed.py b/src/virtualenv/seed/embed/base_embed.py
-index c794e83..bc9cec8 100644
---- a/src/virtualenv/seed/embed/base_embed.py
-+++ b/src/virtualenv/seed/embed/base_embed.py
-@@ -43,11 +43,16 @@ class BaseEmbed(Seeder):
- }
-
- def distribution_to_versions(self):
-- return {
-+ dv = {
- distribution: getattr(self, "{}_version".format(distribution))
- for distribution in self.distributions()
- if getattr(self, "no_{}".format(distribution)) is False
- }
-+ # Debian specific: Since Debian splits out pkg_resources from
-+ # setuptools, for a local virtualenv, we need to add it to the base.
-+ if not self.download:
-+ dv['pkg_resources'] = None
-+ return dv
-
- @classmethod
- def add_parser_arguments(cls, parser, interpreter, app_data):
diff --git a/src/virtualenv/seed/wheels/embed/__init__.py b/src/virtualenv/seed/wheels/embed/__init__.py
index f63ec1d..4c1a4a7 100644
--- a/src/virtualenv/seed/wheels/embed/__init__.py
diff -Nru python-virtualenv-20.4.0+ds/debian/patches/disable-periodic-update.patch python-virtualenv-20.4.0+ds/debian/patches/disable-periodic-update.patch
--- python-virtualenv-20.4.0+ds/debian/patches/disable-periodic-update.patch 2021-01-22 23:40:18.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/patches/disable-periodic-update.patch 2021-06-20 17:31:30.000000000 -0400
@@ -11,7 +11,7 @@
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/virtualenv/seed/embed/base_embed.py b/src/virtualenv/seed/embed/base_embed.py
-index bc9cec8..3f368c4 100644
+index c794e83..fe5239d 100644
--- a/src/virtualenv/seed/embed/base_embed.py
+++ b/src/virtualenv/seed/embed/base_embed.py
@@ -10,7 +10,7 @@ from virtualenv.util.six import ensure_str, ensure_text
diff -Nru python-virtualenv-20.4.0+ds/debian/patches/fix-upgrade-embedded.patch python-virtualenv-20.4.0+ds/debian/patches/fix-upgrade-embedded.patch
--- python-virtualenv-20.4.0+ds/debian/patches/fix-upgrade-embedded.patch 1969-12-31 20:00:00.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/patches/fix-upgrade-embedded.patch 2021-06-20 17:31:30.000000000 -0400
@@ -0,0 +1,53 @@
+From: =?utf-8?b?QmVybsOhdCBHw6Fib3I=?= <bgabor8@bloomberg.net>
+Date: Sun, 20 Jun 2021 11:34:55 -0400
+Subject: Fix upgrade embeded crashes (#2062)
+
+Origin: https://github.com/pypa/virtualenv/pull/2062
+Bug-Upstream: https://github.com/pypa/virtualenv/issues/2058
+---
+ src/virtualenv/seed/wheels/periodic_update.py | 2 ++
+ tests/unit/seed/wheels/test_periodic_update.py | 4 ++--
+ 2 files changed, 4 insertions(+), 2 deletions(-)
+
+diff --git a/src/virtualenv/seed/wheels/periodic_update.py b/src/virtualenv/seed/wheels/periodic_update.py
+index 18fcfa3..45584f9 100644
+--- a/src/virtualenv/seed/wheels/periodic_update.py
++++ b/src/virtualenv/seed/wheels/periodic_update.py
+@@ -6,6 +6,7 @@ from __future__ import absolute_import, unicode_literals
+
+ import json
+ import logging
++import os
+ import ssl
+ import subprocess
+ import sys
+@@ -235,6 +236,7 @@ def _run_do_update(app_data, distribution, embed_filename, for_py_version, perio
+ search_dirs=search_dirs,
+ app_data=app_data,
+ to_folder=wheelhouse,
++ env=os.environ,
+ )
+ if dest is None or (u_log.versions and u_log.versions[0].filename == dest.name):
+ break
+diff --git a/tests/unit/seed/wheels/test_periodic_update.py b/tests/unit/seed/wheels/test_periodic_update.py
+index 6d0b66d..2ea4732 100644
+--- a/tests/unit/seed/wheels/test_periodic_update.py
++++ b/tests/unit/seed/wheels/test_periodic_update.py
+@@ -300,7 +300,7 @@ def test_do_update_first(tmp_path, mocker, freezer):
+ ]
+ download_wheels = (Wheel(Path(i[0])) for i in pip_version_remote)
+
+- def _download_wheel(distribution, version_spec, for_py_version, search_dirs, app_data, to_folder):
++ def _download_wheel(distribution, version_spec, for_py_version, search_dirs, app_data, to_folder, env):
+ assert distribution == "pip"
+ assert for_py_version == "3.9"
+ assert [str(i) for i in search_dirs] == [str(extra)]
+@@ -361,7 +361,7 @@ def test_do_update_skip_already_done(tmp_path, mocker, freezer):
+ extra = tmp_path / "extra"
+ extra.mkdir()
+
+- def _download_wheel(distribution, version_spec, for_py_version, search_dirs, app_data, to_folder): # noqa
++ def _download_wheel(distribution, version_spec, for_py_version, search_dirs, app_data, to_folder, env): # noqa
+ return wheel.path
+
+ download_wheel = mocker.patch("virtualenv.seed.wheels.acquire.download_wheel", side_effect=_download_wheel)
diff -Nru python-virtualenv-20.4.0+ds/debian/patches/include-pkg_resources.patch python-virtualenv-20.4.0+ds/debian/patches/include-pkg_resources.patch
--- python-virtualenv-20.4.0+ds/debian/patches/include-pkg_resources.patch 1969-12-31 20:00:00.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/patches/include-pkg_resources.patch 2021-06-20 17:31:30.000000000 -0400
@@ -0,0 +1,73 @@
+From: Stefano Rivera <stefanor@debian.org>
+Date: Sat, 21 Mar 2020 03:16:18 -0400
+Subject: Include pkg_resources when using Debian wheels
+
+Origin: vendor
+Forwarded: not-needed
+Last-Update: 2021-07-20
+---
+ src/virtualenv/seed/embed/pip_invoke.py | 9 ++++++++-
+ src/virtualenv/seed/embed/via_app_data/via_app_data.py | 9 ++++++++-
+ 2 files changed, 16 insertions(+), 2 deletions(-)
+
+diff --git a/src/virtualenv/seed/embed/pip_invoke.py b/src/virtualenv/seed/embed/pip_invoke.py
+index c935c02..275330b 100644
+--- a/src/virtualenv/seed/embed/pip_invoke.py
++++ b/src/virtualenv/seed/embed/pip_invoke.py
+@@ -5,6 +5,7 @@ from contextlib import contextmanager
+
+ from virtualenv.discovery.cached_py_info import LogCmd
+ from virtualenv.seed.embed.base_embed import BaseEmbed
++from virtualenv.seed.wheels.embed import BUNDLE_FOLDER
+ from virtualenv.util.subprocess import Popen
+
+ from ..wheels import Version, get_wheel, pip_wheel_env_run
+@@ -37,7 +38,8 @@ class PipInvoke(BaseEmbed):
+ if not self.download:
+ cmd.append("--no-index")
+ folders = set()
+- for dist, version in self.distribution_to_versions().items():
++ dists = list(self.distribution_to_versions().items())
++ for dist, version in dists:
+ wheel = get_wheel(
+ distribution=dist,
+ version=version,
+@@ -52,6 +54,11 @@ class PipInvoke(BaseEmbed):
+ raise RuntimeError("could not get wheel for distribution {}".format(dist))
+ folders.add(str(wheel.path.parent))
+ cmd.append(Version.as_pip_req(dist, wheel.version))
++ # Debian specific: Since Debian splits out pkg_resources from
++ # setuptools, for a local virtualenv, we need to add it to the base.
++ if (dist == 'setuptools'
++ and wheel.path.is_relative_to(BUNDLE_FOLDER)):
++ dists.append(('pkg_resources', Version.bundle))
+ for folder in sorted(folders):
+ cmd.extend(["--find-links", str(folder)])
+ yield cmd
+diff --git a/src/virtualenv/seed/embed/via_app_data/via_app_data.py b/src/virtualenv/seed/embed/via_app_data/via_app_data.py
+index 9a98a70..4d82594 100644
+--- a/src/virtualenv/seed/embed/via_app_data/via_app_data.py
++++ b/src/virtualenv/seed/embed/via_app_data/via_app_data.py
+@@ -10,7 +10,8 @@ from threading import Lock, Thread
+
+ from virtualenv.info import fs_supports_symlink
+ from virtualenv.seed.embed.base_embed import BaseEmbed
+-from virtualenv.seed.wheels import get_wheel
++from virtualenv.seed.wheels import Version, get_wheel
++from virtualenv.seed.wheels.embed import BUNDLE_FOLDER
+ from virtualenv.util.path import Path
+
+ from .pip_install.copy import CopyPipInstall
+@@ -123,6 +124,12 @@ class FromAppData(BaseEmbed):
+ thread.start()
+ for thread in threads:
+ thread.join()
++
++ # Debian specific: Since Debian splits out pkg_resources from
++ # setuptools, for a local virtualenv, we need to add it to the base.
++ if name_to_whl['setuptools'].path.is_relative_to(BUNDLE_FOLDER):
++ _get('pkg_resources', Version.bundle)
++
+ if fail:
+ raise RuntimeError("seed failed due to failing to download wheels {}".format(", ".join(fail.keys())))
+ yield name_to_whl
diff -Nru python-virtualenv-20.4.0+ds/debian/patches/series python-virtualenv-20.4.0+ds/debian/patches/series
--- python-virtualenv-20.4.0+ds/debian/patches/series 2021-01-22 23:40:18.000000000 -0400
+++ python-virtualenv-20.4.0+ds/debian/patches/series 2021-06-20 17:31:30.000000000 -0400
@@ -1,3 +1,5 @@
debian_wheel_location.patch
debian_update_for_available_wheels.patch
+include-pkg_resources.patch
disable-periodic-update.patch
+fix-upgrade-embedded.patch
Reply to: