--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: python-apt: relies on MD5 internally to download packages
- From: Cyril Brulebois <cyril@debamax.com>
- Date: Thu, 14 Nov 2019 01:36:14 +0100
- Message-id: <157369177449.9422.1213703436713711073.reportbug@armor.home>
Package: python-apt
Version: 1.8.4
Severity: serious
Justification: some people want to get rid of MD5Sum in indices
Hi,
While debugging a live-wrapper (lwr) failure that started occurring
(literally) overnight, I ended up discovering it was triggered by the
intel-microcode package's getting a security upgrade.
live-wrapper 0.10 isn't affected, but live-wrapper's master branch has
an extra commit that automatically enables security sources for stable
releases.
Here's the traceback for a simple build (with a local mirror but anyone
would do) with that master branch:
$ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode
[…]
DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian'
DEBUG environment: LWR_EXTRA_PACKAGES = ''
DEBUG environment: LWR_BASE_DEBS = ''
DEBUG environment: LWR_DISTRIBUTION = 'buster'
DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode'
DEBUG environment: LWR_TASK_PACKAGES = ''
[…]
Downloading udebs for Debian Installer...
INFO Downloading udebs for Debian Installer...
Updating a local cache for amd64 buster ...
DEBUG Updating local cache...
CRITICAL Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run
self.process_args(args)
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args
self.start_ops()
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops
apt_udeb.download_udebs(exclude_list)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs
self.download_apt_file(pkg_name, pool_dir, False)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file
version.fetch_binary(destdir=pkg_dir)
File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary
if _file_is_same(destfile, self.size, self._records.md5_hash):
SystemError: error return without exception set
After some debugging, it turned out that merely accessing the
self._records.md5_hash item is sufficient to reproduce this issue.
Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for
buster/updates on security.debian.org, one can only see SHA256 entries
in Release and Packages files, which is likely the reason for
python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries
back at least for buster/updates, and will file another bug report for
that in a moment to make sure it isn't lost.
Looking at even the most recent python-apt code in experimental (1.9.0),
MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary():
def fetch_binary(self, destdir='', progress=None):
# type: (str, AcquireProgress) -> str
"""Fetch the binary version of the package.
The parameter *destdir* specifies the directory where the package will
be fetched to.
The parameter *progress* may refer to an apt_pkg.AcquireProgress()
object. If not specified or None, apt.progress.text.AcquireProgress()
is used.
.. versionadded:: 0.7.10
"""
base = os.path.basename(self._records.filename)
destfile = os.path.join(destdir, base)
if _file_is_same(destfile, self.size, self._records.md5_hash):
logging.debug('Ignoring already existing file: %s' % destfile)
return os.path.abspath(destfile)
acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress())
acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # type: ignore # TODO: Do not use MD5 # nopep8
self.size, base, destfile=destfile)
acq.run()
if acqfile.status != acqfile.STAT_DONE:
raise FetchError("The item %r could not be fetched: %s" %
(acqfile.destfile, acqfile.error_text))
return os.path.abspath(destfile)
Notice the TODO on the apt_pkg.AcquireFile(), but it would probably
break in the same way as in the live-wrapper case a few lines before, on
the self._records.md5_hash item.
The same goes for fetch_source().
Since getting rid of MD5Sum entirely is a topic that comes up on a
regular fashion (with fingers being pointed at jigdo in particular), it
looks to me python-apt should get some attention as well; hence filing
at serious severity. Feel free to adjust as required.
Cheers,
--
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/
--- End Message ---
--- Begin Message ---
- To: 944696-close@bugs.debian.org
- Subject: Bug#944696: fixed in python-apt 1.8.5
- From: Julian Andres Klode <jak@debian.org>
- Date: Mon, 20 Jan 2020 09:50:04 +0000
- Message-id: <E1itTh6-0008jj-Nc@fasolo.debian.org>
Source: python-apt
Source-Version: 1.8.5
We believe that the bug you reported is fixed in the latest version of
python-apt, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 944696@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated python-apt package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Wed, 15 Jan 2020 16:46:29 +0100
Source: python-apt
Architecture: source
Version: 1.8.5
Distribution: unstable
Urgency: high
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 944696 947794
Changes:
python-apt (1.8.5) unstable; urgency=high
.
* SECURITY UPDATE: Check that repository is trusted before downloading
files from it (LP: #1858973)
- apt/cache.py: Add checks to fetch_archives() and commit()
- apt/package.py: Add checks to fetch_binary() and fetch_source()
- CVE-2019-15796
* SECURITY UPDATE: Do not use MD5 for verifying downloadeds
(Closes: #944696) (#LP: #1858972)
- apt/package.py: Use all hashes when fetching packages, and
check that we have trusted hashes when downloading
- CVE-2019-15795
* To work around the new checks, the parameter allow_unauthenticated=True
can be passed to the functions. It defaults to the value of the
APT::Get::AllowUnauthenticated option.
- Bump Breaks aptdaemon (<< 1.1.1+bzr982-0ubuntu21.2), as it will have
to set that parameter after having done validation.
* Automatic changes and fixes for external regressions:
- Adjustments to test suite and CI to fix CI regressions
- Automatic mirror list update
- d/tests/control: Add "Restrictions: allow-stderr" (Closes: #947794)
Checksums-Sha1:
f0b49ffa3f186ba0c2d63b80b551970d6f2f8f83 2451 python-apt_1.8.5.dsc
3592b15feb50e9d6d02a87a0447d75d5a814dc9b 343344 python-apt_1.8.5.tar.xz
67cfc6193f15bcefff16d2624553b6ef122a04ab 10358 python-apt_1.8.5_source.buildinfo
Checksums-Sha256:
d57d34982561e6373625b1b22d6d4e0416778eade6f126db702f98bb8b5853be 2451 python-apt_1.8.5.dsc
f7fe0023f9ea2193a8b7a8cdd5be00f88eb44c59b184e8b0d9c64f38e33e353c 343344 python-apt_1.8.5.tar.xz
0873481ceb4b45197e0a58320e675d68da46e582f307f2de22807d2593c498a7 10358 python-apt_1.8.5_source.buildinfo
Files:
6c2120799bb20e99737cdd832d449b50 2451 python optional python-apt_1.8.5.dsc
2f82338e36a8bf76d16e338e8e2a4651 343344 python optional python-apt_1.8.5.tar.xz
fe2e1ac81f6edfc291ef24080e13571b 10358 python optional python-apt_1.8.5_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAl4lchgPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9xOqMQAISpI3BoO/1aoTGQmL+y9TqtotbVUgyvldwn
Be8B7BpUylml2iX8FJoEX7nQxdoDvmBNHhYV6YvZr+seoKn0cPHlSx3OwuzJwzlC
/o/TmbJCrPPMp8drHBRBC/eSVEO/K36koRG+OjR0EaPqsPdA/Fv1cp3UclroEeC6
pWG5WXHerhvKiCdeO5ABuEa1X0Q87fqEGldzH7Z3eC2aWX93ilBAC383QIVwPVcN
fk5WAYQcc8PE/wV1Vzt4JOCPdXuCrw9WECQmvjFTu6pLfWmXioZt4HWmFyGyf5A7
jBNuhX0TWnXV+mFIbjF6+Blg0r3zSMA8omW4m4VEUmiru7K8NDUp+Sa7FLqQAL+f
3PlQhTllo3l3E+MuB42A+v5+gvyfw+a3TcVmdq4iPr/GKk25V3f/zur5mPm5+PPl
lSRdm/8xBcxdcRZqBDsTbp6Lh7UcKJmUWr1Mq6mWrHcZ6jE0R/aAUn1XwD0hL5vh
9tO7wvuy0LbcvvMa7sYCXq1mwBR5exU54CK6mbMlSSaGMDYTGOn9OJWfqmTQlYky
aOmFsccb1jlVxFqbN575K6m3XoOm4rD3i4iZdIizMKy5CsBu6I+kte2l7FGDSFcy
OZT2BJkek1SBNzsjD9RNiw2aKsh8s/aH6OL/ls7XgV9MWIyXXCJz6R7LbzWWV6sv
lHfZixb0
=j9Rj
-----END PGP SIGNATURE-----
--- End Message ---