--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: python-apt: relies on MD5 internally to download packages
- From: Cyril Brulebois <cyril@debamax.com>
- Date: Thu, 14 Nov 2019 01:36:14 +0100
- Message-id: <157369177449.9422.1213703436713711073.reportbug@armor.home>
Package: python-apt
Version: 1.8.4
Severity: serious
Justification: some people want to get rid of MD5Sum in indices
Hi,
While debugging a live-wrapper (lwr) failure that started occurring
(literally) overnight, I ended up discovering it was triggered by the
intel-microcode package's getting a security upgrade.
live-wrapper 0.10 isn't affected, but live-wrapper's master branch has
an extra commit that automatically enables security sources for stable
releases.
Here's the traceback for a simple build (with a local mirror but anyone
would do) with that master branch:
$ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode
[…]
DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian'
DEBUG environment: LWR_EXTRA_PACKAGES = ''
DEBUG environment: LWR_BASE_DEBS = ''
DEBUG environment: LWR_DISTRIBUTION = 'buster'
DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode'
DEBUG environment: LWR_TASK_PACKAGES = ''
[…]
Downloading udebs for Debian Installer...
INFO Downloading udebs for Debian Installer...
Updating a local cache for amd64 buster ...
DEBUG Updating local cache...
CRITICAL Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run
self.process_args(args)
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args
self.start_ops()
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops
apt_udeb.download_udebs(exclude_list)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs
self.download_apt_file(pkg_name, pool_dir, False)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file
version.fetch_binary(destdir=pkg_dir)
File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary
if _file_is_same(destfile, self.size, self._records.md5_hash):
SystemError: error return without exception set
After some debugging, it turned out that merely accessing the
self._records.md5_hash item is sufficient to reproduce this issue.
Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for
buster/updates on security.debian.org, one can only see SHA256 entries
in Release and Packages files, which is likely the reason for
python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries
back at least for buster/updates, and will file another bug report for
that in a moment to make sure it isn't lost.
Looking at even the most recent python-apt code in experimental (1.9.0),
MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary():
def fetch_binary(self, destdir='', progress=None):
# type: (str, AcquireProgress) -> str
"""Fetch the binary version of the package.
The parameter *destdir* specifies the directory where the package will
be fetched to.
The parameter *progress* may refer to an apt_pkg.AcquireProgress()
object. If not specified or None, apt.progress.text.AcquireProgress()
is used.
.. versionadded:: 0.7.10
"""
base = os.path.basename(self._records.filename)
destfile = os.path.join(destdir, base)
if _file_is_same(destfile, self.size, self._records.md5_hash):
logging.debug('Ignoring already existing file: %s' % destfile)
return os.path.abspath(destfile)
acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress())
acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # type: ignore # TODO: Do not use MD5 # nopep8
self.size, base, destfile=destfile)
acq.run()
if acqfile.status != acqfile.STAT_DONE:
raise FetchError("The item %r could not be fetched: %s" %
(acqfile.destfile, acqfile.error_text))
return os.path.abspath(destfile)
Notice the TODO on the apt_pkg.AcquireFile(), but it would probably
break in the same way as in the live-wrapper case a few lines before, on
the self._records.md5_hash item.
The same goes for fetch_source().
Since getting rid of MD5Sum entirely is a topic that comes up on a
regular fashion (with fingers being pointed at jigdo in particular), it
looks to me python-apt should get some attention as well; hence filing
at serious severity. Feel free to adjust as required.
Cheers,
--
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/
--- End Message ---
--- Begin Message ---
- To: 944696-close@bugs.debian.org
- Subject: Bug#944696: fixed in python-apt 1.4.1
- From: Julian Andres Klode <jak@debian.org>
- Date: Sat, 25 Jan 2020 19:03:19 +0000
- Message-id: <E1ivQiF-00024v-Ab@fasolo.debian.org>
Source: python-apt
Source-Version: 1.4.1
We believe that the bug you reported is fixed in the latest version of
python-apt, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 944696@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated python-apt package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Thu, 23 Jan 2020 11:32:18 +0100
Source: python-apt
Binary: python-apt python-apt-doc python-apt-dbg python-apt-dev python-apt-common python3-apt python3-apt-dbg
Architecture: source
Version: 1.4.1
Distribution: stretch-security
Urgency: high
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Description:
python-apt - Python interface to libapt-pkg
python-apt-common - Python interface to libapt-pkg (locales)
python-apt-dbg - Python interface to libapt-pkg (debug extension)
python-apt-dev - Python interface to libapt-pkg (development files)
python-apt-doc - Python interface to libapt-pkg (API documentation)
python3-apt - Python 3 interface to libapt-pkg
python3-apt-dbg - Python 3 interface to libapt-pkg (debug extension)
Closes: 944696
Changes:
python-apt (1.4.1) stretch-security; urgency=high
.
* SECURITY UPDATE: Check that repository is trusted before downloading
files from it (LP: #1858973)
- apt/cache.py: Add checks to fetch_archives() and commit()
- apt/package.py: Add checks to fetch_binary() and fetch_source()
- CVE-2019-15796
* SECURITY UPDATE: Do not use MD5 for verifying downloadeds
(Closes: #944696) (#LP: #1858972)
- apt/package.py: Use all hashes when fetching packages, and
check that we have trusted hashes when downloading
- CVE-2019-15795
* To work around the new checks, the parameter allow_unauthenticated=True
can be passed to the functions. It defaults to the value of the
APT::Get::AllowUnauthenticated option.
* Cherry-pick "add pkgsrcrecord.Files.{hashes,size,path,type} getters" to
enable apt_pkg.SourceRecords to return objects with such getters instead
of just tuples (providing tuple-style backward compatibility).
* Automatic changes and fixes for external regressions:
- Adjustments to test suite and CI to fix CI regressions
- testcommon: Avoid reading host apt.conf files
- Automatic mirror list update
Checksums-Sha1:
fe0374c18168785d7d3a7fd7a2a8d45ef99cdb38 2427 python-apt_1.4.1.dsc
c7eac12a3d9275b7f350e943c5dfd49e91fa40ee 333512 python-apt_1.4.1.tar.xz
dc43a04fd852617e801c5b62218b1bff52e9ae40 9792 python-apt_1.4.1_source.buildinfo
Checksums-Sha256:
8c8bfedba3e76ed59c4d96f3b9c6db22d6193a84468b899527e1add0687c587b 2427 python-apt_1.4.1.dsc
90a10a7daced35cae9096cb0bd87a6bf1c7e11a0cf201d67bcec4b3b15ab8662 333512 python-apt_1.4.1.tar.xz
251b4423e40d91dec2ef17e61afe227b2edcc75922d056594d7c840c742e29b0 9792 python-apt_1.4.1_source.buildinfo
Files:
d75b178165297f2717840ae67300088c 2427 python optional python-apt_1.4.1.dsc
03a95ce40ebf559851ec2897e6e37415 333512 python optional python-apt_1.4.1.tar.xz
0260a7b9a2bf2ef9a5f6d023d62a0619 9792 python optional python-apt_1.4.1_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAl4pdqkPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9x+ogP/icpU6/hCZFTnjCmPeAK9He/ZuThEz0XHNmo
+VzJxI/Zc4As4TW/HE0faFQWx6JePoHWomr3xh5LpQQfW+h3RlILqTjDY3859Ood
RJ7omS5FFKT6n5LTZllL9POETnd1O8WOkl+BzFTfVLW2ss3hkrZWh8b3xvTZaXxQ
uRx6EDgYTxmoUo5rhi5C5mVhslj5ImEuMXqOTGe++J8dPG1ANEJq//cR6IdCi10G
8aIEJECaK7LlVZS4gCID2/IWpNvbEymRokKIfVrPsc4Cjgrcb+VA+4wyKB6GCALH
mMBo0H88pb28P/9CF3IgvlAEJQDR0BWJvPZpksEDfsxObI/b/g5ZeffLIB4AFHuf
zdb5mUPQ1GAhV9UlVyn4UkSkhBpDGr0lnZNAAG8ezFEIXDEERnZ6pK2hzGtmnQhy
EC7MuLfbTUtmzeX6ri57BZ9p/eypi+VqVNwLIoij+U6Lh7KlnSjPsrivFbkknUdd
3Qwy1tiAtdPIyapEF+IRphYXzo3mTNYOMcnHszMxCUrWgjWLflhjRZqHFPb3qZGc
zPXxXo6qZ/C2iEDlsFA42XbR3Jk7IwmY2AWedbyBUR4Lcu8nV4jLX+NSXmCtYLQE
PNqYLLlv1WDvYb7ynY9pPhGDOKAyX/j+2o7JVmy95dJYURMR0Kpib/4qFOZbnm9N
gPdJ8eJR
=Jli7
-----END PGP SIGNATURE-----
--- End Message ---