--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: python-apt: relies on MD5 internally to download packages
- From: Cyril Brulebois <cyril@debamax.com>
- Date: Thu, 14 Nov 2019 01:36:14 +0100
- Message-id: <157369177449.9422.1213703436713711073.reportbug@armor.home>
Package: python-apt
Version: 1.8.4
Severity: serious
Justification: some people want to get rid of MD5Sum in indices
Hi,
While debugging a live-wrapper (lwr) failure that started occurring
(literally) overnight, I ended up discovering it was triggered by the
intel-microcode package's getting a security upgrade.
live-wrapper 0.10 isn't affected, but live-wrapper's master branch has
an extra commit that automatically enables security sources for stable
releases.
Here's the traceback for a simple build (with a local mirror but anyone
would do) with that master branch:
$ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode
[…]
DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian'
DEBUG environment: LWR_EXTRA_PACKAGES = ''
DEBUG environment: LWR_BASE_DEBS = ''
DEBUG environment: LWR_DISTRIBUTION = 'buster'
DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode'
DEBUG environment: LWR_TASK_PACKAGES = ''
[…]
Downloading udebs for Debian Installer...
INFO Downloading udebs for Debian Installer...
Updating a local cache for amd64 buster ...
DEBUG Updating local cache...
CRITICAL Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run
self.process_args(args)
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args
self.start_ops()
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops
apt_udeb.download_udebs(exclude_list)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs
self.download_apt_file(pkg_name, pool_dir, False)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file
version.fetch_binary(destdir=pkg_dir)
File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary
if _file_is_same(destfile, self.size, self._records.md5_hash):
SystemError: error return without exception set
After some debugging, it turned out that merely accessing the
self._records.md5_hash item is sufficient to reproduce this issue.
Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for
buster/updates on security.debian.org, one can only see SHA256 entries
in Release and Packages files, which is likely the reason for
python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries
back at least for buster/updates, and will file another bug report for
that in a moment to make sure it isn't lost.
Looking at even the most recent python-apt code in experimental (1.9.0),
MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary():
def fetch_binary(self, destdir='', progress=None):
# type: (str, AcquireProgress) -> str
"""Fetch the binary version of the package.
The parameter *destdir* specifies the directory where the package will
be fetched to.
The parameter *progress* may refer to an apt_pkg.AcquireProgress()
object. If not specified or None, apt.progress.text.AcquireProgress()
is used.
.. versionadded:: 0.7.10
"""
base = os.path.basename(self._records.filename)
destfile = os.path.join(destdir, base)
if _file_is_same(destfile, self.size, self._records.md5_hash):
logging.debug('Ignoring already existing file: %s' % destfile)
return os.path.abspath(destfile)
acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress())
acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # type: ignore # TODO: Do not use MD5 # nopep8
self.size, base, destfile=destfile)
acq.run()
if acqfile.status != acqfile.STAT_DONE:
raise FetchError("The item %r could not be fetched: %s" %
(acqfile.destfile, acqfile.error_text))
return os.path.abspath(destfile)
Notice the TODO on the apt_pkg.AcquireFile(), but it would probably
break in the same way as in the live-wrapper case a few lines before, on
the self._records.md5_hash item.
The same goes for fetch_source().
Since getting rid of MD5Sum entirely is a topic that comes up on a
regular fashion (with fingers being pointed at jigdo in particular), it
looks to me python-apt should get some attention as well; hence filing
at serious severity. Feel free to adjust as required.
Cheers,
--
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/
--- End Message ---
--- Begin Message ---
- To: 944696-close@bugs.debian.org
- Subject: Bug#944696: fixed in python-apt 1.8.4.1
- From: Julian Andres Klode <jak@debian.org>
- Date: Sat, 25 Jan 2020 19:02:12 +0000
- Message-id: <E1ivQhA-0001gU-4K@fasolo.debian.org>
Source: python-apt
Source-Version: 1.8.4.1
We believe that the bug you reported is fixed in the latest version of
python-apt, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 944696@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated python-apt package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Thu, 23 Jan 2020 11:10:21 +0100
Source: python-apt
Architecture: source
Version: 1.8.4.1
Distribution: buster-security
Urgency: high
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 944696
Changes:
python-apt (1.8.4.1) buster-security; urgency=high
.
* SECURITY UPDATE: Check that repository is trusted before downloading
files from it (LP: #1858973)
- apt/cache.py: Add checks to fetch_archives() and commit()
- apt/package.py: Add checks to fetch_binary() and fetch_source()
- CVE-2019-15796
* SECURITY UPDATE: Do not use MD5 for verifying downloadeds
(Closes: #944696) (#LP: #1858972)
- apt/package.py: Use all hashes when fetching packages, and
check that we have trusted hashes when downloading
- CVE-2019-15795
* To work around the new checks, the parameter allow_unauthenticated=True
can be passed to the functions. It defaults to the value of the
APT::Get::AllowUnauthenticated option.
* Automatic changes and fixes for external regressions:
- Adjustments to test suite and CI to fix CI regressions
- testcommon: Avoid reading host apt.conf files
- Automatic mirror list update
Checksums-Sha1:
d6fbf2cdd32052a4a24f7059be1d25dd99a393c4 2459 python-apt_1.8.4.1.dsc
1e9fbd73773c2f6ce7cfe5d015ce62918218e49b 343332 python-apt_1.8.4.1.tar.xz
9f73fc9364277b8eb5755f392e07c224a32b1f6c 10090 python-apt_1.8.4.1_source.buildinfo
Checksums-Sha256:
5659acc6cb5068dbcfe3aba00d29fa1b82d91f09c2c2ffbee78ebfc96e9803bb 2459 python-apt_1.8.4.1.dsc
e110b3fff9422c5e27b9cbd23f44e3c7f843d4517fef8b3c2058102b115b20b9 343332 python-apt_1.8.4.1.tar.xz
9517b4ebaaf9b88862021e8e89b18d9685d2a38a0f20c8bf4ddcf901062fa584 10090 python-apt_1.8.4.1_source.buildinfo
Files:
f999d2bef849206bd3f37245a7ab08b4 2459 python optional python-apt_1.8.4.1.dsc
d37f1e3142f62a7548b76c4164cd6a19 343332 python optional python-apt_1.8.4.1.tar.xz
b033d832dda3872ffd0e23d3b0d7ed67 10090 python optional python-apt_1.8.4.1_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAl4pcboPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9xw34P/jbhbydKH789ETDETqgrakPWSl+x7OPNgHUs
iH03neDuvkdmEkGKQc1rehAO8XOWnCQB5k1/BPFl6dQiqNa7w7XIA/Gf8Mv4lK2G
tCQ9juRg14LYOCAjM1CMKIy7WDRM1j2BYcqIbxlGcgqwcZa9hwUAO4ZEcjqv1P7g
wfBGC/GrG3fZmkbqHefWuM0lRBuLfcmqu9OHsSeemEosKfPvc0MI8OFFduGkbxPk
IEMwL6IrlpOKH/MWAB9qlzvCDqWojcfI4+ZHOu1XTWIY/n/FSASMJCgbTCVAmg8I
IyoPrsZ58IkNOT3y8luCe6YJE/0DYgoO2c8AR9TsfUxNyNmZW7WmWk1le79Ycj5H
DTM8HTCyil5dHE2ZklTA8OwitYngpgWgoRGduirJCox7iTPeVDM4pcQKUPJ4oRld
+HPN20it8r+hTtiBbgsAh7cKH6kcHQ6oxQalzQ+rrM9eUqZapXl8+qvIkT6t+HOf
SgBEx0uFPUL6TyisSrKbenN8ouDC3Sjh4cLftx2+zCq9qMaN92uaQWRGbFOV05h3
winIKszeFb4pp/YUgPclhIt5xEaSj3NC12Yg2s9NHcXJkAaB9uJuhAUC7wIw5uZy
wAqt9XCf+kgAE/ni+b5JDucgEIJQR5JQFhz827lGeRlOPA8sJoMiCN6balUK/1o1
fr3aW/y1
=IGbT
-----END PGP SIGNATURE-----
--- End Message ---