Bug#944696: python-apt: relies on MD5 internally to download packages

[Cyril Brulebois <cyril@debamax.com>]

Package: python-apt
Version: 1.8.4
Severity: serious
Justification: some people want to get rid of MD5Sum in indices

Hi,

While debugging a live-wrapper (lwr) failure that started occurring
(literally) overnight, I ended up discovering it was triggered by the
intel-microcode package's getting a security upgrade.

live-wrapper 0.10 isn't affected, but live-wrapper's master branch has
an extra commit that automatically enables security sources for stable
releases.

Here's the traceback for a simple build (with a local mirror but anyone
would do) with that master branch:

    $ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode
    […]
    DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian'
    DEBUG environment: LWR_EXTRA_PACKAGES = ''
    DEBUG environment: LWR_BASE_DEBS = ''
    DEBUG environment: LWR_DISTRIBUTION = 'buster'
    DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode'
    DEBUG environment: LWR_TASK_PACKAGES = ''
    […]
    Downloading udebs for Debian Installer...
    INFO Downloading udebs for Debian Installer...
    Updating a local cache for amd64 buster ...
    DEBUG Updating local cache...
    CRITICAL Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run
        self.process_args(args)
      File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args
        self.start_ops()
      File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops
        apt_udeb.download_udebs(exclude_list)
      File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs
        self.download_apt_file(pkg_name, pool_dir, False)
      File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file
        version.fetch_binary(destdir=pkg_dir)
      File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary
        if _file_is_same(destfile, self.size, self._records.md5_hash):
    SystemError: error return without exception set


After some debugging, it turned out that merely accessing the
self._records.md5_hash item is sufficient to reproduce this issue.

Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for
buster/updates on security.debian.org, one can only see SHA256 entries
in Release and Packages files, which is likely the reason for
python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries
back at least for buster/updates, and will file another bug report for
that in a moment to make sure it isn't lost.

Looking at even the most recent python-apt code in experimental (1.9.0),
MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary():


    def fetch_binary(self, destdir='', progress=None):
        # type: (str, AcquireProgress) -> str
        """Fetch the binary version of the package.

        The parameter *destdir* specifies the directory where the package will
        be fetched to.

        The parameter *progress* may refer to an apt_pkg.AcquireProgress()
        object. If not specified or None, apt.progress.text.AcquireProgress()
        is used.

        .. versionadded:: 0.7.10
        """
        base = os.path.basename(self._records.filename)
        destfile = os.path.join(destdir, base)
        if _file_is_same(destfile, self.size, self._records.md5_hash):
            logging.debug('Ignoring already existing file: %s' % destfile)
            return os.path.abspath(destfile)
        acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress())
        acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash,  # type: ignore # TODO: Do not use MD5 # nopep8
                                      self.size, base, destfile=destfile)
        acq.run()

        if acqfile.status != acqfile.STAT_DONE:
            raise FetchError("The item %r could not be fetched: %s" %
                             (acqfile.destfile, acqfile.error_text))

        return os.path.abspath(destfile)


Notice the TODO on the apt_pkg.AcquireFile(), but it would probably
break in the same way as in the live-wrapper case a few lines before, on
the self._records.md5_hash item.

The same goes for fetch_source().


Since getting rid of MD5Sum entirely is a topic that comes up on a
regular fashion (with fingers being pointed at jigdo in particular), it
looks to me python-apt should get some attention as well; hence filing
at serious severity. Feel free to adjust as required.


Cheers,
-- 
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/