Bug#973826: apt update ignores Diff-index files (WeakHashSums)
Package: apt
Version: 2.1.11
Severity: normal
Since a few days I have been observing this bug during regular “apt
update” / “apt upgrade” cycles, without any configuration change on my
side; I am not completely sure if all this started after the last
upgrade to apt 2.1.11.
apt finds (testing, unstable, experimental) InRelease files, downloads
them from the mirror, checks the signatures, and then downloads
Diff-index files that it regularly ignores, be it for Packages, Sources,
or Contents. The result is always a complete new full download, without
ever considering pdiff files and patching.
After investigating a bit I tried using the following Debug options:
apt -o Debug::Hashes=true \
-o Debug::pkgAcquire::Diffs=true \
-o Debug::pkgAcquire=true \
-o Debug::Acquire::gpgv=true update
and this was the result:
[...]
Dequeuing /var/lib/apt/lists/partial/_etc_apt_mirrors_dists_unstable_main_binary-amd64_Packages.diff_Index
pkgAcqDiffIndex failed: https://ftp.halifax.rwth-aachen.de/debian/dists/unstable/main/binary-amd64/Packages.diff/by-hash/SHA256/659c48e636a76058a99bcb9b2d27eb3597ab5d72d6e763f1948541aba99d7618 with 201 URI Done
SHA256-Hash: 659c48e636a76058a99bcb9b2d27eb3597ab5d72d6e763f1948541aba99d7618
Checksum-FileSize-Hash: 9518
MD5Sum-Hash: bf69442a56c7d11ea9c4d036fe1b170a
MD5-Hash: bf69442a56c7d11ea9c4d036fe1b170a
Last-Modified: Wed, 04 Nov 2020 14:13:12 +0000
Size: 9518
Filename: /var/lib/apt/lists/partial/_etc_apt_mirrors_dists_unstable_main_binary-amd64_Packages.diff_Index
URI: https://ftp.halifax.rwth-aachen.de/debian/dists/unstable/main/binary-amd64/Packages.diff/by-hash/SHA256/659c48e636a76058a99bcb9b2d27eb3597ab5d72d6e763f1948541aba99d7618
FailReason: WeakHashSums
Falling back to normal index file acquire
Fetching mirror+file:/etc/apt/mirrors/dists/unstable/main/binary-amd64/by-hash/SHA256/5059f39d518f9a34159b36adb6ab2f094f8ce937f63e6fada7041a1aea7cce76
to /var/lib/apt/lists/partial/_etc_apt_mirrors_dists_unstable_main_binary-amd64_Packages.xz
Queue is: mirror+file
Dequeuing /var/lib/apt/lists/partial/_etc_apt_mirrors_dists_unstable_main_binary-amd64_Packages.xz
Dequeued from mirror+file
Fetching https://ftp.halifax.rwth-aachen.de/debian/dists/unstable/main/binary-amd64/by-hash/SHA256/5059f39d518f9a34159b36adb6ab2f094f8ce937f63e6fada7041a1aea7cce76
[...]
This is only an excerpt, but the same happens for all other suites and
components: the error is always "FailReason: WeakHashSums".
I then tried other mirrors getting the same results, then I manually
downloaded some of the Index files from the mirror and verified their
SHA256 hash sums: they correspond to what is written in the InRelease
files. This makes me think that apt is responsible for the failure in
hash sums checking.
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-.*-5\.8\.0-3-amd64$";
APT::NeverAutoRemove:: "^linux-.*-5\.9\.0-1-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-.*-5\.8\.0-3-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-.*-5\.9\.0-1-amd64$";
APT::NeverAutoRemove:: "^gnumach-.*-5\.8\.0-3-amd64$";
APT::NeverAutoRemove:: "^gnumach-.*-5\.9\.0-1-amd64$";
APT::NeverAutoRemove:: "^.*-modules-5\.8\.0-3-amd64$";
APT::NeverAutoRemove:: "^.*-modules-5\.9\.0-1-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-5\.8\.0-3-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-5\.9\.0-1-amd64$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-.*";
APT::VersionedKernelPackages:: "kfreebsd-.*";
APT::VersionedKernelPackages:: "gnumach-.*";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "contrib/metapackages";
APT::Never-MarkAuto-Sections:: "non-free/metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Move-Autobit-Sections "";
APT::Move-Autobit-Sections:: "oldlibs";
APT::Move-Autobit-Sections:: "contrib/oldlibs";
APT::Move-Autobit-Sections:: "non-free/oldlibs";
APT::Move-Autobit-Sections:: "restricted/oldlibs";
APT::Move-Autobit-Sections:: "universe/oldlibs";
APT::Move-Autobit-Sections:: "multiverse/oldlibs";
APT::Keep-Downloaded-Packages "true";
APT::Update "";
APT::Update::Post-Invoke "";
APT::Update::Post-Invoke:: "[ ! -x /usr/bin/debtags ] || debtags update || true";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "0";
APT::Compressor::zstd "";
APT::Compressor::zstd::Name "zstd";
APT::Compressor::zstd::Extension ".zst";
APT::Compressor::zstd::Binary "zstd";
APT::Compressor::zstd::Cost "60";
APT::Compressor::zstd::CompressArg "";
APT::Compressor::zstd::CompressArg:: "-19";
APT::Compressor::zstd::UncompressArg "";
APT::Compressor::zstd::UncompressArg:: "-d";
APT::Compressor::lz4 "";
APT::Compressor::lz4::Name "lz4";
APT::Compressor::lz4::Extension ".lz4";
APT::Compressor::lz4::Binary "lz4";
APT::Compressor::lz4::Cost "50";
APT::Compressor::lz4::CompressArg "";
APT::Compressor::lz4::CompressArg:: "-1";
APT::Compressor::lz4::UncompressArg "";
APT::Compressor::lz4::UncompressArg:: "-d";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "100";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-6n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "200";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "300";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-6";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "400";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-6";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::netrcparts "auth.conf.d";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Etc::apt-file-main "apt-file.conf";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::planners "";
Dir::Bin::planners:: "/usr/lib/apt/planners";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::gzip "/bin/gzip";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lz4 "/usr/bin/lz4";
Dir::Bin::zstd "/usr/bin/zstd";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/cdrom";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Log::Planner "eipp.log.xz";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.ucf-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::AllowInsecureRepositories "0";
Acquire::AllowWeakRepositories "0";
Acquire::AllowDowngradeToInsecureRepositories "0";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom";
Acquire::IndexTargets "";
Acquire::IndexTargets::deb "";
Acquire::IndexTargets::deb::Packages "";
Acquire::IndexTargets::deb::Packages::MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
Acquire::IndexTargets::deb::Packages::flatMetaKey "Packages";
Acquire::IndexTargets::deb::Packages::ShortDescription "Packages";
Acquire::IndexTargets::deb::Packages::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
Acquire::IndexTargets::deb::Packages::flatDescription "$(RELEASE) Packages";
Acquire::IndexTargets::deb::Packages::Optional "0";
Acquire::IndexTargets::deb::Translations "";
Acquire::IndexTargets::deb::Translations::MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatMetaKey "$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::ShortDescription "Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatDescription "$(RELEASE) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Contents-deb "";
Acquire::IndexTargets::deb::Contents-deb::MetaKey "$(COMPONENT)/Contents-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-deb::ShortDescription "Contents-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-deb::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Contents (deb)";
Acquire::IndexTargets::deb::Contents-deb::flatMetaKey "Contents-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-deb::flatDescription "$(RELEASE) Contents (deb)";
Acquire::IndexTargets::deb::Contents-deb::PDiffs "true";
Acquire::IndexTargets::deb::Contents-deb::KeepCompressed "true";
Acquire::IndexTargets::deb::Contents-udeb "";
Acquire::IndexTargets::deb::Contents-udeb::MetaKey "$(COMPONENT)/Contents-udeb-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-udeb::ShortDescription "Contents-udeb-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-udeb::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Contents (udeb)";
Acquire::IndexTargets::deb::Contents-udeb::flatMetaKey "Contents-udeb-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-udeb::flatDescription "$(RELEASE) Contents (udeb)";
Acquire::IndexTargets::deb::Contents-udeb::KeepCompressed "true";
Acquire::IndexTargets::deb::Contents-udeb::PDiffs "true";
Acquire::IndexTargets::deb::Contents-udeb::DefaultEnabled "false";
Acquire::IndexTargets::deb::Contents-deb-legacy "";
Acquire::IndexTargets::deb::Contents-deb-legacy::MetaKey "Contents-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-deb-legacy::ShortDescription "Contents-$(ARCHITECTURE)";
Acquire::IndexTargets::deb::Contents-deb-legacy::Description "$(RELEASE) $(ARCHITECTURE) Contents (deb)";
Acquire::IndexTargets::deb::Contents-deb-legacy::PDiffs "true";
Acquire::IndexTargets::deb::Contents-deb-legacy::KeepCompressed "true";
Acquire::IndexTargets::deb::Contents-deb-legacy::Fallback-Of "Contents-deb";
Acquire::IndexTargets::deb::Contents-deb-legacy::Identifier "Contents-deb";
Acquire::IndexTargets::deb-src "";
Acquire::IndexTargets::deb-src::Sources "";
Acquire::IndexTargets::deb-src::Sources::MetaKey "$(COMPONENT)/source/Sources";
Acquire::IndexTargets::deb-src::Sources::flatMetaKey "Sources";
Acquire::IndexTargets::deb-src::Sources::ShortDescription "Sources";
Acquire::IndexTargets::deb-src::Sources::Description "$(RELEASE)/$(COMPONENT) Sources";
Acquire::IndexTargets::deb-src::Sources::flatDescription "$(RELEASE) Sources";
Acquire::IndexTargets::deb-src::Sources::Optional "0";
Acquire::IndexTargets::deb-src::Contents-dsc "";
Acquire::IndexTargets::deb-src::Contents-dsc::MetaKey "$(COMPONENT)/Contents-source";
Acquire::IndexTargets::deb-src::Contents-dsc::ShortDescription "Contents-source";
Acquire::IndexTargets::deb-src::Contents-dsc::Description "$(RELEASE)/$(COMPONENT) source Contents (dsc)";
Acquire::IndexTargets::deb-src::Contents-dsc::flatMetaKey "Contents-source";
Acquire::IndexTargets::deb-src::Contents-dsc::flatDescription "$(RELEASE) Contents (dsc)";
Acquire::IndexTargets::deb-src::Contents-dsc::PDiffs "true";
Acquire::IndexTargets::deb-src::Contents-dsc::KeepCompressed "true";
Acquire::IndexTargets::deb-src::Contents-dsc::DefaultEnabled "false";
Acquire::Changelogs "";
Acquire::Changelogs::URI "";
Acquire::Changelogs::URI::Origin "";
Acquire::Changelogs::URI::Origin::Debian "https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog";;
Acquire::Changelogs::URI::Origin::Ubuntu "https://changelogs.ubuntu.com/changelogs/pool/@CHANGEPATH@/changelog";;
Acquire::Changelogs::AlwaysOnline "";
Acquire::Changelogs::AlwaysOnline::Origin "";
Acquire::Changelogs::AlwaysOnline::Origin::Ubuntu "1";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
Acquire::CompressionTypes "";
Acquire::CompressionTypes::xz "xz";
Acquire::CompressionTypes::bz2 "bzip2";
Acquire::CompressionTypes::lzma "lzma";
Acquire::CompressionTypes::gz "gzip";
Acquire::CompressionTypes::lz4 "lz4";
Acquire::CompressionTypes::zst "zstd";
DPkg "";
DPkg::Path "/usr/sbin:/usr/bin:/sbin:/bin";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -lt 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
DPkg::Tools::Options::/usr/bin/apt-listchanges::InfoFD "20";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "[ -x /usr/lib/libdvd-pkg/b-i_libdvdcss.sh ] && /usr/lib/libdvd-pkg/b-i_libdvdcss.sh || true";
DPkg::Post-Invoke:: "test -x /usr/lib/needrestart/apt-pinvoke && /usr/lib/needrestart/apt-pinvoke || true";
Binary "apt-config";
Binary::apt "";
Binary::apt::APT "";
Binary::apt::APT::Keep-Downloaded-Packages "true";
Binary::apt::APT::Color "1";
Binary::apt::APT::Cache "";
Binary::apt::APT::Cache::Show "";
Binary::apt::APT::Cache::Show::Version "2";
Binary::apt::APT::Cache::AllVersions "0";
Binary::apt::APT::Cache::ShowVirtuals "1";
Binary::apt::APT::Cache::Search "";
Binary::apt::APT::Cache::Search::Version "2";
Binary::apt::APT::Cache::ShowDependencyType "1";
Binary::apt::APT::Cache::ShowVersion "1";
Binary::apt::APT::Get "";
Binary::apt::APT::Get::Upgrade-Allow-New "1";
Binary::apt::APT::Get::Update "";
Binary::apt::APT::Get::Update::InteractiveReleaseInfoChanges "1";
Binary::apt::APT::Cmd "";
Binary::apt::APT::Cmd::Show-Update-Stats "1";
Binary::apt::APT::Cmd::Pattern-Only "1";
Binary::apt::DPkg "";
Binary::apt::DPkg::Progress-Fancy "1";
Binary::apt::DPkg::Lock "";
Binary::apt::DPkg::Lock::Timeout "-1";
apt-file "";
apt-file::Index-Names "deb";
apt-file::Parser "";
apt-file::Parser::Check-For-Description-Header "false";
CommandLine "";
CommandLine::AsString "apt-config dump";
-- /etc/apt/preferences --
Package: *
Pin: release a=testing
Pin-Priority: 900
Package: *
Pin: release a=unstable
Pin-Priority: 800
Package: *
Pin: release o=Debian
Pin-Priority: -10
-- (no /etc/apt/preferences.d/* present) --
-- (no /etc/apt/sources.list present) --
-- (/etc/apt/sources.list.d/01-all.sources present, but not submitted) --
-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (900, 'testing'), (800, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.9.0-1-amd64 (SMP w/4 CPU threads)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages apt depends on:
ii adduser 3.118
ii debian-archive-keyring 2019.1
ii gpgv 2.2.20-1
ii libapt-pkg6.0 2.1.11
ii libc6 2.31-4
ii libgcc-s1 10.2.0-16
ii libgnutls30 3.6.15-4
ii libseccomp2 2.4.4-1+b1
ii libstdc++6 10.2.0-16
ii libsystemd0 246.6-2
Versions of packages apt recommends:
ii ca-certificates 20200601
Versions of packages apt suggests:
ii apt-doc 2.1.11
ii aptitude 0.8.13-2
ii dpkg-dev 1.20.5
ii gnupg 2.2.20-1
ii powermgmt-base 1.36
-- no debconf information
Reply to: