Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
Package: apt
Version: 1.0.9.1
Severity: minor
Hi,
Michael Vogt requested I file this as official bugreport:
The recent revalidation changes lead to a simple apt-get update
on my buildd machine (m68k VM with 198 MHz CPU) with only the
incoming.d-p.o repository modified, but not unstable, taking:
Fetched 156 kB in 8min 33s (304 B/s)
The sources.list on that chroot is just:
deb http://ftp.de.debian.org/debian-ports unstable main
deb http://ftp.de.debian.org/debian-ports unreleased main
deb-src http://ftp.de.debian.org/debian unstable main
# plus the “incoming” repository
With almost 200 MHz, this is one of the faster “machines”.
Our regular buildds have 25, 33, 50 MHz.
The cause for this is that even a not-modified now triggers
reverification of all downloaded files. The visible effect
is a process called “copy” taking >90% CPU for minutes, in
top(1).
Michael and I think that apt should, when the Release and
Release.gpg (or InRelease, but we aren’t using them any
more, IIRC?) files are not modified, just reverify these,
and, if they pass, assume the other data on the disc is
valid/verified (because the files only get moved out of
partial/ once verified AFAIK, and because if an unmodified
Release file passes reverification, it would have passed
initial verification as well, normally).
Michael also thinks apt could cache more information, such
as the original hashes, to speed up things more.
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "m68k";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "0";
APT::Install-Suggests "0";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-image-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^linux-headers-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-headers-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^linux-image-extra-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-image-extra-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^linux-signed-image-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-signed-image-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^kfreebsd-image-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^kfreebsd-image-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^kfreebsd-headers-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^kfreebsd-headers-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^gnumach-image-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^gnumach-image-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^.*-modules-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^.*-modules-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^.*-kernel-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^.*-kernel-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.16-1-m68k$";
APT::NeverAutoRemove:: "^linux-tools-3\.14-2-m68k$";
APT::NeverAutoRemove:: "^linux-tools-3\.16-1-m68k$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Get "";
APT::Get::AllowUnauthenticated "false";
APT::Architectures "";
APT::Architectures:: "m68k";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "4";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "5";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "/var/cache/pbuilder/aptcache";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::PDiffs "false";
Acquire::CompressionTypes "";
Acquire::CompressionTypes::Order "";
Acquire::CompressionTypes::Order:: "gz";
Acquire::http "";
Acquire::http::Pipeline-Depth "0";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
debug "";
debug::pkgproblemresolver "true";
CommandLine "";
CommandLine::AsString "apt-config dump";
-- /etc/apt/preferences --
Package: glib-networking-common
Pin: version 2.36.1-2~m68k.1
Pin-Priority: 1001
-- /etc/apt/sources.list --
# Debian/m68k /etc/apt/sources.list for regular use
#-
# Debian/m68k binaries come from debian-ports or one of its mirrors
deb http://ftp.de.debian.org/debian-ports unstable main
deb http://ftp.de.debian.org/debian-ports unreleased main
#deb http://ftp.de.debian.org/debian-ports experimental main
# Debian arch:all binaries can ingeniously come from a normal debian
# mirror, when dpo is not updating timely
#deb [arch=all] http://ftp.de.debian.org/debian unstable main
# Debian/m68k unstable/experimental source comes from a normal debian mirror
#T#deb-src http://ftp.de.debian.org/debian unstable main
#deb-src http://ftp.de.debian.org/debian experimental main
# Debian/m68k unreleased source comes from debian-ports mirrors
#T#deb-src http://ftp.de.debian.org/debian-ports unreleased main
# [ buildd stuff removed in reportbug ]
# If really needed: Allow etch-m68k to satisfy dependencies
#deb http://archive.debian.org/debian etch-m68k main contrib non-free
#deb-src http://archive.debian.org/debian etch-m68k main contrib non-free
### All following repositories need the 'wtf-debian-keyring' package
### installed (0x405422DD which is signed by 0xE99007E0 on DD keyring)
# Debian unstable + Debian-Ports unreleased + src:wtf-debian-keyring
# Only official Debian packages (and the keyring), long-lived, rarely
# updated; contains everything needed for debootstrap; edos-debcheck
# is used to keep all packages from this repository installable
deb http://shore.naturalnet.de/~tg/DP sid main
#deb-src deb http://shore.naturalnet.de/~tg/DP sid main
# Debian unstable + Debian-Ports unreleased (short-lived: just built
# and not on either dpo or the above one yet)
deb http://shore.naturalnet.de/~tg/DP m68k main
#deb-src http://shore.naturalnet.de/~tg/DP m68k main
# Mostly when we use older (source or binary) packages from proper Debian
deb http://shore.naturalnet.de/~tg/DP hacks clean
#deb-src http://shore.naturalnet.de/~tg/DP hacks clean
# Anything goes, even hex-edited .deb files and cross-built packages
#deb http://shore.naturalnet.de/~tg/DP hacks dirty
#deb-src http://shore.naturalnet.de/~tg/DP hacks dirty
# mirabilos' private packages (not official Debian)
deb http://shore.naturalnet.de/~tg/Debs sid wtf
#deb-src http://shore.naturalnet.de/~tg/Debs sid wtf
-- System Information:
Debian Release: jessie/sid
APT prefers unreleased
APT policy: (500, 'unreleased'), (500, 'unstable')
Architecture: m68k
Kernel: Linux 3.16-1-m68k
Locale: LANG=C, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/lksh
Versions of packages apt depends on:
ii debian-archive-keyring 2014.1
ii gnupg 1.4.18-4
ii libapt-pkg4.12 1.0.9.1
ii libc6 2.19-7
ii libgcc2 4.9.1-5
ii libstdc++6 4.9.1-5
apt recommends no packages.
Versions of packages apt suggests:
pn apt-doc <none>
pn aptitude | synaptic | wajig <none>
ii dpkg-dev 1.17.13
pn python-apt <none>
-- no debconf information
Reply to: