Bug#762178: apt: massive a-g update slowdowns due to recent security fixes
On Fri, Sep 19, 2014 at 09:00:43AM +0000, Thorsten Glaser wrote:
> Package: apt
> Version: 1.0.9.1
> Severity: minor
Thanks for your bugreport.
> Michael Vogt requested I file this as official bugreport:
>
> The recent revalidation changes lead to a simple apt-get update
> on my buildd machine (m68k VM with 198 MHz CPU) with only the
> incoming.d-p.o repository modified, but not unstable, taking:
>
> Fetched 156 kB in 8min 33s (304 B/s)
>
> The sources.list on that chroot is just:
>
> deb http://ftp.de.debian.org/debian-ports unstable main
> deb http://ftp.de.debian.org/debian-ports unreleased main
> deb-src http://ftp.de.debian.org/debian unstable main
> # plus the “incoming” repository
>
> With almost 200 MHz, this is one of the faster “machines”.
> Our regular buildds have 25, 33, 50 MHz.
[..]
This is indeed a pretty big hit on this hardware :/
The following patch should fix most of this:
diff --git a/apt-pkg/acquire-item.cc b/apt-pkg/acquire-item.cc
index 5df4372..28ede30 100644
--- a/apt-pkg/acquire-item.cc
+++ b/apt-pkg/acquire-item.cc
@@ -1558,8 +1558,9 @@ void pkgAcqMetaIndex::AuthDone(string Message)
/*{{{*/
}
- // Download further indexes with verification
- QueueIndexes(true);
+ // Download further indexes with verification if something has changed
+ if(IMSHit == false)
+ QueueIndexes(true);
// is it a clearsigned MetaIndex file?
if (DestFile == SigFile)
The downside if of course that if a Packages file is missing (because
of a earlier download error for example) then it won't be fetched
again, so the patch is probably too simplistic. But I think something
like this should be done to avoid the extra verify step if we know
that nothing has changed.
Cheers,
Michael
> The cause for this is that even a not-modified now triggers
> reverification of all downloaded files. The visible effect
> is a process called “copy” taking >90% CPU for minutes, in
> top(1).
>
> Michael and I think that apt should, when the Release and
> Release.gpg (or InRelease, but we aren’t using them any
> more, IIRC?) files are not modified, just reverify these,
> and, if they pass, assume the other data on the disc is
> valid/verified (because the files only get moved out of
> partial/ once verified AFAIK, and because if an unmodified
> Release file passes reverification, it would have passed
> initial verification as well, normally).
>
> Michael also thinks apt could cache more information, such
> as the original hashes, to speed up things more.
>
>
> -- Package-specific info:
>
> -- apt-config dump --
>
> APT "";
> APT::Architecture "m68k";
> APT::Build-Essential "";
> APT::Build-Essential:: "build-essential";
> APT::Install-Recommends "0";
> APT::Install-Suggests "0";
> APT::NeverAutoRemove "";
> APT::NeverAutoRemove:: "^firmware-linux.*";
> APT::NeverAutoRemove:: "^linux-firmware$";
> APT::NeverAutoRemove:: "^linux-image-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-image-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^linux-headers-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-headers-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^linux-image-extra-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-image-extra-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^linux-signed-image-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-signed-image-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^kfreebsd-image-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^kfreebsd-image-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^kfreebsd-headers-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^kfreebsd-headers-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^gnumach-image-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^gnumach-image-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^.*-modules-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^.*-modules-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^.*-kernel-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^.*-kernel-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.16-1-m68k$";
> APT::NeverAutoRemove:: "^linux-tools-3\.14-2-m68k$";
> APT::NeverAutoRemove:: "^linux-tools-3\.16-1-m68k$";
> APT::VersionedKernelPackages "";
> APT::VersionedKernelPackages:: "linux-image";
> APT::VersionedKernelPackages:: "linux-headers";
> APT::VersionedKernelPackages:: "linux-image-extra";
> APT::VersionedKernelPackages:: "linux-signed-image";
> APT::VersionedKernelPackages:: "kfreebsd-image";
> APT::VersionedKernelPackages:: "kfreebsd-headers";
> APT::VersionedKernelPackages:: "gnumach-image";
> APT::VersionedKernelPackages:: ".*-modules";
> APT::VersionedKernelPackages:: ".*-kernel";
> APT::VersionedKernelPackages:: "linux-backports-modules-.*";
> APT::VersionedKernelPackages:: "linux-tools";
> APT::Never-MarkAuto-Sections "";
> APT::Never-MarkAuto-Sections:: "metapackages";
> APT::Never-MarkAuto-Sections:: "restricted/metapackages";
> APT::Never-MarkAuto-Sections:: "universe/metapackages";
> APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
> APT::Never-MarkAuto-Sections:: "oldlibs";
> APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
> APT::Never-MarkAuto-Sections:: "universe/oldlibs";
> APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
> APT::Get "";
> APT::Get::AllowUnauthenticated "false";
> APT::Architectures "";
> APT::Architectures:: "m68k";
> APT::Compressor "";
> APT::Compressor::. "";
> APT::Compressor::.::Name ".";
> APT::Compressor::.::Extension "";
> APT::Compressor::.::Binary "";
> APT::Compressor::.::Cost "1";
> APT::Compressor::gzip "";
> APT::Compressor::gzip::Name "gzip";
> APT::Compressor::gzip::Extension ".gz";
> APT::Compressor::gzip::Binary "gzip";
> APT::Compressor::gzip::Cost "2";
> APT::Compressor::gzip::CompressArg "";
> APT::Compressor::gzip::CompressArg:: "-9n";
> APT::Compressor::gzip::UncompressArg "";
> APT::Compressor::gzip::UncompressArg:: "-d";
> APT::Compressor::bzip2 "";
> APT::Compressor::bzip2::Name "bzip2";
> APT::Compressor::bzip2::Extension ".bz2";
> APT::Compressor::bzip2::Binary "bzip2";
> APT::Compressor::bzip2::Cost "3";
> APT::Compressor::bzip2::CompressArg "";
> APT::Compressor::bzip2::CompressArg:: "-9";
> APT::Compressor::bzip2::UncompressArg "";
> APT::Compressor::bzip2::UncompressArg:: "-d";
> APT::Compressor::xz "";
> APT::Compressor::xz::Name "xz";
> APT::Compressor::xz::Extension ".xz";
> APT::Compressor::xz::Binary "xz";
> APT::Compressor::xz::Cost "4";
> APT::Compressor::xz::CompressArg "";
> APT::Compressor::xz::CompressArg:: "-6";
> APT::Compressor::xz::UncompressArg "";
> APT::Compressor::xz::UncompressArg:: "-d";
> APT::Compressor::lzma "";
> APT::Compressor::lzma::Name "lzma";
> APT::Compressor::lzma::Extension ".lzma";
> APT::Compressor::lzma::Binary "xz";
> APT::Compressor::lzma::Cost "5";
> APT::Compressor::lzma::CompressArg "";
> APT::Compressor::lzma::CompressArg:: "--format=lzma";
> APT::Compressor::lzma::CompressArg:: "-9";
> APT::Compressor::lzma::UncompressArg "";
> APT::Compressor::lzma::UncompressArg:: "--format=lzma";
> APT::Compressor::lzma::UncompressArg:: "-d";
> Dir "/";
> Dir::State "var/lib/apt/";
> Dir::State::lists "lists/";
> Dir::State::cdroms "cdroms.list";
> Dir::State::mirrors "mirrors/";
> Dir::State::extended_states "extended_states";
> Dir::State::status "/var/lib/dpkg/status";
> Dir::Cache "var/cache/apt/";
> Dir::Cache::archives "/var/cache/pbuilder/aptcache";
> Dir::Cache::srcpkgcache "srcpkgcache.bin";
> Dir::Cache::pkgcache "pkgcache.bin";
> Dir::Etc "etc/apt/";
> Dir::Etc::sourcelist "sources.list";
> Dir::Etc::sourceparts "sources.list.d";
> Dir::Etc::vendorlist "vendors.list";
> Dir::Etc::vendorparts "vendors.list.d";
> Dir::Etc::main "apt.conf";
> Dir::Etc::netrc "auth.conf";
> Dir::Etc::parts "apt.conf.d";
> Dir::Etc::preferences "preferences";
> Dir::Etc::preferencesparts "preferences.d";
> Dir::Etc::trusted "trusted.gpg";
> Dir::Etc::trustedparts "trusted.gpg.d";
> Dir::Bin "";
> Dir::Bin::methods "/usr/lib/apt/methods";
> Dir::Bin::solvers "";
> Dir::Bin::solvers:: "/usr/lib/apt/solvers";
> Dir::Bin::dpkg "/usr/bin/dpkg";
> Dir::Bin::bzip2 "/bin/bzip2";
> Dir::Bin::xz "/usr/bin/xz";
> Dir::Bin::lzma "/usr/bin/xz";
> Dir::Media "";
> Dir::Media::MountPath "/media/apt";
> Dir::Log "var/log/apt";
> Dir::Log::Terminal "term.log";
> Dir::Log::History "history.log";
> Dir::Ignore-Files-Silently "";
> Dir::Ignore-Files-Silently:: "~$";
> Dir::Ignore-Files-Silently:: "\.disabled$";
> Dir::Ignore-Files-Silently:: "\.bak$";
> Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
> Dir::Ignore-Files-Silently:: "\.save$";
> Dir::Ignore-Files-Silently:: "\.orig$";
> Dir::Ignore-Files-Silently:: "\.distUpgrade$";
> Acquire "";
> Acquire::cdrom "";
> Acquire::cdrom::mount "/media/cdrom/";
> Acquire::PDiffs "false";
> Acquire::CompressionTypes "";
> Acquire::CompressionTypes::Order "";
> Acquire::CompressionTypes::Order:: "gz";
> Acquire::http "";
> Acquire::http::Pipeline-Depth "0";
> Acquire::Languages "";
> Acquire::Languages:: "en";
> Acquire::Languages:: "none";
> DPkg "";
> DPkg::Pre-Install-Pkgs "";
> DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
> debug "";
> debug::pkgproblemresolver "true";
> CommandLine "";
> CommandLine::AsString "apt-config dump";
>
> -- /etc/apt/preferences --
>
> Package: glib-networking-common
> Pin: version 2.36.1-2~m68k.1
> Pin-Priority: 1001
>
>
> -- /etc/apt/sources.list --
>
> # Debian/m68k /etc/apt/sources.list for regular use
> #-
>
> # Debian/m68k binaries come from debian-ports or one of its mirrors
> deb http://ftp.de.debian.org/debian-ports unstable main
> deb http://ftp.de.debian.org/debian-ports unreleased main
> #deb http://ftp.de.debian.org/debian-ports experimental main
>
> # Debian arch:all binaries can ingeniously come from a normal debian
> # mirror, when dpo is not updating timely
> #deb [arch=all] http://ftp.de.debian.org/debian unstable main
>
> # Debian/m68k unstable/experimental source comes from a normal debian mirror
> #T#deb-src http://ftp.de.debian.org/debian unstable main
> #deb-src http://ftp.de.debian.org/debian experimental main
>
> # Debian/m68k unreleased source comes from debian-ports mirrors
> #T#deb-src http://ftp.de.debian.org/debian-ports unreleased main
>
> # [ buildd stuff removed in reportbug ]
>
> # If really needed: Allow etch-m68k to satisfy dependencies
> #deb http://archive.debian.org/debian etch-m68k main contrib non-free
> #deb-src http://archive.debian.org/debian etch-m68k main contrib non-free
>
> ### All following repositories need the 'wtf-debian-keyring' package
> ### installed (0x405422DD which is signed by 0xE99007E0 on DD keyring)
>
> # Debian unstable + Debian-Ports unreleased + src:wtf-debian-keyring
> # Only official Debian packages (and the keyring), long-lived, rarely
> # updated; contains everything needed for debootstrap; edos-debcheck
> # is used to keep all packages from this repository installable
> deb http://shore.naturalnet.de/~tg/DP sid main
> #deb-src deb http://shore.naturalnet.de/~tg/DP sid main
>
> # Debian unstable + Debian-Ports unreleased (short-lived: just built
> # and not on either dpo or the above one yet)
> deb http://shore.naturalnet.de/~tg/DP m68k main
> #deb-src http://shore.naturalnet.de/~tg/DP m68k main
>
> # Mostly when we use older (source or binary) packages from proper Debian
> deb http://shore.naturalnet.de/~tg/DP hacks clean
> #deb-src http://shore.naturalnet.de/~tg/DP hacks clean
>
> # Anything goes, even hex-edited .deb files and cross-built packages
> #deb http://shore.naturalnet.de/~tg/DP hacks dirty
> #deb-src http://shore.naturalnet.de/~tg/DP hacks dirty
>
> # mirabilos' private packages (not official Debian)
> deb http://shore.naturalnet.de/~tg/Debs sid wtf
> #deb-src http://shore.naturalnet.de/~tg/Debs sid wtf
>
> -- System Information:
> Debian Release: jessie/sid
> APT prefers unreleased
> APT policy: (500, 'unreleased'), (500, 'unstable')
> Architecture: m68k
>
> Kernel: Linux 3.16-1-m68k
> Locale: LANG=C, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/lksh
>
> Versions of packages apt depends on:
> ii debian-archive-keyring 2014.1
> ii gnupg 1.4.18-4
> ii libapt-pkg4.12 1.0.9.1
> ii libc6 2.19-7
> ii libgcc2 4.9.1-5
> ii libstdc++6 4.9.1-5
>
> apt recommends no packages.
>
> Versions of packages apt suggests:
> pn apt-doc <none>
> pn aptitude | synaptic | wajig <none>
> ii dpkg-dev 1.17.13
> pn python-apt <none>
>
> -- no debconf information
>
>
> --
> To UNSUBSCRIBE, email to deity-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: [🔎] 20140919090043.24295.29004.reportbug@ara4.mirbsd.org">https://lists.debian.org/[🔎] 20140919090043.24295.29004.reportbug@ara4.mirbsd.org
>
Reply to: