Bug#784906: apt does not need to repeatedly re-download package file information when switching between apt servers

To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Bug#784906: apt does not need to repeatedly re-download package file information when switching between apt servers
From: Faheem Mitha <faheem@faheem.info>
Date: Sun, 10 May 2015 18:15:05 +0530
Message-id: <[🔎] 20150510124505.23186.15518.reportbug@orwell.homelinux.org>
Reply-to: Faheem Mitha <faheem@faheem.info>, 784906@bugs.debian.org
Package: apt
Version: 1.0.9.8
Severity: wishlist

Dear Maintainer,

I noticed that if I switch servers, apt goes ahead and re-downloads
the package file information from scratch, completely ignoring the
already existing file. But usually these files are similar, if not
identical. Also, as of the just released jessie, these files are
increasingly significant in size, currently of the order of 20 MB.

As described in the IRC snippet below, one possibiliy would be to
check hashes of files, and if they are identical, then hardlink to an
existing file.

###################################################################

Snipper from #debian-apt follow:

<faheem> I'm sure I'm not saying anything that others haven't thought
of before, but I noticed (not for the first time) that apt downloads
the same package information from every mirror, even though each
mirror has probably similar if not identical information. Can this
step not be reduced or skipped?

<mvo> faheem: that is a interessting idea, we know what hashes to
expect and we could simply hardlink a existing identical file if the
hash is the same


-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-image-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-headers-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-headers-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-image-extra-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-image-extra-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-signed-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-signed-image-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-image-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-headers-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^kfreebsd-headers-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^gnumach-image-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^gnumach-image-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-modules-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-modules-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^.*-kernel-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-backports-modules-.*-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-tools-3\.16\.0-4-amd64$";
APT::NeverAutoRemove:: "^linux-tools-3\.2\.0-4-amd64$";
APT::NeverAutoRemove:: "^postgresql-";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Update "";
APT::Update::Post-Invoke-Success "";
APT::Update::Post-Invoke-Success:: "test -x /usr/bin/apt-show-versions || exit 0 ; apt-show-versions -i";
APT::Update::Post-Invoke-Success:: "[ ! -f /var/run/dbus/system_bus_socket ] || /usr/bin/dbus-send --system --dest=org.debian.apt --type=signal /org/debian/apt org.debian.apt.CacheChanged || true";
APT::Update::Post-Invoke-Success:: "/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
APT::Update::Post-Invoke "";
APT::Update::Post-Invoke:: "[ ! -x /usr/bin/debtags ] || debtags update || true";
APT::Periodic "";
APT::Periodic::Unattended-Upgrade "1";
APT::Get "";
APT::Get::Build-Dep-Automatic "true";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Architectures:: "i386";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "4";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "5";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/cdrom";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
DPkg "";
DPkg::Pre-Invoke "";
DPkg::Pre-Invoke:: "if [ -x /usr/bin/etckeeper ]; then etckeeper pre-install; fi";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -x /usr/bin/etckeeper ]; then etckeeper post-install; fi";
DPkg::Post-Invoke:: "/usr/bin/test -e /usr/share/dbus-1/system-services/org.freedesktop.PackageKit.service && /usr/bin/test -S /var/run/dbus/system_bus_socket && /usr/bin/gdbus call --system --dest org.freedesktop.PackageKit --object-path /org/freedesktop/PackageKit --timeout 4 --method org.freedesktop.PackageKit.StateHasChanged cache-update > /dev/null; /bin/echo > /dev/null";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
RPM "";
RPM::Pre-Invoke "";
RPM::Pre-Invoke:: "if [ -x /usr/bin/etckeeper ]; then etckeeper pre-install; fi";
RPM::Post-Invoke "";
RPM::Post-Invoke:: "if [ -x /usr/bin/etckeeper ]; then etckeeper post-install; fi";
Unattended-Upgrade "";
Unattended-Upgrade::Origins-Pattern "";
Unattended-Upgrade::Origins-Pattern:: "origin=Debian,archive=${distro_codename},label=Debian-Security";
Unattended-Upgrade::Origins-Pattern:: "l=smuxi-daily";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- /etc/apt/preferences --

Package: *
Pin: release a=testing
Pin-Priority: 50

# Package: *
# Pin: release n=jessie
# Pin-Priority: 50

Package: *
Pin: release a=unstable
Pin-Priority: 50

Package: *
Pin: origin backports.debian.org
Pin-Priority: 50

Package: *
Pin: release a=experimental
Pin-Priority: 1

Package: *
Pin: release a=snapshot.debian.net
Pin-Priority: 1

Package: *
Pin: origin www.deb-multimedia.org
Pin-Priority: 1

Package: *
Pin: origin deb.grml.org
Pin-Priority: 1

-- /etc/apt/sources.list --

#http://debian.mirrors.ovh.net/debian/
#http://httpredir.debian.org/

## wheezy sources

deb http://httpredir.debian.org/debian/ wheezy main non-free contrib
deb-src http://httpredir.debian.org/debian/ wheezy main non-free contrib

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

# wheezy-updates, previously known as 'volatile'
deb http://httpredir.debian.org/debian/ wheezy-updates main contrib non-free
deb-src http://httpredir.debian.org/debian/ wheezy-updates main contrib non-free

deb http://httpredir.debian.org/debian/ wheezy-backports main contrib non-free
deb-src http://httpredir.debian.org/debian/ wheezy-backports main contrib non-free

deb http://lib.stat.cmu.edu/R/CRAN/bin/linux/debian wheezy-cran3/
deb-src http://lib.stat.cmu.edu/R/CRAN/bin/linux/debian wheezy-cran3/

deb http://www.deb-multimedia.org wheezy main non-free
deb-src http://www.deb-multimedia.org wheezy main non-free

## jessie sources

deb http://httpredir.debian.org/debian/ jessie main non-free contrib
deb-src http://httpredir.debian.org/debian/ jessie main non-free contrib

deb http://security.debian.org/ jessie/updates main contrib non-free
deb-src http://security.debian.org/ jessie/updates main contrib non-free

# jessie-updates, previously known as 'volatile'
deb http://httpredir.debian.org/debian/ jessie-updates main contrib non-free
deb-src http://httpredir.debian.org/debian/ jessie-updates main contrib non-free

# unstable source

deb http://httpredir.debian.org/debian/ unstable main non-free contrib
deb-src http://httpredir.debian.org/debian/ unstable main non-free contrib

# experimental sources

deb http://httpredir.debian.org/debian/ experimental main non-free contrib
deb-src http://httpredir.debian.org/debian/ experimental main non-free contrib

#smuxi
deb http://ppa.launchpad.net/meebey/smuxi-daily/ubuntu precise main
deb-src http://ppa.launchpad.net/meebey/smuxi-daily/ubuntu precise main

#deb-src http://llvm.org/apt/unstable/ llvm-toolchain main

#deb http://dl.google.com/linux/chrome/deb/ stable main

-- System Information:
Debian Release: 8.0
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'oldstable-updates'), (500, 'stable'), (500, 'oldstable'), (50, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/6 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apt depends on:
ii  debian-archive-keyring  2014.3
ii  gnupg                   1.4.18-7
ii  libapt-pkg4.12          1.0.9.8
ii  libc6                   2.19-18
ii  libgcc1                 1:4.9.2-10
ii  libstdc++6              4.9.2-10

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc     <none>
ii  aptitude    0.6.11-1+b1
ii  dpkg-dev    1.17.25
ii  python-apt  0.9.3.11
ii  synaptic    0.81.2
ii  wajig       2.17

-- no debconf information
Reply to:
Prev by Date: Bug#784310: apt update always fails on experimental/non-free
Next by Date: Bug#784910: python-apt-doc: Wrong indentation and doctring formating cause Sphinx to emit 3 errors and 2 warnings
Previous by thread: python-apt_0.9.4_amd64.changes ACCEPTED into unstable
Next by thread: Bug#784910: python-apt-doc: Wrong indentation and doctring formating cause Sphinx to emit 3 errors and 2 warnings
Index(es):
- Date
- Thread