Bug#688303: fails on concatenated bzip2 compressed Packages files (regression)

To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Bug#688303: fails on concatenated bzip2 compressed Packages files (regression)
From: Thijs Kinkhorst <thijs@debian.org>
Date: Fri, 21 Sep 2012 11:28:30 +0200
Message-id: <20120921092830.27166.15273.reportbug@dixan.uvt.nl>
Reply-to: Thijs Kinkhorst <thijs@debian.org>, 688303@bugs.debian.org

Package: apt
Version: 0.9.7.4
Severity: important

Hi,

Using apt from wheezy with our APT repository fails:
201 URI Done: bzip2:/var/lib/apt/lists/partial/non-free.uvt.nl_debian_dists_squeeze_uvt_binary-amd64_Packages
RecivedHash: SHA256:99efd14ebb9fe582cc986f50502540030299595badefad71f5819e78c86a0c8d
ExpectedHash: SHA256:56ea2d1eb412a7121ad50f96e389c587e870ed7e80e893de0a66b3dbd9793015
[...]
W: Failed to fetch bzip2:/var/lib/apt/lists/partial/non-free.uvt.nl_debian_dists_squeeze_uvt_binary-amd64_Packages  Hash Sum mismatch

The problem is that our APT archive contains a Packages.bz2 which is a
concatenation of other bz2 files (of sub-archives). Two concatenated
bzip2 files again form a valid bzip2 file. The b(un)zip2 command line
utils support this and it's a documented feature. APT only decompresses
the first bzip2 file in Packages.bz2 leading indeed to a different
hash sum than expected.

This is a regression from squeeze, because apt in squeeze used the
command line utils which do this correctly. In 0.9, apt switched to
using libbz2, however, it only calls the function that decompresses
one bzip2 stream. The bzip2 command line utils handle the looping over
input themselves with the appropriate library calls (search for
bzReadGetUnused).


Cheers,
Thijs

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "false";
APT::Install-Suggests "0";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^kfreebsd-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
APT::NeverAutoRemove:: "^linux-ubuntu-modules-.*";
APT::NeverAutoRemove:: "^gnumach$";
APT::NeverAutoRemove:: "^gnumach-image.*";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Periodic "";
APT::Periodic::AutocleanInterval "14";
APT::Cache-Limit "134217728";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "1";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "2";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-9n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "3";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-9";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "4";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "5";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-9";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
APT::Compressor::::Name "";
APT::Compressor::::Extension ".";
APT::Compressor::::Binary "";
APT::Compressor::::Cost "100";
APT::Compressor::::CompressArg "";
APT::Compressor::::CompressArg:: "-9";
APT::Compressor::::UncompressArg "";
APT::Compressor::::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::Pdiffs "false";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
DPkg "";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "exec sync";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
Aptitude "";
Aptitude::Recommends-Important "false";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- System Information:
Debian Release: wheezy/sid
  APT prefers testing
  APT policy: (400, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-3-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash

Versions of packages apt depends on:
ii  debian-archive-keyring  2012.4
ii  gnupg                   1.4.12-4+b1
ii  libapt-pkg4.12          0.9.7.4
ii  libc6                   2.13-35
ii  libgcc1                 1:4.7.1-7
ii  libstdc++6              4.7.1-7

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc     <none>
ii  aptitude    0.6.8.1-1
ii  dpkg-dev    1.16.8
ii  python-apt  0.8.4
ii  xz-utils    5.1.1alpha+20120614-1

-- no debconf information

Reply to:

Follow-Ups:
- Bug#688303: fails on concatenated bzip2 compressed Packages files (regression)
  - From: David Kalnischkies <kalnischkies+debian@gmail.com>

Prev by Date: elings do also allow denied Will to be expressed. We sense your confusion, one
Next by Date: Bug#676302: apt: apt-cdrom add couldn't be used several times
Previous by thread: elings do also allow denied Will to be expressed. We sense your confusion, one
Next by thread: Bug#688303: fails on concatenated bzip2 compressed Packages files (regression)
Index(es):
- Date
- Thread