Bug#632271: apt fails to install very large packages: file size overflow?
Package: apt
Version: 0.8.10.3+squeeze1
Severity: important
Tags: lfs
I have created a package for a very large piece of software and am trying to distribute it using an in-house repository. The package works fine when installed using "dpkg -i", but fails when I try to install it through apt-get or aptitude. The package in question is 4.2 GB in size.
Apt gives me the error message "Bizarre Error - File size is not what the server reported", then proceeds to list the correct file size followed by a bogus filesize (generally deviating from the correct value by about 1 MB). However, the downloaded file, as observed in /var/cache/apt, is not corrupted (correct filesize and md5sum). The server and repository both appear to be configured correctly as well.
There are also minor issues with the progress indicator during the package download, but I am unsure if these are related or a separate issue.
I have tested the versions of apt in stable, testing, and experimental with similar results.
.... the following is a detailed copy/paste description of the problem. The package in question is ansys_13.0sp2-1_amd64.deb and has a size of 4489033432 bytes.
Attempting to install the large package looks like this:
# apt-get install ansys
[...]
Get:1 http://internal_server/debian/ puc-ems/main ansys amd64 13.0sp2-1 [4,489 MB
2% [1 ansys 97.2 MB/194 MB 50%] 11.3 MB/s 6min 28s
[...later...]
41% [1 ansys 1,879 MB/1,879 MB 100%] 11.4 MB/s 3min 48s
[...later...]
Fetched 194 MB in 6min 46s (477 kB/s)
Failed to fetch http://emssupport.purduecal.edu/debian/pool/main/a/ansys/ansys_13.0sp2-1_amd64.deb Size mismatch
W: Bizarre Error - File size is not what the server reported 4489033432 4485863888
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Note that the progress indicator seemed to indicate a total filesize of 194 MB on one line, yet the full 4,489 MB size was listed on the line above it. Once the download passed the 194 MB mark, the amount downloaded and total size increased together until the download was complete. (at which point the incorrect size was displayed)
However, the downloaded file (located at /var/cache/apt/archives/partial/ansys_13.0sp2-1_amd64.deb) matches the filesize and md5sum observed on the server... and it installs properly with "dpkg -i".
# ls -l /var/cache/apt/archives/partial/ansys_13.0sp2-1_amd64.deb
-rw-r--r-- 1 root root 4489033432 Jun 30 03:39 /var/cache/apt/archives/partial/ansys_13.0sp2-1_amd64.deb
The output from apt-get is identical with the version of apt in squeeze/stable (0.8.10.3+squeeze1) and wheezy/testing (0.8.14.1).
The version in experimental also seems to have the bug, but it behaves slightly differently. Its error message reads:
W: Size of file /var/cache/apt/archives/partial/ansys_13.0sp2-1_amd64.deb is not what the server reported 4489033432 4487664456
....and it seems to disregard the error and install the package anyway. The downloaded file (this time located at /var/cache/apt/archives/ansys_13.0sp2-1_amd64.deb) is the correct size and has the correct md5sum.
I don't think this is a problem with the server. To verify this, I checked the headers it was generating:
# wget -S http://internal_server/debian/pool/main/a/ansys/ansys_13.0sp2-1_amd64.deb
[...]
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Content-Type: application/x-debian-package
Accept-Ranges: bytes
ETag: "3495314447"
Last-Modified: Thu, 30 Jun 2011 08:39:56 GMT
Content-Length: 4489033432
Connection: keep-alive
Date: Fri, 01 Jul 2011 01:47:48 GMT
Server: lighttpd/1.4.28
Length: 4489033432 (4.2G) [application/x-debian-package]
[...]
Similarly, I see the correct size and md5sum in the binary-amd64/Packages file on the server.
Any ideas on what I should check next?
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Acquire "";
APT::Acquire::Translation "environment";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^kfreebsd-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
APT::NeverAutoRemove:: "^linux-ubuntu-modules-.*";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Never-MarkAuto-Sections:: "oldlibs";
APT::Never-MarkAuto-Sections:: "restricted/oldlibs";
APT::Never-MarkAuto-Sections:: "universe/oldlibs";
APT::Never-MarkAuto-Sections:: "multiverse/oldlibs";
APT::Periodic "";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Update "";
APT::Update::Post-Invoke "";
APT::Update::Post-Invoke:: "touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";
APT::Update::Post-Invoke-Success "";
APT::Update::Post-Invoke-Success:: "[ ! -f /var/run/dbus/system_bus_socket ] || /usr/bin/dbus-send --system --dest=org.debian.apt --type=signal /org/debian/apt org.debian.apt.CacheChanged || true";
APT::Archives "";
APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::mirrors "mirrors/";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Media "";
Dir::Media::MountPath "/media/cdrom";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Acquire "";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi ";
Unattended-Upgrade "";
Unattended-Upgrade::Allowed-Origins "";
Unattended-Upgrade::Allowed-Origins:: "${distro_id} stable";
Unattended-Upgrade::Allowed-Origins:: "${distro_id} ${distro_codename}-security";
CommandLine "";
CommandLine::AsString "apt-config dump";
-- /etc/apt/preferences --
Package: *
Pin: release a=testing
Pin-Priority: 99
-- /etc/apt/sources.list --
#
# deb cdrom:[Debian GNU/Linux 6.0.1a _Squeeze_ - Official Multi-architecture amd64/i386 NETINST #1 20110320-15:14]/ squeeze main
#deb cdrom:[Debian GNU/Linux 6.0.1a _Squeeze_ - Official Multi-architecture amd64/i386 NETINST #1 20110320-15:14]/ squeeze main
deb http://debian.osuosl.org/debian/ squeeze main
deb-src http://debian.osuosl.org/debian/ squeeze main
deb http://security.debian.org/ squeeze/updates main
deb-src http://security.debian.org/ squeeze/updates main
# squeeze-updates, previously known as 'volatile'
deb http://debian.osuosl.org/debian/ squeeze-updates main
deb-src http://debian.osuosl.org/debian/ squeeze-updates main
deb http://internal_server/debian/ puc-ems main
deb http://ftp.us.debian.org/debian/ testing main
deb-src http://ftp.us.debian.org/debian/ testing main
-- System Information:
Debian Release: 6.0.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable'), (99, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages apt depends on:
ii debian-archive-keyring 2010.08.28 GnuPG archive keys of the Debian a
ii gnupg 1.4.10-4 GNU privacy guard - a free PGP rep
ii libc6 2.11.2-10 Embedded GNU C Library: Shared lib
ii libgcc1 1:4.4.5-8 GCC support library
ii libstdc++6 4.6.0-10 The GNU Standard C++ Library v3
ii zlib1g 1:1.2.3.4.dfsg-3 compression library - runtime
apt recommends no packages.
Versions of packages apt suggests:
pn apt-doc <none> (no description available)
ii aptitude 0.6.3-3.2 terminal-based package manager (te
ii bzip2 1.0.5-6 high-quality block-sorting file co
pn dpkg-dev <none> (no description available)
pn lzma <none> (no description available)
ii python-apt 0.7.100.1+squeeze1 Python interface to libapt-pkg
ii synaptic 0.70~pre1+b1 Graphical package manager
-- no debconf information
Reply to: