Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org

To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
From: James Addison <jay@jp-hosting.net>
Date: Sun, 03 May 2020 09:49:42 +0000
Message-id: <[🔎] 158849938210.2189.16048687147754811739.reportbug@3c7e4d80d05d>
Reply-to: James Addison <jay@jp-hosting.net>, 959518@bugs.debian.org
Package: apt
Version: 1.8.2
Severity: normal

Dear Maintainer,

During installation of packages under Debian Buster[1], I've encountered a
repeatable (non-TLS) HTTP download error that occurs during download of
openjdk-11-jdk-headless.

It's possible this may expose a rare edge case in apt's HTTP client, and/or
it may be a transient network condition; either way it seems worth reporting.

The steps to reproduce the problem are:

1. Install GPG

    apt-get update && apt-get install -y gpg

2. Approve apt GPG keys for snapshot.debian.org

    gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 5E61B217265DA9807A23C5FF4DFAB270CAA96DFA
    gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
    gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE
    gpg -a --export 5E61B217265DA9807A23C5FF4DFAB270CAA96DFA | apt-key add -
    gpg -a --export 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517 | apt-key add -
    gpg -a --export 80D15823B7FD1561F9F7BCDDDC30D7C23CBBABEE | apt-key add -

NB: gpg will fail to import, with a 'new key but contains no user ID - skipped'
    message if the '--keyserver' option is omitted.

3. Configure retrieval of snapshot packages

    export SNAPSHOT="20200502T085134Z"
    rm /etc/apt/sources.list
    printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/ buster main\n" >> /etc/apt/sources.list
    printf "deb http://snapshot.debian.org/archive/debian-security/${SNAPSHOT}/ buster/updates main\n" >> /etc/apt/sources.list
    printf "deb http://snapshot.debian.org/archive/debian/${SNAPSHOT}/ buster-updates main\n" >> /etc/apt/sources.list

4. Attempt installation of openjdk-11-jdk

    apt-get update && apt-get install -y openjdk-11-jdk


Expected Results:

Successful installation of packages.


Actual Results:

The following output appears during processing of the command:

    Get:185 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 libxt-dev amd64 1:1.1.5-1+b3 [426 kB]
    Get:186 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jre amd64 11.0.7+10-3~deb10u1 [34.3 kB]
    Get:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1 [215 MB]
    Err:187 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk-headless amd64 11.0.7+10-3~deb10u1
      Undetermined Error [IP: 193.62.202.27 80]
    Get:188 http://snapshot.debian.org/archive/debian-security/20200502T085134Z buster/updates/main amd64 openjdk-11-jdk amd64 11.0.7+10-3~deb10u1 [2607 kB]
    Get:189 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 publicsuffix all 20190415.1030-1 [116 kB]
    Get:190 http://snapshot.debian.org/archive/debian/20200502T085134Z buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB]


Anecdotally, after retrying this process a number of times yesterday, my ISP
appeared to introduce some rate limiting (unfortunately I didn't record any
statistics).

During the slower/limited network condition, I was not able to
reproduce the problem.

Restrictions appear to have been lifted today and the problem reliably
appears each time I attempt the installation.


Related Issues:

I have tried applying a workaround for a similar-sounding issue[2] by
adding '-o Acquire::http::Pipeline-Depth="0"' to the apt-get install
command, but this does not resolve the issue.

Thank you,
James

[1] - https://hub.docker.com/layers/debian/library/debian/buster-slim/images/sha256-ed8df275d4736ad00e69c4f58c2d9aa3f917911be47753e6d8d25368bcafa38d

[2] - https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1801338

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-image";
APT::VersionedKernelPackages:: "linux-headers";
APT::VersionedKernelPackages:: "linux-image-extra";
APT::VersionedKernelPackages:: "linux-modules";
APT::VersionedKernelPackages:: "linux-modules-extra";
APT::VersionedKernelPackages:: "linux-signed-image";
APT::VersionedKernelPackages:: "linux-image-unsigned";
APT::VersionedKernelPackages:: "kfreebsd-image";
APT::VersionedKernelPackages:: "kfreebsd-headers";
APT::VersionedKernelPackages:: "gnumach-image";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::VersionedKernelPackages:: "linux-backports-modules-.*";
APT::VersionedKernelPackages:: "linux-modules-.*";
APT::VersionedKernelPackages:: "linux-tools";
APT::VersionedKernelPackages:: "linux-cloud-tools";
APT::VersionedKernelPackages:: "linux-buildinfo";
APT::VersionedKernelPackages:: "linux-source";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "contrib/metapackages";
APT::Never-MarkAuto-Sections:: "non-free/metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Move-Autobit-Sections "";
APT::Move-Autobit-Sections:: "oldlibs";
APT::Move-Autobit-Sections:: "contrib/oldlibs";
APT::Move-Autobit-Sections:: "non-free/oldlibs";
APT::Move-Autobit-Sections:: "restricted/oldlibs";
APT::Move-Autobit-Sections:: "universe/oldlibs";
APT::Move-Autobit-Sections:: "multiverse/oldlibs";
APT::AutoRemove "";
APT::AutoRemove::SuggestsImportant "false";
APT::Update "";
APT::Update::Post-Invoke "";
APT::Update::Post-Invoke:: "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Architectures:: "i386";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "0";
APT::Compressor::zstd "";
APT::Compressor::zstd::Name "zstd";
APT::Compressor::zstd::Extension ".zst";
APT::Compressor::zstd::Binary "false";
APT::Compressor::zstd::Cost "60";
APT::Compressor::lz4 "";
APT::Compressor::lz4::Name "lz4";
APT::Compressor::lz4::Extension ".lz4";
APT::Compressor::lz4::Binary "false";
APT::Compressor::lz4::Cost "50";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "100";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-6n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "200";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "300";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-6";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "400";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-6";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "";
Dir::Cache::pkgcache "";
Dir::Etc "etc/apt";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::netrcparts "auth.conf.d";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::planners "";
Dir::Bin::planners:: "/usr/lib/apt/planners";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::gzip "/bin/gzip";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lz4 "/usr/bin/lz4";
Dir::Bin::zstd "/usr/bin/zstd";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/apt";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Log::Planner "eipp.log.xz";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.ucf-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::AllowInsecureRepositories "0";
Acquire::AllowWeakRepositories "0";
Acquire::AllowDowngradeToInsecureRepositories "0";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom/";
Acquire::IndexTargets "";
Acquire::IndexTargets::deb "";
Acquire::IndexTargets::deb::Packages "";
Acquire::IndexTargets::deb::Packages::MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
Acquire::IndexTargets::deb::Packages::flatMetaKey "Packages";
Acquire::IndexTargets::deb::Packages::ShortDescription "Packages";
Acquire::IndexTargets::deb::Packages::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
Acquire::IndexTargets::deb::Packages::flatDescription "$(RELEASE) Packages";
Acquire::IndexTargets::deb::Packages::Optional "0";
Acquire::IndexTargets::deb::Translations "";
Acquire::IndexTargets::deb::Translations::MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatMetaKey "$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::ShortDescription "Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatDescription "$(RELEASE) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb-src "";
Acquire::IndexTargets::deb-src::Sources "";
Acquire::IndexTargets::deb-src::Sources::MetaKey "$(COMPONENT)/source/Sources";
Acquire::IndexTargets::deb-src::Sources::flatMetaKey "Sources";
Acquire::IndexTargets::deb-src::Sources::ShortDescription "Sources";
Acquire::IndexTargets::deb-src::Sources::Description "$(RELEASE)/$(COMPONENT) Sources";
Acquire::IndexTargets::deb-src::Sources::flatDescription "$(RELEASE) Sources";
Acquire::IndexTargets::deb-src::Sources::Optional "0";
Acquire::Changelogs "";
Acquire::Changelogs::URI "";
Acquire::Changelogs::URI::Origin "";
Acquire::Changelogs::URI::Origin::Debian "https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog";;
Acquire::Changelogs::URI::Origin::Ubuntu "https://changelogs.ubuntu.com/changelogs/pool/@CHANGEPATH@/changelog";;
Acquire::Changelogs::AlwaysOnline "";
Acquire::Changelogs::AlwaysOnline::Origin "";
Acquire::Changelogs::AlwaysOnline::Origin::Ubuntu "1";
Acquire::GzipIndexes "true";
Acquire::Languages "";
Acquire::Languages:: "none";
Acquire::CompressionTypes "";
Acquire::CompressionTypes::xz "xz";
Acquire::CompressionTypes::bz2 "bzip2";
Acquire::CompressionTypes::lzma "lzma";
Acquire::CompressionTypes::gz "gzip";
Acquire::CompressionTypes::lz4 "lz4";
Acquire::CompressionTypes::zst "zstd";
DPkg "";
DPkg::Path "/usr/sbin:/usr/bin:/sbin:/bin";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true";
Binary "apt-config";
Binary::apt "";
Binary::apt::APT "";
Binary::apt::APT::Color "1";
Binary::apt::APT::Cache "";
Binary::apt::APT::Cache::Show "";
Binary::apt::APT::Cache::Show::Version "2";
Binary::apt::APT::Cache::AllVersions "0";
Binary::apt::APT::Cache::ShowVirtuals "1";
Binary::apt::APT::Cache::Search "";
Binary::apt::APT::Cache::Search::Version "2";
Binary::apt::APT::Cache::ShowDependencyType "1";
Binary::apt::APT::Cache::ShowVersion "1";
Binary::apt::APT::Get "";
Binary::apt::APT::Get::Upgrade-Allow-New "1";
Binary::apt::APT::Get::Update "";
Binary::apt::APT::Get::Update::InteractiveReleaseInfoChanges "1";
Binary::apt::APT::Cmd "";
Binary::apt::APT::Cmd::Show-Update-Stats "1";
Binary::apt::APT::Keep-Downloaded-Packages "0";
Binary::apt::DPkg "";
Binary::apt::DPkg::Progress-Fancy "1";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- (no /etc/apt/preferences present) --


-- (no /etc/apt/preferences.d/* present) --


-- /etc/apt/sources.list --

deb http://snapshot.debian.org/archive/debian/20200502T085134Z/ buster main
deb http://snapshot.debian.org/archive/debian-security/20200502T085134Z/ buster/updates main
deb http://snapshot.debian.org/archive/debian/20200502T085134Z/ buster-updates main

-- (no /etc/apt/sources.list.d/* present) --


-- System Information:
Debian Release: 10.3
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.5.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages apt depends on:
ii  adduser                 3.118
ii  debian-archive-keyring  2019.1
ii  gpgv                    2.2.12-1+deb10u1
ii  libapt-pkg5.0           1.8.2
ii  libc6                   2.28-10
ii  libgcc1                 1:8.3.0-6
ii  libgnutls30             3.6.7-4+deb10u3
ii  libseccomp2             2.3.3-4
ii  libstdc++6              8.3.0-6

Versions of packages apt recommends:
ii  ca-certificates  20190110

Versions of packages apt suggests:
pn  apt-doc                      <none>
pn  aptitude | synaptic | wajig  <none>
pn  dpkg-dev                     <none>
ii  gnupg                        2.2.12-1+deb10u1
pn  powermgmt-base               <none>

-- no debconf information
Reply to:
Follow-Ups:
- Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
  - From: James Addison <jay@jp-hosting.net>
Prev by Date: Investment Proposal
Next by Date: Processed: reassign 959518 to apt,snapshot.debian.org
Previous by thread: Investment Proposal
Next by thread: Bug#959518: apt-transport-http: Repeatable 'Undetermined Error' during package download from snapshot.debian.org
Index(es):
- Date
- Thread