Bug#973861: apt: http acquire method still failing with "Undetermined error" or "Data left in the buffer"
Package: apt
Version: 2.1.11
Severity: important
User: devel@kali.org
Usertags: origin-kali
Hello,
while the frequence of the error has really decreased since the last set
of fixes, we still have occasional failures where apt ends
up with "Undetermined error" or "Data left in the buffer".
It's pretty annoying for APT to be unreliable in that way.
The last case that triggered this bug for us in Kali was within a net-install
in d-i:
[...]
Nov 5 23:15:48 in-target: Err:2130 http://kali.download/kali kali-rolling/main amd64 llvm-10 amd64 1:10.0.1-6
Nov 5 23:15:48 in-target: Undetermined Error [IP: 104.18.102.100 80]
[...]
Nov 5 23:16:14 in-target: Fetched 2,651 MB in 5min 20s (8,274 kB/s)
Nov 5 23:16:14 in-target: E: Failed to fetch http://kali.download/kali/pool/main/l/llvm-toolchain-10/llvm-10_10.0.1-6_amd64.deb Undetermined Error [IP: 104.18.102.100 80]
Nov 5 23:16:14 in-target: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Nov 5 23:16:14 in-target: tasksel: apt-get failed (100)
To try to reproduce it I invite you to configure a sources.lists with
kali.download as mirror. Do this on a server with a very good connection.
deb http://kali.download/kali kali-rolling main contrib non-free
And try to run this in a loop until you reproduce it:
# apt --download-only install kali-linux-large
# apt clean
It took me two tries to reproduce it... I had a download rate of more than
50MB/s.
I tried to reproduce it with debugging options enabled:
# while true; do apt -y --download-only install kali-linux-large -o Debug::Acquire::http=true -o Debug::pkgAcquire::Worker=true 2>log || break; apt clean; done
And I managed to reproduce it too but after a dozen of tries this time.
The log file is huge but it failed with this:
E: Failed to fetch http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb Undetermined Error [IP: 104.18.102.100 80]
And here are the lines that seem relevant for that specific file:
-> http:600%20URI%20Acquire%0aURI:%20http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb%0aFilename:%20/var/cache/apt/archives/partial/libjs-highlight.js_9.12.0+dfsg1-5_all.deb%0aExpected-SHA256:%202347b5fd2bb741cf87f01b4243e591d094445227bea63de5d53555752b322a45%0aExpected-SHA1:%20c465af27c6a567320661436f509043587735f996%0aExpected-MD5Sum:%206d11858477ba16d6b2ffb2d518dd2735%0aExpected-Checksum-FileSize:%20300172%0aTarget-Repo-URI:%20http://kali.download/kali/%0aTarget-Site:%20http://kali.download/kali%0aTarget-Release:%20kali-rolling%0aTarget-Base-URI:%20http://kali.download/kali/%0aTarget-Component:%20main%0aTarget-Codename:%20kali-rolling%0aTarget-Suite:%20kali-rolling%0aTarget-Architecture:%20all%0aTarget-Type:%20deb%0a%0a
<- http:102%20Status%0aURI:%20http://kali.download/kali/pool/main/x/xorg-sgml-doctools/xorg-sgml-doctools_1.11-1_all.deb%0aMessage:%20Waiting%20for%20headers
GET /kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0%2bdfsg1-5_all.deb HTTP/1.1^M
Host: kali.download^M
User-Agent: Debian APT-HTTP/1.3 (2.1.11)^M
^M
[...]
<- http:102%20Status%0aURI:%20http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb%0aMessage:%20Waiting%20for%20headers
GET /kali/pool/main/i/imagemagick/libmagickcore-6.q16-6-extra_6.9.11.24%2bdfsg-1%2bb1_amd64.deb HTTP/1.1^M
Host: kali.download^M
User-Agent: Debian APT-HTTP/1.3 (2.1.11)^M
^M
Answer for: http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb
HTTP/1.1 200 OK^M
Date: Fri, 06 Nov 2020 08:10:09 GMT^M
Content-Type: application/octet-stream^M
Content-Length: 300172^M
Connection: close^M
Set-Cookie: __cfduid=dcd44a60d11f41fb354d70f7609be4f1a1604650209; expires=Sun, 06-Dec-20 08:10:09 GMT; path=/; domain=.kali.download; HttpOnly; SameSite=Lax^M
Last-Modified: Sun, 29 Dec 2019 15:40:32 GMT^M
ETag: "5e08c8f0-4948c"^M
Expires: Mon, 04 Nov 2030 08:10:09 GMT^M
Cache-Control: public, max-age=315360000^M
CF-Cache-Status: HIT^M
Age: 264899^M
Accept-Ranges: bytes^M
cf-request-id: 063e342a5d0000a30f222e8000000001^M
Server: cloudflare^M
CF-RAY: 5edd5623cfbda30f-ORD^M
^M
<- http:200%20URI%20Start%0aLast-Modified:%20Sun,%2029%20Dec%202019%2015:40:32%20+0000%0aSize:%20300172%0aURI:%20http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb
<- http:400%20URI%20Failure%0aTransient-Failure:%20true%0aMessage:%20Undetermined%20Error%20[IP:%20104.18.102.100%2080]%0aURI:%20http://kali.download/kali/pool/main/h/highlight.js/libjs-highlight.js_9.12.0+dfsg1-5_all.deb
-> http:600%20URI%20Acquire%0aURI:%20http://kali.download/kali/pool/main/libn/libnet-smtp-ssl-perl/libnet-smtp-ssl-perl_1.04-1_all.deb%0aFilename:%20/var/cache/apt/archives/partial/libnet-smtp-ssl-perl_1.04-1_all.deb%0aExpected-SHA256:%20cf23f2c340b048177ef3060644ec759a9002932f8c97889089d741723f8ada6c%0aExpected-SHA1:%20b02e232125c7a35b665bfeeacc2691646aeda44d%0aExpected-MD5Sum:%2042fd123d655b3ea46850ccaf5e9a5a63%0aExpected-Checksum-FileSize:%206184%0aTarget-Repo-URI:%20http://kali.download/kali/%0aTarget-Site:%20http://kali.download/kali%0aTarget-Release:%20kali-rolling%0aTarget-Base-URI:%20http://kali.download/kali/%0aTarget-Component:%20main%0aTarget-Codename:%20kali-rolling%0aTarget-Suite:%20kali-rolling%0aTarget-Architecture:%20all%0aTarget-Type:%20deb%0a%0a
Cheers,
-- System Information:
Debian Release: bullseye/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.19.0-6-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_UNSIGNED_MODULE
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages apt depends on:
ii adduser 3.118
ii debian-archive-keyring 2019.1
ii gpgv 2.2.20-1
ii libapt-pkg6.0 2.1.11
ii libc6 2.31-4
ii libgcc-s1 10.2.0-15
ii libgnutls30 3.6.15-4
ii libseccomp2 2.4.4-1+b1
ii libstdc++6 10.2.0-15
ii libsystemd0 246.6-2
Versions of packages apt recommends:
ii ca-certificates 20200601
Versions of packages apt suggests:
ii apt-doc 2.1.11
ii aptitude 0.8.13-2
ii dpkg-dev 1.20.5
ii gnupg 2.2.20-1
ii gnupg2 2.2.20-1
ii powermgmt-base 1.36
ii synaptic 0.90+nmu1
-- no debconf information
Reply to: