Bug#990281: APT make duplicate HTTP requests with mirrorbits
Package: apt
Version: 2.2.3
Severity: important
User: devel@kali.org
Usertags: origin-kali
X-Debbugs-Cc: hertzog@debian.org
Control: found -1 apt/2.3.6
Hi,
in Kali we have been testing "mirrorbits" as a replacement for
"mirrorbrain". Our current mirrorbrain setup runs on http.kali.org
and mirrorbits runs on http-staging.kali.org.
We have seen strange behaviour from APT with mirrorbits where APT
seems to send some HTTP requests multiple times. Whereas with mirrorbrain,
it's fine.
$ cat /etc/apt/sources.list
deb http://http-staging.kali.org/kali kali-rolling main contrib non-free
$ apt install --dry-run aria2
The following additional packages will be installed:
libaria2-0 libsqlite3-0 libssh2-1
The following NEW packages will be installed:
aria2 libaria2-0 libsqlite3-0 libssh2-1
0 upgraded, 4 newly installed, 0 to remove.
$ sudo apt -y -d -q -o Debug::Acquire::http=true install aria2 2>&1 | grep -A1 ^GET
GET /kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com
As you can see, APT seems to send the same HTTP request to mirrorbits
multiple times, but the final download happens only once per file.
Depending on the exact case, we get fewer duplicate requests, but it's
easily reproducible.
We tried to compare the HTTP headers returned by the two servers and
mirrorbits/nginx sets those headers in addition to the usual ones when it
sends its redirect answer (compared to mirrorbrain/apache):
Content-Length: 0
Connection: keep-alive
Cache-Control: private, no-cache
I have also reproduced the issue with apt 2.3.6 in unstable.
If you want to reproduce, just tweak your sources.list and install
http://http.kali.org/pool/main/k/kali-archive-keyring/kali-archive-keyring_2020.2_all.deb
to have the kali archive key to authenticate the repository.
-- System Information:
Distributor ID: Kali
Description: Kali GNU/Linux Rolling
Release: 2021.1
Codename: kali-rolling
Architecture: x86_64
Kernel: Linux 5.10.0-7-amd64 (SMP w/16 CPU threads)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: unable to detect
Versions of packages apt depends on:
ii adduser 3.118
ii debian-archive-keyring 2019.1
ii gpgv 2.2.20-1
ii libapt-pkg6.0 2.1.18
ii libc6 2.31-9
ii libgcc-s1 10.2.1-6
ii libgnutls30 3.7.0-5
ii libseccomp2 2.5.1-1
ii libstdc++6 10.2.1-6
ii libsystemd0 247.2-5
Versions of packages apt recommends:
ii ca-certificates 20210119
Versions of packages apt suggests:
pn apt-doc <none>
pn aptitude | synaptic | wajig <none>
ii dpkg-dev 1.20.7.1kali1
ii gnupg 2.2.20-1
pn powermgmt-base <none>
-- no debconf information
Reply to: