[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#990281: APT make duplicate HTTP requests with mirrorbits



Package: apt
Version: 2.2.3
Severity: important
User: devel@kali.org
Usertags: origin-kali
X-Debbugs-Cc: hertzog@debian.org
Control: found -1 apt/2.3.6

Hi,

in Kali we have been testing "mirrorbits" as a replacement for
"mirrorbrain". Our current mirrorbrain setup runs on http.kali.org
and mirrorbits runs on http-staging.kali.org.

We have seen strange behaviour from APT with mirrorbits where APT
seems to send some HTTP requests multiple times. Whereas with mirrorbrain,
it's fine.

$ cat /etc/apt/sources.list
deb http://http-staging.kali.org/kali kali-rolling main contrib non-free
$ apt install --dry-run aria2

The following additional packages will be installed:
  libaria2-0 libsqlite3-0 libssh2-1
The following NEW packages will be installed:
  aria2 libaria2-0 libsqlite3-0 libssh2-1
0 upgraded, 4 newly installed, 0 to remove.

$ sudo apt -y -d -q -o Debug::Acquire::http=true install aria2 2>&1 | grep -A1 ^GET
GET /kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com


As you can see, APT seems to send the same HTTP request to mirrorbits
multiple times, but the final download happens only once per file.
Depending on the exact case, we get fewer duplicate requests, but it's
easily reproducible.

We tried to compare the HTTP headers returned by the two servers and
mirrorbits/nginx sets those headers in addition to the usual ones when it
sends its redirect answer (compared to mirrorbrain/apache):

    Content-Length: 0
    Connection: keep-alive
    Cache-Control: private, no-cache

I have also reproduced the issue with apt 2.3.6 in unstable.

If you want to reproduce, just tweak your sources.list and install
http://http.kali.org/pool/main/k/kali-archive-keyring/kali-archive-keyring_2020.2_all.deb
to have the kali archive key to authenticate the repository.

-- System Information:
Distributor ID:	Kali
Description:	Kali GNU/Linux Rolling
Release:	2021.1
Codename:	kali-rolling
Architecture: x86_64

Kernel: Linux 5.10.0-7-amd64 (SMP w/16 CPU threads)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages apt depends on:
ii  adduser                 3.118
ii  debian-archive-keyring  2019.1
ii  gpgv                    2.2.20-1
ii  libapt-pkg6.0           2.1.18
ii  libc6                   2.31-9
ii  libgcc-s1               10.2.1-6
ii  libgnutls30             3.7.0-5
ii  libseccomp2             2.5.1-1
ii  libstdc++6              10.2.1-6
ii  libsystemd0             247.2-5

Versions of packages apt recommends:
ii  ca-certificates  20210119

Versions of packages apt suggests:
pn  apt-doc                      <none>
pn  aptitude | synaptic | wajig  <none>
ii  dpkg-dev                     1.20.7.1kali1
ii  gnupg                        2.2.20-1
pn  powermgmt-base               <none>

-- no debconf information


Reply to: