Bug#990281: marked as done (APT make duplicate HTTP requests with mirrorbits)

To: Julian Andres Klode <jak@debian.org>
Subject: Bug#990281: marked as done (APT make duplicate HTTP requests with mirrorbits)
From: "Debian Bug Tracking System" <owner@bugs.debian.org>
Date: Mon, 18 Oct 2021 14:51:05 +0000
Message-id: <[🔎] handler.990281.D990281.16345685112706.ackdone@bugs.debian.org>
Reply-to: 990281@bugs.debian.org
References: <E1mcTwC-000A4d-5Y@fasolo.debian.org> <162454128782.89858.6157103509010019034.reportbug@t14-buxy.home.ouaza.com>

Your message dated Mon, 18 Oct 2021 14:48:28 +0000
with message-id <E1mcTwC-000A4d-5Y@fasolo.debian.org>
and subject line Bug#990281: fixed in apt 2.3.10
has caused the Debian Bug report #990281,
regarding APT make duplicate HTTP requests with mirrorbits
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
990281: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990281
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems

--- Begin Message ---

To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: APT make duplicate HTTP requests with mirrorbits
From: Raphaël Hertzog <hertzog@debian.org>
Date: Thu, 24 Jun 2021 15:28:07 +0200
Message-id: <162454128782.89858.6157103509010019034.reportbug@t14-buxy.home.ouaza.com>

Package: apt
Version: 2.2.3
Severity: important
User: devel@kali.org
Usertags: origin-kali
X-Debbugs-Cc: hertzog@debian.org
Control: found -1 apt/2.3.6

Hi,

in Kali we have been testing "mirrorbits" as a replacement for
"mirrorbrain". Our current mirrorbrain setup runs on http.kali.org
and mirrorbits runs on http-staging.kali.org.

We have seen strange behaviour from APT with mirrorbits where APT
seems to send some HTTP requests multiple times. Whereas with mirrorbrain,
it's fine.

$ cat /etc/apt/sources.list
deb http://http-staging.kali.org/kali kali-rolling main contrib non-free
$ apt install --dry-run aria2

The following additional packages will be installed:
  libaria2-0 libsqlite3-0 libssh2-1
The following NEW packages will be installed:
  aria2 libaria2-0 libsqlite3-0 libssh2-1
0 upgraded, 4 newly installed, 0 to remove.

$ sudo apt -y -d -q -o Debug::Acquire::http=true install aria2 2>&1 | grep -A1 ^GET
GET /kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/s/sqlite3/libsqlite3-0_3.34.1-3_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /pub/Linux/kali/pool/main/libs/libssh2/libssh2-1_1.9.0-2_amd64.deb HTTP/1.1
Host: ftp.jaist.ac.jp
--
GET /kali/pool/main/a/aria2/libaria2-0_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: http-staging.kali.org
--
GET /kali/pool/main/a/aria2/aria2_1.35.0-3_amd64.deb HTTP/1.1
Host: mirror.anigil.com


As you can see, APT seems to send the same HTTP request to mirrorbits
multiple times, but the final download happens only once per file.
Depending on the exact case, we get fewer duplicate requests, but it's
easily reproducible.

We tried to compare the HTTP headers returned by the two servers and
mirrorbits/nginx sets those headers in addition to the usual ones when it
sends its redirect answer (compared to mirrorbrain/apache):

    Content-Length: 0
    Connection: keep-alive
    Cache-Control: private, no-cache

I have also reproduced the issue with apt 2.3.6 in unstable.

If you want to reproduce, just tweak your sources.list and install
http://http.kali.org/pool/main/k/kali-archive-keyring/kali-archive-keyring_2020.2_all.deb
to have the kali archive key to authenticate the repository.

-- System Information:
Distributor ID:	Kali
Description:	Kali GNU/Linux Rolling
Release:	2021.1
Codename:	kali-rolling
Architecture: x86_64

Kernel: Linux 5.10.0-7-amd64 (SMP w/16 CPU threads)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: unable to detect

Versions of packages apt depends on:
ii  adduser                 3.118
ii  debian-archive-keyring  2019.1
ii  gpgv                    2.2.20-1
ii  libapt-pkg6.0           2.1.18
ii  libc6                   2.31-9
ii  libgcc-s1               10.2.1-6
ii  libgnutls30             3.7.0-5
ii  libseccomp2             2.5.1-1
ii  libstdc++6              10.2.1-6
ii  libsystemd0             247.2-5

Versions of packages apt recommends:
ii  ca-certificates  20210119

Versions of packages apt suggests:
pn  apt-doc                      <none>
pn  aptitude | synaptic | wajig  <none>
ii  dpkg-dev                     1.20.7.1kali1
ii  gnupg                        2.2.20-1
pn  powermgmt-base               <none>

-- no debconf information

--- End Message ---

--- Begin Message ---

To: 990281-close@bugs.debian.org
Subject: Bug#990281: fixed in apt 2.3.10
From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>
Date: Mon, 18 Oct 2021 14:48:28 +0000
Message-id: <E1mcTwC-000A4d-5Y@fasolo.debian.org>
Reply-to: Julian Andres Klode <jak@debian.org>

Source: apt
Source-Version: 2.3.10
Done: Julian Andres Klode <jak@debian.org>

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 990281@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Mon, 18 Oct 2021 16:35:21 +0200
Source: apt
Architecture: source
Version: 2.3.10
Distribution: unstable
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 989558 990281 990555
Changes:
 apt (2.3.10) unstable; urgency=medium
 .
   [ Julian Andres Klode ]
   * basehttp: Turn HaveContent into a TriState
   * Set haveContent to FALSE on `Content-Length: 0` (Closes: #990281)
   * Add support for embedding PGP keys into Signed-By in deb822 sources
 .
   [ David Kalnischkies ]
   * All pkgCaches are MultiArch caches
   * Do not strip M-A for native build-dep resolution
   * Do not make provides of M-A:allowed implicit M-A:foreign
   * Barbarian M-A:allowed don't satisfy :any deps of other archs
   * Streamline access to barbarian architecture functionality
   * Read and work with canonical file-URIs from sources.lists
   * Use https config on https proxies for http servers (Closes: #990555)
   * Add AllowRange option to disable HTTP Range usage
   * Disable HTTP Range usage if varnish < 6.4 is involved
   * Use exact If-Range match in our test webserver
 .
   [ Johannes Schauer Marin Rodrigues ]
   * add pattern to select packages by priority (closes: #989558)
Checksums-Sha1:
 ff394f947012a3cbdb48cf300e984217ec6dab95 2801 apt_2.3.10.dsc
 c8f4eb4bc07561c0b3cde3ab545a66755196ae82 2210032 apt_2.3.10.tar.xz
 28b58837dfa8ae367c501638fa1e50c088926f0d 7439 apt_2.3.10_source.buildinfo
Checksums-Sha256:
 2e9d0653225719d65892256b823e251c855100e83e6231b2bd4977e7dd6f7b45 2801 apt_2.3.10.dsc
 145c02b998c52b11a49d2cf845c7d4fd85201c4c182c3779502c8e05602d4935 2210032 apt_2.3.10.tar.xz
 4c917abf7cc58d2af7503f0d320c08da085ddad552a573e9abd22f57ba15ceba 7439 apt_2.3.10_source.buildinfo
Files:
 dfc5280a1f8c03fa606227c2c7cdff01 2801 admin important apt_2.3.10.dsc
 48357ffef0f3adfb29fba983438bfea0 2210032 admin important apt_2.3.10.tar.xz
 33b32eb7d1221f9438c63b55ead4fd35 7439 admin important apt_2.3.10_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAmFthrcPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9xCJAQAKSW3p0q+vo4yXcafD9ruxDUeS1o0OeruYd6
7mSZM413cjsXP2SgInG/QKX1Q6g2vd/t11/AQ9y6tM3L3Vc5hCB46VXa3vofTy/X
NE7bPi+ISr06TjzgkWtpADC+cfAaeW35C2wFGFqx4kv9OTExJyWH0tl1kHZVNZ5I
j5+9m+rDlLsaLW/6YPme1aVkwyfhVmjpazROzVH6IptxqGvsTVl4YIkEZypwW7F6
v3B6NfLvSzm1fPrvjQsNiTRnfZ23tiTyu0INTSROkGtl7kAEHE98d3qjmKnvYPfc
K8aG0rxKBBs5d/2MLF0HF5ns+0FWFQMYTCy+9jwP+C1ftGLvvg891BTzwLbaTp0E
nab3DW2ezXWQEuoQdj75jddGPRGoplKFPZMsHc0P2yHsWJYPRpoA8ncZtpJMkOEM
JeMa97TmV/u9IbSC5yjI8yOBW61DmQujbLC2TnCyIDGY8+F9ZzPPBdAtdFP1wEk3
E7qH6oWhMDjpZg+PXEKpL3KeUrYvjE5jnO8/F2t/MW76vI5x4RsSAcfRMWyPr/iJ
0QudPCiF694+IIUUnX1kxnmVGlzPHCRCMMWcqM0gaB/cZth8KqcUlCFM26aKuxms
MKRAMjCvMaSxsSuu2B7g7/NpfFn2rVT8lmwXg1Wmgtd0b7BasyUsSo9C0UGRnDbW
BVTRhwLc
=OWNc
-----END PGP SIGNATURE-----

--- End Message ---

Reply to:

Prev by Date: Bug#989558: marked as done (apt: please add a pattern that allows one to select packages by priority)
Next by Date: Bug#990555: marked as done (When using an HTTPS proxy for HTTP repositories, APT ignores CaInfo (and possibly other Acquire::https options))
Previous by thread: Bug#989558: marked as done (apt: please add a pattern that allows one to select packages by priority)
Next by thread: Bug#990555: marked as done (When using an HTTPS proxy for HTTP repositories, APT ignores CaInfo (and possibly other Acquire::https options))
Index(es):
- Date
- Thread