Bug#960304: snapshot.debian.org: Snapshot repo repeatedly cutting off connection, returning partial content

Severity: serious

This problem is still present, and is making the snapshot service unusable for many people.

At the time of writing, this is the status:

$ wget https://snapshot.debian.org/archive/debian/20200917T084540Z/pool/main/libr/libreoffice/libreoffice-dbg_4.3.3-2%2Bdeb8u11_amd64.deb
--2020-10-13 12:50:28-- https://snapshot.debian.org/archive/debian/20200917T084540Z/pool/main/libr/libreoffice/libreoffice-dbg_4.3.3-2%2Bdeb8u11_amd64.deb
Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 739029194 (705M)
Saving to: ‘libreoffice-dbg_4.3.3-2+deb8u11_amd64.deb.4’

libreoffice-dbg_4.3.3-2+deb8 3%[> ] 25,03M 11,1MB/s in 2,3s

2020-10-13 12:50:31 (11,1 MB/s) - Connection closed at byte 26247168. Retrying.

--2020-10-13 12:50:32-- (try: 2) https://snapshot.debian.org/archive/debian/20200917T084540Z/pool/main/libr/libreoffice/libreoffice-dbg_4.3.3-2%2Bdeb8u11_amd64.deb
Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 739029194 (705M), 712782026 (680M) remaining
Saving to: ‘libreoffice-dbg_4.3.3-2+deb8u11_amd64.deb.4’

libreoffice-dbg_4.3.3-2+deb8 7%[+=> ] 52,22M 10,9MB/s in 2,5s

2020-10-13 12:50:34 (10,9 MB/s) - Connection closed at byte 54755328. Retrying.

--2020-10-13 12:50:36-- (try: 3) https://snapshot.debian.org/archive/debian/20200917T084540Z/pool/main/libr/libreoffice/libreoffice-dbg_4.3.3-2%2Bdeb8u11_amd64.deb
Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 739029194 (705M), 684273866 (653M) remaining
Saving to: ‘libreoffice-dbg_4.3.3-2+deb8u11_amd64.deb.4’

libreoffice-dbg_4.3.3-2+deb8 9%[+++> ] 66,02M 9,89MB/s in 1,4s

2020-10-13 12:50:38 (9,89 MB/s) - Connection closed at byte 69222400. Retrying.

--2020-10-13 12:50:41-- (try: 4) https://snapshot.debian.org/archive/debian/20200917T084540Z/pool/main/libr/libreoffice/libreoffice-dbg_4.3.3-2%2Bdeb8u11_amd64.deb
Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 739029194 (705M), 669806794 (639M) remaining
Saving to: ‘libreoffice-dbg_4.3.3-2+deb8u11_amd64.deb.4’

libreoffice-dbg_4.3.3-2+deb8 12%[++++> ] 88,97M 10,5MB/s in 2,2s

2020-10-13 12:50:44 (10,5 MB/s) - Connection closed at byte 93290496. Retrying.

^C
$

As you can see, the connection is forcibly closed by the server and wget has to retry repeatedly. Apt does not retry connections by default, instead it fails completely. You have to rerun the apt command again and again until all packages are successfully downloaded, which can require a large number of retries and a very long time.

It is not always trivial to trigger this bug. It comes and goes with time, perhaps due to general network load. One rather reliable way of triggering it seems to be to install buster on a machine and then dist-upgrade to a recent snapshot. This results in a large number of downloads, which snapshot eventually starts to abort. Once it has started aborting downloads, the bug can be triggered by plain wget or curl calls.

The source of the problem is unidentified, but one guess is that Varnish somehow erroneously interprets the rapid file accesses as a denial-of-service attack and applies restrictions.

/Björn