overly short max-age of archive -> file redirect
Hi,
while debugging rate-limiting issues on snapshot.d.o, I noticed that I
run into the rate-limiting not due to the file-transfers, but due
archive/.deb > /file/<hash> redirects. When apt accesses a file on the
pool, this is not returned directly but via a 302 redirect (see curl
log below). This redirect however has a max-age of just 10mins. The
corresponding code is probably in [1]. This raises the following
questions:
1. Why return a 302 (temporary moved) instead of a 301 (permanently
moved)?
2. Why not use the same max-age (31536000)?
Further, the rate-limits should be precisely documented so clients /
caching proxies can adapt to this. The limits also need to match the
retry-after header in the 429 responses. Currently s.d.o responds with
retry-after 5 (seconds), which is by far to short to overcome the
limit.
If rate-limiting would be implemented correctly, downstream caches
could properly cache the results and clients like apt could behave
nicely. I further recommend to use WAY higher request limits in
combination with a moving average limit on the amount of transferred
data. By that, the cheap "is my cache still valid" requests could pass,
while the more heavy payload transfers are avoided. Also clients could
hit s.d.o without reduced transfer rates, hence reducing the amount of
open handles on the server.
[1]
https://salsa.debian.org/snapshot-team/snapshot/-/blob/master/web/app/snapshot/views/archive.py?ref_type=heads#L96
curl -v -L http://snapshot.debian.org/archive/deb
ian-security//20240702T082400Z/pool/updates/main/j/json-c/libjson-c-
dev_0.
15-2%2bdeb11u1_amd64.deb > /dev/null
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-
- 0* Trying 185.213.153.170:80...
* Connected to snapshot.debian.org (185.213.153.170) port 80 (#0)
> GET /archive/debian-
security//20240702T082400Z/pool/updates/main/j/json-c/libjson-c-
dev_0.15-2%2bdeb11u1_amd64.deb HTTP/1.1
> Host: snapshot.debian.org
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 302 FOUND
< Date: Tue, 05 Nov 2024 08:41:38 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Referrer-Policy: no-referrer
< X-Xss-Protection: 1
< Permissions-Policy: interest-cohort=()
< Content-Length: 279
< Location: /file/a152d21f12e32c7f40cb03220502e238dbb264fd
< Cache-Control: max-age=600, public
< Expires: Tue, 05 Nov 2024 08:51:38 GMT
< X-Clacks-Overhead: GNU Terry Pratchett
< Content-Type: text/html; charset=utf-8
< X-Varnish: 135556047
< Age: 0
< Via: 1.1 varnish (Varnish/7.1)
< Xonnection: close
< Connection: Keep-Alive
Best regards,
Felix Moessbauer
--
Siemens AG, Technology
Linux Expert Center
Reply to: