Re: frequent gateway timeout on snapshots
On 6/19/2023 4:54 AM, Appu Goundan wrote:
Our project is experiencing higher failure rates than usual on
downloading package indexes from snapshots.debian.org
<http://snapshots.debian.org> in the last few weeks. Distroless
downloads the latest packages index for all architectures (arm, amd64,
arm 64, ppc64el, s390x) to update our multi-architecture container
images. Failures seem to be happening significantly more frequently at
the index downloading phase rather than the package download phase.
Download rates are super slow and sometimes just pause for no reason. I
don't know if we're getting prematurely throttled (this happens both in
CI and on my laptop) or there are other issues with the service at the
moment?
Also seeing this in both GitHub Actions CI Runners (Azure IPs I believe)
as well as my home ISP. The problem seems more pronounced in GitHub Actions.
Over the weekend I retried simple Dockerfile builds dozens of times and
only got 1-2 successes. And those took 10-30m (they used to take 1-2m).
http://snapshot.debian.org/archive/debian/20221105T150728Z if you want
to search logs.
https://github.com/indygreg/python-build-standalone/actions/runs/5663496252/job/15345326212
for a CI job that downloads at <10 KB/s.
It certainly feels like throttling or a very unhealthy server.
https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/blob/production/modules/roles/manifests/snapshot_web.pp
seems to contain the server config and does apply throttling. But I feel
like things regressed in the past month and this file hasn't changed. So
unsure what changed.
Reply to: