Dear Snapshot people,
TL;DR Questions:
Q1. how to "throttle" connection or otherwise overcome problem of
snapshots cutting of our connection for a heavy "apt-get install"?
Q2. is there work being done to establish/provide some kind of CDN
or caching proxy setup for snapshot.debian.org?
Details:
To refresh memory about myself: I have been a "user" of
snapshot.debian.org service for long time, thank you for developing and
keeping it running:
- we run our "fork" http://snapshot-neuro.debian.net/
- https://packages.debian.org/neurodebian-freeze package provides
nd_freeze which given a date would freeze debian/neurodebian APT
config (old fashioned one, pre deb822 adoption) to use debian and
neurodebian snapshots repos.
We promote such use of nd_freeze largely in the scope of creating
reproducible container recipes -- so you could rebuild container from
its e.g. Dockerfile while minimizing any changes due to changes in
upstream APT repos.
As such, those rebuilds of containers happen rarely but could be quite
"heavy".
One immediate use-case is our CI setup for building/testing git-annex of
Joey Hess: https://github.com/datalad/git-annex . There we rely on a
development container image
https://github.com/datalad/git-annex/blob/master/.github/workflows/tools/containers/buildenv-git-annex/Dockerfile
But even despite our added throttling and retries for APT
RUN echo 'Acquire::http::Dl-Limit "1000";' >| /etc/apt/apt.conf.d/20snapshots \
&& echo 'Acquire::https::Dl-Limit "1000";' >> /etc/apt/apt.conf.d/20snapshots \
&& echo 'Acquire::Retries "5";' >> /etc/apt/apt.conf.d/20snapshots
which I think used to have desired effect and letting us reliably build
that container. But currently we get (while running on github CI) into
#7 590.3 Could not connect to snapshot.debian.org:80 (185.17.185.185). - connect (111: Connection refused) [IP: 185.17.185.185 80]
#7 590.3 Err:120 http://snapshot.debian.org/archive/debian/20230829T151102Z bookworm/main amd64 libipc-system-simple-perl all 1.30-2
full log: https://github.com/datalad/git-annex/actions/runs/10258125473/job/28382982779 .
Q1: Do you think throttling even further would be of help? any
other means to overcome?
Q2: I know that snapshot.debian.org is sizeable (135T IIRC), but I
wonder if someone checked/looking into getting some CDN or may some
setting up some caching proxy (for most frequent/popular requests) to
improve its accessibility?
Thank you in advance for your time,
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW: http://www.linkedin.com/in/yarik
Attachment:
signature.asc
Description: PGP signature