Re: short-term: a way to overcome limits (?); long-term: CDN?
On Sat, 10 Aug 2024, Simon Josefsson wrote:
> Yaroslav Halchenko <debian@onerussian.com> writes:
> > Q2. is there work being done to establish/provide some kind of CDN
> > or caching proxy setup for snapshot.debian.org?
> I've been using http://snapshot-cloudflare.debian.org/ and the majority
> of files are possible to download, but there are some files that are
> notorious hard to get that way and always tend to raise 503 errors or
> just times out. I haven't been able to find a pattern, but I've managed
> to download around 150TB of snapshot's 180TB of data via cloudflare so
> it is reasonable usable. The speed from my point of view seems limited
> to around 50MB/s so there is room for improvement still.
Thank you Simon!
I guess I could immediately "take advantage" from it but adding more of
APT lines, i.e. pointing to the official repo and your mirror.
Is your mirror updated daily as for the metadata/new snapshots?
Could official snapshot folks assist in transferring those remaining
TBs which cause 503s?
PS on my end I mitigated by adjusting limits even further down:
-RUN echo 'Acquire::http::Dl-Limit "1000";' >| /etc/apt/apt.conf.d/20snapshots \
- && echo 'Acquire::https::Dl-Limit "1000";' >> /etc/apt/apt.conf.d/20snapshots \
- && echo 'Acquire::Retries "5";' >> /etc/apt/apt.conf.d/20snapshots
+RUN echo 'Acquire::http::Dl-Limit "200";' >| /etc/apt/apt.conf.d/20snapshots \
+ && echo 'Acquire::https::Dl-Limit "200";' >> /etc/apt/apt.conf.d/20snapshots \
+ && echo 'Acquire::Retries "10";' >> /etc/apt/apt.conf.d/20snapshots
and managed to build that docker image on CI.
Cheers,
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW: http://www.linkedin.com/in/yarik
Reply to: