Yaroslav Halchenko <debian@oneukrainian.com> writes: > On Sat, 10 Aug 2024, Simon Josefsson wrote: >> Yaroslav Halchenko <debian@onerussian.com> writes: >> > Q2. is there work being done to establish/provide some kind of CDN >> > or caching proxy setup for snapshot.debian.org? > >> I've been using http://snapshot-cloudflare.debian.org/ and the majority >> of files are possible to download, but there are some files that are >> notorious hard to get that way and always tend to raise 503 errors or >> just times out. I haven't been able to find a pattern, but I've managed >> to download around 150TB of snapshot's 180TB of data via cloudflare so >> it is reasonable usable. The speed from my point of view seems limited >> to around 50MB/s so there is room for improvement still. > > Thank you Simon! > > I guess I could immediately "take advantage" from it but adding more of > APT lines, i.e. pointing to the official repo and your mirror. > Is your mirror updated daily as for the metadata/new snapshots? > > Could official snapshot folks assist in transferring those remaining > TBs which cause 503s? I suspect that cloudflare provide an on-demand mirror of snapshot and when the rate-limiting of the real server hit cloudflare, the files doesn't reach cloudflare's mirror. Sometimes files eventually make it to cloudflare when you retry during a couple of times over a couple of hours, but I recall having some files being unavailable for several days and I eventually gave up and now fetches bulk from cloudflare and the failed files from the real server. Btw, since the last e-mail I'm now at 160TB so I'm reaching closure :) /Simon > PS on my end I mitigated by adjusting limits even further down: > > -RUN echo 'Acquire::http::Dl-Limit "1000";' >| /etc/apt/apt.conf.d/20snapshots \ > - && echo 'Acquire::https::Dl-Limit "1000";' >> /etc/apt/apt.conf.d/20snapshots \ > - && echo 'Acquire::Retries "5";' >> /etc/apt/apt.conf.d/20snapshots > +RUN echo 'Acquire::http::Dl-Limit "200";' >| /etc/apt/apt.conf.d/20snapshots \ > + && echo 'Acquire::https::Dl-Limit "200";' >> /etc/apt/apt.conf.d/20snapshots \ > + && echo 'Acquire::Retries "10";' >> /etc/apt/apt.conf.d/20snapshots > > and managed to build that docker image on CI. > > Cheers,
Attachment:
signature.asc
Description: PGP signature