[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Unique kernel with my own backup of all Debian repositories



Dear Étienne Mollier, 

Aha, a logical response. Thank you for shedding some light on this for me. That is probably the case. My access could perhaps be misinterpreted as an attack. Maybe I misunderstand the concept of a mirror, but I do not wish to maintain a server which allows the public to download Debian repositories. I'll look into it, in any case. If I find it is possible to simply download the entire collection, without having to host a mirror, I may very well go that route.

If I continue the scraping route, would adding wait time in my loop between downloads make my repeated access less of a problem? I would like to let it run until it is finished. It is tedious to restart my scrape periodically.

Thanks,
John

On Sat, Jun 12, 2021 at 10:35 AM Étienne Mollier <etienne.mollier@mailoo.org> wrote:
Hi John,

John E Petersen, on 2021-06-12:
> Hey folks, I’m developing a unique kernel based on Debian Linux, and I’ve
> been scraping the website for repositories. After a few thousand, the
> servers start to block my ip.

I'm not too sure what you are trying to achieve.  It sounds to
me like you wish to either develop a Debian derivative, or make
a backup copy of Debian.  The IP blocking you see is probably
automated, and the result of your having done repeated access on
a system that might not have been sized to be mirrored directly.
I would suppose this is in place, so regular users can access to
these ressources without being impacted by numerous background
download tasks hammering the websites.

Please have a look at the mirroring page[1] to assess whether
you want to mirror the packages archive, and if so, how to do it
with tools tailored for such task.

[1]: https://www.debian.org/mirror/ftpmirror

In hope this helps!
--
Étienne Mollier <etienne.mollier@mailoo.org>
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.

Reply to: