[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Download a whole gopherhole using wget/curl?



Greetings.

For the crawling I already proposed for eomyidae and what if
follows some robots.txt for gopherspace.

See gopher://gopherproject.org/1/eomyidae for the details. All
automatic crawlers or archivers should keep to this standard.


Sincerely,

Christoph Lohmann


Reply to: