Re: robots.txt disallow: /Lists-Archives/
On Thu, Dec 14, 2000 at 01:30:14PM +0100, Thomas Guettler wrote:
> Why do you disallow /Lists-Archives/? I used to search the archive
> with google. I prefered this to http://lists.debian.org/search.html
> because google searches in all lists and all archives by default.
> (site:debian.org)
Because of severe problems with robots (spiders) -- googlebot brought the
whole machine down last time. I'm still wary of allowing robots access to
that, even though we have robots.txt for BTS pages now...
BTW it's "lists.debian.org" now, not "www.debian.org/Lists-Archives/".
--
Digital Electronic Being Intended for Assassination and Nullification
Reply to: