Re: robots.txt contents.
On Fri, Jul 21, 2000 at 04:26:29PM -0300, Nicolás Lichtmaier wrote:
> This is the content of the robots.txt:
>
> User-agent: *
> Disallow: /Bugs/
> Disallow: /Lists-Archives/
> Disallow: /Packages/
> Disallow: /security/
> Disallow: /news.html
> Disallow: /consultants.html
> Disallow: /consultant_info/
> Disallow: /people.html
> Disallow:
>
> It shows, IMO, a misunderstanding of what the web is. Why would we disallow
> access to e.g. /Packages/ or /Lists-Archives/ ?
>
> Could we change this?
>
I don't know why Lists-Archives is in there, but I wouldn't put Packages in
there. Those pages are updated frequently enough that I see little reason
in having them indexed by outsiders.
--
James (Jay) Treacy
treacy@debian.org
Reply to: