RE: news gateways
Add the line
User-agent: *
Disallow: /BugsDisallow: /Bugs
/Disallow: /Bugs/
On 13-Dec-97 bruce@va.debian.org wrote:
> Does this mean that they are finding it through our own
> WWW archives of the mailing lists?
Yes. Search indexes like Yahoo index the whole site.
> Do we need to add something there to keep them from being indexed?
>
> Bruce
>
Add the line:
Disallow: /Lists-Archives/
to the robots.txt in the root directory of the web site. It currently reads:
User-agent: *
Disallow: /Bugs/
This will stop the "robots" from the search engines indexing that part of the
website. The only problem is that it won't clear what they previously have
indexed. Also, the change needs to be made to all site that mirror the mailing
lists.
----------------------------------
Please do not look directly into laser with remaining eye.
E-Mail: Mark Ciciretti <mciciretti@dzn.com>
Date: 14-Dec-97
Time: 21:04:01
----------------------------------
--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org .
Trouble? e-mail to templin@bucknell.edu .
Reply to: