Re: Debian in Google's Index
On Wed, Mar 19, 2003 at 10:20:56AM -0800, Gabriela Valdes wrote:
> As you may know, Google's mission is to deliver the best search
> experience on the Internet by making the world's information universally
> accessible and useful. Currently, our Google.com site is within the top
> 10 sites in all major markets worldwide with over 200 million searches
> per day.
>
> We believe that www.debian.org is a great site and have discovered that
> Google is currently blocked from crawling your site by the robot.txt
> that is on your site. I believe we can drive a lot of traffic and
> awareness to your organization and would like to find a mutually
> beneficial way to work together.
How's that? http://www.debian.org/robots.txt says only:
User-agent: *
Disallow: /security/
Disallow:
I'm not sure why we ban /security/, but otherwise it should be perfectly
possible to crawl the remaining 773 MB of www.debian.org...
--
2. That which causes joy or happiness.
Reply to: