Michael Marsh wrote:
> I'd say the most optimistic explanation is that google is finding what
> it's able to find while respecting site owners' wishes.
Which is, incidentally, more than MSNBot does. It slammed my forums for
months, several connections at a time. I put in a robots.txt to tell it to
sod off and for the next month it still slammed my site, 4-5 connections at a
time. Sad part is the forums they were trolling never showed up on their
search. So after several months I started blocking MSN's IPs at the firewall.
T'Hell with them and their lame spider.
--
Steve C. Lamb | I'm your priest, I'm your shrink, I'm your
PGP Key: 8B6E99C5 | main connection to the switchboard of souls.
-------------------------------+---------------------------------------------
Attachment:
signature.asc
Description: OpenPGP digital signature