Re: LinkWalker
On Mon, 7 Jan 2002 23:31, Nathan Strom wrote:
> > I have a nasty web spider with an agent name of "LinkWalker" downloading
> > everything on my site (including .tgz files). Does anyone know anything
> > about it?
>
> It's apparantly a link-validation robot operated by a company called
> SevenTwentyFour Incorporated, see:
> http://www.seventwentyfour.com/tech.html
Oops.
Actually they sent me an offer of a free trial to their service (which seems
quite useful). The free trial gave me some useful stats and let me fix a
bunch of broken links (of course I didn't pay).
Hmm, I wonder if they REALLY downloaded those files or aborted the transfers
after the first few K (needed to verify that the link was correct).
Anyway I'll remove that line from my iptables configuration now!
> Personally, I think this is a rogue organization -- there was an entry
> from this spider in our logs coming from a Seven24 IP with a HTTP
> referrer of
> www.adultinterracialsexvideos.com/interracialsex/interracialgroupsexsen.htm
>l. Needless to say, we do not run an adult web site and that referrer site
> does NOT have a link to us. Likely Seven24 is trying to clutter people's
> logs with references as a form of advertising.
A single entry in web logs does not mean much. If I blocked every origin of
a bad entry in my web logs I'd be busy all day doing it...
--
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/ My home page
Reply to: