Re: Stopping webcrawlers.
On Sun, Nov 03, 2019 at 12:26:01PM -0500, Gene Heskett wrote:
> I'll trim mine forthwith to the last entry. I've wondered if that was
> too long a list. And restart apache2 of course.
Hopefully it'll fix it for you too.
> But now I see the next
> access is not a 200, but a 404, that not intended. From the access log:
>
> coyote.coyote.den:80 209.197.24.34 - -
> [03/Nov/2019:12:19:55 -0500] "GET /gene/lathe-stf/linuxcnc4rpi4
> HTTP/1.1" 404 498 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0;
> rv:11.0) like Gecko"
>
> that directory exists, shouldn't that have been a 200?
I'd disregard it, based on user-agent only.
It's not a browser anyway (it's IE), so why bother?
Reco
Reply to: