Re: [gopher] gopher proxies
On Tue, 23 Jul 2013, Cameron Kaiser wrote:
I think these are good thoughts to consider, but the problem is that most of
the "overly permissive" proxies are on servers run by folks who either aren't
on these lists, or disagree with the premise that the content should not be
indexed. Realistically, asking them to do more work won't fly, unless there
were a concerted cooperative effort to simply drop requests from them on the
floor and refuse to serve their proxies data until changes were made.
had hoped if there was a sample implementation that parsed, check and
generated that unifed robots.txt, it would be easy enough for them to
implement it, but you are right, in the end we are at their mercy and
blocking is the only option.
Have been using the redirection code I made, that servering up nonsense to
them, I even though about serving them piratebay data, but that would at best
get them delisted from google and bing but I doubt its ether of those that is
causing those none stop crawer storms.
I see the same behaviour on http from:
user-agent: Lightspeed
user-agent: SISTRIX Crawler
user-agent: Baiduspider
user-agent: YandexBot
user-agent: Ezooms
user-agent: Exabot
user-agent: AhrefsBot
--
Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: