Re: [gopher] New Gopher Proxy
On Sat, Dec 05, 2009 at 02:38:07PM -0800, Cameron Kaiser wrote:
> If those people are really interested in
> gopher, then they need to be spidering gopherspace directly.
Why bother with an HTTP proxy to gopherspace if we want to prevent
HTTP-based spiders from accessing?
What might be useful is to figure out a way to serve up robot.txt to
HTTP-based spiders to prevent crawling of certain gopherspace pages
(as you brought up in your previous e-mail).
--Brian
--
Don't have gopher? Visit the world's first wiki-based gopher proxy!
http://www.pongonova.org/gopherwiki
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/gopher-project
Reply to: