|
Very nice! On 12/08/2011 08:38 AM, Wolfgang F. wrote: The gopher proxy & archive ( http://gopher-project:gpa-test-675@gpa.internetearth.com/ ) should now pay attention to robots.txt exclusion rules. I haven't tested this extensively, but it seems to work. Does anybody know of any gopher pages that have been blocked by robots.txt so I can test? |
_______________________________________________ Gopher-Project mailing list Gopher-Project@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project