[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

[gopher] Gopher Proxy & Archive robots.txt

The gopher proxy & archive ( http://gopher-project:gpa-test-675@gpa.internetearth.com/ ) should now pay attention to robots.txt exclusion rules. I haven't tested this extensively, but it seems to work. Does anybody know of any gopher pages that have been blocked by robots.txt so I can test?

Also, the spider will begin crawling gopherspace shortly.
Gopher-Project mailing list

Reply to: