please use robots.txt for your gopher apps
I love gopher apps and seeing them, but it is very hard for V2's robot to
automatically recognize them, requiring lots of manual work to pull stuff
out of the index that should never have been there in the first place. Please
use a robots.txt selector to keep the V2 robot out of these areas; I'm
considering a policy requirement that sites to be accepted to the new servers
page must have some sort of robots.txt up since this is becoming a (happy)
problem.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
-- Remember, kids: for great justice take off every zig! ----------------------
Reply to: