[gopher] gopher proxies
I would like to suggest a means grande gopher site, a way of controling
access through the proxies.
with http we have the robots.txt file, and the gopher proxies have one
such file, but as the individual gopher sites are presenteted as pages
beloning to proxy site, we have to kindly ask the manageres of those pages
to include out pages there to limited access from roaming web crawler bots.
We need a standalized way of placing a .robots.txt file in the root of out
site, and say once a week, month such a file would be automatical
incorperated into the robots.txt on the proxy site, so we need a simple
parse, and a set of rules as to just how many linies we can have in that
file.
at the same time, we could need some way of getting a bit header
infomation also, useragent string is the one we need for this, if just the
proxy site would transmite the useragent of every 50th access by the means
of a sperat access to a comon select with the useragent appeded by the
means of the web get hack.
Say access to proxy?useragent=<bingbot,googlebot, fake german search
enginen etc etc>
Could all be done over the caps.txt file, by means or robot= lines and the
webget argument could be pass to that file also.
--
Jacob Dahl Pind | telefisk.org | fidonet 2:230/38.8
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: