Re: [gopher] Strange implementations of "Missing file"
> > Thanks for the source code, I will try to use something similar.
> > If your server is a finger service, I will remove this address of the
> > crawler.
>
> The right way is to just ignore robots.txt for ports 79. The gopher request
> (and crawling) should be carried on normally.
I agree. This is exactly how V-2 does it.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
-- I know we can make it together! -- "Shogo" ---------------------------------
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: