Re: My IP address seems listed as a spammer address by bugs.debian.org
On Mon, Nov 21, 2005 at 10:35:01AM -0500, Joey Hess wrote:
> Brian M. Carlson wrote:
>> Additionally, the robots.txt is being violated by "bts cache", so
>> perhaps someone should file a bug.
> "bts cache" is not a web spider, it only downloads the bugs that you
> tell it to. It violates the robots.txt much less than eg, wget.
wget (in non-recursive behaviour, never following any link) downloads
only the URLs you tell it too, so according to the same argument,
doesn't violate robots.txt . In recursive mode (following links), wget
respects robots.txt (by default).
--
Lionel
Reply to: