[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: My IP address seems listed as a spammer address by bugs.debian.org

On Mon, Nov 21, 2005 at 10:35:01AM -0500, Joey Hess wrote:
> Brian M. Carlson wrote:

>> Additionally, the robots.txt is being violated by "bts cache", so
>> perhaps someone should file a bug.

> "bts cache" is not a web spider, it only downloads the bugs that you
> tell it to. It violates the robots.txt much less than eg, wget.

wget (in non-recursive behaviour, never following any link) downloads
only the URLs you tell it too, so according to the same argument,
doesn't violate robots.txt . In recursive mode (following links), wget
respects robots.txt (by default).


Reply to: