[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: please use robots.txt for your gopher apps

Let me just add that it seems to me that globbing might be safer than regular expressions for the owner of the crawler because of denial of service attacks due to some regular expressions. Making sure these matches terminate is a challenge, I think. 

Reply to: