Let me just add that it seems to me that globbing might be safer than regular expressions for the owner of the crawler because of denial of service attacks due to some regular expressions. Making sure these matches terminate is a challenge, I think.