On Sat, Nov 13, 2004 at 03:43:43PM -0800, Paul Johnson wrote: > Pigeon <jah.pigeon@ukonline.co.uk> writes: > > > I must say I found it vaguely amusing to see that while so far Google > > has only indexed the top-level page of my website, MSN-Search has > > crawled the whole thing, including the odd references to "Microshite" > > and plugs for Debian... :-) > > I saw no reason to make Microsoft's search engine more useful (or make > money off my content by extension), so here's my robots.txt. Given > that Google delivers most of my hits from search engines, and MSN > usually only gives me four or five a year... > > User-agent: MSNBot > Disallow: / I see your point... but on the other hand, given that my website (a) has very little on it and (b) what is on it includes stuff like this: <A HREF="http://www.debian.org";>Debian GNU/Linux</A> Fed up with Microshite crashes, viruses, trojans, spyware, total ignorance of security and general endemic dysfunctionality? Linux is a REAL operating system. Find out more here.<BR> ...I think I'll leave it as it is! -- Pigeon Be kind to pigeons Get my GPG key here: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x21C61F7F
Attachment:
signature.asc
Description: Digital signature