Web crawler engine.
There are lot of web search engines around, i mean like altavista's
google's infoseek's etc..
What does one do when she wants to build one herself.
I mean what do you people use when you build local search engine
( city, locality, my needs are about 100 000 pages?webs? for the
beggining).
I've been looking at some open-source engines like htdig, swish-e,
swish++, udmsearch and the like..
all of them have some flaws which make them unusable -
htdig keeps it's data in db-files not in sql, same goes for swishes,
and udmsearch is written in highly undocumented and hairy c.
byuing services from altavista seems to be out of my league,
google would be nice, but building local service I need to
make it index only local things, and that can be done only
be using long list of ip-networks. So I need to rewrite indexing
part of engine to implement that.
Anyone doing sth like that? any comments and help appreciated.
regards, Eyck
Reply to: