[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: [gopher] Spidering teh gopherspace



> Anyway, on to a few questions: What kind of spidering rate would the 
> admins here accept? The spider will index types 0 and 1 (text documents 
> and menus) and currently does three hits per second (actually, a hit and 
> a 1/3 second delay). I think that's too fast - so how does a hit per 
> second sound like? I'll take forever to spider things, but at least it 
> wouldn't kill anyones server....

V-2 uses five seconds, actually, and it only grabs menus (in keeping with
the original Veronica concept), so it is a pretty low impact spider.

-- 
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
-- Artificial intelligence is no match for natural idiocy. --------------------

_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/gopher-project




Reply to: