Re: [gopher] GopherMole - a gopher media crawler
On 01/13/2015 12:10 PM, James Mills wrote:
Q: What languages and libraries did you use to implement Gophermole?
ANSI C89. No libraries, besides libc and a SQL connector.
I assume the "sites to crawl" is some kind of FIFO Queue?
It's a queue, yes, but not FIFO. It's randomized, but with some weights
for servers that weren't ever polled (so discovery of new servers is
quick), and then priorities on sites that haven't been revalidated for
the longest time.
Do you run the spider(s) on some cloud?
Nope, all the magic happens on my server.
Mateusz
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: