[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: [gopher] A little Gopher crawler

Hi again,

So I wrote that crawler and improved upon it
(it's still single threaded) but here's what I have to share.

prologic@daisy
Thu Dec 25 20:06:53 
~/tmp
$ time ./gspider.py &> index

real 0m6.244s
user 0m0.120s
sys 0m0.047s

prologic@daisy
Thu Dec 25 20:07:16 
~/tmp
$ wc -l index
3039 index

This is run on localhost on the same machine against
my cgod python gopher server (see other thread).

The performance is *MUCH* better :)

cheers
James


James Mills / prologic

E: prologic@shortcircuit.net.au
W: prologic.shortcircuit.net.au

On Mon, Dec 15, 2014 at 3:23 PM, James Mills <prologic@shortcircuit.net.au> wrote:
Hi All and Kim (author of Gophernicus!),

I wrote this little cralwer today: https://gist.github.com/b781e02b0299fef1f3f6

I'm a bit disappointed in performance though of crawling my local Gopehrspace (basically via localhost):

prologic@daisy
Mon Dec 15 15:13:08 
~/tmp
$ time ./gspider.py &> index

real 5m27.825s
user 0m6.126s
sys 0m5.825s

prologic@daisy
Mon Dec 15 15:18:51 
~/tmp
$ wc -l index 
355 index

Any comments? :)

cheers
James
_______________________________________________
Gopher-Project mailing list
Gopher-Project@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project
Reply to: