Re: Anyone want a utility to find the best mirror?

[Seth Cohn <scohn@clipper.net>]

On Thu, 28 Sep 2000, Joe Emenaker wrote:

> Some time ago, I asked about any utility that could take a list of
> machines and find the one that's closest, so that I could find the
> best mirror to point dselect to. Someone responded that I should try
> "netselect", which did help a lot. However, netselect has a couple of
> shortcomings:

>   o It only tests ping times, not actual transfer rates of data

Right, because ping times are a good first step.  If the box pings really
slowly, the odds are the transfer rate will be greatly affected.
I don't care if the lines you and the mirror are on are T1s, if a critical
link in the middle is causing slow pings, the odds are the transfer will
be slow.

>   o It doesn't check to see if there's an ftp server there.
>   o It doesn't check to see if the ftp server accepts anonymous login
>   o It doesn't check to see if the ftp server even *still* mirrors the
> debian tree

Right, netselect is about as general as possible.  You can use it for web
site mirrors, file mirrors, gnutella sites, and much more.

> These last two are a real pain... especially the last one. Many of the sites
> I tried first in the mirrrors list didn't have the debian tree anymore.
> Hrmph!

So the list is out of date.  That isn't netselect's fault.

Can I suggest you focus on that side of the problem....

> So, anyway.... I broke down and wrote a little perl script which takes a
> README.mirrors.txt and tries to connect to each of the mirrors listed. It
> tries to connect anonymously and tries to navagate to the directory where
> the debian tree is supposed to be

This is a nice idea.  You are doing both http _and_ ftp, right?

> and then it actually tries to download a
> file from the site to see what kind of transfer rate it gets.

I suspect the mirrors admins would prefer this:

Use netselect to narrow down to 5 or sites from a list (or even 2-3?) and
_then_ do a transfer rate test.  I guarantee you: the results will be the
same, and more so, you will have skipped transfering data from dozens or
more other sites.

> It does *not* check for *currency* of the mirror. In other words, if the
> files in the mirror are all 6 months old, the script will not pick that
> up.... yet. I do plan, however, to make it sense that and take that into
> account.

Actually, I think the answer to this should be done on a server level: the
mirror _should_ have a timestamp of last update.  This would be all that
is needed.  If a site stopped mirroring, the timestamp would be way off.

Check with mirrors@debian.org (I bcc:ed this to them) and see if they are
doing this, and if not, why not?  That would be extremely useful to know.

> Anyway, if anybody wants to give it a whirl, I'm open to some
> beta-testing.....

I'd be interesting in seeing it.... but perl scripts that use netselect
to do the pinging make more sense.

I got this from either debian-user or debian-devel (can't recall) in the
last few weeks.  DNS on http.us.debian.org is round-robin, and returns
different mirrors each time.  Run this 3 or 4 times and put the mirror(s)
in place of the http.us.debian.org part of the /etc/apt/sources.list.  
Also use http.us.debian.org at the end of the sources.list file if you
want to be sure to always have a valid mirror somewhere... it'll only use
that if the other mirrors are down...


#!/usr/bin/perl -w
# Uses netselect to find "best" (fastest, closest) server.
# 'apt-get install netselect'
#
# Evan Martin <eeyem@u.washington.edu>
# Nov-11-1999

$host = 'http.us.debian.org';

print "Looking up $host...\n";
$iplist = nslookup($host, 'Addresses');
print "Available hosts: " . $iplist . "\n";
print "netselect'ng best host...\n";

open(NETSEL, "netselect " . join(' ', split(/\s*,\s*/, $iplist)) . "|") || die "netselect error";
$ip = (split(' ', <NETSEL>))[1];
close(NETSEL);

$besthost = nslookup($ip, 'Name');
print "Best host: $besthost\n";

sub nslookup() {
	my ($host, $word) = @_;
	open(NSLOOKUP, "nslookup $host 2>&1 |") || die "nslookup error";
	while (<NSLOOKUP>) {
		last if /$word/;
		if (/\*\*\*/) {
			print "nslookup says: $_";
			die "*** nslookup error";
		}
	}
	close(NSLOOKUP);
	s/$word:\s*//;
	chomp;
	return $_;
}