debmirror script responsiveness [PATCH]

[Sam Vilain <sam@vilain.net>]

Joey,

For the poor punters using FTP mirroring on slow connections (eg, T1
or less), the following patch to debmirror-20020427 provides the
following enhancements:

  - ftp_get() function checks size and MDTM of files so that
    Contents-*.gz, etc are not transferred twice if it is not needed.
    This was the first thing that annoyed me as I had to wait about a
    minute for the script to download a file it had already
    downloaded :-).

  - `--debug' split into `--debug' and `--ftpdebug'

  - `--debug' displays output when checking file MD5 sums

I think it makes the script a bit more responsive, especially in debug
mode.

--- debmirror.orig      Thu Jun 13 15:02:50 2002
+++ debmirror   Thu Jun 13 15:32:38 2002
@@ -50,7 +50,7 @@
 sub usage {
         warn join(" ", @_)."\n" if @_;
         warn <<EOF;
-Usage: $0 mirrordir [--debug] [--progress] [--source|--nosource]
+Usage: $0 mirrordir [--debug] [--ftpdebug] [--progress] [--source|--nosource]
           [--md5sums|--nomd5sums] [--passive] [--host=remotehost]
           [--user=remoteusername] [--method=ftp|rsync] [--timeout=seconds]
           [--root=directory] [--dist=foo[,bar,..] ...]
@@ -77,7 +77,11 @@
 
 =item --debug
 
-Enables verbose debug output, including ftp protocol dump.
+Enables verbose debug output, excluding ftp protocol dump.
+
+=item --ftpdebug
+
+Enables verbose debug output of the ftp connection
 
 =item --progress -p
 
@@ -225,7 +229,7 @@
 
 # Yeah, I use too many global variables in this program.
 my ($debug, $progress, $check_md5sums, $passive, $skippackages,
-    $getcontents);
+    $getcontents, $ftpdebug);
 my (@dists, @sections, @arches, @extra_dirs, @ignores, @excludes, @includes);
 my $cleanup=1;
 my $do_source=1;
@@ -244,6 +248,7 @@
 my $help;
 GetOptions(
        'debug'         => \$debug,
+       'ftpdebug'      => \$ftpdebug,
        'progress|p'    => \$progress,
        'verbose'       => \$progress,
        'source!'       => \$do_source,
@@ -322,7 +327,7 @@
 
 # Start up ftp.
 my $ftp;
-my %opts = (Debug => $debug, Passive => $passive, Timeout => $timeout);
+my %opts = (Debug => $ftpdebug, Passive => $passive, Timeout => $timeout);
 
 my $rsynctempfile;
 END { unlink $rsynctempfile if $rsynctempfile }
@@ -539,11 +544,15 @@
        my ($filename, $size, $md5sum)=@_;
        if (-f $filename and $size == -s _) {
                if ($check_md5sums) {
+                       debug ("MD5 CHK - $filename");
                        open HANDLE, $filename or
                                die "$filename: $!";
                        $md5->addfile(*HANDLE);
                        if ($md5sum eq $md5->hexdigest) {
                                return 0;
+                       } else {
+                               debug ("MD5 NOT OK - $filename");
+                               return 1;
                        }
                }
                else {
@@ -576,17 +585,40 @@
        my $oldautoflush = $|;
        $| = 1;
        my $file=shift;
-       print "$file: " if $progress;
-       my $ret=$ftp->get($file, $file);
-       if ($ret) {
-               my $mtime=$ftp->mdtm($file);
-               utime($mtime, $mtime, $file) if defined $mtime;
+       my $force = shift;
+
+       my ($skip, $mtime);
+       if (!$force and my @stat = stat $file) {
+               my $size = $ftp->size($file);
+               if ( $size == $stat[7] ) {
+                       if ($mtime = $ftp->mdtm($file)) {
+                               if (abs($mtime-$stat[9]) < 3600) {
+                                       # close enuf :-)
+                                       $skip = 1;
+                               }
+                       } else {
+                               # some might prefer $skip = 0 here
+                               $skip = 1;
+                       }
+               }
        }
-       else {
-               warn "$file failed\n";
+
+       if ($skip) {
+                debug("Updating $file [OK]");
+               return 1;
+       } else {
+                debug("Updating $file [FETCHING]");
+               print "$file: " if $progress;
+               my $ret=$ftp->get($file, $file);
+               if ($ret) {
+                       $mtime ||= $ftp->mdtm($file);
+                       utime($mtime, $mtime, $file) if defined $mtime;
+               } else {
+                       warn "$file failed\n";
+               }
+               $| = $oldautoflush;
+               return $ret;
        }
-       $| = $oldautoflush;
-       return $ret;
 }
 
 sub rsync_get {


--
   Sam Vilain, sam@vilain.net     WWW: http://sam.vilain.net/
    7D74 2A09 B2D3 C30F F78E      GPG: http://sam.vilain.net/sam.asc
    278A A425 30A9 05B5 2F13

  Never invest your money in anything that eats or needs painting.
BILLY ROSE


-- 
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org