debmirror script responsiveness [PATCH]
Joey,
For the poor punters using FTP mirroring on slow connections (eg, T1
or less), the following patch to debmirror-20020427 provides the
following enhancements:
- ftp_get() function checks size and MDTM of files so that
Contents-*.gz, etc are not transferred twice if it is not needed.
This was the first thing that annoyed me as I had to wait about a
minute for the script to download a file it had already
downloaded :-).
- `--debug' split into `--debug' and `--ftpdebug'
- `--debug' displays output when checking file MD5 sums
I think it makes the script a bit more responsive, especially in debug
mode.
--- debmirror.orig Thu Jun 13 15:02:50 2002
+++ debmirror Thu Jun 13 15:32:38 2002
@@ -50,7 +50,7 @@
sub usage {
warn join(" ", @_)."\n" if @_;
warn <<EOF;
-Usage: $0 mirrordir [--debug] [--progress] [--source|--nosource]
+Usage: $0 mirrordir [--debug] [--ftpdebug] [--progress] [--source|--nosource]
[--md5sums|--nomd5sums] [--passive] [--host=remotehost]
[--user=remoteusername] [--method=ftp|rsync] [--timeout=seconds]
[--root=directory] [--dist=foo[,bar,..] ...]
@@ -77,7 +77,11 @@
=item --debug
-Enables verbose debug output, including ftp protocol dump.
+Enables verbose debug output, excluding ftp protocol dump.
+
+=item --ftpdebug
+
+Enables verbose debug output of the ftp connection
=item --progress -p
@@ -225,7 +229,7 @@
# Yeah, I use too many global variables in this program.
my ($debug, $progress, $check_md5sums, $passive, $skippackages,
- $getcontents);
+ $getcontents, $ftpdebug);
my (@dists, @sections, @arches, @extra_dirs, @ignores, @excludes, @includes);
my $cleanup=1;
my $do_source=1;
@@ -244,6 +248,7 @@
my $help;
GetOptions(
'debug' => \$debug,
+ 'ftpdebug' => \$ftpdebug,
'progress|p' => \$progress,
'verbose' => \$progress,
'source!' => \$do_source,
@@ -322,7 +327,7 @@
# Start up ftp.
my $ftp;
-my %opts = (Debug => $debug, Passive => $passive, Timeout => $timeout);
+my %opts = (Debug => $ftpdebug, Passive => $passive, Timeout => $timeout);
my $rsynctempfile;
END { unlink $rsynctempfile if $rsynctempfile }
@@ -539,11 +544,15 @@
my ($filename, $size, $md5sum)=@_;
if (-f $filename and $size == -s _) {
if ($check_md5sums) {
+ debug ("MD5 CHK - $filename");
open HANDLE, $filename or
die "$filename: $!";
$md5->addfile(*HANDLE);
if ($md5sum eq $md5->hexdigest) {
return 0;
+ } else {
+ debug ("MD5 NOT OK - $filename");
+ return 1;
}
}
else {
@@ -576,17 +585,40 @@
my $oldautoflush = $|;
$| = 1;
my $file=shift;
- print "$file: " if $progress;
- my $ret=$ftp->get($file, $file);
- if ($ret) {
- my $mtime=$ftp->mdtm($file);
- utime($mtime, $mtime, $file) if defined $mtime;
+ my $force = shift;
+
+ my ($skip, $mtime);
+ if (!$force and my @stat = stat $file) {
+ my $size = $ftp->size($file);
+ if ( $size == $stat[7] ) {
+ if ($mtime = $ftp->mdtm($file)) {
+ if (abs($mtime-$stat[9]) < 3600) {
+ # close enuf :-)
+ $skip = 1;
+ }
+ } else {
+ # some might prefer $skip = 0 here
+ $skip = 1;
+ }
+ }
}
- else {
- warn "$file failed\n";
+
+ if ($skip) {
+ debug("Updating $file [OK]");
+ return 1;
+ } else {
+ debug("Updating $file [FETCHING]");
+ print "$file: " if $progress;
+ my $ret=$ftp->get($file, $file);
+ if ($ret) {
+ $mtime ||= $ftp->mdtm($file);
+ utime($mtime, $mtime, $file) if defined $mtime;
+ } else {
+ warn "$file failed\n";
+ }
+ $| = $oldautoflush;
+ return $ret;
}
- $| = $oldautoflush;
- return $ret;
}
sub rsync_get {
--
Sam Vilain, sam@vilain.net WWW: http://sam.vilain.net/
7D74 2A09 B2D3 C30F F78E GPG: http://sam.vilain.net/sam.asc
278A A425 30A9 05B5 2F13
Never invest your money in anything that eats or needs painting.
BILLY ROSE
--
To UNSUBSCRIBE, email to debian-devel-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Reply to: