FWD: Mirror script

[Joey Hess <joeyh@debian.org>]

No, David, debian-devel is not a closed list.

----- Forwarded message from David Luyer <david_luyer@pacific.net.au> -----

From: David Luyer <david_luyer@pacific.net.au>
Date: Wed, 20 Dec 2000 03:12:42 +1100
To: joeyh@debian.org
Subject: Mirror script
X-Mailer: exmh version 2.2 06/23/2000 (debian 2.2-1) with nmh-1.0.4+dev


Re: http://lists.debian.org/debian-devel-0011/msg01827.html

Since my only interface to debian-devel is via reading the weekly news, and I 
suspect it's a closed list, I can't exactly follow up your example mirror
script.

But, anyway, here's a script that's almost the exact opposite - mirror by 
specifying what you don't want.  It doesn't allow you to specify a 
distribution though - just to narrow down to only binary-i386 (a local APT
source for a farm of i386 machines, some running stable, unstable, etc, so
a full i386 mirror is required but no sources/other architectures since
they're too rare in our environment).

BTW: Re: Waste bandwidth. Put a partial mirror on your laptop today!

Those were the days, when my Libretto 70CT could fit a full Debian mirror
(0.93r6 with far to spare... then things grew to the point I was excluding
EMACS and IRAF to fit on a Libretto 110CT... then I had to write a smarter
script to only mirror one revision at a time, resolving symlinks into older
versions, with the help of a modified ftp-mirror script... then I gave up
on keeping a full copy on my laptop because I had minimum 2Mbps anywhere
I went anyway... :-)

#!/bin/sh

# Script to mirror debian archive (binary-i386 by exclusion list only)

LOCK=/home/mirror/tmp/debian
EXCLUDE=/home/mirror/tmp/debian-exclude
LOG=/home/mirror/log/debian
DEST=/home/mirror/pub/linux/debian/
RSYNC_HOST=rsync.kernel.org
RSYNC_DIR=mirrors/debian/
RSYNC_OPTS="-zvvrlpt --force --delete --delete-excluded"
RSYNC=/usr/local/bin/rsync
OTHERS='{alpha,arm,hppa,hurd-i386,m68k,mips,mipsel,powerpc,sh,sparc,sparc64}'
SOURCES='{diff.gz,orig.tar.gz,dsc}'
INSTALL='{debian-installer/binary,disks,upgrade}'
EXCLUDES="# Exclude list
          pool/*/*/*/*_${OTHERS}.deb
          pool/*/*/*/*.${SOURCES}
          project/*/*_${OTHERS}.deb
          project/*/*.${SOURCES}
          dists/*-proposed-updates/*_${OTHERS}.{changes,deb}
          dists/*-proposed-updates/*.${SOURCES}
          dists/*/Contents-${OTHERS}.gz
          dists/*/main/${INSTALL}-${OTHERS}
          dists/*/*/binary-${OTHERS}
          dists/*/*/source"

if expr "`id`" : "uid=[0-9]*(mirror) gid=[0-9]*(.*)" >/dev/null 2>&1
then
  :
else
  echo "$0 must be run as user mirror not `id`"
  exit 1
fi

if [ -f $LOCK ]; then
  echo "Lock problem with $LOCK" | mail -s 'Debian Mirror' david_luyer@pacific.net.au
  exit 1
fi

touch $LOCK
rm -f $EXCLUDE
echo "$EXCLUDES" | perl -e '
use strict;

sub patex {
  if ($_[0] =~ /^([^{]*)\{([^}]*)\}(.*)$/) {
    my @lhs = map { $1.$_ } split(/,/, $2);
    my @rhs = patex($3);
    my (@ret, $lhs);
    for $lhs (@lhs) {
      push @ret, map { $lhs.$_ } @rhs;
    }
    return @ret;
  } else {
    return ($_[0]);
  }
}

while (<>) {
  chop;
  s/^\s+//g;
  next if /^$|^#/;
  print join ("\n", patex($_))."\n";
}' > $EXCLUDE
$RSYNC $RSYNC_OPTS --exclude-from $EXCLUDE $RSYNC_HOST::$RSYNC_DIR $DEST 2>&1 > $LOG
rm -f $LOCK

exit 0


-- 
David Luyer                                        Phone:   +61 3 9674 7525
Senior Network Engineer        P A C I F I C       Fax:     +61 3 9699 8693
Pacific Internet (Australia)  I N T E R N E T      Mobile:  +61 4 1111 2983
http://www.pacific.net.au/                         NASDAQ:  PCNTF


----- End forwarded message -----
-- 
see shy jo