anonftpsync update, 2nd try
On Fri, Nov 10, 2006 at 02:57:07PM +0100, Simon Paillard wrote:
> On Mon, Nov 06, 2006 at 12:10:45AM +0100, Josip Rodin wrote:
> > On Sun, Nov 05, 2006 at 02:43:57PM +0100, Dan Ohnesorg wrote:
> > > > They should first pull the debs, then update the Packages* files, then
> > > > delete. --delete-after --delay-updates to rsync should do the trick.
> > > > I'm CC'ing the admins.
> > >
> > > Thank you for the report, I will check it, we are using oficial mirror
> > > script, which suggest only --delete-after:
> >
> > You can optimize that further, by doing a two stage sync.
>
> Hi,
>
> Maybe we should update the official script provided on the website at
> http://www.debian.org/mirrors/anonftpsync to take better options into
> account.
>
> Here is a try, taking into account this discussion.
>
> Three stage update :
> - It first add new packages to the pool.
> - Then update the rest (excluding the pool) with --delay-updates and --delete-after
> - endly delete old packages in /pool.
As Guido S. Nickels told me, the third stage can be merged with the
second one.
Here is the diff againt anonftpsync.
--
Simon Paillard
Index: anonftpsync
===================================================================
RCS file: /cvs/webwml/webwml/english/mirror/anonftpsync,v
retrieving revision 1.21
diff -u -r1.21 anonftpsync
--- anonftpsync 26 Oct 2006 16:53:35 -0000 1.21
+++ anonftpsync 11 Nov 2006 13:59:56 -0000
@@ -117,8 +117,13 @@
# If that's the case on your system, make sure hostname prints the full
# hostname, and remove the -f option. If there's no hostname command,
# explicitly replace `hostname -f` with the hostname.
+
HOSTNAME=`hostname -f`
+# The hostname must match the "Site" field written in the list of mirrors.
+# If hostname doesn't returns the correct value, fill and uncomment below
+# HOSTNAME=mirror.domain.tld
+
LOCK="${TO}/Archive-Update-in-Progress-${HOSTNAME}"
# Exclude architectures defined in $ARCH_EXCLUDE
@@ -158,17 +163,16 @@
set +e
# First sync /pool
-rsync --recursive --links --hard-links --times --verbose --delete \
+rsync --recursive --links --hard-links --times --verbose \
$EXCLUDE $SOURCE_EXCLUDE \
$RSYNC_HOST::$RSYNC_DIR/pool/ $TO/pool/ >> $LOGFILE 2>&1
result=$?
if [ 0 = $result ]; then
# Now sync the remaining stuff
- rsync --recursive --links --hard-links --times --verbose --delete-after \
+ rsync --recursive --links --hard-links --times --verbose --delay-updates --delete-after \
--exclude "Archive-Update-in-Progress-${HOSTNAME}" \
--exclude "project/trace/${HOSTNAME}" \
- --exclude "/pool/" \
$EXCLUDE \
$RSYNC_HOST::$RSYNC_DIR $TO >> $LOGFILE 2>&1
#! /bin/sh
set -e
# This script originates from http://www.debian.org/mirror/anonftpsync
# modified by Martin Zobel-Helas <zobel@debian.org>, 2005-01-16
# these modifications are published under the terms of the GNU GPL
# Modifications:
# + some more documentation about variables
# + added ARCH_EXCLUDE
# + mirror in a safe way, first /pool, then /dists and the rest
# + add documentation where to find /usr/bin/procmail
# Version: $Id: anonftpsync,v 1.21 2006/10/26 16:53:35 spaillar Exp $
# Note: You MUST have rsync 2.0.16-1 or newer, which is available in slink
# and all newer Debian releases, or at http://rsync.samba.org/
# Note: You need to install the package procmail, it contains
# /usr/bin/lockfile, which is needed by this script.
# Set the variables below to fit your site. You can then use cron to have
# this script run daily to automatically update your copy of the archive.
# Don't forget:
# chmod 744 anonftpsync
# TO is the destination for the base of the Debian mirror directory
# (the dir that holds dists/ and ls-lR).
# (mandatory)
TO=
# RSYNC_HOST is the site you have chosen from the mirrors file.
# (http://www.debian.org/mirror/list-full)
# (mandatory)
RSYNC_HOST=
# RSYNC_DIR is the directory given in the "Packages over rsync:" line of
# the mirrors file for the site you have chosen to mirror.
# (mandatory)
RSYNC_DIR=debian/
# LOGDIR is the directory where the logs will be written to
# (mandatory)
LOGDIR=
# ARCH_EXCLUDE can be used to exclude a complete architecture from
# mirrorring. Please use as space seperated list.
# Possible values are:
# alpha, amd64, arm, hppa, hurd-i386, i386, ia64, m68k, mipsel, mips, powerpc, s390, sh and sparc
#
# There is one special value: source
# This is not an architecture but will exclude all source code in /pool
#
# eg.
# ARCH_EXCLUDE="alpha arm hppa hurd-i386 ia64 m68k mipsel mips s390 sparc"
#
# With a blank ARCH_EXCLUDE you will mirror all available architectures
# (optional)
ARCH_EXCLUDE=
# EXCLUDE is a list of parameters listing patterns that rsync will exclude.
# The following example would exclude mostly everything:
#EXCLUDE="\
# --exclude binary-alpha/ --exclude binary-arm/ --exclude binary-i386/ \
# --exclude binary-m68k/ --exclude binary-powerpc/ --exclude binary-sparc/ \
# --exclude binary-ia64/ --exclude binary-mips*/ --exclude binary-hppa/ \
# --exclude binary-sh/ --exclude binary-s390/ \
# --exclude binary-hurd-i386/ \
# --exclude *_alpha.deb --exclude *_arm.deb --exclude *_i386.deb \
# --exclude *_m68k.deb --exclude *_powerpc.deb --exclude *_sparc.deb \
# --exclude *_ia64.deb --exclude *_hppa.deb --exclude *_sh.deb \
# --exclude *_mips.deb --exclude *_mipsel.deb --exclude *_s390.deb \
# --exclude *_hurd-i386.deb \
# --exclude disks-alpha/ --exclude disks-arm/ --exclude disks-i386/ \
# --exclude disks-ia64/ --exclude disks-m68k/ --exclude disks-mips*/ \
# --exclude disks-powerpc/ --exclude disks-s390/ --exclude disks-sparc/ \
# --exclude stable/ --exclude testing/ --exclude unstable/ \
# --exclude source/ \
# --exclude *.orig.tar.gz --exclude *.diff.gz --exclude *.dsc \
# --exclude /contrib/ --exclude /non-free/ \
# "
# With a blank EXCLUDE you will mirror the entire archive.
# (optional)
EXCLUDE=
# MAILTO is the address to send logfiles to;
# if it is not defined, no mail will be sent
# (optional)
MAILTO=
# There should be no need to edit anything below this point, unless there
# are problems.
#-----------------------------------------------------------------------------#
# Check for some environment variables
if [ -z $TO ] || [ -z $RSYNC_HOST ] || [ -z $RSYNC_DIR ] || [ -z $LOGDIR ]; then
echo "One of the following variables seems to be empty:"
echo "TO, RSYNC_HOST, RSYNC_DIR or LOGDIR"
exit 2
fi
if ! [ -d ${TO}/project/trace/ ]; then
# we are running mirror script for the first time
mkdir -p ${TO}/project/trace
fi
# Note: on some non-Debian systems, hostname doesn't accept -f option.
# If that's the case on your system, make sure hostname prints the full
# hostname, and remove the -f option. If there's no hostname command,
# explicitly replace `hostname -f` with the hostname.
HOSTNAME=`hostname -f`
# The hostname must match the "Site" field written in the list of mirrors.
# If hostname doesn't returns the correct value, fill and uncomment below
# HOSTNAME=mirror.domain.tld
LOCK="${TO}/Archive-Update-in-Progress-${HOSTNAME}"
# Exclude architectures defined in $ARCH_EXCLUDE
for ARCH in $ARCH_EXCLUDE; do
EXCLUDE=$EXCLUDE"\
--exclude binary-$ARCH/ \
--exclude disks-$ARCH/ \
--exclude installer-$ARCH/ \
--exclude Contents-$ARCH.gz \
--exclude *_$ARCH.deb \
--exclude *_$ARCH.udeb "
if [ "$ARCH" == "source" ]; then
SOURCE_EXCLUDE="\
--exclude *.tar.gz \
--exclude *.diff.gz \
--exclude *.dsc "
fi
done
# Logfile
LOGFILE=$LOGDIR/debian-mirror.log
# Get in the right directory and set the umask to be group writable
#
cd $HOME
umask 002
# Check to see if another sync is in progress
if lockfile -! -l 43200 -r 0 "$LOCK"; then
echo ${HOSTNAME} is unable to start rsync, lock file exists
exit 1
fi
# Note: on some non-Debian systems, trap doesn't accept "exit" as signal
# specification. If that's the case on your system, try using "0".
trap "rm -f $LOCK > /dev/null 2>&1" exit
set +e
# First sync /pool
rsync --recursive --links --hard-links --times --verbose \
$EXCLUDE $SOURCE_EXCLUDE \
$RSYNC_HOST::$RSYNC_DIR/pool/ $TO/pool/ >> $LOGFILE 2>&1
result=$?
if [ 0 = $result ]; then
# Now sync the remaining stuff
rsync --recursive --links --hard-links --times --verbose --delay-updates --delete-after \
--exclude "Archive-Update-in-Progress-${HOSTNAME}" \
--exclude "project/trace/${HOSTNAME}" \
$EXCLUDE \
$RSYNC_HOST::$RSYNC_DIR $TO >> $LOGFILE 2>&1
LANG=C date -u > "${TO}/project/trace/${HOSTNAME}"
else
echo "ERROR: Help, something weird happened" | tee -a $LOGFILE
echo "mirroring /pool exited with exitcode" $result | tee -a $LOGFILE
fi
if ! [ -z $MAILTO ]; then
mail -s "debian archive synced" $MAILTO < $LOGFILE
fi
savelog $LOGFILE
Reply to: