Improved anonftpsync script
[Please Cc me for replies, I'm not subscibed to this list]
Hello All,
The anonymous rsysnc mirroring script available from
http://www.debian.org/mirror/anonftpsync
has several problems:
- It fails to exclude some arch-specific stuff, namely *.changes and
Contrents-*.gz
- It also doesn't exclude arch-specific parts of debian-installer,
that is *.udeb and installer-*
- The "set +e" before the rsync prevents the removal of the lockfile
to get triggered, so stale lockfiles may remain.
I guess the "set +e" was done to get a saved log in case of errors, but
moving that upwards is better, and also prevents timestamp updates when
the rsync went wrong.
In the end, I rewrote the whole script in order to fix those flaws.
It's appended at this mail, and I think it's a better example for
a mirror script. I haven't tested it on non-Debian systems, though,
so I may have missed some portability issues.
Thiemo
#! /bin/sh
#
# Copyright 2004 Thiemo Seufer <seufer@csv.ica.uni-stuttgart.de>
#
# This file is distributed under the terms of the GNU General Public License,
# Version 2, as published at http://www.gnu.org/licenses/gpl.html
#
# This script originates from http://www.debian.org/mirror/anonftpsync
#
# Note: You MUST have rsync 2.0.16-1 or newer, which is available in slink
# and all newer Debian releases, or at http://rsync.samba.org/
#
# Set the variables at the end of the file to fit your site. You can then use
# cron to have this script run daily to automatically update your copy of the
# archive.
#
# Don't forget:
# chmod 744 anonftpsync
# ---------------------------------------------------------------------------
# There should be no need to edit anything in this section, unless there are
# problems. See the end of the file for customization.
set -e
exclude_arch ()
{
while [ $# -ge 1 ]; do
echo -n "--exclude binary-${1}/ --exclude *_${1}.deb --exclude *_${1}.changes --exclude Contents-${1}.gz --exclude disks-${1}/ --exclude *_${1}.udeb --exclude installer-${1}/ "
shift
done
return 0
}
exclude_source ()
{
case "$EXCLUDE_SOURCE" in
"y*" | "Y*") echo -n "--exclude source/ --exclude *.orig.tar.gz --exclude *.diff.gz --exclude *.dsc" ;;
esac
}
do_mirror ()
{
_rsync_host=$1
_rsync_dir=$2
_target_dir=$3
# Get in the right directory and set the umask to be group writable.
cd $HOME
umask 002
# Note: on some non-Debian systems, hostname doesn't accept -f option.
# If that's the case on your system, make sure hostname prints the full
# hostname, and remove the -f option. If there's no hostname command,
# explicitly replace `hostname -f` with the hostname.
_target_host=`hostname -f`
LOCK="${_target_dir}/Archive-Update-in-Progress-${_target_host}"
mkdir -p ${_target_dir}/project/trace
# Check to see if another sync is in progress
if lockfile -! -l 43200 -r 0 "$LOCK"; then
echo ${_target_host} is unable to start rsync, lock file exists
return 1
fi
# Note: on some non-Debian systems, trap doesn't accept "EXIT"
# as signal specification. If that's the case on your system,
# try using "0".
trap "rm -f $LOCK > /dev/null 2>&1" EXIT
# Note: if you don't have savelog, use any other log rotation facility, or
# comment this out, the log will simply be overwritten each time.
savelog ${_target_dir}/project/trace/rsync.log > /dev/null 2>&1
rsync --recursive --links --hard-links --times --verbose \
--compress --delete $RSYNC_OTHEROPTS \
--exclude "Archive-Update-in-Progress-${_target_host}" \
--exclude "project/trace/*" \
`exclude_arch $EXCLUDE_ARCH` \
`exclude_source` \
$EXCLUDE_OTHER \
${_rsync_host}::${_rsync_dir} ${_target_dir} > ${_target_dir}/project/trace/rsync.log 2>&1
date -u > "${_target_dir}/project/trace/${_target_host}"
rm -f $LOCK > /dev/null 2>&1
return 0
}
# ---------------------------------------------------------------
# Customize this section
# Things to exclude.
# With blank EXCLUDE_* you will mirror the entire archive.
# This sample would exclude all architectures:
# EXCLUDE_ARCH="alpha arm m68k mips mipsel powerpc sparc i386 ia64 hppa sh s390 hurd-i386"
EXCLUDE_ARCH=
# This sample would exclude the source code:
# EXCLUDE_SOURCE="y"
EXCLUDE_SOURCE="n"
# Exclude other things, like some symlinks or sections.
# The --exclude option must be given in this case.
# Samples:
# EXCLUDE_OTHER="--exclude stable/ --exclude testing/ --exclude unstable/"
# EXCLUDE_OTHER="--exclude /contrib/ --exclude /non-free/"
#
# Exclude old distributions. Note that this won't work well for newer
# ones which have their files in a pool directory.
# EXCLUDE_OTHER="--exclude /slink/ --exclude /slink-proposed-updates/"
# EXCLUDE_OTHER="--exclude /potato/ --exclude /poatato-proposed-updates/"
EXCLUDE_OTHER=
# Additional options for rsync
# Common options might include bwlimit (but see Debian bug #181336)
# Sample:
# RSYNC_OTHEROPTS="--bwlimit=23 --safe-links --delete-after"
RSYNC_OTHEROPTS=
# Trees to mirror
#
# do_mirror RSYNC_HOST RSYNC_DIR TARGET_DIR
#
# RSYNC_HOST is the site you have chosen from the mirrors file.
# (http://www.debian.org/mirror/list-full)
#
# RSYNC_DIR is the directory given in the "Packages over rsync:" line of
# the mirrors file for the site you have chosen to mirror.
#
# TARGET_DIR is the destination for the base of the Debian mirror directory
# (the dir that holds dists/ and ls-lR).
#
# Samples:
# do_mirror rsync.example.org debian/ /bigdisk/mirror/debian
# do_mirror rsync.example.org debian-non-US/ /bigdisk/mirror/debian-non-US
# do_mirror rsync.example.org debian-security/ /bigdisk/mirror/debian-security
# End of customize section
# ---------------------------------------------------------------
exit 0
Reply to: