Re: backup strategy (was Re: reconstruct the system)

[Marty <martyb@ix.netcom.com>]

I (Marty) wrote:
  (The scripts "make-md5sums," "check-backup" and a few other related scripts
> were posted in my message to debian-user called "My local debian archive maintenance
> scripts," posted on May 12 and accessable in the archives.

Sorry, this statement is incorrect.  Below are my scripts for making and checking md5sums
for debian and arbitrary file archives, which probably should have been included in the
May 12 posting.

(Note that one of my motivations for posting these is to let any repressed shellscript
wizards have at it, since I'm no shell script wizard myself and I'm sure there are
cleaner ways to do these tasks.  Hope that I can get some ideas from other list members
about how to clean them up partially offsets the embarassment of posting them here.
Second note: Another reason for posting these is that I have spent inordinate amounts
of time getting them to handle arbitrary file names, and they are close to perfect now,
if not 100% perfect.  If anyone finds remaining failure cases, in the form of pathological
file names that these scritps can't handle, then I would appreciate if they notify me so
that I can debug the scripts accordingly.)


Contents of script /mnt/install/test/make-md5sums:
-------------------------------------------
rm $1/md5sums.gz
DATE=`date +%j%H%M%S`
pushd $1; find -exec /mnt/install/test/make-md5sum {} \; | cat >/tmp/md5sums.$DATE;popd
gzip -c /tmp/md5sums.$DATE >$1/$2/md5sums.gz
-------------------------------------------

Contents of make-md5sum:
-------------------------------------------
#if path is not a normal file then assign either 0 or 1 as its md5sum depending on its type
if [ -h "$1" ]
then
	echo "00000000000000000000000000000000  $1"
else
if [ -f "$1" ]
then
	md5sum "$1"
else
if [ -e "$1" ]
then
	echo "00000000000000000000000000000001  $1"
else
	echo ----------  path not found ----------- >&2
	echo "---->>>>>  $1  <<<<<-----" >&2
	echo --------- possible name problem ------------>&2
fi
fi
fi
-------------------------------------------

Here are the checking scripts:

Contents of script /mnt/install/test/check-backup:
zcat $1/md5sums.gz | sed 's/  /&"/' | sed 's/$/"&/' |  sed 's/\\/\\\\/g' |  /mnt/install/test/md5chk $1
-------------------------------------------
(notes: One function check-backup is to make arbitrary file name handling more robust
by adding the double quote characters.  I'm sure this can be more cleanly done with
awk or cut and tr, but sed is the one I always seem to remember.  Any suggestions for
cleanup are appreciated. :-)

Contents of script /mnt/install/test/md5chk:
-------------------------------------------
cd $1
while read md5 filep
do
	#echo "$md5  $filep"
	filepath=`echo "$filep" | sed 's/"//g'`

	if [ -h "$filepath" ]
	then
		testmd5="00000000000000000000000000000000"
	else
	if [ -f "$filepath" ]
	then
                testmd5=`md5sum "$filepath" | (read md5 filepath; echo $md5;)`
	else
	if [ -e "$filepath" ]
	then
		testmd5="00000000000000000000000000000001"
	else
		echo "$filepath" not found | tee >&2
		echo
		continue
	fi
	fi
	fi

	if [ "$testmd5" != "$md5" ]
	then
		echo "$filepath md5sums don't match" | tee >&2
		echo "orig md5sum= $md5" | tee >&2
		echo "test md5sum= $testmd5" | tee >&2
		echo
	fi
	#echo "$testmd5  $filepath found"
done
---------------------------------------------------

For good measure, here are test scripts which use some of the previous scripts
here and in my May 12 posting, to check my (x86-specific) debian archives:

Contents of script chkloop:
----------------------------------
echo starting at `date`
while true
do
time ./check-debian ../debian indices
time ./check-debian ../debian-non-US indices-non-US
echo done at `date`
echo
echo
echo
sleep 5
done

(Note: this script is in a looped version because it doubles as a
stress test which I use to load systems to test for intermittent
hardware failures, in combination with other stress tests.)

Contents of script check-debian:
---------------------------
zcat $1/$2/md5sums.gz |egrep -v '_arm|\-arm|\_alpha|\-alpha|hurd|powerpc|m68k|hppa|mips|mipsel|sparc|ia64|s390|potato|slink'| /mnt/install/test/md5chk $1
---------------------------