Re: In House Incremental Backup System - bash script

[CJ van den Berg <cj@vdbonline.com>]

On Fri, Aug 25, 2006 at 09:50:53AM +0100, Hagakure wrote:
> Hi,
> 
> I've got to upgrade our backup system (bash script) that currently uses
> rsync to backup two sites to a storage server.
> 
> Currently rsync is used to do an incremental copy of the data and then
> we archive it using tar. Unfortunately, the archives are now taking up a
> huge amount of hdd space.
> 
> What I am looking for is a pointer so that we still use rsync to copy
> the data, but rather than archive the entire backup, we have a tar file
> for each days differences.

I suggest you look at the --link-dest option of rsync. That tells rsync to
reuse identical files from previous backups by hardlinking to them. It
really works well, and is a very efficient way of doing on disk backups.
Especially if you want to do daily, or even hourly backups.

> Ideally, we would be able to insert the differences (file names only!)
> into a mysql db so we can recover files quickly.

If you use the rsync --link-dest method then it's not really necessary. Each
incremental backup is a complete tree that represents a point in time of the
backup source. So finding files is dead easy. Just use find or locate. :-)

> I looked into various options, but wondered if anyone might have a
> pointer to enable us to do this.

I've attached one of our backup scripts that uses rsync --link-dest to do
incremental local backups. The same concept can be used over the network to.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com

#!/bin/bash
SOURCE=/home/
BACKUPDIR=/backups

echo backing up to ${BACKUPDIR}

if [ ! -d ${BACKUPDIR} ]
then
    echo ERROR: Backup dir does not exist!
    exit 1
fi

if [ -e ${BACKUPDIR}/0 ]
then
    echo shifting backup dirs...
    pushd ${BACKUPDIR} > /dev/null
    for i in *; do mv ${i} $(( ${i} + 100000 + 1 )); done
    for i in *; do mv ${i} $(( ${i} - 100000 )); done
    popd > /dev/null
fi

TARGET=${BACKUPDIR}/0
LINKDEST=--link-dest=../1

echo backing up ${SOURCE}...
rsync -a ${LINKDEST} ${SOURCE} ${TARGET} || exit 1
echo syncing...
sync
echo done.

Attachment: signature.asc
Description: Digital signature