Re: backup of backup or alternating backups?
Hi,
On Mon, Oct 07, 2024 at 07:52:55PM -0600, Charles Curley wrote:
> I've used rsnapshot for several years now with no such issue. My
> rsnapshot repository resides on ext4, on its own LVM logical volume, on
> top of an encrypted RAID 5 array on four four terabyte spinning rust
> drives.
>
> /crc/rsnapshot root@hawk:~# df -i /crc/rsnapshot/
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/mapper/hawk--vg--raid-rsnapshot 16M 3.2M 13M 21%
This really isn't that much data and you have four drives to spread
random reads across, so I'm not surprised that you don't really feel it
yet.
When you have hundreds of millions of files in rsnapshot it really
starts to hurt because every backup run involves:
- Deleting the oldest tree of files;
- Walking the entire tree of the most recent backup once to cp -l it and
then;
- Walking it all again when rsync compares the new data to your previous
iteration.
Worse, it's all small, largely random IO which is worst case for
spinning media. It easily gets to the point where the copy and compare
steps take much longer than the actual data transfer.
Other backup solutions get better performance by using some sort of
index, manifest or other database, not just by walking every inode in
the filesystem. But are then more complicated.
This rsnapshot I have is really quite slow with only two 7200rpm HDDs.
It spends way longer walking its data store than actually backing up any
data. I could definitely make it speedier by switching to something
else. But I like rsnapshot for this particular case.
$ sudo find /data/backup/rsnapshot -print0 | grep -zc '.'
202326554
(This is a btrfs filesystem which doesn't report an inode count with df
-i)
Although it probably matters most how many files you have only in the
most recent backup iteration rather than the entire rsnapshot store. For
me that is approx 5.8 million.
Thanks,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Reply to: