[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Problems with making hardlink-based backups

Hi list.

Until recently I was using rdiff-backup for backing up our servers.
But after a certain point, the rdiff-backup process would take days,
use up huge amounts of CPU and RAM, and the debug logs were very
opaque. And the rdiff-backup mailing lists weren't very helpful

So, for servers where that's a problem, I changed over to a hardlink
snapshot-based aproach.

That's been working fine for the past few weeks. But now, the backup
server is running out of harddrive space and we need to check which
backup histories is taking up the most space, so we can prune the
oldest week or two from them.

The problem now is, that 'du' takes days to run, and now generates 4-5
GB text files. This is of course caused by du having to now process
dozens of additional snapshot directories, many of them with a large
number of files.

What I've been doing, is writing helper scripts to help prune the
earlier directories. Something like this:

1) Compare files under snapshot1 & snapshot2. If any files under
snapshot1 are hardlinks to the same files under snapshot2, then remove
them from snapshot1, and add an entry to a text file (for possible
later regeneration).

2) Remove empty directories (and add their details to a text file)

3) 7zip-compress the text files containing recovery info.

4) Possibly later (before empty directories): Remove hardlinks and
symlinks from snapshot dirs, and add them to compressed text files.

There's also scripts to reverse the above operations. Of course, it
takes a while to get a complete snapshot of a given server from a few
weeks back, but at least it's possible.

I'm thinking that this is a lot of work, and there must be better ways
of handling this kind of problem. I don't like reinventing the wheel.

Given the above, are there any suggestions? eg:

1) Another tool similar to rdiff-backup, which has easier-to-understand logs.
2) A quicker way of running DU for my use case (huge number of
hardlinks in directories)
3) Existing tools for managing hardlink-based snapshot directories

Thanks in advance.


Reply to: