Re: Backup solutions without reinventig the wheel these days

--lnk-dest is a nice feature of rsync, however, it would require me to write yet another custom script that so many people already have and reinvent the wheel again. I'm trying to find a ready and time-tested and maintained solution before I go down that path. The rsynch handler in backupninja does pretty much the same thing, but uses cp -l as --link-dest did not exist back when it was written. Nevertheless, I'd prefer block-level deduplication and compression.

Your btrfs setup sounds interesting, Sarunas. Are you running those btrfs volumes on Jessie? Any stability issues? Do you think it could work with just 1GB RAM + 1 GB swap? I was thinking about doing out-of-band deduplication to conserve RAM. Do you use both compression and deduplication (file or block level)? I heard they don't play nicely together. Could you share your mount options?

Kind regards,

Ondřej Grover

On Tue, Oct 20, 2015 at 8:35 PM, Sarunas Burdulis <sarunas@math.dartmouth.edu> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/20/2015 12:57 PM, Ondřej Grover wrote:
> Hello,
>
> I'm looking for recommendations for backup solutions that don't reinvent
> the wheel and are reliable and used. I want to backup two servers to a
> backup server. The main data content is several hundred GB in many very
> small files.
>
> [...]
>
> I've also looked at the new kids on the block like obnam, attick and
> borgbackup. They look interesting, but I prefer time-tested SW for backups.
> After realizing that these new backup programs pretty much try to
> replicate features of btrfs or ZFS (incremental snapshots, block-level
> compression and deduplication) I started thinking that I could perhaps
> just send the data to the backup server via rsync and save them to a
> btrfs or ZFS (but the backup server may not have enough RAM for ZFS) and
> create daily snaphosts on the server. If memory will permit (if I
> optimize it), I'd go with ZFS as it should be more reliable. Does
> anybody use such a solution?

Yes, we do use pretty much the setup you describe, i.e. rsync push to
backup server, btrfs volume with daily snapshot subvolumes on the
server. This particular setup has been in use for more than two years by
now, and is working well with 8-10 servers (totals for all backed-up
filesystems is about 1.2TiB/2.5M files of varying size; transferred
daily delta is of course just a fraction of this). The backup server is
a rather modest self-built Mini-ITX Core i5 with 8GiB RAM and 6TiB
hardware (3ware) RAID-10 storage volume.

Using btrfs volume in btrfs-raid configuration (raid-1 for data and
metadata) would provide protection against data bit-rot. We use several
such volumes on some of our severs, though not in combination with btrfs
snapshot subvolumes.

- --
Sarunas Burdulis
http://math.dartmouth.edu/~sarunas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWJollAAoJEAcPjabqyGA9iD4H/2Hxa4IqYxRRLi55kg8c/tOS
UafxgMMnUB498HHVGMldDZdjqmzBd4djHuVOeJPguu7RaSVKOIBBdpmZdI44LXoj
y3P0m7Ig1cg3kGB7tZTgG6mzsgZQln6F8uW/lNox9EikmWUQ2eQZYIDVbLTEOLXt
pMKVTKQeR1Jdl9HvbIqlPBpniqHn+JvesWUeTDdo47i45jcfRonYfPay0/ua7ql0
n80ObcOLlYSCw/rKZbkspHz7k9PMEEdeosbkvUkw2ZiGq13E1j6DFwSUTxcrMPhR
Ffd+bL5c0CtKOTVrvtZdahJZ5U/Kl/QG1c78TsJagCN7YTSXRWnDSn6acdDRPmM=
=Edp3
-----END PGP SIGNATURE-----