Re: Backup ideas

[Andy Smith <andy@strugglers.net>]

Hello,

On Tue, Apr 28, 2020 at 11:43:22AM -0400, Default User wrote:
> Andy, you mentioned restic, which I am not familiar with.  Similar
> considerations would seem to apply to that also.  But I might also try
> that out later.

Note that you did not state any of these requirements for only using
tools you are familiar with, so we cannot really be blamed for
suggesting things just because we think they are good. You didn't
really state many requirements at all, only asked for pointers as to
what makes a good backup system, so only generalities were possible.
I did say there are many many different solutions.

> 4 - So I am still using rsync.  I back up a single system to a local
> external usb drive.  That drive is then backed up to another local
> external usb drive. So I do have some redundancy.  I could go even
> further, backing up to a third drive, and using the --checksum option.
> But that might be excessive, and I do have a life to live.

If you wish to continue using rsync then I would suggest looking at
rsnapshot. It uses rsync. It provides for incremental backups. It
stores things as the actual files which makes them easy to
investigate and restore.

When you come to features like deduplication and encryption either
the remote filesystem has to support it, or else the backup tool has
to support it. That is why so many of these solutions store data in
big blocks in their own format, so they can provide those features.

rsnapshot gives you some limited form of deduplication by hard
linking identical versions of each file together. It is far from
perfect but is better than no deduplication.

> Andy, you suggested having off-site backups.  You are correct that
> local-only backups do not protect against things like fire or theft.
> 
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.

The thing about cloud storage is that it is generic and it is easy.
Once you come up with a means to backup to an S3-compatible storage,
should your data disappear you can just point it at another
S3-compatible storage. Amazon's S3 was of course the original but
there are many competitors to Amazon now that use the same protocol.

Anyone using cloud ("someone else's computer") services needs to
think about what they will do if/when the cloud ("someone else's
computer") goes away. The usual answer is "switch everything over to
another cloud".

So the concern when using such a service usually isn't whether they
are going to go out of business, but more often about keeping
control of costs. At some point there may be a time where it's
cheaper to do yourself, or scale up the amount of responsibility you
take. For example, if you require many terabytes of storage then it
will be cheaper for you to rent a whole server from someone instead.
That server will almost certainly have a lower reliability than the
cloud storage vendor, but if it's only for backups then maybe that
is not as important to you as the cost saving.

I would not suggest having "the cloud" as one's ONLY backup of
important data. Multiple different clouds may be safe enough, though
the advantage of having a copy of the data local is large.

> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.

Any organisation that wants to spend that amount of resources
accessing your data has already been in your home and cloned your
storage while you were out/asleep.

> I'm not sure my threat model really justifies off-site backups at this point.

Fair enough. Fire and theft seem pretty common to me, but it's your
call.

> 5 - You also mentioned encrypted backups.  Of course, you have to to
> trust the encryption methodology.  And it should not be such a hassle
> that you don't actually use it.

It is perfectly feasible to have it be entirely automated. I would
not recommend any kind of backup that's not automated!

> Finally, after well over seven hours into rsyncing (with no end in
> sight) from the first external usb drive to the second one (both are
> HDD), I am beginning to wonder if that is a good idea.  Those first
> full backups always take forever . . .

…and USB is a fairly awful way to use storage beyond personal
gadgets (camera, phone, etc)…

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting