Re: backup archive format saved to disk

[Douglas Tutty <dtutty@porchlight.ca>]

On Wed, Dec 06, 2006 at 09:02:37PM -0600, Reid Priedhorsky wrote:
 
> No, you _should_ compress it and then use some of the space you saved to
> add some carefully chosen redundancy which will allow you to reconstruct
> everything, not just some things, in case of failure. (E.g., using par2.)
 
> Scenario C: Compression plus redundancy
> 
>   Suppose you have 100 megabytes of files, uncompressed. You create a tar
>   archive and compress it down to 75M. You then create 10M of redundancy
>   using (e.g.) par2, for a total of 85M. A failure occurs, and 2M of data
>   is lost. You use par2 to reconstruct the archive, and nothing is lost.
>   (You can do this regardless of whether data, redundancy, or both are
>   destroyed.) You are happy.
> 

Hi Reid,

I've been looking at par2.  The question remains how to apply it to data
stored on media where the potential failure is one of media not
transmittion.  If I only protect the tar.bz2 file and a media failure
occurs, how could I have set up the par2 redundancy files to allow me to
recover the data.

Apparently, hard disks use FEC themselves so that they either can fix
the data or there is too much damage and the drive is inaccessible.  It
seems to be an all-or-nothing propositition.  If someone has experience
of FEC drive failures that refutes this I'd be very interested.

The only disk failures I have experienced are on older drives without
FEC that for a given sector return an error about bad CRC but one can
carry on and read the rest of the disk.  It was from this perspective
that I proposed the question that led to this thread.

If drives are atomic in this way, it seems that the only way to achieve
redundancy is through multiple copies (either manually done or via
raid1).

I'm still hoping that someone who knows how linux software raid work can
tell me how it decides that a drive has failed.  This question was posed
in a thread about raid1 internals.

Thanks,

Doug.