Re: backup archive format saved to disk
Douglas Tutty wrote:
On Wed, Dec 06, 2006 at 09:02:37PM -0600, Reid Priedhorsky wrote:
No, you _should_ compress it and then use some of the space you saved to
add some carefully chosen redundancy which will allow you to reconstruct
everything, not just some things, in case of failure. (E.g., using par2.)
Scenario C: Compression plus redundancy
Suppose you have 100 megabytes of files, uncompressed. You create a tar
archive and compress it down to 75M. You then create 10M of redundancy
using (e.g.) par2, for a total of 85M. A failure occurs, and 2M of data
is lost. You use par2 to reconstruct the archive, and nothing is lost.
(You can do this regardless of whether data, redundancy, or both are
destroyed.) You are happy.
Hi Reid,
I've been looking at par2. The question remains how to apply it to data
stored on media where the potential failure is one of media not
transmittion. If I only protect the tar.bz2 file and a media failure
occurs, how could I have set up the par2 redundancy files to allow me to
recover the data.
Apparently, hard disks use FEC themselves so that they either can fix
the data or there is too much damage and the drive is inaccessible. It
The sector is unreadable. It is possible to command discs to do "long
reads" which return the data in the sector along with the check bits,
and possibly recover some of the data if one can recognize other
redundancy present in it. For example, ASCII text is often so redundant
that it can be reconstructed with even half of the data corrupted.
seems to be an all-or-nothing propositition. If someone has experience
of FEC drive failures that refutes this I'd be very interested.
I'm not sure that what I wrote above is a refutation of what you said.
The only disk failures I have experienced are on older drives without
Hmm? Discs have had FEC on them since 1982 at least (when I got involved
with them).
FEC that for a given sector return an error about bad CRC but one can
carry on and read the rest of the disk. It was from this perspective
that I proposed the question that led to this thread.
There are many kinds of disc failures. If you have a failing bearing,
then you may have speed variations that the data resynchronizer cannot
ovecome. If this happens, you may have sporadic ability to read any
given sector, and if you try often enough, you may get everything off
the disc.
If you have a medium failure, then you may have weak or migrating bits
which are sometimes readable without errror, and sometimes not. Or you
may have a hard error. If the FEC can correct it, then the data get
recovered, and the sector gets remapped. This happens without any
notification, unless you ask the disc for its information.
If the power on the disc fails, then the disc is unreadable in any
fashion at all.
If drives are atomic in this way, it seems that the only way to achieve
redundancy is through multiple copies (either manually done or via
raid1).
Well, that's not quite the case. But if you want to protect against
single point failures, like the uC on the disc goes South, or the
power supply on it takes a nosedive, then the only way to do that
is via redundancy.
I'm still hoping that someone who knows how linux software raid work can
tell me how it decides that a drive has failed. This question was posed
in a thread about raid1 internals.
Your best bet may be to get the code and read it.
Mike
--
p="p=%c%s%c;main(){printf(p,34,p,34);}";main(){printf(p,34,p,34);}
This message made from 100% recycled bits.
You have found the bank of Larn.
I can explain it for you, but I can't understand it for you.
I speak only for myself, and I am unanimous in that!
Reply to: