Re: RAID1 problem - server freezes on md data-check
Hello Ross,
> > Ross Halliday wrote:
> > > Aside from any bugs that checkarray
> > > function is definitely a pain on a production system.
> >
> > Well, it's even more a pain to have no monthly check at all, and have
> > your drive silently die without a warning. [...]
[...]
> I have to seriously question the value of this once-a-month check as the
> other 27-30 days of the month your disk could be half-dead, spewing
> corrupt data and you'd never know until it was Sunday at 1:06 AM. It
> seems like a sort of after-thought hack that renders your disks unusable
> for a few hours. It would make more sense if the check was run nightly,
> but you would probably see a lot of upset people complaining that
> backups take forever and after-hours performance would take a massive
> hit. Perhaps there is some other way to check data integrity more than
> approximately once per full moon that doesn't destroy I/O performance?
Since the re-check is comparable to (identical to?) the re-sync that's done
after a spare has been added, you ought to be able to control the I/O rate
using the usual /proc/sys/dev/raid/speed_limit_{min,max}-controls. I, however,
have never seen those controls actually have any effect.
The re-check is however a guard against silent failures, in that it guarantees
that all logical chunks have been touched upon, on all member disks. Had it not
been there, some server somewhere might have that the i'th block on the j'th
participant disk has somehow become unreadable---except this isn't known yet,
because nobody has accessed it for years.
When the k'th (for j!=k) participant disk experiences e.g. a head crash, the
recombination algorithm (be it raid1, raid5, whatever) cannot reconstruct the
logical data that has come to depend on the i'th block in the j'th disk. The
result is that silent failure combines with a non-silent failure, and there
isn't enough information available to reconstruct the stored data. Data loss
results.
As a sidenote, the sort of corruption we're talking about here is something
that the disk reports (as opposed to some checksum that doesn't add up in the
md layer). So the same purpose as the first-Sunday-of-month check could be
fulfilled by issuing the command to the disk to do a self-check:
mdadm --detail /dev/md0 | egrep -o "/dev/sd[a-z]+" | sort -u | \
while read disk; do smartctl -t long $disk; done
This should not cause I/O death, since no data is ever communicated over the
I/O channel. But the disk itself will see that its CRC's don't add up, and
you'd be getting mails from smartd. Of course, the md layer won't take notice.
Regards, skrewz.
Reply to: