Re: RAID-1 and disk I/O
David Christensen <dpchrist@holgerdanske.com> writes:
> You should consider upgrading to Debian 10 -- more people run that and
> you will get better support.
It's on my TODO list. As well as upgrading the very old hardware.
Currently, it's a Gigabyte P35-DS3L with an Intel Core2Duo E8400 CPU
and 8 GB RAM. It's only my private home server and performance is
still sufficient but I hope to reduce power consumption considerably.
> > the storage setup is as follows:
> > Two identical SATA disks with 1 partition on each drive spanning the
> > whole drive, i.e. /dev/sda1 and /dev/sdb1. Then, /dev/sda1 and
> > /dev/sdb1 form a RAID-1 /dev/md0 with LVM on top of it.
>
>
> ext4? That lacks integrity checking.
>
>
> btrfs? That has integrity checking, but requires periodic balancing.
Mostly ext4 for / /var /var/spool/news /usr /usr/local and /home file
systems. The /usr/src file system is btrfs and some test file systems
also. There are also 4 VMs, FreeBSD and NetBSD with their partitions
and slices and ufs file systems, one Linux VM with ext4 and one very
old Linux VM (kernel 2.4) with its own LVM in two LVs and 10 ext3 file
systems.
> Are both your operating system and your data on this array? I always
> use a single, small solid-state device for the system drive, configure
> my hardware so that it is /dev/sda, and use separate drive(s) for data
> (/dev/sdb, /dev/sdc, etc.). Separating these concerns simplifies
> system administration and disaster preparedness/ recovery.
Yes, everything is in the LVs on /dev/md0. Except for some external
USB hard drives for backup (4 TB) and some other seldomly used stuff
(e.g. NTFS drive with some old data of my wife's laptop, I cannot
persuade her to use Linux).
> > but I found the following with
> > smartctl:
> > ------------------------------------------------------------------------------
> > # diff -U20 <(smartctl -x /dev/sda) <(smartctl -x /dev/sdb)
>
>
> Why limit unified context to 20 lines? You may be missing information
> (I have not counted the differences, below). I suggest '-U' alone.
20 lines are just enough to get all. You can see this because there
are less than 20 context lines at the beginning and end of the diff
and only one hunk. GNU diff doesn't allow -U without a line count.
> You have a SATA transfer speed mismatch -- 6.0 Gbps drives running at
> 3.0 Gbps. If your ports are 3 Gbps, fine. If your ports are 6 Gbps,
> you have bad ports, cables, racks, docks, trays, etc..
Yes, the old Gigabyte mainboard has only 3 Gbps ports. I wasn't aware
of this but have just looked up the specs.
> Seek_Error_Rate indicates those drives have seen better days, but are
> doing their job.
>
>
> Power_On_Hours indicates those drives have seen lots of use.
> Power_Cycle_Count indicates that the machine runs 24x7 for long
> periods without rebooting.
Yes, the server runs 24/7 except for kernel updates, and a power
outage 2 weeks ago (my UPS batteries also need replacement... )-:
And the server is about 8 years old, initially with only 1 hard drive
which crashed while my backup was too small to hold everything. This
meant a lot of work (and quite some money) to get everything running
again and to recover data which wasn't in the backup.
This was almost 6 years ago and I then bought 2 Seagate Barracuda
drives for RAID-1 and a larger backup drive. One of the two Seagate
drives is still running and is /dev/sda. The other drive /dev/sdb
crashed after only 9.5 months of operation and I got it replaced by
the dealer. This was when I loved my decision to setup RAID-1. With
no downtime I pulled the failed drive, returned it to the dealer, ran
the system a week or two with only one drive, got the replacement
drive from the dealer hot-plugged it in, synced, and was happy :-)
Only short time after this I also bought a 3.5" removable mounting
frame for 2 drives to swap drives even more easily.
> Runtime_Bad_Block looks acceptable.
> End-to-End_Error and Reported_Uncorrect look perfect. The drives
> should not have corrupted or lost any data (other hardware and/or
> events may have).
OK.
> Airflow_Temperature_Cel and Temperature_Celsius are higher than I
> like. I suggest that you dress cables, add fans, etc., to improve
> cooling.
OK, I'll have a look at that.
> UDMA_CRC_Error_Count for /dev/sda looks worrisome, both compared to
> /dev/sdb and compared to reports for my drives.
>
>
> Total_LBAs_Written for /dev/sda is almost double that of
> /dev/sdb. Where those drives both new when put into RAID1?
Yes, see above. But /dev/sdb was replaced after 9.5 months, so it has
shorter life-time. Also, /dev/sda began to fail every couple of
months about a year ago. I could always fix this by pulling the
drive, re-inserting and re-syncing it. This also caused more
write-traffic to /dev/sda.
> > SMART Extended Self-test Log Version: 1 (1 sectors)
> > Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
> > -# 1 Short offline Completed without error 00% 21808 -
> > +# 1 Short offline Completed without error 00% 14254 -
>
>
> LifeTime for /dev/sda is ~50% higher than /dev/sdb. So, those drives
> were not both new when put into RAID1?
As David has clarified, these are the times of self-tests and current
life times are 51289 hours = 5.9 years and 43740 hours = 5.0 years,
resp.
I have now run a short and a long self-test on both drives and both
drives seem to have no problems:
root@bit:~# smartctl -l selftest /dev/sda ; smartctl -l selftest /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.13.1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 51316 -
# 2 Short offline Completed without error 00% 51312 -
# 3 Short offline Completed without error 00% 21808 -
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.13.1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 43767 -
# 2 Short offline Completed without error 00% 43764 -
# 3 Short offline Completed without error 00% 14254 -
root@bit:~# smartctl -l error /dev/sda ; smartctl -l error /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.13.1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.13.1] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged
> The smartctl(8) RAW_VALUE column is tough to read. Sometimes it looks
> like an integer. Other times, it looks like a bitmap or big-endian/
> little-endian mix-up. The VALUE column is easier. Both 119 and 117
> are greater than 100, so I would not worry.
Hm, in some cases the RAW_VALUE looked somehow "more readable, and the
VALUE looked suspicous to me. And here I found the explanation in the
smartctl(8) man page:
Each Attribute has a "Raw" value, printed under the heading
"RAW_VALUE", and a "Normalized" value printed under the
heading "VALUE".
[...]
Each vendor uses their own algorithm to convert this "Raw"
value to a "Normalized" value in the range from 1 to 254.
[...]
So to summarize: the Raw Attribute values are the ones that
might have a real physical interpretation, such as
"Temperature Celsius", "Hours", or "Start-Stop Cycles".
> I replaced all of my SATA I/II/III cables in all of my computers about
> a year ago with Cable Matters black 6G cables with locking straight
> and/or 90 degree connectors. Life got much better.
I think I should replace my cables too, probably not the cheapest ons
I can find.
And I'll also run self-tests and store and keep the results somewhere
more regularly.
Thanks for all your answers, hints, suggestions. With that, and
reading the man page more carefully (mostly motivate by your and
other's answers) I learned quite a lot new about SMART and how to
use/read it.
urs
Reply to: