Re: raid1 issue, somewhat related to recent "debian on big machines"
It came to mind how to check the raid. Below the key tests. I am not
sure if the disk removed by the system (sda) has to be replaced with a
new one or it could be forced into place, tried, reformatted, and what
else should be better tried. If to be replaced, it would be the second
time in a less than 2000 hours service, while there is a good UPS and
the disks are mounted on Zahlman coolers with a strong fan directly on
them.
Unfortunately I have a badly planned huge raid. If trying to reuse
sda, is it correct to command directly:
# sfdisk -d /dev/sdb > /tmp/partitions
# sfdisk --force /dev/sda < /tmp/partitions
# mdadm -add /dev/md0 /dev/sdb1
# mdadm -add /dev/md1 /dev/sdb2
# mdadm -add /dev/md2 /dev/sdb3
# mdadm -add /dev/md3 /dev/sdb5
# mdadm -add /dev/md4 /dev/sdb6
# mdadm -add /dev/md5 /dev/sdb7
# mdadm -add /dev/md6 /dev/sdb8
or should sda be first ext3 formatted?
=========================================================================
cat /proc/mdstat
Personalities : [raid1]
md6 : active raid1 sdb8[1]
102341952 blocks [2/1] [_U]
md5 : active raid1 sdb7[1]
1951744 blocks [2/1] [_U]
md4 : active raid1 sdb6[1]
2931712 blocks [2/1] [_U]
md3 : active raid1 sdb5[1]
14651136 blocks [2/1] [_U]
md1 : active (auto-read-only) raid1 sdb2[1]
6835584 blocks [2/1] [_U]
md0 : active raid1 sdb1[1]
2931712 blocks [2/1] [_U]
md2 : active raid1 sdb3[1]
14651200 blocks [2/1] [_U]
unused devices: <none>
========================================================================
mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Wed Sep 13 16:50:11 2006
Raid Level : raid1
Array Size : 2931712 (2.80 GiB 3.00 GB)
Used Dev Size : 2931712 (2.80 GiB 3.00 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Mar 3 19:05:21 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : 5afe70f3:fa1dcb8c:5cc73e84:1c5db3e0
Events : 0.912
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1
==================================================================
Similar situation with all other raids
==================================================================
smartctl --all /dev/sda (the disk removed):
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8
Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Raptor family
Device Model: WDC WD1500ADFD-00NLR1
Serial Number: WD-WMAP41185084
Firmware Version: 20.07P20
User Capacity: 150,039,945,216 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: ATA/ATAPI-7 published, ANSI INCITS 397-2005
Local Time is: Tue Mar 3 19:41:15 2009 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (4783) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 72) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0007 168 166 021 Pre-fail
Always - 4625
4 Start_Stop_Count 0x0032 099 099 040 Old_age
Always - 1031
5 Reallocated_Sector_Ct 0x0033 189 189 140 Pre-fail
Always - 168
7 Seek_Error_Rate 0x000a 200 200 051 Old_age
Always - 0
9 Power_On_Hours 0x0032 091 091 000 Old_age
Always - 7259
10 Spin_Retry_Count 0x0012 100 100 051 Old_age
Always - 0
11 Calibration_Retry_Count 0x0012 100 100 051 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age
Always - 1031
194 Temperature_Celsius 0x0022 123 106 000 Old_age
Always - 24
196 Reallocated_Event_Count 0x0032 180 180 000 Old_age
Always - 20
197 Current_Pending_Sector 0x0012 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0012 200 200 000 Old_age
Always - 0
199 UDMA_CRC_Error_Count 0x000a 200 253 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 051 Old_age
Offline - 0
SMART Error Log Version: 1
ATA Error Count: 12 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 12 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
Error 11 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
Error 10 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
Error 9 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
Error 8 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
ec 00 08 4a 9d 9b 00 00 00:52:36.600 IDENTIFY DEVICE
c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
c8 00 08 3f 9d 9b 4c 00 00:52:36.600 READ DMA
c8 00 08 37 9d 9b 4c 00 00:52:36.600 READ DMA
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
============================================================
sdb reported no errors
============================================================
thanks
francesco pietra
On Tue, Mar 3, 2009 at 5:02 PM, Francesco Pietra <chiendarret@gmail.com> wrote:
> How to run selftest with raid1?
>
> smartctl -t /dev/sda (or sda1, sda2, ..)
>
> is incorrect. I don't remember what should first be done with mdadm.
>
> I would like to run the test because
>
> smartctl --all /dev/sd#
>
> report errors for only sda, as follows:
>
> SMART Error Log Version: 1
> ATA Error Count: 12 (device log contains only the most recent five errors)
> CR = Command Register [HEX]
> FR = Features Register [HEX]
> SC = Sector Count Register [HEX]
> SN = Sector Number Register [HEX]
> CL = Cylinder Low Register [HEX]
> CH = Cylinder High Register [HEX]
> DH = Device/Head Register [HEX]
> DC = Device Command Register [HEX]
> ER = Error register [HEX]
> ST = Status register [HEX]
> Powered_Up_Time is measured from power on, and printed as
> DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
> SS=sec, and sss=millisec. It "wraps" after 49.710 days.
>
> Error 12 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
> When the command that caused the error occurred, the device was
> active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:49.600 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:49.600 READ DMA
>
> Error 11 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
> When the command that caused the error occurred, the device was
> active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:46.400 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:46.400 READ DMA
>
> Error 10 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
> When the command that caused the error occurred, the device was
> active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:42.950 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:42.950 READ DMA
>
> Error 9 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
> When the command that caused the error occurred, the device was
> active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:39.800 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:39.800 READ DMA
>
> Error 8 occurred at disk power-on lifetime: 1940 hours (80 days + 20 hours)
> When the command that caused the error occurred, the device was
> active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 08 4a 9d 9b ec Error: UNC 8 sectors at LBA = 0x0c9b9d4a = 211524938
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
> ec 00 08 4a 9d 9b 00 00 00:52:36.600 IDENTIFY DEVICE
> c8 00 08 47 9d 9b 4c 00 00:52:36.600 READ DMA
> c8 00 08 3f 9d 9b 4c 00 00:52:36.600 READ DMA
> c8 00 08 37 9d 9b 4c 00 00:52:36.600 READ DMA
>
> SMART Self-test log structure revision number 1
> No self-tests have been logged. [To run self-tests, use: smartctl -t]
> =======
>
> I.e., I am still uncertain if sda has to be replaced with a new disk,
> or the errors reported were temporary and have been removed by raid1.
>
> thanks
>
> francesco
==============================================================================================
===============================================================================================>
>
> ---------- Forwarded message ----------
> From: Francesco Pietra <chiendarret@gmail.com>
> Date: Tue, Mar 3, 2009 at 11:21 AM
> Subject: Re: raid1 issue, somewhat related to recent "debian on big machines"
> To: Ron Johnson <ron.l.johnson@cox.net>
>
>
> On Tue, Mar 3, 2009 at 10:08 AM, Ron Johnson <ron.l.johnson@cox.net> wrote:
>> On 03/03/2009 02:53 AM, Francesco Pietra wrote:
>>>
>>> lupus in fabula as a follow up of my short intervention on raid1 with
>>> my machine to the thread "Debian on big systems".
>>>
>>> System: supermicro H8QC8 m.board, two WD Raptor SATA 150GB, Debian
>>> amd64 lenny, raid1
>>>
>>> While running an electronic molecular calculation - estimated to four
>>> days time - I noticed by chance on the screen (what is not in the out
>>> file of the calculation) that there was a disk problem. I took some
>>> scattered notes from the scree:
>>>
>>> RAID1 conf printout
>>>
>>> wd: 1 rd:2
>>
>> [snip]
>>
>> What you are looking for should be in syslog, not your application's log.
>
>
> OK, but /var/log/syslog
>
> tels nothing more that I took notice about from the screen: sda sector
> 0 problematic, disk failure, continuing on one disk. My question is,
> what does the lshw -disk output mean (SCSI vs SATA, as I have shown),
> and if one disk has to be replaced with a new one. If so, to identify
> which is which can I detach the SATA connection to the disks and see
> which one works?
>
> thanks
> francesco
>
>
>>
>> --
>> Ron Johnson, Jr.
>> Jefferson LA USA
>>
>> The feeling of disgust at seeing a human female in a Relationship
>> with a chimp male is Homininphobia, and you should be ashamed of
>> yourself.
>>
>>
>> --
>> To UNSUBSCRIBE, email to debian-amd64-REQUEST@lists.debian.org
>> with a subject of "unsubscribe". Trouble? Contact
>> listmaster@lists.debian.org
>>
>>
>
Reply to: