mdadm /dev/md1 -a /dev/sdf2 results in a rebuild of md1. that stops while dmesg shows:
[15454.971105] md: bind<sdf2>
[15455.019843] RAID1 conf printout:
[15455.019846] --- wd:1 rd:2
[15455.019847] disk 0, wo:0, o:1, dev:sde2
[15455.019849] disk 1, wo:1, o:1, dev:sdf2
[15455.019911] md: recovery of RAID array md1
[15455.019913] md: minimum _guaranteed_ speed: 500000 KB/sec/disk.
[15455.019915] md: using maximum available idle IO bandwidth (but not more than 800000 KB/sec) for recovery.
[15455.019918] md: using 128k window, over a total of 976655488 blocks.
[15492.547520] svc: failed to register lockdv1 RPC service (errno 97).
[16092.596235] svc: failed to register lockdv1 RPC service (errno 97).
[16693.457632] svc: failed to register lockdv1 RPC service (errno 97).
[17296.691140] svc: failed to register lockdv1 RPC service (errno 97).
[17444.763190] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17444.763200] ata11.00: irq_stat 0x40000008
[17444.763207] ata11.00: failed command: READ FPDMA QUEUED
[17444.763216] ata11.00: cmd 60/00:00:4d:80:d9/04:00:09:00:00/40 tag 0 ncq 524288 in
[17444.763217] res 41/40:00:03:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17444.763234] ata11.00: status: { DRDY ERR }
[17444.763240] ata11.00: error: { UNC }
[17444.767842] ata11.00: configured for UDMA/133
[17444.767855] ata11: EH complete
[17447.367615] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17447.367627] ata11.00: irq_stat 0x40000008
[17447.367634] ata11.00: failed command: READ FPDMA QUEUED
[17447.367643] ata11.00: cmd 60/00:10:4d:80:d9/04:00:09:00:00/40 tag 2 ncq 524288 in
[17447.367644] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17447.367662] ata11.00: status: { DRDY ERR }
[17447.367667] ata11.00: error: { UNC }
[17447.372640] ata11.00: configured for UDMA/133
[17447.372652] ata11: EH complete
[17449.288020] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17449.288030] ata11.00: irq_stat 0x40000008
[17449.288037] ata11.00: failed command: READ FPDMA QUEUED
[17449.288045] ata11.00: cmd 60/00:00:4d:80:d9/04:00:09:00:00/40 tag 0 ncq 524288 in
[17449.288046] res 41/40:00:03:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17449.290577] ata11.00: status: { DRDY ERR }
[17449.290584] ata11.00: error: { UNC }
[17449.295561] ata11.00: configured for UDMA/133
[17449.295575] ata11: EH complete
[17451.883179] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17451.883188] ata11.00: irq_stat 0x40000008
[17451.883195] ata11.00: failed command: READ FPDMA QUEUED
[17451.883204] ata11.00: cmd 60/00:10:4d:80:d9/04:00:09:00:00/40 tag 2 ncq 524288 in
[17451.883205] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17451.883229] ata11.00: status: { DRDY ERR }
[17451.883235] ata11.00: error: { UNC }
[17451.888136] ata11.00: configured for UDMA/133
[17451.888147] ata11: EH complete
[17454.462913] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17454.462923] ata11.00: irq_stat 0x40000008
[17454.462930] ata11.00: failed command: READ FPDMA QUEUED
[17454.462939] ata11.00: cmd 60/00:00:4d:80:d9/04:00:09:00:00/40 tag 0 ncq 524288 in
[17454.462939] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17454.462959] ata11.00: status: { DRDY ERR }
[17454.462965] ata11.00: error: { UNC }
[17454.467508] ata11.00: configured for UDMA/133
[17454.467522] ata11: EH complete
[17456.379426] ata11.00: exception Emask 0x0 SAct 0x7 SErr 0x0 action 0x0
[17456.379436] ata11.00: irq_stat 0x40000008
[17456.379443] ata11.00: failed command: READ FPDMA QUEUED
[17456.379452] ata11.00: cmd 60/00:10:4d:80:d9/04:00:09:00:00/40 tag 2 ncq 524288 in
[17456.379452] res 41/40:00:03:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17456.379470] ata11.00: status: { DRDY ERR }
[17456.379476] ata11.00: error: { UNC }
[17456.384416] ata11.00: configured for UDMA/133
[17456.384468] sd 10:0:0:0: [sde] Unhandled sense code
[17456.384470] sd 10:0:0:0: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[17456.384472] sd 10:0:0:0: [sde] Sense Key : Medium Error [current] [descriptor]
[17456.384475] Descriptor sense data with sense descriptors (in hex):
[17456.384476] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[17456.384480] 09 d9 84 03
[17456.384482] sd 10:0:0:0: [sde] Add. Sense: Unrecovered read error - auto reallocate failed
[17456.384485] sd 10:0:0:0: [sde] CDB: Read(10): 28 00 09 d9 80 4d 00 04 00 00
[17456.384489] end_request: I/O error, dev sde, sector 165250051
[17456.384517] ata11: EH complete
[17458.309172] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17458.309182] ata11.00: irq_stat 0x40000008
[17458.309189] ata11.00: failed command: READ FPDMA QUEUED
[17458.309198] ata11.00: cmd 60/08:00:fd:83:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17458.309199] res 41/40:00:03:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17458.309216] ata11.00: status: { DRDY ERR }
[17458.309222] ata11.00: error: { UNC }
[17458.314542] ata11.00: configured for UDMA/133
[17458.314552] ata11: EH complete
[17460.217302] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17460.217312] ata11.00: irq_stat 0x40000008
[17460.217318] ata11.00: failed command: READ FPDMA QUEUED
[17460.217327] ata11.00: cmd 60/08:00:fd:83:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17460.217328] res 41/40:00:03:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17460.217345] ata11.00: status: { DRDY ERR }
[17460.217351] ata11.00: error: { UNC }
[17460.222238] ata11.00: configured for UDMA/133
[17460.222248] ata11: EH complete
[17462.161310] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17462.161319] ata11.00: irq_stat 0x40000008
[17462.161326] ata11.00: failed command: READ FPDMA QUEUED
[17462.161335] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17462.161336] res 41/40:00:06:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17462.161353] ata11.00: status: { DRDY ERR }
[17462.161359] ata11.00: error: { UNC }
[17462.166396] ata11.00: configured for UDMA/133
[17462.166406] ata11: EH complete
[17464.417353] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17464.417363] ata11.00: irq_stat 0x40000008
[17464.417370] ata11.00: failed command: READ FPDMA QUEUED
[17464.417379] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17464.417380] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17464.417397] ata11.00: status: { DRDY ERR }
[17464.417403] ata11.00: error: { UNC }
[17464.422434] ata11.00: configured for UDMA/133
[17464.422442] ata11: EH complete
[17466.453444] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17466.453453] ata11.00: irq_stat 0x40000008
[17466.453460] ata11.00: failed command: READ FPDMA QUEUED
[17466.453469] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17466.453470] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17466.453487] ata11.00: status: { DRDY ERR }
[17466.453493] ata11.00: error: { UNC }
[17466.457832] ata11.00: configured for UDMA/133
[17466.457843] ata11: EH complete
[17468.361305] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17468.361315] ata11.00: irq_stat 0x40000008
[17468.361322] ata11.00: failed command: READ FPDMA QUEUED
[17468.361331] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17468.361332] res 41/40:00:06:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17468.361349] ata11.00: status: { DRDY ERR }
[17468.361355] ata11.00: error: { UNC }
[17468.365684] ata11.00: configured for UDMA/133
[17468.365691] ata11: EH complete
[17470.637379] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17470.637388] ata11.00: irq_stat 0x40000008
[17470.637395] ata11.00: failed command: READ FPDMA QUEUED
[17470.637404] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17470.637405] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17470.637423] ata11.00: status: { DRDY ERR }
[17470.637428] ata11.00: error: { UNC }
[17470.642626] ata11.00: configured for UDMA/133
[17470.642635] ata11: EH complete
[17472.945322] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[17472.945331] ata11.00: irq_stat 0x40000008
[17472.945338] ata11.00: failed command: READ FPDMA QUEUED
[17472.945347] ata11.00: cmd 60/08:00:05:84:d9/00:00:09:00:00/40 tag 0 ncq 4096 in
[17472.945348] res 41/40:00:07:84:d9/00:00:09:00:00/40 Emask 0x409 (media error) <F>
[17472.945365] ata11.00: status: { DRDY ERR }
[17472.945371] ata11.00: error: { UNC }
[17472.949708] ata11.00: configured for UDMA/133
[17472.949717] sd 10:0:0:0: [sde] Unhandled sense code
[17472.949719] sd 10:0:0:0: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[17472.949721] sd 10:0:0:0: [sde] Sense Key : Medium Error [current] [descriptor]
[17472.949723] Descriptor sense data with sense descriptors (in hex):
[17472.949724] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[17472.949729] 09 d9 84 07
[17472.949730] sd 10:0:0:0: [sde] Add. Sense: Unrecovered read error - auto reallocate failed
[17472.949734] sd 10:0:0:0: [sde] CDB: Read(10): 28 00 09 d9 84 05 00 00 08 00
[17472.949738] end_request: I/O error, dev sde, sector 165250055
[17472.949754] ata11: EH complete
[17472.949757] raid1: sde: unrecoverable I/O read error for block 165041152
[17472.956352] md: md1: recovery done.
[17478.467735] RAID1 conf printout:
[17478.467737] --- wd:1 rd:2
[17478.467739] disk 0, wo:0, o:1, dev:sde2
[17478.467741] disk 1, wo:1, o:1, dev:sdf2
[17478.481529] RAID1 conf printout:
[17478.481533] --- wd:1 rd:2
[17478.481535] disk 0, wo:0, o:1, dev:sde2
[17896.660112] svc: failed to register lockdv1 RPC service (errno 97).
[18496.966137] svc: failed to register lockdv1 RPC service (errno 97).
[18569.140016] ata11.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[18569.140026] ata11.00: irq_stat 0x40000001
[18569.140033] ata11.00: failed command: READ SECTOR(S)
[18569.140042] ata11.00: cmd 20/00:01:03:84:d9/00:00:00:00:00/e9 tag 0 pio 512 in
[18569.140043] res 51/40:01:03:84:d9/00:00:09:00:00/e9 Emask 0x9 (media error)
[18569.140060] ata11.00: status: { DRDY ERR }
[18569.140066] ata11.00: error: { UNC }
[18570.917046] ata11.00: configured for UDMA/133
[18570.917066] ata11: EH complete
root@bigpeat:~# hdparm --read-sector 165250051 /dev/sde
/dev/sde:
reading sector 165250051: FAILED: Input/output error
root@bigpeat:~# hdparm --write-sector 165250051 /dev/sde
/dev/sde:
Use of --write-sector is VERY DANGEROUS.
You are trying to deliberately overwrite a low-level sector on the media.
This is a BAD idea, and can easily result in total data loss.
Please supply the --yes-i-know-what-i-am-doing flag if you really want this.
Program aborted.
root@bigpeat:~# hdparm --write-sector 165250051 --yes-i-know-what-i-am-doing /dev/sde
/dev/sde:
re-writing sector 165250051: succeeded
root@bigpeat:~# hdparm --read-sector 165250051 /dev/sde
/dev/sde:
reading sector 165250051: succeeded
0000 0000 0000 0000 0000 0000 0000 0000
<snap> some more of those lines
then:
mdadm /dev/md1 --fail /dev/sdf2
mdadm /dev/md1 --remove /dev/sdf2
mdadm /dev/md1 --add /dev/sdf2
leads strait to the same error.