Re: Mdadm drive fail power cord issue.

On Sat, Jun 15, 2013 at 1:07 AM, Bob Proulx <bob@proulx.com> wrote:

Muhammad Yousuf Khan wrote:
> one of our server's drive failed due to power cord issue.
> however i plug it back and due to less experience with "parted" i messed
> the whole thing.

If it was a power cord issue, and you plugged it back in, then there
should be no need to use parted. Simply mdadm re-add the drive to the
raid array. If the drive was running correctly before then the
partitions on the drive would have been correct. There should be no
need to make any changes to the partition table.

> i have 2tbx2 RAID 1 mirror

Nice box.

Thanks :)

> and have 4x500GB partitions.

And understand that each of those partitions are set up as RAID 1 mirror.

> even i selected the right drive by "select command" and deleted the right
> partition but parted did something worst though, i take it as my own
> mistake as i am not that experience with parted i am usually using fdisk.

Why were you trying to delete, add, or modify any partition? You said
you had a power cord issue. Did you replace the disk with a different
disk? If you replaced the disk with a different disk then you will
need to clone the partition table. If you are using the same disk as
before then do not modify the partition table.

actually i did that by mistake :)

> now the problem part is i can not re attach the fail drive partition with
> RAID /dev/md[2,3,4] devices.
>
> Personalities : [raid1]
> md1 : active (auto-read-only) raid1 sdc1[0]
> 488147776 blocks super 1.2 [2/1] [U_]
>
> md4 : active raid1 sdb4[0]
> 488670072 blocks super 1.2 [2/1] [U_]
>
> md3 : active raid1 sdb3[0]
> 488279928 blocks super 1.2 [2/1] [U_]
>
> md2 : active raid1 sdb2[0]
> 488279928 blocks super 1.2 [2/1] [U_]

This shows two devices. It shows /dev/sdb and /dev/sdc. All four
raid partitions are operating in degraded mode using only one device.
Note that the sizes are not quite identical.

But that is only part of the data. You need to use mdadm to display
the data that you need. Please show the output of each of the following
commands:

mdadm --detail /dev/md1
mdadm --detail /dev/md2
mdadm --detail /dev/md3
mdadm --detail /dev/md4

root@nasbox:~# mdadm --detail /dev/md1
/dev/md1:
        Version : 1.2
Creation Time : Thu Jun 13 23:46:28 2013
     Raid Level : raid1
     Array Size : 488147776 (465.53 GiB 499.86 GB)
Used Dev Size : 488147776 (465.53 GiB 499.86 GB)
   Raid Devices : 2
Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Jun 13 23:46:28 2013
          State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       0        0        1      removed
root@nasbox:~#   mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
Creation Time : Wed May 9 20:19:59 2012
     Raid Level : raid1
     Array Size : 488279928 (465.66 GiB 500.00 GB)
Used Dev Size : 488279928 (465.66 GiB 500.00 GB)
   Raid Devices : 2
Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Fri Jun 21 17:20:34 2013
          State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

           Name : archive:2
           UUID : c417ee7a:ad1eedf5:73043caa:e15bceb3
         Events : 3320

    Number   Major   Minor   RaidDevice State
       0       8       18        0      active sync   /dev/sdb2
       1       0        0        1      removed
root@nasbox:~#   mdadm --detail /dev/md3
/dev/md3:
        Version : 1.2
Creation Time : Wed May 9 20:20:11 2012
     Raid Level : raid1
     Array Size : 488279928 (465.66 GiB 500.00 GB)
Used Dev Size : 488279928 (465.66 GiB 500.00 GB)
   Raid Devices : 2
Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Fri Jun 21 12:38:13 2013
          State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

           Name : archive:3
           UUID : 6e63959f:b2947908:46bbbe04:211639f6
         Events : 6206

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3
       1       0        0        1      removed
root@nasbox:~#   mdadm --detail /dev/md4
/dev/md4:
        Version : 1.2
Creation Time : Wed May 9 20:39:03 2012
     Raid Level : raid1
     Array Size : 488670072 (466.03 GiB 500.40 GB)
Used Dev Size : 488670072 (466.03 GiB 500.40 GB)
   Raid Devices : 2
Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Fri Jun 21 12:38:13 2013
          State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

           Name : archive:4
           UUID : cdb71a07:2a18cc4a:bf139b42:a819fab1
         Events : 3434

    Number   Major   Minor   RaidDevice State
       0       8       20        0      active sync   /dev/sdb4
       1       0        0        1      removed
root@nasbox:~#

The critical information is at the bottom of each of the output. On
my system it shows this for an example:

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1

my second line state "Removed" :(

You will need to know that information in order to proceed successfully.

I would like to see the output of:

mdadm --examine /dev/sdc2

root@nasbox:~# mdadm --examine /dev/sdc2
mdadm: No md superblock detected on /dev/sdc2.

I am hoping that would show something useful. I would also like to
see the exact sizes of the partitions. Please show the output of the
following which will show the exact sizes. I worry that sdc2 is not
large enough and is smaller than sdb2.

sfdisk -d /dev/sdb
sfdisk -d /dev/sdc

root@nasbox:~# sfdisk -d /dev/sdb

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util sfdisk doesn't support GPT. Use GNU Parted.

# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start=        1, size=3907029167, Id=ee
/dev/sdb2 : start=        0, size=        0, Id= 0
/dev/sdb3 : start=        0, size=        0, Id= 0
/dev/sdb4 : start=        0, size=        0, Id= 0
root@nasbox:~# sfdisk -d /dev/sdc
read: Input/output error

sfdisk: read error on /dev/sdc - cannot read sector 0
/dev/sdc: unrecognized partition table type
No partitions found
root@nasbox:~#

It is possible that the partition table on the disk and the operating
system's view of it are out of sync with each other. You may need to
poke at the OS and have it scan the disk.

partprobe /dev/sdc

Hope that helps.
Bob

Thanks bob for all your interest. :)