Re: mdadm doing strange things

[Andrew Reid <reidac@bellatlantic.net>]

On Saturday 19 June 2010 14:20:27 Alan Chandler wrote:

[ Details elided ]

> HOWEVER (the punch line).  When this system booted, it was not the old
> reverted one but how it was before I started this cycle. In other words
> it looked as though the disk which I had failed and removed was being used
>
> If I did mdadm --detail /dev/md1 (or any of the other devices) it shows
> /dev/sdb as the only device on the raid pair.  To sync up again I am
> having to add in the various /dev/sda partitions.
>
> SO THE QUESTION IS.  What went wrong.  How does a failed device end up
> being used to build the operational arrays, and the other devices end up
> not being included.

  My understanding of how mdadm re-arranges the array (including for
failures, etc.) is that it writes metadata into the various partitions,
so I agree with you that this is weird -- I would have expected the
RAID array to come up with the sda devices as the only devices present.

  There are two things I can think of, neither quite right, but maybe
they'll motivate someone else to figure it out:

 (1) Device naming can be tricky when you're unplugging drives.  
Maybe the devices now showing up as "sdb" actually are the original
"sda" devices.  Can you check UUIDs?  This explanation also requires
that you didn't actually revert the disk, you only thought you did,
but then didn't catch it because the conjectural device-renaming 
convinced you that the RAID was being weird.

 (2) How did you revert the root partition?  If you copied all the 
files, then I have nothing else to add.  If you did "dd" between the
partitions, however, you may have creamed the md metadata, and caused
the system to think the sdb device was the "good" one.  This explanation
is unsatisfactory because, even if it's right, it only explains why
that partition should be reversed, not the others, although if you didn't
revert the others, they're copies, and you can't tell them apart anyways.

  Also, what happened to /etc/mdadm/mdadm.conf on the reverted root
partition?  Is it nonexistent on the one you're now booting from?  
There's potential for confusion there also, although I think the 
initramfs info will suffice until the next kernel update.

				-- A.
-- 
Andrew Reid / reidac@bellatlantic.net