progress, but... - re. fixing LVM/md snafu

[Miles Fidelman <mfidelman@traversetechnologies.com>]

Hello again Folks,

So.. I'm getting closer to fixing this messed up machine.

Where things stand:

I have root defined as an LVM2 LV, that should use /dev/md2 as it's PV.
/dev/md2 in turn is a RAID1 array built from /dev/sda3 /dev/sdb3 and 
/dev/sdc3

Instead, LVM is reporting: "Found duplicate PV 
2ppSS2q0kO3t0tuf8t6S19qY3ypWBOxF: using /dev/sdb3 not /dev/sda3"
and the /dev/md2 is reporting itself as inactive (cat /proc/mdstat) and 
active,degraded (mdadm --detail)

---
I'm guessing that, during boot:

- the raid array failed to start
- LVM found both copies of the PV, and picked one (/dev/sdb3)
- everything then came up and my server is humming away

but: the md array can't rebuild because the most current device in it is 
already in use

so...  I'm looking for the right sequence of events, with the minimum 
downtime to:

1. stop changes to /dev/sdb3 (actually, to / - which complicates things)
2. rebuild the RAID1 array, making sure to use /dev/sdb3 as the starting 
point for current data
3. restart in such a way that LVM finds /dev/md2 as the right PVM 
instead of one of its components

Each of these is just tricky enough that I'm sure there are lots of 
gotchas to watch out for.

So.. any suggestions?

Thanks very much,

Miles Fidelman