Re: Very slow LVM performance
On Mon, Jul 12, 2010 at 22:28, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> I'm curious as to why you're (apparently) wasting 2/3 of your storage for
> redundancy. Have you considered a straight RAID 10 across those 30
> disks/LUNs?
This is a very good question. And the answer is: because Linux's MD
does not implement RAID10 the way we expected (as you have found out
for yourself). We started out thinking exactly that we'd have a
RAID10 stripe with cardinality of 3, instead of the multi-layered MD
design. But for us it's important to have full control over what
physical disks form the triplets (see below for discussion); instead,
MD's so-called RAID10 only guarantees that there will be exactly N
copies of each chunk on N different drives, but makes on promise as to
on *which* drives.
The reason the drive assignment is important to us is that we can
achieve more data redundancy if we form each triplet from an iSCSI
disk that lives on a different iSCSI target (host).
Suppose that you have six iSCSI target hosts h0 through h5, and each
of them has five disks d0 through d4. Then if you form the first
triplet as (h0:d0, h1:d0, h2:d0), and so forth until (h3:d4, h4:d4,
h5:d4), then if any iSCSI host goes down for whatever reason, then all
triplets still stay up and are still redundant, only running on two
copies instead of three.
Linux's RAID10 implementation did not allow us to do this. So we had
to layer by first creating RAID1 (or RAID10 with n=3) triplets, and
striping them in a higher layer.
> I'm also curious as to why you're running software RAID at all given the fact
> than pretty much every iSCSI target is itself an array controller with built
> in hardware RAID. Can you tell us a little bit about your iSCSI target devices?
Our boss wanted us to only use commodity hardware to build this
solution, so we don't employ any fancy RAID controllers - all drives
are connected to on-board SATA ports. Staying away from the "black
box" implementations as much as possible was also part of the wish
list.
After dealing with all the idiosyncrasies of iSCSI and software RAID
under Linux I am a bit skeptical whether what we are building is going
to actually be better than a black-box fiber-attached RAID solution,
but it surely is cheaper and more expandable.
--
Arcady Genkin
Reply to: