Re: Very slow LVM performance

[Arcady Genkin <agenkin@gmail.com>]

On Mon, Jul 12, 2010 at 14:54, Aaron Toponce <aaron.toponce@gmail.com> wrote:
> Can you provide the commands from start to finish when building the volume?
>
> fdisk ...
> mdadm ...
> pvcreate ...
> vgcreate ...
> lvcreate ...

Hi, Aaron, I already provided all of the above commands in earlier
messages (except for fdisk, since we are giving the entire disks to
MD, not partitions).  I'll repeat them here for your convenience:

Creating the ten 3-way RAID1 triplets - for N in 0 through 9:
mdadm --create /dev/mdN -v --raid-devices=3 --level=raid10 \
 --layout=n3 --metadata=0 --bitmap=internal --bitmap-chunk=2048 \
 --chunk=1024 /dev/sdX /dev/sdY /dev/sdZ

Then the big stripe:
mdadm --create /dev/md10 -v --raid-devices=10 --level=stripe \
 --metadata=1.0 --chunk=1024 /dev/md{0,5,1,6,2,7,3,8,4,9}

Then the LVM business:
pvcreate /dev/md10
vgcreate vg0 /dev/md10
lvcreate -l 102389 vg0

Note that the file system is not being created on top of LVM at this
point, and I ran the test by simply dd-ing /dev/vg0/lvol0.

> My experience has been that LVM will introduce about a 1-2% performance
> hit compared to not using it

This is what we were expecting, it's encouraging.

> On a side note, I've never seen any reason to increase or decrease the
> chunk size with software RAID. However, you may want to match your chunk
> size with '-c' for 'lvcreate'.

We have tested a variety of chunk sizes (from 64K to 4MB) with
bonnie++ and found that 1MB chunks worked the best for our usage,
which is a general purpose NFS server, so it's mainly small random
reads.  In this scenario it's best to tune the chunk size to increase
the probability that a small read from the stripe would result in only
one read from the disk.  If the chunk size is too small, then a 1KB
read has a pretty high chance to be fragmented between two chunks,
and, thus, require two I/Os to service instead of one I/O (and, thus,
most likely two drive head seeks instead of just one). Modern
commodity drives can do about only 100-120 seeks per second.  But this
is a side note for your side note. :))

>From the man page to 'lvcreate' it seems that the -c option sets the
chunk size for something snapshot-related, so it should have no
bearing in our performance testing, which involved no snapshots.  Am I
misreading the man page?

Thanks!
-- 
Arcady Genkin