Very slow LVM performance
I'm seeing a 10-fold performance hit when using an LVM2 logical volume
that sits on top of a RAID0 stripe. Using dd to read directly from
the stripe (i.e. a large sequential read) I get speeds over 600MB/s.
Reading from the logical volume using the same method only gives
around 57MB/s. I am new to LVM and I need to for the snapshots.
Would anyone suggest where to start looking for the problem?
The server runs the amd64 version of Lenny. Most packages (including
lvm2) are stock from Lenny, but we had to upgrade the kernel to the
one from lenny-backports (2.6.32).
There are ten RAID1 triplets: md0 through md9 (that's 30 physical
disks arranged into ten 3-way mirrors), connected over iSCSI from six
targets. The ten triplets are then striped together into a RAID0
stripe /dev/md10. I don't think we have any issues with the MD
layers, because each of them seems to perform fairly well; it's when
we add LVM into the soup the speeds start getting slow.
test4:~# uname -a
Linux test4 2.6.32-bpo.4-amd64 #1 SMP Thu Apr 8 10:20:24 UTC 2010
x86_64 GNU/Linux
test4:~# dd of=/dev/null bs=8K count=2500000 if=/dev/md10
2500000+0 records in
2500000+0 records out
20480000000 bytes (20 GB) copied, 33.4619 s, 612 MB/s
test4:~# dd of=/dev/null bs=8K count=2500000 if=/dev/vg0/lvol0
2500000+0 records in
2500000+0 records out
20480000000 bytes (20 GB) copied, 354.951 s, 57.7 MB/s
I used the following commands to create the volume group:
pvcreate /dev/md10
vgcreate vg0 /dev/md10
lvcreate -l 102389 vg0
Here's what LVM reports of its devices:
test4:~# pvdisplay
--- Physical volume ---
PV Name /dev/md10
VG Name vg0
PV Size 399.96 GB / not usable 4.00 MB
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 102389
Free PE 0
Allocated PE 102389
PV UUID ocIGdd-cqcy-GNQl-jxRo-FHmW-THMi-fqofbd
test4:~# vgdisplay
--- Volume group ---
VG Name vg0
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 399.96 GB
PE Size 4.00 MB
Total PE 102389
Alloc PE / Size 102389 / 399.96 GB
Free PE / Size 0 / 0
VG UUID o2TeAm-gPmZ-VvJc-OSfU-quvW-OB3a-y1pQaB
test4:~# lvdisplay
--- Logical volume ---
LV Name /dev/vg0/lvol0
VG Name vg0
LV UUID Q3nA6w-0jgw-ImWY-IYJK-kvMJ-aybW-GAdoOs
LV Write Access read/write
LV Status available
# open 0
LV Size 399.96 GB
Current LE 102389
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 254:0
Many thanks in advance for any pointers!
--
Arcady Genkin
Reply to: