Hi,
Say I've got a handful of servers hosting virtual machines. They're
typical 1U boxes with 4 SATA disks and a 3ware RAID card configured
as RAID-10. The local storage is put into LVM and logical volumes
exported to the VMs.
The servers are split across two racks in different suites of a
datacentre. Each rack has two switches for redundancy, and the two
racks are cross connected with two links from each switch also.
I would like to investigate moving to shared storage and would be
very interested in hearing people's opinions of the best way to go.
The goals/requirements are as follows:
- Easier manageability; moving VMs between hosts is currently a bit
of a hassle involving creating new LVs on the target and copying
the data.
- Better provision for hardware failure; at the moment I need to
keep spare servers but if the storage were more flexible then I
could go to an N+1 arrangement of hosts and quickly bring up VMs
from a failed host on all of the other hosts.
- Lower the cost of scaling up; cheaper CPU nodes with little or no
local disk. I have some hope of reduced power consumption also,
since I am billed per Volt-Amp and this represents over 60% of my
recurring colo charges.
- Should be as cheap as possible while not being any less resilient
than the current setup. If I have to hand build it out of Linux
and NBD then that's fine.
- The current choice of SATA is due to customer demand for large
amounts of storage. It is not economical in the current setup for
me to go to SAS or SCSI even though it has higher performance, so
it is unlikely to be economical to do it with shared storage.
- No requirement for cluster operation; each block
device/LUN/whatever will only ever be in use by one host at a
time.
Local disks in a RAID-10 is probably one of the most performant
configurations so I have no expectation of greater performance, but
obviously it needs to not totally suck in that regard.
For redundancy purposes there most likely needs to be one disk box
per rack, with servers from both racks able to use either disk box.
Power failures on a rack or suite basis do happen from time to time,
so if there were only one disk box in that scenario then dual power
would not help and the resultant outage to servers in the unaffected
suite would be unacceptable.
The immediate question then is how to do that. Take for example
this disk box:
http://www.span.com/catalog/product_info.php?products_id=4770
Two of those could be used, each in RAID-10, exported by iSCSI and
then software RAID-1 on the servers would allow for operation even
in the face of the complete death of either disk box.
The downside is that 75% of the raw capacity is gone. Does anyone
have any feel for how much of a performance penalty would be
incurred by configuring each one as say a RAID-50 (two 5-spindle
RAID-5s, striped) in each with 2 hot spares and then software RAID-1
on the servers?
Given 12x500G disks in each box, this would result in
(((12-2)/2)-1)x2x500G = 4T usable for 12T raw. The
previously-mentioned RAID-10, RAID-1 configuration would result in
(12-2)/2x500G = 2.5T usable for 12T raw. A straight up 10-disk
RAID-5 on each disk box would give (12-2-1)x500G = 4.5T usable for
12T raw, but 10 spindles seems too big for a RAID-5 to me, plus
RAID-5 write performance sucks and I understand -50 goes some way to
mitigate that.
Still 4T usable seems like a poor amount to end up with after buying
12T of storage, but I can't see how anything except RAID-1 across the
two disk boxes would allow for one of them to die. With 6T written
off to start with, perhaps getting 4T out of the remaining 6T does
not seem so bad.
A crazy idea would be to set both disk boxes up as JBOD and export
all 24 disks out, handling all the redundancy on the servers using
MD. That really does sound crazy and hard to manage though!
As for the server end, is software RAID of iSCSI exports the right
choice here? Would I be better off doing multipath?
My next concern is iSCSI. I've not yet played with that in Debian.
How usable is it in Debian Etch, assuming commodity hardware and a
dedicated 1GbE network with jumbo frames? Would I be better off
building my own Linux-based disk box and going with AoE or NBD? The
downside is needing to buy something like two of:
http://www.span.com/catalog/product_info.php?cPath=18_711_2401&products_id=15975
plus two storage servers with SAS to export out AoE or NBD.
At the moment I am gathering I/O usage statistics from one of my
busiest servers and I'll respond with those later if they will help.
If anyone has any experience of any of this, or any thoughts, I'd
love to hear what you have to say.
Cheers,
Andy
Attachment:
signature.asc
Description: Digital signature