Re: thoughts on moving to shared storage for VM hosting
Hi Andy,
Local disks in a RAID-10 is probably one of the most performant
configurations so I have no expectation of greater performance, but
obviously it needs to not totally suck in that regard.
Your best solution for this would be iSCSI, preferrably over a
separate network.
The immediate question then is how to do that. Take for example
this disk box:
http://www.span.com/catalog/product_info.php?products_id=4770
Two of those could be used, each in RAID-10, exported by iSCSI and
then software RAID-1 on the servers would allow for operation even
in the face of the complete death of either disk box.
Sounds like a good plan, but I'd spend a lot of time testing what
happens when one of the machines goes down. Does the software RAID
detect this, and what happens to your performance when tens or even
hundreds of exported disks start resyncing?
The downside is that 75% of the raw capacity is gone. Does anyone
have any feel for how much of a performance penalty would be
incurred by configuring each one as say a RAID-50 (two 5-spindle
RAID-5s, striped) in each with 2 hot spares and then software RAID-1
on the servers?
I'd suggest choosing a platform that supports RAID-6 instead of
RAID-5, and use that, optionally with a hot spare. You might even skip
the hot spare, since you can lose up to two active disks in a RAID-6
array.
If you choose RAID-5 with a 12-disk array, sooner or later Murphy will
catch up with you. The chance of a second drive in your RAID-5 array
failing while it's doing a rebuild to the hot spare are larger than
you might think.
Given 12x500G disks in each box, this would result in
(((12-2)/2)-1)x2x500G = 4T usable for 12T raw. The
previously-mentioned RAID-10, RAID-1 configuration would result in
(12-2)/2x500G = 2.5T usable for 12T raw. A straight up 10-disk
RAID-5 on each disk box would give (12-2-1)x500G = 4.5T usable for
12T raw, but 10 spindles seems too big for a RAID-5 to me, plus
RAID-5 write performance sucks and I understand -50 goes some way to
mitigate that.
The disk statistics you're gathering will help a lot in evaluating
your future storage platform. /proc/diskstats will give you a lot of
info on this. If you use cacti for monitoring trends on your servers I
can send you some scripts that will help you graph disk I/O both in
megabytes/second and disk I/O operations.
A crazy idea would be to set both disk boxes up as JBOD and export
all 24 disks out, handling all the redundancy on the servers using
MD. That really does sound crazy and hard to manage though!
As for the server end, is software RAID of iSCSI exports the right
choice here? Would I be better off doing multipath?
As I said before, test this well. You might also look at handling the
replication and fail-over on the storage servers instead of the
clients; you'll need to build your own storage boxes for that, and
invest a lot of time in testing the failover scenario's, but it'll be
easier to manage.
You can run RAID-1 over NBD on your storage servers themselves, or use
DRBD to handle the synchronization.
My next concern is iSCSI. I've not yet played with that in Debian.
How usable is it in Debian Etch, assuming commodity hardware and a
dedicated 1GbE network with jumbo frames? Would I be better off
building my own Linux-based disk box and going with AoE or NBD? The
downside is needing to buy something like two of:
http://www.span.com/catalog/product_info.php?cPath=18_711_2401&products_id=15975
plus two storage servers with SAS to export out AoE or NBD.
You don't have to use external storage; there are lots of
manufacturers that offer servers with 12 or 16 hot-swappable disks.
Supermicro comes to mind: http://www.supermicro.com/products/chassis/3U/836/SC836TQ-R800.cfm
There are a couple of other suppliers, it shouldn't be too hard to
find one in the UK.
Regards,
Maarten
Reply to: