Re: Optimal Storage Server
On 7/22/2012 6:41 AM, Shaffin Bhanji wrote:
> Hello,
>
> I am trying to put together a 2U storage server for data. I have
> previously invested in NAS equipment such as the Netgear NAS 1100 that I
> have been disappointed in to say the least - data write speed of 5MB/s.
>
> This time around I want to build something that I have control over
> hardware than to rely on equipment that I am locked in with, and not to
> mention limitations.
>
> I would like an opinion from this group on successful implementation as
> I will highly be using the server for vitalization disks (iSCSI),
> backup, file share, etc. I want to make sure that I chose the right
> hardware to get the best read/write performance.
>
> Experiences shared will greatly be appreciated.
If building a NFS/CIFS server from channel parts, I recommend the
following. Newegg only ships to N. America but these parts are top
shelf and common, and most if not all should be available from a local
vendor worldwide.
http://www.newegg.com/Product/Product.aspx?Item=N82E16811152478
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182245
http://www.newegg.com/Product/Product.aspx?Item=N82E16819105277
http://www.newegg.com/Product/Product.aspx?Item=N82E16835114078
4x http://www.newegg.com/Product/Product.aspx?Item=N82E16820148382
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118169
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118162
2x http://www.newegg.com/Product/Product.aspx?Item=N82E16816116098
8x http://www.newegg.com/Product/Product.aspx?Item=N82E16822236243
http://www.newegg.com/Product/Product.aspx?Item=N82E16827151242
Summary: 4 core Opteron @2.2GHz, 16GB ECC DDR1333, dual GbE
LSI dual core RAID PCIe x8 1GB DDR1333 BBWC
8x 10k RPM 1TB WD Raptor drives in hot swap
This hardware gives you the most performance bang for the buck in a 2U
chassis.
Configure the drives as a single RAID10 array in the LSI webBIOS,
yielding 4TB of usable space. Use a relatively small strip (not stripe)
size of 64KB, for a stripe size of 4x64= 256KB. This will yield the
best read performance for virtual machine disk files and general file
sharing. With the battery module you will enable write back caching,
which will dramatically increase write performance for all workloads due
to the 1GB of cache.
Create a single 4TB virtual disk of the array and export it a single
LUN. Configure the mobo BIOS to boot from the LSI controller. Boot
your install DVD, create a small partition for /boot (10MB), one for the
root filesystem (50GB), and install the bootloader (I prefer LILO for
hardware RAID) in the MBR. Create a 4GB partition as swap. Pick
whatever filesystem you prefer for /boot and the root filesystem. I
normally use EXT2 for /boot, and EXT3/4 for the root filesystem. Finish
your Debian install.
Once finished, and assuming the reboot from the controller LUN works
fine, create a partition of the remaining space on the LUN. Then create
one large XFS filesystem of the remaining ~3.9TB of space, with this
command:
~$ mkfs.xfs -d su=64k,sw=4 /dev/sda4
The su and sw parameters are somewhat critical, as they align all writes
to the hardware stripe width of the RAID device, allowing the hardware
write cache to perform optimally.
Edit fstab and mount the XFS filesystem as such:
/dev/sda4 /data xfs nobarrier,inode64 0 0
Since you have BBWC, nobarrier tells XFS to not use write barriers when
writing to the journal or when doing unwritten extent conversion. This
can speed up metadata (directory) operations significantly, as XFS will
not flush the hardware write cache, as it would normally do without this
setting. inode64 changes the allocator behavior, putting metadata and
files in the same allocation group, decreasing read/write latency. The
default inode32 allocator puts all metadata in AG0, causing head
thrashing and increases latency with common workloads.
Now create your data directories, virtual machine disk files, and iSCSI
LUN target exports as you normally would, along with your samba shares.
And enjoy the phenomenal IO performance this relatively inexpensive 8
drive machine will give you.
If 4TB isn't enough, you can use 2TB RE4 drives:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136579
and double your storage to 8T. Due to the 1GB of BBWC on the LSI your
write performance may not decrease much, but your read performance will
drop by about 50%. The RE4s will also save about $400 on the 8 drives,
at Newegg pricing, but given the overall cost of the machine that's not
much savings.
If you saying to yourself "I'll just use RAID6 and get 2 more TB from
the fast drives". If you do this I've just wasted more than an hour of
my time here. Using RAID6 will cause a read-modify-write cyle every
time you modify an existing file--which is all the time with virtual
machine disk files. That RMW cycle increases latency per write
operation by anywhere from 6-10 fold and decreases write throughput by
about the same. The latest generation of LSI controllers, including the
9266-8i, do have pretty fantastic performance with RAID6 RMW due to the
large cache and fast ASICs, but the write performance under heavy load
is still less than half that of RAID10, which isn't affected by
RMW--there's no parity to recalculate.
Hope you've found this information and useful, given the time I spent
putting it all together for you.
--
Stan
Reply to: