Opinions on ext3 vs XFS vs reiserfs for LAMP server
Hi,
I am wondering if anyone has any real-world advice on the best
filesystem to use for an AMD64 LAMP server. I know that the different
systems have their pros and cons, but a lot of the comparisons out there
seem to be rather old. So I'll describe below exactly what it is I'm doing.
The server is classic LAMP, with Etch AMD64 stable. It's running current
MySQL 5.0. There is also a large number of images being served, which
exist under a deep directory hierarchy that is based on database id,
split up to stop individual directories from getting too large - for
example:
/pics/crazyguyonabike/docs/00/00/13/78/small/P6270050.JPG
There are over 190,000 pics, with three versions of each (large, small,
thumbnail). So more like 600,000 total files, many of them small, in a
deep directory structure. The website is www.crazyguyonabike.com, it
gets something on the order of 80-100,000 page views per day.
The server is a dual Opteron 265 (1.8 GHz), so 4 cores total, with 4 GB
RAM and 4 x 10k SCSI 73GB drives configured on an Adaptec 2015S as a
single RAID 10 array. The machine has been up since 2005 using Sarge
AMD64, and recently I was in the datacenter to rebuild so I could
upgrade to Etch and repartition (the original partition sizes were badly
chosen and the pics were running out of space). I had had problems
installing AMD64 on the RAID card, since it uses dpt_i2o and this is not
included in AMD64 (they put i2o_block instead). Anyway, I decided to
just try Etch 32-bit, since I had a suspicion that maybe it wouldn't
have much impact on performance and none of my processes would need more
than 3GB anyway. So I installed with 32-bit, also using XFS instead of
ext3. Here is a thread that includes some munin snapshots from the old
and new configs:
http://www.crazyguyonabike.com/forum/board/message/?thread_id=60689
The cpu usage seems to have gone up, but the IO wait has gone down a
little. It would seem that the old AMD64 was a little better on cpu
usage and load average; it uses a little more memory (apps) than 32-bit,
but that's ok. So I am going to rebuild again using AMD64 and just see
how things go with i2o_block for the RAID driver.
Now the decision is about partitions and filesystems.
I chose XFS because one of the guys at the datacenter was very
enthusiastic about it for performance, and since I use rsync and MySQL
replication to back up the server very regularly (even pics are mirrored
every minute, using change records in mysql), it would seem that I could
probably recover even a hard reset without much if any data loss. It
seems that XFS has a bit of a reputation for zeroing out open files on
hard resets. I don't know if that's still true or not, though, since all
info out there is anecdotal.
I'm thinking about the following partitioning scheme:
/boot ext3
/ ext3
/mysql ? XFS ?
/pics ? reiserfs ?
/var ? ???
So use ext3 for reliability and stability, and other faster (?) systems
for data that has a big impact on IO performance and can be relatively
easily restored in the event of total loss.
What do you think? Is XFS a good choice for a MySQL partition? XFS is a
lot slower when deleting files, but that doesn't happen as much with
MySQL (I don't have a good handle on how often mysql creates temp files,
though). XFS seems to have been designed by SGI for handling large file
throughput. Is it good for MySQL database performance?
Is reiserfs good for the part of the server that is effectively a
fileserver (i.e. static images being served directly)?
What is a good choice for /var? (assuming, of course, that it's actually
a good idea to split that one out).
Whenever people ask about "which filesystem is best", people usually say
"it depends what you're doing", so I hope I've given enough context here.
Finally, is it even a good idea to "mix" filesystems like this? Are
there any downsides (e.g. multiple caches eating up and contending for
memory)?
I'm mostly interested in experience with performance over time - it
seems like as filesystems age, performance goes down. So "real world"
experience of these would be much appreciated.
Thanks!
/Neil
Reply to: