Re: Opinions on ext3 vs XFS vs reiserfs for LAMP server
On Thu, Aug 23, 2007 at 10:19:44AM -0500, Neil Gunton wrote:
> I am wondering if anyone has any real-world advice on the best
> filesystem to use for an AMD64 LAMP server. I know that the different
> systems have their pros and cons, but a lot of the comparisons out there
> seem to be rather old. So I'll describe below exactly what it is I'm doing.
>
> The server is classic LAMP, with Etch AMD64 stable. It's running current
> MySQL 5.0. There is also a large number of images being served, which
> exist under a deep directory hierarchy that is based on database id,
> split up to stop individual directories from getting too large - for
> example:
>
> /pics/crazyguyonabike/docs/00/00/13/78/small/P6270050.JPG
>
> There are over 190,000 pics, with three versions of each (large, small,
> thumbnail). So more like 600,000 total files, many of them small, in a
> deep directory structure. The website is www.crazyguyonabike.com, it
> gets something on the order of 80-100,000 page views per day.
>
> The server is a dual Opteron 265 (1.8 GHz), so 4 cores total, with 4 GB
> RAM and 4 x 10k SCSI 73GB drives configured on an Adaptec 2015S as a
> single RAID 10 array. The machine has been up since 2005 using Sarge
> AMD64, and recently I was in the datacenter to rebuild so I could
> upgrade to Etch and repartition (the original partition sizes were badly
> chosen and the pics were running out of space). I had had problems
> installing AMD64 on the RAID card, since it uses dpt_i2o and this is not
> included in AMD64 (they put i2o_block instead). Anyway, I decided to
> just try Etch 32-bit, since I had a suspicion that maybe it wouldn't
> have much impact on performance and none of my processes would need more
> than 3GB anyway. So I installed with 32-bit, also using XFS instead of
> ext3. Here is a thread that includes some munin snapshots from the old
> and new configs:
>
> http://www.crazyguyonabike.com/forum/board/message/?thread_id=60689
>
> The cpu usage seems to have gone up, but the IO wait has gone down a
> little. It would seem that the old AMD64 was a little better on cpu
> usage and load average; it uses a little more memory (apps) than 32-bit,
> but that's ok. So I am going to rebuild again using AMD64 and just see
> how things go with i2o_block for the RAID driver.
>
> Now the decision is about partitions and filesystems.
>
> I chose XFS because one of the guys at the datacenter was very
> enthusiastic about it for performance, and since I use rsync and MySQL
> replication to back up the server very regularly (even pics are mirrored
> every minute, using change records in mysql), it would seem that I could
> probably recover even a hard reset without much if any data loss. It
> seems that XFS has a bit of a reputation for zeroing out open files on
> hard resets. I don't know if that's still true or not, though, since all
> info out there is anecdotal.
>
> I'm thinking about the following partitioning scheme:
>
> /boot ext3
> / ext3
> /mysql ? XFS ?
> /pics ? reiserfs ?
> /var ? ???
>
> So use ext3 for reliability and stability, and other faster (?) systems
> for data that has a big impact on IO performance and can be relatively
> easily restored in the event of total loss.
>
> What do you think? Is XFS a good choice for a MySQL partition? XFS is a
> lot slower when deleting files, but that doesn't happen as much with
> MySQL (I don't have a good handle on how often mysql creates temp files,
> though). XFS seems to have been designed by SGI for handling large file
> throughput. Is it good for MySQL database performance?
>
> Is reiserfs good for the part of the server that is effectively a
> fileserver (i.e. static images being served directly)?
>
> What is a good choice for /var? (assuming, of course, that it's actually
> a good idea to split that one out).
>
> Whenever people ask about "which filesystem is best", people usually say
> "it depends what you're doing", so I hope I've given enough context here.
>
> Finally, is it even a good idea to "mix" filesystems like this? Are
> there any downsides (e.g. multiple caches eating up and contending for
> memory)?
>
> I'm mostly interested in experience with performance over time - it
> seems like as filesystems age, performance goes down. So "real world"
> experience of these would be much appreciated.
Well personally I have had nasty data corruption and data loss in the
past with reiserfs, so I don't touch that anymore. Unless they have
changed the behaviour of reiserfs 3.6 in the last couple of years to be
like ext3's ordered writes, then I would not put any data on a reiserfs
filesystem.
I used XFS for a while, but due to some bugs in earlier 2.6 kernels it
would frequently freeze and corrupt the filesystem (although it always
seemed fixable without data loss, although the server hanging was not a
good thing). I have no idea if those bugs have been fixed since, as I
moved to ext3 and haven't looked back.
For me ext3 just works and has been the least problematic so far.
I have in a previous job even used ext3 on a 1.5TB LV to store digital
photos. Never had a problem, and performance seemed only limited by the
slow poke IBM SAN we had to deal with.
I have never tried JFS personally. No idea about it.
For storing mysql, just about anything should do since it is just a few
large files in general.
For storing pictures, again probably just lots of fairly large files.
As long as you don't make a stupid system with a million files per
directory, anything should do. I have used something like:
/pics/00/01/43/6e/f3.jpg in the past, where the path was the image
number on the system in hex, and it kept each directory at no more than
256 entries, which made it very fast, and very easy to find things. I
suppose optimally you might even want to try and figure out how many
entries you can store in a directory using one 4k block, and aim to stay
just under that. After all smaller directories are faster to
update/sort/etc.
So my opinion is to use ext3 for it all. Then one repair tool is all
you will ever need (and in my experience you are unlikely to really ever
need that one tool either). The repair tools for reiserfs have
historically been awful (in many cases making the problem worse rather
than better), and the XFS tools have had a tendancy to require an insane
amount of ram on larger filesystems which can make repairs very
difficult unless you have a lot of ram if using a large drive.
--
Len Sorensen
Reply to: