[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: file systems



In <4DBE75DD.80900@hardwarefreak.com>, Stan Hoeppner wrote:
>On 5/1/2011 10:40 AM, Boyd Stephen Smith Jr. wrote:
>> In<4DBD0D23.1080903@hardwarefreak.com>, Stan Hoeppner wrote:
>>> Independent Linux filesystem tests performed by an IBM engineer to track
>>> BTRFS performance during development.  XFS trounces the others in most
>> 
>>> tests:
>> These results are interesting and useful, but I think "trounces" is a poor
>> description for what XFS does.
>> 
>> Not using barriers undermines data consistency guarantees, I think it is
>> best to ignore the 2.6.35-rc5-autokern1-ext3-*-ext3,
>> 2.6.35-rc5-autokern1-ext4-*- ext4-nobarrier, and
>> 2.6.35-rc5-autokern1-xfs-*-xfx-nobarrier entries.
>
>It would be best for you to state something like "which results are
>relevant to you depend on your workload and hardware environment".
>Generally, anyone running a server with persistent storage is interested
>in the nobarrier results.  Those without persistent storage should be
>interested in the barrier results.  There are exceptions to this general
>guideline and I mention at least one below.
>
>Barriers are used to flush storage device cache to avoid data loss due
>to a system crash or power loss.

That's not all they are used for.  They are also used to makes sure that 
writes occur in the correct order.

>Barriers are great for low end systems
>lacking persistent storage.

They are also essential for any journaled filesystem to have correct behavior 
in the face of sudden pwoer loss.   Barriers ensure, (e.g.) that the journal 
entry creating a file is flushed to the backing store before the journal 
entry.  Or, that the journal entry

>However, barriers should never be used with
>high performance persistent storage, i.e. RAID/SAN controllers w/
>battery or flash backed write cache.

Even with a a battery-packed RAID cache, like I have in my desktop, executing 
without barrier can result in extra data loss that executing with a barrier 
prevents.

Never run without barriers if you value your data.

Of course, even with out barriers a properly journaled or log-structed 
filesystem should be able to immediately and silently recover.

>Using barriers with such storage
>arrays simply costs you between 50-90% of your random write performance,
>especially with shared SAN storage and many hosts, as issuing a single
>barrier flushes the entire BBWC in the array controller.  May enterprise
>arrays contain 32GB or more of BBWC.

Barriers do have a cost, however, on modern file systems it is not anwhere 
near 50% even in the random write case, since a barrier doesn't have to occur 
between every journal entry.  
<http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-
rc5_Large_file_random_writes._num_threads=128.html> shows less than a 10% 
difference between XFS performance with barriers and without.

From this LWN article <http://lwn.net/Articles/283161/>: "The filesystem code 
must, before writing the [journaling] commit record, be absolutely sure that 
all of the transaction's information has made it to the journal. Just doing 
the writes in the proper order is insufficient; contemporary drives maintain 
large internal caches and will reorder operations for better performance. So 
the filesystem must explicitly instruct the disk to get all of the journal 
data onto the media before writing the commit record; if the commit record 
gets written first, the journal may be corrupted. The kernel's block I/O 
subsystem makes this capability available through the use of barriers; in 
essence, a barrier forbids the writing of any blocks after the barrier until 
all blocks written before the barrier are committed to the media. By using 
barriers, filesystems can make sure that their on-disk structures remain 
consistent at all times." 

>The nobarrier results are far more relevant than the barrier results,
>especially the 16 and 128 thread results, for those SAs with high
>performance persistent storage.

I disagree entirely.  You should be looking at the threaded results, probably 
128 threads (depending on what the server does), but you should also be using 
barriers.

>For desktop users the single thread
>barrier results are probably the most relevant.

Agreed.

>> I wouldn't say that is a "trouncing", since it doesn't even win in many
>> categories.
>
>In those test where XFS has the best 'score' it typically beats most
>others by a wide margin.

No, it doesn't.  On a few tests, it does amazingly outpace other file systems.  
On most of the others, even when it is first, one of the other filesystems is 
a close second.

>>> http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5
>>> _Ma il_server_simulation._num_threads=128.html
>> 
>> 4th, 2nd, 4th, 4th, 2nd
>> 
>If you could interpret a graph correctly Boyd maybe you'd see it
>differently.  :)  Just taking the last one above as an example, you
>listed XFS as 4th place when it's clearly in first place, by a huge
>margin over all but JFS.  XFS 32k, JFS 30k, EXT4 20k.

XFS without barriers is first -- but I prefaced my statements with the fact 
that I was ignoring -nobarrier variants and the -nocow variant of btrfs.

JFS (default): 30k
BtrFS (default): 10.8k
Ext3 (barrier): 6.1k
XFS (default, barrier): 5k
Ext4 (default, barrier): 3.5k

XFS = 4th.
-- 
Boyd Stephen Smith Jr.                   ,= ,-_-. =.
bss@iguanasuicide.net                   ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy         `-'(. .)`-'
http://iguanasuicide.net/                    \_/

Attachment: signature.asc
Description: This is a digitally signed message part.


Reply to: