[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: file systems



On 5/2/2011 4:02 PM, Boyd Stephen Smith Jr. wrote:

They are also essential for any journaled filesystem to have correct behavior
in the face of sudden pwoer loss.

This is true only if you don't have BBWC.

Barriers ensure, (e.g.) that the journal
entry creating a file is flushed to the backing store before the journal
entry.

XFS only journals metadata not file data. Barriers are not used by XFS to guarantee write order. It uses another mechanism for this. The reason is that XFS allows the log journal to be on a different physical device/path than the filesystem device in order to increase performance.

Even with a a battery-packed RAID cache, like I have in my desktop, executing
without barrier can result in extra data loss that executing with a barrier
prevents.

Then I'd say you have a problem with your BBWC RAID controller in your desktop. Which BBWC RAID card do you have?

Never run without barriers if you value your data.

Again, with a quality BBWC implementation, using barriers simply decreases performance. Disabling barriers with BBWC does not expose you to any additional filesystemm inconsistency risk due to cached data.

Of course, even with out barriers a properly journaled or log-structed
filesystem should be able to immediately and silently recover.

This contradicts what you stated above.

Barriers do have a cost, however, on modern file systems it is not anwhere
near 50% even in the random write case, since a barrier doesn't have to occur
between every journal entry.
<http://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-
rc5_Large_file_random_writes._num_threads=128.html>  shows less than a 10%
difference between XFS performance with barriers and without.

This is but one synthetic test case. Do an 'rm -rf' of a subdirectory with 1,000,000 entries on a shared filesystem where other multiple users are performing medium to heavy metadata operations, with and without barriers, and you'll see what I'm talking about. Such a million file delete is actually pretty common with many research applications.

 From this LWN article<http://lwn.net/Articles/283161/>: "The filesystem code
must, before writing the [journaling] commit record, be absolutely sure that
all of the transaction's information has made it to the journal. Just doing
the writes in the proper order is insufficient; contemporary drives maintain
large internal caches and will reorder operations for better performance.

This is why (good) BBWC enabled RAID cards automatically disable the caches on all the drives, and thus why it is recommended to disable barriers for filesystems on BBWC RAID cards. With some BBWC RAID cards you have to manually disable the drive's caches in the controller BIOS or driver.

The nobarrier results are far more relevant than the barrier results,
especially the 16 and 128 thread results, for those SAs with high
performance persistent storage.

I disagree entirely.  You should be looking at the threaded results, probably
128 threads (depending on what the server does), but you should also be using
barriers.

You just said you "disagree entirely" and then say 128 threads, same thing I said. But then you recommend barriers, which is the disagreement.

In those test where XFS has the best 'score' it typically beats most
others by a wide margin.

No, it doesn't.  On a few tests, it does amazingly outpace other file systems.
On most of the others, even when it is first, one of the other filesystems is
a close second.

Considering the fact that JFS was abandoned by IBM over 2 years ago, is no longer developed, is thus deprecated, and is used by no one, my statement is correct. Across the board JFS is the only FS close to XFS in the more than 1 thread tests.

XFS without barriers is first -- but I prefaced my statements with the fact
that I was ignoring -nobarrier variants and the -nocow variant of btrfs.

This is because you don't yet understand the significance of barriers in relation to write cache WRT enterprise hardware with BBWC. The whole point of BBWC is avoiding the use of barriers. Point of fact: most of the very high end SAN array controllers, and even some of the midrange controllers, discard all barrier commands, period, to prevent flushing the mammoth (4-64GB) caches on such controllers. With some it's a config option. Many low end units don't offer it, so you have to disable barriers at the filesystem level.

--
Stan


Reply to: