Re: Storage server

[Stan Hoeppner <stan@hardwarefreak.com>]

On 9/16/2012 7:38 AM, Martin Steigerwald wrote:

> I have always recommended to leave at least 10-15% free, but from a 
> discussion on XFS mailinglist where you took part, I learned that 
> depending on use case for large volumes even more free space might be 
> necessary for performant long term operation.

And this is due the allocation group design of XFS.  When the filesystem
is used properly, its performance with parallel workloads simply runs
away from all other filesystems.  When using LVM in the manner I've been
discussing, the way the OP of this thread wants to use it, you end up
with the following situation and problem:

1.  Create 1TB LVM and format with XFS.
2.  XFS creates 4 allocation group
3.  XFS spreads directories and files fairly evenly over all AGs
4.  When the XFS gets full, you end up with inode/files/free space
    badly fragmented over the 4 AGs and performance suffers when reading
    these back, or when trying to write new, or modify existing
5.  So you expand the LV by 1TB and then grow the XFS over the new space
6.  This operation simply creates 4 new AGs in the new space
7.  New inode/extent creation to these new AGs is fast and reading back
    is also fast.
8.  But, here's the kicker, reading the fragmented files from the first
    4 AGs is still dog slow, as well as modifying metadata in those AGs

Thus, the moral of the story is that adding more space to an XFS via LVM
can't fix performance problems that one has created while reaching the
"tank full" marker on the original XFS.  The result is fast access to
the new AGs in the new LVM sliver, but slow access to the original 4 AGs
in the first LVM sliver.  So as one does the LVM rinse/repeat growth
strategy, one ends up with slow access to all of their AGs in the entire
filesystem.  Thus, this method of "slice/dice" expansion for XFS is insane.

This is why XFS subject matter experts and power users do our best to
educate beginners about the aging behavior of XFS.  This is why we
strongly recommend that users create one large XFS of the maximum size
they foresee needing in the long term instead of doing the expand/grow
dance with LVM or doing multiple md/RAID reshape operations.

Depending on the nature of the workload, and careful, considerate,
judicious use of XFS grow operations, it is safe to grow an XFS without
the performance problems.  This should be done long before one hits the
~90% full mark.  Growing before it hit ~70% is much better.  But one
should still never grow an XFS more than a couple of times, as a general
rule, if one wishes to maintain relatively equal performance amongst all
AGs.

-- 
Stan