Re: Storage tuning
Greg wrote:
> Hi there,
>
> I have a "server" running 24/7 with a lot of RAM. I would like to speed up
> disk system by giving much higher priority to reads and delaying writes.
>
> YES, I KNOW THE RISK!
>
> As I understand, there are two things to tune:
>
> 1. I/O Scheduler. The default is mq-deadline. Let say I have md_raid array
> /dev/md0 consisting of /dev/sd[ab]. Should I keep it for both, md0 and
> physical disks? Which I/O scheduler parameters should I change in case of
> md0 and which in case of sd[ab]?
>
> 2. Kernel runtime parameters. As I understand I should focus on vm.dirty_*
> parameters. My Idea is to set vm.dirty_ratio=70 and
> vm.dirty_expire_centisecs to something like 10 to 60min. Should I change
> anything else?
>
> PS. Among others, I'm trying to learn something about Linux caching. So
> please stick to above questions.
The first thing you should do is establish whether you have a
problem.
cat /proc/vmstat | egrep "dirty|writeback"
nr_dirty is the number of pages waiting to be written, which
will be written out when vm.dirty_expire_centisecs goes off.
If this number is low while your server is actually doing
things, then there is no point in trying to delay writes further
- it is not pushing enough data to disk to be worthwhile.
What's low? Well, a page is typically 4KB, so anything that is
less than a tenth of a second of writing is going to be fast. If
you have a RAID that can write at 150MB/s (a typical speed for a
single rotating disk) then less than 15MB is negligible. That
would be an nr_dirty around 1000.
If you are using a SATA SSD, a write speed of 500MB/s is a good
assumption, so an nr_dirty exceeding 6000 would hit a 0.1 second
threshold. If you are using PCIe 4 NVMe SSDs, 1-2 GB/s is
plausible, and the nr_dirty would have to be 12000 to 25000.
All this assumes a fairly linear write pattern. If you are using
rotating disks and all your writes are small and to random
locations - the worst case - then an nr_dirty of 100 might be
interesting.
If you want to gather stats for a longer period of time while
you run a typical workload, the command you want is called sar.
First establish that you have a problem.
-dsr-
Reply to: