| Hi all, I'm experiencing some unexpected disk I/O latency on a newly built server. Here are some of the details: - Supermicro motherboard with Intel PCH RAID [1] - Debian 13.6 installed - / is mounted on /dev/nvme0n1p2, which is an NVMe SSD, formatted with ext4 - /home is mounted on /dev/md125, which is a RAID 5 consisting of three disks, formatted with ext4 $ sudo mdadm -D /dev/md125 /dev/md125: Container : /dev/md/imsm1, member 0 Raid Level : raid5 Array Size : 9279152128 (8.64 TiB 9.50 TB) Used Dev Size : 4639576064 (4.32 TiB 4.75 TB) Raid Devices : 3 Total Devices : 3 State : active Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-asymmetric Chunk Size : 64K Consistency Policy : resync UUID : 2fbd736b:9b750e2a:e723b80c:df2f2c33 Number Major Minor RaidDevice State 2 8 0 0 active sync /dev/sda 1 8 16 1 active sync /dev/sdb 0 8 32 2 active sync /dev/sdc I'm getting some weird latency issues with fsync on the RAID disk (/home on /dev/md125): (Caveat: I'm doing a file transfer that is writing to /home, so I might expect some latency, but nothing like this.) $ for i in {1..5}; do time sync; done 2>&1 | grep real real 0m21.647s real 0m11.770s real 0m5.234s real 0m3.023s real 0m0.462s Even when I turn off the file transfer, I still get something like this: $ for i in {1..10}; do time sync; done 2>&1 | grep real real 0m0.147s real 0m0.077s real 0m0.003s real 0m0.004s real 0m0.003s I've tried turning on and off smart monitoring with smartctl to see if that would help, and it didn't seem to help. I'm not really sure what else I might try, and I'm reaching out to the community to see if anyone has any suggestions for identifying and fixing the problem. Thanks, Casey |