Re: Increased read IO wait times after Bullseye upgrade
On Thu 10 Nov 2022, at 07:04, Vukovics Mihaly <vm@informatik.hu> wrote:
> Hi Gareth,
>
> - Smartmon/smarctl does not report any hw issues on the HDDs.
> - Fragmentation score is 1 (not fragmented at all)
> - 18% used only
> - RAID status is green (force-resynced)
> - rebooted several times
> - the IO utilization is almost zero(!) - chart attached
> - tried to change the io scheduler of the disks from mq-deadline to noop:
> does not bring change.
> - the read latency increased on ALL 4 discs by the bullseye upgrade!
> - no errors/warnings in kern.log/syslog/messages
>
> Br,
> Mihaly
Hi Mihaly,
People here often recommend SATA cable/connection checks - might this be worthwhile?
"[using]
iostat -mxy 10
[...]
If %util is consistently under 30% most of the time, most likely you don’t have a problem with disk i/o. If you’re on the fence, you can also look at r_await and w_await columns — the average amount of time in milliseconds a read or write disk request is waiting before being serviced — to see if the drive is able to handle requests in a timely manner. A value less than 10ms for SSD or 100ms for hard drives is usually not cause for concern,"
(-- from the article linked in my first response)
Are the {r,w}_await values what is shown in your first chart, inter alia? I imagine so.
Does performance actually seem to be suffering?
Unfortunately I can't try to replicate this as I don't have a HDD system, let alone MDRAID5, hence the rather basic questions, which you may well already have considered.
Write wait time has also increased somewhat, according to your first graph.
Is anything hogging CPU?
Free memory/swap usage seems unlikely to be the issue after several reboots.
I might be barking up the wrong tree here but do you know which kernel version you were using on Buster?
There were minor version changes to mdadm in Bullseye
https://packages.debian.org/buster/mdadm
https://packages.debian.org/bullseye/mdadm
which made me wonder if in-kernel parts of MD changed too.
It might be interesting to diff the relevant kernel sources between Buster and Bullseye, perhaps starting with drivers/md/raid5.c extracted from
/usr/src/linux-source-{KERNEL-VERSION}.tar.xz
https://packages.debian.org/search?keywords=linux-source&searchon=names&suite=oldstable§ion=all
https://packages.debian.org/bullseye/linux-source-5.10
This assumes the identification of the driver in [3] (below) is anything to go by.
$ apt-file search md/raid456
[on Bullseye] seems to agree.
[though are sources up to date?...
$ uname -a
... 5.10.149-2 ...
vs
"Package: ... (5.10.140-1)"
https://packages.debian.org/bullseye/linux-source-5.10]
I'm somewhat out of my range of experience here and this would be very much "an exercise" for me.
I'm sorry I can't try to replicate the issue. Can you, on another system, if you have time?
Best wishes,
G
[1] "Initial driver: /lib/modules/4.18.0-305.el8.x86_64/kernel/drivers/md/raid456.ko.xz
I think this has been changed?"
https://grantcurell.github.io/Notes%20on%20mdraid%20Performance%20Testing/
Reply to: