Re: Increased read IO wait times after Bullseye upgrade

[Vuki <szun@informatik.hu>]

Hi Nicholas,

in longer term the load of the raid members are equal:
DSK |          sde | busy      7% |              | read  102017 | write 
217744 | KiB/r      9 | KiB/w      6 |              | MBr/s    0.0 | 
MBw/s    0.0 | avq     2.60 |              | avio 5.91 ms |
DSK |          sdb | busy      7% |              | read   98877 | write 
210703 | KiB/r      9 | KiB/w      5 |              | MBr/s    0.0 | 
MBw/s    0.0 | avq     2.76 |              | avio 6.05 ms |
DSK |          sdc | busy      7% |              | read   89963 | write 
219237 | KiB/r     10 | KiB/w      5 |              | MBr/s    0.0 | 
MBw/s    0.0 | avq     2.71 |              | avio 6.03 ms |
DSK |          sda | busy      7% |              | read  100267 | write 
211566 | KiB/r      9 | KiB/w      5 |              | MBr/s    0.0 | 
MBw/s    0.0 | avq     2.55 |              | avio 5.93 ms |

The filesystems are clear.

Br,
Mihaly

On 2022. 11. 12. 2:27, Gareth Evans wrote:
> > On 11 Nov 2022, at 16:59, Vukovics Mihály <vm@informatik.hu> wrote:
> >
> > 
> >
> > Hi Gareth,
> >
> > dmesg is "clean", there disks are not shared in any way and there is 
> > no virtualization layer installed.
> Hello, but the message was from Nicholas :)
>
> Looking at your first graph, I noticed the upgrade seems to introduce 
> a broad correlation between read and write iowait.  Write wait before 
> the uptick is fairly consistent and apparently unrelated to read wait.
>
> Does that suggest anything to anyone?
>
> In your atop stats, one drive (sdc) is ~50% more "busy" than the 
> others, has double the number of writes, a higher avq value and lower 
> avio time.  Is it normal for raid devices to vary this much?
>
> Is this degree of difference consistent over time?  Might atop stats 
> during some eg. fio tests be useful?
>
> Have you done any filesystem checks?
>
> Thanks,
> Gareth