filesystem slowdown with backports kernel

[Jens Holzkämper <jens.holzkaemper@zbmath.org>]

Hi,

we have a NAS system acting as a place to store our server's backups
(via rsync with link-dest). On that NAS we switched from the stable
kernel (4.9) to the one provided by backports (4.18) because of an
unrelated problem. When we do that, we see a slowdown of our backup
process, from the backup via rsync itself to deleting old backup
directories. The slowdown seems to be connected to the number of
files/directories as backups of systems with less files seem less
affected than the ones with many files.


So we started benchmarking and the following seems to do the trick in
showing our problem by creating about 100k directories and files (10
dirs containing 10000 directories and files for easier deleting between
tries):

#!/bin/bash
time (
	for i in {0..9};do
        	for j in {0000..9999};do
                	mkdir -p $i/$j
	                touch $i/$j/1
		done
	done
)


We get the following results (with a variance within a few seconds)

4.9 ext4:
real	2m13.303s
user	0m4.976s
sys	0m20.424s

4.9 xfs:
real	2m7.416s
user	0m5.076s
sys	0m20.960s

4.18 ext4:
real	4m3.276s
user	2m46.401s
sys	1m12.546s

4.18 xfs:
real	3m53.430s
user	2m46.841s
sys	1m12.716s

About a 50% slowdown in time elapsed and quite an increase in user and sys.


To rule out something like spectre/meltdown-mitigations we tried the
oldest kernel package that's a higher version number than in stable we
could find on http://snapshot.debian.org from July 2017.

4.11 ext4:
real	3m28.443s
user	2m29.551s
sys	1m0.924s

4.11 xfs
real	3m32.438s
user	2m31.349s
sys	1m3.333s

It's a little faster than 4.18 but the problem still persists.


The NAS is using a software RAID 6 via MD, and we tested with the same
script on a desktop system to rule out the RAID as a problem source and
see the same thing:

4.9 ext4 desktop:
real	2m22.525s
user	0m6.176s
sys	0m20.872s

4.18 ext4 desktop:
real	4m16.412s
user	3m2.282s
sys	1m19.308s


So to us at looks like something is seriously wrong somewhere but have
no clue where exactly to look for anymore. Is the test flawed, did we
miss something about an expected slowdown in the news, is it really a
bug and if so where can we look to locate it more precisely?

Thanks in advance,
Jens Holzkämper