Bug#1026870: linux-image-6.0.0-6-amd64 - significant performance degradation
Hi,
On Thu, Dec 22, 2022 at 10:20:44PM +0100, Kamila Szewczyk wrote:
> Package: src:linux
> Version: 6.0.12-1
> Severity: important
> X-Debbugs-Cc: kspalaiologos@gmail.com
>
> Dear Maintainer,
>
> I have recently upgraded my Linux kernel to package version 6.0.12-1
> (6.0.0-6) from version 6.0.8 (6.0.0-4). Since then I am experiencing
> significant performance degradation. I have tried to observe the
> cause of it using the `perf' tool, however, I haven't had any luck
> with this so far, and the kernel has been logging the following
> messages throughout the procedure:
>
> >>>
> Message from syslogd@laplace at Dec 22 21:51:20 ...
> kernel:[ 5212.738589] Uhhuh. NMI received for unknown reason 2d on CPU 9.
>
> Message from syslogd@laplace at Dec 22 21:51:20 ...
> kernel:[ 5212.738607] Dazed and confused, but trying to continue
>
> Message from syslogd@laplace at Dec 22 21:51:21 ...
> kernel:[ 5214.422563] Uhhuh. NMI received for unknown reason 2d on CPU 7.
>
> Message from syslogd@laplace at Dec 22 21:51:21 ...
> kernel:[ 5214.422582] Dazed and confused, but trying to continue
> <<<
>
> So far I have observed that returning to the previously installed
> kernel version solves most of my problems. I have also compared the
> execution times using `perf' on both of the kernels.
>
> New kernel:
> <<<
> Reading package lists... Done
> Building dependency tree... Done
> Reading state information... Done
> E: Unable to locate package asdfghjkl
>
> Performance counter stats for 'apt install asdfghjkl':
>
> 2,409.28 msec task-clock # 0.999 CPUs utilized
> 11 context-switches # 4.566 /sec
> 4 cpu-migrations # 1.660 /sec
> 2,526 page-faults # 1.048 K/sec
> 961,017,382 cycles # 0.399 GHz (83.39%)
> 15,353,337 stalled-cycles-frontend # 1.60% frontend cycles idle (83.40%)
> 226,730,321 stalled-cycles-backend # 23.59% backend cycles idle (83.40%)
> 2,018,766,682 instructions # 2.10 insn per cycle
> # 0.11 stalled cycles per insn (83.39%)
> 383,290,701 branches # 159.089 M/sec (83.35%)
> 1,819,464 branch-misses # 0.47% of all branches (83.24%)
>
> 2.411797623 seconds time elapsed
>
> 2.349918000 seconds user
> 0.062534000 seconds sys
> >>>
>
> Old kernel:
> <<<
> Reading package lists... Done
> Building dependency tree... Done
> Reading state information... Done
> E: Unable to locate package asdfghjkl
>
> Performance counter stats for 'apt install asdfghjkl':
>
> 852.22 msec task-clock # 0.940 CPUs utilized
> 392 context-switches # 459.975 /sec
> 3 cpu-migrations # 3.520 /sec
> 2,548 page-faults # 2.990 K/sec
> 1,172,831,665 cycles # 1.376 GHz (83.08%)
> 43,524,987 stalled-cycles-frontend # 3.71% frontend cycles idle (83.76%)
> 412,509,320 stalled-cycles-backend # 35.17% backend cycles idle (82.83%)
> 2,060,235,707 instructions # 1.76 insn per cycle
> # 0.20 stalled cycles per insn (83.60%)
> 396,100,764 branches # 464.786 M/sec (83.18%)
> 1,892,879 branch-misses # 0.48% of all branches (83.77%)
>
> 0.906939283 seconds time elapsed
>
> 0.795226000 seconds user
> 0.059521000 seconds sys
> >>>
>
> If there is any troubleshooting or important information
> I could provide, please let me know.
>
> I have noticed that the clock speeds tend to be much lower
> on the new Linux kernel, however, I don't know the cause of it.
> All the system data (including e.g. CPU governor configuration)
> stays the same.
As we will soonish move to the 6.1.y series (which as LTS kernel
upstream is aimed to be the one we will use for bookworm), do you
observe the same regression when moving to the kernel available in
experimental? (Ideally test the just uploaded 6.1.1-1~exp1 once it is
built).
If it is still reproducible with latest kernel, would you be able to
bisect the changes between 6.0.8 and 6.0.12 to isolate the change
introducing the regression for you?
The following resource might be helpful for the later case:
https://wiki.debian.org/DebianKernel/GitBisect
Regards,
Salvatore
Reply to: