Debian-13 bad performance on IPC
Sorry for the lengthy message, but I don't know how to better compress
the facts...
- I have a data aquisition device which delivers data with ~2.8MB/s to
an IPC (fanless Industrial PC, Intel core i7-1185G7E).
- When receiving a START signal, data are stored to SSD, and an analysis
program is started. The analysis program in turn starts several
sub-processes, which in turn start several sub-sub-processes.
All sub-sub-processes read the recorded data files in a "tail -f" fashion.
- When receiving the STOP signal after 5secs, recording is stopped and
the analysis result is collected from the sub-sub-processes.
* Crucial point is the time it takes from STOP to RESULT-RECEIVED:
We started on Windows-11:
- IPC, Windows-11 with all updates
(11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)
Result-recv[s] Count
----------------------
2.20 2
2.25 82
2.30 328
2.35 767
2.40 857
2.45 632
2.50 372
2.55 136
2.60 40
2.65 16
2.70 1
2.80 1
=> Approx. 2.4s to "Result-received", with some variation,
but fairly constant. So far so good.
The idea now was that Linux/Debian-13 should perform better on
identical hardware.
- the very same IPC, Debian-13, with all patches
(11th Gen Intel(R) Core(TM) i7-1185G7E @ 2.80GHz, 8 logical processors)
Result-recv[s] Count
----------------------
0.60 1
0.65 75
0.70 137
0.75 109
0.80 21
0.85 8
0.90 29
0.95 18
1.00 5
1.45 1
1.50 1
1.55 2
1.60 10
1.65 17
1.70 27
1.75 177
1.80 532
1.85 384
1.90 92
1.95 39
2.00 15
2.45 1
2.50 1
2.55 4
2.60 6
2.65 102
2.70 365
2.75 395
2.80 604
2.85 562
2.90 115
2.95 19
3.00 7
3.45 3
3.50 9
3.55 12
3.60 17
3.65 15
3.70 21
3.75 31
3.80 39
3.85 21
3.90 7
3.95 1
=> There are several clusters of answer times, all ~1s apart.
=> the majority of answers are *later* than on Windows
I don't understand
- why the performance varies that much,
- why the performance is worse than on Windows on the very same PC hardware.
I have tried:
- echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
- no difference
- sudo powerprofilesctl set performance
- no difference
- cpupower-gui: set performance mode, limit min-CPU-Freq to 2500
- no difference
- intel_pstate=disabled at boot
- performance even worse
- intel_pstate=passive at boot, cpu-governor schedutil
- slightly worse performance
- Debian-12, same software
- no difference
For comparison, the very same program on my Desktop PC (with
cpu-governor 'schedutil', since there is a different kernel module at work)
- Desktop PC, Debian-13 with all patches
(Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 8 logical processors)
Result-recv[s] Count
0.40 2
0.45 1
0.50 1
0.55 4
0.60 14
0.65 21
0.70 46
0.75 68
0.80 82
0.85 55
0.90 16
0.95 8
1.00 3
1.05 1
This is what I had expected on the IPC as well.
The problem seems to be that some of the sub-sub-processes are started
very late, not until STOP is received:
local_time[ms] what
----------------------
17817764 68904 MAIN "START" sub-1 started
17817764 69213 sub-1: OPENED sub-1-1
17817764 69215 sub-1: OPENED sub-1-2
17817764 69216 sub-1: OPENED SUB-1-3
17817764 71455 sub-1-2: opened sub-1-2-1
17817764 71455 sub-1-2: opened sub-1-2-2
17817764 73451 sub-1-3: opened sub-1-3-1
17817764 73452 sub-1-3: opened sub-1-3-2
17817764 73905 MAIN "STOP" sub-1 close, wait for result
Note: NO "sub-1-1 OPENED" yet!
17817764 74068 sub-1-3: finished sub-1-3-1
17817764 74072 sub-1-2: finished sub-1-2-1
17817764 74076 sub-1-3: finished sub-1-3-2
17817764 74081 sub-1-2: finished sub-1-3-2
17817764 74165 sub-1: CLOSED sub-1-2
17817764 74166 sub-1: CLOSED sub-1-3
Only now arrive the "sub-1-1: opened..." ! It looks like the
scheduler postponed running sub-1-1 until now...
17817764 74457 sub-1-1: opened sub1-1-1
17817764 74457 sub-1-1: opened ...
17817764 74457 sub-1-1: opened sub1-1-9
17817764 74484 sub-1-1: closed sub1-1-1
17817764 75737 sub-1-1: closed sub1-1-2
...
17817764 75739 sub-1-1: closed sub1-1-9
17817764 76657 sub-1: CLOSED sub-1-1
Which program starts 'late' varies, sometimes it is sub-1-1, sometimes
sub-1-2, sometimes sub-1-3. Sometimes all three start early, this is
when the low response times are achieved.
On my Desktop-PC and on Windows on the same hardware:
- all sub-programs (sub and sub-sub) start immediately after START
- all sub-programs are done quickly after STOP (since processing time
is less than recording time)
So I would be grateful for any hint on how to make the IPC under
Debian-13 as performant (or better) as under Windows, at best as
perfomant as my ancient desktop PC.
TNX
R'
Reply to: