Bug#1001818: linux-image-amd64: Syscalls on the seconds thread are slow with many open files

[Thomas Otto <th1000s-2020@posteo.net>]

Package: linux-image-amd64
Version: 5.10.70-1
Severity: normal
Tags: upstream
X-Debbugs-Cc: th1000s-2020@posteo.net

Dear Maintainer,

included is a minimal example program which demonstrates the issue.
Apart from the 5.10 kernel I could reproduce this on 4.9 (oldstable),
4.18 and 5.14 on various amd64 distributions. It did not show up on
3.10-amd64 or on 5.11-arm64/aarch64.

This program contains a function which repeatedly opens a file, reads
from it and keeps the fd open (set `ulimit -n X` if you increase that).

When this runs on the main thread, everything works as expected, but
when started on a second thread, it runs much slower, in my case by a
factor of over 10.

According to `strace -f -o log --syscall-times=us ..` this is caused by
syscalls taking longer, some even taking milliseconds to complete.


------------- %< --------------- %< -------------

/* gcc -std=c11 slow_read_in_thread.c -lpthread
 * g++ -std=c++11 slow_read_in_thread.c -lpthread
 *
 * time ./a.out        # run in thread, slow
 *   vs
 * time ./a.out main   # run in main, fast
 */

#include <assert.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#if __cplusplus >= 201103L
#include <thread>
#else
#include <threads.h>
#endif

int read_stat(void *_arg) {
  int i;
  for (i = 0; i < 999; ++i) {
    char buf[2048];
    // identical behavior if the pid is advanced, or
    // another filesystem is used.
    int fd = open("/proc/1/stat", O_RDONLY);
    ssize_t n = read(fd, buf, 2048);
    assert(n > 1);
    // keep files open:
    // close(fd);
  }
  return i;
}

int main(int argc, char **_argv) {
  void *arg = NULL;
  if (argc > 1) {
    printf("main thread\n");
    read_stat(arg);
  } else {
    printf("2nd thread\n");
#if __cplusplus >= 201103L
    std::thread t(read_stat, arg);
    t.join();
#else
    thrd_t t;
    int res;
    thrd_create(&t, read_stat, arg);
    thrd_join(t, &res);
#endif
  }
}


-- System Information:
Debian Release: 11.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-9-amd64 (SMP w/16 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-image-amd64 depends on:
ii  linux-image-5.10.0-9-amd64  5.10.70-1

linux-image-amd64 recommends no packages.

linux-image-amd64 suggests no packages.

-- no debconf information