Bug#1088964: libc6: system() shall return -1 if a child process cannot be created

[Arnaud Giersch <arnaud.giersch@univ-fcomte.fr>]

Package: libc6
Version: 2.36-9+deb12u9
Severity: normal

Dear Maintainer,

The system() library function gives the wrong result when a child
process cannot be created.

According to the man page system(3posix):
"If a child process cannot be created, [...], system() shall return -1 and
 set errno to indicate the error."

Actually (libc6 2.36), it returns 32512 instead of -1.

The following code can be used as a test case:

/*=== cut here ===*/
#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>

int main(void)
{
    int a = system("true");

    // force failure for the next call
    const struct rlimit zerorl = {0, 0};
    if (setrlimit(RLIMIT_NPROC, &zerorl) == -1)
        perror("setrlimit(RLIMIT_CORE)");

    int b = system("true");
    printf("> success:%d failure:%d (%#x)\n", a, b, b);
}
/*=== cut here ===*/

Returns:
> success:0 failure:32512 (0x7f00)

Expected result:
> success:0 failure:-1 (0xffffffff)

Regards,
Arnaud Giersch


-- System Information:
Debian Release: 12.8
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-security'), (500, 'stable-debug'), (500, 'proposed-updates-debug'), (500, 'proposed-updates'), (500, 'stable'), (50, 'unstable'), (40, 'experimental'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.1.0-27-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libc6 depends on:
ii  libgcc-s1  12.2.0-14

Versions of packages libc6 recommends:
ii  libidn2-0  2.3.3-1+b1

Versions of packages libc6 suggests:
ii  debconf [debconf-2.0]  1.5.82
ii  glibc-doc              2.36-9+deb12u9
ii  libc-l10n              2.36-9+deb12u9
ii  libnss-nis             3.1-4
ii  libnss-nisplus         1.3-4
ii  locales                2.36-9+deb12u9

-- debconf-show failed