Bug#787807: libc6: mq_open unexpectedly fails with errno 24

[Tim Ruehsen <tim.ruehsen@gmx.de>]

Package: libc6
Severity: normal

Dear Maintainer,

I have an application that uses Posix message queues since a while.
Today I realized that mq_open fails with errno 24 where it worked before.
(errno 24: Too many open files)

Here is a short C file to reproduce the problem.
Before you start it, you have to increase msg_max system default:
sysctl -w fs.mqueue.msg_max=1000

Now (as normal user) execute the following code:
#include <stdio.h>
#include <mqueue.h>
#include <errno.h>

void main(void) {
  struct mq_attr attr={.mq_maxmsg=900,.mq_msgsize=1020};
  int rc=mq_open("/xxx",O_CREAT | O_RDONLY, 0600, &attr);
  printf("rc=%d errno=%d\n",rc,errno);
}

It returns errno 24. Using a value of 800 for .mq_maxmsg returns success.
Compile with gcc x.c -o x -lrt.

The system settings should allow the above values without problem:
$ ls /proc/sys/fs/mqueue/|more
msg_default
msg_max
msgsize_default
msgsize_max
queues_max

$ cat /proc/sys/fs/mqueue/*
10
1000
8192
8192
256

Regards, Tim


-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)