Apache 2.4 hits RLIMIT_NPROC: hidden processes?

[Jeroen Ooms <jeroen.ooms@stat.ucla.edu>]

My webapp allows users to execute some arbitrary code in a sandbox. To
prevent forkbombs, the application calls [setrlimit][1] and limits
`RLIMIT_NPROC` to 50 before executing user code. This worked great in
Ubuntu 12.04 up till Ubuntu 13.04. However, after upgrading to Ubuntu
13.10 (which ships with Apache 2.4 and Linux 3.11), we hit the limit
of 50 `www-data` processes, even when Apache2 is idle!

The problem is most easily reproduced by running `bash` as user
`www-data` with `ulimit`. First switch into user `www-data` and start
`bash`:

    jeroen@Ubuntu:/$ sudo su www-data
    $ bash
    www-data@Ubuntu:/$

Now gradually lower `RLIMIT_NPROC` until we hit problems:

#RLIMIT_NPROC=100: works fine
www-data@Ubuntu:/$ ulimit -u 100
www-data@Ubuntu:/$ ls
bin    dev   initrd.img      lib64 mnt   root  srv  usr   vmlinuz.old
boot   etc   initrd.img.old  lost+found  opt   run   sys  var
cdrom  home  lib     media proc  sbin  tmp  vmlinuz

#RLIMIT_NPROC=50: limit reached
www-data@Ubuntu:/$ ulimit -u 50
www-data@Ubuntu:/$ ls
bash: fork: retry: No child processes
bash: fork: retry: No child processes
bash: fork: Resource temporarily unavailable

Hence after setting `RLIMIT_NPROC` to 50, the process can no longer
fork. This implies that there are already 50 or more processes running
as user `www-data`. However, this does not seem to be the case, the
server is just a blank idle Apache 2.4. According to `ps`, there are
currently only 2 procs owned by `www-data`:

jeroen@Ubuntu:~$ ps aux | grep www-data
www-data 11473  0.0  0.5 631296 46164 ?        Sl   14:28   0:01
/usr/sbin/apache2 -k start
www-data 11474  0.0  0.5 565656 45632 ?        Sl   14:28   0:01
/usr/sbin/apache2 -k start
jeroen   12136  0.0  0.0  13644   956 pts/4    S+   14:51   0:00 grep
--color=auto www-data


So why is `www-data` is hitting the `RLIMIT_NPROC` limit of 50 in
Apache 2.4, even when idle?


  [1]: http://linux.die.net/man/2/setrlimit