Resource limit question

[Martin Weinberg <weinberg@osprey.astro.umass.edu>]

I have set a hard limit on memoryuse to 10000 (10M).  limits (or
ulimit -a) shows the limit but, it doesn't seem to be working.
For example, with the limit set the following bit of code

============================================================
/* Name: tst.c */

#include <sys/time.h>
#include <sys/resource.h>
#include <unistd.h>
#include <stdio.h>

static char x[20*1024*1024];

int main()
{
  int i;
  struct rlimit limit;

  getrlimit(RLIMIT_RSS, &limit);
  printf("RSS limit curr, max: %d, %d\n", limit.rlim_cur, limit.rlim_max);

  if (! malloc(20000000))
    printf("malloc not successfull - good\n");

  for (i=0; i < sizeof(x); i+=4096)
    x[i] = 1;
  
  system("ps gauxww | grep a.out | grep -v grep");
  
  exit(0);
}
============================================================

compiled as follows:

	 cc tst.c

give me the output:

RSS limit curr, max: 10240000, 10240000
weinberg 25601 11.0 16.2 41000 20836 pts/1   S    16:25   0:00 ./a.out

Note the RSS limit of 10M.  But the code as a data size of
20M to start and mallocs an additional 20M.  What am I missing?


P.S. This is a pure potato system.  The limit itself is set by
pam_limits from the limits.conf file.  I assume that this part
is fine because the "limits" or "ulimit -a" correctly reports
the values set in limits.conf and /var/log/auth.log reports that
pam has read and set the value (with the debug flag set).