Re: Bug Report?! Part II.
On Wed, 21 Oct 1998, Helmut Metzdorf wrote:
> so i tried a closer look by starting top before firing up my programm.
> now things became somewhat clearer. after boot - i'm running xdm -
> when just the X server is running besides the other system-programms,
> top shows 18000K free memory (40M total).
> when i now fired up my own programm i could observe the figures
> representing the free memory dropping down slowly but steadily. when
> at least the numbers reach zero the system of course starts using
> swap and that is the point when the accompaniing io-demands seem to
> increase immense and the system is doing nearly nothing more than
> waiting for IO.
that sounds like a memory leak, OR a program which just needs more
memory than you have OR both.
> now, there's just one plausible explanation for that, the system is
> trying to cache all used files. ok: i quite can hear your comments
> from "did you expect somewhat else" to "i thought that was common
> knowledge" not to forget "and where, please are you considering a
yes, this is true. the linux kernel dynamically caches disk I/O. Any
memory not being used by a program can and will be used by the kernel as
disk cache/buffers, but when a program needs that memory, then it has
priority over the caching. linux's dynamic buffering has worked like
this for years (since before linux 1.0 i think).
the key point here is that programs ALWAYS have priority over caching
- if a program needs more memory then it is taken from the free pool
and/or the caching pool and given to that program. if necessary, idle
programs are paged out to increase the free memory pool.
(NOTE: this is a gross simplification of what actually happens but is
what this means is that your problem is likely to be caused by a memory
leak bug in your program - you are allocating memory but neglecting to
free it. in a short-lived program this isn't a big problem, but in a
long-lived program or a loop this can be catastrophic, rapidly using up
all available memory.
> i'll tell you: i consider it to be a bug that the system tries to
> cache even pure input files (InPut = fopen(XFileName, "r");) and i
> cant see the rationale behind it, especially if the reults can be such
> dissapointing. in this case it's the main input-file that is rather
> huge, making up more than 95 % off all memory used.
check your source code. i suspect that you will find a memory leak bug
in your code, and that the problem has nothing to do with the kernel's
you might want to use a debugging library like electric fence which
helps in tracking down memory allocation problems.
also, rethink your algorithm - do you REALLY need to have the entire
file loaded into memory at once, or can you process it line-by-line? if
you want to load the entire file into memory then OF COURSE it is going
to use memory, how can it not do so?
as far as i can tell from your comments, your problems could be solved
with some debugging and some memory-usage optimisation. consider the
resource constraints of the system you are working with (i.e. RAM +
swap) and write your code accordingly.