Re: 'out of memory' errors

[tom arnall <kloro@cox.net>]

On Saturday 29 April 2006 04:22 pm, Linas ?virblis wrote:
> tom arnall wrote:
> > I am trying to us a perl application that handles large files (.5GB) in
> > scalar variables.  When I try to  'slurp' one of these files into a
> > variable (e.g., '$_ = `cat filename`) I get an out of memory error.
>
> You should consider alternative solutions, like processing files line by
> line for example. It highly depends on what you are trying to achieve,
> but loading half a gig into memory is not something you should do
> without having a very good reason for doing so.

Understood, but I am doing a regex on text units which span more than one 
line. Trying to process this text w' line-by-line io would be possible but, 
as far as I can see, very awkward.

>
> > is there not some way to get linux to use swap space when physical memory
> > is exceeded?
>
> It does that by default, assuming you do have swap partitions or files
> available. Note that this can be slow, and I assume this is what you
> described as a "freeze".

Looking at 'top' and the 'MEM%' column, it seems to me that the system is 
finally swapping, but only after allowing the perl applicationg to hog all 
available memory, which then of course has everyone else swapping and 
therefore grinding to a virtual halt. What I want is for system to be 
swapping mostly for just the perl application. 


> There are many tunable Linux virtual memory parameters you can set via
> sysctl interface. You can get a list of them with "sysctl -a | grep vm".
> The most useful one is vm.swappiness. You can set it to a number from 0
> to 100. 0 means "try not to use swap at all" and 100 - "swap as much as
> possible" with all other values in between. You can set it (and other
> sysctl options) like this:
>
>  # sysctl -w vm.swappiness=60
>

I tried setting it to 100, but i get an 'out of memory' error anyway. But 
shouldn't 'limits.conf' also be tuned to deal with the situation? My current 
settings:

	# /etc/security/limits.conf
	kloro	hard   	rss	   	256000
	kloro	hard   	memlock     256000
	kloro	hard   	as	       	768000

The 'as' parameter, as set above, causes an 'out of memory' error when the 
perl appl' runs. If I make the 'as' parameter to be very big (e.g., 2G), the 
system allows the perl appl' to run but also to hog memory. To my way of 
thinking, there should be a way to put a limit on the physical memory an 
appl' is allowed to use, and to cause swap space to be used when that limit 
is reached, i.e., a way to put a limit on the physical memory which the 
individual application is allowed to use. 

Thanks,

Tom Arnall
north spit, ca