Bug#707770: sosospider+gitweb caused apache memory use to balloon and not go back down

[Stefan Fritsch <sf@sfritsch.de>]

On Saturday 11 May 2013, Joey Hess wrote:
> I don't understand why apache is using all that memory. Could it be
> trying to buffer the cgi's output? If apache mallocs a lot of
> memory for such a buffer, will it ever free it? Perhaps sosospider
> is doing additional evil things beyond ignoring robots.txt, that
> cause this behavior.

Apache 2.2 has some problems with generators like cgi or php scripts 
generating large content slowly. It will then allocate lots of memory 
management structs and only free them after the request is finished. 
And freeing them does only mean that it will re-use the memory later, 
not that it gives the memory back to the OS.

The latter issue can be fixed by specifying something like "MaxMemFree 
2048" in the configuration. I am not sure how much that buys you if 
the spider does lots of parallel requests of this type, though.

The former issue has been fixed in 2.4, but the fixes have not been 
backported so far. In 2.4, MaxMemFree no longer defaults to 
'unlimited', either.