Re: apache & log files

[Craig Sanders <cas@taz.net.au>]

On Wed, Nov 03, 2004 at 11:11:13PM +0100, Marek Podmaka wrote:
>   I have apache 1.3 webserver hosting about 150 domains (more than 400
>   virtual hosts). Now I have separate error log for each domain
>   (something.sk) and separate combined log for each virtual host (for
>   example www.abcq.sk and new.abcq.sk). This has many positives for
>   me: easy to find some data related to each virtual host and that I
>   can make seaparate statistics for each virtual host. I use awstats.
>   And now the bad side - the number of open files in each apache
>   process is more than 500 just for these log files. It's no problem
>   for now, but with more domains in future it will hit the 1024 per
>   process limit of open files.
> 
>   And now the questions :)
>   1) Where does that 1024 open files limit come from? Is it somewhere
>   configurable? 

edit /etc/init.d/apache and add a line like "ulimit -n 4096"

"-n" is the maximum number of open file descriptors per process.  use a value
that's about twice as much as you think you'll need.

you also need to set /proc/sys/fs/file-max to some suitably high value (again,
calculate how many you think you'll need and double it).  this can be set in
/etc/sysctl.conf


> Or do you think it's totally bad idea to have such number of log
> files?

until recently, i ran a web server with about 600 virtual hosts on it, each
with its own access.log and error.log files.

with 200 max apache children, that worked out as up to about 240,000 (600 x 200
x 2) file handles opened by apache processes for logging at any given time.
this was on a dual p3-866 with 512MB RAM.   it worked.

it bothered me a little that it wasn't really scalable and that eventually i'd
have to do something about logging.  i had some ideas on what to do, but was
limited by the fact that i wanted to have separate error.log files for each
virtual host.  overall, though, my attitude was "it aint broke, so don't fix
it".

this wouldn't be a problem if apache could be configured to prefix each
error.log line with the virtual host's domain name......then you could have a
single pipe to an error logging script which wrote the error lines to the right
file, same as you can do for the access.log.

but apache can't be configured to do that, and i never bothered looking at the
source to see how easy it would be to hack it in, so that means you either have
a shared error.log for all vhosts or you put up with having lots of open file
handles.  i chose the latter, and occasionally increased both "ulimit -n" and
/proc/sys/fs/file-max as required....i never did run into any limit.


craig

-- 
craig sanders <cas@taz.net.au>           (part time cyborg)