[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Splitting Apache logs

I am in the middle of writing a program to split CLF format log files (as 
produced by Apache and most other web servers) based on the domain name of 
the server.  The idea is that log data for http://www.coker.com.au/ will go 
into the file coker.com.au and log data for http://www.workbenelux.nl/ will 
go into the file workbenelux.nl.
Before someone suggests making Apache log to multiple files, this doesn't 
work if you one of the following situations:
1)  More domains on the web server than Apache can have open file handles (IE 
more than 1020 domains for older kernels/libc/Apache or 60,000 to 80,000 
domains for if you are running the latest software).
2)  If you are dynamically adding domains in bulk, adding 100 domains a day 
would require 100 changes to config files and 100 restarts if you have a file 
per domain - this is not practical.

I plan to add my domain-based log splitting program to my logtools package in 
the near future.  I have attached the man page for what I plan to write.

I would like to receive suggestions both on functionality and on any issues 
regarding the documentation (if you don't understand the man page then let me 

Please respond to this message directly to me.  I am not subscribed to 

http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Attachment: clfdomainsplit.1
Description: clfdomainsplit.1

Reply to: