Re: Rotating logs (and keeping the log's tail)
On Thu, 2009-05-07 at 14:01 +0200, Sjoerd Hardeman wrote:
> Frank Lin PIAT wrote:
> > Hello,
> >
> > I use a wiki engine that produces an event log, which I want to rotate.
> >
> > Because the event-log is used to display the "PageHits", I don't want to
> > truncate the log every Monday. I wish the log could contain the last
> > week, plus the current week, so the statistics are never empty.
> >
> > The scenario would be:
> >
> > On The 2nd Monday: store the last line of the log (i.e "tag" it)
> >
> > On the next Mondays:
> > * Save the top of the file (above the tag)
> > * Delete the top of the current log.
> > * Update the tag (to the bottom of the log file).
> Can't you just write a script that updates the tag to a separate file,
> and use logrotate to do the normal logrotation?
I was hoping that someone already wrote that script ;-) Anyway, here it
is (for those who may need it):
#!/bin/sh
#
# rotate-and-keep.sh - Rotate a log file, keeping the last period.
# This script is useful when you want to rotate a file
# periodically, but you want the "current" log file to always
# contain two periods. (sometime needed for statistics)
# Copyright 2009 Franklin Piat <URL:http://www.klabs.be/~fpiat/>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
version=0.1
# LIMITATION
#
# Unique lines only
# Use this script on logfiles that don't produce duplicate line
# (i.e Each line MUST contain date and time, at least)
#
# Disk Space
# The partition must have free space to make a copy of the last logfile
#
# ,--[ KNOWN BUG / DISCLAIMER ]------------------------------------------.
# | |
# | The traditional way to rotate a log file is to move it. Which is |
# | guaranteed to be atomic and immediate. |
# | |
# | ** THIS SCRIPT ISN'T ATOMIC AND ISN'T IMMEDIATE ** |
# | You may lose some new lines, appended while the rotation occurs. |
# | (because it has to copy and manipulate the log file. Typically if |
# | you have more than 10 lines of log per seconds... that is > 100000 |
# | page hit per day!) |
# `----------------------------------------------------------------------'
LOGFILE=event-log
ARCHIVE=event-log.$(date +%F_%T)
TMPFILE=event-log.tmp
LASTLINE=lastline
set -e
pivotline=0
if [ -s "$LASTLINE" -a -s "$LOGFILE" ]; then
pivotline=$(grep -s -n -F -f $LASTLINE $LOGFILE | tail -n 1 | cut -d ":" -f 1)
#DEBUG: printf "Rotate line #$pivotline: " >&2 ; cat "$LASTLINE" > &2
fi
if [ "$pivotline" -gt 1 ] ; then
pivotline=$(( $pivotline + 1 ))
mv "$LOGFILE" "$TMPFILE"
if sed -n $pivotline,\$p "$TMPFILE" > "$ARCHIVE" ; then
cp "$ARCHIVE" "$TMPFILE"
## TEST: for test purpose, you can insert a sleep here to
## simulate the copy of large log file.
#sleep 1
# Try to catch new lines (that occured during the "cp" above)
if [ -s "$LOGFILE" ]; then
cat "$LOGFILE" 2>/dev/null | tee -a "$ARCHIVE" >> "$TMPFILE"
fi
mv "$TMPFILE" "$LOGFILE"
tail -n 1 "$ARCHIVE" > "$LASTLINE"
else
#Restore the logfile in case of error (especially disk-space!)
echo "LOG ROTATION ABORTED: probably insufficient disk-space." >&2
mv "$TMPFILE" "$LOGFILE"
exit 1
fi
else
if [ -s "$LOGFILE" ]; then
cp "$LOGFILE" "$ARCHIVE"
tail -n 1 "$ARCHIVE" > "$LASTLINE"
else
cp /dev/null "$LASTLINE"
fi
fi
#
# To test/stress this script you can generate a fake log using
# while true; do date +%s.%N | tee -a event-log >> log; sleep 0.01 ; done
##END##
Thanks,
Franklin
[Please CC me, I am not subscribed to this list]
Reply to: