error running shared postrotate script for /var/log/apache/*.log

["David Christensen" <dpchrist@holgerdanske.com>]

debian-user:

I have a Debian 3.0r2 box that I use as primarily as a CVS server, and
secondarily for messing around with Linux-Apache-MySQL-Perl development.  It has
been running smoothly for many months.  cron started sending me the following
e-mails a few days ago:

    /etc/cron.daily/logrotate:
    error running shared postrotate script for /var/log/apache/*.log 
    run-parts: /etc/cron.daily/logrotate exited with return code 1


I tried RTFM and STFW, but didn't find any direct answers.  So, I beat my head
against it and determined that the problem is that I removed the entry for my
host's name in /etc/hosts while rearranging my network (right about the same
time logrotate broke; imagine that).  Putting the entry back seems to have fixed
things.


My troubleshooting log follows FWIW.


Here is my /etc/cron.daily/logrotate:

    root@p166d302:~# cat /etc/cron.daily/logrotate
    #!/bin/sh

    /usr/sbin/logrotate /etc/logrotate.conf


This should be the relevant portions of /etc/logrotate.conf:

    root@p166d302:~# grep -v '^#' /etc/logrotate.conf | grep -v '^$'
    weekly
    rotate 4
    create
    include /etc/logrotate.d
    /var/log/wtmp {
        monthly
        create 0664 root utmp
        rotate 1
    }
    /var/log/btmp {
        missingok
        monthly
        create 0664 root utmp
        rotate 1
    }

Here is my /etc/logrotate.d:

    root@p166d302:~# l /etc/logrotate.d
    ./  ../  apache  base-config


Here is /etc/logrotate.d/apache:

    root@p166d302:~# cat /etc/logrotate.d/apache
    /var/log/apache/*.log {
            weekly
            missingok
            rotate 52
            compress
            delaycompress
            notifempty
            create 640 root adm
            sharedscripts
            postrotate
                    /etc/init.d/apache reload > /dev/null
            endscript
    }


Here is my /var/log/apache/*.log:

    root@p166d302:~# l /var/log/apache/
    ./               error.log.18.gz  error.log.31.gz  error.log.45.gz
    ../              error.log.19.gz  error.log.32.gz  error.log.46.gz
    access.log       error.log.2.gz   error.log.33.gz  error.log.47.gz
    access.log.1     error.log.20.gz  error.log.34.gz  error.log.48.gz
    access.log.2.gz  error.log.21.gz  error.log.35.gz  error.log.49.gz
    error.log        error.log.22.gz  error.log.36.gz  error.log.5.gz
    error.log.1      error.log.23.gz  error.log.37.gz  error.log.50.gz
    error.log.10.gz  error.log.24.gz  error.log.38.gz  error.log.51.gz
    error.log.11.gz  error.log.25.gz  error.log.39.gz  error.log.52.gz
    error.log.12.gz  error.log.26.gz  error.log.4.gz   error.log.6.gz
    error.log.13.gz  error.log.27.gz  error.log.40.gz  error.log.7.gz
    error.log.14.gz  error.log.28.gz  error.log.41.gz  error.log.8.gz
    error.log.15.gz  error.log.29.gz  error.log.42.gz  error.log.9.gz
    error.log.16.gz  error.log.3.gz   error.log.43.gz
    error.log.17.gz  error.log.30.gz  error.log.44.gz


Perhaps logrotate is barfing because 52 copies of error.log exist?


I'll try running logrotate from the command line:

    root@p166d302:~# /usr/sbin/logrotate /etc/logrotate.conf
    error running shared postrotate script for /var/log/apache/*.log


I think it's complaining about the "/etc/init.d/apache reload > /dev/null" in
/etc/logrotate.d/apache, so I'll try running that:

    root@p166d302:~# /etc/init.d/apache reload
    Reloading apache configurationNo /usr/sbin/apache found running; non
e killed.
    failed


I think it's complaining that apache is not running.  I'll try browsing to that
box -- nope; apache is down.  Try starting it:

    root@p166d302:~# apachectl start
    /usr/sbin/apachectl start: httpd could not be started


Nope.  Check the config file:

    root@p166d302:~# apachectl configtest
    Syntax OK


No clues there.  The relevant portions of /usr/sbin/apachectl would seem to be:

    HTTPD='/usr/sbin/apache'
    ...
            fi
            if $HTTPD ; then
                echo "$0 $ARG: httpd started"
            else
                echo "$0 $ARG: httpd could not be started"
                ERROR=3
            fi

Trying to invoke /usr/sbin/apache directly:

    root@p166d302:~# /usr/sbin/apache


No complaints, but my browser indicates it isn't running.  Same goes for ps:

    root@p166d302:~# ps -A
      PID TTY          TIME CMD
        1 ?        00:00:07 init
        2 ?        00:00:00 kflushd
        3 ?        00:00:00 kupdate
        4 ?        00:00:07 kswapd
        5 ?        00:00:00 keventd
      100 ?        00:00:00 portmap
      160 ?        00:00:00 syslogd
      163 ?        00:00:00 klogd
      168 ?        00:00:00 rpc.statd
      178 ?        00:00:00 inetd
      182 ?        00:00:00 lpd
      191 ?        00:00:00 sshd
      194 ?        00:00:00 ntpd
      197 ?        00:00:00 atd
      200 ?        00:00:00 cron
      206 tty1     00:00:00 getty
      207 tty2     00:00:00 getty
      208 tty3     00:00:00 getty
      209 tty4     00:00:00 getty
      210 tty5     00:00:00 getty
      211 tty6     00:00:00 getty
     1060 ?        00:00:00 sshd
     1062 ?        00:00:01 sshd
     1063 pts/0    00:00:00 bash
     1064 pts/0    00:00:00 bash
     1081 ?        00:00:00 sshd
     1083 ?        00:00:00 sshd
     1084 pts/1    00:00:00 bash
     1085 pts/1    00:00:00 bash
     1172 pts/1    00:00:00 man
     1176 pts/1    00:00:00 pager
     1212 pts/0    00:00:00 ps


Try deleting the 52nd error.log file and starting apache:

    root@p166d302:~# rm -f /var/log/apache/error.log.52.gz

    root@p166d302:~# apachectl start
    /usr/sbin/apachectl start: httpd could not be started


Nope.  RTFM apache doesn't seem to have a "verbose" option.  Look at the end of
Apache's error log:

    root@p166d302:~# tail /var/log/apache/error.log
    [Mon Apr 11 19:34:21 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")
    [Mon Apr 11 19:39:29 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")
    [Mon Apr 11 19:39:38 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")
    [Mon Apr 11 19:46:37 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")
    [Mon Apr 11 19:47:17 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")
    [Mon Apr 11 19:55:55 2005] [alert] mod_unique_id: unable to
gethostbyname("p166d302")


There's a clue.  Here's my /etc/hosts:

    root@p166d302:~# cat /etc/hosts
    127.0.0.1       localhost

    # The following lines are desirable for IPv6 capable hosts
    # (added automatically by netbase upgrade)

    ::1     ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts


I seem to be missing an entry for the hostname (p166d302; did I remove that?).
Add it and try again:

    root@p166d302:~# vi /etc/hosts

    root@p166d302:~# cat /etc/hosts
    127.0.0.1       localhost
    192.168.254.2   p166d302.holgerdanske.com p166d302
    ...

    root@p166d302:~# apachectl start
    /usr/sbin/apachectl start: httpd started


D'Oh!  [sound of hand slapping forehead]  I removed that entry while rebuilding
my firewall/ router/ DNS cache machine, thinking I could put it there.


Make sure logrotate is happy:

    root@p166d302:~# /usr/sbin/logrotate -f /etc/logrotate.conf


Okay.  Check with browser -- okay.


HTH,

David