Re: system gobbles disk space

[<tomas@tuxteam.de>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Sep 27, 2016 at 10:55:52AM +0200, Tony van der Hoff wrote:
> On 26/09/16 17:04, tomas@tuxteam.de wrote:
> > On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
> >> On 26/09/16 16:03, tomas@tuxteam.de wrote:
> >>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
> >>>
> >>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
> >>>
> 
> Thanks again to all who have helped with this.
> 
> After the overnight run, I'm now seeing this:

[...]

Bad Apache, bad :-)

> So, I can see this growing to an impossible size, which is probably
> where my problem lies.
> 
> These files are created by a apache-driven PERL application, and the log
> files are renamed/compressed overnight by a cron job.

Seems this "cron job" is doing logrotate's job (or I am misunderstanding).

> This setup has existed for a number of years unchanged, so what's caused
> it to start misbehaving?
> 
> How to stop it happening?
> Should I be stopping Apache while compressing the logs?

There is a way to tell long-running processes to close their log files
and then re-open them (typically sending a HUP or USR1 signal, this has
established itself as a kind of "standard protocol"). This is usually
part of the logrotate process. You have logrotate to take care of your
log files, have you?

This is an example off logrotate's man page:

       /var/log/messages {
           rotate 5
           weekly
           postrotate
               /usr/bin/killall -HUP syslogd
           endscript
       }

Meaning "take care of log messages in this-and-that way and *after* that
send a HUP signal to syslogd". You should have a similar entry for your
Apache in /etc/logrotate.d or thereabouts.

If your Apache's log files aren't managed by logrotate (someone *is*
moving them around, after all) then go find out what Apache needs to
release its grip on log files and arrange for that to happen after
they have been moved around.

If I read [1] correctly, the Apache folks recommend a graceful restart
of the server after log rotation.

[1] https://httpd.apache.org/docs/2.4/logs.html#rotation

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfqU20ACgkQBcgs9XrR2kYgiQCeM0oR8IdvvzWce7ey/0LXeTIY
zOMAniQRK/N43LW2sNCp34aiboJToVa8
=4ccB
-----END PGP SIGNATURE-----