[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Q: 'df' reports partition is full but 'du' shows not full partition -- how to find where the problem

Snow Leopard wrote:
> You have a web server and monitoring system reports that you short on disk
> space. You login into web server computer and command 'df' indeed shows that
> some partition is almost full.
> How would you resolve the issue if 'du' does not confirm that disk is full.
> You 'du' all directories in the root of the partition but sum does not adds
> up to match 'Use%' reported by 'df'.

I read your case and it reminds me immediately of many questions that
often get asked on the coreutils mailing list.  This is one of the
FAQs there.  Review this entry and the next one after it and verify
that neither case applies to your case.


> For such situation I would assume that some process holds huge open file
> which was deleted (for example your web server was compromised).

Hmm...  Possible.  But I think less likely.  If your web server is
running as www-data then it is non-priviledged and the attack will be
limited in what it can do.  Removing a file it has permissions to
remove is one but I think it unlikely this to be large enough to be a
problem.  I am often wrong but I think this is unlikely.

The most often problem with deleted large files still holding disk
space are log files.  A process writes a log file.  The admin has
configured it to be in a custom location and has forgotten to set up a
lotrotate rule for it.  The log file grows and grows.  The local admin
then sees this and thinks, aha, there is the disk space, and uses 'rm'
to remove the file not realizing that the program is still writing the
file and will continue to do so.  In hindsight the admin should
truncate the file first to force the disk space to be freed

  # > /var/log/largelogfile

By truncating the large file it doesn't matter if it is open and being
written to by other processes.  The open(2) with O_TRUNC will zero it

And of course to keep this from happening again something like
logrotate or savelog needs to be set up for routine rotation of the
log file.

> Question: how would you find the process which holds 'deleted' file?

As Stack suggested lsof works well.


Attachment: signature.asc
Description: Digital signature

Reply to: