Re: system gobbles disk space

[<tomas@tuxteam.de>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Sep 26, 2016 at 11:42:41AM +0200, Tony van der Hoff wrote:
> Hi,
> 
> For the last few days, my jessie VM has apparently run out of disk space
> overnight. df shows 100% usage on / but du does not show any enormous files.
> 
> If I reboot, the disk usage goes back to 50% on a 40GB volume.
> 
> How can I determine what is eating the space?

Hm. Tough question. One traditional culprit for that kind of behaviour is
a file which has been deleted (thus having no entry in a directory in the
file system) but which is kept open by some process. One way to find that
would be to look into the /proc file system. Each process there (represented
by a directory named after the PID, the process ID) has a subdirectory called
'fd' containing a symlink to each file the process has open. Look for dangling
symlinks and ask yourself in each case whether the file behind this symlink
migh be responsible for the missing space.

A practical example:

First, I create a process which keeps a file open, writing into it from
time to time:

 | tomas@rasputin:~$ ( while true ; do echo "bip"; sleep 5 ; done ) > /tmp/logfile &
 | [1] 3642

This process has the PID 3642. In /proc/3642/fd, I find an entry for this
file:

 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

Note that this is linked from the file named '1', which is the process's
standard output, as one would expect from the shell jargon '>' (redirect
stdout).

Now I get nasty and remove /tmp/logfile. Note that this doesn't remove
the file itself -- just that one directory entry (the file might have
more than one: it gets removed whenever nobody needs it). The process
continues running as if nothing happened. The file keeps growing (albeit
slowly in my case) and using up space:

 | tomas@rasputin:~$ rm /tmp/logfile
 | tomas@rasputin:~$ ls -l /proc/3642/fd
 | total 0
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 0 -> /dev/pts/3
 | l-wx------ 1 tomas tomas 64 Sep 26 11:53 1 -> /tmp/logfile (deleted)
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 2 -> /dev/pts/3
 | lrwx------ 1 tomas tomas 64 Sep 26 11:53 255 -> /dev/pts/3

See? Link -> is now marked (deleted). In my version of ls and console
it's printed now in a happy red colour. I can end the nonsense by
killing the process:

 | tomas@rasputin:~$ fg
 | ( while true; do
 |     echo "bip"; sleep 5;
 | done ) > /tmp/logfile
 | ^C

Of course, you'll have to become root to be able to move around
other process's entrails this way. In my small example I just
created the process, that made it easy.

There are other ways of making "disk space disappear", among others
mounting something over a non-empty directory.

Regards
- -- t
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfo8zsACgkQBcgs9XrR2kakRgCfT0BNO3HAjTdYqb17vQXU6o1e
RdQAnimGR73J4lmXH3/4rtYoXDvrkc7u
=n1gG
-----END PGP SIGNATURE-----