Re: system gobbles disk space
On 26/09/16 17:04, tomas@tuxteam.de wrote:
> On Mon, Sep 26, 2016 at 04:38:01PM +0200, Tony van der Hoff wrote:
>> On 26/09/16 16:03, tomas@tuxteam.de wrote:
>>> On Mon, Sep 26, 2016 at 12:54:49PM +0200, Erwan David wrote:
>>>
>>>> A possibility is that you have processes writing into deleted files. You can see them with lsof +L1 (as root)
>>>
Thanks again to all who have helped with this.
After the overnight run, I'm now seeing this:
root@shell:~# lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
mysqld 3164 mysql 4u REG 254,16 0 0 589839
/tmp/ibvh7MKT (deleted)
mysqld 3164 mysql 5u REG 254,16 473 0 589840
/tmp/ibT7D133 (deleted)
mysqld 3164 mysql 6u REG 254,16 0 0 589845
/tmp/ibO9pgne (deleted)
mysqld 3164 mysql 7u REG 254,16 0 0 589851
/tmp/ibOgMl6y (deleted)
mysqld 3164 mysql 11u REG 254,16 0 0 589852
/tmp/ibON9JEJ (deleted)
/usr/sbin 4643 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 4643 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 4643 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 4643 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 4643 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 4643 www-data 38w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 38w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13918 www-data 39w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 13920 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 13920 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 13920 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 13920 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 38w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 39w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 13920 www-data 40w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 33w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 34w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 38w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 15766 www-data 39w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 29w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16711 www-data 30w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16711 www-data 31w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16711 www-data 32w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16711 www-data 33w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 34w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16711 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16768 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16768 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16768 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16768 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 37w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16768 www-data 38w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16770 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16770 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16770 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16770 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16770 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 16798 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 16798 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 16798 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 16798 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 16798 www-data 36w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data 30w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18785 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18785 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18785 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18785 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
/usr/sbin 18785 www-data 35w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data 2w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data 29w REG 254,16 22055 0 2180962
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/error.log_20160927
(deleted)
/usr/sbin 18929 www-data 31w REG 254,16 0 0 2180306
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/root.log_20160927
(deleted)
/usr/sbin 18929 www-data 32w REG 254,16 0 0 2180396
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/application.log_20160927
(deleted)
/usr/sbin 18929 www-data 33w REG 254,16 0 0 2180397
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/lib.log_20160927
(deleted)
/usr/sbin 18929 www-data 34w REG 254,16 10374 0 2180686
/home/vanderhoff/webs/www.rutland-sailing.org.uk/htdocs/rutland_members/log/audit.log_20160927
(deleted)
So, I can see this growing to an impossible size, which is probably
where my problem lies.
These files are created by a apache-driven PERL application, and the log
files are renamed/compressed overnight by a cron job.
This setup has existed for a number of years unchanged, so what's caused
it to start misbehaving?
How to stop it happening?
Should I be stopping Apache while compressing the logs?
--
Tony van der Hoff | mailto:tony@vanderhoff.org
Ariège, France |
Reply to: