Re: A full /var partition destroyed 3 hours of my life!

[Borden Rhodes <jrvp@bordenrhodes.com>]

> [rsyslog maintainer speaking here]
>
> Am 15.11.2016 um 06:00 schrieb Borden Rhodes:
>> One of the culprits in my full /var partition was a 3 gig syslog file
>> which has only been getting bigger since January despite running
>> logrotate -f. I try to run it this time but I'm told that it can't
>
> I'd be interested to find out, why logrotation was not done
> automatically. Do you have cron installed and running?
> Do you have  /etc/cron.daily/logrotate which works when executed and a
> corresponding /etc/logrotate.d/rsyslog?
>
> Any idea why logrotate was not run or failed to do its job?

Here's the contents of /etc/cron.daily/logrotate:
#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

and /etc/logrotate.d/rsyslog:
/var/log/syslog
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog rotate > /dev/null
        endscript
}

Both looked normal to me and, without knowing more about the structure
of logrotate config files, didn't pick further. When I logrotate -f ,
it runs and finishes without complaining, but syslog doesn't seem to
get smaller. I think it just kept getting bigger.

>> My question, therefore, is whether this is a btrfs bug that got
>> triggered by the full /var partition or whether Debian is designed to
>> break irrecoverably when /var fills up. Any ideas of what happened?
>>=20
>
> That sounds like a btrfs issue. Which kernel is that?
> I do remember btrfs having problems when the disk runs full.

I'm running a 4.8.0-1-amd64 kernel. I'm on the testing branch. It
makes me feel better knowing that it may be a btrfs bug (or at least
not part of the Linux design) since that's a rough edge I can (try to)
work around by checking /var every so often. Still, "A Cowboy's Guide
to Cleaning /var and /tmp" would help in cases where some process gets
greedy with space.

>> My question, therefore, is whether this is a btrfs bug that got
>> triggered by the full /var partition or whether Debian is designed to
>> break irrecoverably when /var fills up. Any ideas of what happened?
>>
>
> Does anything on the Debian Wiki on Btrfs [1] seem familiar? Other than
> that I can only guess, but maybe check the SMART information of your
> disk(s) for excessive errors, as it _could_ be that defective sectors
> prevent Btrfs from doing it's COW magic.

I don't think it's that, unless smartctl is lying to me. It passes all
of the test and the only historical failure (which I think has almost
always been there) is an airflow warning. Error logs are empty. If I
start getting strange behaviour, I can do a more comprehensive SMART
scan.

> [1] https://wiki.debian.org/Btrfs#WARNINGS

Nothing seems on point here. My configuration is btrfs partitions
within an LVM within an MBR hard drive. I'm not doing any fancy RAID
or anything.

Thank you for the hints!