A full /var partition destroyed 3 hours of my life!
I tried booting up into Debian and got all sorts of systemd breakages
apparently because my /var partition was full. That's fair, but the
pain started when Debian frustrated any attempt to free up space. I'm
wondering if this is a 'feature' that needs removing or if there might
be a bug in the underlying filesystem. I really don't want anymore
finite hours of my life (or anyone else's) lost in this problem if I
can find the cause.
One of the culprits in my full /var partition was a 3 gig syslog file
which has only been getting bigger since January despite running
logrotate -f. I try to run it this time but I'm told that it can't
rotate anything because there's no space left of the device. OK, Plan
B. Another thing that the interwebs say is to run apt-get clean to
sweep out downloaded packages, of which I collected hundreds of
megabytes. Again, this command failed because there was no space left
on the device.
Since there's almost no documentation as to what can be safely rm'd in
/var without breaking your system, I decide the least risky choice is
to sudo rm -rf the offending 3-gig syslog file from single-user mode
and the systemd debug shell. But *THIS* command failed because there
was 'no space left on the device'. Is this right? Does rm need space
on a drive to free other space? If so, how on earth can you fix a full
partition if you can't remove anything from it?!
Since Debian can't delete files from its own partitions, I have to
boot from a Ubuntu DVD. I'm able to rm -rf the syslog file from that,
but when I reboot into Debian, I get the same 'no space left on
device' errors. That's weird, so I df -h to figure out what's going on
and df correctly reports a 5G var partition, of which under 3G are now
used and avail space is 0G. Whoa, wait, what?!?! How can 5G - 3G =
0G?!
I start blindly casting whatever btrfs spells I can find on the
Internet to fix 'no space left on device' errors. One of them
eventually works and df -h correctly reports the free space in my /var
partition and Debian boots normally again.
My question, therefore, is whether this is a btrfs bug that got
triggered by the full /var partition or whether Debian is designed to
break irrecoverably when /var fills up. Any ideas of what happened?
Reply to: