[RESOLVED] Disk quota exceeded ... but no quota are defined.

The server is a VPS (Virtual Private Server) one.
Indeed the inodes were full.

sudo df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 500000 500000 100% /
..etc..

So filesystem needs some files cleanup...
I identified which directory was full of files with following command executed from root location /:

/: sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

After few iterations I found the directory : /usr/local/pnp4nagios/var/spool which contains more than 300.000 files.
Need to work on this later, but to resolve the current issue I stopped nagios and delete its spool directory.
Then everything is up & running OK.

sudo df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 167715 332285 34% /
..etc..

Thanks for your attention.
Bruno

On Tue, Mar 24, 2015 at 12:07 PM, Darac Marjal <mailinglist@darac.org.uk> wrote:

On Tue, Mar 24, 2015 at 05:05:46AM +0100, Bruno Costacurta wrote:
> Hello to All,
>
> my debian server is freezed with quota exceeded but there is no quota
> installed on it.
>
> Debian version is :
> uname -a
> Linux vps622 2.6.32-042stab103.6 #1 SMP Wed Jan 21 13:07:39 MSK 2015 i686
> GNU/Linux
>
> No quota are setup and disk is not full as only 3% of usage are reported :
>
> sudo df -h
>
> Filesystem      Size Used Avail Use% Mounted on
> /dev/simfs      160G 3.8G 157G   3% /
> tmpfs           205M   72K 205M   1% /run
> tmpfs           5.0M     0 5.0M   0% /run/lock
> tmpfs           410M     0 410M   0% /run/shm
>
> mount does not indicate quota setup :
>
> /vz/private/622 on / type simfs (rw,relatime)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=209716k,mode=755)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /run/shm type tmpfs
> (rw,nosuid,nodev,noexec,relatime,size=419420k)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> sudo du -ch /run
> 72K    /run
> 72K    total
>
> But all services are freezed with quota exeeded errors.
> when I log :
> zsh: locking failed for /home/bruno/.histfile: disk quota exceeded:
> reading anyway
>
> Or in postfix logs :
> Mar 24 04:44:52 vps622 postfix/cleanup[2727]: warning: mail_queue_enter:
> create file incoming/181079.2727: Disk quota exceeded
> ..etc..
>
> Also MySQL also cannot be started.
> Once rebooted the server is fine only few minutes before facing quota
> exceeded.
> I cannot find any quota setup.
> And indeed there is no quota packages installed neither.
> dpkg -l | grep quota
> ---> nothing.
>
> How to check this quota problem ?
> Thanks for any help or clue.

I don't know much about OpenVZ, but a quick bit of searching tells me
that your container has run out of space/inodes quota. That is, although
the operating system within the container can see plenty of space, the
supervising operating system is refusing to allow more allocation and
this is filtering down to the guest.

Apparently, running something like "vzctl set <VZID> --save -diskinodes
xxxxx:yyyyy" might help.

>
> Bye
> Bruno

Yes the server is a VPS (Virtual Private Server) one.
Indeed the inodes were full.

sudo df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 500000 500000 100% /
..etc..

So filesystem needs some files cleanup...
I identified which directory was full of files with following command executed from root location /:

/: sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

After few iterations I found the directory : /usr/local/pnp4nagios/var/spool which contains more than 300.000 files.
Need to work on this later, but to resolve the current issue I stopped nagios and delete its spool directory.
Then everything is up & running OK.

sudo df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 167715 332285 34% /
..etc..

Thanks for your attention.
Bruno