Disk space issues but 10To available

[Andrimont]

I noticed that all the VM start to have issues concerning missing disk space. Even the Shell in the main SD server became unusable because of insufficient disk space. Still there is more than 10To available on the disks !
So far what could be the reason ?
This happened several time. I could get control of the server after freeing some disk space (removing some back-up). But the issue comes back regularly.
Thanks for any help on this issue.

> The Virtual Environment 6.3-3
> the HD space(root) is 36To with 59.27% (used 21.48 TiB of 36.23 TiB)

Would this might be a hint on the issue ?
df -h gives :
/dev/mapper/system--acrob-root 37T 22T 13T 63% /

df -hi
/dev/mapper/system--acrob-root 583M 579M 4.1M 100% /

 

[apoc]

Well if you are running out of inodes you are in trouble.
What are you storing on that volume to reach the inode limit??? I never got into that situation and I am storing millions of files due to rsync backup and hardlinks.

"df -i" wil print the actual inode numbers.

My advice would be to shrink the partition and add a second partition to the array/disk. Then move parts of the data to the new partition. This will free up inodes on the currently existing partition while adding more inodes for the remaining 10TB you have.
You also can specify number of inodes on formatting of your filesystem but it seems you can't that for an existing FS.
If you need to have all data in one FS then it gets more challenging, but you could likely also achieve this by moving things around incrementally and shrinking / growing partitions step by step...

 

[Andrimont]

Thank you for your answer, @tburger. It was helpful as it shows that the issue was indeed the server running out of inodes.
I could locate a large bunch of unnecessary rsync backup. I am deleting them slowly and already df -ih
shows that the server recovered /dev/mapper/system--acrob-root 583M 530M 53M 92% /.
92% is much better than 100% !

EDIT : I got finally /dev/mapper/system--acrob-root 583M 40M 543M 7% /. So everything is back to normal in the use of the server. There were much too frequent backups of web sites that made the server running out of inodes. So beware of too much rsync backup. Use with moderation.