Storage suddenly completely full

[Heino77]

I lost connection to my host and wanted to access it through the Webgui.
I couldn't login.
I went to the terminal and did some checks which I read about here on the forums.
df -h
tells me that 98% of root is full.
I checked further and found out that /mnt takes up 80% of my storage.
I was finally able to login today and saw that the SSDs in ZFS are completely full. Also the HDD which is only there for backup reasons is 90% full.
A backup was made when I first installed the server but there was an installation of applications afterwards so I can't use this one.

I wanted to restart the VM to see if Veeam uses up all the disk space but I can't start vncproxy due to insufficient space.
Also I'm not able to install apt-get ncdu to check since I still have the enterprise repository.
If I want to change it it says permisison denied even though there is only root access on the ssh.

 

[Dunuin]

If your root pool is completely full proxmox will switch to read-only mode.

Most of the time when your root pool gets full it's because you got some kind of mount problem. Lets say you got a SMB share or USB disk mounted to /mnt/example or /media/example. If the share/disk got mounted everything written to /mnt/example or /media/example will be stored on the disk/share. If it is not mounted and you write to these folders everything will be written to you root storage (so your proxmox system SSD) instead.

So first unmount everything in /mnt and /media and then check if there is still something stored. If a mountpoint isn't empty then something was written there while unmounted and you need to delete it before remounting everything.
But make sure it is really unmounted so that you don't delete stuff you want to keep.

 

[Heino77]

Thanks for your help and explanation.
I unmounted /media and /mnt. Fstab shows no mounted devices.
How do I delete the files? Do you have a command?

can I just do rm -r? Should be safe because all drives are unmounted? Is there a way to double check?

 

[Dunuin]

You can run df -aTh or findmnt to get a list what is mounted where.