Storage usage unusually high compared to actual VM storage usage?

[kardigan42]

So recently my VM storage disk has been growing almost full. The disk in question is a 1TB disk, and proxmox claims that 728GB is being used. However, this number is not accurate, and I should have much more storage available.

I've attempted to try and figure out why by running ncdu on each VM and container, as well as on the host, however none of the VMs nor containers are actually using 728GB in total. They are all using less than 50GB storage each (10 VMs and containers total). I'm really confused as to what is using so much storage? I've deleted all unused VMs and containers as well.

Below are screenshots showing the ouput of some commands I think might help provide more information, please let me know if I need to provide anything else. Thanks in advance, and happy new year!

Here is the output of ncdu on the host:

The output of lsblk:

Output of vgs:

Output of lvs:

The content of /etc/pve/storage.cfg:

 

[Dunuin]

Check if there are snapshots as these would prevent stuff from actually getting deleted. Similar case when you run VMs and didn't set them properly up for discard/TRIM. In that the space of delete data won't be freed up.

 

[kardigan42]

Thanks for the reply!
I tried deleting old snapshots as well as enabling discards on the VM that is using the most storage (the rest are fresh installs and a couple DNS servers not using any storage really), and rebooted it and ran fstrim -a, however this only brought down the storage usage from 728GB to 675GB. I wasn't able to run fstrim on the containers, it gave operation not permitted error (I assume LXC doesn't support doing it?). The underlying hardware is a HDD, if it makes any difference. Any ideas?

 

[Dunuin]

For containers on LVM you need to trim them via running pct fstrim YourVMID on the host.

 

[kardigan42]

Wow, thank you so much! Worked like a charm, freeing up tons of storage. TIL! Freed up 828GB on the container that was used as a backup server.

Is there any risk with automating trimming containers like this? Or should I stick do doing it manually once in a while?

 

[Dunuin]

Is there any risk with automating trimming containers like this? Or should I stick do doing it manually once in a while?

You could automate it via cron. One problem I encountered is that when trimming is running the LXC will be locked and a backup of that LXC will fail.
So plan your automation of "pct fstrim" when there is no backup job running.

 

[kardigan42]

You could automate it via cron. One problem I encountered is that when trimming is running the LXC will be locked and a backup of that LXC will fail.
So plan your automation of "pct fstrim" when there is no backup job running.

Nice, makes sense. Thank you again for the help and valuable input!