SSD ZFS Pool keeps increasing in usage space

[awirthy]

I have a 3 node cluster with a SSD ZFS pool that keeps increasing in size.
Each server has 24 SSD drives in a ZFS raid.
The 'autotrim' setting is set to on for each server node.
The two attachment show the space being slowly used up on the server node.
All disks on VMs on all server nodes have 'discard' set to 'on'.

I also ran the command 'zpool status -t ssd-zfs' and get the result from the 'PVE3 SSD-ZFS Status' image. This says 'untrimmed'.
It is the same on PVE2, But PVE1 says 'trim unsupported' instead.

Can anyone help me with why this is happening.

 

[IsThisThingOn]

You would have to provide a lot more information than what you posted here. Otherwise we have to make educated guesses

I have a 3 node cluster

So just cluster, no HA?

Each server has 24 SSD drives in a ZFS raid.

So you use ZFS and VMs use local RAW drives on ZFS?
There are a few problems with that.

Short:
A: Don't use RAIDZ for VM! Use mirror instead.
B: Don't put data into VMs!

Long:
A: RAW ZFS drives use by default 16k volblock storage. Pool geometry will not work out for you. You expect to get 22/24*100=91.667% storage efficiency.
In reality you get 66.66%. Your 16k volblock can't make use of the wide pool. Each stripe will have four 4k data blocks and two 4k parity blocks. That is only 66.66%. For an even longer explanation see this: https://github.com/jameskimmel/opinions_about_tech_stuff/blob/main/ZFS/The problem with RAIDZ.md#the-problem-with-raidz-or-why-you-probably-wont-get-the-storage-efficiency-you-think-you-will-get but TLDR is use mirrors for Proxmox not RAIDZ.

B: Volblocksize from above is one thing, but huge VM disks with data also suck to backup and move. Instead offload the data out of VMs. But it into smb or nfs shares or create a dataset on Proxmox and access it over VirtioFS. That is not easy to manage for beginners. I get an all in one machine sounds great on paper, but Proxmox is a great hypervisor, not so great NAS. While TrueNAS is a great NAS, not so great hypervisor. Unless you really know what you are doing, I recommend to start with two systems.

Now to get back to your problem
My guess is this has nothing to do with TRIM, but with problem A.
The mean part about ZFS is, that it will not show you the problem A.
AFAIK ZFS calculates the available storage based on the default 128k record.
So when you put a 1TB VM disk on your pool, you expect it to use 1TB of your storage, right?
Well that is not what will happen. Because you don't get the expected 91.667% storage efficiency, but 66%.
So instead, the 1TB VM RAW disk will not only use 1TB but roughly 1.4TB.

Expected: 1/91.66*100=1.091TB of raw space
Reality with 16k volblock size: 1/66*100=1.515TB of raw space
1.515/100*91.66=1.389TB usage on your disk.

I am at my first coffee, so take my math with a buttload of salt

 

[Impact]

The 'autotrim' setting is set to on for each server node.

I'd recommed setting up scheduled trim instead.

Please share

zfs list -ospace,refreservation -rS used
qm config VMIDHEREOFAVMWITHLOTSOFUSEDDATA

Also read this about how to properly fstrim/discarding. A zpool trim does not affect the disk usage for your virtual guest drives unless you use ZFS on top of ZFS.