ZFS inside VM: thin provisioning & trimming

Hello,

we're using ZFS (single disk) inside some VMs for huge disks. The reason is, this way we can easily do snapshots and send them over to our zpool on the backup server. No rsync-fiddling with ACLs / permission / etc necessary. This works really great.

One thing we noticed is, that thin provisioning does not seem to work correctly. We have a Debian VM with a 5 TB disk configured like this:
Code:
agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=scsi0
cores: 24
cpu: host
efidisk0: replicated:vm-119-disk-0,size=1M
ide0: none,media=cdrom
memory: 32768
meta: creation-qemu=9.0.2,ctime=1742653482
name: mail-1-DMZ
net0: virtio=00:50:56:97:6c:8b,bridge=vmbr4
onboot: 1
ostype: l26
scsi0: replicated:vm-119-disk-1,aio=native,discard=on,iothread=1,size=110G,ssd=1
scsi1: replicated:vm-119-disk-2,aio=native,discard=on,iothread=1,size=5T,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=42172fec-e2f2-683f-c6a7-3ad6bfb31ba3
sockets: 1
vmgenid: c133273b-3f0c-449d-8d1c-380706c831b0

Discard and SSD-Emulation are on! The underlying storage is a ceph pool (named "replicated" with a 3/2 replication). `rbd du` says that 5079.51 GiB / 5120.00 GiB are used for disk 2.
Inside the VM zfs has this to say:
Code:
NAME      USED  AVAIL  REFER  MOUNTPOINT
storage  3.25T  1.61T  2.38T  /storage

I've manually ran `zpool trim storage` inside the VM in order to trigger trimming... sadly this did not reduce the used space on the ceph pool.

Is there anything I can do to regain the 1,6 TB or is it impossible because ZFS is not made for VMs?

Thanks for your help!
 
Last edited: