[SOLVED] zfs storage full but i don't understand why

[kcalvat]

Hello proxmox

I am a novice with proxmox, ZFS and I have to understand our situation because our physical (Debian 9.7) server space is full. This is not my own installation and configuration, i don't have any history about its management.

When i audit, one the VMs (102) has 7TB disks at proxmox level (this is a file server with windows 2k16) but when i look at ZFS level, storage/vm-102-disk-2 is 9.25TB, and when i look at the windows level, the data used 5.5TB. I am not able to understand why ZFS used space is almost twice of "real" windows used space.

There are no snapshots, no copies, compression is lz4.
So does the ZFS metadata use all this space ? Does the parity with raidz1 is poor optimization ? Does block size is miss configure somewhere ? Something wrong with ZFS configuration or proxmox ?

Thank you very much for your help
Best regards, Kenny.


cat /etc/pve/qemu-server/102.conf

scsi0: storage:vm-102-disk-1,discard=on,size=100G
scsi1: storage:vm-102-disk-2,size=7000G

zfs list -o space -r storage

NAME                     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
storage                  3,51G  9,32T        0B    162K             0B      9,32T
storage/base-101-disk-1  3,51G  14,0G     13,5K   14,0G             0B         0B
storage/vm-100-disk-1    3,51G  38,7G        0B   38,7G             0B         0B
storage/vm-102-disk-1    3,51G  22,7G        0B   22,7G             0B         0B
storage/vm-102-disk-2    3,51G  9,25T        0B   9,25T             0B         0B

zfs list

NAME                      USED  AVAIL  REFER  MOUNTPOINT
storage                  9,32T  3,50G   162K  /storage
storage/base-101-disk-1  14,0G  3,50G  14,0G  -
storage/vm-100-disk-1    38,7G  3,50G  38,7G  -
storage/vm-102-disk-1    22,7G  3,50G  22,7G  -
storage/vm-102-disk-2    9,25T  3,50G  9,25T  -

zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  11,4T  11,1T   370G         -    51%    96%  1.00x  ONLINE  -

zpool status

      pool: storage
      state: ONLINE
      scan: scrub repaired 0B in 14h21m with 0 errors on Sun Feb 13 14:45:40 2022
      config:

            NAME                            STATE     READ WRITE CKSUM
            storage                         ONLINE       0     0     0
              raidz1-0                      ONLINE       0     0     0
                scsi-350000398581be825      ONLINE       0     0     0
                scsi-35000039858207311      ONLINE       0     0     0
                scsi-350000398582084c9      ONLINE       0     0     0
                scsi-350000398581be1a1      ONLINE       0     0     0
                scsi-35000039858207a05      ONLINE       0     0     0
                scsi-350000398581b7209      ONLINE       0     0     0
                scsi-35000c500a10c210b      ONLINE       0     0     0
            logs
              scsi-33001438042b37703-part1  ONLINE       0     0     0
            cache
              scsi-33001438042b37703-part2  ONLINE       0     0     0

    errors: No known data errors

 

[fabian]

see https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_zfs_raid_considerations

 

[Dunuin]

The problem should be padding overhead caused by using raidz with a too small volblocksize. Read this: https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

You probably want atleast 8 times your ashift as volblocksize to only loose 20% raw capacity instead of 50% which you probably loose now with the default 8K volblocksize.
In case you use ashift=12 so 4K sectors used:

	Parity+Padding loss of raw capacity:
Volblocksize 4K/8K:	50% (14% parity + 36% padding)
Volblocksize 16K:	33% (14% parity + 19% padding)
Volblocksize 32K/64K:	20% (14% parity + 6% padding)
Volblocksize 128K/256K:	16% (14% parity + 2% padding)
Volblocksize 512K/1M:	15% (14% parity + 1% padding)

 

[kcalvat]

thank you @fabian for your quick response I read it.
So if i understand, this is parity blocks with raidz1 usage and some metadatas and others ZFS things ?

zfs get volsize,refreservation,used,volblocksize storage/vm-102-disk-2

NAME                   PROPERTY        VALUE      SOURCE
storage/vm-102-disk-2  volsize         6,84T      local
storage/vm-102-disk-2  refreservation  none       default
storage/vm-102-disk-2  used            9,25T      -
storage/vm-102-disk-2  volblocksize    8K         default -

 

[kcalvat]

Finally, it was only dirty free space with thin provisioning for my file server VM. I use https://docs.microsoft.com/en-us/sysinternals/downloads/sdelete with -z option and I reclaim 1,4TB.

Thank you very much @fabian and @Dunuin for your help.
@Dunuin, I also read your article and it was very interesting !

 

[Dunuin]

I usually enable "SSD emulation" for my virtual disks, even if that pool only consists of HDDs. That way Windows should think its an SSD so it should send TRIM commands by default needed by thin-provisioning and it won't try to defrag the virtual disk which you don't want with a Copy-on-Write storage.

 

[kcalvat]

Thank you very much @Dunuin for your tip, very usefull !