RPool don't display good size

[jmorfali]

Good morning,

When I'm in Proxmox and I'm going to look at my server information on the ZFS. It tells me that I have 112TB of space and 99.11TB free. Which is normal. But when I go to see the information of the rpool that I created. It tells me I have 94.14TB and 73.68TB used. Yet the two current VMs only use 50 TB... I don't understand why.

Thank you very much for your help.

 

[Dunuin]

Output of zpool list -v and zfs list -o space, zpool get ashift and zfs get volblocksize would help to understand the situation better.

 

[jmorfali]

Yes of course, here they are:

# zpool list -v
NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                        102T  11.7T  90.1T        -         -     0%    11%  1.00x    ONLINE  -
  raidz1-0                   102T  11.7T  90.1T        -         -     0%  11.5%      -    ONLINE
    scsi-35000cca2906fdf78  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca2906fe064  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca290702b6c  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca290700b48  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca2904b4a50  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca2ad2f1b48  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca2ad329448  12.7T      -      -        -         -      -      -      -    ONLINE
    scsi-35000cca2904ac568  12.7T      -      -        -         -      -      -      -    ONLINE



# zfs list -o space
NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool                  18.6T  67.0T        0B    162K             0B      67.0T
rpool/vm-1001-disk-0   27.2T  16.8T        0B   8.13T          8.62T         0B
rpool/vm-90806-disk-0  67.1T  50.3T        0B   1.74T          48.5T         0B


# zpool get ashift
NAME   PROPERTY  VALUE   SOURCE
rpool  ashift    12      local



# zpool get ashift
NAME   PROPERTY  VALUE   SOURCE
rpool  ashift    12      local


# zfs get volblocksize
NAME                   PROPERTY      VALUE     SOURCE
rpool                  volblocksize  -         -
rpool/vm-1001-disk-0   volblocksize  8K        default
rpool/vm-90806-disk-0  volblocksize  8K        default

Thank you very much

 

[Dunuin]

You got ashift=12 (so 4K "sectors") and default volblocksize=8K with a 8 disk raidz1. That means you will indirectly lose 50% of the raw capacity. 12,5% raw capacity loss because of parity data and every zvol (so VM virtual disks) will be 175% in size, as for each 1 TB of data blocks there will be additional 0,75TB of padding blocks.
To only lose 20% of your raw capacity you would need to increase the volblocksize to at least 32K. To only lose 16% it would need to be 128K. For 14% it needs to be 256K.
The volblocksize can only be set at the creation of a zvol and is defined by the "Block Size" field of your ZFS storage. So you would need to change that to something way higher and after that you need to destroy and recreate all those VMs (easiest would be to backup and restore them with the same VMID).

Search this forum for "padding overhead" for more info.

And VM 90806 got 48.5TB of refreservation. So you either didn't checked the "thin" checkbox when creating the ZFS storage so all virtual disks are thick-provisioned or you didn't set up discard/trim inside your VM so ZFS will never free up data that got deleted. As you are using the default rpool I would guess it's the latter one. In that case make sure you use virtio SCSI and verify that the "discard" checkbox of your VMs virtual disk is checked. And check that every guestOS will trim/discard regularily by mounting your guests filesystem with a "discard" option or by running something like a daily fstrim -a.