ZFS usage incorrect for a VM in RAIDZ1

[Autrui]

Hi,

I'm currently using a zfs pool as RAIDZ1.

The ZFS pool has the thin provision argument set.

I created a hard disk on this zfs pool, with discard=1. In the VM, df -h tells me there is a 295G disk, with 188G used.

In the pool in proxmox, the summary says 294Gb used, but the vm disk (raw format) is at 322Gb.

zpool list says there is 377G allocated. zfs list says there is 274G allocated.

There is absolutely no other file in the zfs pool.

zpool list tells me there is 6.90T free, but zfs list tells me it's 4.89T. The summary of the zfs pool in Proxmox says it's 5.67T

Could someone tell me what's going on there ? I'm having a hard time understanding if the thin provisioning is working, and why the used space is not the same everywhere. Thanks !

 

[LnxBil]

why the used space is not the same everywhere.

if running with raidz, zpool shows the raw space of your pool before any vdev specification. zfs list show this after the vdev specification, so that those two differ.

Can you show us the output of (in CODE tags)

• zpool list -v
• zfs list

 

[Dunuin]

Also search this forum for "padding overhead". If you didn't manually changed the blocksize of the raidz1 storage, every virtual disk will consume way more space.

 

[LnxBil]

Also search this forum for "padding overhead". If you didn't manually changed the blocksize of the raidz1 storage, every virtual disk will consume way more space.

We still a page we can refer to ... instead of answering the same question over and over again.

 

[Autrui]

Hi,

Thank you for your answer.

Here are the outputs :

root@proxmox:~# zpool list -v zfs-data
NAME                                           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs-data                                      7.27T   377G  6.90T        -         -     0%     5%  1.00x    ONLINE  -
  raidz1-0                                    7.27T   377G  6.90T        -         -     0%  5.06%      -    ONLINE
    ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1KN5EJS  3.64T      -      -        -         -      -      -      -    ONLINE
    ata-ST4000NM0165_ZAD0BS74                 3.64T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107TZ7    1.82T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107V4C    1.82T      -      -        -         -      -      -      -    ONLINE

root@proxmox:~# zfs list zfs-data
NAME       USED  AVAIL     REFER  MOUNTPOINT
zfs-data   274G  4.89T      151K  /zfs-data

@Dunuin I found some threads minutes ago talking about it (and you in it !), and I just set the block size from 8k to 16k. I think it's the correct value ?

 

[LnxBil]

Here are the outputs

at least a subset. Don't you want to show all of the output? I did not say to add the pool name.

 

[Dunuin]

And you know that you will lose 4TB because only 2TB of each 4TB disk can be used when mixing disks with different sized in a raidz1? I would create a raid10 where you get more performance while also getting more usable capacity and better reliability, shorter resilvering times and would allow for smaller blocksizes.

 

[Autrui]

at least a subset. Don't you want to show all of the output? I did not say to add the pool name.

Sorry, I was focused on the pool I was questionning about !

Here is the full output :

root@proxmox:~# zfs list
NAME                              USED  AVAIL     REFER  MOUNTPOINT
rpool                            18.3G   427G      104K  /rpool
rpool/ROOT                       5.08G   427G       96K  /rpool/ROOT
rpool/ROOT/pve-1                 5.08G   427G     5.08G  /
rpool/data                       13.2G   427G      104K  /rpool/data
rpool/data/subvol-102-disk-0      571M  7.45G      568M  /rpool/data/subvol-102-disk-0
rpool/data/subvol-105-disk-0      491M  7.52G      491M  /rpool/data/subvol-105-disk-0
rpool/data/subvol-106-disk-0      570M  1.44G      570M  /rpool/data/subvol-106-disk-0
rpool/data/vm-100-disk-0         3.31G   427G     3.31G  -
rpool/data/vm-101-disk-0         2.15G   427G     2.15G  -
rpool/data/vm-103-disk-0           56K   427G       56K  -
rpool/data/vm-103-disk-1         2.86G   427G     2.86G  -
rpool/data/vm-104-disk-0         3.28G   427G     3.28G  -
zfs-backup                        384G  1.38T      104K  /zfs-backup
zfs-backup/backup                 384G  1.38T     5.75G  /zfs-backup/backup
zfs-backup/backup/vm-104-disk-0   191G  1.38T      191G  -
zfs-backup/backup/vm-104-disk-1   187G  1.38T      187G  -
zfs-backup/data                    96K  1.38T       96K  /zfs-backup/data
zfs-data                          274G  4.89T      151K  /zfs-data
zfs-data/data                     274G  4.89T      151K  /zfs-data/data
zfs-data/data/vm-104-disk-0       274G  4.89T      274G  -



root@proxmox:~# zpool list -v
NAME                                           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                                          460G  18.3G   442G        -         -     3%     3%  1.00x    ONLINE  -
  mirror-0                                     460G  18.3G   442G        -         -     3%  3.98%      -    ONLINE
    nvme-TS512GMTE110S_F976830179              477G      -      -        -       12G      -      -      -    ONLINE
    nvme-WDS500G3X0C-00SJG0_203814803380       466G      -      -        -         -      -      -      -    ONLINE
zfs-backup                                    1.81T   384G  1.44T        -         -     0%    20%  1.00x    ONLINE  -
  mirror-0                                    1.81T   384G  1.44T        -         -     0%  20.7%      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A310765A    1.82T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107R6R    1.82T      -      -        -         -      -      -      -    ONLINE
zfs-data                                      7.27T   377G  6.90T        -         -     0%     5%  1.00x    ONLINE  -
  raidz1-0                                    7.27T   377G  6.90T        -         -     0%  5.06%      -    ONLINE
    ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1KN5EJS  3.64T      -      -        -         -      -      -      -    ONLINE
    ata-ST4000NM0165_ZAD0BS74                 3.64T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107TZ7    1.82T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107V4C    1.82T      -      -        -         -      -      -      -    ONLINE

And you know that you will lose 4TB because only 2TB of each 4TB disk can be used when mixing disks with different sized in a raidz1? I would create a raid10 where you get more performance while also getting more usable capacity and better reliability, shorter resilvering times and would allow for smaller blocksizes.

Yes I know, I'm planning to replace the 2 Tb disks for 4 Tb disks. Is a RAIDZ1 a good idea with 4 Tb disks then ?

 

[Dunuin]

Yes I know, I'm planning to replace the 2 Tb disks for 4 Tb disks. Is a RAIDZ1 a good idea with 4 Tb disks then ?

Not if you care about IOPS performance, you got alot of small writes or if you want to run some applications backed by DBs like MySQL/PostgreSQL.
With HDDs only useful as a cold storage.

 

[Autrui]

I was thinking to use the zfs-data pool for data like files in nextcloud, openmediavault, etc.
For DB like MySQL, I wanted to use the rpool pool, which is two NVME in mirror.

I'm looking at a RAID10 calculator, with 4 Tb disks I see 7.3 Tb of usable storage, but with a RAIDZ1 it's 11.23. Am I missing something ?

 

[Dunuin]

Depends on the volblocksize. Might be 50-75% of the raw capacity. With raid10 it't always 50%.

 

[LnxBil]

Why don't you just raid10 all 6 disks to one pool? I don't see how that should be worse than the availabiliy now. Backups should be stored off-site.

 

[Autrui]

So, I followed your advice and deleted my zfs-data RAIDZ and created a RAID10 one :

root@proxmox:~# zpool list -v
NAME                                           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                                          460G   206G   254G        -         -     3%    44%  1.00x    ONLINE  -
  mirror-0                                     460G   206G   254G        -         -     3%  44.9%      -    ONLINE
    nvme-TS512GMTE110S_F976830179              477G      -      -        -       12G      -      -      -    ONLINE
    nvme-WDS500G3X0C-00SJG0_203814803380       466G      -      -        -         -      -      -      -    ONLINE
zfs-data                                      7.25T  78.1G  7.17T        -         -     0%     1%  1.00x    ONLINE  -
  mirror-0                                    3.62T  20.0G  3.61T        -         -     0%  0.53%      -    ONLINE
    ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1KN5EJS  3.64T      -      -        -         -      -      -      -    ONLINE
    ata-ST4000NM0165_ZAD0BS74                 3.64T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                    1.81T  29.2G  1.78T        -         -     0%  1.57%      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107TZ7    1.82T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A310765A    1.82T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                    1.81T  29.0G  1.78T        -         -     0%  1.56%      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107R6R    1.82T      -      -        -         -      -      -      -    ONLINE
    ata-WDC_WD20EZBX-08AYR_WD-WXJ2A3107V4C    1.82T      -      -        -         -      -      -      -    ONLINE


root@proxmox:~# zfs list
NAME                           USED  AVAIL     REFER  MOUNTPOINT
rpool                          206G   239G      104K  /rpool
rpool/ROOT                    5.15G   239G       96K  /rpool/ROOT
rpool/ROOT/pve-1              5.15G   239G     5.15G  /
rpool/data                     201G   239G      104K  /rpool/data
rpool/data/subvol-102-disk-0   571M  7.45G      568M  /rpool/data/subvol-102-disk-0
rpool/data/subvol-102-disk-1    96K  50.0G       96K  /rpool/data/subvol-102-disk-1
rpool/data/subvol-105-disk-0   491M  7.52G      491M  /rpool/data/subvol-105-disk-0
rpool/data/subvol-106-disk-0   571M  1.44G      571M  /rpool/data/subvol-106-disk-0
rpool/data/vm-100-disk-0      3.41G   239G     3.41G  -
rpool/data/vm-101-disk-0      2.15G   239G     2.15G  -
rpool/data/vm-103-disk-0        56K   239G       56K  -
rpool/data/vm-103-disk-1      2.86G   239G     2.86G  -
rpool/data/vm-104-disk-0      3.28G   239G     3.28G  -
rpool/data/vm-104-disk-1       188G   239G      188G  -
zfs-data                      78.1G  7.05T     78.1G  /zfs-data

I went from 7.27 + 1.81 = 9.08 T of available space to 7.25 T.

I set atime=off, relatime=on, and block size = 16k.

Is there some tweaks I can do to get some more available space ?

 

[Dunuin]

atime=off, relatime=on

With "atime=off" the "relatime=on" will be ignored, as no atime will be used at all.

block size = 16k

Would be fine with the default 8K nut 16K might help with better compression ratios.

Is there some tweaks I can do to get some more available space ?

You could switch from default LZ4 to ZSTD compression but that with a noticable performance hit.