Does zfs mirror read performance scale by vdevs num?

gladtent7039

New Member
Apr 26, 2024
2
0
1
I googled and was told that N-way mirror should improve read performance, but test result indicate otherwise.
Is my testing method incorrect, or is there a misunderstanding?

https://openzfs.org/wiki/Features#Improve_N-way_mirror_read_performance

test result:

PVE 8.2.2
2 nvme ssd zfs mirror

Code:
pool: data
 state: ONLINE
config:


        NAME                                               STATE     READ WRITE CKSUM
        data                                               ONLINE       0     0     0
          mirror-0                                         ONLINE       0     0     0
            nvme-SanDisk_Ultra_3D_NVMe_2TB_23452H800317    ONLINE       0     0     0
            nvme-SanDisk_Ultra_3D_NVMe_2TB_23452H800837_1  ONLINE       0     0     0

fio benchmark
Code:
[global]
name=nvme-seq-read
time_based
ramp_time=5
runtime=90
readwrite=read
bs=256k
ioengine=libaio
direct=1
numjobs=1
iodepth=32
group_reporting=1
[nvme0]
filename=/data/benchmark.dat
size=256G

fio test got about 3300m/s

iostat:
Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.17    0.00   13.93    0.00    0.00   85.91
Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
nvme0n1       13095.00   1676160.00         0.00         0.00    1676160          0          0
nvme1n1       13131.00   1680768.00         0.00         0.00    1680768          0          0

If I tear down zfs pool and run same fio test on single ssd driver, also get 3500m/s
Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    1.09    0.00    0.00   98.91
Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
nvme0n1           0.00         0.00         0.00         0.00          0          0          0
nvme1n1       27622.00   3535616.00         0.00         0.00    3535616          0          0

So, my expectation is that in a ZFS mirror setup, I should achieve speeds of around 6000~7000 MB/s.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!