Poor NFS Performance To TrueNAS Share

E

EG777

New Member
Aug 8, 2025
5
0
1
Hello all, new to the Proxmox world (recently migrated from the ruin that is ESXI under Broadcom). I'm hoping someone can help me pinpoint my bottleneck.

  • My host (Proxmox 8.4.0) at 10.0.30.12 is connected to a 10Gbps (USW-Aggregation) switch over DAC cables.
  • The storage is TrueNAS Scale at 10.0.30.240 connected to the same aggregation switch also over fiber.
    • The storage is a RaidZ1 Pool with 3x Crucial BX500 2TB Drives, and a 2TB Samsung EVO970 Pro SSD as cache.
  • My LAG ports are all registering 10GBe.
  • I have tried Unraid with a R5 Pool of all drives, and averaged 4-7Gbps.
  • I have tried OpenMediaVault with a RZ1 pool, and averaged 3-5Gbps.
  • I am currently on TrueNAS Scale, with an RZ1 pool & cache drive, and getting the speeds below.
My questions are: Is this even a Proxmox issue based on the findings below? Why when I move a disk does it constantly try to copy the same bits without the percentage or total increasing? Going to NFS was abysmal.

My goal is:
  • Use an NFS mount for every day VM storage that can effectively utilize the 10Gbps bandwidth available.
  • Have "mission critical" VMs on replicated local ZFS storage across the 3 nodes in cluster.
I'm just not really sure of the next steps, and I hope someone can shed some light here.

iperf3 results from PVE02 (client) -> TrueNAS (server)
Code: 
Connecting to host 10.0.30.240, port 5201
[  5] local 10.0.30.12 port 52920 connected to 10.0.30.240 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.27 Gbits/sec  266    790 KBytes      
[  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec   32   1.10 MBytes      
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec   64    714 KBytes      
[  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec   51   1010 KBytes      
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec   48   1.05 MBytes      
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec   64    776 KBytes      
[  5]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec   48    861 KBytes      
[  5]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec   48    806 KBytes      
[  5]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec   52    732 KBytes      
[  5]   9.00-10.00  sec  1.09 GBytes  9.36 Gbits/sec   40   1.04 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec  713             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec                  receiver

iperf Done.

iperf3 results from TrueNAS (client) -> PVE02 (server)
Code: 
Connecting to host 10.0.30.12, port 5201
[  5] local 10.0.30.240 port 56532 connected to 10.0.30.12 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.35 Gbits/sec  425   1.73 MBytes      
[  5]   1.00-2.00   sec  1.09 GBytes  9.35 Gbits/sec  322   1.10 MBytes      
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.66 MBytes      
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec    0   2.00 MBytes      
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec    0   2.05 MBytes      
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec    0   2.07 MBytes      
[  5]   6.00-7.00   sec  1.09 GBytes  9.34 Gbits/sec  400   1.51 MBytes      
[  5]   7.00-8.00   sec  1.09 GBytes  9.34 Gbits/sec   94   1.35 MBytes      
[  5]   8.00-9.00   sec  1.08 GBytes  9.26 Gbits/sec  198   1.56 MBytes      
[  5]   9.00-10.00  sec  1.09 GBytes  9.36 Gbits/sec  179   1.08 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.36 Gbits/sec  1618             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec                  receiver

iperf Done.


From Local ZFS share on host -> NFS (too long to post raw)

From NFS -> Local ZFS share on host (too long to post raw)
 
Last edited:
EG777 said:
RaidZ1 Pool with 3x Crucial BX500 2TB Drives
Click to expand...
:eek:
Crucial BX500 are the slowest disks you can find, they writes data slower than HDD, except the first GBs, they use QLC flash.
Running ZFS over these is the best to test the worst case.

EDIT: I don't see network problem as iPerf show 9,3 Gbits/s for your 10Gb network as excepted.
 
Last edited:
_gabriel said:
:eek:
Crucial BX500 are the slowest disks you can find, they writes data slower than HDD, except the first GBs, they use QLC flash.
Running ZFS over these is the best to test the worst case.
Click to expand...
Well damn, I had sort of thought storage devices might be the case and not so much the network, but wanted to collect some additional intel. Would the Samsung 970 EVO's be appropriate replacements? I'm not looking for the most top end of SSD's (I have a budget to consider) but I also would like something more middle of the road that will help saturate (~70-80%) of my 10Gbps link.
 
An update from tonight. I removed the BX500 drives, and tried a single mirror (2x500GB Samsung 970 EVO's) with a 2TB cache drive, also a Samsung 970 EVO. Turning off Sync gave me the best read and write performance. I also noticed the same bottlenecks with an iSCSI target mapped to an LVM.

If I understand correctly, that indicates the drives are still the bottle neck and I need better drives, such as HGST/Intel, something enterprise grade that will take advantage of the random read/write traits of a VMFS.

Sync (Standard)
  • ZFS -> NFS 80 GB - 3.52s - Mirror - No Cache - No Slog - Samsung 970 - 348 MB/s
  • NFS -> ZFS 80 GB - 3.59s - RZ1 - N/A - N/A - Samsung 970 - 334 MB/s
  • ZFS -> NFS 80GB - 3.54s - Mirror - 2TB Cache - No Slog - Samsung 970 - 346 MB/s
  • NFS -> ZFS 80GB - 3.58s - RZ1 - N/A - N/A - Samsung 970 - 335 MB/s
Sync (Disabled) - ASYNC
  • ZFS -> NFS 80GB - 2.20s - Mirror - 2TB Cache - No Slog - Samsung 970 - 570 MB/s
  • NFS -> ZFS 80 GB - 3.17s - RZ1- N/A - N/A - Samsung 970 - 405 MB/s
  • ZFS -> ISCSI 80GB - 4.48s - Mirror - 2TB Cache - No Slog - Samsung 970 - 277 MB/s
  • ISCSI - ZFS 80GB - 3.07s - RZ1 - N/A - N/A - Samsung 970 - 427 MB/s
 
You must log in or register to reply here.
Top Bottom