Dell R760 High I/O Pressure with NFS, but No Measurable VM Impact

[orkiniec]

I am seeing sustained I/O Pressure and I/O Delay on one Proxmox node only when its VM disks are stored on NFS.
These are standalone nodes and are not in a cluster.
Node 1: Proxmox VE 9.2.2, kernel 7.0.2-6-pve, Dell PowerEdge R760, dual Xeon Gold 6426Y, 512 GB RAM, Intel E810 25 Gb NIC, 4 VMs.
Node 2: Proxmox VE 9.2.2, kernel 7.0.2-6-pve, Fujitsu RX2540, dual Xeon Silver 4410, 256 GB RAM, Intel X722 10 Gb NIC, 13 VMs.
Storage: Both nodes use the same TrueNAS H20 Enterprise NVMe storage, the same NFS export, NFS 4.2, and a dataset sync setting of standard.

All VMs use the same general configuration: qcow2 disks, VirtIO SCSI Single, I/O thread enabled, and the same Proxmox, kernel, QEMU, and NFS mount options.
All VMs on both nodes are running QEMU 11.0.0. Both nodes have the same relevant package versions:
proxmox-kernel-helper: 9.1.0+fde2
proxmox-kernel-7.0: 7.0.2-6
proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6
pve-qemu-kvm: 11.0.0-3
qemu-server: 9.1.15

The affected VMs have been fully stopped and started since the QEMU update, so they are not still running an older QEMU process.

On the Dell R760, I/O Pressure is typically sustained around 5–25%, with occasional higher spikes.
When the same VMs are moved from NFS to the Dell’s local NVMe ZFS storage, the pressure drops significantly.
The Dell originally used a Broadcom BCM57504 25 Gb NIC. I suspected the adapter, so I replaced it with an Intel E810 25 Gb NIC, but the behavior did not change.

The Fujitsu node uses the same TrueNAS server, NFS export, NFS 4.2 mount options, qcow2 format, VirtIO SCSI configuration, and software versions.
It has more active VMs and a 10 Gb connection, but it does not show the same sustained I/O Pressure.
I understand that network storage and file-backed virtual disks add latency compared with local block storage.
However, I have not found evidence of a performance bottleneck in the NFS or TrueNAS path.

During elevated pressure on the Dell, nfsiostat showed zero retransmissions, zero RPC backlog, zero errors, read and write RTT generally below 1 ms, and very low NFS queue time.
In one sample, the Dell exceeded 2,500 NFS write operations per second while write RTT remained around 0.4 ms.
The Intel E810 shows no meaningful CRC errors, drops, resets, link faults, or timeouts.
Per-VM PSI shows that several VMs are accumulating I/O pressure.
However, Windows PerfMon captures from the three main affected VMs showed typical write latency below 1 ms during active workloads, low disk queue depth, no sustained storage latency, no timeout or retry events, and no user-reported application problems.

Even during workloads exceeding 1,500–2,000 writes per second, guest-observed write latency remained around 0.5–1 ms.
QEMU block statistics confirmed that the affected workloads perform many small writes, generally around 5–6 KB each, but these are the same workloads that previously ran on local ZFS without elevated pressure.

The Dell is therefore recording real PSI I/O stalls when the VM disks are on NFS, but I have not found corresponding evidence that the TrueNAS storage, NFS connection, host, or guests are suffering from an actual performance problem.

NFS is clearly the condition that exposes the behavior, but the available measurements do not indicate that the NFS server or network path is slow or saturated.
The same storage and configuration also operate without sustained PSI on the Fujitsu node under a heavier VM load.

Has anyone seen one Proxmox host report significantly higher I/O PSI than another host using the same NFS storage?
Could this be related to QEMU, io_uring, NUMA placement, CPU scheduling, Linux PSI accounting, or something specific to the Dell R760 platform?
At this point, I suspect something host-specific on the Dell R760, but I have not been able to identify it.
Are there additional tracing or diagnostic tools that would show exactly which wait state PSI is accounting for?

 

[waltar]

ZFS don't like nfs default export sync behavior which can be a bit alleviated with a slog device.
Your 25Gb node1 still shows more I/O pressure as data would come faster as on the other and so normally you see the higher pressure there.
Download the parallel filesystem testtool elbencho (just 1 tarfile to extract to 1 all inclusive executable) and run locally on your truenas and your 2 pve nodes via nfs mounts - normally should be the same bandwitdh for local and nfs tests if network is fast enough but see it's harder to reach when nfs-server is exported zfs, but test, eg:
elbencho --help

./elbencho -r -w -d -t 8 -n 8 -N 10 -s 1G -b 128k -F -D /mnt
OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
MKDIRS      Elapsed time     :         1ms         4ms
            Dirs/s           :        8240       15263
            Dirs total       :          14          64
---
WRITE       Elapsed time     :   2m51.808s   2m56.677s
            Files/s          :           3           3
            IOPS             :       29902       29674
            Throughput MiB/s :        3737        3709
            Total MiB        :      642193      655360
            Files total      :         624         640
---
READ        Elapsed time     :   2m25.973s   2m34.637s
            Files/s          :           4           4
            IOPS             :       34109       33904
            Throughput MiB/s :        4263        4238
            Total MiB        :      622381      655360
            Files total      :         606         640
---
RMFILES     Elapsed time     :       506ms      3.071s
            Files/s          :        1216         208
            Files total      :         616         640
---
RMDIRS      Elapsed time     :         5ms         7ms
            Dirs/s           :       10882        8111
            Dirs total       :          64          64

 

[orkiniec]

Thanks for this suggestion. I will need to run this tonight during a quieter period since i do have production running. Would a smaller sample, such as 32 GiB, still provide a useful comparison between TrueNAS, PVE01 (node 1), and PVE99 (node 2), or is the larger test necessary?

 

[waltar]

As you see that runs just under 3 mins for write and for the read one but you could change everything, maybe going from 8 to just 4 threads will result in half of benchmark size (above doing 640x 1GB). If you test to small you test caching mechanism and not your real I/O capability.

 

[orkiniec]

I was able to complete the full benchmark this morning on Node1, a Dell R760, and locally on the TrueNAS Enterprise system. I was not able to complete the full run on Node2, but I did run a smaller 100 GiB test last night, which is attached.
While the benchmark was running on Node1, I also monitored /proc/pressure/io. During the test, both some and full PSI climbed into the 90% range.
I also logged into a Windows VM running on Node1 while the benchmark was active. I did not notice any major slowdown or responsiveness issues. Windows Resource Monitor showed disk response times in the 200–400 ms range, which I expected under this level of load.
What surprised me was that the VM remained usable even though PSI was reporting approximately 90% I/O pressure. Based on that number alone, I expected the VM to be nearly unresponsive.

Node1 Dell R760
./elbencho -r -w -d -t 8 -n 8 -N 10 -s 1G -b 128k -F -D /mnt/pve/truepool1-tank01

OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
MKDIRS      Elapsed time     :         6ms         9ms
            Dirs/s           :        5524        6627
            Dirs total       :          34          64
            Dirs latency     : [ min=413us avg=636us max=1.02ms ]
---
WRITE       Elapsed time     :   6m38.795s   6m44.874s
            Files/s          :           1           1
            IOPS             :       13023       12949
            Throughput MiB/s :        1627        1618
            Total MiB        :      649216      655360
            Files total      :         632         640
            Files latency    : [ min=1.12s avg=5.01s max=9.19s ]
            IO latency       : [ min=28us avg=39us max=21.1ms ]
---
READ        Elapsed time     :  14m38.846s  14m59.089s
            Files/s          :           0           0
            IOPS             :        5954        5831
            Throughput MiB/s :         744         728
            Total MiB        :      654118      655360
            Files total      :         633         640
            Files latency    : [ min=3.93s avg=11.1s max=95.1s ]
            IO latency       : [ min=6us avg=1.35ms max=1.06s ]
---
RMFILES     Elapsed time     :       888ms      1.130s
            Files/s          :         566         566
            Files total      :         503         640
            Files latency    : [ min=1.37ms avg=13.1ms max=37.7ms ]
---
RMDIRS      Elapsed time     :         4ms         7ms
            Dirs/s           :       14946        8740
            Dirs total       :          64          64
            Dirs latency     : [ min=323us avg=494us max=843us ]
---

TrueNAS
OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
MKDIRS      Elapsed time     :         0ms         1ms
            Dirs/s           :       35851       42838
            Dirs total       :          28          64
            Dirs latency     : [ min=35us avg=45us max=64us ]
---
WRITE       Elapsed time     :   4m20.377s   4m20.946s
            Files/s          :           2           2
            IOPS             :       20100       20091
            Throughput MiB/s :        2512        2511
            Total MiB        :      654211      655360
            Files total      :         633         640
            Files latency    : [ min=2.31s avg=3.26s max=3.42s ]
            IO latency       : [ min=94us avg=396us max=90.3ms ]
---
READ        Elapsed time     :   1m52.617s   1m53.264s
            Files/s          :           5           5
            IOPS             :       46312       46288
            Throughput MiB/s :        5789        5786
            Total MiB        :      651945      655360
            Files total      :         633         640
            Files latency    : [ min=936ms avg=1.41s max=2.30s ]
            IO latency       : [ min=26us avg=171us max=33.4ms ]
---
RMFILES     Elapsed time     :        74ms       227ms
            Files/s          :        6472        2807
            Files total      :         484         640
            Files latency    : [ min=542us avg=1.62ms max=85.1ms ]
---
RMDIRS      Elapsed time     :         0ms         1ms
            Dirs/s           :       58519       34744
            Dirs total       :          34          64
            Dirs latency     : [ min=33us avg=112us max=239us ]

Node 2 Fujitsu 100GiB test
OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========

MKDIRS      Elapsed time     :         0ms         2ms
            Dirs/s           :       33973       29244
            Dirs total       :          57          64
            Dirs latency     : [ min=29us avg=67us max=1.18ms ]

---

WRITE       Elapsed time     :   6m38.795s   6m44.874s
            Files/s          :           2           2
            IOPS             :       13023       12949
            Throughput MiB/s :        1627        1618
            Total MiB        :      649216      655360
            Files total      :         634         640
            Files latency    : [ min=1.12s avg=5.01s max=9.19s ]
            IO latency       : [ min=28us avg=39us max=21.1ms ]

---

READ        Elapsed time     :  14m38.846s  14m59.089s
            Files/s          :           0           0
            IOPS             :        5954        5831
            Throughput MiB/s :         744         728
            Total MiB        :      654118      655360
            Files total      :         639         640
            Files latency    : [ min=3.93s avg=11.1s max=95.1s ]
            IO latency       : [ min=6us avg=1.35ms max=1.06s ]

---

RMFILES     Elapsed time     :        40ms       654ms
            Files/s          :       12257         978
            Files total      :         490         640
            Files latency    : [ min=427us avg=5.08ms max=502ms ]

---

RMDIRS      Elapsed time     :         0ms         2ms
            Dirs/s           :       51284       29908
            Dirs total       :          63          64
            Dirs latency     : [ min=34us avg=123us max=1.46ms ]

---

 

[waltar]

Hi orkiniec, first you have a real bad nfs transfer as local on truenas versus nfs4.2 remote pve1 has very high difference, should be less than 50MB/s if not limited by network interface which is the case with 25Gb - limited to 3GB/s - and comes to read result :
Truenas write 2500 MB/s but just 1600 MB/s at node1 --> do mount with "option nconnect=2" in your /etc/pve/storage.cfg nfs section.
Truenas read 5700 MB/s but just 700 MB/s at node1 --> can just get ~2900 MB/s because of 25Gbit interface but far away, nconnect fix.
The cmd "mount" should show somethink like this :
... nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=2,timeo=600,retrans=2,sec=sys, ...)
Don't know your zpool config and used disks but 5700 MB/s read vs. just 2500 MB/s write would just be acceptable if using some kind of ssd's/nvme's for vdev configs as for hdd's results should be near together. Or maybe you have lot of RAM, eg 512 GB and while compressed could easily cache the 640 GB benchmark data ... Did you check I/O on truenas while running with zpool iostat running ? And for pve's let sar -n DEV 1 running to see the network kB's ?
PS: For your node2 the write and read results are exactly the same as of node1, maybe a cp/paste error ... as even that 1600 MB/s write would not fit into your 10Gb interface at the fujitsu ...

Get this 2 files for systemctl and do after the 2 cmd's. At last rebench node1 (and show config of your zpool and used disk for):
systemctl daemon-reload
systemctl enable --now nfs-readahead-scan.timer

cat /etc/systemd/system/nfs-readahead-scan.service
[Unit]
Description=Set NFS read_ahead_kb for all current bdis

[Service]
Type=oneshot
ExecStart=/bin/bash -c "for m in $(cat /proc/fs/nfsfs/volumes|awk '{print $4}'|grep -v DEV);do if [ $(cat /sys/class/bdi/$m/read_ahead_kb) == 128 ];then echo 8192|tee /sys/class/bdi/$m/read_ahead_kb >/dev/null;fi;done"

cat /etc/systemd/system/nfs-readahead-scan.timer
[Unit]
Description=Periodically set read_ahead_kb for NFS bdis

[Timer]
OnBootSec=60s
OnUnitActiveSec=60s
AccuracySec=5s
Unit=nfs-readahead-scan.service

[Install]
WantedBy=timers.target

 

[orkiniec]

Thanks for looking at this. I will add nconnect=2 on PVE01 and rerun the test.
I did not have zpool iostat running on truenas during the original benchmark.
I did have PRTG monitoring the network interface that the truenas is connected to.
During the test it showed average traffic of approximately 14 Gbit/s.
There are currently 18 VMs running from this truenas datastore across the two Proxmox nodes, so the benchmark was not the only workload using the pool.
During the next test, I will capture the network kbs.
I monitored the network infrastructure and did not observe any interface errors, dropped packets, crc errors, or switch port errors on either node1, the truenas system, or the switch, if that matters?

The truenas has 128GB memory.
7x7.6tb nvme, single 6 drive RAIDZ1 vdev with hot spare

zpool status
 pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 00:16:39 with 0 errors on Sun Jul 12 00:16:40 2026
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            ba412352-ca57-4712-a2a2-42d9191433ec  ONLINE       0     0     0
            d4d20dcd-ee54-46f7-b686-74ae37ae502b  ONLINE       0     0     0
            6eb8c899-7ef6-43d1-a9d9-7ad396473114  ONLINE       0     0     0
            f85fb08a-84b7-4de8-a0fa-1f040ba8dfa1  ONLINE       0     0     0
            8013194d-be8c-4962-b431-a49450762406  ONLINE       0     0     0
            2f28d626-cc02-4789-93c5-c90d751f3c41  ONLINE       0     0     0
        spares
          983a0a97-a820-4f89-a269-c423b2b7e9ef    AVAIL   

errors: No known data errors

Yes Node2 was a paste error.

OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========

MKDIRS      Elapsed time     :         4ms         9ms
            Dirs/s           :        3170        4396
            Dirs total       :          15          40
            Dirs latency     : [ min=412us avg=718us max=1.06ms ]

---

WRITE       Elapsed time     :    2m1.896s    2m2.140s
            Files/s          :           0           0
            IOPS             :        8064        8048
            Throughput MiB/s :        1008        1006
            Total MiB        :      122880      122880
            Files total      :         117         120
            Files latency    : [ min=7.81s avg=8.13s max=8.63s ]
            IO latency       : [ min=66us avg=107us max=5.99ms ]

---

READ        Elapsed time     :   1m57.037s   1m57.889s
            Files/s          :           0           1
            IOPS             :        8334        8338
            Throughput MiB/s :        1041        1042
            Total MiB        :      121927      122880
            Files total      :         113         120
            Files latency    : [ min=6.66s avg=7.84s max=15.6s ]
            IO latency       : [ min=14us avg=956us max=639ms ]

---

RMFILES     Elapsed time     :        87ms       154ms
            Files/s          :        1250         778
            Files total      :         109         120
            Files latency    : [ min=1.95ms avg=7.57ms max=68.3ms ]

---

RMDIRS      Elapsed time     :         3ms         6ms
            Dirs/s           :       11979        5966
            Dirs total       :          40          40
            Dirs latency     : [ min=420us avg=607us max=807us ]

 

[waltar]

I monitored the network infrastructure and did not observe any interface errors, dropped packets, crc errors, or switch port errors on either node1, the truenas system, or the switch, if that matters?

Of course that matters but you observed nothing there which is fine !

During the test it showed average traffic of approximately 14 Gbit/s.
There are currently 18 VMs running from this truenas datastore across the two Proxmox nodes, so the benchmark was not the only workload using the pool.
During the next test, I will capture the network kbs.

Just 14Gbit during test is the result of your slow nfs connection yet, would be full at 23 Gbit when fixed (even with/without vm load at test time).

The truenas has 128GB memory.
7x7.6tb nvme, single 6 drive RAIDZ1 vdev with hot spare

nvme's have ~1/3 less write to read performance, so the difference for write/read 2500 vs 5700 MB/s still too high but maybe ok for a raidz1 and bench 640 GB for 128GB ram is a minimize cache effects - it is like it is.
So get the nfs readahead running and after find some off-workhours to again rebench pve1.

 

[orkiniec]

I have already added the nconnect=2 into the storage.cfg and will remount the nfs tonight and test that first.
Then this weekend i can do the nfs readahead setup this weekend and benchmark it.
I might also just move the vm off this node so i can test during the day.
I will say that the proxmox and truenas setup, even with the current issue I'm investigating is performing WAY better than setup on my other two dell r760 with esxi and onboard NVME storage. For the past 2 years i have been fighting with dell and vmware about the latency on the storage to no solution from either. On this platform most vm are showing 300 to 1500ms latency. I can't even run a windows update on the servers because it takes around 4 hours to do...
So crap i am happy with the speed i am seeing so far, i figured i just need to optimize everything.
This is a perfect example of what i am dealing with in my vmware enviroment....

 

[orkiniec]

ok here is the results with nconnect=2 and read ahead set to 8192.
I shutdown the 4 virtual machines on the server during this test.
It seems the write phase didn't increase but the read phase basically maxed out the connection.
The psi still was high during the test at 50-60%.

Switch port monitor updated every 10 seconds and reflected the same stats as below
Write Phase: 13Gbit/s
Read Phase: 22.56Gbit/s

the sar -n dev 1 file was fairly large so i had ai summarize it for me.

Phase    Time    Direction    Avg throughput    Peak    Avg %ifutil
Write Phase    07:24:54–07:31:56    TX-heavy (outbound)    ~1.5 GB/s tx    2.75 GB/s tx    52% (oscillating, ~14% of seconds near-idle)
Read Phase    07:31:57–07:35:40    RX-heavy (inbound)    ~2.86 GB/s rx    2.96 GB/s rx    96%, peaking 99.4%

Truenas iostat

Phase    Duration    Dominant I/O    Avg    Peak
Write-heavy    ~420s (~7 min)    Writes    ~2.7 GB/s write, negligible reads    3.30 GB/s write, up to ~33.8K write IOPS
Read-heavy    ~226s (~3m46s)    Reads    ~2.9 GB/s read, negligible writes    3.32 GB/s read, up to ~71K read IOPS

OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
MKDIRS      Elapsed time     :         1ms         2ms
            Dirs/s           :       16990       22137
            Dirs total       :          21          64
            Dirs latency     : [ min=94us avg=150us max=265us ]
---
WRITE       Elapsed time     :   6m59.739s    7m3.500s
            Files/s          :           1           1
            IOPS             :       12432       12379
            Throughput MiB/s :        1554        1547
            Total MiB        :      652288      655360
            Files total      :         633         640
            Files latency    : [ min=3.18s avg=5.27s max=8.48s ]
            IO latency       : [ min=33us avg=39us max=2.07ms ]
---
READ        Elapsed time     :   3m44.795s   3m45.241s
            Files/s          :           2           2
            IOPS             :       23277       23276
            Throughput MiB/s :        2909        2909
            Total MiB        :      654099      655360
            Files total      :         633         640
            Files latency    : [ min=2.16s avg=2.81s max=3.09s ]
            IO latency       : [ min=3us avg=337us max=227ms ]
---
RMFILES     Elapsed time     :       234ms       381ms
            Files/s          :        2584        1675
            Files total      :         606         640
            Files latency    : [ min=1.06ms avg=3.60ms max=133ms ]
---

 

[waltar]

So nfs read is good yet but write still lags with 2500 MB/s on server vs just 1500 MB/s on nfs client.
Just for a test set your truenas dataset to async by

zfs set sync=disabled yourpool/dataset

and benchmark once again.
After set back to default

zfs set sync=standard yourpool/dataset

Hopefully get then near 2500 MB/s at nfs client side and then you know if zfs or nfs is slowing down your write performance.

 

[waltar]

Apply the observed settings to your pve2 node and maybe get a 25Gbit card there too as you see that one is limited in performance as of lack network bandwidth yet because it's not that kind money to get it faster.

 

[orkiniec]

So nfs read is good yet but write still lags with 2500 MB/s on server vs just 1500 MB/s on nfs client.
Just for a test set your truenas dataset to async by

zfs set sync=disabled yourpool/dataset

and benchmark once again.
After set back to default

zfs set sync=standard yourpool/dataset

Hopefully get then near 2500 MB/s at nfs client side and then you know if zfs or nfs is slowing down your write performance.

actually, I already tested that earlier in my setup.
the dataset was originally configured with sync=always, and changing it to sync=standard made a very noticeable improvement in the day-to-day performance of the VMs.
I also tested sync=disabled, the benchmark results improved quite a bit, but I did not notice a major improvement in normal day-to-day vm performance compared to sync=standard.
I ended up switching it back to sync=standard because I have a truenas support contract, and that is the configuration they recommended and to stay within their supported configuration, I left it set to standard.
so I guess my real question at this point is about the proxmox io psi. should I just stop worrying about that metric?
if the io psi graph didn't exist, I honestly wouldn't know there was anything wrong.
The vm are responsive, users haven't reported any performance issues (rds servers), and after the nconnect=2 and read-ahead changes the read performance is now close to saturating the 25 gb link.
The write performance is also much better than what I was seeing with sync=always.
i'm just trying to determine if I'm chasing a metric at this point rather than an actual performance problem.
Today i was going to make the changes to the other node, that node is being used as a staging node for the conversion and will just be a dr node in the end so it won't be running day to day operations.

 

[waltar]

So if you happy with your environment and behavior of your vm's now you should not more worrying about.
Just for me I'm worrying about 6x 7,8TB nvme getting just 5700 MB/s read and 2500 MB/s write in raidz1 as a kind of raid5 which is just 1100 MB/s/nvme read and 500 MB/s/nvme write (1 for parity) which is the price to pay for zfs checksums etc instead of a hw-raid card, eg the older perc h965i give you 12-13 GB/s write in raid5/6 and up to 27 GB/s read (a h975i even double perf because gen5) before the bus bottleneck you if you have enough nvme's.

 

[orkiniec]

the local nvme storage in the dell has 8 × 3.5 tb nvme drives with the h965i in non-raid mode. zfs is configured as four mirrored pairs, so i would expect it to perform differently than the raidz1 setup on the truenas.
one of my main goals with this project was to have all three dell servers using shared storage. i could have gone with ceph for ha, but after the storage latency issues i have been fighting with under esxi, i did not want to depend on the local nvme until i knew whether the same issue would show up under proxmox.
Before doing anything i had prtg monitoring the bandwidth being used on my esxi server for about 3 weeks. in my esxi production environment all 36 vms combined averaged around 471 mb/s of storage bandwidth during business hours. so even though the truenas benchmark numbers are lower than the local mirrored nvme, the current nfs performance is still well above what my normal production workload is using.

regardless, i am going to contact truenas support about the local read and write numbers to see what they say and whether there is a reason they are not higher.

waltar, i cannot thank you enough for all of your help with this. you have been an amazing wealth of information, and i really appreciate you taking the time to help a random guy work through all of this.
you are a huge asset to these forums, and i truly appreciate all of the time you've spent helping me.

 

[waltar]

in my esxi production environment all 36 vms combined averaged around 471 mb/s of storage bandwidth during business hours. so even though the truenas benchmark numbers are lower than the local mirrored nvme, the current nfs performance is still well above what my normal production workload is using.

Thanks. But be carefull with your before measured workload and possible nfs bandwidth because if (the vm remote) I/O reaction is low you just can measure small numbers and then don't think of these is what you only need of - that's s an easy fallacy ... like eg now downgrade your network to just 4Gbit as that's even enough for "my workload as I measured already" ...
VM responsible comes from low image latency and the faster it is the less is the possibility to overlap one vm I/O with that of another which when occur combined latency go up for both (and in reality there are many vm's fighting for I/O on a hypervisor), so tuning I/O is always the best and normally cheapest way for nice running env.