TrueNAS - Same host low read normal write via smb.

[spdy42]

Hello,

i`m currently not sure where to look at.
Running a workstation with multiple vm guests and gpu-passthrough, as storage one of these vms is a TrueNAS instance with two 7.68TB SSD via passthrough on the same host.
They can achiev 3000/2000 read/write which they did before handing them over to TrueNAS.

But now i`m only seeing ~1000/2000 read/write when accessed via SMB from a Windows VM on the same host.

As network i configured them all as paravirtualized (Virtio), im kinda baffled why i`m seeing expected write speeds but only 1000 instead of 3000 on read (testing with CrystalDiskMark).

Can someone please point me into a direction how to diagnose this problem, machine has plenty of cpu/ram for this usecase.

thanks

 

[bbgeek17]

Running a workstation with multiple vm guests and gpu-passthrough, as storage one of these vms is a TrueNAS instance with two 7.68TB SSD via passthrough on the same host.

Do you have enough CPU horsepower for all of them? Is it assigned in optimal way to avoid cross-NUMA talk and prevent.
Try to reduce number of variables (number of VMs), test with different protocols (ftp/nfs), clients (linux vs windows), etc.
Test just the network with iperf type program.
The simpler that you make your environment for troubleshooting, the easier it will be to find the culprit.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[spdy42]

It cant be an cpu issue epyc 7302p on an h12ssl-nt with 256gb ecc ddr4 more than enough pci lanes and this should not congest the bus.
Truenas instance has 8 cores with 32 gb ram and never goes above 40% cpu.

SSD are 2x WD server ssds connected via u.2

And looking at iperf3 from the linux vm to truenas vm the bandwith is there even if it should probably be more.

iperf3 -c 10.0.3.3 -f g

Connecting to host 10.0.3.3, port 5201
[ 5] local 10.0.2.144 port 43364 connected to 10.0.3.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.66 GBytes 22.9 Gbits/sec 0 3.09 MBytes
[ 5] 1.00-2.00 sec 2.60 GBytes 22.3 Gbits/sec 0 3.09 MBytes
[ 5] 2.00-3.00 sec 2.57 GBytes 22.1 Gbits/sec 0 3.09 MBytes
[ 5] 3.00-4.00 sec 2.29 GBytes 19.7 Gbits/sec 0 3.09 MBytes
[ 5] 4.00-5.00 sec 2.30 GBytes 19.8 Gbits/sec 0 3.09 MBytes
[ 5] 5.00-6.00 sec 2.38 GBytes 20.4 Gbits/sec 0 3.09 MBytes
[ 5] 6.00-7.00 sec 2.42 GBytes 20.8 Gbits/sec 0 3.09 MBytes
[ 5] 7.00-8.00 sec 2.33 GBytes 20.0 Gbits/sec 0 3.09 MBytes
[ 5] 8.00-9.00 sec 2.31 GBytes 19.9 Gbits/sec 0 3.09 MBytes
[ 5] 9.00-10.00 sec 2.28 GBytes 19.6 Gbits/sec 0 3.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 24.1 GBytes 20.7 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 24.1 GBytes 20.6 Gbits/sec receiver

But it looks like the windows vm has some issues.

.\iperf3.exe -c 10.0.3.3
Connecting to host 10.0.3.3, port 5201
[ 4] local 10.0.2.146 port 57478 connected to 10.0.3.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 788 MBytes 6.61 Gbits/sec
[ 4] 1.00-2.00 sec 742 MBytes 6.23 Gbits/sec
[ 4] 2.00-3.00 sec 733 MBytes 6.15 Gbits/sec
[ 4] 3.00-4.00 sec 771 MBytes 6.47 Gbits/sec
[ 4] 4.00-5.00 sec 786 MBytes 6.60 Gbits/sec
[ 4] 5.00-6.00 sec 778 MBytes 6.53 Gbits/sec
[ 4] 6.00-7.00 sec 816 MBytes 6.84 Gbits/sec
[ 4] 7.00-8.00 sec 812 MBytes 6.81 Gbits/sec
[ 4] 8.00-9.00 sec 823 MBytes 6.90 Gbits/sec
[ 4] 9.00-10.00 sec 812 MBytes 6.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.68 GBytes 6.59 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.68 GBytes 6.59 Gbits/sec receiver