[SOLVED] Truenas VM Only 1Gbe instead of 10Gbe

[kocy]

Hello,

I have a Truenas VM that only gets very poor network performance of only 1gbe while the VM is connected at 10gbe.
I even passed through the network interface to the vm so that not even the bridge is a potential issue.
I first thought maybe the RaidZ2 throughput is to blame, but this appears to be very bad performance for a vdev 6 (6drive array).
I have tried copying files form multiple machines in different networks.
1. I tried from Wifi and I get approx. 150mb/s (from a Unifi U7 Pro XGS with 10gbe)
2. I tried from a cabled Windows client. 120mb/s
3. I tried from another VM on the same network bridge of Truenas. 350mb/s. (that seems better but still slow)

Then i tried a iperf from the Wlan Device. this is the result:
C:\iperf>.\iperf3.exe -c 192.168.50.9
Connecting to host X, port 5201
[ 5] local X port 49229 connected to X port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 105 MBytes 869 Mbits/sec
[ 5] 1.01-2.01 sec 146 MBytes 1.23 Gbits/sec
[ 5] 2.01-3.00 sec 144 MBytes 1.21 Gbits/sec
[ 5] 3.00-4.01 sec 161 MBytes 1.34 Gbits/sec
[ 5] 4.01-5.01 sec 162 MBytes 1.36 Gbits/sec
[ 5] 5.01-6.00 sec 166 MBytes 1.41 Gbits/sec
[ 5] 6.00-7.00 sec 158 MBytes 1.33 Gbits/sec
[ 5] 7.00-8.01 sec 154 MBytes 1.28 Gbits/sec
[ 5] 8.01-9.02 sec 166 MBytes 1.39 Gbits/sec
[ 5] 9.02-10.00 sec 160 MBytes 1.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.49 GBytes 1.28 Gbits/sec sender
[ 5] 0.00-10.01 sec 1.48 GBytes 1.27 Gbits/sec receiver

I thought maybe the PCIe lanes are amiss and i checked if the throughput of the network card is to blame but:
lspci -vv -s 42:00.0 | grep LnkSta
LnkSta: Speed 5GT/s, Width x8
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-

All devices show 10gbe in there gui as negotiated speed.
I am baffled what else it could be.
As the hba the network card is passedthrough, so maybe that is a Truenas bug and I should redirect this problem there?
Post in their forum?

Thanks in advance for any advice as i am out of ideas...

Specs of the system:
proxmox Virtual Environment 8.4.14
Linux pve 6.8.12-15-pve
Truenas: ElectricEel-24.10.2
Network card: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Output of ethtool
root@pve:~# ethtool -k enp66s0f1
Features for enp66s0f1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: on [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: on
esp-tx-csum-hw-offload: on
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

 

[DerekG]

The iperf test is showing you the problem, here is what I get with iperf3 over 10Gbe

root@pve-5:~# iperf3 -c 192.168.10.28
Connecting to host 192.168.10.28, port 5201
[  5] local 192.168.10.50 port 39354 connected to 192.168.10.28 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.07 GBytes  9.15 Gbits/sec  152    634 KBytes       
[  5]   1.00-2.00   sec  1.07 GBytes  9.15 Gbits/sec  147    221 KBytes       
[  5]   2.00-3.00   sec  1.04 GBytes  8.97 Gbits/sec  702    576 KBytes       
[  5]   3.00-4.00   sec  1.04 GBytes  8.92 Gbits/sec  738    396 KBytes       
[  5]   4.00-5.00   sec  1.05 GBytes  8.98 Gbits/sec  571    399 KBytes       
[  5]   5.00-6.00   sec  1.03 GBytes  8.83 Gbits/sec  145    679 KBytes       
[  5]   6.00-7.00   sec   997 MBytes  8.37 Gbits/sec   46    936 KBytes       
[  5]   7.00-8.00   sec   944 MBytes  7.92 Gbits/sec  167    576 KBytes       
[  5]   8.00-9.00   sec  1.02 GBytes  8.78 Gbits/sec  142    628 KBytes       
[  5]   9.00-10.00  sec  1.06 GBytes  9.12 Gbits/sec  197    568 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.82 Gbits/sec  3007            sender
[  5]   0.00-10.00  sec  10.3 GBytes  8.82 Gbits/sec                  receiver

Do you have a 1Gbe media converter plugged into the SFP+ port???

 

[kocy]

ok... Sorry it wasn't a Proxmox issue it seems. Marking this as solved.
I dug deeper and I upgraded the firmware of my old aruba 2930f.
It was showing 10gbe link before so I didnt think of it being cause of concern.
After the updade I was checking smb speed to truenas and it was still only 1gbe.

Then i enabled smb multipath and then I got the full expected speed from a cabled client!

I tried from WLAN again and the Speed didnt improve.
Until I looked at the "Wireless Lan Card" of the Laptop and realized that its a "MediaTek Wi-Fi 7 MT7925" and I guess it won't perfom to well...

I am still searching for a performance smb client for android to use my mobile, to check If the U7 Pro XGS is limiting. Is there an Iperf3 for android?

Thank you DerekG for trying to help, wishing a nice weekend.