Very slow iperf speeds (~370Mbps) with proxmox host 10Gbps NIC

[ramreddy]

Only thing left is to test in direct wired connection mode. Will reply once I get the results.

 

[_gabriel]

what is the ip of your Windows host ?

 

[Frew]

I have the same problem. Still trying to find the solution.
Try not to use Linux Bridge and run iperf again.

Upd. The problem was in my main router, not in proxmox.

 

[ramreddy]

what is the ip of your Windows host ?

192.168.50.55 is my windows IP. For some reason, with automatic DHCP I was not able to connect to Proxmox portal or SSH.
I have statically configured IP and now I am able to SSH and connect to web UI.

iPerf speeds still stayed low though. Did iperf tests with a 2nd windows machine which also resulted in same low speeds.

 

[ramreddy]

I have the same problem. Still trying to find the solution.
Try not to use Linux Bridge and run iperf again.

Going through bit of learning curve, will have to read on how to replace current linux bridge.
Any pointers you can share on migrating from Linux bridge to OVS bridge will help a lot. Thanks

 

[Dunuin]

In case CPU performance is bottlenecking it might help to use jumboframes. With 1500 MTU my 10Gbit NIC on Xeon E5 v4 can only handle 3-4 Gbit. With 9000 MTU the CPU needs to process less packets, as the packets are bigger and I get the full 10Gbit. But of cause only in case all your hardware supports jumboframes.

 

[ramreddy]

In case CPU performance is bottlenecking it might help to use jumboframes. With 1500 MTU my 10Gbit NIC on Xeon E5 v4 can only handle 3-4 Gbit. With 9000 MTU the CPU needs to process less packets, as the packets are bigger and I get the full 10Gbit. But of cause only in case all your hardware supports jumboframes.

I have tried setting MTU to 9000 already. No difference in performance.
Would be happy if I got 3-4 Gbps but speed I am getting is in the range of 0.3 Gbps.

 

[Dunuin]

Is there maybe any routing involved? Fast speed you ususally only get when directly communicating within the same subnet.

 

[ramreddy]

Is there maybe any routing involved? Fast speed you ususally only get when directly communicating within the same subnet.

Not sure how to check this, but both proxmox host and windows machine have same gateway and subnet mask.

 

[Dunuin]

Then this usually should be fine. Just strange that you told us that webUI and SSH won'T work when using DHCP but are working with a static IP. I thought maybe DHCP and static IP were using different subnets or something similar.

 

[ramreddy]

Then this usually should be fine. Just strange that you told us that webUI and SSH won'T work when using DHCP but are working with a static IP. I thought maybe DHCP and static IP were using different subnets or something similar.

Home network is using pretty basic setup of single router, single gateway. Proxmox Server connected to 10gbps port and rest of machines through wifi. No additional customization on defaults.

I disabled firewall fully on widows machines, but static IP config works. I also have no clue on what is happening

 

[Frew]

Going through bit of learning curve, will have to read on how to replace current linux bridge.
Any pointers you can share on migrating from Linux bridge to OVS bridge will help a lot. Thanks

https://pve.proxmox.com/wiki/Open_vSwitch
You can try to use the official guide.

 

[_gabriel]

edit manually /etc/network/interfaces
to try with the 1GB nic
then try the 10 GB without vmbr0

 

[ramreddy]

Will take sometime to get back with results given I might need to learn and try out some stuff as suggested above.

 

[Frew]

If I were you I would run WinPE (LiveCD) and run iperf there. Thus you can understand what type of problem it is hardware or software.

And your card has 2 ports, right?

 

[ramreddy]

If I were you I would run WinPE (LiveCD) and run iperf there. Thus you can understand what type of problem it is hardware or software.

And your card has 2 ports, right?

Yes, my Ethernet card has two ports and router has two ports for 10gbps.

Are you suggesting that I run live cd connected to proxmox host or my desktop? I didnt understand where I should run winPE and how it's network should be connected with 10gbps ports.

 

[_gabriel]

indeed missing the second port, post your /etc/network /interfaces into the code tag (not the inline code)

 

[ramreddy]

indeed missing the second port, post your /etc/network /interfaces into the code tag (not the inline code)

ens3f0 and ens3f1 are 10gbps ports.

root@pve-1:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface ens3f0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.50.50/24
        gateway 192.168.50.1
        bridge-ports ens3f0
        bridge-stp off
        bridge-fd 0

iface eno1 inet manual

iface eno2 inet manual

iface ens3f1 inet manual

 

[_gabriel]

replace ens3f1 line by

auto ens3f1
iface ens3f1 inet static
        address 192.168.50.2/24

then Reboot, then iperf3 to 192.168.50. 2
this will bypass the Linux bridge.

 

[ramreddy]

replace ens3f1 line by

auto ens3f1
iface ens3f1 inet static
        address 192.168.50.2/24

then Reboot, then iperf3 to 192.168.50. 2
this will bypass the Linux bridge.

Had to wait for additional cat8 cable to connect to another 10gbps port from router to NIC.

PS C:\Users\ramap\Downloads\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.50.2
Connecting to host 192.168.50.2, port 5201
[  4] local 192.168.50.100 port 56998 connected to 192.168.50.2 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  40.2 MBytes   337 Mbits/sec
[  4]   1.00-2.00   sec  47.8 MBytes   400 Mbits/sec
[  4]   2.00-3.00   sec  47.4 MBytes   397 Mbits/sec
[  4]   3.00-4.00   sec  45.1 MBytes   379 Mbits/sec
[  4]   4.00-5.00   sec  48.0 MBytes   403 Mbits/sec
[  4]   5.00-6.00   sec  44.9 MBytes   376 Mbits/sec
[  4]   6.00-7.00   sec  45.8 MBytes   385 Mbits/sec
[  4]   7.00-8.00   sec  42.9 MBytes   359 Mbits/sec
[  4]   8.00-9.00   sec  42.5 MBytes   357 Mbits/sec
[  4]   9.00-10.00  sec  42.1 MBytes   354 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   447 MBytes   375 Mbits/sec                  sender
[  4]   0.00-10.00  sec   447 MBytes   375 Mbits/sec                  receiver

iperf Done.