Slow 100G network performance

[deepcloud]

Hi Folks,

We have 100G Ethernet (Mellanox HP SFP28 100G Cards) on each Node of our Proxmox Nodes (Running AMD EPYC Processors, so PCI-E Bandwidth is not a problem)

We have 6.4TB SN630 Enterprise NVME SSD in the nodes on a CEPH Cluster so this is not a bottleneck either as its on the 100G Link too.

If we do a VM to VM copy (with Virtio Network card and the latest drivers installed).

The speed is only at 7-8 Gbps (Gigabits) on a iptraf test on a Windows and Linux mix and 13-14 Gbps on a Linux / Linux VM mix.

This is no way near the 100G or even 25G (as 100G is basically 25G x 4).

Any ideas on how we can get good speeds on this kind of hardware.

By the way we have a Arista 7060CX 32 Port 100GBE Switch

 

[spirit]

iperf2 or iperf3 ?

iperf3 is single threaded, so it's possible to be cpu/core limited. you need to use multiple iperf3 in parallel.

iperf2 is multi-threaded, you can use -P option to use multiple steams.

 

[deepcloud]

ok let me re-check with iperf2

 

[bobdvb]

I just found this question and wanted to add that I have 25G networking and wanted to give one of my VMs faster iSCSI and I notice that the VirtIO driver in Windows says it is 10GbE. I am now wondering if there is a way to make VirtIO 25G into the virtual switch?

 

[Emilien]

With latest Virtio driver show speed link at 10Gbit but they can go over, give a try with your 25Gbit

 

[bobdvb]

With latest Virtio driver show speed link at 10Gbit but they can go over, give a try with your 25Gbit

I suspect my NAS is running out of CPU, I get iperf3 at about 6Gbps and iSCSI at 9.36Gbps, not yet reaching the 25Gbps potential.

Irrelevant to the original topic, so sorry if that offends anyone. But interesting to try and demonstrate more than 10G on my rig.

 

[Emilien]

I suspect my NAS is running out of CPU, I get iperf3 at about 6Gbps and iSCSI at 9.36Gbps, not yet reaching the 25Gbps potential.

Irrelevant to the original topic, so sorry if that offends anyone. But interesting to try and demonstrate more than 10G on my rig.

WIth iperf try to give more than 1 in parallel:

-P, --parallel n

The number of simultaneous connections to make to the server. Default is 1.

 

[bobdvb]

WIth iperf try to give more than 1 in parallel:

-P, --parallel n

The number of simultaneous connections to make to the server. Default is 1.

Yea, I went from 4Gbps to 6Gbps with 4 parallel and 6 made no more difference.

 

[Emilien]

Yea, I went from 4Gbps to 6Gbps with 4 parallel and 6 made no more difference.

Ah ok, maybe you can try to pcie passthrough the network card directly to the Windows VM and check the speed with iperf...

 

[tjk]

I would eliminate the storage and vm for a baseline test and do iperf or nuttcp testing proxmox node to node, eliminate any network/node issues before you try inside the vm's.