Slow 100G network performance

deepcloud

Member
Feb 12, 2021
67
11
13
India
deepcloud.in
Hi Folks,

We have 100G Ethernet (Mellanox HP SFP28 100G Cards) on each Node of our Proxmox Nodes (Running AMD EPYC Processors, so PCI-E Bandwidth is not a problem)

We have 6.4TB SN630 Enterprise NVME SSD in the nodes on a CEPH Cluster so this is not a bottleneck either as its on the 100G Link too.

If we do a VM to VM copy (with Virtio Network card and the latest drivers installed).

The speed is only at 7-8 Gbps (Gigabits) on a iptraf test on a Windows and Linux mix and 13-14 Gbps on a Linux / Linux VM mix.

This is no way near the 100G or even 25G (as 100G is basically 25G x 4).

Any ideas on how we can get good speeds on this kind of hardware.

By the way we have a Arista 7060CX 32 Port 100GBE Switch
 
iperf2 or iperf3 ?

iperf3 is single threaded, so it's possible to be cpu/core limited. you need to use multiple iperf3 in parallel.

iperf2 is multi-threaded, you can use -P option to use multiple steams.
 
I just found this question and wanted to add that I have 25G networking and wanted to give one of my VMs faster iSCSI and I notice that the VirtIO driver in Windows says it is 10GbE. I am now wondering if there is a way to make VirtIO 25G into the virtual switch?
 
With latest Virtio driver show speed link at 10Gbit but they can go over, give a try with your 25Gbit ;)
I suspect my NAS is running out of CPU, I get iperf3 at about 6Gbps and iSCSI at 9.36Gbps, not yet reaching the 25Gbps potential.

Irrelevant to the original topic, so sorry if that offends anyone. But interesting to try and demonstrate more than 10G on my rig.
 
I suspect my NAS is running out of CPU, I get iperf3 at about 6Gbps and iSCSI at 9.36Gbps, not yet reaching the 25Gbps potential.

Irrelevant to the original topic, so sorry if that offends anyone. But interesting to try and demonstrate more than 10G on my rig.
WIth iperf try to give more than 1 in parallel:

-P, --parallel nThe number of simultaneous connections to make to the server. Default is 1.
 
  • Like
Reactions: bobdvb
WIth iperf try to give more than 1 in parallel:

-P, --parallel nThe number of simultaneous connections to make to the server. Default is 1.
Yea, I went from 4Gbps to 6Gbps with 4 parallel and 6 made no more difference.
 
I would eliminate the storage and vm for a baseline test and do iperf or nuttcp testing proxmox node to node, eliminate any network/node issues before you try inside the vm's.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!