VM Networking Performance with Mellanox Connect X-3

[hello_world6767]

Hello, I am relatively new to Proxmox and virtualization in general and wanted some advice regarding networking performance with the card mentioned in the title, they are Oracle-branded Mellanox Connect X-3 cards cross flashed to Mellanox firmware running in VPI mode connected to a Mellanox SX6036 running stock firmware in Ethernet mode. I am having trouble pushing past 20Gb/s between VMs and hosts (benchmarked using iperf3), but was able to achieve 39.5Gb/s between hosts. In the VM's NIC I set multiqueue to equal the number of vCPUs, which is 12, and set the MTU to 9000 throughout the whole system (on the physical interface to any bridges and vlans derived from it, as well as the switch). I ensured that the multiqueues were actually being used by the interface within the VM with ethtool -l and -L.

That is my current situation but I started out with 6Gb/s single stream performance and was able to get it up to 20Gb/s, and noticed that if I increased the number of client streams the individual stream performance decreases such that the combined stream performance still remains at around 20Gb/s. I also tested two clients simultaneously (with the VM as the server, and two iperf3 server processes), and was able to get ~15Gb/s per client, which combined is more than 20Gb/s.

Does anyone have any experience with this? Have I hit a performance limit with my hardware with VMs?

 

[complexplaster27]

What driver are you using to run the Mellanox cards?

For reference this is what I'm using in our production servers for ceph storage and I can't saturate the mesh network that way using the Debian Linux Firmware drivers, granted I can't tinker with these as they are deployed:

lspci -nnk | grep -A3 -i mell
0000:5c:00.0 Ethernet controller [0200]: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] [15b3:101d]
        Subsystem: Mellanox Technologies Device [15b3:0042]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
0000:5c:00.1 Ethernet controller [0200]: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] [15b3:101d]
        Subsystem: Mellanox Technologies Device [15b3:0042]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

I did read on this forum that Nvidia DOCA_OFED may run faster https://developer.nvidia.com/networking/doca but you have to be willing to play around and install the drivers correctly https://developer.nvidia.com/doca-downloads?deployment_platform=Host-Server

Here's a thread on it if you want to tinker around with it: https://forum.proxmox.com/threads/installing-doca-ofed-on-proxmox-9-0.173376/

But personally I'm happy to run with the mlx5_core drivers as they get updated with each linux firmware upgrade and I've never had breakages. Straying away from the mlx5_core on Proxmox is very much uncharted territory and you may need to get your hands dirty.