How to create 100Gbit/s network in Proxmox between VM`s?

[MacronAVG]

Good day.
I have two node`s with Mellanox ConnectX-5 100Gbit/s EDR InfiniBand. I want to have 100Gbit/s between VM`s. I`m using VirtIO Network driver and get only 20Gbit/s. I`m using "Iperf" for calculating network speed.
Between two physical hosts I have 93Gbit/s (7 threads)
When i was checking Network Interface and bridge (Physical hosts) I got 100000MB/s (look at screenshot)
Who know`s how to overcome this speed limit?

 

[Dunuin]

Virtio NICs are bascially limited by the performance of your CPU. So I guess your CPU is the bottleneck there. You might need to passthrough the NIC (or a function when using SR-IOV) to the VMs for better performance. So the VMs can directly access the physical hardware without the virtualization slowing it down in between.

 

[MacronAVG]

Virtio NICs are bascially limited by the performance of your CPU. So I guess your CPU is the bottleneck there. You might need to passthrough the NIC (or a function when using SR-IOV) to the VMs for better performance. So the VMs can directly access the physical hardware without the virtualization slowing it down in between.

Thank you for your reply. I`ll try to do it and answer you

 

[spirit]

you can also enable multiqueues on vm nic.

about iperf:
you need use parallel stream (-P) to be able to use multiqueue.

also, iperf3 is single threaded (even with -P), so it can be bottleneck by cpu, so you need to launch multiple iperf3 in //.

iperf2 is multithreaded (when using -P)


Now, 100gbit/s in vm with virtio will be difficult to achieve; (it's really depend of the packet size).

 

[MacronAVG]

you can also enable multiqueues on vm nic.

about iperf:
you need use parallel stream (-P) to be able to use multiqueue.

also, iperf3 is single threaded (even with -P), so it can be bottleneck by cpu, so you need to launch multiple iperf3 in //.

iperf2 is multithreaded (when using -P)


Now, 100gbit/s in vm with virtio will be difficult to achieve; (it's really depend of the packet size).

Hello.
Thank you for your answer.
I`ve already checked network speed with multiqueues on a vm nic. It has got +10-15Gbit/s (general 40Gbit/s but unstably).
About multithread: All my network`s speed tests were multithread (5-10 threads).
I checked passthrough and got 100Gbit/s. But I got some new problems:
1) Only one VM can use one PCI-E device:
2) My node can`t use this PCI-E device when it`s using on a VM.

 

[MacronAVG]

Virtio NICs are bascially limited by the performance of your CPU. So I guess your CPU is the bottleneck there. You might need to passthrough the NIC (or a function when using SR-IOV) to the VMs for better performance. So the VMs can directly access the physical hardware without the virtualization slowing it down in between.

Hello.
I`ve checked passthrough and got 100Gbit/s. But I got some new problems:
1) Only one VM can use one PCI-E device. When I was trying to start second VM with this PCI-E adapter:

"qemu-system-x86_64: -device vfio-pci,host=0000:5e:00.0,id=hostpci0.0,bus=pci.0,addr=0x10.0,multifunction=on: vfio 0000:5e:00.0: failed to open /dev/vfio/94: Device or resource busy"
TASK ERROR: start failed: QEMU exited with code 1

2) Physical node can`t use this PCI-E device when it`s using on a VM (I can`t ping other devices in the local network).
Do you know how to overcome these problems?

 

[spirit]

if you want to share 1 nic with multiple vms, you need some kind of virtual functions support in your nic drivers.

It'll regenerate something like 0000:5e:00.1, 0000:5e:00.2, 0000:5e:00.3

look this doc for mellanox card :

https://shawnliu.me/post/configuring-sr-iov-for-mellanox-adapters/

 

[MacronAVG]

if you want to share 1 nic with multiple vms, you need some kind of virtual functions support in your nic drivers.

It'll regenerate something like 0000:5e:00.1, 0000:5e:00.2, 0000:5e:00.3

look this doc for mellanox card :

https://shawnliu.me/post/configuring-sr-iov-for-mellanox-adapters/

Hello.
Thank you very much for information.
I did it and got many Virtual devices but there are strange things:
1) Windows Server was creating many virtual devices but I`d added only one virtual interface (screenshot);
2) VM can ping only itself. I can`t ping other devices (Exceptiong: Ubuntu Desktop (screenshot)). In Physical host I can`t ping VM`s but I can ping other devices;
3) After restart all virtual interfaces disappear and I need to change this file again: "echo 100 > /sys/class/net/enp94s0f1np1/device/sriov_numvfs"

 

[spirit]

Hello.
Thank you very much for information.
I did it and got many Virtual devices but there are strange things:
1) Windows Server was creating many virtual devices but I`d added only one virtual interface (screenshot);

uncheck "all functions" in the gui

3) After restart all virtual interfaces disappear and I need to change this file again: "echo 100 > /sys/class/net/enp94s0f1np1/device/sriov_numvfs"

you can add a post-up script in /etc/network/interfaces

auto enp94s0f1np1
inet iface enp94s0f1np1 ...
     ....
     post-up "echo 100 > /sys/class/net/enp94s0f1np1/device/sriov_numvfs"

it seem also possible to use "max_vfs" as module option

try to add a file "/etc/modprobe.d/mlx5_core.conf
with

options mlx5_core max_vfs="100"

 

[alpha754293]

Virtio NICs are bascially limited by the performance of your CPU. So I guess your CPU is the bottleneck there. You might need to passthrough the NIC (or a function when using SR-IOV) to the VMs for better performance. So the VMs can directly access the physical hardware without the virtualization slowing it down in between.

My two Ryzen 9 5950X, inside the VM, according to htop wasn't using more than 25% CPU on any particular core.

I'm limited to 23.4-ish Gbps going VM-to-VM through a Linux Network Bridge (which has one of the two 100 Gbps ports from my Mellanox ConnectX-4 card, set to ETH link type), and connected to each other via a DAC. (that's with 1 stream. 4 parallel streams, that drops down to 23.1 Gbps, and with 8 parallel streams, that drops that to 23.0 Gbps.)



Host to host, I can hit 96.9 Gbps, also tested using iperf.


re: virtual functions
So, as I've mentioned, I've enabled SR-IOV (at least on the IB side of things).

The problem that I am running into now is that I can't set the 20-byte IB MAC address for the virtual functions, because when I tried to do that via, the error message that I get is:

# ip link set ibp8s0f0 vf 0 mac 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:22:33:44:55:66
RTNETLINK answers: Operation not supported

So, I haven't been able to really "use" said virtual functions with a VM, because it inherited the MAC address from port0 on the physical function.

The instructions that I have found are either for Red Hat based linux distros and/or that it is the official documentation from Mellanox's OFED drivers, which as I've been googling this, trying to find an answer, someone else had said that they kept "breaking" their system and that Mellanox's OFED driver apparently doesn't play nicely with Debian and/or Debian-based linux distros.

(This is why I ended up setting one of the ports on each node/NIC to be the ETH link type, so that I can create a Linux (ethernet) network bridge, for use between VMs and/or CTs, inside of Proxmox.)