Linux Bridge between 2 VMs - What is maximum speed?

[telesteven]

Hi
I installed two linux guest VMs (Debian 11) and connected them to a Linux Bride Network device that I created in Proxmox.
Using iperf3 I get around 20 GBit/s between the VMs and I'm wondering if this is really the max or if I am missing something. I couldn't find any hint about theoretical maximum values elsewhere...
The main problem with only 20 GBit/s is the limit is that I want to transfer data from one VM to the other which is connected to a fast NVMe raid array which can handle around 10 GB/s

Thanks

 

[Dunuin]

As far as I understand with virtio NICs and bridges you are basically limited to the performance of your CPU. Which is usually around 20Gbit. You could try to setup multiqueue for your virtio NICs so the guest could use multiple vCPUs for processing receiving packets.

 

[telesteven]

Hm, I set queues=4 on both NICs, still the same.
I also noticed that the CPU core is only around 40-50% when reaching the 20 Gbit/s

 

[androme13]

Hello, with Intel X710 DA2 NIC, and enabled sr-iov and intel nic virtual function i can reach ~40 Gbits between 2 VMs without hammering my AMD 3900X CPU (It reduces overhead of guests too), with normal linux bridge you will be limited by your host's CPU.

 

[telesteven]

I noticed that I can get around 40 Gbit/s between two containers... so I'm not sure if CPU is really the limiting factor between the two VMs.
sr-iov is an ideal. I'll give it a try later

 

[Dunuin]

LXCs shouldn't use NIC virtualizations. So CPUs singlethreaded performance could still bottleneck virtio NICs.