Poor VM networking performance

[ams123]

I am running into some VM network speed issues that I first noticed when SMB share upload speeds on my OpenMediaVault VM dropped, previously I could saturate a 1 Gbits/sec connection shown by 120 MBytes/sec upload speeds. Doing some investigating with iPerf3 I think I have isolated the issue to the bridge performance.

Results
-Proxmox host iperf3 results shows it will saturate the 1 gbps connection, 940 Mbits/sec.
-Running only a single VM on the same connection as host, vmbr0, is about 660 Mbits/sec but it’s highly variable
-Running VM to VM on the same bridge, vmbr1, is about 1.14 Gbits/sec but again highly variable
-I switched vmbr1 from a linux bridge to a OVS bridge and did see some improvement but that is reflected in the above results

Proxmox Install

​

CPU(s) 4 x Intel(R) Atom(TM) CPU C2550 @ 2.40GHz (1 Socket)​

Kernel Version Linux 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100)​

PVE Manager Version pve-manager/7.3-3/c3928077​

Repository Status Proxmox VE updates Production-ready Enterprise repository enabled​

/etc/network/interfaces

auto lo​

iface lo inet loopback​

​

iface eno1 inet manual​

​

auto eno2​

iface eno2 inet manual​

​

auto eno3​

iface eno3 inet manual​

​

iface eno4 inet manual​

​

auto bond0​

iface bond0 inet manual​

ovs_bonds eno2 eno3​

ovs_type OVSBond​

ovs_bridge vmbr1​

ovs_options lacp=active bond_mode=balance-slb​

​

auto vmbr0​

iface vmbr0 inet static​

address 192.168.10.3/24​

gateway 192.168.10.1​

bridge-ports eno1​

bridge-stp off​

bridge-fd 0​

hwaddress ac:1f:6b:66:b7:24​

#Proxmox Server​

​

auto vmbr1​

iface vmbr1 inet manual​

ovs_type OVSBridge​

ovs_ports bond0​

​

auto vmbr2​

iface vmbr2 inet manual​

bridge-ports eno4​

bridge-stp off​

bridge-fd 0​

#WAN​

Proxmox host iperf3

​

Connecting to host 192.168.10.21, port 5201​

[ 5] local 192.168.10.3 port 58232 connected to 192.168.10.21 port 5201​

[ ID] Interval Transfer Bitrate Retr Cwnd​

[ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 0 577 KBytes​

[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 699 KBytes​

[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 891 KBytes​

[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 891 KBytes​

[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 891 KBytes​

[ 5] 5.00-6.00 sec 110 MBytes 923 Mbits/sec 0 936 KBytes​

[ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 1.01 MBytes​

[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 1.06 MBytes​

[ 5] 8.00-9.00 sec 111 MBytes 936 Mbits/sec 76 571 KBytes​

[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 39 431 KBytes​

[ 5] 10.00-11.00 sec 112 MBytes 944 Mbits/sec 0 465 KBytes​

[ 5] 11.00-12.00 sec 111 MBytes 933 Mbits/sec 0 496 KBytes​

[ 5] 12.00-13.00 sec 111 MBytes 933 Mbits/sec 0 506 KBytes​

[ 5] 13.00-14.00 sec 112 MBytes 944 Mbits/sec 0 513 KBytes​

[ 5] 14.00-15.00 sec 111 MBytes 933 Mbits/sec 0 515 KBytes​

[ 5] 15.00-16.00 sec 111 MBytes 933 Mbits/sec 0 515 KBytes​

[ 5] 16.00-17.00 sec 111 MBytes 933 Mbits/sec 0 518 KBytes​

[ 5] 17.00-18.00 sec 110 MBytes 923 Mbits/sec 36 396 KBytes​

[ 5] 18.00-19.00 sec 110 MBytes 923 Mbits/sec 0 451 KBytes​

[ 5] 19.00-20.00 sec 112 MBytes 944 Mbits/sec 0 486 KBytes​

- - - - - - - - - - - - - - - - - - - - - - - - -​

[ ID] Interval Transfer Bitrate Retr​

[ 5] 0.00-20.00 sec 2.18 GBytes 937 Mbits/sec 151 sender​

[ 5] 0.00-20.05 sec 2.18 GBytes 934 Mbits/sec receiver​

VM to iperf3 (basically the same for vmbr0 & vmbr1)

​

Connecting to host 192.168.10.21, port 5201​

[ 5] local 192.168.10.22 port 39740 connected to 192.168.10.21 port 5201​

[ ID] Interval Transfer Bitrate Retr Cwnd​

[ 5] 0.00-1.00 sec 88.5 MBytes 743 Mbits/sec 0 3.00 MBytes​

[ 5] 1.00-2.00 sec 98.8 MBytes 828 Mbits/sec 0 3.00 MBytes​

[ 5] 2.00-3.00 sec 80.0 MBytes 671 Mbits/sec 0 3.00 MBytes​

[ 5] 3.00-4.00 sec 71.2 MBytes 598 Mbits/sec 0 3.00 MBytes​

[ 5] 4.00-5.00 sec 104 MBytes 870 Mbits/sec 0 3.00 MBytes​

[ 5] 5.00-6.00 sec 70.0 MBytes 587 Mbits/sec 0 3.00 MBytes​

[ 5] 6.00-7.00 sec 73.8 MBytes 619 Mbits/sec 0 3.00 MBytes​

[ 5] 7.00-8.00 sec 75.0 MBytes 627 Mbits/sec 0 3.00 MBytes​

[ 5] 8.00-9.00 sec 57.5 MBytes 484 Mbits/sec 0 3.00 MBytes​

[ 5] 9.00-10.00 sec 77.5 MBytes 650 Mbits/sec 0 3.00 MBytes​

[ 5] 10.00-11.00 sec 75.0 MBytes 629 Mbits/sec 0 3.00 MBytes​

[ 5] 11.00-12.00 sec 70.0 MBytes 587 Mbits/sec 0 3.00 MBytes​

[ 5] 12.00-13.00 sec 68.8 MBytes 577 Mbits/sec 0 3.00 MBytes​

[ 5] 13.00-14.00 sec 70.0 MBytes 587 Mbits/sec 0 3.00 MBytes​

[ 5] 14.00-15.00 sec 78.8 MBytes 659 Mbits/sec 0 3.00 MBytes​

[ 5] 15.00-16.00 sec 112 MBytes 946 Mbits/sec 0 3.00 MBytes​

[ 5] 16.00-17.00 sec 77.5 MBytes 650 Mbits/sec 0 3.00 MBytes​

[ 5] 17.00-18.00 sec 83.8 MBytes 702 Mbits/sec 0 3.00 MBytes​

[ 5] 18.00-19.00 sec 78.8 MBytes 661 Mbits/sec 0 3.00 MBytes​

[ 5] 19.00-20.00 sec 78.8 MBytes 661 Mbits/sec 0 3.00 MBytes​

- - - - - - - - - - - - - - - - - - - - - - - - -​

[ ID] Interval Transfer Bitrate Retr​

[ 5] 0.00-20.00 sec 1.55 GBytes 667 Mbits/sec 0 sender​

[ 5] 0.00-20.04 sec 1.55 GBytes 665 Mbits/sec receiver​

VM to VM on the same bridge (vmbr1)

​

Connecting to host 192.168.10.7, port 5201​

[ 5] local 192.168.10.22 port 44918 connected to 192.168.10.7 port 5201​

[ ID] Interval Transfer Bitrate Retr Cwnd​

[ 5] 0.00-1.00 sec 199 MBytes 1.67 Gbits/sec 0 3.06 MBytes​

[ 5] 1.00-2.00 sec 224 MBytes 1.87 Gbits/sec 0 3.06 MBytes​

[ 5] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec 0 3.06 MBytes​

[ 5] 3.00-4.00 sec 119 MBytes 995 Mbits/sec 0 3.06 MBytes​

[ 5] 4.00-5.00 sec 124 MBytes 1.04 Gbits/sec 0 3.06 MBytes​

[ 5] 5.00-6.00 sec 124 MBytes 1.04 Gbits/sec 0 3.06 MBytes​

[ 5] 6.00-7.00 sec 110 MBytes 923 Mbits/sec 0 3.06 MBytes​

[ 5] 7.00-8.00 sec 109 MBytes 910 Mbits/sec 0 3.06 MBytes​

[ 5] 8.00-9.00 sec 154 MBytes 1.29 Gbits/sec 0 3.06 MBytes​

[ 5] 9.00-10.01 sec 68.8 MBytes 573 Mbits/sec 0 3.06 MBytes​

[ 5] 10.01-11.00 sec 115 MBytes 972 Mbits/sec 0 3.06 MBytes​

[ 5] 11.00-12.00 sec 109 MBytes 912 Mbits/sec 0 3.06 MBytes​

[ 5] 12.00-13.00 sec 109 MBytes 912 Mbits/sec 0 3.06 MBytes​

[ 5] 13.00-14.00 sec 124 MBytes 1.04 Gbits/sec 1 3.06 MBytes​

[ 5] 14.00-15.02 sec 68.8 MBytes 563 Mbits/sec 0 3.06 MBytes​

[ 5] 15.02-16.00 sec 106 MBytes 913 Mbits/sec 0 3.06 MBytes​

[ 5] 16.00-17.00 sec 188 MBytes 1.57 Gbits/sec 0 3.06 MBytes​

[ 5] 17.00-18.00 sec 128 MBytes 1.07 Gbits/sec 0 3.06 MBytes​

[ 5] 18.00-19.00 sec 169 MBytes 1.42 Gbits/sec 0 3.06 MBytes​

[ 5] 19.00-20.00 sec 252 MBytes 2.12 Gbits/sec 0 3.06 MBytes​

- - - - - - - - - - - - - - - - - - - - - - - - -​

[ ID] Interval Transfer Bitrate Retr​

[ 5] 0.00-20.00 sec 2.66 GBytes 1.14 Gbits/sec 1 sender​

[ 5] 0.00-20.05 sec 2.65 GBytes 1.14 Gbits/sec receiver​

 

[Dunuin]

The performance of the virtual NICs and bridges is usually limited by the single-thread performance of your CPU. The Atom got a really terrible single-thread performance, but that still sound too slow.

 

[ams123]

I should have added that prior performance was better. While I was not testing with iperf3 I was able to saturate a 1 Gbits/sec conection when uploading to OpenMediaVault, 120 MBytes/sec.