[SOLVED] Advice for vmbr1 at 25 Gbit/s

[v95klima]

Hi
seeking advice how to share a 25Gbit/s to several VMs/LXC?
I get full 25Gbit/s speed when doing passthrough of the NIC to VM, but I would like several VMs/LXC to have access at full speed?
Advice greatly appreciated.
Thanks in advance

 

[dubios]

Could you please elaborate which exact NIC you are using? Depending on its capabilities you have two options:

1. Best Performance: SR-IOV to pass-through a single NIC to multiple VMs https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_sr_iov
2. Best Compatibility: Software-based using Bridges https://pve.proxmox.com/wiki/Network_Configuration#_default_configuration_using_a_bridge

Nonetheless, you could pass the same NIC to multiple VMs but the traffic will never exceed the max. rated speed of 25Gbit/s TOTAL

 

[v95klima]

Thank for reply.
Hoping to get same high quality performance as passtrough to Windows Server 2022 to work with vmbr1 with many VMs/LXCs in Proxmox?
Thanks

Please see below exact card and that
SR-IOV = 1 enabled
with vfs=8

Firmware has been upgraded.

The card works at full speed 25Gbit/s when pass-through to Windows Server 2022, with RDMA successfully enabled.

This is the vmbr1 setting, it works but never at the same high/full speeds of Windows Server 2022:

auto vmbr1
iface vmbr1 inet static
address 192.168.3.3/24
bridge-ports enp2s0f0np0
bridge-stp off
bridge-fd 0
mtu 9000



root@epyc5:~/mft-4.28.0-92-x86_64-deb# flint -d 02:00.0 dc
;; Generated automatically by iniprep tool on Wed Dec 01 14:51:14 IST 2021 from ./cx4-lx_MCX4121A_2p_x8_25g.prs;; FW version: 14.32.1010
;; Mellanox Technologies LTD
;;[PS_INFO]
;;Name = MCX4121A-ACA_Ax
;;Description = ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
[image_info]
;;;;; This section contains info which is shared by FW and burning tool
psid = MT_2420110034
name = MCX4121A-ACA_Ax
description = ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
prs_name = cx4-lx_MCX4121A_2p_x8_25g.prs
[boot_record]
;;;;; 1. Boot record endianes: reserved1 is the fist Byte that should be written on the NVRAM (address 0);2. Each line is protected by parity bit (bit 31) the xor of the 32 read bits should be 1
[mfg_info]
guids.guids.num_allocated = 4
guids.macs.num_allocated = 4
[device_info]
guids.guids.num_allocated = 0x4
guids.macs.num_allocated = 0x4
[fw_boot_config]
;;;;; boot + iron fw config data
nv_config.global.pci.settings.log_vf_uar_bar_size = 0x0
nv_config.global.pci.settings.sriov_en = 1
nv_config.global.pci.settings.total_vfs = 8

 

[v95klima]

OK I ran:
echo 4 > /sys/bus/pci/devices/0000:02:00.0/sriov_numvfs
resulting in more virtual functions per below.
I ll try passing them through to VMs and measure performance.
Anything else I can do to improve performance in Proxmox for 25Gbit/s?
Thanks


lspci | grep Ethernet
02:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
02:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
02:00.2 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.3 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.4 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.5 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
44:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
44:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)

 

[dubios]

Glad it helped !

Anything else I can do to improve performance in Proxmox for 25Gbit/s?

Without knowing your workload it's hard to answer. I would advice some general performance recommendations first:

1. Dedicated nic for cluster network: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvecm
2. When using bridges on Windows use the "VirtIO (paravirtualized)" controller: https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers

And before we're going down doing fine tuning, you would've to provide some benchmarks/data on why you think that the linux birdges are to slow for your workload.

Please provide output of the following commands:

$ bridge link
$ ethtool enp2s0f0np0

And be aware of the limits of your ConnectX-4 Lx:

Virtualization
– SR-IOV: Up to 256 Virtual Functions
– SR-IOV: Up to 8 Physical Functions per port
– Virtualization hierarchies (e.g. NPAR)
• Virtualizing Physical Functions on a physical port
• SR-IOV on every Physical Function
– Ingress and egress QoS levels
– Guaranteed QoS for VMs

 

[v95klima]

Thanks,
I have not tested the added ConnectX-4 Lx Virtual Function yet. Will try later.

Yes, the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

With full hardware Passthrough "All Functions" I do get 25 Gbit/s with a VM Windows Server 2022, hence the guess it has to do with Proxmox VMBR1 settings or something else at Proxmox level?



root@epyc5:~# ethtool enp2s0f0np0
Settings for enp2s0f0np0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: RS
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 25000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Link detected: yes

root@epyc5:~# bridge link
2: enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 5
2: enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 hwmode VEB
3: enp68s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 hwmode VEB
4: enp2s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 state forwarding priority 32 cost 1
4: enp2s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 hwmode VEB
5: enp2s0f1np1: <BROADCAST,MULTICAST> mtu 1500 hwmode VEB
9: veth102i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
10: veth103i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
11: veth104i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
12: veth106i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
13: veth106i1@enp68s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 state forwarding priority 32 cost 2
14: veth107i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2

 

[v95klima]

Btw, no proxmox cluster in use.
The 25Gbit/s is primarily for NVME NAS server purpose.

 

[dubios]

the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

Yes, it's software and therefore creating an immense workload for the CPU. The latency when comparing pass-through (SR-IOV or not) will always be way better, which is why SR-IOV got developed. Therefore despite any fine tuning, the software based bridges will not be as performant as pass-through.

It could be questioned why you have to pass-through the NVMEs instead of using ZFS or ZFS over iSCSi but again, that mainly depends on your workload.

Further optimizations are beyond my pay grade and should better be answered by someone with more experience with 25Gbit/s and above workloads.

 

[mow]

Yes, the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

View attachment 70317

Did you actually test the speed or are you just going by what it says in the properties (which is not necessarily the actual speed)?

 

[v95klima]

With Connectx-4 Virtual function, SR-IOV between VF on host 192.168.3.3 and one inside same Proxmox Host Debian LXC 192.168.3.6.
To get this full speed like this Trunk interface should not be used for vmbr1 (not even present, just present slowed things down despite not being in the path). Just VF to VF locally, no fiber or second Connectx-4 card.

root@epyc5:~# iperf3 -c 192.168.3.3
Connecting to host 192.168.3.3, port 5201
[ 5] local 192.168.3.3 port 39132 connected to 192.168.3.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.04 GBytes 26.1 Gbits/sec 0 895 KBytes
[ 5] 1.00-2.00 sec 3.09 GBytes 26.5 Gbits/sec 0 895 KBytes
[ 5] 2.00-3.00 sec 3.11 GBytes 26.7 Gbits/sec 0 895 KBytes
[ 5] 3.00-4.00 sec 3.09 GBytes 26.6 Gbits/sec 0 1.25 MBytes
[ 5] 4.00-5.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.25 MBytes
[ 5] 5.00-6.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.25 MBytes
[ 5] 6.00-7.00 sec 3.11 GBytes 26.7 Gbits/sec 0 1.25 MBytes
[ 5] 7.00-8.00 sec 3.12 GBytes 26.8 Gbits/sec 0 1.25 MBytes
[ 5] 8.00-9.00 sec 3.11 GBytes 26.7 Gbits/sec 0 1.25 MBytes
[ 5] 9.00-10.00 sec 3.11 GBytes 26.7 Gbits/sec 0 1.25 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.9 GBytes 26.6 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 30.9 GBytes 26.6 Gbits/sec receiver

iperf Done.
root@epyc5:~# iperf3 -c 192.168.3.3 -R
Connecting to host 192.168.3.3, port 5201
Reverse mode, remote host 192.168.3.3 is sending
[ 5] local 192.168.3.3 port 34306 connected to 192.168.3.3 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.87 GBytes 24.7 Gbits/sec
[ 5] 1.00-2.00 sec 2.87 GBytes 24.7 Gbits/sec
[ 5] 2.00-3.00 sec 2.82 GBytes 24.3 Gbits/sec
[ 5] 3.00-4.00 sec 2.83 GBytes 24.3 Gbits/sec
[ 5] 4.00-5.00 sec 2.84 GBytes 24.4 Gbits/sec
[ 5] 5.00-6.00 sec 2.84 GBytes 24.4 Gbits/sec
[ 5] 6.00-7.00 sec 2.83 GBytes 24.3 Gbits/sec
[ 5] 7.00-8.00 sec 2.81 GBytes 24.2 Gbits/sec
[ 5] 8.00-9.00 sec 2.84 GBytes 24.4 Gbits/sec
[ 5] 9.00-10.00 sec 2.83 GBytes 24.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 28.4 GBytes 24.4 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 28.4 GBytes 24.4 Gbits/sec receiver

 

[v95klima]

From second ConnectX-4, VM of windows server 2022 on different Proxmox physically 192.168.3.4

to first ConnectX-4 192.168.3.3 Proxmox Host through fiber.

C:\Users\Administrator> .\iperf3.exe -c 192.168.3.3 -R
Connecting to host 192.168.3.3, port 5201
Reverse mode, remote host 192.168.3.3 is sending
[ 5] local 192.168.3.4 port 57920 connected to 192.168.3.3 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.35 GBytes 20.2 Gbits/sec
[ 5] 1.00-2.00 sec 2.54 GBytes 21.7 Gbits/sec
[ 5] 2.00-3.00 sec 2.55 GBytes 21.9 Gbits/sec
[ 5] 3.00-4.00 sec 2.65 GBytes 22.8 Gbits/sec
[ 5] 4.00-5.00 sec 2.69 GBytes 23.1 Gbits/sec
[ 5] 5.00-6.00 sec 2.72 GBytes 23.3 Gbits/sec
[ 5] 6.00-7.00 sec 2.65 GBytes 22.8 Gbits/sec
[ 5] 7.00-8.00 sec 2.36 GBytes 20.3 Gbits/sec
[ 5] 8.00-9.00 sec 1.88 GBytes 16.1 Gbits/sec
[ 5] 9.00-10.00 sec 2.60 GBytes 22.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 25.0 GBytes 21.4 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 25.0 GBytes 21.4 Gbits/sec receiver

iperf Done.
PS C:\Users\Administrator> .\iperf3.exe -c 192.168.3.3
Connecting to host 192.168.3.3, port 5201
[ 5] local 192.168.3.4 port 57926 connected to 192.168.3.3 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.15 GBytes 18.5 Gbits/sec
[ 5] 1.00-2.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 2.00-3.00 sec 2.16 GBytes 18.5 Gbits/sec
[ 5] 3.00-4.00 sec 2.16 GBytes 18.5 Gbits/sec
[ 5] 4.00-5.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 5.00-6.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 6.00-7.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 7.00-8.00 sec 2.17 GBytes 18.6 Gbits/sec
[ 5] 8.00-9.00 sec 2.17 GBytes 18.6 Gbits/sec
[ 5] 9.00-10.00 sec 2.13 GBytes 18.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 21.6 GBytes 18.5 Gbits/sec sender
[ 5] 0.00-10.00 sec 21.6 GBytes 18.5 Gbits/sec receiver

iperf Done.
PS C:\Users\Administrator>

 

[v95klima]

From second ConnectX-4, VM of windows server 2022 on different Proxmox physically 192.168.3.4

to Debian LXC on the first ConnectX-4 192.168.3.6 Proxmox Host through fiber.

PS C:\Users\Administrator> .\iperf3.exe -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.4 port 58100 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.13 GBytes 18.3 Gbits/sec
[ 5] 1.00-2.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 2.00-3.00 sec 2.13 GBytes 18.4 Gbits/sec
[ 5] 3.00-4.00 sec 2.16 GBytes 18.6 Gbits/sec
[ 5] 4.00-5.00 sec 2.16 GBytes 18.5 Gbits/sec
[ 5] 5.00-6.00 sec 2.14 GBytes 18.4 Gbits/sec
[ 5] 6.00-7.00 sec 2.18 GBytes 18.6 Gbits/sec
[ 5] 7.00-8.00 sec 2.16 GBytes 18.5 Gbits/sec
[ 5] 8.00-9.00 sec 2.15 GBytes 18.5 Gbits/sec
[ 5] 9.00-10.00 sec 2.15 GBytes 18.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec sender
[ 5] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec receiver

iperf Done.
PS C:\Users\Administrator> .\iperf3.exe -c 192.168.3.6 -R
Connecting to host 192.168.3.6, port 5201
Reverse mode, remote host 192.168.3.6 is sending
[ 5] local 192.168.3.4 port 58104 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.73 GBytes 14.8 Gbits/sec
[ 5] 1.00-2.00 sec 1.73 GBytes 14.9 Gbits/sec
[ 5] 2.00-3.00 sec 1.72 GBytes 14.8 Gbits/sec
[ 5] 3.00-4.00 sec 1.73 GBytes 14.9 Gbits/sec
[ 5] 4.00-5.00 sec 1.73 GBytes 14.8 Gbits/sec
[ 5] 5.00-6.00 sec 1.73 GBytes 14.9 Gbits/sec
[ 5] 6.00-7.00 sec 1.72 GBytes 14.9 Gbits/sec
[ 5] 7.00-8.00 sec 1.73 GBytes 14.8 Gbits/sec
[ 5] 8.00-9.00 sec 1.79 GBytes 15.4 Gbits/sec
[ 5] 9.00-10.00 sec 2.61 GBytes 22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 18.2 GBytes 15.7 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 18.2 GBytes 15.7 Gbits/sec receiver

iperf Done.

 

[v95klima]

My success getting this to work on LXC
is credited to Epijunkie's direct support and posted guide.

https://www.reddit.com/r/Proxmox/comments/1ak26yg/guide_configure_sriov_virtual_functions_vf_in_lxc/

 

[v95klima]

Ohh and to close the Subject topic,

I could not get close to 25Gbit/s with vmbr1.
But same function and performance was solved with SR-IOV and vfs. 25 Gbit/s between VMs and LXCs locally or to separate host.

VMBR1 had 30-50% loss in performance when putting physical, trunc/virtual on vmbr1, with my Server CPU running max 3 Ghz.
Offloading the "switching" down to Connectx-4 hardware on NIC, increased performance.

And the nice thing, you get EIGHT! 8 vfs with this cheap old card from the SR-VIO, ( enp2s0f0v3 is passed to Debian LXC and therefor not seen in list below). If eight is not enought the newer ones like intel 810 you get a couple of hundred of vfs!
This is clearly they way to go long term for high performance!

root@epyc5:~# lshw -class network | grep logical
logical name: enp2s0f0np0
logical name: enp2s0f1np1
logical name: enp2s0f0v0
logical name: enp2s0f0v1
logical name: enp2s0f0v2
logical name: enp2s0f0v4
logical name: enp2s0f0v5
logical name: enp2s0f0v6
logical name: enp2s0f0v7
logical name: enp68s0f0np0
logical name: enp68s0f1np1

 

[v95klima]

From Proxmox Host VM win Server 2022 Share to another physically separate Proxmox Host VM Windows 2022 Desktop through fiber.

Both VMs are using SR-IOV with vfs.

As you can see full speed when Cached file transfer. The sub 2 GByte/s is non-cached, limited by CPU/NVME/IOPS.
Same performance as physical passthrough of ConnectX-4, before SR-IOV and vfs. No penalty for SR-IOV with vfs.