Advice for vmbr1 at 25 Gbit/s

[v95klima]

Hi
seeking advice how to share a 25Gbit/s to several VMs/LXC?
I get full 25Gbit/s speed when doing passthrough of the NIC to VM, but I would like several VMs/LXC to have access at full speed?
Advice greatly appreciated.
Thanks in advance

 

[dubios]

Could you please elaborate which exact NIC you are using? Depending on its capabilities you have two options:

1. Best Performance: SR-IOV to pass-through a single NIC to multiple VMs https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_sr_iov
2. Best Compatibility: Software-based using Bridges https://pve.proxmox.com/wiki/Network_Configuration#_default_configuration_using_a_bridge

Nonetheless, you could pass the same NIC to multiple VMs but the traffic will never exceed the max. rated speed of 25Gbit/s TOTAL

 

[v95klima]

Thank for reply.
Hoping to get same high quality performance as passtrough to Windows Server 2022 to work with vmbr1 with many VMs/LXCs in Proxmox?
Thanks

Please see below exact card and that
SR-IOV = 1 enabled
with vfs=8

Firmware has been upgraded.

The card works at full speed 25Gbit/s when pass-through to Windows Server 2022, with RDMA successfully enabled.

This is the vmbr1 setting, it works but never at the same high/full speeds of Windows Server 2022:

auto vmbr1
iface vmbr1 inet static
address 192.168.3.3/24
bridge-ports enp2s0f0np0
bridge-stp off
bridge-fd 0
mtu 9000



root@epyc5:~/mft-4.28.0-92-x86_64-deb# flint -d 02:00.0 dc
;; Generated automatically by iniprep tool on Wed Dec 01 14:51:14 IST 2021 from ./cx4-lx_MCX4121A_2p_x8_25g.prs;; FW version: 14.32.1010
;; Mellanox Technologies LTD
;;[PS_INFO]
;;Name = MCX4121A-ACA_Ax
;;Description = ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
[image_info]
;;;;; This section contains info which is shared by FW and burning tool
psid = MT_2420110034
name = MCX4121A-ACA_Ax
description = ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
prs_name = cx4-lx_MCX4121A_2p_x8_25g.prs
[boot_record]
;;;;; 1. Boot record endianes: reserved1 is the fist Byte that should be written on the NVRAM (address 0);2. Each line is protected by parity bit (bit 31) the xor of the 32 read bits should be 1
[mfg_info]
guids.guids.num_allocated = 4
guids.macs.num_allocated = 4
[device_info]
guids.guids.num_allocated = 0x4
guids.macs.num_allocated = 0x4
[fw_boot_config]
;;;;; boot + iron fw config data
nv_config.global.pci.settings.log_vf_uar_bar_size = 0x0
nv_config.global.pci.settings.sriov_en = 1
nv_config.global.pci.settings.total_vfs = 8

 

[v95klima]

OK I ran:
echo 4 > /sys/bus/pci/devices/0000:02:00.0/sriov_numvfs
resulting in more virtual functions per below.
I ll try passing them through to VMs and measure performance.
Anything else I can do to improve performance in Proxmox for 25Gbit/s?
Thanks


lspci | grep Ethernet
02:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
02:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
02:00.2 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.3 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.4 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
02:00.5 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
44:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)
44:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01)

 

[dubios]

Glad it helped !

Anything else I can do to improve performance in Proxmox for 25Gbit/s?

Without knowing your workload it's hard to answer. I would advice some general performance recommendations first:

1. Dedicated nic for cluster network: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvecm
2. When using bridges on Windows use the "VirtIO (paravirtualized)" controller: https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers

And before we're going down doing fine tuning, you would've to provide some benchmarks/data on why you think that the linux birdges are to slow for your workload.

Please provide output of the following commands:

$ bridge link
$ ethtool enp2s0f0np0

And be aware of the limits of your ConnectX-4 Lx:

Virtualization
– SR-IOV: Up to 256 Virtual Functions
– SR-IOV: Up to 8 Physical Functions per port
– Virtualization hierarchies (e.g. NPAR)
• Virtualizing Physical Functions on a physical port
• SR-IOV on every Physical Function
– Ingress and egress QoS levels
– Guaranteed QoS for VMs

 

[v95klima]

Thanks,
I have not tested the added ConnectX-4 Lx Virtual Function yet. Will try later.

Yes, the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

With full hardware Passthrough "All Functions" I do get 25 Gbit/s with a VM Windows Server 2022, hence the guess it has to do with Proxmox VMBR1 settings or something else at Proxmox level?



root@epyc5:~# ethtool enp2s0f0np0
Settings for enp2s0f0np0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None RS BASER
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: RS
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 25000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Link detected: yes

root@epyc5:~# bridge link
2: enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 5
2: enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 hwmode VEB
3: enp68s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 hwmode VEB
4: enp2s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 state forwarding priority 32 cost 1
4: enp2s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 hwmode VEB
5: enp2s0f1np1: <BROADCAST,MULTICAST> mtu 1500 hwmode VEB
9: veth102i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
10: veth103i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
11: veth104i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
12: veth106i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2
13: veth106i1@enp68s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 master vmbr1 state forwarding priority 32 cost 2
14: veth107i0@enp68s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master vmbr0 state forwarding priority 32 cost 2

 

[v95klima]

Btw, no proxmox cluster in use.
The 25Gbit/s is primarily for NVME NAS server purpose.

 

[dubios]

the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

Yes, it's software and therefore creating an immense workload for the CPU. The latency when comparing pass-through (SR-IOV or not) will always be way better, which is why SR-IOV got developed. Therefore despite any fine tuning, the software based bridges will not be as performant as pass-through.

It could be questioned why you have to pass-through the NVMEs instead of using ZFS or ZFS over iSCSi but again, that mainly depends on your workload.

Further optimizations are beyond my pay grade and should better be answered by someone with more experience with 25Gbit/s and above workloads.

 

[mow]

Yes, the Redhat controller seems capped at 10Gbit/s inside Windows Server 2022 for virtual NIC.
So I get the full 10Gbit/s using Red Hat VirtIO Ethernet, but not the full 25Gbit/s, hence the problem and origin seeking advice.

View attachment 70317

Did you actually test the speed or are you just going by what it says in the properties (which is not necessarily the actual speed)?