Mellanox ConnectX-3 SR-IOV Questions

[payne0420]

Hey everyone,

I'm having trouble understanding how SR-IOV works on the Mellanox CX3 (non-pro) cards.
The cards I have are the VPI 40GbIB/10GbE flashed to 56/40.
Enabled SR-IOV in the bios, proxmox and got it showing inside proxmox.

The question is related to how Nvidia handles SR-IOV in this case.
The system has two CX3 cards installed which live under bus 00:04:00.0 and 00:82:00.0
Both ports of each card show under the same PCI bus, see lshw output below.

Kernel:

uname -r 
5.15.116-1-pve

Driver options:

cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core num_vfs=4,4,0 port_type_array=2,2 probe_vf=4,4,0 log_num_mgm_entry_size=-1

lshw output, sorted and renamed by interface:



Notice how the VFs show all on enp130s0 and none on the enp130s0d1 port.
Same for the other card.

if I do ip link show, they show somewhat correct, but still not according to the modprobe settings - should've been 4 VFs per port:



What's causing all this?
Any hints and/or brainstorming is much appreciated!

Thanks in advance

 

[payne0420]

Update:
Started playing around assigning VLANs to the VFs and noticed the following:

38: enp130s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:bb:37:01 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
39: enp130s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:bb:37:02 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 4095, spoof checking off, link-state auto
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, vlan 99, spoof checking off, link-state auto

If you do ip link set enp130s0d1 vf {0..7} vlan 99 it will actually assign them on each port/device as shown above.
So on enp130s0 the active VFs are 0-3 and on enp130s0d1 there's 4-7.

This is weird and definitely looks like a bug to me. Can someone else confirm they see the same things on their side?
It seems like a nightmare to manage and document properly inside Proxmox.


Another edit:
https://docs.openstack.org/neutron/...-connectx-3-connectx-3-pro-dual-port-ethernet

Found an article outlining this exact issue.