Proxmox 8.2.2 SR-IOV x550-BT2 onboard NIC, only VF v0 works

[dek0]

Hello everyone,

Before I start, my setup: Proxmox 8.2.2, epc621d6u-2t motherboard with Intel x550-BT2 onboard NIC. (2 ports)
I went ahead and updated the firmware and driver as per suggestion here: https://forum.proxmox.com/threads/support-issues-with-sr-iov-on-proxmox-8-2-2.149365/

I was able to activate SR-IOV and spawn 8 virtual functions by setting "ixgbe.max_vfs=8" in /etc/default/grub. So far so good, I see the VFs in Proxmox and I was able to add the VFs to the opnsense VM. Everything works fine there.

Now if I want to add additional VFs to other VMs (Ubuntu), I'm able to and I see the device when I type ip link, but I don't get an IP and of course no connection.

Troubleshooting done: added the same VF I used on opnsense to the Ubuntu machine, voila, it works.
So the issue I have is, VFs work only on one VM at a time.

From the ixgbe driver readme I found the following:
SR-IOV virtual functions have identical MAC addresses
-----------------------------------------------------


When you create multiple SR-IOV virtual functions, the VFs may have
identical MAC addresses. Only one VF will pass traffic, and all
traffic on other VFs with identical MAC addresses will fail. This is
related to the "MACAddressPolicy=persistent" setting in
"/usr/lib/systemd/network/99-default.link".


To resolve this issue, edit the
"/usr/lib/systemd/network/99-default.link" file and change the
"MACAddressPolicy" line to "MACAddressPolicy=none". For more
information, see the systemd.link man page.

I applied this fix, but unfortunately it didn't help.

Any ideas?

 

[dek0]

Did some more troubleshooting and can confirm only virtual function 0 works, the remaining 7 don't.

 

[dek0]

Update: Pinned an older kernel, 6.5.13-5-pve , with this kernel I only see VFs 0, 2, 4 and 6. All of them work.
I reverted to the newest kernel and I see 8 VFs but only 0, 2, 4 and 6 work!

At least I have 4 VFs working now, but this is really strange behavior. Go figure...