SR-IOV with Intel X552/82599 vs X710 - Communication issue between VF VM and host bridge

A

alhaddar

Active Member
Sep 17, 2019
3
0
41
58
Hello,

I have been testing SR-IOV on two different Proxmox nodes and found a significant difference in behavior between Intel NICs. I would like to share my findings and ask if there is a solution for the X552/82599.

---

Hardware Setup:

Node 1 - pve (Supermicro X10SDV-16C-TLN2F):
- Intel X552/X557-AT 10GbE (ixgbe driver)
- Intel 82599ES X520 SFP+ (ixgbe driver)
- Proxmox 9.x, Kernel 6.17.13-11-pve

Node 2 - pve2 (Protectli VP6650):
- Intel X710 10GbE SFP+ (i40e driver)
- Proxmox 9.x, Kernel 7.0.6-2-pve

---

Results:

X710 (pve2) - SR-IOV WORKS perfectly:
- VFs created successfully
- Kernel driver: iavf
- LXC container with VF passthrough gets IP via DHCP
- Full connectivity from host, LAN, and other VMs
- ping works perfectly

X552/82599 (pve) - SR-IOV DOES NOT work with bridge:
- VFs created successfully
- After blacklisting ixgbevf: Kernel driver: vfio-pci
- VM gets IP via DHCP on VF
- BUT: VM cannot ping host, gateway, or any other device on the same bridge
- ARP shows: (incomplete)
- tcpdump on PF shows ICMP requests arriving but no replies

Steps already tried:
1. Blacklisted ixgbevf
2. Set spoofchk off and trust on for VFs
3. bridge fdb add with static MAC
4. hookscript to add MAC to FDB post-start
5. unbind VF and rebind to vfio-pci manually
6. options ixgbe max_vfs=4 in modprobe

None of these solved the issue.

---

Root Cause (as I understand it):

The X552/82599 uses a static hardware switching table. When SR-IOV is enabled, the NIC does not automatically learn MAC addresses of devices connected to the Linux bridge on the PF. This means the VF VM is isolated from the bridge.

This issue is referenced in this thread:
C

[SOLVED] Thread 'communication issue between SRIOV VM VF and CT on PF bridge'

I'm passing through a VF using SRIOV to a VM. This VM works fine, I can reach it from Proxmox (mgmt interface on vmbr0 of the PF) and other hosts on the network.
However, a CT attached to the very same vmbr0 cannot reach the VM, but also everything else on the network. According to tcpdump packets from the CT are getting to the VF and the VF is responding, but those response packets are then not visible on the PF anymore and as result do not reach the CT.

Does that ring any bells?

---

Question:

Is there any known solution or workaround to get SR-IOV working on Intel X552/82599 (ixgbe driver) with a Linux bridge on the same NIC in Proxmox?

Or is this a fundamental hardware limitation with no fix?

Thank you for any help!
 
I had exactly this on my own two nodes, and it cost me a weekend before I
understood it. So here is the mechanism, and a two-minute test that will tell
you where you stand.

The card's internal switch (a VEB) forwards between the PF, the VFs and the wire
using a table it is *told* about - it does not learn. It knows the VF's own
address and the PF's, and nothing else. Your container and the host's address
live behind the Linux bridge, so the VEB has never heard of them. A frame from
the VF to such an address is a miss, and a miss leaves through the physical
port. You can watch it go with tcpdump on the uplink, addressed to something
that is not out there.

That is also why ARP resolves while ping does not: ARP is broadcast and gets
flooded, the unicast reply does not.

The fix is to put the address of the peer *behind the bridge* into the uplink's
unicast filter:

bridge fdb add <mac of the container/host> dev <uplink> self permanent

Two things trip this up when done by hand, and both match "I tried FDB entries
and it did not help":

- `<uplink>` is the interface the bridge sits on - the PF, or the VF you use as
the bridge uplink. Not the VF passed into the VM, and not the bridge itself.
- `<mac>` is the *peer's* address (container, host, printer), not the VM's.
Registering the VM's own address on the uplink is worse than doing nothing: it
tells the card that address lives on the uplink vport, and traffic arriving
from the wire for that VM stops being delivered to the VF.

Worth confirming in this exact order before you change anything permanently:

1. from inside the VF guest, ping the container - should fail
2. `bridge fdb add <container mac> dev <uplink> self permanent`
3. ping again - should work
4. `bridge fdb del <container mac> dev <uplink> self` - should fail again

If step 3 changes nothing, this whole approach does not work on your hardware
and nothing below will help either. If it does work, what is left is
bookkeeping: those entries are static, and the set of addresses behind a bridge
is not. A new container, a device that moves, an interface that goes down and
comes back - and the filter is out of step again.

On "the X710 is fine": I would be careful with that. I see the same failure and
the same fix on i40e (X710) and on ixgbe (82599ES), and on mlx4 and mlx5 as
well. If your X710 node genuinely does not need it, I would look for a
difference in the setup - most often whether anything is actually bridged behind
that uplink at all - rather than assume the driver is immune.

Because I got tired of maintaining those entries by hand on my own machines, I
ended up writing a small daemon for it (disclosure: it is mine, MIT):

https://github.com/Jimbambuli/sriov-mac-sync

It watches the bridge's forwarding database over netlink and keeps the uplink's
unicast filter in step - including addresses that are only ever learnt (the host
itself, a wifi client, a printer on a second NIC) and setups where the NIC sits
a layer below a VLAN-aware bridge or an SDN vnet. Single static binary, .deb and
OpenWrt packages, no dependencies:

curl -LO https://github.com/Jimbambuli/sriov-mac-sync/releases/latest/download/sriov-mac-sync_amd64.deb
dpkg -i sriov-mac-sync_amd64.deb

That is the whole installation: the package starts the service, and
`sriov-mac-sync --status` shows what it decided. Removing the package takes its
entries back out of the card, so trying it costs you nothing you cannot undo.

One alternative worth knowing before you take mine: jdlayman/pve-hookscript-sriov
is smaller and does the job if all you need is your configured guests on plain
bridges. There is also yujincheng08/mlx4_br, a C++ daemon on the same idea, but
check it against your setup first: it picks its targets from a fixed list of
driver names - mlx4_core, mlx5_core, iavf, ixgbevf - so on Intel it only matches
a bridge port that is itself a virtual function. If your bridge sits on the PF,
which is the usual arrangement, it will not find anything to do.

If the four steps above behave oddly, post `bridge fdb show dev <uplink>` and
`ip -d link show <uplink>` and I will have a look.
 
You must log in or register to reply here.
Top Bottom
Back