[SOLVED] Issues with SRIOV-based NIC-passthrough to firewall

[ctr]

I'm passing through a VF from my Intel X722-based NIC to a firewall (OPNsense) qemu vm which uses CARP for high-availability.
However, due to filtering by the PF, the packets destined to the virtual CARP MAC addresses do not reach the VF/guest.
This is "by design" and if such functionality is intended, the PF must be configured in a certain way (https://www.kernel.org/doc/Documentation/networking/i40e.rst VLAN Packet Steering).
However, either the version of ethtool in Proxmox or the used driver version (2.8.20-k) in the Kernel does not support this special configuration:
ethtool -set-priv-flags <interface-name> vf-true-promisc-support on
Would be possible to update the affected component(s)? I will try to compile the Intel driver out-of-tree (latest is 2.10.19.82) to see if this resolves the issue.

Not directly Proxmox related, but related. If someone has an idea I'd be grateful.
On the host (proxmox) I receive kernel messages like:
i40e 0000:b5:00.2: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on VF 0 i40e 0000:b5:00.2: Error I40E_AQ_RC_ENOSPC adding RX filters on VF 0, please set promiscuous on manually for VF 0

which (as per above doc) would require:
ip link set <interface> promisc on ip link set <interface> allmulticast on
However, "ip link" does not exist in FreeBSD/OPNsense. ifconfig <interface> promisc should be the equivalent for the first, but I cannot find anything for "allmulticast on"

 

[Stoiko Ivanov]

From a quick grepping through the current kernel source - the flag should be available in the driver (and I did not find a kernel config option that should need to be set).

One thing you could try as well is getting a newer version of ethtool (e.g. the one from bullseye - since the dependencies seem to not have changed from buster) - https://packages.debian.org/bullseye/ethtool
AFAIR the featureset present in ethtools output is also dependent on its version

i40e 0000:b5:00.2: Error I40E_AQ_RC_ENOSPC, forcing overflow promiscuous on VF 0

not too sure - but seems it could be related (not being able to set true promisc mode on a VF)?

However, "ip link" does not exist in FreeBSD/OPNsense.

from the respective man-pages of the drivers https://www.freebsd.org/cgi/man.cgi...opos=0&manpath=FreeBSD+12.1-RELEASE+and+Ports and https://www.freebsd.org/cgi/man.cgi?query=ixl&sektion=4&manpath=freebsd-release-ports - I would guess that there simply is no such option - maybe try if CARP works once the true-promisc flag on the PF is set?

(Sadly I don't have such a NIC around to try to reproduce your config)

I hope this helps!

 

[ctr]

Never mind, supplied driver and ethtool support it. But (as documented) it only works for the "first PF on a device" (which means the first port on a multi-NIC card.
However, even with true-promisc enabled the VF doesn't receive the traffic for the secondary MAC (regardless of a PF-defined or dynamic mac being used as primary MAC)

 

[Stoiko Ivanov]

just to be sure - you checked with tcpdump in the guest that the packets don't arrive inside? - and the guests vf is set to promisc ?

 

[ctr]

Yes, the guest/VF doesn't see the packets, but the the PF does. And the guest says:
ixlv0_vlan1234: permanently promiscuous mode enabled ixlv0: permanently promiscuous mode enabled
on the relevant interfaces

 

[Stoiko Ivanov]

from https://downloadmirror.intel.com/24411/eng/readme.txt:

To set a VF as trusted or untrusted, enter the following command in the
Hypervisor:

# ip link set dev eth0 vf 1 trust [on|off]

* you could try to set the promisc mode on the vf before starting the VM
* apart from that you could also try to disable various offloading features on the pf (and the vf)
* maybe also try reproducing the config with a linux guest

I hope this helps!

 

[ctr]

from https://downloadmirror.intel.com/24411/eng/readme.txt:
To set a VF as trusted or untrusted, enter the following command in the
Hypervisor:

# ip link set dev eth0 vf 1 trust [on|off]

Yes that is configured (as well as spoof_check off),

* you could try to set the promisc mode on the vf before starting the VM

As per the doc:

The vf-true-promisc-support priv-flag does not enable promiscuous mode; rather,
it designates which type of promiscuous mode (limited or true) you will get
when you enable promiscuous mode using the ip link commands above.

The referenced "ip link" command has to be set in the guest, so I don't think setting promisc mode on the VF before starting the VM is an option

* apart from that you could also try to disable various offloading features on the pf (and the vf)

Already tried that. I think vlan hw filtering is required to let the PF know which VLANs the VF is looking for. If I remember correctly, before having that enabled, VLANs where not working on the VF at all. I can play around with TSO/LRO and CRC though.

* maybe also try reproducing the config with a linux guest

I'm afraid this is what I will have to do anyway to get the attention of the Intel e1000 dev team...
They already responded on the mailing list, saying that the ixlv driver is deprecated and they ask me to use iavf instead. Not sure if this is support by FreeBSD 11.2 though. I tried to compile it on OPNsense but it failed.

 

[ctr]

They already responded on the mailing list, saying that the ixlv driver is deprecated and they ask me to use iavf instead. Not sure if this is support by FreeBSD 11.2 though. I tried to compile it on OPNsense but it failed.

I managed to get iavf compiled an loading, but it doesn't take ownership of the interfaces as ixlv is still loaded.

Any recommendations how to get rid of ixlv?
hints.ixlv.0.disabled = 1 left the interface assigned to ixvl (but disabled / unusuable)
if_ixlv_load="NO" in /boot/loader.conf.local seems to have no effect

I did not dare to delete/move /boot/kernel/if_ixlv.ko yet, but I assume it is compiled into the kernel anyway as I do not normally see the module when looking at kldstat output.

 

[Stoiko Ivanov]

hmm - maybe it would be easier just to create a FreeBSD12 VM (no opnsense - but the minimal config should be doable in FreeBSD as well)

 

[ctr]

It works with a Linux guest, so I have to assume the guest/VF driver is causing the problem. Trying with a vanilla FreeBSD 12 doesn't really help me as regardless of the results I don't have a solution.
Instead I'd like to try iavf on OPNSense itself. As mentioned the module loads, but as the device has already been used by (kernel-included) I don't get enay iavf interfaces. So I'd need an OPNSense kernel without built-in ixlv.

Do you have any tutorials how to rebuild the kernel? I already have https://github.com/opnsense/src cloned to /usr/src (to be able to build iavf).
Is it really as simple as running "make buildkernel" and then using the resulting "/usr/obj/usr/src/sys/HARDENEDBSD/kernel"? Or do I need to install everything ("make kernel")?

To exclude ixlv I suppose I have to write an own KERNCONF (derived from GENERIC) that doesn't include ixlv?!?

 

[ctr]

Just tried the 20.7a (which already comes with iavf instead of ixlv) and this works perfectly fine (from VF/CARP perspective). Lack of available packages prevent me from performing more thorough testing, but I'd say it looks good and I'll probably switch to the 20.7 alpha/beta as soon as packages are available.

 

[Stoiko Ivanov]

Great to hear!
I think waiting a bit more for a supported release based on FreeBSD 12 might be more future-proof than starting to compile you kernel now
(should you still wish - I can really recommend the FreeBSD handbook as far as documentation goes - sadly I'm not too familiar with Opensense to tell you how far it differs - but if you have a backup - you could try the steps for plain FreeBSD)

 

[ctr]

OPNsense 20.7b indeed works fine, so this one is resolved.

 

[ctr]

I may have been a bit too optimistic here. While it works fine on a single node, on a cluster I do have issues (and this probably where CARP is really needed). Still need to dig into the details and this may not be Proxmox related at all, but I wanted to leave the latest update here since this thread seems to be quite prominent when searching for SR-IOV and CARP.

 

[ctr]

It does work. One just has to be careful with the VLAN filter to not overload the NIC capabilities, otherwise it will just stop working.

 

[fhloston]

It does work. One just has to be careful with the VLAN filter to not overload the NIC capabilities, otherwise it will just stop working.

Could you please elaborate a bit? How do you configure the VLAN filter?

I am trying something similar but with Intel X553, ixgbe (linux/proxmox) and iavf (opnsense). So far I have not succeeded in getting a trunk port to work in opnsense.

The multicast issue for carp can be solved with

post-up ip maddress add 01:00:5e:00:00:12 dev $IFACE || :

in eni.

Trust is on, spoofcheck off.


Regards

Martin

 

[cyruspy]

Hello!, any update on this?. Does the VLAN filter comment apply if you do tagging/untagging in the guest?

Currently I have everything on a virtio based NIC doing tagging on the OPNsense VM.

I would like to move everything to an aggregation of two VFs with Intel NICs. I'm heavily using CARP.