Opnsense with virtual functions

[nooke]

Hey guys,

I'm virtualizing my opnsense instance via proxmox and had some issues regarding performance in the past.
therfor I wanted to switch to SR-IOV -> virtual functions.

I have multiple Intel NIC's
I350
82599ES 10gig SFI/SFP+
X550T

For all 3 I created virtual functions and tried to passthrough those to an opnsense VM.
First of all they were not visible as NICs at all when using Q35 with PCIe passthrough.
So I went with i440fx pci -> voila NICs are now displayed in freebsd/opnsense.

Now the real problem occured. I'm using Vlans in my network so in opnsense configuration manager I set up multiple Vlans for those virtual functions.
No matter which virtual function I use (from whatever NIC) I can't communicate with any device in my network.

After digging around for hours making changes on proxmox for the specified NICs (blacklisting drivers, setting trust mode on, setting spoofchecking off etcpp) none seem to help.

But if I spin up a fresh opnsense VM, skipping the configuration manager and go straight into shell. setting up a vlan via "ifconfig ixv0.10 create vlan 10 vlandev ixv0 inet 10.10.10.20/24" and try to reach a machine in my network (ping/icmp) it's working.
So vlan with virtual functions does work - but as soon as I use the cli tool from opnsense to create vlans even those manually configured vlan doesn't work anymore.

anyone ever experience this issue?
what can I do from there?

root@THOR:~# lspci | grep -i Eth
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:10.3 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
06:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
06:10.1 Ethernet controller: Intel Corporation X550 Virtual Function
06:10.3 Ethernet controller: Intel Corporation X550 Virtual Function
06:10.5 Ethernet controller: Intel Corporation X550 Virtual Function
06:10.7 Ethernet controller: Intel Corporation X550 Virtual Function


root@THOR:~# lspci -s 01:10.3 -nbk
01:10.3 0200: ffff:ffff (rev 01)
        Subsystem: 15d9:0000
        Kernel driver in use: vfio-pci
        Kernel modules: igbvf

cheers