NIC passthrough for OPNSense VM, Intel NIC 2 ports

[SpookyAction]

Hi,

Using Proxmox 8.0.4, planning to install latest OPNSense 23.7.x. I am familiar with Linux but new to both Proxmox and OPNSense. Looking for advice for the general direction to read further documentation and practice.

The Proxmox host has 3 physical NICs

• enp4s0 = Realtek RTL8125 built-in in the motherboard. Currently is used by vmbr0 bridge, created at Proxmox installation time.
• enp2s0f0, enp2s0f1 = Intel card, PCIe 2 ports. Currently not used by Proxmox

My plan is to create a VM and install OPNSense. Which would use

• enp2s0f0 = for OPNSense LAN interface, connected to a new linux bridge vmbr1. This vmbr1 bridge would also be used later for other VMs and LXC containers, as network device.
• enp2s0f1 = for OPNSense WAN interface. This NIC will be passthrough, only be used by the OPNSense VM. For beginning while I am learning OPNSense, this NIC will be connected to the home router to get internet. Later on, it will be connected directly to the cable modem. At that point the OPNSense VM will replace the home router.

QUESTIONS:

Q1. Is the plan above doable? Especially the way I plan to use the 2 ports of the same Intel NIC PCIe card: one port for the vmbr1 bridge. One port as PCIe passthrough.

Q2. In case Q1 is possible, I would appreciate if you can suggest links to documentation how to create a new vmbr1 bridge.

Q3. For PCIe passthrough. I find this page Enable Proxmox PCIe Passthrough easier to follow than the Proxmox Wiki, PCI Passthrough. Nevertheless, if you know of any beginner friendly doc to setup NIC passthrough for Proxmox VM, I would greatly appreciate.

Thanks very much for any help.

# On Proxmox host
cat /etc/network/interfaces

auto lo
iface lo inet loopback

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.80/24
gateway 192.168.1.250
bridge-ports enp4s0
bridge-stp off
bridge-fd 0

iface enp2s0f0 inet manual
iface enp2s0f1 inet manual

 

[wallacio]

You can't passthrough a single ethernet port on a NIC - you pass through the whole device.
Regardless, I'd have thought all the answers to your pretty vanilla sounding setup would be answered here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

 

[SpookyAction]

@wallacio thanks for the documentation, appreciated a lot.

You can't passthrough a single ethernet port on a NIC - you pass through the whole device.

What a disappointment. I could passthrough the built-in Realtek NIC. But unfortunately OPNSense uses FreeBSD which doesn't have good driver for Realtek devices. So I guess my best option for now is to give up on passthrough NICs and use linux bridges instead.

 

[showiproute]

I agree with @SpookyAction .

If you want to get into passing through PCIe devices or VFs (Virtual Functions) it's the best to get Intel devices as they usually have good driver support in mostly any OS.

 

[SpookyAction]

If you want to get into passing through PCIe devices or VFs (Virtual Functions) it's the best to get Intel devices as they usually have good driver support in mostly any OS.

What is the relationship between "Virtual Functions" and PCIe passthrough? Can you please direct me to some documentation?

 

[Nollimox]

What is the relationship between "Virtual Functions" and PCIe passthrough?

Virtual function is when you assigned ens2f0 to vtnet0, you can plug cable from your ISP in and it automatically pass through whereas with PCIe passthrough, you passthrough the entire device. In my setup that's how I have mine, instead of passing through the entire NIC, I let the NIC stay with the kernel and just passthrough the Ethernet port. For instance, I have a Intel i350xt4 and assigned ens2f0=vtnet0=vmbr1 just plug the cable in and pfSense WAN has an IP. In your case, WAN would be: enp2s0f0=vtnet0=vmbr1...
Ideally, one needs at least three Ethernet ports.

 

[showiproute]

I meant VF in context of SRIOV. So you have one physical card but can create multiple "virtual cards" out of it which can be passed to a VM.
So a VM would be able to see and access the real hardware.

 

[miles267]

I have a Protectli VP4650 (fast) hardware device. Am currently passing through NICs "raw" devices (PCIe NICs) to my Untangle UTM. The NICs are all Intel. For such fast hardware, does passing through physical NICs really offer much performance gain for a 1GB/1GB fiber service? It certainly complicates the setup (mapping the right hardware NICs) or is it an unnecessary complication?

 

[spetrillo]

I guess that is my question also. I have enabled SR-IOV and have 2 VF per physical interface. Now when setting up OPNsense I am not sure whether I use the physical interfaces or the VF interfaces? Second nothing shows up under Mapped Device...they show under Raw Device. Is my setup ok?

 

[waynevm]

You can't passthrough a single ethernet port on a NIC - you pass through the whole device.
Regardless, I'd have thought all the answers to your pretty vanilla sounding setup would be answered here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
({nodename} of your node)
If this shows the nic ports are in separate IOMMU groups, then you can pass them through individually according to:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough

 

[spetrillo]

Ok...thanks!

How about the question of mapped vs raw devices? I find all my devices under raw. Is that correct?

 

[Ramalama]

https://forum.proxmox.com/threads/script-service-passthrough-helper.85488/

You can passthrough whatever you want, there are multiple approaches, one of them i made 2021 already.
But there are a ton of other ways, you just have to google or search in this forum a bit.

That above if a method with a script/service, im not sure but in the meantime there are probably even better ways.

However, i personally don't passthrough myself just only one port of intel cards, or not using sr-iov passthrough with intel cards, because there are some bugs.
Those bugs have nothing todo with passthrough themself, just with the unthought incomplete intel driver itself. Like you need to adjust the bridge fdb table for example...
However, not saying that the driver is bad, just that there are issues with likely everything that is an uncommon configuration.
Mellanox is even worse here.

However, the best approach is to try simply out and check yourself if it works as you want.

Cheers