OPNsense and PCI Pass-through

[spetrillo]

Morning all,

I have a 4 port Intel I350-T4 pci card in my Proxmox server. 3 of the 4 ports will be virtualized for OPNsense internal vlans. The 4th port will be the WAN port but I am considering whether I could connect it to the vm via pci pass through. It will only be used for WAN and no other vms will use it.

Would you pci pass through this port or just virtualize it?

Steve

 

[Spaneta]

Hello,

Sorry but I don't understand what you mean virtualizing 3 of the 4 ports ... Could you clarify based on the examples below ?

Example 1 :
- i350-T4 solely for OPNsense VM, 3 ports for internal trafic and 1 port for WAN
- other NIC's in the PVE server for other VM's trafic
- in this case you can pass-through the whole i350-T4 to the OPNsense VM
- the trafic from other VM's to OPNsense and back have to go through switch/cables which seems bad (for performance but not only) if OPNsense serves as L3 switch/inter-VLAN routing for example.

Example 2 :
- i350-T4 is the only NIC in PVE server for VM trafic
- 3 ports of the NIC are used by Proxmox for bonds / vmbr's for internal trafic of the VM's including OPNsense internal VLAN's
- port 4 dedicated to WAN trafic of OPNsense
- (AFAIK on these cards) you can pass-through only the 4th port to OPNsense as WAN port ...
- ... or simply link the 4th port in PVE to a vmbr dedicated to WAN trafic so when you plan/test a new VM (newer version of OPNsense or other firewall distribution) you can easily "plug" the new VM to all needed LAN segments including the WAN.

Have a nice day,

 

[spetrillo]

What I meant by virtualizing the first 3 ports is setting up vmbrs for each and assigning internal VLANs to them. Since the fourth port of the I350 will be WAN only to OPNsense, and never used by any other vm or container, I thought I could passthrough that port to OPNsense directly. The I350 is only for OPNsense. I have another set of NICs for eveything else.

 

[Spaneta]

Thanks for clarifying.

Setting vmbr's for each implicitely give 1 Gbps per (group of) VLAN which is not best IMHO for performance and redundancy.

In this case my next questions is about the role of the OPNsense and the data-flows between the VM's with again few examples :
- if OPN is firewall between 3 (mostly) isolated areas (prod, guest, homelab for example) and mainly give Internet access and only minimal trafic between the 3 areas you can pass-through the whole card to OPN (less overhead for handling packets, bandwidth not important between areas)
- in this scenario you can also look toward SR-IOV and give 1 virtual function of each port to OPN VM for the same "less overhead" and then PVE can also use the card for LACP bond for example (just theory, I never have done this) thus giving more redundancy and consolidated bandwidth
- if OPN is used mainly for inter-VLAN routing, thus like a L3 switch + Internet access I would not got this route. Using virtio virtual NIC's connected to the same vmbr will give (above) 10 Gbps performances between VM's where dedicated NIC's will force 1 Gbps trafic betwwen _physical_ cards for local L3 trafic
- another variation, without knowing the intended topology of your network, is to use 1 port (untagged) for management, 1 port for WAN and 2 ports bonded and tagged for "internal" trafic.

What is the closest use-case ?

 

[spetrillo]

Option 1 is better suited, however I have a streaming vlan, that has my Plex server and media. My Home wireless vlan talks to this vlan when streaming content. Other than that there is a little bit of traffic between the vlans. My thought now would be to SR-IOV and bond the 3 ports together, still leaving the WAN port alone.

So now if I went down the SR-IOV path I have a few questions:

1) You mention giving 1 virtual function of each port to OPNsense VM. Would I enable just 1 virtual function or would I enable 4 per port on the I350?
2) When I add a PCI device to my vm should the list come from the Mapped List or the Raw List?
3) I have seen alot of write ups on using or not using Q35. What say you?
4) If I went down the path of SR-IOV and 1 virtual function to OPNsense does that mean I could use the other virtual functions for other VMs? If that were the case I might be able to remove my 2 port I350, bc those ports were for server management and any other vms I needed to build.

Appreciate you staying with me on this. If I can get this config right I am then going to spin up a second PVE and have a HA OPNsense world, with one VM on each physical host.

 

[Spaneta]

Never done SR-IOV myself but based on my readings and planning to do something similar for labbing ...

- begin with THIS doc from Proxmox to enable it and (basic) instruction to pass-through
- AFAIK you enable all VF or not, then pass 1 VF to a VM, another VF to another VM, ... and leave the other VF "unused"
- if I correctly understand the VF are meant _solely_ for pass-through to VM and PVE uses the "Physical" function of the NIC for itself ...
- ... or is it first VF ?
- thus you could use all 4 "PF" of the T4 for PVE and 1 VF for each port for the OPN VM
- keep in mind that there is no virtual switch between the VF (from what I have read it is unsure there one in the 1 Gbps generation of Intel cards)
- so using VF for all VM's will isolate the trafic between them and force it to go through the switch each time and direction
- AFAIK using all VF distributed to different VM's is only used for lowering overhead for high networking needs between numerous (virtual) servers and clients located on other hardware
- for new VM type I tend to always go to Q35 except if something push me to 440FX (nothing currently)
- based on the above link (top of document) Q35 may be better because passing-through a device as a pure PCIe "Some guest applications benefit from this" although not describing if it is lower overhead, faster performance, removing some bottleneck if device is passed-through like a "really fast legacy PCIdevice".

If you only have 2 NIC's, the T2 and the T4, then yes moving the T2 to make a second host may be a fun or usefull project, depending of your willing. If you have the hardware on hand to make a second node I would recommend making the second node a test bench for SR-IOV before risking anything to your "prod" node, then move the NIC's, VM's, ... per you project plan.

Hope this helps !

 

[spetrillo]

It definitely helps alot...and I have a physical OPNsense firewall, so any playing around is going to be on this PVE host. Gotta just jump in.

One question you did not answer is this: when I add a PCI device to my vm should the list come from the Mapped List or the Raw List?

 

[Spaneta]

I never used the mapped list as of now but AFAIK there is 2 differences (but I don't find a source for this) :

- only root user can assign a raw device to a VM (be it at creation or restore from backup)

- mapped devices are used to present the same "mapped-device-id" between different nodes, where the same hardware model can be in different slot and thus have different "low-level-name" from the hypervisor stand-point, to a (maybe HA) VM that would migrate from a node to another (although this does not help with SATA CTRL pass-through in this case but is more meant for NIC pass-through).

Good labbing to you !

 

[spetrillo]

Ok I am back...and so far its been a fun ride. I do have one question about the config file that establishes the vfs. Right now I have options igb max_vfs=4. How can I specify this on a per physical port basis, if that is allowed. As mentioned port 4 of my PCI card is for the WAN, so I do not need vfs on that port.

 

[Spaneta]

Sorry but I don't know if this is possible.
Having VF enabled for the whole card, thus on all ports, don't prevent you choosing to pass-through or not on a per VF basis.
So this shouldn't be a problem AFAIK.