OPNsense on PVE - PCIe Passthrough

[spetrillo]

Has anyone successfully gotten OPNsense to run reliably in a PCIe pass-through environment?

I have been trying to pass-through the 4 ports of my Intel I350 PCIe network card and it hangs up both the vm and then PVE, as it enumerates the NIC hardware and vlans.

I don't even get to install OPNsense. Getting very frustrated with this and am ready to dump it.

 

[ualex]

Is your i350 card the only network interface in the node?

I personally use the standard bridge option with OPNsense (so no hardware pass-through), this makes moving the VM easier to another node (bridge is generic) and performance impact bridge vs pass-through is not that much.

 

[leesteken]

Has anyone successfully gotten OPNsense to run reliably in a PCIe pass-through environment?

Yes, but that was years ago.

I have been trying to pass-through the 4 ports of my Intel I350 PCIe network card and it hangs up both the vm and then PVE, as it enumerates the NIC hardware and vlans.

Not all hardware is well suited for passthrough. Did you check your IOMMU groups? Did you early-bind the NIC to vfio-pci? Does the NIC reset properly? PCIe passthrough is always trial and error and can be hit or miss depending on the motherboard, its BIOS and the device.

I don't even get to install OPNsense. Getting very frustrated with this and am ready to dump it.

Maybe install it without passthrough and try to add passthrough later. Or test passthrough by booting a Ubuntu installer ISO (without installing) in the VM to prevent filesystem corruption by all the hard resets.

 

[spetrillo]

Is your i350 card the only network interface in the node?

I personally use the standard bridge option with OPNsense (so no hardware pass-through), this makes moving the VM easier to another node (bridge is generic) and performance impact bridge vs pass-through is not that much.

My I350 is dedicated to vms only. I have an onboard NIC that I am using for PVE functions.

Did you just setup standard Linux bridges with OPNsense? Do you use vlans with your OPNsense setup?

 

[spetrillo]

Yes, but that was years ago.

Not all hardware is well suited for passthrough. Did you check your IOMMU groups? Did you early-bind the NIC to vfio-pci? Does the NIC reset properly? PCIe passthrough is always trial and error and can be hit or miss depending on the motherboard, its BIOS and the device.

Maybe install it without passthrough and try to add passthrough later. Or test passthrough by booting a Ubuntu installer ISO (without installing) in the VM to prevent filesystem corruption by all the hard resets.

1) All 4 ports of the I350 NIC are in their own IOMMU group
2) Each port of the I350 NIC shows vfio-pci as the kernel driver in use
3) How do I check if the NIC reset properly??
4) When using PCIe passthrough I select each individual port and leave ROM-Bar and PCI Express unchecked. Is this correct?

I absolutely agree its very trial and error. We are trying to do server stuff on consumer kit and that does not always work out well. If I cannot get this going soon I am going to abort and just use standard networking.

 

[leesteken]

1) All 4 ports of the I350 NIC are in their own IOMMU group

And you're not using pcie_acs_override (check with cat /proc/cmdline)? I was half expecting that the Proxmox and VM crash/freeze was because of this. None of the other things typically causes a crash or freeze (of the Proxmox host).

2) Each port of the I350 NIC shows vfio-pci as the kernel driver in use

Before starting the VM? Then that's fine. Then the device(s) does not need to reset before being passed to the VM.

3) How do I check if the NIC reset properly??

I think I350 is know to work in a VM and also after stopping and restarting the VM but I cannot check for myself. If the device(s) only works once (because it does not reset), you'll need to reboot the Proxmox host after each run of the VM.
Is it one device with four functions (or two devices with two functions or four devices)? Try to pass the device(s) with All Functions instead of each function separately (as it allows for resetting the whole device).

4) When using PCIe passthrough I select each individual port and leave ROM-Bar and PCI Express unchecked. Is this correct?

I don't know. Maybe the drivers inside the VM expect/assume the NIC to be connected to PCIe (instead of PCI) but otherwise it does not really matter.
I would always enable ROM unless there is a specific reason not to.

I absolutely agree its very trial and error. We are trying to do server stuff on consumer kit and that does not always work out well. If I cannot get this going soon I am going to abort and just use standard networking.

Maybe search this forum for I350 to see if other people needed some work-arounds or specific settings to get it to work with passthrough.

 

[spetrillo]

And you're not using pcie_acs_override (check with cat /proc/cmdline)? I was half expecting that the Proxmox and VM crash/freeze was because of this. None of the other things typically causes a crash or freeze (of the Proxmox host).

Before starting the VM? Then that's fine. Then the device(s) does not need to reset before being passed to the VM.

I think I350 is know to work in a VM and also after stopping and restarting the VM but I cannot check for myself. If the device(s) only works once (because it does not reset), you'll need to reboot the Proxmox host after each run of the VM.
Is it one device with four functions (or two devices with two functions or four devices)? Try to pass the device(s) with All Functions instead of each function separately (as it allows for resetting the whole device).

I don't know. Maybe the drivers inside the VM expect/assume the NIC to be connected to PCIe (instead of PCI) but otherwise it does not really matter.
I would always enable ROM unless there is a specific reason not to.

Maybe search this forum for I350 to see if other people needed some work-arounds or specific settings to get it to work with passthrough.

Do I need to remove the I350 interfaces from Proxmox networking config all together?

 

[leesteken]

Do I need to remove the I350 interfaces from Proxmox networking config all together?

Since you plan to remove the device(s) from the host (by passing it through to a VM), it would be a good idea. It's always a good idea to prevent the host from touching devices that you passthrough.

EDIT: To prevent the host from loading drivers for the device(s), use early-binding to vfio-pci (which sometimes also needs a softdep): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_host_configuration

 

[spetrillo]

And you're not using pcie_acs_override (check with cat /proc/cmdline)? I was half expecting that the Proxmox and VM crash/freeze was because of this. None of the other things typically causes a crash or freeze (of the Proxmox host).

I am actually using the ACS override. Without the override my I350 ports are in the same IOMMU group. With the override the ports are in separate IOMMU groups. If I do not use the override is it ok for all ports to be in the same IOMMU group?

 

[spetrillo]

Here is my Grub cmdline:

GRUB_CMDLINE_LINUX_DEFAULT="quiet pci=assign-busses intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction pci_pt_e820_access=on pci=assign-busses i915.enable_gvt=1"

 

[leesteken]

I am actually using the ACS override. Without the override my I350 ports are in the same IOMMU group. With the override the ports are in separate IOMMU groups. If I do not use the override is it ok for all ports to be in the same IOMMU group?

You cannot share devices from the same group between VMs and/or the host. If they are in the same group, then it's fine. Please try not to use the override, it breaks the security isolation that is the whole point of doing passthrough.

 

[spetrillo]

I have had enough...this is the log from the last lock up. It shows nothing to be able to tshoot on:

Jul 07 13:45:57 pve01.petrillo.home pvedaemon[1064]: <root@pam> update VM 100: -hostpci0 0000:01:00,pcie=1
Jul 07 13:46:04 pve01.petrillo.home pvedaemon[1064]: <root@pam> starting task UPIDve01:0000049A:000012B5:668AD45C:qmstart:100:root@pam:
Jul 07 13:46:04 pve01.petrillo.home pvedaemon[1178]: start VM 100: UPIDve01:0000049A:000012B5:668AD45C:qmstart:100:root@pam:
Jul 07 13:46:05 pve01.petrillo.home systemd[1]: Created slice qemu.slice - Slice /qemu.
Jul 07 13:46:05 pve01.petrillo.home systemd[1]: Started 100.scope.
Jul 07 13:46:07 pve01.petrillo.home pvedaemon[1064]: <root@pam> end task UPIDve01:0000049A:000012B5:668AD45C:qmstart:100:root@pam: OK
Jul 07 13:46:07 pve01.petrillo.home pvedaemon[1202]: starting vnc proxy UPIDve01:000004B2:000013A0:668AD45F:vncproxy:100:root@pam:
Jul 07 13:46:07 pve01.petrillo.home pvedaemon[1064]: <root@pam> starting task UPIDve01:000004B2:000013A0:668AD45F:vncproxy:100:root@pam:
Jul 07 13:46:07 pve01.petrillo.home pveproxy[1075]: proxy detected vanished client connection
Jul 07 13:46:07 pve01.petrillo.home pvedaemon[1206]: starting vnc proxy UPIDve01:000004B6:000013A9:668AD45F:vncproxy:100:root@pam:
Jul 07 13:46:07 pve01.petrillo.home pvedaemon[1065]: <root@pam> starting task UPIDve01:000004B6:000013A9:668AD45F:vncproxy:100:root@pam:
Jul 07 13:46:17 pve01.petrillo.home pvedaemon[1202]: connection timed out
Jul 07 13:46:17 pve01.petrillo.home pvedaemon[1064]: <root@pam> end task UPIDve01:000004B2:000013A0:668AD45F:vncproxy:100:root@pam: connection timed out
Jul 07 13:47:51 pve01.petrillo.home pvedaemon[1065]: <root@pam> end task UPIDve01:000004B6:000013A9:668AD45F:vncproxy:100:root@pam: OK
Jul 07 13:47:52 pve01.petrillo.home pvedaemon[1066]: <root@pam> starting task UPIDve01:000005AD:00003CC4:668AD4C8:vncproxy:100:root@pam:
Jul 07 13:47:52 pve01.petrillo.home pvedaemon[1453]: starting vnc proxy UPIDve01:000005AD:00003CC4:668AD4C8:vncproxy:100:root@pam:
Jul 07 13:48:53 pve01.petrillo.home pveproxy[1075]: problem with client ::ffff:192.168.2.12; Connection reset by peer

I am going to work on trying to get OPNsense to run with normal Linux bridge interfaces. While this was a complete waste of time and I went deep into the rabbit hole, I did learn alot about PCIe passthrough and SRIOV. Maybe I will work on this when I can get OPNsense properly virtualized and then take my original OPNsense hardware and play with that.

 

[ualex]

@spetrillo

My setup is super simple, see attached HW I am using with Proxmox/OPNsense. I just map my 3 bridges to the VM. vmbr0 == lan, vmbr1=internet and vmbr2=internet2 (I have failover)

 

[spetrillo]

@spetrillo

My setup is super simple, see attached HW I am using with Proxmox/OPNsense. I just map my 3 bridges to the VM. vmbr0 == lan, vmbr1=internet and vmbr2=internet2 (I have failover)

Yes I finally built one using virtualized networking. Curious...you have the firewall option on I guess your WAN connection? Why do you do that for a firewall?