Proxmox with OPNsense Passthrough

[uniketou]

Hello all,

I've bought the box below (MGNAS 12 ; with i7 1355u; 32Go; 10G SFP+).

The goal is :
- Install Proxmox on the device
- Install OPNsense on Proxmox
- Configure the 2 x SFP+ in Passthough to OPNsense (for WAN & LAN access)
- WAN : Public IP
- LAN : 192.168.1.x /24

I want also to have the Proxmox MGMT on Port 1 (RJ45).

=> As the both SFP+ to join the router will be in Passthrough, is it possible to have the Proxmox MGMT on Port 1 with IP 192.168.1.x /24 ?

Here the diagram :

 

[LnxBil]

As the both SFP+ to join the router will be in Passthrough, is it possible to have the Proxmox MGMT on Port 1 with IP 192.168.1.x /24 ?

That depends on the IOMMU groups, if you really want PCIe passthrough. If you just want to connect the posts, create a bridge for each and use them on the OPNsense VM as virtual nics. This is the easiest setup and will also not break the layers of virtualization.

 

[uniketou]

I agree, it's easiest with virtual NIC.

But I want to use the full 10G interfaces. Not sure that the virtual NIC can support 10G.

 

[BobhWasatch]

Why not? It is just a copy in memory. Try iperf between two VM's if you don't believe it.

 

[uniketou]

Ok I'll try.
Does it needs more cpu/ram ?
I plan to use 4 cores + 8Go RAM

 

[BobhWasatch]

I would think that is plenty.

 

[uniketou]

Ok, I keep you posted when I'll test

 

[patch]

s the both SFP+ to join the router will be in Passthrough, is it possible to have the Proxmox MGMT on Port 1 with IP 192.168.1.x /24 ?

I believe so however you would need to connect the two using an external switch.

• OPSense Lan port is passed through so Proxmox has no internal connection to it.
• OPSense allocates IP addressing in 192.168.1.x /24 on LAN
• Proxmox MGMT on Port 1 set within Proxmox to an address in 192.168.1.x /24 on ETH0 NIC, resulting in a fixed IP address
• External switch connects ETH0 and SFP+1 to enable devices on Lan to Layer 1 access Proxmox MGMT
• Probably worth also setting / reserving Proxmox MGMT IP on OPSense DHCP server to prevent duplicate IP allocation

 

[uniketou]

Yes that sound good !

Regarding the pass through and OPNsense VM , I need to :
- enable IOMMU
- Create the VM with : CPU host, PCI device ( 2x10G ), q35 enable

Is it enough ?

 

[patch]

Regarding the pass through and OPNsense VM , I need to :

See https://pve.proxmox.com/wiki/PCI_Passthrough

• Enable IOMMU in hardware bios
• Enable IOMMU in Proxmox
• I have passed through i210 & i226 NIC and tried both VM bios and Machine options but couldn't notice any difference in pfsense VM function.

 

[uniketou]

That means you can reach 10G bandwidth ?

 

[patch]

That means you can reach 10G bandwidth ?

Those NIC are limited to 1G, But function in an equivalent set up without issue.

 

[uniketou]

Great !
For Proxmox IP, you connect to another switch port and use the same IP network as OPNsense LAN ?

 

[patch]

yes

• Wan and LAN ports in pfsense are passed through in Proxmox
• Main Lan port on pfsense actually configured to for VLan, with one of the Vlan connected to the NIC used by Proxmox for management via and external switch

 

[LnxBil]

Is it enough ?

As I've written in my first post: it depends on the IOMMU groups if they are in seperate groups, if they're not you cannot pcie passthrough the NICS seperately. To check that should have been your first thing to do. If they're not this whole discussion about passthrough is moot.

 

[uniketou]

Yes that's right !
I'll check and come back to you afterwards

 

[uniketou]

Hello,

I've installed proxmox + opnsense with 10G interface Passthrough , all is OK !

But one question :
When I enable the IPS/IDS, the bandwidth drop a lot... Does the CPU is the problem ?

I've 4 CPU cores (i7 1355U).
I tried to boost with 6 CPU cores and reboot, but still 4 cores on Opnsense (orange line in proxmox)

 

[Azunai333]

I tried to boost with 6 CPU cores and reboot, but still 4 cores on Opnsense (orange line in proxmox)

You need to restart the whole VM process, not only the OS inside of it.

Do a right click in the PVE GUI on the VM and select "Reboot". That should do it.

 

[uniketou]

I confirm it's better for the CPU
Thanks !

I need to perform some other bandwidth tests to see the result