PVE/OPNsense Network Questions

[spetrillo]

Hello all,

I am running my OPNsense firewall on my PVE server. I have now added a second PVE server for HA. What is the best way to set my networking on my PVE servers, since OPNsense is live on one of them? Is there anything in particular? Second OPNsense handles my DHCP. It seems building a VM on the same PVE server as my OPNsense VM does not pick up DHCP. Has anyone run into this also?

Thanks,
Steve

 

[andrew-transparent]

We have two OPNSense firewalls in our deployment and at a basic level have 4 virtual adapters assigned: LAN, WAN, DMZ, CARP. We assign any public IPs to the WAN interface and do not have any public IPs on any of our PVE hosts. We use Wireguard to access anything management wise.

The LAN interface(s) are given a VLAN id and all PVE hosts have a bridge with an IP in that VLAN for managment.

CARP interface is used for HA between the firewalls on a seperate VLAN.

DMZ interface is used in a VLAN for various VMs that might serve up resources and need to be isolated (ie Docker containers etc).

This is a pretty high level overview and it has worked well for us over the years.

 

[proxmrmomba]

added a second PVE server for HA.

Hi,

in my opinion, i think that will not work, because you have no wittness. Proxmox need 3 Servers, or 2 Servers and a witness-side ( i have forget the name: q-device?)

An other thing is, normaly the active opnsense do all the work, the passive opnsense is only hot-standby. Did you have tested your ha config? Are you realy use a real HA with CARP?
Has every interface the same name (Firewall 1: ETH0 LAN / Firewall 2: EHT0 LAN)?

Next thing: LAN Interface from OPNSENSE have to use the bridge, where your server will be connected.

 

[andrew-transparent]

proxmrmomba make some good points here - I assumed when you said you were adding second for HA you were referring to OPNSense, not PVE. If you're doing HA for OPNSense generally you don't want to setup any services/configs on the secondary FW as those settings will get synced over from the primary FW to keep their states the same.

 

[andrew-transparent]

Just to add, you will need to put some thought on where you are doing your VLANs. OPNSense can be attached to a VLAN aware bridge and you can manage VLAN membership inside OPNSense.

We prefer to manage VLANs at the host level/Datacenter level, and assign appropriate VLANs via virtual adapters/bridges on the guest. Meaning each VLAN would get its own dedicated interface in OPNSense. We generally like to have less layers of abstraction at the VLAN level and use SDN in Proxmox for assigning the right interafces to our guests. There are a lot of ways to do this with the same effect, but be mindful of how you want to manage this in the future.

That being said, if you plan on having a lot of VLANs it gets combersome to have a lot of virtual adapters inside OPNSense, so it may be easier to just manage VLAN membership there.

 

[spetrillo]

Its actually HA for both...but I am not there yet. I was actually asking a different question and probably did not explain it properly. So let me try again.

My production OPNsense VM sits on my production PVE server. Originally my PVE server got its mgmt IP from DHCP, but since the DHCP server is on my OPNsense VM it cannot get DHCP when the OPNsense VM is not active....think reboot of PVE server. I then changed the mgmt IP from DHCP to static, and that seemed to cure the PVE mgmt access.

The onboard NIC has three vlans on it. The config is as follows:

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1 20 30

auto vmbr0.1
iface vmbr0.1 inet static
address 192.168.1.5/24
gateway 192.168.1.1

When I try to build a VM using a vlan of 20 or 30 I cannot get a DHCP IP, bc I believe the OPNsense VM(DHCP server) is on the same PVE as the VM I am trying to build. Has anyone run into this issue?