PVE/OPNsense Network Questions

S

spetrillo

Member
Feb 15, 2024
248
13
18
Hello all,

I am running my OPNsense firewall on my PVE server. I have now added a second PVE server for HA. What is the best way to set my networking on my PVE servers, since OPNsense is live on one of them? Is there anything in particular? Second OPNsense handles my DHCP. It seems building a VM on the same PVE server as my OPNsense VM does not pick up DHCP. Has anyone run into this also?

Thanks,
Steve
 
We have two OPNSense firewalls in our deployment and at a basic level have 4 virtual adapters assigned: LAN, WAN, DMZ, CARP. We assign any public IPs to the WAN interface and do not have any public IPs on any of our PVE hosts. We use Wireguard to access anything management wise.

The LAN interface(s) are given a VLAN id and all PVE hosts have a bridge with an IP in that VLAN for managment.

CARP interface is used for HA between the firewalls on a seperate VLAN.

DMZ interface is used in a VLAN for various VMs that might serve up resources and need to be isolated (ie Docker containers etc).

This is a pretty high level overview and it has worked well for us over the years.
 
  • Like
Reactions: carles89
spetrillo said:
added a second PVE server for HA.
Click to expand...
Hi,

in my opinion, i think that will not work, because you have no wittness. Proxmox need 3 Servers, or 2 Servers and a witness-side ( i have forget the name: q-device?)

An other thing is, normaly the active opnsense do all the work, the passive opnsense is only hot-standby. Did you have tested your ha config? Are you realy use a real HA with CARP?
Has every interface the same name (Firewall 1: ETH0 LAN / Firewall 2: EHT0 LAN)?

Next thing: LAN Interface from OPNSENSE have to use the bridge, where your server will be connected.
 
  • Like
Reactions: andrew-transparent and Johannes S
Just to add, you will need to put some thought on where you are doing your VLANs. OPNSense can be attached to a VLAN aware bridge and you can manage VLAN membership inside OPNSense.

We prefer to manage VLANs at the host level/Datacenter level, and assign appropriate VLANs via virtual adapters/bridges on the guest. Meaning each VLAN would get its own dedicated interface in OPNSense. We generally like to have less layers of abstraction at the VLAN level and use SDN in Proxmox for assigning the right interafces to our guests. There are a lot of ways to do this with the same effect, but be mindful of how you want to manage this in the future.

That being said, if you plan on having a lot of VLANs it gets combersome to have a lot of virtual adapters inside OPNSense, so it may be easier to just manage VLAN membership there.
 
Last edited:
Its actually HA for both...but I am not there yet. I was actually asking a different question and probably did not explain it properly. So let me try again.

My production OPNsense VM sits on my production PVE server. Originally my PVE server got its mgmt IP from DHCP, but since the DHCP server is on my OPNsense VM it cannot get DHCP when the OPNsense VM is not active....think reboot of PVE server. I then changed the mgmt IP from DHCP to static, and that seemed to cure the PVE mgmt access.

The onboard NIC has three vlans on it. The config is as follows:

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1 20 30

auto vmbr0.1
iface vmbr0.1 inet static
address 192.168.1.5/24
gateway 192.168.1.1

When I try to build a VM using a vlan of 20 or 30 I cannot get a DHCP IP, bc I believe the OPNsense VM(DHCP server) is on the same PVE as the VM I am trying to build. Has anyone run into this issue?
 
Last edited:
That shouldn't be an issue. We have multiple OPNsense VM's running on various hosts all handing out DHCP to various VMs running on the same/different hosts. Idealy it should be the same network fabric across all hosts.

Its probably a VLAN issue and I would confirm you have the right tagging on the guest adapter if you're using VLAN aware bridge (you are given the example above with vmbr0). You can try assigning a static IP to your VM and try to ping the OPNsense IP to ensure you have connectivity. If you do, then you can look at your DHCP service on OPNsense and ensure its running on the right interface with the right scope.

I'm assuming vmbr0.1 is for your PVE host?
 
Hi,

From the beginning, you should have approached this differently, considering how your servers would communicate with each other.
The Proxmox documentation on clustering explains this well. Ideally, you should connect two PVE servers on a separate network, using different NICs. However, if this isn't feasible (and I assume you have only two NICs, Public and LAN), you can create a VLAN for both PVE servers on the LAN NIC, physically connecting the two servers (e.g., VLAN100) and assigning a subnet (e.g., 10.10.100.0/24). Then, assign 10.10.100.10/24 to the first PVE server and 10.10.100.20/24 to the second.
This will enable both servers to communicate independently of OPNsense.
Afterward, you can experiment with both OPNsense instances. However, in my experience, OPNsense has been quite buggy in various network topologies. I eventually switched back to pfSense with HA between two firewalls, which has worked smoothly. Additionally, the DHCP KEA in FreeBSD can be problematic in certain scenarios. I reverted to ISC, which has proven more reliable. Both OPNsense and pfSense have well-documented guides. Remember to add a failover IP and gateway in the DHCP advanced settings to ensure HA functionality.
 
Yup...I figured it out. DHCP server on OPNsense for vlan 20 or 30 was not enabled....duhhh! Thanks all!
 
You must log in or register to reply here.
Top Bottom