OPNsense: barebone or include in Proxmox cluster?

[K2V2]

After using Proxmox for ~6 months in my homelab i am quite tempted to install my new OPNsense on top of Proxmox. 1M$ question: is this a good idea? (I don't expect a yes no answer as that doesn't exist).

Current setup:

• 3 x Odroid H3 with each 32 GB Ram and 2 TB NMVE drives.
• Proxmox 8 installed on a small 32 GB partition.
• CEPH on the remaining almost 2 TB of the NMVE drives in the default 3/2 replicated mode.
• CEPH uses 1 of the 2.5GBe nics via a dedicated switch.
• The other 2.5 GBe nic hosts the management interface and all regular use of the systems.
• A Synology NAS houses the backups and all installation files.

This has been working great, especially the ease of spinning up new LXC containers to setup services has 100% won me over versus single machines with lots of services, some via Docker.

Now a new Topton Core i5 1235U system with 6 x 2.5GBe nics will be used to host OPNsense. i will install 32 GB ram and and the same 2TB NMVE disk.

It would be rather easy to integrate this into the Proxmox cluster thus expanding the CEPH cluster to a 4th drive. I hope the nics can be passed trough directly but even virtual nics will do the job, I have a test OPNsense running now with virtual nics on the H3's which works really well (this nic is even shared with the management interface which won't be needed on the Topton as it has 6 nic's).

The big disadvantage is that if my cluster of H3's goes down (for instance power supply failure) the router will go down as well as quorum is lost. A single H3 node failure is no problem. Dual H3 node failure can be absorbed with a Qdevice (running on my Synology NAS for instance or one of the in the meantime unused rPi's). To go even further i could increase the votes number of the router node such that the H3 cluster could totally fail (sure if also other nodes fail even this doesn't work).

If I am not mistaken CEPH will also stop working with only a single OSD left in the cluster however. This could be overcome by installing the NMVE adapter and a second NMVE drive in the Topton (disadvantage i loose the 3x4 PCIe connection and am down to 1x4, alternative a SATA drive could do the trick as well). This will give CEPH 2 OSD's but still only a single monitor. If I understood things well this means that CEPH will go down and there is no "Qdevice" for CEPH as far as I know.

No internet means bad WAF factor. All other services are less critical in that sense .

Which way would you go (and why):

• Integrate in Proxmox thus expanding the cluster and adding storage space, reliability and computing power?
• Keep OPNsense standalone and have only 1 point of failure for the incoming internet?

 

[supermicro_server]

install opnsense on a proxmox VM is a good idea, but if using your cluster enviroment is not safe enoght for you, Y can use the carp solution of opnsense with a slave opnsense on a different HW system. Network interfaces, must be configured on the same broadcast domain.

 

[K2V2]

I looked at the carp solution and indeed that gives you redundant routers, however I would need multiple IP's which my ISP doesn't provide. Moreover my ISP uses pppoe for internet on vlan 6 and regular DHCP on vlan 4 for IPTV. Both have their own IP and somewhat complex configuration.
I would hence need a router in front capable of separating both. The IPTV then routes directly from the first router and internet is forwarded on 2 ports to both OPNsense boxes. I would however end up with double nat and still have a single point of failure.
The additional box can also serve as an extra node giving CEPH its minimum of 2 disks. Giving both these boxes 2 quorum votes or adding Qdevices would then indeed make the system resilient against a failure of the 3 H3's or the failure of one of the OPNsense boxes.

Probably better (=avoiding double nat) is to make a second small (32GB) ZFS partition for Proxmox on the Topton box avoiding a CEPH failure from taking the router down. The Topton part of the cluster can be kept alive by giving it 2 additional votes plus a Qdevice (to have some redundancy in the voting).

Am I correct that giving the Router box 3 votes plus a Qdevice and having the router on a ZFS partition will keep the router alive? The router box has 3 votes plus the Qdevice = 4 votes out of 7 so Proxmox and thus the router keeps running on the Topton. If the Topton box goes down the H3's keep running as well (just think of giving the topton 2 votes and the Qdevice 2 that would make 2 failures possible as long as it's not the Topton and Qdevice).

 

[Dunuin]

I looked at the carp solution and indeed that gives you redundant routers, however I would need multiple IP's which my ISP doesn't provide.

CARP works with virtual IPs and while both OPNsense VMs will be running 24/7, only one at a time is actively working as your router and the viretual IP will always be pointing to the single active OPNsense. So a single public IP is totally fine.

See: https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration