Help with HA setup

[DJFraz]

Hello,

Running PVE 7.1-8 (3 Node Cluster) I'm looking for the best method of getting pfSense failover for my situation. I've tried just about every configuration I could Google up and can't get it working. I'm hoping someone may have an idea of what I direction I could go.

I only have one DHCP IP address (that rarely ever changes) available to me. I've attempted using local addresses for WAN and a VLAN on my switch along with CARP, but neither pfSense install can see the modem on the VLAN through my Unifi switch. I currently have two PCIe NICs passed through to each pfSense VM. When I attempted to utilize a Linux Bridge and the entire PVE server rebooted, I would also have to reboot the modem to be able to grab a connection. When I used passthrough it resolved this issue. Because I'm using physical PCIe passthrough, I can't use one VM in HA mode.

I've run out of directions to go to get HA for pfSense on Proxmox. Aside from two physical pfSense servers, does anyone have any additional ideas on how to have HA for my pfsense setup?

Thanks in advance.

 

[Dunuin]

Doesn't pfsense has build in HA? OPNsense does and its based on pfsense. You could run 3 OPNsense VMs in parallel and keep them in sync with CARP and pfsync so only one of the 3 VMs is the "master" router and the other 2 VMs are idleing as "backup" routers.

 

[DJFraz]

It does have built in HA but requires all nodes to be able to communicate with the modem. I haven’t been able to get this to function properly as I only have a single port modem and none of the nodes can talk to the modem through my managed switch

 

[Astraea]

The way that I did my setup for HA pfSense while only running a single pfSense VM is to make use of my managed switches VLAN and LACP abilities. I'll simplify my install a little for the length of my post but it won't change the design. So for this post, I have 1 managed switch (called MAIN), a modem and 3 ProxmoxVE nodes (PVE1, PVE2, PVE3), I will also be using a NAS device for shared storage (let's call this one NAS1).

I connect my modem to the switch let's say on ports 1 and 2 on each device and the ports on the switch are configured so that port 1 is only using VLAN WAN1 and port 2 is using WAN2 VLAN only. I have 2 WAN addresses (a static and a dynamic) so I have each of those on their VLAN on the switch. I have NAS1 connected to the switch say on ports 24 and is configured to be on the STORAGE VLAN when untagged,

I then have my ProxmoxVE nodes connected to the switch with 2x1GB connections (say ports 3, 4 and 5, 6, and 7, 8) that are LACP bonded together (each configured in ProxmoxVE as vmbr0) and is set to use the LAN network as its untagged network and has access to the WAN1 and WAN2 VLANs using tags. I then connect the ProxmoxVE nodes to the switch again (ports 9, 10, 11) on another interface (set to use the management VLAN as untagged) this is how I access the ProxmoxVE WebUI. The only difference between the LAN and the MANAGEMENT VLANs is that only specified devices can access the management VLAN. the MANAGEMENT VLAN is also the same VLAN that my switch management WebUI is on too. I use the 4th interface on the ProxmoxVE nodes to connect to the STORAGE VLAN to configure the shared storage with NAS1.

vmbr0 is assigned to the pfSense VM using three separate virtual network devices, the first is configured to use the WAN1 VLAN by adding the appropriate tag and the second is untagged so it used the LAN VLAN (though has access to the MANAGEMENT VLAN) and the third is cnfigured to use the WAN2 VLAN via the tag on the network device. I then configure everything in pfSense as it was a "physical interface" and can also configure the MANAGEMENT VLAN in pfSEnse as well and set it to use the LAN as the parent interface.

As the ProxmoxVE nodes are all configured the same and the pfSense VM can access everything it needs no matter the node it is running on I can not only migrate the VM if I need to do maintenance or have a node failure. I have the VM running under HA so it can move when needed to keep it running and also will autostart when storage is available should I be starting things up from cold and dark.