pfsense on a 3 node cluster

May 14, 2019
43
1
13
59
Hello, I am considering running a pfsense vm on proxmox 7 and would like some feed back on using 3 NICS, each attached to a WAN linux bridge.
Each node will have a dedicated internet connected NIC, with pfsense actively connected and protecting it.

My concern is the WAN nic's and WAN bridges on the other 2 nodes, which have no configuation of any kind or any active network vm connections until the pfsense vm migrates to another node.

Is this a 'safe' configuration ?

Thanks for you input
 

MrPete

Member
Aug 6, 2021
57
7
8
64
I'm using HA CARP which is built into pfSense... in my case on 2 proxmox VM's.
This has some advantages over vm migration. Most particularly: CARP maintains a live backup of all connections. Rather nice when a cutover maintains all connections.

Do you have 2+ WAN connections? If so, it's simple. If not... it's supposed to work and I almost have that part nailed.
 
May 14, 2019
43
1
13
59
I'm using HA CARP which is built into pfSense... in my case on 2 proxmox VM's.
This has some advantages over vm migration. Most particularly: CARP maintains a live backup of all connections. Rather nice when a cutover maintains all connections.

Do you have 2+ WAN connections? If so, it's simple. If not... it's supposed to work and I almost have that part nailed.
I had not considered the HA angle, I could run 3 pfsense vm's in HA CARP mode and not worry about the potential issues I raised.
Any advice as I have never worked with HA CARP before ?

thanks
 

MrPete

Member
Aug 6, 2021
57
7
8
64
Follow the pfSense CARP cookbook carefully.
  • The MOST important thing by far: all network devices must be in the exact same order internally to pfSense, as that is the assumption for sync. If that's a problem, there's a workaround using LAGGs.
  • If you end up having to reconfigure network interfaces, a hint: the easy/correct way is to save the config (it's an XML file.) Edit the config.xml, load on a USB stick, and make it available during pfSense boot -- it will fully reload the config and all is well. This is documented if obscure.
  • If performance is an issue, PCI(e) passthrough is a real benefit (and impossible for vm migration). If you can do SR-IOV, that's even better for some applications. I'm heading in the SR-IOV direction myself. I easily max out my WAN fiber gigabit link, even though it is pppoe.
  • Consider carefully how your network will function when one or more pfSense's are down. At the moment, I'm using raw IP addresses to access proxmox when there's no pfSense (particularly because I use HAproxy for "nice" handling of HTTPS ;) ). I could probably do some of myDNS stuff in proxmox itself... I'm obviously not 100% done. I do have other priorities in life, as this is my HomeLab (which serves my sweetie of 42 years, and I want to keep it that way so I try not to have everything die toooo often ;) )
  • There are some oddities: DNS comes from Primary. DHCP tends to come from both Primary/Backup IP's but is kept in sync. Not sure why.
  • I'm told IPv6 works fine... I've not yet joined the world of IPv6. :-D
  • Note that ifconfig is fully CARP aware (in FreeBSD)
 
Last edited:
May 14, 2019
43
1
13
59
Follow the pfSense CARP cookbook carefully.
  • The MOST important thing by far: all network devices must be in the exact same order internally to pfSense, as that is the assumption for sync. If that's a problem, there's a workaround using LAGGs.
  • If you end up having to reconfigure network interfaces, a hint: the easy/correct way is to save the config (it's an XML file.) Edit the config.xml, load on a USB stick, and make it available during pfSense boot -- it will fully reload the config and all is well. This is documented if obscure.
  • If performance is an issue, PCI(e) passthrough is a real benefit (and impossible for vm migration). If you can do SR-IOV, that's even better for some applications. I'm heading in the SR-IOV direction myself. I easily max out my WAN fiber gigabit link, even though it is pppoe.
  • Consider carefully how your network will function when one or more pfSense's are down. At the moment, I'm using raw IP addresses to access proxmox when there's no pfSense (particularly because I use HAproxy for "nice" handling of HTTPS ;) ). I could probably do some of myDNS stuff in proxmox itself... I'm obviously not 100% done. I do have other priorities in life, as this is my HomeLab (which serves my sweetie of 42 years, and I want to keep it that way so I try not to have everything die toooo often ;) )
  • There are some oddities: DNS comes from Primary. DHCP tends to come from both Primary/Backup IP's but is kept in sync. Not sure why.
  • I'm told IPv6 works fine... I've not yet joined the world of IPv6. :-D
  • Note that ifconfig is fully CARP aware (in FreeBSD)
Thanks for you time and info, I will follow the CARP cookbook and see if I can get this to work.
 

MrPete

Member
Aug 6, 2021
57
7
8
64
On the LAN side, each accessible LAN interface requires both:
* A shared CARP IP, that switches back and forth (I use *.1)
* A unique IP for that interface (I use *.2 for primary and *.3 for secondary)
Since pfSense supplies DHCP and local DNS, that's pretty easy.

On the WAN side,
a) it's *simpler* if you have (at least) two ISP static IP's. However,
b) pfSense now supports use of a single WAN IP plus non-routable local IP's for the per-interface part.
c) What is NOT yet supported is static pppoe IP... which is what I have with my gigabit fiber-to-doorstep. I have worked around that limitation, kinda-sorta.
 
May 14, 2019
43
1
13
59
On the LAN side, each accessible LAN interface requires both:
* A shared CARP IP, that switches back and forth (I use *.1)
* A unique IP for that interface (I use *.2 for primary and *.3 for secondary)
Since pfSense supplies DHCP and local DNS, that's pretty easy.

On the WAN side,
a) it's *simpler* if you have (at least) two ISP static IP's. However,
b) pfSense now supports use of a single WAN IP plus non-routable local IP's for the per-interface part.
c) What is NOT yet supported is static pppoe IP... which is what I have with my gigabit fiber-to-doorstep. I have worked around that limitation, kinda-sorta.
ok, now you have my attention, I can work with single WAN IP scenario, but not the multiple static situation, I will look for references to the single IP WAN solution.

Thanks
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!