pfsense on a 3 node cluster

[David Thistlethwaite]

Hello, I am considering running a pfsense vm on proxmox 7 and would like some feed back on using 3 NICS, each attached to a WAN linux bridge.
Each node will have a dedicated internet connected NIC, with pfsense actively connected and protecting it.

My concern is the WAN nic's and WAN bridges on the other 2 nodes, which have no configuation of any kind or any active network vm connections until the pfsense vm migrates to another node.

Is this a 'safe' configuration ?

Thanks for you input

 

[MrPete]

I'm using HA CARP which is built into pfSense... in my case on 2 proxmox VM's.
This has some advantages over vm migration. Most particularly: CARP maintains a live backup of all connections. Rather nice when a cutover maintains all connections.

Do you have 2+ WAN connections? If so, it's simple. If not... it's supposed to work and I almost have that part nailed.

 

[David Thistlethwaite]

I'm using HA CARP which is built into pfSense... in my case on 2 proxmox VM's.
This has some advantages over vm migration. Most particularly: CARP maintains a live backup of all connections. Rather nice when a cutover maintains all connections.

Do you have 2+ WAN connections? If so, it's simple. If not... it's supposed to work and I almost have that part nailed.

I had not considered the HA angle, I could run 3 pfsense vm's in HA CARP mode and not worry about the potential issues I raised.
Any advice as I have never worked with HA CARP before ?

thanks

 

[MrPete]

Follow the pfSense CARP cookbook carefully.

• The MOST important thing by far: all network devices must be in the exact same order internally to pfSense, as that is the assumption for sync. If that's a problem, there's a workaround using LAGGs.
• If you end up having to reconfigure network interfaces, a hint: the easy/correct way is to save the config (it's an XML file.) Edit the config.xml, load on a USB stick, and make it available during pfSense boot -- it will fully reload the config and all is well. This is documented if obscure.
• If performance is an issue, PCI(e) passthrough is a real benefit (and impossible for vm migration). If you can do SR-IOV, that's even better for some applications. I'm heading in the SR-IOV direction myself. I easily max out my WAN fiber gigabit link, even though it is pppoe.
• Consider carefully how your network will function when one or more pfSense's are down. At the moment, I'm using raw IP addresses to access proxmox when there's no pfSense (particularly because I use HAproxy for "nice" handling of HTTPS ). I could probably do some of myDNS stuff in proxmox itself... I'm obviously not 100% done. I do have other priorities in life, as this is my HomeLab (which serves my sweetie of 42 years, and I want to keep it that way so I try not to have everything die toooo often )
• There are some oddities: DNS comes from Primary. DHCP tends to come from both Primary/Backup IP's but is kept in sync. Not sure why.
• I'm told IPv6 works fine... I've not yet joined the world of IPv6. :-D
• Note that ifconfig is fully CARP aware (in FreeBSD)

 

[David Thistlethwaite]

Follow the pfSense CARP cookbook carefully.

• The MOST important thing by far: all network devices must be in the exact same order internally to pfSense, as that is the assumption for sync. If that's a problem, there's a workaround using LAGGs.
• If you end up having to reconfigure network interfaces, a hint: the easy/correct way is to save the config (it's an XML file.) Edit the config.xml, load on a USB stick, and make it available during pfSense boot -- it will fully reload the config and all is well. This is documented if obscure.
• If performance is an issue, PCI(e) passthrough is a real benefit (and impossible for vm migration). If you can do SR-IOV, that's even better for some applications. I'm heading in the SR-IOV direction myself. I easily max out my WAN fiber gigabit link, even though it is pppoe.
• Consider carefully how your network will function when one or more pfSense's are down. At the moment, I'm using raw IP addresses to access proxmox when there's no pfSense (particularly because I use HAproxy for "nice" handling of HTTPS ). I could probably do some of myDNS stuff in proxmox itself... I'm obviously not 100% done. I do have other priorities in life, as this is my HomeLab (which serves my sweetie of 42 years, and I want to keep it that way so I try not to have everything die toooo often )
• There are some oddities: DNS comes from Primary. DHCP tends to come from both Primary/Backup IP's but is kept in sync. Not sure why.
• I'm told IPv6 works fine... I've not yet joined the world of IPv6. :-D
• Note that ifconfig is fully CARP aware (in FreeBSD)

Thanks for you time and info, I will follow the CARP cookbook and see if I can get this to work.

 

[David Thistlethwaite]

Thanks for you time and info, I will follow the CARP cookbook and see if I can get this to work.

quick question, it looks like each pfsense node requires it's own non-CARP IP, do these have to be static (it seems that way)
thanks again

 

[MrPete]

On the LAN side, each accessible LAN interface requires both:
* A shared CARP IP, that switches back and forth (I use *.1)
* A unique IP for that interface (I use *.2 for primary and *.3 for secondary)
Since pfSense supplies DHCP and local DNS, that's pretty easy.

On the WAN side,
a) it's *simpler* if you have (at least) two ISP static IP's. However,
b) pfSense now supports use of a single WAN IP plus non-routable local IP's for the per-interface part.
c) What is NOT yet supported is static pppoe IP... which is what I have with my gigabit fiber-to-doorstep. I have worked around that limitation, kinda-sorta.

 

[David Thistlethwaite]

On the LAN side, each accessible LAN interface requires both:
* A shared CARP IP, that switches back and forth (I use *.1)
* A unique IP for that interface (I use *.2 for primary and *.3 for secondary)
Since pfSense supplies DHCP and local DNS, that's pretty easy.

On the WAN side,
a) it's *simpler* if you have (at least) two ISP static IP's. However,
b) pfSense now supports use of a single WAN IP plus non-routable local IP's for the per-interface part.
c) What is NOT yet supported is static pppoe IP... which is what I have with my gigabit fiber-to-doorstep. I have worked around that limitation, kinda-sorta.

ok, now you have my attention, I can work with single WAN IP scenario, but not the multiple static situation, I will look for references to the single IP WAN solution.

Thanks