Isolated Proxmox Cluster with virtualised pfsense HA

[waltk]

Hi folks,

I’m new to Proxmox (previously XCP-ng, Xen). I’ve taken inspiration from some great threads here (thank you) and would like to get some feedback/advice from anyone experienced with Proxmox and Hetzner.

Goals:

• pfsense HA – with everything behind it, so no exposed Proxmox hosts.
• A Public /27 subnet assigned to pfsense WAN. With MGMT, LAN, and DMZ on the other side, plus OpenVPN for remote access.
• 3 node Proxmox cluster – HA is not essential. Internally, I’ll be running 2x HAProxy VMs (keepalived) on separate Hosts that load-balance services.
• Avoid reassigning MACs with Hetzner (manually or API) for the Public Subnet in case of failover and avoid abuse notifications from Hetzner.
• Keep it simple.

See diagram below. Hope it’s clear – in short:

• All hosts are connected via 10G managed switch, so VLAN separation, SDN management within Proxmox, etc.
• 2x hosts running virtualised pfsense in HA. Third host has a cold spare which can be powered up and import configs if needed.
• Main uplink NIC of each host is PCI passed through to the pfsense VM.
• Everything is in the same rack/dc. SPOF I know, but latency across DCs is not ideal for some of our services, and it’s possible to have a similar setup in another DC with a fibre interconnect (surprisingly inexpensive).

Questions:

• According to Hetzner, IPs from Public VLAN Subnets assigned to a vSwitch can be used by any Host or VMs connected to the vSwitch. What’s your expereince with the switch over time for IPs on Hetzner’s side if say, root-server-1 dies and the backup pfsense on root-server-2 becomes the master?
  
  
• How to configure the WAN side of pfsense? Usually, I use 3 IPs from the Public Subnet for a CARP WAN setup but with Hetzner, everything must go the Main IP/MAC of the Host which I’m passing through to pfsense. What are the alternatives given I want to avoid Proxmox having a Public interface? I considered using separate dedicated servers for the pfsenses (not virtualised), so networking is completely independent of Proxmox, but thought I’d start with this.
  
  
• What would you do differently? It would be nice to avoid the 1TB outbound bandwidth limit of vSwitches but I couldn’t figure out how to do this without having to reassign the Public Subnet in case of failover.

Thanks, and appreciate any comments.