Isolated Proxmox Cluster with virtualised pfsense HA

waltk

New Member
May 27, 2024
1
0
1
Hi folks,

I’m new to Proxmox (previously XCP-ng, Xen). I’ve taken inspiration from some great threads here (thank you) and would like to get some feedback/advice from anyone experienced with Proxmox and Hetzner.

Goals:
  • pfsense HA – with everything behind it, so no exposed Proxmox hosts.
  • A Public /27 subnet assigned to pfsense WAN. With MGMT, LAN, and DMZ on the other side, plus OpenVPN for remote access.
  • 3 node Proxmox cluster – HA is not essential. Internally, I’ll be running 2x HAProxy VMs (keepalived) on separate Hosts that load-balance services.
  • Avoid reassigning MACs with Hetzner (manually or API) for the Public Subnet in case of failover and avoid abuse notifications from Hetzner.
  • Keep it simple.

See diagram below. Hope it’s clear – in short:
  • All hosts are connected via 10G managed switch, so VLAN separation, SDN management within Proxmox, etc.
  • 2x hosts running virtualised pfsense in HA. Third host has a cold spare which can be powered up and import configs if needed.
  • Main uplink NIC of each host is PCI passed through to the pfsense VM.
  • Everything is in the same rack/dc. SPOF I know, but latency across DCs is not ideal for some of our services, and it’s possible to have a similar setup in another DC with a fibre interconnect (surprisingly inexpensive).

Questions:
  • According to Hetzner, IPs from Public VLAN Subnets assigned to a vSwitch can be used by any Host or VMs connected to the vSwitch. What’s your expereince with the switch over time for IPs on Hetzner’s side if say, root-server-1 dies and the backup pfsense on root-server-2 becomes the master?

  • How to configure the WAN side of pfsense? Usually, I use 3 IPs from the Public Subnet for a CARP WAN setup but with Hetzner, everything must go the Main IP/MAC of the Host which I’m passing through to pfsense. What are the alternatives given I want to avoid Proxmox having a Public interface? I considered using separate dedicated servers for the pfsenses (not virtualised), so networking is completely independent of Proxmox, but thought I’d start with this.

  • What would you do differently? It would be nice to avoid the 1TB outbound bandwidth limit of vSwitches but I couldn’t figure out how to do this without having to reassign the Public Subnet in case of failover.

Thanks, and appreciate any comments.
 

Attachments

  • Hetzner-Proxmox-virtualised-pfsense.png
    Hetzner-Proxmox-virtualised-pfsense.png
    89.9 KB · Views: 34
Last edited: