Distributing Public Network Within Cluster

[markc]

I've got a private 4 node homelab cluster, and I'd like to set up a public cluster at a hosting provider. The problem is that they can only BGP route a /28 network to a single node, so I am struggling to understand how best to take advantage of those public IPs within a 3 node (or more) cluster. I have installed libpve-network-perl on all my homelab nodes so the SDN options are available for testing from the Datacenter dashboard, but I obviously can't emulate a public /28 example with my current homelab. I do have access to a 2nd NIC via a VPC at the hosting provider, so assigning the /28 to it, with some firewall/routing rules to expose the /28 to the outside world, may be one way to allow the /28 to be distributed cluster wide. A dedicated reverse proxy in front of the cluster may also be an option, but either way it looks like I have to deal with a single point of failure for the exposed /28 ingress/egress point.

Could anyone suggest the best, or any, approach to take advantage of and distribute a public subnet within a cluster?

 

[HLPCLC]

I am currently working on a similar problem but with my home lab. My home is behind an AT&T BGW-320 FTTH ONT/Gateway. I can pass my public block to my network via the public subnet hosts feature, via either DHCP or as a static option. I am sure there are more ways. I can also use IP passthrough to bind the ONT gateway's public IP to a specific MAC address, for a total of 6 routable IPs, 5 from my /28 and another from my home's /30 or whatever they use

Anyways, I have the proxmox itself on the /30 through my single port, realtek NIC, and it runs on NAT behind the AT&T gateway. I also have a dual port Intel NIC I am using for WAN and LAN for a pfSense VM. I got it working by assigning two virtual interfaces, one for WAN and one for LAN, neither with any designation directly in proxmox. It works perfectly fine when I set the WAN static IP within pfSense, and set its own default gateway, which is different than the proxmox's default gateway. Because of this stroke of luck, I can set virtual IP Aliases within the pfSense, assign LAN interfaces public IPs, or 1:1 nat public IPs to devices AND virtual machines. I give the VMs the interface as the pfSense's LAN virtual interface. I haven't tried anything with VLANs yet.

I also want to harden the pfSense firewall with the Proxmox firewall. The listed protocols are:

tcp, udp, icmp, igmp, ggp, ipencap, st egp, igp, pup, hmp, xns-idp, rdp, iso-tp4, dccp, xtp, ddp, idpr-cmtp, idrp, rsvp, gre, esp, ah, skip, vmtp, eigrp, ospf, ax.25, ipip, etherip, encap, pim, ipcomp, vrrp, l2tp, isis, sctp, fc, mobility-header, udp-lite, mpls-in-ip, hip, shim6, wesp, rohc.

pfSense mentions absolutely nothing about these and I am not 100% certain if Wireshark recognizes them all. I think if you want to assign individual nodes public IPs, you'd want to first assign a Static Public IP to your initial setup node, and then just assign interfaces within the same subnet a public IP from the /28. Otherwise you get errors trying to use multiple default gateways in proxmox. It definitely doesn't seem secure this way, because the public can literally poke at your management node and data center. Also, maybe try using a VM like pfSense to route them for you. You would have to have one node's pfSense controlling all of the others, however, you might be able to set up CARP or similar to sync all of the pfSenses virtual machines on each of the nodes too.

https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html

I am using a virtual interface for the pfSense VM NIC, but I think you can also use IOMMU and PCIe passthrough. I am not 100% certain about which is more or less secure. Also, I don't know how to use a reverse proxy yet. I use Squid Proxy which is a forward proxy for HTTP and HTTPs to make downloads for my videogames faster. (MUCH, much faster, like wow).

Essentially with my public block my pfSense VM (and/or my bare-metal pfSense) is the new border gateway. I have no idea how to secure a border gateway with those firewall protocols. Do you have any recommendations?

 

[spirit]

I've got a private 4 node homelab cluster, and I'd like to set up a public cluster at a hosting provider. The problem is that they can only BGP route a /28 network to a single node, so I am struggling to understand how best to take advantage of those public IPs within a 3 node (or more) cluster. I have installed libpve-network-perl on all my homelab nodes so the SDN options are available for testing from the Datacenter dashboard, but I obviously can't emulate a public /28 example with my current homelab. I do have access to a 2nd NIC via a VPC at the hosting provider, so assigning the /28 to it, with some firewall/routing rules to expose the /28 to the outside world, may be one way to allow the /28 to be distributed cluster wide. A dedicated reverse proxy in front of the cluster may also be an option, but either way it looks like I have to deal with a single point of failure for the exposed /28 ingress/egress point.

Could anyone suggest the best, or any, approach to take advantage of and distribute a public subnet within a cluster?

you could use evpn zone in sdn to redistribute the /28.

in zone option, enable exit-node option on the node where the /28 is coming.

(for redundancy, you can enable multiple exit-node, but it's really depend of your hosting provider if you announce the /28 at the same time on 2 exit-node, or if your hosting provider is active/passive and can do failover if one node goes down. Bgp could be used too.).

Then in the zone, create 1 vnet + 1 subnet /28 + 1ip from the subnet for the gateway.

 

[tomcatcher]

Hi Spirit,

you could use evpn zone in sdn to redistribute the /28.

Is there more details on this option? I have a setup, where this whould be beneficiary. Can You help me implement it or at least point me to the right direction? Thanks

Bgp could be used too

I think eVPN in Proxmox is using BGP (or eBGP) for MAC/IP routing.

Then in the zone, create 1 vnet + 1 subnet /28 + 1ip from the subnet for the gateway.

In my setup (OVH dedicated server), when I am bridging a VM's virtual interface into the vmbr0 bridge on eth0 as the main IP (public) of the node, I need to set a specific GW address and predefined MAC address to make it work. Why do I need a GW address also in thesubnet of the vnet in the eVPN zone?

I would really appreciate more information and technical/implementation details about this setup. I have a Proxmox cluster with 3 nodes, from which only one has additional public IPs and I am thinking about a way of destributing this public IPs on all cluster nodes.

Thank You very much in advance. I am studying about eVPN and BGP and I think this is the way. As far as I see it now, I'd like to turn my Proxmox cluster into an AS (Autonomous System). I thinking that I might be able to use a eVPN controller to set up this and another VXLAN for each user to have it's own private network. New instances and appliances will be placed into this network (each of them), plus the ones with public IPs will be in the other too. Or should I use 1:1 NAT? Which way would You recommend.

Tom