Trouble mixing public and private IP spaces on PVE host

[gregwbrooks]

Summary: Having trouble mixing public and private IPs on a PVE host. Help!

Details: I have a PVE host with a public IP and an additional /28 block (not contiguous with with host's IP). Everything works fine and all provisioned VMs have public IPs, which is appropriate for their use cases.

The problem: I want to use 10.10.x.x IPs for some containers and VMs that will run Kubernetes, likely with a simple Nginx Proxy Server in front or, if it's the only way to do it, by assigning a public IP to an Opnsense VM and using it as a gateway into the private IP space.

Working /etc/network/interfaces file -- I've never had to do anything with the /28 block of public IPs other than assign IPs to the VMs:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
# dns-nameservers 127.0.0.1
# dns-* options are implemented by the resolvconf package, if installed

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
address PVE.HOST.PUBLIC.IP/30
gateway PVE.HOST.PUBLIC.GATEWAY
bridge-ports eno1
bridge-stp off
bridge-fd 0

My problem: Everything I've tried to create a bridge into the private IP space hasn't worked. The following doesn't work, either specifying eno1 or vmbr0 in the masquerade lines:

auto vmbr1
iface vmbr1 inet static
address 10.10.0.1/16
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.0.0/16' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.0.0/16' -o vmbr0 -j MASQUERADE

(I know I'd need additional lines for routing specific ports; those are deleted here.)

 

[sb-jw]

I don't understand why you want to usw NAT. You only need this if you only have one external IP. You can also mount multiple IPs on one interface, it doesn't matter whether public or private IP. You can also work with VLAN within a node.

 

[gregwbrooks]

I don't understand why you want to usw NAT. You only need this if you only have one external IP. You can also mount multiple IPs on one interface, it doesn't matter whether public or private IP. You can also work with VLAN within a node.

A few reasons...

The static IPs cost me money -- the fewer I use, the happier I'll be.

Some services (mail server comes to mind) really benefit from a static, public IP. But the rest? I want to try moving them to k3s within Proxmox.

While I could, in theory, have every k3s node on a public IP, it seems easier to manage if there's a single point of entry.

As much as anything else? Once I realized there were things keeping me from doing it, I wanted to see if I could do it.

Since my post, I've figured out that I can use a SDN to handle DHCP provision and the private-IP subnet. Seems to work like a charm, although I'm still figuring out the best way to route traffic into and out of the subnet. Perhaps a firewall rule taking all traffic from a single public IP and pointing it into the subnet?

 

[sb-jw]

A few reasons...

I don't need an explanation as to why this makes sense. NAT is simply not necessary for your goal. You can simply create an interface or assign another IP on top of the existing one. You have absolutely no reason for NAT or SDN with this requirement.

Perhaps a firewall rule taking all traffic from a single public IP and pointing it into the subnet?

Why don't you just give the PfSense a virtual IP, e.g. 192.168.1.1/24 as the gateway IP, then on your VMs you set the 192.168.1.2 as the IP and the 192.168.1.1/24 as the gateway. All devices can now communicate externally.
On the PfSense you can also set up a HAProxy and simply enter the corresponding IPs in the backend. However, I would recommend that you set up an additional instance for HAProxy and not let it run on the firewall.

By the way, you can also do DHCP with PfSense or with SDN. You could use PfSense to store an entry in the DNS and then have everything resolvable directly.

 

[muekno]

I hav e similar problem, I only have a /28 public range and want to connect different hosts on a privat subnet from different public IPs of the /28 range