Multiple Public IP, Single Interface, Some Guests with Public and Private IP

[mightyschwartz]

I have been using Proxmox in my homelab for around 2 years. At my home, I have a proper router/switch/etc so this is a bit easier to do. Recently my company tasked me with putting a Proxmox box in a local datacenter to get more reliable power/internet.

I am going to try to be as thorough as possible here, while masking public IP info.

The setup from the datacenter is as such.

• Usable public IPs: 5 (all on the same subnet)
• Subnet: 255.255.255.248
• Gateway: 66.X.X.225
• Usable IPs: 66.X.X.226-230
• One network interface port.

My public network on the Proxmox box is vmbr0
My private network(vmbr1) is 10.0.0.0

I am trying to set my network up using only Proxmox networking and firewall whereas before I've relied on pfsense. I'm hesitant to do so now because pfSense would be virtualized and if there were an issue, I'd be cut off from my server until the next business day.

My goal is to have my Proxmox host on a public IP 66.X.X.226 while allowing certain VMs to pull from the remaining 4 if I want them to be public facing. Additionally I'd like to have them all on a Local Network as well. Lastly, I'd like to be able to use NAT to push ports on the main proxmox IP 66.X.X.226 to guests that do not have a public IP. For instance, push requests on 66.X.X.226:8888 => 10.0.0.100:3306

Proxmox Host /etc/network/interfaces file:

auto lo
iface lo inet loopback

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 66.X.X.226
        netmask 255.255.255.248
        gateway 66.X.X.225
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

auto vmbr1
iface vmbr1 inet static
        address 10.0.0.1
        netmask 255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' ! -d '10.0.0.0/244' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE

Guest VM1 /etc/network/interfaces

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
        address 66.X.X.227
        netmask 255.255.255.248
        gateway 66.X.X.225
        dns-nameservers 8.8.8.8
        dns-search domain.com

auto ens19
iface ens19 inet static
        address 10.0.0.100
        netmask 255.255.255.0
        broadcast 10.0.0.255
        gateway 10.0.0.1
        dns-nameservers 8.8.8.8
        dns-search domain.com

Guest VM2 /etc/network/interfaces

auto lo
iface lo inet loopback

auto ens18
iface ens18 inet static
        address 66.X.X.228
        netmask 255.255.255.248
        gateway 66.X.X.225
        dns-nameservers 8.8.8.8
        dns-search domain.com

auto ens19
iface ens19 inet static
        address 10.0.0.101
        netmask 255.255.255.0
        broadcast 10.0.0.255
        gateway 10.0.0.1
        dns-nameservers 8.8.8.8
        dns-search domain.com

With my current configuration, the IP addresses are pulled by the VMs and I can SSH to whichever one goes up first (227 or 228) from outside of the datacenter network(routing issue?).

However, I can SSH to both from the Proxmox box(66.X.X.226). It seems I have a routing issue a in how that's handled and I'd rather learn a little here than break down and install pfSense as a VM. Lastly, the NAT forward of ports from the main IP(66.X.X.226), I have no idea how to do this. I've tried a few things and failed. I am open to anyone's advice here.

 

[Nikolay]

First of all, u dont need to use gateway option at ens19, local network will be available, coz it`s directly connected.

Can u show routing table from one of ur vm?

For port forwarding u can try iptables rules:

iptables -A PREROUTING -t nat -i vmbr0 -p tcp --dport 8888 -j DNAT --to-destination 10.0.0.100:3306
iptables -A FORWARD -p tcp -d 10.0.0.100:3306 --dport 3306 -j ACCEPT
iptables -A POSTROUTING -t nat -s 10.0.0.100 -o vmbr0 -j MASQUERADE

 

[mightyschwartz]

First of all, u dont need to use gateway option at ens19, local network will be available, coz it`s directly connected.

Can u show routing table from one of ur vm?

For port forwarding u can try iptables rules:

iptables -A PREROUTING -t nat -i vmbr0 -p tcp --dport 8888 -j DNAT --to-destination 10.0.0.100:3306
iptables -A FORWARD -p tcp -d 10.0.0.100:3306 --dport 3306 -j ACCEPT
iptables -A POSTROUTING -t nat -s 10.0.0.100 -o vmbr0 -j MASQUERADE

Nikolay: True. I had actually tried adding the gateway based on another thread.

What I have done in the interest of time is setup pfSense on a separate IP.

so I have public IP: 66.X.X.226 hosting proxmox, then I have a pfsense vm on there taking 66.X.X.227 and using virtual IPs for 66.X.X.228-230

This seems to be working and I think my colleagues will appreciate the GUI