Dedicated Server: Single NIC. Use 2nd IP for VM traffic

gregg098

Well-Known Member
Apr 5, 2018
39
5
48
44
Update: Working now. Working configuration shown at the bottom.

Looking for some assistance to see what Im doing wrong in this setup.

Dedicated server with ColoCrossing with 2 public IPs. Single network interface enp3s0.

The following setup works just fine. VMs are setup with an IP of 10.0.0x and gateway of 10.0.0.1. Traffic originates from <IP1>
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

auto enp3s0
iface enp3s0 inet static
    address <IP1>
    gateway <GW>

auto vmbr0
iface vmbr0 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    address  10.0.0.1/24
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j MASQUERADE

I would like to make it so all traffic on vmbr0 uses the 2nd IP address. Essentially making <IP1> for Proxmox management only. <IP2> to be the source of all VM traffic.

From research on forums and documentation, it looks like I should change my NAT rules to something like this:
Code:
post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j SNAT --to-source <IP2>
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j SNAT --to-source <IP2>

But when I do this, I dont get internet in any of the VMs. VMs are still set to using IPs from the range 10.0.0.0/24, with gateway 10.0.0.1.

What am I doing wrong?

As an additional test, I setup an OPNsense VM tied to the bridge with the first NAT setup. I created a virtual IP alias of <IP2> and an outgoing NAT rule to use <IP2>, and it works just fine. So the second IP is routed properly. I just cant get it to work in a Proxmox only environment (no OPNsense as a middle man for the VMs). I would prefer not to manage the OPNsense VM as well just for this.

Thank you.

UPDATE1:

I did get it working finally after messing around for awhile. Below is my working config. I think what really made it work was adding enp3s0's mac address to vmbr0 due to it being a hosting providers network.
If anyone sees anything wrong with this configuration, please let me know.

Code:
auto enp3s0
iface enp3s0 inet static
    address <IP1>
    gateway <GW>
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

auto vmbr0
iface vmbr0 inet manual
    hwaddress <mac address of enp3s0>
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    address  10.0.0.1/24
    up ip route add <IP2> dev vmbr0
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j SNAT --to-source <IP2>
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp3s0 -j SNAT --to-source <IP2>

    post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
 
Last edited: