Routing 2 public IP ranges on 2 bridges

[richard.keller]

I've set up a new Proxmox 8 server, and I'm struggling to get my networking set up correctly. A normal bridge interface with a single IP range is easy enough, but I can't seem to get my 2 IP ranges to route correctly when using 2 bridges.

Background:

• The Proxmox host has a single physical interface, eno1, which plugs directly into my hosting provider's switch
• My hosting provider has given me 2 IP ranges.
  • 129.212.157.200/29 (Gateway is 129.212.157.201)
  • 129.212.175.24/29 (Gateway is 129.212.175.25)
• I have created two bridges, vmbr0 and vmbr1. This is to provide layer 2 isolation, since I don't entirely trust the VMs on vmbr1, since a third-party vendor has root access to them.
• The primary IP address to access the Proxmox server should be 129.212.157.202.

My understanding is that this situation requires a routed configuration. However, this is where things start to fall apart.

My /etc/network/interfaces config looks like this:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address 129.212.157.202/29
    gateway 129.212.157.201

auto vmbr0
iface vmbr0 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

Ideally, I would like to mix-and-match IPs from both IP ranges on both vmbr0 and vmbr1. In other words, I don't want one IP range to be assigned exclusively to vmbr0 and the other IP range to be assigned exclusively to vmbr1.

I have manually added the following routes using ip ro:

default via 129.212.157.201 dev eno1 proto kernel onlink
129.212.157.200/29 dev eno1 proto kernel scope link src 129.212.157.202
129.212.175.24/29 via 129.212.175.25 dev eno1
129.212.175.25 dev eno1 scope link

Now, I have tried to create a VM and bound the virtual NIC to vmbr0, and I've also tried vmbr1. For both of these combinations, I have tried configuring the VM with IPs from both the 129.212.157.200/29 range as well as from the 129.212.175.24/29 range, using the gateways 129.212.157.201 and 129.212.175.25 respectively. No matter what combination of vmbr and IP address I use for the VM, I can't seem to get connectivity. I can't even ping the gateway IP from within the VM.

For comparison, and to test that the two IP ranges are in fact routable via the provided gateways, I have successfully set up a simple bridged implementation with a single vmbr0 per the Proxmox documentation, and that works fine, for both IP ranges and both gateways. Things only seem to fall apart when I switch to the routed approach.

I've also confirmed that Proxy ARP and IP Forwarding are both enabled on the Proxmox host.

To me, this looks like a routing issue on the Proxmox host, but for the life of me I can't see what I'm doing wrong.

As far as I can tell, I haven't configured anything special in terms of firewall rules. I do have the firewall enabled at the datacentre level, the host level, and the VM level. I have tried turning off firewalling at the host and VM level, but that also doesn't seem to make a difference.

I've sat for hours quizzing ChatGPT about this already, and haven't made much headway. One suggestion I keep getting is to implement NATting, but I don't see why that would be necessary, since each of the VMs will only have a public IP, and no private IP ranges.

 

[gurubert]

Do both of the gateway IPs belong to the router of your provider? Then this is not a routed setup, it's a bridged setup.
You need to configure eno1 as bridge port for vmbr0 and bind the IP .202 to vmbr0. The VMs can then use one IP on either of the networks.
You cannot separate the VM traffic in this case because everything runs on one layer 2 network via eno1.

 

[gurubert]

I have successfully set up a simple bridged implementation with a single vmbr0 per the Proxmox documentation, and that works fine, for both IP ranges and both gateways.

Then why don't you use that?