Add vmbr1 bridge, same subnet vmbr0, why could not access internet?

[SpookyAction]

When I installed Proxmox 8.04, the vmbr0 was created automatically. Now I added a new network card and created a new bridge vmbr1



After applying the config, the /etc/network/interfaces is updated accordingly

auto lo
iface lo inet loopback
iface enp4s0 inet manual
iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.80/24
gateway 192.168.1.250
bridge-ports enp4s0
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.1.81/24
bridge-ports enp5s0
bridge-stp off
bridge-fd 0
#Proxmox Management

I could access the Proxmox Admin UI and SSH to Proxmox using the new static IP address 192.168.1.81. However the vmbr1 cannot access the internet

curl -sSL --interface vmbr1 --connect-timeout 5 https://pve.proxmox.com/wiki | grep -Eoi '<title>.+</title>'
#curl: (7) Failed to connect to pve.proxmox.com port 443 after 3090 ms: Couldn't connect to server

# But vmbr0 can access internet
curl -sSL --interface vmbr0 --connect-timeout 5 https://pve.proxmox.com/wiki | grep -Eoi '<title>.+</title>'
#<title>Proxmox VE</title>

QUESTION: vmbr1 has no gateway definition. Because if I set the same gateway than vmbr0, Proxmox will reject the creation of vmbr1 (gateway 192.168.1.250 is already used). However the IP address 192.168.1.81 is in the same subnet than the enp4s0 interface underlying the vmbr0 bridge. Should it use automatically the same gateway defined for the subnet 192.168.1.0/24 ?

I suppose this question is more of a general linux networking question rather than Proxmox specific. I wonder what is the purpose of the vmbr1 bridge, because for managing Proxmox server, I think I could use the enp5s0 interface directly without creating a bridge.

The reason I create the new bridge vmbr1, is in fact extracted from a tutorial How to Run OPNsense in a Proxmox Virtual Machine for Evaluation Purposes Before I go through with the exercise, I'd like to play around a bit to get familiarized with Proxmox networking.

 

[Dunuin]

An OS shouldn't got two IPs in the same subnet. This will screw up routing.
Put your management on a dedicated management subnet.

 

[SpookyAction]

An OS shouldn't got two IPs in the same subnet. This will screw up routing.

How come? If the computer has 2 or more network interfaces. Each connected to the home network via a switch. Each could acquire a different IP address and they are all in the same subnet. The OS allows this easily. It is more likely I mis-interpret your answer. Can you please be more specific?

Put your management on a dedicated management subnet.

Can you please suggest how to setup the dedicated management subnet?

- How to get DHCP on this different subnet? Because for now the home network subnet gets DHCP from the router.

- Let's assume I can install a service (which and how?) which fulfills the DHCP function on this new subnet. How can I use a home machine to manage Proxmox server? Because the home machine is on a different subnet.

 

[B.Otto]

You are right, this is more a Linux Networking question (and general Networking) than a Proxmox Networking question.

How come? If the computer has 2 or more network interfaces. Each connected to the home network via a switch. Each could acquire a different IP address and they are all in the same subnet. The OS allows this easily. It is more likely I mis-interpret your answer. Can you please be more specific?

A Linux Bridge is both a 'virtual switch' and a virtual interface held by your Proxmox Host. If you give that Bridge an IP address, this IP is held by your Proxmox OS. In your case your Proxmox has the two IP addresses 192.168.1.80 and 192.168.1.81 on different interfaces.

The basic way the routing table on your Proxmox works is that it associates a subnet with an interface, like so:

• Target Subnet 192.168.1.0/24 => send via Interface vmbr0

Now that your Host has two different interfaces in the same subnet, it doesn't know whether it should send packets through the interface vmbr0 or vmbr1. The result is an erratic behavior where you can't really predict which interface it will take.

So the rule of thumb is to never assign IPs in the same subnet to different interfaces.

QUESTION: vmbr1 has no gateway definition. Because if I set the same gateway than vmbr0, Proxmox will reject the creation of vmbr1 (gateway 192.168.1.250 is already used). However the IP address 192.168.1.81 is in the same subnet than the enp4s0 interface underlying the vmbr0 bridge. Should it use automatically the same gateway defined for the subnet 192.168.1.0/24 ?

Gateways are global and there can only be one Gateway set at a time. Gateway-IPs are also just a 'catch-all-remaining' line in the routing table:

• Target Subnet 192.168.1.0/24 => send via Interface vmbr0
• Target Subnet *.*.*.* => send to Gateway IP (via an appropiate Interface assigned from the Subnet of the Gateway IP)

---
Do note that you don't actually have to assign an IP address to a Linux Bridge. Bridges work on Layer2, and the only reason for assigning IPs to them is that your Proxmox can talk through it. For the classic 'Private network behind OPNsense firewall' setting you put the private stuff on vmbr1 and don't give it any IP address.