PVE host cannot access internet through second bridge

[Qlii256]

I have a PVE host running with two NICs. I only use one at the moment and it has three virtual bridges. NIC1 has an untagged network (home network) and two tagged VLANs (VM network and VM isolated network). The purpose is that I can create VMs that can access my LAN (home network) and create other VMs that can access the internet using the VM network, where they can also access each other through that network. The third network is for VMs that can only access the internet, but not each other.

I have no rules in my gateway/firewall and still run into issues with my configuration. The PVE host is set up correctly AFAIK, as it has two NICs (only one use atm), two VLANs and three (virtual) bridges. When I open a shell on the host and issue the follow command:

ping -I vmbr0 google.com

I get a response from the google servers.

However, when I issue the following command:

ping -I vmbr1 google.com
ping -I vmbr2 google.com

I get no response and the ping failed.

From my LAN (home network) I can access the server on all of its IP addresses (vmbr0, vmbr1 and vmbr2) without problems. From the host, I can NOT access other LAN devices other than through vmbr0. My firewall (Unifi USG 4 Pro) does not have any rules applied to any of the networks, therefor each VLAN should be able to talk to one another. However, there seems to be an issue with the host OS when routing through different bridges other than the default gateway. I've encountered this issue on a VM as well when I pass two NICs to that VM, and I have to edit Netplan to account for multiple gateways each with a lower/higher metric/priority. However, I'm not sure how to do this on the PVE host.

Another issue I'm facing is when I forward (open) port 80 on my firewall to vmbr0, I can access the web interface of the host from my LAN when using my public IP address, however when I forward it to any of vmbr1 or vmbr2, it does not work. This might be related due to the host not having access to LAN through those bridges.

Clearly something is wrong and I'd like to resolve this so that my host can also access internet and LAN through vmbr1 and vmbr2 just like vmbr0. After, I'll create firewall rules on my gateway to force VLANs not being able to talk to each other, other than certain ports/protocols which I'll allow (such as port 80 for the web interface). I suspect this has something to do with routing. It's coming in on vmbr1, but trying to go out on vmbr0?

# configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno8303 inet manual iface eno8403 inet manual auto eno8303.10 iface eno8303.10 inet manual auto eno8303.20 iface eno8303.20 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.1.130/24 gateway 192.168.1.1 bridge-ports eno8303 bridge-stp off bridge-fd 0 #Home Network auto vmbr1 iface vmbr1 inet static address 192.168.10.100/24 bridge-ports eno8303.10 bridge-stp off bridge-fd 0 #VM Network auto vmbr2 iface vmbr2 inet static address 192.168.20.100/24 bridge-ports eno8303.20 bridge-stp off bridge-fd 0 #VM Network (gateway)

 

[UdoB]

When you ping via vmbr0 the IP (Internet Protocol) knows that there is an exit (gateway) at 192.168.1.1.

How can a ping leaving your host via vmbr1/vmbr2 know where to step forward? You need a router for this task...

 

[Qlii256]

When you ping via vmbr0 the IP (Internet Protocol) knows that there is an exit (gateway) at 192.168.1.1.

How can a ping leaving your host via vmbr1/vmbr2 know where to step forward? You need a router for this task...

Ok, so this is normal behaviour? But what I cannot figure out is why I can go from LAN to the IP of vmbr1 and vmbr2 no problem (because it's going via router?) from my Windows pc. But when I do a port forward on my gateway for say port 8006 to vmbr1 IP, I cannot access it with my public ip (both outside and inside my LAN).

EDIT: It makes more sense to me now, thank you. With the help of the following link I was able to add another routing table and have a default route set-up for vmbr1 (VLAN 10). This seems to be working now! I can now access the web ui using my external IP address, as well as ping outside the vmbr1 using:

ping -I 192.168.10.100 google.com

Note that using the interface name, vmbr1, does not seem to work. Not sure what's up with that?

Also, I'd like to know if this is the correct way to do something like this. I've removed all the bridges and VLANs and added only a vmbr0 that is vlan-aware. I now have only vmbr1 NIC added to each VM and I put the VLAN-tag in there. It's a bit clearer and that way I only need to manage one bridge device.

https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System

 

[Qlii256]

Anyone got any idea why I cannot simply do:

ping -I vmbr1 google.com

while

ping -I vmbr0 google.com

works, and (192.168.10.100 is the IPv4 address used for the vmbr1 bridge)

ping -I 192.168.10.100 google.com

also works?

 

[cvonfeldt]

having the same issue - did you solve?

 

[joltman]

I know the original post is nearly two years old, but there is so much info missing here. OK, you have a Unifi firewall. What kind of switch do you have? Do you have layer two VLANs on this switch? Or is the switch a layer three switch with VLAN interfaces on it? Or does the switch have a trunk to the firewall, and the VLAN interfaces are on the firewall? Why would you have IPs on each bridge if you have VLAN interfaces on the switch or on the firewall? The only reason to have bridges with their own IPs is to:

1. Make the PVE host GUI/SSH available on multiple networks.
2. Trying to make the PVE host do some crazy routing using FFR _between_ the VMBRs (and hopefully the upstream router too). I doubt that's something a novice would want to do.

The problem with all VMBRs having an IP is that it allows your DMZ machines access to the PVE host's interface (that's a no no). I would think the most common configuration would be:

vmbr0: this is the bridge that has the bond on it
vmbr0.2: this is only used for PVE host connectivity. It will have a static IP on it of 192.168.1.130/24.

You will assign vmbr0 to every VM/CT, but you will assign a VLAN tag to the network interface on those VMs/CTs.

The Unifi would have VLAN interfaces for all of these, and then have rules for VLAN interface 20 to block all access to 192.168.1.0/24, and 192.168.10.0/24, but allow everything else (ie, the Internet).

You're also not even using the interface 8403 either. If it were me, I'd setup an LACP bond of those two interfaces 8303 and 8403 and create bridges with the appropriate VLANs. Something like this example (backup your config _before_ making any changes!!):

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

# This configures LACP mode between the host and the switch.  Verify your switch can handle that, and if not, change the bond mode using the documentation.
auto bond0
iface bond0 inet manual
      bond-slaves eno1 eno2
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer3+4

# Should never use VLAN1 for anything so change that on the switch side and Unifi side too.  Used _only_ to access PVE on VLAN 2, internal LAN facing only.
# Always tag traffic on the switch side and the PVE side

auto bond0.2
iface bond0.2 inet manual

auto vmbr0v2
iface vmbr0v2 inet static
        address  192.168.1.130/24
        gateway  192.168.1.1
        bridge-ports bond0.2
        bridge-stp off
        bridge-fd 0

# This is the bridge that is VLAN aware.  You assign this to the VMs/CTs and add a VLAN tag to their network interface in the PVE GUI.
auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

Then on your VMs/CTs, just enter in the appropriate VLAN tag for the bridge you care about. Or look into SDN so you don't have to remember VLAN tags.

Honestly, you should rethink your IP scheme, too. Nothing about this is easily supernetted. It would make more sense to have the following networks:

SUPERNET: 192.168.0.0/22
SUBNET1: 192.168.0.0/24
SUBNET2: 192.168.1.0/24
SUBNET3: 192.168.2.0/24
SUBNET4: 192.168.3.0/24

If you need more VLANs than that, you could move to a /21 which would have an end subnet of
SUBNET8: 192.168.7.0/24

The reason to supernet is so that if you have another site (friend, family, data center, whatever) and you want a VPN tunnel between the two, you can assign that other site to the next supernet. Picking up from our /21 example:

SITE 2 SUPERNET: 192.168.8.0/21
SUBNET1: 192.168.8.0/24
SUBNET2: 192.168.9.0/24
...
SUBNET8: 192.168.15.0/24

This means there's no overlapping subnets between the sites. Refer to the PVE documentation for BOND config:

https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_bond

EDIT: vmbr0 should _not_ be VLAN aware if you want to enable SDN