2 nearly identical bridges. One works, the other doesn't. Why?

garyd9

Member
Nov 10, 2023
44
3
8
(Edit: before replying, please read posts #4 and #5 below. I found a workaround...)

My interfaces file is below, but the problem I'm encountering is that any VM or LXC container that's assigned to vmbr0 has ZERO network connectivity (outside the vmbr0 bridge.) Any VM or LXC container that's assigned to vmbr1 works fine. All other config options (vlan id) are the same.

Both sets have a bond that's assigned to a bridge, and an extra vlan on the bond that gives an IP to the host. _BOTH_ IP addresses assigned to the host work. (I can connect to the pve config page from either vlan31 using IP 192.168.31.98, or from vlan 6 using the IP address assigned by dhcp.)

bond1 -> vmbr1 (with bond1.6 giving an IP to the host properly. In my testing, that IP was 192.168.6.11)

bond0 -> vmrb1 (with bond0.31 giving an IP to the host properly. Static IP here.)

Yet, as mentioned, any CT (or VM) assigned to use vmbr0 with any vlan specified will have ZERO network connectivity outside of that specific bridge. Any CT (or VM) assigned to use vmbr1 with any vlan specified works perfectly.

On the network switch, all the ports attached to the NICs are properly configured as TRUNK ports

Please help! (vmbr1 is only a test interface using a USB NIC... not something I want to use long-term)

Here's the interfaces file:


Code:
auto lo
iface lo inet loopback

iface wlo1 inet manual

auto enp129s0
iface enp129s0 inet manual

auto enp130s0
iface enp130s0 inet manual

auto enx3c18a0d526c9
iface enx3c18a0d526c9 inet manual

auto bond1
iface bond1 inet manual
        bond-slaves enx3c18a0d526c9
        bond-miimon 100
        bond-mode balance-rr

auto bond1.6
iface bond1.6 inet dhcp

auto bond0
iface bond0 inet manual
        bond-slaves enp129s0 enp130s0
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        hwaddress ether 00:e0:4c:88:31:00

auto bond0.31
iface bond0.31 inet static
        address 192.168.31.98/24
        gateway 192.168.31.1
#for pve host access only
 
auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1 6 7 31
        bridge-allow-untagged no

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 31 6 7 1
        bridge-allow-untagged no
 
Last edited:
For completeness, here's one of the container configs. This works. If I simply replace "vmbr1" with "vmbr0", it fails.

Code:
arch: amd64
cores: 2
features: nesting=1
hostname: testmacs
memory: 512
net0: name=eth0,bridge=vmbr1,ip=dhcp,ip6=auto,tag=31,type=veth
ostype: debian
rootfs: local-zfs:subvol-500-disk-0,size=8G
swap: 512
unprivileged: 1

Also the result of "ip a" on the host:
Code:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp129s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:e0:4c:88:31:00 brd ff:ff:ff:ff:ff:ff permaddr 00:e0:4c:88:31:98
3: enp130s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:e0:4c:88:31:00 brd ff:ff:ff:ff:ff:ff permaddr 00:e0:4c:88:31:99
4: enx3c18a0d526c9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond1 state UP group default qlen 1000
    link/ether 3c:18:a0:d5:26:c9 brd ff:ff:ff:ff:ff:ff
5: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 40:1c:83:cf:6a:66 brd ff:ff:ff:ff:ff:ff
    altname wlp0s20f3
6: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether 3c:18:a0:d5:26:c9 brd ff:ff:ff:ff:ff:ff
7: bond1.6@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:18:a0:d5:26:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.11/24 brd 192.168.6.255 scope global dynamic bond1.6
       valid_lft 41541sec preferred_lft 41541sec
    inet6 fe80::3e18:a0ff:fed5:26c9/64 scope link 
       valid_lft forever preferred_lft forever
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:18:a0:d5:26:c9 brd ff:ff:ff:ff:ff:ff
    inet6 2601:549:c101:79c0:3e18:a0ff:fed5:26c9/64 scope global dynamic mngtmpaddr 
       valid_lft 86371sec preferred_lft 86371sec
    inet6 fe80::3e18:a0ff:fed5:26c9/64 scope link 
       valid_lft forever preferred_lft forever
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 00:e0:4c:88:31:00 brd ff:ff:ff:ff:ff:ff
10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:e0:4c:88:31:00 brd ff:ff:ff:ff:ff:ff
    inet6 2601:549:c101:79c0:2e0:4cff:fe88:3100/64 scope global dynamic mngtmpaddr 
       valid_lft 86371sec preferred_lft 86371sec
    inet6 fe80::2e0:4cff:fe88:3100/64 scope link 
       valid_lft forever preferred_lft forever
12: veth102i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:35:44:fc:60:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0
13: bond0.31@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:e0:4c:88:31:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.31.98/24 scope global bond0.31
       valid_lft forever preferred_lft forever
    inet6 2601:549:c101:79c1:2e0:4cff:fe88:3100/64 scope global dynamic mngtmpaddr 
       valid_lft 86260sec preferred_lft 86260sec
    inet6 fe80::2e0:4cff:fe88:3100/64 scope link 
       valid_lft forever preferred_lft forever
14: veth201i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether fe:9a:4d:b1:a7:47 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 26:3d:74:9c:77:49 brd ff:ff:ff:ff:ff:ff
21: veth500i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:69:b1:1a:f2:ef brd ff:ff:ff:ff:ff:ff link-netnsid 2
 
Last edited:
edited the top post to specify that any VM/CT on vmbr0 has no connectivity outside the vmbr0. If two CT's are both on vmbr0, they can talk to each other (assuming they get IP addresses) but neither can communicate outside the bridge.
 
... and more info: It's only vlan 31 that vmbr0 isn't allowing me to access. That's odd. If I set up a CT to use vlan 7, it works fine.
 
...and because I just can't ask a question and stop.. I've kept playing around. If I delete the bond0.31 and replace it with the snippet below, things start to work as I expect. My best guess (and it's only a WAG) is that "bond0.31" was somehow blocking vlan31 from moving over the vmbr0.

Code:
auto vmbr0.31
iface vmbr0.31 inet static
        address 192.168.31.98/24
        gateway 192.168.31.1
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!