[SOLVED] possible bug proxmox 7, active-backup bond with VLAN aware bridge causes no internet

[se_marc]

------------
UPDATE:
------------

On the working 6.4 server, i used https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 to update to 7.0-9 and networking stopped working. this seems like a bug


----------------
ORIGINAL:
----------------


hey all,

I did a fresh install of proxmox 7 on a dell r630. below is my interfaces config. when i try to make vmbr0 VLAN aware, it causes the host to lose networking. I read the documentation and couldn't find any limitations for making a bridge VLAN aware to an active-backup bond. I flashed 6.4 onto the r630 and everything works as intended - I am able to create an active-backup bond and then create a VLAN aware linux bridge.

i am selecting the option to make the bridge VLAN aware in the GUI and it adds the following lines to vmbr0

    bridge-vlan-aware yes
    bridge-vids 2-4094

can anyone please provide any insight?

thanks!

root@proxmox-5:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno3
iface eno3 inet manual

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto eno4
iface eno4 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2 eno3 eno4
    bond-miimon 100
    bond-mode active-backup
    bond-primary eno1

auto vmbr0
iface vmbr0 inet static
    address 10.10.3.17/24
    gateway 10.10.3.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
root@proxmox-5:~#

here is the working config on proxmox 6.4-4 (also 6.4-13)

root@proxmox-6:/# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno2
iface eno2 inet manual

auto eno1
iface eno1 inet manual

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2 eno3 eno4
        bond-miimon 100
        bond-mode active-backup
        bond-primary eno1

auto vmbr0
iface vmbr0 inet static
        address 10.10.3.18/24
        gateway 10.10.3.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

root@proxmox-6:/#

 

[spirit]

do you use ifupdown2 ?

 

[se_marc]

do you use ifupdown2 ?

yes, i also rebooted the server

 

[ph0x]

Try removing all lines beginning with auto eno.

 

[se_marc]

Try removing all lines beginning with auto eno.

i commented them all out (auto eno1 through auto eno4), then did ifdown -a; ifup -a

still unable to ping 8.8.8.8

 

[ph0x]

What does cat /proc/net/bonding/bond0 say?

 

[se_marc]

What does cat /proc/net/bonding/bond0 say?

had to attach the file, image is to large in size

one moment

 

[se_marc]

What does cat /proc/net/bonding/bond0 say?

https://imgur.com/eDatW17

 

[ph0x]

Okay, the bond is up.
What do ip a and ip r say?

 

[se_marc]

Okay, the bond is up.
What do ip a and ip r say?

https://imgur.com/0nEGSRI

 

[ph0x]

Sorry to tell you, but this looks promising.
Can you ping 10.10.3.1?

 

[se_marc]

Sorry to tell you, but this looks promising.
Can you ping 10.10.3.1?

I cannot ping 10.10.3.1

also, this may or may not matter but I noticed a difference in Ethernet Channel Bonding Driver from 6.4 to 7.0

6.4 is using Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
7.0 is using Ethernet Channel Bonding Driver: v5.11.22-1-pve

let me see if i can save tcpdump to a file and retrieve it. not being able to ping the gateway is gross

 

[ph0x]

Or a firewall issue.

 

[t.lamprecht]

What environment are those servers in? Firewalled? I ask due to the change in how the MAC for virtual network devices (bridge and bond is set): https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Linux_Bridge_MAC-Address_Change

 

[se_marc]

What environment are those servers in? Firewalled? I ask due to the change in how the MAC for virtual network devices (bridge and bond is set): https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0#Linux_Bridge_MAC-Address_Change

pfsense > unifi switch (no aggregation on the 4 ports set due to this being active-backup) > proxmox server

I don't have any special pfsense rules set or haven't made any changes when upgrading from 6.4 to 7.0.

 

[ph0x]

Any MAC access rules on the switch? Or port security enabled? Or a static MAC-IP-entry in pfSense?

 

[se_marc]

Any MAC access rules on the switch? Or port security enabled? Or a static MAC-IP-entry in pfSense?

the MAC allowed list is blank which should mean anything is allowed.

LLDP MED and STP are enabled on the 4 switch ports which the proxmox server is connected to.

 

[se_marc]

I may have not made this too clear but when I remove VLAN Aware from vmbr0, it works without an issue on 7.0. Also, the proxmox server is NOT any any tagged network, its on the primary LAN.

 

[ph0x]

Just to rule that out I would still define a static LAG for those four ports in the switch, even if they are active-backup. Just to be sure.

 

[se_marc]

Just to rule that out I would still define a static LAG for those four ports in the switch, even if they are active-backup. Just to be sure.

so i created a LAG on the switch on the 4 ports.
next, i did ifdown -a; ifup -a on proxmox
still unable to ping.

i changed bond0 to bond-mode 802.3ad, ifdown/up again. - still unable to ping.

i removed bridge-vlan-aware yes and bridge-vids 1-4094 from vmbr0 - i AM able to ping.

however, if i remove the LAG from the switch and change bond0 mode back to active-backup, leave off vlan aware, i can still ping.