[SOLVED] possible bug proxmox 7, active-backup bond with VLAN aware bridge causes no internet
- Thread starter se_marc
- Start date
Nothing should be left unchecked. But to be honest, I'm done with suggestions. 
If everything works until the point where you enable vlan awareness, I have no clue where to look for possible errors.
Config looks good, so yes, maybe a firmware issue.
If everything works until the point where you enable vlan awareness, I have no clue where to look for possible errors.
Config looks good, so yes, maybe a firmware issue.
thanks for your time and effort!Nothing should be left unchecked. But to be honest, I'm done with suggestions.
If everything works until the point where you enable vlan awareness, I have no clue where to look for possible errors.
Config looks good, so yes, maybe a firmware issue.
Set the MAC address of the bridge to the same as the bond with the `hwaddress` option (see `man ifupdown-addons-interfaces`).
See also the following thread on the pve-user list: https://lists.proxmox.com/pipermail/pve-user/2021-July/172651.html
See also the following thread on the pve-user list: https://lists.proxmox.com/pipermail/pve-user/2021-July/172651.html
Last edited:
if you enable vlan-aware option, the ip address on vmbr0 will be on default vlan 1. (so you need to support vlan=1 tagging on your physical switch as default vlan too)
The native vlan id is pretty irrelevant, I would say. I have vlan 99 as native vlan on my trunks and everything works out of the box.
Its working!
NOTE: making any changes to networking will wipe this config out as of version 7.0-9
Here's what I did:
NOTE: making any changes to networking will wipe this config out as of version 7.0-9
Here's what I did:
- To get the permanent mac addresses for each physical NIC ,and the mac addresses of the bridge and bond (note, the bridge and bond are the same), I ran
ip -d addr |grep 'vmbr0\|eno\|ether'
Code:root@proxmox-6:~# ip -d addr |grep 'vmbr0\|eno\|ether' 2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff permaddr b8:2a:72:d7:1e:c8 promiscuity 0 minmtu 46 maxmtu 9600 3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff permaddr b8:2a:72:d7:1e:ca promiscuity 0 minmtu 46 maxmtu 9600 4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff permaddr b8:2a:72:d7:1e:cc promiscuity 0 minmtu 46 maxmtu 9600 5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff permaddr b8:2a:72:d7:1e:ce promiscuity 0 minmtu 46 maxmtu 9600 22: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bond mode active-backup active_slave eno1 miimon 100 updelay 0 downdelay 0 peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none arp_all_targets any primary eno1 primary_reselect always fail_over_mac none xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select stable tlb_dynamic_lb 1 23: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:7c:1b:7c:c9:e0 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 inet 10.10.3.18/24 scope global vmbr0 root@proxmox-6:~#
nano /etc/network/interfaces
I removedauto eno1,auto eno2,auto eno3,auto eno4
I addedhwaddress <mac address>to each of the interfaces
-
Code:
root@proxmox-6:~# cat /etc/network/interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno2 inet manual hwaddress b8:2a:72:d7:1e:ca iface eno1 inet manual hwaddress b8:2a:72:d7:1e:c8 iface eno3 inet manual hwaddress b8:2a:72:d7:1e:cc iface eno4 inet manual hwaddress b8:2a:72:d7:1e:ce auto bond0 iface bond0 inet manual bond-slaves eno1 eno2 eno3 eno4 hwaddress fa:7c:1b:7c:c9:e0 bond-miimon 100 bond-mode active-backup bond-primary eno1 auto vmbr0 iface vmbr0 inet static address 10.10.3.18/24 gateway 10.10.3.1 hwaddress fa:7c:1b:7c:c9:e0 bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 root@proxmox-6:~#
- restart networking by running
ifdown -a; ifup -a
I believe I've tried everything in this thread regarding this issue, still no luck. ifupdown has been removed and purged. ifupdown2 has been reinstalled.
The messages I get upon trying to bring up (ifup) the vmbr0 interface:
/proc/net/bonding/ appears to have no files in it.
My section for my bond in /etc/network/interfaces:
I've added bonding and bridge to /etc/modules and ran update-initramfs -u.
As far as I know none of my configuration has changed. I did try to add the hwaddress info per this and a couple of other threads but have not had any luck. This is my current interfaces file excluding the USB NIC I have setup just to get the system online:
Running and 'ip ad' shows the vmbrX devices, but show as "<NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000".
The bond device does not appear at all.
The messages I get upon trying to bring up (ifup) the vmbr0 interface:
Code:
error: vmbr0: netlink: cannot create bridge or set attributes: operation failed with 'Numerical result out of range' (34)
warning: bond0: interface not recognized - please check interface configuration/proc/net/bonding/ appears to have no files in it.
My section for my bond in /etc/network/interfaces:
Code:
auto bond0
iface bond0 inet manual
hwaddress 66:88:e0:41:00:f6
slaves enp2s0 enp3s0
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4I've added bonding and bridge to /etc/modules and ran update-initramfs -u.
As far as I know none of my configuration has changed. I did try to add the hwaddress info per this and a couple of other threads but have not had any luck. This is my current interfaces file excluding the USB NIC I have setup just to get the system online:
Code:
auto lo
iface lo inet loopback
#auto enp2s0
iface enp2s0 inet manual
hwaddress 68:05:ca:1e:8a:e2
#auto enp3s0
iface enp3s0 inet manual
hwaddress 68:05:ca:0b:ab:5b
auto bond0
iface bond0 inet manual
hwaddress 66:88:e0:41:00:f6
slaves enp2s0 enp3s0
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
hwaddress 66:88:e0:41:00:f6
address x.x.x.x
netmask x.x.x.x
gateway x.x.x.x
broadcast x.x.x.x
bridge_ports bond0
bridge_stp on
bridge_fd 0
network x.x.x.x
dns-nameservers x.x.x.x
dns-nameservers x.x.x.x
dns-domain xxx.xxx
dns-search xxx.xxx
auto vmbr1
iface vmbr1 inet static
hwaddress aa:47:37:38:fb:7e
address x.x.x.x
netmask x.x.x.x
bridge_ports none
bridge_stp on
bridge_fd 0Running and 'ip ad' shows the vmbrX devices, but show as "<NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000".
The bond device does not appear at all.
What I did was just removing the auto eno1 auto eno2 etc, basically remove the auto command for each hardware interface. I did not even add the hwaddress command for the bond.
There is a thread on this problem
https://forum.proxmox.com/threads/u...ond-lacp-interface-not-working-anymore.92060/
Upgraded my R630's as well today, ran into the same problem, solved it (for now) by removing the two lines regarding VLAN:
It's not a big issue since I haven't been using VLANs yet—but I'd love to know what's wrong there.
PS: No bonds on vmbr0, not firewalled.
Code:
bridge-vlan-aware yes
bridge-vids 2-4094It's not a big issue since I haven't been using VLANs yet—but I'd love to know what's wrong there.
PS: No bonds on vmbr0, not firewalled.
Last edited:
I would think commenting out the line would serve the same purpose. Should removing the "auto" lines be done for only the interfaces that are part of the bond? Again I've tried this several ways, and I'm running off a USB NIC at this point just to have connectivity.
So, while writing this, for shits and giggles, I removed ifupdown2 and reinstalled ifupdown. Still no dice. Then I found this:
https://forum.proxmox.com/threads/ifdownup2-breaks-network-on-nodes.64463/
OVS....'member berries....did I install that at one point? No, pretty sure I didn't...
https://pve.proxmox.com/wiki/Open_vSwitch
Didn't think there was openvswitch installed, but went ahead and ran and apt-get remove on it.....turned out it was...removed/purged openvswitch, rebooted and BAM, I've got pings to my VMBRx interfaces and can finally get into the web interface.
I found problem if set bridge-vids 2-4094 but if I keep line and set the range 1-4094. Everything goes fine
We also did update from 6.4 to 7.0 and the bonding doesn't work.
When the primary uplink goes down the server doesn't fall back to the second uplink.
I changed the primary uplink to the second one and disabled the first uplink everything works fine.
auto lo
iface lo inet loopback
iface enp68s0f0 inet manual
iface enp68s0f1 inet manual
auto enp161s0f0
iface enp161s0f0 inet manual
auto enp161s0f1
iface enp161s0f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp161s0f0 enp161s0f1
bond-miimon 100
bond-mode active-backup
bond-primary enp161s0f0
auto vmbr0
iface vmbr0 inet static
address 45.151.xx.xx/24
gateway 45.151.xx.xx
bridge-ports bond0
bridge-stp off
bridge-fd 0
iface vmbr0 inet6 static
address xxxx:xxxx:11:6::1/48
gateway xxxx:xxxx:11::1
---
root@kvm110:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.13.19-1-pve
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp161s0f0 (primary_reselect always)
Currently Active Slave: enp161s0f0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
Slave Interface: enp161s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:38:ce:c4
Slave queue ID: 0
Slave Interface: enp161s0f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:38:ce:c5
Slave queue ID: 0
When the primary uplink goes down the server doesn't fall back to the second uplink.
I changed the primary uplink to the second one and disabled the first uplink everything works fine.
auto lo
iface lo inet loopback
iface enp68s0f0 inet manual
iface enp68s0f1 inet manual
auto enp161s0f0
iface enp161s0f0 inet manual
auto enp161s0f1
iface enp161s0f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp161s0f0 enp161s0f1
bond-miimon 100
bond-mode active-backup
bond-primary enp161s0f0
auto vmbr0
iface vmbr0 inet static
address 45.151.xx.xx/24
gateway 45.151.xx.xx
bridge-ports bond0
bridge-stp off
bridge-fd 0
iface vmbr0 inet6 static
address xxxx:xxxx:11:6::1/48
gateway xxxx:xxxx:11::1
---
root@kvm110:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.13.19-1-pve
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp161s0f0 (primary_reselect always)
Currently Active Slave: enp161s0f0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
Slave Interface: enp161s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:38:ce:c4
Slave queue ID: 0
Slave Interface: enp161s0f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 3c:ec:ef:38:ce:c5
Slave queue ID: 0
Thanks, removing auto on the HW interfaces without adding the mac addresses worked for me, just needed to do an "ifdown -a" and "ifup -a"!