Setting Up LACP Bond with VLAN Trunk and Bridge

F

firewire10000

Member
Jul 25, 2021
5
1
6
Hi

I'm having a really hard time trying to get the network setup on Proxmox despite looking at all of the forums and tweaks I've been doing.

I have configured a 2Gbps LACP bond on my Netgear GS724T managed switch and is also setup as a VLAN trunk with it's PVID (native VLAN) on 8 so it can talk to the internet. Essentially I would like to be able to run virtual machines on various VLANs configured in Proxmox whilst the server still talks on its VLAN 8. I'd like the virtual machines to talk to the VLAN's DHCP server as though the virtual machine is directly on that subnet as a real machine requesting an IP address.

Please see my diagram below which better explains this:
Proxmox VLAN Bond.png

Below is my current network configuration:

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug eno1
iface eno1 inet manual

allow-hotplug eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
    bond-mode 4
    slaves eno1 eno2
    bond-miimon 100
    bond-downdelay 200
    bond-updelay 200
    bond-lacp-rate 1
    bond-xmit-hash-policy layer2+3

auto bond0.8
iface bond0.8 inet static
    address 192.168.8.122/24
    gateway 192.168.8.1
    dns-nameservers 192.168.8.1

Now the bond does work with a static IP address but for some reason I couldn't get the bond to work on DHCP. To be able to give my virtual machine a network I need to setup a bridge which is were the confusion starts as I also need to set a VLAN.

I've looked at the Proxmox Network Configuration Wiki found here but my virtual machines never get an IP address so it seems my VLANs aren't working.

Any help much appreciated.
 
The Proxmox host itself will not rely on DHCP to get an address. It really likes static configuration.

You need to add a VLAN-aware bridge. Install the package ifupdown2 for that.

Configure the bridge with bond0 as one port. Then you can set the VLAN ID in the configuration of the virtual NIC of the VM.
 
Last edited:
QUOTE="gurubert, post: 406194, member: 31712"]
The Proxmox host itself will not rely on DHCP to get an address. It really likes static configuration.
[/QUOTE]
I was trying to establish a DCHP on a bond interface within Debian 10 itself before I even installed Proxmox. I had a trial of Windows Server 2016 and I did eventually manage to establish a DHCP connection on what Windows calls it a LAG (link aggregation)/team interface and was hoping Linux could do something similar.

QUOTE="gurubert, post: 406194, member: 31712"]
You need to add a VLAN-aware bridge. Install the package ifupdown2 for that.

Configure the bridge with bond0 as one port. Then you can set the VLAN ID in the configuration of the virtual NIC of the VM.
[/QUOTE]

Would the following config be sufficient?

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug eno1
iface eno1 inet manual

allow-hotplug eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
	bond-mode 4
	slaves eno1 eno2
	bond-miimon 100
	bond-downdelay 200
	bond-updelay 200
	bond-lacp-rate 1
	bond-xmit-hash-policy layer2+3
	
auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes

auto vmbr0.8
iface vmbr0.8 inet static
        address 192.168.8.122/24
        gateway 192.168.8.1
        dns-nameservers 192.168.8.1

I take it by setting the VLAN on the VM's NIC settings, the VM then will then communicate with the DHCP for that LAN as though it's directly connecting to the network?
 
I think you cannot use a VLAN tag on a bridge interface. We always use a VLAN tag on the bond and use it as a port in a bridge.

Code: 
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer2+3

auto bond0.4
iface bond0.4 inet manual
#priv-compute

auto bond0.100
iface bond0.100 inet static
    address a.b.c.d/m
#priv-vmotion

auto vmbr0
iface vmbr0 inet static
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 1-3 5-99 101-4094
    bridge-pvid 4063
#all VLANs

auto vmbr4
iface vmbr4 inet static
    address w.x.y.z/n
    gateway w.x.y.z.1
    bridge-ports bond0.4
    bridge-stp off
    bridge-fd 0
#priv-compute

vmbr4 is the management interface for the Proxmox node. bond0.100 is for migration. vmbr0 is a VLAN aware bridge that can be used with VMs. The VLAN tag is then specified in the VM network configuration. bridge-vids and the bridge-pvid have to be adjusted to your environment.
 
Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        bond-downdelay 200
        bond-updelay 200
        band-lacp-rate 1

auto bond0.8
iface bond0.8 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1-4094
        bridge-pvid 8

auto vmbr8
iface vmbr8 inet static
        address 192.168.8.122/24
        gateway 192.168.8.1
        bridge-ports bond0.8
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        dns-nameservers 192.168.8.1

I can say I have finally got it working. I can set the VLAN e.g. VLAN 5 on the VM and it communicates with the DHCP server on that VLAN and dynamically acquires an IP address.

The only thing I can't get my head around is why I need two bridges? Why couldn't I assign a static IP address onto the VLAN-aware bridge like shown in the below code? bond0 is the trunk containing all of the VLANs, the bridge is VLAN-aware and I've set the PVID to 8 so that the bridge gets untagged packets allowing the Proxmox management interface to work.

Code: 
auto vmbr0
iface vmbr0 inet static
        address 192.168.8.122/24
        gateway 192.168.8.1
        dns-nameservers 192.168.8.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1-4094
        bridge-pvid 8
 
I'm trying to do something similar, however I'm getting the following error when trying to apply the configuration:
Code: 
root@n01:~# ifreload -a
error: netlink: bond0.611: cannot create vlan bond0.611 611: operation failed with 'Operation not supported' (95)

As per the last example on the Wiki, I'm trying to add a VLAN to an active bond interface. Here's our config:

Code: 
auto lo
iface lo inet loopback

auto ens4f1
iface ens4f1 inet manual

iface ens3f0 inet manual

auto ens3f1
iface ens3f1 inet manual

iface ens4f0 inet manual

iface ens2f0 inet manual

iface ens2f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves ens3f1 ens4f1
        bond-miimon 100
        bond-mode 802.3ad

iface bond0.611 inet manual

auto vmbr0v611
iface vmbr0v611 inet static
    address xx.xx.11.1/24
    # gateway # comment it if you want to have any gateway
    bridge-ports bond0.611
    bridge-stp off
    bridge-fd 0

auto vmbr0
iface vmbr0 inet static
        address xx.xx.xx.31/24
        gateway xx.xx.xx.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Any clues?
 
Sincere said:
I'm trying to do something similar, however I'm getting the following error when trying to apply the configuration:
Code: 
root@n01:~# ifreload -a
error: netlink: bond0.611: cannot create vlan bond0.611 611: operation failed with 'Operation not supported' (95)

As per the last example on the Wiki, I'm trying to add a VLAN to an active bond interface.
Click to expand...

I suspect the Wiki is simply incorrect. I have no sense one can add a VLAN directly to a bond. Maybe it has changed with ProxMox 8???

What DOES work (I've not repeated exactly yours, but this is like mine using my card, vmbr and vlan numbers):
Code: 
...
auto bond0
iface bond0 inet manual
        bond-slaves ens3f1 ens4f1
        bond-miimon 100
        bond-mode 802.3ad

auto vmbr0
iface vmbr0 inet static
        address xx.xx.xx.31/24
        gateway xx.xx.xx.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.611
iface vmbr0.611 inet static
    address xx.xx.11.1/24
    # gateway # comment it if you want to have any gateway

AND:
  • For my switch at the other end, my bond0 def includes bond-xmit-hash-policy layer2 (because Netgear smart managed switches only support that for LACP)
  • I had to add bridge-pvid NN to vmbr0 -- to ensure correct VLANS are tagged and the PVID is not
NOTE that I can add vmbr0 to any VM and give it a VLAN tag to include that VM on any particular VM. No need to define that in the host.
 
Last edited:
If you're to do both link aggregation and vlan tags, I've found it *far* easy to use Open-vSwitch.

Code: 
apt update
apt install openvswitch-switch openvswitch-common

Here's an example showing two ethernet interfaces, bundled into an 802.3ad/LACP bundle, with tagged VLAN's, including proxmox management on VLAN 8

Code: 
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
    mtu 1500
    ovs_mtu 1500

auto eno2
iface eno2 inet manual
        mtu 1500
        ovs_mtu 1500

auto bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eno1 eno2
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
        ovs_options tag=1 vlan_mode=native-untagged
        ovs_mtu 1500
        pre-up ifconfig eno1 mtu 1500
        pre-up ifconfig eno2 mtu 1500

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0
        ovs_mtu 1500

auto vlan8
iface vlan8 inet static
        address 10.10.10.111/24
        gateway 10.10.10.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=8
    ovs_mtu 1500

auto vlan500
iface vlan500 inet static
        address 192.168.50.111/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=500
    ovs_mtu 1500

The bonus to doing it this way -- in the VM config, you choose "vmbr0" as the network interface, then choose the VLAN Tag you want. Nothing else needs added to this config file.

Screen Shot 2024-03-12 at 4.51.55 PM.png
 
Last edited:
dlasher said:
If you're to do both link aggregation and vlan tags, I've found it *far* easy to use Open-vSwitch.

Code: 
apt update
apt install openvswitch-switch openvswitch-common

Here's an example showing two ethernet interfaces, bundled into an 802.3ad/LACP bundle, with tagged VLAN's, including proxmox management on VLAN 8

Code: 
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
    mtu 1500
    ovs_mtu 1500

auto eno2
iface eno2 inet manual
        mtu 1500
        ovs_mtu 1500

auto bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eno1 eno2
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
        ovs_options tag=1 vlan_mode=native-untagged
        ovs_mtu 1500
        pre-up ifconfig eno1 mtu 1500
        pre-up ifconfig eno2 mtu 1500

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0
        ovs_mtu 1500

auto vlan8
iface vlan8 inet static
        address 10.10.10.111/24
        gateway 10.10.10.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=8
    ovs_mtu 1500

auto vlan500
iface vlan500 inet static
        address 192.168.50.111/24
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=500
    ovs_mtu 1500

The bonus to doing it this way -- in the VM config, you choose "vmbr0" as the network interface, then choose the VLAN Tag you want. Nothing else needs added to this config file.

View attachment 64560
Click to expand...

Hi, thanks for the sharing.

I one more thing to clarify
So i dont need to add vlan tag 10 in bond or bridge config or create vlan10 ?
Is this part of the config that makes any vlan in vm define can pass
“ovs_options tag=1 vlan_mode=native-untagged”

And thank you again.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom