Setting up bridge, bonding and vlan

[spinner]

Hi.

Still very new to Proxmox I'm trying to get the network setup correctly before getting into the storage part.

I have a simple LACP (linux) bonding setup on two NICs (bond0), and then added two Linux Bridges on top of that bond.

The bridge that operates on the untagged works just fine (port/slave: bond0)

But the bridge that I would like to operate on vlan 50 isn't working and it's probably due to a misconfiguration.

I read somewhere that you can use vlans by just dotting the vlan after the interface.

So on the bridge with vlan, I set port/slave to bond0.50.

Should this be working?

Also, when are you suppose to use the vlan aware option?

 

[Richard]

Add the following lines to your /etc/network/interfaces:

auto bond0.50
iface bond0.50 inet manual
       vlan-raw-device bond0

 

[spinner]

Thanks for your reply.

I actually got this to work on another setup I tried.

What seems to cause me the most problems is that I have 4 NICs that I would like to setup with 2 bonds.

Apparently thats not supported out of the box, and I haven't successfully been able to configure 2 bonds without problems.

As soon as I setup the second bond, one or both of them will start acting weird, where a bond will work if one of the cables are in, but as soon as both are in, the link is down.

I'd really like this Proxmox setup to work, but have to admit that there's a long way from the experience you get with ie. ESXi and I'm beginning to lose patience :/

 

[Richard]

As soon as I setup the second bond, one or both of them will start acting weird, where a bond will work if one of the cables are in, but as soon as both are in, the link is down.

In order to verify this I created a setup with 4 bonds all together, see the relevant part of /etc/network/interfaces

auto bond1
iface bond1 inet manual
        slaves eth6 eth7
        bond_miimon 100
        bond_mode 4

auto bond1.112
iface bond1.112 inet static
        vlan-raw-device bond1
        address 192.168.112.112
        netmask 255.255.255.0

auto bond1.95
iface bond1.95 inet manual
        vlan-raw-device bond1

auto bond2
iface bond2 inet static
        slaves eth8 eth9
        bond_miimon 100
        bond_mode 4
        address 192.168.66.1
        netmask 255.255.255.0

auto bond3
iface bond3 inet static
        slaves eth10 eth11
        bond_miimon 100
        bond_mode 4
        address 192.168.67.1
        netmask 255.255.255.0

auto vmbr10
iface vmbr10 inet static
        address  192.168.95.131
        netmask  255.255.255.0  
        bridge_ports bond1.95
        bridge_stp off
        bridge_fd 0

Works fine at all 3 bonds - as you can see I made some variant: with/without vlan and bridge.

If really one bond stops working when the other is active: consider whether there is a problem in the NIC card (you told all 4 interfaces are at the same card) and try to use more NIC cards.

I'd really like this Proxmox setup to work, but have to admit that there's a long way from the experience you get with ie. ESXi and I'm beginning to lose patience :/

The necessity of editing /etc/network/interfaces manually is a possible source of mistakes. ESXi has possibly (I am not so familiar with it) GUI support for such a setup ....[/CODE]

 

[superawesomeusername]

I spent a lot of time trying to get a "LACP bond + vlan + bridge" set up running using native Linux bridges, but had several issues when guest VMs shared the same VLAN as Proxmox.

I solved this using OpenVSwitch instead. In the example below, my vmbr0 is set up for VMs (as usual) and I am able to select the VLAN using the Proxmox GUI (even the same VLAN used by the Proxmox)

Step 1. Ensure your switch has a proper LACP Port-Channel set up. Place the port-channel into trunk mode and enable the proper VLANs. For Cisco Nexus switches, that would look like this:

interface port-channel3
  description kvm1.san
  switchport mode trunk
  switchport trunk allowed vlan 5,9
  spanning-tree port type edge trunk

interface Ethernet1/3
  description kvm1.san
  switchport mode trunk
  switchport trunk allowed vlan 5,9
  spanning-tree port type edge trunk
  channel-group 3 mode active

interface Ethernet1/4
  description kvm1.san
  switchport mode trunk
  switchport trunk allowed vlan 5,9
  spanning-tree port type edge trunk
  channel-group 3 mode active

Step 2. Install OpenVSwitch on Proxmox (you need to have basic networking set up, or you could mount a disk and install the deb that way).
apt install openvswitch-switch

Step 3. Update your /etc/network/interfaces file and reboot

auto lo
iface lo inet loopback


allow-vmbr0 bond0
iface bond0 inet manual
    ovs_bridge vmbr0
    ovs_type OVSBond
    ovs_bonds eth0 eth1
    ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
    mtu 9000


auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
    ovs_type OVSBridge
    ovs_ports bond0 vlan5 vlan9


allow-vmbr0 vlan5
iface vlan5 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=5
    ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
    address 10.1.5.22
    netmask 255.255.255.0
    gateway 10.1.5.1
    mtu 1500


allow-vmbr0 vlan9
iface vlan9 inet static
    ovs_type OVSIntPort
    ovs_bridge vmbr0
    ovs_options tag=9
    ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
    address 10.1.9.22
    netmask 255.255.255.0
    mtu 1500

This example with have your Proxmox node connected to VLAN 5 and VLAN 9 using their respect IP addresses (above) as well as provide a vmbr0 for Guest VMs to access a VLAN of their choice (just ensure it is trunked up to Proxmox in your switch config).

Another bond could be easily added to this set up