Multi-VLAN - CSR1000v + Trunk and bonding dilemma

[hanoon]

Hello, I have been struggling for a while to getting this configuration to work, Searched forums and compared multiple times with existing configs without luck

2 x Cisco switches with an ether-channel trunk for HSRP
Proxmox hosts connected to port 5 on each switch eth0 to sw1 and eth1 to sw2 respectively

VLAN:2 (management)
VLAN 3
VLAN 4
VLAN 5

My goal:
1- Must-Have redundancy in case of switch failure for VLANs 2,3,4
2- CSR1000 need to be only on one interface eth1 with access to the 3 VLANs 2,3,5

Here are my current configurations:


auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth1.2
iface eth1.2 inet manual

auto eth1.3
iface eth1.3 inet manual

auto eth1.5
iface eth1.5 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eth0 eth1
bond-miimon 100
bond-mode active-backup
bond_downdelay 200
bond_updelay 200

iface bond0.2 inet manual

iface bond0.3 inet manual

iface bond0.4 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.100.7/24
gateway 192.168.100.3
bridge-ports bond0.2
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0.3
bridge-stp off
bridge-fd 0

auto vmbr2
iface vmbr2 inet manual
bridge-ports bond0.4
bridge-stp off
bridge-fd 0

#CSR bridges
auto vmbr4
iface vmbr4 inet manual
bridge-ports eth1.2
bridge-stp off
bridge-fd 0

auto vmbr5
iface vmbr5 inet manual
bridge-ports eth1.5
bridge-stp off
bridge-fd 0

auto vmbr6
iface vmbr6 inet manual
bridge-ports eth1.3
bridge-stp off
bridge-fd 0


SW1 and SW2 port configurations

interface GigabitEthernet1/0/5
switchport access vlan 2
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 2,3,4,5,6
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk

==================================
When I restart networking on PX i get this error on the console :

bond0: option mode : unable to set cause the bond device is up
1340.215262 bond0: (slave eth1): Invalid new link 3 on slave
==================================

This setup is accepted however CSR is not getting traffic passed properly - is the proper recommended setup? Any advice would be appreciated.

Thanks

 

[spirit]

I dont think you can't add vlan on eth1.X , if eth1 is member of a bond.

I don't known if if it's a linux kernel limitation, or simply because of ordering on nic handle by ifupdown from the configuration.
Maybe can you try to install ifupdown2 package (apt install ifupdown2), and send result of "ifreload -a -d".
Maybe it'll handle it correctly.

How do you have failover of eth1 for your CSR, if you don't use bonding for it ?

 

[hanoon]

CSR doesn't need redundancy - I have another router on the other switch for that purpose - however having it on a bond would route traffic over the ether-channel.

I tried ifupdonw2 and the network doesn't come up so I uninstalled it.

What is the proper configuration for my setup?

 

[spirit]

What is the proper configuration for my setup?

I think you need a third link for this traffic. I'm really not sure you can use directly an ethX if it's enslaved in a bond.

Note that you could use eth1 as primary for your bond, so traffic should go directly to your csr on the same swith, until a failover.

 

[hanoon]

eth1 is the backup router link - so the eth0 should be the primary - Also when we connect CSR on the bridge vmbr0 - SW2 learns the mac address from CSR over the etherchannel and not to the directly connected layer 2 port.

Compared to ESXi we are able to create as many port groups as we need with specific VLANs and assign to VMs - There must be a way to properly get that on our case

 

[spirit]

I think you could create a lot of bond with eth0/1.X pair

lke

auto eth0.3
iface eth0.3 inet manual

auto eth1.3
iface eth1.3 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eth0.3 eth1.3
...

auto eth0.4
iface eth0.4 inet manual

auto eth1.4
iface eth1.4 inet manual

auto bond1
iface bond1 inet manual
bond-slaves eth0.4 eth1.4
...

 

[hanoon]

Can the same interface be used in multiple bonds?

For example

auto eth0.3
iface eth0.3 inet manual

auto eth1.3
iface eth1.3 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eth0.3 eth1.3

auto bond1
iface bond1 inet manual
bond-slaves eth1.3

If yes, Do i need to repeat the VLAN interface initialization? like this

auto eth1.3
iface eth1.3 inet manual

auto eth1.3
iface eth1.3 inet manual

 

[hanoon]

Basically I been trying and spending hours on this setup and can't get it to work, Any help here would be appreciated.

Note: My host management network is on VLAN 2 - switches to host are trunk ports... without native VLANs

The problem is I want a VM with access to either a trunk or specific VLAN bridge on one single eth1 interface then the other VMs should be on a bridge attached to the bond interface.