Proxmox with Cisco Nexus LACP

filips

Member
May 25, 2021
10
0
21
36
Hello, I trying to connect two switches in VPC mode with proxmox with LACP. I get port status :
Code:
Ethernet1/5 is down (suspended(no LACP PDUs))
the same on the other switch

Code:
Nexus-02# show running-config interface ethernet 1/5

!Command: show running-config interface Ethernet1/5
!Running configuration last done at: Wed Nov 26 18:03:35 2025
!Time: Wed Nov 26 19:02:43 2025

version 9.3(13) Bios:version 05.51

interface Ethernet1/5
  switchport mode trunk
  channel-group 17 mode active
  no shutdown

Nexus-02# show running-config interface po17

!Command: show running-config interface port-channel17
!Running configuration last done at: Wed Nov 26 18:03:35 2025
!Time: Wed Nov 26 19:03:14 2025

version 9.3(13) Bios:version 05.51

interface port-channel17
  switchport mode trunk
  vpc 17

Nexus-02# show vpc 17


vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
17    Po17          down*  success     success               -

On proxmox server :

Code:
auto lo
iface lo inet loopback

iface eno8303 inet manual
iface eno8403 inet manual
iface eno12399np0 inet manual
iface eno12409np1 inet manual
iface ens2f0np0 inet manual
iface ens2f1np1 inet manual

auto ens1f0np0
iface ens1f0np0 inet manual

auto ens1f1np1
iface ens1f1np1 inet manual

# Bond w OVS z LACP
auto bond0
iface bond0 inet manual
    ovs_type OVSBond
    ovs_bridge vmbr1
    ovs_bonds ens1f0np0 ens1f1np1
    ovs_options bond_mode=balance-tcp lacp=active

# Linux bridge do zarządzania
auto vmbr0
iface vmbr0 inet static
    address 10.9.254.185/24
    gateway 10.9.254.1
    bridge-ports eno8303
    bridge-stp off
    bridge-fd 0

# VLAN-aware OVS bridge (trunk)
auto vmbr1
iface vmbr1 inet manual
    ovs_type OVSBridge
    ovs_ports bond0

source /etc/network/interfaces.d/sdn

Code:
root@srv-07:/etc/network# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno8303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether b4:e9:b8:03:5b:2a brd ff:ff:ff:ff:ff:ff
    altname enp99s0f0
    altname enxb4e9b8035b2a
3: eno8403: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:e9:b8:03:5b:2b brd ff:ff:ff:ff:ff:ff
    altname enp99s0f1
    altname enxb4e9b8035b2b
4: eno12399np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 6c:92:cf:5d:28:c0 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0np0
    altname enx6c92cf5d28c0
5: eno12409np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 6c:92:cf:5d:28:c1 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1np1
    altname enx6c92cf5d28c1
6: ens2f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:83:51:23:af:a8 brd ff:ff:ff:ff:ff:ff
    altname enp33s0f0np0
    altname enxb4835123afa8
7: ens2f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:83:51:23:af:a9 brd ff:ff:ff:ff:ff:ff
    altname enp33s0f1np1
    altname enxb4835123afa9
8: ens1f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN mode DEFAULT group default qlen 1000
    link/ether b4:83:51:23:ac:8e brd ff:ff:ff:ff:ff:ff
    altname enp34s0f0np0
    altname enxb4835123ac8e
:2b brd ff:ff:ff:ff:ff:ff
17: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master fwbr101i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1a:43:e5:d2:b1:68 brd ff:ff:ff:ff:ff:ff
18: fwbr101i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 1a:43:e5:d2:b1:68 brd ff:ff:ff:ff:ff:ff
20: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 1e:8a:7c:bc:b0:98 brd ff:ff:ff:ff:ff:ff
21: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether b4:83:51:23:ac:8e brd ff:ff:ff:ff:ff:ff
22: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ea:66:c4:85:f4:c1 brd ff:ff:ff:ff:ff:ff
23: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether b4:e9:b8:03:5b:2a brd ff:ff:ff:ff:ff:ff
24: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master fwbr102i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:e4:d7:1b:c9:7e brd ff:ff:ff:ff:ff:ff
25: fwbr102i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fe:e4:d7:1b:c9:7e brd ff:ff:ff:ff:ff:ff
26: fwln102o0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr102i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 8a:29:30:3b:9b:16 brd ff:ff:ff:ff:ff:ff
root@srv-07:/etc/network#
[/code


Why LACP not working ?
 
Hi,

your bond on Proxmox VE uses ovs_bonds ens1f0np0 ens1f1np1
but the interface configuration a few lines above state
iface ens2f0np0 inet manual
iface ens2f1np1 inet manual
 
Hi,

your bond on Proxmox VE uses ovs_bonds ens1f0np0 ens1f1np1
but the interface configuration a few lines above state
iface ens2f0np0 inet manual
iface ens2f1np1 inet manual

These interfaces belongs to the other network card
 
I pointed there because your posted output for ip link show is missing interface ens1f1np1 completely and ens1f0np0 has NO-CARRIER.
 
I removed bonding and trying to set bridge on only one physical interface with vlans but no luck.
Previous one the same hardware configuration was hyper-v and we had no problems.
Switches are cisco nexus 9k and network cards intel E810-XXVDA2 with the newest drivers.


Current config :

Code:
auto lo
iface lo inet loopback

iface ens1f1np1 inet manual

iface eno12399np0 inet manual

iface eno12409np1 inet manual

iface eno8303 inet manual

iface eno8403 inet manual

iface ens1f0np0 inet manual

iface ens2f0np0 inet manual

iface ens2f1np1 inet manual

iface ens6f0np0 inet manual

iface ens6f1np1 inet manual

auto vmbr0
iface vmbr0 inet static
<------>address 10.9.254.185/24
<------>gateway 10.9.254.1
<------>bridge-ports eno8303
<------>bridge-stp off
<------>bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
<------>bridge-ports ens1f1np1
<------>bridge-stp off
<------>bridge-fd 0
<------>bridge-vlan-aware yes
<------>bridge-vids 2-4094

auto vmbr1.20
iface vmbr1.20 inet manual

auto vmbr1.254
iface vmbr1.254 inet manual

auto vmbr1.116
iface vmbr1.116 inet manual

source /etc/network/interfaces.d/sdn

Virtual machines dosn't receive any ip address from dhcp.
Can someone help me ?
 
Code:
  auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
#Mgmt to 3750

iface eno2 inet manual

auto eno3
iface eno3 inet static
        address 172.31.1.30/24
#CoroSync1

auto eno4
iface eno4 inet static
        address 172.31.2.30/24
#CoroSync2

auto eno49
iface eno49 inet manual
        mtu 9216

auto eno50
iface eno50 inet manual
        mtu 9216

auto ceph
iface ceph inet static
        address 172.16.50.30/24
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        ovs_options tag=950
#CEPH Public

auto bond0
iface bond0 inet manual
        ovs_bonds eno49 eno50
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_mtu 9216
        ovs_options bond_mode=balance-tcp lacp=active
#To Nexus

auto vmbr0
iface vmbr0 inet static
        address 10.60.5.33/24
        gateway 10.60.5.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
#Mgmt network

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 ceph
        ovs_mtu 9216
#VM/LXC networks
 
I removed bonding and trying to set bridge on only one physical interface with vlans but no luck.
Previous one the same hardware configuration was hyper-v and we had no problems.
Switches are cisco nexus 9k and network cards intel E810-XXVDA2 with the newest drivers.


Current config :

Code:
auto lo
iface lo inet loopback

iface ens1f1np1 inet manual

iface eno12399np0 inet manual

iface eno12409np1 inet manual

iface eno8303 inet manual

iface eno8403 inet manual

iface ens1f0np0 inet manual

iface ens2f0np0 inet manual

iface ens2f1np1 inet manual

iface ens6f0np0 inet manual

iface ens6f1np1 inet manual

auto vmbr0
iface vmbr0 inet static
<------>address 10.9.254.185/24
<------>gateway 10.9.254.1
<------>bridge-ports eno8303
<------>bridge-stp off
<------>bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
<------>bridge-ports ens1f1np1
<------>bridge-stp off
<------>bridge-fd 0
<------>bridge-vlan-aware yes
<------>bridge-vids 2-4094

auto vmbr1.20
iface vmbr1.20 inet manual

auto vmbr1.254
iface vmbr1.254 inet manual

auto vmbr1.116
iface vmbr1.116 inet manual

source /etc/network/interfaces.d/sdn

Virtual machines dosn't receive any ip address from dhcp.
Can someone help me ?
vmbr1 has no interfaces.
 
Hello @filips,
have you tried using a normal Linux Bond before?

Example:
Code:
auto ens1f0np0
iface ens1f0np0 inet manual

auto ens1f1np1
iface ens1f1np1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves ens1f0np0 ens1f1np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
 
Last edited: