Network instability on Proxmox Infrastructure in HA X Mikrotik

[babine]

Hello,

I'm experiencing strange behavior on my Proxmox infrastructure configured with 3 hosts and 4 Mikrotik switches: 2 dedicated to storage and 2 for VM networking. Each pair of switches is linked via a bonding.

In normal operation, everything runs smoothly. However, during high-availability (HA) tests involving the rebooting of one or more hosts, a server may lose its network connection altogether. Even after several reboots, connectivity does not return, unless I manually disable and then re-enable a bonding port at the switch.

This problem has occurred several times. So I'm wondering if this is a configuration error on my part or a limitation related to the current infrastructure, but it seems unreliable in an HA context.

I'd be grateful for your advice or recommendations to stabilize this configuration.

 

[LnxBil]

Can you explain what bonding mode you used and how you configured it on the switch?

 

[babine]

an exemple of a bond configuration :
add comment="VM NETWORK PROX10" mlag-id=1 mode=802.3ad mtu=9000 name=mbond1 \
slaves=sfp-sfpplus1 transmit-hash-policy=layer-2-and-3

About the configuration, it is 2 times two switchs in mlag with a peerbond .

 

[LnxBil]

and on the linux side? Do you use slow or fast?

 

[babine]

Bond configuration proxmox side.

 

[LnxBil]

What about the contents of /proc/net/bonding/bond0 ?

 

[babine]

root@prox11:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.8.12-10-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: ec:e7:a7:10:05:f0
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 15
Partner Key: 15
Partner Mac Address: 78:9a:18:39:52:b9

Slave Interface: ens1f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: ec:e7:a7:10:05:f0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: ec:e7:a7:10:05:f0
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 65535
system mac address: 78:9a:18:39:52:b9
oper key: 15
port priority: 255
port number: 1
port state: 61

Slave Interface: ens6f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: ec:e7:a7:08:37:f8
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: ec:e7:a7:10:05:f0
port key: 15
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 65535
system mac address: 78:9a:18:39:52:b9
oper key: 15
port priority: 255
port number: 1
port state: 61
root@prox11:~#

 

[Joris L.]

@babine was this set-up at one point stable and functioning ?

Was the MTU validated to be supported across all parts and paths ?