[SOLVED] Linux bond and ceph timeouts

[tiopz]

Hi,

I have a cluster of 4 servers running Proxmox 9.2.5.
These servers are running ceph 19.2.5 and each server is connected to two 10gb ethernet switchs : one ethernet port on switch A and the other on switch B. These switch are dedicated to the ceph network and switch A and B are not connected together.

The switchs are the same brand and model, same firmware version and same configuration (except management IP).

Here is the network config of the server 1

auto enp4s0f0
iface enp4s0f0 inet manual
        mtu 9000

auto enp4s0f1
iface enp4s0f1 inet manual
        mtu 9000
        
auto bond1
iface bond1 inet static
        address 192.168.1.1/24
        bond-slaves enp4s0f0 enp4s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp4s0f0
        mtu 9000

The configuration is identical for the other servers, except the interface name and IP address.
I checked that the primary interface configured on the bonds is the one connected to switch A on all servers to make sure that they are all using the same switch to communicate.

However, after a couple of minutes, ceph hangs because one or two servers stopped communicating on switch A and they are now using switch B. No changes were made in topology or configuration when this happened. Unplugging switch B makes all servers use switch A again and everything works properly.

An other solution is to remove enp4s0f1 from bond-slaves but I am losing the why I've made the bond in the first place.

Any ideas on what could be happening ?

 

[aaron]

Anything in the kernel logs about the interfaces maybe losing the connection? That could trigger the switch to the other interface.

If you do a multi-switch setup, the best approach would be to configure the two switches in an MLAG configuration and then LACP to the hosts.
The switches would then of course need to support that, because in the end, they would be also connected and know about each other and that each switch holds one of the two physical connections of the LACP bond to the host.

Active-Backup bonds with two dumb switches is not the best idea as you saw in this situation.

 

[tiopz]

There are some logs of the interfaces going up/down but I don't remember if this happened when I tested it and/or plugged/unplugged the backup switch. Since I removed the backup link inside the bond, no more issues were found.

From what I saw and what you said, MLAG is clearly the solution here

Thank you