Corosync Stability issues on large cluster

[Nips]

Hello

We have a cluster of > 30 servers. Initially it did NOT have a dedicated Corosync network as we felt the host network would be sufficient (2 X 10Gbe interfaces each), however we started having Corosync issues once we got to about 30 servers so then decided to put in a dedicated Corosync network. We added 2 X 48 Port 1Gbe switches, cabled up each host to go to both switches and set up Corosync to use both interfaces. In other words we now have a dedicated Corosync network with 2 X 1Gbe interfaces per host. This however doesn't seem to have made a difference.

We still end up with hosts that seem to "fall out" of the cluster. In other words in the Web Gui they show up as red, if you log into one of those hosts directly it appears like they have established themsleves in their own cluster.

I tried stopping pve-cluster and cororsync services on all hosts, then slowly starting it up 1 by 1. As I start them up on hosts it all looks good, with hosts appearing in the cluster as expected, until we get between 20-30 hosts running then it starts happening again

Running Proxmox 7.4 with latest patches.

Any advice?

 

[spirit]

Personnality, I never do more than 20 nodes clusters.

It's really depend of network latency and cpu frequence to handle corosync traffic. (bandwith is not important, but you really need low latency and no network saturation).

Note that also, sometime 1 bad node can impact the whole cluster. (bad nic driver, wrong cable,a node really older vs the others). Do you have same nodes models everywhere ? same cpu generation ?

do you have any retransmit corosync logs in /var/log/daemon.log ?


maybe you could try to use sctp instead udp, I had some retransmist error in the past, fixed by it.

/etc/pve/corosync.conf

totem {
...
interface {
knet_transport: sctp
linknumber: 0
}
...
}


(this need a full restart of corosync on every node, so be carefull to not have HA enabled before change this)

 

[Nips]

Thanks. That is at least something to try.