Good way to modify corosync.conf in our cluster

[sebastien@domalys]

Hello to the Proxmox and users teams,

I want to change the number of retry before corosync consider the node must be fenced.

The context :

Currently, we have 2 nodes and a qdevice. Each equipment is located in different city and different country (France and Sweden).

(A) Proxmox Server A (Provider One) ---------------------- (B) Proxmox Server B (Provider Two)
                 |                                                          |
                 \----------------------------------------------------------/
                                               |
                  (C) Qdevice on Debian server (in the company) + NFS share

Between the nodes, the network is not enough good for corosync with default parameters. We have a average of 50 ms of ping, and the documentation said that a maximum of 2 ms must be respected between nodes.

So the HA functionality is unstable and, randomly, the nodes are fenced. Some times, all nodes is fenced in 1 or 2 minutes.

As I can't modify the network architecture, I read the corosync documentation and I found a option that I think it can help.

The option is "token_retransmits_before_loss_const". The default is 4 retransmissions before forming new configuration. I want try a value of 40 .

So, here is question : Is the following is the best way to change de corosync configuration on our current cluster ?

My source of information : https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_edit_corosync_conf

(The following will be done on only one node)

• Backup the current configuration

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.bak

• Prepare the new configuration file :

cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new

• Edit the corosync.conf.new file with the following modifications :

Original :​

...
totem {
cluster_name: xxx
config_version: 23
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}

​

Modified :​

totem {
cluster_name: xxx
config_version: 24
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
token_retransmits_before_loss_const: 40
token : 30000
}

• Replace the old configuration file with the new one:

mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf

• Restart corosync :

systemctl restart corosync

As I don't want to break the cluster, if you see a error, don't hesitate to tell me !

Thanks in advance for the help or correction.

Best regards.

 

[David Herselman]

We mitigated against false positive fencing scenarios by making the following changes:

Overview:
https://forum.proxmox.com/threads/unexpected-restart-of-all-cluster-nodes.97021/#post-426562

Restart corosync if it ever crashes:
https://forum.proxmox.com/threads/unexpected-restart-of-all-cluster-nodes.97021/post-426580

Making corosync less sensitive:
https://forum.proxmox.com/threads/pve-5-4-11-corosync-3-x-major-issues.56124/post-269235

 

[spirit]

when you modifiy file in /etc/pve/corosync.conf, proxmox copy it ot /etc/corosync/corosync.conf to all pve nodes and restart corosync.
but for the qdevice, you need to do it manually. (as you don't have /etc/pve/ ).

also, shutdown HA first to avoid fencing.

systemctl stop pve-ha-lrm on all nodes

then

systemctl stop pve-ha-crm on all nodes