Proxmox cluster reboot

[itvietnam]

Hi,

I have 10 nodes in cluster, is this reliable quorum?

We are usually facing reboot on the weekend (during backup VM). Currently we have stopped backup all nodes and create backup cron per node to avoid this random reboot. It has been up more than 2 weeks.

May we know how to debug this?

 

[mir]

An even number of nodes is always a bad thing for deciding quorum.

 

[udo]

Hi,
yes an odd number is better, but with this amount of nodes no problem.
It must be sure, that 6 nodes are running (with corosync). That should be easy with 10 nodes.

Your issue sounds, that you use the same nic for storage and cluster communication. If you saturate the link with storage traffic, the corosync traffic are not fast enough and you lost the quorum (and hosts are reboot).

Split the networks and you should be happy!

Udo

 

[itvietnam]

use the same nic for storage and cluster communication

Thanks Udo, we solved a lot of case from your reply on this community.

Correct. But we have 8 x 10Gbps per node:

• bond0: vmbr0 for VM public network connection. This bond run as mode active-backup with dual 10Gbps SFP+
• bond1: this bond works as cluster network, migration network, 2 differences external storage and backup. This bond run as mode LACP with 4 x 10Gbps SFP+
• bond2: vmbr1 for VM internal network connection. This bond run as mode active-backup with dual 10Gbps SFP+

Can we change cluster network to vmbr1 without downtime?

 

[itvietnam]

If you saturate the link with storage traffic, the corosync traffic are not fast enough and you lost the quorum (and hosts are reboot).

But do you think all node will reboot in this case?

Our last 2 incidents has been reboot all nodes in this case.

 

[czechsys]

I am suggesting dedicated second corosync ring on different switch - you have stacked switches or multichasis LACP?

 

[itvietnam]

Hello,

We use multichasis LACP.

Can we change corosync network to differences interface without downtime?

Thanks,

 

[Belokan]

I added a corosync ring on a second network on live cluster using this document:

https://pve.proxmox.com/wiki/Separate_Cluster_Network#On_Running_Cluster

Just don't forget to add "rrp_mode: passive" and to increase config_version: ##" to avoid most issues in the procedure.

Once done, just evacuate/move VMs/Containers and reboot nodes one by one.

 

[itvietnam]

I added a corosync ring on a second network on live cluster using this document:

https://pve.proxmox.com/wiki/Separate_Cluster_Network#On_Running_Cluster

Just don't forget to add "rrp_mode: passive" and to increase config_version: ##" to avoid most issues in the procedure.

Once done, just evacuate/move VMs/Containers and reboot nodes one by one.

Thanks for shed some light, i have added to our note to making change in this week.

Thanks again,