Proxmox cluster reboot

itvietnam

Renowned Member
Aug 11, 2015
132
4
83
Hi,

I have 10 nodes in cluster, is this reliable quorum?

We are usually facing reboot on the weekend (during backup VM). Currently we have stopped backup all nodes and create backup cron per node to avoid this random reboot. It has been up more than 2 weeks.

May we know how to debug this?
 
Hi,
yes an odd number is better, but with this amount of nodes no problem.
It must be sure, that 6 nodes are running (with corosync). That should be easy with 10 nodes.

Your issue sounds, that you use the same nic for storage and cluster communication. If you saturate the link with storage traffic, the corosync traffic are not fast enough and you lost the quorum (and hosts are reboot).

Split the networks and you should be happy!

Udo
 
  • Like
Reactions: itvietnam
use the same nic for storage and cluster communication
Thanks Udo, we solved a lot of case from your reply on this community.

Correct. But we have 8 x 10Gbps per node:

  • bond0: vmbr0 for VM public network connection. This bond run as mode active-backup with dual 10Gbps SFP+
  • bond1: this bond works as cluster network, migration network, 2 differences external storage and backup. This bond run as mode LACP with 4 x 10Gbps SFP+
  • bond2: vmbr1 for VM internal network connection. This bond run as mode active-backup with dual 10Gbps SFP+
Can we change cluster network to vmbr1 without downtime?
 
If you saturate the link with storage traffic, the corosync traffic are not fast enough and you lost the quorum (and hosts are reboot).
But do you think all node will reboot in this case?

Our last 2 incidents has been reboot all nodes in this case.
 
I am suggesting dedicated second corosync ring on different switch - you have stacked switches or multichasis LACP?
 
Hello,

We use multichasis LACP.

Can we change corosync network to differences interface without downtime?

Thanks,
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!