Cluster fails to restart/recover after network disconection

jinjer

Active Member
Oct 4, 2010
203
6
38
Hi,

I have a few proxmox nodes running in a cluster, with the cluster on a separate network interface.
If a cluster node looses connection to this interface temporarily (like when a cable is disconnected and reconnected), the node will never see the other nodes until reboot.

I tried issuing "service proxmox-cluster restart" to no avail but that does not fix the problem.

The only way is a reboot of the node.

Is this normal, or are there other services I need to restart?
Code:
# pveversion -v
proxmox-ve: 5.0-19 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-30 (running version: 5.0-30/5ab26bc)
pve-kernel-4.10.17-2-pve: 4.10.17-19
pve-kernel-4.10.17-1-pve: 4.10.17-18
libpve-http-server-perl: 2.0-5
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-3
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.9-pve16~bpo90
jijner
 

manu

Proxmox Staff Member
Mar 3, 2015
806
66
28
when you disconnect / reconnect the cable, the cluster software should detect this and rebuild the cluster
after the cable is plugged in again check that:

1) the link is active on the NIC
ip link show dev your_NIC
should have LOWER_UP in the output
2) if the link is active, verify if the network cluster between the nodes is active by pinging each other nodes
3) then check in the corosync log for two things:
A) on cable disconnect, you should get the message that a new membership is formed ( on the other nodes) including the two remaining nodes
B) on cable reconnect you should see again the message new membership is formed, this time including the third node
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!