Cluster fails to restart/recover after network disconection

[jinjer]

Hi,

I have a few proxmox nodes running in a cluster, with the cluster on a separate network interface.
If a cluster node looses connection to this interface temporarily (like when a cable is disconnected and reconnected), the node will never see the other nodes until reboot.

I tried issuing "service proxmox-cluster restart" to no avail but that does not fix the problem.

The only way is a reboot of the node.

Is this normal, or are there other services I need to restart?

# pveversion -v
proxmox-ve: 5.0-19 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-30 (running version: 5.0-30/5ab26bc)
pve-kernel-4.10.17-2-pve: 4.10.17-19
pve-kernel-4.10.17-1-pve: 4.10.17-18
libpve-http-server-perl: 2.0-5
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-3
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.9-pve16~bpo90

jijner

 

[manu]

when you disconnect / reconnect the cable, the cluster software should detect this and rebuild the cluster
after the cable is plugged in again check that:

1) the link is active on the NIC
ip link show dev your_NIC
should have LOWER_UP in the output
2) if the link is active, verify if the network cluster between the nodes is active by pinging each other nodes
3) then check in the corosync log for two things:
A) on cable disconnect, you should get the message that a new membership is formed ( on the other nodes) including the two remaining nodes
B) on cable reconnect you should see again the message new membership is formed, this time including the third node