Disconnected Proxmox Host in Cluuster after losing network config

[fpa-mhb]

Hi,

we have a PoC running with a 3-Node Cluster with PVE 8.4.
We have on this cluster also CEPH in version 18.2.4

The Installation was running fine until 2 Weeks ago.
There i recognized in the GUI , that the networking config of on of the hosts was gone, except the static physical network interfaces.
Everythin beyond like Bonds, Bridges, VLANs were gone.
At this point i took a look at /etc/network/interfaces on this host and saw, that this file was completely empty.

unfortunately i treid to reboot the system. as expected, it came back without any networking at all.
Then i rebuilt the network config based on one of the other hosts.
Antoher reboot later, the system was back online with its complete network config.

To my current problem, the 2 remaining cluster nodes don't recognize the reconfigured , former failed , cluster node.
it is shown as discennected.
The networking is working fine.
how can i reconnect this node to the PVE and the ceph cluster ?

Kind Regards
Frank

 

[Moayad]

Hi,

First of all I would check the bash/zsh history to see why the interfaces file was empty!


Regarding the issue, could you please post the output of the following commands:

pvecm status
cat /etc/pve/corosync.conf
pveceph status

 

[fpa-mhb]

Hi Moyad,

attached the output of the commands executed on all 3 nodes one after the other.
xypvele50 is the failed node
xypvel52 and xypve58 are the remaining nodes

Best Regards
Frank

 

[Moayad]

Thank you!

Could please check the syslog and the corosync status `systemctl status corosync` on the `xypvele50` node, plus test ping to .43 and .48.
You can check the syslog using journalctl tool as the following command:

journalctl --since "2 days ago"