How to re-join same node to the cluster (Proxmox 8)

[pidumenk]

Hello,

I faced with a problem, where I had to rejoin same node to the cluster, but the following errors appeared:

Please enter superuser (root) password for 'IP ADDR': **************
detected the following error(s):
* authentication key '/etc/corosync/authkey' already exists
* cluster config '/etc/pve/corosync.conf' already exists
* this host already contains virtual guests
* corosync is already running, is this node already in a cluster?!
Check if node may join a cluster failed!

I don't want to change IP address or anything else in the server configuration. I simply executed pvecm delnode and then I did pvecm add IP-ADDR. The reason why I'm rejoining the server is a typo in my /etc/hosts and /etc/hostname. When I fixed that, the cluster couldn't read certificates, cause it had been issued for a different domain name which wasn't match a new one. So, I just decided to delete node from the cluster, change names, restart server and join back.

 

[pidumenk]

The following solution was found by me. I fixed that just following these steps:

1) First, stop the corosync and pve-cluster services on the node:

systemctl stop pve-cluster
systemctl stop corosync

2) Start the cluster file system again in local mode:

pmxcfs -l

3) Delete the corosync configuration files:

rm /etc/pve/corosync.conf
rm -r /etc/corosync/*
rm /var/lib/corosync/*

# Delete all old cluster configuration files of you want to join the node back to the same cluster
rm -r /etc/pve/nodes/*

4) Start the file system again as a normal service:

killall pmxcfs
systemctl start pve-cluster

5) Join node to the cluster

pvecm add IP-ADDRESS-CLUSTER

6) Update cluster certificates on all nodes within the custer

pvecm updatecerts

 

[esi_y]

6) Update cluster certificates on all nodes within the custer

To avoid future issues, I would suggest you check content of your /etc/pve/known_hosts for duplicate entries (both IP and alias for the rejoined node). Duplicate entries in itself would not pose a problem, but there's a bug in a certain codepath of pvecm that will later on retain only the oldest entry of the same name in the file, drop the most recent, and you can start getting SSH conn errors.

EDIT: More details here: https://forum.proxmox.com/threads/cannot-migrate-vm-ct-due-to-ssh-key-error.133560/#post-614183