Cluster issues after migration to new cluster

[mervin]

Hi there,

We are in the process of migrating servers to a new 8.2 cluster.
The steps are as follows:

1. Shutdown the server
2. Remove the server from the cluster via : pvecm delnode mynode
3. Reinstall the server with the same name and network configuration as before
4. Add the node to the new cluster via the GUI

On two occasions this whent well, the last two times it didn't. The symptoms after adding are:

• Question marks in the GUI at the node and container status
• Unresponsive PVE services
• Not able to run pct commands on the nodes
• Not able to login to the GUI

We managed to get the cluster working again by restarting pve services many times in different orders etc.
At this moment the last added node has a green status in the GUI but the network config is not visible and when we select containers we get a connection error timeout and communication failure (0) message.

Do you have any idea what we're doing wrong, what step we might have missed and what we can do about the connection error timeouts?

Thank you in advance!

 

[mervin]

Update:
I noticed that in /etc/pve/priv/authorized_keys the key of the new node was not there on all nodes.
I ran pvecm updatecerts on the new node, after that the key was immediately visible on all other nodes.

The connection time outs in the GUI are now gone!

I have a feeling that this was the root cause of our problems all along.

 

[mervin]

Hi there,
Yesterday we unfortunately experienced the same issues after rebooting a node.
This time all SSH keys where present.
Could it be that we're missing a step before rebooting the node?
With version 6.4 this did not happen, we could just reboot the node without issues.
Any ideas are greatly appreciated.