[SOLVED] Trouble with SSH Keys and Certs in Cluster

[Hubguru]

Hey Folks,

I'm running a 6.4 cluster with 8 nodes, I'm in the middle of server upgrades, I shutdown old server, delnode from cluster, then bring up new server with same hostname. I did this for a separate cluster with 14 nodes, everything went well and did not have any issues.

But with this particular 8 node cluster, my ssh keys and certs are working, then failing, I am having to remove ssh keys and regenerate certs just to get VMs moved around.

I'm using 'pvecm updatecerts -f' on nodes that will not allow ssh, and this works for a few days then I have to do it all over again. I also have to 'systemctl restart pveproxy; systemctl restart pvestatd' to get the certs to load in the cluster portal.

What am I doing wrong or is there a specific sequence of task to ensure all certs are newly generated on all nodes once and for all?

Thanks.
JR

 

[shanreich]

Have you run pvecm updatecerts on the newly added node?

Since you are using the same hostname it would be possible that there are still old entries in the known_hosts file that contain the old SSH key. You can find the file under /etc/pve/priv/known_hosts

 

[Hubguru]

Yes, I've run 'pvecm updatecerts -f' in newly added nodes. It works for a while, maybe a few days, then I'll go into the cluster to move some VMs from node a to one of the new nodes and get the cert errors. It's really kind of random and I'm struggling to get a handle on what is exactly going on.

Is there a method to clear out all old certs and ssh keys then update certs or could this break my cluster. This is a production cluster with 30+ VMs.

 

[Hubguru]

What I found was an endless back and forth with having to remove the offending host_keys and known_hosts plus I was also using hostname sometimes and when using the 'pvecm updatecerts' this was adding the same hosts using IP so really frustrated. What I did to resolve the issue was to remove the offending keys and known hosts for both the node hostname and IP Address. Then I could connect to the offending node, accept the new key and all worked as expected.