[SOLVED] SSH Host key verification error

[Quim Roscas]

Hello fellow Proxmox'ers,

I added a new node on a 7.4 cluster (I updated it fully before adding it), and I'm seeing the famous SSH Host key verification failed.

Did all the troubleshooting I found in the forums, including "pvecm updatecerts" on all nodes. Can access every node via CLI without complaints about keys, EXCEPT for the new node... None of the other nodes can access the new node, not even the new node can access it self via SSH (error below).

HOWEVER, I still cannot install CEPH on the new node I added... it keeps complaining about the error below. Any merciful soul out there with any idea on how to fix this?

(I know there is no more support for 7.4.xxx but I really need to add this new node before going for the upgrade to 8)

admin@pve3:~$ sudo ssh pve3
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:gEmrJ1X+FOS9LtclMGcEg87osbGdXAdsRfczeDviEUw.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:7
remove with:
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "pve3"
RSA host key for pve3 has changed and you have requested strict checking.
Host key verification failed.
admin@pve3:~$

Also, the following commands yield the same key in all nodes:
admin@pve3:~$ sudo ssh-keygen -F "192.168.0.XXX" -f /etc/pve/priv/known_hosts -H
# Host 192.168.0.XXX found: line 8
|1|wsuLkvrdAseW+5yxN34mNQQkMiM=|9a98/DW2sXjtx4mNlqvFw8rPfN0= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCTpVB9iv/ar1nZKZbraqyeQw6K/4/saRvBeulYCjJPgoa+6d+TeNneecI8E+OO1oC21cbot8OJwXnsUJijoeqoHgmY3DBrGl4LaGxSzJK23s7lmMmfCk+j/RF+u99YgSPnnPhxAKTLldUJ67yKugrfeg0Tjs0T4cxusrdKkCrQKnaFSCse1/VJGoBFB3GG/rZFAAWPFxTZYWF8Baf8RF6vWXQnT/uRqR3aZ6l0Q2BliWkW0n7qxNgKUWThqT9Os+1ypf7fqAIh5wdxR23jlm3itKGlBUyo1Nsh9FrUg8G2gEN6AIeR5V7gjddvY1R0A9SkOGkNHxv8jAPBVbK4cX7f

admin@pve3:~$ sudo ssh-keygen -F "pve3" -f /etc/pve/priv/known_hosts -H
# Host pve3 found: line 7
|1|zfOxuz71U/1Xaq1BwSsH07OZNa4=|qKVJFSkcKJt5HOhQp4N+PqY10Cw= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCTpVB9iv/ar1nZKZbraqyeQw6K/4/saRvBeulYCjJPgoa+6d+TeNneecI8E+OO1oC21cbot8OJwXnsUJijoeqoHgmY3DBrGl4LaGxSzJK23s7lmMmfCk+j/RF+u99YgSPnnPhxAKTLldUJ67yKugrfeg0Tjs0T4cxusrdKkCrQKnaFSCse1/VJGoBFB3GG/rZFAAWPFxTZYWF8Baf8RF6vWXQnT/uRqR3aZ6l0Q2BliWkW0n7qxNgKUWThqT9Os+1ypf7fqAIh5wdxR23jlm3itKGlBUyo1Nsh9FrUg8G2gEN6AIeR5V7gjddvY1R0A9SkOGkNHxv8jAPBVbK4cX7f
pvadmin@pve3:~$

 

[Quim Roscas]

I believe my problem is deriving from reusing an IP from an old removed node... The old ssh key from the removed node must be forgotten somewhere... but where? I already searched it an there is nothing to be found.

I did delete /etc/pve/nodes/<OLD-NODE> ... do I have to reboot the cluster?!

 

[Quim Roscas]

SOLVED!

Compare your key in "/etc/ssh/ssh_host_rsa_key.pub" with the key in "/etc/ssh/ssh_known_hosts". They where different in my case, and executing "pvecm updatecerts" in all nodes did not fix that. Just replace the key in "/etc/ssh/ssh_known_hosts" with the key in "/etc/ssh/ssh_host_rsa_key.pub"
You don't need to replace in all nodes, just replace in one because that file is automatically replicated to all cluster nodes.