Cluster issue

[34by151]

I have a 3node cluster

I needed to rebuild one node
I did the removal and join using the docs https://pve.proxmox.com/wiki/Cluster_Manager
That all went well

Note: Im using the same name PVE3 and IP for ne rebuilt node as before

Now I have issues manageing the nodes

My 3 modes are PVE1, PVE2, PVE3

If I login via PVE1
Everything i s fine on PVE1 and PVE2
On PVE3 I cant get to the console of the LXC containers

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:1xpJKnujQjC6VuAUpsazcb94/HDf09o7x2v8B+ECWeg.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:17
remove with:
ssh-keygen -f "/root/.ssh/known_hosts" -R "172.30.30.13"
Host key for 172.30.30.13 has changed and you have requested strict checking.
Host key verification failed.

If I login via PV2
Everything works correctly and the above issue goes away

If I login via PV3
Everything works correctly and the above issue goes away

Ive tried a bunch if solutions on the forms without sucess
inclusing the pvecm updatecerts --force and ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "172.30.30.13"

Im running the latest Proxmox 8.2.4 on all nodes with all patches up to date

 

[mira]

Can you run the following command on all cluster nodes and provide the output? ls -l /etc/ssh
What's the output of the following command run on any of the nodes (1 is enough)? ssh-keygen -F "<HOST>" -f /etc/pve/priv/known_hosts -H
Please replace <HOST> in the command above with the IP and the hostname, once each.

 

[34by151]

PVE1
-rw-r--r-- 1 root root 573928 Dec 20 2023 moduli
-rw-r--r-- 1 root root 1650 Dec 20 2023 ssh_config
drwxr-xr-x 2 root root 4096 Dec 20 2023 ssh_config.d
-rw-r--r-- 1 root root 3208 May 5 14:41 sshd_config
drwxr-xr-x 2 root root 4096 Dec 20 2023 sshd_config.d
-rw------- 1 root root 505 May 5 14:41 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 171 May 5 14:41 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 May 5 14:41 ssh_host_ed25519_key
-rw-r--r-- 1 root root 91 May 5 14:41 ssh_host_ed25519_key.pub
-rw------- 1 root root 2590 May 5 14:41 ssh_host_rsa_key
-rw-r--r-- 1 root root 563 May 5 14:41 ssh_host_rsa_key.pub
lrwxrwxrwx 1 root root 25 May 25 18:54 ssh_known_hosts -> /etc/pve/priv/known_hosts

PVE2
-rw-r--r-- 1 root root 573928 Dec 20 2023 moduli
-rw-r--r-- 1 root root 1650 Jul 2 2022 ssh_config
drwxr-xr-x 2 root root 4096 Jul 2 2022 ssh_config.d
-rw-r--r-- 1 root root 3208 Jun 16 15:46 sshd_config
drwxr-xr-x 2 root root 4096 Jul 2 2022 sshd_config.d
-rw------- 1 root root 505 Jun 16 14:59 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 170 Jun 16 14:59 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 Jun 16 14:59 ssh_host_ed25519_key
-rw-r--r-- 1 root root 90 Jun 16 14:59 ssh_host_ed25519_key.pub
-rw------- 1 root root 2590 Jun 16 14:59 ssh_host_rsa_key
-rw-r--r-- 1 root root 562 Jun 16 14:59 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 1681 Jun 16 16:43 ssh_known_hosts

PVE3
-rw-r--r-- 1 root root 573928 Dec 20 2023 moduli
-rw-r--r-- 1 root root 1650 Jul 2 2022 ssh_config
drwxr-xr-x 2 root root 4096 Jul 2 2022 ssh_config.d
-rw-r--r-- 1 root root 3208 Jun 20 16:22 sshd_config
drwxr-xr-x 2 root root 4096 Jul 2 2022 sshd_config.d
-rw------- 1 root root 505 Jun 20 15:49 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 171 Jun 20 15:49 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 Jun 20 15:49 ssh_host_ed25519_key
-rw-r--r-- 1 root root 91 Jun 20 15:49 ssh_host_ed25519_key.pub
-rw------- 1 root root 2590 Jun 20 15:49 ssh_host_rsa_key
-rw-r--r-- 1 root root 563 Jun 20 15:49 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 558 Jun 20 21:10 ssh_known_hosts


No output from these commands on PVE1

root@pve1:~# ssh-keygen -F "172.30.30.12" -f /etc/pve/priv/known_hosts -H
root@pve1:~# ssh-keygen -F "172.30.30.13" -f /etc/pve/priv/known_hosts -H
root@pve1:~# ssh-keygen -F "172.30.30.11" -f /etc/pve/priv/known_hosts -H

Same result
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED
When connected to cluster via PVE1

 

[mira]

Looks like PVE2 and PVE3 no longer use the shared ssh_known_hosts (no symlink).

Can you check the the local known_hosts file for root as well on PVE1?

ssh-keygen -F "172.30.30.13" -f /root/.ssh/known_hosts -H
ssh-keygen -F "pve3" -f /root/.ssh/known_hosts -H

 

[34by151]

From shell on PVE1

ssh-keygen -F "172.30.30.13" -f /root/.ssh/known_hosts -H

# Host 172.30.30.13 found: line 15
|1|8Ab7Les34g1hHPFxND/DgvQx+h0=|vp68GbltdHQVILVxwfmyBOSE7D0= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBu1nd23KJlT8l+wzOtvvcN++8+hxx4s33QpYtDz+aS9
# Host 172.30.30.13 found: line 16
|1|fICwgo2h8seXTrzUQDrubLY22/k=|yOAmiHzYHsBN314LbvR9ZruJ30w= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCr3YVkhcRtlnQkKIWED+KGsPos49dRTQQbnG286XcgVyV3+QM+CILebLUYdrefDRBg5AxTxMbBsFI0/ZpSkVN8BMIoavHpMGIx/M/Xp1C3iKftKZckomLNKbiwd9w7qKFrR7xlkYk1zkyxJspRFbz63sXWSF1SVSmFWAElGQ2Y8DOBseNG4Ev6v6jd0RcoVvwgzskvi74pvqum1Fw7rIAE0N+TRMhpNQqU/+lWLF6l8ePlJTxZrXMnQZ3TJOabDrWSMn21FaschPAyUJDvO25WUDblA8mviWpae4meDvMdxzwpYkClNqkewrOrrZArdp8k10BVtXyaWobayZfv7Iu3LJ8dyltjI2g7y2sExgogTEdCNMYjgfyLLTc5MEEEVpmrNruYjgw6/u/7chCSEljwFN9h0UgRbS/m0s3LIwf6JfpHJBqlqjQb5NbmxQstYsfxOw4vl0yA8clFuVtn000VomcyhfYWDV4oHcHYk48/mj02S4bJFzRqGfpHlthgY8=
# Host 172.30.30.13 found: line 17
|1|wnzkohUBt3S7zGMFLMY71Ja9BQg=|ppLX3gtuesoIz8XTR6dxD7ThqJs= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCm0QN3rk5sUQCm8juYmZA7OBag0dZbFHdSBFepUAlZ0KOcyCt8JK07AFORht+OoLEBKsmjS9/FZwGUIFJSI6Lg=

ssh-keygen -F "pve3" -f /root/.ssh/known_hosts -H

No output

 

[mira]

Please remove those 3 lines from /root/.ssh/known_hosts on PVE1.

After removing those, do you still see the issue?

 

[34by151]

Yes that has fixed it

Thanks a bunch