8 Node Cluster / Host key verification failed.

[TheMrg]

We can migrate from all of our nodes. but from cluster22 to cluster23 it is not working due
Host key verification failed.
Migrate from cluster22 to cluster21 works well. migrate from cluster21 to cluster23 works well too.

check:
/usr/bin/ssh -v -e none -o 'BatchMode=yes' -o 'HostKeyAlias=cluster23' root@51.N.N.N /bin/true

OpenSSH_7.9p1 Debian-10+deb10u1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 51.N.N.N [51.N.N.N] port 63000.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 51.N.N.N:63000 as 'root'
debug1: using hostkeyalias: cluster23
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:WaGhKRWcXLUG+dmaDJkQNaZBQ28amTKz7PNqZ+LdBCU
debug1: using hostkeyalias: cluster23
debug1: Host 'cluster23' is known and matches the RSA host key.
debug1: Found key in /etc/ssh/ssh_known_hosts:26
Host key verification failed.

What is wrong with cluster22?
corosync is working. cat /etc/pve/priv/known_hosts is exaclty the same on all hosts.

################################################################
other problem is, why do proxmox use the public interface instead of the private (we migrate via GUI). we have
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.3
0x00000002 1 192.168.1.1
0x00000003 1 192.168.1.20
0x00000004 1 192.168.1.2
0x00000005 1 192.168.1.21
0x00000006 1 192.168.1.99
0x00000007 1 192.168.1.22 (local)
0x00000008 1 192.168.1.23
################################################################

Thanks so mich.

 

[TheMrg]

ok, after 2 hows, we got it:
edit
/root/.ssh/known_hosts
seemse there are problems
with different host keys from files:
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:23
debug3: load_hostkeys: loaded 1 keys from cluster22
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:4

still a problem:
################################################################
other problem is, why do proxmox use the public interface instead of the private (we migrate via GUI). we have
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.3
0x00000002 1 192.168.1.1
0x00000003 1 192.168.1.20
0x00000004 1 192.168.1.2
0x00000005 1 192.168.1.21
0x00000006 1 192.168.1.99
0x00000007 1 192.168.1.22 (local)
0x00000008 1 192.168.1.23
################################################################

 

[TheMrg]

solved too, was wrong /etc/hosts 2nd line. local name cluster23 was mapped to public ip
, but how can i upgrade the wrong entry in
/etc/pve/.members

"cluster22": { "id": 7, "online": 1, "ip": "192.168.0.22"},
"cluster23": { "id": 8, "online": 1, "ip": "51.N.N.N"},

Thanks.

 

[TheMrg]

ok, we restart the node. now it works as espected.
is there a way without reboot the hole node and withput impact on runnung VMs ?

 

[Stoiko Ivanov]

You should get /etc/pve/.members updated by restarting pmxcfs/pve-cluster (`systemctl restart pve-cluster`) after you've fixed your name resolution

I hope this helps!