[SOLVED] ssh error transferring to new node in cluster

[j4ys0n]

the error only seems to occur when i'm initiating a transfer from one server to another (of a vm or lxc).

i get this message..

022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2022-05-13 00:14:00 @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
2022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2022-05-13 00:14:00 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
2022-05-13 00:14:00 Someone could be eavesdropping on you right now (man-in-the-middle attack)!
2022-05-13 00:14:00 It is also possible that a host key has just been changed.
2022-05-13 00:14:00 The fingerprint for the ECDSA key sent by the remote host is
...

i did have trouble adding the node to the cluster initially. but i deleted it and all of the references to it and added it again with the same name and different ip address. i've done this before with success but, that was on older versions of proxmox. i've made sure the keys are updated and i can connect to this host, by hostname and ip from all nodes in the cluster without issue using ssh from the terminal. but when initiating a transfer, it fails.

i tried

pvecm updatecerts

and

pvecm updatecerts --force

but that hasn't made a difference.

anyone know what i need to do here?

 

[Stoiko Ivanov]

022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2022-05-13 00:14:00 @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ 2022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2022-05-13 00:14:00 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! 2022-05-13 00:14:00 Someone could be eavesdropping on you right now (man-in-the-middle attack)!

usually the context above and below the message states which command exactly is executed (which options to ssh, which command to run on the remote host) - could you share that - and the output when you run it as root on the cli?

additionally please share:
* `ls -la /etc/ssh` from all nodes

 

[j4ys0n]

yes

2022-05-13 00:14:00 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=starhawk' root@10.10.1.17 /bin/true
2022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2022-05-13 00:14:00 @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
2022-05-13 00:14:00 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2022-05-13 00:14:00 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
2022-05-13 00:14:00 Someone could be eavesdropping on you right now (man-in-the-middle attack)!
2022-05-13 00:14:00 It is also possible that a host key has just been changed.
2022-05-13 00:14:00 The fingerprint for the ECDSA key sent by the remote host is
2022-05-13 00:14:00 SHA256:duuksvODLbyOK+uTE2ikxYPMsGstfJgZRT81wiCcwVY.
2022-05-13 00:14:00 Please contact your system administrator.
2022-05-13 00:14:00 Add correct host key in /root/.ssh/known_hosts to get rid of this message.
2022-05-13 00:14:00 Offending RSA key in /etc/ssh/ssh_known_hosts:12
2022-05-13 00:14:00   remove with:
2022-05-13 00:14:00   ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "starhawk"
2022-05-13 00:14:00 ECDSA host key for starhawk has changed and you have requested strict checking.
2022-05-13 00:14:00 Host key verification failed.
2022-05-13 00:14:00 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted

i ran ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "starhawk" but it didn't make a difference.

node 1

total 162
drwxr-xr-x  4 root root     14 Apr 24 15:26 .
drwxr-xr-x 92 root root    189 Apr 24 15:29 ..
-rw-r--r--  1 root root 577771 Mar 13  2021 moduli
-rw-r--r--  1 root root   1650 Mar 13  2021 ssh_config
drwxr-xr-x  2 root root      2 Mar 13  2021 ssh_config.d
-rw-r--r--  1 root root   3274 Apr 24 15:26 sshd_config
drwxr-xr-x  2 root root      2 Mar 13  2021 sshd_config.d
-rw-------  1 root root    505 Sep 20  2020 ssh_host_ecdsa_key
-rw-r--r--  1 root root    174 Sep 20  2020 ssh_host_ecdsa_key.pub
-rw-------  1 root root    399 Sep 20  2020 ssh_host_ed25519_key
-rw-r--r--  1 root root     94 Sep 20  2020 ssh_host_ed25519_key.pub
-rw-------  1 root root   1823 Sep 20  2020 ssh_host_rsa_key
-rw-r--r--  1 root root    394 Sep 20  2020 ssh_host_rsa_key.pub
lrwxrwxrwx  1 root root     25 Jan 10  2021 ssh_known_hosts -> /etc/pve/priv/known_hosts

node 2

total 100
drwxr-xr-x  4 root root     15 Apr 29 14:42 .
drwxr-xr-x 92 root root    188 Apr 29 14:49 ..
-rw-r--r--  1 root root 577771 Mar 13  2021 moduli
-rw-r--r--  1 root root   1650 Mar 13  2021 ssh_config
drwxr-xr-x  2 root root      2 Mar 13  2021 ssh_config.d
-rw-r--r--  1 root root   3235 May 25  2021 sshd_config
drwxr-xr-x  2 root root      2 Mar 13  2021 sshd_config.d
-rw-r--r--  1 root root   3274 Apr 29 14:40 sshd_config.ucf-dist
-rw-------  1 root root    505 May 25  2021 ssh_host_ecdsa_key
-rw-r--r--  1 root root    174 May 25  2021 ssh_host_ecdsa_key.pub
-rw-------  1 root root    399 May 25  2021 ssh_host_ed25519_key
-rw-r--r--  1 root root     94 May 25  2021 ssh_host_ed25519_key.pub
-rw-------  1 root root   1823 May 25  2021 ssh_host_rsa_key
-rw-r--r--  1 root root    394 May 25  2021 ssh_host_rsa_key.pub
lrwxrwxrwx  1 root root     25 May 25  2021 ssh_known_hosts -> /etc/pve/priv/known_hosts

node 3

total 99
drwxr-xr-x   4 root root     14 Apr 24 16:39 .
drwxr-xr-x 110 root root    213 May 12 19:35 ..
-rw-r--r--   1 root root 577771 Mar 13  2021 moduli
-rw-r--r--   1 root root   1650 Mar 13  2021 ssh_config
drwxr-xr-x   2 root root      2 Mar 13  2021 ssh_config.d
-rw-r--r--   1 root root   3274 Apr 24 16:39 sshd_config
drwxr-xr-x   2 root root      2 Mar 13  2021 sshd_config.d
-rw-------   1 root root    505 Oct  5  2019 ssh_host_ecdsa_key
-rw-r--r--   1 root root    176 Oct  5  2019 ssh_host_ecdsa_key.pub
-rw-------   1 root root    411 Oct  5  2019 ssh_host_ed25519_key
-rw-r--r--   1 root root     96 Oct  5  2019 ssh_host_ed25519_key.pub
-rw-------   1 root root   1823 Oct  5  2019 ssh_host_rsa_key
-rw-r--r--   1 root root    396 Oct  5  2019 ssh_host_rsa_key.pub
lrwxrwxrwx   1 root root     25 Jan 10  2021 ssh_known_hosts -> /etc/pve/priv/known_hosts

node 4

total 94
drwxr-xr-x  2 root root     12 May 25  2021 .
drwxr-xr-x 92 root root    182 Apr 24 17:24 ..
-rw-r--r--  1 root root 565189 Oct  6  2019 moduli
-rw-r--r--  1 root root   1580 Oct  6  2019 ssh_config
-rw-r--r--  1 root root   3235 Aug 22  2020 sshd_config
-rw-------  1 root root    505 Aug 22  2020 ssh_host_ecdsa_key
-rw-r--r--  1 root root    173 Aug 22  2020 ssh_host_ecdsa_key.pub
-rw-------  1 root root    399 Aug 22  2020 ssh_host_ed25519_key
-rw-r--r--  1 root root     93 Aug 22  2020 ssh_host_ed25519_key.pub
-rw-------  1 root root   1823 Aug 22  2020 ssh_host_rsa_key
-rw-r--r--  1 root root    393 Aug 22  2020 ssh_host_rsa_key.pub
lrwxrwxrwx  1 root root     25 May 25  2021 ssh_known_hosts -> /etc/pve/priv/known_hosts

node 5

total 152
drwxr-xr-x  4 root root     15 May 12 23:58 .
drwxr-xr-x 89 root root    182 May 12 23:19 ..
-rw-r--r--  1 root root 577771 Mar 13  2021 moduli
-rw-r--r--  1 root root   1650 Mar 13  2021 ssh_config
drwxr-xr-x  2 root root      2 Mar 13  2021 ssh_config.d
-rw-r--r--  1 root root   3274 Apr 24 23:02 sshd_config
drwxr-xr-x  2 root root      2 Mar 13  2021 sshd_config.d
-rw-------  1 root root    505 Apr 24 23:02 ssh_host_ecdsa_key
-rw-r--r--  1 root root    172 Apr 24 23:02 ssh_host_ecdsa_key.pub
-rw-------  1 root root    399 Apr 24 23:02 ssh_host_ed25519_key
-rw-r--r--  1 root root     92 Apr 24 23:02 ssh_host_ed25519_key.pub
-rw-------  1 root root   2590 Apr 24 23:02 ssh_host_rsa_key
-rw-r--r--  1 root root    564 Apr 24 23:02 ssh_host_rsa_key.pub
lrwxrwxrwx  1 root root     25 May 12 23:58 ssh_known_hosts -> /etc/pve/priv/known_hosts
lrwxrwxrwx  1 root root     25 May 12 23:38 ssh_known_hosts.old -> /etc/pve/priv/known_hosts

node 6

total 151
drwxr-xr-x  4 root root     14 May 12 20:18 .
drwxr-xr-x 86 root root    177 May 12 20:24 ..
-rw-r--r--  1 root root 577771 Mar 13  2021 moduli
-rw-r--r--  1 root root   1650 Mar 13  2021 ssh_config
drwxr-xr-x  2 root root      2 Mar 13  2021 ssh_config.d
-rw-r--r--  1 root root   3274 May 12 20:05 sshd_config
drwxr-xr-x  2 root root      2 Mar 13  2021 sshd_config.d
-rw-------  1 root root    505 May 12 20:04 ssh_host_ecdsa_key
-rw-r--r--  1 root root    175 May 12 20:04 ssh_host_ecdsa_key.pub
-rw-------  1 root root    399 May 12 20:04 ssh_host_ed25519_key
-rw-r--r--  1 root root     95 May 12 20:04 ssh_host_ed25519_key.pub
-rw-------  1 root root   2602 May 12 20:04 ssh_host_rsa_key
-rw-r--r--  1 root root    567 May 12 20:04 ssh_host_rsa_key.pub
lrwxrwxrwx  1 root root     25 May 12 20:18 ssh_known_hosts -> /etc/pve/priv/known_host

 

[j4ys0n]

@Stoiko Ivanov any idea what the problem is? the known_hosts files seem to be fine.

 

[manuelkamp]

Today I wanted to migrate a CT from node 1 to node 2 (where it belongs to, I migrated it 3 days ago from node 2 to 1) and faced the same issue!

 

[j4ys0n]

I got it - had to make sure the old host key was gone from all of the known host files on all of the nodes, which i thought i had done.

ran ssh-keygen -f "/etc/pve/priv/known_hosts" -R "starhawk" and ssh-keygen -f "/root/.ssh/known_hosts" -R "starhawk" on each node, then connected manually with ssh starhawk to regenerate the hashed known host entry. (might have to add an entry in /etc/hosts)

this was handy to see what the hashed known host entry should look like.

echo "starhawk,10.10.1.17 $(cat /etc/ssh/ssh_host_ecdsa_key.pub)" >> ~/known_hosts_test
ssh-keygen -f known_hosts_test -H
cat known_host_test

 

[Stoiko Ivanov]

lrwxrwxrwx 1 root root 25 May 12 23:38 ssh_known_hosts.old -> /etc/pve/priv/known_hosts

this link seems problematic - it should not be there - remove it

ran ssh-keygen -f "/etc/pve/priv/known_hosts" -R "starhawk" and ssh-keygen -f "/root/.ssh/known_hosts" -R "starhawk" on each node, then connected manually with ssh starhawk to regenerate the hashed known host entry. (might have to add an entry in /etc/hosts)

glad you figured it out

 

[manuelkamp]

well that did not work for me. I cleared the known_hosts file content and ran the two commands (of course with my node name). also ssh'd to the other node successfully. but in pve ui I still can not migrate CTs to the node?

<code>2022-05-16 14:27:19 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=srv2' root@10.0.1.4 /bin/true
2022-05-16 14:27:19 Host key verification failed.
2022-05-16 14:27:19 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted/code>

 

[manuelkamp]

Ok-forget it, this weird instructions just confused me. The only thing you need to do is remove the key (as it is stated in the error message) from etc/ssh/ssh_known_hosts and ssh to the node again to store the new key - no need to use the different commands from @j4ys0n for the other ssh_known_hosts files.