[SOLVED] Migration ok from PVE1 to PVE2 but fails from PVE2 to PVE1

[genesio]

Hello,
I have a cluster with two nodes (no HA):
- pve1 (192.168.0.4)
- pve2 (192.168.0.6)

I can successfully migrate VMs and containers from pve1 to pve2.
I now need to migrate a container from pve2 to pve1 but the operations fails with this error:

2019-02-06 17:49:26 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve1' root@192.168.0.4 /bin/true
2019-02-06 17:49:26 Host key verification failed.
2019-02-06 17:49:26 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted


Important: I am able to connect with ssh from pve2 to pve1:

root@pve2:~# ssh 192.168.0.4
Linux pve1 4.15.18-10-pve #1 SMP PVE 4.15.18-32 (Sat, 19 Jan 2019 10:09:37 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 6 17:54:09 2019
root@pve1:~#


What could I be missing?
Thank you

 

[Chris]

Hi,
please check the fingerprints in '/etc/pve/priv/known_hosts'. There should be two entries per node, one with hostname and one with ip, both with identical fingerprint. Eventually also check in '/root/.ssh/known_hosts'. Finally, verify that the file '/etc/ssh/ssh_known_hosts' is a symlink to '/etc/pve/priv/known_hosts'. You can get the fingerprint of the nodes by running `ssh-keyscan -t ssh-rsa hostname/ip`.

 

[genesio]

Hi,
please check the fingerprints in '/etc/pve/priv/known_hosts'. There should be two entries per node, one with hostname and one with ip, both with identical fingerprint. Eventually also check in '/root/.ssh/known_hosts'. Finally, verify that the file '/etc/ssh/known_hosts' is a symlink to '/etc/pve/priv/known_hosts'. You can get the fingerprint of the nodes by running `ssh-keyscan -t ssh-rsa hostname/ip`.

The known_hosts file was missing the entry for pve1 (hostname) and 192.168.0.4 ip address
running ssh-keyscan and adding the resulting key to the file solved the issue

I still don't know what could have caused the known_hosts file to "loose" one of the keys.
I bet I did something wrong when I had some issues last week after installing a new internal DNS server

Thank you
Genesio

 

[huky]

I met the same problem

2019-04-10 15:10:21 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=ynode002' root@172.16.100.2 /bin/true
2019-04-10 15:10:21 Host key verification failed.
2019-04-10 15:10:21 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted

try ssh in shell:
key is differs from two known_hosts?

root@ynode001:~# ssh -o 'HostKeyAlias=ynode002' root@172.16.100.2
Warning: the RSA host key for 'ynode002' differs from the key for the IP address '[172.16.100.2]:60019'
Offending key for IP in /root/.ssh/known_hosts:8
Matching host key in /etc/ssh/ssh_known_hosts:3
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.

this is a way to rolve it:

root@ynode001:~# > /etc/ssh/ssh_known_hosts
root@ynode001:~# ssh -o 'HostKeyAlias=ynode002' root@172.16.100.2
The authenticity of host 'ynode002 ([172.16.100.2]:60019)' can't be established.
ECDSA key fingerprint is SHA256:2EAmzapvZCeCpFRG9TuR3Z/8uYPznFeAe9yGCD3VBZc.
Are you sure you want to continue connecting (yes/no)? yes

 

[Chris]

I met the same problem



try ssh in shell:
key is differs from two known_hosts?


this is a way to rolve it:

This will repopulate the /root/.ssh/known_hosts and not the /etc/ssh/ssh_known_hosts which is symlinked to /etc/pve/priv/known_hosts and might cause you troubles as well as not being shared in the cluster.
IMHO it is better to make sure that /etc/pve/priv/known_hosts is correct.