VM migration failed - host key verification failed

[andreisrr]

I have a 3 node cluster set up for testing: pve01 and pve02 running 8.1.3 and pve03 added later running 8.2.2

Trying to migrate a VM to or from pve03 fails with the following error:

2024-09-10 13:02:35 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve03' root@A.B.C.D /bin/true
2024-09-10 13:02:35 Host key verification failed.
2024-09-10 13:02:35 ERROR: migration aborted (duration 00:00:00): Can't connect to destination address using public key
TASK ERROR: migration aborted

On all nodes there is a /etc/pve/priv/authorized_keys file with apparently the same contents across all 3 nodes.
All keys are apparently ssh-rsa, with pve03 having a longer key (in bits).

However /etc/pve/priv/known_hosts contains only entries for pve01 and pve02 (across all 3 nodes).

So far all VM migrations I did using WebUI.

What am I missing?
Thank you.

 

[fabian]

please upgrade pve01 and pve02 (or manually add pve03's host key to the legacy shared known hosts file).

 

[andreisrr]

please upgrade pve01 and pve02 (or manually add pve03's host key to the legacy shared known hosts file).

I see. Thank you.

Based on this should I infer that adding pve03 being at a different version to the cluster was wrong ?

 

[andreisrr]

Also, is there something else involved?
A few hours before I did apt update and apt dist-upgrade on pve02 which apparently didn't upgrade to 8.2.x.

Before today I could access the shell on any other cluster member while logged in on any closter member. (webUI). Now, possibly related, I can't access the shell of pve01 and pve02 if I am logged in on pve03 and vice versa.

I am underlining these as well, because I just started adding the missing ssh pubkey and hit this other problem, and now I am questioning what happened behind the scenes and what can i do to prevent it hapening in the future.

 

[fabian]

I see. Thank you.

Based on this should I infer that adding pve03 being at a different version to the cluster was wrong ?

it can sometimes cause issues like this, yes. it's best to keep nodes at a similar version for this reason.

 

[fabian]

A few hours before I did apt update and apt dist-upgrade on pve02 which apparently didn't upgrade to 8.2.x.

maybe your repository configuration is incomplete? see https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_package_repositories

migration and shell access across node use the same SSH helper, so likely both was broken before between the new node and the old ones and you just didn't notice..

 

[andreisrr]

After further investigations I discovered the following:

• on all hosts I have /etc/pve/priv/known_hosts in the classical format (hostname/IP key type pubkey) containing only pve01 and pve02
• on pve01 and pve02 I have /root/.ssh/known_hosts with 3 entries in a format I haven't seen before, anyway hostnames or IPs are not discernable
• on pve03 there is no /root/.ssh/known_hosts file (odd ...)

Logged in on pve01 trying to access shell on pve03 now yielded like unknown host key, do you want to add it? Afterwards it worked. Not the reverse though.

Note: what has happened comes under part of what I am testing as potential real life incidents during the lifetime of a datacenter. So indeed, solving is high priority.

Note2: I can't upgrade (at least without disk wipe and install from scratch).
According to what's written at https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_package_repositories in my /etc/apt/sources.list things look good. (using local romanian mirror instead of pure debian.org origin).
pve-no-subscription not included, and as I understand it is not a requirement.

I applied all updates that are apparently available. Still pve01 and pve02 are shown to be Virtual Environment 8.1.3. (reboot included)

 

[fabian]

you either need pve-enterprise and a subscription, or pve-no-subscription (or pvetest ) enabled, else you won't get any updates for PVE itself..

 

[andreisrr]

Interesting.
While in WebUI on pve03, I can't see the shell for pve01.

While in a shell on pve03, I did
ssh pve01
It behaved like first time connection to a host, added host key.
Now I can see in WebUI on pve03 the shell for pve01.

 

[andreisrr]

you either need pve-enterprise and a subscription, or pve-no-subscription (or pvetest ) enabled, else you won't get any updates for PVE itself..

Aaaaaa, now I get it.

It worked now! (upgrade part). Onward to the rest.

 

[andreisrr]

Thank you very much for the guidance.