WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED

[nekton]

I recently have been getting this message from the console of 2 of my 3 node cluster. The node I created the cluster from has not displayed this message yet.

I have made two changes to my cluster. I removed one of the nodes and then added it back in. And I added the Proxmox Backup Server to the mix.

My cluster is behind a firewall and not available to the public so I don't know why it is suggesting a man-in-the-middle attack.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:<SHA256 code>.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:6
  remove with:
  ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.201.21"
RSA host key for 192.168.201.21 has changed and you have requested strict checking.
Host key verification failed.

The node this is happening on in this case is the 192.168.201.21 node. I can't tell from this message which is the local host and which is the remote host. Why would the message about the node's own RSA key be flagged as suspect? Again, this happens on two of the three nodes in the cluster. Restarting the node removes the message.

I looked at other threads and someone suggested that running this command would help. I don't see why though.

pvecm updatecerts

That doesn't seem to have helped with the problem. One more thing. After I reboot the node, I might not see the issue until 24 hours passes. And it only happens on one node at a time.

Any suggestions would be welcome. I haven't a clue what to try.

 

[leesteken]

The system that gives you the warning has a key stored for the remote system (192.169.201.21) but notices that the remote system at 192.168.201.21 now has a different key. Maybe some systems changed their IP address and/or some systems changed their key. You cannot tell whether this is accidental or malice just from this.
It's up to you to determine whether 192.168.201.21 is the system you actually want to connect to and whether to trust it (by removing the local copy of the old key using ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.201.21"). SSH cannot determine it is the same system as before automatically.

 

[reto]

Today I had the same issue, all vms (from two pve hosts) seemed to have changed their host key after a host reboot. :/

reto@carpo:~$ sudo journalctl --list-boots 
-2 04bfdaec3fff4559a2957d5fa74af48a Fri 2022-05-13 14:41:49 CEST—Wed 2022-05-25 06:17:35 CEST
-1 1e4ade28c95c47a388c52680ffd874d1 Wed 2022-05-25 06:17:41 CEST—Thu 2022-06-02 14:44:36 CEST
 0 7d5e0e09f155459c8980be7cbcb15b02 Fri 2022-06-03 00:00:21 CEST—Fri 2022-06-03 00:02:11 CEST
reto@carpo:~$ sudo journalctl  | grep 'Generating public/private rsa key pair'
May 13 14:42:07 carpo cloud-init[571]: Generating public/private rsa key pair.
Jun 03 00:00:38 carpo cloud-init[398]: Generating public/private rsa key pair.

I found this other forum post: https://forum.proxmox.com/threads/changing-instance-id-cloud-init.101012/

I compared my qm config vs a backup from yesterday night. No changes for vmgenid and smbios1. Weird.


The situation occured after an apt dist-upgrade on the host and a restart afterwards.

-2 628808a5d586476ab64e7203b64fbd1f Fri 2022-05-06 16:13:54 CEST—Thu 2022-06-02 14:46:18 CEST
2022-05-12 22:55:04 upgrade libproxmox-rs-perl:amd64 0.1.0 0.1.1
2022-06-02 14:42:18 upgrade dpkg:amd64 1.20.9 1.20.10
2022-06-02 14:42:18 upgrade rsyslog:amd64 8.2102.0-2 8.2102.0-2+deb11u1
2022-06-02 14:42:18 upgrade libssl1.1:amd64 1.1.1n-0+deb11u1 1.1.1n-0+deb11u2
2022-06-02 14:42:19 upgrade libcups2:amd64 2.3.3op2-3+deb11u1 2.3.3op2-3+deb11u2
2022-06-02 14:42:19 upgrade libldap-2.4-2:amd64 2.4.57+dfsg-3 2.4.57+dfsg-3+deb11u1
2022-06-02 14:42:19 upgrade libproxmox-backup-qemu0:amd64 1.2.0-1 1.3.1-1
2022-06-02 14:42:19 upgrade libpve-common-perl:all 7.1-6 7.2-2
2022-06-02 14:42:19 upgrade libpve-access-control:all 7.1-8 7.2-1
2022-06-02 14:42:19 upgrade proxmox-backup-client:amd64 2.1.8-1 2.2.1-1
2022-06-02 14:42:19 upgrade proxmox-backup-file-restore:amd64 2.1.8-1 2.2.1-1
2022-06-02 14:42:20 upgrade libpve-http-server-perl:all 4.1-1 4.1-2
2022-06-02 14:42:20 upgrade libpve-storage-perl:all 7.2-2 7.2-4
2022-06-02 14:42:20 upgrade libxml2:amd64 2.9.10+dfsg-6.7+deb11u1 2.9.10+dfsg-6.7+deb11u2
2022-06-02 14:42:20 upgrade needrestart:all 3.5-4 3.5-4+deb11u1
2022-06-02 14:42:20 upgrade openssl:amd64 1.1.1n-0+deb11u1 1.1.1n-0+deb11u2
2022-06-02 14:42:20 upgrade proxmox-widget-toolkit:all 3.4-10 3.5.1
2022-06-02 14:42:20 upgrade pve-i18n:all 2.7-1 2.7-2
2022-06-02 14:42:20 upgrade pve-kernel-5.15.35-1-pve:amd64 5.15.35-2 5.15.35-3
2022-06-02 14:42:26 upgrade pve-qemu-kvm:amd64 6.2.0-5 6.2.0-8
2022-06-02 14:42:28 upgrade qemu-server:amd64 7.2-2 7.2-3
2022-06-02 14:42:28 upgrade pve-manager:amd64 7.2-3 7.2-4
 -1 c18036a108654fbb8afd1c07549f94c7 Thu 2022-06-02 14:46:46 CEST—Fri 2022-06-03 00:37:45 CEST
  0 03732e03fcc541fcb112d05f6807c72d Fri 2022-06-03 00:38:13 CEST—Fri 2022-06-03 00:39:16 CEST

[code]

 

[fabian]

seems like you accidentally installed cloud-init on the PVE system itself? it's only meant to be installed inside the guest..

 

[reto]

@fabian

nope, just the VMs have cloud-init installed.

# on one of the  vms
ii  cloud-init     22.1-14-g2e17a0d6-0ubuntu1~22.04.5 all          initialization and customization tool for cloud instances

# on the host
un  cloud-init     <none>       <none>       (no description available)

here the timeline again a little

• 06-02 14:42 First I run an apt dist-upgrade on host (hostname little-foot) (see dpkg log) in previous posts
• 06-02 14:46 reboot host
• 06-02 14:47 vm (hostname metis) wakes up and regenerates ssh key, see log section below

metis logs

# first boot of metis

-21 0a41ecc56a1744f6b33df8468df2cc58 Sun 2022-01-23 22:27:45 CET—Sun 2022-01-23 22:42:43 CET
Jan 23 22:27:50 metis cloud-init[601]: Generating public/private rsa key pair.

( ... )

[ many more boots, no ssh key regenerated]

( ... )

 0 dfeb7e3a42da44aab21bbab1c58485ca Thu 2022-06-02 14:47:01 CEST—Thu 2022-06-02 19:12:33 CEST

Jun 02 14:47:05 metis cloud-init[430]: Generating public/private rsa key pair.

 

[fabian]

well, the logs say the key is re-generated by cloud init, so at least at that point, it must have been installed

 

[reto]

@fabian Ive extended my answer, I can provide more logs if you like

 

[fabian]

ah, you wrote "you had the same issue", but you actually don't - the original issue is about cluster nodes changing their SSH keys, not VMs.. if you reboot metis once more, does it regenerate the key again?

 

[reto]

@fabian sorry, yes, the link to the other issue is probably misleading (it was late)

and yes, I tried both restarting the host AND the guest, and in both cases the ssh key was NOT regenerated. So the issue it self seems to be 'solved', but I'm really wondering what caused it.

 

[reto]

@fabian all my ubuntu 22.04 and 20.04 vms were affected, but not my single debian host

 

[fabian]

if you are sure the config(s) didn't change, the next step would be to correlate both in-VM upgrades and host upgrades and reboots.

 

[reto]

@fabian i've installed 7.1, set everything up, did the upgrade to pve 7.2 (plus/minus as yesterday) and did a reboot. But no luck, no host key regeneration...

Very strange, but at this point I just wait for it to happen again (or you have more affected users).

But thanks anyway!

 

[nekton]

The system that gives you the warning has a key stored for the remote system (192.169.201.21) but notices that the remote system at 192.168.201.21 now has a different key. Maybe some systems changed their IP address and/or some systems changed their key. You cannot tell whether this is accidental or malice just from this.
It's up to you to determine whether 192.168.201.21 is the system you actually want to connect to and whether to trust it (by removing the local copy of the old key using ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.201.21"). SSH cannot determine it is the same system as before automatically.

Sorry for the delay in the follow up. I removed the key as the command requested. Next time I logged on to that node through the console a new key was generated and all seems well with the cluster. Thanks for your help @leesteken !

 

[reto]

@fabian it happened again, this time after restarting some vms on the node.

I think I will just remove cloud-init after the initial boot, this is too annyoing.

 

[Dragoon]

Hello
The way to fix this issues regenerate all certificates on all nodes (run below command on all nodes):

pvecm updatecerts -F

systemctl restart pvedaemon pveproxy

 

[simonjan]

Hello
The way to fix this issues regenerate all certificates on all nodes (run below command on all nodes):

pvecm updatecerts -F

systemctl restart pvedaemon pveproxy

Thanks, That's solved the problem.

 

[esi_y]

Very old bug: https://forum.proxmox.com/threads/s...ass-ssh-known_hosts-bug-s.137809/#post-640203

 

[benka]

For others having the "REMOTE HOST IDENTIFICATION HAS CHANGED" message. What I've done to solve that problem:

If the IP address of new node was used before in the cluster or the node was set up again run on all nodes of the cluster:

1. Remove host key from known_hosts files:
   

   ssh-keygen -f /etc/ssh/ssh_known_hosts -R "node"
   ssh-keygen -f /root/.ssh/known_hosts -R "node"
   ssh-keygen -f /etc/pve/priv/known_hosts -R "node"

2. Add key again:
   

   /usr/bin/ssh -e none -o 'HostKeyAlias=node' root@node /bin/true

3. Update node certificates
   

   pvecm updatecerts -F

 

[esi_y]

For others having the "REMOTE HOST IDENTIFICATION HAS CHANGED" message. What I've done to solve that problem:

If the IP address of new node was used before in the cluster or the node was set up again run on all nodes of the cluster:

1. Remove host key from known_hosts files:
   

   ssh-keygen -f /etc/ssh/ssh_known_hosts -R "node"
   ssh-keygen -f /root/.ssh/known_hosts -R "node"
   ssh-keygen -f /etc/pve/priv/known_hosts -R "node"

So this is tantamount to destroying the file:
https://bugzilla.proxmox.com/show_bug.cgi?id=4252

1. Add key again:
   

   /usr/bin/ssh -e none -o 'HostKeyAlias=node' root@node /bin/true

This does not work as intended.

1. Update node certificates
   

   pvecm updatecerts -F

This restores the symlink.

It's all wrong.