Error when openning console on other node CT

[rob320]

Now here's a weird one (for me).
I have a 2 node cluster. The nodes are called PVE1 and PVE2. both nodes have the latest Proxmox 8.2.2.
When I log in to PVE1 and try to open the console of a CT in PVE2, I get a red bar at the top of the console window saying "Connection failed (code: 1006)"
and in the console itself :

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:nVVT9f9IyKwZBHXkBf8SG1M2VzLDeRPQtZf8hswqFV4.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:4
  remove with:
  ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.32.5"
Host key for 192.168.32.5 has changed and you have requested strict checking.
Host key verification failed.

This only happens with CTs,
I can open any VM console with no problem.
I can open the PVE2 shell with no problem
I can migrate the a CT to PVE1 and open the console, but if I migrate it back I get the error again.
If I log in to PVE2, I can open any console in PVE1 with no problem.

I looked around in the forums and tried

pvecm updatecerts -F
systemctl restart pvedaemon pveproxy

on both nodes with no luck.

Can anyone help?

 

[Hannes Laimer]

Hey,

looks like 192.168.32.5 was already used previously by at different server/VM/CT and you connected to it using SSH. The fingerprint of that was saved and associated with the IP, now the IP is the same but its fingerprint is a new one, and SSH warns you that the fingerprints don't match. As the message suggests, you can

ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.32.5"

and the next time you connect it'll ask you to trust the new fingerprint.

 

[rob320]

Thanks Hannes.
192.168.32.5 is the address of PVE2.

The thing is that I am able to open the shell of that node with no problem.

The error only happens when trying to open the console of a CT. It happens with all CTs and they give me the exact same ip address.
It does NOT happen when opening the PVE2 shell (that is at IP: 192.168.32.5) or when opening the console of any VM.
Shouldn't I have the same problem with all consoles and shells?

I would like to be double sure because I am afraid of breaking the cluster by using

ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.32.5"

As it is now, this is just an annoyance, I don't want it to become a major headache

 

[bferrell]

I'm getting this same error on all of my 5 nodes now as well, but only in Firefox (on MacOS), Chome does not give it to me. I tried the fix in this other thread, but it does not seem to have helped.

https://forum.proxmox.com/threads/server-disconnected-code-1006.20036/

UPDATE: That's incorrect, it actually seems I had to update ALL my nodes. I can now connect to my consoles, on all the nodes. Strange b/c I didn't change anything this week...

UPDATE2: I can connect to the shell, but the update task fails with the below, but again only in FireFox, Chrome executes the update fine.

"TASK ERROR: command '/usr/bin/termproxy 5900 --path /nodes/svr-01 --perm Sys.Console -- /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=svr-01' -o 'UserKnownHostsFile=/etc/pve/nodes/svr-01/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' -t root@192.168.100.11 -- /usr/bin/pveupgrade --shell' failed: exit code 1"