Losing SSH Keys Arbitrarily

[nexusguy59]

This makes no sense as to why this would happen. First I lost or it said my ssh key on node 3 changed. I couldn't figure out why and got frustrated and just moved the VMs off of it and rebuilt it and moved them back after, everything was fine for about 2 weeks. Then node 5 lost it's ssh key now I am at a loss as to why this is happening just out of the blue and it is really truly very irritating at this point. I shouldn't have to keep rebuilding these nodes. Now if I can't get this fixed I will be forced to go to a commercial product which really sucks for me because Promox has the best footprint. Anyone have any ideas about this problem. This happens when I ssh or migrate from the offending node.


@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:VdwFCpM1Lyzur3E/k1JTh+UFMIKozFqMjUKmEXWQ/Do.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:5
remove with:
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.xxx.xxx.xxx"
RSA host key for 192.xxx.xxx.xxx has changed and you have requested strict checking.
Host key verification failed.

Yes I have tried to remove the key to no avail.

Thanks in advance from a very frustrated user.

 

[t.lamprecht]

Now if I can't get this fixed I will be forced to go to a commercial product which really sucks for me because Promox has the best footprint

First-off, you can also get commercial enterprise support for Proxmox, having enterprise support for still a small footprint: https://www.proxmox.com/en/proxmox-ve/pricing

First I lost or it said my ssh key on node 3 changed. I couldn't figure out why and got frustrated and just moved the VMs off of it and rebuilt it and moved them back after, everything was fine for about 2 weeks. Then node 5 lost it's ssh key now I am at a loss as to why this is happening just out of the blue and it is really truly very irritating at this point. I shouldn't have to keep rebuilding these nodes.

So host-keys are do not re-generated themself, never. Their initially generated at installation and then not touched, at least if a root user, or root running process, does not manually delete or modifies them.

What's your pveversion -v, and how long did this cluster ran before issue happened? That could help to clear if there is some sort of issue in the stack or if other forces are at work.

This message can generally happen if:
* the host key was change, as said that's normally something which needs to be done actively.
* man in the middle attack
* hijacking of the node (then doing the manual change)

You said you rebuilt the node, meaning re-install, I'd guess. And nobody else has root-like or physical access to them?

No extra metric or configuration-management daemons installed?