Proxmox 7.4 RSA Fingerprint Keeps Changing in cluster after upgrade

[BeDazzler]

Hi All,

I have a cluster of 5x PVE 7.4 nodes which we recently upgraded onto new hardware.

We did this host by host and it all worked fine, however every time I try to remote console (NoVNC) I get "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! "

This is likely due to old host keys still being present in /etc/ssh/ssh_known_hosts.

I've removed them one by one and moved to each host to update certs, however I am still receiving the same message.

Is there something else I need to do ?

Many thanks.

 

[shanreich]

It's quite likely that there are still leftovers of the key in /root/.ssh/known_hosts as well

 

[BeDazzler]

It's quite likely that there are still leftovers of the key in /root/.ssh/known_hosts as well

Sure, I've looked in there and I do see entries for each host using it's IP and name.

Which ones do I remove ??

Can I just remove all and run updatecerts to refresh them all ?

 

[shanreich]

Sure, I've looked in there and I do see entries for each host using it's IP and name.

Interesting, on my machine the entries are hashed which makes this a bit tricky usually.

In any case you should be able to remove the entries via the following command:

ssh-keygen -f /root/.ssh/known_hosts -R <host>

You should remove the hostname as well as the IP(s) of all hosts.

 

[BeDazzler]

Interesting, on my machine the entries are hashed which makes this a bit tricky usually.

In any case you should be able to remove the entries via the following command:

ssh-keygen -f /root/.ssh/known_hosts -R <host>

You should remove the hostname as well as the IP(s) of all hosts.

I see, I was only removing the host name. I will try again and also remove the IPs. Thanks for your reply.

 

[tempacc375924]

@BeDazzler See https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-2#post-606349

 

[BeDazzler]

@BeDazzler See https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-2#post-606349

Thank you. Yes I knew the symlink was broken/being overwritten with a local file. As a short term workaround I added 'StrictHostKeyChecking no' to each ssh config file since my PVEs are running on their own internal network. I will have a look at this patch.

 

[tempacc375924]

Thank you. Yes I knew the symlink was broken/being overwritten with a local file. As a short term workaround I added 'StrictHostKeyChecking no' to each ssh config file since my PVEs are running on their own internal network. I will have a look at this patch.

No worries! I actually encountered the same and found quite a few posts on the forum, so wondered if it was boiling down to one and the same. The symlink is actually not really much of a problem, after running ssh-keygen -R on the /etc located file, you may (have also in the past) run the pvecm udpatecerts (just like that, the -f really only applies to SSL certificates).

The patch is for the updatecerts so that it does not go on dropping out keys - which is why you were getting the MITM warnings. You get the warning when NONE of the keys in none of the known_hosts match. This means you had some old keys there, but the new were missing. Of course, after manually removing all of them (as some people did), you could just accept the prompt (when the StrictHostKeyChecking is the default ask) and happy days, but then surprisingly (also depending on which node it was done from) the updatecerts might have broken it yet again.

This applies to anyone whether they ever called a new node the same like dead (previously correctly removed one) or they had simply more keys under the same alias in the known_hosts - it's not like the alias has to be fqdn. Glad someone actually really experiencing this got help with this (or will, if you want to wait for them to okay it and backport), but it's really cosmetic.

 

[BeDazzler]

No worries! I actually encountered the same and found quite a few posts on the forum, so wondered if it was boiling down to one and the same. The symlink is actually not really much of a problem, after running ssh-keygen -R on the /etc located file, you may (have also in the past) run the pvecm udpatecerts (just like that, the -f really only applies to SSL certificates).

The patch is for the updatecerts so that it does not go on dropping out keys - which is why you were getting the MITM warnings. You get the warning when NONE of the keys in none of the known_hosts match. This means you had some old keys there, but the new were missing. Of course, after manually removing all of them (as some people did), you could just accept the prompt (when the StrickhostKeyChecking is the default ask) and happy days, but then surprisingly (also depending on which node it was done) the updatecerts might have broken it yet again.

This applies to anyone whether they ever called a new node the same like dead (previously correctly removed one) or they had simply more keys under the same alias in the known_hosts - it's not like the alias has to be fqdn. Glad someone actually really experiencing this got help with this (or will, if you want to wait for them to okay it and backport), but it's really cosmetic.

Yes I just read through the bugfix correspondence, very interesting communications in there for sure.
Yes, I also found the issue whilst upgrading infrastructure and worked out that old keys were appearing despite removing and re-generating.
In the end it became annoying and such a waste of time (with no resolution) since the symlinked file is required for the correct information to be shared across PVE hosts, hence disabling checks was the easiest way forward and stopped the complaints from admins.

 

[tempacc375924]

Yes I just read through the bugfix correspondence, very interesting communications in there for sure.
Yes, I also found the issue whilst upgrading infrastructure and worked out that old keys were appearing despite removing and re-generating.
In the end it became annoying and such a waste of time (with no resolution) since the symlinked file is required for the correct information to be shared across PVE hosts, hence disabling checks was the easiest way forward and stopped the complaints from admins.

I would totally consider deploying SSH certificates for the cluster (and another for VMs but that's out of scope here), which also scales well. It basically takes just a CA signing the host keys and then whatever is signed is accepted by the known_hosts (@ca *.fqdn.tld key ... etc entry, just one for each host and well - they are shared as of now, so one, but it's easy to drop in when joining the cluster.) CAs can have long-term keys and if you operate it on a secure network it's not high risk to have them very long-term). Because otherwise yeah, complaints make people do workarounds that are ... sub-optimal so to say.

I am really glad it helped another person!

 

[esi_y]

https://forum.proxmox.com/threads/s...s-how-to-bypass-ssh-known_hosts-bug-s.137809/