[SOLVED] Safely changing ssh keys

Proximate

Member
Feb 13, 2022
219
11
23
64
Solved: https://forum.proxmox.com/posts/606419/

Some time back, I upgraded my hosts to 8.0.3 and all seemed to be fine.
All of sudden, I'm not able to migrate a host and keep getting the ssh key warning.
I logged into both hosts that I wanted to migrate between and tried to ssh from one to the other and was told I had to delete the ssh key.
I did that but migration still doesn't work.

I'm able to ssh between hosts without needing password but cannot migrate.
I think there are two ssh keys in play no? One for ssh'ing and the other for the cluster?
How do I fix this without breaking my cluster?

It would be nice if there was some sort of option to help with this since it's something I've seen posted many times but it's never clear if I'll break the cluster in those posts so have to ask the same question again.
 
Last edited:
Hi,
please try to run pvecm updatecert -f. If that does not solve your issue, check the entries in /etc/pve/priv/authorized_keys and /etc/pve/priv/known_hosts.
 
Hi,

Thanks for your reply. I'm kind of stuck until I solve this :).

I ran this on every host. Then I tried to migrate again and got this;

1693591066160.png
Then, logged into the first host, I clicked on another host and got this;

1693591198375.png

So now it seems it's even more broken.
 
Last edited:
Sorry, I updated my previous comment several times.
Cluster seems fine, from the command line.

1693591381628.png
1693591436500.png
 
I re-ran the command on just one host and still seeing the same. Now I'm getting a certificate failed when clicking on any other node than the one I'm logged into (GUI). The cli command shows the cluster is up and running with quorum so really not sure what to do now.
 
On host 01, I see multiple keys for the same hosts in
/etc/pve/priv/authorized_keys but in /etc/pve/priv/known_hosts, only one of each.
Same on all of the hosts in the cluster.

Is it safe to remove the multiples in the first file and will that solve the issue?
 
Last edited:
Without changing anything in the certs files, I ran the following;
pvecm updatecert -f
systemctl stop pve-cluster
systemctl start pve-cluster

No change.

I've been reading the link you gave me but it's hard to know what to do/try without taking chances of breaking something.
I don't have enough experience with proxmox to know what I should be doing at this point.

I found this but I'm nervous about trying this solution, especially having to restart a host. Since you would be restarting the services, why would you need to reboot?

https://codingpackets.com/blog/proxmox-certificate-error-fix-after-node-replacement/
 
Last edited:
I tried some of the above post without rebooting hosts;

# pvecm updatecert -f
# systemctl restart pvedaemon pveproxy pve-cluster

No change. No idea how to get past this after a full day of reading so far.

The doc says if you unattach a node, you can run pvecm updatecerts to update its certs when re-attaching. No idea how safe this is in my case.

I wish this seemingly very common problem could be solved through the GUI. One click, all done for those of us who aren't full experts yet.
 
Last edited:
Hi,
so it seems that you did not flush the browser cache and accept the self-signed cert after running the pvecm updatecerts, therefore your browser did not accept any connection/API requests.

I rebooted the two hosts I wanted to migrate between and now everything works again.
Glad to hear that everything is working as expected now.
 
Actually, I used control-F5 multiple times to refresh the browser every time I did anything with the certs.
So, I can only guess but it seems the reboot is what finally did the trick?
If that's the case, it's too bad to even have to reboot a host, ever, unless there is something physically wrong with it.
 
Last edited:
Months later and this comes back to haunt me again.
I'm able to log into any host, control all hosts from there but cannot migrate or do anything else that is inter-host communications.

I thought this was solved, but it's back again to haunt me. I wish the devs would consider some sort of re-sync option where you can click on it from any one host you've authorized yourself on and immediately sync all hosts ssh keys again.

I really don't know what to do now that won't break everything again.
I started another thread about it.

https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-2
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!