[SOLVED] Safely changing ssh keys

[Proximate]

Solved: https://forum.proxmox.com/posts/606419/

Some time back, I upgraded my hosts to 8.0.3 and all seemed to be fine.
All of sudden, I'm not able to migrate a host and keep getting the ssh key warning.
I logged into both hosts that I wanted to migrate between and tried to ssh from one to the other and was told I had to delete the ssh key.
I did that but migration still doesn't work.

I'm able to ssh between hosts without needing password but cannot migrate.
I think there are two ssh keys in play no? One for ssh'ing and the other for the cluster?
How do I fix this without breaking my cluster?

It would be nice if there was some sort of option to help with this since it's something I've seen posted many times but it's never clear if I'll break the cluster in those posts so have to ask the same question again.

 

[Chris]

Hi,
please try to run pvecm updatecert -f. If that does not solve your issue, check the entries in /etc/pve/priv/authorized_keys and /etc/pve/priv/known_hosts.

 

[Proximate]

Hi,

Thanks for your reply. I'm kind of stuck until I solve this .

I ran this on every host. Then I tried to migrate again and got this;


Then, logged into the first host, I clicked on another host and got this;



So now it seems it's even more broken.

 

[Chris]

Hi,

Thanks for your reply. I'm kind of stuck until I solve this .

Should I run this on all hosts then or just one?

Running this on one host should be enough, as long as your pmxcfs is running and your cluster is quorate (which you can quickly check via pvecm status). See also the docs, if you are not familiar with the cluster filesystem https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pmxcfs

 

[Proximate]

Sorry, I updated my previous comment several times.
Cluster seems fine, from the command line.

 

[Proximate]

I re-ran the command on just one host and still seeing the same. Now I'm getting a certificate failed when clicking on any other node than the one I'm logged into (GUI). The cli command shows the cluster is up and running with quorum so really not sure what to do now.

 

[Proximate]

On host 01, I see multiple keys for the same hosts in
/etc/pve/priv/authorized_keys but in /etc/pve/priv/known_hosts, only one of each.
Same on all of the hosts in the cluster.

Is it safe to remove the multiples in the first file and will that solve the issue?

 

[Proximate]

Without changing anything in the certs files, I ran the following;
pvecm updatecert -f
systemctl stop pve-cluster
systemctl start pve-cluster

No change.

I've been reading the link you gave me but it's hard to know what to do/try without taking chances of breaking something.
I don't have enough experience with proxmox to know what I should be doing at this point.

I found this but I'm nervous about trying this solution, especially having to restart a host. Since you would be restarting the services, why would you need to reboot?

https://codingpackets.com/blog/proxmox-certificate-error-fix-after-node-replacement/

 

[Proximate]

I tried some of the above post without rebooting hosts;

# pvecm updatecert -f
# systemctl restart pvedaemon pveproxy pve-cluster

No change. No idea how to get past this after a full day of reading so far.

The doc says if you unattach a node, you can run pvecm updatecerts to update its certs when re-attaching. No idea how safe this is in my case.

I wish this seemingly very common problem could be solved through the GUI. One click, all done for those of us who aren't full experts yet.

 

[Proximate]

I rebooted the two hosts I wanted to migrate between and now everything works again.

 

[Chris]

Hi,
so it seems that you did not flush the browser cache and accept the self-signed cert after running the pvecm updatecerts, therefore your browser did not accept any connection/API requests.

I rebooted the two hosts I wanted to migrate between and now everything works again.

Glad to hear that everything is working as expected now.

 

[Proximate]

Actually, I used control-F5 multiple times to refresh the browser every time I did anything with the certs.
So, I can only guess but it seems the reboot is what finally did the trick?
If that's the case, it's too bad to even have to reboot a host, ever, unless there is something physically wrong with it.

 

[Proximate]

Months later and this comes back to haunt me again.
I'm able to log into any host, control all hosts from there but cannot migrate or do anything else that is inter-host communications.

I thought this was solved, but it's back again to haunt me. I wish the devs would consider some sort of re-sync option where you can click on it from any one host you've authorized yourself on and immediately sync all hosts ssh keys again.

I really don't know what to do now that won't break everything again.
I started another thread about it.

https://forum.proxmox.com/threads/pvecm-updatecert-f-not-working.135812/page-2

 

[Proximate]

Solved
https://forum.proxmox.com/posts/606419/