SSH Key problems for new PVE Node joining existing cluster that had its hostname renamed before joining

[BloodyIron]

So I have a PVE server that I've been working on that I just joined to an existing PVE Cluster. Another person did the initial install of the PVE OS on the system, and I followed the Proxmox VE Documentation for renaming said node ( https://pve.proxmox.com/wiki/Renaming_a_PVE_node ) to the hostname we _actually_ want it to be in the cluster. Did other unrelated config stuff (LACP, etc). Then when we got to the point of joining to this new PVE Node to the existing PVE Cluster a bunch of problems cropped up that were not noticed in planning. This thread is about one in particular.

When I looked at the authorised_keys (for SSH) on /etc/pve/priv/ I see the latest entry is tied to the hostname that was set when the PVE OS was first installed by "Another person" mentioned above, had does not reflect any of the hostname change stuff I did. So I have a hunch a bunch of my problems are caused by SSH key issues.

I tried to look up appropriate steps to fully regenerate keys in this scenario etc, and ran out of time before finding a solution. I am unsure the actual best steps to take here as this scenario does not look to have appropriate documentation.

What exactly _should_ I actually be doing here? "pvecm updatecerts -f" did not seem to resolve the situation, and it is a command I tried when trying to resolve this.

Any help appreciated, thanks!

 

[bbgeek17]

Hi @BloodyIron , you'd have to add more technical facts for anyone to be able to point you in a specific direction.

You didn't mention in your post what error you ran into.

You stated that authorised_keys refers to a prior hostname. However, the last part of a key line is a comment and is not used for anything.

Things you should check:
context of /etc/hostname on each node
context of /etc/hosts on each node
context of /etc/network/interfaces on each node
ping by hostname from each node to another node
ssh by IP and hostname on each node to another node

That's just the start. It sounds like more than one person worked on this host over an extended period. Since you can't join a non-empty node to the cluster, why not just reinstall the new node without any baggage?

Good luck

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[BloodyIron]

/etc/hostname was updated to the correct hostname when I renamed the node
/etc/hosts was updated to the correct hostname when I renamed the node, and was not automatically populated with the other PVE Nodes when it joined the cluster (I guess that's not a thing though)
/etc/network/interfaces what information exactly do you want to know there?

To be clear a lot of this contains sensitive information so I'm not blind-pasting the contents of this specific node, let alone the contents for these files for the whole cluster.

I was able to get the new node to a state of being able to ping, by hostname, to each node, and other nodes back to the new node. SSH I ran out of time before testing that comprehensively but some levels of success there.

And there's only been one person working on this system this whole time... EXCEPT for when the OS was initially installed. That installation was handled by another person, but I reviewed how it was performed and apart from the hostname not being the desired value, no problems stuck out with me.

 

[esi_y]

When I looked at the authorised_keys (for SSH) on /etc/pve/priv/ I see the latest entry is tied to the hostname that was set when the PVE OS was first installed by "Another person" mentioned above, had does not reflect any of the hostname change stuff I did.

This is a known bug, benign though. You will be left with skeletons from dead nodes unless you manually prune it, however ...

So I have a hunch a bunch of my problems are caused by SSH key issues.

Is anything to that effect in the logs at all?

 

[esi_y]

You didn't mention in your post what error you ran into.

It would really help to answer this question directly. What exactly is not working as expected?

 

[BloodyIron]

Look I have a lot of different errors here that I'm working through and I really would appreciate just having the instructions needed to properly cycle the SSH Keys for a single node to the rest of the cluster. I'm asking because I could not find this sufficiently outlined in the Proxmox VE Documentation. I can appreciate that it's typical to share more insights into what errors I'm having but right now I really am just looking for the correct steps I need to take for this function, and really cannot get into more details at this time. If anyone can actually tell me that information it would be appreciated, otherwise, I'm not sharing logs or errors I'm seeing at this time.

 

[BloodyIron]

Alright well the issues I've been facing were MTU related (but not in my realm as the path between nodes I'm not responsible for looks to have been messing with MTUs). Still would be nice to have documentation outline SSH key cycling for PVE... Thanks anyways folks.

 

[pveuser113]

thanks for the update, after sending everyone on a wild goose chase...
You must be your helpdesk team's favorite person to hear from...
Happy it wasnt your fault/realm...

 

[esi_y]

Look I have a lot of different errors here

You did not provide sufficient information to even reply your "ticket".

I really would appreciate just having the instructions needed to properly cycle the SSH Keys for a single node to the rest of the cluster.

Like on any other Debian.

I'm asking because I could not find this sufficiently outlined in the Proxmox VE Documentation.

See above.

otherwise, I'm not sharing logs or errors I'm seeing at this time.

Had you mentioned this in your original post, you would have saved everyone a lot of effort.

 

[BloodyIron]

You did not provide sufficient information to even reply your "ticket".



Like on any other Debian.



See above.



Had you mentioned this in your original post, you would have saved everyone a lot of effort.

Are you just married to trying to lecture me? I'm going to ignore you because you seem more fixated on me fitting into your box of "help" than trying to actually get anything accomplished done. Go away troll, I have actual work to do, not placate your pedantic ego stroking.

 

[BloodyIron]

thanks for the update, after sending everyone on a wild goose chase...
You must be your helpdesk team's favorite person to hear from...
Happy it wasnt your fault/realm...

Feel free to move along and not even post if you don't see value in participating. I was simply asking for the proper process to do something that wasn't documented. And you're here to do... what exactly? Lecture me because I didn't say exactly what you wanted to hear? Why are you even bothering if this is what you're going to say? If you don't know the process, then say nothing. Lecturing me on this bs helps nobody, especially yourself.

 

[esi_y]

Are you just married to trying to lecture me? I'm going to ignore you because you seem more fixated on me fitting into your box of "help" than trying to actually get anything accomplished done. Go away troll, I have actual work to do, not placate your pedantic ego stroking.

I do not understand why you had started the post the way you did. If you really did not want to share anything, just wanted to know how to rotate a key on Debian, you might have skipped all the additional information.

Naturally, people want to help with the reported error, your hunch was very obviously wrong and the troubleshooting would have continued faster - consider nothing of what you had provided would have allowed anyone to help you find it was MTU.

Also note I had not asked for entire log, it is certainly possible to sanitize e.g. ssh -vvv output.

I understand you may not appreciate the follow-up, but others who will find this thread later and SSH & MTU topic might. They would have also benefited with more information provided.

As for key rotation: https://www.ssh.com/academy/ssh/keygen