[SOLVED] One Proxmox node can't reach the other one via SSH

[hohl]

It seems one Proxmox node can't reach the other one via SSH. That blocks features such as the VNC console (on the other node in the same cluster) and live migrations. For example:

root@5.255.77.104: Permission denied (publickey).

TASK ERROR: Failed to run vncproxy.

I've found a few seemingly similar post here on the forum and there it always was fixed by running `pvecm updatecerts`. I've tried so multiple times on both nodes and it each time seems to succeed:

root@pve-node-1:~# pvecm updatecerts
(re)generate node files
merge authorized SSH keys and known hosts

However, in my case this doesn't seem to fix the issue. When I try to connect:

root@pve-node-1:~# ssh pve-node-2.full.tld
Warning: the ECDSA host key for 'pve-node-2.full.tld' differs from the key for the IP address '1.2.3.4'
Offending key for IP in /etc/ssh/ssh_known_hosts:3
Matching host key in /root/.ssh/known_hosts:2
Are you sure you want to continue connecting (yes/no)? yes
Permission denied (publickey).

`/var/log/auth.log` only says:

Jun 14 18:30:02 pve-nl-2 sshd[50918]: ROOT LOGIN REFUSED FROM X.X.X.X port 35790
Jun 14 18:30:02 pve-nl-2 sshd[50918]: ROOT LOGIN REFUSED FROM X.X.X.X port 35790 [preauth]
Jun 14 18:30:02 pve-nl-2 sshd[50918]: Connection closed by authenticating user root X.X.X.X port 35790 [preauth]

`/etc/pve/priv/authorized_keys` also shows same keys on both nodes and these are exactly those that are inside the `/root/.ssh/id_rsa.pub`. And btw. `root@pve-node-1:~# ssh pve-node-1.full.tld` also does yield a permission denied error. And `/root/.ssh/authorized_keys` is a symlink:

root@pve-node-1:~# ls -l /root/.ssh/authorized_keys
lrwxrwxrwx 1 root root 29 Aug  5  2020 /root/.ssh/authorized_keys -> /etc/pve/priv/authorized_keys

Seems all perfectly fine? Help.

Any ideas what else I can try so that they each other can reach themselves again? I can use my administrative user to connect from my local workstation to both and everything else seems to work too. Just the two nodes each other disagreeing on their keys. Also tried rebooting the nodes. Didn't change anything.

EDIT: Even tried to manually recreate the keys, but didn't help although it successfully exchanges the keys via the `pvecm updatecerts` (so I reverted to the old keys).

EDIT2: That's the output of `ssh -vv pve-node-2.domain.tld`:

(...)
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:nC7CBXy3WfGeYn8pXyGFsnka757qZjGjU3ok0K/ZOck
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /root/.ssh/id_rsa RSA SHA256:nC7CBXy3WfGeYn8pXyGFsnka757qZjGjU3ok0K/ZOck
debug1: Authentications that can continue: publickey
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@pve-node-2: Permission denied (publickey).

 

[hohl]

Oh no, I found it. Recently introduced some SSH hardening policy which is enforced via automation on all Linux machine and that policy includes `PermitRootLogin no`. Since it's automatically applied to all Linux machines, it was also applied to the Proxmox hypervisors. If anybody else stumbles over this issue in the future: check you `/etc/ssh/ssh_config`.