Error Connection error 596 after upgrading to 7.2

[rentel]

Hi everyone!, we are in the process of upgrading our cluster to Proxmox version 7.2 and after upgrading one of the nodes we have encountered this error.

Error Connection error 596: tls_process_server_certificate: certificate verify failed

Curiously I see the status of all virtual machines and LXC, and these are working normally, although I can not perform any operation on these from another node, nor can you perform migrations or other operations involving more than the node from which you are connected. When I try to access a console of an LXC located on another node, the following message is displayed on the console.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:Yhk4KVsxo3BQiUHKxpE55xJCVOQ832fyIzVPFCAARik.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:24
  remove with:
  ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.2.242"
RSA host key for 192.168.2.242 has changed and you have requested strict checking.
Host key verification failed.

The following message also appears constantly in the Proxmox logs.

Jun 17 17:27:14 px1 pvedaemon[11443]: <root@pam> successful auth for user 'root@pam'
Jun 17 17:27:40 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:27:50 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:28:44 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:28:48 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:15 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:51 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:59 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:30:22 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:30:52 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:31:11 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:31:17 px1 pvecm[64413]: got inotify poll request in wrong process - disabling inotify

I have tried to re-generate the certificates unsuccessfully using 'pvecm updatecerts --force', applying it on each node, but it doesn't seem to fix anything. From the interface of each node you can access the machines and execute actions on them, but not operations between nodes.

Cluster information
-------------------
Name:             datarentel
Config Version:   20
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Jun 17 17:43:40 2022
Quorum provider:  corosync_votequorum
Nodes:            9
Node ID:          0x00000005
Ring ID:          1.4d3
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   9
Highest expected: 9
Total votes:      9
Quorum:           5
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 162.162.1.202
0x00000002          1 162.162.1.203
0x00000003          1 162.162.1.239
0x00000004          1 162.162.1.205
0x00000005          1 162.162.1.208 (local)
0x00000006          1 162.162.1.242
0x00000007          1 162.162.1.204
0x00000008          1 162.162.1.233
0x00000009          1 162.162.1.235

 

[rentel]

The 2 nodes that are still on version 6.4 do work correctly with each other, the problem only appears on the nodes that are already upgraded to 7.2.

 

[rentel]

When I came back on Monday I could access all the nodes without problems through the web, it seems to be a temporary problem after the update.

However, there is one machine that I still can't access the console of the containers it hosts from another node, and I get the following error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

WARNING: THE REMOTE HOST ID HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT'S POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping right now (man-in-the-middle attack).

It is also possible that the host key has been changed.

The fingerprint of the RSA key sent by the remote host is

SHA256:Yhk4KVsxo3BQiUHKxpE55xJCVOQ832fyIzVPFCAARik.

Contact your system administrator.

Add the correct host key in /root/.ssh/known_hosts to get rid of this message.

Breaching RSA key in /etc/ssh/ssh_known_hosts:24

  delete with:

  ssh-keygen -f "/etc/ssh/ssh/ssh_known_hosts" -R "192.168.2.242".

The RSA host key for 192.168.2.242 has changed and has requested strict verification.

The host key verification has failed.

Interestingly I can access the console of the node itself, I can also control the machines via the web without problems.

 

[Dragoon]

Hello
The way to fix this issues regenerate all certificates on all nodes (run below command on all nodes):

pvecm updatecerts -F

systemctl restart pvedaemon pveproxy