Error Connection error 596 after upgrading to 7.2

rentel

New Member
Nov 13, 2021
4
1
3
38
Hi everyone!, we are in the process of upgrading our cluster to Proxmox version 7.2 and after upgrading one of the nodes we have encountered this error.

Code:
Error Connection error 596: tls_process_server_certificate: certificate verify failed

Curiously I see the status of all virtual machines and LXC, and these are working normally, although I can not perform any operation on these from another node, nor can you perform migrations or other operations involving more than the node from which you are connected. When I try to access a console of an LXC located on another node, the following message is displayed on the console.

Code:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:Yhk4KVsxo3BQiUHKxpE55xJCVOQ832fyIzVPFCAARik.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:24
  remove with:
  ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "192.168.2.242"
RSA host key for 192.168.2.242 has changed and you have requested strict checking.
Host key verification failed.

The following message also appears constantly in the Proxmox logs.

Code:
Jun 17 17:27:14 px1 pvedaemon[11443]: <root@pam> successful auth for user 'root@pam'
Jun 17 17:27:40 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:27:50 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:28:44 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:28:48 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:15 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:51 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:29:59 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:30:22 px1 pveproxy[11445]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:30:52 px1 pveproxy[11444]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:31:11 px1 pveproxy[11446]: Could not verify remote node certificate '20:82:BB:C8:71:DE:59:DC:58:33:6E:4C:AE:AB:05:4E:E0:73:B5:E6:B4:8A:29:65:19:38:02:B4:BA:9E:8D:99' with list of pinned certificates, refreshing cache
Jun 17 17:31:17 px1 pvecm[64413]: got inotify poll request in wrong process - disabling inotify

I have tried to re-generate the certificates unsuccessfully using 'pvecm updatecerts --force', applying it on each node, but it doesn't seem to fix anything. From the interface of each node you can access the machines and execute actions on them, but not operations between nodes.

Code:
Cluster information
-------------------
Name:             datarentel
Config Version:   20
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Jun 17 17:43:40 2022
Quorum provider:  corosync_votequorum
Nodes:            9
Node ID:          0x00000005
Ring ID:          1.4d3
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   9
Highest expected: 9
Total votes:      9
Quorum:           5
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 162.162.1.202
0x00000002          1 162.162.1.203
0x00000003          1 162.162.1.239
0x00000004          1 162.162.1.205
0x00000005          1 162.162.1.208 (local)
0x00000006          1 162.162.1.242
0x00000007          1 162.162.1.204
0x00000008          1 162.162.1.233
0x00000009          1 162.162.1.235
 
The 2 nodes that are still on version 6.4 do work correctly with each other, the problem only appears on the nodes that are already upgraded to 7.2.
 
When I came back on Monday I could access all the nodes without problems through the web, it seems to be a temporary problem after the update.

However, there is one machine that I still can't access the console of the containers it hosts from another node, and I get the following error:

Code:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

WARNING: THE REMOTE HOST ID HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT'S POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping right now (man-in-the-middle attack).

It is also possible that the host key has been changed.

The fingerprint of the RSA key sent by the remote host is

SHA256:Yhk4KVsxo3BQiUHKxpE55xJCVOQ832fyIzVPFCAARik.

Contact your system administrator.

Add the correct host key in /root/.ssh/known_hosts to get rid of this message.

Breaching RSA key in /etc/ssh/ssh_known_hosts:24

  delete with:

  ssh-keygen -f "/etc/ssh/ssh/ssh_known_hosts" -R "192.168.2.242".

The RSA host key for 192.168.2.242 has changed and has requested strict verification.

The host key verification has failed.

Interestingly I can access the console of the node itself, I can also control the machines via the web without problems.
 
  • Like
Reactions: meichthys

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!