[SOLVED] proxy detected vanished client connection

[EvertM]

Hi all,

We have a 2-node cluster, down from 3. On node2 all is fine, but on node1 the log keeps filling with messages like these:

Feb 28 14:54:17 node1 pveproxy[1057325]: Clearing outdated entries from certificate cache
Feb 28 14:54:29 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:54:33 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:54:36 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:54:40 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:55:29 node1 pveproxy[983812]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:55:33 node1 pveproxy[983812]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:55:36 node1 pveproxy[983812]: proxy detected vanished client connection
Feb 28 14:55:36 node1 pveproxy[983812]: proxy detected vanished client connection
Feb 28 14:56:29 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:56:33 node1 pveproxy[1057325]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:56:36 node1 pveproxy[1057325]: proxy detected vanished client connection
Feb 28 14:56:36 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:57:27 node1 pvedaenode3[755194]: <root@pam> successful auth for user 'root@pam'
Feb 28 14:57:29 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:57:33 node1 pveproxy[1057325]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:57:36 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:57:36 node1 pveproxy[1057325]: proxy detected vanished client connection
Feb 28 14:58:29 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:58:33 node1 pveproxy[1017690]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:58:36 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:58:36 node1 pveproxy[1017690]: proxy detected vanished client connection
Feb 28 14:59:29 node1 pveproxy[1057325]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:59:33 node1 pveproxy[1057325]: '/etc/pve/nodes/node3/pve-ssl.pem' does not exist!
Feb 28 14:59:37 node1 pveproxy[1057325]: proxy detected vanished client connection
Feb 28 14:59:40 node1 pveproxy[1057325]: proxy detected vanished client connection

Before you ask, /etc/pve/nodes on node1 & node2 contain only 2 folders, node1 & node2.

How do I get rid of these remnants of node3 on node1?

 

[Moayad]

Hi,

I would check for any remaining configurations or references to node3 within your Proxmox VE cluster's configuration files.

Specifically, you might want to look into the cluster configuration e.g., `/etc/pve/corosync.conf` to see if node3 is still listed there.

 

[EvertM]

Hi,

I went thoroughly through all files in /etc/pve on node1, but I only found references to node3 in

• /etc/pve/priv/known_hosts
• /etc/pve/priv/authorized_keys

Could any of these be the issue?

 

[Moayad]

Hi,

No. However, consider to get rid of them.

Since the issue related to the `pveproxy` have you tried to issue `pvecm updatecerts --force` command?

I would also check the below files:

cat /etc/pve/.clusterlog
cat /etc/pve/.members

 

[EvertM]

* Removed mentions of node3 from known_hosts & authorized_keys.
* Double-checked .clusterlog & .members: no mentions there.

Still seeing the errors in the log.

 

[Moayad]

Have you run the `pvecm updatecerts` command?

 

[EvertM]

Yes, I ran that again after removing mentions of node3 from known_hosts & authorized_keys

 

[EvertM]

Update: As of yesterday, around 1 p.m., the errors stopped appearing. There were no config changes, no restarts of services or servers, just like that.

Perhaps something timed out? I don't recall the exact date/time we decommissioned node3, so I can't be sure.

 

[Moayad]

Thank you for the update.

You can check the syslog using journalctl e.g.:

journalctl --since "2024-03-06 01:00" --until "2024-03-07 10:00"

 

[EvertM]

Thank you for the update.

You can check the syslog using journalctl e.g.:

journalctl --since "2024-03-06 01:00" --until "2024-03-07 10:00"

Oh, I checked the logs, that's for sure. That's how I know the errors just 'went away' around 1 pm yesterday. The logs make no mention of anything that could be the cause.