Connection error 596: Broken pipe

[syfy323]

Hi!

I am managing a cluster running PVE v6, which currently includes two servers. My third node has been replaced but I am unable to join it.
When I access hv102 using the GUI on hv101, I get "Connection error 596: Broken pipe".
If I try to join hv103, I get this:

root@hv103 ~ # pvecm add 192.168.120.11 -force
Please enter superuser (root) password for '192.168.120.11': **********
Establishing API connection with host '192.168.120.11'
The authenticity of host '192.168.120.11' can't be established.
X509 SHA256 key fingerprint is 94:DC:85:CD:88:FA:4D:80:8F:CA:37:49:C0:C7:26:4B:5E:EE:28:A8:4B:C0:AD:95:E9:F1:B3:43:2B:6D:3F:C7.
Are you sure you want to continue connecting (yes/no)? yes
500 500 Server closed connection without sending any data back

The pvecm status is fine, I can even run "pvesh get /nodes/hv102/status" from hv101 but I can see it fail, when pveproxy needs to process it for the webif.

This cluster has been updated from v5 in the past but hv103 is a fresh install.

Looks like something is broken. What do I need to do?

 

[syfy323]

Anyone?

 

[Stoiko Ivanov]

Check the journal of all nodes - anything related to problems around the time you tried to join the third node?
look for messages from pveproxy, pmxcfs (pve-cluster), corosync, pvedaemon

 

[syfy323]

No issues, just a restart of the replication service.
What I do not understand is that the status query works using shell and failes when run by pveproxy (when the webif needs the data).
This will most likely be the same issue.