random "api error (status = 400: client error (SendRequest))" pop-ups!

[jt_telrite]

I have 3 new installs of PVE 8.4.14 in one cluster, a PDM 1.0.1 VM in that cluster and another 3 installs of PVE 8.4.1 in a second cluster , and a bare metal PBS v3.3.6
as I navigate around , I occasionally get the error... api error (status = 400: client error (SendRequest)) but in the end everything loads and seems to work.

 

[Bob.Dig]

Get this too all the time, just two PVEs, no cluster.

 

[Bob.Dig]

Logs says this but doesn't make sense to me. Connection is over WireGuard. PDM runs as LXC from community-scripts.

Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: marking client 10.3.9.50 as unreachable
Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: skipping host 10.3.9.50 - marked unreachable
Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: client error on request /api2/extjs/cluster/resources, giving up - client error (SendRequest)

Caused by:
    connection closed before message completed
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: marking client 10.3.9.50 as unreachable
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: skipping host 10.3.9.50 - marked unreachable
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: client error on request /api2/extjs/cluster/resources, giving up - client error (SendRequest)

Caused by:
    connection closed before message completed

10.3.9.50 is the remote PVE, PDM is running as a LXC on the local PVE, where there is no connection problem.

 

[Curt Hall]

This started for us this week, all our PVE's are 9.1.4. Our PDM alpha and a newly spun up version 1.0.1 neither one will talk to the 3 PVE's now. certificate error? or kernel update/ firewall? not sure what's happening.

EDIT: we (my tech) had wrong password, PDM 1.0.1 does connect now, issue seems restricted to PDM Alpha, at least in our environment.

 

[ozgurerdogan]

We have same thing. I thought it is because PDM is behind NAT.

 

[Puliper]

The log shows the connection is closed before the response completes, so PDM marks the node unreachable. This points to a network issue.

If you are using WireGuard, lower the MTU to around 1380 on both sides and test. MTU mismatch is a common cause of this exact behavior.

Also check for firewall or NAT timeouts on port 8006. From inside the PDM LXC, curl the remote API and see if it intermittently fails.