random "api error (status = 400: client error (SendRequest))" pop-ups!

[jt_telrite]

I have 3 new installs of PVE 8.4.14 in one cluster, a PDM 1.0.1 VM in that cluster and another 3 installs of PVE 8.4.1 in a second cluster , and a bare metal PBS v3.3.6
as I navigate around , I occasionally get the error... api error (status = 400: client error (SendRequest)) but in the end everything loads and seems to work.

 

[Bob.Dig]

Get this too all the time, just two PVEs, no cluster.

 

[Bob.Dig]

Logs says this but doesn't make sense to me. Connection is over WireGuard. PDM runs as LXC from community-scripts.

Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: marking client 10.3.9.50 as unreachable
Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: skipping host 10.3.9.50 - marked unreachable
Dec 28 10:36:52 datacenter-manager proxmox-datacenter-api[187]: client error on request /api2/extjs/cluster/resources, giving up - client error (SendRequest)

Caused by:
    connection closed before message completed
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: marking client 10.3.9.50 as unreachable
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: skipping host 10.3.9.50 - marked unreachable
Dec 28 10:37:36 datacenter-manager proxmox-datacenter-api[187]: client error on request /api2/extjs/cluster/resources, giving up - client error (SendRequest)

Caused by:
    connection closed before message completed

10.3.9.50 is the remote PVE, PDM is running as a LXC on the local PVE, where there is no connection problem.

 

[Curt Hall]

This started for us this week, all our PVE's are 9.1.4. Our PDM alpha and a newly spun up version 1.0.1 neither one will talk to the 3 PVE's now. certificate error? or kernel update/ firewall? not sure what's happening.

EDIT: we (my tech) had wrong password, PDM 1.0.1 does connect now, issue seems restricted to PDM Alpha, at least in our environment.

 

[ozgurerdogan]

We have same thing. I thought it is because PDM is behind NAT.

 

[manoritltd]

We hit the same "api error (status = 400: client error (SendRequest))" pop-ups with PDM 1.1.9 and a PVE 9.2 remote over a WAN link (~110-175 ms RTT), both over WireGuard and public HTTPS. LAN remotes were never affected.

A packet capture on the PDM showed the cause: pveproxy closes idle keep-alive connections after 5 seconds (default in pve-http-server), and PDM sometimes reuses the pooled connection at almost exactly that moment. The request crosses the server's close on the wire, hyper reports "connection closed before message completed", and for a single-endpoint remote PDM 1.1.9 does not retry, so it shows the error and marks the remote unreachable until the next success. Higher RTT widens the window, which is why it shows up on remote sites and over VPNs. It is not a firewall or NAT problem.

Workaround: add a second endpoint (e.g. VPN and public address) for the same node in the remote's settings. PDM then fails over GET requests and most pop-ups disappear.

Bug report with details and suggested fixes: <add Bugzilla link here>