No error content when a 500 error happens on the HTTP API

[robinstraub]

When interacting with the Proxmox HTTP API and submitting a request that produces an error (e.g. creating a VM with a "vmid" that is already taken, I am not getting any error message from the api.

curl --request POST --url https://proxmox.acme/api2/json/nodes/pve-node1/qemu --header 'authorization: PVEAPIToken=root@pam!api=obfuscated-token' --header 'content-type: application/json' --data '{ "vmid":100}'

The following request produces a 500 error with content

{"data": null}

.

But when trying to produce the same outcome with pvesh:

pvesh create /nodes/pve-node1/qemu --vmid 100

I obtain a more qualified error message:

unable to create VM 100 - VM 100 already exists on node 'pve-node1'

Is this behavior expected ? Does the API not return any error message when something bad happens? It complicates my debugging. Also, I desperately tried to find some thorough logs that could help me debug errors but all I could find for the API is the /var/log/pveproxy/access.log, which gives me incoming request and their HTTP status code, but no additional context.

To be precise, I don't need fixing that specific error: the context is clear, the vm id is already taken. What I want is, when an error occurs, get that error from the HTTP API, and not having to mimic that request through pvesh to obtain an error message.

My best,

 

[bbgeek17]

No, something is not working right in your test:

root@pve-1:~/auth_test# bash test.txt
{"data":"UPID:pve-1:003D199A:3B1C84AD:67CDFCF7:qmcreate:7001:root@pam:"}

root@pve-1:~/auth_test# bash test.txt
{"message":"unable to create VM 7001 - VM 7001 already exists on node 'pve-1'\n","data":null}

curl --silent --insecure --data "username=root@pam&password=$PASSWORD" \
 https://$APINODE:8006/api2/json/access/ticket\
| jq --raw-output '.data.ticket' | sed 's/^/PVEAuthCookie=/' > cookie

curl --silent --insecure --data "username=root@pam&password=$PASSWORD" \
 https://$APINODE:8006/api2/json/access/ticket \
| jq --raw-output '.data.CSRFPreventionToken' | sed 's/^/CSRFPreventionToken:/' > csrftoken

curl  --insecure --cookie "$(<cookie)"  --header "$(<csrftoken)"  --request POST https://$APINODE:8006/api2/json/nodes/$APINODE/qemu --data vmid=7001

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[robinstraub]

Thanks for your reply!

I tried running your script? 'im still getting `{"data": null}`. Is your HTTP status code a 500 as well or a 400. Do you have any direction to point me on how to debug this behavior ? logs to investigate, configuration to look into...

 

[bbgeek17]

yes, its 500: "POST /api2/json/nodes/pve-1/qemu HTTP/1.1" 500 93

data is null, message is the information

bash test.txt |jq
{
  "data": null,
  "message": "unable to create VM 7001 - VM 7001 already exists on node 'pve-1'\n"
}

curl -sk --request POST --url https://localhost:8006/api2/json/nodes/pve-1/qemu --header 'authorization: PVEAPIToken=root@pam!test=f4xxx' --header 'content-type: application/json' --data '{ "vmid":7001}'|jq
{
  "data": null,
  "message": "unable to create VM 7001 - VM 7001 already exists on node 'pve-1'\n"
}

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[robinstraub]

But when I issue a failing request, e.g. your example (with a valid header, and the request succeeds when the vmid does not exist)

curl -sk --request POST --url [URL]https://localhost:8006/api2/json/nodes/pve-1/qemu[/URL] --header 'authorization: PVEAPIToken=root@pam!test=f4xxx' --header 'content-type: application/json' --data '{ "vmid":7001}'|jq

I get ONLY a `"data": null`, no message whatsoever (my bad for not precising this). Also the error is a 500, I would have expected a 400, idk if the status code is the normal behavior

{
    "data": null
}

Again, do you have any clues where in the logs or in the config I could look to explain this behavior ?

 

[robinstraub]

Here is a stack trace from curl below. The key takeaway is: < HTTP/1.1 500 unable to create VM 100 - VM 100 already exists on node 'pve-node3'. That would mean that proxmox passes the error message through the HTTP status code description.


I have a HAProxy in front of the nodes that would "lose" that header description, and when pveproxy picks up the response it has no error message to provide... Am I right?

*   Trying XXX.XXX.XXX.XXX:8006...
* Connected to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=pve-node1.local
*  start date: Sep  5 19:08:39 2024 GMT
*  expire date: Sep  5 19:08:39 2026 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /api2/json/nodes/pve-node1/qemu HTTP/1.1
> Host: XXX.XXX.XXX.XXX:8006
> User-Agent: curl/7.81.0
> Cookie: PVEAuthCookie=PVE:root@pam:obfuscated
> CSRFPreventionToken:obfuscated
> Accept: application/json
> Content-Length: 8
> Content-Type: application/x-www-form-urlencoded
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 unable to create VM 100 - VM 100 already exists on node 'pve-node3'
< Cache-Control: max-age=0
< Connection: close
< Date: Mon, 10 Mar 2025 10:45:07 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 13
< Content-Type: application/json;charset=UTF-8
< Expires: Mon, 10 Mar 2025 10:45:07 GMT
<
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, decode error (562):
{"data":null}%

 

[bbgeek17]

Also the error is a 500, I would have expected a 400,

I believe, that currently PVE returns 500 as catch-all for various conditions. There is work happening in this direction to improve it, IIRC.

That would mean that proxmox passes the error message through the HTTP status code description.

The fact that you had no port specified in your example in original post was something that caught my eye and was next on the list of suspects. You clearly had some sort of intermediary between your client and PVE.

I have a HAProxy in front of the nodes that would "lose" that header description, and when pveproxy picks up the response it has no error message to provide...

Sounds like HAproxy problem. I would recommend to always test with as few components as possible to get the baseline.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[robinstraub]

I believe, that currently PVE returns 500 as catch-all for various conditions. There is work happening in this direction to improve it, IIRC.


The fact that you had no port specified in your example in original post was something that caught my eye and was next on the list of suspects. You clearly had some sort of intermediary between your client and PVE.

Sounds like HAproxy problem. I would recommend to always test with as few components as possible to get the baseline.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I understand, thank you for your help and the follow-up.

Can you confirm Proxmox carries messages by design as a status code description, and thus I should stick to the HTTP/1.1 protocol, as HTTP/2 does not define a way to carry the version or reason phrase ?

My best,

 

[bbgeek17]

Can you confirm Proxmox carries messages by design as a status code description, and thus I should stick to the HTTP/1.1 protocol, as HTTP/2 does not define a way to carry the version or reason phrase ?

I am not that familiar with the code. I suspect what you said is correct. Perhaps one of the Proxmox staff members can clarify further.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[robinstraub]

Ok thanks. Anyways that settles my issue, we can consider it as closed (idk if there is a close-up mecanism for posts here).

Again, thanks a lot for your assistance!

 

[robinstraub]

ACTUALLY: upgrading from 8.3.2 to 8.3.4 fixed the issue .

Operating behind a HAProxywas causing the error, because the proxy was rewriting the protocol to HTTP/2 and dropping the status code description... Works fine on 8.3.4 so I guess they fixed something to allow operating on HTTP/2