Uploading an ISO file to Proxmox VE 8 using the API

[rudolfbyker]

I used to be able to do stuff like this on Proxmox VE 7:

https --verify=no --form POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload 'Authorization:PVEAPIToken=github-webhooks@pve!mytoken=secret' content=iso filename@foo.iso

Now my old server crashed, and I built a new one using Proxmox VE 8. Now the same command gives me this:

https: error: SSLError: HTTPSConnectionPool(host='hv1.aa', port=8006): Max retries exceeded with url: /api2/json/nodes/hv1/storage/hdd3/upload (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)'))) while doing a POST request to URL: https://hv1.aa:8006/api2/json/nodes/hv1/storage/hdd3/upload

I don't know what this error means.

 

[LnxBil]

I don't know what this error means.

Try to enable debugging on your https tool, or try to recreate the error with curl.

 

[rudolfbyker]

I already had the --verbose flag in there. Here is the exact command that I ran:

https --verify=no --verbose --form POST hv1.aa:8006/api2/json/nodes/hv1/storage/hdd3/upload 'Authorization:PVEAPIToken=github-webhooks@pve!mytoken=<redacted>' content=iso filename@github-runner-secrets.iso

And here is the full output:

POST /api2/json/nodes/hv1/storage/hdd3/upload HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: PVEAPIToken=github-webhooks@pve!mytoken=<redacted>
Connection: keep-alive
Content-Length: 551211
Content-Type: multipart/form-data; boundary=dc7aa5efa2574eb982c14a7d329aa71e
Host: hv1.aa:8006
User-Agent: HTTPie/2.6.0



+-----------------------------------------+  ← These three lines are repeated a few times
| NOTE: binary data not shown in terminal |  ←
+-----------------------------------------+  ←
https: error: SSLError: HTTPSConnectionPool(host='hv1.aa', port=8006): Max retries exceeded with url: /api2/json/nodes/hv1/storage/hdd3/upload (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)'))) while doing a POST request to URL: https://hv1.aa:8006/api2/json/nodes/hv1/storage/hdd3/upload

Here is the same thing using curl:

curl -k -v -X POST https://hv1.aa:8006/api2/json/nodes/hv1/storage/hdd3/upload -H 'Authorization: PVEAPIToken=github-webhooks@pve!mytoken=<redacted>' -F "content=iso" -F "filename=@github-runner-secrets.iso"

And here is the output:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <redacted>:8006...
* Connected to hv1.aa (<redacted>) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=<redacted>
*  start date: Jul  9 19:23:10 2024 GMT
*  expire date: Jul  9 19:23:10 2026 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=2aee7b0c-49d5-428c-828c-f9bef130c525; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /api2/json/nodes/hv1/storage/hdd3/upload HTTP/1.1
> Host: hv1.aa:8006
> User-Agent: curl/7.81.0
> Accept: */*
> Authorization: PVEAPIToken=github-webhooks@pve!mytoken=<redacted>
> Content-Length: 551232
> Content-Type: multipart/form-data; boundary=------------------------06e7cc285967d6cb
>
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* OpenSSL SSL_write: Broken pipe, errno 32
* Closing connection 0
curl: (55) OpenSSL SSL_write: Broken pipe, errno 32

 

[LnxBil]

Thank you, I was hoping for more useful information from debugging. Have you tried another PVE in order to see if this is a local or a more general problem?

I hope I've time today so that I can try to replicate the problem. It looks interesting, though.

 

[LnxBil]

So, I just tested it and I just works as it should:

> POST /api2/json/nodes/proxmox/storage/local/upload HTTP/1.1
> Host: proxmox.local:8006
> User-Agent: curl/8.1.2
> Accept: */*
> Authorization: PVEAPIToken=user@pve!test=abcdefgh-ijkl-mnop-qrst-uvwxyz123456
> Content-Length: 41853233
> Content-Type: multipart/form-data; boundary=------------------------a726ff89e8c706ef
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: close

Therefore I cannot help you any further if I cannot reproduce the problem.

 

[rudolfbyker]

Sorry for the late reply.

Have you tried another PVE in order to see if this is a local or a more general problem?

I built another Proxmox VE 8 server, and it does the exact same thing. I tried with different types of storage, different ISO files, and with different types of credentials in my Authorization header. But it looks like none of that makes a difference, because the error comes from OpenSSL.

I had someone else test it who is on the same LAN as the Promox server, and they get the same error, so it's not my local machine or the internet connection or the VPN that's to blame.

When I try uploading the ISO from the Proxmox server itself (accessing the API as https://localhost:8006), I get a different error:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:8006...
* Connected to localhost (127.0.0.1) port 8006 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=hv2.example.com
*  start date: Jun  9 10:49:50 2024 GMT
*  expire date: Jun  9 10:49:50 2026 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=f817b60a-5d88-4d97-8875-83cd656c56df; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> POST /api2/json/nodes/hv2/storage/hdd3/upload HTTP/1.1
> Host: localhost:8006
> User-Agent: curl/7.88.1
> Accept: */*
> Authorization: PVEAPIToken=root@pam!my-token=<redacted>
> Content-Length: 551215
> Content-Type: multipart/form-data; boundary=------------------------d55db2e78fa414f1
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Empty reply from server
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (52) Empty reply from server

 

[LnxBil]

Content-Length: 551215

What small iso is this?

I just tried again (I wrote a script last time), and it's not working anymore. Perfect. Yet I deleted the API access token, so that was almost easy to fix. In the end my problems were just permission problems, yet I looked at /var/log/pveproxy/access.log, what error was in there. Can you have a look there?

 

[rudolfbyker]

What small iso is this?

A programmatically created ISO containing a single text file containing a secret that needs to be temporarily injected into an ephemeral VM. But I tried with other, larger ISOs too. Same result.

I'm not sure what you mean by the rest of your message. The error is on the SSL layer, so permissions don't come into play yet, as far as I can tell.

 

[LnxBil]

Puhh ... I don't know what could be the problem here. Have you tried a newer curl version? What about tcpdumping the session and analysing if there is somethine present there? Have you any entry in the log about the connection attempt? Your curl says it has uploaded the file, so there may hopefully be some information in the log.