image upload fails after upgrading from 7.1 to 7.3

[jkonieczny]

Hi,

We have upgraded our cluster from 7.1 to 7.3 and everything is fine… except ISO image uploads via API do not work any more for us.

We get an error:

{"data":null,"errors":{"content":"invalid format - invalid content type 'Content-Type: text/plain; charse
t=ISO-8859-1\r\nContent-Transfer-Encoding: 8bit\r\n\r\niso'\n"}}

The exact some upload requests (multipart/form-data with content=iso in one part and the file data in the other)
worked for PVE 7.1 and earlier.

Any ideas what is going on?

I have looked into the code and it seems the 'invalid content type' error message is not about the 'Content-Type' header, but the data received which should be just 'iso' but includes the headers for some reason.

 

[Matthias.]

Hi,

there have been some changes in that code and we (probably I) must have missed something.

Could you share a minimal example request that fails when trying to upload?
Can you confirm that uploads via the GUI still work?

 

[jkonieczny]

Manual upload does work. Requests made with curl also work But our client code uses Apache HttpClient Java library to build the requests which are somehow different form those of curl.

It seems like pveproxy is confused by the 'Content-Type:' and 'Content-Transfer-Encoding:' headers added by HttpClient in one of the multipart/form-data parts. They are not needed there, but should not cause the request to fail.

Making a minimum example may take a bit time, as it is not a simple JSON request one can write by hand, but we will try to prepare one.

 

[Matthias.]

pveproxy is confused by the 'Content-Type:' and 'Content-Transfer-Encoding:' headers

Most likely, yes. If I know where they are inserted, I'll adapt the code to ignore them

but we will try to prepare one.

Great! It should already help if you can give us the request in plain text, e.g. when uploading an empty iso (to keep it small, and because content doesn't matter).

 

[jkonieczny]

Here they are:

Failing request:

POST /api2/json/nodes/NODE/storage/STORAGE/upload HTTP/1.1
Host: HOST:8006
User-Agent: curl/7.75.0
Accept: */*
CSRFPreventionToken: [REDACTED]
Cookie: PVEAuthCookie=[REDACTED]
Content-Type: multipart/form-data; boundary=yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Length: CONTENT_LENGTH

--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Disposition: form-data; name="content"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

iso
--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Disposition: form-data; name="filename"; filename="empty.iso"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary


--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz--

Working request:

POST /api2/json/nodes/NODE/storage/STORAGE/upload HTTP/1.1
Host: HOST:8006
User-Agent: curl/7.75.0
Accept: */*
CSRFPreventionToken: [REDACTED]
Cookie: PVEAuthCookie=[REDACTED]
Content-Type: multipart/form-data; boundary=yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Length: CONTENT_LENGTH

--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Disposition: form-data; name="content"

iso
--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz
Content-Disposition: form-data; name="filename"; filename="empty.iso"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary


--yERt1dYNCtjX46clu06uO3ym2Y0cLUwb3nz--

Note the difference in headers of the first part.

 

[Matthias.]

Thanks a lot for your information!
I sent a fix to the list: https://lists.proxmox.com/pipermail/pve-devel/2022-December/055113.html

I tested it with a small Apache HttpClient program, that's as close as I could get to your request.
You can apply it yourself (fast, but as it's not yet reviewed it might contain another bug) or wait until it gets packaged (might take a bit).
Either way, please report back whether it fixes your problem

 

[jkonieczny]

I have applied the patch and now the upload fails with:

pveproxy[1730382]: problem with client ::ffff:10.28.45.85; No space left on device

Which is strange, as there is plenty of disk space available there.

 

[jkonieczny]

The problem is that this line from the patch:

$hdl->{rbuf} =~ s/^(.*?)(${delim_re})/$2/s;

Won't match anything if the part delimiter does not come in the same data chunk as the one currently processed. The next line becomes nonsense, as '$1' is not what it is supposed to be, $hdl->{rbuf} is not cut properly and then the code just tries to match regexps with bigger and bigger chunks of data, eventually getting 'No space left on device' (provided the upload size is big enough).

Is this whole reinvention of HTTP requests parsing really needed? Shouldn't some standard library provide that?

 

[jkonieczny]

I have reverted the patch to try to work around the original problem. And I found out things are worse than we thought. After removing the extra headers from the 'content' part the request is accepted and a file is uploaded. But the uploaded file is corrupted – it has part of the headers of the 'filename' part included at the beginning and is not a valid ISO file any more.

 

[Matthias.]

Ugh, that's not good. You're right of course. I'll try to fix this.

As a work around, please use the GUI, curl or scp. They should all work.
To avoid corruption, you can pass a hash that will be verified serverside and fail the upload if it doesn't. It's not the solution to your problem, but at least you won't get any broken files that way.

Shouldn't some standard library provide that?

Yes, that would certainly be nice