Two extra bytes in ISO file when uploading via API / Proxmoxer

[rudolfbyker]

When I upload the same ISO file manually through the GUI and through the API, I get two slightly different results:

• Manual upload: The ISO works.
• Upload using API: The file has two extra leading bytes, specifically `0x0d0a` (`\r\n`), and it's seen as a corrupt ISO file.

More details at https://github.com/proxmoxer/proxmoxer/issues/139 . Reposting here because I'm not 100% convinced that the problem is with the Proxmoxer package.

 

[fweber]

Hi, could you post the output of pveversion -v? Have you also tried uploading the file via curl or e.g. httpie [1]? I just tried using httpie to upload an ISO to a Proxmox VE 7.4 host via the API, and it worked fine (i.e. local and uploaded file were identical).

[1] https://github.com/httpie/httpie

 

[rudolfbyker]

Versions:

proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.4
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

I'm not familiar with HTTPie. Would you send me a sample of the commands you used?

 

[rudolfbyker]

One of the reasons I'm using proxmoxer, is that I could not find in the docs, nor figure out for myself, how to authenticate requests to the API once I got my ticket from `POST /api2/json/access/ticket`.

 

[Dunuin]

One of the reasons I'm using proxmoxer, is that I could not find in the docs, nor figure out for myself, how to authenticate requests to the API once I got my ticket from `POST /api2/json/access/ticket`.

It also works without tickets when using API tokens. For example to shutdown a VM via API:
curl -k -X POST -H 'Authorization: PVEAPIToken=<ApiToken>=<TokenSecret>' https://<NodeIP>:8006/api2/json/nodes/<NodeName>/qemu/<VMID>/status/shutdown

 

[rudolfbyker]

With some trial and error I ended up with this:

https --verify=no --session=dolf POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload content=iso filename=foo.iso Cookie:PVEAuthCookie=PVE:username@pve:redacted==

But I'm getting

https: error: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) while doing a POST request to URL: https://192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload

Same results when using a Token instead of a Ticket.

 

[rudolfbyker]

Another variant I tried:

https --verify=no --session=dolf POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload Cookie:PVEAuthCookie=PVE:username@pve:redacted== < foo.iso

But this gave me

https: error: ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)) while doing a POST request to URL: https://192.168.0.2:8006/api2
/json/nodes/hv1/storage/local/upload

Same results when using a Token instead of a Ticket.

 

[fweber]

Hi, to upload an ISO using httpie you need to pass the --form flag, and the path to the ISO file as filename@<path to your ISO>, e.g.:

http <...> --form POST 'https://<...>/api2/json/nodes/<...>/storage/<...>/upload' Cookie:<...> filename@/path/to/your.iso

The http manpage has some more details.

 

[rudolfbyker]

Hi, to upload an ISO using httpie you need to pass the --form flag, and the path to the ISO file as filename@<path to your ISO>, e.g.:

http <...> --form POST 'https://<...>/api2/json/nodes/<...>/storage/<...>/upload' Cookie:<...> filename@/path/to/your.iso

Aha! This time, it worked. I get the same problem as when using proxmoxer. The file is two bytes longer on the server. Somehow, `\r\n` is being prepended.

The http manpage has some more details.

Interesting... I was following the examples at https://httpie.io/docs/cli/examples . I did not expect the manpage to say something different than their online docs.

 

[jhollowe]

I imagine this is related to changes in http-server related to handling `Content-Type` on the file's multipart. RFC 2388 specifies that the `Content-Type` on multi-parts is optional. Chrome (and chromium-based browsers) have semi-recently started adding this header to its file uploads, but other tools like `proxmoxer` and I assume `http` do not add this header.

I think the upload working in a browser but not in other tools indicates that the regex-ing of the `Content-Type` (and its \r\n\r\n separator) is not quite right.

 

[rudolfbyker]

Thanks for the help and info! If anyone finds or creates a bug report for this, please link it here.

EDIT: jhollowe, Thanks, I saw you commented on Github, too.

 

[fweber]

Aha! This time, it worked. I get the same problem as when using proxmoxer. The file is two bytes longer on the server. Somehow, `\r\n` is being prepended.

Thanks for checking! That you also see this issue with httpie is strange -- I still cannot reproduce it.

I think the upload working in a browser but not in other tools indicates that the regex-ing of the `Content-Type` (and its \r\n\r\n separator) is not quite right.

Good call! There might be some subtle issue with header processing.

@rudolfbyker, to troubleshoot this further:

• Could you post your httpie version (output of http --version)?
• Could you show your final httpie command (of course you can censor the IP and authorization cookie)?
• Could you check if httpie sends Content-Type headers for multiparts? One way to do this would be to add the -pHB flag and grep its (partly binary) output for 'Content-' lines, that is, run and post of the output of:
  http -pHB <remaining arguments> | grep -aE '^Content-'
• Does this happen for all ISO files, or just a specific one?
• Is this a clustered setup? If yes, do you send the HTTP request to the node referenced in the API path (in .../nodes/<name>/storage...), or is it a different one?

 

[rudolfbyker]

Version:

http --version
3.2.1

Final command:

https --verify=no --form POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload 'Authorization:PVEAPIToken=github-webhooks@pve!mytoken=secret' content=iso filename@foo.iso

My ISO file: https://www.dropbox.com/s/dfuowhz5rt4yfp3/foo.iso?dl=0 Yes, it happens with different but similar ISO files. They are tiny, each containing one or two text files with secrets, used for ephemeral VMs. For more about my use case, see https://forum.proxmox.com/threads/injecting-secrets-into-windows-vms.125303/ . However, I suspect that this is not relevant.

Output of

https -pHB ... > debug.txt

https://www.dropbox.com/s/kqg9ydnbh0aghvf/debug.txt?dl=0

Checking for headers:

grep -aE '^Content-' debug.txt
Content-Length: 551150
Content-Type: multipart/form-data; boundary=d653dc22969b420c823f1a2fa8c29121
Content-Disposition: form-data; name="content"
Content-Disposition: form-data; name="filename"; filename="foo.iso"

No clustering. I have a single node at 192.168.0.2, although it is in another city, on the other side of an OpenVPN tunnel.

 

[fweber]

Thank you for the information! I managed to reproduce the issue. Indeed, multipart requests are currently parsed incorrectly if the file part does not contain a Content-Type, as it is the case for the request you've posted:

grep -aE '^Content-' debug.txt
Content-Length: 551150
Content-Type: multipart/form-data; boundary=d653dc22969b420c823f1a2fa8c29121
Content-Disposition: form-data; name="content"
Content-Disposition: form-data; name="filename"; filename="foo.iso"

Note that there is no Content-Type: application/x-iso9660-image line after the ...filename="foo.iso" line.

I'm currently working on a patch to fix this, and will keep you up-to-date in this thread.

One more thing, out of curiosity: Apparently, httpie (including 3.2.1) tries to guess the MIME type of the input file using the Python mimetypes module [1], and omits the file part's Content-Type only if mimetypes fails to guess it. On my system (Proxmox VE 7.4), mimetype guesses application/x-iso9660-image and httpie sends this as the Content-Type. I can only reproduce the bug if I manually edit /etc/mime.types and comment out the iso line. I'm wondering why mimetypes cannot guess the MIME type on your system -- what kind of system are you running (Windows)? What does python3 -c 'import mimetypes; print(mimetypes.guess_type("some-file.iso", False))' print?

[1] https://docs.python.org/3/library/mimetypes.html

 

[rudolfbyker]

what kind of system are you running (Windows)? What does python3 -c 'import mimetypes; print(mimetypes.guess_type("some-file.iso", False))' print?

Yes, Windows 10. It seems that my Python can't guess the MIME type:

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mimetypes
>>> print(mimetypes.guess_type("foo.iso", False))
(None, None)

… but I had the same problem inside a python alpine docker image.

 

[fweber]

I sent a patch [1] that was already applied [2], so the issue should be fixed in libpve-http-server-perl >= 4.2-2. Thanks for the report and your help troubleshooting it!

Yes, Windows 10. It seems that my Python can't guess the MIME type:

I see -- that explains why httpie does not send a Content-Type on your machine.

[1]: https://lists.proxmox.com/pipermail/pve-devel/2023-April/056531.html
[2]: https://git.proxmox.com/?p=pve-http-server.git;a=commit;h=602eb8aabdc04180962d5c3b6f1c66358a8d2690

 

[rudolfbyker]

Perfect, thanks. After apt update && apt upgrade I got the latest version of libpve-http-server-perl and now all is well.