Two extra bytes in ISO file when uploading via API / Proxmoxer

rudolfbyker

New Member
Apr 3, 2023
13
3
3
South Africa
When I upload the same ISO file manually through the GUI and through the API, I get two slightly different results:
  • Manual upload: The ISO works.
  • Upload using API: The file has two extra leading bytes, specifically `0x0d0a` (`\r\n`), and it's seen as a corrupt ISO file.
More details at https://github.com/proxmoxer/proxmoxer/issues/139 . Reposting here because I'm not 100% convinced that the problem is with the Proxmoxer package.
 
Last edited:
Hi, could you post the output of pveversion -v? Have you also tried uploading the file via curl or e.g. httpie [1]? I just tried using httpie to upload an ISO to a Proxmox VE 7.4 host via the API, and it worked fine (i.e. local and uploaded file were identical).

[1] https://github.com/httpie/httpie
 
  • Like
Reactions: pvps1
Versions:

Code:
proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.6.4
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.11-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

I'm not familiar with HTTPie. Would you send me a sample of the commands you used?
 
One of the reasons I'm using proxmoxer, is that I could not find in the docs, nor figure out for myself, how to authenticate requests to the API once I got my ticket from `POST /api2/json/access/ticket`.
 
One of the reasons I'm using proxmoxer, is that I could not find in the docs, nor figure out for myself, how to authenticate requests to the API once I got my ticket from `POST /api2/json/access/ticket`.
It also works without tickets when using API tokens. For example to shutdown a VM via API:
curl -k -X POST -H 'Authorization: PVEAPIToken=<ApiToken>=<TokenSecret>' https://<NodeIP>:8006/api2/json/nodes/<NodeName>/qemu/<VMID>/status/shutdown
 
With some trial and error I ended up with this:
Code:
https --verify=no --session=dolf POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload content=iso filename=foo.iso Cookie:PVEAuthCookie=PVE:username@pve:redacted==
But I'm getting
Code:
https: error: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) while doing a POST request to URL: https://192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload

Same results when using a Token instead of a Ticket.
 
Last edited:
  • Like
Reactions: Tmanok
Another variant I tried:
Code:
https --verify=no --session=dolf POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload Cookie:PVEAuthCookie=PVE:username@pve:redacted== < foo.iso
But this gave me
Code:
https: error: ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)) while doing a POST request to URL: https://192.168.0.2:8006/api2
/json/nodes/hv1/storage/local/upload

Same results when using a Token instead of a Ticket.
 
Last edited:
Hi, to upload an ISO using httpie you need to pass the --form flag, and the path to the ISO file as filename@<path to your ISO>, e.g.:
Code:
http <...> --form POST 'https://<...>/api2/json/nodes/<...>/storage/<...>/upload' Cookie:<...> filename@/path/to/your.iso

The http manpage has some more details.
 
Hi, to upload an ISO using httpie you need to pass the --form flag, and the path to the ISO file as filename@<path to your ISO>, e.g.:
Code:
http <...> --form POST 'https://<...>/api2/json/nodes/<...>/storage/<...>/upload' Cookie:<...> filename@/path/to/your.iso
Aha! This time, it worked. I get the same problem as when using proxmoxer. The file is two bytes longer on the server. Somehow, `\r\n` is being prepended.

The http manpage has some more details.
Interesting... I was following the examples at https://httpie.io/docs/cli/examples . I did not expect the manpage to say something different than their online docs.
 
I imagine this is related to changes in http-server related to handling `Content-Type` on the file's multipart. RFC 2388 specifies that the `Content-Type` on multi-parts is optional. Chrome (and chromium-based browsers) have semi-recently started adding this header to its file uploads, but other tools like `proxmoxer` and I assume `http` do not add this header.

I think the upload working in a browser but not in other tools indicates that the regex-ing of the `Content-Type` (and its \r\n\r\n separator) is not quite right.
 
  • Like
Reactions: Dunuin
Thanks for the help and info! If anyone finds or creates a bug report for this, please link it here.

EDIT: jhollowe, Thanks, I saw you commented on Github, too.
 
Last edited:
Aha! This time, it worked. I get the same problem as when using proxmoxer. The file is two bytes longer on the server. Somehow, `\r\n` is being prepended.
Thanks for checking! That you also see this issue with httpie is strange -- I still cannot reproduce it.
I think the upload working in a browser but not in other tools indicates that the regex-ing of the `Content-Type` (and its \r\n\r\n separator) is not quite right.
Good call! There might be some subtle issue with header processing.

@rudolfbyker, to troubleshoot this further:
  • Could you post your httpie version (output of http --version)?
  • Could you show your final httpie command (of course you can censor the IP and authorization cookie)?
  • Could you check if httpie sends Content-Type headers for multiparts? One way to do this would be to add the -pHB flag and grep its (partly binary) output for 'Content-' lines, that is, run and post of the output of:
    http -pHB <remaining arguments> | grep -aE '^Content-'
  • Does this happen for all ISO files, or just a specific one?
  • Is this a clustered setup? If yes, do you send the HTTP request to the node referenced in the API path (in .../nodes/<name>/storage...), or is it a different one?
 
Version:

Code:
http --version
3.2.1

Final command:
Code:
https --verify=no --form POST 192.168.0.2:8006/api2/json/nodes/hv1/storage/local/upload 'Authorization:PVEAPIToken=github-webhooks@pve!mytoken=secret' content=iso filename@foo.iso

My ISO file: https://www.dropbox.com/s/dfuowhz5rt4yfp3/foo.iso?dl=0 Yes, it happens with different but similar ISO files. They are tiny, each containing one or two text files with secrets, used for ephemeral VMs. For more about my use case, see https://forum.proxmox.com/threads/injecting-secrets-into-windows-vms.125303/ . However, I suspect that this is not relevant.

Output of
Code:
https -pHB ... > debug.txt
https://www.dropbox.com/s/kqg9ydnbh0aghvf/debug.txt?dl=0

Checking for headers:
Code:
grep -aE '^Content-' debug.txt
Content-Length: 551150
Content-Type: multipart/form-data; boundary=d653dc22969b420c823f1a2fa8c29121
Content-Disposition: form-data; name="content"
Content-Disposition: form-data; name="filename"; filename="foo.iso"

No clustering. I have a single node at 192.168.0.2, although it is in another city, on the other side of an OpenVPN tunnel.
 
  • Like
Reactions: fweber
Thank you for the information! I managed to reproduce the issue. Indeed, multipart requests are currently parsed incorrectly if the file part does not contain a Content-Type, as it is the case for the request you've posted:
Code:
grep -aE '^Content-' debug.txt
Content-Length: 551150
Content-Type: multipart/form-data; boundary=d653dc22969b420c823f1a2fa8c29121
Content-Disposition: form-data; name="content"
Content-Disposition: form-data; name="filename"; filename="foo.iso"
Note that there is no Content-Type: application/x-iso9660-image line after the ...filename="foo.iso" line.

I'm currently working on a patch to fix this, and will keep you up-to-date in this thread.

One more thing, out of curiosity: Apparently, httpie (including 3.2.1) tries to guess the MIME type of the input file using the Python mimetypes module [1], and omits the file part's Content-Type only if mimetypes fails to guess it. On my system (Proxmox VE 7.4), mimetype guesses application/x-iso9660-image and httpie sends this as the Content-Type. I can only reproduce the bug if I manually edit /etc/mime.types and comment out the iso line. I'm wondering why mimetypes cannot guess the MIME type on your system -- what kind of system are you running (Windows)? What does python3 -c 'import mimetypes; print(mimetypes.guess_type("some-file.iso", False))' print?

[1] https://docs.python.org/3/library/mimetypes.html
 
what kind of system are you running (Windows)? What does python3 -c 'import mimetypes; print(mimetypes.guess_type("some-file.iso", False))' print?

Yes, Windows 10. It seems that my Python can't guess the MIME type:

Code:
Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mimetypes
>>> print(mimetypes.guess_type("foo.iso", False))
(None, None)

… but I had the same problem inside a python alpine docker image.
 
  • Like
Reactions: fweber
I sent a patch [1] that was already applied [2], so the issue should be fixed in libpve-http-server-perl >= 4.2-2. Thanks for the report and your help troubleshooting it!

Yes, Windows 10. It seems that my Python can't guess the MIME type:
I see -- that explains why httpie does not send a Content-Type on your machine.

[1]: https://lists.proxmox.com/pipermail/pve-devel/2023-April/056531.html
[2]: https://git.proxmox.com/?p=pve-http-server.git;a=commit;h=602eb8aabdc04180962d5c3b6f1c66358a8d2690
 
  • Like
Reactions: pvps1

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!