[SOLVED] Proxmox VE 9.0 ISO Upload Corrupts Some Windows Server 2025 ISOs

B

brendanw36

New Member
Aug 10, 2025
6
0
1
I've recently done a fresh install of Proxmox VE 9.0 and came across this issue while trying to upload ISOs for a Windows Server 2025 VM. When I try to upload either SW_DVD9_Win_Server_STD_CORE_2025_24H2.4_64Bit_English_DC_STD_MLF_X23-96921.ISO or SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO, the resulting ISO that ends up in /var/lib/vz/template/iso is about 3 GB larger than the original ISO, no longer matches the SHA256 checksum of the original ISO, and attempting to boot from it shows a "Boot Configuration Data" error screen. The oldest Windows Server 2025 ISO I have, SW_DVD9_Win_Server_STD_CORE_2025_24H2_64Bit_English_DC_STD_MLF_X23-81891.ISO, uploads just fine. When I upload the ISOs, I can specify a SHA256 checksum, and it will pass the check, but the resulting file will still be corrupted. In an attempt to reproduce this issue, I used FileZilla to upload the ISO to /var/tmp, confirmed that the checksum was good, and then ran cp -- /var/tmp/source.iso /var/lib/vz/template/iso/dest.iso. The resulting ISO experienced the same corruption that occurs when uploading the ISO through the Proxmox VE GUI. Doing the same copy procedure with rsync doesn't result in a corrupted ISO.

Has anyone else experienced this or is anyone able to help me find out why cp is having issues with these specific ISOs?

Edit: Since the copy command always generates the same bad ISO, I figure it might be useful to share the SHA256 checksum of the good and bad ISO.
SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO:
2509618770af72a6335f975beb9f3caacfecbd60c448cafa37959aabccb0d12d

SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO.CORRUPTED:
5ac07a753e55848f344b7273e4c0a167dd64ffe9033e58df79ffefd0981cbb4f
 
Last edited:
what kind of storage are you using for /var/lib/vz?
 
I installed Proxmox on two 1 TB SATA SSDs using the "zfs (RAID1)" filesystem option in the installer. df reports the filesystem as "rpool/var/lib/vz".
 
can you try the `cp` with `--sparse=never`?
 
If it is disk corruption (due to drive/flash, cable, controller) then a scrub of the pool would show errors. If it is memory corruption then anything you do with the computer will make things worse, so maybe first run a memtest.
 
fabian said:
can you try the `cp` with `--sparse=never`?
Click to expand...
cp --sparse=never -- /var/tmp/source.iso /var/lib/vz/template/iso/dest.iso works. The resulting file has the same hash as the original. I also tested copying from /var/tmp to /var/tmp and from /var/tmp to /root with the normal cp invocation, and neither of those result in bad copies.

Edit: I also just tested copying a good ISO from /var/lib/vz/template/iso to /var/tmp without using --sparse=never, and it corrupted that to. The resulting bad ISO has the same checksum as the bad ISO that is created when I copy from /var/tmp to /var/lib/vz/template/iso.
 
Last edited:
leesteken said:
If it is disk corruption (due to drive/flash, cable, controller) then a scrub of the pool would show errors. If it is memory corruption then anything you do with the computer will make things worse, so maybe first run a memtest.
Click to expand...
The resulting bad file always has the same hash every time I test it. I don't know if that would be congruent with disk or memory corruption.
 
I don't know if this helps, but I did a hexdump of both files and diffed them. According to the diff, the first differences appear at 0x1896a0000. The good hexdump shows "1896a0000 4c00 2637 b051 436e 4ce6 37a9 5ba7 2c9d" and the bad hexdump shows "1896a0000 0000 0000 0000 0000 0000 0000 0000 0000". Here's the surrounding bytes from both dumps.

Edit: I've added a section of the hexdump from another affected ISO.

Code: 
SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO
good.dump
18969ff80 21c8 d76a 5e6d 31ac 0691 9944 00b1 9821
18969ff90 8b79 305a 7938 0030 2848 500c e51a 4107
18969ffa0 defc 73d0 30ba 560c 1b94 1183 c463 6420
18969ffb0 3119 3130 1188 1811 9880 3484 9631 424c
18969ffc0 a619 9746 dd19 14aa 55ab daa8 6ea4 eaf1
18969ffd0 bc21 ad51 a246 4d6a 2640 5586 79e9 0658
18969ffe0 1a45 6f33 9e49 f944 3412 94a9 018a d108
18969fff0 f041 2e0c 250b 1b35 e946 23a4 013c 0727
1896a0000 4c00 2637 b051 436e 4ce6 37a9 5ba7 2c9d
1896a0010 b245 1a94 6d16 a441 b309 4210 14c8 4e6f
1896a0020 0282 eab1 1b14 fb52 abd1 2fd2 9942 5979
1896a0030 ca60 7820 53e6 798d 091c 2133 02de 2f04
1896a0040 00ae e050 bc10 1c02 a597 a0df c561 94ca
1896a0050 5029 4264 31bd 6a23 ca0c 6f56 bc46 b33e
1896a0060 cd8a d5ab 5124 1ca9 4655 c8cd a794 a351
1896a0070 0362 2e00 218a 529c 823a 031e 2020 0243
-------------------------------------------------
bad.dump
18969ff80 21c8 d76a 5e6d 31ac 0691 9944 00b1 9821
18969ff90 8b79 305a 7938 0030 2848 500c e51a 4107
18969ffa0 defc 73d0 30ba 560c 1b94 1183 c463 6420
18969ffb0 3119 3130 1188 1811 9880 3484 9631 424c
18969ffc0 a619 9746 dd19 14aa 55ab daa8 6ea4 eaf1
18969ffd0 bc21 ad51 a246 4d6a 2640 5586 79e9 0658
18969ffe0 1a45 6f33 9e49 f944 3412 94a9 018a d108
18969fff0 f041 2e0c 250b 1b35 e946 23a4 013c 0727
1896a0000 0000 0000 0000 0000 0000 0000 0000 0000
*
18971a2c0 0000 ffff ffff ffff ffff ffff ffff ffff
18971a2d0 ffff ffff ffff ffff ffff ffff ffff ffff
*
18979a2c0 ffff ffff 0f00 0000 0000 0000 0000 0000
18979a2d0 0000 0000 0000 0000 0000 0000 0000 0000
*
18979a4c0 0000 4946 454c 0030 0003 17a2 0010 0000
18979a4d0 0000 0001 0001 0038 0001 01a0 0000 0400
18979a4e0 0000 0000 0000 0000 0000 0006 0000 0000
Code: 
SW_DVD9_Win_Server_STD_CORE_2025_24H2.4_64Bit_English_DC_STD_MLF_X23-96921.ISO
good2.dump
1bf1dff80 72d6 4815 c283 4827 498b 48f8 c12b 8348
1bf1dff90 f8c0 8348 1ff8 6877 37e8 ed08 0fff c057
1bf1dffa0 0ff3 457f 4cbf 7d89 48cf ff85 2f74 2b48
1bf1dffb0 48df e383 48f8 c78b 3b48 72de 4815 c383
1bf1dffc0 4827 7f8b 48f8 c72b 8348 f8c0 8348 1ff8
1bf1dffd0 3477 8b48 48d3 cf8b f7e8 ed07 33ff 48c0
1bf1dffe0 4d8b 48ff cc33 c5e8 ed07 48ff c481 00e8
1bf1dfff0 0000 5f41 5e41 5d41 5c41 5e5f 5d5b ccc3
1bf1e0000 f7e8 ec48 90ff f1e8 ec48 90ff 5be8 f70e
1bf1e0010 ccff cccc cccc cccc 5540 5653 4157 4856
1bf1e0020 ec8b 8348 50ec 8b48 2059 8b48 45f2 f633
1bf1e0030 8b48 441b 7338 0f19 3d85 0001 4800 438b
1bf1e0040 4820 788b 4810 3f8b 3844 1977 850f 009f
1bf1e0050 0000 8b4c 2047 8b48 48ce 538b 4d20 488b
1bf1e0060 4d10 408b 4808 128b 6fe8 ff15 48ff c085
1bf1e0070 3074 b880 0158 0000 7506 4827 508b 4868
-------------------------------------------------
bad2.dump
1bf1dff80 72d6 4815 c283 4827 498b 48f8 c12b 8348
1bf1dff90 f8c0 8348 1ff8 6877 37e8 ed08 0fff c057
1bf1dffa0 0ff3 457f 4cbf 7d89 48cf ff85 2f74 2b48
1bf1dffb0 48df e383 48f8 c78b 3b48 72de 4815 c383
1bf1dffc0 4827 7f8b 48f8 c72b 8348 f8c0 8348 1ff8
1bf1dffd0 3477 8b48 48d3 cf8b f7e8 ed07 33ff 48c0
1bf1dffe0 4d8b 48ff cc33 c5e8 ed07 48ff c481 00e8
1bf1dfff0 0000 5f41 5e41 5d41 5c41 5e5f 5d5b ccc3
1bf1e0000 0000 0000 0000 0000 0000 0000 0000 0000
*
1bf2459c0 0000 0000 0000 0000 0000 0000 0000 ffff
1bf2459d0 ffff ffff ffff ffff ffff ffff ffff ffff
*
1bf2c59d0 0f00 0000 0000 0000 0000 0000 0000 0000
1bf2c59e0 0000 0000 0000 0000 0000 0000 0000 0000
*
1bf2c5bc0 0000 0000 0000 0000 0000 0000 0000 4946
1bf2c5bd0 454c 0030 0003 17a2 0010 0000 0000 0001
1bf2c5be0 0001 0038 0001 01a0 0000 0400 0000 0000
 
Last edited:
It's a bit unexpected that the bad iso is the one with holes.
It would be interesting to see
- the file's allocation information before the copy
- how cp sees the holes.

Could you provide the following:
- The output of the breaking cp command(strace -f cp ...)
- The output of: zdb -Ovv rpool/ROOT/pve-1 var/tmp/SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO
(For the latter i assumed a default zfs raid1 installation from the ISO as you described - if you created more zfs datasets via zfs create or other ways, you'll need to adapt the dataset and path - the path is relative to the zfs dataset and should point to the *good* file)

I tried to just build a large ISO with some holes in it and upload that, but could not reproduce this on a virtual pve installation.
 
wbumiller said:
It's a bit unexpected that the bad iso is the one with holes.
It would be interesting to see
- the file's allocation information before the copy
- how cp sees the holes.

Could you provide the following:
- The output of the breaking cp command(strace -f cp ...)
- The output of: zdb -Ovv rpool/ROOT/pve-1 var/tmp/SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO
(For the latter i assumed a default zfs raid1 installation from the ISO as you described - if you created more zfs datasets via zfs create or other ways, you'll need to adapt the dataset and path - the path is relative to the zfs dataset and should point to the *good* file)

I tried to just build a large ISO with some holes in it and upload that, but could not reproduce this on a virtual pve installation.
Click to expand...
It looks like my strace and zdb outputs are too large to upload. They're about 3 MB and 8 MB respectively. Is there another way to send them to you, or is there a small section of them that you're interested in?

Edit: These files exceeded the upload limits of Pastebin and GitHub Gists so I've made a GitHub repo for them. The repo contains the output of strace -f cp -- /var/tmp/SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO /var/lib/vz/template/iso/SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.BAD.ISO and zdb -Ovv rpool/ROOT/pve-1 var/tmp/SW_DVD9_Win_Server_STD_CORE_2025_24H2.9_64Bit_English_DC_STD_MLF_X24-08243.ISO. I did confirm that strace didn't affect the corruption of the ISO. The bad ISO still has the same checksum as before. As for my ZFS setup, I'm pretty sure I didn't do anything fancy. All I did was select "zfs (RAID1)", select the two SSDs I wanted to use, and moved to the next screen. I don't think I even looked at the advanced options. Unfortunately I can't find a public download link for these ISOs straight from Microsoft, so I don't know how to share these files.
 
Last edited:
Okay I managed to reproduce the issue by producing a file with enough hole and large-enough filled extents. This seems to be an interaction between zfs and glibc (and the buggy behavior seems to be in glibc from initial analysis).
If you intend to upload more large files, you'll want to mount some non-ZFS file system on /var/tmp for the time being.
 
  • Like
Reactions: Kurgan, fiona, _gabriel and 2 others
You must log in or register to reply here.
Top Bottom