Add a Windows partitioned Hard Drive to a VM to copy data from it

[Leo_1o]

I'm working on a Minecraft Server, which i installed per Proxmox on a Ubuntu VM. One Project is to map out a whole custom world with dynmap to play on it. My Friend renderd all the data for it when I was doing Vacation on his Gaming PC (Windows) and it still took him 3 Weeks.
So now I have 32 Million little jpgs of the World, which all together create the map. I did the Maths and even if I'm in the same Network with the Server PC, it'll take 1,5 Weeks of 24/7 Networktraffic with FileZilla to move all this data.

So... The Threadtitle could be my solution (if it works...). I couldn't find any Advice on the Internet for this Specific use case, so i'll give this a Try!
Other Solutions (mabye simpler) are surely welcome!

 

[Neobin]

What is the overall size we are talking about?

Before fiddling around with the drive and risking messing up the partition table and/or loosing data while trying to passthrough the disk for example, I would probably pack (without compression) the files, transfer the one big file over the LAN to the VM and unpack it there.

Hopefully those million little files are currently and will be on NVMes/SSDs.

 

[Dunuin]

With default "jpg-85" settings it should be around 40kb per title. In such a case that would mean roughly 1.2TB. Biggest problem should be the IOPS performance of the HDD. Not sure if it would be faster by directly attaching the disk to your VM, which you could do by installing cifs-utils package and then using disk passthrough like described here: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM))
Lets say that HDD can handle 100 IOPS, then copying 32 Mio files would still need 320,000 seconds or 3.7 days.
Better tell your friend next time to use a SSD for such workloads

And I hope your friend used a well tested quality setting. Set the quality too low and you need to render it all again because it looks like scrap with compression artifacts. Set the quality too high and you also need to render it again because it's wasting hundreds of gigabytes of valuable SSD storage.
I did some test renderings with "png", "jpg-q100", "jpg-q95", "jpg-q90", "jpg-q85", "jpg-q80", "jpg-q75", "jpg-q70" and even "webp-q85". "png"of cause looked best but also wasted way too much space. Roughly 2.5x the size of the default "jpg-85". "webp-q85" looked a bit better than "jpg-q85" but I got alot of strange rendering errors so it wasn't usable and looked blurry at some tiles. In the end I sticked with "jpg-q90" because as it wasn't that much bigger but with noticable better image quality.

 

[Leo_1o]

What is the overall size we are talking about?

Before fiddling around with the drive and risking messing up the partition table and/or loosing data while trying to passthrough the disk for example, I would probably pack (without compression) the files, transfer the one big file over the LAN to the VM and unpack it there.

Hopefully those million little files are currently and will be on NVMes/SSDs.

Well because of that, i bought a Corsair MP600 Nvme SSD, but my friend unfortunately couldn't provide the files on an SSD. So we ended up with HDD speeds. The Files are 200GB big (But they are packed in a folderlabyrinth because of the structure the Rendersoftware renders stuff).

 

[Leo_1o]

With default "jpg-85" settings it should be around 40kb per title. In such a case that would mean roughly 1.2TB. Biggest problem should be the IOPS performance of the HDD. Not sure if it would be faster by directly attaching the disk to your VM, which you could do by installing cifs-utils package and then using disk passthrough like described here: https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM))
Lets say that HDD can handle 100 IOPS, then copying 32 Mio files would still need 320,000 seconds or 3.7 days.
Better tell your friend next time to use a SSD for such workloads

And I hope your friend used a well tested quality setting. Set the quality too low and you need to render it all again because it looks like scrap with compression artifacts. Set the quality too high and you also need to render it again because it's wasting hundreds of gigabytes of valuable SSD storage.
I did some test renderings with "png", "jpg-q100", "jpg-q95", "jpg-q90", "jpg-q85", "jpg-q80", "jpg-q75", "jpg-q70" and even "webp-q85". "png"of cause looked best but also wasted way too much space. Roughly 2.5x the size of the default "jpg-85". "webp-q85" looked a bit better than "jpg-q85" but I got alot of strange rendering errors so it wasn't usable and looked blurry at some tiles. In the end I sticked with "jpg-q90" because as it wasn't that much bigger but with noticable better image quality.

I know this sounds stupid, but we rendered all the files on default settings (hi-res all 3 Maps (Flat, 3D, Cave)). Then we discovered that you can change that in the configs... We didn't even enabled the "bigworld" feature (which would be better for literally everything we have problems with).