Disk format to recommend use with oversize files

[asukarei]

Hi,

I am new to Proxmox.
We intend to build a small lab to try this great product.
The business scenario is that we have high CPU core usage and large file (txt like) to process.
Each file can range from 1GB to 5 GB.
What will be the recommend storage format to do this with focus on speed (processing)?
Raid1 will be considered as well.


Any thoughts are welcome.

 

[bbgeek17]

Each file can range from 1GB to 5 GB.
What will be the recommend storage format to do this with focus on speed (processing)?
Raid1 will be considered as well.

Any modern filesystem can handle this. The bottleneck will be the disk hardware and controller. Pick a good enterprise NVMe, you can add R1 for redundancy.
Assuming that this workload is running inside the VM, the rest will depend on the appropriate VM configuration.
As far as disk configuration, there are many options: pass-through, LVM, ZFS, etc.

It seems like you are very performance-focused, where microseconds count - the best way to pick storage is to try your exact workload on each of the variants.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[asukarei]

let me thank you for such a quick reply.

What if I want to add some sort of NAS to store that file.
Let’s say I can do 2.5GB to 10 GB network
Will a particular format better than another (zfs, LVN and etc)?

 

[bbgeek17]

What if I want to add some sort of NAS to store that file.
Let’s say I can do 2.5GB to 10 GB network

Do you mean Gbit?

Will a particular format better than another (zfs, LVN and etc)?

Likely neither of the above. NAS, generally, implies an NFS or a CIFS type storage, which provides file access. In most cases you don't have control over the underlying NAS filesystem.

If your "NAS" provides block type storage (iSCSI, NVMe/TCP), then the choice of the filesystem will depend on your use-case. Specifically, whether or not you will dedicate a LUN/Disk to a VM, or would like to use Shared storage.

Its very unlikely that your bottleneck will be the filesystem type. The best approach is to try your particular workload and compare. There is no one right answer - it always "depends".

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox