XFS Or ZFS On Consumer SSDs For VMs & Containers

[WhosTheBosch]

I'm trying to figure out the best way to configure the following storage devices for ~30 VMs / ~30 containers. I currently have:

Supermicro X11SPM-TF
Intel Silver 4210
128 GB of Ram (24GB max to ZFS)
32GB Supemicro SATADOM
240GB Intel S3520 SATA II
512GB Samsung 960 Pro NVME
2TB Samsung 970 Evo Plus NVME
4 x 10TB HDDs (Going to setup ZFS RAIDZ on them for photos / backup, not VM / Containers)

I'm planning on using the SATADOM to run Proxmox 6.1, and it will be ZFS as that's the best way to boot I've found using UEFI.

VMs: Elasticsearch, Windows Server 2019, Windows 10 GPU passthrough, Cent OS, Security Onion, Kali
Containers: Elasticsearch, Emby, Apache

Questions:

1) I was wondering if there's any benefit to using ZFS over XFS?

2) In Proxmox are ZFS datasets / volumes more performant than using an XFS directory?

3) What would be the best way to incorporate the S3520?

4) Is it worth using LVM thin pools or ZFS sparse volumes, or just leave that up to the VMs?

5) I've read that consumer drives would be slower than enterprise. Is that just for ZFS or also XFS? I'm not too worried about the data so endurance isn't a priority as I'll have backups (Borg) of anything important. What speeds could I reasonably expect from them?

6) When setting up whole disks for both VMs & containers, is there any benefit to directories compared to LVM?

 

[Dominic]

1, 2, 3) ZFS on Linux in Proxmox VE, also take a look at the section about caching
4) Do you need thin provisioning?
5) Enterprise drives are generally faster independent of the file system. See the fio section in our Ceph benchmark.
6) Directory storages can hold more content types than LVM. Take a look at the respective wiki pages: Directory, LVM