Storage configuration for consumer SSDs (ZFS vs LVM)

[Santig]

Hi everyone,

I am trying to set up a new Proxmox server, but the last installation failed because one of the drives in my ZFS pool corrupted.
With this in mind, I researched ZFS a bit more and started hearing talk about write amplification in consumer-grade SSDs.
I don't fully understand how write amplification occurs. I seem to understand that 1 write to ZFS can be amplified into many writes to the physical disks.
A friend also said he heard something about write amplification being mitigated by using raw image instead of qcow2 (I don't fully understand what he meant; he just referenced this video)

So my question is, what would be the best way to configure storage on my server?
I have:
4x SSDs (Samsung 850 PRO 256GB) (one I intend on using for the OS)
4x HDDs (Seagate 500GB)
Should I use ZFS, thin-LVM, or LVM?
I will be setting up my disks in RAID5, and another consideration is that I have a PERC RAID controller in front of them.

 

[fstrankowski]

Which CPU, how much RAM, how many VMs, workload on the VMs, etc. Please provider a bigger picture. By just naming 2 type of drives there is no way to provide a proper answer to your (very broad) question

 

[Santig]

Sorry, I will try to provide as much info as I can.
CPU : Intel(R) Xeon(R) CPU E5630@2.53GHz (2 Sockets)
RAM: 144 GB
In my previous setup, I used 5/6 VMs, but I'm intent on having about 8
Regarding the VM workload, I intend on using this machine to run OpenCloud, Opensense, and some Docker containers on a Ubuntu VM (like Linkwarden)
Probably some automation like n8n and Jenkins
It is mostly a machine to run services for myself
My main goal is to minimize physical contact with the machine, given that I don't have 24/7 access to it

Also, thank you for dedicating your time to helping me with what I believe is a very broad and complex question.
I've been having a hard time committing to any decision over the last week, because I don't fully understand the trade-offs and keep getting conflicting information.
Hardware, ZFS, and low-level is not really a specialty for me, so I feel a be a bit out of my depth
So once again, thank you, and don't hesitate to ask for more info.

 

[networkguy3]

ZFS does not work/play well with RAID controllers. To use ZFS, you should have a disk controller in HBA mode

Proxmox in general does not work and play well with consumer SSDs. The boot drives get a surprising amount of traffic. It is significantly lower in stand alone hosts than in a cluster.

We still need more information to help you out. How much disk space do you need for VMs and for data. What are your performance goals?

However, just as a shot in the dark, my first step would be to flash the RAID controller to conver it to HBA mode. The next thing I would do is get some Intel SATA / NVME SSDs for both booting the machine and to store the VMs. I’d then use the spinning rust for data. I’d use ZFS all around. (I’ve been doing ZFS for well over a decade and am very comfortable with it.).