Need storage advice for new setup

[flbit]

Hi guys,
I am new to Proxmox, coming from Unraid. I switched mostly because I need to study setting up VMs using packer, terraform, ansible etc ...
to make a long story short, I did some research on the right storage layout for me. I started to learn zfs, barely scratching the surface. Here is my issue:
I do not have ECC ram and I know that it’s very recommended if I want to use zfs. I cannot justified spending for an enterprise server these days.
So, here is what I have for storage:
- 1x120gb ssd (I guess for Proxmox)
- 4x500gb ssd
- 3x10tb hdd
I created a raidz1 pool for the 500gb and another raidz1 pool for the 10tb but now I don’t know if it’s a safe layout for my specs.
Basically I need some advice for the best storage layout I can use considering that I don’t want to lose data due to non ECC ram if I chose ZFS. What are my options if I want to use all my drives without having data corruption down the road ? Thanks

 

[Dunuin]

If that are consumer SSDs keep in mind that ZFS might kill them very fast depending on your workload. Here are enterprise grade SSDs recommended too because they much more durable (for example will last 30 times longer). You will get alot of writes because ZFS is a copy on write filesystem and mixed blocksizes and virtualization will cause overhead too. Especially small sync writes if you don't got a good SSD with powerloss protection so your SSDs can't use the internal RAM cache to optimize writes to lower the write amplification.

Also keep in mind to increase the volblocksize of your pools (for example 32K for your 3 HDDs and 64K for your 4 SSDs) or you will loose additional 33% of your capacity due to bad padding. And volblocksize can't be easily changed later because it can only be set at the creation of your VMs virtual disks. And a stripped mirror (raid10) might be a better idea if you want to use your SSDs as a VM storage. With a 4 SSDs as a stripped mirror 8K should be fine.

ZFS should run fine without ECC but if a bit flip occures in memory before creating the checksum your data is corrupted and ZFS will think everything is fine. So you will never absolutely know if or how much of of your data is damaged.

And don't forget to manually edit the ARC size. By default ZFS will use 50% or your hosts RAM. Rule of thumb would be 4GB + 1GB RAM per 1TB of raw storage. So 36GB ARC size would be a good value to start with. If everything is working fine you can run "arc_summary" and look at your hit rates and ARC usage and try to lower it if your are low on memory and don't need that storage to be fast.

 

[flbit]

useful info, thanks! I forgot to mention that I only have 64gb ram. My SSDs are just regular Samsung 870evo, I thought about running my VMs on the ssd pool and yes I am aware that zfs won’t warn me if my data is corrupted. I’d more concerned about losing anything on my 10tb pool where I would store my documents, media etc.. I also have a nvme 1tb on the mobo but I guess that is no substitute for not having ECC ram.

 

[Dunuin]

I’d more concerned about losing anything on my 10tb pool where I would store my documents, media etc

Raid never replaces a backup. Its very unlikely that a bit flip will degrade your complete pool but it could happen. But if you backup your data that wouldn't be a big problem either. You always could loose the complete pool due to ransomware, hardware failure, power outage without UPS, critical ZFS bugs, theft, fire, ... that should all be more likely.

 

[flbit]

Noted. Thanks @Dunuin !