Recommended Storage Type, RAID vs No RAID, ZFS vs no ZFS

[gorillaz]

Hello everyone,

I'm new to proxmox and am not sure about whether I should go with ZFS and/or RAID or without for my hardware.

My hardware:
Lenovo ThinkCentre with Intel i5-9400T 1.80 Ghz
32 GB Non-ECC RAM
1x 256 GB M2-SSD for Proxmox installation
4x 4TB HDD

What I already know is that ZFS has many advantages over the other options but I'm not sure if it is suitable for my hardware because of the resource overhead and Non-ECC memory (although data corruption because of that should be rather rare as far as i know).
Also I'm not sure what would happen if I use ZFS without RAID and one drive fails. Is the whole pool then non-recoverable or just the failed drive? Because if not, I'd rather go with no RAID (I've had trouble with RAID before, being stuck in a reconstruction loop and other things) and have an extra 4TB cause I would have backup if a failure happens.
And is it true that if the proxmox installation is broken or my machine stops working, I'm not able to access my data on another machine when using ZFS with or without RAID?

I'm really interested in your long-term experiences of those different options (ZFS, ZFS with RAID, no ZFS like lvm, ...) and recommendations for my specific hardware.

Thanks for your help

 

[Blueloop]

Ideally you need to address redundancy from the other side, ie not: "I have this and" but "I want this". However you have some hardware and want to make the best of it.

You have a M2 and a bunch of spinning rust and you have decided to use the M2 for the OS and it looks like you want to use the spinning discs for data.

If you have a hardware RAID controller with battery backed cache then use that: Four HDDs as RAID 5 and slap an ext4 or xfs on it. If you do not have a hardware RAID controller then you should consider a software alternative. ZFS is indicated. Don't worry about ECC too much unless you are going to massively increase your budget.

In the end it really boils down to how important your data is to you and what your budget looks like. Don't mix funky filesystems like ZFS that can do their own RAID with hardware RAID - its a waste. ext4 or xfs on a RAID or ZFS, BTRFS, or BCacheFS on disparate discs.

Don't overthink it when starting out with this stuff. Make sure you have backups of important data and then experiment.

 

[gorillaz]

Thanks for your response.
And yes I think you’re right, I am overthinking this actually.
And since I have backups of all my important data it shouldn’t be a problem to experiment.
So I’ll just go with ZFS, since I don’t have hardware RAID, and see how it goes.

 

[Blueloop]

Do make sure that you can restore from your backups first before mucking about.

 

[_gabriel]

Four HDDs as RAID 5

IOPS is slower than RAID10.
RAID10 is recommended for virtualization.
in ZFS it's called striped mirrored disks.
ZFS slowdown disks, if you host file sharing why not, but if you host many guests they will crawl down.

 

[louie1961]

Personally I would go with a larger NVMe drive (1-2 TB) and put both Proxmox and your VMs on it. I wouldn't probably wouldn't use ZFS since its just one drive and you don't have much memory. I would dedicate one of those 4TB spinners to Proxmox as a data store for ISOs, CT templates, etc. The other three spinners I would pass through to something like OpenMediaVault to provide NFS shares to your VMs and docker. I say OMV instead of TrueNAS since OMV is a lot less resource intensive. Inside of OMV, you could create a software raid with those three disks using MDADM then format the file system with BTRFS. This way you get snapshots and data scrubs but without the problems that BTRFS sometimes has with creating raid arrays. I nhave done this and it works great. If you absolutely can not get a larger NVME, then I guess you will have to put your VMs on one of those 4TB spinners. But you will be much happier running your VMs from an NVME.