[Storage] Root FS on consumer hardware

[wilcomir]

Hello everyone,

I am rebuilding my home server from scratch - the one I am using right now will be demoted to a bare metal PBS host.

I will use consumer hardware, currently the leading option is an i9-13700 + 96 GB of DDR5.
As for storage, I plan to have an HBA passed through some flavour of NAS, but then there is the root FS, and the VM/LXC storage.

I have been reading quite a bit around, and I would like to understand whether I get any particular benefit in using NVMe storage for the root filesystem. The motherboard I plan to use has 4 NVMe slots, therefore I initially thought to use 2x 256GB in mirror for root, and 2x 2T in mirror for the VMs, but it seems to me that NVMe is not strictly necessary for the root FS.
As a data point, I currently have two hosts (that I want to consolidate), and in the root FS they run at about 100 GB per day of data written. It should be possible to find cheap SSDs in the hundredth of TBW; those would last on average 3 years. To mitigate this, I can even use four drives in mirror + stripe; this would even help with the IOPS problem hopefully.

To sum it up, my latest strategy would be:

- buy 2x 512 G/1 TB sata SSD (to get a lot of TBW), zfs mirror, root FS
- buy 2x 1 TB/2 TB NVMe, zfs mirror, VMs
- after maybe 1yr add 2 SSDs to the root pool, for striping, and to avoid all SSDs failing at once
- adding further NVMe would be done on a need-for-storage basis, but I am going to backup my VMs anyway so even a catastrophic failure would be recoverable there

Would this be a sensible plan?

Thanks a lot!
V

 

[Hqu]

Hey,

sounds like a valid plan. Check if needed/wanted some SSDs with power-loss protection, just in case of power outage as you write 100GB data per day IMO or a UPS

Best

 

[wilcomir]

Thanks for the reply!

Power loss protection might be interesting. I will search for sone consumer options then.

A UPS might also be a nice to have, but I am lucky enough to live in an area where power losses happen rarely if ever… maybe once every couple of years.

 

[Astraea]

Here is how my workstation system is setup, I have 2 smaller SSDs (500 GB) that I use for booting Proxmox VE and also storing the few ISO images that I need and the VM disks for the few VMs that run on the system (Firewall [], TrueNAS, Windows [Workstation with PCIe pass-through for a graphics card], Linux [Workstation with PCIe pass-through for a graphics card]. I then have 4 x 12TB HDDs and 2 x 2TB NVMe and 2 x 1TB SSDs all passed though to the TrueNAS VM, I use the HDDs for personal storage and the NVMe drives for fast storage (Steam game library) and the other SSDs for shared general storage.

I was also able to setup encryption on all the TrueNAS disks using the native tooling in TrueNAS and have the key files stored on the TrueNAS OS virtual disk and a backup on 2 other storage locations. I have the OS disks for Proxmox in a ZFS mirror with ZFS encryption and it has been fantastic with no issues after using the native installer and then modifying the ZFS pool to be encrypted.

 

[Hqu]

Thanks for the reply!

Power loss protection might be interesting. I will search for sone consumer options then.

A UPS might also be a nice to have, but I am lucky enough to live in an area where power losses happen rarely if ever… maybe once every couple of years.

Check geizhals.at (austrian skinflint price comparison website) if this can help you decide what's on the marked right now https://geizhals.at/?cat=hdssd&xf=4...z=&dist=&mail=&sort=p&bl1_id=1000#productlist

Best