Advice on PVE Storage Configuration for new Homelab

[Markus87]

Dear all,

For some first learnings I have PVE currently running on an OPNsense DEC-740 in a small scale scenario, mainly as OpenWrt router/firewall (1Gb/s fiber) and some smaller LXCs. As a next step, I would like to build a new server for an enthusiast Homelab, dealing with the following main use cases:

1. OpenWrt router/firewall for upgraded internet to fiber 25Gb/s (Initi7, living in Switzerland)
2. Potentially some IDS/IPS and VPN
3. Windows 11 VM (for various use cases, e.g. CAD) - I am primarily a MAC user
4. Home Assistant
5. NVR: Frigate w/ analysis (maybe Blue Iris in addition)
6. Plex server (I know that AMD is not the best for transcoding w/ Plex but main use case is watching raw data)
7. NAS/file share (home network and via VPN), current thought is OMV or possibly TrueNAS?
8. some smaller LXCs like ad guard, mail server, etc

So far, my plans are:

• CPU: AMD Ryzen 9 9950X w/ Noctua air cooler TBC
• RAM: 128GB (4 x 32GB) Crucial or Kingston DDR5 ECC UDIMMs
• Mainboard: ASRock Rack B650D4U w/ following usable interfaces/lanes
  • 1 x PCIe 5.0 x16 @ CPU --> bifurcating to x4 x4 x4 x4 w/ PCIe to 2 x SFF-8654-8i adapter
  • 1 x PCIe 5.0 x4 @ CPU --> reserved for Intel E810-XXVDA2
  • 1 x M.2 PCIe 5.0 x4 @ CPU --> possibly adding M.2 to SFF-8654-4i adapter
  • 1 x M.2 PCIe 4.0 x4 @ PCH (shared PCIe 4.0 x4) --> possibly adding M.2 to SFF-8654-4i adapter
  • 4 x SATA 6Gb/s @ PCH (shared PCIe 4.0 x4)
• U.2/U.3 cage: Icy Box ToughArmor MB118VP-B (hot swap for 6 x U.2/U.3 w/ 3 x SFF-8654-8i)
• SATA cage: TBD
• Nic: 25Gbe Intel E810-XXVDA2 (internet in / to main switch)
• Case: 3U or 4U TBC
• PSU: Seasonic Prime TX-750W
• AI accelerator (Frigate): Google Coral TPU USB

I am looking for some advice on storage configuration now. My current thoughts/questions are:

1. PVE/OS: 2 x SATA SSD in ZFS mirror
   1. I read that SATA SSDs are well suited as OS drives - no need for NVMe, correct? No limitation for 25Gb/s internet, correct?
   2. Would Micron 5400 Pro 480GB (1324TB TBW) be a good choice (I could get them for 80EUR per drive new)? Or is it worth going to the 5400 Max or any other advice?
2. Storage for VMs/LXCs: 4 x 7.68TB U.2/U.3 SSDs in parity (15TB usable)
   1. Which configuration would be best? 2 VDEVs w/ mirror or RAID-Z2 or any other?
   2. Does it make sense to have one shared storage pool for all VMs/LXCs (except NVR)?
   3. Would Micron 7400 Pro be suited for this? Or better the 7400 Max (6.4TB)?
   4. What kind of setup would make sense? ZFS via PVE or creating the storage pool in something like TrueNAS?
3. Storage for NVR (6-10 5MP cams) w/ AI analysis: 1 or 2 (parity?) x Micron 7400 Max 6.4TB U.2/U.3 SSD for short-term storage, clips & AI analysis + possibly add. Seagate Skyhawk AI HDD for "offload" storage up to 2 weeks (e.g. vacation)
   1. Does such a combination make sense?
4. Backup: The desired way to backup the PVE/OS drive and the VMs/LXCs is to have have a separate PBS w/ big HDDs in RAID or mirror configuration doing e.g. nightly backups, correct?
   1. What is best practice here? 2 HDDs in ZFS mirror? More HDDs in RAID?
   2. Which size should I go for considering the mentioned disk sizes above?
   3. Which (enterprise) disc models would you recommend considering durability, (minimal) loudness - see below, energy consumption, etc?

First sight, the server will be placed in my home office/ guest room and therefore needs to be reasonably quiet. In some years - in our own house - it will move into a rack.

Thanks a lot for any advice!

BR Markus

 

[Markus87]

Any advice would be highly appreciated

 

[czechsys]

ad 1] if only OS, any good sata ssd will work. 25Gbps network has nothing with it.
ad 2] shared storage for all VMs is normal. Better to use SSD with capacitor (Intel DC, Kingston DC etc) - depends on iops/size workload.

ad 4] Again, use case depends on iops/size requirements. PBS has SSD as requirement, HDD can be slow. PBS has it's cons/prons. Standard vzdump backup can work with hdd but it has limited functions vs pbs.

 

[Markus87]

Hi @czechsys,

Thanks a lot for your replies!
ad 1] thanks a lot for the confirmation!
ad 2] you're referring to power loss protection (PLP), correct? both the Micron 7400 Pro and Max are enterprise SSDs and feature PLP. But maybe I misunderstood your comment?
ad 4] I actually missed the SSD prerequisite for PBS, so the whole backup strategy is definitely something I have to re-consider

Does anybody have any further advice on the other questions?

BR Markus