Help me on deciding storage options.

[chun02160]

Hello

I am building a new homelab server machine and I am trying to pick the storage. My main usage will be VMs as well as Containers...like Kubernetes Cluster in Proxmox, Home Assistant, TrueNas, Plex, Unifi Controller, AIML stuff, and many more. I am a Software person so when it comes to Hardware and all, I am still learning. After few search, here is the rough draft of what I am thinking of the storage:

• 2 x 250GB NVMe for boot drive: These two will be used for mirroring
• 3 x 8TB HDD: ZFS Pool with RAID1. This is where rest of stuff sits, VM, ISO, storages...
• 3 x 8TB HDD with HBA cable: This is future plan but I saw some post about TrueNAS separating storage using HBA,

I am not sure if this makes sense? I am open to suggestions! I've also seen some posts about having another 2xSSD for VM and then the rest goes to HDD?

Thank you so much!

 

[louie1961]

I am less than a year into my Proxmox journey so take this all with a grain of salt. Other more experienced folks may have a different point of view. That being said, 3 x 8TB in a raidz1 array will yield around 14-15 TB of usable storage. That is a LOT in my experience. Maybe too much? The thing to consider is how much memory are you going to put into the server, and how many VMs will you run. I have 64GB of ram, and I am running six VMs, two LXC containers, and two of my VMs run docker. I have 8 apps running in docker. I have something like 12 applications that I use or experiment with. I keep two backups and two snapshots for each VM. I am currently using less than 1TB of storage. I will run out of memory long before I run out of storage. But I don't run Plex, so YMMV

A couple of things I would recommend: Set up your router and your NAS on separate devices. I have been on a learning curve with Proxmox and I don''t want to take down my NAS or my home internet every time I blow up Proxmox or do a re-install because I didn't like my storage configuration for one reason or another. You can mount NFS storage from a standalone NAS into your Proxmox server, and store your backups and ISOs there. You don't need a lot of performance for those items, so NFS works fine. If I was starting from scratch I would do two really small SSDs for boot drives, in a ZFS mirror (and by small I mean maybe even 128GB. The OS doesn't take up hardly any space) and two SSDs in a ZFS mirror for the VMs. Me personally I would use the NVME slots for the largest drives you can afford and run the VMs on NVME. But there may be some risk to that if the NVMEs are not enterprise grade (which I am not sure they even make?). Its what I do but I am willing to take the risk, since all my critical data actually resides in the NAS (a separate box). I find running the VMs on NVME to be very performant. I would save the 3+ raid arrays for your NAS.

For what its worth, I am running NextCloud, Wordpress, Homeassistant, OpenMediaVault (for experimentation, not for production-I have a synology NAS for that, and a second NAS I built for onsite backup. I also back up to Amazon Glacier nightly), Portainer, Heimdall, Guacamole, Photoprism, Monica, Grocy, Mealie, Tracks, LeanTime, and a couple of flavors of desktop Linux that I am test driving.

 

[Dunuin]

For ZFS you usually want SSDs with a power-loss protection, which only Enterprise/Datacenter grade SSD got. And virtualization needs IOPS performance, where HDDs are terrible. So I would buy two Enterprise SSDs for the PVE system + VMs/LXCs and use them as a ZFS mirror. HDD I would only use for cold storage. You then probably want all of them on the HBA, so TrueNAS can share them back to the PVE host as well as all those VMs using SMB/NFS/iSCSI.
NVMe SSD as boot drives would be a waste of money and hardware capabilities. Even some slow HDDs would be fine for that. And you don'T need dedicated boot disks. If you want it fast, get two U.2 Enterprise SSDs for system and VMs and get some M.2 to U.2 adapters. There are a few M.2 enterprise SSDs, but then you are very limited in options drive capacities. If you want it cheap, don't use the M.2 slots at all and get some refurbished SATA enterprise SSD.

 

[chun02160]

For ZFS you usually want SSDs with a power-loss protection, which only Enterprise/Datacenter grade SSD got. And virtualization needs IOPS performance, where HDDs are terrible. So I would buy two Enterprise SSDs for the PVE system + VMs/LXCs and use them as a ZFS mirror. HDD I would only use for cold storage. You then probably want all of them on the HBA, so TrueNAS can share them back to the PVE host as well as all those VMs using SMB/NFS/iSCSI.
NVMe SSD as boot drives would be a waste of money and hardware capabilities. Even some slow HDDs would be fine for that. And you don'T need dedicated boot disks. If you want it fast, get two U.2 Enterprise SSDs for system and VMs and get some M.2 to U.2 adapters. There are a few M.2 enterprise SSDs, but then you are very limited in options drive capacities. If you want it cheap, don't use the M.2 slots at all and get some refurbished SATA enterprise SSD.

Great! So From what I am understanding and thinking (Please correct me if I am wrong,),

• 2 Enterprise SSD where I put all the proxmox stuff (boot, VM, and etc). ZFS pool RAID1 mirror.
• 2 regular HDD through HBA (TrueNas, VM using SMB/FHS/iSCI). Another ZFS pool RAID1

What does this sound like? Appreciate your help again!

Also would a refurbished enterprise still be decent enough? I am bit scared since it's harddrive and used?

 

[chun02160]

I am less than a year into my Proxmox journey so take this all with a grain of salt. Other more experienced folks may have a different point of view. That being said, 3 x 8TB in a raidz1 array will yield around 14-15 TB of usable storage. That is a LOT in my experience. Maybe too much? The thing to consider is how much memory are you going to put into the server, and how many VMs will you run. I have 64GB of ram, and I am running six VMs, two LXC containers, and two of my VMs run docker. I have 8 apps running in docker. I have something like 12 applications that I use or experiment with. I keep two backups and two snapshots for each VM. I am currently using less than 1TB of storage. I will run out of memory long before I run out of storage. But I don't run Plex, so YMMV

A couple of things I would recommend: Set up your router and your NAS on separate devices. I have been on a learning curve with Proxmox and I don''t want to take down my NAS or my home internet every time I blow up Proxmox or do a re-install because I didn't like my storage configuration for one reason or another. You can mount NFS storage from a standalone NAS into your Proxmox server, and store your backups and ISOs there. You don't need a lot of performance for those items, so NFS works fine. If I was starting from scratch I would do two really small SSDs for boot drives, in a ZFS mirror (and by small I mean maybe even 128GB. The OS doesn't take up hardly any space) and two SSDs in a ZFS mirror for the VMs. Me personally I would use the NVME slots for the largest drives you can afford and run the VMs on NVME. But there may be some risk to that if the NVMEs are not enterprise grade (which I am not sure they even make?). Its what I do but I am willing to take the risk, since all my critical data actually resides in the NAS (a separate box). I find running the VMs on NVME to be very performant. I would save the 3+ raid arrays for your NAS.

For what its worth, I am running NextCloud, Wordpress, Homeassistant, OpenMediaVault (for experimentation, not for production-I have a synology NAS for that, and a second NAS I built for onsite backup. I also back up to Amazon Glacier nightly), Portainer, Heimdall, Guacamole, Photoprism, Monica, Grocy, Mealie, Tracks, LeanTime, and a couple of flavors of desktop Linux that I am test driving.

Thank you so much! After reading the post, I might overthink about the size and layout (Thanks for your help again).

I mentioned my plan in comment above, but thinking about going 2 SSD (as you and another user mentioned) in enterprise grade SSD to do ZFS pool RAID1 which has boot, VM, and all the heavy stuff, and regular HDD for none heavy stuff. I am also curious about your thought?

Thank you!

 

[Dunuin]

Great! So From what I am understanding and thinking (Please correct me if I am wrong,),

• 2 Enterprise SSD where I put all the proxmox stuff (boot, VM, and etc). ZFS pool RAID1 mirror.
• 2 regular HDD through HBA (TrueNas, VM using SMB/FHS/iSCI). Another ZFS pool RAID1

Yes. But if you want the HDDs managed by TrueNAS you don't need to create a ZFS pool with them on PVE. You would use PCI passthrough to passthrough the whole HBA with all disks attached to it, into your TrueNAS VM. TrueNAS then could directly access the unpartitioned physical HDDs and you would let TrueNAS create the ZFS pool. For the HDDs a raid1 would be fine and you could add pairs of disks later and convert it into raid10 when running out of space (keep in mind that you usually don't want to fill the pool more than 80-90% or it will become slower and snapshotting can also eat up alot of space). As a cold storage for big files like videos I would probably prefer something like a 6 or 8 disk raidz2 for better space efficiency and additional reliability as the resilvering of a pool of 20TB HDDs might take days or even weeks to finish. But a raidz1/2/3 isn't that easy expandable, so you then probably want to buy all the disks as once you think you will need the next years.

Also would a refurbished enterprise still be decent enough? I am bit scared since it's harddrive and used?

Good old enterprise SSDs are way more durable compared to new consumer SSDs. A SSD with eMLC NAND for example can be written 30 times as often as a SSD with QLC NAND before failing. So even if such a eMLC SSD shows 90% wear, it might still survice 3 times more writes than a never used new QLC SSD. New is of cause always better (controller chips and so on could fail of old age too, even if the NAND is still fine) but if I need to decide between a new consumer SSD and a used enterprise SSD in good condition, I would always prefer the used enterprise SSD. But a good idea to ask the seller for a SMART report so you can check the disks metrics before buying.