Disk Selection Help Please

[AnThRaX]

I know this question has been asked a bunch of times, as I've read tons of posts about it. However, reading them and seeing so many different answers, I'm a little confused as to what the best disk in my setup would be for installation. My disks will be as follows:
14x 6TB EXOS drives
2x 1TB SATA SSD
1x 250GB NVME SSD

I have 12 other disks, ranging from 2-10TB, however I don't plan to use any of those except maybe 3x 4TB disks maybe for backups if that's a viable option.

Should I setup the 2x 1TB SATA SSD's in a raid and use those for the installation of Proxmox or should I use a different route?

Again, I know this kind of question has been asked many times and I'm sorry to ask AGAIN, but with all of the varying answers, I'm still a bit confused.
Thanks in advance for your time!

 

[Dunuin]

What are you planning to store on which disks?
Do you want to use HW raid or ZFS?
What reliablity do you need?

Best would be to install PVE to a dedicated mirror just for the OS. You don't need much space for it (16-32GB should be fine if you don't want to store ISOs/templates/backups there) and performance isn't really important (HDDs would be fine).
Then a enterprise SSD only storage for your guests virtual disks.
And a HDD based storage (maybe with a pair of SSD as a special device for storing metadata in case of ZFS) for cold data.
And a dedicated array/pool for your backups in case you don't also store them offsite or on another machine.

 

[AnThRaX]

What are you planning to store on which disks?
Do you want to use HW raid or ZFS?
What reliablity do you need?

Best would be to install PVE to a dedicated mirror just for the OS. You don't need much space for it (16-32GB should be fine if you don't want to store ISOs/templates/backups there) and performance isn't really important (HDDs would be fine).
Then a enterprise SSD only storage for your guests virtual disks.
And a HDD based storage (maybe with a pair of SSD as a special device for storing metadata in case of ZFS) for cold data.
And a dedicated array/pool for your backups in case you don't also store them offsite or on another machine.

I plan to just have media such as shows and such on the disks, and photos, phone backups, what-not. I will be using ZFS. I plan to have it setup so I'm able to lose 2 disks. I'll be using LXC with docker for Authelia/Traefik various things. I'm coming from unRaid so this is a curve for me. Trying to learn! I don't plan on using any VM's actually, other than the LXC. Basically this server is for personal media/movie/tb library, a couple of self-hosted game servers and phone backups.

 

[Dunuin]

Installing just PVE to those two 1TB SSDs is a bit of a waste because you don't need all the speed and capacity. And that NVMe is in my opinion only useful for caching if you don't got a second NVMe slot to mirror it with a second NVMe SSD. I guess I would use that NVMe as a swap partition or as a SLOG for you HDD pool in case you got alot of sync writes (wont help anything with async writes).

The HDDs you could then use as a 14 disk raidz2 + 3 (Enterprise) 500GB (around 300GB would be needed) SSDs as special devices in a threeway mirror. That way your HDDs should be alot faster as they don't need to store any metadata and are that way not hit by so much IO operations.
But it might also be not a bad idea to use a striped mirror or atleast striping some smaller raidz1 or raidz2 vdevs. In case you loose a disk a resivering can take quite a long time (for example weeks) and this would be faster when striping smaller vdevs together. And the more vdevs you stripe together, the better your HDDs can handle small files (= more IOPS performance).
A new alternative to a raidz2 would be a draid which will help with resilvering times: http://storagegaga.com/raidz-expansion-and-draid-excellent-openzfs-adventure/

You could use the two 1TB SSDs in a mirror for both your PVE OS and VM/LXC storage. But keep in mind that ZFS got alot of overhead and will wear the SSDs really fast. Enterprise SSDs are really highly recommended here. First they are 3 to 30 times more durable, then they got way better sync write performance, can sustain continous writes with good performance and not just short write burst and the got powerloss protection, so you don't get corrupted data (or loose the whole data) on power outages.

 

[AnThRaX]

Installing just PVE to those two 1TB SSDs is a bit of a waste because you don't need all the speed and capacity. And that NVMe is in my opinion only useful for caching if you don't got a second NVMe slot to mirror it with a second NVMe SSD. I guess I would use that NVMe as a swap partition or as a SLOG for you HDD pool in case you got alot of sync writes (wont help anything with async writes).

The HDDs you could then use as a 14 disk raidz2 + 3 (Enterprise) 500GB (around 300GB would be needed) SSDs as special devices in a threeway mirror. That way your HDDs should be alot faster as they don't need to store any metadata and are that way not hit by so much IO operations.
But it might also be not a bad idea to use a striped mirror or atleast striping some smaller raidz1 or raidz2 vdevs. In case you loose a disk a resivering can take quite a long time (for example weeks) and this would be faster when striping smaller vdevs together. And the more vdevs you stripe together, the better your HDDs can handle small files (= more IOPS performance).
A new alternative to a raidz2 would be a draid which will help with resilvering times: http://storagegaga.com/raidz-expansion-and-draid-excellent-openzfs-adventure/

You could use the two 1TB SSDs in a mirror for both your PVE OS and VM/LXC storage. But keep in mind that ZFS got alot of overhead and will wear the SSDs really fast. Enterprise SSDs are really highly recommended here. First they are 3 to 30 times more durable, then they got way better sync write performance, can sustain continous writes with good performance and not just short write burst and the got powerloss protection, so you don't get corrupted data (or loose the whole data) on power outages.

Thank you very much for your time and response. I will take these into consideration. I don't really have the money to afford some enterprise SSD's, so I'll have to figure out what to do instead, don't really wanna kill my SSD's in a month or somethin crazy LOL.

 

[Dunuin]

What SSDs do you actually got there? On the long run enterprise SSDs are way cheaper compared to consumer SSD. For example:

	Price:	Price per TB capacty:	Durability rating (TBW):	Price per 1TB written:
Kingston A400 480GB (bad consumer QLC SSD; QLC):	44€	92€	160 TB	0.275€
Samsung 870 EVO 500GB (reasonable consumer SSD; TLC):	55€	110€	300 TB	0.1833 €
Samsung 860 PRO 512GB (good consumer SSD; MLC):	440€	859€	600 TB	0.7333€
Intel SSD D3-S4510 480GB (cheap enterprise SSD; TLC):	130€	271€	1,200 TB	0.1083€
Intel SSD D3-S4610 480GB (reasonable enterprise SSD; TLC):	251€	523€	3,000 TB	0.0837€
Intel SSD DC S3710 400GB (good enterprise SSD; MLC):	224€	560€	8,300 TB	0.027€
Intel Optane DC P5800X 400GB (highend enterprise SSD; SLC):	1,504€	3,760€	73,000 TB	0.0206€

Sadly Intel and Samsung discontinued the MLC SSD ranges. So the S3710 is way cheaper and the 860 PRO way more expensive than initially as they are discontinued and are only selling remainders.

So you initially pay more for the enterprise SSD but in the long term its still cheaper as the SSD should life longer. Better buying a D3-S4610 all 10 years for 251€ than buying each year a new 870 EVO for 55€.

But if you see that a P5800X can handle 73,000 TB and a A400 just 160 TB, even if the P5800X got less capacity, you might understand why consumer SSDs might die quickly when used for server workloads and using it with enterprise grade storage solutions like ZFS.

 

[AnThRaX]

What SSDs do you actually got there? On the long run enterprise SSDs are way cheaper compared to consumer SSD. For example:

	Price:	Price per TB capacty:	Durability rating (TBW):	Price per 1TB written:
Kingston A400 480GB (bad consumer QLC SSD; QLC):	44€	92€	160 TB	0.275€
Samsung 870 EVO 500GB (reasonable consumer SSD; TLC):	55€	110€	300 TB	0.1833 €
Samsung 860 PRO 512GB (good consumer SSD; MLC):	440€	859€	600 TB	0.7333€
Intel SSD D3-S4510 480GB (cheap enterprise SSD; TLC):	130€	271€	1,200 TB	0.1083€
Intel SSD D3-S4610 480GB (reasonable enterprise SSD; TLC):	251€	523€	3,000 TB	0.0837€
Intel SSD DC S3710 400GB (good enterprise SSD; MLC):	224€	560€	8,300 TB	0.027€
Intel Optane DC P5800X 400GB (highend enterprise SSD; SLC):	1,504€	3,760€	73,000 TB	0.0206€

Sadly Intel and Samsung discontinued the MLC SSD ranges. So the S3710 is way cheaper and the 860 PRO way more expensive than initially as they are discontinued and are only selling remainders.

So you initially pay more for the enterprise SSD but in the long term its still cheaper as the SSD should life longer. Better buying a D3-S4610 all 10 years for 251€ than buying each year a new 870 EVO for 55€.

But if you see that a P5800X can handle 73,000 TB and a A400 just 160 TB, even if the P5800X got less capacity, you might understand why consumer SSDs might die quickly when used for server workloads and using it with enterprise grade storage solutions like ZFS.

I have some cheap Crucial MX500 1TB's.