SSD NVME Best Practices

[gneto]

I would like a suggestion what the best practice for 5 2tb nvme ssd on a single node if I use ZFSz, LVM or other practice.
I need basic security and data corruption protection.
I will have the Proxmox native Backup server separate.
Thanks

 

[Dunuin]

First a raidz of 5 SSDs will result in either alot of wasted space (you loose half of your capacity like a raid10 but with worse performance) or you are forced to a high blocksize (32K or more; so you don't want to run anything inside VMs that uses a blocksize of 16K or less) so running alot of workloads like Postgres/MySQL DBs will be terrible. And ZFS got alot of overhead and is doing alot of sync writes, so consumer SSDs aren't recommended to use.
LVM on the other hand won't wear your disks that bad, but you are missing the bit rot (corruption) protection.

 

[gneto]

I understand, the recommended would be to use LVM to reduce the loss of space with the consequence of having the files rotten.
So, would it be a better solution to attach a TrueNas to solve this using its compression and ZFS itself?
Or do I directly configure this internal LVM?
Thank you for your cooperation and congratulations for your knowledge.

 

[Dunuin]

TrueNAS is also just using ZFS. Its the same as running ZFS directly on the PVE host. You can also use compression with ZFS on PVE, but that won't help much. You still got the problem of either padding overhead or very big blocksizes. Any kind of raidz1/2/3 or draid just isn't great if you want to run VMs on it (and especialy database VMs).
Best would be to use a striped mirror (raid10) and not a raidz1/2 (raid5/6). Then you don't get this padding overhead while still being able to use a relatively small volblocksize (8K for 4 disks or 16K for 6/8 disks). "1x" means performance of a single SSD, "2x" double the performance of a singe SSD and so on.

	5x 2TB raidz1 @ 8K volblocksize:	5x 2TB raidz1 @ 32K volblocksize:	4x 2TB striped mirror @ 8K volblocksize:	6x 2TB striped mirror @ 16K volblocksize:	8x 2TB striped mirror @ 16k volblocksize:
Theoretical IOPS performance for 32+K IO:	1x	1x	2x	3x	4x
Theoretical Write Throughput Performance for 32+K IO:	4x	4x	2x	3x	4x
Theoretical Read Throughput performance for 32+K IO:	4x	4x	4x	6x	8x
MySQL Performance might look more like this (16K sync IO):	1x	0.5x	2x	3x	4x
Postgres Performance might look more like this (8K sync IO):	1x	0.25x	2x	1.5x	2x
Padding Overhead loss (only for VMs/zvols):	30%	0%	0%	0%	0%
Parity loss:	20%	20%	50%	50%	50%
Real usable capacity for LXCs:	64% or 6.4 TB	64% or 6.4 TB	40% or 3.2TB	40% or 4.8TB	40% or 6.4TB
Real usable capacity for VMs:	40% or 4TB	64% or 6.4TB	40% or 3.2TB	40% or 4.8 TB	40% or 6.4TB
Disks may fail:	1	1	1-2	1-3	1-4

But keep in mind that there is additional massive write amplification when writing with a lower blocksize to a bigger blocksize. So doing random 4k sync writes to a zvol using a 32K volblocksize you might only get something like 12.5% of the IOPS/Throughput performance while increasing disk wear by factor 8.

So if you want to stick with raidz1 you have to choose if you either want good capacity with bad performance and high wear (volblocksize=32k) or bad capacity with ok performance and ok wear (volblocksize=8k).

Or getting more SSDs and switch to a striped mirror for bad capacity but good performance and ok wear.

 

[gneto]

Thank you very much, I will try here to use Raid 10 as you suggested, I believe that besides being coherent, it seems to be the best solution for the short and long term