Advice for disk layout and RAID type

[sudoku]

Hello,

I understand that this question has been asked many times in the forum but still I'm not so sure about what I'm planning to do. I'm wondering if the disk layout I'm thinking about make sense. Basically, I have the following disks:
- 2 NVMe
- 12 SSDs (8 on the motherboard SATA and 4 on HBA PCIe).

My plan:
- Proxmox OS: 2 NVMe -- ZFS -- RAID1
- Data storage: 8 SSD -- ZFS -- RAID10
- VM storage: 4 SSD from HBA PCIe -- ZFS -- RAID10

I'm going to use Proxmox principally with VMs (for servers) and the data storage is not for a NAS-like storage but for the VMs data.

Is my plan relevant ?

 

[sudoku]

I would like to add. As I'm using an HBA, is it ok if mix my SSDs plug on motherboard and the HBA or should I use the HBA for any specific role ?

 

[VictorSTS]

Looks ok to me, mainly because you dont use RAIDz as VM disks storage. You should be able of mixing disks of the HBA and the mobo in the same zpool, but to give exact recomendation on performance, etc you must know the exact NUMA configuration and PCIe lanes of the server.

 

[sudoku]

Great, thank you.

 

[DanTheServerMan]

Looks acceptable to me.

 

[bbrendon]

Hello,

I understand that this question has been asked many times in the forum but still I'm not so sure about what I'm planning to do. I'm wondering if the disk layout I'm thinking about make sense. Basically, I have the following disks:
- 2 NVMe
- 12 SSDs (8 on the motherboard SATA and 4 on HBA PCIe).

My plan:
- Proxmox OS: 2 NVMe -- ZFS -- RAID1
- Data storage: 8 SSD -- ZFS -- RAID10
- VM storage: 4 SSD from HBA PCIe -- ZFS -- RAID10

I'm going to use Proxmox principally with VMs (for servers) and the data storage is not for a NAS-like storage but for the VMs data.

Is my plan relevant ?

No, doesn't make sense to me.

Why waste NVMEs on a boot OS you can run on a USB stick?

I would use the NVMEs on either VMs or use it as a ZIL for ZFS.

 

[DanTheServerMan]

No, doesn't make sense to me.

Why waste NVMEs on a boot OS you can run on a USB stick?

I would use the NVMEs on either VMs or use it as a ZIL for ZFS.

The OP stated that they already have the NVMe drives, therefore they have two specific disks that must find a purpose for.

Why run anything on a USB in 2025? ProxMox doesn't run in memory like ESXi does (and even they don't recommend USBs anymore..). USB's are horrendously slow, things like package updates are going to be slow, and USBs do not have near the endurance or reliability as a NVMe drive.

You have a point regarding the ZIL, perhaps they could use one NVMe for the OS and the other for ZIL. However, unless they're doing lots of sync writes, then a ZIL may not benefit them much. Plus, we don't even know if they're enterprise NVMes. If they're consumer grade, that NVMe may get shredded as a single disk.

It's not my server, and you can discuss with the OP if you have your own opinions. I am just explaining why I said it makes sense, given that they said they already owned the hardware. It would not be how I personally set it up from scratch, but we have no info on the platform or context to give a better answer.

 

[VictorSTS]

Why waste NVMEs on a boot OS you can run on a USB stick?

Running PVE on SD card or USB drive is not supported nor recommended due to the use PVE/Debian does of the OS disk. Also, not having redundant OS disk is nonsense for me for anything except testing machines.

I would use the NVMEs on either VMs or use it as a ZIL for ZFS.

Having SSD also for the main storage, having ZIL on NVMe will add nothing but complexity, as those SATA SSD will be more than capable of doing sync writes fast enough. Not to mention that ZIL is only useful when the workload does sync writes only.

As a final note, you can partiion the NVMe drives and leave say 50G for the OS and use the rest of the NVMe drives for any other purpose. And use enterprise drives with PLP!

 

[sudoku]

Thanks guys !!

It's just an homelab, if I'm not optimizing and the layout is still relevant its fine for me. I just wanted other opinions so I can make a (relevant) choice.

FYI, I own the following hardware:
- motherboard: X570D4U-2L2T
- 2 NVMe Western Digital Blue SN580
- 12 SSDs MX500 (4 on HBA, 8 on the motherboard)
- CPU Ryzen 5 - 3600 (single numa node design, configuration auto)