ZFS RAID 1 configuration with SSDs and HDDs: Help needed!

[MatzeK1]

Hello Proxmox community,

I hope someone can help me with my storage configuration or understanding issue. I'm either making a mistake or making a mistake. I would be very grateful for your tips and assistance, including any experience with storage sizes.

I have an Intel N100 system with 16GB DDR5 RAM, 2x 500GB M.2 NVME SSDs, and 2x 8TB NAS HDDs, and I would like to install Proxmox and Encrypted NVR on it.

My goal is to create a secure and efficient storage solution for my Proxmox system. Here is my concept:
A ZFS RAID1 configuration (mirror) consisting of the two M.2 NVME SSDs (nvme pool) with a total size of 500 GB.
A ZFS RAID1 configuration (mirror) consisting of the two NAS HDDs (hdd pool) with a total size of 8 TB. I planned the following partitioning and file systems:

For the nvme pool (SSD1 + SSD2 = 500 GB ZFS Raid1):
One partition for the Proxmox operating system (named: proxmox-system) with a size of ? GB.
One partition for the ZFS Metadata Special Device (Z) (named: zfs-metadata) with a size of ? GB to improve performance and reduce latency.
One partition for the cache memory (named: nvme-cache) with a size of ? GB to increase read and write speeds and reduce HDD utilization.

For the HDD pool (HDD1 + HDD2 = 8 TB ZFS RAID 1):
One partition for the encrypted NVR recordings (named scrypted-recordings) with a minimum size of 6 TB. (For the encrypted NVR recordings. The documentation states that it's best to pass a dedicated hard drive through to Scrypted.)
One partition for the backups and snapshots (named backup) with a size of approximately 2 TB.
One partition for the Proxmox database (named proxmox-db) with a size of ? TB, which is placed on the HDDs to protect the SSDs and reduce the number of read and write accesses to them.

The ZFS Metadata Special Device (Z) and the NVMe cache are located on the SSDs to improve system performance and reliability.

Unfortunately, I'm not making any progress implementing this concept. I've successfully installed Proxmox on the ZFS RAID 1 (2x SSDs). The ZFS pool is now called rpool because I created the ZFS mirror during the Proxmox installation, but that's not a problem. However, I'm stuck setting up the ZFS partitions. When I create hdds-pool as a ZFS RAID 1, I can't create a directory on it. When adding a new directory in Proxmox, it says that all disks will be used. I would appreciate it if someone could help me implement my concept or suggest alternative solutions.

Thank you in advance for your help!

Best regards,
Matthias

 

[mgabriel]

For the nvme pool (SSD1 + SSD2 = 500 GB ZFS Raid1):
One partition for the Proxmox operating system (named: proxmox-system) with a size of ? GB.
One partition for the ZFS Metadata Special Device (Z) (named: zfs-metadata) with a size of ? GB to improve performance and reduce latency.
One partition for the cache memory (named: nvme-cache) with a size of ? GB to increase read and write speeds and reduce HDD utilization.

First, I assume the zfs special device and cache device should speed up the hdd pool, correct? It doesn't make sense to have a special device or cache pool for the nvme devices.

Second, do you want to passthrough the hdd devices or not? You wrote that passthrough is recommended by the software you use, but if the devices are managed within your VM, your VM would need to take care of that ZFS also. If you want to manage your ZFS on the Proxmox VE Node, you may want to create VM-images as datastores for your software.

1. Don't use a cache device for your workload.
2. The special device should be 2% of your slow backend devices (2% of 8 TB = 160 GB).

I'd recommend that you let the Installer do it's magic for you. On page 2 -> Options -> ZFS -> Advanced Options set the hdsize to 500 GB - 160 GB = 340 GB and you'll be fine. That 160 GB of space is unused. You can create a partition there and use it as a special device mirror for your hdds. You can use the root ZFS also to host VMs or containers within that remaining 340 GB.

For the HDD pool (HDD1 + HDD2 = 8 TB ZFS RAID 1):
One partition for the encrypted NVR recordings (named scrypted-recordings) with a minimum size of 6 TB. (For the encrypted NVR recordings. The documentation states that it's best to pass a dedicated hard drive through to Scrypted.)
One partition for the backups and snapshots (named backup) with a size of approximately 2 TB.
One partition for the Proxmox database (named proxmox-db) with a size of ? TB, which is placed on the HDDs to protect the SSDs and reduce the number of read and write accesses to them.

What do you mean with proxmox-db? If you mean the Proxmox Cluster File System config.db, just leave it on the root partition. It does get some reads and writes, but that should not be a problem, and you don't want them to be on the slow devices.