Best Setup for ZFS

[JohnD]

Hello everybody,

i am currenlty planning a new Proxmox Host and i am unsure about the best setup.

I have an 8 Core AMD Ryzen System (3700x) with 64 GB RAM (i could upgrade it 128 GB when needed).
It will run about 4-5 Debian based and 4-5 Ubuntu based virtual machines and containers.

I can install up to 4 NVMe or SATA based SSDs but i am very unsure about the best solution. 2x 1TB NVMe comes with the server and not changeable. I can either add 2 additonal 1 TB NVMe disks or go with 2x 2 TB Sata SSDs. I need around 2 TB usable storage for my VMs and containers. More would be better.

1) Is it a good idea or even possible to use only one ZFS pool for the whole system (Proxmox and VMs, e.g. 4 whole disks in one mirror or z1 setup). Do i need a separated partition to boot from?

2)Is there any performance difference between using the whole Disk for an ZFS and using just a partition?

3) Should i put the Swap volumes of the proxmoxhost on ZFS? What about the swap volumes of the VMs?

I am currently thinking about two different hardware options with different possible configurations:

a) Use the two 1TB NVMe drives with MDADM and ext4 for the proxmox host/swap, for swap volumes of the VMs and maybe as Slog ZIL devices for the the Storage Pool. Create a mirrored ZFS storage Pool with 2x 2 TB Sata SSDs.

b) Use 4x 1 TB NVMe drives in one large mirrored or Z1 pool (if possible). Alternatively create two partitions per NVMe disk (32 GB, rest) and use the first two 32 GB partitions in a mdadm mirror for the host the second two for Swap. The 4 Partitions with the majority of the capacity go to one mirrored or Z1 ZFS pool.

Which setup would you recommend and why. Any help is appreciated. I will provide some benchmarks of the final setup if anyone is interested.

Thank you!

 

[aaron]

Beforehand a few hints:
Install ProxmoxVE v6 or higher in UEFI mode. This will make it use systemd-boot as bootloader and store the kernel image on the EFI service partition (ESP). If you install it in BIOS mode GRUB will be used as bootloader and the boot volume will be on the ZPool. This will limit you in terms of features you can use in the zpool (for example encryption) because GRUB only supports a very limited subset of ZFS features.

1) Is it a good idea or even possible to use only one ZFS pool for the whole system (Proxmox and VMs, e.g. 4 whole disks in one mirror or z1 setup). Do i need a separated partition to boot from?

It is possible. For 4 Disks either Raid10 or RaidZ1 are a sensible choice.

2)Is there any performance difference between using the whole Disk for an ZFS and using just a partition?

Performance-wise there should be no difference. One reason to use a partition is so it can be a bit smaller than the disk. This helps in case a replacement is needed that is not exactly the same size.

3) Should i put the Swap volumes of the proxmoxhost on ZFS? What about the swap volumes of the VMs?

We do not use swap on ZFS anymore because even with enough RAM and swappiness set to 0 swap space will still be used, resulting in poorer performance.

Swap of the VMs will be inside their virtual disks.

I am currently thinking about two different hardware options with different possible configurations:

Option a is not supported. ProxmoxVE does not come with MDADM by itself and we do not support it. A dedicated SLOG ZIL device does not make a lot of sense if only have fast SSDs. It can benefit to use a small SSD for it if the main storage consists of slow spinning HDDs.

I would recommend to use the installer and depending on your needs a RAID10, or RAID-Z1/2 variants.

RAID10 is faster but only the right 2 disks can fail.
RAID-Z2 offers the same space as RAID10 with 4 disks but any 2 disks can fail.
RAID-Z1 will give you more space but only 1 disk can fail.