New pve cluster advice needed, zfs or ceph


May 8, 2023
Hi all,

So I am setting up a new pve cluster. I have a few questions I'm trying to wrap my head around if anyone doesn't mind helping.

Cluster will be 4 or 5 large x86 boxes with varying configurations and 4 mini PCs

Large boxes are all backed by a 10gb links. Mini PCs are on a 1gb network. All nodes have seperate management and data NICs.

Boxes are:

2950x w 128gb ram
5900x w 128gb ram
I5 8400 w 48gb ram
I5 4590 w 32gb ram
5600g w 64gb ram

3x Mini PC i5 6500t w 16gb ram
1x mini PC i7 6700t w 32gb ram
Nuc i7 6770 w 32gb ram

I need to decide on a few things

A) install pve on nvme or sata zfs mirror? I noticed there was almost no performance benefit when using nvme with zfs and running fio 4k. I plan to pass though either the nvme drives or the SATA controller to a truenas VM

B) ceph or zfs/truenas? I have the following disks available:
2 x 2tb SATA
2 x 4tb SATA
4 x 6tg SATA
2 x 8tb SATA
2 x 1tb SATA SSD (MX500)
A bunch of 400gb enterprise ssds
A bunch of 500gb consumer ssds and nvmes

Option 1 pile all drives into on virtualised truenas box and share to the pve cluster with NFS
Option 2 use ceph

Would you suggest a centralised Nas or a ceph cluster for such a setup?
I understand you can add a disk at a time to the ceph cluster. Is that correct?
What happens to performance in such a mixed cluster?
How is usable storage calculated? I have read about max safe thresholds etc but can't quite understand what they mean.
How do ceph failure scenarios work in a mixed disk size cluster? Say the largest disk or node fails, does the cluster account for this when presenting usable storage?
Are 3 replicas a hard requirement with ceph in this scenario? What would happen in a failure I'd using only 2
If each node has two NICs would management + vms exist on one and storage on the other or should storage and VM management be isolate from management.

I will be running media streaming, vms and a bunch of lab stuff on this cluster (nested Hyper-V, esxi, nutanix)

Thanks in advance.
Would you suggest a centralised Nas or a ceph cluster for such a setup?
Is HA a requirement? If so, then CEPH, otherwise ZFS-over-iSCSI SAN is also a workable and desireable solution. You always have to imagine what can fail and everything still works.

install pve on nvme or sata zfs mirror?
if the nvme is also mirrored, then it does not matter. In general a mirrored harddisk would be enough. You don't need flash storage for your PVE boot volume.


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!