Hi all,
So I am setting up a new pve cluster. I have a few questions I'm trying to wrap my head around if anyone doesn't mind helping.
Cluster will be 4 or 5 large x86 boxes with varying configurations and 4 mini PCs
Large boxes are all backed by a 10gb links. Mini PCs are on a 1gb network. All nodes have seperate management and data NICs.
Boxes are:
2950x w 128gb ram
5900x w 128gb ram
I5 8400 w 48gb ram
I5 4590 w 32gb ram
5600g w 64gb ram
3x Mini PC i5 6500t w 16gb ram
1x mini PC i7 6700t w 32gb ram
Nuc i7 6770 w 32gb ram
I need to decide on a few things
A) install pve on nvme or sata zfs mirror? I noticed there was almost no performance benefit when using nvme with zfs and running fio 4k. I plan to pass though either the nvme drives or the SATA controller to a truenas VM
B) ceph or zfs/truenas? I have the following disks available:
2 x 2tb SATA
2 x 4tb SATA
4 x 6tg SATA
2 x 8tb SATA
2 x 1tb SATA SSD (MX500)
A bunch of 400gb enterprise ssds
A bunch of 500gb consumer ssds and nvmes
Option 1 pile all drives into on virtualised truenas box and share to the pve cluster with NFS
Option 2 use ceph
Would you suggest a centralised Nas or a ceph cluster for such a setup?
I understand you can add a disk at a time to the ceph cluster. Is that correct?
What happens to performance in such a mixed cluster?
How is usable storage calculated? I have read about max safe thresholds etc but can't quite understand what they mean.
How do ceph failure scenarios work in a mixed disk size cluster? Say the largest disk or node fails, does the cluster account for this when presenting usable storage?
Are 3 replicas a hard requirement with ceph in this scenario? What would happen in a failure I'd using only 2
If each node has two NICs would management + vms exist on one and storage on the other or should storage and VM management be isolate from management.
I will be running media streaming, vms and a bunch of lab stuff on this cluster (nested Hyper-V, esxi, nutanix)
Thanks in advance.
So I am setting up a new pve cluster. I have a few questions I'm trying to wrap my head around if anyone doesn't mind helping.
Cluster will be 4 or 5 large x86 boxes with varying configurations and 4 mini PCs
Large boxes are all backed by a 10gb links. Mini PCs are on a 1gb network. All nodes have seperate management and data NICs.
Boxes are:
2950x w 128gb ram
5900x w 128gb ram
I5 8400 w 48gb ram
I5 4590 w 32gb ram
5600g w 64gb ram
3x Mini PC i5 6500t w 16gb ram
1x mini PC i7 6700t w 32gb ram
Nuc i7 6770 w 32gb ram
I need to decide on a few things
A) install pve on nvme or sata zfs mirror? I noticed there was almost no performance benefit when using nvme with zfs and running fio 4k. I plan to pass though either the nvme drives or the SATA controller to a truenas VM
B) ceph or zfs/truenas? I have the following disks available:
2 x 2tb SATA
2 x 4tb SATA
4 x 6tg SATA
2 x 8tb SATA
2 x 1tb SATA SSD (MX500)
A bunch of 400gb enterprise ssds
A bunch of 500gb consumer ssds and nvmes
Option 1 pile all drives into on virtualised truenas box and share to the pve cluster with NFS
Option 2 use ceph
Would you suggest a centralised Nas or a ceph cluster for such a setup?
I understand you can add a disk at a time to the ceph cluster. Is that correct?
What happens to performance in such a mixed cluster?
How is usable storage calculated? I have read about max safe thresholds etc but can't quite understand what they mean.
How do ceph failure scenarios work in a mixed disk size cluster? Say the largest disk or node fails, does the cluster account for this when presenting usable storage?
Are 3 replicas a hard requirement with ceph in this scenario? What would happen in a failure I'd using only 2
If each node has two NICs would management + vms exist on one and storage on the other or should storage and VM management be isolate from management.
I will be running media streaming, vms and a bunch of lab stuff on this cluster (nested Hyper-V, esxi, nutanix)
Thanks in advance.
