OS on ZFS and VMs on CEPH

[stpmuskego]

I'm looking to move from vmware/vsphere to proxmox and am wondering, would it make sense to put the OS on a mirrored zfs pool and then have a separate drive used for vm's that would be setup with ceph? for the ceph interlinks i would be looking at a 25gb mesh network since we only have 3 servers. Additionally, for zfs i would install with a mirror, but for ceph that doesn't look like it is necessary since things are mirrored on our other nodes right?

also, looking at enterprise nvme's, does anyone have any good recommendations? during the conversion from vmware to proxmox, we are moving from some very old HP proliants with 10k HDD's to in-house built systems with nvme SSD's and i want to make sure that what i get will make sense.

thanks!

 

[VictorSTS]

At least 3 servers with OS on ZFS mirror + some drives for Ceph is a typical setup for a minimum PVE HCI cluster. So yes, that will work. Make sure to use redundant corosync links even when using Mesh. Also, remember that in a 3 node Ceph cluster if one node fails Ceph can't recover redundancy (the "third copy" of data) until that node is up again.

For the drives use enterprise ones with PLP and at least 1DWPD for typical workloads and 4k sectors. Most will work perfectly fine and there won't really be much of a performace difference as the bottleneck is usually the network, not the drives.

 

[stpmuskego]

for the corosync links, is there a primary and backup or is it load balanced? i'm wondering if i can use my 10g primary network as a corosync fallback if the 25g mesh falls down.

 

[VictorSTS]

Can have up to 10, used by priorities from 0 to 9. Review cluster requirements in [1] or practice beforehand using virtual machines to avoid errors.

[1] https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy