5 Node Proxmox Cluster, 4 Node CEPH

[johnoatwork]

Hi All,

I've read a few posts about running an even number of nodes in a Proxmox cluster and issues with split brain.

I'm building a 5 node Proxmox cluster where 4 of the nodes have SSDs and the 5th node has 3.5" HDDs for bulk storage.

So, the 4 x SSD nodes will run CEPH and all 5 nodes will run Proxmox.

I'm planning to install CEPH and a monitor on the 5th node but it won't have any OSDs.

Is this likely to be an issue for CEPH? It seems to me if I have a monitor on at least 3 nodes it should be OK.

Thank you.

 

[gurubert]

if I have a monitor on at least 3 nodes it should be OK.

Yes. Make sure to have an odd number of MONs for the Ceph cluster.

 

[johnoatwork]

Yes. Make sure to have an odd number of MONs for the Ceph cluster.

That makes sense. Thanks for the feedback gurubert.

 

[jsterr]

Hi All,

I've read a few posts about running an even number of nodes in a Proxmox cluster and issues with split brain.

I'm building a 5 node Proxmox cluster where 4 of the nodes have SSDs and the 5th node has 3.5" HDDs for bulk storage.

So, the 4 x SSD nodes will run CEPH and all 5 nodes will run Proxmox.

I'm planning to install CEPH and a monitor on the 5th node but it won't have any OSDs.

Is this likely to be an issue for CEPH? It seems to me if I have a monitor on at least 3 nodes it should be OK.

Thank you.

What are you using storage-wise for the hdds?

 

[johnoatwork]

What are you using storage-wise for the hdds?

I've got half a dozen Seagate IronWolf HDDs in an R730xd. They IronWolf's aren't real quick but they have been pretty reliable so far. The server takes 12 x 3.5" drives and I'll likely fill the remaining bays with some Dell 3TB SAS drives that are laying around.

 

[jsterr]

I've got half a dozen Seagate IronWolf HDDs in an R730xd. They IronWolf's aren't real quick but they have been pretty reliable so far. The server takes 12 x 3.5" drives and I'll likely fill the remaining bays with some Dell 3TB SAS drives that are laying around.

I meant, what storage-software are u using? You will go for zfs on that?

 

[johnoatwork]

I meant, what storage-software are u using? You will go for zfs on that?

Ah yes, will run zfs on these.

 

[jsterr]

Ah yes, will run zfs on these.

Interesting. Dont forget to make sure that when enabling HA, VMs that use HDD in vm-config, should be restricted to fifth node only, as they can not be started on the other 4 nodes. Or simply dont enable ha for vms that use hdd

 

[johnoatwork]

Thanks for the tip, but I won't be running any VMs on node 5. All the VM's will be (are) on CEPH SSD's on nodes 1 - 4.

The HDD storage is primarily for Proxmox Backup Server datasets, but will also provide shared HDD storage over NFS for VM bulk data disks. Using zfs send to sync the pools to a second bulk storage server for redundancy. I do love zfs