3 nodes cluster with local shared storage and HA

[Moatasem]

Hello

I’ve 3 servers each with 2 x 1TB SSD and 1 x 4TB HDD.

I want to build a Proxmox VE cluster with HA utilising the storage on each node.

What is the best way to create a shared storage from the 3 nodes and present it Proxmox?

Regards
Moatasem

 

[bbgeek17]

Since you are limited to local storage on each node, your only two "supported" options from this page: https://pve.proxmox.com/wiki/Storage
Are: GlusterFS and Ceph.

Ceph has a much more robust support in PVE: https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jdancer]

This.

I run a full-mesh 3-node Ceph cluster on 12-year old servers.

Since it's running Debian, I have zero issues.

 

[Moatasem]

Thanks guys for your replays, Ceph with full-mesh works perfect.

 

[jdancer]

Thanks guys for your replays, Ceph with full-mesh works perfect.

Since it seems you are going with Ceph, I suggest the following optimizations to get better IOPS:

1. Set VM cache to none
2. VirtIO SCSI Single controller with discard and IO thread enabled
3. On Linux VMs, set the IO scheduler to none or noop
4. Turn on write-cache enable (WCE) on SAS drives with "sdparm --set WCE=1 --save /dev/sd[x]". SATA drives usually have their write cache enabled already but can run "hdparm -W 1 /dev/sd[x]"

 

[Moatasem]

Thank guys much apricate all your help.
One more question, the Ceph OSDs is up and running fine but only one manager and one monitor are running.


The others fail to start, is this normal or something is wrong.

Regards
Moatasem

 

[Moatasem]

I managed to get the managers going.
But I can't get the monitors started.. If I try to destroy to recreate, I get "no such monitor id"

 

[Moatasem]

All sorted..