Clarification about Ceph suitability for a 2-node storage architecture with a witness/quorum node

[mjanati]

Hello Proxmox community,

I would like to ask for a technical clarification regarding the use of Ceph in a Proxmox VE architecture with only two storage nodes.

We are currently evaluating a virtualization infrastructure where the target architecture is based on:

• 2 Proxmox VE nodes used for virtualization and storage
• 1 additional witness/quorum node without storage OSDs
• Synchronous replication between the two storage nodes
• High availability for virtual machines
• Split-brain protection
• Continuity of service in case one storage node fails

The proposed storage solution is Ceph. However, I would like to confirm whether Ceph is really suitable for this kind of production architecture.

My understanding is that Ceph generally requires at least three storage nodes with OSDs in order to provide proper high availability and data redundancy. With only two storage nodes, using a replication size of 2 may lead to problems when one node fails, because Ceph may not be able to continue write operations safely while maintaining the required number of replicas. On the other hand, reducing min_size to 1 may allow degraded operation, but it also means that data may temporarily exist with only one valid copy, which could increase the risk of data loss.

Therefore, I would appreciate clarification on the following points:

1. Is Ceph recommended for production with only two storage nodes and one witness/quorum node without OSDs?
2. Can a witness/quorum node without storage OSDs replace a third Ceph storage node from a data availability point of view?
3. In case one of the two storage nodes fails, can Ceph continue to provide safe write operations while still maintaining proper data redundancy?
4. Is the recommended Proxmox/Ceph architecture for production high availability based on at least three full storage nodes with OSDs?
5. Would you consider a 2-node Ceph storage architecture suitable for a production virtualization platform requiring HA, or should another SDS/vSAN solution designed for 2-node deployments be considered?

The goal of my question is to assess whether Ceph is technically compliant with a target architecture limited to two storage nodes plus a witness/quorum node.

Thank you in advance for your clarification and recommendations.

Best regards,

 

[UdoB]

Two years ago I tried to use Ceph in my homelab. My experience: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

I am not a Ceph user now. But I would not start with fewer than five active nodes.

Just my irrelevant two €¢...

 

[MarkusKo]

No, Ceph needs at least 3 nodes, for production 5 nodes or more (odd number) are recommended: PVE docs

 

[Johannes S]

Don't. Ceph needs at least three nodes unless you change the defaults ( and doing this is a bad idea, see: https://forum.proxmox.com/threads/ceph-pool-size-is-2-1-really-a-bad-idea.68939/ )

The recommended requirements in the documentation are here, the network and storage part are especially important:
https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster

Three nodes can be ok if you are aware of the involved tradeoff compared to more nodes ( like that you can only loose one node and still have your vms running. This is ok for taking down a node for maintenance or if you can live with the risc that another node might fail and thus breaking things) see the Proxmox wiki for one example of a three-node cluster with a mesh network: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server