New Proxmox Cluster with Ceph

[parker0909]

Hello All,

I am planning to setup the new Proxmox Cluster with 3 hosts with Ceph Storage. Each host will be installed 4x 2.5" SATA SSD and 1 x NvME SSD.
I have a question about it is possible to setup any read/write cache in Ceph, so i can assign Nvme SSD to cache role and another 2.5" SSD to OSD. May i know it is possible to support this design for Ceph?

Thank you
Parker

 

[Lukas Moravek]

Hello @parker0909

in each OSD you can use separate device for "Block and WAL"

https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_osd_create

But I never do that in configuration as you described, as this is single point of failure scenario (1 x NvME SSD shared for OSDs), not even mentioning if it will help to performance (same data to be written in to one device for all OSDs)

Lukas

 

[parker0909]

Thank you

 

[parker0909]

Hello,

I have one further question. May i know one NvME SSD can share with different OSDs? For example, i have using 4 x 2.5" SSD ( OSDs) and 1 x NvMe SSD(Block.db and block.wal).

Thank you

Parker

 

[shbaek]

Hello,

I have one further question. May i know one NvME SSD can share with different OSDs? For example, i have using 4 x 2.5" SSD ( OSDs) and 1 x NvMe SSD(Block.db and block.wal).

Thank you

Parker

Yes, an NVMe SSD can be shared among multiple OSDs by using it for Block.db and Block.wal, but this setup introduces a single point of failure—if the NVMe fails, all associated OSDs will go down.

For better redundancy, consider using multiple NVMe devices or mirroring the WAL/DB across different drives.

 

[Lukas Moravek]

Hi @parker0909,

This is same scenario I described in my first post. Still same problem with single point of failure and also I am afraid that you will not gain so much performance, as the data will be under write / read operation four times (four OSDs) on one single device NVMe in same time. Which might be bottleneck.

You have options:

1) Change SSDs for NVMe drives to gain super speed.
2) Use 4x small NVMe for "Wal and DB" for 4 OSDs (one NVMe for one OSD).
3) Use current the NVMe for critical fast operation with ZFS and use replication in cluster. No additional costs.

L

 

[ness1602]

This is okay setup for me, usually we don't have more than 4xdisks per one nvme for caching . If you need more than that we usually recommend adding more nvme caching drives. Yes, if this nvme dies, all osds which are cached on it die, but this is acceptable in CEPH.