Ceph how to ? scenario

[txsastre]

hi there.

I read everywhere about Ceph, I understand what's the meaning of it, but I don't know how to use it.

for example, I have 2 CPD, 3 servers and a SAN storage in each. and also both CPD are connected with 10GbE

could I use ceph in this scenario ?

if not, could you tell me a typical scenario of ceph ?

appreciate it, thank you very much.

 

[sb-jw]

CEPH is a distributed storage, you can use it with a single node and one disk, this is not recommend, but it works.

If you want to use it in production, i would recommend to use it with a minimum of 3 nodes with 1 Disk per Node. The crush rule should be distribute the data per node and you should use an replication of 3.
If you have 1TB disk per Node, you can use around 850 G overall. There are some limitations like Nearfull and Full, which your are not try to hit, it can bring your cluster to stop serving I/O. But you have to keep in mind, if a disk or server failed completely and you have to redistribute the data to the last existing 2 drives, you are not able to store 850 G on these drives.

Scale the cluster, pools, disks etc correctly is much more complicated then my example. You should setup a CEPH Cluster and try to work with this to get an better view of it all.

Keep in mind, the performance of CEPH is not similar to local disks, to get an good value you should use 10g Network and Flash only, no RAID controller only real HBA.

 

[Alwin]

And see below link, if you're looking for instructions on how-to do it on Proxmox VE.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html

 

[txsastre]

thanks for your answers.
I now the fundamentals of Cehp but may be I have not correctly made my questions.

In my scenario 2 CPD, 3 servers with SAN storage at 10 Gb/s each.

how should I use ceph ? as I have 1 SAN and 3 servers and in another building another san and 3 more servers, connected this buildings by a 10Gb/s fibber.

so, how should I configure everything ? also, If CPD 1 gets down... I will get everyting working on CDP 2 ?

or should I contemplate a simple replication between them ?

 

[Alwin]

In my scenario 2 CPD, 3 servers with SAN storage at 10 Gb/s each.

Use the SAN as a separate means of storage, not with Ceph. See my previous link.

how should I use ceph ? as I have 1 SAN and 3 servers and in another building another san and 3 more servers, connected this buildings by a 10Gb/s fibber.

Don't connect them as one, use rbd-mirror for offsite replication.
https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

so, how should I configure everything ? also, If CPD 1 gets down... I will get everyting working on CDP 2 ?

That's the idea behind a Proxmox VE + Ceph hyper-converged cluster.

Please read up on ceph and its capabilities, while the post is a little older, the links should still be valid.
https://forum.proxmox.com/threads/ceph-raw-usage-grows-by-itself.38395/#post-189842

Also a little more information on what Proxmox VE can do.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_introduction