Proxmox Ceph Cluster Configuration

[alangv]

Hi all,
I read a lot of documentation about Proxmox Ceph Cluster but I can't figure out the best solution for my case.
I would like to configure a little 3 node cluster using Proxmox and Ceph with the ability of add in the future more nodes (so more CPU and RAM to the cluster) if needed or more Storage (so more TB for File Storage).

Reading the documentation seems that is better to have similar SSD or HDD on each node, so I can start with 1 SSD for the system (128 GB) and 1 SSD for data (2 TB).

Now I have 2 question:
If I need more Storage Volume I need to add another 2 TB SSD on each node (6 TB) to add only 2 TB ? And If I have 10 nodes I need to add 2 SSD on each node (20 TB) to add 2 TB to the cluster? And what id I have 100 nodes? It seems a little bit redundant!

It is a good practice in proxmox ceph cluster to have an external shared storage like a NAS or similar that can be easily scalable and can use RAID (5, 6, 10) ? There are some cons?

Which is the best configuration recommended?

 

[ilia987]

if you are on replication 3, each data is copied 3 times + some overhead
for each 1 TB of RAW data you will get approx 300MB usable data
adding 10 hdds of 2TB Each will give you approx 6TB usable data

 

[alangv]

Ah ok, so during the configuration of CEPH I need to set the number of replicas (3 recommended) then I can calculate the space that approximately is:

( Number of nodes * HDD/SSD capacity ) / Number of replicas

Anyway is a good idea setting up a shared storage or is not really useful?

Thanks!

 

[ilia987]

Ah ok, so during the configuration of CEPH I need to set the number of replicas (3 recommended) then I can calculate the space that approximately is:

( Number of nodes * HDD/SSD capacity ) / Number of replicas

Anyway is a good idea setting up a shared storage or is not really useful?

Thanks!

yes but also take approx 10% of the total value (for journaling and such)

we moved to ceph from freenass (was good but it is limited in performance and capacity as single node.), and we are mainly need to as READ to provide enoth data to our claster, so ceph was ideal, i am still figuring out some stuff regarding it(not as strait forward as nfs ) but it worth the effort (i hope )

 

[finish06]

Look into erasure coding: https://docs.ceph.com/docs/master/rados/operations/erasure-code/. I use it on my cephFS implementation on a 3 node, 4 OSDs per node cluster. It works well. I would not use erasure coding for VM storage.